Technical Documentation: google/syzkaller
ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference
Overview (`README.md`)
syzkaller - kernel fuzzer
syzkaller ([siːzˈkɔːlə]) is an unsupervised coverage-guided kernel fuzzer.
Supported OSes: FreeBSD, Fuchsia, gVisor, Linux, NetBSD, OpenBSD, Windows.
Mailing list: [email protected] (join on web or by email).
Found bugs: Darwin/XNU, FreeBSD, Linux, NetBSD, OpenBSD, Windows.
Documentation
Initially, syzkaller was developed with Linux kernel fuzzing in mind, but now
it's being extended to support other OS kernels as well.
Most of the documentation at this moment is related to the Linux kernel.
For other OS kernels check:
Darwin/XNU,
FreeBSD,
Fuchsia,
NetBSD,
OpenBSD,
Starnix,
Windows,
gVisor,
Akaros.
- How to install syzkaller
- How to use syzkaller
- How syzkaller works
- How to install syzbot
- How to contribute to syzkaller
- How to report Linux kernel bugs
- Tech talks and articles
- Research work based on syzkaller
Disclaimer
This is not an official Google product.
Overview (`syz-agent/README.md`)
syz-agent
syz-agent is an AI agent running as part of the continuous fuzzing
infrastructure. It continuously polls the syzbot web dashboard for tasks and
executes agentic workflows.
Running Locally with Docker
You can easily run syz-agent locally without Kubernetes. This is the best
approach for testing and development.
Build the Docker image from the root of the syzkaller repository:
bashmake -C syz-agent containerNote: The default image name is
local/syz-agent:latest. You can override it by passingIMAGE_NAME=your-image-name IMAGE_TAG=your-tag.Prepare a configuration file (e.g.
config.json):json{ "http": ":8080", "dashboard_client": "my-local-agent", "dashboard_addr": "https://syzkaller.appspot.com", "dashboard_key": "YOUR_KEY", "targets": { "linux/amd64": { "image": "/disk-images/buildroot_amd64", "kernel_config": "/kernel-configs/upstream-apparmor-kasan.config", "type": "qemu", "vm": { "cpu": 2, "mem": 2048, "cmdline": "root=/dev/sda1" } } }, "cache_size": 107374182400 }Run the container, mounting your configuration file:
bashdocker run -it --rm \ -p 8080:8080 \ --privileged \ -e GOOGLE_API_KEY=$GOOGLE_API_KEY \ -v $(pwd)/config.json:/etc/syz-agent/config.json:ro \ local/syz-agent:latest \ -name=$MY_UNIQUE_AGENT_NAME \ -config=/etc/syz-agent/config.json
Note: pkg/updater is bypassed inside Docker because the
-syzkaller=/syzkaller flag is passed via run.sh which serves as the Dockerfile's
ENTRYPOINT. syz-agent will use the pre-built binaries inside the container.
Running on Kubernetes
syz-agent can be deployed as a StatefulSet on Kubernetes (Minikube or GKE) for persistent identity.
Local Development (Minikube)
To deploy to a standard minikube cluster:
# 1. Build and load image into Minikube
make container
minikube image load local/syz-agent:latest
# 2. Deploy (automatically binds host's GOOGLE_API_KEY and DASHBOARD_KEY)
GOOGLE_API_KEY=xxx DASHBOARD_KEY=yyy make k8s-minikube | kubectl apply -f -Overview (`syz-cluster/README.md`)
syz-cluster is a distributed patch series fuzzing that is relying on syzkaller.
It's deployed at https://ci.syzbot.org.
Overview
The system is to be deployed on a K8S cluster. The main services are:
dashboard: the web interface, read-only.controller: manages the state of the system, provides
API for other components, schedules fuzzing sessions.series-tracker: polls LKML git archives for the new series.reporter-server: generates new reports, provides API for actual reporter
implementations.email-reporter: sends reports over email, handles incoming email commands.
The actual patch processing is orchestrated by Argo Workflows: see the
template. It relies on the following processing
steps:
workflow/triageworkflow/buildworkflow/bootworkflow/fuzz
Triage and build steps need the actual kernel checkouts. The base kernel repo is
hosted on a shared network disk and is regularly updated by the scripts inkernel-disk.
The system can be deployed in multiple environments, which is achieved with the
help of Kustomize. Depending on the actual deployment target, different pieces of
configuration are applied:
overlays/local/minikube: the local dev environment.overlays/gke/staging: the staging prod environment.overlays/gke/prod: https://ci.syzbot.org.
global-config.yaml is the main configuration file of the system - it
determines the mailing lists to poll, configures resource usage and the actual
reporting.
Local installation steps
- Install and start minikube: https://minikube.sigs.k8s.io/docs/start/
$ minikube start --cni=cilium --container-runtime=docker --memory=128g--cni=cilium enables the use of a more advanced Network plugin that supports
the emulation of network policies.
--container-runtime=docker is needed for minikube docker-env (the default
is containerd since minikube v1.39).
--memory=128g limits the memory of the whole cluster. Fuzzing may use a lot
of it, and on OOM random pods get killed (e.g. the Spanner emulator).
- Build all docker containers (might take a while):
$ eval $(minikube docker-env)
$ make all-containers- Deploy local infrastructure (Argo, Spanner, Service Accounts):
$ kubectl create namespace argo
$ make k8s-config-local-infra
$ make migrate-local- Deploy the rest of the cluster:
$ make k8s-config-dev | kubectl apply -f -- Pre-fetch the kernel git repository:
$ make fetch-kernels-once.yaml | kubectl create -f -Note that actual series processing won't start until the job created in (5)
finishes.
Developmental tips
- Install Argo Workflows client: https://github.com/argoproj/argo-workflows/releases
Then you can use the argo tool like this:
$ argo list- Forward the dashboard port:
$ kubectl port-forward service/web-dashboard-service --address 0.0.0.0 50123:80Overview (`dashboard/app/README.md`)
Dashboard
dashboard is an App Engine app that powers syzbot.
The main deployment is at syzkaller.appspot.com.
It is so-called Standard environment Go app.
To deploy and manage the app you need to install Google Cloud SDK.
For more details about App Engine refer to the docs.
Note: The app is not stable and is not officially supported. It's here only to power the main deployment.
Here is "big" picture of a possible setup:
Note: the vector source is here;
to update: make a copy of the source, edit, download a png, update the png and include a link to your vector copy into the PR.
To deploy the app you need to add a .go file with production config. The config specifies kernel namespaces,
bug reporting details, API keys, etc. Tests contain a config example, but it's not ready for
production use.
The app also needs one or more syz-ci instances running elsewhere. The syz-ci instances
do the actual fuzzing, bisection, patch testing, etc.
The app can be deployed by gcloud app deploy ./dashboard/app/app.yaml.
The following optional flags are available:
- "--no-promote" to test the app firs and migrate the traffic to it later.
- "--verbosity=info" to see what files are going to be deployed.
The app tests can be run with:
go test github.com/google/syzkaller/dashboard/appDuring development it's handy to use -short flag to not run the longest tests.
If any of the tests fail, use -v flag to see log of what happens and -run flag
to run a single test, e.g.:
go test -short -v -run=TestEmailReport github.com/google/syzkaller/dashboard/appLocal Test Deployment
It's possible to run the dashboard locally for testing purposes.
However, note that it won't have any data, so you would need to connect syz-ci
instances so that they populate database with some bugs.
First, you need to install Google Cloud SDK (gcloud command, and required components,
this is one time step).
Then, create emulator config (this is one time step):
gcloud config configurations create emulator
gcloud config set auth/disable_credentials true
gcloud config set project syzkaller
gcloud config set api_endpoint_overrides/spanner http://localhost:9020/Then, start local spanner emulator in one console:
gcloud emulators spanner start
gcloud spanner instances create syzbot --config=emulator --nodes=1
gcloud spanner databases create ai --instance=syzbotThen, initialize the schema from another console:
for SQL in dashboard/app//aidb/migrations/*.up.sql; do \
gcloud spanner databases ddl update ai \
--instance=syzbot --ddl-file ${SQL}; doneFinally, start the web server:
SPANNER_EMULATOR_HOST="localhost:9010" \
GOOGLE_CLOUD_SPANNER_MULTIPLEXED_SESSIONS=false \
google-cloud-sdk/bin/dev_appserver.py --application=syzkaller \
--host=0.0.0.0 --enable_host_checking=false dashboard/app/Overview (`docs/akaros/README.md`)
Akaros support
Akaros support was removed.
You can find it in git history, if needed.
Found bugs are here,
here
and here.
Overview (`docs/darwin/README.md`)
Darwin/XNU
It turned out to be unreasonably hard to bootstrap a VM image usable for fuzzing from the XNU source drops without using any of the proprietary kernel extensions shipped with macOS. This guide is therefore based on a normal macOS installation. Unfortunately Apples macOS EULA makes this unsuitable for fuzzing XNU on Google Cloud Platform.
Prepare a macOS installation disk image
Nowadays Apple mainly distributes macOS updates via the Mac App Store. This will however only give us the latest builds. Luckily the Munki people are kind enough to maintain a script, allowing us to fetch a macOS build of our choice from the shell.
We'll need a macOS build that has a kernel version close to the one we will be building. You can see Apples most recent source drops on this page. At the time of writing the most recent version is macOS 11.5 containing kernel xnu-7195.141.2. The Munki download script can only tell the macOS version and build number, but not the XNU version. Unfortunately you might occasionally download a matching macOS release, but the kernel you build won't boot anyway. Among many other reasons this can be caused by this mismatch in kernel and bootloader versions. Some trial and error can be involved in getting the correct build. Sometimes the correct build might no longer be available. At the time of writing 11.5 build 20G71 was available and worked with the 11.5 xnu source drop.
In the instructions below I will assume you have VMware Fusion installed on your host macOS for creating the VM disk image. This is for convenience sake as Fusion allows us to simply drag and drop in the macOS installer App we downloaded. If you want to use another tool like Qemu for this, take note of Apples own process for creating a bootable install media. I had trouble generating bootable ISOs from certain macOS builds using Apples method, hence I just always let Fusion create the installation medium for me.
Additionally the below instructions ask you to disable System Integrity Protection and Authenticated Root inside the VM. We need to do this in order to run the DIY kernel we will build in a bit. Executive Summary on these features:
- In OS X 10.11 Apple introduced System Integrity Protection, a feature that (among other things) limits even root from writing to certain critical system directories during normal operations. We need to disable it to write our kernel to disk
- In macOS 11 Apple introduced Authenticated Root. Starting with this version only a cryptographically signed read only snapshot of the root filesystem is mounted during boot. We need to disable it in order to remount a writable version and take a new snapshot to boot from later
Tl;Dr: To create the VM image:
- Clone Munkis macadmin-script repo
- Run
installinstallmacos.pyand choose a version matching the last kernel source drop - Mount and open the downloaded
Install_macOS_<version>-.dmg - Open VMware Fusion and create a new VM via the File menu
- Drag and Drop the
Install macOS <name>App from the mounted disk image into theSelect the Installation MethodFusion dialog - I suggest cranking up the VMs CPU and Memory at this point
- After picking your language in the macOS installer open
Utilities -> Terminal - Enter
csrutil disableto disable System Integrity Protection - Enter
csrutil authenticated-root disableto disable Authenticated Root - Quit the Terminal app via Cmd+Q
- Follow through with the macOS installation, creating a user called
user. If you don’t have any disk available to install to, you might need to use the Disk Utility to format the virtual disk first - Go to
System Preferences -> Software Update -> Advanced…and uncheckCheck for updates - Go to
System Preferences -> Energy Saverand checkPrevent computer from sleeping automatically when the display is off - Go to
System Preferences -> Sharingand checkRemote Loginto enable sshd - Add your ssh pubkey to both your users and roots authorized_keys files
- Optionally disable WindowServer and other non-essential services via launchd. Note that you will obviously loose the GUI:
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.WindowServer.plist
Check that everything looks alright:
Prepare a Kernel optimized for fuzzing
You might be wondering why we aren’t using one of the precompiled kernels available in Apples Kernel Development Kits. However those don’t include any kernel built with the KSANCOV feature flag. KSANCOV is Apples take on an API that allows userspace to request the kernel to start tracing which kernel code a given thread touched and exposing that information to userspace. This information is required by Syzkaller to be really effective at fuzzing.
Luckily afrojer@ is releasing semi-regularly updated instructions on building XNU from source and installing it on macOS on his blog. At the time of writing he is lagging three minor source drop versions behind. The most recent instructions are for macOS 11.2. We’ll cover some additional required changes in this text.
Building and testing a XNU useful for fuzzing:
Download a somewhat recent Xcode from Apples Xcode versions archive (Apple ID login required) to your VM. I’m using Xcode 12.5. I had issues with 12.5.1 and 13 beta 4
Open the
Xcode_<version>.xipto extract. Make a coffee ⏳Drag and Drop the extracted Xcode app into your VMs Applications folder
Start Xcode, agree to the license and quit it after it finishes installation
Create and join a directory named
kernelin the users home dir and cd into itDownloading afrojer@s Makefile:
curl https://jeremya.com/sw/Makefile.xnudeps > Makefile.xnudepsNote this file is not versioned, but always updated in place. Here is an archive link to the version at the time of writing.make -f Makefile.xnudeps macos_version=11.5 xnudepsThis will get the dependencies for building the 11.5 XNU. Check out the original blog post for details on how to fetch dependencies for a specific versioncd into
~/kernel/xnu-<version>/Apply required XNU patches manually or via
git am. Applying via git requires you to init the repo and commit all files first. That's probably a good idea anyway to track further changes you might make- The patches for
MakeInc.defandkasan.care required for the KASAN kernel to build. KASAN is short for KernelAddressSANitizer - a feature to detect a bunch of memory safety issues inside the kernel during runtime - The
ksancov.hpatch is required for building syzkallers executor. Executor is C++ and hence doesn't like the void pointer casting - Finally the
cpuid.candcpu_threads.cpatches are required for our kernel to boot on Qemu
- The patches for
Run
mountand look for the roots mount device. On my VM it looks like this/dev/disk2s5s1 on / (apfs, sealed, local, read-only, journaled). Now remember the devices name, ignoring the last sN part. So I note down/dev/disk2s5Cd into
~/kernel/xnu-<version>/and run the following, replacing<your_disk>to build and install your kernel
mkdir -p BUILD/mnt
sudo mount -o nobrowse -t apfs /dev/<your_disk> $PWD/BUILD/mnt
make SDKROOT=macosx TARGET_CONFIGS="KASAN X86_64 NONE" KSANCOV=1
kmutil create -a x86_64 -Z -n boot sys \
-B BUILD/BootKernelExtensions.kc.kasan \
-S BUILD/SystemKernelExtensions.kc.kasan \
-k BUILD/obj/kernel.kasan \
--elide-identifier com.apple.driver.AppleIntelTGLGraphicsFramebuffer
sudo ditto BUILD/BootKernelExtensions.kc.kasan "$PWD/BUILD/mnt/System/Library/KernelCollections/"
sudo ditto BUILD/SystemKernelExtensions.kc.kasan "$PWD/BUILD/mnt/System/Library/KernelCollections/"
sudo ditto BUILD/obj/kernel.kasan "$PWD/BUILD/mnt/System/Library/Kernels/"
sudo bless --folder $PWD/BUILD/mnt/System/Library/CoreServices --bootefi --create-snapshot
sudo nvram boot-args="-v kcsuffix=kasan wlan.skywalk.enable=0"After rebooting you should see your shiny new kernel when running uname -a: Darwin users-Mac.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 9 16:12:43 PDT 2021; user:xnu-7195.141.2/BUILD/obj/KASAN_X86_64 x86_64
For effective fuzzing we'll need the kernels binary, symbols and source on the host. Copy them like this:
mkdir -p ~/115/src/Users/user/kernel/ ~/115/obj
rsync -r mac:/Users/user/kernel/xnu-7195.141.2 ~/115/src/Users/user/kernel/
mv ~/115/src/Users/user/kernel/xnu-7195.141.2/BUILD/obj/KASAN_X86_64/kernel.kasan ~/115/obj/
mv ~/115/src/Users/user/kernel/xnu-7195.141.2/BUILD/obj/KASAN_X86_64/kernel.kasan.dSYM/ ~/115/obj/Preparing VM for Qemu
Even though Macs are AMD64 machines with EFIs (at least the once we care about here), they aren't exactly IBM PC compatible. So far VMWare Fusion did all the trickery necessary to virtualize macOS for us, but qemu-system-x86_64 does not.
To make macOS boot we'll first start Qemu with OVMF (tianocore based UEFI for qemu). Next we boot OpenCore, which will do some trickery making it possible to chainload Apples stock AMD64 EFI bootloader. It also does some binary kernel patching, making it possible to load the RELEASE kernel shipped with macOS, should we want that.
OpenCore is rather configurable, but we don't care about real hardware. I'm using this version prebuild and configured to work inside Qemu. We can simply overwrite the EFI partition on our VMs disk with the EFI partition from one of the images from this repo.
Let's first find out which partition we will overwrite in a minute. From the following output from within the macOS VM (still booted via Fusion for now) we can see that in my case the EFI partition is at /dev/disk0s1:
user@users-Mac ~ % diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *69.8 GB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_APFS Container disk1 69.6 GB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +69.6 GB disk1
Physical Store disk0s2
1: APFS Volume macos - Data 43.2 GB disk1s1
2: APFS Volume Preboot 385.6 MB disk1s2
3: APFS Volume Recovery 623.2 MB disk1s3
4: APFS Volume VM 1.1 MB disk1s4
5: APFS Volume macos 16.0 GB disk1s5
6: APFS Snapshot com.apple.bless.4099... 16.0 GB disk1s5s1Now download OpenCore-v13.iso.gz and extract the image via gzip -d OpenCore-v13.iso.gz. Display the partition map to find out the images blocksize and the EFI partitions offset and size.
user@users-Mac ~ % hdiutil pmap ./OpenCore-v13.iso
MEDIA: ""; Size 150 MB [307200 x 512]; Max Transfer Blocks 2048
SCHEME: 1 GPT, "GPT Partition Scheme" [16]
SECTION: 1 Type:'MAP'; Size 150 MB [307200 x 512]; Offset 34 Blocks (307133 + 67) x 512
ID Type Offset Size Name (1)
-- -------------------- ------------ ------------ -------------------- --------
1 EFI 40 307120 disk imageNow put all those values together in a dd command like so: sudo dd if=./OpenCore-v13.iso of=/dev/disk0s1 bs=512 iseek=40 count=307120
Now let's mount the EFI disk via sudo mount -t msdos /dev/disk0s1 ~/mnt/. We have to edit OpenCores config file a tiny bit. We disable the boot device selector, as that will prevent us from starting the VMs during fuzzing completely automatically. Additionally note how we set boot-args here. In VMware Fusion we were able to use the normal macOS tools like nvram and csrutil. In OpenCore we need to set these settings in the config.plist instead.
Edit ~/mnt/EFI/OC/config.plist like so:
index 8537ca8..a46de97 100755
--- a/Users/user/mnt/EFI/OC/config.plist
+++ b/Users/user/mnt/EFI/OC/config.plist
@@ -799,7 +799,7 @@
<key>PollAppleHotKeys</key>
<true/>
<key>ShowPicker</key>
- <true/>
+ <false/>
<key>TakeoffDelay</key>
<integer>0</integer>
<key>Timeout</key>
@@ -944,7 +944,7 @@
<key>SystemAudioVolume</key>
<data>Rg==</data>
<key>boot-args</key>
- <string>keepsyms=1</string>
+ <string>-v kcsuffix=kasan wlan.skywalk.enable=0 keepsyms=1 debug=0x100008 kasan.checks=4294967295</string>
<key>csr-active-config</key>
<data>Jg8=</data>
<key>prev-lang:kbd</key>At this point you should still be able to (re)boot the VM in Fusion. It will just ignore OpenCore however. That's fine.
Prepare isa-applesmc
On boot macOS checks whether it is booted on a proper Mac by reading a value from its System Management Controller and comparing it with the value it expects. We'll retrieve this value now and pass it to qemu later. To retrieve the value:
- Download the smc_read.c source from this site.
gcc -Wall -o smc_read smc_read.c -framework IOKit./smc_read
That will produce a single line of text which you will later have to substitute in for a place marked <YOUR_APPLE_SMC_HERE>.
Booting macOS via Qemu
- Setup Homebrew on your host macOS
- Install
qemuvia homebrew - Your VMs disk should be somewhere like this on your host
~/Virtual\ Machines.localized/macOS-11.5-20G71.vmwarevm/Virtual\ Disk.vmdk. Convert it to qcow2 via something like thisqemu-img convert -U ./Virtual\ Disk.vmdk -O qcow2 ~/115/mac_hdd.qcow - Unfortunately OVMF is currently not packaged in Homebrew. Download the
ovmfpackage from ubuntu. Extract it viaar -xv ./ovmf_2020.11-4_all.debandtar -xvf ./data.tar.xz. Finallymv ./usr/share/OVMF /usr/local/share/OVMF
That's pretty much all you should need in order to boot the VM image we build. Start Qemu like so. Remember to substitute <YOUR_APPLE_SMC_HERE> and the username in the disk image path:
qemu-system-x86_64 \
-device isa-applesmc,osk="<YOUR_APPLE_SMC_HERE>" \
-accel hvf -machine q35 -smp "2",cores="2",sockets="1" -m "4096" \
-cpu Penryn,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,"+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check" \
-drive if=pflash,format=raw,readonly=on,file="/usr/local/share/OVMF/OVMF_CODE.fd" \
-drive if=pflash,format=raw,readonly=on,file="/usr/local/share/OVMF/OVMF_VARS.fd" \
-device ich9-intel-hda -device hda-duplex -device ich9-ahci,id=sata \
-device ide-hd,bus=sata.4,drive=MacHDD \
-drive id=MacHDD,if=none,file="/Users/user/115/macos_11_5.qcow",format=qcow2 \
-netdev user,id=net0,hostfwd=tcp::1042-:22, -device e1000-82545em,netdev=net0,id=net0 \
-device usb-ehci,id=ehci -usb -device usb-kbd -device usb-tablet \
-monitor stdio -vga vmwareYou should both see the macOS UI and be able to ssh user@localhost -p 1042. Confirm we are booted into your KASAN kernel:
user@users-Mac ~ % uname -a
Darwin users-Mac.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 9 16:12:43 PDT 2021; user:xnu-7195.141.2/BUILD/obj/KASAN_X86_64 x86_64Shut down your VM now. We'll let syzkaller boot it back up soon.
Building Syzkaller
- Install
govia homebrew - Add something like this to your .zshrc:
export GOPATH=/Users/user/go
export PATH=$GOPATH/bin:$PATH- Relogin and build syzkaller like this:
git clone https://github.com/google/syzkaller
cd syzkaller
make HOSTOS=darwin HOSTARCH=amd64 TARGETOS=darwin TARGETARCH=amd64 SOURCEDIR=/Users/user/115/src/Users/user/kernel/xnu-7195.141.2Fuzzing with Syzkaller
- We need g++ to make C reproducers work. Install
gcc@11via homebrew - We need addr2line from binutils to make the
/coverendpoint work. Installbinutilsvia homebrew - Add something like this to your .zshrc
export PATH="/usr/local/opt/binutils/bin:$PATH". Restart your shell - Save the following to
~/115/syzkaller.cfg. Remember to substitute<YOUR_APPLE_SMC_HERE>:
{
"target": "darwin/amd64",
"http": "127.0.0.1:56741",
"sshkey": "/Users/user/.ssh/id_macos115",
"workdir": "/Users/user/sk_darwin/",
"kernel_obj": "/Users/user/115/obj/",
"kernel_src": "/Users/user/115/src/",
"syzkaller": "/Users/user/go/src/github.com/google/syzkaller",
"procs": 2,
"type": "qemu",
"cover": true,
"image": "/Users/user/115/macos_11_5.qcow",
"vm": {
"count": 2,
"cpu": 2,
"mem": 4096,
"efi_code_device": "/usr/local/share/OVMF/OVMF_CODE.fd",
"efi_vars_device": "/usr/local/share/OVMF/OVMF_VARS.fd",
"apple_smc_osk": "<YOUR_APPLE_SMC_HERE>"
}
}Start syzkaller via ~/115/bin/syz-manager -config=/root/115/syzkaller.cfg and open http://localhost:56741 in your browser.
Overview (`docs/freebsd/README.md`)
FreeBSD
This page contains instructions to set up syzkaller to run on a FreeBSD or Linux host and fuzz an amd64 FreeBSD kernel running in a virtual machine.
Currently, syzkaller can fuzz FreeBSD running under bhyve, QEMU or GCE (Google Compute Engine). Regardless of the mode of operation, some common steps must be followed.
Setting up a host
syz-manager is the component of syzkaller that manages target VMs. It runs on a host system and automatically creates, runs and destroys VMs which share a user-specified image file.
Setting up a FreeBSD host
To build syzkaller out of the box, a recent version of FreeBSD 13.0-CURRENT must be used for the host. Older versions of FreeBSD can be used but will require manual tweaking.
The required dependencies can be installed by running:
# pkg install bash gcc git gmake go golangci-lint llvmWhen using bhyve as the VM backend, a DHCP server must also be installed:
# pkg install dnsmasqTo checkout the syzkaller sources, run:
$ git clone https://github.com/google/syzkallerand the binaries can be built by running:
$ cd syzkaller
$ gmakeOnce this completes, a syz-manager executable should be available under bin/.
Setting up a Linux host
To build Go binaries do:
make manager fuzzer execprog TARGETOS=freebsdTo build C syz-executor binary, copy executor/* files to a FreeBSD machine and build there with:
c++ executor/executor.cc -o syz-executor -O1 -lpthread -DGOOS_freebsd=1 -DGOARCH_amd64=1 -DGIT_REVISION=\"CURRENT_GIT_REVISION\"Then, copy out the binary back to host into bin/freebsd_amd64 dir.
Setting up the FreeBSD VM
It is easiest to start with a snapshot image of FreeBSD. Fetch a QCOW2 disk image for QEMU or a raw image for GCE or bhyve.
Fetch a copy of the FreeBSD kernel sources and place them in /usr/src. You will likely need to expand the disk of the VM. Before booting the VM, run:
# truncate -s 15G $IMAGEFILETo enable KCOV on FreeBSD, a custom kernel must be compiled. It is easiest to do this on the host.
Before booting the VM, compile a custom kernel on the host and install it on the VM. On the host:
# mdconfig -a -f $IMAGEFILE
# mount /dev/md0p4 /mntThis will create a memory device for the VM file and allow host to install the custom kernel source on the VM.
To get a copy of the current development sources:
# pkg install git
# git clone --depth=1 --branch=main https://github.com/freebsd/freebsd-src /usr/srcTo create a custom kernel configuration file for syzkaller and build a new kernel, run:
# cd /usr/src/sys/amd64/conf
# cat <<__EOF__ > SYZKALLER
include "./GENERIC"
ident SYZKALLER
options COVERAGE
options KCOV
__EOF__
# cd /usr/src
# make -j $(sysctl -n hw.ncpu) KERNCONF=SYZKALLER buildkernel
# make KERNCONF=SYZKALLER installkernel DESTDIR=/mntBefore booting the VM, make sure to run:
# umount /mntThe md device will linger and you can use it again later if you want. Otherwise destroy it:
# mdconfig -d -u 0Use QEMU to start a VM using the downloaded image:
$ qemu-system-x86_64 -hda $IMAGEFILE -nographic -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic,model=e1000When the boot loader menu is printed, escape to the loader prompt and enter the commands set console="comconsole" and boot. Once you reach a login prompt, log in as root and add a couple of configuration parameters to /boot/loader.conf:
# cat <<__EOF__ >>/boot/loader.conf
autoboot_delay="-1"
console="comconsole"
__EOF__After VM is booted, /etc/rc.d/growfs should haven grown its file system automatically. Otherwise run:
# /etc/rc.d/growfs onestartVerify that uname -i prints SYZKALLER to confirm that your newly built kernel is running.
Then, to permit remote access to the VM, you must configure DHCP and enable sshd:
# sysrc sshd_enable=YES
# sysrc ifconfig_DEFAULT=DHCPIf you plan to run the syscall executor as root, ensure that root SSH logins are permitted by adding PermitRootLogin without-password to /etc/ssh/sshd_config. Otherwise, create a new user with adduser. Install an ssh key for the user and verify that you can SSH into the VM from the host. Note that bhyve requires the use of the root user for the time being.
Running Under bhyve
Some additional steps are required on the host in order to use bhyve. First, since bhyve currently does not support disk image snapshots, ZFS must be used to provide equivalent functionality. Create a ZFS data set and copy the VM image there. The data set can also be used to store the syzkaller workdir. For example, with a zpool named data mounted at /data, write:
# zfs create data/syzkaller
# cp FreeBSD-13.3-CURRENT-amd64.raw /data/syzkallerThird, configure bridged networking and DHCP for the VM instances. This is no longer required on FreeBSD 14.1 and 15.0-CURRENT, as bhyve now supports libslirp-based user networking similar to the QEMU backend.
# ifconfig bridge create
bridge0
# ifconfig bridge0 inet 169.254.0.1
# echo 'dhcp-range=169.254.0.2,169.254.0.254,255.255.255.0' > /usr/local/etc/dnsmasq.conf
# echo 'interface=bridge0' >> /usr/local/etc/dnsmasq.conf
# sysrc dnsmasq_enable=YES
# service dnsmasq start
# echo 'net.link.tap.up_on_open=1' >> /etc/sysctl.conf
# sysctl net.link.tap.up_on_open=1To enable automatic configuration of bridged network every time the system boots, add the following to /etc/rc.conf:
# cloned_interfaces="bridge0 tap0"
# ifconfig_bridge0="inet 169.254.0.1 addm tap0 up"
# ifconfig_tap0="up"Finally, ensure that the bhyve kernel module is loaded:
# kldload vmmPutting It All Together
If all of the above worked, create a freebsd.cfg configuration file with the following contents (alter paths as necessary):
{
"name": "freebsd",
"target": "freebsd/amd64",
"http": ":10000",
"workdir": "/workdir",
"syzkaller": "/gopath/src/github.com/google/syzkaller",
"sshkey": "/freebsd_id_rsa",
"sandbox": "none",
"procs": 8,
}If running the fuzzer under QEMU, add:
"image": "/FreeBSD-13.0-CURRENT-amd64.qcow2",
"type": "qemu",
"vm": {
"count": 10,
"cpu": 4,
"mem": 2048
}For GCE, add the following instead (alter the storage bucket path as necessary):
"image": "/FreeBSD-13.0-CURRENT-amd64.raw",
"type": "gce",
"vm": {
"count": 10,
"instance_type": "n1-standard-4",
"gcs_path": "syzkaller"
}For bhyve, we need to specify the VM image snapshot name and networking info (alter the dataset name and paths as necessary):
"image": "/data/syzkaller/FreeBSD-13.0-CURRENT-amd64.raw",
"type": "bhyve",
"vm": {
"count": 10,
"hostip": "169.254.0.1",
"dataset": "data/syzkaller"
}If using bridged networking, the name of the bridge interface needs to be included in the "vm" block:
"vm": {
"count": 10,
"hostip": "169.254.0.1",
"dataset": "data/syzkaller",
"bridge": "bridge0"
}Then, start syz-manager with:
$ bin/syz-manager -config freebsd.cfgIt should start printing output along the lines of:
booting test machines...
wait for the connection from test machine...
machine check: 253 calls enabled, kcov=true, kleakcheck=false, faultinjection=false, comps=false
executed 3622, cover 1219, crashes 0, repro 0
executed 7921, cover 1239, crashes 0, repro 0
executed 32807, cover 1244, crashes 0, repro 0
executed 35803, cover 1248, crashes 0, repro 0If something does not work, try adding the -debug flag to syz-manager.
Missing things
- System call descriptions. The initial list of FreeBSD system calls was a copy-and-paste of Linux's, and while they have been cleaned up over time they should be audited more carefully. We are also still missing many system call descriptions.
- We should support fuzzing the Linux compatibility subsystem.
- We should provide instructions for fuzzing a FreeBSD system on ZFS
pkg/hostneeds to be taught how to detect supported syscalls/devices.- KASAN and KCSAN for FreeBSD would be useful.
- On Linux we have emission of external networking/USB traffic into kernel using tun/gadgetfs. Implementing these for FreeBSD could uncover a number of high-profile bugs.
Overview (`docs/fuchsia/README.md`)
Fuchsia support
For information about checking out and building Fuchsia see Get started with
Fuchsia and Download the Fuchsia
source code.
Caveat
Please note that Fuchsia support is currently incomplete, and may break at any
time due to changes in Fuchsia and/or Syzkaller.
Some known issues include:
- System call definitions require manual updates.
- Crash parsing does not work reliably.
- Coverage feedback is not supported.
Prerequisites
To run Syzkaller with a Fuchsia target, you will need a checkout of the Fuchsia
source repository.
The rest of this document will use the environment variable SOURCEDIR to
identify the path to your Fuchsia checkout (e.g. /home/you/fuchsia). The
commands below assume you have set SOURCEDIR, like so:
export SOURCEDIR=/home/you/fuchsiaQuick and easy version
The shell script setup.sh automates the building and runningsyz-manager steps described below. Give it a try, and see if it works for you.
For usage help, invoke it with no arguments.
Building Fuchsia
To build Fuchsia for x64, run:
fx --dir "out/x64" set core.x64 \
--with-base "//bundles/tools" \
--with-base "//src/testing/fuzzing/syzkaller" \
--args=syzkaller_dir='"/full/path/to/syzkaller"' \
--variant=kasan
fx buildAlternatively, for arm64, run:
fx --dir "out/arm64" set core.arm64 \
--with-base "//bundles/tools" \
--with-base "//src/testing/fuzzing/syzkaller" \
--args=syzkaller_dir='"/full/path/to/syzkaller"' \
--variant=kasan
fx clean-buildBuilding binaries for Fuchsia
To build all the binaries required for running Syzkaller in Fuchsia, run this
from inside your Syzkaller checkout (assuming you built Fuchsia for x64):
make TARGETOS=fuchsia TARGETARCH=amd64 SOURCEDIR=${SOURCEDIR}Running syz-manager
Running syz-manager requires you to have built Fuchsia previously, and added the
ssh keys to the fuchsia.zbi image:
The output of the Fuchsia build is a product bundle. Look up the image files needed
via ffx product get-image-path:
product_bundle_path="$(ffx config get product.path | tr -d '"')"
fxfs_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type fxfs)"
zbi_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type zbi)"
multiboot_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type qemu-kernel)"Make sure the ssh keys exist and are properly configured:
# Make sure there are ssh keys available
ffx config check-ssh-keys
auth_keys_path="$(ffx config get ssh.pub | tr -d '"')"
priv_key_path="$(ffx config get ssh.priv | tr -d '"')"Add the authorized keys to the zbi.
fx host-tool zbi -o "${SOURCEDIR}/out/x64/fuchsia-ssh.zbi" \
"${zbi_path}" \
--entry "data/ssh/authorized_keys=${auth_keys_path}"Note: This needs to be repeated after each fx build.
Create a syz-manager configuration file, using this as a starting point:
{
"name": "fuchsia",
"target": "fuchsia/amd64",
"http": ":12345",
"workdir": "/workdir.fuchsia",
"kernel_obj": "/fuchsia/out/syz/kernel_x64-kasan/obj/zircon/kernel",
"syzkaller": "/syzkaller",
"image": "$fxfs_path",
"sshkey": "$priv_key_path",
"reproduce": false,
"cover": false,
"procs": 8,
"type": "qemu",
"vm": {
"count": 10,
"cpu": 4,
"mem": 2048,
"kernel": "${multiboot_path}",
"initrd": " ${SOURCEDIR}/out/x64/fuchsia-ssh.zbi"
}
}Run syz-manager with that config:
bin/syz-manager -config manager.cfgNote: You may need to modify your PATH so that qemu can be found, e.g.
export `PATH="$SOURCEDIR/prebuilt/third_party/qemu/linux-x64/bin:$PATH"`Update syscall and FIDL definitions
Syscall descriptions live in the sys/fuchsia folder. To update a syscall, you
need to modify the .txt file that contains it, make sure your new definition
matches the one in Zircon's
syscalls.abigen
file. If the syscall was used in executor/common_fuchsia.h, you need to
update the usages there as well. FIDL definitions do not need manual updating
because they are extracted automatically when you run make extract, but they
require Fuchsia to be rebuilt for each architecture (see Building
Fuchsia above).
Once you updated the syscall definitions, everything can be regenerated by
running (in your Syzkaller checkout):
make extract TARGETOS=fuchsia SOURCEDIR=${SOURCEDIR}
make generateCaveat: This command does not currently work.
How to generate syscall definitions from FIDL
TODO: This section is out of date.
Syscall descriptions for FIDL are automatically generated as part of make extract as described above.
However, if you wish to manually generate syscall descriptions for a given.fidl file, do the following.
FIDL files should first be compiled into FIDL intermediate representation (JSON)
files using fidlc:
${SOURCEDIR}/out/x64/host_x64/fidlc --json /tmp/io.json --files ${SOURCEDIR}/zircon/system/fidl/fuchsia-io/io.fidlThen run the FIDL compiler backend fidlgen with the syzkaller generator, which
compiles a FIDL IR file into a syscall description file:
${SOURCEDIR}/out/x64/host_x64/fidlgen -generators syzkaller -json /tmp/io.json -output-base fidl_io -include-base fidl_ioRunning syz-ci locally
To run syz-ci locally for Fuchsia, you need:
- bootstrapped Fuchsia checkout (in
/bootstrap/fuchsiadir in the example below) - bootstrap
syz-cibinary (in the current dir, build withmake ci) syz-ciconfig similar to the one below (inci.cfgfile in the current dir)
{
"name": "testci",
"http": ":50000",
"manager_port_start": 50001,
"syzkaller_repo": "https://github.com/google/syzkaller.git",
"managers": [
{
"name": "fuchsia",
"repo": "https://fuchsia.googlesource.com",
"manager_config": {
"target": "fuchsia/amd64",
"type": "qemu",
"cover": false,
"procs": 8,
"vm": {
"count": 4,
"cpu": 4,
"mem": 1024
}
}
}
]
}Run syz-ci as:
SOURCEDIR=/bootstrap/fuchsia ./syz-ci -config ci.cfgTroubleshooting
While running the make extract step, it's possible that the FIDL definitions
are not up to date. It could happen that they have been removed or renamed.
If this is the case, you would see an error mentioning that the fidl.json file
could not be found:
go generate ./sys/fuchsia
cannot find /path-to-fuchsia/out/x64/fidling/gen/zircon/public/fidl/zircon-ethernet/zircon-ethernet.fidl.json
exit status 1You can search for the string in the Fuchsia code search
interface or in the
Gerrit code-review tool to see what
happened to it. If the FIDL interface was renamed or removed, you should updatesys/fuchsia/fidlgen/main.go to reflect this change, and remove the stale
autogenerated files.
Overview (`docs/gvisor/README.md`)
gVisor
gVisor is a user-space kernel, written in
Go, that implements a substantial portion of the Linux system surface.
gVisor uses linux OS, but the special gvisor VM type. There is nothing
special regarding gVisor besides that. The image is a gVisor release
tarball (gvisor.tar.bz2, see
installation instructions),
which syzkaller extracts into the workdir. There is no kernel object, sokernel_obj is not needed; for coverage, point kernel_src to the gVisor
source tree. Here is an example manager config:
{
"name": "gvisor",
"target": "linux/amd64",
"http": ":12345",
"workdir": "/workdir",
"image": "/path/to/gvisor.tar.bz2",
"syzkaller": "/gopath/src/github.com/google/syzkaller",
"cover": false,
"procs": 8,
"type": "gvisor",
"vm": {
"count": 5,
"runsc_args": "-platform=kvm"
}
}Reproducing crashes
syz-execprog can be used inside gVisor to (hopefully) reproduce crashes.
To run a single program inside a minimal gVisor sandbox, do the following.
- Build all of the syzkaller tools:
$ cd $SYZKALLER_DIR
$ make- Build the gVisor release tarball and extract it:
$ cd $GVISOR_DIR
$ make release-tarball DESTINATION=bin/
$ tar -C bin/ -xf bin/gvisor.tar.bz2- Create a
bundle/directory with a config like the one below. Be sure to
update the paths to thelinux_amd64directory and input log/program file.
$ mkdir bundle
$ $EDITOR bundle/config.json- Run gVisor:
$ sudo bin/runsc \
-platform=ptrace \
-file-access=shared \
-network=host \
run \
-bundle /PATH/TO/bundle/ \
syzkaller- Remove container:
$ sudo bin/runsc delete -force syzkallerNote that you'll want to adjust the runsc args to match the config in which
the crash was discovered. You may also want to add -debug -strace for more
debugging information.
You can also adjust the args to syz-execprog in config.json. e.g., add-repeat to repeat the program.
config.json
/* Detailed source-code truncated for AI context efficiency. */syzkaller way gVisor use
To build a specific gVisor version:
git clone https://github.com/google/gvisor
git checkout be6ffa78e4df78df13d004a17f2a8833305285c4syzkaller uses gVisor release tarballs as images. pkg/build builds them from
the gVisor sources with make copy (see pkg/build/gvisor.go), e.g. for the
coverage-instrumented build that syzbot uses:
make copy TARGETS=//debian:gvisor-release-coverage-tar-bz2 DESTINATION=gvisor-coverage.tar.bz2-race and -race-coverage variants of the tarball are available as well.
The kernel config of a gVisor manager consists of the -cover and -race
flags that select the variant.
To get coverage symbols, extract the tarball and run runsc symbolize:
tar -xf gvisor-coverage.tar.bz2
./runsc symbolize -all > symbolize_all_gvisor_be6ffa78e4df78df13d004a17f2a8833305285c4.txtOverview (`docs/hafnium/README.md`)
Hafnium
Hafnium is a hypervisor.
This is work-in-progress, see #996.
For details on how to build/run Hafnium see GettingStarted, HafniumRamDisk and PreparingLinux.
For now we only have descriptions for AF_HF protocol provided by Hafnium linux driver.
Overview (`docs/netbsd/README.md`)
NetBSD
Instructions to set up syzkaller for a Linux host and a NetBSD guest.
Preparing the NetBSD sources
Get the NetBSD kernel source (preferably HEAD).
shhost$ mdkir $HOME/netbsd host$ cd $HOME/netbsd host$ git clone https://github.com/NetBSD/src.gitBuild the tools. You will have the toolchain in
$HOME/netbsd/tools.shhost$ cd src host$ ./build.sh -j4 -m amd64 -U -T ../tools toolsBuild the distribution. This might take a while.
shhost$ ./build.sh -j4 -m amd64 -U -T ../tools -D ../dest distribution
At this point you should have a NetBSD distribution in $HOME/netbsd/dest.
Installing and building syzkaller on the Linux host
Install all the dependencies for syzkaller.
Clone the syzkaller repository.
shhost$ git clone https://github.com/google/syzkaller host$ cd syzkallerCompile syzkaller for NetBSD.
shhost$ make TARGETOS=netbsd SOURCEDIR=$HOME/netbsd
The above steps should have built the syzkaller binaries for NetBSD.
You can see the compiled binaries in bin/netbsd_amd64.
Setting up a NetBSD VM with qemu
You can use the script given here to create a disk image with NetBSD installed.
The script would also automatically give you a ssh key to ssh into the VM.
Alternatively, you can follow the tutorial given here to
set up a basic NetBSD VM with qemu.
After installing and running the NetBSD VM on qemu, please follow the steps below to
configure ssh.
Create a ssh-keypair on the host and save it as
netbsdkey.shhost$ ssh-keygen -f netbsdkey -t rsa -N ""Make sure you have a NAT enabled in your Qemu command line. Typically to
forward the host port10022to the guest port22:shhost$ qemu-system-x86_64 ... -netdev user,id=mynet0,hostfwd=tcp:127.0.0.1:10022-:22 -device e1000,netdev=mynet0Append the following lines to
/etc/rc.confon the guest. You can use thevieditor to do that.textsshd=YES dhcpcd=YES ifconfig_wm0="inet 10.0.2.15 netmask 255.255.255.0"Append this to
/etc/ssh/sshd_configon the guest.textPort 22 ListenAddress 10.0.2.15 PermitRootLogin yes PermitRootLogin without-passwordNow you should be able to ssh into the NetBSD VM.
shhost$ ssh -p 10022 [email protected]Copy and paste your public key to
/root/.ssh/authorized_keyson the guest
andrebootthe VM.After reboot make sure that ssh is working properly. Replace the port with what
you have configured.shhost$ ssh -i path/to/netbsdkey -p 10022 [email protected]
If the last command returns a proper shell it means the VM has been configured.
Compiling a NetBSD kernel (Optional)
You can compile a kernel with KASAN to increase the chances of finding bugs.
Make a copy of the config file.
shhost$ cd $HOME/netbsd/src host$ cp sys/arch/amd64/conf/GENERIC sys/arch/amd64/conf/SYZKALLERUncomment the following lines in
sys/arch/amd64/conf/SYZKALLERto enable KASAN.text#makeoptions KASAN=1 # Kernel Address Sanitizer #options KASAN #no options SVSCompile the kernel with KASAN (assuming you have followed the initial steps to
build tools).shhost$ cd $HOME/netbsd/src host$ ./build.sh -m amd64 -U -T ../tools -j4 kernel=SYZKALLERAt this point you should have the new compiled kernel image which can be found in
$HOME/netbsd/src/sys/arch/amd64/compile/SYZKALLERand should have the namenetbsd. You need to copy it to the installed VM and reboot the VM.
Running syzkaller
If all of the above worked,
poweroffthe VM and create thenetbsd.cfgconfig
file on the host with the following contents (alter paths as necessary):text{ "name": "netbsd", "target": "netbsd/amd64", "http": ":10000", "workdir": "work", "syzkaller": "$GOPATH/src/github.com/google/syzkaller", "image": "path/to/netbsd.img", "sshkey": "/path/to/netbsdkey", "sandbox": "none", "procs": 2, "cover": false, "type": "qemu", "vm": { "qemu": "qemu-system-x86_64", "count": 2, "cpu": 2, "mem": 2048 } }The above directories have to be specified to the exact locations and the ssh keys
must be in a separate directory with chmod 700 permissions set to that directory
and chmod 600 permissions to the files in both the guest and the host.Then, inside the syzkaller folder where the
netbsd.cfgfile also exists, startsyz-managerwith:shhost$ bin/syz-manager -config netbsd.cfgYou can add a
-debugflag to the above command to view the log if any issues arise.Once syzkaller has started executing, it should start printing output along the lines of:
textbooting test machines... wait for the connection from test machine... machine check: 253 calls enabled, kcov=true, kleakcheck=false, faultinjection=false, comps=false executed 3622, cover 1219, crashes 0, repro 0 executed 7921, cover 1239, crashes 0, repro 0 executed 32807, cover 1244, crashes 0, repro 0 executed 35803, cover 1248, crashes 0, repro 0
syzbot
syzbot tests NetBSD and reports bugs to
syzkaller-netbsd-bugs mailing list
(also can be seen on dashboard).
The image syzbot uses can be downloaded here (266MB, includes root ssh key). The image was built using this script.
The image can be used with qemu as follows:
qemu-system-x86_64 -m 1024 -smp 2 -nographic -enable-kvm \
-netdev user,id=mynet0,hostfwd=tcp:127.0.0.1:10022-:22 \
-device e1000,netdev=mynet0 -hda netbsd-image.rawAnd then you can ssh/scp into the VM using:
ssh -i netbsd-image.key -p 10022 -o IdentitiesOnly=yes root@localhost
scp -i netbsd-image.key -P 10022 -o IdentitiesOnly=yes FILE root@localhost:/root/Note: the image contains a stock kernel, so if you are reproducing a bug
most likely you want to update kernel as the first step:
scp -i netbsd-image.key -P 10022 -o IdentitiesOnly=yes \
src/sys/arch/amd64/compile/obj/GENERIC_SYZKALLER/netbsd root@localhost:/netbsd
ssh -i netbsd-image.key -p 10022 -o IdentitiesOnly=yes root@localhost /sbin/rebootMissing things
- Automating the configuration changes (like appending to config files), generating the json config file on the fly (with customizable values to the keys using command line parameters) and calling syz-manager with
anitausing just a single command. - System call descriptions.
sys/netbsd/*.txtis a dirty copy fromsys/linux/*.txtwith everything that does not compile dropped. We need to go through syscalls and verify/fix/extend them, including devices/ioctls/etc. - Currently only
amd64arch is supported. Supporting386would be useful, because it should cover compat paths. Also, we could do testing of the linux-compatibility subsystem. pkg/hostneeds to be taught how to detect supported syscalls/devices.- On Linux we have emission of external networking/USB traffic into kernel using tun/gadgetfs. Implementing these for NetBSD could uncover a number of high-profile bugs.
Overview (`docs/starnix/README.md`)
starnix support
Prerequisites
To run syzkaller for fuzzing starnix, you will need a checkout of the Fuchsia
source repository.
The rest of this document will use the environment variable SOURCEDIR to
identify the path to your Fuchsia checkout (e.g. /home/you/fuchsia). The
commands below assume you have set SOURCEDIR, like so:
export SOURCEDIR=/home/you/fuchsiaTo build Fuchsia for qemu-x64, run:
fx --dir "out/qemu-x64" set workstation_eng.qemu-x64 \
--with "//bundles/tools" \
--with "//src/proc/bin/starnix"
fx buildYou will also need to follow the instructions in the sections GCC and Kernel
of the setup_ubuntu-host_qemu-vm_x86-64-kernel.md file.
syzkaller
Building binaries for starnix
First, you need to build all the binaries required for running syzkaller in starnix.
For that, you only need to run this from inside your syzkaller checkout (assuming you built Fuchsia for x64):
SYZ_STARNIX_HACK=1 make TARGETOS=linux TARGETARCH=amd64Configuration file
Create a manager config like the following, replacing the environment variables $SYZKALLER, $KERNEL and $IMAGE with their actual values.
NOTE:
ffxis still under development, for that reason VM count of 1 is recommended until fxbug.dev/118926 is solved.
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "$SYZKALLER/workdir",
"kernel_obj": "$KERNEL",
"kernel_src": "$SOURCEDIR",
"syzkaller": "$SYZKALLER",
"procs": 8,
"type": "starnix",
"vm": {
"count": 1
},
"cover": false
}Running
Lastly, just run the command below to start fuzzing.
SYZ_STARNIX_HACK=1 bin/syz-manager -config=./starnix.cfgOverview (`docs/trusty/README.md`)
Trusty support
Trusty is a set of software
components supporting a Trusted Execution Environment (TEE) on mobile devices.
This is work-in-progress, see #933. For now we only support testing Trusty via actual application ports.
Building kernel with Trusty IPC support
git remote add android https://android.googlesource.com/kernel/common
git fetch android
git checkout android/android-trusty-4.9
make distclean
# TODO: consider using trusty_qemu_defconfig instead.
make ARCH=arm64 ranchu64_defconfig
# Required to enable qemu networking:
make ARCH=arm64 defconfig
make ARCH=arm64 kvmconfig
# Some custom configs:
ARCH=arm64 ./scripts/kconfig/merge_config.sh .config trusty.config
# We don't install modules:
sed -i 's#^\(.*\)=m$#\# \1 is not set#g' .config
make ARCH=arm64 olddefconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j64# trusty.config
CONFIG_TRUSTY=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_INFO=y
CONFIG_KCOV=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_VM=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=140
CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y
CONFIG_WQ_WATCHDOG=yBuilding Trusty
mkdir trusty; cd trusty
repo init -u https://android.googlesource.com/trusty/manifest -b master
repo sync -j32
source trusty/vendor/google/aosp/scripts/envsetup.sh
make -j32 generic-arm64
# Build Trusty and qemu images:
trusty/vendor/google/aosp/scripts/build.py qemu-generic-arm64-test-debug
# Create qemu-comb.dtb:
KERNEL_DIR=$KERNEL build-root/build-qemu-generic-arm64-test-debug/run-qemuBuilding arm64 image
git clone git://git.buildroot.net/buildroot
cd buildroot
make qemu_aarch64_virt_defconfig
support/kconfig/merge_config.sh .config syzkaller.config
make -j64# syzkaller.config
BR2_cortex_a57=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_ARM_AARCH64=y
BR2_TARGET_GENERIC_HOSTNAME="syzkaller"
BR2_TARGET_GENERIC_ISSUE="syzkaller"
BR2_PACKAGE_DHCPCD=y
BR2_PACKAGE_OPENSSH=y
BR2_TARGET_ROOTFS_EXT2_SIZE="1G"
BR2_ROOTFS_POST_FAKEROOT_SCRIPT="./syzkaller.sh"
# BR2_LINUX_KERNEL is not set# syzkaller.sh
set -eux
# Mount debugfs for KCOV.
echo "debugfs /sys/kernel/debug debugfs defaults 0 0" >> $1/etc/fstab
# Generate and install ssh key.
rm -f key key.pub
ssh-keygen -f key -t rsa -N ""
mkdir -p $1/root/.ssh
cp key.pub $1/root/.ssh/authorized_keysTesting build
TODO: where does the firmware come from?
Boot in qemu:
cd $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug
$TRUSTY/build-root/build-qemu-generic-arm64-test-debug/qemu-build/aarch64-softmmu/qemu-system-aarch64 -m 1024 -smp 1 -net nic -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -display none -serial stdio -no-reboot -machine virt,secure=on,virtualization=on -cpu cortex-a57 -bios $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/bl1.bin -d unimp -semihosting-config enable,target=native -no-acpi -dtb $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/qemu-comb.dtb -hda $BUILDROOT/output/images/rootfs.ext4 -snapshot -kernel $KERNEL/arch/arm64/boot/Image -append "androidboot.hardware=qemu_trusty earlyprintk=serial console=ttyAMA0,38400 root=/dev/vda"SSH into the VM:
ssh -i $BUILDROOT/key -p 10022 -o IdentitiesOnly=yes root@localhostRunning syzkaller
Build and run syzkaller as:
cd $SYZKALLER
make TARGETARCH=arm64
cd $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug
$SYZKALLER/bin/syz-manager -config trusty.cfgusing config along the lines of (substitute actual values for $KERNEL, $SYZKALLER, $BUILDROOT and $TRUSTY):
{
"name": "trusty",
"target": "linux/arm64",
"http": ":10000",
"workdir": "/workdir",
"kernel_obj": "$KERNEL",
"syzkaller": "$SYZKALLER",
"image": "$BUILDROOT/output/images/rootfs.ext4",
"sshkey": "$BUILDROOT/key",
"cover": false,
"procs": 4,
"type": "qemu",
"vm": {
"count": 4,
"cpu": 1,
"mem": 1024,
"qemu": "$TRUSTY/build-root/build-qemu-generic-arm64-test-debug/qemu-build/aarch64-softmmu/qemu-system-aarch64",
"qemu_args": "-machine virt,secure=on,virtualization=on -cpu cortex-a57 -bios $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/bl1.bin -d unimp -semihosting-config enable,target=native -no-acpi -dtb $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/qemu-comb.dtb",
"cmdline": "androidboot.hardware=qemu_trusty console=ttyAMA0,38400 root=/dev/vda",
"kernel": "$KERNEL/arch/arm64/boot/Image"
},
"enable_syscalls": [
"openat$trusty*",
"write$trusty*",
"read",
"ioctl$TIPC_IOC_CONNECT*",
"ppoll",
"dup3",
"tkill",
"gettid",
"close"
]
}Overview (`docs/windows/README.md`)
Windows
Windows support is very raw and preliminary (read, non-working).
There is a closed-source port at Microsoft.
There is a more complete
closed-source Windows port
done by Fritz and zer0mem.
The port has found 6 bugs including
CVE-2018-8441.
Also, BSoDs in WSL: 1, 2.
See BUGS ON THE WINDSHIELD: FUZZING THE WINDOWS KERNEL presentation.
To update descriptions run (assumes cl cross-compiler is in PATH):
syz-extract -os=windows
syz-sysgensys/windows/windows.txt was auto-extracted from windows headers with tools/syz-declextract.
To build binaries:
make fuzzer execprog stress TARGETOS=windows
REV=git rev-parse HEAD
cl executor\executor_windows.cc /EHsc -o bin\windows_amd64\syz-executor.exe \
-DGIT_REVISION=\"$REV\" \
kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib \
shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib \
winmm.lib rpcrt4.lib Crypt32.lib imm32.lib Urlmon.lib Oleaut32.lib \
Winscard.lib Opengl32.lib Mpr.lib Ws2_32.lib Bcrypt.lib Ncrypt.lib \
Synchronization.lib Shell32.lib Rpcns4.lib Mswsock.lib Mincore.lib \
Msimg32.lib RpcRT4.lib Rpcrt4.lib lz32.libTo run syz-execprog:
bin\windows_amd64\syz-execprog.exe -executor c:\full\path\to\bin\windows_amd64\syz-executor.exe -stressWindows is supported by only gce VMs at the moment.
To use gce, create a Windows GCE VM, inside of the machine:
- Enable serial console debugging (see this for details):
bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200 /noumexDisable automatic restart in
sysdm.cpl -> Advanced -> Startup and RecoverySetup sshd with key auth, these instructions worked for me.
Preferably use non-admin user. Save private ssh key.
Then shutdown the machine, stop the instance and create an image from the disk.
Then start syz-manager with config similar to the following one:
{
"name": "windows",
"target": "windows/amd64",
"http": ":20000",
"workdir": "/workdir",
"syzkaller": "/syzkaller",
"sshkey": "/id_rsa",
"ssh_user": "you",
"cover": false,
"procs": 8,
"type": "gce",
"vm": {
"count": 10,
"machine_type": "n1-highcpu-2",
"gce_image": "your-gce-image"
}
}Overview (`pkg/coveragedb/README.md`)
Prepare coverage aggregation pipeline
Assuming you have the coverage *.jsonl files in some bucket:
- Create BigQuery table.
- Start data transfers from the bucket to BigQuery table.
Coverage merger job is consuming data from BQ table and store aggregations
in the Spanner DB.
The dots in BQ table name are not supported, thus:
- For the namespace "upstream" the expected BQ table name is "upstream".
- For the namespace "android-6.12" the expected BQ table name is "android-6-12".
Create new BigQuery table for coverage data
bq mk \
--table \
--description "android 6.12" \
--time_partitioning_field timestamp \
--time_partitioning_type DAY \
--require_partition_filter=true \
--clustering_fields file_path,kernel_commit,hit_count \
syzkaller:syzbot_coverage.android-6-12 \
./pkg/coveragedb/bq-schema.jsonAdd new data transfer
bq mk \
--transfer_config \
--display_name=ci-android-6-12-bucket-to-syzbot_coverage \
--params='{"destination_table_name_template":"android-6-12",
"data_path_template": "gs://$COVERAGE_STREAM_BUCKET/ci-android-6.12/*.jsonl",
"allow_jagged_rows": false,
"allow_quoted_newlines": false,
"delete_source_files": true,
"encoding": "UTF8",
"field_delimiter": ",",
"file_format": "JSON",
"ignore_unknown_values": false,
"max_bad_records": "0",
"parquet_enable_list_inference": false,
"parquet_enum_as_string": false,
"preserve_ascii_control_characters": false,
"skip_leading_rows": "0",
"use_avro_logical_types": false,
"write_disposition": "APPEND"
}' \
--project_id=syzkaller \
--data_source=google_cloud_storage \
--target_dataset=syzbot_coverageList BigQuery data transfers
bq ls \
--transfer_config \
--transfer_location=us-central1 \
--project_id=syzkallerOverview (`pkg/report/README.md`)
Bugs scoring
Until triaged we don't really know the bug impact. But we can learn a lot from the bug title.
Syzbot scoring is based on our understanding of what bug class looks historically more impactful. It allows to
prioritize the triaging queue.
Heuristics
KASAN > KMSAN > KCSAN
KASAN detected bugs are typically more dangerous than KMSAN detected bugs. And KMSAN detected bugs are typically more
dangerous than KCSAN detected bugs.
Invalid-free (double-free) >= use-after-free write > use-after-free read.
KASAN write > KASAN read
KASAN write indicates an out-of-bounds or use-after-free write operation. Any uncontrolled write to kernel memory is
extremely dangerous because it can corrupt data or code pointers, making it a high-value target for exploitation
and leading to system compromise. KASAN read indicates an out-of-bounds or use-after-free read. This is generally
less critical. It can crash the system (DoS) or leak sensitive data, but it doesn't provide a direct path for an
attacker to execute their own code.
Memory Safety bugs > DoS bugs.
This heuristic establishes a broad priority between two major classes of bugs based on their ultimate impact.
Memory Safety bugs: This category includes all the issues mentioned above—use-after-free, double-free, out-of-bounds
reads/writes, etc. These are considered more severe because they represent a potential system compromise. A successful
exploit can allow an attacker to escalate privileges and gain complete control over the kernel and the entire system.
DoS bugs (Denial of Service): This category includes bugs like kernel hangs, crashes, or resource exhaustion
(e.g., memory leaks). While they are serious because they disrupt system availability, they typically do not allow an
attacker to execute code or steal data. The impact is usually temporary and can be resolved by rebooting the system.
They disrupt the service but don't compromise its integrity.
Information Leaks > Denial of Service (DoS)
Kmsan infoleak and other bugs that leak kernel memory are generally more severe than a typical DoS. These leaks can be
used to bypass security mitigations like Kernel Address Space Layout Randomization (KASLR), which makes exploiting
other vulnerabilities easier.
Concurrency Issues > Simple DoS
Bugs like DataRace and LockdepBug can be more critical than a standard DoS. Data races can lead to unpredictable
behavior, including memory corruption, which might be exploitable.
LockdepBug indicates potential deadlocks, which can cause a more severe system Hang than a resource-exhaustion DoS.
KFENCE reports are high priority
KFENCE is a lighter-weight memory safety detector compared to KASAN. While it may have a lower performance overhead,
the bugs it finds (use-after-free, out-of-bounds) are of the same high-impact nature as those found by KASAN.
Therefore, KFENCE detected bugs should be treated with a similar level of urgency as KASAN reports.
UBSAN reports require careful evaluation
The Undefined Behavior Sanitizer (UBSAN) can detect a wide range of issues. Their severity can vary greatly:
- A shift-out-of-bounds or array-index-out-of-bounds issue can be very severe if it leads to memory corruption.
- An integer-overflow can also be critical if it results in bypassing security checks and leads to a buffer overflow.
- Other UBSAN issues might be less critical but still indicate latent bugs that could become problematic.
LockdepSleeping in Atomic Context is a critical flaw
AtomicSleep is a serious bug that can lead to system-wide hangs and instability. This is because holding a spinlock
or being in another atomic context while sleeping can cause deadlocks.
These are generally more severe than a typical DoS.
Memory Leaks are a form of DoS
MemoryLeak bugs are a type of denial of service where the kernel gradually runs out of memory. While generally less
severe than memory corruption, a fast memory leak that can be triggered by an unprivileged user can be a high-impact
DoS vector.
NULL pointer dereference in reads/writes
They may be exploitable (see proof)
but the exploitation probability is not clear. Because of this uncertainty they are put to the bottom of the high
priority bugs.
Overview (`sys/fuchsia/README.md`)
Fuchsia system call definitions
This directory contains the system call definitions for Fuchsia's kernel,
Zircon. They are currently updated manually, but we hope that will change Real
Soon Now. When/if it does, we'll update this file.
The .fidl files in the vDSO
directory
describe Zircon's system calls in FIDL, and in the comments there are English
and C++ descriptions. For every FIDL file, there should be a corresponding.txt file in this directory that describes the system calls using Syzkaller's
syzlang
language.
Overview (`syz-cluster/dashboard/README.md`)
To update and access the web dashboard during local development:
$ make install-dev-config
$ make deploy-web-dashboard-dev
$ kubectl port-forward service/web-dashboard-service --address 0.0.0.0 50123:80Overview (`tools/docker/README.md`)
Docker images
We provide a set of Docker images that provide dev environment suitable for syzkaller development/testing.
These images are used by CI, but can also be used for local development.
- env includes Go/fuchsia/netbsd toolchains, gcloud sdk, C/C++ cross-compilers, make, git and other essential tools.
- old-env provides essential tools but based on an older disto (ubuntu:16.04).
These images are available as gcr.io/syzkaller/{env,old-env}, respectively.
To download and run locally:
docker pull gcr.io/syzkaller/env
docker run -it gcr.io/syzkaller/envBuilding Multi-arch Images
The syzbot and env images support multiple architectures (amd64, arm64). To build and push them, we use Docker buildx to build a multi-arch image in a way that allows distributing it under one tag name.
1. One-time Setup
Install the QEMU emulators and create a new builder instance:
# Install QEMU emulators for multi-arch support
docker run --privileged --rm tonistiigi/binfmt --install all
# Create and bootstrap a new builder
docker buildx create --name mybuilder --driver docker-container --bootstrap
docker buildx use mybuilder2. Build and Push
Once the builder is configured, you can build and push the images. Ensure you are authenticated with Google Cloud:
gcloud auth login && gcloud auth configure-dockerBuilding the syzbot image:
docker buildx build --platform linux/amd64,linux/arm64 \
-t gcr.io/syzkaller/syzbot \
tools/docker/syzbot \
--pushBuilding the env image:
docker buildx build --platform linux/amd64,linux/arm64 \
-t gcr.io/syzkaller/env \
tools/docker/env \
--push[DEPRECATED] Github Packages
Github packages are not supported (if you can't access gcr.io, please contact us).
docker tag gcr.io/syzkaller/env [docker.pkg.github.com/google/syzkaller/env](https://docker.pkg.github.com/google/syzkaller/env)
docker login [https://docker.pkg.github.com](https://docker.pkg.github.com)
docker push [docker.pkg.github.com/google/syzkaller/env](https://docker.pkg.github.com/google/syzkaller/env)Overview (`tools/syz-aflow/README.md`)
`syz-aflow` - AI Workflow Executor
syz-aflow is a CLI tool designed to execute aflow workflows locally for debugging and testing purposes.
Building
To build syz-aflow, use the syz-env environment from the repository root:
./tools/syz-env go build ./tools/syz-aflowThis will create a syz-aflow binary in the repository root.
Usage
Basic Execution
To run a workflow, you need to specify the workflow name, an input JSON file, and a working directory:
./tools/syz-env ./syz-aflow -workflow <workflow_name> -input <input.json> -workdir <workdir>Batch Execution
If -input is a directory, each *.json file in it is executed as a separate task:
./tools/syz-env ./syz-aflow -workflow seed-gen-file-line -input ./tasks -workdir ./workdir -parallel 4Results are saved to <workdir>/trajectories/<state>/. Completed tasks are skipped on restart.
Batch execution is currently only supported for the seed-gen-file-line workflow and cannot be combined
with -html and -output.
To spread the load over several Gemini API keys, put them one per line into GOOGLE_API_KEY
(e.g. GOOGLE_API_KEY="$(cat keys.txt)"). Each task sticks to one key. Keys only help with rate limits
if they belong to different projects.
With -corpus , all programs executed during the batch run are also saved into the given corpus.db
(without call properties such as fault injection), e.g. to be used as fuzzing seeds.
Workflow Inputs
syz-aflow does not require a standard syz-manager configuration file. Instead, it takes a JSON file containing the arguments specific to the workflow you are running.
If the workflow needs to perform actions that interact with VMs (like reproducing a crash or testing a patch), it will expect fields in that input JSON that describe the environment (e.g., Image, VM type and config, KernelSrc, etc.). The workflow code then takes these individual arguments and builds the necessary manager configuration programmatically on the fly.
Example Input for `patching` Workflow
{
"Syzkaller": "/path/to/syzkaller",
"Image": "/path/to/linux/image",
"Type": "qemu",
"VM": {
"count": 1,
"cpu": 2,
"mem": 2048,
"qemu_args": "-machine q35 -enable-kvm -smp 2,sockets=2,cores=1"
},
"ReproSyz": "syz_open$dir(0x0, 0x1) ...",
"KernelConfig": "@/path/to/linux.config"
}Any string field in the input JSON starting with @ (e.g. "@/path/to/file" or "@./relative/path") will be automatically expanded with the contents of that file. Relative paths are resolved relative to the directory of the -input JSON file. Literal leading @ characters can be escaped with @@ (e.g. "@@literal").
Flags
-workflow: The name of the workflow to execute.-input: Path to a JSON file containing the arguments for the workflow (or a directory of them).-parallel: Number of tasks to run concurrently in batch execution.-corpus: Path to acorpus.dbto save all executed programs into in batch execution.-workdir: Directory where the workflow can perform checkouts, builds, etc.-html: Path to an HTML file where the execution trajectory will be rendered in real-time.-model: Override the default LLM model.-cache-size: Set the maximum cache size (default "10GB").-download-bug: Download bug details from the dashboard by ID or ExtID.-auth: Use gcloud auth token when downloading bugs.
Live Trajectory Visualization
You can monitor the execution of the workflow in real-time by using the -html flag:
./tools/syz-env ./syz-aflow -workflow patching -input input.json -workdir ./workdir -html trajectory.htmlOpen the specified HTML file in your browser to see the charts and steps as they execute.
Overview (`tools/syz-covermerger/README.md`)
Covermerger, how it works
Coverage signal lifetime
Syzkaller relies on KCOV to get information about the actual basic blocks executed
by kernel code.
This tool consumes line:col pairs that are coming from addr2line(pc) where pc
is the actual pc where we observed KCOV instrumentation hit.
What do the dashboard numbers mean
Understanding the coverage percentages displayed on the dashboard can be tricky.
These numbers represent neither pure basic block coverage nor pure line coverage.
Instead, the dashboard shows the percentage of lines with coverage signals where at least one basic block
attributed to that line was executed.
Here's why this matters:
A single line of code can contain multiple basic blocks.
Covermerger reports a "hit" for a line even if only one out of several basic blocks on that line was covered.
For example, if a line has two basic blocks and only one is covered, we report a hit (50% basic block coverage
for that line, but 100% line coverage in our dashboard's context).
Essentially, dashboard coverage will always be greater than or equal to basic block coverage.
How this tool works
Covermerger's primary function is to merge coverage signals from various kernel versions and configurations.
It provides daily, monthly, and quarterly aggregations.
Merging Process
Independent File Processing: Each file is processed on its own.
Same Version: If coverage signals come from the exact same version of a file, their hit counts are simply added together.
Different Versions: When merging signals from different file versions (e.g., if a file changed multiple times within
a month), a diff-tool is used to identify equivalent lines across versions.
Projection to Latest Version: All merged signals are projected onto the latest available kernel version within the
aggregation period. For the current month, the target file version may change with each merge run. For past months,
the latest commit available at the end of that month is used as the target version.
Consider a scenario where a file changed three times in a month, and not all lines were included in every build.
Covermerger intelligently combines the signals to provide a comprehensive view despite these variations.
Update frequency
Merge jobs are automated via cron.yaml and follow this schedule:
- Daily Aggregations: Updated every day. Batch jobs typically refresh "today" and "yesterday" numbers.
- Monthly Aggregations: Updated weekly, with the current month's data refreshed 4-5 times per month.
- Quarterly Aggregations: Also updated weekly.
We retain original coverage signals for at least one year, allowing for regeneration of data if needed.
Overview (`tools/syz-db-export/README.md`)
Syzbot DB export
Every week syzbot runs syz-db-export to export its databases:
- Upstream Linux
db is here. - Contact us if you want see others.
Export structure
DB currently includes:
- Bugs descriptions.
- First C-Reproducer for every bug.
It doesn't include:
- Second+ C-Reproducers for every bug.
- Syz-Reproducers.
- Any reproducer related metadata (like triggering requirements).
How to export more data
The best way to see more data exported is to modify the tool and send us PR with your changes.
To reproduce locally what syzbot is doing for upstream Linux:
go run ./tools/syz-db-export/... -namespace upstreamExtending tools/syz-db-export you can teach syzbot to export more.
Overview (`tools/syz-declextract/README.md`)
syz-declextract
Linux Kernel (for testing purposes)
export KERNEL=$PWD/linux
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $KERNEL
cd $KERNEL
make CC=clang defconfig
./scripts/config -e FTRACE_SYSCALLS
make CC=clang olddefconfig
make CC=clang -j`nproc` vmlinux compile_commands.json # kernel has to be built at least once for the script to workRunning on a single source file
./bin/syz-declextract $KERNEL/fs/read_write.c | less # or any other .c fileCoverage Data
Coverage data (coverage.jsonl) can be obtained from syzbot dashboard using:
curl --header "accept-encoding: gzip" https://syzkaller.appspot.com/upstream/coverage?jsonl=1 | gunzip > coverage.jsonlNote: the coverage is tied to a particular kernel commit. For consistency that commit
should be used for the rest of the process as well.
Running on the whole kernel
go run ./tools/syz-declextract -config=manager.cfg -coverage coverage.jsonl
syz-env make extract SOURCEDIR=$KERNELThe tool caches results of static kernel analysis in manager.workdir/declextract.cache file,
and results of the dynamic kernel probing in manager.workdir/interfaces.json file.
These can be examined for debugging purposes, and will be reused in future runs if exist
(greatly saves time). If the clang tool/kernel has changed, delete these cache files
so that they are updated.
Overview (`tools/syz-uncovered-batch/README.md`)
`syz-uncovered-batch` - Uncovered Target Generator
syz-uncovered-batch extracts candidate uncovered functions and code locations from Syzkaller dashboard coverage data, and generates batch task input files for syz-aflow.
Building
To build syz-uncovered-batch, run:
./tools/syz-env go build ./tools/syz-uncovered-batchUsage
See main.go (or run ./syz-uncovered-batch -help) for all available flags.
1. Prepare Base Configuration
Create a base template base.json defining the VM, kernel source, and image configuration for the target environment:
{
"AgentName": "syzbot",
"Syzkaller": "/path/to/syzkaller",
"Image": "/path/to/image",
"KernelConfig": "@/path/to/linux.config",
"TargetOS": "linux",
"TargetArch": "amd64",
"Snapshot": true,
"Type": "qemu",
"VM": {
"count": 1,
"cpu": 2,
"mem": 2048,
"qemu_args": "-machine q35 -enable-kvm -smp 2,sockets=2,cores=1"
}
}2. Generate Tasks
Extract uncovered targets for a specific manager and subsystems (e.g. KVM and DRM) and generate 50 task files:
./tools/syz-env ./syz-uncovered-batch \
-manager ci-upstream-kasan-gce-root \
-base base.json \
-output-dir ./tasks \
-include-paths virt/kvm,drivers/gpu/drm \
-limit 503. Run Batch with `syz-aflow`
Execute the generated tasks in parallel using syz-aflow:
./tools/syz-env ./syz-aflow \
-workflow seed-gen-file-line \
-input ./tasks \
-workdir ./workdir \
-parallel 4 \
-corpusResults and trajectories will be saved to ./workdir/trajectories/{success,giveup,error}/ and executed programs collected into ./workdir/corpus.db.
Overview (`dashboard/app/dashboard/README.md`)
This dir exists for the purposes of running the dashboard app locally using dev_appserver.py.
The production app sets CWD to the root of the syzkaller repository, so paths that refer to
static resources in app.yaml look like dashboard/app/static/. However, dev_appserver.py
sets CWD to dashboard/app, so these paths do not work. This dir contains soft link
dashboard/app/dashboard/app/static that point to dashboard/app/static, so that
dashboard/app/static/ paths work from both locations.
Overview (`dashboard/config/linux/README.md`)
This dir contains Linux kernel configs used by syzbot.
Configs are auto-generated by syz-kconf utility from declaration file main.yml
and config fragments. main.yml specifies list of target config files (instances) and theirfeatures, as well as order of inclusion of config fragments and their constraints. Config fragment is used for
an instance if all fragment constraints are present in instance features and all negative constraints (-foo) are not
present in instance features.
For example, the following declaration file:
instances:
- instance-a: [x86_64, gcc, upstream]
- instance-b: [x86_64, gcc, upstream, foo]
- instance-c: [arm64, clang, stable]
includes:
- always.yml: []
- only-upstream.yml: [upstream]
- only-clang-arm64.yml: [clang, arm64]
- only-upstream-not-foo.yml: [upstream, -foo]specifies 3 target configs (instances) and built from 4 config fragments. Here fragment always.yml is included first
and for all instances. Fragment only-upstream.yml is included next for instances that have upstream feature listed
(instance-a and instance-b). Fragment only-clang-arm64.yml is included next for instances that have bothclang and arm64 listed (instance-c). Finally, fragment only-upstream-not-foo.yml is included for instances
that have upstream features but don't have foo feature listed (instance-a).
Config fragments can specify several aspects of the target config file. First, they can specify the kernel repository
and revision that will be used for config generation using the following syntax. There must be only one repository
specification for each instance.
kernel:
repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
tag: v5.8-rc7Then, config fragments can specify kernel make commands that will be invoked to create the base config.
For example:
shell:
- make x86_64_defconfig
- make kvm_guest.configShell commands can have constraints similar to includes, for example:
shell:
- make defconfig: [-nodefconfig]Then, fragments can specify text for verbatim inclusion at the end of the resulting config. syz-kconf does not do
anything with these string. For example:
verbatim: |
CONFIG_THIS_CONFIG_DOES_NOT_EXIST_YET=yFinally, fragments can specify individual config options that need to be enabled (or disabled) in the resulting config.
For example:
config:
- CONFIG_WILL_BE_ENABLED
- CONFIG_WILL_BE_DISABLED: n
- CONFIG_WITH_INTEGER_VALUE: 10
- CONFIG_WITH_STRING_VALUE: "foo"Config options can have constraints for more precise control over target instances
(constraints can be combined with values):
config:
- CONFIG_ONLY_FOR_UPSTREAM: [upstream]
- CONFIG_STABLE_AND_ARM64: [stable, arm64]
- CONFIG_NOT_CLANG: [-clang]
- CONFIG_DISABLED_FOR_GCC: [n, gcc]
- CONFIG_STRING_FOO: ["value", foo]Instance features:
nonoise: disables auxiliary debug configs (lockdep, stalls, object debug, etc);
used on instances on which we ignore all crashes except for the main debug tool (KMSAN/KCSAN/LEAK)
and on instances that are too slow (some qemu emulation).reduced: disables some generic (mostly arch-independent) kernel subsystems;
used on slow qemu emulation instances since these subsystems are well testing on fast native instances.syz-kconfknows about this feature and gives it special meaning: all configs in fragments enabled
by this feature are disabled for reduced instances.
Some features/constraints are defined by syz-kconf itself or have special meaning:
overrideon a config allows to override a previous definition of the same config (otherwise it results in an error)optionalon a config disables checking that the resulting kernel config file has the specified value for this
config (otherwise it results in an error)weakmeans bothoverrideandoptionalclang/gccinstance feature controls what compiler is used for config generation- kernel arch name (
x86_64,arm64, etc) instance feature controls architecture of the kernel config - kernel release tag is automatically added to instance features and allows to specify a minimum kernel version
for a config (e.g.v5.5for a config added in that kernel release) or a maximum kernel version (e.g.-v5.5for
a config removed in that kernel release)
syz-kconf generates 2 kernel config files for each instance: one normal (or full) and one baseline.
The baseline config has -base suffix and has baseline and base-config features added. The baseline
config is meant to not contain majority of subsystem configs enabled (should just boot and have debugging
configs enabled), however, that is fully controlled by config fragments based on the presence of the baseline
feature. In order to facilitate even smaller -base.config files, the base-config feature is automatically
set for them. Such configs are intended to be used by the config minimization procedure.
The only exception to the declarative nature of the process is USB configs. They are added procedurally, seeaddUSBConfigs function for details.
syz-kconf switches between gcc and clang compilers automatically and is supposed to be used with
syz-env, which provides correct versions of the compilers.
To update kernel configs:
- update kernel version in linux-next.yml and linux-upstream.yml
- change config fragments as necessary (e.g. add additional configs to subsystems.yml)
- run
./tools/syz-env make configs SOURCEDIR=/path/to/existing/linux/checkout- note: it will be mrproper-ed and a number of remotes will be added
- see this on how to setup/use
syz-env
- resolve problems. You can iterate faster focusing on the specific instance problem.
./tools/syz-env make configs SOURCEDIR=/path/to/existing/linux/checkout INSTANCE=PROBLEM_INSTANCE_NAME
- check in config fragments and changed kernel configs and send a PR
- changes will be deployed to
syzbotwithin a day after merging
Overview (`executor/_include/flatbuffers/README.md`)
This was manually vendored on the version v23.5.26
(which matches the compiler version in the env container)
using the following commands:
git clone --branch=v23.5.26 --depth=1 --single-branch https://github.com/google/flatbuffers.git
cp flatbuffers/LICENSE syzkaller/executor/_include/flatbuffers
cp flatbuffers/include/flatbuffers/*.h syzkaller/executor/_include/flatbuffersOverview (`syz-cluster/tools/db-mgmt/README.md`)
How to run DB migrations locally:
cd $SYZKALLER/syz-cluster
make build-db-mgmt-dev restart-spanner
./run-local.sh db-mgmt migrateOverview (`syz-cluster/workflow/triage/README.md`)
Ephemeral disk size usage estimations for Linux
$ git fetch origin +refs/tags/torvalds-head:refs/tags/torvalds-head --depth=1
$ du -h .
260M .
$ git checkout torvalds-head
$ du -h .
2G .
$ git fetch origin 2dde18cd1d8fac735875f2e4987f11817cc0bc2c --depth=1
$ du -h .
2.1G .
$ git checkout 2dde18cd1d8fac735875f2e4987f11817cc0bc2c
$ du -h .
2G .Without cloning the repository
mkdir ~/shallow-repo ~/shallow-repo/.git ~/shallow-repo/workdir ~/overlayfs
mount -t tmpfs -o size=128M tmpfs /root/overlayfs
mkdir ~/overlayfs/upper ~/overlayfs/work
mount -t overlay overlay -o lowerdir=/kernel-repo,upperdir=/root/overlayfs/upper,workdir=/root/overlayfs/work /root/shallow-repo/.git
git --git-dir=/root/shallow-repo/.git --work-tree=/root/shallow-repo/workdir checkout masterNeeds:
securityContext:
capabilities:
add: ["SYS_ADMIN"]Overview (`tools/syz-declextract/testdata/README.md`)
This dir contains sources of a fake kernel that resembles Linux for testing of the syz-declextract tool.
For each *.c file there 3 golden files:
*.c.jsonwith the expected output of the clang tool*.c.txtwith the expected syzlang descriptions*.c.infowith the expected kernel interface list
Testing is done by tools/syz-declextract tests.
TestClangTool invokes the clang tool and verifies *.c.json contents.
The test requires the clang tool binary specified in the -bin, otherwise it skips testing.
You also want to run it with -count=1 flag since the Go tool does not detect changes in the tool binary,
and will cache and reuse test results:
go test -count=1 ./tools/syz-declextract -bin=llvm/build/bin/syz-declextractTestDeclextract verifies *.c.txt and *.c.info using *.c.json files as inputs
(it does not involve the clang tool and runs always).
All tests also support -update flag, which updates the golden files.
Generally you don't need to update them manually.
Since the test covers multiple packages, it's useful to run coverage as follows:
go test -count=1 -coverprofile=/tmp/cover -coverpkg="github.com/google/syzkaller/tools/syz-declextract,github.com/google/syzkaller/pkg/declextract,github.com/google/syzkaller/pkg/clangtool" ./tools/syz-declextract -bin=llvm/build/bin/syz-declextract
go tool cover -html /tmp/coverOverview (`dashboard/config/openbsd/overlays/README.md`)
File hierarchies to lay on top of default images
pkg/build/openbsd.go replaces the files in the image generated bycreate-openbsd-vmm-worker.sh with the contents of ci-openbsd-multicore/.
A minor bit of missing connection between userspaceDir and this
location is managed outside this repository by the local filesystem
state on the ci-openbsd machine.
Overview (`docs/linux/syzlang/skills/README.md`)
Syzlang Subsystem Skills (Linux)
This directory contains component and subsystem-specific knowledge and
constraints for LLM agents generating or fixing Linux syzlang programs.
Format
Each skill must be a Markdown (.md) file starting with standard YAML
frontmatter defining name and description:
---
name: kvm
description: KVM Virtualization and Guest Constraints (x86/amd64 Focus)
---
# KVM Virtualization and Guest Constraints (x86/amd64 Focus)
... instructions ...How Agents Use Skills
In AFlow, available skills are dynamically listed in SkillsPrompt(osTarget).
Agents can inspect any skill using read-syz-spec with either a short name
(kvm.md or skills/kvm.md) or full relative path
(docs/linux/syzlang/skills/kvm.md).
Overview (`pkg/ifuzz/arm64/gen/json/README.md`)
arm64.json is taken from the Go Language repository:
https://tip.golang.org/src/cmd/vendor/golang.org/x/arch/arm64/arm64asm/inst.json
Please see the LICENSE file (taken from https://tip.golang.org/src/cmd/vendor/golang.org/x/arch/LICENSE)
for the relevant copyright notice and disclaimers.
Adding New Os Support (`docs/adding_new_os_support.md`)
Adding new OS support
Here are the common parts of syzkaller to edit in order to make syzkaller support a new OS kernel. However, there may be some specific changes that will be required for a given kernel (for example, gathering coverage from a given kernel, or some errors that might pop up and give a hint about what to tweak).
syz-executor
For each OS, there is this file executor/executor_GOOS.h where GOOS is the OS name. This file contains two important functions:
os_initwhich is responsible for mapping a virtual address space for the calling process,execute_syscallwhich is responsible for executing system calls for a particular OS kernel.
These two functions, are called in executor/executor.cc, which is mainly responsible for executing the syscalls programs, and managing the threads in which the programs run.
executor_GOOS.h also contains functions related to that operating system such as functions that allow it to gather coverage information, detect bitness, etc. (Example: executor_linux.h ).
The intended function will be called according to the target kernel as defined by the macros in the executor/executor.cc file.
Build files `pkg/`
The OS name is added to
pkg/build/build.goalong with the supported architectureCreating a file that builds the image for the targeted kernel under
pkg/build/. This file contains functions for configuring the build of the bootable image, for building it, and for generate SSH keys which will be used by Syzkaller in order to access the VM. There is a file per each of the supported OSes by Syzkaller where the name pattern isGOOS.go.Adding the given target to the
Makefile.
Report files `pkg/report/`
Creating a file that reports build errors for the targeted kernel under pkg/report/. There is a file per each of the supported OSes by Syzkaller where the name pattern is GOOS.go.
Editing `pkg/host/`
- implement
isSupportedfunction that returns true for a supported syscall, it is located underpkg/host/GOOS.
Creating a file under `sys/GOOS/`
Creating a file init.go for the targeted kernel under sys/GOOS/that included the function initTarget that initializes the target and the different supported architectures.
Editing `sys/syz-extract`
Adding the new kernel name with already existing supported kernels to the file sys/syz-extract/extract.go.
Editing `sys/targets`
Adding the new kernel name with already existing supported kernels to the file targets.go which is located undersys/targets.
Editing `vm/qemu`
Adding the new kernel name with already existing supported kernels to the file qemu.go which is located under vm/qemu.
Syzkaller description & pseudo-syscalls
Check descriptions, and pseudo-syscalls.
Bisect (`docs/bisect.md`)
Syz-bisect
syz-bisect program can be used to bisect culprit and fix commits for
crashes found by syzkaller. It can also identify configuration options
that are triggers for the crash.
Usage
Build syz-bisect with make bisect.
During bisection different compilers depending on kernel revision are
used. These compilers are available
here.
Install ccache to speed up kernel compilations during bisecton.
Create user-space (chroot) using create-image.sh
Create a config file with following lines adjusted for your environment:
{
"bin_dir": "/home/syzkaller/bisect_bin",
"ccache": "/usr/bin/ccache",
"compiler_type": "gcc",
"kernel_repo": "git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git",
"kernel_branch": "master",
"syzkaller_repo": "https://github.com/google/syzkaller",
"userspace": "/home/syzkaller/image/chroot",
"kernel_config": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream-apparmor-kasan.config",
"kernel_baseline_config": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream-apparmor-kasan-base.config",
"syzctl": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream.sysctl",
"cmdline": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream.cmdline",
"manager":
{
"name" : "bisect",
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "/home/syzkaller/workdir",
"kernel_obj": "/home/syzkaller/linux",
"image": "/home/syzkaller/workdir/image/image",
"sshkey": "/home/syzkaller/workdir/image/key",
"syzkaller": "/home/syzkaller/go/src/github.com/google/syzkaller_bisect",
"procs": 8,
"type": "qemu",
"kernel_src": "/syzkaller/linux",
"vm": {
"count": 4,
"kernel": "/home/syzkaller/linux/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048,
"cmdline": "root=/dev/sda1 rw console=ttyS0 kaslr crashkernel=512M minnowboard_1:eth0::: security=none"
}
}
}And run bisection with bin/syz-bisect -config vm_bisect.cfg -crash /syzkaller/workdir/crashes/03ee30ae11dfd0ddd062af26566c34a8c853698d.
Syz-bisect is expecting finding repro.cprog or repro.prog in given
crash directory. It will also utilize repro.opts, but it's not
mandatory.
Additional Arguments
-syzkaller_commit use this if you want to use specific version of syzkaller
-kernel_commit kernel commit where crash is known to reproduce. You
want to use this when bisecting fixing commit
-fix use this if you want to bisect a fixing commit.
Output
It takes some time, but after syz-bisect completes it dumps out it's
results into console It also stores results into files in given crash
directory:
cause.commit commit identified causing the crash or text "the crash
already happened on the oldest tested release"
fix.commit commit identified fixing the crash or text "the crash
still happens on HEAD"
cause.config config options identified working as one trigger for the crash
original.config, baseline.config, minimized.config config files used
in config bisection
Ci (`docs/ci.md`)
Continuous integration fuzzing
syz-ci command provides support for continuous fuzzing with syzkaller.
It runs several syz-manager's, polls and rebuilds images for managers and polls
and rebuilds syzkaller binaries.
Configuration (`docs/configuration.md`)
Configuration
The operation of the syzkaller syz-manager process is governed by a
configuration file, passed at invocation time with the -config option.
This configuration can be based on the example;
the file is in JSON format and contains the the following parameters.
Contributing (`docs/contributing.md`)
How to contribute to syzkaller
If you want to contribute to the project, feel free to send a pull request following the guidelines below.
In case this is your first pull request to syzkaller, you will need to sign Google CLA
and add yourself to AUTHORS AND CONTRIBUTORS files in the first commit.
What to work on
Extending/improving system call descriptions is always a good idea.
Unassigned issues from the bug tracker are worth doing, but some of them might be complicated.
To contribute code or syscall descriptions, at the very least you need to be able to build and run syzkaller, see the instructions here.
Guidelines
If you want to work on something non-trivial, please briefly describe it on the
[email protected] mailing list first,
so that there is an agreement on the high level approach/design and no duplication of work between contributors.
Split large changes into smaller, logically cohesive commits. Small commits are much easier and faster to review and iterate on.
Everything that can be reasonably tested should be tested.
Provide enough documentation for other users to use the new feature.
Keep the style of the code, tests, comments, docs, log/error messages consistent with the existing style.
Continuous Integration (CI) system runs a number of tests and some [opinionated] style checks. They need to pass.
You can test locally with make presubmit, if you don't have some prerequisites installed,
you may use syz-env (see below).
Commits
Commit messages should follow the following template:
dir/path: one-line description
<empty line>
Extended multi-line description that includes
the problem you are solving and how it is solved.dir/path is a relative path to the main dir this commit changes
(look at examples in the commit history).
If several packages/dirs are significantly affected, then the following format is allowed:
dir1/path1, dir2/path2: one-line descriptionThough, dirs should not be included if they have only minor changes.
For pervasive changes the following format is allowed:
all: one-line descriptionPlease pay attention to punctuation. In particular:
one-line descriptionshould not start with a Capital letter.- There is no dot at the end of
one-line description. Extended multi-line descriptionis full English sentences with Capital letters and dots.
Commit message line length is limited to 120 characters.
Also:
- If your commit fixes an issue, please include
Fixes #NNNline into commit message
(whereNNNis the issue number). This will auto-close the issue. If you need to mention
an issue without closing it, addUpdate #NNN. - For syscall descriptions
*.constfiles are checked-in with the*.txtchanges
in the same commit.
Pull requests
- Rebase your working branch onto the master branch before sending a pull request to avoid merge conflicts.
- Run
make presubmitand ensure that it passes before sending a PR.
This may require some additional packages to be installed (trysudo make install_prerequisites). - Provide a brief high-level description in the pull request title.
The pull request text is mostly irrelevant, all the details should be in the commit messages. - If you're asked to add some fixes to your pull request, please squash the fixes into the old commits.
How to create a pull request on Github
First, you need your own git fork of syzkaller repository. Navigate to
github.com/google/syzkaller and pressForkbutton in the top-right corner of
the page. This will createhttps://github.com/YOUR_GITHUB_USERNAME/syzkallerrepository.Checkout main syzkaller repository if you have not already. The simplest way to do it is to run
git clone https://github.com/google/syzkaller, this will checkout
the repository in the current working directory.Remember to
export PATH=$GOPATH/bin:$PATHif you have not already.Then add your repository as an additional origin:
shellcd syzkaller git remote add my-origin https://github.com/YOUR_GITHUB_USERNAME/syzkaller.git git fetch my-origin git checkout -b my-branch my-origin/master
This adds git origin my-origin with your repository and checks out new branch my-branch based on master branch.
- Change/add files as necessary.
- Commit changes locally. For this you need to run
git addfor all changed files, e.g.git add sys/linux/sys.txt. You can rungit statusto see what files were changed/created. When all files are added (git statusshows no files inChanges not staged for commitsection and no relevant files inUntracked filessection), rungit commitand enter commit description in your editor. - Run tests locally (
make install_prerequisitesfollowed bymake presubmit). - Push the commit to your fork on github with
git push my-origin my-branch. - Navigate to github.com/google/syzkaller and you should see green
Compare & pull requestbutton, press it. Then pressCreate pull request. Now your pull request should show up on pull requests page. - If you don't see
Create pull requestbutton for any reason, you can create pull request manually. For that navigate to pull requests page, pressNew pull request, thencompare across forksand choosegoogle/syzkaller/masteras base andYOUR_GITHUB_USERNAME/syzkaller/my-branchas compare and pressCreate pull request. - If you decided to rebase commits in
my-branch(e.g. to rebase them onto updated master) after you created a pull-request, you will need to do a force push:git push -f my-origin my-branch.
Using syz-env
Developing syzkaller requires a number of tools installed (Go toolchain, C/C++ cross-compilers, golangci-lint, etc).
Installing all of them may be cumbersome, e.g. due to broken/missing packages.
syz-env provides a working hermetic development environment based on a Docker container.
If you don't yet have Docker installed, see documentation,
in particular regarding enabling sudo-less
Docker (Googlers see go/docker).
It's recommended to create an alias for syz-env script:
alias syz-env="$(go env GOPATH)/src/github.com/google/syzkaller/tools/syz-env"Then it can be used to wrap almost any make invocation as:
syz-env make format
syz-env make presubmit
syz-env make extract SOURCEDIR=$(readlink -f ~/linux)Or other commands/scripts, e.g.:
syz-env go test -short ./pkg/csourceOr you may run the shell inside of the container with just syz-env and look around.
To update syz-env container to the latest version do:
docker pull gcr.io/syzkaller/envYou can also build the container from the respective Dockerfile by setting the SYZ_ENV_BUILD environment variable, i.e.:
SYZ_ENV_BUILD=1 syz-envThis can be useful to test local changes that have not been pushed to the registry yet.
Using [act](https://github.com/nektos/act)
.github/workflows has more tests compared to syz-env make presubmit. To have the same tests as the workflow, we can run these workflow jobs locally.
# install act
make act
# list all jobs
bin/act -l
# run all jobs
bin/act
# run job with name build
bin/act -j buildCoverage (`docs/coverage.md`)
Coverage
syzkaller uses sanitizer coverage (tracing mode)
and KCOV for coverage collection.
Sanitizer coverage is also supported by gcc and KCOV is supported by some other OSes.
Note: gVisor coverage is completely different.
Coverage is based on tracing coverage points inserted into the object code by the compiler.
A coverage point generally refers to a basic block of code
or a CFG edge
(this depends on the compiler and instrumentation mode used during build,
e.g. for Linux and clang the default mode is CFG edges, while for gcc the default mode is basic blocks).
Note that coverage points are inserted by the compiler in the middle-end after a significant number
of transformation and optimization passes. As the result coverage may poorly relate to the source code.
For example, you may see a covered line after a non-covered line, or you may not see a coverage point
where you would expect to see it, or vice versa (this may happen if the compiler splits basic blocks,
or turns control flow constructs into conditional moves without control flow, etc).
Assessing coverage is still generally very useful and allows to understand overall fuzzing progress,
but treat it with a grain of salt.
See this for Linux kernel specific coverage information.
Web Interface
When clicking on cover link you get view showing each directory located in your kernel build directory. It's showing either percentage number X% of N or ---. X% of N means that X% of N coverage points are covered so far, . --- indicates there is no coverage in that directory.
Directory can be clicked and you get view on files and possible subdirectories. On each source code file there is again either --- or coverage percentage.
If you click on any C files you will get source code view. There is certain coloring used in the source code view. Color definitions can be found in coverTemplate. Coloring is described below.
If you click on percentage number of any listed source file you will get cover percentage for each function in that source file.
Covered: black (#000000)
All PC values associated to that line are covered. There is number on the left side indicating how many programs have triggered executing the PC values associated to this line. You can click on that number and it will open last executed program. Example below shows how single line which is fully covered is shown.
Both: orange (#c86400)
There are several PC values associated to the line and not all of these are executed. Again there is number left to the source code line that can clicked to open last program triggering associated PC values. Example below shows single line which has both executed and non-executed PC values associated to it.
Uncovered: red (#ff0000)
Line is uncovered. Function (symbol) this line is in is executed and one of the PC values associated to this line. Example below shows how single line which is not covered is shown.
Not executed or non-instrumented code: grey (#505050)
If an entire function was never executed, all its lines are shown in grey. This color is also used for lines that are not instrumented or do not generate code (e.g., comments, macros, or empty lines).
Note that if a function was executed but some lines within it were not covered, those lines are shown in red, not grey.
syz-cover
There is small utility in syzkaller repository to generate coverage report based on raw coverage data. This is available in syz-cover and can be built by:
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o ./bin/syz-cover github.com/google/syzkaller/tools/syz-coverRaw coverage data can be obtained from running syz-manager by:
wget http://localhost:<your syz-manager port>/rawcoverNow this raw cover data can be fed to syz-cover to generate coverage report:
./bin/syz-cover --config <location of your syzkaller config> ./rawcoverYou can also export CSV file containing function coverage by:
./bin/syz-cover --config <location of your syzkaller config> --exports funccover ./rawcoverYou can export a JSON file containing line coverage info by:
./bin/syz-cover --config <location of your syzkaller config> --exports json ./rawcoverDb (`docs/db.md`)
syz-db
syz-db program can be used to manipulate corpus.db databases that are used
by syz-managers.
Build
Build syz-db with make db or by changing to tools/syz-db and run go build.
Options
syz-db currently overs the following generic arguments:
-arch string
target arch
-os string
target OS
-version uint
database version
-vv int
verbosityThat can be used with
syz-db pack dir corpus.dbto pack a database
syz-db unpack corpus.db dirto unpack a database. A file containing performed syscalls will be returned.
syz-db merge dst-corpus.db add-corpus.db* add-prog*to merge databases. No additional file will be created: The first file will be replaced by the merged result.
syz-db bench corpus.dbto run a deserialization benchmark. For example:
syz-db -os=linux -arch=amd64 bench corpus.dbcould give an output like
allocs 123 MB (123 M), next GC 123 MB, sys heap 123 MB, live allocs 123 MB (123 M), time 324s.Headerparser Usage (`docs/headerparser_usage.md`)
Headerparser
headerparser is a tool that assists in writing device system call descriptions for syzkaller.
In order to make syzkaller smarter when it comes to fuzzing a device node, you can provide it with
information about the ioctl argument struct types it expects.
However, in certain cases the number of argument struct types might be high, increasing the amount of manual
effort that goes into writing the description files for the struct types.
In order to ease the effort of writing ioctl argument type description files, headerlib does a best-effort job at
generating them for you. You will still need to manually select the appropriate syzkaller data type from the list
of types here.
Dependencies
Headerlib uses pycparser. You can install pycparser using pip.
$ pip install pycparserUsing headerparser
$ python headerparser.py --filenames=./test_headers/th_b.h
B {
B1 len|fileoff|flags|intN #(unsigned long)
B2 len|fileoff|flags|intN #(unsigned long)
}
struct_containing_union {
something len|fileoff|flags|int32 #(int)
a_union.a_char ptr[in|out, string]|ptr[in, filename] #(char*)
a_union.B_ptr ptr|buffer|array #(struct B*)
}You can copy paste the content underneath the Structure Metadata over to your syzkaller device description.
Something breaks
Let us try parsing test_headers/th_a.h header file to generate argument structs.
$ python headerparser.py --filenames=./test_headers/th_a.h
ERROR:root:HeaderFilePreprocessorException: /tmp/tmpW8xzty/source.o:36:2: before: some_type
$ python headerparser.py --filenames=./test_headers/th_a.h --debug
DEBUG:GlobalHierarchy:load_header_files : ['./test_headers/th_a.h']
DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor._mktempfiles: sourcefile=/tmp/tmpbBQYhR/source.cobjectfile=/tmp/tmpbBQYhR/source.o
DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor.execute: cp ./test_headers/th_a.h /tmp/tmpbBQYhR
DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor.execute: gcc -I. -E -P -c /tmp/tmpbBQYhR/source.c > /tmp/tmpbBQYhR/source.o
ERROR:root:HeaderFilePreprocessorException: /tmp/tmpbBQYhR/source.o:36:2: before: some_typeFrom the error message, we can see that the error occurs as pycparser is not aware of the type some_type. We can resolve this by making pycparser aware of the unknown type. In order to do this, we supply headerparser with a include file that contains C declarations and includes that can fix the parse error.
$ cat > include_file
typedef int some_type;
$ python headerparser.py --filenames=./test_headers/th_a.h --include=./include_file
A {
B_item ptr|buffer|array #(struct B*)
char_ptr ptr[in|out, string]|ptr[in, filename] #(char*)
an_unsigned_int len|fileoff|int32 #(unsigned int)
a_bool _Bool #(_Bool)
another_bool _Bool #(_Bool)
var some_type #(some_type)
}Hub (`docs/hub.md`)
Connecting several managers via Hub
syz-hub program can be used to connect several syz-manager's together and
allow them to exchange programs.
Build syz-hub with make hub. Then create a config file along the lines of:
{
"http": ":80",
"rpc": ":55555",
"workdir": "/syzkaller/workdir",
"clients": [
{"name": "manager1", "key": "6sCFsJVfyFQVhWVKJpKhHcHxpCH0gAxL"},
{"name": "manager2", "key": "FZFSjthHHf8nKm2cqqAcAYKM5a3XM4Ao"},
{"name": "manager3", "key": "fTrIBQCmkEq8NsvQXZiOUyop6uWLBuzf"}
]
}And start it with bin/syz-hub -config hub.cfg. Then add the following
additional parameters to syz-manager config files of each manager:
"name": "manager1",
"hub_client": "manager1",
"hub_addr": "1.2.3.4:55555",
"hub_key": "6sCFsJVfyFQVhWVKJpKhHcHxpCH0gAxL",And start managers. Once they triage local corpus, they will connect to the hub
and start exchanging inputs. Both hub and manager web pages will show how many
inputs they send/receive from the hub.
Internals (`docs/internals.md`)
How syzkaller works
Generic description of how syzkaller works are below.
Linux kernel specific internals can be found here.
Overview
The process structure for the syzkaller system is shown in the following diagram;
red labels indicate corresponding configuration options.
syz-manager is responsible for:
- Starting/restarting/monitoring VM instances.
- The actual fuzzing process (input generation, mutation, minimization, etc.).
- Persistent corpus and crash storage.
It runs on a host with a stable kernel which does not experience white-noise fuzzer load.
syz-manager starts syz-executor processes (one inside each VM).syz-executors communicate with syz-manager over RPC to receive the programs
that must be executed and to report back the results (error statuses, collected coverage, etc.).
To execute programs, syz-executor starts transient subprocesses.
Each transient subprocess executes a single input (a sequence of syscalls).
It is designed to be as simple as possible (to not interfere with fuzzing process),
written in C++, compiled as static binary and uses shared memory for communication.
Syscall descriptions
The syz-manager process generates programs based on syscall descriptions described here.
Coverage
Syzkaller is a coverage-guided fuzzer. The details about coverage collection can be found here.
Crash reports
When syzkaller finds a crasher, it saves information about it into workdir/crashes directory.
The directory contains one subdirectory per unique crash type.
Each subdirectory contains a description file with a unique string identifying the crash (intended for bug identification and deduplication);
and up to 100 logN and reportN files, one pair per test machine crash:
- crashes/
- 6e512290efa36515a7a27e53623304d20d1c3e
- description
- log0
- report0
- log1
- report1
...
- 77c578906abe311d06227b9dc3bffa4c52676f
- description
- log0
- report0
...Descriptions are extracted using a set of regular expressions.
This set may need to be extended if you are using a different kernel architecture, or are just seeing a previously unseen kernel error messages.
logN files contain raw syzkaller logs and include kernel console output as well as programs executed before the crash.
These logs can be fed to syz-repro tool for crash location and minimization,
or to syz-execprog tool for manual localization.reportN files contain post-processed and symbolized kernel crash reports (e.g. a KASAN report).
Normally you need just 1 pair of these files (i.e. log0 and report0), because they all presumably describe the same kernel bug.
However, syzkaller saves up to 100 of them for the case when the crash is poorly reproducible, or if you just want to look at a set of crash reports to infer some similarities or differences.
There are 3 special types of crashes:
no output from test machine: the test machine produces no output whatsoeverlost connection to test machine: the ssh connection to the machine was unexpectedly closedtest machine is not executing programs: the machine looks alive, but no test programs were executed for long period of time
Most likely you won't see reportN files for these crashes (e.g. if there is no output from the test machine, there is nothing to put into report).
Sometimes these crashes indicate a bug in syzkaller itself (especially if you see a Go panic message in the logs).
However, frequently they mean a kernel lockup or something similarly bad (here are just a few examples of bugs found this way:
1,
2,
3).
Issue Template (`docs/issue_template.txt`)
Issues are for bugs in syzkaller itself, for everything else please contact [email protected] mailing list.
Before filing an issue please check the existing issues and the mailing list (https://groups.google.com/forum/#!forum/syzkaller).
Please provide syzkaller revision, Go version (output of go version), relevant information about host and target systems.
If syz-manager is not working overall, please provide output of syz-manager with -debug flag.
Kernel Config Update Guidelines (`docs/kernel_config_update_guidelines.md`)
Updating Syzkaller Kernel Configurations Guidelines
These guidelines describe the process for updating the syzkaller dashboard configuration files to support new Linux kernel versions.
Objective
Successfully update configuration fragments (.yml) and generate corresponding .config files for target kernel versions (e.g., linux-upstream and linux-next).
Principles
- Maximize Coverage: We want to maximize the coverage of the new code and maintain the coverage of the old code.
- Enable New Options: If new options are introduced with the kernel version, we want them to be enabled if they help to shift left the bug detection or directly help to detect bugs. If they help with fuzzing but were not enabled, explain why in the commit message. These changes should be grouped into a commit titled
dashboard/config/linux: enable more configs. - Reasoned Disabling: If some options are not available anymore, disable them adding a comment in the YAML file referencing the kernel commit that removed, renamed, or moved them. Prefer formats like:
# Removed in <commit_hash> "<commit_title>",# Renamed to <NEW_CONFIG> in <commit_hash> "<commit_title>", or# Migrated to <NEW_LOCATION> in <commit_hash> "<commit_title>". If the specific commit cannot be found after a reasonable search, use the format:# Removed in vX.Y (exact commit not found). If the option is not mentioned in any generated config files (i.e., it was not enabled in any instance), you can remove it from the YAML file entirely instead of disabling it. In this case, explain the removal in the commit message. - Documenting Constraints: If you disable an option for a specific architecture (e.g.,
-riscv) or configuration (e.g.,-kmsan), or if you make some config 'weak', add a comment explaining this decision. - Explicit Constraints: Prefer explicitly enumerating the supported platforms and/or kernel versions instead of using
[optional]or[weak]constraints when possible. If the option was added inlinux-next, use the tag[linux-next]. If the specific kernel version is not known but it is in a released kernel, use[weak]and add a comment explaining it. Also use[weak]or[optional]if the config option has dependencies that might not be met in all configurations (e.g., depends on specific networking features or hardware support). Do not silently make a failing configweakwithout investigation. Prefer identifying the specific reason (e.g., architecture limitation or kernel version change) and using explicit exclusions if possible. - File-Level Exclusion: If all options in a configuration fragment are disabled for certain kernel versions or instances, prefer excluding the whole file in
main.ymlusing negative tags (e.g.,-upstream) instead of adding version constraints to every individual option in the fragment. Add an explanatory comment above the inclusion line inmain.ymlto explain why it is disabled. - Explanation Headers: Every new file created in
dashboard/config/linux/bitsrequires an explanation header describing the purpose of the file, why we need it, and how it is used. - No Manual Edits: Do NOT manually update the generated
.configfiles. They MUST always be generated by runningmake configs.
Prerequisites
- A local clone of the
linuxkernel repository. - A local clone of the
syzkallerrepository. - Docker installed for running
syz-env.
Procedure
1. Update Kernel Commits
Modify the corresponding YAML files in dashboard/config/linux/bits/ to point to the desired tags. Use tags instead of commit hashes when updating these files.
linux-upstream.ymllinux-next.yml
2. Run Config Generation
Execute the following command from the root of the syzkaller repository:
CI=true ./tools/syz-env make configs SOURCEDIR=/path/to/linux/sourceNote: Replace /path/to/linux/source with the absolute path to your local Linux kernel directory.
3. Troubleshooting Common Issues
A. Kconfig Parser Limitations (`syz-kconf`)
The syz-kconf parser in pkg/kconfig may not support advanced Kconfig syntax such as if in depends on statements.
- Solution: Implement support for the missing syntax in
pkg/kconfig/parser.go. Do NOT modify Kconfig files in the kernel source.
B. Missing Config Options in Final Config
If an option specified in a YAML fragment is not present in the generated .config, make configs will fail.
- Check Dependencies: Verify if the option depends on another option not enabled, or if it is not supported on the target architecture or kernel version.
- Version Constraints: Add version constraints to the option in the YAML file, e.g.,
[-v7.1]to disable it for versions starting fromv7.1. - Weak Constraints: If the option is optional and may not be present in all versions, make it weak:
[value, weak]. This prevents failure if the option is missing. Avoid usingweakjust to bypass errors. Investigate why the option is missing or failing to be set. If it is due to architecture or version specific behavior, use explicit exclusions (e.g.[-arm]) instead ofweak.
C. Git Fetch Failures
If syz-kconf fails to checkout a repository or commit (e.g., Android kernels):
- Diagnosis: Check the error message to find the auto-generated remote name (often a hash).
- Workaround: Manually run
git fetch <remote_name>in the kernel source directory to pull all objects, then retrymake configs.
4. Finalizing Changes
Once make configs completes successfully:
- Stage and commit the changes to YAML files first.
- Stage and commit the generated
.configfiles as the VERY LAST commit in the PR. The commit message MUST bedashboard/config/linux: update generated configs. - Push to a separate branch and create a Pull Request.
Kfuzztest (`docs/kfuzztest.md`)
KFuzzTest Integration With syzkaller
KFuzzTest, introduced initially in this RFC
is a framework for exposing internal kernel functions to a userspace fuzzing
engine like syzkaller. As the kernel docs put it:
The Kernel Fuzz Testing Framework (KFuzzTest) is a framework designed to
expose internal kernel functions to a userspace fuzzing engine.It is intended for testing stateless or low-state functions that are difficult
to reach from the system call interface, such as routines involved in file
format parsing or complex data transformations. This provides a method for
in-situ fuzzing of kernel code without requiring that it be built as a
separate userspace library or that its dependencies be stubbed out.
This document introduces how syzkaller integrates with KFuzzTest.
Getting Started
Firstly, ensure that the KFuzzTest patch series has been applied to your Linux
tree.
As of the 26th of Semptember 2025, the most up-to-date version can be found in
this Linux Kernel patch series.
Once this is done, KFuzzTest targets can be defined on arbitrary kernel
functions using the FUZZ_TEST macro as described in the kernel docs inDocumentation/dev-tools/kfuzztest.rst.
Configuration Options
Ensure that the following KConfig options are enabled for your kernel image:
CONFIG_DEBUG_FS(used as a communication interface by KFuzzTest).CONFIG_DEBUG_KERNEL.CONFIG_KFUZZTEST.
It is also highly recommended to enable the following KConfig options for
more effective fuzzing.
CONFIG_KASAN(catch memory bugs such as out-of-bounds-accesses).CONFIG_KCOV(to enable coverage guided fuzzing).
Fuzzing KFuzzTest Targets
Syzkaller implements three ways to fuzz KFuzzTest targets:
syz-managerintegration with static targetssyz-managerwith dynamic targetssyz-kfuzztest: a standalone tool that runs inside a VM, discovers KFuzzTest
targets dynamically, and fuzzes them.
1. `syz-manager` with static targets
Configuration for this method is identical to syz-manager, and is designed to
make it easy to integrate KFuzzTest fuzzing into existing continuous fuzzing
deployments.
One must first write a syzlang description for the KFuzzTest target(s) of
interest, for example in /sys/linux/my_kfuzztest_target.txt. Each target
should have the following format:
some_buffer {
buf ptr[inout, array[int8]]
buflen len[buf, int64]
}
kfuzztest_underflow_on_buffer(name ptr[in, string["test_underflow_on_buffer"]], data ptr[in, some_buffer], len bytesize[data], buf ptr[in, array[int8, 65536]]) (kfuzz_test)Where:
- The first argument should be a string pointer to the name of the fuzz target,
i.e,. the name of itsdebugfsinput directory in the kernel. - The second should be a pointer to a struct of the type that the fuzz
target accepts as input. - The third should be the size in bytes of the input argument.
- The call is annotated with attribute
kfuzz_test.
The final buf argument is a buffer of sizeKFUZZTEST_MAX_INPUT_SIZE = 16 * PAGE_SIZE and is used internally to ensure
that enough space is available in a program for the entire flattened input that
is sent into a KFuzzTest target.
For more information on writing syzkaller descriptions attributes, consult the
syscall description and syscall description syntax
documentation files.
To facilitate the tedious task of writing syz_kfuzztest_run descriptions, a
tool (tools/kfuzztest-gen) is provided to automatically generate these from avmlinux binary. One can run the tool and paste the output into a syzlang file.
go run ./tools/kfuzztest-gen --vmlinux=path/to/vmlinuxAfter writing these descriptions to a file under the /sys/linux/ directory
(for example, /sys/linux/my_fuzz_targets.txt), they need to be compiled withmake descriptions.
Finally, the targets can be enabled in syz-manager config file in theenable_syscalls field, e.g.
{
"enable_syscalls": [ "syz_kfuzztest_run$test_underflow_on_buffer" ]
}2. `syz-manager` with dynamic discovery
This feature greatly reduces the amount of setup needed for fuzzing KFuzzTest
targets, by discovering them all dynamically at launch.
This approach is considered less stable than the previous as it involves
generating descriptions for KFuzzTest targets without human input and then
immediately fuzzing them. It does, however, better reflect our intentions for
KFuzzTest: continuously fuzzing the kernel with a dynamically changing set of
targets with little intervention from syzkaller maintainers.
To enable this feature, configure the experimental enable_kfuzztest option in
the manager configuration, which enables all discovered KFuzzTest targets by
default.
{
"enable_kfuzztest": true
}You must also enable pseudo-syscall syz_kfuzztest_run, like so:
{
"enable_syscalls": [
"syz_kfuzztest_run"
],
}IMPORTANT: for dynamic discovery to work, it is essential for the kernel
image pointed to by the manager configuration is built with CONFIG_DWARF4 orCONFIG_DWARF5 enabled, as dynamic target discovery depends on these symbols
being emitted.
3. `syz-kfuzztest`, an in-VM standalone tool
In contrast with syz-manager, syz-kfuzztest is designed to perform coverage
guided fuzzing from within a VM directly rather than orchestrating a fleet of
VMs. It is primarily targeted at development-time fuzzing, rather than long-term
continuous fuzzing.
For more information, consult the syz-kfuzztest documentation.
Maintaining (`docs/maintaining.md`)
Maintainer's Guide
Linear history
We preserve linear history and have merge commits disabled in repository settings.
Pull requests (PR) have only 2 options:
- Rebase and merge
This is preferred if all commits in the PR are properly squashed/fixed already,
as it preserves commits in the PR (takes all commits in the PR as is and adds them
on top of the master HEAD). - Squash and merge
This is preferred if the PR contains fix ups as separate commits and/or other clutter
in commit organization (squashes all commits in the PR into one commit and adds it
on top of the master HEAD, also allows to edit commit subject/description).
PR checks (CI)
cla/google check needs to pass before merging.
CI testing generally needs to pass before merging.
Exceptions may be infrastructure flakes (especially in external services: codecov, ci/fuzzit);
one-off timeouts/OOMs (but not if this PR itself makes them much more frequent).
All static checking warnings and testing errors are considered hard errors.
Tests
Adding tests for new code and bug fixes is generally encouraged. Ask contributors to add tests.
However, some code is easier to test, while some is harder. Some examples of cases where
it's easier to add tests (should be added): abstract functionalities without external dependencies
(e.g. parsers, data transformations, calculations); code with established testing infrastructure
(adding new tests is just adding one more of the same). Examples of cases where it's harder
to add tests (may be not added, but still welcome if one wants to go above and beyond):
code with external dependencies that are not easy to fake out (qemu, kernel, image, etc);
code without established testing infrastructure where adding one test would require building
the whole infrastructure first.
Use your judgement
There are no strict rules for reviews/ownership at the moment. Use your judgement.
If you are maintaining a particular area of the project (e.g. support for one OS),
it is OK to merge your own changes without further review (especially smaller and
if CI gives green light). It's also OK to review and merge changes to other parts
of the project. But loop in other maintainers if you don't feel confident or need
additional feedback/review.
Mcp (`docs/mcp.md`)
syz-agent in MCP mode
Example use with Gemini CLI:
Run
make agent.Create
mcp.configfile, for example:
{
"http": "localhost:59999",
"mcp": true,
"syzkaller_repo": "[email protected]:syzkaller/syzkaller.git",
"target": "linux/amd64",
"image": "/buildroot_amd64-2022.11.clean",
"kernel_config": "upstream-apparmor-kasan.config",
"type": "qemu",
"vm": {
"count": 10,
"cpu": 2,
"mem": 2048,
"cmdline": "root=/dev/sda1"
},
"cache_size": 107374182400
}- Start
syz-agent:
bin/syz-agent -config mcp.config- Add the following to
~/.gemini/settings.json:
{
"mcpServers": {
"syzkaller": {
"httpUrl": "http://127.0.0.1:59999",
"trust": true
}
}
}- Start Gemini CLI:
geminiTest that
geminihas connected tosyz-agentby executing/mcpor/mcp schema.Try the following prompt:
/* Detailed source-code truncated for AI context efficiency. */A manual unit test for the MCP server:
/* Detailed source-code truncated for AI context efficiency. */Program Syntax (`docs/program_syntax.md`)
Program syntax
Syzkaller uses a compact domain-specific language (DSL) for programs
to log executed programs, test its code, and persist programs in the
corpus. This page provides a brief description of the corresponding
syntax. Some useful information can also be found in the
existing examples and in the program
deserialization code.
Together with execution options, the DSL provides everything that
syz-executor needs to run a program.
For example, consider the program:
r0 = syz_open_dev$loop(&(0x7f00000011c0), 0x0, 0x0)
r1 = openat$6lowpan_control(0xffffffffffffff9c, &(0x7f00000000c0), 0x2, 0x0)
ioctl$LOOP_SET_FD(r0, 0x4c00, r1)Each line in this program describes a particular syscall invocation,
with the first two calls saving the result in temporary variables r0
and r1, which are passed to the third call.
line = assignment | call
assignment = variable " = " call
call = syscall-name "(" [arg ["," arg]*] ")" ["(" [call-prop ["," call-prop*] ")"]
arg = "nil" | "AUTO" | const-arg | resource-arg | result-arg | pointer-arg | string-arg | struct-arg | array-arg | union-arg
const-arg = integer
resource-arg = variable ["/" hex-integer] ["+" hex-integer]
result-arg = "<" variable "=>" arg
pointer-arg = "&" pointer-arg-addr ["=ANY"] "=" arg
pointer-arg-addr = "AUTO" | "(" pointer-addr ["/" region-size] ")"
string-arg = "'" escaped-string "'" | "\"" escaped-string "\"" | "\"$" escaped-string "\""
struct-arg = "{" [arg ["," arg]*] "}"
array-arg = "[" [arg ["," arg]*] "]"
union-arg = "@" field-name ["=" arg]
call-prop = prop-name ": " prop-value
variable = "r" dec-integer
pointer-addr = integer
region-size = integer
integer = dec-integer | oct-integer | "0x" hex-integerPrograms may also contain blank lines and comments.
# Obtain a file handle
r0 = openat(0xffffffffffffff9c, &AUTO='./file1\x00', 0x42, 0x1ff)
# Perform a write operation
write(r0, &AUTO="01010101", 0x4)Memory management
Memory management is performed by syzkaller itself. It will allocate
virtual memory regions of the necessary size and set the final values
of pointer arguments.
By using the AUTO keyword, programs can give syzkaller the full
control over storing the data. This may be convenient e.g. when a
parameter must be passed by reference, but the exact location of its
value is not of particular importance.
r1 = syz_genetlink_get_family_id$nl80211(&AUTO='nl80211\x00', 0xffffffffffffffff)
ioctl$sock_SIOCGIFINDEX_80211(r0, 0x8933, &AUTO={'wlan0\x00', <r2=>0x0})Alternatively, some data can be "anchored" to specific addresses. It
may be especially important when a memory region must be shared
between multiple calls. In this case, pointer addresses must be given
at the 0x7f0000000000 offset. Before the actual execution, syzkaller
will adjust pointers to the start of the actual mmap'ed region.
Call properties
Call properties specify extra information about how a specific call
must be executed. Each call within a program has its own set of call
properties. If no properties are provided, syzkaller takes the default
ones.
Currently, syzkaller supports the following call properties.
Fault injection
Syntax: fail_nth: N.
It takes an integer (base 10) argument N. If the argument is
non-negative, a fault will be injected into the N-th occasion.
r0 = openat$6lowpan_control(0xffffffffffffff9c, &(0x7f00000000c0), 0x2, 0x0)
ioctl$LOOP_SET_FD(r0, 0x4c00, r0) (fail_nth: 5)Async
Syntax: async.
Instructs syz-executor not to wait until the call completes and
to proceed immediately to the next call.
r0 = openat(0xffffffffffffff9c, &AUTO='./file1\x00', 0x42, 0x1ff)
write(r0, &AUTO="01010101", 0x4) (async)
read(r0, &AUTO=""/4, 0x4)
close(r0)When setting async flags be aware of the following considerations:
- Such programs should only be executed in threaded mode (i.e.
-threaded
flag must be passed tosyz-executor. - Each
asynccall is executed in a separate thread and there's a
limited number of available threads (kMaxThreads = 16). - If an
asynccall produces a resource, keep in mind that some other call
might take it as input andsyz-executorwill just pass 0 if the resource-
producing call has not finished by that time.
Prometheus (`docs/prometheus.md`)
Prometheus metrics
syz-manager metrics are exposed at the URI /metrics on the http endpoint.
Currently exported prometheus metrics from the manager are syz_exec_total, syz_corpus_cover and syz_crash_total.
These metrics can be ingested using following prometheus client configuration:
scrape_configs:
- job_name: syzkaller
scrape_interval: 10s
static_configs:
- targets:
- localhost:56741Values are reset to zero at syz-manager restart and only reflect for current execution of syz-manager.
Pseudo Syscalls (`docs/pseudo_syscalls.md`)
Pseudo-syscalls
Besides regular system calls, a syscall
description file can also contain
pseudo-syscalls. These are C functions defined in the
executor. When a test program uses a pseudo-syscall, the executor
will generate the pseudo-syscall function code in the resulting C program.
This allows a test program to have specific code blocks to perform
certain actions, they may also be used as more test-friendly wrappers
for primitive syscalls.
Use of pseudo-syscalls is generally discouraged because they ruin all
advantages of the declarative descriptions (declarativeness, conciseness,
fuzzer control over all aspects, possibility of global improvements to
the logic, static checking, fewer bugs, etc), increase maintenance burden,
are non-reusable and make C reproducers longer. However, syzlang is not
expressive enough to cover all possible cases, so use of pseudo-syscalls
needs to be considered on a case-by-cases basis (additional benefit,
amount of code, possibility of extending syzlang to cover this case, etc).
How to add a pseudo-syscall to the executor
First, think about the scope of the pseudo-syscall and which systems and
subsystems it will be related to. The executor includes a fixed set of C
header files executor/common*.h containing the code of the pseudo-syscalls.
Check if the new one can fit in one of the existing files before creating
a new one.
For instance, if our new pseudo-syscall is Linux-specific, then
common_linux.h would be the place to put it.
The actual pseudo-syscall function may look something like this:
#if SYZ_EXECUTOR || __NR_syz_mycall
/* Add all the necessary #include and #define headers */
static long syz_mycall(volatile long a0, volatile long a1)
{
/* Function body */
}
#endifMake sure that all the function requirements are met and that it can
be compiled. Note that the function name must start with "syz_". It may
also take a different number of arguments. Type of arguments must bevolatile long, return type - long. long is required to avoid
potential calling convention issues because it is casted to a function
pointer that accepts long's. The reason for volatile is interesting:
lots of libc functions are annotated with various argument constraints
(e.g. this argument should not be NULL, or that argument must be a
valid file descriptor); C reproducers may call these functions with
constant arguments and compiler may see that some of these constraints
are violated (e.g. passing NULL to a non-NULL argument, or passing-1 as file descriptor) and produce errors/warnings. volatile prevents
that.
Now, to handle the pseudo-syscall properly we have to update thelinuxSyscallChecks in
linux_syscalls.go and add a particular
case for this syscall, enabling it when necessary. If we want to enable
it unconditionally we can simply use alwaysSupported for it.
Finally, run make generate. Now you can use it in a syscall
description file as if it was a regular system call:
syz_mycall(arg0 pid, arg1 const[0])External Dependencies
The implementation must not use any external libraries nor external headers,
except for the most basic and standard ones (like `` and<sys/mman.h>). In particular, it must not depend on libraries/headers
installed by additional packages nor on headers for recently added kernel
subsystems. External dependencies have proved to be brittle and easily cause
build breakage because all dependencies will be required for any build/run on
the fuzzer and any C reproducer. For example, packages/headers may be missing
on some distros, named differently, be of a wrong version, broken, or conflict
with other headers. Unfortunately, there is no way to reliably specify such
dependencies and requirements for C programs. Therefore, if the pseudo-syscall
requires definitions of some structures, constants, or helper functions, these
should be described in the executor code itself as minimally as possible (they
will be part of C reproducers).
Testing
Each new pseudo-syscall should have at least one test in sys/OS/test.
See Linux tests for an example. A tests is just a program
with checked syscall return values. There should be at least one test
that contains "the main successful scenario" of using the pseudo-syscall.
See io_uring test as a good example.
Such tests are important because they ensure that the pseudo-syscall code
does not contain "stupid" bugs (e.g. crash on NULL-deref each time),
that it is possible for the fuzzer to come up with the successful scenario
(as a combination of the pseudo-syscall and the surrounding descriptions)
and that it will continue to work in future.
See Testing of descriptions
for details about the tests.
Pull Request Template (`docs/pull_request_template.txt`)
Before sending a pull request, please review Contribution Guidelines:
https://github.com/google/syzkaller/blob/master/docs/contributing.md
Reproducing Crashes (`docs/reproducing_crashes.md`)
How to reproduce syzkaller crashes
Using a C reproducer
If the bug was reported by syzbot, you first need to build the kernel used by
the tool. Syzbot provides the necessary information in its report:
Hello,
syzbot found the following issue on:
HEAD commit: ae58226b89ac Add linux-next specific files for 20241118
git tree: linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=14a67378580000
kernel config: https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba
dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=177beac0580000In this case, you would run:
$ git checkout ae58226b89ac
$ wget -O '.config' 'https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba'
$ make CC=clang LD=ld.lld olddefconfig
$ make CC=clang LD=ld.lld -j$(nproc)You also need a bootable disk image. Syzbot currently uses small Buildroot-based
images that you can either build locally or
download.
Download and build the reproducer:
$ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=177beac0580000'
$ gcc repro.c -lpthread -static -o reproRun the VM:
$ export DISK_IMAGE='buildroot_amd64_2024.09'
$ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=$DISK_IMAGE,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1Run the reproducer:
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro [email protected]:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes [email protected] 'chmod +x ./repro && ./repro'Using a Syz reproducer
Syzkaller always generates a "Syz" reproducer first (in Syzkaller
DSL). Afterwards, syzkaller attempts to convert the
Syz reproducer into C code. The process does not always succeed due to the
differences between the syz-executor environment and the environment emulated
in the C reproducer. Therefore, in some cases, only the Syz version is
available.
To run a Syz reproducer locally, the required actions are mostly similar to
those in the previous section.
Download and build syzkaller. If you
have Docker installed, the instructions are simpler:
$ git clone https://github.com/google/syzkaller.git
$ cd syzkaller
$ ./tools/syz-env makeBuild the kernel and boot the VM as described in the section above.
Download the reproducer:
$ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000'Copy the reproducer and the syzkaller binaries to the test machine:
$ export SYZKALLER_PATH="~/syzkaller"
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes $SYZKALLER_PATH/bin/linux_amd64/* ./repro.syz [email protected]:/root/Now you can use the syz-execprog tool to actually execute the program.
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes [email protected] './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz'Several useful syz-execprog flags:
-procs int
number of parallel processes to execute programs (default 1)
-repeat int
repeat execution that many times (0 for infinite loop) (default 1)
-sandbox string
sandbox for fuzzing (none/setuid/namespace) (default "setuid")
-threaded
use threaded mode in executor (default true)If you pass -threaded=0, all syscalls will be executed in the same thread.-threaded=1 forces execution of each syscall in a separate thread, so that
execution can proceed over blocking syscalls.
Before 2021, syz-execprog also supported the following flag:
-collide
collide syscalls to provoke data races (default true)-collide=1 forced second round of execution of syscalls when pairs of syscalls
are executed concurrently.
Starting from the revision
fd8caa54,
the behavior is controlled directly in syzlang.
If you are running older reproducers, you might still need to set the -collide=1 flag.
If you are replaying a reproducer program that contains a header along the
following lines:
# {Threaded:true Repeat:true RepeatTimes:0 Procs:8 Slowdown:1 Sandbox:none Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}then you need to adjust syz-execprog flags based on the values in the
header. Namely, Threaded/Procs/Sandbox directly relate to-threaded/-procs/-sandbox flags. If Repeat is set to true, add-repeat=0 flag to syz-execprog.
Using ktest
ktest is a collection
of tests for Linux and an infrastructure that simplifies running them locally.
Ktest includes a special syzbot-repro.ktest test that automates building the
kernel, booting the VM, fetching syzbot bug report details and running the
reproducer.
Installation instructions:
$ git clone git://evilpiepirate.org/ktest.git
$ cd ktest
$ export KTEST_PATH=$(pwd)
$ sudo ./root_image init
$ sudo ./root_image create
$ cargo install --path $KTEST_PATHInstructions to reproduce a syzbot bug:
$ cd ~/linux
$ git checkout <kernel-commit>
$ $KTEST_PATH/build-test-kernel run $KTEST_PATH/tests/syzbot-repro.ktest <bug-id>`` can be taken from syzbot bug reports:
dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053In this case, bug-id is 2159cbb522b02847c053.
Using downloadable assets
In each report, syzbot shares the exact disk image, kernel image and the vmlinux
file that were used to find it.
See the corresponding documentation on how you can
use those files to reproduce bugs locally.
From execution logs
The process of creating reproducer programs for syzkaller bugs is automated, but
it's not perfect. In some cases, the tool cannot narrow down the kernel crash to
a single program.
Obtaining execution logs
A local syzkaller instance
Crash logs created in managerworkdir/crashesdir contain programs executed
just before a crash. In parallel execution mode (whenprocsparameter in
manager config is set to value larger than 1), program that caused the crash
does not necessary immediately precedes it; the guilty program can be somewhere
before.Syzbot shares execution logs in its reports:
console output: https://syzkaller.appspot.com/x/log.txt?x=148914c0580000Crafting reproducers manually
There are two tools that can help you identify and minimize the program that
causes a crash: syz-execprog and syz-prog2c. You can build them with make execprog and make prog2c, respectively.
syz-execprog executes a single syzkaller program or a set of programs in
various modes (once or loop indefinitely; in threaded/collide mode (see below),
with or without coverage collection).
You can start by running all programs in the crash log in a loop to check that
at least one of them indeed crashes kernel:
./syz-execprog -executor=./syz-executor -repeat=0 -procs=8 -cover=0 crash-log-file.txtNote: syz-execprog executes programs locally. So you need to copysyz-execprog and syz-executor into a VM with the test kernel and run it
there. See the Using a Syz reproducer section.
To identify the single program that causes the crash, you can cut out individual
programs from crash-log-file.txt and run syz-execprog separately.
Once you have a single program that causes the crash, you can try to minimize it by:
- Removing individual syscalls from the program (you can comment out single lines
with#at the beginning of line) - By removing unnecessary data (e.g. replacing
&(0x7f0000001000)="73656c6600"
syscall argument with&(0x7f0000001000)=nil). - You can also try to coalesce all mmap calls into a single mmap call that maps
whole required area.
Don't forget to test minimization results with the syz-execprog tool.
Now that you have a minimized program, check if the crash still reproduces with./syz-execprog -threaded=0 -collide=0 flags. If not, then you will need to do
some additional work later.
Now, run the syz-prog2c tool on the program. It will give you an executable C
source code. If the crash reproduces with -threaded/collide=0 flags, then this C
program should cause the crash as well.
If the crash is not reproducible with -threaded/collide=0 flags, then you need
this last step. You can think of threaded mode as if each syscall is
executed in its own thread. To model such execution mode, move individual
syscalls into separate threads. You can see an example here:
https://groups.google.com/d/msg/syzkaller/fHZ42YrQM-Y/Z4Xf-BbUDgAJ.
This process is automated to some degree in the syz-repro utility. You need to
give it your manager config and a crash report file. And you can refer to the
example config file.
./syz-repro -config my.cfg crash-qemu-1-1455745459265726910It will try to find the offending program and minimize it. But since there are
lots of factors that can affect reproducibility, it does not always work.
Research (`docs/research.md`)
Research work based on syzkaller
Dear researchers, feel free to contact us at [email protected] if you need any assistance.
newer first
- Unlocking Low Frequency Syscalls in Kernel Fuzzing with Dependency-Based RAG (pdf, source code)
- A Little Goes a Long Way: Tuning Configuration Selection for Continuous Kernel Fuzzing
- SyzDirect: Directed Greybox Fuzzing for Linux Kernel
- KIT: Testing OS-Level Virtualization for Functional Interference Bugs
- SyzDescribe: Principled, Automated, Static Generation of Syscall Descriptions for Kernel Drivers
- GREBE: Unveiling Exploitation Potential for Linux Kernel Bugs
- Precise Detection of Kernel Data Races with Probabilistic Lockset Analysis
- Linux Kernel Enriched Corpus : corpus.db
- HotBPF - An On-demand and On-the-fly Memory Protection
- KASPER: Scanning for Generalized Transient Execution Gadgets in the Linux Kernel
- VaultFuzzer: A state-based approach for Linux kernel
- Demystifying the Dependency Challenge in Kernel Fuzzing
- SyzVegas: Beating Kernel Fuzzing Odds with Reinforcement Learning
- SyzScope: Revealing High-Risk Security Impacts of Fuzzer-Exposed Bugs in Linux kernel
- Rtkaller: State-aware Task Generation for RTOS Fuzzing
- BSOD: Binary-only Scalable fuzzing Of device Drivers
- Torpedo: A Fuzzing Framework for Discovering Adversarial Container Workloads
- Healer is a kernel fuzzer inspired by syzkaller. (pdf)
- SyzGen: Automated Generation of Syscall Specification of Closed-Source macOS Drivers (source code)
- Snowboard: Finding Kernel Concurrency Bugs through Systematic Inter-thread Communication Analysis
- Undo Workarounds for Kernel Bugs (source code)
- HFL: Hybrid Fuzzing on the Linux Kernel
- A Novel Dynamic Analysis Infrastructure to Instrument Untrusted Execution Flow Across User-Kernel Spaces
- Industry Practice of Coverage-Guided Enterprise Linux Kernel Fuzzing
- Agamotto: Accelerating Kernel Driver Fuzzing with Lightweight Virtual Machine Checkpoints (source code)
- Empirical Notes on the Interaction Between Continuous Kernel Fuzzing and Development
- FastSyzkaller: Improving Fuzz Efficiency for Linux Kernel Fuzzing
- Charm: Facilitating Dynamic Analysis of Device Drivers of Mobile Systems
(video,
slides,
source code) - ALEXKIDD-FUZZER: Kernel Fuzzing Guided by Symbolic Information
- DIFUZE: Interface Aware Fuzzing for Kernel Drivers
- MoonShine: Optimizing OS Fuzzer Seed Selection with Trace Distillation
- RAZZER: Finding Kernel Race Bugs through Fuzzing
- SemFuzz: Semantics-based Automatic Generation of Proof-of-Concept Exploits
- Towards Automating Exploit Generation for Arbitrary Types of Kernel Vulnerabilities
- KOOBE: Towards Facilitating Exploit Generation of Kernel Out-Of-Bounds Write Vulnerabilities
- Synthesis of Linux Kernel Fuzzing Tools Based on Syscall
- Drill the Apple Core: Up & Down
- WSL Reloaded
Other kernel fuzzing work
- Hydra: Finding Semantic Bugs in File Systems with an Extensible Fuzzing Framework (github)
- Janus: Fuzzing File Systems via Two-Dimensional Input Space Exploration (github)
- CoLaFUZE: Coverage-Guided and Layout-Aware Fuzzing for Android Drivers
- KRACE: Data Race Fuzzing for Kernel File Systems
- trinity
- kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels (bridges AFL and Intel PT)
- kernel-fuzzing (bridges AFL and KCOV)
- A gentle introduction to Linux Kernel fuzzing (bridges AFL and KCOV)
- IMF: Inferred Model-based Fuzzer
Also see tech talks page.
Review Guidelines (`docs/review_guidelines.md`)
Automated Code Review Guidelines (DVYUKOV.md)
This file contains curated recommendations extracted from automated pull request reviews. Ingest this file into your LLM pipeline to ensure compliance with repository style and best practices!
1. Go Coding Style & Idioms
- Naming:
- Lowercase package names (no underscores or MixedCaps).
- Use
NewFooorMakeFoofor constructors; return pointers if methods are defined on pointers. - Avoid repeating type names in method names (e.g.,
linux.Foo()notlinux.FooLinux()). - Avoid "syz" prefixes for internal structs/fields (e.g.,
exec_totalnotsyz_exec_total). - Use
ifor loop indices; use descriptive names for slice elements. - Favor positive boolean naming (
activevsnot_inactive).
- Declarations:
- Combine declaration and initialization (
x := foo()). - Rely on zero-value defaults; do not manually initialize to 0, "", or
map[type]type{}. - Unexport symbols (types, fields, constants, functions) not needed externally.
- Use
iotafor enums and makeInvalidthe first value (0) to catch uninitialized fields. - Use
go:embedfor embedding static assets like HTML or text templates.
- Combine declaration and initialization (
- Functions & Control Flow:
- Return early on errors (standard
if err != nilcheck); avoid deep nesting. - Avoid naked or named returns unless they significantly simplify repetitive code.
- Drop unnecessary
elseafterreturn,break, orcontinue. - Prefer free/normal functions over closures if no context is captured from the outer scope.
- Use closures to deduplicate scoped logic within a single function.
- Return early on errors (standard
- Miscellaneous:
- Cap Go lines at 120 columns; use tabs for indentation.
- Use
%sfor struct string conversion infmtfunctions (idiomaticv.String()call). - Avoid unneeded parentheses in expressions and
()to refer to functions in comments.
2. Error Handling & Reliability
- Return Errors: Never swallow errors or return
nilon unexpected conditions. - Fail Fast:
- Validate invariants early (e.g., during initialization or deserialization).
- Panic on impossible internal logic corruption or broken engine invariants.
- For external tools/plugins, use
SYZFAIL:prefix for internal bugs to distinguish from target crashes.
- Error Context:
- Don't wrap errors without adding specific, helpful context (Syzkaller libs often add basic context).
- Use
aflow.BadCallErrorfor invalid tool inputs in agent workflows.
- Validation:
- Always validate inputs at boundaries (datastores, APIs, CLI flags, regexes).
- Guard against underflows/overflows in length checks and binary formats.
3. Testing Best Practices
- Assertions: Use
github.com/stretchr/testify/require(orassert) for single-line, readable assertions. - Structure:
- Use table-driven tests with
t.Runfor isolation and subtest selection. - Keep helper functions at the bottom of the test file; use
t.Cleanupfor teardown.
- Use table-driven tests with
- Data Handling:
- Use backtick raw string literals (
`) for multi-line test data, regexes, and assembly. - Use
AUTOin syzlang tests for auto-computing lengths and offsets. - Use
timeNowovertime.Nowfor deterministic test stubbing.
- Use backtick raw string literals (
- Robustness:
- Run tests with
-raceto surface data races. - Check for goroutine leaks in tests using background concurrency.
- Test boundary conditions (e.g., empty slices, index limits, or corrupted inputs).
- Never just disable failing tests; fix them or document why they are disabled.
- Include sample reports in
pkg/report/testdatafor new crash parsers.
- Run tests with
4. Architecture & Design
- YAGNI (You Ain't Gonna Need It):
- Don't add code "just in case"; remove unused types, fields, and dead code immediately.
- Avoid unnecessary tunables/config flags; pick sensible defaults and scale timeouts.
- Decoupling:
- Keep logic uniform across different execution modes (e.g., fork-server vs no-fork-server).
- Use standard interfaces (e.g.,
io.Closer) over custom cleanup methods. - Prefer specific, descriptive package names over generic ones like
utilorgeneric. - Prioritize larger, coherent packages over many micro-packages.
- State Management:
- Favor simple state tracking (single counter/slice) over complex nested maps.
- Use fields in the
targets.Targetstruct (often referred to as Arch struct) instead of logic-driven switches for architecture differences. - Pass objects/pointers early instead of passing IDs and doing repeated lookups.
5. Performance & Resource Management
- Efficiency:
- Use map lookups (O(1)) instead of double loops (O(N^2)) for filtering.
- Avoid casting large buffers to strings if only byte processing is needed.
- Prefer direct slicing over
io.Readerfor byte processing to reduce garbage. - Avoid
bufio.Writerwhen writing tobytes.Buffer.
- Resource Cleanup:
- Always
Wait()on launched commands; usedeferto close file descriptors and pipes. - Extract loop bodies into functions if they use
deferto ensure per-iteration cleanup.
- Always
- Concurrency:
- Use channels for semaphores or async notifications; use capacity 1 to prevent blocking.
- Use
nilchannels to disableselectcases. - Ensure condition variables are checked in a loop to prevent race hangs.
- Avoid infinite goroutine spawning in pools; track wait states.
6. Logging & Observability
- Hygiene:
- Use the internal
logpackage consistently; avoid mixing multiple logger libraries. - Log only actionable information at verbosity 0; use info logs for invalid user requests.
- Don't suffix log messages with newlines (
\n). - Format string safety: use
Logf(1, "%s", out)instead ofLogf(1, out).
- Use the internal
- Determinism: Always sort map keys when serializing to persistent formats (configs, logs, coverage).
- Metrics: Rename existing metrics rather than adding confusing duplicates when semantics change.
- Reporting: Keep email prefixes short; e.g. explain why something was skipped in bisection logs.
7. Syzkaller Specifics (Syzlang & Executor)
- Syzlang (files in
sys/*/*.txt):- Use exact kernel field names; use descriptive prefixes to avoid collisions.
- Use
const[0]for reserved/unused fields to optimize mutation. - Use
ignore_returnfor syscalls returning random system IDs or times.
- Executor (C++ files in
executor):- Keep
executor.ccfree of OS-specific#ifdefs; use OS-specific headers. - Use
const std::string&orstd::string_viewfor immutable strings. - Check for short writes in output loops.
- Avoid C89 start-of-block declarations; combine declaration and initialization.
- Keep
8. Tooling & Infrastructure
- CLI Tools:
- Standardize on
tool.Init()andtool.Failf()frompkg/tool. - Declare CLI flags in
main()to avoid global namespace pollution. - Use
runtime.NumCPU()for default worker counts.
- Standardize on
- OS Operations:
- Use
filepath.Joinfor OS-agnostic paths; useosutil.IsExistfor existence checks. - Combine multiple shell commands (e.g.,
adb shell 'cmd1; cmd2') to reduce overhead.
- Use
- HTML/UI:
- Use standard
andfor collapsibles. - Use tabs for HTML templates.
- Use standard
Setup (`docs/setup.md`)
How to set up syzkaller
Generic setup instructions for fuzzing Linux kernel are outlined here.
For other kernels see:
FreeBSD,
Darwin/XNU,
Fuchsia,
NetBSD,
OpenBSD,
Windows.
After following these instructions you should be able to run syz-manager, see it executing programs, and be able to access statistics exposed at http://127.0.0.1:56741 (or whatever address you've specified in the manager config).
If everything is working properly, a typical execution log would look like:
$ ./bin/syz-manager -config=my.cfg
2017/06/14 16:39:05 loading corpus...
2017/06/14 16:39:05 loaded 0 programs (0 total, 0 deleted)
2017/06/14 16:39:05 serving http on http://127.0.0.1:56741
2017/06/14 16:39:05 serving rpc on tcp://127.0.0.1:34918
2017/06/14 16:39:05 booting test machines...
2017/06/14 16:39:05 wait for the connection from test machine...
2017/06/14 16:39:59 received first connection from test machine vm-9
2017/06/14 16:40:05 executed 293, cover 43260, crashes 0, repro 0
2017/06/14 16:40:15 executed 5992, cover 88463, crashes 0, repro 0
2017/06/14 16:40:25 executed 10959, cover 116991, crashes 0, repro 0
2017/06/14 16:40:35 executed 15504, cover 132403, crashes 0, repro 0At this point it's important to ensure that syzkaller is able to collect code coverage of the executed programs
(unless you specified "cover": false in the config or coverage is not yet supported for the kernel you're fuzzing).
The cover counter on the web page should be non zero.
More information on the configuration file format is available here.
See this page for troubleshooting tips.
Setup Syzbot (`docs/setup_syzbot.md`)
How to set up syzbot
This doc will be useful to you:
- should you wish to hack on user interface bits like the dashboard / mailing list integration or
- should you wish to continuously run a separate syzbot dashboard for your own kernels
Note: For most development purposes you don't need a full syzbot setup. The meat of syzkaller is really located
in syz-manager and syz-executor. You can run syz-manager directly which is usually what you will want to do during
fuzzer development. See this documentation for syz-manager setup instructions.
This doc assumes that you:
- have a GCP account and billing setup
- created a GCP project for running syzbot in
- are running a reasonably modern linux distro
- locally installed
gcloud,ssh,goandbuild-essential - may need to install
google-cloud-sdk-app-engine-gofor the GAE deployment to work - ran
gcloud auth loginto run authenticated gcloud commands - read go/syzbot-setup if you are a Googler
While most syzkaller bits happily run on various operating systems, the syzbot dashboard does not. The dashboard is a Google App Engine or GAE project. GAE allows developers to develop web applications without needing to worry about the underlying servers. Instead developers just push their code and GAE takes care of web servers, load balancers and more. Hence this document is more Google Cloud focused than the rest of our documentation.
We will also deploy a syz-ci instance. syz-ci keeps track of the syzkaller and kernel repositories and continuously rebuilds the kernel under test, itself and other syzkaller components when new commits land in the upstream repositories. syz-ci also takes care of (re)starting syz-manager instances, which in turn (re)start VMs fuzzing the target kernel. For simplicity we will run everything in this doc on GCP even though syz-ci could run elsewhere.
Deploying Syz-ci
[local] First prepare an initial syz-ci build locally (later syz-ci rebuilds itself) and a rootfs:
# Most syzkaller components can be build even outside of the GOPATH, however
# the syzbot app engine deployment only works from the GOPATH right now..
export GOOGLE_GO=$HOME/gopath/src/github.com/google/
mkdir -p $GOOGLE_GO
git clone https://github.com/google/syzkaller.git
mv syzkaller $GOOGLE_GO/
cd $GOOGLE_GO/syzkaller
make ci
cd ~/repos
git clone git://git.buildroot.net/buildroot
cd buildroot
$GOOGLE_GO/syzkaller/tools/create-buildroot-image.sh[local] Enable various services in the project, create a VM, storage bucket, scp assets and login:
/* Detailed source-code truncated for AI context efficiency. */[syz-ci] Let's install and configure the syz-ci service on our syz-ci VM:
/* Detailed source-code truncated for AI context efficiency. */Check the syc-ci journal logs at this point to see if the service comes up fine. Now syz-ci needs to do a bunch of time consuming stuff like building the kernel under test, so be patient.
If you want to hack on syz-ci you can stop here. Otherwise the next section builds on the syz-ci instructions and extends the setup with a dashboard deployment.
Deploying Syzbot dashboard
[locally] deploy the dashboard to Google App Engine:
/* Detailed source-code truncated for AI context efficiency. */Integrating Syz-ci with syzbot
[locally] Prepare config and login to syz-ci VM:
/* Detailed source-code truncated for AI context efficiency. */[syz-ci] Reconfigure syz-ci to start sending results to the dashboard:
sudo mv ~/config.ci /syzkaller/
sudo systemctl restart syz-ci
sudo journalctl -fu syz-ci[locally] Open the dashboard in your browser:
gcloud app browse --project=$PROJECTOnce syzkaller finds the first crashes they should show up here. This might take a while.
Strace (`docs/strace.md`)
Strace
Syzkaller can be instructed to execute programs under
strace and capture the output.
If the strace_bin is set to an strace binary, syzkaller will automatically
run each reproducer it managed to find under the strace binary.
- If a syz-manager is attached to a
dashboard, syzkaller will upload the
resulting output as a normal log file if the generated reproducer still
managed to produce the same crash. - Otherwise, the output of strace will be saved to a separate file and will be
accessible through the syz-manager's web interface.
How to compile the strace binary
It is safer to compile strace as a statically linked binary in order to
prevent problems with mismatching libc versions on the kernel image used for
fuzzing.
git clone https://github.com/strace/strace.git
cd strace
./bootstrap
./configure --enable-mpers=no LDFLAGS='-static -pthread'
make -j`nproc`The resulting binary can be found at src/strace.
syz-crush
It's possible to instruct syz-crush to run the attached repro under strace. In
order to do so, make sure strace_bin is specified in the syz-manager config
file and pass an extra -strace argument to the command arguments.
syz-repro
If -strace file-name.log is appended to the syz-repro's arguments, the tool
will run the resulting repro (if it managed to generate one) under strace and
save its output.
Syscall Descriptions (`docs/syscall_descriptions.md`)
Syscall descriptions
syzkaller uses declarative description of syscall interfaces to manipulate
programs (sequences of syscalls). Below you can see (hopefully self-explanatory)
excerpt from the descriptions:
open(file filename, flags flags[open_flags], mode flags[open_mode]) fd
read(fd fd, buf buffer[out], count len[buf])
close(fd fd)
open_mode = S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTHThe descriptions are contained in sys/$OS/*.txt files.
For example see the sys/linux/dev_snd_midi.txt file
for descriptions of the Linux MIDI interfaces.
A more formal description of the description syntax can be found here.
Programs
The translated descriptions are then used to generate, mutate, execute, minimize, serialize
and deserialize programs. A program is a sequences of syscalls with concrete values for arguments.
Here is an example (of a textual representation) of a program:
r0 = open(&(0x7f0000000000)="./file0", 0x3, 0x9)
read(r0, &(0x7f0000000000), 42)
close(r0)For actual manipulations syzkaller uses in-memory AST-like representation consisting ofCall and Arg values defined in prog/prog.go. That representation is used to
analyze, generate, mutate,
minimize, validate, etc programs.
The in-memory representation can be transformed to/from
textual form to store in on-disk corpus, show to humans, etc.
There is also another binary representation
of the programs (called exec), that is much simpler, does not contain rich type information (irreversible)
and is used for actual execution (interpretation) of programs by executor.
Describing new system calls
This section describes how to extend syzkaller to allow fuzz testing of more kernel interfaces.
This is particularly useful for kernel developers who are proposing new system calls.
Currently all syscall descriptions are manually-written. There is an
open issue to provide some aid
for this process and some ongoing work, but we are not there yet to have a
fully-automated way to generate descriptions.
There is a helper headerparser utility that can auto-generate
some parts of descriptions from header files. Visual Studio Code has syz-lang extension for highlighting syntax.
To enable fuzzing of a new kernel interface:
Study the interface, find out which syscalls are required to use it. Sometimes there is nothing besides the source code, but here are some things that may help:
- Searching the Internet for the interface name and/or some unique constants.
- Grepping Documentation/ dir in the kernel.
- Searching tools/testing/ dir in the kernel.
- Looking for large comment blocks in the source code.
- Finding commit that added the interface via
git blameorgit logand reading the commit description. - Reading source code of or tracing libraries or applications that are known to use this interface.
Using syntax documentation and
existing descriptions as an example, add a declarative
description of this interface to the appropriate file:sys/linux/<subsystem>.txtfiles hold system calls for particular kernel
subsystems, for example bpf.txt or socket.txt.- sys/linux/sys.txt holds descriptions for more general system calls.
- An entirely new subsystem can be added as a new
sys/linux/<new>.txtfile. - If subsystem descriptions are split across multiple files, prefix the name of each file with the name of the subsystem (e.g. use
dev_*.txtfor descriptions of/dev/devices, usesocket_*.txtfor sockets, etc).
After adding/changing descriptions run:
bashmake extract TARGETOS=linux SOURCEDIR=$KSRC make generate makeRun syzkaller. Make sure that the newly added interface in being reached by
syzkaller using the coverage information page.
In the instructions above make extract generates/updates the *.const files.$KSRC should point to the latest kernel checkout.
Note: for Linux the latest kernel checkout generally means the
mainline tree.
However, in some cases we add descriptions for interfaces that are not in the mainline tree yet,
so if make extract complains about missing header files or constants undefined on all architectures,
try to use the latest linux-next
tree (or if it happens to be broken at the moment, try a slightly older linux-next tree).
Note: make extract overwrites .config in $KSRC and mrproper's it.
Note: *.const files are checked-in with the *.txt changes in the same commit.
Then make generate updates generated code and make rebuilds binaries.
Note: make generate does not require any kernel sources, native compilers, etc
and is pure text processing.
Note: make generate also updates the SYZ_REVISION under executor/defs.h, which
is required for machine check while running syz-manager. This should be taken care
of especially if you are trying to rebase with your own change on syscall description.
Note: make extract extracts constants for all architectures which requires
installed cross-compilers. If you get errors about missing compilers/libraries,
try sudo make install_prerequisites or install equivalent package for your distro.
Note: sudo make install_prerequisites will success even with some package failed to
install, sudo apt-get update && sudo apt-get upgrade might be required to make this
more efficient.
If you want to fuzz only the new subsystem that you described locally, you may
find the enable_syscalls configuration parameter useful to specifically target
the new system calls. All system calls in the enable_syscalls list
will be enabled if their requirements are met (ie. if they are supported
in the target machine and any other system calls that need to run in
order to provide inputs for them are also enabled). You can also include
wildcard definitions to enable multiple system calls in a single line,
for example: "ioctl" will enable all the described ioctls syscalls
that have their requirements met, "ioctl$UDMABUF_CREATE" enables
only that particular ioctl call, "write$UHID_*" enables all write
system calls that start with that description identifier.
When updating existing syzkaller descriptions, note, that unless there's a drastic
change in descriptions for a particular syscall, the programs that are already in
the corpus will be kept there, unless you manually clear them out (for example by
removing the corpus.db file).
Description tips and FAQ
Syscall, struct, field, flags names
Stick with existing kernel names for things, don't invent new names if possible.
Following established naming conventions provides the following benefits:
(1) consistency and familiarity of names used across kernel interfaces,
which also enables searching kernel sources for related names; and
(2) enable static checking of descriptions (e.g. missed flags or mistyped fields)
with syz-check.
For example, if there is an existing enum v4l2_buf_type in the kernel headers,
use this name for flags in descriptions as well. The same for structs, unions,
fields, etc. For syscall and struct variants, append the variant name after the $ sign.
For example, fcntl$F_GET_RW_HINT, ioctl$FIOCLEX, setsockopt$SO_TIMESTAMP.
Resources for syscall ordering
Resources and resource directions (in, out, inout) impose implicit ordering
constraints on involved syscalls.
If a syscall accepts a resource of a particular type (e.g. has fd_cdrom as an input),
then it will be generally placed after a syscall that has this resource as output,
so that the resource value can be passed between syscalls. For example:
r0 = openat$cdrom(...)
ioctl$CDROMPAUSE(r0, 0x123)
close(r0)Syscall arguments are always in, return values are out and pointer indirections
have explicit direction as ptr type attribute. Also, it is possible to specify
direction attribute individually for struct fields to account for more complex
producer/consumer scenarios with structs that include both input/output resources.
Use of unexpected/undeclared values
When specifying integer/string flags or integer fields stick with the official expected values only.
Commonly, bugs are triggered by unexpected inputs. With that in mind, it can be too tempting to introduce
some unexpected values to descriptions (e.g. -1 or INT_MAX). This is not encouraged for several reasons.
First, this is a cross-cutting aspect and these special unexpected values are applicable to just
any flags and integer fields. Manually specifying them thousands of times is not scalable and
is not maintainable. Second, It's hard for the fuzzer to come up with correct complex syscall sequences,
and the descriptions are meant to help with this. Coming up with unexpected integer values is easy
and the fuzzer does not need help here. Overall the idea is to improve the generic fuzzer logic
to handle these cases better, which will help all descriptions, rather than over-specializing each
individual integer separately. Fuzzer already has several tricks to deal with this, e.g. comparison
operand value interception and list of typical magic values.
Note: some values for flags may be undocumented only as an oversight. These values should be added to descriptions.
Flags/enums
The flags type is used for all of:
- sets of mutually exclusive values, where only one of them should be chosen (like C enum);
- sets of bit flags, where multiple values can be combined with bitwise OR (like mmap flags);
- any combination of the above.
The fuzzer has logic to distinguish enums and bit flags, and generates values
accordingly. So the general guideline is just to enumerate the meaningful values
in flags without adding any "special" values to "help" the current fuzzer logic.
When/if the fuzzer logic changes/improves, these manual additions may become
unnecessary, or, worse, interfere with the fuzzer ability to generate good values.
Declaration order
syzlang does not require declaring entities before use (like C/C++ does), entities can refer to entities
declared later (like in Go). It's recommended to declare things in the order of importance so that the reader
sees the most important things first and then proceeds to finer and finer implementation details. For example,
system calls usually should go before flag declarations used in these system calls. Note: this order is usually
the exact opposite of how things are declared in C: the least important things go first.
Description compilation internals
The process of compiling the textual syscall descriptions into machine-usable
form used by syzkaller to actually generate programs consists of 2 steps.
The first step is extraction of values of symbolic constants from kernel sources using
syz-extract utility. syz-extract generates a small C program that
includes kernel headers referenced by include directives, defines macros as specified
by define directives and prints values of symbolic constants.
Results are stored in .const files, one per arch.
For example, sys/linux/dev_ptmx.txt is translated into
sys/linux/dev_ptmx.txt.const.
The second step is translation of descriptions into Go code using
syz-sysgen utility (the actual compiler code lives in
pkg/ast and pkg/compiler).
This step uses syscall descriptions and the const files generated during the first step
and produces instantiations of Syscall and Type types defined in prog/types.go.
You can see an example of the compiler output for Linux/AMD64 in sys/linux/gen/amd64.go.
This step also generates some minimal syscall metadata for C++ code in executor/syscalls.h.
Non-mainline subsystems
make extract extracts constants for all *.txt files and for all supported architectures.
This may not work for subsystems that are not present in mainline kernel or if you have
problems with native kernel compilers, etc. In such cases the syz-extract utility
used by make extract can be run manually for single file/arch as:
make bin/syz-extract
bin/syz-extract -os linux -arch $ARCH -sourcedir $KSRC -builddir $LINUXBLD <new>.txt
make generate
make$ARCH is one of amd64, 386 arm64, arm, ppc64le, mips64le.
If the subsystem is supported on several architectures, then run syz-extract for each arch.$KSRC should point to kernel source checkout, which is configured for the
corresponding arch (i.e. you need to run make ARCH=arch someconfig && make ARCH=arch there first,
remember to add CROSS_COMPILE=arm-linux-gnueabi-/aarch64-linux-gnu-/powerpc64le-linux-gnu- if needed).
If the kernel was built into a separate directory (with make O=output_dir, remember to put .config
into output_dir, this will be helpful if you'd like to work on different arch at the same time)
then also set $LINUXBLD to the location of the build directory.
Testing of descriptions
Descriptions themselves may contain bugs. After running syz-manager with the new descriptions
it's always useful to check the kernel code coverage report available in the syz-manager web UI.
The report allows to assess if everything one expects to be covered is in fact covered,
and if not, where the fuzzer gets stuck. However, this is a useful but quite indirect assessment
of the descriptions correctness. The fuzzer may get around some bugs in the descriptions by diverging
from what the descriptions say, but it makes it considerably harder for the fuzzer to progress.
Tests stored in sys/OS/test/* provide a more direct testing of the descriptions. Each test is just
a program with checked syscall return values. The syntax of the programs is briefly described here.
You can also look at the existing examples and at the program deserialization code.AUTO keyword can be used as a value for consts and pointers, for pointers it will lead to
some reasonable sequential allocation of memory addresses.
It's always good to add a test at least for "the main successful scenario" for the subsystem.
It will ensure that the descriptions are actually correct and that it's possible for the fuzzer
to come up with the successful scenario. See io_uring test as a good example.
The tests can be run with the run-tests functionality of syz-manager:
make && bin/syz-manager -config manager.config -mode run-testsIt will boot multiple VMs and runs these tests in different execution modes inside of the VMs.
However, the full run takes significant time, so while developing the test, it's more handy to run it
using the syz-execprog utility. To run the test, copy syz-execprog, syz-executor and the test
into a manually booted VM and then run the following command inside of the VM:
syz-execprog -debug -threaded=0 mytestIt will show results of all executed syscalls. It's also handy for manual debugging of pseudo-syscall code:
if you add some temporal debug calls to the pseudo-syscall, syz-execprog -debug will show their output.
The test syntax can be checked by running:
go test -run=TestParsing ./pkg/runtestSyscall Descriptions Syntax (`docs/syscall_descriptions_syntax.md`)
Syscall description language
aka syzlang ([siːzˈlæŋg])
Pseudo-formal grammar of syscall description:
syscallname "(" [arg ["," arg]*] ")" [type] ["(" attribute* ")"]
arg = argname type
argname = identifier
type = typename [ "[" type-options "]" ]
typename = "const" | "intN" | "intptr" | "flags" | "array" | "ptr" |
"string" | "filename" | "glob" | "len" |
"bytesize" | "bytesizeN" | "bitsize" | "vma" | "proc" |
"compressed_image"
type-options = [type-opt ["," type-opt]]common type-options include:
"opt" - the argument is optional (like mmap fd argument, or accept peer argument)rest of the type-options are type-specific:
/* Detailed source-code truncated for AI context efficiency. */flags/len/flags also have trailing underlying type type-option when used in structs/unions/pointers.
Flags are described as:
flagname = const ["," const]*or for string flags as:
flagname = "\"" literal "\"" ["," "\"" literal "\""]*Call attributes are:
/* Detailed source-code truncated for AI context efficiency. */Ints
int8, int16, int32 and int64 denote an integer of the corresponding size.intptr denotes a pointer-sized integer, i.e. C long type.
By appending be suffix (e.g. int16be) integers become big-endian.
It's possible to specify a range of values for an integer in the format of int32[0:100] or int32[0:4096, 512] for a 512-aligned int.
Integers can also take a reference to flags description or a value as its first type-option.
In that case, the alignment parameter is not supported.
To denote a bitfield of size N use int64:N.
It's possible to use these various kinds of ints as base types for const, flags, len and proc.
example_struct {
f0 int8 # random 1-byte integer
f1 const[0x42, int16be] # const 2-byte integer with value 0x4200 (big-endian 0x42)
f2 int32[0:100] # random 4-byte integer with values from 0 to 100 inclusive
f3 int32[1:10, 2] # random 4-byte integer with values {1, 3, 5, 7, 9}
f4 int64:20 # random 20-bit bitfield
f5 int8[10] # const 1-byte integer with value 10
f6 int32[flagname] # random 4-byte integer from the set of values referenced by flagname
}Structs
Structs are described as:
structname "{" "\n"
(fieldname type ("(" fieldattribute* ")")? (if[expression])? "\n")+
"}" ("[" attribute* "]")?Fields can have attributes specified in parentheses after the field, independent
of their type. in/out/inout attribute specify per-field direction, for example:
foo {
field0 const[1, int32] (in)
field1 int32 (inout)
field2 fd (out)
}You may specify conditions that determine whether a field will be included:
foo {
field0 int32
field1 int32 (if[value[field0] == 0x1])
}See the corresponding section
for more details.
out_overlay attribute allows to have separate input and output layouts for the struct.
Fields before the out_overlay field are input, fields starting from out_overlay are output.
Input and output fields overlap in memory (both start from the beginning of the struct in memory).
For example:
foo {
in0 const[1, int32]
in1 flags[bar, int8]
in2 ptr[in, string]
out0 fd (out_overlay)
out1 int32
}Structs can have attributes specified in square brackets after the struct.
Attributes are:
packed: the struct does not have paddings between fields and has alignment 1; this is similar to GNU C__attribute__((packed)); struct alignment can be overridden withalignattributealign[N]: the struct has alignment N and padded up to multiple ofN; contents of the padding are unspecified (though, frequently are zeros); similar to GNU C__attribute__((aligned(N)))size[N]: the struct is padded up to the specified sizeN; contents of the padding are unspecified (though, frequently are zeros)
Unions
Unions are described as:
unionname "[" "\n"
(fieldname type (if[expression])? "\n")+
"]" ("[" attribute* "]")?During fuzzing, syzkaller randomly picks one of the union options.
You may also specify conditions that determine whether the corresponding
option may or may not be selected, depending on values of other fields. See
the corresponding section
for more details.
Unions can have attributes specified in square brackets after the union.
Attributes are:
varlen: union size is the size of the particular chosen option (not statically known); without this attribute unions are statically sized as maximum of all options (similar to C unions)size[N]: the union is padded up to the specified sizeN; contents of the padding are unspecified (though, frequently are zeros)
Resources
Resources represent values that need to be passed from output of one syscall to input of another syscall. For example, close syscall requires an input value (fd) previously returned by open or pipe syscall. To achieve this, fd is declared as a resource. This is a way of modelling dependencies between syscalls, as defining a syscall as the producer of a resource and another syscall as the consumer defines a loose sense of ordering between them. Resources are described as:
"resource" identifier "[" underlying_type "]" [ ":" const ("," const)* ]underlying_type is either one of int8, int16, int32, int64, intptr or another resource (which models inheritance, for example, a socket is a subtype of fd). The optional set of constants represent resource special values, for example, 0xffffffffffffffff (-1) for "no fd", or AT_FDCWD for "the current dir". Special values are used once in a while as resource values. If no special values specified, special value of 0 is used. Resources can then be used as types, for example:
resource fd[int32]: 0xffffffffffffffff, AT_FDCWD, 1000000
resource sock[fd]
resource sock_unix[sock]
socket(...) sock
accept(fd sock, ...) sock
listen(fd sock, backlog int32)Resources don't have to be necessarily returned by a syscall. They can be used as any other data type. For example:
resource my_resource[int32]
request_producer(..., arg ptr[out, my_resource])
request_consumer(..., arg ptr[inout, test_struct])
test_struct {
...
attr my_resource
}For more complex producer/consumer scenarios, field attributes can be utilized.
For example:
resource my_resource_1[int32]
resource my_resource_2[int32]
request_produce1_consume2(..., arg ptr[inout, test_struct])
test_struct {
...
field0 my_resource_1 (out)
field1 my_resource_2 (in)
}Each resource type must be "produced" (used as an output) by at least one syscall
(outside of unions and optional pointers) and "consumed" (used as an input)
by at least one syscall.
Type Aliases
Complex types that are often repeated can be given short type aliases using the
following syntax:
type identifier underlying_typeFor example:
type signalno int32[0:65]
type net_port proc[20000, 4, int16be]Then, type alias can be used instead of the underlying type in any contexts.
Underlying type needs to be described as if it's a struct field, that is,
with the base type if it's required. However, type alias can be used as syscall
arguments as well. Underlying types are currently restricted to integer types,ptr, ptr64, const, flags and proc types.
There are some builtin type aliases:
type bool8 int8[0:1]
type bool16 int16[0:1]
type bool32 int32[0:1]
type bool64 int64[0:1]
type boolptr intptr[0:1]
type fileoff[BASE] BASE
type filename string[filename]
type buffer[DIR] ptr[DIR, array[int8]]Type Templates
Type templates can be declared as follows:
type buffer[DIR] ptr[DIR, array[int8]]
type fileoff[BASE] BASE
type nlattr[TYPE, PAYLOAD] {
nla_len len[parent, int16]
nla_type const[TYPE, int16]
payload PAYLOAD
} [align_4]and later used as follows:
syscall(a buffer[in], b fileoff[int64], c ptr[in, nlattr[FOO, int32]])There is builtin type template optional defined as:
type optional[T] [
val T
void void
] [varlen]Length
You can specify length of a particular field in struct or a named argument by
using len, bytesize and bitsize types, for example:
write(fd fd, buf ptr[in, array[int8]], count len[buf])
sock_fprog {
len len[filter, int16]
filter ptr[in, array[sock_filter]]
}If len's argument is a pointer, then the length of the pointee argument is used.
To denote the length of a field in N-byte words use bytesizeN, possible values
for N are 1, 2, 4 and 8.
To denote the length of the parent struct, you can use len[parent, int8].
To denote the length of the higher level parent when structs are embedded into
one another, you can specify the type name of the particular parent:
s1 {
f0 len[s2] # length of s2
}
s2 {
f0 s1
f1 array[int32]
f2 len[parent, int32]
}len argument can also be a path expression which allows more complex
addressing. Path expressions are similar to C field references, but also allow
referencing parent and sibling elements. A special reference syscall used
in the beginning of the path allows to refer directly to the syscall arguments.
For example:
s1 {
a ptr[in, s2]
b ptr[in, s3]
c array[int8]
}
s2 {
d array[int8]
}
s3 {
# This refers to the array c in the parent s1.
e len[s1:c, int32]
# This refers to the array d in the sibling s2.
f len[s1:a:d, int32]
# This refers to the array k in the child s4.
g len[i:j, int32]
# This refers to syscall argument l.
h len[syscall:l, int32]
i ptr[in, s4]
}
s4 {
j array[int8]
}
foo(k ptr[in, s1], l ptr[in, array[int8]])Proc
The proc type can be used to denote per process integers.
The idea is to have a separate range of values for each executor, so they don't interfere.
The simplest example is a port number.
The proc[20000, 4, int16be] type means that we want to generate an int16be
integer starting from 20000 and assign 4 values for each process.
As a result the executor number n will get values in the [20000 + n * 4, 20000 + (n + 1) * 4) range.
Integer Constants
Integer constants can be specified as decimal literals, as 0x-prefixed
hex literals, as '-surrounded char literals, or as symbolic constants
extracted from kernel headers or defined by define directives. For example:
foo(a const[10], b const[-10])
foo(a const[0xabcd])
foo(a int8['a':'z'])
foo(a const[PATH_MAX])
foo(a int32[PATH_MAX])
foo(a ptr[in, array[int8, MY_PATH_MAX]])
define MY_PATH_MAX PATH_MAX + 2Conditional fields
In structures
In syzlang, it's possible to specify a condition for every struct field that
determines whether the field should be included or omitted:
header_fields {
magic const[0xabcd, int16]
haveInteger int8
} [packed]
packet {
header header_fields
integer int64 (if[value[header:haveInteger] == 0x1])
body array[int8]
} [packed]
some_call(a ptr[in, packet])In this example, the packet structure will include the field integer only
if header.haveInteger == 1. In memory, packet will have the following
layout:
| header.magic = 0xabcd | header.haveInteger = 0x1 | integer | body |
|---|
That corresponds to e.g. the following program:
some_call(&AUTO={{AUTO, 0x1}, @value=0xabcd, []})If header.haveInteger is not 1, syzkaller will just pretend that the fieldinteger does not exist.
some_call(&AUTO={{AUTO, 0x0}, @void, []})| header.magic = 0xabcd | header.haveInteger = 0x0 | body |
|---|
Every conditional field is assumed to be of variable length and so is the struct
to which this field belongs.
When a variable length field appears in the middle of a structure, the structure
must be marked with [packed].
Conditions on bitfields are prohibited:
struct {
f0 int
f1 int:3 (if[value[f0] == 0x1]) # It will not compile.
}But you may reference bitfields in your conditions:
struct {
f0 int:1
f1 int:7
f2 int (if[value[f0] == value[f1]])
} [packed]In unions
Let's consider the following example.
struct {
type int
body alternatives
}
alternatives [
int int64 (if[value[struct:type] == 0x1])
arr array[int64, 5] (if[value[struct:type] == 0x2])
default int32
] [varlen]
some_call(a ptr[in, struct])In this case, the union option will be selected depending on the value of thetype field. For example, if type is 0x1, then it can be either int ordefault:
some_call(&AUTO={0x1, @int=0x123})
some_call(&AUTO={0x1, @default=0x123})If type is 0x2, it can be either arr or default.
If type is neither 0x1 nor 0x2, syzkaller may only select default:
some_call(&AUTO={0x0, @default=0xabcd})To ensure that a union can always be constructed, the last union field must always
have no condition.
Thus, the following definition would fail to compile:
alternatives [
int int64 (if[value[struct:type] == 0x1])
arr array[int64, 5] (if[value[struct:type] == 0x1])
] [varlen]During prog mutation and generation syzkaller will select a random union field
whose condition is satisfied.
Expression syntax
Currently, only ==, !=, & and || operators are supported. However, the
functionality was designed in such a way that adding more operators is easy.
Feel free to file a GitHub issue or write us an email in case it's needed.
Expressions are evaluated as int64 values. If the final result of an
expression is not 0, it's assumed to be satisfied.
If you want to reference a field's value, you can do it viavalue[path:to:field], which is similar to the len[] argument.
sub_struct {
f0 int
# Reference a field in a parent struct.
f1 int (if[value[struct:f2]]) # Same as if[value[struct:f2] != 0].
}
struct {
f2 int
f3 sub_struct
f4 int (if[value[f2] == 0x2]) # Reference a sibling field.
f5 int (if[value[f3:f0] == 0x1]) # Reference a nested field.
f6 int (if[value[f3:f0] == 0x1 || value[f3:f0] == 0x2]) # Reference a nested field which either equals to 0x1 or 0x2.
} [packed]
call(a ptr[in, struct])The referenced field must be of integer type and there must be no
conditional fields in the path to it. For example, the following
descriptions will not compile.
struct {
f0 int
f1 int (if[value[f0] == 0x1])
f2 int (if[value[f1] == 0x1])
}You may also reference constants in expressions:
struct {
f0 int
f1 int
f2 int (if[value[f0] & SOME_CONST == OTHER_CONST])
}Meta
Description files can also contain meta directives that specify meta-information for the whole file.
meta noextractTells make extract to not extract constants for this file.
Though, syz-extract can still be invoked manually on this file.
meta arches["arch1", "arch2"]Restricts this file only to the given set of architectures.make extract and make generate will not use it on other architectures.
Misc
Description files also contain include directives that refer to Linux kernel header files,incdir directives that refer to custom Linux kernel header directories
and define directives that define symbolic constant values.
The syzkaller executor defines some pseudo system calls
that can be used as any other syscall in a description file. These pseudo
system calls expand to literal C code and can perform user-defined
custom actions. You can find some examples in
executor/common_linux.h.
Also see tips on writing good descriptions.
Syz Dungeon (`docs/syz-dungeon.md`)
Syzkaller Dungeon
Overview
The Syzkaller Dungeon (/syz-dungeon) ranks contributors (kernel developers) based on their achievements in fixing bugs found by Syzkaller. It uses a retro "terminal hacker" aesthetic with green-on-black text and ASCII art.
Note: Currently, the Dungeon is only available for the upstream namespace.
Core Mechanics
1. Character Generation
Each contributor is assigned a "Character" based on their bug fixing history during a specific time period.
Eras (Time Ranges)
The Dungeon provides two separate views or "Eras" for statistics:
1 Year (Default): A rolling window of exactly the last 365 days from today.
All Time: A comprehensive view of all bugs fixed since the beginning of tracking.
Era Scoping & Decay: In the 1-Year Era, the entire character sheet (including XP, Level, Class, Badges, and Stats like STR/INT/WIS) is dynamically recalculated based exclusively on the bugs currently within the 365-day rolling window. As fixes age out of the window, these attributes decay. In the All-Time Era, bugs never fall out of the window, providing a permanent, non-decaying global view of the contributor's achievements.
Eligibility: Bugs must have landed a fix commit within the given period.
- Criteria: Includes both fully fixed bugs (on the Fixed page) and partially fixed bugs (still Open but with upstream commits landing in the given period).
- Co-Authored Fixes: If a bug is resolved by multiple commits, all unique commit authors recorded in the bug's history are independently granted 1 bug count and the associated XP/Score. This means a single bug can boost multiple heroes simultaneously, occasionally resulting in leaderboard totals that are higher than naive SQL queries that only map one author per bug.
- Date: The "date fixed" is determined strictly by
FixTime(when the fix commit was discovered).Closeddate is ignored.
Matching: Identity resolution uses strict exact matching on the lowercase Email address (
Authorfield).- No Email Grouping: Contributors using different email addresses (e.g.,
@gmail.comvs a corporate domain) are treated as separate characters.
- No Email Grouping: Contributors using different email addresses (e.g.,
Hero ID: Each hero is assigned a stable, 16-character hexadecimal ID. This is generated by calculating the hash of their lowercased email address (using SHA-1) and extracting the first 16 characters of the hexadecimal string representation of the hash. This provides a secure, deterministic identifier for individual Hero Profile Pages without exposing raw email addresses in URLs.
Hero Name: Derived from the
AuthorNamefield in the DatastoreCommitInfoentity, bound to the matched Email.
* If a contributor uses multiple names for the same email (e.g.John DoevsJ. Doe), the system deterministically assigns the most frequently used name to the Hero. Ties are broken alphabetically.- Fallback: If an email never encounters a valid author name across all of its recorded bugs, the hero's name will gracefully fall back to displaying just the raw email address.
Bots: Bots are implicitly excluded as only humans land fixing commits in the Linux kernel (even bot-generated patches are signed by a human).
Heroic Title: An adjective is appended to the name based on the first letter, e.g., "Eric Dumazet The Epic".
- Full List:
A: The ArcaneB: The BraveC: The CosmicD: The DevotedE: The EpicF: The FierceG: The GallantH: The HeroicI: The ImmortalJ: The JustK: The KnightlyL: The LegendaryM: The MysticN: The NobleO: The OmniscientP: The PrimeQ: The QuickR: The RadiantS: The SacredT: The TitanU: The UndyingV: The ValiantW: The WiseX: The XenialY: The YouthfulZ: The Zealous
- Identity: Defined by the dynamic Hero Name calculated above (which selects the first valid name encountered for the email, or falls back to the raw email).
- No Title for Fallbacks: If the Hero Name had to fall back to the raw Email because no valid name was found across their commits, no heroic title modifier is added. The hero is simply displayed as their address (e.g.
[email protected]).
- No Title for Fallbacks: If the Hero Name had to fall back to the raw Email because no valid name was found across their commits, no heroic title modifier is added. The hero is simply displayed as their address (e.g.
- Logic: The title is determined by the first letter of the resolved name.
- Normalization: The first letter is lowercased before lookup.
- Map: 26 adjectives corresponding to 'a'-'z'.
- Fallback: If the name begins with a non-alphabetic character (numbers, symbols, or non-Latin scripts not in the a-z map), the title defaults to "The Adventurer". This fallback applies universally without transliteration (no UTF-8 awareness).
- Note: There is currently no mechanism for manual identity overrides or merging separate profiles.
- Full List:
Class: Determined by analyzing the existing tags attached to the bug.
- Multi-Class Credit: A contributor receives exactly 1 point for every subsystem tag present on a fixed bug. For example, a bug tagged with
netandmmincrements the contributor's count for bothnetandmmby 1. - Dynamic Class: The "Main Class" is determined after calculating the totals for all valid bugs. The system finds the single subsystem tag that appears most frequently across all of the contributor's fixes, and assigns the Class corresponding to that winning tag.
- Source: Classes are determined by mapping the winning tag (subsystem) to the RPG categories below.
- Weaver (🕸️): Networking & BPF.
- Tags:
net,ipv4,ipv6,tcp,udp,sctp,bpf,wireless,bluetooth,can,tipc,nfc,bridge,netlink.
- Tags:
- Geomancer (🏔️): Filesystems & Storage.
- Tags:
fs,ext4,btrfs,xfs,vfs,block,scsi,nvme,io-uring,fuse,overlayfs.
- Tags:
- Voidwalker (🔮): Memory Management.
- Tags:
mm,kasan,slab,kcsan,kmemleak.
- Tags:
- Paladin (🛡️): Security & Crypto.
- Tags:
security,selinux,apparmor,lsm,audit,keyrings,crypto.
- Tags:
- Bard (🎻): Media, Sound, Input, Human Interface.
- Tags:
sound,usb,media,video,drm,gpu,input,hid,drivers(generic).
- Tags:
- Artificer (🛠️): Hardware Buses & Infrastructure.
- Tags:
pci,i2c,spi,gpio,clk,pm,dmaengine,iio,rtc,watchdog.
- Tags:
- Warlock (🌀): Virtualization.
- Tags:
kvm,xen,virt,hyperv.
- Tags:
- Rogue (🗡️): Architecture Specific.
- Tags:
arch,arm,arm64,x86,riscv,s390.
- Tags:
- Druid (🌿): Core Kernel & Sync.
- Tags:
kernel,sched,rcu,locking,trace,cgroups.
- Tags:
- Warrior (⚔️): Default class for other subsystems or if no tags match any class.
- Tie-Breaking: If a contributor has multiple tags with the exact same highest frequency count, the first subsystem tag alphabetically is chosen as the winner. Warrior is only used if there are no specialized tags.
- Multi-Class Credit: A contributor receives exactly 1 point for every subsystem tag present on a fixed bug. For example, a bug tagged with
2. Experience (XP) Calculation
Score is calculated for each fixed bug:
- Base XP: 100
- Boss Bonus: 10% of crash count (Capped at 500 XP).
- Formula:
ceil(NumCrashes * 0.1) - Note: If the bug data is missing crash counts or has 0 crashes, this bonus is 0.
- Formula:
- Ancient Evil Bonus: 1 XP for every day (measured as a precise 24-hour elapsed duration, i.e.,
hours / 24) the bug was open (Capped at 300 XP).- Note: if
FirstTimeis missing, this bonus defaults to 0. - Floor: If duration is negative (due to clock skew), it is treated as 0 days (0 bonus XP). This prevents "negative XP".
- Note: if
- Windwalker Bonus: +250 XP flat bonus if the bug was fixed within 7 days. This bonus defaults to 0 if either the
FirstTimeorFixTimetimestamps are missing. - Blind Bonus: 150 XP if the bug had no reproducer (
ReproLevelNone).
3. Leveling
- Level 1: 0 XP
- Progressive Level Base: 1,000 XP per level up to Level 50.
- Legendary Scaling: Starting at Level 50, each subsequent level requires 500 more XP than the previous level's requirement (e.g., Level 50 -> 51 costs 1,500 XP, 51 -> 52 costs 2,000 XP).
- Mathematical Formula: The total XP required to reach Level $N$, denoted as $TotalXP(N)$, is calculated strictly by the following piecewise function:
- For $N \le 50$: $TotalXP(N) = (N - 1) \times 1000$
- For $N > 50$: Let $m = N - 50$. The total XP is defined by the arithmetic series sum:
$TotalXP(N) = 49000 + 1000m + 250m(m + 1)$
- Concrete Threshold Table:
Level 1: 0 XPLevel 2: 1,000 XPLevel 49: 48,000 XPLevel 50: 49,000 XPLevel 51: 50,500 XPLevel 52: 52,500 XPLevel 53: 55,000 XP
- Cap: There is no hard level cap.
- Note: Rapid leveling (e.g., jumping 2.5 levels from a single legendary bug fix) is intentional to reward high-impact contributions.
4. Inventory (Badges)
Contributors earn badges for specific achievements. Badges are purely cosmetic and do not affect XP or Rank.
General Badge Rules:
- All string/title matching is case-insensitive.
- Bugs missing
FirstTimeorFixTimefields cannot be evaluated and therefore do not count toward time-based badges. - Dragon Vanquisher (🐉): Fixed a bug with >1000 crashes.
- Necromancer (💀): Fixed a bug open for >365 days.
- Diviner (👁️): Fixed a bug with no reproducer.
- Windwalker (🪽): Fixed a bug within 7 days (measured as precise 24-hour duration chunks) of it being reported (based on
FirstTime).- Logic:
FixTime - FirstTime < 7 days. - Data Integrity: This calculation is skipped (badge not awarded) if either
FirstTimeorFixTimetimestamps are missing.
- Logic:
- 🔥 Hydra Hunter: Fixed 10 or more bugs overall.
- 🧌 Troll Crusher: Fixed 50 or more bugs overall.
- 🐙 Kraken Bane: Fixed 100 or more bugs overall.
- 👹 Demon Slayer: Fixed 250 or more bugs overall.
- 🔱 Leviathan Conqueror: Fixed 500 or more bugs overall.
- 🏺 Sealer: Fixed a memory leak.
- Logic: Title or Fix Commit Title contains tokens
leak,leaks,memory-leak, ormemory-leaks(with word boundaries checking).
- Logic: Title or Fix Commit Title contains tokens
- 📜 Keeper of Secrets: Fixed an infoleak vulnerability.
- Logic: Title or Fix Commit Title contains
infoleak(matched as a simple substring, without word boundaries or special character tokenization).
- Logic: Title or Fix Commit Title contains
- ⏳ Master of Time: Fixed a time-related bug.
- Logic: Title or Fix Commit Title contains
time:ortimer:(matched as a simple substring, without word boundaries or special character tokenization, matching things likehrtimer:orposix-timer:).
- Logic: Title or Fix Commit Title contains
- 🛶 Pathfinder: Fixed an out-of-bounds error. (String match:
out-of-boundsanywhere in bug title/commit titles). - 🦄 Beast Tamer: Fixed a data race condition. (String match:
data-raceanywhere in bug title/commit titles). - ⚒️ Armorer: Fixed a general protection fault. (String match:
general protection faultanywhere in bug title/commit titles). - 🪦 Grave Robber: Fixed a use-after-free error. (String match:
use-after-freeanywhere in bug title/commit titles). - 🗝️ Locksmith: Fixed a locking issue. (Matches tokens:
lock,deadlock,lockup,locking,mutex,unlock,locks,rwlock,spinlockin bug title/commit titles). - 🕯️ Exorcist: Fixed a hung task or zombie process. (String match:
hung task,task hung, orzombieanywhere in bug title/commit titles). - 🕳️ Abyss Walker: Fixed a NULL pointer dereference. (String match:
null-ptr-derefornull pointeranywhere in bug title/commit titles). - ⚗️ Alchemist: Fixed an uninitialized value bug. (String match:
uninitorkmsananywhere in bug title/commit titles). - 🪓 Executioner: Fixed a double free bug. (String match:
double-freeordouble freeanywhere in bug title/commit titles).
5. General Rules
- Alphabetical Tie-Breakers: Unless otherwise specified, all "alphabetical" sorting and tie-breaking used throughout the Dungeon (for Names, Kingdom Main Classes, Guilds Array, Kingdom Ranks, etc.) is strictly case-sensitive and lexicographically ascending. This guarantees absolute determinism derived from underlying ASCII/UTF-8 byte representations (e.g., uppercase 'Z' sorts before lowercase 'a').
6. Kingdoms
Players can view aggregate statistics via two toggles in the dashboard menu:
- Entities: Choose between viewing the standard individual Heroes leaderboard, or the Kingdoms ranking.
- Kingdoms: View aggregated points and heroic classes across different organizations playing the dungeon.
- Clicking a Kingdom's name takes you to the Kingdom Profile Page where you can see their overall Trophies, Hero Count, Total XP, and established Guilds.
- Employees are assigned to their company based on a curated list of email domains. The matching rule requires an exact string match against the entire portion of the email address following the
@symbol (e.g.,[email protected]will not matchcompany.com).- The Google Kingdom:
google.com,chromium.org - The Red Hat Kingdom:
redhat.com - The Huawei Kingdom:
huawei.com,hisilicon.com - The Linux Foundation Kingdom:
linuxfoundation.org,kernel.org - The Meta Kingdom:
meta.com,fb.com - The Intel Kingdom:
intel.com,linux.intel.com - The AMD Kingdom:
amd.com - The ARM Kingdom:
arm.com - The Oracle Kingdom:
oracle.com - The SUSE Kingdom:
suse.com,suse.de,suse.cz - The Linaro Kingdom:
linaro.org - The Alibaba Kingdom:
linux.alibaba.com - The NVIDIA Kingdom:
nvidia.com,mellanox.com - The IBM Kingdom:
linux.ibm.com,linux.vnet.ibm.com - The Bytedance Kingdom:
bytedance.com - The Canonical Kingdom:
canonical.com,ubuntu.com - The Amazon Kingdom:
amazon.com,amazon.de,amazon.co.uk,amazon.co.jp - The Tencent Kingdom:
tencent.com - The Samsung Kingdom:
samsung.com
- The Google Kingdom:
- Any hero not in the curated list belongs to
The Independent Mercenaries. - Royal Champion: A Kingdom's Royal Champion is determined by finding the single hero associated with that Kingdom who has the highest
Score(XP) contribution in the given Era. The Champion's Name, XP contribution, and a link to their profile are featured on the Kingdom's Profile Page. - Total Hero XP: A Kingdom's total
Scoreis purely the combined sum of theScoreof all its affiliatedHeroes. This means all dynamic bug bonuses (like the Boss Bonus for crashes, Ancient Evil bonus for old bugs, etc.) earned by a hero directly benefit their aligned Kingdom. However, the Kingdom's hero count and specific class composition metric will correctly reflect all distinct individuals who participated. If the fix is co-authored by heroes from different Kingdoms, both Kingdoms independently receive full credit for the bug. - Trophy Deduplication: To prevent double-counting, if a bug is resolved jointly by multiple heroes who belong to the same Kingdom, the Kingdom is credited exactly 1 Trophy for resolving the bug.
- Stats: Kingdoms do not possess individual stats like STR, INT, or WIS.
- Instead of "Inventory" (Badges), Kingdoms display "Guilds", which details the composition of Hero Classes based on the total count of players in the Kingdom (e.g., "3 Weavers", "1 Geomancer"). The array is sorted descending by the count of players, and then alphabetically by the resulting string representation in the event of a tie. Pluralization is handled simply: an
"s"is appended to the Class Name if the player count is strictly greater than 1. - Sorting: Kingdoms are ranked and sorted descending by their Total XP (Score). Ties are broken alphabetically by the Kingdom Name.
- Territory Tiers: A Kingdom evaluates its overall combined
Score(XP) strictly into one of the following brackets, displaying an associated name and emoji. On the Kingdom Profile page, the tier icon is displayed prominently with the tier name below and an XP range tooltip. On the global Leaderboard, the tier emoji is displayed as an inline badge next to the Kingdom's name (with a tooltip revealing the tier name).- 0 - 49,999: Outpost (⛺)
- 50,000 - 149,999: Barony (🏘️)
- 150,000 - 499,999: Duchy (🏛️)
- 500,000 - 999,999: Kingdom (🏰)
- 1,000,000+: Empire (👑)
Technical Implementation
Backend (`dashboard/app/dungeon.go`)
- JSON API:
/syz-dungeon?json=1returns the calculated scaled stats (RPG range [3-30]) forstr,int,wis. - Endpoint:
/syz-dungeon(Leaderboard Overview) - Hero Profile Endpoint:
/syz-dungeon/hero/{id}(Individual Hero Sheet)- Parameters: Supports
?json=1(JSON dump) and?era=1yor?era=all(defaults toall). Hero pages will display an interactive[1 YEAR]|[ALL TIME]toggle if the hero has contributions mapped into both era scopes. - Class Box: Displays the hero's class emoji, name, and a tooltip containing the class's flavor text (description).
- Attributes Box: Displays the hero's STR, INT, and WIS properties. Hovering over each stat reveals a tooltip explaining its corresponding metric (e.g., "Max Crashes (Scaled)").
- Badges Box: Displays earned badges. Hovering over a badge reveals its name and an extended description characterizing the achievement.
- Click-to-Copy ID: The
IDdisplayed in the footer is clickable and will copy the URL to the profile into the user's clipboard.
- Parameters: Supports
- Data Source: Queries the
Bugentity in Google Cloud Datastore.- Definition:
dashboard/app/entities_datastore.go - Key Fields Used:
Namespace,Status: For Datastore query filtering. (Note: Only theupstreamnamespace is currently supported).- Logic:
Statusmust be Fixed (or Open if fixing commits are present). - Index: Requires composite index on
(Namespace, Status)inindex.yaml. FixTime: To filter bugs within the given period (in UTC).- Decay: Since the cache refreshes hourly, players viewing the 1-Year Era will see their entire character sheet—including stats, badges, levels, classes, and ranks—actively decay day-to-day as old bugs fall out of the 365-day window. No notifications are sent for level down; users monitor their status on the dashboard.
Tags: For Class determination (instead of Title).CommitInfo: For authorship identification (AuthorNameandAuthorfields).Title: For "Sealer", "Keeper of Secrets", and "Master of Time" badge logic.NumCrashes: For "Boss Bonus", "Dragon Slayer" badge, and STR.ReproLevel: For "Blind Bonus", "Diviner" badge, and INT.FirstTime&FixTime: For "Ancient Evil" bonus, "Windwalker"/"Necromancer" badges, and WIS.
- Definition:
- Caching: Results are cached in shared App Engine Memcache (using compressed JSON) for 1 hour. The cache is segregated by the viewer's
AccessLevelso that privileged bug data (like undisclosed bugs visible to admins) does not leak to public viewers.- Warm-up/Preheating: A cron job (
/cron/dungeon_preheat) runs hourly to precompute and warm the cache for all access levels simultaneously, ensuring fast page loads for all users across multiple live instances.
- Warm-up/Preheating: A cron job (
- Logic:
- Identity Resolution: Aggregates bugs strictly by the lowercased Email address.
- Rule: "Linus Torvalds " and "Linus Torvalds " are treated as separate characters.
- Name Selection: If an author uses multiple names for the same email address across different commits, the system assigns the most frequently used name to the Hero. Ties are broken alphabetically.
- Determines the "Main Spec" (class) based on subsystem counts.
- Calculates XP and assigns badges.
- Sorts players by Score (descending), then Name (ascending), then Email (ascending).
- Pagination: The backend server returns the complete active roster of players. The frontend JavaScript dynamically partitions the tables into interactive pages of 100 rows.
- Identity Resolution: Aggregates bugs strictly by the lowercased Email address.
Frontend (`dashboard/app/templates/dungeon.html`)
- Style: Dark mode with
Courier New/ monospace font.- Accent Color: Neon Green (
#0f0). - Banner: Retro pixel-art PNG image "SYZKALLER DUNGEON" (responsive, located at
/static/dungeon.png).
- Accent Color: Neon Green (
- Interactivity:
- Hover Tooltips: Hovering over a character row reveals a "Character Sheet":
- Desktop Users: Appears on Hover.
- Mobile/Touch Users: Appears on Click/Tap. (Tapping a row toggles the sheet; opening a new one closes the previous one).
- NAME: "Name Adjective" (e.g., "Eric Dumazet The Epic")
- CLASS: "Emoji (Class)" (e.g., "🕸️ (Weaver)")
- LEVEL: Current Level
- Stats Scaling:
- Instead of raw counts, metrics are mapped to an RPG-like scale [3, 30] using a logarithmic function.
- STR (Strength): Based on Max Crashes of a single fixed bug.
- Formula: $STR = \min(30, \max(3, \lfloor 3 + 4 \times \log_{10}(\max(1, \text{MaxCrashes})) \rfloor))$
- Example: 1 crash = 3, 100 crashes = 11, 1,000,000 crashes = 27.
- INT (Intelligence): Based on total Blind Fixes (no reproducer).
- Formula: $INT = \min(30, \max(3, \lfloor 3 + 10 \times \log_{10}(\text{BlindFixes} + 1) \rfloor))$
- Example: 0 blind fixes = 3, 2 blind fixes = 7, 10 blind fixes = 13, 500 blind fixes = 30.
- WIS (Wisdom): Based on Oldest Kill (maximum duration of a single bug).
- Formula: $WIS = \min(30, \max(3, \lfloor 3 + 5 \times \log_{10}(\text{DaysOpen} + 1) \rfloor))$
- Example: 1 day = 4, 100 days = 13, 1000 days = 18, 250,000 days = 30.
- Inventory Tooltips: Hovering over badges shows their name/description.
- Hover Tooltips: Hovering over a character row reveals a "Character Sheet":
Configuration
- JSON Output: The endpoint supports
?json=1to return the leaderboard data as JSON.- Schema:text
- Schema:
/* Detailed source-code truncated for AI context efficiency. */
* **Hero Profile Schema** (`/syz-dungeon/hero/{id}?json=1`): Returns a single JSON object mirroring the `heroes_1y` / `heroes_all` definition depending on the evaluated `?era=` query parameter.
```json
{
"id": "string",
"name": "string",
"rank": "int",
"score": "int",
"level": "int",
"class_name": "string",
"class_emoji": "string",
"class_desc": "string",
"badges": [{ "emoji": "string", "name": "string", "desc": "string" }],
"bug_count": "int",
"str": "int",
"int": "int",
"wis": "int"
}
```
* **Kingdom Profile Schema** (`/syz-dungeon/kingdom/{id}?json=1`): Returns a single JSON object mirroring the `kingdoms_1y` / `kingdoms_all` definition depending on the evaluated `?era=` query parameter.
```json
{
"id": "string",
"name": "string",
"rank": "int",
"score": "int",
"tier": "string",
"tier_emoji": "string",
"tier_desc": "string",
"champion_name": "string",
"champion_score": "int",
"champion_id": "string",
"trophies": "int",
"heroes": "int",
"guilds": [{ "emoji": "string", "name": "string", "desc": "string" }]
}
```
* **Heroic Adjectives**: Hardcoded map in `dungeon.go` for A-Z initials.
---
### Syz Kfuzztest (`docs/syz-kfuzztest.md`)
# `syz-kfuzztest`
`syz-kfuzztest` is a standalone tool for fuzzing KFuzzTest targets from within
the kernel being fuzzed (e.g., a VM).
It is intended to be used for development-time fuzzing rather than continuous
fuzzing like `syz-manager`.
For more information on KFuzzTest, consult the [dedicated readme](kfuzztest.md)
or the Kernel documentation.
## Usage (in-VM fuzzing)
### Getting the Kernel Ready
It is important that the target Kernel image has the correct KConfig options
enabled. Namely
- `CONFIG_KFUZZTEST`
- `CONFIG_DEBUG_FS`
- `CONFIG_DEBUG_KERNEL`
- `CONFIG_KCOV`
- `CONFIG_DWARF4` or `CONFIG_DWARF5`
- `CONFIG_KASAN` _(optional, choose your favorite sanitizers for a better shot
at finding bugs!)_
Furthermore, as you will need to connect to the VM being tested through SSH and
launch `syz-kfuzztest` _(a Go binary with LIBC dependencies)_, it is recommended
to create an image for the kernel being fuzzed (e.g., a Debian Trixie image).
Detailed instructions on how to do this can be found in
[this setup guide](linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md).
### Building and Launching the Binary
The `syz-kfuzztest` binary is built with `make syz-kfuzztest`, and is intended
to run on the Kernel fuzzed. The common case for this is within a VM _(after
all, the tool is trying to make the Kernel crash)_.
Then, ensure that the `syz-kfuzztest` binary and `vmlinux` image are copied
over into the VM. E.g.,
```sh
scp $KERNEL/vmlinux root@my-vm:~/syz-kfuzztest/vmlinux
scp $SYZKALLER/bin/syz-kfuzztest root@lmy-vm:~/syz-kfuzztest/syz-kfuzztestThen launched like this:
usage: ./bin/syz-kfuzztest [flags] [enabled targets]
Args:
One fuzz test name per enabled fuzz test arg. If empty, defaults to
all discovered targets.
Example:
./syz-kfuzztest -vmlinux ~/kernel/vmlinux fuzz_target_0 fuzz_target_1
Flags:
-display int
Number of seconds between console outputs (default 5)
-threads int
Number of threads (default 2)
-timeout int
Timeout between program executions in seconds (default 0)
-vmlinux string
Path to vmlinux binary (default "vmlinux")
-vv int
verbosityThe enabled targets, which are listed after the flag arguments, are the names of
the enabled fuzz targets. For example given some KFuzzTest targets:
FUZZ_TEST(kfuzztest_target_1, struct input_arg_type)
{
/* ... */
}
FUZZ_TEST(kfuzztest_target_2, struct input_arg_type)
{
/* ... */
}Can be fuzzed with:
./syz-kfuzztest -vmlinux path/to/vmlinux -threads 4 kfuzztest_target_1 kfuzztest_target_2If the enabled targets list is left empty, syz-kfuzztest will fuzz all
discovered targets in the kernel.
On exit, syz-kfuzztest will write the collected program counters (which are
collected with KCOV) into a file called pcs.out. These program counters can
be fed into syz-cover to generate an HTML
visualization of the lines that were covered during fuzzing. It is recommended
to do this on the host machine rather than the VM.
For example:
scp root@my-vm:~/syz-kfuzztest/pcs.out .
go run tools/syz-cover -config my.cfg pcs.out # May require the -force flag.Syz Testbed (`docs/syz_testbed.md`)
syz-testbed
syz-testbed is the tool that simplifies the process of evaluating the
performance of different syzkaller versions (or configurations) against each
other. The tool automates checking out syzkaller repos, building them, runningsyz-managers and collecting/summarizing their results.
Configuring syz-testbed
syz-testbed requires a JSON config file. An example of such a file:
{
"workdir": "/tmp/syz-testbed-workdir/",
"corpus": "/tmp/corpus.db",
"target": "syz-manager",
"max_instances": 5,
"run_time": "24h",
"http": "0.0.0.0:50000",
"checkouts": [
{
"name": "first",
"repo": "https://github.com/google/syzkaller.git",
},
{
"name": "second",
"repo": "https://github.com/google/syzkaller.git",
"branch": "some-dev-branch",
}
],
"manager_config": {
"target": "linux/amd64",
"kernel_obj": "/tmp/linux-stable",
"image": "/tmp/kernel-image/trixie.img",
"sshkey": "/tmp/kernel-image/trixie.id_rsa",
"procs": 8,
"type": "qemu",
"vm": {
"count": 2,
"kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}
}Given such a configuration file, syz-testbed will do the following:
- Check out the
masterbranch ofhttps://github.com/google/syzkaller.git
into/tmp/syz-testbed-workdir/checkouts/first/and build it. - Check out the
some-dev-branchofhttps://github.com/google/syzkaller.git
into/tmp/syz-testbed-workdir/checkouts/second/and build it. - Set up and run 3 instances of
firstand 2 instances ofsecond
(max_instances = 5).
The directory structure looks as follows:
/tmp/syz-testbed-workdir/
└── checkouts
├── first
│ ├── run-first-0
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
│ ├── run-first-1
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
│ └── run-first-4
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
└── second
├── run-second-2
│ ├── log.txt
│ ├── manager.cfg
│ └── workdir
└── run-second-3
├── log.txt
├── manager.cfg
└── workdir- After 24 hours (as
run_hoursis 24), stop those 5 instances. - Create and run 2 instances of
firstand 3 instances ofsecond. - Repeat those steps over and over
The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM
signal. Also, if one of the instances has exited due to some error, this also
stops the whole experiment.
Web interface
The tool has a simple web interface that displays the current information about
the experiment (the number of active and finished instances, the time until
instances are stopped, etc.) and the latest statistics collected from thesyz-managers.
If the benchmp parameter points to the syz-benchcmp executable, then the web
interface can also generate graphs of various parameters over time or the number
of executions.
In order to enable the interface, set the http parameter to the IP address and
port to which syz-testbed should bind. E.g. "http": "0.0.0.0:50000".
Statistics
syz-testbed provides two "views" of the statistics:
complete- only includes data from the finished instances (i.e. those that
have been running forrun_hours).all- also includes the data from the currently active instances. The
statistics from the finished instances is winded back to match the current
uptime of the active instances.
Therefore, the statistics is laid out the following way.
$ tree -L 2 /tmp/syz-testbed-workdir/
/tmp/syz-testbed-workdir/
├── stats_all
│ ├── benches
│ │ ├── avg_first.txt
│ │ ├── avg_second.txt
│ ├── bugs.csv
│ ├── checkout_stats.csv
│ └── instance_stats.csv
├── stats_completed
│ ├── benches
│ │ ├── avg_first.txt
│ │ ├── avg_second.txt
│ ├── bugs.csv
│ ├── checkout_stats.csv
│ └── instance_stats.csv
└── testbed.csvbugs.csvcontains all the bugs found by the running instances. If a single
checkout has several instances (i.e.count> 1),syz-testbedtakes a
union of bugs found by them. The purpose is ultimately to collect all bugs
that could be found by that version of syzkaller.- Statistics that is generated by individual
syz-managers is saved intoinstance_stats.csv. The same data is also averaged among instances that
belong to the same checkouts and saved into thecheckout_stats.csvfile. - Bench files (see
tools/syz-benchcmp) of allsyz-managers belonging to a
single checkout are averaged and saved into the corresponding files in thebenchesfolder.
The statics is updated once every 90 seconds.
Running syz-testbed
First, checkout the most recent version of syzkaller itself:
$ git clone https://github.com/google/syzkaller.gitThen, build syz-testbed:
$ cd syzkaller/tools/syz-testbed/
$ go buildWrite and save the configuration file (e.g. into the config.json file). Then,syz-testbed can be run using the following command:
$ ./syz-testbed -config config.jsonStopping the syz-testbed process results in stopping all the syzkaller instances.
Testing syz-repro
syz-testbed can also be used to test syzkaller's ability to reproduce bugs. To do
so, set the target property in the syz-testbed's config file to syz-repro.
One can also specify the source of the crash log files. This is either just a folder,
whose files will be treated accordingly or it can be a syzkaller's workdir.input_logs must point to the folder with crash logs - syz-testbed will traverse
it and treat each file as an input. Otherwise, input_workdir must be used.
For example:
"repro_config": {
"input_workdir": "/tmp/some-syzkaller-workdir",
"crashes_per_bug": 2,
"skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"]
},In this case, syz-testbed will traverse all bugs found by the syzkaller, skip
those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then
pick 2 random crash logs for each such bug for later processing.
syz-testbed will check out and compile the specified syzkaller instances and will
go on executing their syz-repros on each picked up crash log file, as long as
the tool is not stopped.
Syz Verifier (`docs/syz_verifier.md`)
syz-verifier
Many bugs are easy to detect: they might cause assertions failures, crash our
system, or cause other forms of undefined behaviour detectable by various
dynamic analysis tools. However, certain classes of bugs, referred to as
semantic bugs, cause none of these while still resulting in a misbehaving
faulty system.
To find semantic bugs, one needs to establish a specification of the system's
intended behaviour. Depending on the complexity of the system, creating and
centralising such specifications can be difficult. For example, the
"specification" of the Linux kernel is not found in one place, but is rather a
collection of documentation, man pages, and the implied expectations of a vast
collection of user space programs. As such, detecting semantic bugs in the
Linux kernel is significantly harder than other classes of bugs. Indeed, many
test suites are meant to detect regressions, but creating and maintaining test
cases, as well as covering new features requires significant amounts of
engineering effort.
One way to automate detection of semantic bugs is to provide the same input to
different implementations, or different versions of the same system, and then
cross-compare the resulting behaviour. In case the systems disagree, at least
one of them is assumed to be wrong.
The current syz-verifier is not fuzzing new inputs. Instead, it loads a
corpus of existing syzkaller programs and cross-compares their execution on
different versions of the Linux kernel to detect semantic bugs.
The architecture of syz-verifier is shown in the following diagram.
The syz-verifier process starts and manages VM instances with the kernels to
be cross-compared. It also starts the syz-runner process on the VMs.
Communication between the host and the guest is done via RPCs.
At startup, syz-verifier loads programs from a syzkaller corpus database
(corpus.db). It then sends each corpus program to syz-runner on every VM
via RPCs while syz-runner is responsible for starting syz-executor
processes and turning the program into input for those.syz-executor processes the input, which triggers a sequence of syscalls in
the kernel. Then, syz-runner collects the results and sends them back to the
host.
At the moment, the results contain the errnos returned by each system call.
When syz-verifier has received results from all the kernels for a specific
program, it verifies them to ensure they are identical. If a mismatch is found,syz-verifier creates a report for the founded mismatch including the program mismatch details.
How to use `syz-verifier`
After cloning the repository (see how
here), build the tool as:
make verifier runner executorTo start using the tool, separate configuration files need to be created for
each kernel you want to include in the verification. An example of Linux
configs can be found here. The configuration files
are identical to those used by syz-manager.
All compared kernels must use the same workdir. syz-verifier loads the
corpus from corpus.db in that shared workdir, so the first config must
point at the workdir whose corpus you want to verify.
If you want to constrain verification to programs that use a specific set of
system calls, list them in the kernel config files using theenable_syscalls option. If you want to disable some system calls, use thedisable_syscalls option.
Start syz-verifier as:
./bin/syz-verifier -configs=kernel0.cfg,kernel1.cfgsyz-verifier logs its progress throughout execution, including corpus loading
and per-program comparison progress.
How to interpret the results
Results can be found in workdir/results.
When syz-verifier finds a mismatch in a program, it will create a report for
that program. The report lists the results returned for each system call, by
each of the cross-compared kernels, highlighting the ones were a mismatch was
found. The system calls are listed in the order they appear in the program.
An extract of such a report is shown below:
ERRNO mismatches found for program:
========== ERRNO MISMATCH DETECTED ==========
Between: Kernel 0 (kernel-6.17) and Kernel 1 (kernel-6.2)
Complete Program Sequence:
-------------------------------------------
[0] capset(&(0x7f0000000380)={0x19980330}, &(0x7f00000003c0))
Result: errno=0, flags=0x3
>>> [1] r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
>>> ┌─ : errno=0, flags=0x3
>>> └─ : errno=2, flags=0x3
>>> [2] setsockopt$inet6_tcp_TCP_CONGESTION(r0, 0x6, 0xd, &(0x7f0000000000)='cubic', 0x3)
>>> ┌─ : errno=2, flags=0x3
>>> └─ : errno=9, flags=0x3
-------------------------------------------
Kernel Outputs:
: ""
: ""
=============================================
...In this report, lines prefixed with >>> highlight the calls whose returned
errnos differed between kernels. The two indented result lines under such a
call show the values returned by each compared kernel for that exact syscall.
The Kernel Outputs section records the raw runner output for each kernel,
which is useful when a mismatch is caused by a crash, timeout, or some other
execution problem rather than a pure semantic difference.
Syzbot (`docs/syzbot.md`)
syzbot
syzbot system continuously fuzzes main Linux kernel branches and automatically
reports found bugs to kernel mailing lists.
syzbot dashboard shows current statuses of
bugs. All syzbot-reported bugs are also CCed to
syzkaller-bugs mailing list.
Direct all questions to [email protected].
Bug status tracking
syzbot needs to know when a bug is fixed in order to (1) verify that it is
in fact fixed and (2) be able to report other similarly-looking crashes
(while a bug is considered open all similarly-looking crashes are merged into
the existing bug). To understand when a bug is fixed syzbot needs to know
what commit fixes the bug; once syzbot knows the commit it will track when
the commit reaches all kernel builds on all tracked branches. Only when the
commit reaches all builds, the bug is considered closed (new similarly-looking
crashes create a new bug).
Communication with syzbot
If you fix a bug reported by syzbot, please add the provided Reported-by
tag to the commit. You can also communicate with syzbot by replying
to its emails. The commands are:
- to attach a fixing commit to the bug (if you forgot to add
Reported-bytag):
#syz fix: exact-commit-titleIt's enough that the commit is merged into any tree or you are reasonably sure
about its final title, in particular, you don't need to wait for the commit to
be merged into upstream tree. syzbot only needs to know the title by which
it will appear in tested trees. In case of an error or a title change, you can
override the commit simply by sending another #syz fix command.
- to undo a previous fix command and remove any fixing commits:
#syz unfix- to mark the bug as a duplicate of another
syzbotbug:
#syz dup: exact-subject-of-another-report- to undo a previous dup command and turn it into an independent bug again:
#syz undup- to mark the bug as a one-off invalid report (e.g. induced by a previous memory corruption):
#syz invalidNote: if the crash happens again, it will cause creation of a new bug report.
Note: all commands must start from beginning of the line.
Note: please keep at least [email protected] mailing list in CC.
It serves as a history of what happened with each bug report. Keeping the main kernel
mailing list (e.g. [email protected] ) in CC is useful as well so that
it's searchable in those archives as well.
Note: syzbot identifies bugs by the HASH in the syzbot+HASH@ receiver email address.
So, strictly saying, you don't need to reply to emails (e.g. if you did not receive them),
you can send a new email to the syzbot+HASH@ email address, which you can find as Sender
in email archives or as Reported-by email on the dashboard page for each bug.
Testing patches
syzbot can test patches for bugs with reproducers. This can be used for
testing of fix patches, or just for debugging (i.e. adding additional checks to
code and testing with them), or to check if the bug still happens. To test on
a particular git tree and branch reply with:
#syz test: git://repo/address.git branchor alternatively, to test on exact commit reply with:
#syz test: git://repo/address.git commit-hashYou can also completely omit these parameters:
#syz testIn this case, syzbot will check out the latest commit from the branch where the
issue was detected.
If you also provide a patch with the email, syzbot will apply it on top of the
tree before testing. The patch can be provided inline in email text or as
a text attachment (which is more reliable if your email client messes with
whitespaces).
If you want to include the patch directly in the email body, just paste the diff
somewhere under the #syz test command line, e.g.
#syz test: git://repo/address.git branch
--- a/mm/kasan/kasan.c
+++ b/mm/kasan/kasan.c
- current->kasan_depth++;
+ current->kasan_depth--;Here are
some real examples of #syz test commands for syzbot-reported bugs.
If you don't provide a patch, syzbot will test the tree as is.
This is useful if this is your own tree which already contains the patch,
or to check if the bug is already fixed by some recent commit.
After sending an email you should typically get a reply email with results within
an hour.
Note: you may send the request only to syzbot email address, as patches sent
to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork.
Note: when testing a patch, syzbot uses the newest reproducer and the matching
kernel config that are listed on the dashboard for this bug. As a result, specifying
a repo, branch or commit id that are different from the ones that were used for
reproducing, can result in false-positive Tested-by responses. For example this
happens, when the bug is not reproducible on a specified kernel tree, with or without
the supplied patch.
Note: see below for KMSAN bugs testing.
Note: see below for USB bugs testing.
Subsystems
For all its bugs, syzbot automatically assigns kernel subsystems tags. For Linux,
the predefined list of kernel subsystems can be found at
https://syzkaller.appspot.com/upstream/subsystems.
By clicking on a name in the subsystem list or by following a tag after a bug's
title, you can get the full list of bugs belonging to the subsystem. For example,
all nfc bugs are listed here: https://syzkaller.appspot.com/upstream/s/nfc.
syzbot includes subsystem tags into email subject as well, with ? indicating
that it's an automatic guess: [syzbot] [ntfs?] kernel BUG in ntfs_iget.
Over time, as we improve the classification rules or as syzbot obtains more
information about the bug (e.g. finds a reproducer), syzbot will update tags.
You can also manually override the automatic guess by replying to the syzbot email:
#syz set subsystems: net, mmNames of subsystems must be taken from the subsystem list page on the syzbot web
dashboard.
Bug labels
It is possible to assign labels to syzkaller-reported bugs. These labels
are displayed near bug titles on the bug lists and on individual bug pages.
There are two types of labels:
- Flags
- Send
#syz set <label>to set the flag. - Send
#syz unset <label>to drop the flag.
- Send
- Label with values
- Send
#syz set <label>: <value>[, <value2> ...]to overwrite the value list. #syz unset <label>would drop the label with all its values.
- Send
There's a fixed list of supported labels (*). Most important ones:
| Key | Description | Values | Example |
|---|---|---|---|
subsystems |
A comma-separated list of subsystems of the bug | See the list on the syzbot dashboard | #syz set subsystems: mm, reiserfs |
prio |
The priority of the bug | One of low, normal, high |
#syz set prio: low``#syz unset prio |
no-reminders |
Don't include the bug into monthly reminders | No values, just a flag | #syz set no-reminders``#syz unset no-reminders |
It is also possible to set and unset labels for individual bugs from a monthly
subsystem report. Let's consider an example.
Some of the still happening issues:
Ref Crashes Repro Title
<1> 10 No WARNING in __brelse (2)
https://syzkaller.appspot.com/bug?id=cd9cb7a620dcfbf46a5eaf201031acaf3aeda28e
<2> 3 No WARNING in kernfs_get (4)
https://syzkaller.appspot.com/bug?id=b4278401038872458a20f08210e46f3ab519b786One can send the following email to disable reminders for the first bug and to
change the subsystem of the second bug:
#syz set <1> no-reminders
#syz set <2> subsystems: kernfs(*) If you need other labels to facilitate your work, feel free to contact us.
Rebuilt trees/amended patches
There are several additional aspects if the tree is rebuilt/rebased or contains
amended/folded patches (namely, linux-next).
First, adding Reported-by tags to amended commits may be misleading.
A Reported-by tag suggests that the commit fixes a bug in some previous
commit, but here it's not the case (it only fixes a bug in a previous version
of itself which is not in the tree). In such case it's recommended to include
a Tested-by or a Reviewed-by tag with the hash instead. Technically,syzbot accepts any tag, so With-inputs-from would work too.
Then, if the guilty commit is completely dropped (actually removed from the
tree during rebuild), then there is effectively no fixing commit. There is no
good way to handle such cases at the moment. One possibility is to mark them
with #syz invalid. However this needs to be done only when syzbot picks up
the new tree in all builds (current kernel commits can be seen on dashboard).
Otherwise, the occurrences of the crash that are still happening in the current
build will immediately create a new bug report. Another possibility is to mark
it as fixed with some unrelated later commit using#syz fix: some-later-commit. This way syzbot will wait until the commit
propagates to all tested trees automatically.
In any case the relation between the report and the fix can later be fixed up
using #syz fix: commit-title commands.
Bisection
syzbot bisects bugs with reproducers to find commit that introduced the bug.syzbot starts with the commit on which the bug was discovered, ensures that it
can reproduce the bug and then goes back release-by-release to find the first
release where kernel does not crash. Once such release is found, syzbot starts
bisection on that range. syzbot has limitation of how far back in time it can
go (currently v4.19), going back in time is very hard
because of incompatible compiler/linker/asm/perl/make/libc/etc, kernel
build/boot breakages and large amounts of bugs.
The predicate for bisection is binary (crash/doesn't crash), syzbot does not
look at the exact crash and does not try to differentiate them. This is
intentional because lots of bugs can manifest in different ways (sometimes 50+
different ways). For each revision syzbot repeats testing 10 times and
a single crash marks revision as bad (lots of bugs are due to races and are
hard to trigger).
During bisection syzbot uses different compilers depending on kernel revision
(a single compiler can't build all revisions). These compilers are available
here.
Exact compiler used to test a particular revision is specified in the bisection
log.
Bisection is best-effort and may not find the right commit for multiple reasons,
including:
- hard to reproduce bugs that trigger with very low probability
- bug being introduced before the tool that reliably detects it (LOCKDEP, KASAN,
FAULT_INJECTION, WARNING, etc);
such bugs may be bisection to the addition/improvement of the tool - kernel build/boot errors that force skipping revisions
- some kernel configs are disabled as bisection goes back
in time because they build/boot break release tags;
bugs in these subsystems may be bisected to release tags - reproducers triggering multiple kernel bugs at once
- unrelated kernel bugs that break even simple programs
A single incorrect decision during bisection leads to an incorrect result,
so please treat the results with understanding. You may consult the providedbisection log to see how/why syzbot has arrived to a particular commit.
Suggestions and patches that improve bisection quality for common cases are
welcome.
syzbot supports cause bisection (find the commit that introduces a bug) and
fix bisection (find the commit that fixes a bug).
The web UI for a specific kernel
(say upstream linux) shows theBisected status for all bugs.
Fix bisection
syzbot will perform fix bisection on bugs that meet the following criterion:
- The kernel tree corresponding to the bug has fix bisection enabled (for
instance, fix bisection might be disabled on kernel trees that have commits
which are force pushed). - The bug must have a reproducer.
- The bug must not have occurred for at least 30 days. If the bug still occurs
on HEAD upon fix bisection, it will be retried in another 30 days.
If you receive an email with fix bisection results you think is correct, please
reply with a #syz fix: commit-title so that syzbot can close the bug report.
Running reproducers
Detailed instructions on running reproducers can be found here.
syzbot aims at providing stand-alone C reproducers for all reported bugs.
However, sometimes it can't extract a reproducer at all, or can only extract a
syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller
notation and they can be executed on the target system with a little bit more
effort.
A syskaller program can also give you an idea as to what syscalls with what
arguments were executed (note that some calls can actually be executed in
parallel).
A syzkaller program can be converted to an almost equivalent C source usingsyz-prog2c utility. syz-prog2c has lots of flags in common with
syz-execprog,
e.g. -threaded which controls if the syscalls are executed sequentially or
concurrently.
An example invocation:
syz-prog2c -prog repro.syz.txt -enable=all -threaded -repeat -procs=8 -sandbox=namespace -segv -tmpdir -waitrepeatHowever, note that if syzbot did not provide a C reproducer, it wasn't able to
trigger the bug using the C program (it might also be the case that the bug is
triggered by a rare race condition).
Downloadable assets
Syzbot shares links to the bootable disk image, kernel object file and other
relevant files that can be used to reproduce and debug the issue locally.
See this tutorial on how to use them.
Crash does not reproduce?
If the provided reproducer does not work for you, most likely it is related to the
fact that you have slightly different setup than syzbot. syzbot has obtained
the provided crash report on the provided reproducer on a freshly-booted
machine, so the reproducer worked for it somehow.
Note: if the report contains userspace arch: i386,
then the program needs to be built with -m32 flag.
syzbot uses GCE VMs for testing, but usually it is not important.
If the reproducer exits quickly, try to run it several times, or in a loop.
There can be some races involved.
Sometimes it might be important to build the kernel using the exact same
compiler that was used by syzbot. Normally that information is included in every
email report, e.g.:
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40The latest compilers used by syzbot are contained in thegcr.io/syzkaller/syzbot docker image. For in-tree kernel build in current
directory it can be used as follows:
docker pull gcr.io/syzkaller/syzbot
docker run -it --user $(id -u ${USER}):$(id -g ${USER}) \
--volume "$PWD:/syzkaller/pwd" --workdir /syzkaller/pwd \
gcr.io/syzkaller/syzbot:latest
makeFor an out-of-tree build one needs to proxy more host directories using --volume flag.
Older compilers used by syzbot can be found here:
- gcc 7.1.1 20170620 (245MB)
- gcc 8.0.1 20180301 (286MB)
- gcc 8.0.1 20180412 (33MB)
- gcc 9.0.0 20181231 (30MB)
- gcc 10.1.0-syz (20200507) (220MB)
- gcc 10.2.0 (165MB)
- clang 7.0.0 (trunk 329060) (44MB)
- clang 7.0.0 (trunk 334104) (44MB)
- clang 8.0.0 (trunk 343298) (45MB)
- clang 10.0.0 (c2443155)
- clang 11.0.0 (git ca2dcbd030e) (682MB)
- clang 11.0.1
A QEMU-suitable Debian Stretch image can be found
here (241 MB).
A reference qemu command line to run it is as follows:
qemu-system-x86_64 -smp 2 -m 4G -enable-kvm -cpu host \
-net nic -net user,hostfwd=tcp::10022-:22 \
-kernel arch/x86/boot/bzImage -nographic \
-device virtio-scsi-pci,id=scsi \
-device scsi-hd,bus=scsi.0,drive=d0 \
-drive file=stretch-amd64.img,format=raw,if=none,id=d0 \
-append "root=/dev/sda1 console=ttyS0 earlyprintk=serial"And then you can ssh into it using:
ssh -p 10022 root@localhostNote: before March 25th 2020 Debian Wheezy image was used for testing, so some of the bugs reported before that date
might only be reproducible on Wheezy. That image is here
and the key for it is here.
No reproducer at all?
Reproducers are best-effort. syzbot always tries to create reproducers, and
once it has one it adds it to the bug. If there is no reproducer referenced in a
bug, a reproducer does not exist. There are multiple reasons why syzbot can
fail to create a reproducer: some crashes are caused by subtle races and are
very hard to reproduce in general; some crashes are caused by global accumulated
state in kernel (e.g. lockdep reports); some crashes are caused by
non-reproducible coincidences (e.g. an integer 0x12345 happened to reference an
existing IPC object) and there is long tail of other reasons.
Moderation queue
Bugs with reproducers are automatically reported to kernel mailing lists.
Bugs without reproducers are first staged in moderation queue to filter out
invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard
in moderation section
and mailed to
syzkaller-upstream-moderation
mailing list. Staged bugs accept all commands supported for reported bugs
(fix, dup, invalid) with a restriction that bugs reported upstream
can't be dup-ed onto bugs in moderation queue. Additionally, staged bugs
accept upstream command:
#syz upstreamwhich sends the bug to kernel mailing lists.
KMSAN bugs
KMSAN is a dynamic, compiler-based tool (similar to KASAN) that detects
uses of uninitialized values. As compared to (now deleted) KMEMCHECK which
simply detected loads of non-stored-to memory, KMSAN tracks precise
propagation of uninitialized values through memory and registers and only flags
actual eventual uses of uninitialized values. For example, KMSAN will detect
a branch on or a copy_to_user() of values that transitively come from
uninitialized memory created by heap/stack allocations. This ensures
/theoretical/ absence of both false positives and false negatives (with some
implementation limitations of course). Note that KMSAN requires clang compiler.
KMSAN is not upstream yet, though, we want to upstream it later. For now,
it lives in github.com/google/kmsan and is
based on a reasonably fresh upstream tree. As the result, any patch testing
requests for KMSAN bugs need to go to KMSAN tree
(https://github.com/google/kmsan.git repo, master branch).
A standard way for triggering the test with KMSAN tree is to send an
email to syzbot+HASH address containing the following line:
#syz test: https://github.com/google/kmsan.git masterand attach/inline your test patch in the same email.
Report explanation. The first call trace points to the use of the uninit value
(which is usually a branching or copying it to userspace). Then there are 0 or
more "Uninit was stored to memory at:" stacks which denote how the unint value
traveled through memory. Finally there is a "Uninit was created at:"
section which points either to a heap allocation or a stack variable which
is the original source of uninitialized-ness.
USB bugs
syzkaller has an ability to perform fuzzing of the Linux kernel USB stack, see
the details here. As of now all kernel
changes required for USB fuzzing have been merged into the mainline (the last one
during the 5.8-rc1 merge window), so the USB fuzzing instance has been switched
to target the usb-testing tree.
Testing kernel patches on the USB instance follows the same principle as on the
mainline instances, with a few caveats:
You may specify any kernel tree for
syz testas long as it includes all
mainline patches up to 5.8-rc1.Some of the bugs have reproducers generated on kernel versions with custom
kernel (when fuzzing was performed with non-yet-mainlined kernel patches), thus
those reproducers might no longer work. The recommended workflow is to: first,
execute asyz testcommand on a target tree to make sure that the bug
reproduces, and then execute asyz testcommand with a fix/debug patch.
If the bug was triggered on the KMSAN tree, follow the instructions above,
with the exception that you must also use commit-hash instead of the master
branch when testing patches.
Memory leaks
syzbot uses KMEMLEAK to find memory leaks in the Linux kernel.KMEMLEAK kernel config is stored here.
See KMEMLEAK docs
for general info, algorithm overview and usage instructions.
Memory leaks may be tricky to debug because we have only the allocation stack,
but we don't see where kernel code forgot to free the object or drop a reference.KMEMLEAK can have some false positives on tricky kernel code that hides
pointers to live objects and due to memory scanning non-atomicity.
But don't write off everything to false positives right away, the rate of
false positives is observed to be very low. In particular, KMEMLEAK is
not confused
by pointers stored in a middle of another object; and it's
not confused
if several pointer low bits are used as flags because a pointer into
a middle of an object also marks the target as reachable.
A useful litmus test is to remove KMEMLEAK code from the reproducer
and run it for longer and/or multiple times. If memory consumption and number
of live objects in /proc/slabinfo steadily grow, most likely the leak is real.
KCSAN bugs
The Kernel Concurrency Sanitizer (KCSAN)
is a dynamic data-race detector. Reproduction of data-races is currently
unsupported, and syzbot is unable to test patches.
No custom patches
While syzbot can test patches that fix bugs, it does not support applying
custom patches during fuzzing. It always tests vanilla unmodified git trees.
There are several reasons for this:
- custom patches may not apply tomorrow
- custom patches may not apply to all of the tested git trees
- it's hard to communicate exact state of the code with bug reports (not just hash anymore)
- line numbers won't match in reports (which always brings suspicion as to the quality of reports)
- custom patches can also introduce bugs, and even if they don't a developer may (rightfully)
question validity of and may not want to spend time on reports obtained
with a number of out-of-tree patches - order of patch application generally matters, and at some point patches
need to be removed, there is nobody to manage this
We've experimented with application of custom patches in the past and it lead
to unrecoverable mess. If you want syzbot to pick up patches sooner,
ask tree maintainers for priority handling.
However, syzbot kernel config always includes CONFIG_DEBUG_AID_FOR_SYZBOT=y setting,
which is not normally present in kernel. What was used for particularly elusive bugs in the past
is temporary merging some additional debugging code into linux-next under this config setting
(e.g. more debug checks and/or debug output) and waiting for new crash reports from syzbot.
One can also always run syzkaller locally on any kernel for better stress testing
of a particular subsystem and/or patch.
Kernel configs
Kernel configs, sysctls and command line arguments that syzbot uses are available in /dashboard/config.
Is syzbot code available?
Yes, it is here.
Syzbot Ai Patches (`docs/syzbot_ai_patches.md`)
AI-generated patches
syzbot investigates kernel bugs and tries to generate fix patches using AI.
To ensure patch quality and adhere to kernel community standards, the AI
patching process follows a two-stage workflow:
- Moderation: Newly generated patches are first posted to an internal moderation
list, where human reviewers can collaborate with the AI, provide feedback, and
trigger automated revisions. - Upstream submission: Once reviewed and signed off by a human developer
(in accordance with the kernel's AI Coding Assistants
guidelines), the patch is sent to the Linux Kernel Mailing List (LKML) and
relevant subsystem lists.
Once sent upstream, patches follow the standard kernel review process: the developer
who signed off on the patch acts as the submitter and handles reviewer feedback,
discussion, and any follow-up revisions.
The Intended Workflow
syzbot's AI patch reporting follows a two-stage pipeline to refine the patch
before publishing it to the wider community.
1. Moderation Stage
Newly generated patches are first sent to an internal moderation mailing list
([email protected]) where the bot actively
listens and participates. You can:
Reply to the patch email with text comments to provide feedback and request
changes. The AI model will read the comments and reply or send a new patch
version (e.g. an RFC v2) if necessary.Reject a fundamentally flawed patch:
#syz rejectYou can provide the reason for rejection in the email body for better accounting.
Once a patch is rejected, the AI will stop reacting to further comments on this patch version.
- If you accidentally rejected a patch, you can undo it:
#syz unreject- Approve a patch and send it to the public mailing lists:
#syz upstream2. Linux Kernel Mailing List (LKML)
Once a patch is approved and upstreamed from the moderation list, it is sent to
the public mailing lists (LKML / subsystem lists).
Important: syzbot does not automatically reply to review comments or send
patch iterations on LKML. The developer who signed off on the patch is
responsible for addressing review feedback, answering questions, and submitting
any necessary follow-up versions.
While syzbot administrators actively monitor upstream patches and can manually
trigger AI iterations internally if appropriate, all interactions and patch
submissions on LKML remain human-driven.
System Invariants and Rules
syzbot maintains strict rules about tracking and handling AI patches based on
email replies.
Which patch version am I interacting with?
Each patch version (e.g., v1, v2) is sent as a separate email thread. syzbot
uses the In-Reply-To email header to identify exactly which version of the
patch you are interacting with.
You can reply to any patch version's thread, not just the latest one. Commands
like #syz upstream or #syz reject will only apply to the specific version
you are replying to.
Can I upstream multiple versions of the same patch?
No. syzbot prevents upstreaming multiple patch iterations for the same bug
simultaneously to avoid spamming the upstream list.
If you upstream "v1" and later decide "v2" is better, an attempt to#syz upstream "v2" will be blocked. You must first reply to the previously
upstreamed version ("v1") with:
#syz rejectOnce rejected, the system clears the conflict, and you can successfully#syz upstream the new version.
What happens to my tags (Reviewed-by, Acked-by)?
During the review process, reviewers often provide standard tags likeReviewed-by:, Acked-by:, Tested-by:, Reported-by:, or Suggested-by:.
syzbot automatically parses and accumulates these tags from review comments.
When a newer iteration of the patch is generated, these tags are reliably
preserved and appended to the commit message's trailer.
Who gets the Signed-off-by tag?
When you approve a patch by replying with #syz upstream, syzbot incorporates
your email and name into a standard Signed-off-by: tag and appends it to the
commit message when sending the patch to the next stage (e.g., LKML).
If the patch goes through further iterations on public mailing lists, thisSigned-off-by: tag is preserved.
What happens if I reply without a command?
- On the moderation mailing list: Your reply is recorded as review feedback
on the AI patch job.syzbottriggers an iteration job where the AI evaluates
your comments, addresses questions or change requests, and sends a new version
of the patch if appropriate (accumulating any provided tags likeReviewed-by).
A plain reply does not advance (#syz upstream) or reject (#syz reject) the patch. - On LKML / public lists:
syzbotdoes not automatically reply or iterate;
feedback is handled directly by the developer who signed off on the patch.
What are the requirements for upstreaming?
The AI job must have successfully produced a patch. You cannot use#syz upstream on an AI run that only replied with a textual comment and no
code changes.
If you send #syz upstream and do not receive a reply, the command was
processed successfully. The system does not reply back to confirm; it simply
pushes the patch to the next reporting stage.
Syzbot Assets (`docs/syzbot_assets.md`)
Reproduce a bug with syzbot's downloadable assets
As a part of every bug report, syzbot shares downloadable assets -- that is,
disk images and kernel binaries on which the bug was originally found.
This document serves as a guide on how to use those assets to reproduce such bugs
locally.
A sample report
To be more specific, let's take this syzbot report: [syzbot] [hfs?] kernel BUG
in hfsplus_bnode_put.
syzbot has found a reproducer for the following issue on:
HEAD commit: 40f71e7cd3c6 Merge tag 'net-6.4-rc7' of git://git.kernel.o..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=10482ae3280000
kernel config: https://syzkaller.appspot.com/x/.config?x=7ff8f87c7ab0e04e
dashboard link: https://syzkaller.appspot.com/bug?extid=005d2a9ecd9fbf525f6a
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gzThere are 4 linked assets:
- The bootable VM disk image on which the bug was found:
https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz- The image is suitable both for GCE and for qemu.
- The
vmlinuxfile that can be used e.g. for report symbolization or forgdb-based debugging:https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xz - The separate
bzImagefile (it is already included in the disk image):https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xz - The filesystem image that is mounted in the reproducer:
https://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gz
All these links are also reachable from the web dashboard.
Run a C reproducer
Boot a VM:
$ wget 'https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz'
$ unxz disk-40f71e7c.raw.xz
$ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=./disk-40f71e7c.raw,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1Build and run the C reproducer:
$ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000'
$ gcc repro.c -lpthread -static -o repro
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro [email protected]:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes [email protected] 'chmod +x ./repro && ./repro'Wait a minute and notice a crash report in the qemu's serial output:
[ 91.956238][ T81] ------------[ cut here ]------------
[ 91.957508][ T81] kernel BUG at fs/hfsplus/bnode.c:618!
[ 91.958645][ T81] invalid opcode: 0000 [#1] PREEMPT SMP KASAN
[ 91.959861][ T81] CPU: 0 PID: 81 Comm: kworker/u5:3 Not tainted 6.4.0-rc6-syzkaller-00195-g40f71e7cd3c6 #0Run a syz reproducer directly
For some bugs, there's either no C reproducer or it's not reliable enough. In
that case, syz reproducers might be useful.
You'll need to check out and build
syzkaller first. The fastest way to do it is as follows (assuming Docker is
installed and configured on your machine):
$ git clone https://github.com/google/syzkaller.git
$ cd syzkaller
$ ./tools/syz-env makeThen boot a VM exactly like in the previous section.
Download and run the syz reproducer:
$ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000'
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./bin/linux_amd64/* ./repro.syz [email protected]:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes [email protected] './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz'In some time, you'll see the same bug report in the VM's serial output.
The commands above execute the ./syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz
command inside the VM. More details can be found in this document.
Use the `tools/syz-crush` tool
The syz-crush automatizes the steps above: it sets up and boots a pool of VMs
and runs the given C or syz reproducer in them.
First, download the disk image and reproducers (see instructions above).
Then, go to the syzkaller checkout and build the syz-crush tool:
$ make crushPrepare a config file (let it be config.json):
{
"name": "test",
"http": "0.0.0.0:0",
"target": "linux/amd64",
"image": "/tmp/disk-40f71e7c.raw",
"syzkaller": "/tmp/syzkaller",
"workdir": "/tmp/syzkaller/workdir",
"type": "qemu",
"procs": 6,
"vm": {
"count": 5,
"cmdline": "root=/dev/sda1",
"cpu": 2,
"mem": 2048,
"qemu_args": "-machine pc-q35-7.1 -enable-kvm"
}
}You need to replace /tmp/syzkaller with the location of your syzkaller
checkout and /tmp/disk-40f71e7c.raw with the location of the bootable disk
image.
Run the tool:
$ mkdir workdir
$ ./bin/syz-crush -config config.json repro.syzReproducing ARM64 bugs
If the bug was found on an ARM64 instance (e.g. the manager name isci-upstream-gce-arm64), you may use the following qemu command as a reference:
$ qemu-system-aarch64 -machine virt -cpu cortex-a57 -smp 4 -m 4G -nographic -drive file=disk.raw,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -kernel Image -net user,hostfwd=tcp::10023-:22 -net nic -append "root=/dev/vda2" -accel tcg,thread=multiProblems
The bug doesn't reproduce
If the C reproducer did not work, try to run the syz reproducer.
If there's still no success, it might be that relatively rare case when the
execution environment becomes important. Syzbot fuzzes kernels on GCE VMs, which
might have a different instruction set / execution speed than locally run qemu
VMs. These changes might be critical for the generated reproducer.
There's unfortunately no universal solution.
Note that you can always ask syzbot to
apply your git patch and re-run the reproducer.
It will be run in the same GCE environment where the bug was originally found.
See also this document.
Assets are not downloadable
The downloadable assets are not stored infinitely. Syzbot keeps them until the
bug is fixed or marked as invalid + 30 days after that.
So if you cannot download the assets using the links from the email, this might
be a sign that the bug is actually no longer worth looking at.
Qemu doesn't boot
A recent qemu problem
may prevent it from booting large kernel images. Add -machine pc-q35-7.1 to
the qemu args to make it work.
Syzos (`docs/syzos.md`)
SYZOS Technical Documentation
1. System Overview
Concept
SYZOS is not a traditional operating system but an immutable C library designed to run as a Guest (L1) within a KVM virtual machine. Its primary purpose is to expose an easy-to-fuzz API to the Host (L0) fuzzer (syzkaller), allowing for state-aware interactions that are difficult to achieve with raw instruction fuzzing.
In this architecture, the Host (syz-executor) acts as the orchestrator, while the Guest (SYZOS) acts as the execution engine for a pre-defined sequence of commands.
Execution Flow
SYZOS leverages syzkaller's standard execution model, where the fuzzer generates a sequence of syscalls (a syzlang program) to be executed by the host executor. For SYZOS, this program constructs the VM and defines the guest's internal logic via pseudo-syscalls.
- VM Creation: The fuzzer calls standard KVM ioctls (e.g.,
openat,KVM_CREATE_VM) to create the VM container. - Environment Setup (
syz_kvm_setup_syzos_vm): This pseudo-syscall automates the complex setup of Guest memory, ensuring the VM has valid code and stack regions. - VCPU & Program Loading (
syz_kvm_add_vcpu, see 3.2):- Instead of a bare
KVM_CREATE_VCPU, the fuzzer callssyz_kvm_add_vcputhat creates a new VCPU in the VM and initializes its state.- This call takes the entire sequence of SYZOS commands as an argument. This sequence effectively becomes the "program" the guest will execute.
- Concurrency: SYZOS supports up to 4 separate VCPUs sharing the same address space, allowing the fuzzer to schedule concurrent guest operations.
- Instead of a bare
- Execution (
KVM_RUN): The fuzzer triggers execution via standardKVM_RUNcalls. The Guest executes its pre-loaded commands step-by-step.- Yielding: When the Guest needs to perform an action that requires Host intervention (e.g., a transition during Nested Virtualization), it yields to L0 via
UEXIT. - Resumption: If the program contains multiple
KVM_RUNcalls, they are used to resume the Guest until the pre-loaded program completes.
- Yielding: When the Guest needs to perform an action that requires Host intervention (e.g., a transition during Nested Virtualization), it yields to L0 via
Design Philosophy
- Logical Mutation: Instead of fuzzing raw assembly bytes, SYZOS exposes high-level primitives to the fuzzer. The fuzzer mutates the arguments of the SYZOS commands.
- State Validity: By implementing setup sequences in C, SYZOS ensures that complex structures like IRQ tables or Page Tables are valid enough to reach deep kernel code paths.
2. Memory Layout & ABI (Communication Interface)
The Host and Guest communicate via a shared memory protocol. The Host writes commands and arguments into specific physical memory addresses, which the Guest maps and reads.
Communication Interface
- Command Channel: A dedicated memory region where the Host writes the commands and their arguments.
- Result Channel: Mechanism for the Guest to report status back to the Host, piggybacked on the
UEXITmechanism. - Scratch Space: Mutable memory used by the Guest to generate dynamic code blobs or store temporary data needed for operations like
MSRwrites.
ARM64 Memory Map
The ARM64 implementation relies on a static physical memory layout to ensure the Host knows exactly where to place data.
| Physical Address | Description | Usage |
|---|---|---|
0x08000000 |
GIC v3 Distributor | Interaction with Generic Interrupt Controller |
0x080a0000 |
GIC v3 Redistributor | Per-CPU Interrupt Controller interface |
0xdddd0000 - 0xeeee0000 |
Read-only / Command Page | Host writes SYZOS commands here. Also used to trigger page faults for UEXIT |
0xeeee8000 - 0xeeef0000 |
Code / Scratch Space | Where SYZOS resides. Also used for generated code (e.g., MSR trampolines) |
0xffff1000 |
EL1 Stack | Stack space for the SYZOS Guest execution |
x86 Memory Map
While the exact addresses may vary by implementation, the x86 layout follows similar principles:
- Guest Code: Allocated via
KVM_SET_USER_MEMORY_REGION(typically 1 page). - Page Tables: Setup by the Host to allow virtual-to-physical translation required for long mode.
- IDT (Interrupt Descriptor Table): Setup by the Host to handle exceptions within the Guest.
3. Host-Side Implementation (`syz-executor`)
The Host side is responsible for the heavy lifting of VM initialization. This is achieved through "pseudo-syscalls" - functions implemented in syz-executor that look like syscalls to the fuzzer but perform complex setup logic.
3.1 VM Initialization: `syz_kvm_setup_syzos_vm()`
This pseudo-syscall creates the VM and prepares the environment.
- Memory Allocation: Calls
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &memreg)multiple times to map the Guest physical memory slots (Code, Stack, MMIO). - Image Loading: Copies the compiled SYZOS C library binary into the allocated Guest Code region.
3.2 VCPU Initialization: `syz_kvm_add_vcpu()`
This function adds a virtual CPU to the VM and configures its initial state to jump into the SYZOS entry point.
- Program Loading: It parses the
syzlang-generated argument structure, which contains the sequence of SYZOS commands, and copies them into the Guest's Command Page. - Context Setup:
- x86: Sets guest CPUID to
KVM_GET_SUPPORTED_CPUID, then sets up SREGS (64-bit mode, Segments, Page Tables) and IDT, and points RIP atguest_main. The order matters: KVM validates the EFER/CR4 bits passed toKVM_SET_SREGSagainst guest CPUID (e.g.EFER.LMErequiresLM,EFER.SVMErequiresSVM), so CPUID must be set first. If any step fails,syz_kvm_add_vcpufails instead of returning a half-initialized VCPU. - ARM64: Sets PC to the entry point and SP to the stack.
- x86: Sets guest CPUID to
3.3 The Handshake Mechanism (`UEXIT`)
The core synchronization primitive is the UEXIT.
- Trigger: The Guest reads from a specific unmapped or read-only address (e.g., inside
0xdddd0000on ARM) or executes a specific instruction sequence. - Detection:
KVM_RUNreturns on the Host. The Host may checkkvm_run->exit_reason. - Handling:
- If the exit indicates a Page Fault (EPT violation) at the specific
UEXITaddress, the Host treats this as a voluntary yield. - The Host reads the exit qualification or register state to retrieve the "return argument" passed by the Guest.
- If the exit indicates a Page Fault (EPT violation) at the specific
4. Guest-Side Implementation (SYZOS Library)
Source Organization & `GUEST_CODE`
SYZOS guest handlers are defined directly in architecture-specific executor headers (e.g., executor/common_kvm_amd64_syzos.h).
- The
GUEST_CODEMacro: Functions intended to run inside the guest are marked withGUEST_CODE(e.g.,GUEST_CODE static void guest_handle_...). This instructs the compiler/linker to place these functions in a specific section thatsyz-executorcopies into the Guest's physical memory. - Header-Based Implementation: The entire SYZOS codebase is contained within header files included by the executor. This architecture is necessitated by
syz-prog2c, a tool that convertssyzlangreproducers into standalone C programs. By concatenating these headers (via#includeexpansion),syz-prog2ccan produce a single, build-system-independent C source file that compiles anywhere without external dependencies.
The Dispatch Loop (`guest_main`)
The entry point guest_main iterates through the command buffer that was populated by syz_kvm_add_vcpu.
- Command Routing (If/Else Chain): The routing is strictly implemented as a series of
if/else ifstatements rather than aswitch. The reason for this is that aswitchstatement can be optimized by compilers into a jump table stored in the executable's.rodatasection. Since the global data sections are not mapped into the Guest address space, accessing a jump table would cause an immediate Page Fault. - Argument Parsing: Commands are cast to specific structures (e.g.,
struct api_call_5*) to access arguments safely. - Execution: The handler performs the logic and the loop advances to the next command in the buffer.
Core Primitives
SYZOS_API_UEXIT: - Triggers a specific exception that the Host recognizes as a "yield". It passes a return value (1 argument) back to the Host to signal success/failure or data.SYZOS_API_CODE: - Executes a raw blob of machine code supplied by the Host. This can be used to emit exact instruction sequences not covered by high-level APIs.
5. Platform Specifics
x86 (Intel & AMD)
Privileged Operations
SYZOS exposes specific APIs to fuzz privileged x86 instructions:
SYZOS_API_CPUID: Executes theCPUIDinstruction.SYZOS_API_WRMSR/SYZOS_API_RDMSR: Reads/Writes Model Specific Registers.SYZOS_API_WR_CRN/SYZOS_API_WR_DRN: Writes to Control Registers and Debug Registers.SYZOS_API_IN_DX/SYZOS_API_OUT_DX: Executes I/O port operations.
Nested Virtualization (NV) Engine
SYZOS acts as a lightweight L1 hypervisor to fuzz L2 guests, abstracting the architectural differences between Intel VMX and AMD SVM. It provides a uniform API for the VM lifecycle while offering architecture-specific commands for state mutation.
VM Lifecycle & Execution
The following primitives control the nested guest's existence and execution flow:
SYZOS_API_ENABLE_NESTED: Enables the virtualization extensions (VMXON on Intel, EFER.SVME on AMD).SYZOS_API_NESTED_CREATE_VM: Initializes the necessary control structures (VMCS for Intel, VMCB for AMD) and sets up Nested Page Tables.SYZOS_API_NESTED_LOAD_CODE: Injects a sequence of instructions into the L2 guest's memory, defining what code the nested machine will execute.SYZOS_API_NESTED_VMLAUNCH: Performs the initial VM Entry, transferring control to the L2 guest.SYZOS_API_NESTED_VMRESUME: Resumes execution of the L2 guest after it has exited back to L1.
State Mutation (Architecture Specific)
To stress the host's handling of invalid or edge-case states, SYZOS allows direct mutation of the hardware control structures. This is done by applying the "set/unset/flip" mask logic: new_val = (old_val & ~unset_mask) | set_mask ^ flip_mask.
The SYZOS commands are SYZOS_API_NESTED_INTEL_VMWRITE_MASK (mutates the VMCS fields on Intel) and SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK (VMCB on AMD).
ARM64
Device Emulation
A significant portion of ARM64 KVM code is device emulation. SYZOS provides specialized APIs to fuzz these complex interactions.
- GICv3 & ITS:
SYZOS_API_IRQ_SETUP: Sets up the VGICv3 distributor and installs the guest IRQ table.SYZOS_API_ITS_SETUP: Allocates translation tables and configures the Interrupt Translation Service (ITS) base.SYZOS_API_ITS_SEND_CMD: Injects structured GIC commands (e.g.,MAPD,MOVI) into the command queue.
Hypervisor Interface
SYZOS targets the boundary between the Guest and EL2/Firmware.
- Hypercalls:
SYZOS_API_HVC: Executeshvc #0with fuzzer-controlled parameters in registersx0-x5.SYZOS_API_SMC: Executessmc #0(Secure Monitor Call) with parameters inx0-x5.
6. Developer Guide: How to Add a New Command
This guide details the process of adding a new SYZOS command, using SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK as a reference case.
Step 1: Define API ID and Handler Prototype
Modify the architecture-specific executor header (e.g., executor/common_kvm_amd64_syzos.h) to register the new command.
- Add the Enum ID: Add a new entry to the
syzos_api_idenum.ctypedef enum { // ... SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK = 380, // New ID SYZOS_API_STOP, } syzos_api_id; - Declare the Handler: Add a forward declaration using the
GUEST_CODEmacro.cGUEST_CODE static void guest_handle_nested_amd_vmcb_write_mask(struct api_call_5* cmd, uint64 cpu_id);
Note: make sure to choose the optimal api_call_N structure that exactly matches the number of arguments required by your new primitive (e.g., use struct api_call_2 for a command needing two arguments). If no arguments are required, omit the cmd parameter altogether. If the guest code does not access VMCB/VMCS, omit the cpu_id parameter.
Step 2: Implement Guest Logic and Dispatch
In the same file (or corresponding source), implement the guest logic.
- Add Dispatch Case: Update
guest_main.c} else if (call == SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK) { guest_handle_nested_amd_vmcb_write_mask((struct api_call_5*)cmd, cpu); } - Implement Handler: Write the function logic. Strict guest-safe code restrictions apply.c
GUEST_CODE static noinline void guest_handle_nested_amd_vmcb_write_mask(struct api_call_5* cmd, uint64 cpu_id) { if (get_cpu_vendor() != CPU_VENDOR_AMD) return; // ... parse args and perform logic ... vmcb_write64(vmcb_addr, offset, new_value); }
Step 3: Define syzlang Description
Expose the new command to syzkaller in the description file (e.g., sys/linux/dev_kvm_amd64.txt).
Define Structures: Define any necessary constants or structures.
textsyzos_api_nested_amd_vmcb_write_mask { vm_id syzos_api_vm_id offset vmcb_offset set_mask int64 unset_mask int64 flip_mask int64 }Map Command ID: Add the command to the
syzos_api_callunion. Crucial: The ID (e.g.,380) must match the enum in the C header.textsyzos_api_call$x86 [ nested_amd_vmcb_write_mask syzos_api$x86[380, syzos_api_nested_amd_vmcb_write_mask] ] [varlen]
7. Validation & Regression Testing
The system includes a regression testing framework located in sys/linux/test/. New commands must include a test case to verify they trigger the expected Hypervisor behavior.
Test File Structure
Tests are syzlang programs with special assertions.
- Header: Requires metadata, e.g.,
# requires: arch=amd64 -threaded. - Setup: Standard boilerplate creates a VM and enters SYZOS.
- Logic: The test configures the guest to perform a specific action (e.g., executing
HLTin a nested L2 guest).
Assertions
Tests use specialized pseudo-syscalls to assert the VM's exit state:
syz_kvm_assert_syzos_uexit$x86(fd, code): Asserts that the guest voluntarily yielded with a specificUEXITcode (e.g.,0xe2e20001).syz_kvm_assert_syzos_kvm_exit$x86(fd, exit_reason): Asserts that the guest triggered a standard KVM exit (e.g.,0x5forKVM_EXIT_HLT) that was trapped by L0.
Talks (`docs/talks.md`)
Tech talks
newer first
- Head First Reporting of Linux Kernel CVEs: Practical Use of the Kernel Fuzzer at SOSS Community Day (Japan, 2024)
- Syzbot: 7 years of continuous kernel fuzzing at Linux Plumbers 2023
- Designing UAPI for Fuzz-ability at Linux Plumbers 2022
- Making syzbot reports more developer-friendly at Linux Plumbers 2022
- syzkaller: adventures in continuous coverage-guided kernel fuzzing
at BlueHatIL 2020
[presentation]
[slides] - Bugs on the Windshield: Fuzzing the Windows Kernel
at BlueHatIL 2020
[slides] - Fuzzing the OpenBSD Kernel
- syzkaller in FreeBSD
- Фаззинг ядра Linux на практике at ISPRASOPEN-2019 (in Russian)
- syzbot: update and open problems
at Linux Plumbers 2019 - GWP-ASAN
at Linux Plumbers 2019 - syzkaller: update and open problems
at Linux Security Summit NA 2019 - WSL Reloaded
- Drill Apple Core: Up and Down
at Blackhat Europe 2018 [slides] - Evolution of kernel fuzzers in NetBSD
- syzbot: automated kernel testing
at Linux Plumbers 2018 - syzbot and the tale of thousand kernel bugs
at Linux Security Summit NA 2018 [slides] - syzkaller+KASAN
at SIL2LinuxMP Workshop 2018 - ubsan, kasan, syzkaller und co
- syzkaller: the next gen kernel fuzzer
at Qualcomm Mobile Security Summit 2017 - syzkaller: future development
at Linux Plumbers 2016 - Sanitizers: a new generation of bug finding tools
at Linux Plumbers 2016 - Kernel Sanitizers: a new generation of bug finding tools for Linux kernel
at GeeKon 2015 - AddressSanitizer/ThreadSanitizer for Linux Kernel
at Linux Collaboration Summit 2013
Articles
- Your NVMe Had Been Syz’ed: Fuzzing NVMe-oF/TCP Driver for Linux with Syzkaller
- Looking for Remote Code Execution bugs in the Linux kernel
- From Collabora:
- From HardenedLinux project:
- Kernel QA with syzkaller and qemu (tutorial on how to setup syzkaller with qemu)
- Syzkaller crash DEMO (tutorial on how to extend syzkaller with new syscalls)
- Kernel debug tool with syzkaller (debugging qemu VM created by syz-manager with gdb)
- Explanation of some syzkaller internals
- A example of fuzzing the ceph filesystem
- Exploiting the Linux kernel via packet sockets
- Coverage-guided kernel fuzzing with syzkaller (by David Drysdale)
- Debugging a kernel crash found by syzkaller (by Quentin Casasnovas)
Also see research page.
Troubleshooting (`docs/troubleshooting.md`)
Troubleshooting
Here are some things to check if there are problems running syzkaller.
Use the
-debugcommand line option to make syzkaller print all possible debug output,
from both thesyz-managertop-level program and thesyz-executorinstances.
With this option syzkaller will only run one VM instance.Use the
-vv Ncommand line option to increase the amount of logging output, from both
thesyz-managertop-level program and thesyz-executorinstances (which go to the
output files in thecrashessubdirectory of the working directory). Higher values of
N give more output.If logging indicates problems with the executor program (e.g.
executor failure),
try manually running a short sequence of system calls:- Copy
syz-executorandsyz-execproginto a running VM. - In the VM run
./syz-execprog -executor ./syz-executor -debug sampleprogwhere
sampleprog is a simple system call script (e.g. just containinggetpid()). - For example, if this reports that
clonehas failed, this probably indicates
that the test kernel does not include support for all of the required namespaces.
In this case, running thesyz-execprogtest with the-sandbox=setuidoption fixes the problem,
so the main configuration needs to be updated to setsandboxtosetuid.
- Copy
If syzkaller prints the
failed to copy binaryerror shortly after VM has booted:- If you're using Buildroot images and the error output contains the
subsystem request failed on channel 0line, this can be due to the OpenSSH 9.0 changes
that force the use of the SFTP protocol. Upgrade your Buildroot image to the latest version and
make sure SFTP is enabled there.
- If you're using Buildroot images and the error output contains the
Also see this for Linux kernel specific troubleshooting advice.
If none of the above helps, file a bug on the bug tracker
or ask us directly on the [email protected] mailing list.
Please include syzkaller commit id that you use and syz-manager output with -debug flag enabled if applicable.
Usage (`docs/usage.md`)
How to use syzkaller
Running
Start the syz-manager process as:
./bin/syz-manager -config my.cfgThe syz-manager process will wind up VMs and start fuzzing in them.
The -config command line option gives the location of the configuration file, which is described here.
Found crashes, statistics and other information is exposed on the HTTP address specified in the manager config.
Crashes
Once syzkaller detected a kernel crash in one of the VMs, it will automatically start the process of reproducing this crash (unless you specified "reproduce": false in the config).
By default it will use 4 VMs to reproduce the crash and then minimize the program that caused it.
This may stop the fuzzing, since all of the VMs might be busy reproducing detected crashes.
The process of reproducing one crash may take from a few minutes up to an hour depending on whether the crash is easily reproducible or non-reproducible at all.
Since this process is not perfect, there's a way to try to manually reproduce the crash, as described here.
If a reproducer is successfully found, it can be generated in one of the two forms: syzkaller program or C program.
Syzkaller always tries to generate a more user-friendly C reproducer, but sometimes fails for various reasons (for example slightly different timings).
In case syzkaller only generated a syzkaller program, there's a way to execute them to reproduce and debug the crash manually.
Hub
In case you're running multiple syz-manager instances, there's a way to connect them together and allow to exchange programs and reproducers, see the details here.
Reporting bugs
Check here for the instructions on how to report Linux kernel bugs.
Darwin Found Bugs (`docs/darwin/found_bugs.md`)
Found bugs
panicall has ported
([1]
(video),
[2])
syzkaller to Darwin/XNU and that has found more than
50 bugs includingCVE-2018-4447 and CVE-2018-4435 mentioned in
Apple security updates. However he
didn't upstream his work.
Since 2021 Syzkaller is able to fuzz macOS, however it's not integrated into
syzbot for licensing reasons.
Freebsd Found Bugs (`docs/freebsd/found_bugs.md`)
Found Bugs
Newer bugs come first
- Fix udp_output() lock inconsistency.
- Fix IPV6_CHECKSUM computation.
- When sending a routing message, don't allow the user to set the RTF_RNH_LOCKED flag in rtm_flags.
- Fix an SCTP related locking issue.
- Reinitialize multicast source filter structures after invalidation.
- Fix a small locking bug in tcp_log_id().
- Fix a double free of an SCTP association in an error path.
- Reject F_SETLK_REMOTE commands when sysid == 0.
- Initialize scheduler specific data for the FCFS scheduler.
- Improve locking when tearing down an SCTP association.
- Fix more signed unsigned issues in SCTP.
- Fix a signed/unsigned bug when receiving SCTP messages.
- Limit the size of messages sent on 1-to-many style SCTP sockets with the SCTP_SENDALL flag.
- Limit the number of bytes which can be queued for SCTP sockets.
- Fix a KASSERT() in tcp_output().
- Disallow preemptive creation of wired superpage mappings.
- vm_fault_copy_entry: accept invalid source pages.
- Fix a bug in the SCTP stream schedulers.
- Allocate an association id and register the stcb with holding the lock.
- Allow SCTP stream reconfiguration operations only in ESTABLISHED state.
- Fix handling of the SCTP_STATUS socket option in early states.
- Honor the memory limits provided when processing the SCTP_GET_LOCAL_ADDRESSES socket option.
- Check the index hasn't changed after writing the cmp entry.
- Improve input validation for raw IPv4 socket using the IP_HDRINCL option.
- Fix a locking issue in the IPPROTO_SCTP level SCTP_PEER_ADDR_THLDS socket.
- Fix a locking bug in the IPPROTO_SCTP level SCTP_EVENT socket option.
- Fix locking for IPPROTO_SCTP level SCTP_DEFAULT_PRINFO socket option.
- Fix an off-by-one error in the input validation of the SCTP_RESET_STREAMS socketoption.
- Limit the user-controllable amount of memory the kernel allocates via IPPROTO_SCTP level socket options.
- Fix getsockopt() for IP_OPTIONS/IP_RETOPTS.
- Avoid overflow in vtruncbuf().
- Limit option_len for the TCP_CCALGOOPT.
- Correct vm_fault_copy_entry() handling of backing file truncation after the file mapping was wired.
- In vm_fault_copy_entry(), we should not assert that entry is charged if the dst_object is not of swap type.
- Handle a guest executing a vm instruction by trapping and raising an undefined instruction exception.
- disallow clock_settime too far in the future to avoid panic.
- Fix parsing error when processing cmsg in SCTP send calls.
Linux Coverage (`docs/linux/coverage.md`)
Coverage
Syzkaller uses kcov to collect coverage from the kernel. kcov exports the address of each executed basic block, and syzkaller runtime uses tools from binutils (objdump, nm, addr2line and readelf) to map these addresses to lines and functions in the source code.
Binutils
Note that if you are fuzzing in cross-arch environment you need to provide correct binutils cross-tools to syzkaller before starting syz-manager:
mkdir -p ~/bin/mips64le
ln -s `which mips64el-linux-gnuabi64-addr2line` ~/bin/mips64le/addr2line
ln -s `which mips64el-linux-gnuabi64-nm` ~/bin/mips64le/nm
ln -s `which mips64el-linux-gnuabi64-objdump` ~/bin/mips64le/objdump
ln -s `which mips64el-linux-gnuabi64-readelf` ~/bin/mips64le/readelf
export PATH=~/bin/mips64le:$PATHThe target-triple prefix is determined based on the target config option.
readelf
readelf is used to detect virtual memory offset.
readelf -SW kernel_imageThe meaning of the flags is as follows:
- `-S' - list section headers in the kernel image file
- `-W' - output each section header entry in a single line
Example output of the command:
There are 59 section headers, starting at offset 0x3825258:
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000 e010f7 00 AX 0 0 4096
[ 2] .rela.text RELA 0000000000000000 23ff488 684720 18 I 56 1 8
[ 3] .rodata PROGBITS ffffffff82000000 1200000 2df790 00 WA 0 0 4096
[ 4] .rela.rodata RELA 0000000000000000 2a83ba8 0d8e28 18 I 56 3 8
[ 5] .pci_fixup PROGBITS ffffffff822df790 14df790 003180 00 A 0 0 16
[ 6] .rela.pci_fixup RELA 0000000000000000 2b5c9d0 004a40 18 I 56 5 8
[ 7] .tracedata PROGBITS ffffffff822e2910 14e2910 000078 00 A 0 0 1
[ 8] .rela.tracedata RELA 0000000000000000 2b61410 000120 18 I 56 7 8
[ 9] __ksymtab PROGBITS ffffffff822e2988 14e2988 011b68 00 A 0 0 4
[10] ...Executor truncates PC values into uint32 before sending them to syz-manager and syz-manager uses section header information to recover the offset. Only the section headers of type PROGBITS are considered. The Address field represents the virtual address of a section in memory (for the sections that are loaded). It is required that all PROGBITS sections have same upper 32 bits in the Address field. These 32 bits are used as recovery offset.
Reporting coverage data
MakeReportGenerator factory creates an object database for the report. It requires target data, as well as information on the location of the source files and build directory. The first step in building this database is
extracting the function data from the target binary.
nm
nm is used to parse address and size of each function in the kernel image
nm -Ptx kernel_imageThe meaning of the flags is as follows:
-P- use the portable output format (Standard Output)-tx- write the numeric values in the hex format
Output is of the following form:
tracepoint_module_nb d ffffffff84509580 0000000000000018
...
udp_lib_hash t ffffffff831a4660 0000000000000007The first column is a symbol name and the second column is its type (e.g. text section, data section, debugging symbol, undefined, zero-init section, etc.). The third column is the symbol value in hex format while the forth column contains its size. The size is always rounded to up to 16 in syzkaller. For the report, we are only interested in the code sections so the nm output is filtered for the symbols with type t or T.
The final result is a map with symbol names as keys, values being starting and ending address of a symbol. This information is used to map coverage data to symbols (functions). This step is needed to find out whether certain functions are called at all.
Object Dump and Symbolize
In order to provide the necessary information for tracking the coverage information with syzkaller, the compiler is instrumented to insert the __sanitizer_cov_trace_pc call into every basic block generated during the build process. These instructions are then used as anchor points to backtrack the covered code lines.
objdump
objdump is used to parse PC value of each call to __sanitizer_cov_trace_pc in the kernel image. These PC values are representing all code that is built into kernel image. PC values exported by kcov are compared against these to determine coverage.
The kernel image is disassembled using the following command:
objdump -d --no-show-raw-insn kernel_imageThe meaning of the flags is as follows:
-d- disassemble executable code blocks-no-show-raw-insn- prevent printing hex alongside symbolic disassembly
Excerpt output looks like this:
...
ffffffff81000f41: callq ffffffff81382a00 <__sanitizer_cov_trace_pc>
ffffffff81000f46: lea -0x80(%r13),%rdx
ffffffff81000f4a: lea -0x40(%r13),%rsi
ffffffff81000f4e: mov $0x1c,%edi
ffffffff81000f53: callq ffffffff813ed680 <perf_trace_buf_alloc>
ffffffff81000f58: test %rax,%rax
ffffffff81000f5b: je ffffffff8100110e <perf_trace_initcall_finish+0x2ae>
ffffffff81000f61: mov %rax,-0xd8(%rbp)
ffffffff81000f68: callq ffffffff81382a00 <__sanitizer_cov_trace_pc>
ffffffff81000f6d: mov -0x40(%r13),%rdx
ffffffff81000f71: mov 0x8(%rbp),%rsi
...From this output coverage trace calls are identified to determine the start of the executable block addresses:
ffffffff81000f41: callq ffffffff81382a00 <__sanitizer_cov_trace_pc>
ffffffff81000f68: callq ffffffff81382a00 <__sanitizer_cov_trace_pc>addr2line
addr2line is used for mapping PC values exported by kcov and parsed by objdump to source code files and lines.
addr2line -afi -e kernel_imageThe meaning of the flags is as follows:
-afi- means show addresses, function names and unwind inlined functions-e- is switch for specifying executable instead of using default
addr2line reads hexadecimal addresses from standard input and prints the filename
function and line number for each address on standard output. Example usage:
>> ffffffff8148ba08
<< 0xffffffff8148ba08
<< generic_file_read_iter
<< /home/user/linux/mm/filemap.c:2363where >> represents the query and << is the response from the addr2line.
The final goal is to have a hash table of frames where key is a program counter
and value is a frame array consisting of a following information:
PC- 64bit program counter value (same as key)Func- function name to which the frame belongsFile- file where function/frame code is locatedLine- Line in a file to which program counter mapsInline- boolean inlining information
Multiple frames can be linked to a single program counter value due to inlining.
Creating report
Once the database of the frames and function address ranges is created the next step is to determine the program coverage. Each program is represented here as a series of program counter values. As the function address ranges are known at this point it is easy to determine which functions were called by simply comparing the program counters against these address intervals. In addition, the coverage information is aggregated over the source files based on the program counters that are keys in the frame hash map. These are marked as coveredPCs. The resulting coverage is not line based but the basic block based. The end result is stored in the file struct containing the following information:
lines- lines covered in the filetotalPCs- total program counters identified for this filecoveredPCs- the program counters that were executed in the program runtotalInline- total number of program counters mapped to inlined framescoveredInline- the program counters mapped to inlined frames that were executed in the program run
Linux External Fuzzing Network (`docs/linux/external_fuzzing_network.md`)
External network fuzzing for Linux kernel
syzkaller has support for external fuzzing of the network stack.
This is achieved by using the TUN/TAP interface.
It allows to set up a virtual network interface and send packets to the kernel as they are being received from an external network.
This triggers the same paths as a real packet delivered through a real network interface (except for the driver layer).
You need to enable the CONFIG_TUN kernel config to enable external network fuzzing.
See initialize_tun() in executor/common_linux.h for the exact way the virtual interface is set up.
The template descriptions can be found in sys/linux/vnet.txt.
At this moment there are 2 fake syscalls: syz_emit_ethernet and syz_extract_tcp_res.
The first one externally sends a packet through the virtual interface.
The second one tries to externally receive a packet back and parse TCP sequence numbers from it for use in subsequent packets.
There are many protocols or protocol extensions that are not described yet, so the additions are welcome!
Since fuzzing may be done in multiple executor processes within the same VM instance, we need a way to isolate the virtual networks for different executors.
Right now this is done by creating one virtual interface per executor and assigning different MAC, IPv4 and IPv6 addresses to each of these interfaces.
Then the template descriptions make use of the proc type to generate proper addresses for each executor.
Since many network protocols require checksum fields to be embedded into packets, there's a support for describing such fields.
There's a csum type, which right now supports two different kinds of checksumming:
the Internet checksum: csum[parent, inet, int16be],
and TCP-like pseudo header checksum: csum[tcp_packet, pseudo, IPPROTO_TCP, int16be].
The checksums are computed and embedded right before emitting a packet through the virtual interface.
There's also a nice feature: when syzkaller generates a C reproducer, it generates code to compute checksums in runtime as well.
By using syz_emit_ethernet and syz_extract_tcp_res the following syzkaller program is able to establish a TCP connection over IPv4:
mmap(&(0x7f0000000000/0x10000)=nil, (0x10000), 0x3, 0x32, 0xffffffffffffffff, 0x0)
r0 = socket$inet_tcp(0x2, 0x1, 0x0)
bind$inet(r0, &(0x7f0000001000)={0x2, 0x0, @empty=0x0, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, 0x10)
listen(r0, 0x5)
syz_emit_ethernet(0x36, &(0x7f0000002000)={@local={[0xaa, 0xaa, 0xaa, 0xaa, 0xaa], 0x0}, @random="4c6112cc15d8", [], {{0x800, @ipv4={{0x5, 0x4, 0x0, 0x0, 0x28, 0x0, 0x0, 0x0, 0x6, 0x0, @remote={0xac, 0x14, 0x0, 0xbb}, @local={0xac, 0x14, 0x0, 0xaa}, {[]}}, @tcp={{0x1, 0x0, 0x42424242, 0x42424242, 0x0, 0x0, 0x5, 0x2, 0x0, 0x0, 0x0, {[]}}, {""}}}}}})
syz_extract_tcp_res(&(0x7f0000003000)={<r1=>0x42424242, <r2=>0x42424242}, 0x1, 0x0)
syz_emit_ethernet(0x38, &(0x7f0000004000)={@local={[0xaa, 0xaa, 0xaa, 0xaa, 0xaa], 0x0}, @remote={[0xbb, 0xbb, 0xbb, 0xbb, 0xbb], 0x0}, [], {{0x800, @ipv4={{0x5, 0x4, 0x0, 0x0, 0x2a, 0x0, 0x0, 0x0, 0x6, 0x0, @remote={0xac, 0x14, 0x0, 0xbb}, @local={0xac, 0x14, 0x0, 0xaa}, {[]}}, @tcp={{0x1, 0x0, r2, r1, 0x0, 0x0, 0x5, 0x10, 0x0, 0x0, 0x0, {[]}}, {"0c10"}}}}}})
r3 = accept$inet(r0, &(0x7f0000005000)={0x0, 0x0, @multicast1=0x0, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, &(0x7f0000006000)=0x10)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:20000 0.0.0.0:* LISTEN 5477/a.out
tcp 2 0 172.20.0.170:20000 172.20.0.187:20001 ESTABLISHED 5477/a.outSimilar program for IPv6:
mmap(&(0x7f0000000000/0x10000)=nil, (0x10000), 0x3, 0x32, 0xffffffffffffffff, 0x0)
r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
bind$inet6(r0, &(0x7f0000000000)={0xa, 0x1, 0x0, @empty={[0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, 0x0}, 0x1c)
listen(r0, 0x5)
syz_emit_ethernet(0x4a, &(0x7f0000001000)={@local={[0xaa, 0xaa, 0xaa, 0xaa, 0xaa], 0x0}, @random="de895db1468d", [], {{0x86dd, @ipv6={0x0, 0x6, "a228af", 0x14, 0x6, 0x0, @remote={0xfe, 0x80, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0], 0x0, 0xbb}, @local={0xfe, 0x80, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0], 0x0, 0xaa}, {[], @tcp={{0x0, 0x1, 0x42424242, 0x42424242, 0x0, 0x0, 0x5, 0x2, 0x0, 0x0, 0x0, {[]}}, {""}}}}}}})
syz_extract_tcp_res(&(0x7f0000002000)={<r1=>0x42424242, <r2=>0x42424242}, 0x1, 0x0)
syz_emit_ethernet(0x4a, &(0x7f0000003000)={@local={[0xaa, 0xaa, 0xaa, 0xaa, 0xaa], 0x0}, @random="de895db1468d", [], {{0x86dd, @ipv6={0x0, 0x6, "a228af", 0x14, 0x6, 0x0, @remote={0xfe, 0x80, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0], 0x0, 0xbb}, @local={0xfe, 0x80, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0], 0x0, 0xaa}, {[], @tcp={{0x0, 0x1, r2, r1, 0x0, 0x0, 0x5, 0x10, 0x0, 0x0, 0x0, {[]}}, {""}}}}}}})
r3 = accept$inet6(r0, &(0x7f0000004000)={0x0, 0x0, 0x0, @empty={[0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, 0x0}, &(0x7f0000005000)=0x1c)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::20001 :::* LISTEN 5527/a.out
tcp6 0 0 fe80::aa:20001 fe80::bb:20000 ESTABLISHED 5527/a.outLinux External Fuzzing Usb (`docs/linux/external_fuzzing_usb.md`)
External USB fuzzing for Linux kernel
syzkaller supports fuzzing the Linux kernel USB subsystem from the external USB device side.
Instead of relying on physical hardware (like Facedancer-based boards) or VM management software features (like QEMU usbredir), syzkaller fuzzes USB fully within a (potentially-virtualized) environment that runs the Linux kernel.
The USB fuzzing support in syzkaller is based on:
- Raw Gadget — a kernel module that implements a low-level interface for the Linux USB Gadget subsystem (now in the mainline kernel);
- Dummy HCD/UDC — a kernel module that sets up virtual USB Device and Host controllers that are connected to each other inside the kernel (existed in the mainline kernel for a long time);
- KCOV changes that allow collecting coverage from background kernel threads and interrupts (now in the mainline kernel);
- syzkaller changes built on top of the other parts; see the Internals section.
Besides this documentation page, for details, see:
Coverage-Guided USB Fuzzing with Syzkaller talk (video) from OffensiveCon 2019 (was given while the USB fuzzing support was a work-in-progress, so some details are outdated);
Fuzzing USB with Raw Gadget talk (video) from BSides Munich 2022 (more up-to-date, but less in-depth);
External fuzzing of Linux kernel USB drivers with syzkaller talk (outlines steps for adding new basic USB descriptions).
See this page for the list of bugs found in the Linux kernel USB stack so far.
Setting up
Use the kernel version 5.7 or later.
It's also recommended to backport all kernel patches that touch
drivers/usb/gadget/legacy/raw_gadget.c,drivers/usb/gadget/udc/dummy_hcd.c, andkernel/kcov.c;Enable
CONFIG_USB_RAW_GADGET=yandCONFIG_USB_DUMMY_HCD=yin the kernel config.Optionally, also enable the config options for the specific USB drivers that you wish to fuzz.
As an alternative, you can use the config from the syzbot USB fuzzing instance.
This config has many USB drivers commonly-used in distro kernels enabled;Build the kernel;
Optionally, update syzkaller descriptions by extracting the USB IDs.
This step is recommended if you wish to just rely on the existing syzlang descriptions to fuzz all USB drivers enabled in your kernel config.
If you plan to add new syzlang descriptions that target a specific USB driver, this step can be skipped;
Optionally, write syzlang descriptions for the USB driver you wish to fuzz;
Enable
syz_usb_connect,syz_usb_disconnect,syz_usb_control_io,syz_usb_ep_writeandsyz_usb_ep_readpseudo-syscalls (or any of their specific variants) in the manager config;Set
sandboxtononein the manager config;Pass
dummy_hcd.num=4(or whichever number you use forprocs) to the kernel command-line parameters in the manager config;Run syzkaller.
Make sure that you see both
USBEmulationandExtraCoverageenabled in themachine checksection in the log.You should also see a decent amount of coverage in
drivers/usb/core/*after the first few programs get added to the corpus.
Internals
syzkaller defines 5 main pseudo-syscalls for fuzzing USB drivers (see the syzlang descriptions and the C implementations):
syz_usb_connect— handles the enumeration process of a new USB device (in simple terms: connects a new USB device; in detail: handles all control requests up until aSET_CONFIGURATIONrequest is received);syz_usb_disconnect— disconnects a USB device;syz_usb_control_io— handles a post-enumeration control request (INorOUT);syz_usb_ep_write— handles a non-controlINrequest on an endpoint;syz_usb_ep_read— handles a non-controlOUTrequest on an endpoint.
Additionally, there is the syz_usb_connect_ath9k pseudo-syscall targeted to handle a few post-enumeration control requests the ath9k driver expects.
The syzlang descriptions for these pseudo-syscalls are targeted at a few different parts of the USB subsystem:
The common USB enumeration code is targeted by the generic
syz_usb_connectvariant.In addition, this generic variant also briefly touches the enumeration code in various USB drivers: the USB device descriptor fields get patched during the program generation;
The code of the class-specific drivers is targeted by
syz_usb_connect$hid,syz_usb_connect$cdc_ecm, and other variants (accompanied by matchingsyz_usb_control_io$*andsyz_usb_ep_read/write$*pseudo-syscalls).The descriptor fields for these
syz_usb_connectvariants are also intended to be patched during the program generation based on the matching rules specific to the driver class (to exercise various driver quirks).
So far, this has only been implemented for the HID (seegenerateUsbHidDeviceDescriptor) and the printer (seegenerateUsbPrinterDeviceDescriptor) classes;The code of the device-specific drivers is intended to be targeted by more
syz_usb_connectvariants whose descriptors do not get patched and are fully defined in syzlang instead. (However, they can be patched as well for drivers that define quirks.)So far, the only such (partial) descriptions have been added for the
ath9k,rtl8150,sierra_net, andlan78xxdrivers.
Dealing with complicated descriptors
Many USB drivers expect a fixed USB descriptors structure from the connected USB device.
Writing syzlang descriptions for these drivers is straightforward: just describe the required structures in syzlang.
However, some drivers allow various permutations of the interface/endpoint descriptors (typically the case for class-specific drivers like UVC).
Describing a fixed descriptor structure for such drivers in syzlang would allow passing the driver descriptor checks and efficiently fuzz the post-enumeration communication.
But it would also prevent syzkaller from exploring the various error-checking paths within the driver's enumeration code.
On the other hand, defining a relaxed structure for the descriptors would make syzkaller often fail the driver's enumeration and thus would not allow efficiently fuzzing the post-enumeration communication.
There are two proposed ways to handle such drivers:
Writing relaxed descriptions and adding seed programs (aka runtests).
This works by only defining a single
syz_usb_connectvariant with the relaxed descriptions of the USB descriptors but also adding a seed program (intosys/linux/test/) that contains thesyz_usb_connectpseudo-syscall with the specific fixed values to pass the driver's enumeration checks;Adding two
syz_usb_connectvariants.One with relaxed descriptions to explore the error-checking paths; and the other with fixed descriptions to allow fuzzing the post-enumeration communication.
Handling post-enumeration control requests
Many USB drivers finish their enumeration procedure with the SET_CONFIGURATION request.
Following this, these drivers start functioning normally and are ready to accept both control and non-control requests.
However, some USB drivers expect certain control requests to be handled by the device directly following the SET_CONFIGURATION request.
Without these requests being handled, these drivers abort their execution and disconnect the device.
There are two proposed ways to handle such drivers:
Adding seed programs (aka runtests).
This works by adding a seed program (into
sys/linux/test/) that contains both thesyz_usb_connectpseudo-syscall for the specific driver and a fewsyz_usb_control_iopseudo-sycalls that handle the required control requests.This is the currently recommended way of handling such drivers, as it allows syzkaller to permute the expected post-
SET_CONFIGURATIONcontrol requests and thus possibly trigger unexpected driver behavior during their handling.See #6283 for a reference implementation of this approach;
Modifying the behavior of the
syz_usb_connectpseudo-syscall.An alternative approach is to modify the C implementation of the
syz_usb_connectpseudo-syscall (or add a similar new pseudo-syscall) to handle the required post-SET_CONFIGURATIONrequests directly from C.Right now, this approach is only implemented for the
ath9kdriver viasyz_usb_connect_ath9k.
Unlikesyz_usb_connect,syz_usb_connect_ath9kalso handles the post-SET_CONFIGURATIONfirmware download requests expected byath9k.If this approach is to be used for other drivers, the proper implementation should rework and extend
syz_usb_connect_ath9kinstead of adding more similar pseudo-syscalls.
For example, add a new argument tosyz_usb_connectthat allows specifying (in syzlang) the responses to post-SET_CONFIGURATIONcontrol requests and which one of them is the last one (i.e., the condition for exiting the pseudo-syscall).
And then port the hardcoded responses from the C implementation ofsyz_usb_connect_ath9kinto syzlang and extend the functionality based on the specific new encountered cases.
Note: There is also a way to describe unusual pre-SET_CONFIGURATION GET_DESCRIPTOR requests via the conn_descs argument of syz_usb_connect, but none of the class/driver-specific descriptions use this feature at the moment.
However, this approach does not allow specifying the order of the responses if there are multiple requests of the same type and also does not allow handling pre-SET_CONFIGURATION non-GET_DESCRIPTOR requests, as no described drivers required this so far.
USB IDs patching
Many USB drivers implement various quirks depending of which device is connected.
Such quirks are either defined in the matching rule table for the driver (a table of usb_device_id structures; see e.g. usb_audio_ids) or hardcoded in the driver code (by manually checking e.g. the Vendor and Product IDs against certain values; see e.g. quirk_printers).
To exercise these driver quirks, syzkaller has to provide certain values (aka IDs) within the USB descriptors for the emulated device.
Listing these USB IDs manually in syzlang descriptions is inefficient (there are too many), so syzkaller employs the way of automatically extracting the IDs before building and then patching them in during the program generation.
The IDs defined in the driver matching rules can be extracted (and patched in) automatically; see Updating USB IDs.
However, the hardcoded IDs need to be specified manually (extracting them automatically is hard, as they are often embedded into the driver code in non-standard ways).
The proposed approach is to read the driver code to find out the hardcoded IDs and then modify the program generation code to embed them into the appropriate USB descriptors (see e.g. generateUsbPrinterDeviceDescriptor).
Updating USB IDs
syzkaller relies on a list of automatically-extracted USB IDs that are patched into syz_usb_connect (for the generic, the HID, and the printer variants) during the program generation.
To update the syzkaller USB IDs based on the USB drivers enabled in your kernel config:
Apply this kernel patch;
Build and boot the kernel;
Connect a USB HID device.
Assuming you have
CONFIG_USB_RAW_GADGET=yenabled, you can just run the keyboard emulation program;Use syz-usbgen to update sys/linux/init_vusb_ids.go:
bash./bin/syz-usbgen $KERNEL_LOG ./sys/linux/init_vusb_ids.goRevert the applied kernel patch and rebuild the kernel before starting syzkaller.
Seed programs
There are a few seed programs (aka runtests) for the USB pseudo-syscalls (named starting with the vusb prefix).
These seed programs serve two purposes:
Allow syzkaller to go deeper into the driver code without having to write detailed syzlang descriptions;
Allow verifying the USB fuzzing functionality and catching potential description-breaking changes by running the seed programs as runtests:
bash./bin/syz-manager -config usb-manager.cfg -mode run-tests -tests vusb
Limitations
Most of the limitations of the USB fuzzing support in syzkaller stem from the features missing from the Raw Gadget and Dummy HCD/UDC implementations (USB 3, isochronous transfers, etc).
In addition, syz_usb_connect only supports devices with a single configuration (but this can be fixed).
This is not a critical limitation, as most kernel drivers are tied to specific interfaces and do not care about the configurations.
However, there are USB devices whose drivers assume the device to have multiple specific configurations.
Things to improve
The core support for USB fuzzing is in place, but there's still space for improvement:
Remove the device from
usb_devicesinsyz_usb_disconnectand don't calllookup_usb_indexmultiple times withinsyz_usb_connect.
Currently, this causes some reproducers to have therepeatflag set when it's not required;Add descriptions for more relevant USB classes and drivers and resolve TODOs from sys/linux/vusb.txt;
Implement a proper way for dynamically extracting relevant USB ids from the kernel (a related discussion);
Add a mode for standalone fuzzing of physical USB hosts (from a Linux-based board with a UDC).
This includes: a. Making sure that the current waysyz_usb_connecthandles the enumeration works properly with different OSes (there are some differences);
b. Using USB requests coming from the host as a signal (like coverage) to enable "signal-driven" fuzzing;
c. Making UDC driver name configurable forsyz-execprogandsyz-prog2c;Generate syzkaller programs from a
usbmontrace produced by real USB devices (this should make syzkaller go significantly deeper into the USB drivers code).
Running reproducers on Linux-based boards
It is possible to run the reproducers for USB bugs generated by syzkaller on a Linux-based board plugged into a physical USB host.
See Running syzkaller USB reproducers in the Raw Gadget repository for the instructions.
Linux External Fuzzing Usbip (`docs/linux/external_fuzzing_usbip.md`)
USB/IP Fuzzing for Linux Kernel
Syzkaller supports fuzzing the Linux kernel USB/IP subsystem externally. We can set up a virtual network and send USB/IP packets to the client kernel as they are being received from an external server.
USB/IP fuzzing needs USB/IP configurations to be enabled. You can find the list in the configurations part.
Currently syzkaller only includes support for fuzzing the client side of USB/IP, which consists of 2 main parts:
- USB/IP pseudo-syscalls.
- Syzkaller descriptions.
Configurations
Following configurations should be enabled for USB/IP.
CONFIG_USBIP_CORE=y
CONFIG_USBIP_VHCI_HCD=y
CONFIG_USBIP_VHCI_HC_PORTS=8
CONFIG_USBIP_VHCI_NR_HCS=8
CONFIG_USBIP_HOST=y
CONFIG_USBIP_VUDC=y
CONFIG_USBIP_DEBUG=yPseudo-syscalls
Currently syzkaller defines one USB/IP pseudo-syscall and one USB/IP specific write syscall (see this for the pseudo-syscall and this for its syzkaller descriptions):
syz_usbip_server_init sets up USB/IP server. It creates a pair of connected socket and opens the /sys/devices/platform/vhci_hcd.0/attach file. Later, this pseudo-syscall writes the USB/IP client’s socket descriptor as well as port number used for USB/IP connection, USB device id and USB device speed into this file so that the USB/IP communication between client and server can start and client’s kernel can receive USB/IP packets from the server.
write$usbip_server sends USB/IP packets to client by using server's socket descriptor. (Particularly, USBIP_RET_SUBMIT and USBIP_RET_UNLINK packets.) We assume that the server can send arbitrary USB/IP packets instead of emulating a real device. These packets end up in the client's kernel and get parsed there.
**Further Improvements**
- Fuzzing the server side of USB/IP.
- Collect coverage from USB/IP kernel code.
Linux Found Bugs (`docs/linux/found_bugs.md`)
Found bugs
Most latest bugs are reported by syzbot and are listed here and on the dashboard.
Additional USB bugs are here.
newer first
- KASAN: use-after-free Read in screen_glyph_unicode
- KASAN: use-after-free Read in vc_do_resize
- KASAN: use-after-free in usb_hcd_unlink_urb
- KASAN: slab-out-of-bounds Read in gadget_dev_desc_UDC_store
- KASAN: use-after-free Write in snd_rawmidi_kernel_write1
- KASAN: use-after-free Write in config_item_get
- KASAN: use-after-free Read in f_hidg_poll
- KASAN: use-after-free Read in printer_ioctl
- KASAN: null-ptr-deref Read in tty_wakeup
- KASAN: use-after-free in afs_wake_up_async_call
- KASAN: use-after-free Read in gs_flush_chars
- kernel BUG at net/core/skbuff.c
- io_uring: avoid page allocation warnings
- io_uring: free allocated io_memory once
- io_uring: fix SQPOLL cpu validation
- locks: use-after-free in perf_trace_lock_acquire CVE-2019-19769
- cirrusfb: divide errors in cirrusfb_check_var/cirrusfb_check_pixclock/cirrusfb_set_par_foo
- floppy: fix out-of-bounds read in copy_buffer
- floppy: fix invalid pointer dereference in drive_name
- floppy: fix out-of-bounds read in next_valid_format
- floppy: fix div-by-zero in setup_format_params
- bpf: BPF_PROG_TEST_RUN leads to unkillable process
- timer_settime leads to unkillable process
- UBSAN: Undefined behaviour in drivers/scsi/sr_ioctl.c
- KASAN: use-after-free Read in ata_scsi_mode_select_xlat
- UBSAN: Undefined behaviour in fs/f2fs/extent_cache.c
- UBSAN: Undefined behaviour in drivers/input/misc/uinput.c
- general protection fault in spk_ttyio_ldisc_close
- rtnetlink: give a user socket to get_target_net() CVE-2018-14646
- tipc: NULL deref in tipc_net_finalize
- Kernel crash at i2cdev_ioctl_rdwr in drivers/i2c/i2c-dev.c
- UBSAN: Undefined behaviour in drivers/input/mousedev.c
- UBSAN: Undefined behaviour in mm/page_alloc.c
- WARNING in pkt setup dev
- UBSAN: Undefined behaviour in drivers/net/ppp/ppp_generic.c
- KASAN: use-after-free Read in raw_cmd_done
- KMSAN: uninit-value in selinux_socket_bind, selinux_socket_connect_helper
- UBSAN: Undefined behaviour in drivers/block/floppy.c
- net: BUG still has locks held in unix_stream_splice_read
- general protection fault in sockfs_setattr CVE-2018-12232
- KASAN: slab out of bounds Write in __jfs_setxattr CVE-2018-12233
- RDMA/mlx5: Fix NULL dereference while accessing XRC_TGT QPslogin
- KASAN: use-after-free Read in set_page_dirty_lock
- System freeze and NULL pointer dereference
- RDS: WARNING in rds_recv_hs_exthdrs
- RDS: slab-out-of-bounds Read in rds_rdma_extra_size
- netfilter: fix out-of-bounds accesses in clusterip_tg_check()
- net: hang in unregister_netdevice: waiting for lo to become free
- scsi: sg: assorted memory corruptions
- kcm: memory leak in kcm_sendmsg
- AF_KEY: memory leak in key_notify_policy
- sctp: memory leak in sctp_endpoint_init
- tipc: memory leak in tipc_nl_node_get_link
- tun: memory leak in tun_set_iff
- net/8021q: memory leak in register_vlan_dev
- net: memory leak in socket
- scsi: memory leak in sg_start_req
- sunrpc: infinite unkillable console spam in xs_tcp_setup_socket
- fs: possible deadlock in do_iter_write/do_splice
- net/ipv6: warning in __alloc_pages_slowpath/ipip6_tunnel_get_prl
- net/ipv6: GPF in rt6_ifdown
- net/ipv4: trying to register non-static key in ip_mc_clear_src
- net/can: trying to register non-static key in can_rx_register
- net: general protection fault in deactivate_slab
- net/ipv4: use-after-free in add_grec
- net/ipv6: use-after-free in ip6_dst_ifdown
- tty: possible deadlock in tty_buffer_flush
- net/ipv6: general protection fault in skb_release_data CVE-2017-9242
- drivers/net/hamradio: divide error in hdlcdrv_ioctl
- tty: fix port buffer locking
- kvm: warning in kvm_load_guest_fpu
- drivers/scsi: GPF in sg_read
- net/ipv4: use-after-free in ip_mc_drop_socket CVE-2017-8890 CVE-2017-9075 CVE-2017-9076 CVE-2017-9077
- net/ipv6: GPF in rt6_device_match
- x86: warning: kernel stack regs has bad 'bp' value
- net/key: slab-out-of-bounds in pfkey_compile_policy
- net/ipv6: warning in inet6_ifa_finish_destroy
- net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu
- net/ipv6: slab-out-of-bounds in ip6_tnl_xmit
- net/rose: null-ptr-deref in rose_route_frame
- time: hang due to timer_create/timer_settime
- net/core: BUG in unregister_netdevice_many
- net/xfrm: stack-out-of-bounds in xfrm_state_find
- net/bonding: stack-out-of-bounds in bond_enslave
- net: ipv6: RTF_PCPU should not be settable from userspace
- fs/notify/inotify: slab-out-of-bounds write in strcpy CVE-2017-7533
- net/ipv6: slab-out-of-bounds read in seg6_validate_srh
- kernel BUG at mm/hugetlb.c:742!
- net/key: slab-out-of-bounds in parse_ipsecrequests
- net/ipv4: use-after-free in ipv4_datagram_support_cmsg
- net/ipv4: use-after-free in ip_queue_xmit
- net: use-after-free in __ns_get_path
- net/ipv4: use-after-free in ip_check_mc_rcu
- net/ipv6: use-after-free in ipv6_sock_ac_close
- net/ipv4: use-after-free in ipv4_mtu
- net/dccp: BUG in tfrc_rx_hist_sample_rtt
- net/sctp: list double add warning in sctp_endpoint_add_asoc
- kvm: use-after-free in srcu_reschedule
- ata: WARNING in ata_bmdma_qc_issue
- net/sched: GPF in qdisc_hash_add
- sg: random memory corruptions
- fs: GPF in deactivate_locked_super
- loop: WARNING in sysfs_remove_group
- lib, fs, cgroup: WARNING in percpu_ref_kill_and_confirm
- ata: WARNING in ata_qc_issue
- security, hugetlbfs: write to user memory in hugetlbfs_destroy_inode
- netlink: NULL timer crash
- kvm: use-after-free function call in kvm_io_bus_destroy
- sound: use-after-free in snd_seq_cell_alloc
- usb: use-after-free write in usb_hcd_link_urb_to_ep
- net/kcm: double free of kcm inode
- crypto: out-of-bounds write in pre_crypt
- security: double-free in superblock_doinit
- kvm: WARNING in kvm_apic_accept_events
- tcp: fix potential double free issue for fastopen_req
- net/udp: slab-out-of-bounds Read in udp_recvmsg
- net: deadlock between ip_expire/sch_direct_xmit
- srcu: BUG in __synchronize_srcu
- net/sctp: recursive locking in sctp_do_peeloff
- kvm: WARNING in vmx_handle_exit
- futex: use-after-free in futex_wait_requeue_pi
- kvm/arm64: use-after-free in kvm_vm_ioctl/vmacache_update
- kvm/arm64: use-after-free in kvm_unmap_hva_handler/unmap_stage2_pmds
- local privilege escalation flaw in n_hdlc CVE-2017-2636
- netlink: GPF in netlink_unicast
- perf: use-after-free in perf_release
- net/ipv6: null-ptr-deref in ip6mr_sk_done
- bpf: kernel NULL pointer dereference in map_get_next_key
- crypto: deadlock between crypto_alg_sem/rtnl_mutex/genl_mutex
- kvm: use-after-free in vmx_check_nested_events/vmcs12_guest_cr0
- sound: another deadlock in snd_seq_pool_done
- rcu: WARNING in rcu_seq_end
- fs: use-after-free in path_lookupat
- ucount: use-after-free read in inc_ucount & dec_ucount
- net/ipv4: division by 0 in tcp_select_window
- net: heap out-of-bounds in fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone
- mm: use-after-free in zap_page_range
- net/kcm: use-after-free in kcm_wq
- idr: use-after-free write in ida_get_new_above
- sg: stack out-of-bounds write in sg_write CVE-2017-7187
- cgroup: WARNING in cgroup_kill_sb
- net/rds: use-after-free in rds_find_bound/memcmp
- net: sleeping function called from invalid context in net_enable_timestamp
- net: use-after-free in neigh_timer_handler/sock_wfree
- net/sctp: use-after-free in sctp_association_put
- fs: use-after-free in userfaultfd_exit
- net/ipv4: inconsistent lock state in tcp_conn_request/inet_ehash_insert
- net/ipv4: suspicious RCU usage in ip_ra_control
- net/ipv4: deadlock in ip_ra_control
- net/dccp: dccp_create_openreq_child freed held lock
- nested_vmx_merge_msr_bitmap
- ipc: use-after-free in shm_get_unmapped_area
- sounds: deadlocked processed in snd_seq_pool_done
- net/atm: vcc_sendmsg calls kmem_cache_alloc in non-blocking context
- ata: WARNING in ata_sff_qc_issue
- net/rds: use-after-free in inet_create
- mm: fault in __do_fault
- kvm: WARNING in nested_vmx_vmexit
- net: GPF in rt6_nexthop_info
- sound: spinlock lockup in snd_timer_user_tinterrupt
- mm: GPF in bdi_put
- net/sctp: use-after-free in sctp_hash_transport
- net/bridge: warning in br_fdb_find
- net/ipv6: null-ptr-deref in ip6_route_del/lock_acquire
- net: possible deadlock in skb_queue_tail
- DCCP double-free vulnerability (local root) CVE-2017-6074
- net: warning in inet_sock_destruct
- net/pptp: use-after-free in dst_release
- net/udp: slab-out-of-bounds in udp_recvmsg/do_csum CVE-2017-6347
- WARNING in skb_warn_bad_offload
- tty: panic in tty_ldisc_restore
- net: BUG in __skb_gso_segment
- net/dccp: use-after-free in dccp_feat_activate_values
- net/kcm: GPF in kcm_sendmsg
- net/xfrm: stack out-of-bounds in xfrm_flowi_sport
- net/llc: BUG in llc_sap_state_process/skb_set_owner_r CVE-2017-6345
- net/llc: bug in llc_pdu_init_as_xid_cmd/skb_over_panic
- net/packet: use-after-free in packet_rcv_fanout
- net: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected in skb_array_produce
- net/ipv4: null-ptr-deref in udp_rmem_release/sk_memory_allocated_sub
- net/sctp: null-ptr-deref in sctp_put_port/sctp_endpoint_destroy
- net/ipv4: warning in nf_nat_ipv4_fn
- net/ipv6: double free in ipip6_dev_free
- sound: use-after-free in snd_seq_queue_alloc
- loop: divide error in transfer_xor
- net/xfrm: use of uninit spinlock in xfrm_policy_flush
- mm: double-free in cgwb_bdi_init
- packet: round up linear to header len
- net/icmp: null-ptr-deref in ping_v4_push_pending_frames
- net/kcm: WARNING in kcm_write_msgs
- tcp: avoid infinite loop in tcp_splice_read() CVE-2017-6214
- tun: read vnet_hdr_sz once
- macvtap: read vnet_hdr_size once
- udp: properly cope with csum errors
- ipv6: tcp: add a missing tcp_v6_restore_cb()
- ip6_gre: fix ip6gre_err() invalid reads CVE-2017-5897
- ipv4: keep skb->dst around in presence of IP options CVE-2017-5970
- net: use a work queue to defer net_disable_timestamp() work
- netlabel: out of bound access in cipso_v4_validate()
- ipv6: pointer math error in ip6_tnl_parse_tlv_enc_lim()
- net: heap out-of-bounds in ip6_fragment CVE-2017-9074
- tcp: fix 0 divide in __tcp_select_window()
- keys: GPF in request_key
- net/tcp: warning in tcp_try_coalesce/skb_try_coalesce
- crypto: NULL deref in sha512_mb_mgr_get_comp_job_avx2
- sound: unable to handle kernel paging request snd_seq_prioq_cell_out
- scsi: BUG in scsi_init_io
- mm: sleeping function called from invalid context shmem_undo_range
- timerfd: use-after-free in timerfd_remove_cancel
- scsi: use-after-free in sg_start_req
- mm: deadlock between get_online_cpus/pcpu_alloc
- BUG at net/sctp/socket.c:7425
- kvm: use-after-free in irq_bypass_register_consumer
- net: suspicious RCU usage in nf_hook
- kvm: fix page struct leak in handle_vmon CVE-2017-2596
- ipv6: fix ip6_tnl_parse_tlv_enc_lim()
- kvm: WARNING in mmu_spte_clear_track_bits
- perf: use-after-free in perf_event_for_each
- net: use-after-free in tw_timer_handler
- namespace: deadlock in dec_pid_namespaces
- sctp: kernel memory overwrite attempt detected in sctp_getsockopt_assoc_stats
- kvm: deadlock in kvm_vgic_map_resources
- net/atm: warning in alloc_tx/__might_sleep
- net/ipv6: use-after-free in sock_wfree
- kvm: kvm: BUG in loaded_vmcs_init
- kvm: NULL deref in vcpu_enter_guest
- kvm: use-after-free in complete_emulated_mmio CVE-2017-2584
- kvm: BUG in kvm_unload_vcpu_mmu
- x86: warning in unwind_get_return_address
- ipc: BUG: sem_unlock unlocks non-locked lock
- kvm: WARNING in mmu_spte_clear_track_bits
- sctp: suspicious rcu_dereference_check() usage in sctp_epaddr_lookup_transport
- kvm: use-after-free in process_srcu
- kvm: assorted bugs after OOMs
- kvm: deadlock between kvm_io_bus_register_dev/kvm_hv_set_msr_common
- netlink: GPF in netlink_dump
- fs, net: deadlock between bind/splice on af_unix
- net: use-after-free in worker_thread
- net: signed overflows in SO_{SND|RCV}BUFFORCE sockopts CVE-2016-9793 CVE-2012-6704
- net/can: warning in raw_setsockopt/__alloc_pages_slowpath
- net/ipv6: null-ptr-deref in ip6_rt_cache_alloc
- net/dccp: use-after-free in dccp_invalid_packet
- net/sctp: vmalloc allocation failure in sctp_setsockopt/xt_alloc_table_info
- net: BUG in unix_notinflight
- net: GPF in eth_header CVE-2016-9755
- net: deadlock on genl_mutex
- net: GPF in rt6_get_cookie
- netlink: GPF in sock_sndtimeo
- scsi: use-after-free in bio_copy_from_iter CVE-2016-9576
- net/udp: bug in skb_pull_rcsum
- net/icmp: null-ptr-deref in icmp6_send CVE-2016-9919
- net/can: use-after-free in bcm_rx_thr_flush
- kvm: slab-out-of-bounds write in __apic_accept_irq CVE-2016-9777
- mm: BUG in pgtable_pmd_page_dtor
- logfs: GPF in logfs_alloc_inode
- mm, floppy: unkillable task faulting on fd0
- kvm: deadlock between kvm_vm_ioctl_get_dirty_log/kvm_hv_set_msr_common/kvm_create_pit
- kvm: WARNING in em_jmp_far CVE-2016-9756
- kvm: WARNING in rtc_status_pending_eoi_check_valid
- kvm: GPF in kvm_ioapic_set_irq
- mm: BUG in munlock_vma_pages_range
- kvm: WARNING in kvm_arch_vcpu_ioctl_run
- kvm: use-after-free/GPF in kvm_irq_delivery_to_apic_fast
- kvm: out-of-bounds write in __rtc_irq_eoi_tracking_restore_one
- kvm: BUG in pte_list_remove
- kvm: recursive lock in kvm_clear_async_pf_completion_queue
- kvm: WARNING in em_ret_far
- kvm: GPF in irqfd_shutdown/eventfd_ctx_remove_wait_queue
- kvm: GPF in gfn_to_rmap
- kvm: paging fault in kvm_gfn_to_hva_cache_init
- kvm: suspicious RCU usage/missed lock in kvm_lapic_set_vapic_addr
- kvm: use-after-free in irq_bypass_register_consumer
- kvm: WARNING in kvm_load_guest_fpu
- kvm: GPF in kvm_pic_set_irq
- kvm: GPF in irq_bypass_unregister_consumer
- kvm: GPF in __get_kvmclock_ns
- kvm: WARNING In kvm_apic_accept_events
- kvm: WARNING in __x86_set_memory_region
- tcp: take care of truncations done by sk_filter()
- net/l2tp: use-after-free write in l2tp_ip6_close
- net/sctp: null-ptr-deref in sctp_inet_listen
- net/tcp: warning in tcp_recvmsg
- net/netlink: another global-out-of-bounds in genl_family_rcv_msg/validate_nla
- bpf: kernel BUG in htab_elem_free
- net/netlink: global-out-of-bounds in genl_family_rcv_msg/validate_nla
- net/ipv6: null-ptr-deref in inet6_bind
- net/dccp: null-ptr-deref in dccp_parse_options
- net/dccp: null-ptr-deref in dccp_v4_rcv/selinux_socket_sock_rcv_skb
- net/tcp: null-ptr-deref in __inet_lookup_listener/inet_exact_dif_match
- net/dccp: warning in dccp_feat_clone_sp_val/__might_sleep
- net/can: warning in bcm_connect/proc_register
- net/ipv4: warning in inet_sock_destruct
- net/sctp: slab-out-of-bounds in sctp_sf_ootb CVE-2016-9555
- net/dccp: warning in dccp_set_state
- net/netlink: bad unlock balance in netlink_diag_dump
- net/netlink: null-ptr-deref in netlink_dump/lock_acquire
- net/ipx: null-ptr-deref in ipxrtr_route_packet
- net/sctp: use-after-free in __sctp_connect
- fs: WARNING in locks_unlink_lock_ctx (not holding proper lock)
- kernel BUG in dio_get_page
- drm: GPF in drm_getcap
- fs: GPF in bd_mount
- tty, fbcon: use-after-free in fbcon_invert_region
- drm: NULL pointer dereference in drm_mode_object_find()
- 6pack: stack-out-of-bounds in sixpack_receive_buf
- logfs: GPF in logfs_init_inode
- tty: use-after-free in n_tty_receive_buf_fast
- sound: divide by 0 in snd_hrtimer_callback (or hang)
- mm: GPF in __insert_vmap_area
- fs, tty: WARNING in devpts_get_priv
- fanotify: unkillable hanged processes
- drm: GPF in drm_context_switch_complete
- drm: GPF in drm_legacy_lock_free
- sound: division by 0 in snd_hrtimer_callback
- perf: WARNING in perf_event_read
- drm: WARNING in drm_irq_by_busid
- dri: WARNING in idr_remove
- mm: use-after-free in collapse_huge_page
- kcm: use-after-free in fput of kcm socket
- bdev: fix NULL pointer dereference in sync()/close() race
- bdev: fix NULL pointer dereference
- BUG: sleeping function called from invalid context at mm/mempolicy.c:553
- use-after-free in ppp_unregister_channel
- net/tipc: NULL-ptr dereference in tipc_nl_publ_dump
- HID: i2c-hid: fix OOB write in i2c_hid_set_or_send_report()
- mm: memory corruption on mmput
- perf: WARNING in perf_event_read
- 9p2000.L stat/unlink race (WARNING: fs/inode.c:280 drop_nlink)
- mm: page fault in __do_huge_pmd_anonymous_page
- usb: memory allocation WARNING in hcd_buffer_alloc
- dccp: potential deadlock in dccp_v4_ctl_send_reset
- mm: GPF in find_get_pages_tag
- mm: BUG in page_move_anon_rmap
- block: GPF in get_task_ioprio
- tty: stall in n_tty_ioctl/inq_canon
- random: negative entropy/overflow: pool input count -40000
- bpf: use after free in array_map_alloc CVE-2016-4794
- kvm: use-after-free in kvm_irqfd_release
- kvm: GPF in kvm_lapic_set_tpr
- sound: use-after-free in hrtimer_cancel
- sound: hang in snd_timer_interrupt
- sound: deadlock involving snd_hrtimer_callback
- fs: GPF in locked_inode_to_wb_and_lock_list
- x86: bad pte in pageattr_test
- tty: memory leak in tty_open
- net: memory leak due to CLONE_NEWNET
- lockdep WARNING in get_online_cpus
- mm: BUG in khugepaged_scan_mm_slot
- sound: use-after-free in snd_timer_interrupt
- scsi: machine hang due to write to /dev/sg0
- AMD newest ucode 0x06000832 for Piledriver-based CPUs seems to behave in a problematic way
- sound: uninterruptible hang in snd_seq_oss_writeq_sync
- fs: uninterruptible hang in handle_userfault
- net: memory leak in N_6PACK driver
- net: memory leak in lapb_register
- net: memory leak in mkiss_open
- sound: list corruption in delete_and_unsubscribe_port
- kvm: GPF in kvm_pic_clear_all
- kvm: GPF in kvm_irq_map_gsi
- tty: memory leak in tty_register_driver
- sound: memory leak in snd_seq_pool_init
- tty: deadlock between tty_buffer_flush/n_tracesink_open
- sound: heap out-of-bounds write in dummy_systimer_prepare
- fs: NULL deref in atime_needs_update
- sound: spinlock lockup in snd_seq_oss_write
- net: memory leak in ip_cmsg_send
- net/irda: BUG: looking up invalid subclass: 4294967295 CVE-2017-6348
- sound: use-after-free in snd_timer_start1
- tty: tty_struct memory leak
- gigaset: memory leak in gigaset_initcshw
- sound: out-of-bounds write in snd_rawmidi_kernel_write1
- mm: uninterruptable tasks hanged on mmap_sem
- sound: another WARNING in rawmidi_transmit_ack
- sound: use-after-free in snd_seq_deliver_single_event
- sound: WARNING in snd_rawmidi_kernel_write1
- sound: deadlock between snd_pcm_oss_write/snd_pcm_oss_mmap
- ata: BUG in ata_sff_hsm_move
- WARNING in set_restore_sigmask
- BUG: bad unlock balance detected in vma_unlock_anon_vma
- bluetooth: use-after-free in vhci_send_frame
- mm: another VM_BUG_ON_PAGE(PageTail(page))
- scsi: NULL deref in sg_start_req
- mm: BUG in expand_downwards
- sound: heap out-of-bounds write in dummy_systimer_prepare
- WARNING in do_jobctl_trap
- mm: VM_BUG_ON_PAGE(PageTail(page)) in mbind
- net/bluetooth: workqueue destruction WARNING in hci_unregister_dev
- gpu: kmalloc size WARNING in vga_arb_write
- net/rfkill: WARNING in rfkill_fop_read
- sound: use-after-free in _snd_timer_stop
- net/irda: use-after-free in ircomm_param_request
- net/sctp: out-of-bounds access in sctp_add_bind_addr
- ext4: BUG: scheduling while atomic in ext4_commit_super
- sound: WARNING in snd_rawmidi_transmit_ack
- floppy: GPF in floppy_rb0_cb
- tty: kmalloc size WARNING in vc_do_resize
- mm: WARNING in __delete_from_page_cache
- sound: WARNING in snd_seq_oss_synth_cleanup
- sound: deadlock between snd_rawmidi_kernel_open/snd_seq_port_connect
- net: GPF in netlink_getsockbyportid
- fs: use-after-free in link_path_walk
- fs: sandboxed process brings host down
- net: use-after-free in recvmmsg
- struct pid memory leak
- net: WARNING in dccp_set_state
- mm: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected in split_huge_page_to_list
- sound: BUG in snd_ctl_find_numid
- net: GPF in __netlink_ns_capable
- crypto: slab-out-of-bounds in skcipher_recvmsg
- net: hang in ip_finish_output
- kvm: access to invalid memory in mmu_zap_unsync_children
- kvm: using uninitialized var in tdp_page_fault
- sound: spinlock lockup in sound/core/timer.c
- sound: GPF in snd_timer_user_params
- sound: use-after-free in snd_timer_interrupt
- sound: use-after-free in snd_timer_user_ioctl
- crypto: use-after-free in skcipher_sock_destruct
- net/sctp: use-after-free in __sctp_connect
- net: WARNING in tcp_recvmsg
- sound: use-after-free in snd_timer_stop
- sound: GPF in snd_seq_fifo_clear
- crypto: ablk_decrypt causes BUG in scatterwalk
- kvm: GPF in native_set_debugreg
- kvm: GPF in kvm_lapic_latched_init
- kvm: WARNING in kvm_apic_accept_events
- kvm: vmalloc allocation failure in kvm_vm_ioctl
- kvm: vmalloc allocation failure in kvm_vcpu_ioctl_set_cpuid
- kvm: WARNING in __x86_set_memory_region
- kvm: WARNING in exception_type
- mm: possible deadlock in mm_take_all_locks
- net/nfc: GPF in llcp_sock_getname
- net/netlink: memory leak in netlink_sendmsg
- net/tipc: memory leak in tipc_release
- memory leak in lapb_create_cb
- net/sctp: sctp_datamsg memory leak
- net/sctp: sock memory leak
- net/nfc: user-controllable kmalloc size in nfc_llcp_send_ui_frame
- tty: deadlock between n_tracerouter_receivebuf and flush_to_ldisc
- crypto: use-after-free in alg_bind
- crypto: deadlock in alg_setsockopt
- crypto: use-after-free in rng_recvmsg
- use-after-free in skcipher_bind
- 9p: sleeping function called from invalid context in v9fs_vfs_atomic_open_dotl
- fs: WARNING in locks_free_lock_context
- net: user-controllable kmalloc size in __sctp_setsockopt_connectx
- GPF in gf128mul_64k_bbe
- use-after-free in hash_sock_destruct
- GPF in lrw_crypt
- bad page state due to PF_ALG socket
- use-after-free in skcipher_sock_destruct
- use-after-free in sixpack_close
- net: heap-out-of-bounds in sock_setsockopt
- BUG_ON(!PageLocked(page)) in munlock_vma_page
- perf: stalls in perf_install_in_context/perf_remove_from_context
- Information leak in sco_sock_bind CVE-2015-8575
- Information leak in llcp_sock_bind/llcp_raw_sock_bind
- Information leak in pptp_bind
- use-after-free in pptp_connect
- GPF in keyctl CVE-2015-7550
- another use-after-free in sctp_do_sm
- use-after-free in inet6_destroy_sock
- WARNING in crypto_wait_for_test
- int overflow in io_getevents
- use-after-free in ip6_xmit
- use-after-free in __perf_install_in_context
- undefined shift in __bpf_prog_run
- signed integer overflow in ktime_add_safe
- jump label: negative count!
- memory leak in alloc_huge_page
- memory leak in do_ipv6_setsockopt
- heap out-of-bounds access in array_map_update_elem
- deadlock in perf_ioctl
- user-controllable kmalloc size in bpf syscall
- net: use after free in ip6_make_skb
- user-controllable kmalloc size in sctp_getsockopt_local_addrs
- use-after-free in ip6_setup_cork
- gigaset: freeing an active object
- Freeing active kobject in pps_device_destruct
- GPF in process_one_work (flush_to_ldisc)
- use-after-free in tty_check_change
- WARNING in tcp_recvmsg
- use-after-free in irtty_open
- use-after-free in sock_wake_async
- WARNING in handle_mm_fault
- WARNING in gsm_cleanup_mux
- use-after-free in sctp_do_sm
- yet another uninterruptable hang in sendfile
- GPF in add_key
- another uninterruptable hang in sendfile
- deadlock during fuseblk shutdown
- tty,net: use-after-free in x25_asy_open_tty
- deadlock between tty_write and tty_send_xchar
- WARNING in shmem_evict_inode
- Deadlock between setsockopt/getsockopt
- Deadlock between bind and splice
- Use-after-free in ipv4_conntrack_defrag
- Use-after-free in selinux_ip_postroute_compat
- Use-after-free in unshare
- GPF in tcp_sk_init/icmp_sk_init
- lockdep warning in ip_mc_msfget
- WARNING in task_participate_group_stop
- Resource leak in unshare
- Paging fault with hard IRQs disabled in getsockopt
- Unkillable processes due to PTRACE_TRACEME
- Use-after-free in ep_remove_wait_queue CVE-2013-7446
- GPF in shm_lock
- GPF in rt6_uncached_list_flush_dev
- Infinite loop in ip6_fragment
- Uninterruptable hang in sendfile
- GPF in keyring_destroy CVE-2015-7872
Linux Found Bugs Usb (`docs/linux/found_bugs_usb.md`)
Found Linux kernel USB bugs
Reported by syzbot
Starting from early 2019, bugs in the USB drivers are being automatically reported by the USB fuzzing instance of syzbot and can be found here. A list of CVEs for some of those can be found here here.
Some of the USB bugs are reported by the KMSAN fuzzing instance and can be either found here or here (via a manual search, e.g. for kernel-usb-infoleak).
Manually reported
These are the bugs that were manually reported before USB fuzzing was integrated into syzbot.
USB drivers
- usb/core: memory corruption due to an out-of-bounds access in usb_destroy_configuration [fix] [CVE-2017-17558]
- usb/net/zd1211rw: possible deadlock in zd_chip_disable_rxtx
- usb/sound: use-after-free in __uac_clock_find_source [fix]
- usb/sound: slab-out-of-bounds in parse_audio_unit [fix]
- usb/media/em28xx: use-after-free in dvb_unregister_frontend [fix]
- usb/media/technisat: slab-out-of-bounds in technisat_usb2_rc_query
- usb/media/tm6000: use-after-free in tm6000_read_write_usb
- usb/net/qmi_wwan: divide error in qmi_wwan_probe/usbnet_probe [fix1, fix2] [CVE-2017-16649, CVE-2017-16650]
- usb/media/uvc: slab-out-of-bounds in uvc_probe
- usb/media/em28xx: use-after-free in em28xx_dvb_fini
- usb/media/em28xx: use-after-free in v4l2_fh_init
- usb/media/pvrusb2: WARNING in pvr2_i2c_core_done/sysfs_remove_group
- usb/sound/usx2y: WARNING in usb_stream_start [fix]
- usb/net/hfa384x: WARNING in submit_rx_urb/usb_submit_urb
- usb/media/dw2102: null-ptr-deref in dvb_usb_adapter_frontend_init/tt_s2_4600_frontend_attach
- usb/net/asix: kernel hang in asix_phy_reset
- usb/media/dtt200u: use-after-free in __dvb_frontend_free [fix] [CVE-2017-16648]
- usb/media/mxl111sf: trying to register non-static key in mxl111sf_ctrl_msg
- usb/media/au0828: use-after-free in au0828_rc_unregister
- usb/input/gtco: slab-out-of-bounds in parse_hid_report_descriptor [fix] [CVE-2017-16643]
- usb/core: slab-out-of-bounds in usb_get_bos_descriptor [fix] [CVE-2017-16535]
- usb/net/asix: null-ptr-deref in asix_suspend [fix] [CVE-2017-16647]
- usb/net/rt2x00: warning in rt2800_eeprom_word_index
- usb/irda: global-out-of-bounds in irda_qos_bits_to_value
- usb/media/imon: global-out-of-bounds in imon_probe/imon_init_intf0
- usb/sound: use-after-free in snd_usb_mixer_interrupt [fix] [CVE-2017-16527]
- usb/net/rtlwifi: trying to register non-static key in rtl_c2hcmd_launcher
- usb/net/prism2usb: warning in hfa384x_usbctlxq_run/usb_submit_urb
- usb/nfs/pn533: use-after-free in pn533_send_complete
- usb/media/imon: null-ptr-deref in imon_probe [fix] [CVE-2017-16537]
- usb/net/prism2usb: warning in hfa384x_drvr_start/usb_submit_urb
- usb/net/ath6kl: GPF in ath6kl_usb_alloc_urb_from_pipe
- usb/net/ar5523: warning in ar5523_submit_rx_cmd/usb_submit_urb
- usb/media/uvc: BUG in uvc_mc_create_links/media_create_pad_link
- usb/media/v4l2: use-after-free in video_unregister_device/device_del
- usb/serial/visor: slab-out-of-bounds in palm_os_3_probe [fix on the way]
- usb/misc/usbtest: null-ptr-deref in usbtest_probe/get_endpoints [fix] [CVE-2017-16532]
- usb/misc/ims-pcu: slab-out-of-bounds in ims_pcu_parse_cdc_data [fix] [CVE-2017-16645]
- usb/serial: use-after-free in usb_serial_disconnect/__lock_acquire [fix1, fix2] [CVE-2017-16525]
- usb/misc/rio500: double-free or invalid-free in disconnect_rio
- usb/sound/caiaq: warning in init_card/usb_submit_urb [fix]
- usb/input/aiptek: warning in aiptek_open/usb_submit_urb
- usb/net/lan78xx: use-after-free in lan78xx_write_reg
- usb/media/b2c2: GPF in flexcop_usb_transfer_init
- usb/media/uvc: warning in uvc_scan_chain_forward/__list_add
- usb/sound/line6: trying to register non-static key in podhd_disconnect
- usb/sound/line6: warning in line6_start_listen/usb_submit_urb [fix]
- usb/media/lmedm04: GPF in lme2510_int_read/usb_pipe_endpoint [fix1, fix2] [CVE-2017-16538]
- usb/sound/bcd2000: warning in bcd2000_idrivers/usb/serial/usb-serial.cnit_device [fix]
- usb/wireless/rsi_91x: use-after-free write in __run_timers
- usb/media/zr364xx: GPF in zr364xx_vidioc_querycap/strlcpy
- usb/media/stkwebcam: use-after-free in v4l2_ctrl_handler_free
- usb/media/dib0700: BUG in stk7070p_frontend_attach/symbol_put_addr [fix] [CVE-2017-16646]
- usb/sounds: slab-out-of-bounds read in snd_usb_create_streams [fix] [CVE-2017-16529]
- usb/media/hdpvr: trying to register non-static key in hdpvr_probe [fix] [CVE-2017-16644]
- usb/net/hso: warning in hso_free_net_device
- usb/net/hso: global-out-of-bounds in hso_probe
- usb/media/smsusb: use-after-free in worker_thread
- usb/storage/uas: slab-out-of-bounds in uas_probe [fix] [CVE-2017-16530]
- usb/sound/usx2y: warning in usb_stream_new/__alloc_pages_slowpath [fix]
- usb/media/pvrusb2: warning in pvr2_send_request_ex/usb_submit_urb [fix]
- usb/media/smsusb: null-ptr-deref in smsusb_init_device
- usb/media/cx231xx: null-ptr-deref in cx231xx_usb_probe [fix] [CVE-2017-16536]
- usb/net/p54: trying to register non-static key in p54_unregister_leds [fix]
- usb/core: slab-out-of-bounds read in cdc_parse_cdc_header [fix] [CVE-2017-16534]
- usb/hid: slab-out-of-bounds read in usbhid_parse [fix] [CVE-2017-16533]
- usb/core: slab-out-of-bounds in usb_set_configuration [fix] [CVE-2017-16531]
- usb/uwb: WARNING in hwarc_neep_init/usb_submit_urb [fix]
- usb/uwb: GPF in uwbd_start [fix] [CVE-2017-16526]
- usb/joystick: warnings in xpad_start_input and xpad_try_sending_next_out_packet [fix]
- usb/midi: use-after-free in snd_rawmidi_dev_seq_free [fix] [CVE-2017-16528]
- usb/core: warning in usb_create_ep_devs/sysfs_create_dir_ns [fix]
GadgetFS
- usb/gadget: stalls in dummy_timer / usbtouch_probe [fix]
- usb/gadget: null-ptr-deref in dev_ioctl [fix]
- usb/gadget: copy_to_user called with spinlock held [fix]
- usb/gadget: potential deadlock in gadgetfs_suspend [fix]
- usb/gadget: another GPF in usb_gadget_unregister_driver [fix]
- usb/gadget: warning in ep_write_iter/__alloc_pages_nodemask [fix]
- usb/gadget: slab-out-of-bounds write in dev_config [fix]
- usb/gadget: warning in dummy_free_request [fix]
- usb/gadget: poor checks of wTotalLength in config descriptors [fix]
- usb/gadget: use-after-free in gadgetfs_setup [fix]
- usb/gadget: GPF in usb_gadget_unregister_driver [fix]
- usb/gadget: warning in dev_config/memdup_user [fix]
Linux Internals (`docs/linux/internals.md`)
Linux-specific syzkaller internals
It's possible to fuzz some external Linux kernel interfaces with syzkaller:
Linux Kernel Configs (`docs/linux/kernel_configs.md`)
Linux kernel configs
List of recommended kernel configs for syzkaller. See syzbot config for a reference config.
Syzkaller features
To enable coverage collection, which is extremely important for effective fuzzing:
CONFIG_KCOV=y
CONFIG_KCOV_INSTRUMENT_ALL=y
CONFIG_KCOV_ENABLE_COMPARISONS=y
CONFIG_DEBUG_FS=yNote that CONFIG_KCOV_ENABLE_COMPARISONS feature also requires gcc8+ and the following commits if you are testing an old kernel:
kcov: support comparison operands collection
kcov: fix comparison callback signatureTo detect memory leaks using the Kernel Memory Leak Detector
(kmemleak):
CONFIG_DEBUG_KMEMLEAK=yTo show code coverage in web interface:
For Linux < 5.12
CONFIG_DEBUG_INFO=yFor Linux >= 5.12
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=yFor detection of enabled syscalls and kernel bitness:
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=yFor better sandboxing:
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG=yFor namespace sandbox:
CONFIG_USER_NS=yFor running in VMs make kvm_guest.config is generally required.
Debian images produced by tools/create-image.sh also require:
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=yIt is recommended to disable the following config (and required if your kernel doesn't have commits arm64: setup: introduce kaslr_offset()
and kcov: make kcov work properly with KASLR enabled):
# CONFIG_RANDOMIZE_BASE is not setIt is also recommended to disable the Predictable Network Interface Names mechanism. This can be done
either via syzkaller configuration (see details here) or by adjusting the following configs:
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="net.ifnames=0"Bug detection configs
Syzkaller is meant to be used with
KASAN (available upstream with CONFIG_KASAN=y),
KTSAN (prototype available),
KMSAN (prototype available),
or KUBSAN (available upstream with CONFIG_UBSAN=y).
Enable KASAN for use-after-free and out-of-bounds detection:
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=yFor testing with fault injection enable the following configs (syzkaller will pick it up automatically):
CONFIG_FAULT_INJECTION=y
CONFIG_FAULT_INJECTION_DEBUG_FS=y
CONFIG_FAULT_INJECTION_USERCOPY=y
CONFIG_FAILSLAB=y
CONFIG_FAIL_PAGE_ALLOC=y
CONFIG_FAIL_MAKE_REQUEST=y
CONFIG_FAIL_IO_TIMEOUT=y
CONFIG_FAIL_FUTEX=yNote: you also need the following commits if you are testing an old kernel:
fault-inject: support systematic fault injection
fault-inject: simplify access check for fail-nth
fault-inject: fix wrong should_fail() decision in task context
fault-inject: add /proc/<pid>/fail-nthAny other debugging configs, the more the better, here are some that proved to be especially useful:
CONFIG_LOCKDEP=y
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_PROVE_RCU=y
CONFIG_DEBUG_VM=y
CONFIG_REFCOUNT_FULL=y
CONFIG_FORTIFY_SOURCE=y
CONFIG_HARDENED_USERCOPY=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_WQ_WATCHDOG=yIncrease hung/stall timeout to reduce false positive rate:
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=140
CONFIG_RCU_CPU_STALL_TIMEOUT=100Linux Reporting Kernel Bugs (`docs/linux/reporting_kernel_bugs.md`)
Reporting Linux kernel bugs
Before reporting a bug make sure nobody else has already reported it. The easiest way to do
this is to search through the syzkaller mailing list,
syzkaller-bugs mailing list,
syzbot dashboard,
and kernel mailing lists for key frames present in the kernel stack traces.
Do NOT report bugs on old kernel versions.
Old kernel generally means older than a week. Reports on old kernels are typically not considered as valid,
and asked to be reproduced on a fresh kernel. Do not report bugs found on stable/LTS kernels.
Bugs on stable/LTS kernels should be reproduced on a fresh upstream kernel, and reported as such.
Or, if they are already fixed upstream, the fix commit should be
submitted to stable.
Please report bugs to the Linux kernel maintainers.
To find out the list of maintainers responsible for a particular kernel subsystem, use the get_maintainer.pl script: ./scripts/get_maintainer.pl -f guilty_file.c. Please add [email protected] to the CC list.
Minimal info to include in the report:
- exact kernel branch and revision where the bug occurred
- exact kernel .config
- kernel OOPS message (BUG, KASAN report, etc), preferably with source files and line numbers
- reproducer, if known (see below)
Properly configure the kernel.
An improperly-configured kernel may lead to bad, and even false reports. There is no official way to get
proper config for testing/fuzzing. You need to tune timeouts, enable debug info, enable lots of debug configs,
disable some other debug configs, etc. You may use one of
syzbot configs, or check guidelines for
individual settings.
Your email client should be configured in plain text mode when sending bug reports.
Kernel mailing lists reject HTML formatted messages. For example, if you use GMail set "Plain text mode" option.
Do NOT mimic syzbot reports.
For example, don't say "syzbot has found". However, you should mention that the bug is found with syzkaller.
Think of what you report. Today, Linux maintainers are overwhelmed with bug reports, so increasing the incoming flow won't help to fix all the bugs.
The more actionable your report is, the higher the chance that it will be addressed.
Note that people are more likely to care about kernel crashes (e.g. use-after-frees or panics) than of INFO: messages and such, unless it is clearly visible from the report what exactly is wrong.
If there are stalls or hangs, only report them if they are frequent enough or have a reliable reproducer.
Overall, bugs without reproducers are way less likely to be triaged and fixed.
If the bug is reproducible, include the reproducer (C source if possible, otherwise a syzkaller program) and the .config you used for your kernel.
If the reproducer is available only in the form of a syzkaller program, please link the instructions on how to execute them in your report.
Check that the reproducer works if you run it manually.
Syzkaller tries to simplify the reproducer, but the result might not be ideal.
You can try to simplify or annotate the reproducer manually, that greatly helps kernel developers to figure out why the bug occurs.
If you want to get extra credit, you can try to understand the bug and develop a fix yourself.
If you can't figure out the right fix, but have some understanding of the bug, please add your thoughts and conclusions to the report, that will save some time for kernel developers.
Reporting security bugs
If you believe that a found bug poses potential security threat, consider following the instructions below.
Note, that these instructions are a work-in-progress and based on my current understanding of the disclosure process.
This instruction is now being discussed here.
If you don't want to deal with this complex disclosure process you can either:
- Report the bug privately to
[email protected]. In this case it should be fixed in the upstream kernel, but there are no guarantees that the fix will be propagated to stable or distro kernels. The maximum embargo on this list is 7 days. - Report the bug privately to a vendor such as Red Hat (
[email protected]) or SUSE ([email protected]). They should fix the bug, assign a CVE, and notify other vendors. The maximum embargo on these lists is 5 weeks. - Report the bug publicly to
[email protected].
If you want to deal with the disclosure yourself, read below.
The three main mailing lists for reporting and disclosing Linux kernel security issues are [email protected], [email protected] and [email protected].
The links for the guidelines for these lists are below, please read them carefully before sending anything to these lists.
[email protected]- https://www.kernel.org/doc/html/latest/admin-guide/security-bugs.html[email protected]- http://oss-security.openwall.org/wiki/mailing-lists/distros[email protected]- http://oss-security.openwall.org/wiki/mailing-lists/oss-security
Reporting minor security bugs
To report minor security bugs (such as local DOS or local info leak):
- Report the bug publicly to kernel developers as described above and wait until a fix is committed. Alternatively, you can develop and send a fix yourself.
- Request a CVE from MITRE through the web form. Describe the bug details and add a link to the fix (from
patchwork.kernel.org,git.kernel.orgorgithub.com) in the request. - Once a CVE is assigned, send the bug details, the CVE number and a link to the fix to
[email protected].
Reporting major security bugs
To report major security bugs (such as LPE, remote DOS, remote info leak or RCE):
- Understand the bug and develop a patch with a fix if possible. Optionally develop a proof-of-concept exploit.
- Notify
[email protected]:- Describe vulnerability details, include the proposed patch and optionally the exploit.
- Ask for 7 days of embargo.
- Work on the patch together with the
[email protected]members.
- Notify
[email protected]:- Describe vulnerability details, include the proposed patch and optionally the exploit.
- Ask them to assign a CVE number.
- Ask for 7 days of embargo.
- Wait 7 days for linux distros to apply the patch.
- Ask
[email protected]to make the CVE description public and roll out the updated kernels. - Send the fix upstream:
- Mention the CVE number in the commit message.
- Mention syzkaller in the commit message.
- Notify
[email protected]:- Describe vulnerability details, include a link to the committed patch.
- Wait 1-3 days for people to update their kernels.
- Optionally publish the exploit on
[email protected].
A few notes:
- There should ideally be no delay between reports to
[email protected]and[email protected]. - When working on the patch together with the
[email protected]members and upstream maintainers, keep the linux-distros aware of the progress. - There should ideally be no delay between CVE description publication, distros' updates, upstream commit and notification to
[email protected]. All of these should be on the same day, at worst. - The moment the issue is made public (e.g. patch is submitted upstream, CVE description published, etc.) it must be reported to
[email protected]right away.
A good example of an LPE announcement structure on [email protected] can be found here, however the timeline doesn't look right there: public announcement should have occurred right after the patch was submitted to netdev.
Linux Setup (`docs/linux/setup.md`)
How to set up syzkaller
Generic instructions on how to set up Linux kernel fuzzing with syzkaller are below.
Instructions for a particular VM type or kernel architecture can be found on these pages:
- Setup: Ubuntu host, QEMU vm, x86-64 kernel
- Setup: Linux host, QEMU vm, arm64 kernel
- Setup: Linux host, QEMU vm, arm kernel
- Setup: Linux host, QEMU vm, loongarch64 kernel
- Setup: Linux host, QEMU vm, riscv64 kernel
- Setup: Linux host, QEMU vm, s390x kernel
- Setup: Linux host, Android device, arm32/64 kernel
- Setup: Linux host, Android virtual device, x86-64 kernel
- Setup: Linux isolated host
- Setup: Ubuntu host, VMware vm, x86-64 kernel
- Setup: Ubuntu host, VirtualBox vm, x86-64 kernel
Install
The following components are needed to use syzkaller:
- Go compiler and syzkaller itself
- C compiler with coverage support
- Linux kernel with coverage additions
- Virtual machine or a physical device
If you encounter any troubles, check the troubleshooting page.
Go and syzkaller
syzkaller is written in Go, and Go 1.23+ toolchain is required for build.
Generally we aim at supporting 2 latest releases of Go.
The toolchain can be installed with:
wget https://dl.google.com/go/go1.23.6.linux-amd64.tar.gz
tar -xf go1.23.6.linux-amd64.tar.gz
export GOROOT=`pwd`/go
export PATH=$GOROOT/bin:$PATHSee Go: Download and install for other options.
To download and build syzkaller:
git clone https://github.com/google/syzkaller
cd syzkaller
makeAs the result compiled binaries should appear in the bin/ dir.
Note: if you want to do cross-OS/arch testing, you need to specify TARGETOS,TARGETVMARCH and TARGETARCH arguments to make. See the Makefile for details.
Environment
You might need to properly setup binutils if you're fuzzing in a cross-arch environment as described here.
C Compiler
Syzkaller is a coverage-guided fuzzer and therefore it needs the kernel to be built with coverage support, which requires a recent GCC version.
Coverage support was submitted to GCC, released in GCC 6.1.0 or later.
Make sure that your GCC meets this requirement, or get a GCC that syzbot uses here.
Linux Kernel
Besides coverage support in GCC, you also need support for it on the kernel side.
KCOV was added into mainline Linux kernel in version 4.6 and is be enabled by CONFIG_KCOV=y kernel configuration option.
For older kernels you need to at least backport commit kernel: add kcov code coverage.
Besides that, it's recommended to backport all kernel patches that touch kernel/kcov.c.
To enable more syzkaller features and improve bug detection abilities, it's recommended to use additional config options.
See this page for details.
VM Setup
Syzkaller performs kernel fuzzing on worker virtual machines or physical devices.
These worker environments are referred to as VMs.
Out-of-the-box syzkaller supports QEMU, kvmtool and GCE virtual machines, Android devices and Odroid C2 boards.
These are the generic requirements for a syzkaller VM:
- The fuzzing processes communicate with the outside world, so the VM image needs to include
networking support. - The program files for the fuzzer processes are transmitted into the VM using SSH, so the VM image
needs a running SSH server. - The VM's SSH configuration should be set up to allow root access for the identity that is
included in thesyz-manager's configuration. In other words, you should be able to dossh -i $SSHID -p $PORT root@localhostwithout being prompted for a password (whereSSHIDis the SSH
identification file andPORTis the port that are specified in thesyz-managerconfiguration
file). - The kernel exports coverage information via a debugfs entry, so the VM image needs to mount
the debugfs filesystem at/sys/kernel/debug.
To use QEMU syzkaller VMs you have to install QEMU on your host system, see QEMU docs for details.
The create-image.sh script can be used to create a suitable Linux image.
See the links at the top of the document for instructions on setting up syzkaller for QEMU, Android and some other types of VMs.
Troubleshooting
QEMU requires root for
-enable-kvm.Solution: add your user to the
kvmgroup (sudo usermod -a -G kvmand relogin).QEMU crashes with:
textqemu-system-x86_64: error: failed to set MSR 0x48b to 0x159ff00000000 qemu-system-x86_64: /build/qemu-EmNSP4/qemu-4.2/target/i386/kvm.c:2947: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.Solution: remove
-cpu host,migratable=offfrom the QEMU command line. The easiest way to do that is to setqemu_argsto-enable-kvmin thesyz-managerconfig file.
Linux Setup Linux Host Android Device Arm Kernel (`docs/linux/setup_linux-host_android-device_arm-kernel.md`)
Setup: Linux host, Android device, arm32/64 kernel
Note: fuzzing the kernel on a real Android device may brick it.
This document details the steps involved in setting up a syzkaller instance fuzzing an arm32/64 linux kernel on an Android device.
Some features of syzkaller may not yet work properly on arm32. For example, not all debugging and test coverage features are available in the Linux kernel for arm32, limiting the efficacy of syskaller in finding bugs fast.
These were tested on an NXP Pico-Pi-IMX7D following the instructions here.
You may find additional details in syzkaller's adb vm implementation here.
Device setup
Follow the instructions for your board to install Android and make sure the device boots properly.
Set up the adb bridge so that adb and fastboot work.
Set up a serial port, following the instructions for your device so that you can monitor kernel log messages. On Android-based boards the serial port is typically exposed as a USB (or some custom) port, or over GPIO pins. On phones you can use Android Serial Cable or Suzy-Q. syzkaller can work without a dedicated serial port as well (by falling back to adb shell dmesg -w), but that is unreliable and turns lots of crashes into "lost connection to test machine" crashes with no additional info.
Get the proper compiler toolchain for your device.
Recompile and reinstall the Linux kernel with debugging kernel options available on your board. You might benefit from backporting KCOV or KASAN patches.
Building syzkaller
Get syzkaller as described here.
Then build it for either arm or arm64 target architecture depending on the device you're using.
make TARGETOS=linux TARGETARCH=armmake TARGETOS=linux TARGETARCH=arm64In case you have old Android /dev/ion driver (kernel <= 3.18) before building syzkaller copy old /dev/ion descriptions:
cp sys/android/* sys/linuxManager config
Create a manager config android.cfg:
{
"target": "linux/arm",
"http": "127.0.0.1:56741",
"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir",
"kernel_obj": "$KERNEL",
"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
"cover": true,
"type": "adb",
"vm": {
"devices": [$DEVICES],
"battery_check": true,
"boot_service": "systemui"
}
}Replace the variables $GOPATH, $KERNEL (path to kernel build directory), and $DEVICES (the device ID for your board as reported by adb devices, e.g. ABCD000010) with their actual values.
For arm64 use "target": "linux/arm64".
If your kernel doesn't support coverage collection (e.g. arm32 without KCOV patches) set "cover": false.
Turn off battery_check if your device doesn't have battery service, see the comment here for details.
The boot_service parameter specifies which service to wait for during boot completion. The default value is "systemui", which waits for the com.android.systemui process. For AOSP builds without SystemUI (e.g., when testing QEMU Android), you can use "servicemanager" or other services instead.
Running syzkaller
Run syzkaller manager:
./bin/syz-manager -config=android.cfgNow syzkaller should be running, you can check manager status with your web browser at 127.0.0.1:56741.
If you get issues after syz-manager starts, consider running it with the -debug flag.
Also see this page for troubleshooting tips and Building a Pixel kernel with KASAN+KCOV or Building a PH-1 kernel with KASAN+KCOV for kernel build/boot instructions.
Linux Setup Linux Host Android Virtual Device X86 64 Kernel (`docs/linux/setup_linux-host_android-virtual-device_x86-64-kernel.md`)
Setup: Linux host, Android virtual device, x86-64 kernel
This document details the steps involved in setting up a syzkaller instance fuzzing an x86-64 linux kernel on an Android virtual device.
In the instructions below, the $VAR notation (e.g. $GSI, $GKI, etc.) is used to denote paths to directories that are either created when executing the instructions, or that you have to create yourself before running the instructions. Substitute the values for those variables manually.
Note:
- All commands below assume root privileges.
- It is recommended to have at least 64 GB of RAM and 500 GB of free disk space.
Install prerequisites
Command:
apt update
apt install sudo git wget curl repo libncurses5 vim gcc make bison bc zip rsync language-pack-en-baseCuttlefish
It is recommended to use Cuttlefish to emulate Android devices. Build and install it from source (v1.16.0 as an example):
Command:
apt install git devscripts equivs config-package-dev debhelper-compat golang curl
git clone -b v1.16.0 https://github.com/google/android-cuttlefish
cd android-cuttlefish
tools/buildutils/build_packages.sh
dpkg -i ./cuttlefish-base_*_*64.deb || sudo apt-get install -y -f
dpkg -i ./cuttlefish-user_*_*64.deb || sudo apt-get install -y -f
usermod -aG kvm,cvdnetwork,render root
rebootGeneric System Images (GSI)
Checkout GSI source
The GSI source checkout is close to 90 GB, and the build can take up about 300 GB of disk space.
Command:
mkdir android13-gsi
cd android13-gsi
repo init -u https://android.googlesource.com/platform/manifest -b android13-gsi
repo sync -cBuild GSI
Refresh the build environment and select the build target:
Command:
source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebugThe output should be as follows (may vary depending on the host):
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=13
TARGET_PRODUCT=aosp_cf_x86_64_phone
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_ARCH=x86_64
TARGET_ARCH_VARIANT=silvermont
TARGET_2ND_ARCH=x86
TARGET_2ND_ARCH_VARIANT=silvermont
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-6.8.0-65-generic-x86_64-Ubuntu-22.04.4-LTS
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=TP1A.220624.019
OUT_DIR=out
PRODUCT_SOONG_NAMESPACES=device/generic/goldfish-opengl device/generic/goldfish device/generic/goldfish-opengl hardware/google/camera hardware/google/camera/devices/EmulatedCamera device/google/cuttlefish/apex/com.google.cf.wifi_hwsim external/mesa3d vendor/google_devices/common/proprietary/confirmatioui_hal
============================================Start building:
Command:
mYou can test your setup by launching the virtual device:
Command:
launch_cvdOpen http://localhost:8443 in your browser, you should see a virtual device. Click Connect to interact with it as you would with a real phone. Press Ctrl-C in the terminal to stop the simulator.
Kernel
Checkout Android Generic Kernel Image (GKI) source
Command:
mkdir common-android13-5.15
cd common-android13-5.15
repo init -u https://android.googlesource.com/kernel/manifest -b common-android13-5.15
repo sync -cBuild GKI
We need to build the Android Kernel with KASAN and KCOV so that syzkaller can get coverage and bug information during fuzzing.
Command:
BUILD_CONFIG=common/build.config.gki_kasan.x86_64 build/build.shBuild vendor modules with KASAN and KCOV:
Command:
BUILD_CONFIG=common-modules/virtual-device/build.config.virtual_device_kasan.x86_64 build/build.shsyzkaller
Build syzkaller
Build syzkaller as described here.
Then create a manager config like the following, replacing the environment
variables $GOPATH and $GKI with their actual values.
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir/android/out",
"kernel_obj": "$GKI/out/android13-5.15/dist",
"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
"cover": true,
"type": "adb",
"vm": {
"devices": ["0.0.0.0:6520"],
"battery_check": true,
"boot_service": "systemui"
}
}The boot_service parameter specifies which service to wait for during boot completion. The default value is "systemui", which waits for the com.android.systemui process. For AOSP builds without SystemUI, you can use "servicemanager" or other services instead. This is particularly useful when testing minimal Android configurations in QEMU.
Launch the virtual device
Launch the Android system with the KASAN and KCOV kernel.
Command:
cd $GSI
source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
launch_cvd -daemon -kernel_path=$GKI/out/android13-5.15/dist/bzImage -initramfs_path=$GKI/out/android13-5.15/dist/initramfs.imgConnect to the virtual device with adb:
Command:
adb connect 0.0.0.0:6520List available virtual devices:
Command:
adb devicesRun syzkaller
Run syzkaller manager:
Command:
cd $GOPATH/src/github.com/google/syzkaller
./bin/syz-manager -config=android.cfgNow syzkaller should be running, you can check manager status with your web browser at 127.0.0.1:56741.
If you get issues after syz-manager starts, consider running it with the -debug flag.
Here are some useful links:
- github - google/android-cuttlefish
- AOSP - Cuttlefish virtual Android devices
- AOSP - Cuttlefish: Get started
- AOSP - Download the Android source
- AOSP - Build Android
- AOSP - Generic system images
- AOSP - Architecture overview
- AOSP - Build kernels
- AOSP - Kernel branches and their build systems
Linux Setup Linux Host Isolated (`docs/linux/setup_linux-host_isolated.md`)
Setup: Linux isolated host
These are the instructions on how to fuzz the kernel on isolated machines.
Isolated machines are separated in a way that limits remote management. They can
be interesting to fuzz due to specific hardware setups.
This syzkaller configuration uses only ssh to launch and monitor an isolated
machine.
Setup reverse proxy support
Given only ssh may work, a reverse ssh proxy will be used to allow the fuzzing
instance and the manager to communicate.
Ensure the sshd configuration on the target machine has AllowTcpForwarding to yes.
machine:~# grep Forwarding /etc/ssh/sshd_config
AllowTcpForwarding yesKernel
The isolated VM does not deploy kernel images so ensure the kernel on the target
machine is build with these options:
CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=yCode coverage works better when KASLR Is disabled too:
# CONFIG_RANDOMIZE_BASE is not setOptional: Reuse existing ssh connection
In most scenarios, you should use an ssh key to connect to the target machine.
The isolated configuration supports ssh keys as described in the generic
setup.
If you cannot use an ssh key, you should configure your manager machine to reuse
existing ssh connections.
Add these lines to your ~/.ssh/config file:
Host *
ControlMaster auto
ControlPath ~/.ssh/control:%h:%p:%rBefore fuzzing, connect to the machine and keep the connection open so all scp
and ssh usage will reuse it.
Optional: Pstore support
If the device under test (DUT) has Pstore support, it is possible to configure syzkaller to
fetch crashlogs from /sys/fs/pstore. You can do this by setting "pstore": true within
the vm section of the syzkaller configuration file.
Optional: Startup script
To execute commands on the DUT before fuzzing (re-)starts,startup_script can be used.
Syzkaller
Build syzkaller as described here.
Use the following config:
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"rpc": "127.0.0.1:0",
"sshkey" : "/path/to/optional/sshkey",
"workdir": "/syzkaller/workdir",
"kernel_obj": "/linux-next",
"syzkaller": "/go/src/github.com/google/syzkaller",
"sandbox": "setuid",
"type": "isolated",
"vm": {
"targets" : [ "10.0.0.1" ],
"pstore": false,
"target_dir" : "/home/user/tmp/syzkaller",
"target_reboot" : false
}
}Don't forget to update:
target(target OS/arch)workdir(path to the workdir)kernel_obj(path to kernel build directory)sshkeyYou can setup an sshkey (optional)vm.targetsList of hosts to use for fuzzingvm.target_dirWorking directory on the target hostvm.target_rebootReboot the machine if remote process hang (useful for wide fuzzing, false by default)
Run syzkaller manager:
./bin/syz-manager -config=my.cfgIf you get issues after syz-manager starts, consider running it with the -debug flag.
Also see this page for troubleshooting tips.
Linux Setup Linux Host Qemu Vm Arm Kernel (`docs/linux/setup_linux-host_qemu-vm_arm-kernel.md`)
Setup: Debian host, QEMU vm, arm kernel
GCC
Obtain a fresh arm-linux-gnueabihf-gcc. Latest Debian distributions provide
version 7.2.0, which should be enough. Otherwise you can download Linaro
compiler here.
Kernel
The instructions are tested with v4.16.1. Check that you have/backport
"arm: port KCOV to arm"
patch. Create kernel config with:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- vexpress_defconfigThen enable the following configs on top:
CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_NAMESPACES=y
CONFIG_USER_NS=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=yAlso check out general kernel configuration recommendations.
Then build kernel with:
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-Image
We will use buildroot to create the disk image. You can obtain buildroot
here. Instructions were tested
with buildroot c665c7c9cd6646b135cdd9aa7036809f7771ab80. First run:
make qemu_arm_vexpress_defconfig
make menuconfigChoose the following options:
Target packages
Networking applications
[*] dhcpcd
[*] iproute2
[*] openssh
Filesystem images
exact size - 1gUnselect:
Kernel
Linux KernelRun make.
Then add the following line to output/target/etc/fstab:
debugfs /sys/kernel/debug debugfs defaults 0 0Then replace output/target/etc/ssh/sshd_config with the following contents:
PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yesRun make again.
Test kernel and image
Run:
qemu-system-arm -m 512 -smp 2 -net nic -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -display none -serial stdio -machine vexpress-a15 -dtb /linux/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb -sd /buildroot/output/images/rootfs.ext2 -snapshot -kernel /linux/arch/arm/boot/zImage -append "earlyprintk=serial console=ttyAMA0 root=/dev/mmcblk0"This should boot the kernel. Wait for login prompt, then in another console run:
ssh -p 10022 root@localhostssh should succeed.
syzkaller
Build syzkaller as described here, with arm target:
make TARGETOS=linux TARGETARCH=armCreate manager config arm.cfg similar to the following one (changing paths as necessary):
{
"name": "arm",
"target": "linux/arm",
"http": ":12345",
"workdir": "/workdir",
"kernel_obj": "/linux",
"syzkaller": "/gopath/src/github.com/google/syzkaller",
"image": "/buildroot/output/images/rootfs.ext2",
"sandbox": "none",
"reproduce": false,
"procs": 4,
"type": "qemu",
"vm": {
"count": 10,
"qemu_args": "-machine vexpress-a15 -dtb /linux/arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb",
"cmdline": "console=ttyAMA0 root=/dev/mmcblk0",
"kernel": "/linux/arch/arm/boot/zImage",
"image_device": "sd",
"mem": 512,
"cpu": 2
}
}Finally, run bin/syz-manager -config arm.cfg.
Linux Setup Linux Host Qemu Vm Arm64 Kernel (`docs/linux/setup_linux-host_qemu-vm_arm64-kernel.md`)
Setup: Linux host, QEMU vm, arm64 kernel
This document will detail the steps involved in setting up a Syzkaller instance fuzzing any ARM64 linux kernel of your choice.
Create a disk image
We will use buildroot to create the disk image.
You can obtain buildroot from here.
Extract the tarball and perform a make menuconfig inside it.
Choose the following options.
Target options
Target Architecture - Aarch64 (little endian)
Toolchain type
External toolchain - Linaro AArch64
System Configuration
[*] Enable root login with password
( ) Root password = set your password using this option
[*] Run a getty (login prompt) after boot --->
TTY port - ttyAMA0
Target packages
[*] Show packages that are also provided by busybox
Networking applications
[*] dhcpcd
[*] iproute2
[*] openssh
Filesystem images
[*] ext2/3/4 root filesystem
ext2/3/4 variant - ext3
exact size in blocks - 6000000
[*] tar the root filesystemRun make. After the build, confirm that output/images/rootfs.ext3 exists.
If you're experiencing a very slow sshd start up time with arm64 qemu running on x86, the reason is probably low entropy and it be "fixed" with installing haveged. It can be found in the buildroot menuconfig:
Target packages
Miscellaneous
[*] havegedGet the ARM64 toolchain from Linaro
You will require an ARM64 kernel with gcc plugin support.
If not, obtain the ARM64 toolchain from Linaro.
Get gcc-linaro-6.1.1-2016.08-x86_64_aarch64-linux-gnu.tar.xz from here.
Extract and add its bin/ to your PATH.
If you have another ARM64 toolchain on your machine, ensure that this newly downloaded toolchain takes precedence.
Compile the kernel
Get the source code of the Linux kernel version you want to fuzz, and do the following.
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
$ vim .configChange the following options :
CONFIG_KCOV=y
CONFIG_KASAN=y
CONFIG_DEBUG_INFO=y
CONFIG_CMDLINE="console=ttyAMA0"
CONFIG_KCOV_INSTRUMENT_ALL=y
CONFIG_DEBUG_FS=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_CROSS_COMPILE="aarch64-linux-gnu-"$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j40If the build was successful, you should have a arch/arm64/boot/Image file.
Obtain qemu for ARM64
Obtain the QEMU source from git or from the latest source release.
$ ./configure
$ make -j40If the build was successful, you should have a aarch64-softmmu/qemu-system-aarch64 binary.
Boot up manually
You should be able to start up the kernel as follows.
$ /path/to/aarch64-softmmu/qemu-system-aarch64 \
-machine virt \
-cpu cortex-a57 \
-nographic -smp 1 \
-hda /path/to/rootfs.ext3 \
-kernel /path/to/arch/arm64/boot/Image \
-append "console=ttyAMA0 root=/dev/vda oops=panic panic_on_warn=1 panic=-1 ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ" \
-m 2048 \
-net user,hostfwd=tcp::10023-:22 -net nicAt this point, you should be able to see a login prompt.
Set up the QEMU disk
Now that we have a shell, let us add a few lines to existing init scripts so that they are executed each time Syzkaller brings up the VM.
At the top of /etc/init.d/S50sshd add the following lines:
ifconfig eth0 up
dhcpcd
mount -t debugfs none /sys/kernel/debug
chmod 777 /sys/kernel/debug/kcovComment out the line
/usr/bin/ssh-keygen -ANext we set up ssh. Create an ssh keypair locally and copy the public key to /authorized_keys in /. Ensure that you do not set a passphrase when creating this key.
Open /etc/ssh/sshd_config and modify the following lines as shown below.
PermitRootLogin yes
PubkeyAuthentication yes
AuthorizedKeysFile /authorized_keys
PasswordAuthentication yesReboot the machine, and ensure that you can ssh from host to guest as.
$ ssh -i /path/to/id_rsa root@localhost -p 10023Build syzkaller
Build syzkaller as described here, with arm64 target:
CC=gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
make TARGETARCH=arm64Modify your config file and start off syzkaller
A sample config file that exercises the required options are shown below. Modify according to your needs.
{
"name": "QEMU-aarch64",
"target": "linux/arm64",
"http": ":56700",
"workdir": "/path/to/a/dir/to/store/syzkaller/corpus",
"kernel_obj": "/path/to/linux/build/dir",
"syzkaller": "/path/to/syzkaller/arm64/",
"image": "/path/to/rootfs.ext3",
"sshkey": "/path/to/id_rsa",
"procs": 8,
"type": "qemu",
"vm": {
"count": 1,
"qemu": "/path/to/qemu-system-aarch64",
"cmdline": "console=ttyAMA0 root=/dev/vda",
"kernel": "/path/to/Image",
"cpu": 2,
"mem": 2048
}
}At this point, you should be able to visit localhost:56700 and view the results of the fuzzing.
If you get issues after syz-manager starts, consider running it with the -debug flag.
Also see this page for troubleshooting tips.
Linux Setup Linux Host Qemu Vm Loongarch64 Kernel (`docs/linux/setup_linux-host_qemu-vm_loongarch64-kernel.md`)
Setup: Linux host, QEMU vm, loongarch64 kernel
This document describes a Linux/QEMU setup for fuzzing a LoongArch64 Linux
kernel with syzkaller.
Toolchain
You need:
- a Go toolchain supported by syzkaller,
loongarch64-linux-gnu-gcc/loongarch64-linux-gnu-g++,qemu-system-loongarch64,- Buildroot or another way to build an SSH-capable root filesystem.
The syzkaller target architecture is loong64, while the external Linux and
toolchain names remain loongarch*.
Use TARGETARCH=loong64 and TARGETVMARCH=loong64 when building syzkaller
target binaries.
Kernel
Build a LoongArch kernel with KCOV and debugfs enabled. The following example
uses a GNU cross toolchain:
make ARCH=loongarch CROSS_COMPILE=loongarch64-linux-gnu- loongson64_defconfig
scripts/config \
--enable KCOV \
--enable KCOV_ENABLE_COMPARISONS \
--enable KCOV_INSTRUMENT_ALL \
--enable DEBUG_FS \
--enable DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
--enable VIRTIO_NET \
--disable RANDOMIZE_BASE
make ARCH=loongarch CROSS_COMPILE=loongarch64-linux-gnu- olddefconfig
make ARCH=loongarch CROSS_COMPILE=loongarch64-linux-gnu- -j$(nproc)The loongson64_defconfig uses 16 KiB pages. Confirm that the final config
contains:
grep '^CONFIG_PAGE_SIZE_16KB=y' .configSee the recommended Linux kernel config options for more
coverage and bug-detection settings.
The QEMU setup below uses the EFI kernel image:
arch/loongarch/boot/vmlinux.efiImage
Buildroot provides a LoongArch virt defconfig:
make O=/buildroot-output qemu_loongarch64_virt_efi_defconfig
make O=/buildroot-output menuconfigChoose the following options:
Target packages
Networking applications
[*] iproute2
[*] openssh
Filesystem images
[*] ext2/3/4 root filesystem
ext2/3/4 variant - ext4
exact size - 1gUnselect:
Kernel
Linux KernelBuild the image once to populate /buildroot-output/target:
make O=/buildroot-output -j$(nproc)Create an SSH key without a passphrase:
ssh-keygen -t rsa -N '' -f /image/id_rsa
chmod 0600 /image/id_rsaAdd the following lines to /buildroot-output/target/etc/fstab:
debugfs /sys/kernel/debug debugfs defaults 0 0
securityfs /sys/kernel/security securityfs defaults 0 0Install the public key in the target filesystem:
install -Dm0600 /image/id_rsa.pub \
/buildroot-output/target/root/.ssh/authorized_keysThen replace /buildroot-output/target/etc/ssh/sshd_config with:
PermitRootLogin yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication noRun Buildroot again so these changes are included in the filesystem image:
make O=/buildroot-output -j$(nproc)Use /buildroot-output/images/rootfs.ext2 as the syzkaller image file.
Test kernel and image
Run the kernel manually before starting syzkaller:
qemu-system-loongarch64 \
-machine virt \
-nographic \
-m 2048 \
-smp 2 \
-kernel /linux/arch/loongarch/boot/vmlinux.efi \
-append "root=/dev/vda console=ttyS0 rw nokaslr" \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0 \
-drive file=/buildroot-output/images/rootfs.ext2,if=none,format=raw,id=hd0 \
-device virtio-blk-pci,drive=hd0 \
-netdev user,id=net0,hostfwd=tcp::10022-:22 \
-device virtio-net-pci,netdev=net0Wait for the login prompt, then test SSH from another terminal:
ssh -i /image/id_rsa -p 10022 root@localhostInside the guest, confirm that KCOV is available:
mount -t debugfs none /sys/kernel/debug 2>/dev/null || true
mount -t securityfs none /sys/kernel/security 2>/dev/null || true
ls -l /sys/kernel/debug/kcovsyzkaller
Build syzkaller as described in the generic Linux setup guide:
make TARGETOS=linux TARGETARCH=loong64 TARGETVMARCH=loong64 target
make managerCreate a manager config similar to the following one:
{
"name": "loong64",
"target": "linux/loong64",
"http": ":56700",
"workdir": "/workdir",
"kernel_obj": "/linux",
"syzkaller": "/syzkaller",
"image": "/buildroot-output/images/rootfs.ext2",
"sshkey": "/image/id_rsa",
"procs": 8,
"type": "qemu",
"vm": {
"count": 1,
"qemu": "qemu-system-loongarch64",
"kernel": "/linux/arch/loongarch/boot/vmlinux.efi",
"cmdline": "nokaslr",
"cpu": 2,
"mem": 2048
}
}Finally, run:
bin/syz-manager -config loong64.cfgIf startup fails, re-run with -debug and check the QEMU command line.
Linux Setup Linux Host Qemu Vm Riscv64 Kernel (`docs/linux/setup_linux-host_qemu-vm_riscv64-kernel.md`)
Setup: Debian/Ubuntu host, QEMU vm, riscv64 kernel
GCC
Obtain riscv64-linux-gnu-gcc at least GCC version 8. The latest Debian/Ubuntu distributions should
provide both cross-compilers in a recent enough version in the gcc-riscv64-linux-gnu package.
Alternatively, you can also build your own
RISC-V GNU compiler toolchain from source.
Kernel
The following instructions were tested with Linux Kernel v5.9-rc1. Create a kernel config with:
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- defconfigAlso enable the recommended Kconfig options for syzkaller.
Then build kernel with:
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- -j $(nproc)OpenSBI
Clone the OpenSBI repository and build the bootable OpenSBI image containing the kernel:
git clone https://github.com/riscv/opensbi
cd opensbi
make CROSS_COMPILE=riscv64-linux-gnu- PLATFORM_RISCV_XLEN=64 PLATFORM=genericSee the OpenSBI documentation for booting on the
QEMU RISC-V Virt Machine Platform
for more information.
Image
We will use buildroot to create the disk image. You can obtain buildroot
here. The following instructions
were tested with buildroot version 2020.05. First run:
make qemu_riscv64_virt_defconfig
make menuconfigChoose the following options:
Target packages
Networking applications
[*] iproute2
[*] openssh
Filesystem images
ext2/3/4 variant - ext4
exact size - 1gUnselect:
Kernel
Linux KernelRun make.
Then add the following line to output/target/etc/fstab:
debugfs /sys/kernel/debug debugfs defaults 0 0Then replace output/target/etc/ssh/sshd_config with the following contents:
PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yesRun make again.
QEMU
The following instructions were tested with QEMU 5.0. At least QEMU 4.1 is needed.
Test kernel and image
Run:
qemu-system-riscv64 \
-machine virt \
-nographic \
-bios /opensbi/build/platform/generic/firmware/fw_jump.bin \
-kernel /linux/arch/riscv/boot/Image \
-append "root=/dev/vda ro console=ttyS0" \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-device,rng=rng0 \
-drive file=/buildroot/output/images/rootfs.ext2,if=none,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-netdev user,id=net0,host=10.0.2.10,hostfwd=tcp::10022-:22 \
-device virtio-net-device,netdev=net0This should boot the kernel. Wait for login prompt, then in another console run:
ssh -p 10022 root@localhostssh should succeed.
syzkaller
Build syzkaller as described here, with riscv64 target:
make TARGETOS=linux TARGETARCH=riscv64Create the manager config riscv64.cfg similar to the following one (adjusting paths as necessary):
{
"name": "riscv64",
"target": "linux/riscv64",
"http": ":56700",
"workdir": "/workdir",
"kernel_obj": "/linux",
"syzkaller": "/gopath/src/github.com/google/syzkaller",
"image": "/buildroot/output/images/rootfs.ext2",
"procs": 8,
"type": "qemu",
"vm": {
"count": 1,
"qemu_args": "-machine virt -bios /opensbi/build/platform/generic/firmware/fw_jump.bin",
"kernel": "/linux/arch/riscv/boot/Image",
"cpu": 2,
"mem": 2048
}
}Alternatively, you may try to use the default OpenSBI firmware provided with QEMU 4.1 and newer by
specifying -machine virt -bios default in qemu_args and pass the kernel image in the kernel
config option:
{
"name": "riscv64",
"target": "linux/riscv64",
"http": ":56700",
"workdir": "/workdir",
"kernel_obj": "/linux",
"syzkaller": "/gopath/src/github.com/google/syzkaller",
"image": "/buildroot/output/images/rootfs.ext2",
"procs": 8,
"type": "qemu",
"vm": {
"count": 1,
"qemu_args": "-machine virt -bios default",
"kernel": "/linux/arch/riscv/boot/Image",
"cpu": 2,
"mem": 2048
}
}This would allow to boot a different kernel without having to re-compile OpenSBI. However, on some
distributions the default OpenSBI firmware required by the -bios default option might not be
available yet.
Finally, run bin/syz-manager -config riscv64.cfg. After it successfully starts, you should be able
to visit localhost:56700 to view the fuzzing results.
In case you encounter issues with starting syz-manager, use the -debug flag and refer to the
troubleshooting guide.
Linux Setup Linux Host Qemu Vm S390x Kernel (`docs/linux/setup_linux-host_qemu-vm_s390x-kernel.md`)
Setup: Debian/Ubuntu/Fedora host, QEMU vm, s390x kernel
GCC
Obtain s390x-linux-gnu-gcc at least GCC version 9. The latest Debian/Ubuntu/Fedora distributions
should provide a recent enough version of a cross-compiler in the gcc-s390x-linux-gnu package.
Kernel
Checkout Linux kernel source:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $KERNELGenerate default configs:
cd $KERNEL
make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- defconfig
make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- kvm_guest.configEnable kernel config options required for syzkaller as described here.
./scripts/config --file .config \
-d MODULES \
-e KCOV \
-e KCOV_INSTRUMENT_ALL \
-e KCOV_ENABLE_COMPARISONS \
-e KASAN \
-e KASAN_INLINE \
-e CONFIGFS_FS \
-e SECURITYFS \
-e DEBUG_INFO \
-e GDB_SCRIPTS \
-e PRINTK \
-e EARLY_PRINTK \
-e DEVTMPFS \
-e TUN \
-e VIRTIO_PCI \
-e VIRTIO_NET \
-e NET_9P_VIRTIO \
-e NET_9P \
-e 9P_FS \
-e BINFMT_MISC \
-e FAULT_INJECTION \
-e FAILSLAB \
-e FAIL_PAGE_ALLOC \
-e FAIL_MAKE_REQUEST \
-e FAIL_IO_TIMEOUT \
-e FAIL_FUTEX \
-e FAULT_INJECTION_DEBUG_FS \
-e FAULT_INJECTION_STACKTRACE_FILTER \
-e DEBUG_KMEMLEAKEdit .config file manually and enable them (or do that through make menuconfig if you prefer).
Since enabling these options results in more sub options being available, we need to regenerate config:
make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- olddefconfigBuild the kernel:
make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- -j$(nproc)Now you should have vmlinux (kernel binary) and bzImage (packed kernel image):
$ ls $KERNEL/vmlinux
$KERNEL/vmlinux
$ ls $KERNEL/arch/s390/boot/bzImage
$KERNEL/arch/s390/boot/bzImageImage
Debian
To create a Debian Linux image with the minimal set of required packages do:
cd $IMAGE/
wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
chmod +x create-image.sh
./create-image.sh -a s390xThe result should be $IMAGE/trixie.img disk image.
For additional options of create-image.sh, please refer to ./create-image.sh -h
QEMU
Debian
Run:
qemu-system-s390x \
-M s390-ccw-virtio -cpu max,zpci=on -m 4G -smp 2 \
-kernel $KERNEL/arch/s390/boot/bzImage \
-drive file=$IMAGE/trixie.img,if=virtio,format=raw \
-append "rootwait root=/dev/vda net.ifnames=0 biosdevname=0" \
-net nic,model=virtio -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
-display none -serial mon:stdio \
-pidfile vm.pid 2>&1 | tee vm.logAfter that you should be able to ssh to QEMU instance in another terminal:
ssh -i $IMAGE/trixie.id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhostIf this fails with "too many tries", ssh may be passing default keys before
the one explicitly passed with -i. Append option -o "IdentitiesOnly yes".
To kill the running QEMU instance press Ctrl+A and then X or run:
kill $(cat vm.pid)If QEMU works, the kernel boots and ssh succeeds, you can shutdown QEMU and try to run syzkaller.
syzkaller
Build syzkaller as described here, with s390x target:
make TARGETOS=linux TARGETARCH=s390xThen create a manager config like the following, replacing the environment
variables $GOPATH, $KERNEL and $IMAGE with their actual values.
{
"target": "linux/s390x",
"http": "127.0.0.1:56741",
"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir",
"kernel_obj": "$KERNEL",
"image": "$IMAGE/trixie.img",
"sshkey": "$IMAGE/trixie.id_rsa",
"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
"procs": 8,
"type": "qemu",
"vm": {
"count": 4,
"kernel": "$KERNEL/arch/s390/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}Run syzkaller manager:
mkdir workdir
./bin/syz-manager -config=my.cfgNow syzkaller should be running, you can check manager status with your web browser at 127.0.0.1:56741.
If you get issues after syz-manager starts, consider running it with the -debug flag.
Also see this page for troubleshooting tips.
Linux Setup Ubuntu Host Qemu Vm X86 64 Kernel (`docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md`)
Setup: Ubuntu host, QEMU vm, x86-64 kernel
These are the instructions on how to fuzz the x86-64 kernel in a QEMU with Ubuntu on the host machine and Debian Trixie in the QEMU instances.
In the instructions below, the $VAR notation (e.g. $GCC, $KERNEL, etc.) is used to denote paths to directories that are either created when executing the instructions (e.g. when unpacking GCC archive, a directory will be created), or that you have to create yourself before running the instructions. Substitute the values for those variables manually.
Install Prerequisites
Command:
sudo apt update
sudo apt install make gcc flex bison libncurses-dev libelf-dev libssl-devGCC
If your distro's GCC is older, it's preferable to get the latest GCC from this list. Download and unpack into $GCC, and you should have GCC binaries in $GCC/bin/
Ubuntu 20.04 LTS: You can ignore this section. GCC is up-to-date.
Command:
ls $GCC/bin/
# Sample output:
# cpp gcc-ranlib x86_64-pc-linux-gnu-gcc x86_64-pc-linux-gnu-gcc-ranlib
# gcc gcov x86_64-pc-linux-gnu-gcc-9.0.0
# gcc-ar gcov-dump x86_64-pc-linux-gnu-gcc-ar
# gcc-nm gcov-tool x86_64-pc-linux-gnu-gcc-nmKernel
Checkout Linux Kernel source
Command:
git clone --branch v6.2 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $KERNELWe recommend to start with the latest stable version. v6.2 is an example here.
Generate default configs
Command:
cd $KERNEL
make defconfig
make kvm_guest.configOr if you want to specify a compiler.
Command:
cd $KERNEL
make CC="$GCC/bin/gcc" defconfig
make CC="$GCC/bin/gcc" kvm_guest.configEnable required config options
Enable kernel config options required for syzkaller as described here.
It's not required to enable all of them, but at the very least you need:
# Coverage collection.
CONFIG_KCOV=y
# Debug info for symbolization.
CONFIG_DEBUG_INFO_DWARF4=y
# Memory bug detector
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
# Required for Debian Stretch and later
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=yEdit .config file manually and enable them (or do that through make menuconfig if you prefer).
Since enabling these options results in more sub options being available, we need to regenerate config:
Command:
make olddefconfigOr if you want to specify a compiler.
Command:
make CC="$GCC/bin/gcc" olddefconfigYou might also be interested in disabling the Predictable Network Interface Names mechanism. This can be disabled either in the syzkaller configuration (see details here) or by updating these kernel configuration parameters:
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="net.ifnames=0"Build the Kernel
Command:
make -j`nproc`Or if you want to specify a compiler.
Command:
make CC="$GCC/bin/gcc" -j`nproc`Now you should have vmlinux (kernel binary) and bzImage (packed kernel image):
Command:
ls $KERNEL/vmlinux
# sample output - $KERNEL/vmlinux
ls $KERNEL/arch/x86/boot/bzImage
# sample output - $KERNEL/arch/x86/boot/bzImageImage
Install debootstrap
Command:
sudo apt install debootstrapCreate Debian Trixie Linux image
Create a Debian Trixie Linux image with the minimal set of required packages.
Command:
mkdir $IMAGE
cd $IMAGE/
wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
chmod +x create-image.sh
./create-image.shThe result should be $IMAGE/trixie.img disk image.
OR Create Debian Linux image with a different version
To create a Debian image with a different version (e.g. bookworm, bullseye, sid), specify the --distribution option.
Command:
./create-image.sh --distribution bookwormImage extra tools
Sometimes it's useful to have some additional packages and tools available in the VM even though they are not required to run syzkaller. To install a set of tools we find useful do (feel free to edit the list of tools in the script):
Command:
./create-image.sh --feature fullTo install perf (not required to run syzkaller; requires $KERNEL to point to the kernel sources):
Command:
./create-image.sh --add-perfFor additional options of create-image.sh, please refer to ./create-image.sh -h
QEMU
Install QEMU
Command:
sudo apt install qemu-system-x86Verify
Make sure the kernel boots and sshd starts.
Command:
qemu-system-x86_64 \
-m 2G \
-smp 2 \
-kernel $KERNEL/arch/x86/boot/bzImage \
-append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=$IMAGE/trixie.img,format=raw \
-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
-net nic,model=e1000 \
-enable-kvm \
-nographic \
-pidfile vm.pid \
2>&1 | tee vm.logearly console in setup code
early console in extract_kernel
input_data: 0x0000000005d9e276
input_len: 0x0000000001da5af3
output: 0x0000000001000000
output_len: 0x00000000058799f8
kernel_total_size: 0x0000000006b63000
Decompressing Linux... Parsing ELF... done.
Booting the kernel.
[ 0.000000] Linux version 4.12.0-rc3+ ...
[ 0.000000] Command line: console=ttyS0 root=/dev/sda debug earlyprintk=serial
...
[ ok ] Starting enhanced syslogd: rsyslogd.
[ ok ] Starting periodic command scheduler: cron.
[ ok ] Starting OpenBSD Secure Shell server: sshd.After that you should be able to ssh to QEMU instance in another terminal.
Command:
ssh -i $IMAGE/trixie.id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhostTroubleshooting
If this fails with "too many tries", ssh may be passing default keys before
the one explicitly passed with -i. Append option -o "IdentitiesOnly yes".
To kill the running QEMU instance press Ctrl+A and then X or run:
Command:
kill $(cat vm.pid)If QEMU works, the kernel boots and ssh succeeds, you can shutdown QEMU and try to run syzkaller.
syzkaller
Build syzkaller as described here.
Then create a manager config like the following, replacing the environment
variables $GOPATH, $KERNEL and $IMAGE with their actual values.
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir",
"kernel_obj": "$KERNEL",
"image": "$IMAGE/trixie.img",
"sshkey": "$IMAGE/trixie.id_rsa",
"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
"procs": 8,
"type": "qemu",
"vm": {
"count": 4,
"kernel": "$KERNEL/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}Run syzkaller manager:
mkdir workdir
./bin/syz-manager -config=my.cfgNow syzkaller should be running, you can check manager status with your web browser at 127.0.0.1:56741.
If you get issues after syz-manager starts, consider running it with the -debug flag.
Also see this page for troubleshooting tips.
Linux Setup Ubuntu Host Virtualbox Vm X86 64 Kernel (`docs/linux/setup_ubuntu-host_virtualbox-vm_x86-64-kernel.md`)
Setup: Ubuntu host, VirtualBox vm, x86-64 kernel
These are the instructions on how to fuzz the x86-64 kernel in VirtualBox with Ubuntu on the host machine and Debian Trixie in the virtual machines.
In the instructions below, the $VAR notation (e.g. $GCC, $KERNEL, etc.) is used to denote paths to directories that are either created when executing the instructions (e.g. when unpacking GCC archive, a directory will be created), or that you have to create yourself before running the instructions. Substitute the values for those variables manually.
GCC and Kernel
You can follow the same instructions for obtaining GCC and building the Linux kernel as when using QEMU.
Image
Install debootstrap:
sudo apt-get install debootstrapTo create a Debian Trixie Linux user space in the $USERSPACE dir do:
sudo mkdir -p $USERSPACE
sudo debootstrap --include=openssh-server,curl,tar,gcc,libc6-dev,time,strace,sudo,less,psmisc,selinux-utils,policycoreutils,checkpolicy,selinux-policy-default,firmware-atheros,open-vm-tools --components=main,contrib,non-free trixie $USERSPACENote: it is important to include the open-vm-tools package in the user space as it provides better VM management.
To create a Debian Trixie Linux VMDK do:
wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-gce-image.sh -O create-gce-image.sh
chmod +x create-gce-image.sh
./create-gce-image.sh $USERSPACE $KERNEL/arch/x86/boot/bzImage
qemu-img convert -f raw -O vdi disk.raw disk.vdiThe result should be disk.vdi for the disk image. You can delete disk.raw if you want.
VirtualBox
Open VirtualBox and start the New Virtual Machine Wizard.
Assuming you want to create the new VM in $VMPATH, complete the wizard as follows:
- Create New Virtual Machine
- Virtual Machine Name and Location: select
$VMPATHas location and "debian" as name - Guest OS type: Debian 64-bit
- Disk: select "Use an existing virtual disk"
- Import the
disk.vdifile, and select the imported.vdifile as an Hard Disk File.
When you complete the wizard, you should have $VMPATH/debian.vbox. From this point onward, you no longer need the VirtualBox UI.
To test the fuzzing environment before getting started, follow the instructions below:
Forwarding port 2222 on your host machine to port 22:
VBoxManage modifyvm debian --natpf1 "test,tcp,,2222,,22"Starting the Debian VM (headless):
VBoxManage startvm debian --type headlessSSH into the VM:
ssh -p 2222 [email protected]Stopping the VM:
VBoxManage controlvm debian poweroffIf all of the above VBoxManage commands work, then you can proceed to running syzkaller.
syzkaller
Create a manager config like the following, replacing the environment variables $GOPATH, $KERNEL and $VMPATH with their actual values.
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "$GOPATH/src/github.com/google/syzkaller/workdir",
"kernel_obj": "$KERNEL",
"sshkey": "$IMAGE/key",
"syzkaller": "$GOPATH/src/github.com/google/syzkaller",
"procs": 8,
"type": "virtualbox",
"vm": {
"count": 4,
"base_vm_name": "debian"
}
}Run syzkaller manager:
mkdir workdir
./bin/syz-manager -config=my.cfgSyzkaller will create full clone VMs from the debian VM