# Technical Documentation: google/syzkaller > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/google/syzkaller) ### Overview (`README.md`) # syzkaller - kernel fuzzer [](https://github.com/google/syzkaller/actions?query=workflow/ci) [](https://bugs.chromium.org/p/oss-fuzz/issues/list?q=label:Proj-syzkaller) [](https://goreportcard.com/report/github.com/google/syzkaller) [](https://codecov.io/gh/google/syzkaller) [](https://godoc.org/github.com/google/syzkaller) [](LICENSE) `syzkaller` (`[siːzˈkɔːlə]`) is an unsupervised coverage-guided kernel fuzzer.\ Supported OSes: `FreeBSD`, `Fuchsia`, `gVisor`, `Linux`, `NetBSD`, `OpenBSD`, `Windows`. Mailing list: [syzkaller@googlegroups.com](https://groups.google.com/forum/#!forum/syzkaller) (join on [web](https://groups.google.com/forum/#!forum/syzkaller) or by [email](mailto:syzkaller+subscribe@googlegroups.com)). Found bugs: [Darwin/XNU](docs/darwin/README.md), [FreeBSD](docs/freebsd/found_bugs.md), [Linux](docs/linux/found_bugs.md), [NetBSD](docs/netbsd/found_bugs.md), [OpenBSD](docs/openbsd/found_bugs.md), [Windows](docs/windows/README.md). ## Documentation Initially, syzkaller was developed with Linux kernel fuzzing in mind, but now it's being extended to support other OS kernels as well. Most of the documentation at this moment is related to the [Linux](docs/linux/setup.md) kernel. For other OS kernels check: [Darwin/XNU](docs/darwin/README.md), [FreeBSD](docs/freebsd/README.md), [Fuchsia](docs/fuchsia/README.md), [NetBSD](docs/netbsd/README.md), [OpenBSD](docs/openbsd/setup.md), [Starnix](docs/starnix/README.md), [Windows](docs/windows/README.md), [gVisor](docs/gvisor/README.md), [Akaros](docs/akaros/README.md). - [How to install syzkaller](docs/setup.md) - [How to use syzkaller](docs/usage.md) - [How syzkaller works](docs/internals.md) - [How to install syzbot](docs/setup_syzbot.md) - [How to contribute to syzkaller](docs/contributing.md) - [How to report Linux kernel bugs](docs/linux/reporting_kernel_bugs.md) - [Tech talks and articles](docs/talks.md) - [Research work based on syzkaller](docs/research.md) ## Disclaimer This is not an official Google product. --- ### Overview (`syz-agent/README.md`) # syz-agent `syz-agent` is an AI agent running as part of the continuous fuzzing infrastructure. It continuously polls the syzbot web dashboard for tasks and executes agentic workflows. ## Running Locally with Docker You can easily run `syz-agent` locally without Kubernetes. This is the best approach for testing and development. 1. Build the Docker image from the root of the syzkaller repository: ```bash make -C syz-agent container ``` *Note: The default image name is `local/syz-agent:latest`. You can override it by passing `IMAGE_NAME=your-image-name IMAGE_TAG=your-tag`.* 2. Prepare a configuration file (e.g. `config.json`): ```json { "http": ":8080", "dashboard_client": "my-local-agent", "dashboard_addr": "https://syzkaller.appspot.com", "dashboard_key": "YOUR_KEY", "targets": { "linux/amd64": { "image": "/disk-images/buildroot_amd64", "kernel_config": "/kernel-configs/upstream-apparmor-kasan.config", "type": "qemu", "vm": { "cpu": 2, "mem": 2048, "cmdline": "root=/dev/sda1" } } }, "cache_size": 107374182400 } ``` 3. Run the container, mounting your configuration file: ```bash docker run -it --rm \ -p 8080:8080 \ --privileged \ -e GOOGLE_API_KEY=$GOOGLE_API_KEY \ -v $(pwd)/config.json:/etc/syz-agent/config.json:ro \ local/syz-agent:latest \ -name=$MY_UNIQUE_AGENT_NAME \ -config=/etc/syz-agent/config.json ``` *Note: `pkg/updater` is bypassed inside Docker because the `-syzkaller=/syzkaller` flag is passed via `run.sh` which serves as the Dockerfile's ENTRYPOINT. `syz-agent` will use the pre-built binaries inside the container.* ## Running on Kubernetes `syz-agent` can be deployed as a `StatefulSet` on Kubernetes (Minikube or GKE) for persistent identity. ### Local Development (Minikube) To deploy to a standard minikube cluster: ```bash # 1. Build and load image into Minikube make container minikube image load local/syz-agent:latest # 2. Deploy (automatically binds host's GOOGLE_API_KEY and DASHBOARD_KEY) GOOGLE_API_KEY=xxx DASHBOARD_KEY=yyy make k8s-minikube | kubectl apply -f - ``` --- ### Overview (`syz-cluster/README.md`) `syz-cluster` is a distributed patch series fuzzing that is relying on syzkaller. It's deployed at https://ci.syzbot.org. ## Overview The system is to be deployed on a K8S cluster. The main services are: * `dashboard`: the web interface, read-only. * `controller`: manages the state of the system, provides [API](./pkg/api/client.go) for other components, schedules fuzzing sessions. * `series-tracker`: polls LKML git archives for the new series. * `reporter-server`: generates new reports, provides API for actual reporter implementations. * `email-reporter`: sends reports over email, handles incoming email commands. The actual patch processing is orchestrated by Argo Workflows: see [the template](./pkg/workflow/template.yaml). It relies on the following processing steps: * `workflow/triage` * `workflow/build` * `workflow/boot` * `workflow/fuzz` Triage and build steps need the actual kernel checkouts. The base kernel repo is hosted on a shared network disk and is regularly updated by the scripts in `kernel-disk`. The system can be deployed in multiple environments, which is achieved with the help of Kustomize. Depending on the actual deployment target, different pieces of configuration are applied: * `overlays/local/minikube`: the local dev environment. * `overlays/gke/staging`: the staging prod environment. * `overlays/gke/prod`: https://ci.syzbot.org. `global-config.yaml` is the main configuration file of the system - it determines the mailing lists to poll, configures resource usage and the actual reporting. ## Local installation steps 1. Install and start minikube: https://minikube.sigs.k8s.io/docs/start/ ``` $ minikube start --cni=cilium --container-runtime=docker --memory=128g ``` `--cni=cilium` enables the use of a more advanced Network plugin that supports the emulation of network policies. `--container-runtime=docker` is needed for `minikube docker-env` (the default is containerd since minikube v1.39). `--memory=128g` limits the memory of the whole cluster. Fuzzing may use a lot of it, and on OOM random pods get killed (e.g. the Spanner emulator). 2. Build all docker containers (might take a while): ``` $ eval $(minikube docker-env) $ make all-containers ``` 3. Deploy local infrastructure (Argo, Spanner, Service Accounts): ``` $ kubectl create namespace argo $ make k8s-config-local-infra $ make migrate-local ``` 4. Deploy the rest of the cluster: ``` $ make k8s-config-dev | kubectl apply -f - ``` 5. Pre-fetch the kernel git repository: ``` $ make fetch-kernels-once.yaml | kubectl create -f - ``` Note that actual series processing won't start until the job created in (5) finishes. ## Developmental tips 1. Install Argo Workflows client: https://github.com/argoproj/argo-workflows/releases Then you can use the `argo` tool like this: ``` $ argo list ``` 2. Forward the dashboard port: ``` $ kubectl port-forward service/web-dashboard-service --address 0.0.0.0 50123:80 ``` --- ### Overview (`dashboard/app/README.md`) # Dashboard `dashboard` is an [App Engine](https://cloud.google.com/appengine) app that powers [syzbot](/docs/syzbot.md). The main deployment is at [syzkaller.appspot.com](https://syzkaller.appspot.com). It is so-called [Standard environment](https://cloud.google.com/appengine/docs/standard) Go app.\ To deploy and manage the app you need to install [Google Cloud SDK](https://cloud.google.com/sdk/install).\ For more details about App Engine refer to the [docs](https://cloud.google.com/appengine/docs/standard/go/). **Note**: The app is not stable and is not officially supported. It's here only to power the main deployment. Here is "big" picture of a possible setup: **Note**: the vector source is [here](https://docs.google.com/drawings/d/16EdqYrWD4PWD2nV_PoDPvC5VPry2H40Sm8Min-RtDdA); to update: make a copy of the source, edit, download a png, update the png and include a link to your vector copy into the PR. To deploy the app you need to add a `.go` file with production config. The config specifies kernel namespaces, bug reporting details, API keys, etc. Tests contain a [config example](app_test.go), but it's not ready for production use. The app also needs one or more [syz-ci](/syz-ci/syz-ci.go) instances running elsewhere. The `syz-ci` instances do the actual fuzzing, bisection, patch testing, etc. The app can be deployed by `gcloud app deploy ./dashboard/app/app.yaml`. The following optional flags are available: 1. "--no-promote" to test the app firs and migrate the traffic to it later. 2. "--verbosity=info" to see what files are going to be deployed. The app tests can be run with: ``` go test github.com/google/syzkaller/dashboard/app ``` During development it's handy to use `-short` flag to not run the longest tests. If any of the tests fail, use `-v` flag to see log of what happens and `-run` flag to run a single test, e.g.: ``` go test -short -v -run=TestEmailReport github.com/google/syzkaller/dashboard/app ``` ## Local Test Deployment It's possible to run the dashboard locally for testing purposes. However, note that it won't have any data, so you would need to connect `syz-ci` instances so that they populate database with some bugs. First, you need to install Google Cloud SDK (`gcloud` command, and required components, this is one time step). Then, create emulator config (this is one time step): ``` gcloud config configurations create emulator gcloud config set auth/disable_credentials true gcloud config set project syzkaller gcloud config set api_endpoint_overrides/spanner http://localhost:9020/ ``` Then, start local spanner emulator in one console: ``` gcloud emulators spanner start gcloud spanner instances create syzbot --config=emulator --nodes=1 gcloud spanner databases create ai --instance=syzbot ``` Then, initialize the schema from another console: ``` for SQL in dashboard/app//aidb/migrations/*.up.sql; do \ gcloud spanner databases ddl update ai \ --instance=syzbot --ddl-file ${SQL}; done ``` Finally, start the web server: ``` SPANNER_EMULATOR_HOST="localhost:9010" \ GOOGLE_CLOUD_SPANNER_MULTIPLEXED_SESSIONS=false \ google-cloud-sdk/bin/dev_appserver.py --application=syzkaller \ --host=0.0.0.0 --enable_host_checking=false dashboard/app/ ``` --- ### Overview (`docs/akaros/README.md`) # Akaros support [Akaros](http://akaros.cs.berkeley.edu/) support was **removed**. You can find it in git history, if needed. Found bugs are [here](https://groups.google.com/forum/#!searchin/akaros/syzbot), [here](https://syzkaller.appspot.com/akaros) and [here](https://github.com/brho/akaros/issues?q=is%3Aissue+syzkaller). --- ### Overview (`docs/darwin/README.md`) # Darwin/XNU It turned out to be unreasonably hard to bootstrap a VM image usable for fuzzing from the XNU source drops without using any of the proprietary kernel extensions shipped with macOS. This guide is therefore based on a normal macOS installation. Unfortunately Apples macOS EULA makes this unsuitable for fuzzing XNU on Google Cloud Platform. ## Prepare a macOS installation disk image Nowadays Apple mainly distributes macOS updates via the Mac App Store. This will however only give us the latest builds. Luckily the Munki people are kind enough to maintain a script, allowing us to fetch a macOS build of our choice from the shell. We'll need a macOS build that has a kernel version close to the one we will be building. You can [see Apples most recent source drops on this page](https://opensource.apple.com/). At the time of writing the most recent version is macOS 11.5 containing kernel xnu-7195.141.2. The Munki download script can only tell the macOS version and build number, but not the XNU version. Unfortunately you might occasionally download a matching macOS release, but the kernel you build won't boot anyway. Among many other reasons this can be caused by this mismatch in kernel and bootloader versions. Some trial and error can be involved in getting the correct build. Sometimes the correct build might no longer be available. At the time of writing 11.5 build 20G71 was available and worked with the 11.5 xnu source drop. In the instructions below I will assume you have VMware Fusion installed on your host macOS for creating the VM disk image. This is for convenience sake as Fusion allows us to simply drag and drop in the macOS installer App we downloaded. If you want to use another tool like Qemu for this, [take note of Apples own process for creating a bootable install media](https://support.apple.com/en-us/HT201372). I had trouble generating bootable ISOs from certain macOS builds using Apples method, hence I just always let Fusion create the installation medium for me. Additionally the below instructions ask you to disable System Integrity Protection and Authenticated Root inside the VM. We need to do this in order to run the DIY kernel we will build in a bit. Executive Summary on these features: - In OS X 10.11 Apple introduced System Integrity Protection, a feature that (among other things) limits even root from writing to certain critical system directories during normal operations. We need to disable it to write our kernel to disk - In macOS 11 Apple introduced Authenticated Root. Starting with this version only a cryptographically signed read only snapshot of the root filesystem is mounted during boot. We need to disable it in order to remount a writable version and take a new snapshot to boot from later **Tl;Dr: To create the VM image:** - [Clone Munkis macadmin-script repo](https://github.com/munki/macadmin-scripts) - Run `installinstallmacos.py` and choose a version matching the last kernel source drop - Mount and open the downloaded `Install_macOS_-.dmg` - Open VMware Fusion and create a new VM via the File menu - Drag and Drop the `Install macOS ` App from the mounted disk image into the `Select the Installation Method` Fusion dialog - I suggest cranking up the VMs CPU and Memory at this point - After picking your language in the macOS installer open `Utilities -> Terminal` - Enter `csrutil disable` to disable System Integrity Protection - Enter `csrutil authenticated-root disable` to disable Authenticated Root - Quit the Terminal app via Cmd+Q - Follow through with the macOS installation, creating a user called `user`. If you don’t have any disk available to install to, you might need to use the Disk Utility to format the virtual disk first - Go to `System Preferences -> Software Update -> Advanced…` and uncheck `Check for updates` - Go to `System Preferences -> Energy Saver` and check `Prevent computer from sleeping automatically when the display is off` - Go to `System Preferences -> Sharing` and check `Remote Login` to enable sshd - Add your ssh pubkey to both your users and roots authorized_keys files - Optionally disable WindowServer and other non-essential services via launchd. Note that you will obviously loose the GUI: `sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.WindowServer.plist` Check that everything looks alright: ## Prepare a Kernel optimized for fuzzing You might be wondering why we aren’t using one of the precompiled kernels available in Apples Kernel Development Kits. However those don’t include any kernel built with the KSANCOV feature flag. KSANCOV is Apples take on an API that allows userspace to request the kernel to start tracing which kernel code a given thread touched and exposing that information to userspace. This information is required by Syzkaller to be really effective at fuzzing. Luckily [afrojer@](https://twitter.com/afrojer) is releasing semi-regularly updated instructions on building XNU from source and installing it on macOS on his blog. At the time of writing he is lagging three minor source drop versions behind. The [most recent instructions are for macOS 11.2](https://web.archive.org/web/20210524205524/https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html). We’ll cover some additional required changes in this text. Building and testing a XNU useful for fuzzing: - Download a somewhat recent Xcode from [Apples Xcode versions archive (Apple ID login required)](https://developer.apple.com/download/all/?q=xcode) to your VM. I’m using Xcode 12.5. I had issues with 12.5.1 and 13 beta 4 - Open the `Xcode_.xip` to extract. Make a coffee ⏳ - Drag and Drop the extracted Xcode app into your VMs Applications folder - Start Xcode, agree to the license and quit it after it finishes installation - Create and join a directory named `kernel` in the users home dir and cd into it - Downloading afrojer@s Makefile: `curl https://jeremya.com/sw/Makefile.xnudeps > Makefile.xnudeps` Note this file is not versioned, but always updated in place. [Here is an archive link to the version at the time of writing.](https://web.archive.org/web/20210210224511/https://jeremya.com/sw/Makefile.xnudeps) - `make -f Makefile.xnudeps macos_version=11.5 xnudeps` This will get the dependencies for building the 11.5 XNU. [Check out the original blog post for details on how to fetch dependencies for a specific version](https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html) - cd into `~/kernel/xnu-/` - Apply [required XNU patches](0001-fuzzing.patch) manually or via `git am`. Applying via git requires you to init the repo and commit all files first. That's probably a good idea anyway to track further changes you might make - The patches for `MakeInc.def` and `kasan.c` are required for the KASAN kernel to build. KASAN is short for KernelAddressSANitizer - a feature to detect a bunch of memory safety issues inside the kernel during runtime - The `ksancov.h` patch is required for building syzkallers executor. Executor is C++ and hence doesn't like the void pointer casting - Finally the `cpuid.c` and `cpu_threads.c` patches are required for our kernel to boot on Qemu - Run `mount` and look for the roots mount device. On my VM it looks like this `/dev/disk2s5s1 on / (apfs, sealed, local, read-only, journaled)`. Now remember the devices name, ignoring the last sN part. So I note down `/dev/disk2s5` - Cd into `~/kernel/xnu-/` and run the following, replacing `` to build and install your kernel ``` mkdir -p BUILD/mnt sudo mount -o nobrowse -t apfs /dev/ $PWD/BUILD/mnt make SDKROOT=macosx TARGET_CONFIGS="KASAN X86_64 NONE" KSANCOV=1 kmutil create -a x86_64 -Z -n boot sys \ -B BUILD/BootKernelExtensions.kc.kasan \ -S BUILD/SystemKernelExtensions.kc.kasan \ -k BUILD/obj/kernel.kasan \ --elide-identifier com.apple.driver.AppleIntelTGLGraphicsFramebuffer sudo ditto BUILD/BootKernelExtensions.kc.kasan "$PWD/BUILD/mnt/System/Library/KernelCollections/" sudo ditto BUILD/SystemKernelExtensions.kc.kasan "$PWD/BUILD/mnt/System/Library/KernelCollections/" sudo ditto BUILD/obj/kernel.kasan "$PWD/BUILD/mnt/System/Library/Kernels/" sudo bless --folder $PWD/BUILD/mnt/System/Library/CoreServices --bootefi --create-snapshot sudo nvram boot-args="-v kcsuffix=kasan wlan.skywalk.enable=0" ``` After rebooting you should see your shiny new kernel when running `uname -a`: `Darwin users-Mac.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 9 16:12:43 PDT 2021; user:xnu-7195.141.2/BUILD/obj/KASAN_X86_64 x86_64` For effective fuzzing we'll need the kernels binary, symbols and source on the host. Copy them like this: ``` mkdir -p ~/115/src/Users/user/kernel/ ~/115/obj rsync -r mac:/Users/user/kernel/xnu-7195.141.2 ~/115/src/Users/user/kernel/ mv ~/115/src/Users/user/kernel/xnu-7195.141.2/BUILD/obj/KASAN_X86_64/kernel.kasan ~/115/obj/ mv ~/115/src/Users/user/kernel/xnu-7195.141.2/BUILD/obj/KASAN_X86_64/kernel.kasan.dSYM/ ~/115/obj/ ``` ## Preparing VM for Qemu Even though Macs are AMD64 machines with EFIs (at least the once we care about here), they aren't exactly IBM PC compatible. So far VMWare Fusion did all the trickery necessary to virtualize macOS for us, but qemu-system-x86_64 does not. To make macOS boot we'll first start Qemu with OVMF (tianocore based UEFI for qemu). Next we boot OpenCore, which will do some trickery making it possible to chainload Apples stock AMD64 EFI bootloader. It also does some binary kernel patching, making it possible to load the RELEASE kernel shipped with macOS, should we want that. OpenCore is rather configurable, but we don't care about real hardware. [I'm using this version prebuild and configured to work inside Qemu](https://github.com/thenickdude/KVM-Opencore/releases). We can simply overwrite the EFI partition on our VMs disk with the EFI partition from one of the images from this repo. Let's first find out which partition we will overwrite in a minute. From the following output from within the macOS VM (still booted via Fusion for now) we can see that in my case the EFI partition is at `/dev/disk0s1`: ``` user@users-Mac ~ % diskutil list /dev/disk0 (internal, physical): #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *69.8 GB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_APFS Container disk1 69.6 GB disk0s2 /dev/disk1 (synthesized): #: TYPE NAME SIZE IDENTIFIER 0: APFS Container Scheme - +69.6 GB disk1 Physical Store disk0s2 1: APFS Volume macos - Data 43.2 GB disk1s1 2: APFS Volume Preboot 385.6 MB disk1s2 3: APFS Volume Recovery 623.2 MB disk1s3 4: APFS Volume VM 1.1 MB disk1s4 5: APFS Volume macos 16.0 GB disk1s5 6: APFS Snapshot com.apple.bless.4099... 16.0 GB disk1s5s1 ``` Now download [OpenCore-v13.iso.gz](https://github.com/thenickdude/KVM-Opencore/releases/download/v13/OpenCore-v13.iso.gz) and extract the image via `gzip -d OpenCore-v13.iso.gz`. Display the partition map to find out the images blocksize and the EFI partitions offset and size. ``` user@users-Mac ~ % hdiutil pmap ./OpenCore-v13.iso MEDIA: ""; Size 150 MB [307200 x 512]; Max Transfer Blocks 2048 SCHEME: 1 GPT, "GPT Partition Scheme" [16] SECTION: 1 Type:'MAP'; Size 150 MB [307200 x 512]; Offset 34 Blocks (307133 + 67) x 512 ID Type Offset Size Name (1) -- -------------------- ------------ ------------ -------------------- -------- 1 EFI 40 307120 disk image ``` Now put all those values together in a dd command like so: `sudo dd if=./OpenCore-v13.iso of=/dev/disk0s1 bs=512 iseek=40 count=307120` Now let's mount the EFI disk via `sudo mount -t msdos /dev/disk0s1 ~/mnt/`. We have to edit OpenCores config file a tiny bit. We disable the boot device selector, as that will prevent us from starting the VMs during fuzzing completely automatically. Additionally note how we set boot-args here. In VMware Fusion we were able to use the normal macOS tools like nvram and csrutil. In OpenCore we need to set these settings in the config.plist instead. Edit `~/mnt/EFI/OC/config.plist` like so: ```diff index 8537ca8..a46de97 100755 --- a/Users/user/mnt/EFI/OC/config.plist +++ b/Users/user/mnt/EFI/OC/config.plist @@ -799,7 +799,7 @@ PollAppleHotKeys ShowPicker - + TakeoffDelay 0 Timeout @@ -944,7 +944,7 @@ SystemAudioVolume Rg== boot-args - keepsyms=1 + -v kcsuffix=kasan wlan.skywalk.enable=0 keepsyms=1 debug=0x100008 kasan.checks=4294967295 csr-active-config Jg8= prev-lang:kbd ``` At this point you should still be able to (re)boot the VM in Fusion. It will just ignore OpenCore however. That's fine. ## Prepare isa-applesmc On boot macOS checks whether it is booted on a proper Mac by reading a value from its System Management Controller and comparing it with the value it expects. We'll retrieve this value now and pass it to qemu later. To retrieve the value: - [Download the smc_read.c source from this site.](https://web.archive.org/web/20200603015401/http://www.osxbook.com/book/bonus/chapter7/tpmdrmmyth/) - `gcc -Wall -o smc_read smc_read.c -framework IOKit` `./smc_read` That will produce a single line of text which you will later have to substitute in for a place marked ``. ## Booting macOS via Qemu - Setup [Homebrew](https://brew.sh/) on your host macOS - Install `qemu` via homebrew - Your VMs disk should be somewhere like this on your host `~/Virtual\ Machines.localized/macOS-11.5-20G71.vmwarevm/Virtual\ Disk.vmdk`. Convert it to qcow2 via something like this `qemu-img convert -U ./Virtual\ Disk.vmdk -O qcow2 ~/115/mac_hdd.qcow` - Unfortunately OVMF is currently not packaged in Homebrew. Download the [`ovmf` package from ubuntu](https://packages.ubuntu.com/hirsute/ovmf). Extract it via `ar -xv ./ovmf_2020.11-4_all.deb` and `tar -xvf ./data.tar.xz`. Finally `mv ./usr/share/OVMF /usr/local/share/OVMF` That's pretty much all you should need in order to boot the VM image we build. Start Qemu like so. Remember to substitute `` and the username in the disk image path: ``` qemu-system-x86_64 \ -device isa-applesmc,osk="" \ -accel hvf -machine q35 -smp "2",cores="2",sockets="1" -m "4096" \ -cpu Penryn,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,"+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check" \ -drive if=pflash,format=raw,readonly=on,file="/usr/local/share/OVMF/OVMF_CODE.fd" \ -drive if=pflash,format=raw,readonly=on,file="/usr/local/share/OVMF/OVMF_VARS.fd" \ -device ich9-intel-hda -device hda-duplex -device ich9-ahci,id=sata \ -device ide-hd,bus=sata.4,drive=MacHDD \ -drive id=MacHDD,if=none,file="/Users/user/115/macos_11_5.qcow",format=qcow2 \ -netdev user,id=net0,hostfwd=tcp::1042-:22, -device e1000-82545em,netdev=net0,id=net0 \ -device usb-ehci,id=ehci -usb -device usb-kbd -device usb-tablet \ -monitor stdio -vga vmware ``` You should both see the macOS UI and be able to `ssh user@localhost -p 1042`. Confirm we are booted into your KASAN kernel: ``` user@users-Mac ~ % uname -a Darwin users-Mac.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 9 16:12:43 PDT 2021; user:xnu-7195.141.2/BUILD/obj/KASAN_X86_64 x86_64 ``` Shut down your VM now. We'll let syzkaller boot it back up soon. ## Building Syzkaller - Install `go` via homebrew - Add something like this to your .zshrc: ``` export GOPATH=/Users/user/go export PATH=$GOPATH/bin:$PATH ``` - Relogin and build syzkaller like this: ``` git clone https://github.com/google/syzkaller cd syzkaller make HOSTOS=darwin HOSTARCH=amd64 TARGETOS=darwin TARGETARCH=amd64 SOURCEDIR=/Users/user/115/src/Users/user/kernel/xnu-7195.141.2 ``` ## Fuzzing with Syzkaller - We need g++ to make C reproducers work. Install `gcc@11` via homebrew - We need addr2line from binutils to make the `/cover` endpoint work. Install `binutils` via homebrew - Add something like this to your .zshrc `export PATH="/usr/local/opt/binutils/bin:$PATH"`. Restart your shell - Save the following to `~/115/syzkaller.cfg`. Remember to substitute ``: ``` { "target": "darwin/amd64", "http": "127.0.0.1:56741", "sshkey": "/Users/user/.ssh/id_macos115", "workdir": "/Users/user/sk_darwin/", "kernel_obj": "/Users/user/115/obj/", "kernel_src": "/Users/user/115/src/", "syzkaller": "/Users/user/go/src/github.com/google/syzkaller", "procs": 2, "type": "qemu", "cover": true, "image": "/Users/user/115/macos_11_5.qcow", "vm": { "count": 2, "cpu": 2, "mem": 4096, "efi_code_device": "/usr/local/share/OVMF/OVMF_CODE.fd", "efi_vars_device": "/usr/local/share/OVMF/OVMF_VARS.fd", "apple_smc_osk": "" } } ``` Start syzkaller via `~/115/bin/syz-manager -config=/root/115/syzkaller.cfg` and open http://localhost:56741 in your browser. --- ### Overview (`docs/freebsd/README.md`) # FreeBSD This page contains instructions to set up syzkaller to run on a FreeBSD or Linux host and fuzz an amd64 FreeBSD kernel running in a virtual machine. Currently, syzkaller can fuzz FreeBSD running under bhyve, QEMU or GCE (Google Compute Engine). Regardless of the mode of operation, some common steps must be followed. ## Setting up a host `syz-manager` is the component of syzkaller that manages target VMs. It runs on a host system and automatically creates, runs and destroys VMs which share a user-specified image file. ### Setting up a FreeBSD host To build syzkaller out of the box, a recent version of FreeBSD 13.0-CURRENT must be used for the host. Older versions of FreeBSD can be used but will require manual tweaking. The required dependencies can be installed by running: ```console # pkg install bash gcc git gmake go golangci-lint llvm ``` When using bhyve as the VM backend, a DHCP server must also be installed: ```console # pkg install dnsmasq ``` To checkout the syzkaller sources, run: ```console $ git clone https://github.com/google/syzkaller ``` and the binaries can be built by running: ```console $ cd syzkaller $ gmake ``` Once this completes, a `syz-manager` executable should be available under `bin/`. ### Setting up a Linux host To build Go binaries do: ``` make manager fuzzer execprog TARGETOS=freebsd ``` To build C `syz-executor` binary, copy `executor/*` files to a FreeBSD machine and build there with: ``` c++ executor/executor.cc -o syz-executor -O1 -lpthread -DGOOS_freebsd=1 -DGOARCH_amd64=1 -DGIT_REVISION=\"CURRENT_GIT_REVISION\" ``` Then, copy out the binary back to host into `bin/freebsd_amd64` dir. ## Setting up the FreeBSD VM It is easiest to start with a [snapshot image](https://ftp.freebsd.org/pub/FreeBSD/snapshots/VM-IMAGES/14.0-CURRENT/amd64/Latest/) of FreeBSD. Fetch a QCOW2 disk image for QEMU or a raw image for GCE or bhyve. Fetch a copy of the FreeBSD kernel sources and place them in `/usr/src`. You will likely need to expand the disk of the VM. Before booting the VM, run: ```console # truncate -s 15G $IMAGEFILE ``` To enable KCOV on FreeBSD, a custom kernel must be compiled. It is easiest to do this on the host.
Before booting the VM, compile a custom kernel on the host and install it on the VM. On the host: ```console # mdconfig -a -f $IMAGEFILE # mount /dev/md0p4 /mnt ``` This will create a memory device for the VM file and allow host to install the custom kernel source on the VM.
To get a copy of the current development sources: ```console # pkg install git # git clone --depth=1 --branch=main https://github.com/freebsd/freebsd-src /usr/src ``` To create a custom kernel configuration file for syzkaller and build a new kernel, run: ```console # cd /usr/src/sys/amd64/conf # cat <<__EOF__ > SYZKALLER include "./GENERIC" ident SYZKALLER options COVERAGE options KCOV __EOF__ # cd /usr/src # make -j $(sysctl -n hw.ncpu) KERNCONF=SYZKALLER buildkernel # make KERNCONF=SYZKALLER installkernel DESTDIR=/mnt ``` Before booting the VM, make sure to run: ```console # umount /mnt ``` The md device will linger and you can use it again later if you want. Otherwise destroy it: ```console # mdconfig -d -u 0 ``` Use QEMU to start a VM using the downloaded image: ```console $ qemu-system-x86_64 -hda $IMAGEFILE -nographic -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic,model=e1000 ``` When the boot loader menu is printed, escape to the loader prompt and enter the commands `set console="comconsole"` and `boot`. Once you reach a login prompt, log in as root and add a couple of configuration parameters to `/boot/loader.conf`: ```console # cat <<__EOF__ >>/boot/loader.conf autoboot_delay="-1" console="comconsole" __EOF__ ``` After VM is booted, /etc/rc.d/growfs should haven grown its file system automatically. Otherwise run: ```console # /etc/rc.d/growfs onestart ``` Verify that `uname -i` prints `SYZKALLER` to confirm that your newly built kernel is running. Then, to permit remote access to the VM, you must configure DHCP and enable `sshd`: ```console # sysrc sshd_enable=YES # sysrc ifconfig_DEFAULT=DHCP ``` If you plan to run the syscall executor as root, ensure that root SSH logins are permitted by adding `PermitRootLogin without-password` to `/etc/ssh/sshd_config`. Otherwise, create a new user with `adduser`. Install an ssh key for the user and verify that you can SSH into the VM from the host. Note that bhyve requires the use of the root user for the time being. ### Running Under bhyve Some additional steps are required on the host in order to use bhyve. First, since bhyve currently does not support disk image snapshots, ZFS must be used to provide equivalent functionality. Create a ZFS data set and copy the VM image there. The data set can also be used to store the syzkaller workdir. For example, with a zpool named `data` mounted at `/data`, write: ```console # zfs create data/syzkaller # cp FreeBSD-13.3-CURRENT-amd64.raw /data/syzkaller ``` Third, configure bridged networking and DHCP for the VM instances. This is no longer required on FreeBSD 14.1 and 15.0-CURRENT, as bhyve now supports libslirp-based user networking similar to the QEMU backend. ```console # ifconfig bridge create bridge0 # ifconfig bridge0 inet 169.254.0.1 # echo 'dhcp-range=169.254.0.2,169.254.0.254,255.255.255.0' > /usr/local/etc/dnsmasq.conf # echo 'interface=bridge0' >> /usr/local/etc/dnsmasq.conf # sysrc dnsmasq_enable=YES # service dnsmasq start # echo 'net.link.tap.up_on_open=1' >> /etc/sysctl.conf # sysctl net.link.tap.up_on_open=1 ``` To enable automatic configuration of bridged network every time the system boots, add the following to /etc/rc.conf: ```console # cloned_interfaces="bridge0 tap0" # ifconfig_bridge0="inet 169.254.0.1 addm tap0 up" # ifconfig_tap0="up" ``` Finally, ensure that the bhyve kernel module is loaded: ```console # kldload vmm ``` ### Putting It All Together If all of the above worked, create a `freebsd.cfg` configuration file with the following contents (alter paths as necessary): ``` { "name": "freebsd", "target": "freebsd/amd64", "http": ":10000", "workdir": "/workdir", "syzkaller": "/gopath/src/github.com/google/syzkaller", "sshkey": "/freebsd_id_rsa", "sandbox": "none", "procs": 8, } ``` If running the fuzzer under QEMU, add: ``` "image": "/FreeBSD-13.0-CURRENT-amd64.qcow2", "type": "qemu", "vm": { "count": 10, "cpu": 4, "mem": 2048 } ``` For GCE, add the following instead (alter the storage bucket path as necessary): ``` "image": "/FreeBSD-13.0-CURRENT-amd64.raw", "type": "gce", "vm": { "count": 10, "instance_type": "n1-standard-4", "gcs_path": "syzkaller" } ``` For bhyve, we need to specify the VM image snapshot name and networking info (alter the dataset name and paths as necessary): ``` "image": "/data/syzkaller/FreeBSD-13.0-CURRENT-amd64.raw", "type": "bhyve", "vm": { "count": 10, "hostip": "169.254.0.1", "dataset": "data/syzkaller" } ``` If using bridged networking, the name of the bridge interface needs to be included in the "vm" block: ``` "vm": { "count": 10, "hostip": "169.254.0.1", "dataset": "data/syzkaller", "bridge": "bridge0" } ``` Then, start `syz-manager` with: ```console $ bin/syz-manager -config freebsd.cfg ``` It should start printing output along the lines of: ``` booting test machines... wait for the connection from test machine... machine check: 253 calls enabled, kcov=true, kleakcheck=false, faultinjection=false, comps=false executed 3622, cover 1219, crashes 0, repro 0 executed 7921, cover 1239, crashes 0, repro 0 executed 32807, cover 1244, crashes 0, repro 0 executed 35803, cover 1248, crashes 0, repro 0 ``` If something does not work, try adding the `-debug` flag to `syz-manager`. ## Missing things - System call descriptions. The initial list of FreeBSD system calls was a copy-and-paste of Linux's, and while they have been cleaned up over time they should be audited more carefully. We are also still missing many system call descriptions. - We should support fuzzing the Linux compatibility subsystem. - We should provide instructions for fuzzing a FreeBSD system on ZFS - `pkg/host` needs to be taught how to detect supported syscalls/devices. - KASAN and KCSAN for FreeBSD would be useful. - On Linux we have emission of external networking/USB traffic into kernel using tun/gadgetfs. Implementing these for FreeBSD could uncover a number of high-profile bugs. --- ### Overview (`docs/fuchsia/README.md`) # Fuchsia support For information about checking out and building Fuchsia see [Get started with Fuchsia](https://fuchsia.dev/fuchsia-src/get-started) and [Download the Fuchsia source code](https://fuchsia.dev/fuchsia-src/get-started/get_fuchsia_source). ## Caveat Please note that Fuchsia support is currently incomplete, and may break at any time due to changes in Fuchsia and/or Syzkaller. Some known issues include: * System call definitions require manual updates. * Crash parsing does not work reliably. * Coverage feedback is not supported. ## Prerequisites To run Syzkaller with a Fuchsia target, you will need a checkout of the Fuchsia source repository. The rest of this document will use the environment variable `SOURCEDIR` to identify the path to your Fuchsia checkout (e.g. `/home/you/fuchsia`). The commands below assume you have set `SOURCEDIR`, like so: ```bash export SOURCEDIR=/home/you/fuchsia ``` ## Quick and easy version The shell script [setup.sh](setup.sh) automates the building and running `syz-manager` steps described below. Give it a try, and see if it works for you. For usage help, invoke it with no arguments. ## Building Fuchsia To build Fuchsia for x64, run: ```bash fx --dir "out/x64" set core.x64 \ --with-base "//bundles/tools" \ --with-base "//src/testing/fuzzing/syzkaller" \ --args=syzkaller_dir='"/full/path/to/syzkaller"' \ --variant=kasan fx build ``` Alternatively, for arm64, run: ```bash fx --dir "out/arm64" set core.arm64 \ --with-base "//bundles/tools" \ --with-base "//src/testing/fuzzing/syzkaller" \ --args=syzkaller_dir='"/full/path/to/syzkaller"' \ --variant=kasan fx clean-build ``` ## Building binaries for Fuchsia To build all the binaries required for running Syzkaller in Fuchsia, run this from inside your Syzkaller checkout (assuming you built Fuchsia for x64): ```bash make TARGETOS=fuchsia TARGETARCH=amd64 SOURCEDIR=${SOURCEDIR} ``` ## Running syz-manager Running syz-manager requires you to have built Fuchsia previously, and added the ssh keys to the fuchsia.zbi image: The output of the Fuchsia build is a _product bundle_. Look up the image files needed via `ffx product get-image-path`: ```bash product_bundle_path="$(ffx config get product.path | tr -d '"')" fxfs_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type fxfs)" zbi_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type zbi)" multiboot_path="$(ffx product get-image-path "$product_bundle_path" --slot a --image-type qemu-kernel)" ``` Make sure the ssh keys exist and are properly configured: ```bash # Make sure there are ssh keys available ffx config check-ssh-keys auth_keys_path="$(ffx config get ssh.pub | tr -d '"')" priv_key_path="$(ffx config get ssh.priv | tr -d '"')" ``` Add the authorized keys to the zbi. ```bash fx host-tool zbi -o "${SOURCEDIR}/out/x64/fuchsia-ssh.zbi" \ "${zbi_path}" \ --entry "data/ssh/authorized_keys=${auth_keys_path}" ``` Note: This needs to be repeated after each `fx build`. Create a `syz-manager` configuration file, using this as a starting point: ```json { "name": "fuchsia", "target": "fuchsia/amd64", "http": ":12345", "workdir": "/workdir.fuchsia", "kernel_obj": "/fuchsia/out/syz/kernel_x64-kasan/obj/zircon/kernel", "syzkaller": "/syzkaller", "image": "$fxfs_path", "sshkey": "$priv_key_path", "reproduce": false, "cover": false, "procs": 8, "type": "qemu", "vm": { "count": 10, "cpu": 4, "mem": 2048, "kernel": "${multiboot_path}", "initrd": " ${SOURCEDIR}/out/x64/fuchsia-ssh.zbi" } } ``` Run `syz-manager` with that config: ```bash bin/syz-manager -config manager.cfg ``` Note: You may need to modify your `PATH` so that qemu can be found, e.g. ```bash export `PATH="$SOURCEDIR/prebuilt/third_party/qemu/linux-x64/bin:$PATH"` ``` ## Update syscall and FIDL definitions Syscall descriptions live in the `sys/fuchsia` folder. To update a syscall, you need to modify the `.txt` file that contains it, make sure your new definition matches the one in Zircon's [syscalls.abigen](https://fuchsia.googlesource.com/fuchsia/+/master/zircon/system/public/zircon/syscalls.abigen) file. **If the syscall was used in `executor/common_fuchsia.h`, you need to update the usages there as well.** FIDL definitions do not need manual updating because they are extracted automatically when you run `make extract`, but they require Fuchsia to be rebuilt for each architecture (see [Building Fuchsia](#building-fuchsia) above). Once you updated the syscall definitions, everything can be regenerated by running (in your Syzkaller checkout): ```bash make extract TARGETOS=fuchsia SOURCEDIR=${SOURCEDIR} make generate ``` **Caveat:** This command does not currently work. ## How to generate syscall definitions from FIDL TODO: This section is out of date. Syscall descriptions for FIDL are automatically generated as part of `make extract` as described above. However, if you wish to manually generate syscall descriptions for a given `.fidl` file, do the following. FIDL files should first be compiled into FIDL intermediate representation (JSON) files using `fidlc`: ```bash ${SOURCEDIR}/out/x64/host_x64/fidlc --json /tmp/io.json --files ${SOURCEDIR}/zircon/system/fidl/fuchsia-io/io.fidl ``` Then run the FIDL compiler backend `fidlgen` with the syzkaller generator, which compiles a FIDL IR file into a syscall description file: ```bash ${SOURCEDIR}/out/x64/host_x64/fidlgen -generators syzkaller -json /tmp/io.json -output-base fidl_io -include-base fidl_io ``` ## Running syz-ci locally To run `syz-ci` locally for Fuchsia, you need: - bootstrapped Fuchsia checkout (in `/bootstrap/fuchsia` dir in the example below) - bootstrap `syz-ci` binary (in the current dir, build with `make ci`) - `syz-ci` config similar to the one below (in `ci.cfg` file in the current dir) ```json { "name": "testci", "http": ":50000", "manager_port_start": 50001, "syzkaller_repo": "https://github.com/google/syzkaller.git", "managers": [ { "name": "fuchsia", "repo": "https://fuchsia.googlesource.com", "manager_config": { "target": "fuchsia/amd64", "type": "qemu", "cover": false, "procs": 8, "vm": { "count": 4, "cpu": 4, "mem": 1024 } } } ] } ``` Run `syz-ci` as: ```bash SOURCEDIR=/bootstrap/fuchsia ./syz-ci -config ci.cfg ``` ## Troubleshooting While running the `make extract` step, it's possible that the FIDL definitions are not up to date. It could happen that they have been removed or renamed. If this is the case, you would see an error mentioning that the fidl.json file could not be found: ```bash go generate ./sys/fuchsia cannot find /path-to-fuchsia/out/x64/fidling/gen/zircon/public/fidl/zircon-ethernet/zircon-ethernet.fidl.json exit status 1 ``` You can search for the string in [the Fuchsia code search interface](https://cs.opensource.google/fuchsia/fuchsia/+/main:) or in [the Gerrit code-review tool](https://fuchsia-review.googlesource.com/) to see what happened to it. If the FIDL interface was renamed or removed, you should update `sys/fuchsia/fidlgen/main.go` to reflect this change, and remove the stale autogenerated files. --- ### Overview (`docs/gvisor/README.md`) # gVisor [gVisor](https://github.com/google/gvisor) is a user-space kernel, written in Go, that implements a substantial portion of the Linux system surface. `gVisor` uses `linux` OS, but the special `gvisor` VM type. There is nothing special regarding `gVisor` besides that. The `image` is a gVisor release tarball (`gvisor.tar.bz2`, see [installation instructions](https://gvisor.dev/docs/user_guide/install/)), which syzkaller extracts into the `workdir`. There is no kernel object, so `kernel_obj` is not needed; for coverage, point `kernel_src` to the gVisor source tree. Here is an example manager config: ``` { "name": "gvisor", "target": "linux/amd64", "http": ":12345", "workdir": "/workdir", "image": "/path/to/gvisor.tar.bz2", "syzkaller": "/gopath/src/github.com/google/syzkaller", "cover": false, "procs": 8, "type": "gvisor", "vm": { "count": 5, "runsc_args": "-platform=kvm" } } ``` ## Reproducing crashes `syz-execprog` can be used inside gVisor to (hopefully) reproduce crashes. To run a single program inside a minimal gVisor sandbox, do the following. 1. Build all of the syzkaller tools: ```bash $ cd $SYZKALLER_DIR $ make ``` 2. Build the gVisor release tarball and extract it: ```bash $ cd $GVISOR_DIR $ make release-tarball DESTINATION=bin/ $ tar -C bin/ -xf bin/gvisor.tar.bz2 ``` 3. Create a `bundle/` directory with a config like the one below. Be sure to update the paths to the `linux_amd64` directory and input log/program file. ```bash $ mkdir bundle $ $EDITOR bundle/config.json ``` 4. Run gVisor: ```bash $ sudo bin/runsc \ -platform=ptrace \ -file-access=shared \ -network=host \ run \ -bundle /PATH/TO/bundle/ \ syzkaller ``` 5. Remove container: ```bash $ sudo bin/runsc delete -force syzkaller ``` Note that you'll want to adjust the `runsc` args to match the config in which the crash was discovered. You may also want to add `-debug -strace` for more debugging information. You can also adjust the args to `syz-execprog` in `config.json`. e.g., add `-repeat` to repeat the program. ### config.json ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## syzkaller way gVisor use To build a specific gVisor version: ``` git clone https://github.com/google/gvisor git checkout be6ffa78e4df78df13d004a17f2a8833305285c4 ``` syzkaller uses gVisor release tarballs as images. `pkg/build` builds them from the gVisor sources with `make copy` (see `pkg/build/gvisor.go`), e.g. for the coverage-instrumented build that syzbot uses: ``` make copy TARGETS=//debian:gvisor-release-coverage-tar-bz2 DESTINATION=gvisor-coverage.tar.bz2 ``` `-race` and `-race-coverage` variants of the tarball are available as well. The kernel config of a gVisor manager consists of the `-cover` and `-race` flags that select the variant. To get coverage symbols, extract the tarball and run `runsc symbolize`: ``` tar -xf gvisor-coverage.tar.bz2 ./runsc symbolize -all > symbolize_all_gvisor_be6ffa78e4df78df13d004a17f2a8833305285c4.txt ``` --- ### Overview (`docs/hafnium/README.md`) # Hafnium [Hafnium](https://hafnium.googlesource.com/hafnium) is a hypervisor. This is work-in-progress, see #996. For details on how to build/run Hafnium see [GettingStarted](https://hafnium.googlesource.com/hafnium/+/HEAD/docs/GettingStarted.md), [HafniumRamDisk](https://hafnium.googlesource.com/hafnium/+/HEAD/docs/HafniumRamDisk.md) and [PreparingLinux](https://hafnium.googlesource.com/hafnium/+/HEAD/docs/PreparingLinux.md). For now we only have descriptions for [AF_HF protocol](/sys/linux/hafnium.txt) provided by Hafnium linux driver. --- ### Overview (`docs/netbsd/README.md`) # NetBSD Instructions to set up syzkaller for a Linux host and a NetBSD guest. ## Preparing the NetBSD sources 1. Get the NetBSD kernel source (preferably HEAD). ```sh host$ mdkir $HOME/netbsd host$ cd $HOME/netbsd host$ git clone https://github.com/NetBSD/src.git ``` 2. Build the tools. You will have the toolchain in `$HOME/netbsd/tools`. ```sh host$ cd src host$ ./build.sh -j4 -m amd64 -U -T ../tools tools ``` 3. Build the distribution. This might take a while. ```sh host$ ./build.sh -j4 -m amd64 -U -T ../tools -D ../dest distribution ``` At this point you should have a NetBSD distribution in `$HOME/netbsd/dest`. ## Installing and building syzkaller on the Linux host 1. Install all the dependencies for syzkaller. 2. Clone the syzkaller repository. ```sh host$ git clone https://github.com/google/syzkaller host$ cd syzkaller ``` 3. Compile syzkaller for NetBSD. ```sh host$ make TARGETOS=netbsd SOURCEDIR=$HOME/netbsd ``` The above steps should have built the syzkaller binaries for NetBSD. You can see the compiled binaries in `bin/netbsd_amd64`. ## Setting up a NetBSD VM with qemu You can use the script given [here](https://github.com/R3x/netbsd-fuzzing-aids/blob/master/install_netbsd.sh) to create a disk image with NetBSD installed. The script would also automatically give you a ssh key to ssh into the VM. Alternatively, you can follow the tutorial given [here](https://wiki.qemu.org/Hosts/BSD#NetBSD) to set up a basic NetBSD VM with qemu. After installing and running the NetBSD VM on qemu, please follow the steps below to configure ssh. 1. Create a ssh-keypair on the host and save it as `netbsdkey`. ```sh host$ ssh-keygen -f netbsdkey -t rsa -N "" ``` 2. Make sure you have a NAT enabled in your Qemu command line. Typically to forward the host port `10022` to the guest port `22`: ```sh host$ qemu-system-x86_64 ... -netdev user,id=mynet0,hostfwd=tcp:127.0.0.1:10022-:22 -device e1000,netdev=mynet0 ``` 3. Append the following lines to `/etc/rc.conf` on the guest. You can use the `vi` editor to do that. ``` sshd=YES dhcpcd=YES ifconfig_wm0="inet 10.0.2.15 netmask 255.255.255.0" ``` 4. Append this to `/etc/ssh/sshd_config` on the guest. ``` Port 22 ListenAddress 10.0.2.15 PermitRootLogin yes PermitRootLogin without-password ``` 5. Now you should be able to ssh into the NetBSD VM. ```sh host$ ssh -p 10022 root@127.0.0.1 ``` 6. Copy and paste your public key to `/root/.ssh/authorized_keys` on the guest and `reboot` the VM. 7. After reboot make sure that ssh is working properly. Replace the port with what you have configured. ```sh host$ ssh -i path/to/netbsdkey -p 10022 root@127.0.0.1 ``` If the last command returns a proper shell it means the VM has been configured. ## Compiling a NetBSD kernel (Optional) You can compile a kernel with KASAN to increase the chances of finding bugs. 1. Make a copy of the config file. ```sh host$ cd $HOME/netbsd/src host$ cp sys/arch/amd64/conf/GENERIC sys/arch/amd64/conf/SYZKALLER ``` 2. Uncomment the following lines in `sys/arch/amd64/conf/SYZKALLER` to enable KASAN. ``` #makeoptions KASAN=1 # Kernel Address Sanitizer #options KASAN #no options SVS ``` 3. Compile the kernel with KASAN (assuming you have followed the initial steps to build tools). ```sh host$ cd $HOME/netbsd/src host$ ./build.sh -m amd64 -U -T ../tools -j4 kernel=SYZKALLER ``` 4. At this point you should have the new compiled kernel image which can be found in `$HOME/netbsd/src/sys/arch/amd64/compile/SYZKALLER` and should have the name `netbsd`. You need to copy it to the installed VM and reboot the VM. ## Running syzkaller 1. If all of the above worked, `poweroff` the VM and create the `netbsd.cfg` config file on the host with the following contents (alter paths as necessary): ``` { "name": "netbsd", "target": "netbsd/amd64", "http": ":10000", "workdir": "work", "syzkaller": "$GOPATH/src/github.com/google/syzkaller", "image": "path/to/netbsd.img", "sshkey": "/path/to/netbsdkey", "sandbox": "none", "procs": 2, "cover": false, "type": "qemu", "vm": { "qemu": "qemu-system-x86_64", "count": 2, "cpu": 2, "mem": 2048 } } ``` The above directories have to be specified to the exact locations and the ssh keys must be in a separate directory with chmod 700 permissions set to that directory and chmod 600 permissions to the files in both the guest and the host. 2. Then, inside the syzkaller folder where the `netbsd.cfg` file also exists, start `syz-manager` with: ```sh host$ bin/syz-manager -config netbsd.cfg ``` You can add a `-debug` flag to the above command to view the log if any issues arise. 3. Once syzkaller has started executing, it should start printing output along the lines of: ``` booting test machines... wait for the connection from test machine... machine check: 253 calls enabled, kcov=true, kleakcheck=false, faultinjection=false, comps=false executed 3622, cover 1219, crashes 0, repro 0 executed 7921, cover 1239, crashes 0, repro 0 executed 32807, cover 1244, crashes 0, repro 0 executed 35803, cover 1248, crashes 0, repro 0 ``` ## syzbot [syzbot](/docs/syzbot.md) tests NetBSD and reports bugs to [syzkaller-netbsd-bugs](https://groups.google.com/forum/#!forum/syzkaller-netbsd-bugs) mailing list (also can be seen on [dashboard](https://syzkaller.appspot.com/netbsd)). The image `syzbot` uses can be downloaded [here](https://storage.googleapis.com/syzkaller/netbsd-image.tar.gz) (266MB, includes root ssh key). The image was built using this [script](https://github.com/R3x/netbsd-fuzzing-aids/blob/master/install_netbsd.sh). The image can be used with qemu as follows: ``` qemu-system-x86_64 -m 1024 -smp 2 -nographic -enable-kvm \ -netdev user,id=mynet0,hostfwd=tcp:127.0.0.1:10022-:22 \ -device e1000,netdev=mynet0 -hda netbsd-image.raw ``` And then you can ssh/scp into the VM using: ``` ssh -i netbsd-image.key -p 10022 -o IdentitiesOnly=yes root@localhost scp -i netbsd-image.key -P 10022 -o IdentitiesOnly=yes FILE root@localhost:/root/ ``` Note: the image contains a stock kernel, so if you are reproducing a bug most likely you want to update kernel as the first step: ``` scp -i netbsd-image.key -P 10022 -o IdentitiesOnly=yes \ src/sys/arch/amd64/compile/obj/GENERIC_SYZKALLER/netbsd root@localhost:/netbsd ssh -i netbsd-image.key -p 10022 -o IdentitiesOnly=yes root@localhost /sbin/reboot ``` ## Missing things - Automating the configuration changes (like appending to config files), generating the json config file on the fly (with customizable values to the keys using command line parameters) and calling syz-manager with `anita` using just a single command. - System call descriptions. `sys/netbsd/*.txt` is a dirty copy from `sys/linux/*.txt` with everything that does not compile dropped. We need to go through syscalls and verify/fix/extend them, including devices/ioctls/etc. - Currently only `amd64` arch is supported. Supporting `386` would be useful, because it should cover compat paths. Also, we could do testing of the linux-compatibility subsystem. - `pkg/host` needs to be taught how to detect supported syscalls/devices. - On Linux we have emission of external networking/USB traffic into kernel using tun/gadgetfs. Implementing these for NetBSD could uncover a number of high-profile bugs. --- ### Overview (`docs/starnix/README.md`) # starnix support ## Prerequisites To run syzkaller for fuzzing starnix, you will need a checkout of the Fuchsia source repository. The rest of this document will use the environment variable `SOURCEDIR` to identify the path to your Fuchsia checkout (e.g. `/home/you/fuchsia`). The commands below assume you have set `SOURCEDIR`, like so: ```bash export SOURCEDIR=/home/you/fuchsia ``` To build Fuchsia for qemu-x64, run: ``` fx --dir "out/qemu-x64" set workstation_eng.qemu-x64 \ --with "//bundles/tools" \ --with "//src/proc/bin/starnix" fx build ``` You will also need to follow the instructions in the sections `GCC` and `Kernel` of the [setup\_ubuntu-host\_qemu-vm\_x86-64-kernel.md](../linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md) file. ## syzkaller ### Building binaries for starnix First, you need to build all the binaries required for running syzkaller in starnix. For that, you only need to run this from inside your syzkaller checkout (assuming you built Fuchsia for x64): ```bash SYZ_STARNIX_HACK=1 make TARGETOS=linux TARGETARCH=amd64 ``` ### Configuration file Create a manager config like the following, replacing the environment variables `$SYZKALLER`, `$KERNEL` and `$IMAGE` with their actual values. > **_NOTE:_** `ffx` is still under development, for that reason VM count of 1 is recommended until fxbug.dev/118926 is solved. ```bash { "target": "linux/amd64", "http": "127.0.0.1:56741", "workdir": "$SYZKALLER/workdir", "kernel_obj": "$KERNEL", "kernel_src": "$SOURCEDIR", "syzkaller": "$SYZKALLER", "procs": 8, "type": "starnix", "vm": { "count": 1 }, "cover": false } ``` ### Running Lastly, just run the command below to start fuzzing. ```bash SYZ_STARNIX_HACK=1 bin/syz-manager -config=./starnix.cfg ``` --- ### Overview (`docs/trusty/README.md`) # Trusty support [Trusty](https://source.android.com/security/trusty) is a set of software components supporting a Trusted Execution Environment (TEE) on mobile devices. This is work-in-progress, see #933. For now we only support testing `Trusty` via actual application ports. # Building kernel with Trusty IPC support ``` git remote add android https://android.googlesource.com/kernel/common git fetch android git checkout android/android-trusty-4.9 make distclean # TODO: consider using trusty_qemu_defconfig instead. make ARCH=arm64 ranchu64_defconfig # Required to enable qemu networking: make ARCH=arm64 defconfig make ARCH=arm64 kvmconfig # Some custom configs: ARCH=arm64 ./scripts/kconfig/merge_config.sh .config trusty.config # We don't install modules: sed -i 's#^\(.*\)=m$#\# \1 is not set#g' .config make ARCH=arm64 olddefconfig make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j64 ``` ``` # trusty.config CONFIG_TRUSTY=y CONFIG_DEBUG_FS=y CONFIG_DEBUG_INFO=y CONFIG_KCOV=y CONFIG_KASAN=y CONFIG_KASAN_INLINE=y CONFIG_PROVE_LOCKING=y CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_DEBUG_VM=y CONFIG_LOCKUP_DETECTOR=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y CONFIG_DETECT_HUNG_TASK=y CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=140 CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y CONFIG_WQ_WATCHDOG=y ``` # Building Trusty ``` mkdir trusty; cd trusty repo init -u https://android.googlesource.com/trusty/manifest -b master repo sync -j32 source trusty/vendor/google/aosp/scripts/envsetup.sh make -j32 generic-arm64 # Build Trusty and qemu images: trusty/vendor/google/aosp/scripts/build.py qemu-generic-arm64-test-debug # Create qemu-comb.dtb: KERNEL_DIR=$KERNEL build-root/build-qemu-generic-arm64-test-debug/run-qemu ``` # Building arm64 image ``` git clone git://git.buildroot.net/buildroot cd buildroot make qemu_aarch64_virt_defconfig support/kconfig/merge_config.sh .config syzkaller.config make -j64 ``` ``` # syzkaller.config BR2_cortex_a57=y BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_ARM_AARCH64=y BR2_TARGET_GENERIC_HOSTNAME="syzkaller" BR2_TARGET_GENERIC_ISSUE="syzkaller" BR2_PACKAGE_DHCPCD=y BR2_PACKAGE_OPENSSH=y BR2_TARGET_ROOTFS_EXT2_SIZE="1G" BR2_ROOTFS_POST_FAKEROOT_SCRIPT="./syzkaller.sh" # BR2_LINUX_KERNEL is not set ``` ``` # syzkaller.sh set -eux # Mount debugfs for KCOV. echo "debugfs /sys/kernel/debug debugfs defaults 0 0" >> $1/etc/fstab # Generate and install ssh key. rm -f key key.pub ssh-keygen -f key -t rsa -N "" mkdir -p $1/root/.ssh cp key.pub $1/root/.ssh/authorized_keys ``` # Testing build TODO: where does the firmware come from? Boot in qemu: ``` cd $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/qemu-build/aarch64-softmmu/qemu-system-aarch64 -m 1024 -smp 1 -net nic -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -display none -serial stdio -no-reboot -machine virt,secure=on,virtualization=on -cpu cortex-a57 -bios $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/bl1.bin -d unimp -semihosting-config enable,target=native -no-acpi -dtb $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/qemu-comb.dtb -hda $BUILDROOT/output/images/rootfs.ext4 -snapshot -kernel $KERNEL/arch/arm64/boot/Image -append "androidboot.hardware=qemu_trusty earlyprintk=serial console=ttyAMA0,38400 root=/dev/vda" ``` SSH into the VM: ``` ssh -i $BUILDROOT/key -p 10022 -o IdentitiesOnly=yes root@localhost ``` # Running syzkaller Build and run `syzkaller` as: ``` cd $SYZKALLER make TARGETARCH=arm64 cd $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug $SYZKALLER/bin/syz-manager -config trusty.cfg ``` using config along the lines of (substitute actual values for `$KERNEL`, `$SYZKALLER`, `$BUILDROOT` and `$TRUSTY`): ``` { "name": "trusty", "target": "linux/arm64", "http": ":10000", "workdir": "/workdir", "kernel_obj": "$KERNEL", "syzkaller": "$SYZKALLER", "image": "$BUILDROOT/output/images/rootfs.ext4", "sshkey": "$BUILDROOT/key", "cover": false, "procs": 4, "type": "qemu", "vm": { "count": 4, "cpu": 1, "mem": 1024, "qemu": "$TRUSTY/build-root/build-qemu-generic-arm64-test-debug/qemu-build/aarch64-softmmu/qemu-system-aarch64", "qemu_args": "-machine virt,secure=on,virtualization=on -cpu cortex-a57 -bios $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/bl1.bin -d unimp -semihosting-config enable,target=native -no-acpi -dtb $TRUSTY/build-root/build-qemu-generic-arm64-test-debug/atf/qemu/debug/qemu-comb.dtb", "cmdline": "androidboot.hardware=qemu_trusty console=ttyAMA0,38400 root=/dev/vda", "kernel": "$KERNEL/arch/arm64/boot/Image" }, "enable_syscalls": [ "openat$trusty*", "write$trusty*", "read", "ioctl$TIPC_IOC_CONNECT*", "ppoll", "dup3", "tkill", "gettid", "close" ] } ``` --- ### Overview (`docs/windows/README.md`) # Windows `Windows` support is very raw and preliminary (read, non-working). There is a [closed-source port at Microsoft](https://github.com/dwizzzle/Presentations/blob/master/David%20Weston%20-%20Keeping%20Windows%20Secure%20-%20Bluehat%20IL%202019.pdf). There is a more complete [closed-source Windows port](https://www.slideshare.net/AnthonyLAOUHINETSUEI/wsl-reloaded) done by [Fritz](https://twitter.com/anarcheuz) and [zer0mem](https://twitter.com/zer0mem). The port has found 6 bugs including [CVE-2018-8441](https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2018-8441). Also, BSoDs in WSL: [1](https://twitter.com/yoavalon/status/1102563655743406082), [2](https://twitter.com/NetanelBenSimon/status/1102563950221316096). See [BUGS ON THE WINDSHIELD: FUZZING THE WINDOWS KERNEL](https://www.offensivecon.org/speakers/2020/netanel-ben-simon-yoav-alon.html) presentation. To update descriptions run (assumes `cl` cross-compiler is in PATH): ``` syz-extract -os=windows syz-sysgen ``` `sys/windows/windows.txt` was auto-extracted from windows headers with `tools/syz-declextract`. To build binaries: ``` make fuzzer execprog stress TARGETOS=windows REV=git rev-parse HEAD cl executor\executor_windows.cc /EHsc -o bin\windows_amd64\syz-executor.exe \ -DGIT_REVISION=\"$REV\" \ kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib \ shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib \ winmm.lib rpcrt4.lib Crypt32.lib imm32.lib Urlmon.lib Oleaut32.lib \ Winscard.lib Opengl32.lib Mpr.lib Ws2_32.lib Bcrypt.lib Ncrypt.lib \ Synchronization.lib Shell32.lib Rpcns4.lib Mswsock.lib Mincore.lib \ Msimg32.lib RpcRT4.lib Rpcrt4.lib lz32.lib ``` To run `syz-execprog`: ``` bin\windows_amd64\syz-execprog.exe -executor c:\full\path\to\bin\windows_amd64\syz-executor.exe -stress ``` Windows is supported by only `gce` VMs at the moment. To use `gce`, create a Windows GCE VM, inside of the machine: - Enable serial console debugging (see [this](https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/boot-parameters-to-enable-debugging) for details): ``` bcdedit /debug on bcdedit /dbgsettings serial debugport:1 baudrate:115200 /noumex ``` - Disable automatic restart in `sysdm.cpl -> Advanced -> Startup and Recovery` - Setup sshd with key auth, [these](https://winscp.net/eng/docs/guide_windows_openssh_server) instructions worked for me. Preferably use non-admin user. Save private ssh key. Then shutdown the machine, stop the instance and create an image from the disk. Then start `syz-manager` with config similar to the following one: ``` { "name": "windows", "target": "windows/amd64", "http": ":20000", "workdir": "/workdir", "syzkaller": "/syzkaller", "sshkey": "/id_rsa", "ssh_user": "you", "cover": false, "procs": 8, "type": "gce", "vm": { "count": 10, "machine_type": "n1-highcpu-2", "gce_image": "your-gce-image" } } ``` --- ### Overview (`pkg/coveragedb/README.md`) # Prepare coverage aggregation pipeline Assuming you have the coverage `*.jsonl` files in some bucket: 1. Create BigQuery table. 2. Start data transfers from the bucket to BigQuery table. Coverage merger job is consuming data from BQ table and store aggregations in the Spanner DB. The dots in BQ table name are not supported, thus: 1. For the namespace "upstream" the expected BQ table name is "upstream". 2. For the namespace "android-6.12" the expected BQ table name is "android-6-12". ### Create new BigQuery table for coverage data ```bash bq mk \ --table \ --description "android 6.12" \ --time_partitioning_field timestamp \ --time_partitioning_type DAY \ --require_partition_filter=true \ --clustering_fields file_path,kernel_commit,hit_count \ syzkaller:syzbot_coverage.android-6-12 \ ./pkg/coveragedb/bq-schema.json ``` ### Add new data transfer ```bash bq mk \ --transfer_config \ --display_name=ci-android-6-12-bucket-to-syzbot_coverage \ --params='{"destination_table_name_template":"android-6-12", "data_path_template": "gs://$COVERAGE_STREAM_BUCKET/ci-android-6.12/*.jsonl", "allow_jagged_rows": false, "allow_quoted_newlines": false, "delete_source_files": true, "encoding": "UTF8", "field_delimiter": ",", "file_format": "JSON", "ignore_unknown_values": false, "max_bad_records": "0", "parquet_enable_list_inference": false, "parquet_enum_as_string": false, "preserve_ascii_control_characters": false, "skip_leading_rows": "0", "use_avro_logical_types": false, "write_disposition": "APPEND" }' \ --project_id=syzkaller \ --data_source=google_cloud_storage \ --target_dataset=syzbot_coverage ``` ### List BigQuery data transfers ```bash bq ls \ --transfer_config \ --transfer_location=us-central1 \ --project_id=syzkaller ``` --- ### Overview (`pkg/report/README.md`) # Bugs scoring Until triaged we don't really know the bug impact. But we can learn a lot from the bug title. Syzbot scoring is based on our understanding of what bug class looks historically more impactful. It allows to prioritize the triaging queue. ## Heuristics ### KASAN > KMSAN > KCSAN KASAN detected bugs are typically more dangerous than KMSAN detected bugs. And KMSAN detected bugs are typically more dangerous than KCSAN detected bugs. ### Invalid-free (double-free) >= use-after-free write > use-after-free read. ### KASAN write > KASAN read KASAN write indicates an out-of-bounds or use-after-free write operation. Any uncontrolled write to kernel memory is extremely dangerous because it can corrupt data or code pointers, making it a high-value target for exploitation and leading to system compromise. KASAN read indicates an out-of-bounds or use-after-free read. This is generally less critical. It can crash the system (DoS) or leak sensitive data, but it doesn't provide a direct path for an attacker to execute their own code. ### Memory Safety bugs > DoS bugs. This heuristic establishes a broad priority between two major classes of bugs based on their ultimate impact. Memory Safety bugs: This category includes all the issues mentioned above—use-after-free, double-free, out-of-bounds reads/writes, etc. These are considered more severe because they represent a potential system compromise. A successful exploit can allow an attacker to escalate privileges and gain complete control over the kernel and the entire system. DoS bugs (Denial of Service): This category includes bugs like kernel hangs, crashes, or resource exhaustion (e.g., memory leaks). While they are serious because they disrupt system availability, they typically do not allow an attacker to execute code or steal data. The impact is usually temporary and can be resolved by rebooting the system. They disrupt the service but don't compromise its integrity. ### Information Leaks > Denial of Service (DoS) Kmsan infoleak and other bugs that leak kernel memory are generally more severe than a typical DoS. These leaks can be used to bypass security mitigations like Kernel Address Space Layout Randomization (KASLR), which makes exploiting other vulnerabilities easier. ### Concurrency Issues > Simple DoS Bugs like DataRace and LockdepBug can be more critical than a standard DoS. Data races can lead to unpredictable behavior, including memory corruption, which might be exploitable. LockdepBug indicates potential deadlocks, which can cause a more severe system Hang than a resource-exhaustion DoS. ### KFENCE reports are high priority KFENCE is a lighter-weight memory safety detector compared to KASAN. While it may have a lower performance overhead, the bugs it finds (use-after-free, out-of-bounds) are of the same high-impact nature as those found by KASAN. Therefore, KFENCE detected bugs should be treated with a similar level of urgency as KASAN reports. ### UBSAN reports require careful evaluation The Undefined Behavior Sanitizer (UBSAN) can detect a wide range of issues. Their severity can vary greatly: 1. A shift-out-of-bounds or array-index-out-of-bounds issue can be very severe if it leads to memory corruption. 2. An integer-overflow can also be critical if it results in bypassing security checks and leads to a buffer overflow. 3. Other UBSAN issues might be less critical but still indicate latent bugs that could become problematic. ### LockdepSleeping in Atomic Context is a critical flaw AtomicSleep is a serious bug that can lead to system-wide hangs and instability. This is because holding a spinlock or being in another atomic context while sleeping can cause deadlocks. These are generally more severe than a typical DoS. ### Memory Leaks are a form of DoS MemoryLeak bugs are a type of denial of service where the kernel gradually runs out of memory. While generally less severe than memory corruption, a fast memory leak that can be triggered by an unprivileged user can be a high-impact DoS vector. ### NULL pointer dereference in reads/writes They may be exploitable (see [proof](https://googleprojectzero.blogspot.com/2023/01/exploiting-null-dereferences-in-linux.html )) but the exploitation probability is not clear. Because of this uncertainty they are put to the bottom of the high priority bugs. --- ### Overview (`sys/fuchsia/README.md`) # Fuchsia system call definitions This directory contains the system call definitions for Fuchsia's kernel, Zircon. They are currently updated manually, but we hope that will change Real Soon Now. When/if it does, we'll update this file. The `.fidl` files in the [vDSO directory](https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/vdso/) describe Zircon's system calls in FIDL, and in the comments there are English and C++ descriptions. For every FIDL file, there should be a corresponding `.txt` file in this directory that describes the system calls using [Syzkaller's syzlang language](https://github.com/google/syzkaller/blob/master/docs/syscall_descriptions_syntax.md). --- ### Overview (`syz-cluster/dashboard/README.md`) To update and access the web dashboard during local development: ``` $ make install-dev-config $ make deploy-web-dashboard-dev $ kubectl port-forward service/web-dashboard-service --address 0.0.0.0 50123:80 ``` --- ### Overview (`tools/docker/README.md`) # Docker images We provide a set of Docker images that provide dev environment suitable for syzkaller development/testing. These images are used by CI, but can also be used for [local development](/docs/contributing.md#using-syz-env). - [env](/tools/docker/env/Dockerfile) includes Go/fuchsia/netbsd toolchains, gcloud sdk, C/C++ cross-compilers, make, git and other essential tools. - [old-env](/tools/docker/old-env/Dockerfile) provides essential tools but based on an older disto (ubuntu:16.04). These images are available as `gcr.io/syzkaller/{env,old-env}`, respectively. To download and run locally: ``` docker pull gcr.io/syzkaller/env docker run -it gcr.io/syzkaller/env ``` ## Building Multi-arch Images The `syzbot` and `env` images support multiple architectures (amd64, arm64). To build and push them, we use [Docker buildx](https://docs.docker.com/build/building/multi-platform/) to build a multi-arch image in a way that allows distributing it under one tag name. ### 1. One-time Setup Install the QEMU emulators and create a new builder instance: ```bash # Install QEMU emulators for multi-arch support docker run --privileged --rm tonistiigi/binfmt --install all # Create and bootstrap a new builder docker buildx create --name mybuilder --driver docker-container --bootstrap docker buildx use mybuilder ``` ### 2. Build and Push Once the builder is configured, you can build and push the images. Ensure you are authenticated with Google Cloud: ```bash gcloud auth login && gcloud auth configure-docker ``` **Building the `syzbot` image:** ```bash docker buildx build --platform linux/amd64,linux/arm64 \ -t gcr.io/syzkaller/syzbot \ tools/docker/syzbot \ --push ``` **Building the `env` image:** ```bash docker buildx build --platform linux/amd64,linux/arm64 \ -t gcr.io/syzkaller/env \ tools/docker/env \ --push ``` ## [DEPRECATED] Github Packages Github packages are not supported (if you can't access gcr.io, please contact us). ``` docker tag gcr.io/syzkaller/env [docker.pkg.github.com/google/syzkaller/env](https://docker.pkg.github.com/google/syzkaller/env) docker login [https://docker.pkg.github.com](https://docker.pkg.github.com) docker push [docker.pkg.github.com/google/syzkaller/env](https://docker.pkg.github.com/google/syzkaller/env) ``` --- ### Overview (`tools/syz-aflow/README.md`) # `syz-aflow` - AI Workflow Executor `syz-aflow` is a CLI tool designed to execute `aflow` workflows locally for debugging and testing purposes. ## Building To build `syz-aflow`, use the `syz-env` environment from the repository root: ```bash ./tools/syz-env go build ./tools/syz-aflow ``` This will create a `syz-aflow` binary in the repository root. ## Usage ### Basic Execution To run a workflow, you need to specify the workflow name, an input JSON file, and a working directory: ```bash ./tools/syz-env ./syz-aflow -workflow -input -workdir ``` ### Batch Execution If `-input` is a directory, each `*.json` file in it is executed as a separate task: ```bash ./tools/syz-env ./syz-aflow -workflow seed-gen-file-line -input ./tasks -workdir ./workdir -parallel 4 ``` Results are saved to `/trajectories//`. Completed tasks are skipped on restart. Batch execution is currently only supported for the `seed-gen-file-line` workflow and cannot be combined with `-html` and `-output`. To spread the load over several Gemini API keys, put them one per line into `GOOGLE_API_KEY` (e.g. `GOOGLE_API_KEY="$(cat keys.txt)"`). Each task sticks to one key. Keys only help with rate limits if they belong to different projects. With `-corpus `, all programs executed during the batch run are also saved into the given `corpus.db` (without call properties such as fault injection), e.g. to be used as fuzzing seeds. ### Workflow Inputs `syz-aflow` does not require a standard `syz-manager` configuration file. Instead, it takes a JSON file containing the arguments specific to the workflow you are running. If the workflow needs to perform actions that interact with VMs (like reproducing a crash or testing a patch), it will expect fields in that input JSON that describe the environment (e.g., `Image`, `VM` type and config, `KernelSrc`, etc.). The workflow code then takes these individual arguments and builds the necessary manager configuration programmatically on the fly. #### Example Input for `patching` Workflow ```json { "Syzkaller": "/path/to/syzkaller", "Image": "/path/to/linux/image", "Type": "qemu", "VM": { "count": 1, "cpu": 2, "mem": 2048, "qemu_args": "-machine q35 -enable-kvm -smp 2,sockets=2,cores=1" }, "ReproSyz": "syz_open$dir(0x0, 0x1) ...", "KernelConfig": "@/path/to/linux.config" } ``` Any string field in the input JSON starting with `@` (e.g. `"@/path/to/file"` or `"@./relative/path"`) will be automatically expanded with the contents of that file. Relative paths are resolved relative to the directory of the `-input` JSON file. Literal leading `@` characters can be escaped with `@@` (e.g. `"@@literal"`). ### Flags - `-workflow`: The name of the workflow to execute. - `-input`: Path to a JSON file containing the arguments for the workflow (or a directory of them). - `-parallel`: Number of tasks to run concurrently in batch execution. - `-corpus`: Path to a `corpus.db` to save all executed programs into in batch execution. - `-workdir`: Directory where the workflow can perform checkouts, builds, etc. - `-html`: Path to an HTML file where the execution trajectory will be rendered in real-time. - `-model`: Override the default LLM model. - `-cache-size`: Set the maximum cache size (default "10GB"). - `-download-bug`: Download bug details from the dashboard by ID or ExtID. - `-auth`: Use gcloud auth token when downloading bugs. ### Live Trajectory Visualization You can monitor the execution of the workflow in real-time by using the `-html` flag: ```bash ./tools/syz-env ./syz-aflow -workflow patching -input input.json -workdir ./workdir -html trajectory.html ``` Open the specified HTML file in your browser to see the charts and steps as they execute. --- ### Overview (`tools/syz-covermerger/README.md`) # Covermerger, how it works ## Coverage signal lifetime Syzkaller relies on KCOV to get information about the actual basic blocks executed by kernel code. This tool consumes `line:col` pairs that are coming from `addr2line(pc)` where `pc` is the actual `pc` where we observed KCOV instrumentation hit. ## What do the dashboard numbers mean Understanding the coverage percentages displayed on the dashboard can be tricky. These numbers represent neither pure basic block coverage nor pure line coverage. Instead, the dashboard shows the percentage of lines with coverage signals where at least one basic block attributed to that line was executed. Here's why this matters: A single line of code can contain multiple basic blocks. Covermerger reports a "hit" for a line even if only one out of several basic blocks on that line was covered. For example, if a line has two basic blocks and only one is covered, we report a hit (50% basic block coverage for that line, but 100% line coverage in our dashboard's context). Essentially, dashboard coverage will always be greater than or equal to basic block coverage. ## How this tool works Covermerger's primary function is to merge coverage signals from various kernel versions and configurations. It provides daily, monthly, and quarterly aggregations. ### Merging Process Independent File Processing: Each file is processed on its own. Same Version: If coverage signals come from the exact same version of a file, their hit counts are simply added together. Different Versions: When merging signals from different file versions (e.g., if a file changed multiple times within a month), a diff-tool is used to identify equivalent lines across versions. Projection to Latest Version: All merged signals are projected onto the latest available kernel version within the aggregation period. For the current month, the target file version may change with each merge run. For past months, the latest commit available at the end of that month is used as the target version. Consider a scenario where a file changed three times in a month, and not all lines were included in every build. Covermerger intelligently combines the signals to provide a comprehensive view despite these variations. ## Update frequency Merge jobs are automated via [cron.yaml](/dashboard/app/cron.yaml) and follow this schedule: 1. Daily Aggregations: Updated every day. Batch jobs typically refresh "today" and "yesterday" numbers. 2. Monthly Aggregations: Updated weekly, with the current month's data refreshed 4-5 times per month. 3. Quarterly Aggregations: Also updated weekly. We retain original coverage signals for at least one year, allowing for regeneration of data if needed. --- ### Overview (`tools/syz-db-export/README.md`) # Syzbot DB export Every week syzbot runs syz-db-export to export its databases: 1. [Upstream Linux](https://syzkaller.appspot.com/upstream) db is [here](https://storage.googleapis.com/artifacts.syzkaller.appspot.com/shared-files/repro-export/upstream.tar.gz). 2. Contact us if you want see others. ## Export structure DB currently includes: 1. Bugs descriptions. 2. First C-Reproducer for every bug. It doesn't include: 1. Second+ C-Reproducers for every bug. 2. Syz-Reproducers. 3. Any reproducer related metadata (like triggering requirements). ## How to export more data The best way to see more data exported is to modify the tool and send us PR with your changes. To reproduce locally what syzbot is doing for upstream Linux: ```golang go run ./tools/syz-db-export/... -namespace upstream ``` Extending tools/syz-db-export you can teach syzbot to export more. --- ### Overview (`tools/syz-declextract/README.md`) # syz-declextract ## Linux Kernel (for testing purposes) ``` export KERNEL=$PWD/linux git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $KERNEL cd $KERNEL make CC=clang defconfig ./scripts/config -e FTRACE_SYSCALLS make CC=clang olddefconfig make CC=clang -j`nproc` vmlinux compile_commands.json # kernel has to be built at least once for the script to work ``` ## Running on a single source file ``` ./bin/syz-declextract $KERNEL/fs/read_write.c | less # or any other .c file ``` ## Coverage Data Coverage data (coverage.jsonl) can be obtained from syzbot dashboard using: ``` curl --header "accept-encoding: gzip" https://syzkaller.appspot.com/upstream/coverage?jsonl=1 | gunzip > coverage.jsonl ``` Note: the coverage is tied to a particular kernel commit. For consistency that commit should be used for the rest of the process as well. ## Running on the whole kernel ``` go run ./tools/syz-declextract -config=manager.cfg -coverage coverage.jsonl syz-env make extract SOURCEDIR=$KERNEL ``` The tool caches results of static kernel analysis in manager.workdir/declextract.cache file, and results of the dynamic kernel probing in manager.workdir/interfaces.json file. These can be examined for debugging purposes, and will be reused in future runs if exist (greatly saves time). If the clang tool/kernel has changed, delete these cache files so that they are updated. --- ### Overview (`tools/syz-uncovered-batch/README.md`) # `syz-uncovered-batch` - Uncovered Target Generator `syz-uncovered-batch` extracts candidate uncovered functions and code locations from Syzkaller dashboard coverage data, and generates batch task input files for `syz-aflow`. ## Building To build `syz-uncovered-batch`, run: ```bash ./tools/syz-env go build ./tools/syz-uncovered-batch ``` ## Usage See `main.go` (or run `./syz-uncovered-batch -help`) for all available flags. ### 1. Prepare Base Configuration Create a base template `base.json` defining the VM, kernel source, and image configuration for the target environment: ```json { "AgentName": "syzbot", "Syzkaller": "/path/to/syzkaller", "Image": "/path/to/image", "KernelConfig": "@/path/to/linux.config", "TargetOS": "linux", "TargetArch": "amd64", "Snapshot": true, "Type": "qemu", "VM": { "count": 1, "cpu": 2, "mem": 2048, "qemu_args": "-machine q35 -enable-kvm -smp 2,sockets=2,cores=1" } } ``` ### 2. Generate Tasks Extract uncovered targets for a specific manager and subsystems (e.g. KVM and DRM) and generate 50 task files: ```bash ./tools/syz-env ./syz-uncovered-batch \ -manager ci-upstream-kasan-gce-root \ -base base.json \ -output-dir ./tasks \ -include-paths virt/kvm,drivers/gpu/drm \ -limit 50 ``` ### 3. Run Batch with `syz-aflow` Execute the generated tasks in parallel using `syz-aflow`: ```bash ./tools/syz-env ./syz-aflow \ -workflow seed-gen-file-line \ -input ./tasks \ -workdir ./workdir \ -parallel 4 \ -corpus ``` Results and trajectories will be saved to `./workdir/trajectories/{success,giveup,error}/` and executed programs collected into `./workdir/corpus.db`. --- ### Overview (`dashboard/app/dashboard/README.md`) This dir exists for the purposes of running the dashboard app locally using dev_appserver.py. The production app sets CWD to the root of the syzkaller repository, so paths that refer to static resources in app.yaml look like dashboard/app/static/*. However, dev_appserver.py sets CWD to dashboard/app, so these paths do not work. This dir contains soft link dashboard/app/dashboard/app/static that point to dashboard/app/static, so that dashboard/app/static/* paths work from both locations. --- ### Overview (`dashboard/config/linux/README.md`) This dir contains Linux kernel configs used by [syzbot](/docs/syzbot.md). Configs are auto-generated by [syz-kconf](/tools/syz-kconf/kconf.go) utility from declaration file [main.yml](main.yml) and config [fragments](bits/). [main.yml](main.yml) specifies list of target config files (instances) and their `features`, as well as order of inclusion of config fragments and their constraints. Config fragment is used for an instance if all fragment constraints are present in instance features and all negative constraints (`-foo`) are not present in instance features. For example, the following declaration file: ``` instances: - instance-a: [x86_64, gcc, upstream] - instance-b: [x86_64, gcc, upstream, foo] - instance-c: [arm64, clang, stable] includes: - always.yml: [] - only-upstream.yml: [upstream] - only-clang-arm64.yml: [clang, arm64] - only-upstream-not-foo.yml: [upstream, -foo] ``` specifies 3 target configs (instances) and built from 4 config fragments. Here fragment `always.yml` is included first and for all instances. Fragment `only-upstream.yml` is included next for instances that have `upstream` feature listed (`instance-a` and `instance-b`). Fragment `only-clang-arm64.yml` is included next for instances that have both `clang` and `arm64` listed (`instance-c`). Finally, fragment `only-upstream-not-foo.yml` is included for instances that have `upstream` features but don't have `foo` feature listed (`instance-a`). Config fragments can specify several aspects of the target config file. First, they can specify the kernel repository and revision that will be used for config generation using the following syntax. There must be only one repository specification for each instance. ``` kernel: repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git tag: v5.8-rc7 ``` Then, config fragments can specify kernel `make` commands that will be invoked to create the base config. For example: ``` shell: - make x86_64_defconfig - make kvm_guest.config ``` Shell commands can have constraints similar to includes, for example: ``` shell: - make defconfig: [-nodefconfig] ``` Then, fragments can specify text for verbatim inclusion at the end of the resulting config. `syz-kconf` does not do anything with these string. For example: ``` verbatim: | CONFIG_THIS_CONFIG_DOES_NOT_EXIST_YET=y ``` Finally, fragments can specify individual config options that need to be enabled (or disabled) in the resulting config. For example: ``` config: - CONFIG_WILL_BE_ENABLED - CONFIG_WILL_BE_DISABLED: n - CONFIG_WITH_INTEGER_VALUE: 10 - CONFIG_WITH_STRING_VALUE: "foo" ``` Config options can have constraints for more precise control over target instances (constraints can be combined with values): ``` config: - CONFIG_ONLY_FOR_UPSTREAM: [upstream] - CONFIG_STABLE_AND_ARM64: [stable, arm64] - CONFIG_NOT_CLANG: [-clang] - CONFIG_DISABLED_FOR_GCC: [n, gcc] - CONFIG_STRING_FOO: ["value", foo] ``` Instance features: - `nonoise`: disables auxiliary debug configs (lockdep, stalls, object debug, etc); used on instances on which we ignore all crashes except for the main debug tool (KMSAN/KCSAN/LEAK) and on instances that are too slow (some qemu emulation). - `reduced`: disables some generic (mostly arch-independent) kernel subsystems; used on slow qemu emulation instances since these subsystems are well testing on fast native instances. `syz-kconf` knows about this feature and gives it special meaning: all configs in fragments enabled by this feature are disabled for reduced instances. Some features/constraints are defined by `syz-kconf` itself or have special meaning: - `override` on a config allows to override a previous definition of the same config (otherwise it results in an error) - `optional` on a config disables checking that the resulting kernel config file has the specified value for this config (otherwise it results in an error) - `weak` means both `override` and `optional` - `clang`/`gcc` instance feature controls what compiler is used for config generation - kernel arch name (`x86_64`, `arm64`, etc) instance feature controls architecture of the kernel config - kernel release tag is automatically added to instance features and allows to specify a minimum kernel version for a config (e.g. `v5.5` for a config added in that kernel release) or a maximum kernel version (e.g. `-v5.5` for a config removed in that kernel release) `syz-kconf` generates 2 kernel config files for each instance: one `normal` (or full) and one `baseline`. The `baseline` config has `-base` suffix and has `baseline` and `base-config` features added. The `baseline` config is meant to not contain majority of subsystem configs enabled (should just boot and have debugging configs enabled), however, that is fully controlled by config fragments based on the presence of the `baseline` feature. In order to facilitate even smaller `-base.config` files, the `base-config` feature is automatically set for them. Such configs are intended to be used by the config minimization procedure. The only exception to the declarative nature of the process is USB configs. They are added procedurally, see `addUSBConfigs` function for details. `syz-kconf` switches between `gcc` and `clang` compilers automatically and is supposed to be used with [syz-env](/docs/contributing.md#using-syz-env), which provides correct versions of the compilers. To update kernel configs: - update kernel version in [linux-next.yml](bits/linux-next.yml) and [linux-upstream.yml](bits/linux-upstream.yml) - change config fragments as necessary (e.g. add additional configs to [subsystems.yml](bits/subsystems.yml)) - run `./tools/syz-env make configs SOURCEDIR=/path/to/existing/linux/checkout` * note: it will be mrproper-ed and a number of remotes will be added * see [this](/docs/contributing.md#using-syz-env) on how to setup/use `syz-env` - resolve problems. You can iterate faster focusing on the specific instance problem. * `./tools/syz-env make configs SOURCEDIR=/path/to/existing/linux/checkout INSTANCE=PROBLEM_INSTANCE_NAME` - check in config fragments and changed kernel configs and send a PR - changes will be deployed to `syzbot` within a day after merging --- ### Overview (`executor/_include/flatbuffers/README.md`) This was manually vendored on the version v23.5.26 (which matches the compiler version in the env container) using the following commands: ``` git clone --branch=v23.5.26 --depth=1 --single-branch https://github.com/google/flatbuffers.git cp flatbuffers/LICENSE syzkaller/executor/_include/flatbuffers cp flatbuffers/include/flatbuffers/*.h syzkaller/executor/_include/flatbuffers ``` --- ### Overview (`syz-cluster/tools/db-mgmt/README.md`) How to run DB migrations locally: ``` cd $SYZKALLER/syz-cluster make build-db-mgmt-dev restart-spanner ./run-local.sh db-mgmt migrate ``` --- ### Overview (`syz-cluster/workflow/triage/README.md`) ## Ephemeral disk size usage estimations for Linux ``` $ git fetch origin +refs/tags/torvalds-head:refs/tags/torvalds-head --depth=1 $ du -h . 260M . $ git checkout torvalds-head $ du -h . 2G . $ git fetch origin 2dde18cd1d8fac735875f2e4987f11817cc0bc2c --depth=1 $ du -h . 2.1G . $ git checkout 2dde18cd1d8fac735875f2e4987f11817cc0bc2c $ du -h . 2G . ``` ## Without cloning the repository ``` mkdir ~/shallow-repo ~/shallow-repo/.git ~/shallow-repo/workdir ~/overlayfs mount -t tmpfs -o size=128M tmpfs /root/overlayfs mkdir ~/overlayfs/upper ~/overlayfs/work mount -t overlay overlay -o lowerdir=/kernel-repo,upperdir=/root/overlayfs/upper,workdir=/root/overlayfs/work /root/shallow-repo/.git git --git-dir=/root/shallow-repo/.git --work-tree=/root/shallow-repo/workdir checkout master ``` Needs: ``` securityContext: capabilities: add: ["SYS_ADMIN"] ``` --- ### Overview (`tools/syz-declextract/testdata/README.md`) This dir contains sources of a fake kernel that resembles Linux for testing of the `syz-declextract` tool. For each `*.c` file there 3 golden files: - `*.c.json` with the expected output of the clang tool - `*.c.txt` with the expected syzlang descriptions - `*.c.info` with the expected kernel interface list Testing is done by `tools/syz-declextract` tests. `TestClangTool` invokes the clang tool and verifies `*.c.json` contents. The test requires the clang tool binary specified in the `-bin`, otherwise it skips testing. You also want to run it with `-count=1` flag since the Go tool does not detect changes in the tool binary, and will cache and reuse test results: ``` go test -count=1 ./tools/syz-declextract -bin=llvm/build/bin/syz-declextract ``` `TestDeclextract` verifies `*.c.txt` and `*.c.info` using `*.c.json` files as inputs (it does not involve the clang tool and runs always). All tests also support `-update` flag, which updates the golden files. Generally you don't need to update them manually. Since the test covers multiple packages, it's useful to run coverage as follows: ``` go test -count=1 -coverprofile=/tmp/cover -coverpkg="github.com/google/syzkaller/tools/syz-declextract,github.com/google/syzkaller/pkg/declextract,github.com/google/syzkaller/pkg/clangtool" ./tools/syz-declextract -bin=llvm/build/bin/syz-declextract go tool cover -html /tmp/cover ``` --- ### Overview (`dashboard/config/openbsd/overlays/README.md`) # File hierarchies to lay on top of default images `pkg/build/openbsd.go` replaces the files in the image generated by `create-openbsd-vmm-worker.sh` with the contents of `ci-openbsd-multicore/`. A minor bit of missing connection between `userspaceDir` and this location is managed outside this repository by the local filesystem state on the `ci-openbsd` machine. --- ### Overview (`docs/linux/syzlang/skills/README.md`) # Syzlang Subsystem Skills (Linux) This directory contains component and subsystem-specific knowledge and constraints for LLM agents generating or fixing Linux syzlang programs. ## Format Each skill must be a Markdown (`.md`) file starting with standard YAML frontmatter defining `name` and `description`: ```markdown --- name: kvm description: KVM Virtualization and Guest Constraints (x86/amd64 Focus) --- # KVM Virtualization and Guest Constraints (x86/amd64 Focus) ... instructions ... ``` ## How Agents Use Skills In AFlow, available skills are dynamically listed in `SkillsPrompt(osTarget)`. Agents can inspect any skill using `read-syz-spec` with either a short name (`kvm.md` or `skills/kvm.md`) or full relative path (`docs/linux/syzlang/skills/kvm.md`). --- ### Overview (`pkg/ifuzz/arm64/gen/json/README.md`) arm64.json is taken from the Go Language repository: https://tip.golang.org/src/cmd/vendor/golang.org/x/arch/arm64/arm64asm/inst.json Please see the LICENSE file (taken from https://tip.golang.org/src/cmd/vendor/golang.org/x/arch/LICENSE) for the relevant copyright notice and disclaimers. --- ### Adding New Os Support (`docs/adding_new_os_support.md`) # Adding new OS support Here are the common parts of syzkaller to edit in order to make syzkaller support a new OS kernel. However, there may be some specific changes that will be required for a given kernel (for example, gathering coverage from a given kernel, or some errors that might pop up and give a hint about what to tweak). ## syz-executor For each OS, there is this file `executor/executor_GOOS.h` where GOOS is the OS name. This file contains two important functions: - `os_init` which is responsible for mapping a virtual address space for the calling process, - `execute_syscall` which is responsible for executing system calls for a particular OS kernel. These two functions, are called in `executor/executor.cc`, which is mainly responsible for executing the syscalls programs, and managing the threads in which the programs run. `executor_GOOS.h` also contains functions related to that operating system such as functions that allow it to gather coverage information, detect bitness, etc. (Example: [executor_linux.h](/executor/executor_linux.h) ). The intended function will be called according to the target kernel as defined by the macros in the `executor/executor.cc` file. ## Build files `pkg/` - The OS name is added to `pkg/build/build.go` along with the supported architecture - Creating a file that builds the image for the targeted kernel under `pkg/build/`. This file contains functions for configuring the build of the bootable image, for building it, and for generate SSH keys which will be used by Syzkaller in order to access the VM. There is a file per each of the supported OSes by Syzkaller where the name pattern is `GOOS.go`. - Adding the given target to the `Makefile`. ## Report files `pkg/report/` Creating a file that reports build errors for the targeted kernel under `pkg/report/`. There is a file per each of the supported OSes by Syzkaller where the name pattern is `GOOS.go`. ## Editing `pkg/host/` - implement `isSupported` function that returns true for a supported syscall, it is located under `pkg/host/GOOS`. ## Creating a file under `sys/GOOS/` Creating a file `init.go` for the targeted kernel under `sys/GOOS/`that included the function `initTarget` that initializes the target and the different supported architectures. ## Editing `sys/syz-extract` Adding the new kernel name with already existing supported kernels to the file `sys/syz-extract/extract.go`. ## Editing `sys/targets` Adding the new kernel name with already existing supported kernels to the file `targets.go` which is located under`sys/targets`. ## Editing `vm/qemu` Adding the new kernel name with already existing supported kernels to the file `qemu.go` which is located under `vm/qemu`. ## Syzkaller description & pseudo-syscalls Check [descriptions](/docs/syscall_descriptions.md), and [pseudo-syscalls](/docs/pseudo_syscalls.md). --- ### Bisect (`docs/bisect.md`) # Syz-bisect `syz-bisect` program can be used to bisect culprit and fix commits for crashes found by syzkaller. It can also identify configuration options that are triggers for the crash. ## Usage Build `syz-bisect` with `make bisect`. During bisection different compilers depending on kernel revision are used. These compilers are available [here](https://storage.googleapis.com/syzkaller/bisect_bin.tar.gz). Install ccache to speed up kernel compilations during bisecton. Create user-space (chroot) using [create-image.sh](../tools/create-image.sh) Create a config file with following lines adjusted for your environment: ```json { "bin_dir": "/home/syzkaller/bisect_bin", "ccache": "/usr/bin/ccache", "compiler_type": "gcc", "kernel_repo": "git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git", "kernel_branch": "master", "syzkaller_repo": "https://github.com/google/syzkaller", "userspace": "/home/syzkaller/image/chroot", "kernel_config": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream-apparmor-kasan.config", "kernel_baseline_config": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream-apparmor-kasan-base.config", "syzctl": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream.sysctl", "cmdline": "/home/syzkaller/go/src/github.com/google/syzkaller/dashboard/config/linux/upstream.cmdline", "manager": { "name" : "bisect", "target": "linux/amd64", "http": "127.0.0.1:56741", "workdir": "/home/syzkaller/workdir", "kernel_obj": "/home/syzkaller/linux", "image": "/home/syzkaller/workdir/image/image", "sshkey": "/home/syzkaller/workdir/image/key", "syzkaller": "/home/syzkaller/go/src/github.com/google/syzkaller_bisect", "procs": 8, "type": "qemu", "kernel_src": "/syzkaller/linux", "vm": { "count": 4, "kernel": "/home/syzkaller/linux/arch/x86/boot/bzImage", "cpu": 2, "mem": 2048, "cmdline": "root=/dev/sda1 rw console=ttyS0 kaslr crashkernel=512M minnowboard_1:eth0::: security=none" } } } ``` And run bisection with `bin/syz-bisect -config vm_bisect.cfg -crash /syzkaller/workdir/crashes/03ee30ae11dfd0ddd062af26566c34a8c853698d`. `Syz-bisect` is expecting finding repro.cprog or repro.prog in given crash directory. It will also utilize repro.opts, but it's not mandatory. ## Additional Arguments `-syzkaller_commit` use this if you want to use specific version of syzkaller `-kernel_commit` kernel commit where crash is known to reproduce. You want to use this when bisecting fixing commit `-fix` use this if you want to bisect a fixing commit. ## Output It takes some time, but after `syz-bisect` completes it dumps out it's results into console It also stores results into files in given crash directory: `cause.commit` commit identified causing the crash or text "the crash already happened on the oldest tested release" `fix.commit` commit identified fixing the crash or text "the crash still happens on HEAD" `cause.config` config options identified working as one trigger for the crash `original.config, baseline.config, minimized.config` config files used in config bisection --- ### Ci (`docs/ci.md`) # Continuous integration fuzzing [syz-ci](../syz-ci/) command provides support for continuous fuzzing with syzkaller. It runs several syz-manager's, polls and rebuilds images for managers and polls and rebuilds syzkaller binaries. --- ### Configuration (`docs/configuration.md`) # Configuration The operation of the syzkaller `syz-manager` process is governed by a configuration file, passed at invocation time with the `-config` option. This configuration can be based on the [example](/pkg/mgrconfig/testdata/qemu-example.cfg); the file is in JSON format and contains the the [following parameters](/pkg/mgrconfig/config.go). --- ### Contributing (`docs/contributing.md`) # How to contribute to syzkaller If you want to contribute to the project, feel free to send a pull request following the [guidelines](contributing.md#guidelines) below. In case this is your first pull request to syzkaller, you will need to [sign Google CLA](https://cla.developers.google.com/) and add yourself to [AUTHORS](/AUTHORS) _AND_ [CONTRIBUTORS](/CONTRIBUTORS) files in the first commit. ## What to work on Extending/improving [system call descriptions](syscall_descriptions.md) is always a good idea. Unassigned issues from the [bug tracker](https://github.com/google/syzkaller/issues) are worth doing, but some of them might be complicated. To contribute code or syscall descriptions, at the very least you need to be able to build and run syzkaller, see the instructions [here](/docs/setup.md). ## Guidelines If you want to work on something non-trivial, please briefly describe it on the [syzkaller@googlegroups.com](https://groups.google.com/forum/#!forum/syzkaller) mailing list first, so that there is an agreement on the high level approach/design and no duplication of work between contributors. Split large changes into smaller, logically cohesive commits. Small commits are much easier and faster to review and iterate on. Everything that can be reasonably tested should be tested. Provide enough documentation for other users to use the new feature. Keep the style of the code, tests, comments, docs, log/error messages consistent with the existing style. Continuous Integration (CI) system runs a number of tests and some [opinionated] style checks. They need to pass. You can test locally with `make presubmit`, if you don't have some prerequisites installed, you may use `syz-env` (see below). ### Commits Commit messages should follow the following template: ``` dir/path: one-line description Extended multi-line description that includes the problem you are solving and how it is solved. ``` `dir/path` is a relative path to the main dir this commit changes (look at examples in the [commit history](https://github.com/google/syzkaller/commits/master)). If several packages/dirs are significantly affected, then the following format is allowed: ``` dir1/path1, dir2/path2: one-line description ``` Though, dirs should not be included if they have only minor changes. For pervasive changes the following format is allowed: ``` all: one-line description ``` Please pay attention to punctuation. In particular: - `one-line description` should *not* start with a Capital letter. - There is *no dot* at the end of `one-line description`. - `Extended multi-line description` is full English sentences with Capital letters and dots. Commit message line length is limited to 120 characters. Also: - If your commit fixes an issue, please include `Fixes #NNN` line into commit message (where `NNN` is the issue number). This will auto-close the issue. If you need to mention an issue without closing it, add `Update #NNN`. - For syscall descriptions `*.const` files are checked-in with the `*.txt` changes in the same commit. ### Pull requests - Rebase your working branch onto the master branch before sending a pull request to avoid merge conflicts. - Run `make presubmit` and ensure that it passes before sending a PR. This may require some additional packages to be installed (try `sudo make install_prerequisites`). - Provide a brief high-level description in the pull request title. The pull request text is mostly irrelevant, all the details should be in the commit messages. - If you're asked to add some fixes to your pull request, please squash the fixes into the old commits. ### How to create a pull request on Github - First, you need your own git fork of syzkaller repository. Navigate to [github.com/google/syzkaller](https://github.com/google/syzkaller) and press `Fork` button in the top-right corner of the page. This will create `https://github.com/YOUR_GITHUB_USERNAME/syzkaller` repository. - Checkout main syzkaller repository if you have not already. The simplest way to do it is to run `git clone https://github.com/google/syzkaller`, this will checkout the repository in the current working directory. - Remember to `export PATH=$GOPATH/bin:$PATH` if you have not already. - Then add your repository as an additional origin: ```shell cd syzkaller git remote add my-origin https://github.com/YOUR_GITHUB_USERNAME/syzkaller.git git fetch my-origin git checkout -b my-branch my-origin/master ``` This adds git origin `my-origin` with your repository and checks out new branch `my-branch` based on `master` branch. - Change/add files as necessary. - Commit changes locally. For this you need to run `git add` for all changed files, e.g. `git add sys/linux/sys.txt`. You can run `git status` to see what files were changed/created. When all files are added (`git status` shows no files in `Changes not staged for commit` section and no relevant files in `Untracked files` section), run `git commit` and enter commit description in your editor. - Run tests locally (`make install_prerequisites` followed by `make presubmit`). - Push the commit to your fork on github with `git push my-origin my-branch`. - Navigate to [github.com/google/syzkaller](https://github.com/google/syzkaller) and you should see green `Compare & pull request` button, press it. Then press `Create pull request`. Now your pull request should show up on [pull requests page](https://github.com/google/syzkaller/pulls). - If you don't see `Create pull request` button for any reason, you can create pull request manually. For that navigate to [pull requests page](https://github.com/google/syzkaller/pulls), press `New pull request`, then `compare across forks` and choose `google/syzkaller`/`master` as base and `YOUR_GITHUB_USERNAME/syzkaller`/`my-branch` as compare and press `Create pull request`. - If you decided to rebase commits in `my-branch` (e.g. to rebase them onto updated master) after you created a pull-request, you will need to do a force push: `git push -f my-origin my-branch`. ### Using syz-env Developing syzkaller requires a number of tools installed (Go toolchain, C/C++ cross-compilers, golangci-lint, etc). Installing all of them may be cumbersome, e.g. due to broken/missing packages. [syz-env](/tools/syz-env) provides a working hermetic development environment based on a Docker container. If you don't yet have Docker installed, see [documentation](https://docs.docker.com/engine/install), in particular regarding enabling [sudo-less](https://docs.docker.com/engine/install/linux-postinstall) Docker (Googlers see go/docker). It's recommended to create an alias for `syz-env` script: ``` alias syz-env="$(go env GOPATH)/src/github.com/google/syzkaller/tools/syz-env" ``` Then it can be used to wrap almost any make invocation as: ``` syz-env make format syz-env make presubmit syz-env make extract SOURCEDIR=$(readlink -f ~/linux) ``` Or other commands/scripts, e.g.: ``` syz-env go test -short ./pkg/csource ``` Or you may run the shell inside of the container with just `syz-env` and look around. To update `syz-env` container to the latest version do: ``` bash docker pull gcr.io/syzkaller/env ``` You can also build the container from the respective `Dockerfile` by setting the `SYZ_ENV_BUILD` environment variable, i.e.: ``` SYZ_ENV_BUILD=1 syz-env ``` This can be useful to test local changes that have not been pushed to the registry yet. ### Using [act](https://github.com/nektos/act) .github/workflows has more tests compared to `syz-env make presubmit`. To have the same tests as the workflow, we can run these workflow jobs locally. ``` # install act make act # list all jobs bin/act -l # run all jobs bin/act # run job with name build bin/act -j build ``` --- ### Coverage (`docs/coverage.md`) # Coverage `syzkaller` uses [sanitizer coverage (tracing mode)](https://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs) and [KCOV](https://www.kernel.org/doc/html/latest/dev-tools/kcov.html) for coverage collection. Sanitizer coverage is also supported by `gcc` and `KCOV` is supported by some other OSes. Note: `gVisor` coverage is completely different. Coverage is based on tracing `coverage points` inserted into the object code by the compiler. A coverage point generally refers to a [basic block](https://en.wikipedia.org/wiki/Basic_block) of code or a [CFG edge](https://en.wikipedia.org/wiki/Control-flow_graph) (this depends on the compiler and instrumentation mode used during build, e.g. for `Linux` and `clang` the default mode is CFG edges, while for `gcc` the default mode is basic blocks). Note that coverage points are inserted by the compiler in the middle-end after a significant number of transformation and optimization passes. As the result coverage may poorly relate to the source code. For example, you may see a covered line after a non-covered line, or you may not see a coverage point where you would expect to see it, or vice versa (this may happen if the compiler splits basic blocks, or turns control flow constructs into conditional moves without control flow, etc). Assessing coverage is still generally very useful and allows to understand overall fuzzing progress, but treat it with a grain of salt. See [this](linux/coverage.md) for Linux kernel specific coverage information. ## Web Interface When clicking on `cover` link you get view showing each directory located in your kernel build directory. It's showing either percentage number `X% of N` or `---`. `X% of N` means that `X%` of `N` coverage points are covered so far, . `---` indicates there is no coverage in that directory. Directory can be clicked and you get view on files and possible subdirectories. On each source code file there is again either `---` or coverage percentage. If you click on any C files you will get source code view. There is certain coloring used in the source code view. Color definitions can be found in [coverTemplate](/pkg/cover/report.go#L504). Coloring is described below. If you click on percentage number of any listed source file you will get cover percentage for each function in that source file. ### Covered: black (#000000) All PC values associated to that line are covered. There is number on the left side indicating how many programs have triggered executing the PC values associated to this line. You can click on that number and it will open last executed program. Example below shows how single line which is fully covered is shown. ### Both: orange (#c86400) There are several PC values associated to the line and not all of these are executed. Again there is number left to the source code line that can clicked to open last program triggering associated PC values. Example below shows single line which has both executed and non-executed PC values associated to it. ### Uncovered: red (#ff0000) Line is uncovered. Function (symbol) this line is in is executed and one of the PC values associated to this line. Example below shows how single line which is not covered is shown. ### Not executed or non-instrumented code: grey (#505050) If an entire function was never executed, all its lines are shown in grey. This color is also used for lines that are not instrumented or do not generate code (e.g., comments, macros, or empty lines). Note that if a function was executed but some lines within it were not covered, those lines are shown in red, not grey. ## syz-cover There is small utility in syzkaller repository to generate coverage report based on raw coverage data. This is available in [syz-cover](/tools/syz-cover) and can be built by: ``` bash GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o ./bin/syz-cover github.com/google/syzkaller/tools/syz-cover ``` Raw coverage data can be obtained from running `syz-manager` by: ``` bash wget http://localhost:/rawcover ``` Now this raw cover data can be fed to `syz-cover` to generate coverage report: ``` bash ./bin/syz-cover --config ./rawcover ``` You can also export CSV file containing function coverage by: ``` bash ./bin/syz-cover --config --exports funccover ./rawcover ``` You can export a JSON file containing line coverage info by: ```bash ./bin/syz-cover --config --exports json ./rawcover ``` --- ### Db (`docs/db.md`) # syz-db `syz-db` program can be used to manipulate corpus.db databases that are used by syz-managers. ## Build Build `syz-db` with `make db` or by changing to `tools/syz-db` and run `go build`. ## Options `syz-db` currently overs the following generic arguments: ```shell -arch string target arch -os string target OS -version uint database version -vv int verbosity ``` That can be used with ``` syz-db pack dir corpus.db ``` to pack a database ``` syz-db unpack corpus.db dir ``` to unpack a database. A file containing performed syscalls will be returned. ``` syz-db merge dst-corpus.db add-corpus.db* add-prog* ``` to merge databases. No additional file will be created: The first file will be replaced by the merged result. ``` syz-db bench corpus.db ``` to run a deserialization benchmark. For example: ``` syz-db -os=linux -arch=amd64 bench corpus.db ``` could give an output like ``` allocs 123 MB (123 M), next GC 123 MB, sys heap 123 MB, live allocs 123 MB (123 M), time 324s. ``` --- ### Headerparser Usage (`docs/headerparser_usage.md`) ## Headerparser headerparser is a tool that assists in writing device system call descriptions for syzkaller. In order to make syzkaller smarter when it comes to fuzzing a device node, you can provide it with information about the ioctl argument struct types it expects. However, in certain cases the number of argument struct types might be high, increasing the amount of manual effort that goes into writing the description files for the struct types. In order to ease the effort of writing ioctl argument type description files, headerlib does a best-effort job at generating them for you. You will still need to manually select the appropriate syzkaller data type from the list of types [here](/docs/syscall_descriptions_syntax.md). ## Dependencies Headerlib uses pycparser. You can install pycparser using pip. ```shell $ pip install pycparser ``` ## Using headerparser ```shell $ python headerparser.py --filenames=./test_headers/th_b.h B { B1 len|fileoff|flags|intN #(unsigned long) B2 len|fileoff|flags|intN #(unsigned long) } struct_containing_union { something len|fileoff|flags|int32 #(int) a_union.a_char ptr[in|out, string]|ptr[in, filename] #(char*) a_union.B_ptr ptr|buffer|array #(struct B*) } ``` You can copy paste the content underneath the `Structure Metadata` over to your syzkaller device description. ## Something breaks Let us try parsing `test_headers/th_a.h` header file to generate argument structs. ```shell $ python headerparser.py --filenames=./test_headers/th_a.h ERROR:root:HeaderFilePreprocessorException: /tmp/tmpW8xzty/source.o:36:2: before: some_type $ python headerparser.py --filenames=./test_headers/th_a.h --debug DEBUG:GlobalHierarchy:load_header_files : ['./test_headers/th_a.h'] DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor._mktempfiles: sourcefile=/tmp/tmpbBQYhR/source.cobjectfile=/tmp/tmpbBQYhR/source.o DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor.execute: cp ./test_headers/th_a.h /tmp/tmpbBQYhR DEBUG:HeaderFilePreprocessor:HeaderFilePreprocessor.execute: gcc -I. -E -P -c /tmp/tmpbBQYhR/source.c > /tmp/tmpbBQYhR/source.o ERROR:root:HeaderFilePreprocessorException: /tmp/tmpbBQYhR/source.o:36:2: before: some_type ``` From the error message, we can see that the error occurs as pycparser is not aware of the type `some_type`. We can resolve this by making pycparser aware of the unknown type. In order to do this, we supply headerparser with a include file that contains C declarations and includes that can fix the parse error. ```shell $ cat > include_file typedef int some_type; $ python headerparser.py --filenames=./test_headers/th_a.h --include=./include_file A { B_item ptr|buffer|array #(struct B*) char_ptr ptr[in|out, string]|ptr[in, filename] #(char*) an_unsigned_int len|fileoff|int32 #(unsigned int) a_bool _Bool #(_Bool) another_bool _Bool #(_Bool) var some_type #(some_type) } ``` --- ### Hub (`docs/hub.md`) # Connecting several managers via Hub `syz-hub` program can be used to connect several `syz-manager`'s together and allow them to exchange programs. Build `syz-hub` with `make hub`. Then create a config file along the lines of: ``` { "http": ":80", "rpc": ":55555", "workdir": "/syzkaller/workdir", "clients": [ {"name": "manager1", "key": "6sCFsJVfyFQVhWVKJpKhHcHxpCH0gAxL"}, {"name": "manager2", "key": "FZFSjthHHf8nKm2cqqAcAYKM5a3XM4Ao"}, {"name": "manager3", "key": "fTrIBQCmkEq8NsvQXZiOUyop6uWLBuzf"} ] } ``` And start it with `bin/syz-hub -config hub.cfg`. Then add the following additional parameters to `syz-manager` config files of each manager: ``` "name": "manager1", "hub_client": "manager1", "hub_addr": "1.2.3.4:55555", "hub_key": "6sCFsJVfyFQVhWVKJpKhHcHxpCH0gAxL", ``` And start managers. Once they triage local corpus, they will connect to the hub and start exchanging inputs. Both hub and manager web pages will show how many inputs they send/receive from the hub. --- ### Internals (`docs/internals.md`) # How syzkaller works Generic description of how syzkaller works are [below](internals.md#overview). Linux kernel specific internals can be found [here](linux/internals.md). ## Overview The process structure for the syzkaller system is shown in the following diagram; red labels indicate corresponding configuration options. `syz-manager` is responsible for: - Starting/restarting/monitoring VM instances. - The actual fuzzing process (input generation, mutation, minimization, etc.). - Persistent corpus and crash storage. It runs on a host with a stable kernel which does not experience white-noise fuzzer load. `syz-manager` starts `syz-executor` processes (one inside each VM). `syz-executor`s communicate with `syz-manager` over RPC to receive the programs that must be executed and to report back the results (error statuses, collected coverage, etc.). To execute programs, `syz-executor` starts transient subprocesses. Each transient subprocess executes a single input (a sequence of syscalls). It is designed to be as simple as possible (to not interfere with fuzzing process), written in C++, compiled as static binary and uses shared memory for communication. ## Syscall descriptions The `syz-manager` process generates programs based on syscall descriptions described [here](syscall_descriptions.md). ## Coverage Syzkaller is a coverage-guided fuzzer. The details about coverage collection can be found [here](coverage.md). ## Crash reports When `syzkaller` finds a crasher, it saves information about it into `workdir/crashes` directory. The directory contains one subdirectory per unique crash type. Each subdirectory contains a `description` file with a unique string identifying the crash (intended for bug identification and deduplication); and up to 100 `logN` and `reportN` files, one pair per test machine crash: ``` - crashes/ - 6e512290efa36515a7a27e53623304d20d1c3e - description - log0 - report0 - log1 - report1 ... - 77c578906abe311d06227b9dc3bffa4c52676f - description - log0 - report0 ... ``` Descriptions are extracted using a set of [regular expressions](/pkg/report/). This set may need to be extended if you are using a different kernel architecture, or are just seeing a previously unseen kernel error messages. `logN` files contain raw `syzkaller` logs and include kernel console output as well as programs executed before the crash. These logs can be fed to `syz-repro` tool for [crash location and minimization](reproducing_crashes.md), or to `syz-execprog` tool for [manual localization](reproducing_crashes.md#from-execution-logs). `reportN` files contain post-processed and symbolized kernel crash reports (e.g. a KASAN report). Normally you need just 1 pair of these files (i.e. `log0` and `report0`), because they all presumably describe the same kernel bug. However, `syzkaller` saves up to 100 of them for the case when the crash is poorly reproducible, or if you just want to look at a set of crash reports to infer some similarities or differences. There are 3 special types of crashes: - `no output from test machine`: the test machine produces no output whatsoever - `lost connection to test machine`: the ssh connection to the machine was unexpectedly closed - `test machine is not executing programs`: the machine looks alive, but no test programs were executed for long period of time Most likely you won't see `reportN` files for these crashes (e.g. if there is no output from the test machine, there is nothing to put into report). Sometimes these crashes indicate a bug in `syzkaller` itself (especially if you see a Go panic message in the logs). However, frequently they mean a kernel lockup or something similarly bad (here are just a few examples of bugs found this way: [1](https://groups.google.com/d/msg/syzkaller/zfuHHRXL7Zg/Tc5rK8bdCAAJ), [2](https://groups.google.com/d/msg/syzkaller/kY_ml6TCm9A/wDd5fYFXBQAJ), [3](https://groups.google.com/d/msg/syzkaller/OM7CXieBCoY/etzvFPX3AQAJ)). --- ### Issue Template (`docs/issue_template.txt`) Issues are for bugs in syzkaller itself, for everything else please contact syzkaller@googlegroups.com mailing list. Before filing an issue please check the existing issues and the mailing list (https://groups.google.com/forum/#!forum/syzkaller). Please provide syzkaller revision, Go version (output of go version), relevant information about host and target systems. If syz-manager is not working overall, please provide output of syz-manager with -debug flag. --- ### Kernel Config Update Guidelines (`docs/kernel_config_update_guidelines.md`) # Updating Syzkaller Kernel Configurations Guidelines These guidelines describe the process for updating the `syzkaller` dashboard configuration files to support new Linux kernel versions. ## Objective Successfully update configuration fragments (`.yml`) and generate corresponding `.config` files for target kernel versions (e.g., `linux-upstream` and `linux-next`). ## Principles - **Maximize Coverage**: We want to maximize the coverage of the new code and maintain the coverage of the old code. - **Enable New Options**: If new options are introduced with the kernel version, we want them to be enabled if they help to shift left the bug detection or directly help to detect bugs. If they help with fuzzing but were not enabled, explain why in the commit message. These changes should be grouped into a commit titled `dashboard/config/linux: enable more configs`. - **Reasoned Disabling**: If some options are not available anymore, disable them adding a comment in the YAML file referencing the kernel commit that removed, renamed, or moved them. Prefer formats like: `# Removed in ""`, `# Renamed to in ""`, or `# Migrated to in ""`. If the specific commit cannot be found after a reasonable search, use the format: `# Removed in vX.Y (exact commit not found)`. If the option is not mentioned in any generated config files (i.e., it was not enabled in any instance), you can remove it from the YAML file entirely instead of disabling it. In this case, explain the removal in the commit message. - **Documenting Constraints**: If you disable an option for a specific architecture (e.g., `-riscv`) or configuration (e.g., `-kmsan`), or if you make some config 'weak', add a comment explaining this decision. - **Explicit Constraints**: Prefer explicitly enumerating the supported platforms and/or kernel versions instead of using `[optional]` or `[weak]` constraints when possible. If the option was added in `linux-next`, use the tag `[linux-next]`. If the specific kernel version is not known but it is in a released kernel, use `[weak]` and add a comment explaining it. Also use `[weak]` or `[optional]` if the config option has dependencies that might not be met in all configurations (e.g., depends on specific networking features or hardware support). **Do not silently make a failing config `weak` without investigation. Prefer identifying the specific reason (e.g., architecture limitation or kernel version change) and using explicit exclusions if possible.** - **File-Level Exclusion**: If all options in a configuration fragment are disabled for certain kernel versions or instances, prefer excluding the whole file in `main.yml` using negative tags (e.g., `-upstream`) instead of adding version constraints to every individual option in the fragment. Add an explanatory comment above the inclusion line in `main.yml` to explain why it is disabled. - **Explanation Headers**: Every new file created in `dashboard/config/linux/bits` requires an explanation header describing the purpose of the file, why we need it, and how it is used. - **No Manual Edits**: Do NOT manually update the generated `.config` files. They MUST always be generated by running `make configs`. ## Prerequisites - A local clone of the `linux` kernel repository. - A local clone of the `syzkaller` repository. - Docker installed for running `syz-env`. ## Procedure ### 1. Update Kernel Commits Modify the corresponding YAML files in `dashboard/config/linux/bits/` to point to the desired tags. Use tags instead of commit hashes when updating these files. - `linux-upstream.yml` - `linux-next.yml` ### 2. Run Config Generation Execute the following command from the root of the `syzkaller` repository: ```bash CI=true ./tools/syz-env make configs SOURCEDIR=/path/to/linux/source ``` *Note: Replace `/path/to/linux/source` with the absolute path to your local Linux kernel directory.* ### 3. Troubleshooting Common Issues #### A. Kconfig Parser Limitations (`syz-kconf`) The `syz-kconf` parser in `pkg/kconfig` may not support advanced Kconfig syntax such as `if` in `depends on` statements. - **Solution:** Implement support for the missing syntax in `pkg/kconfig/parser.go`. Do NOT modify Kconfig files in the kernel source. #### B. Missing Config Options in Final Config If an option specified in a YAML fragment is not present in the generated `.config`, `make configs` will fail. - **Check Dependencies:** Verify if the option depends on another option not enabled, or if it is not supported on the target architecture or kernel version. - **Version Constraints:** Add version constraints to the option in the YAML file, e.g., `[-v7.1]` to disable it for versions starting from `v7.1`. - **Weak Constraints:** If the option is optional and may not be present in all versions, make it weak: `[value, weak]`. This prevents failure if the option is missing. **Avoid using `weak` just to bypass errors. Investigate why the option is missing or failing to be set. If it is due to architecture or version specific behavior, use explicit exclusions (e.g. `[-arm]`) instead of `weak`.** #### C. Git Fetch Failures If `syz-kconf` fails to checkout a repository or commit (e.g., Android kernels): - **Diagnosis:** Check the error message to find the auto-generated remote name (often a hash). - **Workaround:** Manually run `git fetch ` in the kernel source directory to pull all objects, then retry `make configs`. ### 4. Finalizing Changes Once `make configs` completes successfully: 1. Stage and commit the changes to YAML files first. 2. Stage and commit the generated `.config` files as the VERY LAST commit in the PR. The commit message MUST be `dashboard/config/linux: update generated configs`. 3. Push to a separate branch and create a Pull Request. --- ### Kfuzztest (`docs/kfuzztest.md`) # KFuzzTest Integration With syzkaller KFuzzTest, introduced initially in [this RFC](https://lore.kernel.org/all/20250813133812.926145-1-ethan.w.s.graham@gmail.com/) is a framework for exposing internal kernel functions to a userspace fuzzing engine like syzkaller. As the kernel docs put it: > The Kernel Fuzz Testing Framework (KFuzzTest) is a framework designed to > expose internal kernel functions to a userspace fuzzing engine. > > It is intended for testing stateless or low-state functions that are difficult > to reach from the system call interface, such as routines involved in file > format parsing or complex data transformations. This provides a method for > in-situ fuzzing of kernel code without requiring that it be built as a > separate userspace library or that its dependencies be stubbed out. This document introduces how syzkaller integrates with KFuzzTest. ## Getting Started Firstly, ensure that the KFuzzTest patch series has been applied to your Linux tree. As of the 26th of Semptember 2025, the most up-to-date version can be found in [this Linux Kernel patch series](https://lore.kernel.org/all/20250919145750.3448393-1-ethan.w.s.graham@gmail.com/). Once this is done, KFuzzTest targets can be defined on arbitrary kernel functions using the `FUZZ_TEST` macro as described in the kernel docs in `Documentation/dev-tools/kfuzztest.rst`. ### Configuration Options Ensure that the following KConfig options are enabled for your kernel image: - `CONFIG_DEBUG_FS` (used as a communication interface by KFuzzTest). - `CONFIG_DEBUG_KERNEL`. - `CONFIG_KFUZZTEST`. It is also **highly** recommended to enable the following KConfig options for more effective fuzzing. - `CONFIG_KASAN` (catch memory bugs such as out-of-bounds-accesses). - `CONFIG_KCOV` (to enable coverage guided fuzzing). ## Fuzzing KFuzzTest Targets Syzkaller implements three ways to fuzz KFuzzTest targets: 1. `syz-manager` integration with static targets 2. `syz-manager` with dynamic targets 3. `syz-kfuzztest`: a standalone tool that runs inside a VM, discovers KFuzzTest targets dynamically, and fuzzes them. ### 1. `syz-manager` with static targets Configuration for this method is identical to `syz-manager`, and is designed to make it easy to integrate KFuzzTest fuzzing into existing continuous fuzzing deployments. One must first write a syzlang description for the KFuzzTest target(s) of interest, for example in `/sys/linux/my_kfuzztest_target.txt`. Each target should have the following format: ``` some_buffer { buf ptr[inout, array[int8]] buflen len[buf, int64] } kfuzztest_underflow_on_buffer(name ptr[in, string["test_underflow_on_buffer"]], data ptr[in, some_buffer], len bytesize[data], buf ptr[in, array[int8, 65536]]) (kfuzz_test) ``` Where: - The first argument should be a string pointer to the name of the fuzz target, i.e,. the name of its `debugfs` input directory in the kernel. - The second should be a pointer to a struct of the type that the fuzz target accepts as input. - The third should be the size in bytes of the input argument. - The call is annotated with attribute `kfuzz_test`. The final `buf` argument is a buffer of size `KFUZZTEST_MAX_INPUT_SIZE = 16 * PAGE_SIZE` and is used internally to ensure that enough space is available in a program for the entire flattened input that is sent into a KFuzzTest target. For more information on writing syzkaller descriptions attributes, consult the [syscall description](syscall_descriptions.md) and [syscall description syntax](syscall_descriptions_syntax.md) documentation files. To facilitate the tedious task of writing `syz_kfuzztest_run` descriptions, a tool (`tools/kfuzztest-gen`) is provided to automatically generate these from a `vmlinux` binary. One can run the tool and paste the output into a syzlang file. ```sh go run ./tools/kfuzztest-gen --vmlinux=path/to/vmlinux ``` After writing these descriptions to a file under the `/sys/linux/` directory (for example, `/sys/linux/my_fuzz_targets.txt`), they need to be compiled with `make descriptions`. Finally, the targets can be enabled in `syz-manager` config file in the `enable_syscalls` field, e.g. ```json { "enable_syscalls": [ "syz_kfuzztest_run$test_underflow_on_buffer" ] } ``` ### 2. `syz-manager` with dynamic discovery This feature greatly reduces the amount of setup needed for fuzzing KFuzzTest targets, by discovering them all dynamically at launch. This approach is considered less stable than the previous as it involves generating descriptions for KFuzzTest targets without human input and then immediately fuzzing them. It does, however, better reflect our intentions for KFuzzTest: continuously fuzzing the kernel with a dynamically changing set of targets with little intervention from syzkaller maintainers. To enable this feature, configure the experimental `enable_kfuzztest` option in the manager configuration, which enables all discovered KFuzzTest targets by default. ```json { "enable_kfuzztest": true } ``` You must also enable pseudo-syscall `syz_kfuzztest_run`, like so: ```json { "enable_syscalls": [ "syz_kfuzztest_run" ], } ``` **IMPORTANT:** for dynamic discovery to work, it is essential for the kernel image pointed to by the manager configuration is built with `CONFIG_DWARF4` or `CONFIG_DWARF5` enabled, as dynamic target discovery depends on these symbols being emitted. ### 3. `syz-kfuzztest`, an in-VM standalone tool In contrast with `syz-manager`, `syz-kfuzztest` is designed to perform coverage guided fuzzing from within a VM directly rather than orchestrating a fleet of VMs. It is primarily targeted at development-time fuzzing, rather than long-term continuous fuzzing. For more information, consult [the `syz-kfuzztest` documentation](syz-kfuzztest.md). --- ### Maintaining (`docs/maintaining.md`) # Maintainer's Guide ## Linear history We preserve linear history and have merge commits disabled in repository settings. Pull requests (PR) have only 2 options: - **Rebase and merge**\ This is preferred if all commits in the PR are properly squashed/fixed already, as it preserves commits in the PR (takes all commits in the PR as is and adds them on top of the master HEAD). - **Squash and merge**\ This is preferred if the PR contains fix ups as separate commits and/or other clutter in commit organization (squashes all commits in the PR into one commit and adds it on top of the master HEAD, also allows to edit commit subject/description). ## PR checks (CI) `cla/google` check needs to pass before merging. CI testing generally needs to pass before merging.\ Exceptions may be infrastructure flakes (especially in external services: `codecov`, `ci/fuzzit`); one-off timeouts/OOMs (but not if this PR itself makes them much more frequent). All static checking warnings and testing errors are considered hard errors. ## Tests Adding tests for new code and bug fixes is generally encouraged. Ask contributors to add tests. However, some code is easier to test, while some is harder. Some examples of cases where it's easier to add tests (should be added): abstract functionalities without external dependencies (e.g. parsers, data transformations, calculations); code with established testing infrastructure (adding new tests is just adding one more of the same). Examples of cases where it's harder to add tests (may be not added, but still welcome if one wants to go above and beyond): code with external dependencies that are not easy to fake out (qemu, kernel, image, etc); code without established testing infrastructure where adding one test would require building the whole infrastructure first. ## Use your judgement There are no strict rules for reviews/ownership at the moment. Use your judgement. If you are maintaining a particular area of the project (e.g. support for one OS), it is OK to merge your own changes without further review (especially smaller and if CI gives green light). It's also OK to review and merge changes to other parts of the project. But loop in other maintainers if you don't feel confident or need additional feedback/review. --- ### Mcp (`docs/mcp.md`) # syz-agent in MCP mode Example use with Gemini CLI: 1. Run `make agent`. 2. Create `mcp.config` file, for example: ```json { "http": "localhost:59999", "mcp": true, "syzkaller_repo": "git@github.com:syzkaller/syzkaller.git", "target": "linux/amd64", "image": "/buildroot_amd64-2022.11.clean", "kernel_config": "upstream-apparmor-kasan.config", "type": "qemu", "vm": { "count": 10, "cpu": 2, "mem": 2048, "cmdline": "root=/dev/sda1" }, "cache_size": 107374182400 } ``` 3. Start `syz-agent`: ```bash bin/syz-agent -config mcp.config ``` 4. Add the following to `~/.gemini/settings.json`: ```json { "mcpServers": { "syzkaller": { "httpUrl": "http://127.0.0.1:59999", "trust": true } } } ``` 5. Start Gemini CLI: ```bash gemini ``` 6. Test that `gemini` has connected to `syz-agent` by executing `/mcp` or `/mcp schema`. 7. Try the following prompt: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` A manual unit test for the MCP server: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Program Syntax (`docs/program_syntax.md`) # Program syntax Syzkaller uses a compact domain-specific language (DSL) for programs to log executed programs, test its code, and persist programs in the corpus. This page provides a brief description of the corresponding syntax. Some useful information can also be found in the [existing examples](/sys/linux/test) and in the program [deserialization code](/prog/encoding.go). Together with execution options, the DSL provides everything that syz-executor needs to run a program. For example, consider the program: ``` r0 = syz_open_dev$loop(&(0x7f00000011c0), 0x0, 0x0) r1 = openat$6lowpan_control(0xffffffffffffff9c, &(0x7f00000000c0), 0x2, 0x0) ioctl$LOOP_SET_FD(r0, 0x4c00, r1) ``` Each line in this program describes a particular syscall invocation, with the first two calls saving the result in temporary variables `r0` and `r1`, which are passed to the third call. ``` line = assignment | call assignment = variable " = " call call = syscall-name "(" [arg ["," arg]*] ")" ["(" [call-prop ["," call-prop*] ")"] arg = "nil" | "AUTO" | const-arg | resource-arg | result-arg | pointer-arg | string-arg | struct-arg | array-arg | union-arg const-arg = integer resource-arg = variable ["/" hex-integer] ["+" hex-integer] result-arg = "<" variable "=>" arg pointer-arg = "&" pointer-arg-addr ["=ANY"] "=" arg pointer-arg-addr = "AUTO" | "(" pointer-addr ["/" region-size] ")" string-arg = "'" escaped-string "'" | "\"" escaped-string "\"" | "\"$" escaped-string "\"" struct-arg = "{" [arg ["," arg]*] "}" array-arg = "[" [arg ["," arg]*] "]" union-arg = "@" field-name ["=" arg] call-prop = prop-name ": " prop-value variable = "r" dec-integer pointer-addr = integer region-size = integer integer = dec-integer | oct-integer | "0x" hex-integer ``` Programs may also contain blank lines and comments. ``` # Obtain a file handle r0 = openat(0xffffffffffffff9c, &AUTO='./file1\x00', 0x42, 0x1ff) # Perform a write operation write(r0, &AUTO="01010101", 0x4) ``` ### Memory management Memory management is performed by syzkaller itself. It will allocate virtual memory regions of the necessary size and set the final values of pointer arguments. By using the `AUTO` keyword, programs can give syzkaller the full control over storing the data. This may be convenient e.g. when a parameter must be passed by reference, but the exact location of its value is not of particular importance. ``` r1 = syz_genetlink_get_family_id$nl80211(&AUTO='nl80211\x00', 0xffffffffffffffff) ioctl$sock_SIOCGIFINDEX_80211(r0, 0x8933, &AUTO={'wlan0\x00', 0x0}) ``` Alternatively, some data can be "anchored" to specific addresses. It may be especially important when a memory region must be shared between multiple calls. In this case, pointer addresses must be given at the 0x7f0000000000 offset. Before the actual execution, syzkaller will adjust pointers to the start of the actual mmap'ed region. ### Call properties Call properties specify extra information about how a specific call must be executed. Each call within a program has its own set of call properties. If no properties are provided, syzkaller takes the default ones. Currently, syzkaller supports the following call properties. #### Fault injection Syntax: `fail_nth: N`. It takes an integer (base 10) argument `N`. If the argument is non-negative, a fault will be injected into the `N`-th occasion. ``` r0 = openat$6lowpan_control(0xffffffffffffff9c, &(0x7f00000000c0), 0x2, 0x0) ioctl$LOOP_SET_FD(r0, 0x4c00, r0) (fail_nth: 5) ``` #### Async Syntax: `async`. Instructs `syz-executor` not to wait until the call completes and to proceed immediately to the next call. ``` r0 = openat(0xffffffffffffff9c, &AUTO='./file1\x00', 0x42, 0x1ff) write(r0, &AUTO="01010101", 0x4) (async) read(r0, &AUTO=""/4, 0x4) close(r0) ``` When setting `async` flags be aware of the following considerations: * Such programs should only be executed in threaded mode (i.e. `-threaded` flag must be passed to `syz-executor`. * Each `async` call is executed in a separate thread and there's a limited number of available threads (`kMaxThreads = 16`). * If an `async` call produces a resource, keep in mind that some other call might take it as input and `syz-executor` will just pass 0 if the resource- producing call has not finished by that time. --- ### Prometheus (`docs/prometheus.md`) # Prometheus metrics syz-manager metrics are exposed at the URI `/metrics` on the http endpoint. Currently exported prometheus metrics from the manager are `syz_exec_total`, `syz_corpus_cover` and `syz_crash_total`. These metrics can be ingested using following prometheus client configuration: ```yaml scrape_configs: - job_name: syzkaller scrape_interval: 10s static_configs: - targets: - localhost:56741 ``` Values are reset to zero at syz-manager restart and only reflect for current execution of syz-manager. --- ### Pseudo Syscalls (`docs/pseudo_syscalls.md`) # Pseudo-syscalls Besides regular system calls, a [syscall description](syscall_descriptions.md) file can also contain pseudo-syscalls. These are C functions defined in the executor. When a test program uses a pseudo-syscall, the executor will generate the pseudo-syscall function code in the resulting C program. This allows a test program to have specific code blocks to perform certain actions, they may also be used as more test-friendly wrappers for primitive syscalls. Use of pseudo-syscalls is generally **discouraged** because they ruin all advantages of the declarative descriptions (declarativeness, conciseness, fuzzer control over all aspects, possibility of global improvements to the logic, static checking, fewer bugs, etc), increase maintenance burden, are non-reusable and make C reproducers longer. However, syzlang is not expressive enough to cover all possible cases, so use of pseudo-syscalls needs to be considered on a case-by-cases basis (additional benefit, amount of code, possibility of extending syzlang to cover this case, etc). ## How to add a pseudo-syscall to the executor First, think about the scope of the pseudo-syscall and which systems and subsystems it will be related to. The executor includes a fixed set of C header files `executor/common*.h` containing the code of the pseudo-syscalls. Check if the new one can fit in one of the existing files before creating a new one. For instance, if our new pseudo-syscall is Linux-specific, then [common_linux.h](../executor/common_linux.h) would be the place to put it. The actual pseudo-syscall function may look something like this: #if SYZ_EXECUTOR || __NR_syz_mycall /* Add all the necessary #include and #define headers */ static long syz_mycall(volatile long a0, volatile long a1) { /* Function body */ } #endif Make sure that all the function requirements are met and that it can be compiled. Note that the function name must start with "syz_". It may also take a different number of arguments. Type of arguments must be `volatile long`, return type - `long`. `long` is required to avoid potential calling convention issues because it is casted to a function pointer that accepts `long`'s. The reason for `volatile` is interesting: lots of libc functions are annotated with various argument constraints (e.g. this argument should not be `NULL`, or that argument must be a valid file descriptor); C reproducers may call these functions with constant arguments and compiler may see that some of these constraints are violated (e.g. passing `NULL` to a `non-NULL` argument, or passing `-1` as file descriptor) and produce errors/warnings. `volatile` prevents that. Now, to handle the pseudo-syscall properly we have to update the `linuxSyscallChecks` in [linux_syscalls.go](../pkg/vminfo/linux_syscalls.go) and add a particular case for this syscall, enabling it when necessary. If we want to enable it unconditionally we can simply use `alwaysSupported` for it. Finally, run `make generate`. Now you can use it in a syscall description file as if it was a regular system call: syz_mycall(arg0 pid, arg1 const[0])
## External Dependencies The implementation must not use any external libraries nor external headers, except for the most basic and standard ones (like `` and ``). In particular, it must not depend on libraries/headers installed by additional packages nor on headers for recently added kernel subsystems. External dependencies have proved to be brittle and easily cause build breakage because all dependencies will be required for any build/run on the fuzzer and any C reproducer. For example, packages/headers may be missing on some distros, named differently, be of a wrong version, broken, or conflict with other headers. Unfortunately, there is no way to reliably specify such dependencies and requirements for C programs. Therefore, if the pseudo-syscall requires definitions of some structures, constants, or helper functions, these should be described in the executor code itself as minimally as possible (they will be part of C reproducers). ## Testing Each new pseudo-syscall should have at least one test in `sys/OS/test`. See [Linux tests](/sys/linux/test) for an example. A tests is just a program with checked syscall return values. There should be at least one test that contains "the main successful scenario" of using the pseudo-syscall. See [io_uring test](/sys/linux/test/io_uring) as a good example. Such tests are important because they ensure that the pseudo-syscall code does not contain "stupid" bugs (e.g. crash on NULL-deref each time), that it is possible for the fuzzer to come up with the successful scenario (as a combination of the pseudo-syscall and the surrounding descriptions) and that it will continue to work in future. See [Testing of descriptions](syscall_descriptions.md#testing) for details about the tests. --- ### Pull Request Template (`docs/pull_request_template.txt`) ******************************************************************************* Before sending a pull request, please review Contribution Guidelines: https://github.com/google/syzkaller/blob/master/docs/contributing.md ******************************************************************************* --- ### Reproducing Crashes (`docs/reproducing_crashes.md`) # How to reproduce syzkaller crashes ## Using a C reproducer If the bug was reported by syzbot, you first need to build the kernel used by the tool. Syzbot provides the necessary information in its report: ``` Hello, syzbot found the following issue on: HEAD commit: ae58226b89ac Add linux-next specific files for 20241118 git tree: linux-next console+strace: https://syzkaller.appspot.com/x/log.txt?x=14a67378580000 kernel config: https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053 compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=177beac0580000 ``` In this case, you would run: ```shell $ git checkout ae58226b89ac $ wget -O '.config' 'https://syzkaller.appspot.com/x/.config?x=45719eec4c74e6ba' $ make CC=clang LD=ld.lld olddefconfig $ make CC=clang LD=ld.lld -j$(nproc) ``` You also need a bootable disk image. Syzbot currently uses small Buildroot-based images that you can either [build locally](/tools/create-buildroot-image.sh) or [download](https://storage.googleapis.com/syzkaller/images/buildroot_amd64_2024.09.gz). Download and build the reproducer: ```shell $ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=177beac0580000' $ gcc repro.c -lpthread -static -o repro ``` Run the VM: ```shell $ export DISK_IMAGE='buildroot_amd64_2024.09' $ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=$DISK_IMAGE,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1 ``` Run the reproducer: ```shell $ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro root@127.0.0.1:/root/ $ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 'chmod +x ./repro && ./repro' ``` ## Using a Syz reproducer Syzkaller always generates a "Syz" reproducer first (in [Syzkaller DSL](/docs/program_syntax.md)). Afterwards, syzkaller attempts to convert the Syz reproducer into C code. The process does not always succeed due to the differences between the `syz-executor` environment and the environment emulated in the C reproducer. Therefore, in some cases, only the Syz version is available. To run a Syz reproducer locally, the required actions are mostly similar to those in the previous section. Download and [build](/docs/linux/setup.md#go-and-syzkaller) syzkaller. If you have Docker installed, the instructions are simpler: ```shell $ git clone https://github.com/google/syzkaller.git $ cd syzkaller $ ./tools/syz-env make ``` Build the kernel and boot the VM as described in the section above. Download the reproducer: ```shell $ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=137beac0580000' ``` Copy the reproducer and the syzkaller binaries to the test machine: ```shell $ export SYZKALLER_PATH="~/syzkaller" $ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes $SYZKALLER_PATH/bin/linux_amd64/* ./repro.syz root@127.0.0.1:/root/ ``` Now you can use the `syz-execprog` tool to actually execute the program. ```shell $ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz' ``` Several useful `syz-execprog` flags: ``` -procs int number of parallel processes to execute programs (default 1) -repeat int repeat execution that many times (0 for infinite loop) (default 1) -sandbox string sandbox for fuzzing (none/setuid/namespace) (default "setuid") -threaded use threaded mode in executor (default true) ``` If you pass `-threaded=0`, all syscalls will be executed in the same thread. `-threaded=1` forces execution of each syscall in a separate thread, so that execution can proceed over blocking syscalls. Before 2021, `syz-execprog` also supported the following flag: ``` -collide collide syscalls to provoke data races (default true) ``` `-collide=1` forced second round of execution of syscalls when pairs of syscalls are executed concurrently. Starting from the revision [fd8caa54](https://github.com/google/syzkaller/commit/fd8caa5462e64f37cb9eebd75ffca1737dde447d), the behavior is controlled [directly in syzlang](/docs/program_syntax.md#async). If you are running older reproducers, you might still need to set the `-collide=1` flag. If you are replaying a reproducer program that contains a header along the following lines: ``` # {Threaded:true Repeat:true RepeatTimes:0 Procs:8 Slowdown:1 Sandbox:none Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true UseTmpDir:true HandleSegv:true Repro:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}} ``` then you need to adjust `syz-execprog` flags based on the values in the header. Namely, `Threaded`/`Procs`/`Sandbox` directly relate to `-threaded`/`-procs`/`-sandbox` flags. If `Repeat` is set to `true`, add `-repeat=0` flag to `syz-execprog`. ## Using ktest [ktest](https://evilpiepirate.org/git/ktest.git/tree/README.md) is a collection of tests for Linux and an infrastructure that simplifies running them locally. Ktest includes a special `syzbot-repro.ktest` test that automates building the kernel, booting the VM, fetching syzbot bug report details and running the reproducer. **Installation instructions:** ```shell $ git clone git://evilpiepirate.org/ktest.git $ cd ktest $ export KTEST_PATH=$(pwd) $ sudo ./root_image init $ sudo ./root_image create $ cargo install --path $KTEST_PATH ``` **Instructions to reproduce a syzbot bug:** ```shell $ cd ~/linux $ git checkout $ $KTEST_PATH/build-test-kernel run $KTEST_PATH/tests/syzbot-repro.ktest ``` `` can be taken from syzbot bug reports: ``` dashboard link: https://syzkaller.appspot.com/bug?extid=2159cbb522b02847c053 ``` In this case, `bug-id` is `2159cbb522b02847c053`. ## Using downloadable assets In each report, syzbot shares the exact disk image, kernel image and the vmlinux file that were used to find it. See [the corresponding documentation](/docs/syzbot_assets.md) on how you can use those files to reproduce bugs locally. ## From execution logs The process of creating reproducer programs for syzkaller bugs is automated, but it's not perfect. In some cases, the tool cannot narrow down the kernel crash to a single program. ### Obtaining execution logs * **A local syzkaller instance** \ Crash logs created in manager `workdir/crashes` dir contain programs executed just before a crash. In parallel execution mode (when `procs` parameter in manager config is set to value larger than 1), program that caused the crash does not necessary immediately precedes it; the guilty program can be somewhere before. * **Syzbot** shares execution logs in its reports: ``` console output: https://syzkaller.appspot.com/x/log.txt?x=148914c0580000 ``` ### Crafting reproducers manually There are two tools that can help you identify and minimize the program that causes a crash: `syz-execprog` and `syz-prog2c`. You can build them with `make execprog` and `make prog2c`, respectively. `syz-execprog` executes a single syzkaller program or a set of programs in various modes (once or loop indefinitely; in threaded/collide mode (see below), with or without coverage collection). You can start by running all programs in the crash log in a loop to check that at least one of them indeed crashes kernel: ```bash ./syz-execprog -executor=./syz-executor -repeat=0 -procs=8 -cover=0 crash-log-file.txt ``` **Note: `syz-execprog` executes programs locally. So you need to copy `syz-execprog` and `syz-executor` into a VM with the test kernel and run it there.** See the [Using a Syz reproducer](#using-a-syz-reproducer) section. To identify the single program that causes the crash, you can cut out individual programs from `crash-log-file.txt` and run `syz-execprog` separately. Once you have a single program that causes the crash, you can try to minimize it by: * Removing individual syscalls from the program (you can comment out single lines with `#` at the beginning of line) * By removing unnecessary data (e.g. replacing `&(0x7f0000001000)="73656c6600"` syscall argument with `&(0x7f0000001000)=nil`). * You can also try to coalesce all mmap calls into a single mmap call that maps whole required area. Don't forget to test minimization results with the `syz-execprog` tool. Now that you have a minimized program, check if the crash still reproduces with `./syz-execprog -threaded=0 -collide=0` flags. If not, then you will need to do some additional work later. Now, run the `syz-prog2c` tool on the program. It will give you an executable C source code. If the crash reproduces with `-threaded/collide=0` flags, then this C program should cause the crash as well. If the crash is not reproducible with `-threaded/collide=0` flags, then you need this last step. You can think of threaded mode as if each syscall is executed in its own thread. To model such execution mode, move individual syscalls into separate threads. You can see an example here: https://groups.google.com/d/msg/syzkaller/fHZ42YrQM-Y/Z4Xf-BbUDgAJ. This process is automated to some degree in the `syz-repro` utility. You need to give it your manager config and a crash report file. And you can refer to the [example config file](/pkg/mgrconfig/testdata/qemu-example.cfg). ```bash ./syz-repro -config my.cfg crash-qemu-1-1455745459265726910 ``` It will try to find the offending program and minimize it. But since there are lots of factors that can affect reproducibility, it does not always work. --- ### Research (`docs/research.md`) # Research work based on syzkaller Dear researchers, feel free to contact us at syzkaller@googlegroups.com if you need any assistance. _newer first_ * [Unlocking Low Frequency Syscalls in Kernel Fuzzing with Dependency-Based RAG](https://dl.acm.org/doi/abs/10.1145/3728913) ([pdf](https://dl.acm.org/doi/pdf/10.1145/3728913), [source code](https://github.com/QGrain/SyzGPT)) * [A Little Goes a Long Way: Tuning Configuration Selection for Continuous Kernel Fuzzing](https://www.computer.org/csdl/proceedings-article/icse/2025/056900a521/215aWUHqo9i) * [SyzDirect: Directed Greybox Fuzzing for Linux Kernel](https://dl.acm.org/doi/abs/10.1145/3576915.3623146) * [KIT: Testing OS-Level Virtualization for Functional Interference Bugs](https://doi.org/10.1145/3575693.3575731) * [SyzDescribe: Principled, Automated, Static Generation of Syscall Descriptions for Kernel Drivers](https://github.com/seclab-ucr/SyzDescribe) * [GREBE: Unveiling Exploitation Potential for Linux Kernel Bugs](https://zplin.me/papers/GREBE.pdf) * [Precise Detection of Kernel Data Races with Probabilistic Lockset Analysis](https://www.cs.columbia.edu/~gabe/files/oakland2023_pla.pdf) * [Linux Kernel Enriched Corpus](https://github.com/cmu-pasta/linux-kernel-enriched-corpus) : [corpus.db](https://github.com/cmu-pasta/linux-kernel-enriched-corpus/raw/main/corpus.db) * [HotBPF - An On-demand and On-the-fly Memory Protection](https://www.youtube.com/watch?v=1KSLTsgxaSU) * [KASPER: Scanning for Generalized Transient Execution Gadgets in the Linux Kernel](https://www.vusec.net/projects/kasper/) * [VaultFuzzer: A state-based approach for Linux kernel](https://hardenedvault.net/blog/2021-09-13-vaultfuzzer/) * [Demystifying the Dependency Challenge in Kernel Fuzzing](https://conf.researchr.org/details/icse-2022/icse-2022-papers/89/Demystifying-the-Dependency-Challenge-in-Kernel-Fuzzing) * [SyzVegas: Beating Kernel Fuzzing Odds with Reinforcement Learning](https://www.usenix.org/conference/usenixsecurity21/presentation/wang-daimeng) * [SyzScope: Revealing High-Risk Security Impacts of Fuzzer-Exposed Bugs in Linux kernel](https://www.usenix.org/conference/usenixsecurity22/presentation/zou) * [Rtkaller: State-aware Task Generation for RTOS Fuzzing](http://www.wingtecher.com/themes/WingTecherResearch/assets/papers/emsoft21.pdf) * [BSOD: Binary-only Scalable fuzzing Of device Drivers](https://dmnk.co/raid21-bsod.pdf) * [Torpedo: A Fuzzing Framework for Discovering Adversarial Container Workloads](https://vtechworks.lib.vt.edu/handle/10919/104159) * [Healer](https://github.com/SunHao-0/healer) is a kernel fuzzer inspired by syzkaller. ([pdf](http://www.wingtecher.com/themes/WingTecherResearch/assets/papers/healer-sosp21.pdf)) * [SyzGen: Automated Generation of Syscall Specification of Closed-Source macOS Drivers](https://www.cs.ucr.edu/~zhiyunq/pub/ccs21_syzgen.pdf) ([source code](https://github.com/seclab-ucr/SyzGen_setup)) * [Snowboard: Finding Kernel Concurrency Bugs through Systematic Inter-thread Communication Analysis](https://dl.acm.org/doi/10.1145/3477132.3483549) * [Undo Workarounds for Kernel Bugs](https://www.usenix.org/system/files/sec21fall-talebi.pdf) ([source code](https://trusslab.github.io/hecaton)) * [HFL: Hybrid Fuzzing on the Linux Kernel](https://www.ndss-symposium.org/wp-content/uploads/2020/02/24018-paper.pdf) * [A Novel Dynamic Analysis Infrastructure to Instrument Untrusted Execution Flow Across User-Kernel Spaces](https://www.computer.org/csdl/proceedings-article/sp/2021/893400a402/1mbmHSlbmvK) * [Industry Practice of Coverage-Guided Enterprise Linux Kernel Fuzzing](http://wingtecher.com/themes/WingTecherResearch/assets/papers/fse19-linux-kernel.pdf) * [Agamotto: Accelerating Kernel Driver Fuzzing with Lightweight Virtual Machine Checkpoints](https://www.usenix.org/conference/usenixsecurity20/presentation/song) ([source code](https://github.com/securesystemslab/agamotto)) * [Empirical Notes on the Interaction Between Continuous Kernel Fuzzing and Development](http://users.utu.fi/kakrind/publications/19/vulnfuzz_camera.pdf) * [FastSyzkaller: Improving Fuzz Efficiency for Linux Kernel Fuzzing](https://iopscience.iop.org/article/10.1088/1742-6596/1176/2/022013) * [Charm: Facilitating Dynamic Analysis of Device Drivers of Mobile Systems](https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-talebi.pdf) ([video](https://www.usenix.org/conference/usenixsecurity18/presentation/talebi), [slides](https://www.usenix.org/sites/default/files/conference/protected-files/security18_slides_talebi.pdf), [source code](https://trusslab.github.io/charm)) * [ALEXKIDD-FUZZER: Kernel Fuzzing Guided by Symbolic Information](https://www.cerias.purdue.edu/assets/symposium/2018-posters/829-D1B.pdf) * [DIFUZE: Interface Aware Fuzzing for Kernel Drivers](https://acmccs.github.io/papers/p2123-corinaA.pdf) * [MoonShine: Optimizing OS Fuzzer Seed Selection with Trace Distillation](http://www.cs.columbia.edu/~suman/docs/moonshine.pdf) * [RAZZER: Finding Kernel Race Bugs through Fuzzing](https://lifeasageek.github.io/papers/jeong:razzer.pdf) * [SemFuzz: Semantics-based Automatic Generation of Proof-of-Concept Exploits](https://www.informatics.indiana.edu/xw7/papers/p2139-you.pdf) * [Towards Automating Exploit Generation for Arbitrary Types of Kernel Vulnerabilities](https://i.blackhat.com/us-18/Thu-August-9/us-18-Wu-Towards-Automating-Exploit-Generation-For-Arbitrary-Types-of-Kernel-Vulnerabilities-wp.pdf) * [KOOBE: Towards Facilitating Exploit Generation of Kernel Out-Of-Bounds Write Vulnerabilities](https://www.usenix.org/system/files/sec20summer_chen-weiteng_prepub.pdf) * [Synthesis of Linux Kernel Fuzzing Tools Based on Syscall](http://dpi-proceedings.com/index.php/dtcse/article/download/14990/14503) * [Drill the Apple Core: Up & Down](https://i.blackhat.com/eu-18/Wed-Dec-5/eu-18-Juwei_Lin-Drill-The-Apple-Core.pdf) * [WSL Reloaded](https://www.slideshare.net/AnthonyLAOUHINETSUEI/wsl-reloaded) # Other kernel fuzzing work * [Hydra: Finding Semantic Bugs in File Systems with an Extensible Fuzzing Framework](https://squizz617.github.io/pubs/hydra-sosp19.pdf) ([github](https://github.com/sslab-gatech/hydra)) * [Janus: Fuzzing File Systems via Two-Dimensional Input Space Exploration](https://gts3.org/assets/papers/2019/xu:janus.pdf) ([github](https://github.com/sslab-gatech/janus)) * [CoLaFUZE: Coverage-Guided and Layout-Aware Fuzzing for Android Drivers](https://www.jstage.jst.go.jp/article/transinf/E104.D/11/E104.D_2021NGP0005/_pdf) * [KRACE: Data Race Fuzzing for Kernel File Systems](https://www.cc.gatech.edu/~mxu80/pubs/xu:krace.pdf) * [trinity](https://github.com/kernelslacker/trinity) * [kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels](https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-schumilo.pdf) (bridges AFL and Intel PT) * [kernel-fuzzing](https://github.com/oracle/kernel-fuzzing) (bridges AFL and KCOV) * [A gentle introduction to Linux Kernel fuzzing](https://blog.cloudflare.com/a-gentle-introduction-to-linux-kernel-fuzzing/) (bridges AFL and KCOV) * [IMF: Inferred Model-based Fuzzer](https://acmccs.github.io/papers/p2345-hanA.pdf) Also see [tech talks page](/docs/talks.md). --- ### Review Guidelines (`docs/review_guidelines.md`) # Automated Code Review Guidelines (DVYUKOV.md) This file contains curated recommendations extracted from automated pull request reviews. Ingest this file into your LLM pipeline to ensure compliance with repository style and best practices! ## 1. Go Coding Style & Idioms - **Naming:** - Lowercase package names (no underscores or MixedCaps). - Use `NewFoo` or `MakeFoo` for constructors; return pointers if methods are defined on pointers. - Avoid repeating type names in method names (e.g., `linux.Foo()` not `linux.FooLinux()`). - Avoid "syz" prefixes for internal structs/fields (e.g., `exec_total` not `syz_exec_total`). - Use `i` for loop indices; use descriptive names for slice elements. - Favor positive boolean naming (`active` vs `not_inactive`). - **Declarations:** - Combine declaration and initialization (`x := foo()`). - Rely on zero-value defaults; do not manually initialize to 0, "", or `map[type]type{}`. - Unexport symbols (types, fields, constants, functions) not needed externally. - Use `iota` for enums and make `Invalid` the first value (0) to catch uninitialized fields. - Use `go:embed` for embedding static assets like HTML or text templates. - **Functions & Control Flow:** - Return early on errors (standard `if err != nil` check); avoid deep nesting. - Avoid naked or named returns unless they significantly simplify repetitive code. - Drop unnecessary `else` after `return`, `break`, or `continue`. - Prefer free/normal functions over closures if no context is captured from the outer scope. - Use closures to deduplicate scoped logic within a single function. - **Miscellaneous:** - Cap Go lines at 120 columns; use tabs for indentation. - Use `%s` for struct string conversion in `fmt` functions (idiomatic `v.String()` call). - Avoid unneeded parentheses in expressions and `()` to refer to functions in comments. ## 2. Error Handling & Reliability - **Return Errors:** Never swallow errors or return `nil` on unexpected conditions. - **Fail Fast:** - Validate invariants early (e.g., during initialization or deserialization). - Panic on impossible internal logic corruption or broken engine invariants. - For external tools/plugins, use `SYZFAIL:` prefix for internal bugs to distinguish from target crashes. - **Error Context:** - Don't wrap errors without adding specific, helpful context (Syzkaller libs often add basic context). - Use `aflow.BadCallError` for invalid tool inputs in agent workflows. - **Validation:** - Always validate inputs at boundaries (datastores, APIs, CLI flags, regexes). - Guard against underflows/overflows in length checks and binary formats. ## 3. Testing Best Practices - **Assertions:** Use `github.com/stretchr/testify/require` (or `assert`) for single-line, readable assertions. - **Structure:** - Use table-driven tests with `t.Run` for isolation and subtest selection. - Keep helper functions at the bottom of the test file; use `t.Cleanup` for teardown. - **Data Handling:** - Use backtick raw string literals (`` ` ``) for multi-line test data, regexes, and assembly. - Use `AUTO` in syzlang tests for auto-computing lengths and offsets. - Use `timeNow` over `time.Now` for deterministic test stubbing. - **Robustness:** - Run tests with `-race` to surface data races. - Check for goroutine leaks in tests using background concurrency. - Test boundary conditions (e.g., empty slices, index limits, or corrupted inputs). - Never just disable failing tests; fix them or document why they are disabled. - Include sample reports in `pkg/report/testdata` for new crash parsers. ## 4. Architecture & Design - **YAGNI (You Ain't Gonna Need It):** - Don't add code "just in case"; remove unused types, fields, and dead code immediately. - Avoid unnecessary tunables/config flags; pick sensible defaults and scale timeouts. - **Decoupling:** - Keep logic uniform across different execution modes (e.g., fork-server vs no-fork-server). - Use standard interfaces (e.g., `io.Closer`) over custom cleanup methods. - Prefer specific, descriptive package names over generic ones like `util` or `generic`. - Prioritize larger, coherent packages over many micro-packages. - **State Management:** - Favor simple state tracking (single counter/slice) over complex nested maps. - Use fields in the `targets.Target` struct (often referred to as Arch struct) instead of logic-driven switches for architecture differences. - Pass objects/pointers early instead of passing IDs and doing repeated lookups. ## 5. Performance & Resource Management - **Efficiency:** - Use map lookups (O(1)) instead of double loops (O(N^2)) for filtering. - Avoid casting large buffers to strings if only byte processing is needed. - Prefer direct slicing over `io.Reader` for byte processing to reduce garbage. - Avoid `bufio.Writer` when writing to `bytes.Buffer`. - **Resource Cleanup:** - Always `Wait()` on launched commands; use `defer` to close file descriptors and pipes. - Extract loop bodies into functions if they use `defer` to ensure per-iteration cleanup. - **Concurrency:** - Use channels for semaphores or async notifications; use capacity 1 to prevent blocking. - Use `nil` channels to disable `select` cases. - Ensure condition variables are checked in a loop to prevent race hangs. - Avoid infinite goroutine spawning in pools; track wait states. ## 6. Logging & Observability - **Hygiene:** - Use the internal `log` package consistently; avoid mixing multiple logger libraries. - Log only actionable information at verbosity 0; use info logs for invalid user requests. - Don't suffix log messages with newlines (`\n`). - Format string safety: use `Logf(1, "%s", out)` instead of `Logf(1, out)`. - **Determinism:** Always sort map keys when serializing to persistent formats (configs, logs, coverage). - **Metrics:** Rename existing metrics rather than adding confusing duplicates when semantics change. - **Reporting:** Keep email prefixes short; e.g. explain why something was skipped in bisection logs. ## 7. Syzkaller Specifics (Syzlang & Executor) - **Syzlang (files in `sys/*/*.txt`):** - Use exact kernel field names; use descriptive prefixes to avoid collisions. - Use `const[0]` for reserved/unused fields to optimize mutation. - Use `ignore_return` for syscalls returning random system IDs or times. - **Executor (C++ files in `executor`):** - Keep `executor.cc` free of OS-specific `#ifdefs`; use OS-specific headers. - Use `const std::string&` or `std::string_view` for immutable strings. - Check for short writes in output loops. - Avoid C89 start-of-block declarations; combine declaration and initialization. ## 8. Tooling & Infrastructure - **CLI Tools:** - Standardize on `tool.Init()` and `tool.Failf()` from `pkg/tool`. - Declare CLI flags in `main()` to avoid global namespace pollution. - Use `runtime.NumCPU()` for default worker counts. - **OS Operations:** - Use `filepath.Join` for OS-agnostic paths; use `osutil.IsExist` for existence checks. - Combine multiple shell commands (e.g., `adb shell 'cmd1; cmd2'`) to reduce overhead. - **HTML/UI:** - Use standard `
` and `` for collapsibles. - Use tabs for HTML templates. --- ### Setup (`docs/setup.md`) # How to set up syzkaller Generic setup instructions for fuzzing Linux kernel are outlined [here](linux/setup.md). For other kernels see: [FreeBSD](freebsd/README.md), [Darwin/XNU](darwin/README.md), [Fuchsia](fuchsia/README.md), [NetBSD](netbsd/README.md), [OpenBSD](openbsd/setup.md), [Windows](windows/README.md). After following these instructions you should be able to run `syz-manager`, see it executing programs, and be able to access statistics exposed at `http://127.0.0.1:56741` (or whatever address you've specified in the manager config). If everything is working properly, a typical execution log would look like: ``` $ ./bin/syz-manager -config=my.cfg 2017/06/14 16:39:05 loading corpus... 2017/06/14 16:39:05 loaded 0 programs (0 total, 0 deleted) 2017/06/14 16:39:05 serving http on http://127.0.0.1:56741 2017/06/14 16:39:05 serving rpc on tcp://127.0.0.1:34918 2017/06/14 16:39:05 booting test machines... 2017/06/14 16:39:05 wait for the connection from test machine... 2017/06/14 16:39:59 received first connection from test machine vm-9 2017/06/14 16:40:05 executed 293, cover 43260, crashes 0, repro 0 2017/06/14 16:40:15 executed 5992, cover 88463, crashes 0, repro 0 2017/06/14 16:40:25 executed 10959, cover 116991, crashes 0, repro 0 2017/06/14 16:40:35 executed 15504, cover 132403, crashes 0, repro 0 ``` At this point it's important to ensure that syzkaller is able to collect code coverage of the executed programs (unless you specified `"cover": false` in the config or coverage is not yet supported for the kernel you're fuzzing). The `cover` counter on the web page should be non zero. More information on the configuration file format is available [here](configuration.md). See [this page](troubleshooting.md) for troubleshooting tips. --- ### Setup Syzbot (`docs/setup_syzbot.md`) # How to set up syzbot This doc will be useful to you: - should you wish to hack on user interface bits like the dashboard / mailing list integration or - should you wish to continuously run a separate syzbot dashboard for your own kernels Note: For most development purposes you don't need a full syzbot setup. The meat of syzkaller is really located in syz-manager and syz-executor. You can run syz-manager directly which is usually what you will want to do during fuzzer development. [See this documentation for syz-manager setup instructions](setup.md). This doc assumes that you: - have a GCP account and billing setup - created a GCP project for running syzbot in - are running a reasonably modern linux distro - locally installed `gcloud`, `ssh`, `go` and `build-essential` - may need to install `google-cloud-sdk-app-engine-go` for the GAE deployment to work - ran `gcloud auth login` to run authenticated gcloud commands - read [go/syzbot-setup](https://goto.google.com/syzbot-setup) if you are a Googler While most syzkaller bits happily run on various operating systems, the syzbot dashboard does not. The dashboard is a Google App Engine or GAE project. GAE allows developers to develop web applications without needing to worry about the underlying servers. Instead developers just push their code and GAE takes care of web servers, load balancers and more. Hence this document is more Google Cloud focused than the rest of our documentation. We will also deploy a syz-ci instance. syz-ci keeps track of the syzkaller and kernel repositories and continuously rebuilds the kernel under test, itself and other syzkaller components when new commits land in the upstream repositories. syz-ci also takes care of (re)starting syz-manager instances, which in turn (re)start VMs fuzzing the target kernel. For simplicity we will run everything in this doc on GCP even though syz-ci could run elsewhere. ## Deploying Syz-ci [local] First prepare an initial syz-ci build locally (later syz-ci rebuilds itself) and a rootfs: ```sh # Most syzkaller components can be build even outside of the GOPATH, however # the syzbot app engine deployment only works from the GOPATH right now.. export GOOGLE_GO=$HOME/gopath/src/github.com/google/ mkdir -p $GOOGLE_GO git clone https://github.com/google/syzkaller.git mv syzkaller $GOOGLE_GO/ cd $GOOGLE_GO/syzkaller make ci cd ~/repos git clone git://git.buildroot.net/buildroot cd buildroot $GOOGLE_GO/syzkaller/tools/create-buildroot-image.sh ``` [local] Enable various services in the project, create a VM, storage bucket, scp assets and login: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` [syz-ci] Let's install and configure the syz-ci service on our syz-ci VM: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Check the syc-ci journal logs at this point to see if the service comes up fine. Now syz-ci needs to do a bunch of time consuming stuff like building the kernel under test, so be patient. If you want to hack on syz-ci you can stop here. Otherwise the next section builds on the syz-ci instructions and extends the setup with a dashboard deployment. ## Deploying Syzbot dashboard [locally] deploy the dashboard to Google App Engine: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Integrating Syz-ci with syzbot [locally] Prepare config and login to syz-ci VM: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` [syz-ci] Reconfigure syz-ci to start sending results to the dashboard: ```sh sudo mv ~/config.ci /syzkaller/ sudo systemctl restart syz-ci sudo journalctl -fu syz-ci ``` [locally] Open the dashboard in your browser: ```sh gcloud app browse --project=$PROJECT ``` Once syzkaller finds the first crashes they should show up here. This might take a while. --- ### Strace (`docs/strace.md`) # Strace Syzkaller can be instructed to execute programs under [strace](https://strace.io/) and capture the output. If the `strace_bin` is set to an `strace` binary, syzkaller will automatically run each reproducer it managed to find under the `strace` binary. * If a syz-manager is attached to a `dashboard`, syzkaller will upload the resulting output as a normal log file if the generated reproducer still managed to produce the same crash. * Otherwise, the output of strace will be saved to a separate file and will be accessible through the syz-manager's web interface. ## How to compile the strace binary It is safer to compile `strace` as a statically linked binary in order to prevent problems with mismatching libc versions on the kernel image used for fuzzing. ``` git clone https://github.com/strace/strace.git cd strace ./bootstrap ./configure --enable-mpers=no LDFLAGS='-static -pthread' make -j`nproc` ``` The resulting binary can be found at `src/strace`. ## syz-crush It's possible to instruct `syz-crush` to run the attached repro under strace. In order to do so, make sure `strace_bin` is specified in the syz-manager config file and pass an extra `-strace` argument to the command arguments. ## syz-repro If `-strace file-name.log` is appended to the `syz-repro`'s arguments, the tool will run the resulting repro (if it managed to generate one) under strace and save its output. --- ### Syscall Descriptions (`docs/syscall_descriptions.md`) # Syscall descriptions `syzkaller` uses declarative description of syscall interfaces to manipulate programs (sequences of syscalls). Below you can see (hopefully self-explanatory) excerpt from the descriptions: ``` open(file filename, flags flags[open_flags], mode flags[open_mode]) fd read(fd fd, buf buffer[out], count len[buf]) close(fd fd) open_mode = S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH ``` The descriptions are contained in `sys/$OS/*.txt` files. For example see the [sys/linux/dev_snd_midi.txt](/sys/linux/dev_snd_midi.txt) file for descriptions of the Linux MIDI interfaces. A more formal description of the description syntax can be found [here](syscall_descriptions_syntax.md). ## Programs The translated descriptions are then used to generate, mutate, execute, minimize, serialize and deserialize programs. A program is a sequences of syscalls with concrete values for arguments. Here is an example (of a textual representation) of a program: ``` r0 = open(&(0x7f0000000000)="./file0", 0x3, 0x9) read(r0, &(0x7f0000000000), 42) close(r0) ``` For actual manipulations `syzkaller` uses in-memory AST-like representation consisting of `Call` and `Arg` values defined in [prog/prog.go](/prog/prog.go). That representation is used to [analyze](/prog/analysis.go), [generate](/prog/rand.go), [mutate](/prog/mutation.go), [minimize](/prog/minimization.go), [validate](/prog/validation.go), etc programs. The in-memory representation can be [transformed](/prog/encoding.go) to/from textual form to store in on-disk corpus, show to humans, etc. There is also another [binary representation](/prog/decodeexec.go) of the programs (called `exec`), that is much simpler, does not contain rich type information (irreversible) and is used for actual execution (interpretation) of programs by [executor](/executor/executor.cc). ## Describing new system calls This section describes how to extend syzkaller to allow fuzz testing of more kernel interfaces. This is particularly useful for kernel developers who are proposing new system calls. Currently all syscall descriptions are manually-written. There is an [open issue](https://github.com/google/syzkaller/issues/590) to provide some aid for this process and some ongoing work, but we are not there yet to have a fully-automated way to generate descriptions. There is a helper [headerparser](headerparser_usage.md) utility that can auto-generate some parts of descriptions from header files. Visual Studio Code has [syz-lang extension](https://marketplace.visualstudio.com/items?itemName=AndreyArtemiev.syzlang-extension&ssr=false#overview) for highlighting syntax. To enable fuzzing of a new kernel interface: 1. Study the interface, find out which syscalls are required to use it. Sometimes there is nothing besides the source code, but here are some things that may help: - Searching the Internet for the interface name and/or some unique constants. - Grepping Documentation/ dir in the kernel. - Searching tools/testing/ dir in the kernel. - Looking for large comment blocks in the source code. - Finding commit that added the interface via `git blame` or `git log` and reading the commit description. - Reading source code of or tracing libraries or applications that are known to use this interface. 2. Using [syntax documentation](syscall_descriptions_syntax.md) and [existing descriptions](/sys/linux/) as an example, add a declarative description of this interface to the appropriate file: - `sys/linux/.txt` files hold system calls for particular kernel subsystems, for example [bpf.txt](/sys/linux/bpf.txt) or [socket.txt](/sys/linux/socket.txt). - [sys/linux/sys.txt](/sys/linux/sys.txt) holds descriptions for more general system calls. - An entirely new subsystem can be added as a new `sys/linux/.txt` file. - If subsystem descriptions are split across multiple files, prefix the name of each file with the name of the subsystem (e.g. use `dev_*.txt` for descriptions of `/dev/` devices, use `socket_*.txt` for sockets, etc). 3. After adding/changing descriptions run: ``` bash make extract TARGETOS=linux SOURCEDIR=$KSRC make generate make ``` 4. Run syzkaller. Make sure that the newly added interface in being reached by syzkaller using the [coverage](coverage.md) information page. In the instructions above `make extract` generates/updates the `*.const` files. `$KSRC` should point to the _latest_ kernel checkout.\ _Note_: for Linux the _latest_ kernel checkout generally means the [mainline](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/) tree.\ However, in some cases we add descriptions for interfaces that are not in the mainline tree yet, so if `make extract` complains about missing header files or constants undefined on all architectures, try to use the latest [linux-next](https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/) tree (or if it happens to be broken at the moment, try a slightly older linux-next tree).\ _Note_: `make extract` overwrites `.config` in `$KSRC` and `mrproper`'s it. _Note_: `*.const` files are checked-in with the `*.txt` changes in the same commit. Then `make generate` updates generated code and `make` rebuilds binaries.\ Note: `make generate` does not require any kernel sources, native compilers, etc and is pure text processing. Note: `make generate` also updates the SYZ_REVISION under `executor/defs.h`, which is required for machine check while running syz-manager. This should be taken care of especially if you are trying to rebase with your own change on syscall description. Note: `make extract` extracts constants for all architectures which requires installed cross-compilers. If you get errors about missing compilers/libraries, try `sudo make install_prerequisites` or install equivalent package for your distro. Note: `sudo make install_prerequisites` will success even with some package failed to install, `sudo apt-get update && sudo apt-get upgrade` might be required to make this more efficient. If you want to fuzz only the new subsystem that you described locally, you may find the `enable_syscalls` configuration parameter useful to specifically target the new system calls. All system calls in the `enable_syscalls` list will be enabled if their requirements are met (ie. if they are supported in the target machine and any other system calls that need to run in order to provide inputs for them are also enabled). You can also include wildcard definitions to enable multiple system calls in a single line, for example: `"ioctl"` will enable all the described ioctls syscalls that have their requirements met, ``"ioctl$UDMABUF_CREATE"`` enables only that particular ioctl call, ``"write$UHID_*"`` enables all write system calls that start with that description identifier. When updating existing syzkaller descriptions, note, that unless there's a drastic change in descriptions for a particular syscall, the programs that are already in the corpus will be kept there, unless you manually clear them out (for example by removing the `corpus.db` file).
## Description tips and FAQ
### Syscall, struct, field, flags names Stick with existing kernel names for things, don't invent new names if possible. Following established naming conventions provides the following benefits: (1) consistency and familiarity of names used across kernel interfaces, which also enables searching kernel sources for related names; and (2) enable static checking of descriptions (e.g. missed flags or mistyped fields) with [syz-check](/tools/syz-check/check.go). For example, if there is an existing enum `v4l2_buf_type` in the kernel headers, use this name for flags in descriptions as well. The same for structs, unions, fields, etc. For syscall and struct variants, append the variant name after the `$` sign. For example, `fcntl$F_GET_RW_HINT`, `ioctl$FIOCLEX`, `setsockopt$SO_TIMESTAMP`.
### Resources for syscall ordering Resources and resource directions (`in`, `out`, `inout`) impose implicit ordering constraints on involved syscalls. If a syscall accepts a resource of a particular type (e.g. has `fd_cdrom` as an input), then it will be generally placed after a syscall that has this resource as output, so that the resource value can be passed between syscalls. For example: ``` r0 = openat$cdrom(...) ioctl$CDROMPAUSE(r0, 0x123) close(r0) ``` Syscall arguments are always `in`, return values are `out` and pointer indirections have explicit direction as `ptr` type attribute. Also, it is possible to specify direction attribute individually for struct fields to account for more complex producer/consumer scenarios with structs that include both input/output resources.
### Use of unexpected/undeclared values When specifying integer/string flags or integer fields stick with the official expected values only. Commonly, bugs are triggered by unexpected inputs. With that in mind, it can be too tempting to introduce some unexpected values to descriptions (e.g. `-1` or `INT_MAX`). This is not encouraged for several reasons. First, this is a cross-cutting aspect and these special unexpected values are applicable to just any flags and integer fields. Manually specifying them thousands of times is not scalable and is not maintainable. Second, It's hard for the fuzzer to come up with correct complex syscall sequences, and the descriptions are meant to help with this. Coming up with unexpected integer values is easy and the fuzzer does not need help here. Overall the idea is to improve the generic fuzzer logic to handle these cases better, which will help all descriptions, rather than over-specializing each individual integer separately. Fuzzer already has several tricks to deal with this, e.g. comparison operand value interception and list of typical magic values. Note: some values for flags may be undocumented only as an oversight. These values should be added to descriptions.
### Flags/enums The `flags` type is used for all of: - sets of mutually exclusive values, where only one of them should be chosen (like C enum); - sets of bit flags, where multiple values can be combined with bitwise OR (like mmap flags); - any combination of the above. The fuzzer has logic to distinguish enums and bit flags, and generates values accordingly. So the general guideline is just to enumerate the meaningful values in `flags` without adding any "special" values to "help" the current fuzzer logic. When/if the fuzzer logic changes/improves, these manual additions may become unnecessary, or, worse, interfere with the fuzzer ability to generate good values.
### Declaration order `syzlang` does not require declaring entities before use (like C/C++ does), entities can refer to entities declared later (like in Go). It's recommended to declare things in the order of importance so that the reader sees the most important things first and then proceeds to finer and finer implementation details. For example, system calls usually should go before flag declarations used in these system calls. Note: this order is usually the exact opposite of how things are declared in C: the least important things go first. ## Description compilation internals The process of compiling the textual syscall descriptions into machine-usable form used by `syzkaller` to actually generate programs consists of 2 steps. The first step is extraction of values of symbolic constants from kernel sources using [syz-extract](/sys/syz-extract) utility. `syz-extract` generates a small C program that includes kernel headers referenced by `include` directives, defines macros as specified by `define` directives and prints values of symbolic constants. Results are stored in `.const` files, one per arch. For example, [sys/linux/dev_ptmx.txt](/sys/linux/dev_ptmx.txt) is translated into [sys/linux/dev_ptmx.txt.const](/sys/linux/dev_ptmx.txt.const). The second step is translation of descriptions into Go code using [syz-sysgen](/sys/syz-sysgen) utility (the actual compiler code lives in [pkg/ast](/pkg/ast/) and [pkg/compiler](/pkg/compiler/)). This step uses syscall descriptions and the const files generated during the first step and produces instantiations of `Syscall` and `Type` types defined in [prog/types.go](/prog/types.go). You can see an example of the compiler output for Linux/AMD64 in `sys/linux/gen/amd64.go`. This step also generates some minimal syscall metadata for C++ code in `executor/syscalls.h`. ## Non-mainline subsystems `make extract` extracts constants for all `*.txt` files and for all supported architectures. This may not work for subsystems that are not present in mainline kernel or if you have problems with native kernel compilers, etc. In such cases the `syz-extract` utility used by `make extract` can be run manually for single file/arch as: ``` make bin/syz-extract bin/syz-extract -os linux -arch $ARCH -sourcedir $KSRC -builddir $LINUXBLD .txt make generate make ``` `$ARCH` is one of `amd64`, `386` `arm64`, `arm`, `ppc64le`, `mips64le`. If the subsystem is supported on several architectures, then run `syz-extract` for each arch. `$KSRC` should point to kernel source checkout, which is configured for the corresponding arch (i.e. you need to run `make ARCH=arch someconfig && make ARCH=arch` there first, remember to add `CROSS_COMPILE=arm-linux-gnueabi-/aarch64-linux-gnu-/powerpc64le-linux-gnu-` if needed). If the kernel was built into a separate directory (with `make O=output_dir`, remember to put .config into output_dir, this will be helpful if you'd like to work on different arch at the same time) then also set `$LINUXBLD` to the location of the build directory.
### Testing of descriptions Descriptions themselves may contain bugs. After running `syz-manager` with the new descriptions it's always useful to check the kernel code coverage report available in the `syz-manager` web UI. The report allows to assess if everything one expects to be covered is in fact covered, and if not, where the fuzzer gets stuck. However, this is a useful but quite indirect assessment of the descriptions correctness. The fuzzer may get around some bugs in the descriptions by diverging from what the descriptions say, but it makes it considerably harder for the fuzzer to progress. Tests stored in `sys/OS/test/*` provide a more direct testing of the descriptions. Each test is just a program with checked syscall return values. The syntax of the programs is briefly described [here](program_syntax.md). You can also look at the [existing examples](/sys/linux/test) and at the program [deserialization code](/prog/encoding.go). `AUTO` keyword can be used as a value for consts and pointers, for pointers it will lead to some reasonable sequential allocation of memory addresses. It's always good to add a test at least for "the main successful scenario" for the subsystem. It will ensure that the descriptions are actually correct and that it's possible for the fuzzer to come up with the successful scenario. See [io_uring test](/sys/linux/test/io_uring) as a good example. The tests can be run with the `run-tests` functionality of `syz-manager`: ``` make && bin/syz-manager -config manager.config -mode run-tests ``` It will boot multiple VMs and runs these tests in different execution modes inside of the VMs. However, the full run takes significant time, so while developing the test, it's more handy to run it using the `syz-execprog` utility. To run the test, copy `syz-execprog`, `syz-executor` and the test into a manually booted VM and then run the following command inside of the VM: ``` syz-execprog -debug -threaded=0 mytest ``` It will show results of all executed syscalls. It's also handy for manual debugging of pseudo-syscall code: if you add some temporal `debug` calls to the pseudo-syscall, `syz-execprog -debug` will show their output. The test syntax can be checked by running: ``` go test -run=TestParsing ./pkg/runtest ``` --- ### Syscall Descriptions Syntax (`docs/syscall_descriptions_syntax.md`) # Syscall description language aka `syzlang` (`[siːzˈlæŋg]`) Pseudo-formal grammar of syscall description: ``` syscallname "(" [arg ["," arg]*] ")" [type] ["(" attribute* ")"] arg = argname type argname = identifier type = typename [ "[" type-options "]" ] typename = "const" | "intN" | "intptr" | "flags" | "array" | "ptr" | "string" | "filename" | "glob" | "len" | "bytesize" | "bytesizeN" | "bitsize" | "vma" | "proc" | "compressed_image" type-options = [type-opt ["," type-opt]] ``` common type-options include: ``` "opt" - the argument is optional (like mmap fd argument, or accept peer argument) ``` rest of the type-options are type-specific: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` flags/len/flags also have trailing underlying type type-option when used in structs/unions/pointers. Flags are described as: ``` flagname = const ["," const]* ``` or for string flags as: ``` flagname = "\"" literal "\"" ["," "\"" literal "\""]* ``` Call attributes are: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Ints `int8`, `int16`, `int32` and `int64` denote an integer of the corresponding size. `intptr` denotes a pointer-sized integer, i.e. C `long` type. By appending `be` suffix (e.g. `int16be`) integers become big-endian. It's possible to specify a range of values for an integer in the format of `int32[0:100]` or `int32[0:4096, 512]` for a 512-aligned int. Integers can also take a reference to flags description or a value as its first type-option. In that case, the alignment parameter is not supported. To denote a bitfield of size N use `int64:N`. It's possible to use these various kinds of ints as base types for `const`, `flags`, `len` and `proc`. ``` example_struct { f0 int8 # random 1-byte integer f1 const[0x42, int16be] # const 2-byte integer with value 0x4200 (big-endian 0x42) f2 int32[0:100] # random 4-byte integer with values from 0 to 100 inclusive f3 int32[1:10, 2] # random 4-byte integer with values {1, 3, 5, 7, 9} f4 int64:20 # random 20-bit bitfield f5 int8[10] # const 1-byte integer with value 10 f6 int32[flagname] # random 4-byte integer from the set of values referenced by flagname } ``` ## Structs Structs are described as: ``` structname "{" "\n" (fieldname type ("(" fieldattribute* ")")? (if[expression])? "\n")+ "}" ("[" attribute* "]")? ``` Fields can have attributes specified in parentheses after the field, independent of their type. `in/out/inout` attribute specify per-field direction, for example: ``` foo { field0 const[1, int32] (in) field1 int32 (inout) field2 fd (out) } ``` You may specify conditions that determine whether a field will be included: ``` foo { field0 int32 field1 int32 (if[value[field0] == 0x1]) } ``` See [the corresponding section](syscall_descriptions_syntax.md#conditional-fields) for more details. `out_overlay` attribute allows to have separate input and output layouts for the struct. Fields before the `out_overlay` field are input, fields starting from `out_overlay` are output. Input and output fields overlap in memory (both start from the beginning of the struct in memory). For example: ``` foo { in0 const[1, int32] in1 flags[bar, int8] in2 ptr[in, string] out0 fd (out_overlay) out1 int32 } ``` Structs can have attributes specified in square brackets after the struct. Attributes are: - `packed`: the struct does not have paddings between fields and has alignment 1; this is similar to GNU C `__attribute__((packed))`; struct alignment can be overridden with `align` attribute - `align[N]`: the struct has alignment N and padded up to multiple of `N`; contents of the padding are unspecified (though, frequently are zeros); similar to GNU C `__attribute__((aligned(N)))` - `size[N]`: the struct is padded up to the specified size `N`; contents of the padding are unspecified (though, frequently are zeros) ## Unions Unions are described as: ``` unionname "[" "\n" (fieldname type (if[expression])? "\n")+ "]" ("[" attribute* "]")? ``` During fuzzing, syzkaller randomly picks one of the union options. You may also specify conditions that determine whether the corresponding option may or may not be selected, depending on values of other fields. See [the corresponding section](syscall_descriptions_syntax.md#conditional-fields) for more details. Unions can have attributes specified in square brackets after the union. Attributes are: - `varlen`: union size is the size of the particular chosen option (not statically known); without this attribute unions are statically sized as maximum of all options (similar to C unions) - `size[N]`: the union is padded up to the specified size `N`; contents of the padding are unspecified (though, frequently are zeros) ## Resources Resources represent values that need to be passed from output of one syscall to input of another syscall. For example, `close` syscall requires an input value (fd) previously returned by `open` or `pipe` syscall. To achieve this, `fd` is declared as a resource. This is a way of modelling dependencies between syscalls, as defining a syscall as the producer of a resource and another syscall as the consumer defines a loose sense of ordering between them. Resources are described as: ``` "resource" identifier "[" underlying_type "]" [ ":" const ("," const)* ] ``` `underlying_type` is either one of `int8`, `int16`, `int32`, `int64`, `intptr` or another resource (which models inheritance, for example, a socket is a subtype of fd). The optional set of constants represent resource special values, for example, `0xffffffffffffffff` (-1) for "no fd", or `AT_FDCWD` for "the current dir". Special values are used once in a while as resource values. If no special values specified, special value of `0` is used. Resources can then be used as types, for example: ``` resource fd[int32]: 0xffffffffffffffff, AT_FDCWD, 1000000 resource sock[fd] resource sock_unix[sock] socket(...) sock accept(fd sock, ...) sock listen(fd sock, backlog int32) ``` Resources don't have to be necessarily returned by a syscall. They can be used as any other data type. For example: ``` resource my_resource[int32] request_producer(..., arg ptr[out, my_resource]) request_consumer(..., arg ptr[inout, test_struct]) test_struct { ... attr my_resource } ``` For more complex producer/consumer scenarios, field attributes can be utilized. For example: ``` resource my_resource_1[int32] resource my_resource_2[int32] request_produce1_consume2(..., arg ptr[inout, test_struct]) test_struct { ... field0 my_resource_1 (out) field1 my_resource_2 (in) } ``` Each resource type must be "produced" (used as an output) by at least one syscall (outside of unions and optional pointers) and "consumed" (used as an input) by at least one syscall. ## Type Aliases Complex types that are often repeated can be given short type aliases using the following syntax: ``` type identifier underlying_type ``` For example: ``` type signalno int32[0:65] type net_port proc[20000, 4, int16be] ``` Then, type alias can be used instead of the underlying type in any contexts. Underlying type needs to be described as if it's a struct field, that is, with the base type if it's required. However, type alias can be used as syscall arguments as well. Underlying types are currently restricted to integer types, `ptr`, `ptr64`, `const`, `flags` and `proc` types. There are some builtin type aliases: ``` type bool8 int8[0:1] type bool16 int16[0:1] type bool32 int32[0:1] type bool64 int64[0:1] type boolptr intptr[0:1] type fileoff[BASE] BASE type filename string[filename] type buffer[DIR] ptr[DIR, array[int8]] ``` ## Type Templates Type templates can be declared as follows: ``` type buffer[DIR] ptr[DIR, array[int8]] type fileoff[BASE] BASE type nlattr[TYPE, PAYLOAD] { nla_len len[parent, int16] nla_type const[TYPE, int16] payload PAYLOAD } [align_4] ``` and later used as follows: ``` syscall(a buffer[in], b fileoff[int64], c ptr[in, nlattr[FOO, int32]]) ``` There is builtin type template `optional` defined as: ``` type optional[T] [ val T void void ] [varlen] ``` ## Length You can specify length of a particular field in struct or a named argument by using `len`, `bytesize` and `bitsize` types, for example: ``` write(fd fd, buf ptr[in, array[int8]], count len[buf]) sock_fprog { len len[filter, int16] filter ptr[in, array[sock_filter]] } ``` If `len`'s argument is a pointer, then the length of the pointee argument is used. To denote the length of a field in N-byte words use `bytesizeN`, possible values for N are 1, 2, 4 and 8. To denote the length of the parent struct, you can use `len[parent, int8]`. To denote the length of the higher level parent when structs are embedded into one another, you can specify the type name of the particular parent: ``` s1 { f0 len[s2] # length of s2 } s2 { f0 s1 f1 array[int32] f2 len[parent, int32] } ``` `len` argument can also be a path expression which allows more complex addressing. Path expressions are similar to C field references, but also allow referencing parent and sibling elements. A special reference `syscall` used in the beginning of the path allows to refer directly to the syscall arguments. For example: ``` s1 { a ptr[in, s2] b ptr[in, s3] c array[int8] } s2 { d array[int8] } s3 { # This refers to the array c in the parent s1. e len[s1:c, int32] # This refers to the array d in the sibling s2. f len[s1:a:d, int32] # This refers to the array k in the child s4. g len[i:j, int32] # This refers to syscall argument l. h len[syscall:l, int32] i ptr[in, s4] } s4 { j array[int8] } foo(k ptr[in, s1], l ptr[in, array[int8]]) ``` ## Proc The `proc` type can be used to denote per process integers. The idea is to have a separate range of values for each executor, so they don't interfere. The simplest example is a port number. The `proc[20000, 4, int16be]` type means that we want to generate an `int16be` integer starting from `20000` and assign `4` values for each process. As a result the executor number `n` will get values in the `[20000 + n * 4, 20000 + (n + 1) * 4)` range. ## Integer Constants Integer constants can be specified as decimal literals, as `0x`-prefixed hex literals, as `'`-surrounded char literals, or as symbolic constants extracted from kernel headers or defined by `define` directives. For example: ``` foo(a const[10], b const[-10]) foo(a const[0xabcd]) foo(a int8['a':'z']) foo(a const[PATH_MAX]) foo(a int32[PATH_MAX]) foo(a ptr[in, array[int8, MY_PATH_MAX]]) define MY_PATH_MAX PATH_MAX + 2 ``` ## Conditional fields ### In structures In syzlang, it's possible to specify a condition for every struct field that determines whether the field should be included or omitted: ``` header_fields { magic const[0xabcd, int16] haveInteger int8 } [packed] packet { header header_fields integer int64 (if[value[header:haveInteger] == 0x1]) body array[int8] } [packed] some_call(a ptr[in, packet]) ``` In this example, the `packet` structure will include the field `integer` only if `header.haveInteger == 1`. In memory, `packet` will have the following layout: | header.magic = 0xabcd | header.haveInteger = 0x1 | integer | body | | --------------------- | ------------------------ | ------- | ---- | That corresponds to e.g. the following program: ``` some_call(&AUTO={{AUTO, 0x1}, @value=0xabcd, []}) ``` If `header.haveInteger` is not `1`, syzkaller will just pretend that the field `integer` does not exist. ``` some_call(&AUTO={{AUTO, 0x0}, @void, []}) ``` | header.magic = 0xabcd | header.haveInteger = 0x0 | body | | --------------------- | ------------------------ | ---- | Every conditional field is assumed to be of variable length and so is the struct to which this field belongs. When a variable length field appears in the middle of a structure, the structure must be marked with `[packed].` Conditions on bitfields are prohibited: ``` struct { f0 int f1 int:3 (if[value[f0] == 0x1]) # It will not compile. } ``` But you may reference bitfields in your conditions: ``` struct { f0 int:1 f1 int:7 f2 int (if[value[f0] == value[f1]]) } [packed] ``` ### In unions Let's consider the following example. ``` struct { type int body alternatives } alternatives [ int int64 (if[value[struct:type] == 0x1]) arr array[int64, 5] (if[value[struct:type] == 0x2]) default int32 ] [varlen] some_call(a ptr[in, struct]) ``` In this case, the union option will be selected depending on the value of the `type` field. For example, if `type` is `0x1`, then it can be either `int` or `default`: ``` some_call(&AUTO={0x1, @int=0x123}) some_call(&AUTO={0x1, @default=0x123}) ``` If `type` is `0x2`, it can be either `arr` or `default`. If `type` is neither `0x1` nor `0x2`, syzkaller may only select `default`: ``` some_call(&AUTO={0x0, @default=0xabcd}) ``` To ensure that a union can always be constructed, the last union field **must always have no condition**. Thus, the following definition would fail to compile: ``` alternatives [ int int64 (if[value[struct:type] == 0x1]) arr array[int64, 5] (if[value[struct:type] == 0x1]) ] [varlen] ``` During prog mutation and generation syzkaller will select a random union field whose condition is satisfied. ### Expression syntax Currently, only `==`, `!=`, `&` and `||` operators are supported. However, the functionality was designed in such a way that adding more operators is easy. Feel free to file a GitHub issue or write us an email in case it's needed. Expressions are evaluated as `int64` values. If the final result of an expression is not 0, it's assumed to be satisfied. If you want to reference a field's value, you can do it via `value[path:to:field]`, which is similar to the `len[]` argument. ``` sub_struct { f0 int # Reference a field in a parent struct. f1 int (if[value[struct:f2]]) # Same as if[value[struct:f2] != 0]. } struct { f2 int f3 sub_struct f4 int (if[value[f2] == 0x2]) # Reference a sibling field. f5 int (if[value[f3:f0] == 0x1]) # Reference a nested field. f6 int (if[value[f3:f0] == 0x1 || value[f3:f0] == 0x2]) # Reference a nested field which either equals to 0x1 or 0x2. } [packed] call(a ptr[in, struct]) ``` The referenced field must be of integer type and there must be no conditional fields in the path to it. For example, the following descriptions will not compile. ``` struct { f0 int f1 int (if[value[f0] == 0x1]) f2 int (if[value[f1] == 0x1]) } ``` You may also reference constants in expressions: ``` struct { f0 int f1 int f2 int (if[value[f0] & SOME_CONST == OTHER_CONST]) } ``` ## Meta Description files can also contain `meta` directives that specify meta-information for the whole file. ``` meta noextract ``` Tells `make extract` to not extract constants for this file. Though, `syz-extract` can still be invoked manually on this file. ``` meta arches["arch1", "arch2"] ``` Restricts this file only to the given set of architectures. `make extract` and `make generate` will not use it on other architectures. ## Misc Description files also contain `include` directives that refer to Linux kernel header files, `incdir` directives that refer to custom Linux kernel header directories and `define` directives that define symbolic constant values. The syzkaller executor defines some [pseudo system calls](./pseudo_syscalls.md) that can be used as any other syscall in a description file. These pseudo system calls expand to literal C code and can perform user-defined custom actions. You can find some examples in [executor/common_linux.h](../executor/common_linux.h). Also see [tips](syscall_descriptions.md#tips) on writing good descriptions. --- ### Syz Dungeon (`docs/syz-dungeon.md`) # Syzkaller Dungeon ## Overview The **Syzkaller Dungeon** (/syz-dungeon) ranks contributors (kernel developers) based on their achievements in fixing bugs found by Syzkaller. It uses a retro "terminal hacker" aesthetic with green-on-black text and ASCII art. Note: Currently, the Dungeon is only available for the `upstream` namespace. ## Core Mechanics ### 1. Character Generation Each contributor is assigned a "Character" based on their bug fixing history during a specific time period. **Eras (Time Ranges)** The Dungeon provides two separate views or "Eras" for statistics: * **1 Year (Default)**: A rolling window of exactly the last 365 days from today. * **All Time**: A comprehensive view of all bugs fixed since the beginning of tracking. * **Era Scoping & Decay**: In the 1-Year Era, the **entire character sheet** (including XP, Level, Class, Badges, and Stats like STR/INT/WIS) is dynamically recalculated based exclusively on the bugs currently within the 365-day rolling window. As fixes age out of the window, these attributes decay. In the All-Time Era, bugs never fall out of the window, providing a permanent, non-decaying global view of the contributor's achievements. * **Eligibility**: Bugs must have landed a fix commit within the given period. * **Criteria**: Includes both fully fixed bugs (on the **Fixed** page) and partially fixed bugs (still **Open** but with upstream commits landing in the given period). * **Co-Authored Fixes**: If a bug is resolved by multiple commits, **all unique commit authors** recorded in the bug's history are independently granted 1 bug count and the associated XP/Score. This means a single bug can boost multiple heroes simultaneously, occasionally resulting in leaderboard totals that are higher than naive SQL queries that only map one author per bug. * **Date**: The "date fixed" is determined strictly by `FixTime` (when the fix commit was discovered). `Closed` date is ignored. * **Matching**: Identity resolution uses **strict exact matching** on the lowercase **Email** address (`Author` field). * **No Email Grouping**: Contributors using different email addresses (e.g., `@gmail.com` vs a corporate domain) are treated as separate characters. * **Hero ID**: Each hero is assigned a stable, 16-character hexadecimal ID. This is generated by calculating the hash of their **lowercased email address** (using SHA-1) and extracting the first 16 characters of the hexadecimal string representation of the hash. This provides a secure, deterministic identifier for individual Hero Profile Pages without exposing raw email addresses in URLs. * **Hero Name**: Derived from the `AuthorName` field in the Datastore `CommitInfo` entity, bound to the matched Email. * If a contributor uses multiple names for the same email (e.g. `John Doe` vs `J. Doe`), the system deterministically assigns the **most frequently used name** to the Hero. Ties are broken alphabetically. * **Fallback**: If an email *never* encounters a valid author name across all of its recorded bugs, the hero's name will gracefully fall back to displaying just the raw email address. * **Bots**: Bots are implicitly excluded as only humans land fixing commits in the Linux kernel (even bot-generated patches are signed by a human). * **Heroic Title**: An adjective is appended to the name based on the first letter, e.g., "Eric Dumazet The Epic". * **Full List**: * `A`: The Arcane * `B`: The Brave * `C`: The Cosmic * `D`: The Devoted * `E`: The Epic * `F`: The Fierce * `G`: The Gallant * `H`: The Heroic * `I`: The Immortal * `J`: The Just * `K`: The Knightly * `L`: The Legendary * `M`: The Mystic * `N`: The Noble * `O`: The Omniscient * `P`: The Prime * `Q`: The Quick * `R`: The Radiant * `S`: The Sacred * `T`: The Titan * `U`: The Undying * `V`: The Valiant * `W`: The Wise * `X`: The Xenial * `Y`: The Youthful * `Z`: The Zealous * *Identity:* Defined by the dynamic **Hero Name** calculated above (which selects the first valid name encountered for the email, or falls back to the raw email). * **No Title for Fallbacks**: If the Hero Name had to fall back to the raw Email because no valid name was found across their commits, **no heroic title modifier is added**. The hero is simply displayed as their address (e.g. `raw.email@domain.com`). * **Logic**: The title is determined by the **first letter of the resolved name**. * **Normalization**: The first letter is **lowercased** before lookup. * **Map**: 26 adjectives corresponding to 'a'-'z'. * **Fallback**: If the name begins with a non-alphabetic character (numbers, symbols, or non-Latin scripts not in the a-z map), the title defaults to **"The Adventurer"**. This fallback applies universally without transliteration (no UTF-8 awareness). * *Note:* There is currently no mechanism for manual identity overrides or merging separate profiles. * **Class**: Determined by analyzing the **existing tags** attached to the bug. * **Multi-Class Credit**: A contributor receives exactly **1 point** for **every** subsystem tag present on a fixed bug. For example, a bug tagged with `net` and `mm` increments the contributor's count for both `net` and `mm` by 1. * **Dynamic Class**: The "Main Class" is determined **after** calculating the totals for all valid bugs. The system finds the single subsystem tag that appears most frequently across all of the contributor's fixes, and assigns the Class corresponding to that winning tag. * **Source**: Classes are determined by mapping the winning **tag** (subsystem) to the RPG categories below. * **Weaver** (🕸️): Networking & BPF. * Tags: `net`, `ipv4`, `ipv6`, `tcp`, `udp`, `sctp`, `bpf`, `wireless`, `bluetooth`, `can`, `tipc`, `nfc`, `bridge`, `netlink`. * **Geomancer** (🏔️): Filesystems & Storage. * Tags: `fs`, `ext4`, `btrfs`, `xfs`, `vfs`, `block`, `scsi`, `nvme`, `io-uring`, `fuse`, `overlayfs`. * **Voidwalker** (🔮): Memory Management. * Tags: `mm`, `kasan`, `slab`, `kcsan`, `kmemleak`. * **Paladin** (🛡️): Security & Crypto. * Tags: `security`, `selinux`, `apparmor`, `lsm`, `audit`, `keyrings`, `crypto`. * **Bard** (🎻): Media, Sound, Input, Human Interface. * Tags: `sound`, `usb`, `media`, `video`, `drm`, `gpu`, `input`, `hid`, `drivers` (generic). * **Artificer** (🛠️): Hardware Buses & Infrastructure. * Tags: `pci`, `i2c`, `spi`, `gpio`, `clk`, `pm`, `dmaengine`, `iio`, `rtc`, `watchdog`. * **Warlock** (🌀): Virtualization. * Tags: `kvm`, `xen`, `virt`, `hyperv`. * **Rogue** (🗡️): Architecture Specific. * Tags: `arch`, `arm`, `arm64`, `x86`, `riscv`, `s390`. * **Druid** (🌿): Core Kernel & Sync. * Tags: `kernel`, `sched`, `rcu`, `locking`, `trace`, `cgroups`. * **Warrior** (⚔️): Default class for other subsystems or if **no tags** match any class. * *Tie-Breaking:* If a contributor has multiple tags with the exact same highest frequency count, the **first subsystem tag alphabetically** is chosen as the winner. Warrior is only used if there are no specialized tags. ### 2. Experience (XP) Calculation Score is calculated for each fixed bug: * **Base XP**: 100 * **Boss Bonus**: 10% of crash count (Capped at 500 XP). * Formula: `ceil(NumCrashes * 0.1)` * *Note:* If the bug data is missing crash counts or has 0 crashes, this bonus is 0. * **Ancient Evil Bonus**: 1 XP for every day (measured as a precise 24-hour elapsed duration, i.e., `hours / 24`) the bug was open (Capped at 300 XP). * *Note:* if `FirstTime` is missing, this bonus defaults to 0. * *Floor:* If duration is negative (due to clock skew), it is treated as **0 days** (0 bonus XP). This prevents "negative XP". * **Windwalker Bonus**: +250 XP flat bonus if the bug was fixed within 7 days. This bonus defaults to 0 if either the `FirstTime` or `FixTime` timestamps are missing. * **Blind Bonus**: 150 XP if the bug had no reproducer (`ReproLevelNone`). ### 3. Leveling * **Level 1**: 0 XP * **Progressive Level Base**: 1,000 XP per level up to Level 50. * **Legendary Scaling**: Starting at Level 50, each subsequent level requires 500 *more* XP than the previous level's requirement (e.g., Level 50 -> 51 costs 1,500 XP, 51 -> 52 costs 2,000 XP). * **Mathematical Formula**: The total XP required to reach Level $N$, denoted as $TotalXP(N)$, is calculated strictly by the following piecewise function: * For $N \le 50$: $TotalXP(N) = (N - 1) \times 1000$ * For $N > 50$: Let $m = N - 50$. The total XP is defined by the arithmetic series sum: $TotalXP(N) = 49000 + 1000m + 250m(m + 1)$ * **Concrete Threshold Table**: * `Level 1`: 0 XP * `Level 2`: 1,000 XP * `Level 49`: 48,000 XP * `Level 50`: 49,000 XP * `Level 51`: 50,500 XP * `Level 52`: 52,500 XP * `Level 53`: 55,000 XP * **Cap**: There is no hard level cap. * **Note**: Rapid leveling (e.g., jumping 2.5 levels from a single legendary bug fix) is **intentional** to reward high-impact contributions. ### 4. Inventory (Badges) Contributors earn badges for specific achievements. **Badges are purely cosmetic** and do not affect XP or Rank. **General Badge Rules:** * All string/title matching is **case-insensitive**. * Bugs missing `FirstTime` or `FixTime` fields cannot be evaluated and therefore do not count toward time-based badges. * **Dragon Vanquisher** (🐉): Fixed a bug with >1000 crashes. * **Necromancer** (💀): Fixed a bug open for >365 days. * **Diviner** (👁️): Fixed a bug with no reproducer. * **Windwalker** (🪽): Fixed a bug within **7 days** (measured as precise 24-hour duration chunks) of it being reported (based on `FirstTime`). * *Logic:* `FixTime - FirstTime < 7 days`. * *Data Integrity:* This calculation is **skipped** (badge not awarded) if either `FirstTime` or `FixTime` timestamps are missing. * **🔥 Hydra Hunter**: Fixed 10 or more bugs overall. * **🧌 Troll Crusher**: Fixed 50 or more bugs overall. * **🐙 Kraken Bane**: Fixed 100 or more bugs overall. * **👹 Demon Slayer**: Fixed 250 or more bugs overall. * **🔱 Leviathan Conqueror**: Fixed 500 or more bugs overall. * **🏺 Sealer**: Fixed a memory leak. * *Logic:* Title or Fix Commit Title contains tokens `leak`, `leaks`, `memory-leak`, or `memory-leaks` (with word boundaries checking). * **📜 Keeper of Secrets**: Fixed an *infoleak* vulnerability. * *Logic:* Title or Fix Commit Title contains `infoleak` (matched as a simple substring, without word boundaries or special character tokenization). * **⏳ Master of Time**: Fixed a time-related bug. * *Logic:* Title or Fix Commit Title contains `time:` or `timer:` (matched as a simple substring, without word boundaries or special character tokenization, matching things like `hrtimer:` or `posix-timer:`). * **🛶 Pathfinder**: Fixed an *out-of-bounds* error. (String match: `out-of-bounds` anywhere in bug title/commit titles). * **🦄 Beast Tamer**: Fixed a *data race* condition. (String match: `data-race` anywhere in bug title/commit titles). * **⚒️ Armorer**: Fixed a *general protection fault*. (String match: `general protection fault` anywhere in bug title/commit titles). * **🪦 Grave Robber**: Fixed a *use-after-free* error. (String match: `use-after-free` anywhere in bug title/commit titles). * **🗝️ Locksmith**: Fixed a *locking* issue. (Matches tokens: `lock`, `deadlock`, `lockup`, `locking`, `mutex`, `unlock`, `locks`, `rwlock`, `spinlock` in bug title/commit titles). * **🕯️ Exorcist**: Fixed a *hung task* or *zombie process*. (String match: `hung task`, `task hung`, or `zombie` anywhere in bug title/commit titles). * **🕳️ Abyss Walker**: Fixed a *NULL pointer dereference*. (String match: `null-ptr-deref` or `null pointer` anywhere in bug title/commit titles). * **⚗️ Alchemist**: Fixed an *uninitialized value* bug. (String match: `uninit` or `kmsan` anywhere in bug title/commit titles). * **🪓 Executioner**: Fixed a *double free* bug. (String match: `double-free` or `double free` anywhere in bug title/commit titles). ### 5. General Rules * **Alphabetical Tie-Breakers**: Unless otherwise specified, all "alphabetical" sorting and tie-breaking used throughout the Dungeon (for Names, Kingdom Main Classes, Guilds Array, Kingdom Ranks, etc.) is strictly **case-sensitive** and **lexicographically ascending**. This guarantees absolute determinism derived from underlying ASCII/UTF-8 byte representations (e.g., uppercase 'Z' sorts before lowercase 'a'). ### 6. Kingdoms Players can view aggregate statistics via two toggles in the dashboard menu: * **Entities**: Choose between viewing the standard individual **Heroes** leaderboard, or the **Kingdoms** ranking. * **Kingdoms**: View aggregated points and heroic classes across different organizations playing the dungeon. * Clicking a Kingdom's name takes you to the **Kingdom Profile Page** where you can see their overall Trophies, Hero Count, Total XP, and established Guilds. * Employees are assigned to their company based on a curated list of email domains. The matching rule requires an **exact string match** against the entire portion of the email address following the `@` symbol (e.g., `developer@corp.company.com` will *not* match `company.com`). * **The Google Kingdom**: `google.com`, `chromium.org` * **The Red Hat Kingdom**: `redhat.com` * **The Huawei Kingdom**: `huawei.com`, `hisilicon.com` * **The Linux Foundation Kingdom**: `linuxfoundation.org`, `kernel.org` * **The Meta Kingdom**: `meta.com`, `fb.com` * **The Intel Kingdom**: `intel.com`, `linux.intel.com` * **The AMD Kingdom**: `amd.com` * **The ARM Kingdom**: `arm.com` * **The Oracle Kingdom**: `oracle.com` * **The SUSE Kingdom**: `suse.com`, `suse.de`, `suse.cz` * **The Linaro Kingdom**: `linaro.org` * **The Alibaba Kingdom**: `linux.alibaba.com` * **The NVIDIA Kingdom**: `nvidia.com`, `mellanox.com` * **The IBM Kingdom**: `linux.ibm.com`, `linux.vnet.ibm.com` * **The Bytedance Kingdom**: `bytedance.com` * **The Canonical Kingdom**: `canonical.com`, `ubuntu.com` * **The Amazon Kingdom**: `amazon.com`, `amazon.de`, `amazon.co.uk`, `amazon.co.jp` * **The Tencent Kingdom**: `tencent.com` * **The Samsung Kingdom**: `samsung.com` * Any hero not in the curated list belongs to `The Independent Mercenaries`. * **Royal Champion**: A Kingdom's Royal Champion is determined by finding the single hero associated with that Kingdom who has the highest `Score` (XP) contribution in the given Era. The Champion's Name, XP contribution, and a link to their profile are featured on the Kingdom's Profile Page. * **Total Hero XP**: A Kingdom's total `Score` is purely the combined sum of the `Score` of all its affiliated `Heroes`. This means all dynamic bug bonuses (like the Boss Bonus for crashes, Ancient Evil bonus for old bugs, etc.) earned by a hero directly benefit their aligned Kingdom. However, the Kingdom's hero count and specific class composition metric will correctly reflect all distinct individuals who participated. If the fix is co-authored by heroes from **different** Kingdoms, **both** Kingdoms independently receive full credit for the bug. * **Trophy Deduplication**: To prevent double-counting, if a bug is resolved jointly by multiple heroes who belong to the **same** Kingdom, the Kingdom is credited exactly **1 Trophy** for resolving the bug. * **Stats**: Kingdoms do not possess individual stats like STR, INT, or WIS. * Instead of "Inventory" (Badges), Kingdoms display **"Guilds"**, which details the composition of Hero Classes based on the **total count of players** in the Kingdom (e.g., "3 Weavers", "1 Geomancer"). The array is sorted descending by the count of players, and then alphabetically by the resulting string representation in the event of a tie. Pluralization is handled simply: an `"s"` is appended to the Class Name if the player count is strictly greater than 1. * **Sorting**: Kingdoms are ranked and sorted descending by their **Total XP** (Score). Ties are broken alphabetically by the Kingdom Name. * **Territory Tiers**: A Kingdom evaluates its overall combined `Score` (XP) strictly into one of the following brackets, displaying an associated name and emoji. On the **Kingdom Profile** page, the tier icon is displayed prominently with the tier name below and an XP range tooltip. On the global **Leaderboard**, the tier emoji is displayed as an inline badge next to the Kingdom's name (with a tooltip revealing the tier name). * 0 - 49,999: Outpost (⛺) * 50,000 - 149,999: Barony (🏘️) * 150,000 - 499,999: Duchy (🏛️) * 500,000 - 999,999: Kingdom (🏰) * 1,000,000+: Empire (👑) ## Technical Implementation ### Backend (`dashboard/app/dungeon.go`) * **JSON API**: `/syz-dungeon?json=1` returns the **calculated scaled stats** (RPG range [3-30]) for `str`, `int`, `wis`. * **Endpoint**: `/syz-dungeon` (Leaderboard Overview) * **Hero Profile Endpoint**: `/syz-dungeon/hero/{id}` (Individual Hero Sheet) * **Parameters**: Supports `?json=1` (JSON dump) and `?era=1y` or `?era=all` (defaults to `all`). Hero pages will display an interactive `[1 YEAR]` | `[ALL TIME]` toggle if the hero has contributions mapped into both era scopes. * **Class Box**: Displays the hero's class emoji, name, and a tooltip containing the class's flavor text (description). * **Attributes Box**: Displays the hero's STR, INT, and WIS properties. Hovering over each stat reveals a tooltip explaining its corresponding metric (e.g., "Max Crashes (Scaled)"). * **Badges Box**: Displays earned badges. Hovering over a badge reveals its name and an extended description characterizing the achievement. * **Click-to-Copy ID**: The `ID` displayed in the footer is clickable and will copy the URL to the profile into the user's clipboard. * **Data Source**: Queries the `Bug` entity in Google Cloud Datastore. * **Definition**: `dashboard/app/entities_datastore.go` * **Key Fields Used**: * `Namespace`, `Status`: For Datastore query filtering. (Note: Only the `upstream` namespace is currently supported). * *Logic:* `Status` must be Fixed (or Open if fixing commits are present). * *Index:* Requires composite index on `(Namespace, Status)` in `index.yaml`. * `FixTime`: To filter bugs within the given period (in **UTC**). * *Decay:* Since the cache refreshes hourly, players viewing the 1-Year Era will see their entire character sheet—including stats, badges, levels, classes, and ranks—actively decay day-to-day as old bugs fall out of the 365-day window. **No notifications** are sent for level down; users monitor their status on the dashboard. * `Tags`: For Class determination (instead of Title). * `CommitInfo`: For authorship identification (`AuthorName` and `Author` fields). * `Title`: For "Sealer", "Keeper of Secrets", and "Master of Time" badge logic. * `NumCrashes`: For "Boss Bonus", "Dragon Slayer" badge, and **STR**. * `ReproLevel`: For "Blind Bonus", "Diviner" badge, and **INT**. * `FirstTime` & `FixTime`: For "Ancient Evil" bonus, "Windwalker"/"Necromancer" badges, and **WIS**. * **Caching**: Results are cached in shared **App Engine Memcache** (using compressed JSON) for 1 hour. The cache is segregated by the viewer's `AccessLevel` so that privileged bug data (like undisclosed bugs visible to admins) does not leak to public viewers. * *Warm-up/Preheating*: A cron job (`/cron/dungeon_preheat`) runs hourly to precompute and warm the cache for all access levels simultaneously, ensuring fast page loads for all users across multiple live instances. * **Logic**: * **Identity Resolution**: Aggregates bugs strictly by the lowercased **Email** address. * *Rule:* "Linus Torvalds " and "Linus Torvalds " are treated as separate characters. * *Name Selection*: If an author uses multiple names for the same email address across different commits, the system assigns the **most frequently used name** to the Hero. Ties are broken alphabetically. * Determines the "Main Spec" (class) based on subsystem counts. * Calculates XP and assigns badges. * Sorts players by Score (descending), then Name (ascending), then Email (ascending). * **Pagination**: The backend server returns the complete active roster of players. The frontend JavaScript dynamically partitions the tables into interactive pages of 100 rows. ### Frontend (`dashboard/app/templates/dungeon.html`) * **Style**: Dark mode with `Courier New` / monospace font. * Accent Color: Neon Green (`#0f0`). * Banner: Retro pixel-art PNG image "SYZKALLER DUNGEON" (responsive, located at `/static/dungeon.png`). * **Interactivity**: * **Hover Tooltips**: Hovering over a character row reveals a "Character Sheet": * **Desktop Users**: Appears on **Hover**. * **Mobile/Touch Users**: Appears on **Click/Tap**. (Tapping a row toggles the sheet; opening a new one closes the previous one). * **NAME**: "Name Adjective" (e.g., "Eric Dumazet The Epic") * **CLASS**: "Emoji (Class)" (e.g., "🕸️ (Weaver)") * **LEVEL**: Current Level * **Stats Scaling**: * Instead of raw counts, metrics are mapped to an RPG-like scale **[3, 30]** using a logarithmic function. * **STR** (Strength): Based on **Max Crashes** of a single fixed bug. * Formula: $STR = \min(30, \max(3, \lfloor 3 + 4 \times \log_{10}(\max(1, \text{MaxCrashes})) \rfloor))$ * *Example:* 1 crash = 3, 100 crashes = 11, 1,000,000 crashes = 27. * **INT** (Intelligence): Based on total **Blind Fixes** (no reproducer). * Formula: $INT = \min(30, \max(3, \lfloor 3 + 10 \times \log_{10}(\text{BlindFixes} + 1) \rfloor))$ * *Example:* 0 blind fixes = 3, 2 blind fixes = 7, 10 blind fixes = 13, 500 blind fixes = 30. * **WIS** (Wisdom): Based on **Oldest Kill** (maximum duration of a **single** bug). * Formula: $WIS = \min(30, \max(3, \lfloor 3 + 5 \times \log_{10}(\text{DaysOpen} + 1) \rfloor))$ * *Example:* 1 day = 4, 100 days = 13, 1000 days = 18, 250,000 days = 30. * **Inventory Tooltips**: Hovering over badges shows their name/description. ## Configuration * **JSON Output**: The endpoint supports `?json=1` to return the leaderboard data as JSON. * **Schema**: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` * **Hero Profile Schema** (`/syz-dungeon/hero/{id}?json=1`): Returns a single JSON object mirroring the `heroes_1y` / `heroes_all` definition depending on the evaluated `?era=` query parameter. ```json { "id": "string", "name": "string", "rank": "int", "score": "int", "level": "int", "class_name": "string", "class_emoji": "string", "class_desc": "string", "badges": [{ "emoji": "string", "name": "string", "desc": "string" }], "bug_count": "int", "str": "int", "int": "int", "wis": "int" } ``` * **Kingdom Profile Schema** (`/syz-dungeon/kingdom/{id}?json=1`): Returns a single JSON object mirroring the `kingdoms_1y` / `kingdoms_all` definition depending on the evaluated `?era=` query parameter. ```json { "id": "string", "name": "string", "rank": "int", "score": "int", "tier": "string", "tier_emoji": "string", "tier_desc": "string", "champion_name": "string", "champion_score": "int", "champion_id": "string", "trophies": "int", "heroes": "int", "guilds": [{ "emoji": "string", "name": "string", "desc": "string" }] } ``` * **Heroic Adjectives**: Hardcoded map in `dungeon.go` for A-Z initials. --- ### Syz Kfuzztest (`docs/syz-kfuzztest.md`) # `syz-kfuzztest` `syz-kfuzztest` is a standalone tool for fuzzing KFuzzTest targets from within the kernel being fuzzed (e.g., a VM). It is intended to be used for development-time fuzzing rather than continuous fuzzing like `syz-manager`. For more information on KFuzzTest, consult the [dedicated readme](kfuzztest.md) or the Kernel documentation. ## Usage (in-VM fuzzing) ### Getting the Kernel Ready It is important that the target Kernel image has the correct KConfig options enabled. Namely - `CONFIG_KFUZZTEST` - `CONFIG_DEBUG_FS` - `CONFIG_DEBUG_KERNEL` - `CONFIG_KCOV` - `CONFIG_DWARF4` or `CONFIG_DWARF5` - `CONFIG_KASAN` _(optional, choose your favorite sanitizers for a better shot at finding bugs!)_ Furthermore, as you will need to connect to the VM being tested through SSH and launch `syz-kfuzztest` _(a Go binary with LIBC dependencies)_, it is recommended to create an image for the kernel being fuzzed (e.g., a Debian Trixie image). Detailed instructions on how to do this can be found in [this setup guide](linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md). ### Building and Launching the Binary The `syz-kfuzztest` binary is built with `make syz-kfuzztest`, and is intended to run on the Kernel fuzzed. The common case for this is within a VM _(after all, the tool is trying to make the Kernel crash)_. Then, ensure that the `syz-kfuzztest` binary and `vmlinux` image are copied over into the VM. E.g., ```sh scp $KERNEL/vmlinux root@my-vm:~/syz-kfuzztest/vmlinux scp $SYZKALLER/bin/syz-kfuzztest root@lmy-vm:~/syz-kfuzztest/syz-kfuzztest ``` Then launched like this: ``` usage: ./bin/syz-kfuzztest [flags] [enabled targets] Args: One fuzz test name per enabled fuzz test arg. If empty, defaults to all discovered targets. Example: ./syz-kfuzztest -vmlinux ~/kernel/vmlinux fuzz_target_0 fuzz_target_1 Flags: -display int Number of seconds between console outputs (default 5) -threads int Number of threads (default 2) -timeout int Timeout between program executions in seconds (default 0) -vmlinux string Path to vmlinux binary (default "vmlinux") -vv int verbosity ``` The enabled targets, which are listed after the flag arguments, are the names of the enabled fuzz targets. For example given some KFuzzTest targets: ```c FUZZ_TEST(kfuzztest_target_1, struct input_arg_type) { /* ... */ } FUZZ_TEST(kfuzztest_target_2, struct input_arg_type) { /* ... */ } ``` Can be fuzzed with: ```bash ./syz-kfuzztest -vmlinux path/to/vmlinux -threads 4 kfuzztest_target_1 kfuzztest_target_2 ``` If the enabled targets list is left empty, `syz-kfuzztest` will fuzz all discovered targets in the kernel. On exit, `syz-kfuzztest` will write the collected program counters (which are collected with KCOV) into a file called `pcs.out`. These program counters can be fed into [`syz-cover`](../tools/syz-cover/syz-cover.go) to generate an HTML visualization of the lines that were covered during fuzzing. It is recommended to do this on the host machine rather than the VM. For example: ```sh scp root@my-vm:~/syz-kfuzztest/pcs.out . go run tools/syz-cover -config my.cfg pcs.out # May require the -force flag. ``` --- ### Syz Testbed (`docs/syz_testbed.md`) # syz-testbed `syz-testbed` is the tool that simplifies the process of evaluating the performance of different syzkaller versions (or configurations) against each other. The tool automates checking out syzkaller repos, building them, running `syz-manager`s and collecting/summarizing their results. ## Configuring syz-testbed `syz-testbed` requires a JSON config file. An example of such a file: ```json { "workdir": "/tmp/syz-testbed-workdir/", "corpus": "/tmp/corpus.db", "target": "syz-manager", "max_instances": 5, "run_time": "24h", "http": "0.0.0.0:50000", "checkouts": [ { "name": "first", "repo": "https://github.com/google/syzkaller.git", }, { "name": "second", "repo": "https://github.com/google/syzkaller.git", "branch": "some-dev-branch", } ], "manager_config": { "target": "linux/amd64", "kernel_obj": "/tmp/linux-stable", "image": "/tmp/kernel-image/trixie.img", "sshkey": "/tmp/kernel-image/trixie.id_rsa", "procs": 8, "type": "qemu", "vm": { "count": 2, "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage", "cpu": 2, "mem": 2048 } } } ``` Given such a configuration file, `syz-testbed` will do the following: 1. Check out the `master` branch of `https://github.com/google/syzkaller.git` into `/tmp/syz-testbed-workdir/checkouts/first/` and build it. 2. Check out the `some-dev-branch` of `https://github.com/google/syzkaller.git` into `/tmp/syz-testbed-workdir/checkouts/second/` and build it. 3. Set up and run 3 instances of `first` and 2 instances of `second` (`max_instances = 5`). The directory structure looks as follows: ``` /tmp/syz-testbed-workdir/ └── checkouts    ├── first    │   ├── run-first-0    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    │   ├── run-first-1    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    │   └── run-first-4    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    └── second       ├── run-second-2       │   ├── log.txt       │   ├── manager.cfg       │   └── workdir       └── run-second-3          ├── log.txt          ├── manager.cfg          └── workdir ``` 4. After 24 hours (as `run_hours` is 24), stop those 5 instances. 5. Create and run 2 instances of `first` and 3 instances of `second`. 6. *Repeat those steps over and over* The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM signal. Also, if one of the instances has exited due to some error, this also stops the whole experiment. ## Web interface The tool has a simple web interface that displays the current information about the experiment (the number of active and finished instances, the time until instances are stopped, etc.) and the latest statistics collected from the `syz-manager`s. If the `benchmp` parameter points to the `syz-benchcmp` executable, then the web interface can also generate graphs of various parameters over time or the number of executions. In order to enable the interface, set the `http` parameter to the IP address and port to which `syz-testbed` should bind. E.g. `"http": "0.0.0.0:50000"`. ## Statistics `syz-testbed` provides two "views" of the statistics: 1. `complete` - only includes data from the finished instances (i.e. those that have been running for `run_hours`). 2. `all` - also includes the data from the currently active instances. The statistics from the finished instances is winded back to match the current uptime of the active instances. Therefore, the statistics is laid out the following way. ```bash $ tree -L 2 /tmp/syz-testbed-workdir/ /tmp/syz-testbed-workdir/ ├── stats_all │   ├── benches │   │   ├── avg_first.txt │   │   ├── avg_second.txt │   ├── bugs.csv │   ├── checkout_stats.csv │   └── instance_stats.csv ├── stats_completed │   ├── benches │   │   ├── avg_first.txt │   │   ├── avg_second.txt │   ├── bugs.csv │   ├── checkout_stats.csv │   └── instance_stats.csv └── testbed.csv ``` 1. `bugs.csv` contains all the bugs found by the running instances. If a single checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a union of bugs found by them. The purpose is ultimately to collect all bugs that could be found by that version of syzkaller. 2. Statistics that is generated by individual `syz-manager`s is saved into `instance_stats.csv`. The same data is also averaged among instances that belong to the same checkouts and saved into the `checkout_stats.csv` file. 3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a single checkout are averaged and saved into the corresponding files in the `benches` folder. The statics is updated once every 90 seconds. ## Running syz-testbed First, checkout the most recent version of syzkaller itself: ```shell $ git clone https://github.com/google/syzkaller.git ``` Then, build `syz-testbed`: ```shell $ cd syzkaller/tools/syz-testbed/ $ go build ``` Write and save the configuration file (e.g. into the `config.json` file). Then, `syz-testbed` can be run using the following command: ```shell $ ./syz-testbed -config config.json ``` Stopping the `syz-testbed` process results in stopping all the syzkaller instances. ## Testing syz-repro `syz-testbed` can also be used to test syzkaller's ability to reproduce bugs. To do so, set the `target` property in the `syz-testbed`'s config file to `syz-repro`. One can also specify the source of the crash log files. This is either just a folder, whose files will be treated accordingly or it can be a syzkaller's workdir. `input_logs` must point to the folder with crash logs - `syz-testbed` will traverse it and treat each file as an input. Otherwise, `input_workdir` must be used. For example: ```json "repro_config": { "input_workdir": "/tmp/some-syzkaller-workdir", "crashes_per_bug": 2, "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"] }, ``` In this case, `syz-testbed` will traverse all bugs found by the syzkaller, skip those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then pick 2 random crash logs for each such bug for later processing. `syz-testbed` will check out and compile the specified syzkaller instances and will go on executing their `syz-repro`s on each picked up crash log file, as long as the tool is not stopped. --- ### Syz Verifier (`docs/syz_verifier.md`) # syz-verifier Many bugs are easy to detect: they might cause assertions failures, crash our system, or cause other forms of undefined behaviour detectable by various dynamic analysis tools. However, certain classes of bugs, referred to as *semantic bugs*, cause none of these while still resulting in a misbehaving faulty system. To find semantic bugs, one needs to establish a specification of the system's *intended behaviour*. Depending on the complexity of the system, creating and centralising such specifications can be difficult. For example, the "specification" of the Linux kernel is not found in one place, but is rather a collection of documentation, man pages, and the implied expectations of a vast collection of user space programs. As such, detecting semantic bugs in the Linux kernel is significantly harder than other classes of bugs. Indeed, many test suites are meant to detect regressions, but creating and maintaining test cases, as well as covering new features requires significant amounts of engineering effort. One way to automate detection of semantic bugs is to provide the same input to different implementations, or different versions of the same system, and then cross-compare the resulting behaviour. In case the systems disagree, at least one of them is assumed to be wrong. The current `syz-verifier` is not fuzzing new inputs. Instead, it loads a corpus of existing syzkaller programs and cross-compares their execution on different versions of the Linux kernel to detect semantic bugs. The architecture of `syz-verifier` is shown in the following diagram. The `syz-verifier` process starts and manages VM instances with the kernels to be cross-compared. It also starts the `syz-runner` process on the VMs. Communication between the host and the guest is done via RPCs. At startup, `syz-verifier` loads programs from a syzkaller corpus database (`corpus.db`). It then sends each corpus program to `syz-runner` on every VM via RPCs while `syz-runner` is responsible for starting `syz-executor` processes and turning the program into input for those. `syz-executor` processes the input, which triggers a sequence of syscalls in the kernel. Then, `syz-runner` collects the results and sends them back to the host. At the moment, the results contain the errnos returned by each system call. When `syz-verifier` has received results from all the kernels for a specific program, it verifies them to ensure they are identical. If a mismatch is found, `syz-verifier` creates a report for the founded mismatch including the program mismatch details. # How to use `syz-verifier` After cloning the repository (see how [here](/docs/linux/setup.md#go-and-syzkaller)), build the tool as: ``` make verifier runner executor ``` To start using the tool, separate configuration files need to be created for each kernel you want to include in the verification. An example of Linux configs can be found [here](/docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md#syzkaller). The configuration files are identical to those used by `syz-manager`. All compared kernels must use the same `workdir`. `syz-verifier` loads the corpus from `corpus.db` in that shared `workdir`, so the first config must point at the workdir whose corpus you want to verify. If you want to constrain verification to programs that use a specific set of system calls, list them in the kernel config files using the `enable_syscalls` option. If you want to disable some system calls, use the `disable_syscalls` option. Start `syz-verifier` as: ``` ./bin/syz-verifier -configs=kernel0.cfg,kernel1.cfg ``` `syz-verifier` logs its progress throughout execution, including corpus loading and per-program comparison progress. # How to interpret the results Results can be found in `workdir/results`. When `syz-verifier` finds a mismatch in a program, it will create a report for that program. The report lists the results returned for each system call, by each of the cross-compared kernels, highlighting the ones were a mismatch was found. The system calls are listed in the order they appear in the program. An extract of such a report is shown below: ``` ERRNO mismatches found for program: ========== ERRNO MISMATCH DETECTED ========== Between: Kernel 0 (kernel-6.17) and Kernel 1 (kernel-6.2) Complete Program Sequence: ------------------------------------------- [0] capset(&(0x7f0000000380)={0x19980330}, &(0x7f00000003c0)) Result: errno=0, flags=0x3 >>> [1] r0 = socket$inet6_tcp(0xa, 0x1, 0x0) >>> ┌─ : errno=0, flags=0x3 >>> └─ : errno=2, flags=0x3 >>> [2] setsockopt$inet6_tcp_TCP_CONGESTION(r0, 0x6, 0xd, &(0x7f0000000000)='cubic', 0x3) >>> ┌─ : errno=2, flags=0x3 >>> └─ : errno=9, flags=0x3 ------------------------------------------- Kernel Outputs: : "" : "" ============================================= ... ``` In this report, lines prefixed with `>>>` highlight the calls whose returned errnos differed between kernels. The two indented result lines under such a call show the values returned by each compared kernel for that exact syscall. The `Kernel Outputs` section records the raw runner output for each kernel, which is useful when a mismatch is caused by a crash, timeout, or some other execution problem rather than a pure semantic difference. --- ### Syzbot (`docs/syzbot.md`) # syzbot `syzbot` system continuously fuzzes main Linux kernel branches and automatically reports found bugs to kernel mailing lists. [syzbot dashboard](https://syzkaller.appspot.com) shows current statuses of bugs. All `syzbot`-reported bugs are also CCed to [syzkaller-bugs mailing list](https://groups.google.com/forum/#!forum/syzkaller-bugs). Direct all questions to `syzkaller@googlegroups.com`.
## Bug status tracking `syzbot` needs to know when a bug is fixed in order to (1) verify that it is in fact fixed and (2) be able to report other similarly-looking crashes (while a bug is considered open all similarly-looking crashes are merged into the existing bug). To understand when a bug is fixed `syzbot` needs to know what commit fixes the bug; once `syzbot` knows the commit it will track when the commit reaches all kernel builds on all tracked branches. Only when the commit reaches all builds, the bug is considered closed (new similarly-looking crashes create a new bug). ## Communication with syzbot If you fix a bug reported by `syzbot`, please add the provided `Reported-by` tag to the commit. You can also communicate with `syzbot` by replying to its emails. The commands are: - to attach a fixing commit to the bug (if you forgot to add `Reported-by` tag): ``` #syz fix: exact-commit-title ```` It's enough that the commit is merged into any tree or you are reasonably sure about its final title, in particular, you don't need to wait for the commit to be merged into upstream tree. `syzbot` only needs to know the title by which it will appear in tested trees. In case of an error or a title change, you can override the commit simply by sending another `#syz fix` command. - to undo a previous fix command and remove any fixing commits: ``` #syz unfix ```` - to mark the bug as a duplicate of another `syzbot` bug: ``` #syz dup: exact-subject-of-another-report ``` - to undo a previous dup command and turn it into an independent bug again: ``` #syz undup ``` - to mark the bug as a one-off invalid report (e.g. induced by a previous memory corruption): ``` #syz invalid ``` **Note**: if the crash happens again, it will cause creation of a new bug report. **Note**: all commands must start from beginning of the line. **Note**: please keep at least `syzkaller-bugs@googlegroups.com` mailing list in CC. It serves as a history of what happened with each bug report. Keeping the main kernel mailing list (e.g. `linux-kernel@vger.kernel.org `) in CC is useful as well so that it's searchable in those archives as well. **Note**: `syzbot` identifies bugs by the `HASH` in the `syzbot+HASH@` receiver email address. So, strictly saying, you don't need to *reply* to emails (e.g. if you did not receive them), you can send a new email to the `syzbot+HASH@` email address, which you can find as `Sender` in email archives or as `Reported-by` email on the dashboard page for each bug.
## Testing patches `syzbot` can test patches for bugs *with reproducers*. This can be used for testing of fix patches, or just for debugging (i.e. adding additional checks to code and testing with them), or to check if the bug still happens. To test on a particular git tree and branch reply with: ``` #syz test: git://repo/address.git branch ``` or alternatively, to test on exact commit reply with: ``` #syz test: git://repo/address.git commit-hash ``` You can also completely omit these parameters: ``` #syz test ``` In this case, syzbot will check out the latest commit from the branch where the issue was detected. If you also provide a patch with the email, `syzbot` will apply it on top of the tree before testing. The patch can be provided inline in email text or as a text attachment (which is more reliable if your email client messes with whitespaces). If you want to include the patch directly in the email body, just paste the diff somewhere under the `#syz test` command line, e.g. ``` #syz test: git://repo/address.git branch --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c - current->kasan_depth++; + current->kasan_depth--; ``` [Here](https://groups.google.com/g/syzkaller-bugs/search?q=%22%23syz%20test%22) are some real examples of `#syz test` commands for syzbot-reported bugs. If you don't provide a patch, `syzbot` will test the tree as is. This is useful if this is your own tree which already contains the patch, or to check if the bug is already fixed by some recent commit. After sending an email you should typically get a reply email with results within an hour. **Note**: you may send the request only to `syzbot` email address, as patches sent to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork. **Note**: when testing a patch, syzbot uses the newest reproducer and the matching kernel config that are listed on the dashboard for this bug. As a result, specifying a repo, branch or commit id that are different from the ones that were used for reproducing, can result in false-positive Tested-by responses. For example this happens, when the bug is not reproducible on a specified kernel tree, with or without the supplied patch. **Note**: see [below](#kmsan-bugs) for `KMSAN` bugs testing. **Note**: see [below](#usb-bugs) for `USB` bugs testing. ## Subsystems For all its bugs, `syzbot` automatically assigns kernel subsystems tags. For Linux, the predefined list of kernel subsystems can be found at https://syzkaller.appspot.com/upstream/subsystems. By clicking on a name in the subsystem list or by following a tag after a bug's title, you can get the full list of bugs belonging to the subsystem. For example, all `nfc` bugs are listed here: https://syzkaller.appspot.com/upstream/s/nfc. `syzbot` includes subsystem tags into email subject as well, with `?` indicating that it's an automatic guess: `[syzbot] [ntfs?] kernel BUG in ntfs_iget`. Over time, as we improve the classification rules or as syzbot obtains more information about the bug (e.g. finds a reproducer), `syzbot` will update tags. You can also manually override the automatic guess by replying to the `syzbot` email: ``` #syz set subsystems: net, mm ``` Names of subsystems must be taken from the subsystem list page on the syzbot web dashboard.
## Bug labels It is possible to assign labels to syzkaller-reported bugs. These labels are displayed near bug titles on the bug lists and on individual bug pages. There are two types of labels: * Flags * Send `#syz set