bcc Documentation

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

22,566 stars C C 1 file ยท ~56,160 tokens
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy bcc on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run)
docker run -d --name bcc --restart always iovisor/bcc:latest

File: README.md

BPF Compiler Collection (BCC)

BCC is a toolkit for creating efficient kernel tracing and manipulation
programs, and includes several useful tools and examples. It makes use of
extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature
that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1
and above.

eBPF was described by Ingo Molnรกr as:

One of the more interesting features in this cycle is the ability to attach eBPF programs (user-defined, sandboxed bytecode executed by the kernel) to kprobes. This allows user-defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively.

BCC makes BPF programs easier to write, with kernel instrumentation in C
(and includes a C wrapper around LLVM), and front-ends in Python and lua.
It is suited for many tasks, including performance analysis and network
traffic control.

Screenshot

This example traces a disk I/O kernel function, and populates an in-kernel
power-of-2 histogram of the I/O size. For efficiency, only the histogram
summary is returned to user-level.

shell
# ./bitehist.py
Tracing... Hit Ctrl-C to end.
^C
     kbytes          : count     distribution
       0 -> 1        : 3        |                                      |
       2 -> 3        : 0        |                                      |
       4 -> 7        : 211      |**********                            |
       8 -> 15       : 0        |                                      |
      16 -> 31       : 0        |                                      |
      32 -> 63       : 0        |                                      |
      64 -> 127      : 1        |                                      |
     128 -> 255      : 800      |**************************************|

The above output shows a bimodal distribution, where the largest mode of
800 I/O was between 128 and 255 Kbytes in size.

See the source: bitehist.py. What this traces,
what this stores, and how the data is presented, can be entirely customized.
This shows only some of many possible capabilities.

Installing

See INSTALL.md for installation steps on your platform.

FAQ

See FAQ.txt for the most common troubleshoot questions.

Reference guide

See docs/reference_guide.md for the reference guide to the bcc and bcc/BPF APIs.

Contents

Some of these are single files that contain both C and Python, others have a
pair of .c and .py files, and some are directories of files.

Tracing

Examples

Tools

Memory and Process Tools

Performance and Time Tools

CPU and Scheduler Tools

Network and Sockets Tools

Storage and Filesystems Tools

Filesystems Tools

Networking

Examples:

BPF Introspection

Tools that help to introspect BPF programs.

  • introspection/bps.c: List all BPF programs loaded into the kernel. 'ps' for BPF programs. Examples.

Motivation

BPF guarantees that the programs loaded into the kernel cannot crash, and
cannot run forever, but yet BPF is general purpose enough to perform many
arbitrary types of computation. Currently, it is possible to write a program in
C that will compile into a valid BPF program, yet it is vastly easier to
write a C program that will compile into invalid BPF (C is like that). The user
won't know until trying to run the program whether it was valid or not.

With a BPF-specific frontend, one should be able to write in a language and
receive feedback from the compiler on the validity as it pertains to a BPF
backend. This toolkit aims to provide a frontend that can only create valid BPF
programs while still harnessing its full flexibility.

Furthermore, current integrations with BPF have a kludgy workflow, sometimes
involving compiling directly in a linux kernel source tree. This toolchain aims
to minimize the time that a developer spends getting BPF compiled, and instead
focus on the applications that can be written and the problems that can be
solved with BPF.

The features of this toolkit include:

  • End-to-end BPF workflow in a shared library
    • A modified C language for BPF backends
    • Integration with llvm-bpf backend for JIT
    • Dynamic (un)loading of JITed programs
    • Support for BPF kernel hooks: socket filters, tc classifiers,
      tc actions, and kprobes
  • Bindings for Python
  • Examples for socket filters, tc classifiers, and kprobes
  • Self-contained tools for tracing a running system

In the future, more bindings besides python will likely be supported. Feel free
to add support for the language of your choice and send a pull request!

Tutorials

Networking

At Red Hat Summit 2015, BCC was presented as part of a session on BPF.
A multi-host vxlan environment is simulated and a BPF program used to monitor
one of the physical interfaces. The BPF program keeps statistics on the inner
and outer IP addresses traversing the interface, and the userspace component
turns those statistics into a graph showing the traffic distribution at
multiple granularities. See the code here.

Contributing

Already pumped up to commit some code? Here are some resources to join the
discussions in the IOVisor community and see
what you want to work on.

Looking for more information on BCC and how it's being used? You can find links to other BCC content on the web in LINKS.md.


File: docs/kernel_config.md

Kernel Configuration for BPF Features

Functionalities Kernel Configuration Description
Basic CONFIG_BPF_SYSCALL Enable the bpf() system call
CONFIG_BPF_JIT BPF programs are normally handled by a BPF interpreter. This option allows the kernel to generate native code when a program is loaded into the kernel. This will significantly speed-up processing of BPF programs
CONFIG_HAVE_BPF_JIT Enable BPF Just In Time compiler
CONFIG_HAVE_EBPF_JIT Extended BPF JIT (eBPF)
CONFIG_HAVE_CBPF_JIT Classic BPF JIT (cBPF)
CONFIG_MODULES Enable to build loadable kernel modules
CONFIG_BPF BPF VM interpreter
CONFIG_BPF_EVENTS Allow the user to attach BPF programs to kprobe, uprobe, and tracepoint events
CONFIG_PERF_EVENTS Kernel performance events and counters
CONFIG_HAVE_PERF_EVENTS Enable perf events
CONFIG_PROFILING Enable the extended profiling support mechanisms used by profilers
BTF CONFIG_DEBUG_INFO_BTF Generate deduplicated BTF type information from DWARF debug info
CONFIG_PAHOLE_HAS_SPLIT_BTF Generate BTF for each selected kernel module
CONFIG_DEBUG_INFO_BTF_MODULES Generate compact split BTF type information for kernel modules
Security CONFIG_BPF_JIT_ALWAYS_ON Enable BPF JIT and removes BPF interpreter to avoid speculative execution
CONFIG_BPF_UNPRIV_DEFAULT_OFF Disable unprivileged BPF by default by setting
Cgroup CONFIG_CGROUP_BPF Support for BPF programs attached to cgroups
Network CONFIG_BPFILTER BPF based packet filtering framework (BPFILTER)
CONFIG_BPFILTER_UMH This builds bpfilter kernel module with embedded user mode helper
CONFIG_NET_CLS_BPF BPF-based classifier - to classify packets based on programmable BPF (JIT'ed) filters as an alternative to ematches
CONFIG_NET_ACT_BPF Execute BPF code on packets. The BPF code will decide if the packet should be dropped or not
CONFIG_BPF_STREAM_PARSER Enable this to allow a TCP stream parser to be used with BPF_MAP_TYPE_SOCKMAP
CONFIG_LWTUNNEL_BPF Allow to run BPF programs as a nexthop action following a route lookup for incoming and outgoing packets
CONFIG_NETFILTER_XT_MATCH_BPF BPF matching applies a linux socket filter to each packet and accepts those for which the filter returns non-zero
CONFIG_IPV6_SEG6_BPF To support BPF seg6local hook. bpf: Add IPv6 Segment Routing helpersy. Reference
kprobes CONFIG_KPROBE_EVENTS This allows the user to add tracing events (similar to tracepoints) on the fly via the ftrace interface
CONFIG_KPROBES Enable kprobes-based dynamic events
CONFIG_HAVE_KPROBES Check if krpobes enabled
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API This symbol should be selected by an architecture if it supports the API needed to access registers and stack entries from pt_regs. For example the kprobes-based event tracer needs this API.
CONFIG_KPROBES_ON_FTRACE Have kprobes on function tracer if arch supports full passing of pt_regs to function tracing
kprobe multi CONFIG_FPROBE Enable fprobe to attach the probe on multiple functions at once
kprobe override CONFIG_BPF_KPROBE_OVERRIDE Enable BPF programs to override a kprobed function
uprobes CONFIG_UPROBE_EVENTS Enable uprobes-based dynamic events
CONFIG_ARCH_SUPPORTS_UPROBES Arch specific uprobes support
CONFIG_UPROBES Uprobes is the user-space counterpart to kprobes: they enable instrumentation applications (such as 'perf probe') to establish unintrusive probes in user-space binaries and libraries, by executing handler functions when the probes are hit by user-space applications.
CONFIG_MMU MMU-based virtualised addressing space support by paged memory management
Tracepoints CONFIG_TRACEPOINTS Enable inserting tracepoints in the kernel and connect to proble functions
CONFIG_HAVE_SYSCALL_TRACEPOINTS Enable syscall enter/exit tracing
Raw Tracepoints Same as Tracepoints
LSM CONFIG_BPF_LSM Enable instrumentation of the security hooks with BPF programs for implementing dynamic MAC and Audit Policies
LIRC CONFIG_BPF_LIRC_MODE2 Allow attaching BPF programs to a lirc device

File: docs/kernel-versions.md

BPF Features by Linux Kernel Version

eBPF support

Kernel version Commit
3.15 bd4cf0ed331a

JIT compiling

The list of supported architectures for your kernel can be retrieved with:

text
git grep HAVE_EBPF_JIT arch/
Feature / Architecture Kernel version Commit
x86_64 3.16 622582786c9e
ARM64 3.18 e54bcde3d69d
s390 4.1 054623105728
Constant blinding for JIT machines 4.7 4f3446bb809f
PowerPC64 4.8 156d0e290e96
Constant blinding - PowerPC64 4.9 b7b7013cac55
Sparc64 4.12 7a12b5031c6b
MIPS 4.13 f381bf6d82f0
ARM32 4.14 39c13c204bb1
x86_32 4.18 03f5781be2c7
RISC-V RV64G 5.1 2353ecc6f91f
RISC-V RV32G 5.7 5f316b65e99f
PowerPC32 5.13 51c66ad849a7
LoongArch 6.1 5dc615520c4d

Main features

Several (but not all) of these main features translate to an eBPF program type.
The list of such program types supported in your kernel can be found in file
include/uapi/linux/bpf.h:

text
git grep -W 'bpf_prog_type {' include/uapi/linux/bpf.h
Feature Kernel version Commit
AF_PACKET (libpcap/tcpdump, cls_bpf classifier, netfilter's xt_bpf, team driver's load-balancing modeโ€ฆ) 3.15 bd4cf0ed331a
Kernel helpers 3.15 bd4cf0ed331a
bpf() syscall 3.18 99c55f7d47c0
Maps (a.k.a. Tables; details below) 3.18 99c55f7d47c0
BPF attached to sockets 3.19 89aa075832b0
BPF attached to kprobes 4.1 2541517c32be
cls_bpf / act_bpf for tc 4.1 e2e9b6541dd4
Tail calls 4.2 04fd61ab36ec
Non-root programs on sockets 4.4 1be7f75d1668
Persistent maps and programs (virtual FS) 4.4 b2197755b263
tc's direct-action (da) mode 4.4 045efa82ff56
tc's clsact qdisc 4.5 1f211a1b929c
BPF attached to tracepoints 4.7 98b5c2c65c29
Direct packet access 4.7 969bf05eb3ce
XDP (see below) 4.8 6a773a15a1e8
BPF attached to perf events 4.9 0515e5999a46
Hardware offload for tc's cls_bpf 4.9 332ae8e2f6ec
Verifier exposure and internal hooks 4.9 13a27dfc6697
BPF attached to cgroups for socket filtering 4.10 0e33661de493
Lightweight tunnel encapsulation 4.10 3a0af8fd61f9
eBPF support for xt_bpf module (iptables) 4.10 2c16d6033264
BPF program tag 4.10 7bd509e311f4
Tracepoints to debug BPF 4.11 (removedย inย 4.18) a67edbf4fb6d 4d220ed0f814
Testing / benchmarking BPF programs 4.12 1cf1cae963c2
BPF programs and maps IDs 4.13 dc4bb0e23561
BPF support for sock_ops 4.13 40304b2a1567
BPF support for skbs on sockets 4.14 b005fd189cec
bpftool utility in kernel sources 4.15 71bb428fe2c1
BPF attached to cgroups as device controller 4.15 ebc614f68736
bpf2bpf function calls 4.16 cc8b0b92a169
BPF used for monitoring socket RX/TX data 4.17 4f738adba30a
BPF attached to raw tracepoints 4.17 c4f6699dfcb8
BPF attached to bind() system call 4.17 4fbac77d2d09 aac3fc320d94
BPF attached to connect() system call 4.17 d74bad4e74ee
BPF Type Format (BTF) 4.18 69b693f0aefa
AF_XDP 4.18 fbfc504a24f5
bpfilter 4.18 d2ba09c17a06
End.BPF action for seg6local LWT 4.18 004d4b274e2a
BPF attached to LIRC devices 4.18 f4364dcfc86d
Pass map values to map helpers 4.18 d71962f3e627
BPF socket reuseport 4.19 2dbb9b9e6df6
BPF flow dissector 4.20 d58e468b1112
BPF 1M insn limit 5.2 c04c0d2b968a
BPF cgroup sysctl 5.2 7b146cebe30c
BPF raw tracepoint writable 5.2 9df1c28bb752
BPF bounded loop 5.3 2589726d12a1
BPF trampoline 5.5 fec56f5890d9
BPF LSM hook 5.7 fc611f47f218 641cd7b06c91
BPF iterator 5.8 180139dca8b3
BPF socket lookup hook 5.9 e9ddbb7707ff
Sleepable BPF programs 5.10 1e6c62a88215
Mixing bpf2bpf function calls and tailcalls (x86_64) 5.10 e411901c0b77
Mixing bpf2bpf function calls and tailcalls (arm64) 6.0 d4609a5d8c70
Mixing bpf2bpf function calls and tailcalls (s390) 6.3 dd691e847d28
Mixing bpf2bpf function calls and tailcalls (loongarch) 6.4 bb035ef0cc91

Program types

Program type Kernel version Commit Enum
Socket filter 3.19 ddd872bc3098 BPF_PROG_TYPE_SOCKET_FILTER
Kprobe 4.1 2541517c32be BPF_PROG_TYPE_KPROBE
traffic control (TC) 4.1 96be4325f443 BPF_PROG_TYPE_SCHED_CLS
traffic control (TC) 4.1 94caee8c312d BPF_PROG_TYPE_SCHED_ACT
Tracepoint 4.7 98b5c2c65c29 BPF_PROG_TYPE_TRACEPOINT
XDP 4.8 6a773a15a1e8 BPF_PROG_TYPE_XDP
Perf event 4.9 0515e5999a46 BPF_PROG_TYPE_PERF_EVENT
cgroup socket filtering 4.10 0e33661de493 BPF_PROG_TYPE_CGROUP_SKB
cgroup sock modification 4.10 610236587600 BPF_PROG_TYPE_CGROUP_SOCK
lightweight tunnel (IN) 4.10 3a0af8fd61f9 BPF_PROG_TYPE_LWT_IN
lightweight tunnel (OUT) 4.10 3a0af8fd61f9 BPF_PROG_TYPE_LWT_OUT
lightweight tunnel (XMIT) 4.10 3a0af8fd61f9 BPF_PROG_TYPE_LWT_XMIT
cgroup sock ops (per conn) 4.13 40304b2a1567 BPF_PROG_TYPE_SOCK_OPS
stream parser / stream verdict 4.14 b005fd189cec BPF_PROG_TYPE_SK_SKB
cgroup device manager 4.15 ebc614f68736 BPF_PROG_TYPE_CGROUP_DEVICE
socket msg verdict 4.17 4f738adba30a BPF_PROG_TYPE_SK_MSG
Raw tracepoint 4.17 c4f6699dfcb8 BPF_PROG_TYPE_RAW_TRACEPOINT
socket binding 4.17 4fbac77d2d09 BPF_PROG_TYPE_CGROUP_SOCK_ADDR
LWT seg6local 4.18 004d4b274e2a BPF_PROG_TYPE_LWT_SEG6LOCAL
lirc devices 4.18 f4364dcfc86d BPF_PROG_TYPE_LIRC_MODE2
lookup SO_REUSEPORT socket 4.19 2dbb9b9e6df6 BPF_PROG_TYPE_SK_REUSEPORT
flow dissector 4.20 d58e468b1112 BPF_PROG_TYPE_FLOW_DISSECTOR
cgroup sysctl 5.2 7b146cebe30c BPF_PROG_TYPE_CGROUP_SYSCTL
writable raw tracepoints 5.2 9df1c28bb752 BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE
cgroup getsockopt/setsockopt 5.3 0d01da6afc54 BPF_PROG_TYPE_CGROUP_SOCKOPT
Tracing (BTF/BPF trampoline) 5.5 f1b9509c2fb0 BPF_PROG_TYPE_TRACING
struct ops 5.6 27ae7997a661 BPF_PROG_TYPE_STRUCT_OPS
extensions 5.6 be8704ff07d2 BPF_PROG_TYPE_EXT
LSM 5.7 fc611f47f218 BPF_PROG_TYPE_LSM
lookup listening socket 5.9 e9ddbb7707ff BPF_PROG_TYPE_SK_LOOKUP
Allow executing syscalls 5.15 79a7f8bdb159 BPF_PROG_TYPE_SYSCALL

Maps (_a.k.a._ Tables, in BCC lingo)

Map types

The list of map types supported in your kernel can be found in file
include/uapi/linux/bpf.h:

text
git grep -W 'bpf_map_type {' include/uapi/linux/bpf.h
Map type Kernel version Commit Enum
Hash 3.19 0f8e4bd8a1fc BPF_MAP_TYPE_HASH
Array 3.19 28fbcfa08d8e BPF_MAP_TYPE_ARRAY
Prog array 4.2 04fd61ab36ec BPF_MAP_TYPE_PROG_ARRAY
Perf events 4.3 ea317b267e9d BPF_MAP_TYPE_PERF_EVENT_ARRAY
Per-CPU hash 4.6 824bd0ce6c7c BPF_MAP_TYPE_PERCPU_HASH
Per-CPU array 4.6 a10423b87a7e BPF_MAP_TYPE_PERCPU_ARRAY
Stack trace 4.6 d5a3b1f69186 BPF_MAP_TYPE_STACK_TRACE
cgroup array 4.8 4ed8ec521ed5 BPF_MAP_TYPE_CGROUP_ARRAY
LRU hash 4.10 29ba732acbee 3a08c2fd7634 BPF_MAP_TYPE_LRU_HASH
LRU per-CPU hash 4.10 8f8449384ec3 961578b63474 BPF_MAP_TYPE_LRU_PERCPU_HASH
LPM trie (longest-prefix match) 4.11 b95a5c4db09b BPF_MAP_TYPE_LPM_TRIE
Array of maps 4.12 56f668dfe00d BPF_MAP_TYPE_ARRAY_OF_MAPS
Hash of maps 4.12 bcc6b1b7ebf8 BPF_MAP_TYPE_HASH_OF_MAPS
Netdevice references (array) 4.14 546ac1ffb70d BPF_MAP_TYPE_DEVMAP
Socket references (array) 4.14 174a79ff9515 BPF_MAP_TYPE_SOCKMAP
CPU references 4.15 6710e1126934 BPF_MAP_TYPE_CPUMAP
AF_XDP socket (XSK) references 4.18 fbfc504a24f5 BPF_MAP_TYPE_XSKMAP
Socket references (hashmap) 4.18 81110384441a BPF_MAP_TYPE_SOCKHASH
cgroup storage 4.19 de9cbbaadba5 BPF_MAP_TYPE_CGROUP_STORAGE
reuseport sockarray 4.19 5dc4c4b7d4e8 BPF_MAP_TYPE_REUSEPORT_SOCKARRAY
percpu cgroup storage 4.20 b741f1630346 BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE
queue 4.20 f1a2e44a3aec BPF_MAP_TYPE_QUEUE
stack 4.20 f1a2e44a3aec BPF_MAP_TYPE_STACK
socket local storage 5.2 6ac99e8f23d4 BPF_MAP_TYPE_SK_STORAGE
Netdevice references (hashmap) 5.4 6f9d451ab1a3 BPF_MAP_TYPE_DEVMAP_HASH
struct ops 5.6 85d33df357b6 BPF_MAP_TYPE_STRUCT_OPS
ring buffer 5.8 457f44363a88 BPF_MAP_TYPE_RINGBUF
inode storage 5.10 8ea636848aca BPF_MAP_TYPE_INODE_STORAGE
task storage 5.11 4cf1bc1f1045 BPF_MAP_TYPE_TASK_STORAGE
Bloom filter 5.16 9330986c0300 BPF_MAP_TYPE_BLOOM_FILTER
user ringbuf 6.1 583c1f420173 BPF_MAP_TYPE_USER_RINGBUF

Map userspace API

Some (but not all) of these API features translate to a subcommand beginning with BPF_MAP_.
The list of subcommands supported in your kernel can be found in file
include/uapi/linux/bpf.h:

text
git grep -W 'bpf_cmd {' include/uapi/linux/bpf.h
Feature Kernel version Commit
Basic operations (lookup, update, delete, GET_NEXT_KEY) 3.18 db20fd2b0108
Pass flags to UPDATE_ELEM 3.19 3274f52073d8
Pre-alloc map memory by default 4.6 6c9059817432
Pass NULL to GET_NEXT_KEY 4.12 8fe45924387b
Creation: select NUMA node 4.14 96eabe7a40aa
Restrict access from syscall side 4.15 6e71b04a8224
Creation: specify map name 4.15 ad5b177bd73f
LOOKUP_AND_DELETE_ELEM 4.20 bd513cd08f10
Creation: BPF_F_ZERO_SEED 5.0 96b3b6c9091d
BPF_F_LOCK flag for lookup / update 5.1 96049f3afd50
Restrict access from BPF side 5.2 591fe9888d78
FREEZE 5.2 87df15de441b
mmap() support for array maps 5.5 fc9702273e2e
LOOKUP_BATCH 5.6 cb4d03ab499d
UPDATE_BATCH, DELETE_BATCH 5.6 aa2e93b8e58e
LOOKUP_AND_DELETE_BATCH 5.6 057996380a42
LOOKUP_AND_DELETE_ELEM support for hash maps 5.14 3e87f192b405

XDP

An approximate list of drivers or components supporting XDP programs for your
kernel can be retrieved with:

text
git grep -l XDP_SETUP_PROG drivers/
Feature / Driver Kernel version Commit
XDP core architecture 4.8 6a773a15a1e8
Action: drop 4.8 6a773a15a1e8
Action: pass on to stack 4.8 6a773a15a1e8
Action: direct forwarding (on same port) 4.8 6ce96ca348a9
Direct packet data write 4.8 4acf6c0b84c9
Mellanox mlx4 driver 4.8 47a38e155037
Mellanox mlx5 driver 4.9 86994156c736
Netronome nfp driver 4.10 ecd63a0217d5
QLogic (Cavium) qed* drivers 4.10 496e05170958
virtio_net driver 4.10 f600b6905015
Broadcom bnxt_en driver 4.11 c6d30e8391b8
Intel ixgbe* drivers 4.12 924708081629
Cavium thunderx driver 4.12 05c773f52b96
Generic XDP 4.12 b5cdae3291f7
Intel i40e driver 4.13 0c8493d90b6b
Action: redirect 4.14 6453073987ba
Support for tap 4.14 761876c857cb
Support for veth 4.14 d445516966dc
Intel ixgbevf driver 4.17 c7aec59657b6
Freescale dpaa2 driver 5.0 7e273a8ebdd3
Socionext netsec driver 5.3 ba2b232108d3
TI cpsw driver 5.3 9ed4050c0d75
Intel ice driver 5.5 efc2214b6047
Solarflare sfc driver 5.5 eb9a36be7f3e
Marvell mvneta driver 5.5 0db51da7a8e9
Microsoft hv_netvsc driver 5.6 351e1581395f
Amazon ena driver 5.6 838c93dc5449
xen-netfront driver 5.9 6c5aa6fc4def
Marvell mvpp2 driver 5.9 07dd0a7aae7f
Intel igb driver 5.10 9cbc948b5a20
Freescale dpaa driver 5.11 86c0c196cbe4
Intel igc driver 5.13 26575105d6ed
Freescale enetc driver 5.13 d1b15102dd16
STMicro stmmac driver 5.13 5fabb01207a2
bonding driver 5.15 9e2ee5c7e7c3
Marvell otx2 5.16 06059a1a9a4a
Microsoft mana driver 5.17 ed5356b53f07
Fungible fun driver 5.18 db37bc177dae
Aquantia atlantic driver 5.19 0d14657f4083
Mediatek mtk driver 6.0 7c26c20da5d4
Freescale fec_enet driver 6.2 6d6b39f180b8
Microchip lan966x driver 6.2 6a2159be7604
Engleder tsnep driver 6.3 d24bc0bcbbff
Google gve driver 6.4 2e80aeae9f80
VMware vmxnet3 driver 6.6 54f00cce1178

Helpers

The list of helpers supported in your kernel can be found in file
include/uapi/linux/bpf.h:

text
git grep '	FN(' include/uapi/linux/bpf.h

Alphabetical order

Helper Kernel version License Commit
BPF_FUNC_bind() 4.17 d74bad4e74ee
BPF_FUNC_bprm_opts_set() 5.11 3f6719c7b62f
BPF_FUNC_btf_find_by_name_kind() 5.14 3d78417b60fb
BPF_FUNC_cgrp_storage_delete() 6.2 c4bcfb38a95e
BPF_FUNC_cgrp_storage_get() 6.2 c4bcfb38a95e
BPF_FUNC_check_mtu() 5.12 34b2021cc616
BPF_FUNC_clone_redirect() 4.2 3896d655f4d4
BPF_FUNC_copy_from_user() 5.10 07be4c4a3e7a
BPF_FUNC_copy_from_user_task() 5.18 GPL 376040e47334
BPF_FUNC_csum_diff() 4.6 7d672345ed29
BPF_FUNC_csum_level() 5.7 7cdec54f9713
BPF_FUNC_csum_update() 4.9 36bbef52c7eb
BPF_FUNC_current_task_under_cgroup() 4.9 60d20f9195b2
BPF_FUNC_d_path() 5.10 6e22ab9da793
BPF_FUNC_dynptr_data() 5.19 34d4ef5775f7
BPF_FUNC_dynptr_from_mem() 5.19 263ae152e962
BPF_FUNC_dynptr_read() 5.19 13bbbfbea759
BPF_FUNC_dynptr_write() 5.19 13bbbfbea759
BPF_FUNC_fib_lookup() 4.18 GPL 87f5fc7e48dd
BPF_FUNC_find_vma() 5.17 7c7e3d31e785
BPF_FUNC_for_each_map_elem() 5.13 69c087ba6225
BPF_FUNC_get_attach_cookie() 5.15 7adfc6c9b315
BPF_FUNC_get_branch_snapshot() 5.16 GPL 856c02dbce4f
BPF_FUNC_get_current_ancestor_cgroup_id() 5.6 b4490c5c4e02
BPF_FUNC_get_cgroup_classid() 4.3 8d20aabe1c76
BPF_FUNC_get_current_cgroup_id() 4.18 bf6fa2c893c5
BPF_FUNC_get_current_comm() 4.2 ffeedafbf023
BPF_FUNC_get_current_pid_tgid() 4.2 ffeedafbf023
BPF_FUNC_get_current_task() 4.8 GPL 606274c5abd8
BPF_FUNC_get_current_task_btf() 5.11 GPL 3ca1032ab7ab
BPF_FUNC_get_current_uid_gid() 4.2 ffeedafbf023
BPF_FUNC_get_func_arg() 5.17 f92c1e183604
BPF_FUNC_get_func_arg_cnt() 5.17 f92c1e183604
BPF_FUNC_get_func_ip() 5.15 5d8b583d04ae
BPF_FUNC_get_func_ret() 5.17 f92c1e183604
BPF_FUNC_get_retval() 5.18 b44123b4a3dc
BPF_FUNC_get_hash_recalc() 4.8 13c5c240f789
BPF_FUNC_get_listener_sock() 5.1 dbafd7ddd623
BPF_FUNC_get_local_storage() 4.19 cd3394317653
BPF_FUNC_get_netns_cookie() 5.7 f318903c0bf4
BPF_FUNC_get_ns_current_pid_tgid() 5.7 b4490c5c4e02
BPF_FUNC_get_numa_node_id() 4.10 2d0e30c30f84
BPF_FUNC_get_prandom_u32() 4.1 03e69b508b6f
BPF_FUNC_get_route_realm() 4.4 c46646d0484f
BPF_FUNC_get_smp_processor_id() 4.1 c04167ce2ca0
BPF_FUNC_get_socket_cookie() 4.12 91b8270f2a4d
BPF_FUNC_get_socket_uid() 4.12 6acc5c291068
BPF_FUNC_get_stack() 4.18 GPL de2ff05f48af
BPF_FUNC_get_stackid() 4.6 GPL d5a3b1f69186
BPF_FUNC_get_task_stack() 5.9 fa28dcb82a38
BPF_FUNC_getsockopt() 4.15 cd86d1fd2102
BPF_FUNC_ima_file_hash() 5.18 174b16946e39
BPF_FUNC_ima_inode_hash() 5.11 27672f0d280a
BPF_FUNC_inode_storage_delete() 5.10 8ea636848aca
BPF_FUNC_inode_storage_get() 5.10 8ea636848aca
BPF_FUNC_jiffies64() 5.5 5576b991e9c1
BPF_FUNC_kallsyms_lookup_name() 5.16 d6aef08a872b
BPF_FUNC_kptr_xchg() 5.19 c0a5a21c25f3
BPF_FUNC_ktime_get_boot_ns() 5.8 71d19214776e
BPF_FUNC_ktime_get_coarse_ns() 5.11 d05512618056
BPF_FUNC_ktime_get_ns() 4.1 d9847d310ab4
BPF_FUNC_ktime_get_tai_ns() 6.1 c8996c98f703
BPF_FUNC_l3_csum_replace() 4.1 91bc4822c3d6
BPF_FUNC_l4_csum_replace() 4.1 91bc4822c3d6
BPF_FUNC_load_hdr_opt() 5.10 0813a841566f
BPF_FUNC_loop() 5.17 e6f2dd0f8067
BPF_FUNC_lwt_push_encap() 4.18 fe94cc290f53
BPF_FUNC_lwt_seg6_action() 4.18 fe94cc290f53
BPF_FUNC_lwt_seg6_adjust_srh() 4.18 fe94cc290f53
BPF_FUNC_lwt_seg6_store_bytes() 4.18 fe94cc290f53
BPF_FUNC_map_delete_elem() 3.19 d0003ec01c66
BPF_FUNC_map_lookup_elem() 3.19 d0003ec01c66
BPF_FUNC_map_lookup_percpu_elem() 5.19 07343110b293
BPF_FUNC_map_peek_elem() 4.20 f1a2e44a3aec
BPF_FUNC_map_pop_elem() 4.20 f1a2e44a3aec
BPF_FUNC_map_push_elem() 4.20 f1a2e44a3aec
BPF_FUNC_map_update_elem() 3.19 d0003ec01c66
BPF_FUNC_msg_apply_bytes() 4.17 2a100317c9eb
BPF_FUNC_msg_cork_bytes() 4.17 91843d540a13
BPF_FUNC_msg_pop_data() 5.0 7246d8ed4dcc
BPF_FUNC_msg_pull_data() 4.17 015632bb30da
BPF_FUNC_msg_push_data() 4.20 6fff607e2f14
BPF_FUNC_msg_redirect_hash() 4.18 81110384441a
BPF_FUNC_msg_redirect_map() 4.17 4f738adba30a
BPF_FUNC_per_cpu_ptr() 5.10 eaa6bcb71ef6
BPF_FUNC_perf_event_output() 4.4 GPL a43eec304259
BPF_FUNC_perf_event_read() 4.3 GPL 35578d798400
BPF_FUNC_perf_event_read_value() 4.15 GPL 908432ca84fc
BPF_FUNC_perf_prog_read_value() 4.15 GPL 4bebdc7a85aa
BPF_FUNC_probe_read() 4.1 GPL 2541517c32be
BPF_FUNC_probe_read_kernel() 5.5 GPL 6ae08ae3dea2
BPF_FUNC_probe_read_kernel_str() 5.5 GPL 6ae08ae3dea2
BPF_FUNC_probe_read_user() 5.5 GPL 6ae08ae3dea2
BPF_FUNC_probe_read_user_str() 5.5 GPL 6ae08ae3dea2
BPF_FUNC_probe_read_str() 4.11 GPL a5e8c07059d0
BPF_FUNC_probe_write_user() 4.8 GPL 96ae52279594
BPF_FUNC_rc_keydown() 4.18 GPL f4364dcfc86d
BPF_FUNC_rc_pointer_rel() 5.0 GPL 01d3240a04f4
BPF_FUNC_rc_repeat() 4.18 GPL f4364dcfc86d
BPF_FUNC_read_branch_records() 5.6 GPL fff7b64355ea
BPF_FUNC_redirect() 4.4 27b29f63058d
BPF_FUNC_redirect_map() 4.14 97f91a7cf04f
BPF_FUNC_redirect_neigh() 5.10 b4ab31414970
BPF_FUNC_redirect_peer() 5.10 9aa1206e8f48
BPF_FUNC_reserve_hdr_opt() 5.10 0813a841566f
BPF_FUNC_ringbuf_discard() 5.8 457f44363a88
BPF_FUNC_ringbuf_discard_dynptr() 5.19 bc34dee65a65
BPF_FUNC_ringbuf_output() 5.8 457f44363a88
BPF_FUNC_ringbuf_query() 5.8 457f44363a88
BPF_FUNC_ringbuf_reserve() 5.8 457f44363a88
BPF_FUNC_ringbuf_reserve_dynptr() 5.19 bc34dee65a65
BPF_FUNC_ringbuf_submit() 5.8 457f44363a88
BPF_FUNC_ringbuf_submit_dynptr() 5.19 bc34dee65a65
BPF_FUNC_send_signal() 5.3 8b401f9ed244
BPF_FUNC_send_signal_thread() 5.5 8482941f0906
BPF_FUNC_seq_printf() 5.7 GPL 492e639f0c22
BPF_FUNC_seq_printf_btf() 5.10 eb411377aed9
BPF_FUNC_seq_write() 5.7 GPL 492e639f0c22
BPF_FUNC_set_hash() 4.13 ded092cd73c2
BPF_FUNC_set_hash_invalid() 4.9 7a4b28c6cc9f
BPF_FUNC_set_retval() 5.18 b44123b4a3dc
BPF_FUNC_setsockopt() 4.13 8c4b4c7e9ff0
BPF_FUNC_sk_ancestor_cgroup_id() 5.7 f307fa2cb4c9
BPF_FUNC_sk_assign() 5.6 cf7fbe660f2d
BPF_FUNC_sk_cgroup_id() 5.7 f307fa2cb4c9
BPF_FUNC_sk_fullsock() 5.1 46f8bc92758c
BPF_FUNC_sk_lookup_tcp() 4.20 6acc9b432e67
BPF_FUNC_sk_lookup_udp() 4.20 6acc9b432e67
BPF_FUNC_sk_redirect_hash() 4.18 81110384441a
BPF_FUNC_sk_redirect_map() 4.14 174a79ff9515
BPF_FUNC_sk_release() 4.20 6acc9b432e67
BPF_FUNC_sk_select_reuseport() 4.19 2dbb9b9e6df6
BPF_FUNC_sk_storage_delete() 5.2 6ac99e8f23d4
BPF_FUNC_sk_storage_get() 5.2 6ac99e8f23d4
BPF_FUNC_skb_adjust_room() 4.13 2be7e212d541
BPF_FUNC_skb_ancestor_cgroup_id() 4.19 7723628101aa
BPF_FUNC_skb_change_head() 4.10 3a0af8fd61f9
BPF_FUNC_skb_change_proto() 4.8 6578171a7ff0
BPF_FUNC_skb_change_tail() 4.9 5293efe62df8
BPF_FUNC_skb_change_type() 4.8 d2485c4242a8
BPF_FUNC_skb_cgroup_classid() 5.10 b426ce83baa7
BPF_FUNC_skb_cgroup_id() 4.18 cb20b08ead40
BPF_FUNC_skb_ecn_set_ce() 5.1 f7c917ba11a6
BPF_FUNC_skb_get_tunnel_key() 4.3 d3aa45ce6b94
BPF_FUNC_skb_get_tunnel_opt() 4.6 14ca0751c96f
BPF_FUNC_skb_get_xfrm_state() 4.18 12bed760a78d
BPF_FUNC_skb_load_bytes() 4.5 05c74e5e53f6
BPF_FUNC_skb_load_bytes_relative() 4.18 4e1ec56cdc59
BPF_FUNC_skb_output() 5.5 a7658e1a4164
BPF_FUNC_skb_pull_data() 4.9 36bbef52c7eb
BPF_FUNC_skb_set_tstamp() 5.18 9bb984f28d5b
BPF_FUNC_skb_set_tunnel_key() 4.3 d3aa45ce6b94
BPF_FUNC_skb_set_tunnel_opt() 4.6 14ca0751c96f
BPF_FUNC_skb_store_bytes() 4.1 91bc4822c3d6
BPF_FUNC_skb_under_cgroup() 4.8 4a482f34afcc
BPF_FUNC_skb_vlan_pop() 4.3 4e10df9a60d9
BPF_FUNC_skb_vlan_push() 4.3 4e10df9a60d9
BPF_FUNC_skc_lookup_tcp() 5.2 edbf8c01de5a
BPF_FUNC_skc_to_mctcp_sock() 5.19 3bc253c2e652
BPF_FUNC_skc_to_tcp_sock() 5.9 478cfbdf5f13
BPF_FUNC_skc_to_tcp_request_sock() 5.9 478cfbdf5f13
BPF_FUNC_skc_to_tcp_timewait_sock() 5.9 478cfbdf5f13
BPF_FUNC_skc_to_tcp6_sock() 5.9 af7ec1383361
BPF_FUNC_skc_to_udp6_sock() 5.9 0d4fad3e57df
BPF_FUNC_skc_to_unix_sock() 5.16 9eeb3aa33ae0
BPF_FUNC_snprintf() 5.13 7b15523a989b
BPF_FUNC_snprintf_btf() 5.10 c4d0bfb45068
BPF_FUNC_sock_from_file() 5.11 4f19cab76136
BPF_FUNC_sock_hash_update() 4.18 81110384441a
BPF_FUNC_sock_map_update() 4.14 174a79ff9515
BPF_FUNC_spin_lock() 5.1 d83525ca62cf
BPF_FUNC_spin_unlock() 5.1 d83525ca62cf
BPF_FUNC_store_hdr_opt() 5.10 0813a841566f
BPF_FUNC_strncmp() 5.17 c5fb19937455
BPF_FUNC_strtol() 5.2 d7a4cb9b6705
BPF_FUNC_strtoul() 5.2 d7a4cb9b6705
BPF_FUNC_sys_bpf() 5.14 79a7f8bdb159
BPF_FUNC_sys_close() 5.14 3abea089246f
BPF_FUNC_sysctl_get_current_value() 5.2 1d11b3016cec
BPF_FUNC_sysctl_get_name() 5.2 808649fb787d
BPF_FUNC_sysctl_get_new_value() 5.2 4e63acdff864
BPF_FUNC_sysctl_set_new_value() 5.2 4e63acdff864
BPF_FUNC_tail_call() 4.2 04fd61ab36ec
BPF_FUNC_task_pt_regs() 5.15 GPL dd6e10fbd9f
BPF_FUNC_task_storage_delete() 5.11 4cf1bc1f1045
BPF_FUNC_task_storage_get() 5.11 4cf1bc1f1045
BPF_FUNC_tcp_check_syncookie() 5.2 399040847084
BPF_FUNC_tcp_gen_syncookie() 5.3 70d66244317e
BPF_FUNC_tcp_raw_check_syncookie_ipv4() 6.0 33bf9885040c
BPF_FUNC_tcp_raw_check_syncookie_ipv6() 6.0 33bf9885040c
BPF_FUNC_tcp_raw_gen_syncookie_ipv4() 6.0 33bf9885040c
BPF_FUNC_tcp_raw_gen_syncookie_ipv6() 6.0 33bf9885040c
BPF_FUNC_tcp_send_ack() 5.5 206057fe020a
BPF_FUNC_tcp_sock() 5.1 655a51e536c0
BPF_FUNC_this_cpu_ptr() 5.10 63d9b80dcf2c
BPF_FUNC_timer_init() 5.15 b00628b1c7d5
BPF_FUNC_timer_set_callback() 5.15 b00628b1c7d5
BPF_FUNC_timer_start() 5.15 b00628b1c7d5
BPF_FUNC_timer_cancel() 5.15 b00628b1c7d5
BPF_FUNC_trace_printk() 4.1 GPL 9c959c863f82
BPF_FUNC_trace_vprintk() 5.16 GPL 10aceb629e19
BPF_FUNC_user_ringbuf_drain() 6.1 205715673844
BPF_FUNC_xdp_adjust_head() 4.10 17bedab27231
BPF_FUNC_xdp_adjust_meta() 4.15 de8f3a83b0a0
BPF_FUNC_xdp_adjust_tail() 4.18 b32cc5b9a346
BPF_FUNC_xdp_get_buff_len() 5.18 0165cc817075
BPF_FUNC_xdp_load_bytes() 5.18 3f364222d032
BPF_FUNC_xdp_store_bytes() 5.18 3f364222d032
BPF_FUNC_xdp_output() 5.6 GPL d831ee84bfc9
BPF_FUNC_override_return() 4.16 GPL 9802d86585db
BPF_FUNC_sock_ops_cb_flags_set() 4.16 b13d88072172

Note: GPL-only BPF helpers require a GPL-compatible license. The current licenses considered GPL-compatible by the kernel are:

  • GPL
  • GPL v2
  • GPL and additional rights
  • Dual BSD/GPL
  • Dual MIT/GPL
  • Dual MPL/GPL

Check the list of GPL-compatible licenses in your kernel source code.

Program Types

The list of program types and supported helper functions can be retrieved with:

text
git grep -W 'func_proto(enum bpf_func_id func_id' kernel/ net/ drivers/
Program Type Helper Functions
BPF_PROG_TYPE_SOCKET_FILTER BPF_FUNC_skb_load_bytes() BPF_FUNC_skb_load_bytes_relative() BPF_FUNC_get_socket_cookie() BPF_FUNC_get_socket_uid() BPF_FUNC_perf_event_output() Base functions
BPF_PROG_TYPE_KPROBE BPF_FUNC_perf_event_output() BPF_FUNC_get_stackid() BPF_FUNC_get_stack() BPF_FUNC_perf_event_read_value() BPF_FUNC_override_return() Tracing functions
BPF_PROG_TYPE_SCHED_CLS BPF_PROG_TYPE_SCHED_ACT BPF_FUNC_skb_store_bytes() BPF_FUNC_skb_load_bytes() BPF_FUNC_skb_load_bytes_relative() BPF_FUNC_skb_pull_data() BPF_FUNC_csum_diff() BPF_FUNC_csum_update() BPF_FUNC_l3_csum_replace() BPF_FUNC_l4_csum_replace() BPF_FUNC_clone_redirect() BPF_FUNC_get_cgroup_classid() BPF_FUNC_skb_vlan_push() BPF_FUNC_skb_vlan_pop() BPF_FUNC_skb_change_proto() BPF_FUNC_skb_change_type() BPF_FUNC_skb_adjust_room() BPF_FUNC_skb_change_tail() BPF_FUNC_skb_get_tunnel_key() BPF_FUNC_skb_set_tunnel_key() BPF_FUNC_skb_get_tunnel_opt() BPF_FUNC_skb_set_tunnel_opt() BPF_FUNC_redirect() BPF_FUNC_get_route_realm() BPF_FUNC_get_hash_recalc() BPF_FUNC_set_hash_invalid() BPF_FUNC_set_hash() BPF_FUNC_perf_event_output() BPF_FUNC_get_smp_processor_id() BPF_FUNC_skb_under_cgroup() BPF_FUNC_get_socket_cookie() BPF_FUNC_get_socket_uid() BPF_FUNC_fib_lookup() BPF_FUNC_skb_get_xfrm_state() BPF_FUNC_skb_cgroup_id() Base functions
BPF_PROG_TYPE_TRACEPOINT BPF_FUNC_perf_event_output() BPF_FUNC_get_stackid() BPF_FUNC_get_stack() BPF_FUNC_d_path() Tracing functions
BPF_PROG_TYPE_XDP BPF_FUNC_perf_event_output() BPF_FUNC_get_smp_processor_id() BPF_FUNC_csum_diff() BPF_FUNC_xdp_adjust_head() BPF_FUNC_xdp_adjust_meta() BPF_FUNC_redirect() BPF_FUNC_redirect_map() BPF_FUNC_xdp_adjust_tail() BPF_FUNC_fib_lookup() Base functions
BPF_PROG_TYPE_PERF_EVENT BPF_FUNC_perf_event_output() BPF_FUNC_get_stackid() BPF_FUNC_get_stack() BPF_FUNC_perf_prog_read_value() Tracing functions
BPF_PROG_TYPE_CGROUP_SKB BPF_FUNC_skb_load_bytes() BPF_FUNC_skb_load_bytes_relative() BPF_FUNC_get_socket_cookie() BPF_FUNC_get_socket_uid() Base functions
BPF_PROG_TYPE_CGROUP_SOCK BPF_FUNC_get_current_uid_gid() Base functions
BPF_PROG_TYPE_LWT_IN BPF_FUNC_lwt_push_encap() LWT functions Base functions
BPF_PROG_TYPE_LWT_OUT LWT functions Base functions
BPF_PROG_TYPE_LWT_XMIT BPF_FUNC_skb_get_tunnel_key() BPF_FUNC_skb_set_tunnel_key() BPF_FUNC_skb_get_tunnel_opt() BPF_FUNC_skb_set_tunnel_opt() BPF_FUNC_redirect() BPF_FUNC_clone_redirect() BPF_FUNC_skb_change_tail() BPF_FUNC_skb_change_head() BPF_FUNC_skb_store_bytes() BPF_FUNC_csum_update() BPF_FUNC_l3_csum_replace() BPF_FUNC_l4_csum_replace() BPF_FUNC_set_hash_invalid() LWT functions
BPF_PROG_TYPE_SOCK_OPS BPF_FUNC_setsockopt() BPF_FUNC_getsockopt() BPF_FUNC_sock_ops_cb_flags_set() BPF_FUNC_sock_map_update() BPF_FUNC_sock_hash_update() BPF_FUNC_get_socket_cookie() Base functions
BPF_PROG_TYPE_SK_SKB BPF_FUNC_skb_store_bytes() BPF_FUNC_skb_load_bytes() BPF_FUNC_skb_pull_data() BPF_FUNC_skb_change_tail() BPF_FUNC_skb_change_head() BPF_FUNC_get_socket_cookie() BPF_FUNC_get_socket_uid() BPF_FUNC_sk_redirect_map() BPF_FUNC_sk_redirect_hash() BPF_FUNC_sk_lookup_tcp() BPF_FUNC_sk_lookup_udp() BPF_FUNC_sk_release() Base functions
BPF_PROG_TYPE_CGROUP_DEVICE BPF_FUNC_map_lookup_elem() BPF_FUNC_map_update_elem() BPF_FUNC_map_delete_elem() BPF_FUNC_get_current_uid_gid() BPF_FUNC_trace_printk()
BPF_PROG_TYPE_SK_MSG BPF_FUNC_msg_redirect_map() BPF_FUNC_msg_redirect_hash() BPF_FUNC_msg_apply_bytes() BPF_FUNC_msg_cork_bytes() BPF_FUNC_msg_pull_data() BPF_FUNC_msg_push_data() BPF_FUNC_msg_pop_data() Base functions
BPF_PROG_TYPE_RAW_TRACEPOINT BPF_FUNC_perf_event_output() BPF_FUNC_get_stackid() BPF_FUNC_get_stack() BPF_FUNC_skb_output() Tracing functions
BPF_PROG_TYPE_CGROUP_SOCK_ADDR BPF_FUNC_get_current_uid_gid() BPF_FUNC_bind() BPF_FUNC_get_socket_cookie() Base functions
BPF_PROG_TYPE_LWT_SEG6LOCAL BPF_FUNC_lwt_seg6_store_bytes() BPF_FUNC_lwt_seg6_action() BPF_FUNC_lwt_seg6_adjust_srh() LWT functions
BPF_PROG_TYPE_LIRC_MODE2 BPF_FUNC_rc_repeat() BPF_FUNC_rc_keydown() BPF_FUNC_rc_pointer_rel() BPF_FUNC_map_lookup_elem() BPF_FUNC_map_update_elem() BPF_FUNC_map_delete_elem() BPF_FUNC_ktime_get_ns() BPF_FUNC_tail_call() BPF_FUNC_get_prandom_u32() BPF_FUNC_trace_printk()
BPF_PROG_TYPE_SK_REUSEPORT BPF_FUNC_sk_select_reuseport() BPF_FUNC_skb_load_bytes() BPF_FUNC_load_bytes_relative() Base functions
BPF_PROG_TYPE_FLOW_DISSECTOR BPF_FUNC_skb_load_bytes() Base functions
Function Group Functions
Base functions BPF_FUNC_map_lookup_elem() BPF_FUNC_map_update_elem() BPF_FUNC_map_delete_elem() BPF_FUNC_map_peek_elem() BPF_FUNC_map_pop_elem() BPF_FUNC_map_push_elem() BPF_FUNC_get_prandom_u32() BPF_FUNC_get_smp_processor_id() BPF_FUNC_get_numa_node_id() BPF_FUNC_tail_call() BPF_FUNC_ktime_get_boot_ns() BPF_FUNC_ktime_get_ns() BPF_FUNC_trace_printk() BPF_FUNC_spin_lock() BPF_FUNC_spin_unlock()
Tracing functions BPF_FUNC_map_lookup_elem() BPF_FUNC_map_update_elem() BPF_FUNC_map_delete_elem() BPF_FUNC_probe_read() BPF_FUNC_ktime_get_boot_ns() BPF_FUNC_ktime_get_ns() BPF_FUNC_tail_call() BPF_FUNC_get_current_pid_tgid() BPF_FUNC_get_current_task() BPF_FUNC_get_current_uid_gid() BPF_FUNC_get_current_comm() BPF_FUNC_trace_printk() BPF_FUNC_get_smp_processor_id() BPF_FUNC_get_numa_node_id() BPF_FUNC_perf_event_read() BPF_FUNC_probe_write_user() BPF_FUNC_current_task_under_cgroup() BPF_FUNC_get_prandom_u32() BPF_FUNC_probe_read_str() BPF_FUNC_get_current_cgroup_id() BPF_FUNC_send_signal() BPF_FUNC_probe_read_kernel() BPF_FUNC_probe_read_kernel_str() BPF_FUNC_probe_read_user() BPF_FUNC_probe_read_user_str() BPF_FUNC_send_signal_thread() BPF_FUNC_get_ns_current_pid_tgid() BPF_FUNC_xdp_output() BPF_FUNC_get_task_stack()
LWT functions BPF_FUNC_skb_load_bytes() BPF_FUNC_skb_pull_data() BPF_FUNC_csum_diff() BPF_FUNC_get_cgroup_classid() BPF_FUNC_get_route_realm() BPF_FUNC_get_hash_recalc() BPF_FUNC_perf_event_output() BPF_FUNC_get_smp_processor_id() BPF_FUNC_skb_under_cgroup()

File: docs/reference_guide.md

bcc Reference Guide

Intended for search (Ctrl-F) and reference. For tutorials, start with tutorial.md.

This guide is incomplete. If something feels missing, check the bcc and kernel source. And if you confirm we're missing something, please send a pull request to fix it, and help out everyone.

Contents

BPF C

This section describes the C part of a bcc program.

Events & Arguments

1. kprobes

Syntax: kprobe__kernel_function_name

kprobe__ is a special prefix that creates a kprobe (dynamic tracing of a kernel function call) for the kernel function name provided as the remainder. You can also use kprobes by declaring a normal C function, then using the Python BPF.attach_kprobe() (covered later) to associate it with a kernel function.

Arguments are specified on the function declaration: kprobe__kernel_function_name(struct pt_regs *ctx [, argument1 ...])

For example:

c
int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk) {
    [...]
}

This instruments the tcp_v4_connect() kernel function using a kprobe, with the following arguments:

  • struct pt_regs *ctx: Registers and BPF context.
  • struct sock *sk: First argument to tcp_v4_connect().

The first argument is always struct pt_regs *, the remainder are the arguments to the function (they don't need to be specified, if you don't intend to use them).

Examples in situ:
code (output),
code (output)

2. kretprobes

Syntax: kretprobe__kernel_function_name

kretprobe__ is a special prefix that creates a kretprobe (dynamic tracing of a kernel function return) for the kernel function name provided as the remainder. You can also use kretprobes by declaring a normal C function, then using the Python BPF.attach_kretprobe() (covered later) to associate it with a kernel function.

Return value is available as PT_REGS_RC(ctx), given a function declaration of: kretprobe__kernel_function_name(struct pt_regs *ctx)

For example:

c
int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
{
    int ret = PT_REGS_RC(ctx);
    [...]
}

This instruments the return of the tcp_v4_connect() kernel function using a kretprobe, and stores the return value in ret.

Examples in situ:
code (output)

3. Tracepoints

Syntax: TRACEPOINT_PROBE(category, event)

This is a macro that instruments the tracepoint defined by category:event.

The tracepoint name is <category>:<event>.
The probe function name is tracepoint__<category>__<event>.

Arguments are available in an args struct, which are the tracepoint arguments. One way to list these is to cat the relevant format file under /sys/kernel/debug/tracing/events/category/event/format.

The args struct can be used in place of ctx in each functions requiring a context as an argument. This includes notably perf_submit().

For example:

c
TRACEPOINT_PROBE(random, urandom_read) {
    // args is from /sys/kernel/debug/tracing/events/random/urandom_read/format
    bpf_trace_printk("%d\\n", args->got_bits);
    return 0;
}

This instruments the tracepoint random:urandom_read tracepoint, and prints the tracepoint argument got_bits.
When using Python API, this probe is automatically attached to the right tracepoint target.
For C++, this tracepoint probe can be attached by specifying the tracepoint target and function name explicitly:
BPF::attach_tracepoint("random:urandom_read", "tracepoint__random__urandom_read")
Note the name of the probe function defined above is tracepoint__random__urandom_read.

Examples in situ:
code (output),
search /examples,
search /tools

4. uprobes

These are instrumented by declaring a normal function in C, then associating it as a uprobe probe in Python via BPF.attach_uprobe() (covered later).

Arguments can be examined using PT_REGS_PARM macros.

For example:

c
int count(struct pt_regs *ctx) {
    char buf[64];
    bpf_probe_read_user(&buf, sizeof(buf), (void *)PT_REGS_PARM1(ctx));
    bpf_trace_printk("%s %d", buf, PT_REGS_PARM2(ctx));
    return(0);
}

This reads the first argument as a string, and then prints it with the second argument as an integer.

Examples in situ:
code

5. uretprobes

These are instrumented by declaring a normal function in C, then associating it as a uretprobe probe in Python via BPF.attach_uretprobe() (covered later).

Return value is available as PT_REGS_RC(ctx), given a function declaration of: function_name(struct pt_regs *ctx)

For example:

c
BPF_HISTOGRAM(dist);
int count(struct pt_regs *ctx) {
    dist.increment(PT_REGS_RC(ctx));
    return 0;
}

This increments the bucket in the dist histogram that is indexed by the return value.

Examples in situ:
code (output),
code (output)

6. USDT probes

These are User Statically-Defined Tracing (USDT) probes, which may be placed in some applications or libraries to provide a user-level equivalent of tracepoints. The primary BPF method provided for USDT support method is enable_probe(). USDT probes are instrumented by declaring a normal function in C, then associating it as a USDT probe in Python via USDT.enable_probe().

Arguments can be read via: bpf_usdt_readarg(index, ctx, &addr)

For example:

c
int do_trace(struct pt_regs *ctx) {
    uint64_t addr;
    char path[128];
    bpf_usdt_readarg(6, ctx, &addr);
    bpf_probe_read_user(&path, sizeof(path), (void *)addr);
    bpf_trace_printk("path:%s\\n", path);
    return 0;
};

This reads the sixth USDT argument, and then pulls it in as a string to path.

When initializing USDTs via the third argument of BPF::init in the C API, if any USDT fails to init, entire BPF::init will fail. If you're OK with some USDTs failing to init, use BPF::init_usdt before calling BPF::init.

Examples in situ:
code,
search /examples,
search /tools

7. Raw Tracepoints

Syntax: RAW_TRACEPOINT_PROBE(event)

This is a macro that instruments the raw tracepoint defined by event.

The argument is a pointer to struct bpf_raw_tracepoint_args, which is defined in bpf.h. The struct field args contains all parameters of the raw tracepoint where you can found at linux tree include/trace/events
directory.

For example:

c
RAW_TRACEPOINT_PROBE(sched_switch)
{
    // TP_PROTO(bool preempt, struct task_struct *prev, struct task_struct *next)
    struct task_struct *prev = (struct task_struct *)ctx->args[1];
    struct task_struct *next= (struct task_struct *)ctx->args[2];
    s32 prev_tgid, next_tgid;

    bpf_probe_read_kernel(&prev_tgid, sizeof(prev->tgid), &prev->tgid);
    bpf_probe_read_kernel(&next_tgid, sizeof(next->tgid), &next->tgid);
    bpf_trace_printk("%d -> %d\\n", prev_tgid, next_tgid);
}

This instruments the sched:sched_switch tracepoint, and prints the prev and next tgid.

Examples in situ:
search /tools

8. system call tracepoints

Syntax: syscall__SYSCALLNAME

syscall__ is a special prefix that creates a kprobe for the system call name provided as the remainder. You can use it by declaring a normal C function, then using the Python BPF.get_syscall_fnname(SYSCALLNAME) and BPF.attach_kprobe() to associate it.

Arguments are specified on the function declaration: syscall__SYSCALLNAME(struct pt_regs *ctx, [, argument1 ...]).

For example:

c
int syscall__execve(struct pt_regs *ctx,
    const char __user *filename,
    const char __user *const __user *__argv,
    const char __user *const __user *__envp)
{
    [...]
}

This instruments the execve system call.

The first argument is always struct pt_regs *, the remainder are the arguments to the function (they don't need to be specified, if you don't intend to use them).

Corresponding Python code:

python
b = BPF(text=bpf_text)
execve_fnname = b.get_syscall_fnname("execve")
b.attach_kprobe(event=execve_fnname, fn_name="syscall__execve")

Examples in situ:
code (output)

9. kfuncs

Syntax: KFUNC_PROBE(function, typeof(arg1) arg1, typeof(arg2) arge ...)
MODULE_KFUNC_PROBE(module, function, typeof(arg1) arg1, typeof(arg2) arge ...)

This is a macro that instruments the kernel function via trampoline
before the function is executed. It's defined by function name and
the function arguments defined as argX.

For example:

c
KFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode)
{
    ...

This instruments the do_sys_open kernel function and make its arguments
accessible as standard argument values.

Examples in situ:
search /tools

10. kretfuncs

Syntax: KRETFUNC_PROBE(event, typeof(arg1) arg1, typeof(arg2) arge ..., int ret)
MODULE_KRETFUNC_PROBE(module, function, typeof(arg1) arg1, typeof(arg2) arge ...)

This is a macro that instruments the kernel function via trampoline
after the function is executed. It's defined by function name and
the function arguments defined as argX.

The last argument of the probe is the return value of the instrumented function.

For example:

c
KRETFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode, int ret)
{
    ...

This instruments the do_sys_open kernel function and make its arguments
accessible as standard argument values together with its return value.

Examples in situ:
search /tools

11. LSM Probes

Syntax: LSM_PROBE(hook, typeof(arg1) arg1, typeof(arg2) arg2 ...)

This is a macro that instruments an LSM hook as a BPF program. It can be
used to audit security events and implement MAC security policies in BPF.
It is defined by specifying the hook name followed by its arguments.

Hook names can be found in
include/linux/security.h
by taking functions like security_hookname and taking just the hookname part.
For example, security_bpf would simply become bpf.

Unlike other BPF program types, the return value specified in an LSM probe
matters. A return value of 0 allows the hook to succeed, whereas
any non-zero return value will cause the hook to fail and deny the
security operation.

The following example instruments a hook that denies all future BPF operations:

c
LSM_PROBE(bpf, int cmd, union bpf_attr *attr, unsigned int size)
{
    return -EPERM;
}

This instruments the security_bpf hook and causes it to return -EPERM.
Changing return -EPERM to return 0 would cause the BPF program
to allow the operation instead.

LSM probes require at least a 5.7+ kernel with the following configuation options set:

  • CONFIG_BPF_LSM=y
  • CONFIG_LSM comma separated string must contain "bpf" (for example,
    CONFIG_LSM="lockdown,yama,bpf")

Examples in situ:
search /tests

12. BPF ITERATORS

Syntax: BPF_ITER(target)

This is a macro to define a program signature for a bpf iterator program. The argument target specifies what to iterate for the program.

Currently, kernel does not have interface to discover what targets are supported. A good place to find what is supported is in tools/testing/selftests/bpf/prog_test/bpf_iter.c and some sample bpf iter programs are in tools/testing/selftests/bpf/progs with file name prefix bpf_iter.

The following example defines a program for target task, which traverses all tasks in the kernel.

c
BPF_ITER(task)
{
  struct seq_file *seq = ctx->meta->seq;
  struct task_struct *task = ctx->task;

  if (task == (void *)0)
    return 0;

  ... task->pid, task->tgid, task->comm, ...
  return 0;
}

BPF iterators are introduced in 5.8 kernel for task, task_file, bpf_map, netlink_sock and ipv6_route . In 5.9, support is added to tcp/udp sockets and bpf map element (hashmap, arraymap and sk_local_storage_map) traversal.

Data

1. bpf_probe_read_kernel()

Syntax: int bpf_probe_read_kernel(void *dst, int size, const void *src)

Return: 0 on success

This copies size bytes from kernel address space to the BPF stack, so that BPF can later operate on it. For safety, all kernel memory reads must pass through bpf_probe_read_kernel(). This happens automatically in some cases, such as dereferencing kernel variables, as bcc will rewrite the BPF program to include the necessary bpf_probe_read_kernel().

Examples in situ:
search /examples,
search /tools

2. bpf_probe_read_kernel_str()

Syntax: int bpf_probe_read_kernel_str(void *dst, int size, const void *src)

Return:

  • > 0 length of the string including the trailing NULL on success
  • < 0 error

This copies a NULL terminated string from kernel address space to the BPF stack, so that BPF can later operate on it. In case the string length is smaller than size, the target is not padded with further NULL bytes. In case the string length is larger than size, just size - 1 bytes are copied and the last byte is set to NULL.

Examples in situ:
search /examples,
search /tools

3. bpf_ktime_get_ns()

Syntax: u64 bpf_ktime_get_ns(void)

Return: u64 number of nanoseconds. Starts at system boot time but stops during suspend.

Examples in situ:
search /examples,
search /tools

4. bpf_get_current_pid_tgid()

Syntax: u64 bpf_get_current_pid_tgid(void)

Return: current->tgid << 32 | current->pid

Returns the process ID in the lower 32 bits (kernel's view of the PID, which in user space is usually presented as the thread ID), and the thread group ID in the upper 32 bits (what user space often thinks of as the PID). By directly setting this to a u32, we discard the upper 32 bits.

Examples in situ:
search /examples,
search /tools

5. bpf_get_current_uid_gid()

Syntax: u64 bpf_get_current_uid_gid(void)

Return: current_gid << 32 | current_uid

Returns the user ID and group IDs.

Examples in situ:
search /examples,
search /tools

6. bpf_get_current_comm()

Syntax: bpf_get_current_comm(char *buf, int size_of_buf)

Return: 0 on success

Populates the first argument address with the current process name. It should be a pointer to a char array of at least size TASK_COMM_LEN, which is defined in linux/sched.h. For example:

c
#include <linux/sched.h>

int do_trace(struct pt_regs *ctx) {
    char comm[TASK_COMM_LEN];
    bpf_get_current_comm(&comm, sizeof(comm));
[...]

Examples in situ:
search /examples,
search /tools

7. bpf_get_current_task()

Syntax: bpf_get_current_task()

Return: current task as a pointer to struct task_struct.

Returns a pointer to the current task's task_struct object. This helper can be used to compute the on-CPU time for a process, identify kernel threads, get the current CPU's run queue, or retrieve many other pieces of information.

With Linux 4.13, due to issues with field randomization, you may need two #define directives before the includes:

c
#define randomized_struct_fields_start  struct {
#define randomized_struct_fields_end    };
#include <linux/sched.h>

int do_trace(void *ctx) {
    struct task_struct *t = (struct task_struct *)bpf_get_current_task();
[...]

Examples in situ:
search /examples,
search /tools

8. bpf_log2l()

Syntax: unsigned int bpf_log2l(unsigned long v)

Returns the log-2 of the provided value. This is often used to create indexes for histograms, to construct power-of-2 histograms.

Examples in situ:
search /examples,
search /tools

9. bpf_get_prandom_u32()

Syntax: u32 bpf_get_prandom_u32()

Returns a pseudo-random u32.

Example in situ:
search /examples,
search /tools

10. bpf_probe_read_user()

Syntax: int bpf_probe_read_user(void *dst, int size, const void *src)

Return: 0 on success

This attempts to safely read size bytes from user address space to the BPF stack, so that BPF can later operate on it. For safety, all user address space memory reads must pass through bpf_probe_read_user().

Examples in situ:
search /examples,
search /tools

11. bpf_probe_read_user_str()

Syntax: int bpf_probe_read_user_str(void *dst, int size, const void *src)

Return:

  • > 0 length of the string including the trailing NULL on success
  • < 0 error

This copies a NULL terminated string from user address space to the BPF stack, so that BPF can later operate on it. In case the string length is smaller than size, the target is not padded with further NULL bytes. In case the string length is larger than size, just size - 1 bytes are copied and the last byte is set to NULL.

Examples in situ:
search /examples,
search /tools

12. bpf_get_ns_current_pid_tgid()

Syntax: u32 bpf_get_ns_current_pid_tgid(u64 dev, u64 ino, struct bpf_pidns_info* nsdata, u32 size)

Values for pid and tgid as seen from the current namespace will be returned in nsdata.

Return 0 on success, or one of the following in case of failure:

  • -EINVAL if dev and inum supplied don't match dev_t and inode number with nsfs of current task, or if dev conversion to dev_t lost high bits.

  • -ENOENT if pidns does not exists for the current task.

Examples in situ:
search /examples,
search /tools

Debugging

1. bpf_override_return()

Syntax: int bpf_override_return(struct pt_regs *, unsigned long rc)

Return: 0 on success

When used in a program attached to a function entry kprobe, causes the
execution of the function to be skipped, immediately returning rc instead.
This is used for targeted error injection.

bpf_override_return will only work when the kprobed function is whitelisted to
allow error injections. Whitelisting entails tagging a function with
ALLOW_ERROR_INJECTION() in the kernel source tree; see io_ctl_init for
an example. If the kprobed function is not whitelisted, the bpf program will
fail to attach with ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument

c
int kprobe__io_ctl_init(void *ctx) {
	bpf_override_return(ctx, -ENOMEM);
	return 0;
}

Output

1. bpf_trace_printk()

Syntax: int bpf_trace_printk(const char *fmt, ...)

Return: 0 on success

A simple kernel facility for printf() to the common trace_pipe (/sys/kernel/debug/tracing/trace_pipe). This is ok for some quick examples, but has limitations: 3 args max, 1 %s only, and trace_pipe is globally shared, so concurrent programs will have clashing output. A better interface is via BPF_PERF_OUTPUT(). Note that calling this helper is made simpler than the original kernel version, which has fmt_size as the second parameter.

Examples in situ:
search /examples,
search /tools

2. BPF_PERF_OUTPUT

Syntax: BPF_PERF_OUTPUT(name)

Creates a BPF table for pushing out custom event data to user space via a perf ring buffer. This is the preferred method for pushing per-event data to user space.

For example:

c
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);

int hello(struct pt_regs *ctx) {
    struct data_t data = {};

    data.pid = bpf_get_current_pid_tgid();
    data.ts = bpf_ktime_get_ns();
    bpf_get_current_comm(&data.comm, sizeof(data.comm));

    events.perf_submit(ctx, &data, sizeof(data));

    return 0;
}

The output table is named events, and data is pushed to it via events.perf_submit().

Examples in situ:
search /examples,
search /tools

3. perf_submit()

Syntax: int perf_submit((void *)ctx, (void *)data, u32 data_size)

Return: 0 on success

A method of a BPF_PERF_OUTPUT table, for submitting custom event data to user space. See the BPF_PERF_OUTPUT entry. (This ultimately calls bpf_perf_event_output().)

The ctx parameter is provided in kprobes or kretprobes. For SCHED_CLS or SOCKET_FILTER programs, the struct __sk_buff *skb must be used instead.

Examples in situ:
search /examples,
search /tools

4. perf_submit_skb()

Syntax: int perf_submit_skb((void *)ctx, u32 packet_size, (void *)data, u32 data_size)

Return: 0 on success

A method of a BPF_PERF_OUTPUT table available in networking program types, for submitting custom event data to user space, along with the first packet_size bytes of the packet buffer. See the BPF_PERF_OUTPUT entry. (This ultimately calls bpf_perf_event_output().)

Examples in situ:
search /examples,
search /tools

5. BPF_RINGBUF_OUTPUT

Syntax: BPF_RINGBUF_OUTPUT(name, page_cnt)

Creates a BPF table for pushing out custom event data to user space via a ringbuf ring buffer.
BPF_RINGBUF_OUTPUT has several advantages over BPF_PERF_OUTPUT, summarized as follows:

  • Buffer is shared across all CPUs, meaning no per-CPU allocation
  • Supports two APIs for BPF programs
  • BPF APIs do not require access to a CPU ctx argument
  • Superior performance and latency in userspace thanks to a shared ring buffer manager
  • Supports two ways of consuming data in userspace

Starting in Linux 5.8, this should be the preferred method for pushing per-event data to user space.

Example of both APIs:

c
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};

// Creates a ringbuf called events with 8 pages of space, shared across all CPUs
BPF_RINGBUF_OUTPUT(events, 8);

int first_api_example(struct pt_regs *ctx) {
    struct data_t data = {};

    data.pid = bpf_get_current_pid_tgid();
    data.ts = bpf_ktime_get_ns();
    bpf_get_current_comm(&data.comm, sizeof(data.comm));

    events.ringbuf_output(&data, sizeof(data), 0 /* flags */);

    return 0;
}

int second_api_example(struct pt_regs *ctx) {
    struct data_t *data = events.ringbuf_reserve(sizeof(struct data_t));
    if (!data) { // Failed to reserve space
        return 1;
    }

    data->pid = bpf_get_current_pid_tgid();
    data->ts = bpf_ktime_get_ns();
    bpf_get_current_comm(&data->comm, sizeof(data->comm));

    events.ringbuf_submit(data, 0 /* flags */);

    return 0;
}

The output table is named events. Data is allocated via events.ringbuf_reserve() and pushed to it via events.ringbuf_submit().

Examples in situ:
search /examples,

6. ringbuf_output()

Syntax: int ringbuf_output((void *)data, u64 data_size, u64 flags)

Return: 0 on success

Flags:

  • BPF_RB_NO_WAKEUP: Do not sent notification of new data availability
  • BPF_RB_FORCE_WAKEUP: Send notification of new data availability unconditionally

A method of the BPF_RINGBUF_OUTPUT table, for submitting custom event data to user space. This method works like perf_submit(),
although it does not require a ctx argument.

Examples in situ:
search /examples,

7. ringbuf_reserve()

Syntax: void* ringbuf_reserve(u64 data_size)

Return: Pointer to data struct on success, NULL on failure

A method of the BPF_RINGBUF_OUTPUT table, for reserving space in the ring buffer and simultaenously
allocating a data struct for output. Must be used with one of ringbuf_submit or ringbuf_discard.

Examples in situ:
search /examples,

8. ringbuf_submit()

Syntax: void ringbuf_submit((void *)data, u64 flags)

Return: Nothing, always succeeds

Flags:

  • BPF_RB_NO_WAKEUP: Do not sent notification of new data availability
  • BPF_RB_FORCE_WAKEUP: Send notification of new data availability unconditionally

A method of the BPF_RINGBUF_OUTPUT table, for submitting custom event data to user space. Must be preceded by a call to
ringbuf_reserve() to reserve space for the data.

Examples in situ:
search /examples,

9. ringbuf_discard()

Syntax: void ringbuf_discard((void *)data, u64 flags)

Return: Nothing, always succeeds

Flags:

  • BPF_RB_NO_WAKEUP: Do not sent notification of new data availability
  • BPF_RB_FORCE_WAKEUP: Send notification of new data availability unconditionally

A method of the BPF_RINGBUF_OUTPUT table, for discarding custom event data; userspace
ignores the data associated with the discarded event. Must be preceded by a call to
ringbuf_reserve() to reserve space for the data.

Examples in situ:
search /examples,

10. ringbuf_query()

Syntax: u64 ringbuf_query(u64 flags)

Return: Requested value, or 0, if flags are not recognized

Flags:

  • BPF_RB_AVAIL_DATA: Amount of data not yet consumed
  • BPF_RB_RING_SIZE: The size of ring buffer
  • BPF_RB_CONS_POS: Consumer position
  • BPF_RB_PROD_POS: Producer(s) position

A method of the BPF_RINGBUF_OUTPUT table, for getting various properties of ring buffer. Returned values are momentarily snapshots of ring buffer state and could be off by the time helper returns, so this should be used only for debugging/reporting reasons or for implementing various heuristics, that take into account highly-changeable nature of some of those characteristics.

Examples in situ:
search /examples,

Maps

Maps are BPF data stores, and are the basis for higher level object types including tables, hashes, and histograms.

1. BPF_TABLE

Syntax: BPF_TABLE(_table_type, _key_type, _leaf_type, _name, _max_entries)

Creates a map named _name. Most of the time this will be used via higher-level macros, like BPF_HASH, BPF_ARRAY, BPF_HISTOGRAM, etc.

BPF_F_TABLE is a variant that takes a flag in the last parameter. BPF_TABLE(...) is actually a wrapper to BPF_F_TABLE(..., 0 /* flag */).

Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().

Examples in situ:
search /examples,
search /tools

Pinned Maps

Syntax: BPF_TABLE_PINNED(_table_type, _key_type, _leaf_type, _name, _max_entries, "/sys/fs/bpf/xyz")

Create a new map if it doesn't exist and pin it to the bpffs as a FILE, otherwise use the map that was pinned to the bpffs. The type information is not enforced and the actual map type depends on the map that got pinned to the location.

For example:

c
BPF_TABLE_PINNED("hash", u64, u64, ids, 1024, "/sys/fs/bpf/ids");

2. BPF_HASH

Syntax: BPF_HASH(name [, key_type [, leaf_type [, size]]])

Creates a hash map (associative array) named name, with optional parameters.

Defaults: BPF_HASH(name, key_type=u64, leaf_type=u64, size=10240)

For example:

c
BPF_HASH(start, struct request *);

This creates a hash named start where the key is a struct request *, and the value defaults to u64. This hash is used by the disksnoop.py example for saving timestamps for each I/O request, where the key is the pointer to struct request, and the value is the timestamp.

This is a wrapper macro for BPF_TABLE("hash", ...).

Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().

Examples in situ:
search /examples,
search /tools

3. BPF_ARRAY

Syntax: BPF_ARRAY(name [, leaf_type [, size]])

Creates an int-indexed array which is optimized for fastest lookup and update, named name, with optional parameters.

Defaults: BPF_ARRAY(name, leaf_type=u64, size=10240)

For example:

c
BPF_ARRAY(counts, u64, 32);

This creates an array named counts where with 32 buckets and 64-bit integer values. This array is used by the funccount.py example for saving call count of each function.

This is a wrapper macro for BPF_TABLE("array", ...).

Methods (covered later): map.lookup(), map.update(), map.increment(). Note that all array elements are pre-allocated with zero values and can not be deleted.

Examples in situ:
search /examples,
search /tools

4. BPF_HISTOGRAM

Syntax: BPF_HISTOGRAM(name [, key_type [, size ]])

Creates a histogram map named name, with optional parameters.

Defaults: BPF_HISTOGRAM(name, key_type=int, size=64)

For example:

c
BPF_HISTOGRAM(dist);

This creates a histogram named dist, which defaults to 64 buckets indexed by keys of type int.

This is a wrapper macro for BPF_TABLE("histgram", ...).

Methods (covered later): map.increment().

Examples in situ:
search /examples,
search /tools

5. BPF_STACK_TRACE

Syntax: BPF_STACK_TRACE(name, max_entries)

Creates stack trace map named name, with a maximum entry count provided. These maps are used to store stack traces.

For example:

c
BPF_STACK_TRACE(stack_traces, 1024);

This creates stack trace map named stack_traces, with a maximum number of stack trace entries of 1024.

This is a wrapper macro for BPF_TABLE("stacktrace", ...).

Methods (covered later): map.get_stackid().

Examples in situ:
search /examples,
search /tools

6. BPF_PERF_ARRAY

Syntax: BPF_PERF_ARRAY(name, max_entries)

Creates perf array named name, with a maximum entry count provided, which must be equal to the number of system cpus. These maps are used to fetch hardware performance counters.

For example:

c
text="""
BPF_PERF_ARRAY(cpu_cycles, NUM_CPUS);
"""
b = bcc.BPF(text=text, cflags=["-DNUM_CPUS=%d" % multiprocessing.cpu_count()])
b["cpu_cycles"].open_perf_event(b["cpu_cycles"].HW_CPU_CYCLES)

This creates a perf array named cpu_cycles, with number of entries equal to the number of cpus/cores. The array is configured so that later calling map.perf_read() will return a hardware-calculated counter of the number of cycles elapsed from some point in the past. Only one type of hardware counter may be configured per table at a time.

Methods (covered later): map.perf_read().

Examples in situ:
search /tests

7. BPF_PERCPU_HASH

Syntax: BPF_PERCPU_HASH(name [, key_type [, leaf_type [, size]]])

Creates NUM_CPU int-indexed hash maps (associative arrays) named name, with optional parameters. Each CPU will have a separate copy of this array. The copies are not kept synchronized in any way.

Note that due to limits defined in the kernel (in linux/mm/percpu.c), the leaf_type cannot have a size of more than 32KB.
In other words, BPF_PERCPU_HASH elements cannot be larger than 32KB in size.

Defaults: BPF_PERCPU_HASH(name, key_type=u64, leaf_type=u64, size=10240)

For example:

c
BPF_PERCPU_HASH(start, struct request *);

This creates NUM_CPU hashes named start where the key is a struct request *, and the value defaults to u64.

This is a wrapper macro for BPF_TABLE("percpu_hash", ...).

Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().

Examples in situ:
search /examples,
search /tools

8. BPF_PERCPU_ARRAY

Syntax: BPF_PERCPU_ARRAY(name [, leaf_type [, size]])

Creates NUM_CPU int-indexed arrays which are optimized for fastest lookup and update, named name, with optional parameters. Each CPU will have a separate copy of this array. The copies are not kept synchronized in any way.

Note that due to limits defined in the kernel (in linux/mm/percpu.c), the leaf_type cannot have a size of more than 32KB.
In other words, BPF_PERCPU_ARRAY elements cannot be larger than 32KB in size.

Defaults: BPF_PERCPU_ARRAY(name, leaf_type=u64, size=10240)

For example:

c
BPF_PERCPU_ARRAY(counts, u64, 32);

This creates NUM_CPU arrays named counts where with 32 buckets and 64-bit integer values.

This is a wrapper macro for BPF_TABLE("percpu_array", ...).

Methods (covered later): map.lookup(), map.update(), map.increment(). Note that all array elements are pre-allocated with zero values and can not be deleted.

Examples in situ:
search /examples,
search /tools

9. BPF_LPM_TRIE

Syntax: BPF_LPM_TRIE(name [, key_type [, leaf_type [, size]]])

Creates a longest prefix match trie map named name, with optional parameters.

Defaults: BPF_LPM_TRIE(name, key_type=u64, leaf_type=u64, size=10240)

For example:

c
BPF_LPM_TRIE(trie, struct key_v6);

This creates an LPM trie map named trie where the key is a struct key_v6, and the value defaults to u64.

This is a wrapper macro to BPF_F_TABLE("lpm_trie", ..., BPF_F_NO_PREALLOC).

Methods (covered later): map.lookup(), map.lookup_or_try_init(), map.delete(), map.update(), map.insert(), map.increment().

Examples in situ:
search /examples,
search /tools

10. BPF_PROG_ARRAY

Syntax: BPF_PROG_ARRAY(name, size)

This creates a program array named name with size entries. Each entry of the array is either a file descriptor to a bpf program or NULL. The array acts as a jump table so that bpf programs can "tail-call" other bpf programs.

This is a wrapper macro for BPF_TABLE("prog", ...).

Methods (covered later): map.call().

Examples in situ:
search /examples,
search /tests,
assign fd

11. BPF_DEVMAP

Syntax: BPF_DEVMAP(name, size)

This creates a device map named name with size entries. Each entry of the map is an ifindex to a network interface. This map is only used in XDP.

For example:

c
BPF_DEVMAP(devmap, 10);

Methods (covered later): map.redirect_map().

Examples in situ:
search /examples,

12. BPF_CPUMAP

Syntax: BPF_CPUMAP(name, size)

This creates a cpu map named name with size entries. The index of the map represents the CPU id and each entry is the size of the ring buffer allocated for the CPU. This map is only used in XDP.

For example:

c
BPF_CPUMAP(cpumap, 16);

Methods (covered later): map.redirect_map().

Examples in situ:
search /examples,

13. BPF_XSKMAP

Syntax: BPF_XSKMAP(name, size [, "/sys/fs/bpf/xyz"])

This creates a xsk map named name with size entries and pin it to the bpffs as a FILE. Each entry represents one NIC's queue id. This map is only used in XDP to redirect packet to an AF_XDP socket. If the AF_XDP socket is binded to a queue which is different than the current packet's queue id, the packet will be dropped. For kernel v5.3 and latter, lookup method is available and can be used to check whether and AF_XDP socket is available for the current packet's queue id. More details at AF_XDP.

For example:

c
BPF_XSKMAP(xsks_map, 8);

Methods (covered later): map.redirect_map(). map.lookup()

Examples in situ:
search /examples,

14. BPF_ARRAY_OF_MAPS

Syntax: BPF_ARRAY_OF_MAPS(name, inner_map_name, size)

This creates an array map with a map-in-map type (BPF_MAP_TYPE_HASH_OF_MAPS) map named name with size entries. The inner map meta data is provided by map inner_map_name and can be most of array or hash maps except BPF_MAP_TYPE_PROG_ARRAY, BPF_MAP_TYPE_CGROUP_STORAGE and BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE.

For example:

c
BPF_TABLE("hash", int, int, ex1, 1024);
BPF_TABLE("hash", int, int, ex2, 1024);
BPF_ARRAY_OF_MAPS(maps_array, "ex1", 10);

15. BPF_HASH_OF_MAPS

Syntax: BPF_HASH_OF_MAPS(name, key_type, inner_map_name, size)

This creates a hash map with a map-in-map type (BPF_MAP_TYPE_HASH_OF_MAPS) map named name with size entries. The inner map meta data is provided by map inner_map_name and can be most of array or hash maps except BPF_MAP_TYPE_PROG_ARRAY, BPF_MAP_TYPE_CGROUP_STORAGE and BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE.

For example:

c
BPF_ARRAY(ex1, int, 1024);
BPF_ARRAY(ex2, int, 1024);
BPF_HASH_OF_MAPS(maps_hash, struct custom_key, "ex1", 10);

16. BPF_STACK

Syntax: BPF_STACK(name, leaf_type, max_entries[, flags])

Creates a stack named name with value type leaf_type and max entries max_entries.
Stack and Queue maps are only available from Linux 4.20+.

For example:

c
BPF_STACK(stack, struct event, 10240);

This creates a stack named stack where the value type is struct event, that holds up to 10240 entries.

Methods (covered later): map.push(), map.pop(), map.peek().

Examples in situ:
search /tests,

17. BPF_QUEUE

Syntax: BPF_QUEUE(name, leaf_type, max_entries[, flags])

Creates a queue named name with value type leaf_type and max entries max_entries.
Stack and Queue maps are only available from Linux 4.20+.

For example:

c
BPF_QUEUE(queue, struct event, 10240);

This creates a queue named queue where the value type is struct event, that holds up to 10240 entries.

Methods (covered later): map.push(), map.pop(), map.peek().

Examples in situ:
search /tests,

18. BPF_SOCKHASH

Syntax: BPF_SOCKHASH(name[, key_type [, max_entries)

Creates a hash named name, with optional parameters. sockhash is only available from Linux 4.18+.

Default: BPF_SOCKHASH(name, key_type=u32, max_entries=10240)

For example:

c
struct sock_key {
  u32 remote_ip4;
  u32 local_ip4;
  u32 remote_port;
  u32 local_port;
};
BPF_HASH(skh, struct sock_key, 65535);

This creates a hash named skh where the key is a struct sock_key.

A sockhash is a BPF map type that holds references to sock structs. Then with a new sk/msg redirect bpf helper BPF programs can use the map to redirect skbs/msgs between sockets (map.sk_redirect_hash()/map.msg_redirect_hash()).

The difference between BPF_SOCKHASH and BPF_SOCKMAP is that BPF_SOCKMAP is implemented based on an array, and enforces keys to be four bytes. While BPF_SOCKHASH is implemented based on hash table, and the type of key can be specified freely.

Methods (covered later): map.sock_hash_update(), map.msg_redirect_hash(), map.sk_redirect_hash().

search /tests

19. map.lookup()

Syntax: *val map.lookup(&key)

Lookup the key in the map, and return a pointer to its value if it exists, else NULL. We pass the key in as an address to a pointer.

Examples in situ:
search /examples,
search /tools

20. map.lookup_or_try_init()

Syntax: *val map.lookup_or_try_init(&key, &zero)

Lookup the key in the map, and return a pointer to its value if it exists, else initialize the key's value to the second argument. This is often used to initialize values to zero. If the key cannot be inserted (e.g. the map is full) then NULL is returned.

Examples in situ:
search /examples,
search /tools

Note: The old map.lookup_or_init() may cause return from the function, so lookup_or_try_init() is recommended as it
does not have this side effect.

21. map.delete()

Syntax: map.delete(&key)

Delete the key from the hash.

Examples in situ:
search /examples,
search /tools

22. map.update()

Syntax: map.update(&key, &val)

Associate the value in the second argument to the key, overwriting any previous value.

Examples in situ:
search /examples,
search /tools

23. map.insert()

Syntax: map.insert(&key, &val)

Associate the value in the second argument to the key, only if there was no previous value.

Examples in situ:
search /examples,
search /tools

24. map.increment()

Syntax: map.increment(key[, increment_amount])

Increments the key's value by increment_amount, which defaults to 1. Used for histograms.

map.increment() are not atomic. In the concurrency case. If you want more accurate results, use map.atomic_increment() instead of map.increment(). The overhead of map.increment() and map.atomic_increment() is similar.

Note. When using map.atomic_increment() to operate on a BPF map of type BPF_MAP_TYPE_HASH, map.atomic_increment() does not guarantee the atomicity of the operation when the specified key does not exist.

Examples in situ:
search /examples,
search /tools

25. map.get_stackid()

Syntax: int map.get_stackid(void *ctx, u64 flags)

This walks the stack found via the struct pt_regs in ctx, saves it in the stack trace map, and returns a unique ID for the stack trace.

Examples in situ:
search /examples,
search /tools

26. map.perf_read()

Syntax: u64 map.perf_read(u32 cpu)

This returns the hardware performance counter as configured in 5. BPF_PERF_ARRAY

Examples in situ:
search /tests

27. map.call()

Syntax: void map.call(void *ctx, int index)

This invokes bpf_tail_call() to tail-call the bpf program which the index entry in BPF_PROG_ARRAY points to. A tail-call is different from the normal call. It reuses the current stack frame after jumping to another bpf program and never goes back. If the index entry is empty, it won't jump anywhere and the program execution continues as normal.

For example:

c
BPF_PROG_ARRAY(prog_array, 10);

int tail_call(void *ctx) {
    bpf_trace_printk("Tail-call\n");
    return 0;
}

int do_tail_call(void *ctx) {
    bpf_trace_printk("Original program\n");
    prog_array.call(ctx, 2);
    return 0;
}
python
b = BPF(src_file="example.c")
tail_fn = b.load_func("tail_call", BPF.KPROBE)
prog_array = b.get_table("prog_array")
prog_array[c_int(2)] = c_int(tail_fn.fd)
b.attach_kprobe(event="some_kprobe_event", fn_name="do_tail_call")

This assigns tail_call() to prog_array[2]. In the end of do_tail_call(), prog_array.call(ctx, 2) tail-calls tail_call() and executes it.

NOTE: To prevent infinite loop, the maximum number of tail-calls is 32 (MAX_TAIL_CALL_CNT).

Examples in situ:
search /examples,
search /tests

28. map.redirect_map()

Syntax: int map.redirect_map(int index, int flags)

This redirects the incoming packets based on the index entry. If the map is BPF_DEVMAP, the packet will be sent to the transmit queue of the network interface that the entry points to. If the map is BPF_CPUMAP, the packet will be sent to the ring buffer of the index CPU and be processed by the CPU later. If the map is BPF_XSKMAP, the packet will be sent to the AF_XDP socket attached to the queue.

If the packet is redirected successfully, the function will return XDP_REDIRECT. Otherwise, it will return XDP_ABORTED to discard the packet.

For example:

c
BPF_DEVMAP(devmap, 1);

int redirect_example(struct xdp_md *ctx) {
    return devmap.redirect_map(0, 0);
}
int xdp_dummy(struct xdp_md *ctx) {
    return XDP_PASS;
}
python
ip = pyroute2.IPRoute()
idx = ip.link_lookup(ifname="eth1")[0]

b = bcc.BPF(src_file="example.c")

devmap = b.get_table("devmap")
devmap[c_uint32(0)] = c_int(idx)

in_fn = b.load_func("redirect_example", BPF.XDP)
out_fn = b.load_func("xdp_dummy", BPF.XDP)
b.attach_xdp("eth0", in_fn, 0)
b.attach_xdp("eth1", out_fn, 0)

Examples in situ:
search /examples,

29. map.push()

Syntax: int map.push(&val, int flags)

Push an element onto a Stack or Queue table.
Passing BPF_EXIST as a flag causes the Queue or Stack to discard the oldest element if it is full.
Returns 0 on success, negative error on failure.

Examples in situ:
search /tests,

30. map.pop()

Syntax: int map.pop(&val)

Pop an element from a Stack or Queue table. *val is populated with the result.
Unlike peeking, popping removes the element.
Returns 0 on success, negative error on failure.

Examples in situ:
search /tests,

31. map.peek()

Syntax: int map.peek(&val)

Peek an element at the head of a Stack or Queue table. *val is populated with the result.
Unlike popping, peeking does not remove the element.
Returns 0 on success, negative error on failure.

Examples in situ:
search /tests,

32. map.sock_hash_update()

Syntax: int map.sock_hash_update(struct bpf_sock_ops *skops, &key, int flags)

Add an entry to, or update a sockhash map referencing sockets. The skops is used as a new value for the entry associated to key. flags is one of:

text
BPF_NOEXIST: The entry for key must not exist in the map.
BPF_EXIST: The entry for key must already exist in the map.
BPF_ANY: No condition on the existence of the entry for key.

If the map has eBPF programs (parser and verdict), those will be inherited by the socket being added. If the socket is already attached to eBPF programs, this results in an error.

Return 0 on success, or a negative error in case of failure.

Examples in situ:
search /tests,

33. map.msg_redirect_hash()

Syntax: int map.msg_redirect_hash(struct sk_msg_buff *msg, void *key, u64 flags)

This helper is used in programs implementing policies at the socket level. If the message msg is allowed to pass (i.e. if the verdict eBPF program returns SK_PASS), redirect it to the socket referenced by map (of type BPF_MAP_TYPE_SOCKHASH) using hash key. Both ingress and egress interfaces can be used for redirection. The BPF_F_INGRESS value in flags is used to make the distinction (ingress path is selected if the flag is present, egress path otherwise). This is the only flag supported for now.

Return SK_PASS on success, or SK_DROP on error.

Examples in situ:
search /tests,

34. map.sk_redirect_hash()

Syntax: int map.sk_redirect_hash(struct sk_buff *skb, void *key, u64 flags)

This helper is used in programs implementing policies at the skb socket level. If the sk_buff skb is allowed to pass (i.e. if the verdict eBPF program returns SK_PASS), redirect it to the socket referenced by map (of type BPF_MAP_TYPE_SOCKHASH) using hash key. Both ingress and egress interfaces can be used for redirection. The BPF_F_INGRESS value in flags is used to make the distinction (ingress path is selected if the flag is present, egress otherwise). This is the only flag supported for now.

Return SK_PASS on success, or SK_DROP on error.

Examples in situ:
search /tests,

Licensing

Depending on which BPF helpers are used, a GPL-compatible license is required.

The special BCC macro BPF_LICENSE specifies the license of the BPF program. You can set the license as a comment in your source code, but the kernel has a special interface to specify it programmatically. If you need to use GPL-only helpers, it is recommended to specify the macro in your C code so that the kernel can understand it:

c
// SPDX-License-Identifier: GPL-2.0+
#define BPF_LICENSE GPL

Otherwise, the kernel may reject loading your program (see the error description below). Note that it supports multiple words and quotes are not necessary:

c
// SPDX-License-Identifier: GPL-2.0+ OR BSD-2-Clause
#define BPF_LICENSE Dual BSD/GPL

Check the BPF helpers reference to see which helpers are GPL-only and what the kernel understands as GPL-compatible.

If the macro is not specified, BCC will automatically define the license of the program as GPL.

Rewriter

One of jobs for rewriter is to turn implicit memory accesses to explicit ones using kernel helpers. Recent kernel introduced a config option ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE which will be set for architectures whose user address space and kernel address are disjoint. x86 and arm has this config option set while s390 does not. If ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE is not set, the bpf old helper bpf_probe_read() will not be available. Some existing users may have implicit memory accesses to access user memory, so using bpf_probe_read_kernel() will cause their application to fail. Therefore, for non-s390, the rewriter will use bpf_probe_read() for these implicit memory accesses. For s390, bpf_probe_read_kernel() is used as default and users should use bpf_probe_read_user() explicitly when accessing user memories.

bcc Python

Initialization

Constructors.

1. BPF

Syntax: BPF({text=BPF_program | src_file=filename} [, usdt_contexts=[USDT_object, ...]] [, cflags=[arg1, ...]] [, debug=int])

Creates a BPF object. This is the main object for defining a BPF program, and interacting with its output.

Exactly one of text or src_file must be supplied (not both).

The cflags specifies additional arguments to be passed to the compiler, for example -DMACRO_NAME=value or -I/include/path. The arguments are passed as an array, with each element being an additional argument. Note that strings are not split on whitespace, so each argument must be a different element of the array, e.g. ["-include", "header.h"].

The debug flags control debug output, and can be or'ed together:

  • DEBUG_LLVM_IR = 0x1 compiled LLVM IR
  • DEBUG_BPF = 0x2 loaded BPF bytecode and register state on branches
  • DEBUG_PREPROCESSOR = 0x4 pre-processor result
  • DEBUG_SOURCE = 0x8 ASM instructions embedded with source
  • DEBUG_BPF_REGISTER_STATE = 0x10 register state on all instructions in addition to DEBUG_BPF
  • DEBUG_BTF = 0x20 print the messages from the libbpf library.

Examples:

python
# define entire BPF program in one line:
BPF(text='int do_trace(void *ctx) { bpf_trace_printk("hit!\\n"); return 0; }');

# define program as a variable:
prog = """
int hello(void *ctx) {
    bpf_trace_printk("Hello, World!\\n");
    return 0;
}
"""
b = BPF(text=prog)

# source a file:
b = BPF(src_file = "vfsreadlat.c")

# include a USDT object:
u = USDT(pid=int(pid))
[...]
b = BPF(text=bpf_text, usdt_contexts=[u])

# add include paths:
u = BPF(text=prog, cflags=["-I/path/to/include"])

Examples in situ:
search /examples,
search /tools

2. USDT

Syntax: USDT({pid=pid | path=path})

Creates an object to instrument User Statically-Defined Tracing (USDT) probes. Its primary method is enable_probe().

Arguments:

  • pid: attach to this process ID.
  • path: instrument USDT probes from this binary path.

Examples:

python
# include a USDT object:
u = USDT(pid=int(pid))
[...]
b = BPF(text=bpf_text, usdt_contexts=[u])

Examples in situ:
search /examples,
search /tools

Events

1. attach_kprobe()

Syntax: BPF.attach_kprobe(event="event", fn_name="name")

Instruments the kernel function event() using kernel dynamic tracing of the function entry, and attaches our C defined function name() to be called when the kernel function is called.

For example:

python
b.attach_kprobe(event="sys_clone", fn_name="do_trace")

This will instrument the kernel sys_clone() function, which will then run our BPF defined do_trace() function each time it is called.

You can call attach_kprobe() more than once, and attach your BPF function to multiple kernel functions.
You can also call attach_kprobe() more than once to attach multiple BPF functions to the same kernel function.

See the previous kprobes section for how to instrument arguments from BPF.

Examples in situ:
search /examples,
search /tools

2. attach_kretprobe()

Syntax: BPF.attach_kretprobe(event="event", fn_name="name" [, maxactive=int])

Instruments the return of the kernel function event() using kernel dynamic tracing of the function return, and attaches our C defined function name() to be called when the kernel function returns.

For example:

python
b.attach_kretprobe(event="vfs_read", fn_name="do_return")

This will instrument the kernel vfs_read() function, which will then run our BPF defined do_return() function each time it is called.

You can call attach_kretprobe() more than once, and attach your BPF function to multiple kernel function returns.
You can also call attach_kretprobe() more than once to attach multiple BPF functions to the same kernel function return.

When a kretprobe is installed on a kernel function, there is a limit on how many parallel calls it can catch. You can change that limit with maxactive. See the kprobes documentation for its default value.

See the previous kretprobes section for how to instrument the return value from BPF.

Examples in situ:
search /examples,
search /tools

3. attach_tracepoint()

Syntax: BPF.attach_tracepoint(tp="tracepoint", fn_name="name")

Instruments the kernel tracepoint described by tracepoint, and when hit, runs the BPF function name().

This is an explicit way to instrument tracepoints. The TRACEPOINT_PROBE syntax, covered in the earlier tracepoints section, is an alternate method with the advantage of auto-declaring an args struct containing the tracepoint arguments. With attach_tracepoint(), the tracepoint arguments need to be declared in the BPF program.

For example:

python
# define BPF program
bpf_text = """
#include 

struct urandom_read_args {
    // from /sys/kernel/debug/tracing/events/random/urandom_read/format
    u64 __unused__;
    u32 got_bits;
    u32 pool_left;
    u32 input_left;
};

int printarg(struct urandom_read_args *args) {
    bpf_trace_printk("%d\\n", args->got_bits);
    return 0;
};
"""

# load BPF program
b = BPF(text=bpf_text)
b.attach_tracepoint("random:urandom_read", "printarg")

Notice how the first argument to printarg() is now our defined struct.

Examples in situ:
code,
search /examples,
search /tools

4. attach_uprobe()

Syntax: BPF.attach_uprobe(name="location", sym="symbol", fn_name="name" [, sym_off=int]), BPF.attach_uprobe(name="location", sym_re="regex", fn_name="name"), BPF.attach_uprobe(name="location", addr=int, fn_name="name"), BPF.attach_uprobe(name="location", sym="symbol", fn_name="name", [, pid=int])

Instruments the user-level function symbol() from either the library or binary named by location using user-level dynamic tracing of the function entry, and attach our C defined function name() to be called whenever the user-level function is called. If sym_off is given, the function is attached to the offset within the symbol.

The real address addr may be supplied in place of sym, in which case sym must be set to its default value. If the file is a non-PIE executable, addr must be a virtual address, otherwise it must be an offset relative to the file load address.

Instead of a symbol name, a regular expression can be provided in sym_re. The uprobe will then attach to symbols that match the provided regular expression.

Uprobes can be attached to a specific process by passing pid to attach_uprobe.
By default pid is set to -1, indicating the uprobe will be attached to all processes.
For libraries, the uprobe will attach to the version of the library used by the process if pid was given.
For how pid is used, see examples in funcinterval.

Libraries can be given in the name argument without the lib prefix, with a versioned SONAME (c.so.6), or with the full path (/usr/lib/...). Binaries can be given only with the full path (/bin/sh).

For example:

python
b.attach_uprobe(name="c", sym="strlen", fn_name="count")

This will instrument strlen() function from libc, and call our BPF function count() when it is called. Note how the "lib" in "libc" is not necessary to specify.

Other examples:

python
b.attach_uprobe(name="c", sym="getaddrinfo", fn_name="do_entry")
b.attach_uprobe(name="/usr/bin/python", sym="main", fn_name="do_main")

You can call attach_uprobe() more than once, and attach your BPF function to multiple user-level functions.

See the previous uprobes section for how to instrument arguments from BPF.

Examples in situ:
search /examples,
search /tools

5. attach_uretprobe()

Syntax: BPF.attach_uretprobe(name="location", sym="symbol", fn_name="name")

Instruments the return of the user-level function symbol() from either the library or binary named by location using user-level dynamic tracing of the function return, and attach our C defined function name() to be called whenever the user-level function returns.

For example:

python
b.attach_uretprobe(name="c", sym="strlen", fn_name="count")

This will instrument strlen() function from libc, and call our BPF function count() when it returns.

Other examples:

python
b.attach_uretprobe(name="c", sym="getaddrinfo", fn_name="do_return")
b.attach_uretprobe(name="/usr/bin/python", sym="main", fn_name="do_main")

You can call attach_uretprobe() more than once, and attach your BPF function to multiple user-level functions.
BPF.attach_uretprobe can also be used for a specific process.

See the previous uretprobes section for how to instrument the return value from BPF.

Examples in situ:
search /examples,
search /tools

6. USDT.enable_probe()

Syntax: USDT.enable_probe(probe=probe, fn_name=name)

Attaches a BPF C function name to the USDT probe probe.

Example:

python
# enable USDT probe from given PID
u = USDT(pid=int(pid))
u.enable_probe(probe="http__server__request", fn_name="do_trace")

To check if your binary has USDT probes, and what they are, you can run readelf -n binary and check the stap debug section.

Examples in situ:
search /examples,
search /tools

7. attach_raw_tracepoint()

Syntax: BPF.attach_raw_tracepoint(tp="tracepoint", fn_name="name")

Instruments the kernel raw tracepoint described by tracepoint (event only, no category), and when hit, runs the BPF function name().

This is an explicit way to instrument tracepoints. The RAW_TRACEPOINT_PROBE syntax, covered in the earlier raw tracepoints section, is an alternate method.

For example:

python
b.attach_raw_tracepoint("sched_switch", "do_trace")

Examples in situ:
search /tools

8. attach_raw_socket()

Syntax: BPF.attach_raw_socket(fn, dev)

Attaches a BPF function to the specified network interface.

The fn must be the type of BPF.function and the bpf_prog type needs to be BPF_PROG_TYPE_SOCKET_FILTER (fn=BPF.load_func(func_name, BPF.SOCKET_FILTER))

fn.sock is a non-blocking raw socket that was created and bound to dev.

All network packets processed by dev are copied to the recv-q of fn.sock after being processed by bpf_prog. Try to recv packet form fn.sock with rev/recvfrom/recvmsg. Note that if the recv-q is not read in time after the recv-q is full, the copied packets will be discarded.

We can use this feature to capture network packets just like tcpdump.

We can use ss --bpf --packet -p to observe fn.sock.

Example:

python
BPF.attach_raw_socket(bpf_func, ifname)

Examples in situ:
search /examples

9. attach_xdp()

Syntax: BPF.attach_xdp(dev="device", fn=b.load_func("fn_name",BPF.XDP), flags)

Instruments the network driver described by dev , and then receives the packet, run the BPF function fn_name() with flags.

Here is a list of optional flags.

python
# from xdp_flags uapi/linux/if_link.h
XDP_FLAGS_UPDATE_IF_NOEXIST = (1 << 0)
XDP_FLAGS_SKB_MODE = (1 << 1)
XDP_FLAGS_DRV_MODE = (1 << 2)
XDP_FLAGS_HW_MODE = (1 << 3)
XDP_FLAGS_REPLACE = (1 << 4)

You can use flags like this BPF.attach_xdp(dev="device", fn=b.load_func("fn_name",BPF.XDP), flags=BPF.XDP_FLAGS_UPDATE_IF_NOEXIST)

The default value of flags is 0. This means if there is no xdp program with device, the fn will run with that device. If there is an xdp program running with device, the old program will be replaced with new fn program.

Currently, bcc does not support XDP_FLAGS_REPLACE flag. The following are the descriptions of other flags.

1. XDP_FLAGS_UPDATE_IF_NOEXIST

If an XDP program is already attached to the specified driver, attaching the XDP program again will fail.

2. XDP_FLAGS_SKB_MODE

Driver doesnโ€™t have support for XDP, but the kernel fakes it.
XDP program works, but thereโ€™s no real performance benefit because packets are handed to kernel stack anyways which then emulates XDP โ€“ this is usually supported with generic network drivers used in home computers, laptops, and virtualized HW.

3. XDP_FLAGS_DRV_MODE

A driver has XDP support and can hand then to XDP without kernel stack interaction โ€“ Few drivers can support it and those are usually for enterprise HW.

4. XDP_FLAGS_HW_MODE

XDP can be loaded and executed directly on the NIC โ€“ just a handful of NICs can do that.

For example:

python
b.attach_xdp(dev="ens1", fn=b.load_func("do_xdp", BPF.XDP))

This will instrument the network device ens1 , which will then run our BPF defined do_xdp() function each time it receives packets.

Don't forget to call b.remove_xdp("ens1") at the end!

Examples in situ:
search /examples,
search /tools

10. attach_func()

Syntax: BPF.attach_func(fn, attachable_fd, attach_type [, flags])

Attaches a BPF function of the specified type to a particular attachable_fd. if the attach_type is BPF_FLOW_DISSECTOR, the function is expected to attach to current net namespace and attachable_fd must be 0.

For example:

python
b.attach_func(fn, cgroup_fd, BPFAttachType.CGROUP_SOCK_OPS)
b.attach_func(fn, map_fd, BPFAttachType.SK_MSG_VERDICT)

Note. When attached to "global" hooks (xdp, tc, lwt, cgroup). If the "BPF function" is no longer needed after the program terminates, be sure to call detach_func when the program exits.

Examples in situ:

search /examples,

11. detach_func()

Syntax: BPF.detach_func(fn, attachable_fd, attach_type)

Detaches a BPF function of the specified type.

For example:

python
b.detach_func(fn, cgroup_fd, BPFAttachType.CGROUP_SOCK_OPS)
b.detach_func(fn, map_fd, BPFAttachType.SK_MSG_VERDICT)

Examples in situ:

search /examples,

12. detach_kprobe()

Syntax: BPF.detach_kprobe(event="event", fn_name="name")

Detach a kprobe handler function of the specified event.

For example:

python
b.detach_kprobe(event="__page_cache_alloc", fn_name="trace_func_entry")

13. detach_kretprobe()

Syntax: BPF.detach_kretprobe(event="event", fn_name="name")

Detach a kretprobe handler function of the specified event.

For example:

python
b.detach_kretprobe(event="__page_cache_alloc", fn_name="trace_func_return")

Debug Output

1. trace_print()

Syntax: BPF.trace_print(fmt="fields")

This method continually reads the globally shared /sys/kernel/debug/tracing/trace_pipe file and prints its contents. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.

Arguments:

  • fmt: optional, and can contain a field formatting string. It defaults to None.

Examples:

python
# print trace_pipe output as-is:
b.trace_print()

# print PID and message:
b.trace_print(fmt="{1} {5}")

Examples in situ:
search /examples,
search /tools

2. trace_fields()

Syntax: BPF.trace_fields(nonblocking=False)

This method reads one line from the globally shared /sys/kernel/debug/tracing/trace_pipe file and returns it as fields. This file can be written to via BPF and the bpf_trace_printk() function, however, that method has limitations, including a lack of concurrent tracing support. The BPF_PERF_OUTPUT mechanism, covered earlier, is preferred.

Arguments:

  • nonblocking: optional, defaults to False. When set to True, the program will not block waiting for input.

Examples:

python
while 1:
    try:
        (task, pid, cpu, flags, ts, msg) = b.trace_fields()
    except ValueError:
        continue
    [...]

Examples in situ:
search /examples,
search /tools

Output APIs

Normal output from a BPF program is either:

  • per-event: using PERF_EVENT_OUTPUT, open_perf_buffer(), and perf_buffer_poll().
  • map summary: using items(), or print_log2_hist(), covered in the Maps section.

1. perf_buffer_poll()

Syntax: BPF.perf_buffer_poll(timeout=T)

This polls from all open perf ring buffers, calling the callback function that was provided when calling open_perf_buffer for each entry.

The timeout parameter is optional and measured in milliseconds. In its absence, polling continues indefinitely.

Example:

python
# loop with callback to print_event
b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit();

Examples in situ:
code,
search /examples,
search /tools

2. ring_buffer_poll()

Syntax: BPF.ring_buffer_poll(timeout=T)

This polls from all open ringbuf ring buffers, calling the callback function that was provided when calling open_ring_buffer for each entry.

The timeout parameter is optional and measured in milliseconds. In its absence, polling continues until
there is no more data or the callback returns a negative value.

Example:

python
# loop with callback to print_event
b["events"].open_ring_buffer(print_event)
while 1:
    try:
        b.ring_buffer_poll(30)
    except KeyboardInterrupt:
        exit();

Examples in situ:
search /examples,

3. ring_buffer_consume()

Syntax: BPF.ring_buffer_consume()

This consumes from all open ringbuf ring buffers, calling the callback function that was provided when calling open_ring_buffer for each entry.

Unlike ring_buffer_poll, this method does not poll for data before attempting to consume.
This reduces latency at the expense of higher CPU consumption. If you are unsure which to use,
use ring_buffer_poll.

Example:

python
# loop with callback to print_event
b["events"].open_ring_buffer(print_event)
while 1:
    try:
        b.ring_buffer_consume()
    except KeyboardInterrupt:
        exit();

Examples in situ:
search /examples,

Map APIs

Maps are BPF data stores, and are used in bcc to implement a table, and then higher level objects on top of tables, including hashes and histograms.

1. get_table()

Syntax: BPF.get_table(name)

Returns a table object. This is no longer used, as tables can now be read as items from BPF. Eg: BPF[name].

Examples:

python
counts = b.get_table("counts")

counts = b["counts"]

These are equivalent.

2. open_perf_buffer()

Syntax: table.open_perf_buffer(callback, page_cnt=N, lost_cb=None)

This operates on a table as defined in BPF as BPF_PERF_OUTPUT(), and associates the callback Python function callback to be called when data is available in the perf ring buffer. This is part of the recommended mechanism for transferring per-event data from kernel to user space. The size of the perf ring buffer can be specified via the page_cnt parameter, which must be a power of two number of pages and defaults to 8. If the callback is not processing data fast enough, some submitted data may be lost. lost_cb will be called to log / monitor the lost count. If lost_cb is the default None value, it will just print a line of message to stderr.

Example:

python
# process event
def print_event(cpu, data, size):
    event = ct.cast(data, ct.POINTER(Data)).contents
    [...]

# loop with callback to print_event
b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

Note that the data structure transferred will need to be declared in C in the BPF program. For example:

c
// define output data structure in C
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(events);
[...]

In Python, you can either let bcc generate the data structure from C declaration automatically (recommended):

python
def print_event(cpu, data, size):
    event = b["events"].event(data)
[...]

or define it manually:

python
# define output data structure in Python
TASK_COMM_LEN = 16    # linux/sched.h
class Data(ct.Structure):
    _fields_ = [("pid", ct.c_ulonglong),
                ("ts", ct.c_ulonglong),
                ("comm", ct.c_char * TASK_COMM_LEN)]

def print_event(cpu, data, size):
    event = ct.cast(data, ct.POINTER(Data)).contents
[...]

Examples in situ:
code,
search /examples,
search /tools

3. items()

Syntax: table.items()

Returns an array of the keys in a table. This can be used with BPF_HASH maps to fetch, and iterate, over the keys.

Example:

python
# print output
print("%10s %s" % ("COUNT", "STRING"))
counts = b.get_table("counts")
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
    print("%10d \"%s\"" % (v.value, k.c.encode('string-escape')))

This example also uses the sorted() method to sort by value.

Examples in situ:
search /examples,
search /tools

4. values()

Syntax: table.values()

Returns an array of the values in a table.

5. clear()

Syntax: table.clear()

Clears the table: deletes all entries.

Example:

python
# print map summary every second:
while True:
    time.sleep(1)
    print("%-8s\n" % time.strftime("%H:%M:%S"), end="")
    dist.print_log2_hist(sym + " return:")
    dist.clear()

Examples in situ:
search /examples,
search /tools

6. items_lookup_and_delete_batch()

Syntax: table.items_lookup_and_delete_batch()

Returns an array of the keys in a table with a single call to BPF syscall. This can be used with BPF_HASH maps to fetch, and iterate, over the keys. It also clears the table: deletes all entries.
You should rather use table.items_lookup_and_delete_batch() than table.items() followed by table.clear(). It requires kernel v5.6.

Example:

python
# print call rate per second:
print("%9s-%9s-%8s-%9s" % ("PID", "COMM", "fname", "counter"))
while True:
    for k, v in sorted(b['map'].items_lookup_and_delete_batch(), key=lambda kv: (kv[0]).pid):
        print("%9s-%9s-%8s-%9d" % (k.pid, k.comm, k.fname, v.counter))
    sleep(1)

7. items_lookup_batch()

Syntax: table.items_lookup_batch()

Returns an array of the keys in a table with a single call to BPF syscall. This can be used with BPF_HASH maps to fetch, and iterate, over the keys.
You should rather use table.items_lookup_batch() than table.items(). It requires kernel v5.6.

Example:

python
# print current value of map:
print("%9s-%9s-%8s-%9s" % ("PID", "COMM", "fname", "counter"))
while True:
    for k, v in sorted(b['map'].items_lookup_batch(), key=lambda kv: (kv[0]).pid):
        print("%9s-%9s-%8s-%9d" % (k.pid, k.comm, k.fname, v.counter))

8. items_delete_batch()

Syntax: table.items_delete_batch(keys)

It clears all entries of a BPF_HASH map when keys is None. It is more efficient than table.clear() since it generates only one system call. You can delete a subset of a map by giving an array of keys as parameter. Those keys and their associated values will be deleted. It requires kernel v5.6.

Arguments:

  • keys is optional and by default is None.

9. items_update_batch()

Syntax: table.items_update_batch(keys, values)

Update all the provided keys with new values. The two arguments must be the same length and within the map limits (between 1 and the maximum entries). It requires kernel v5.6.

Arguments:

  • keys is the list of keys to be updated
  • values is the list containing the new values.

10. print_log2_hist()

Syntax: table.print_log2_hist(val_type="value", section_header="Bucket ptr", section_print_fn=None)

Prints a table as a log2 histogram in ASCII. The table must be stored as log2, which can be done using the BPF function bpf_log2l().

Arguments:

  • val_type: optional, column header.
  • section_header: if the histogram has a secondary key, multiple tables will print and section_header can be used as a header description for each.
  • section_print_fn: if section_print_fn is not None, it will be passed the bucket value.

Example:

python
b = BPF(text="""
BPF_HISTOGRAM(dist);

int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
	dist.increment(bpf_log2l(req->__data_len / 1024));
	return 0;
}
""")
[...]

b["dist"].print_log2_hist("kbytes")

Output:

text
kbytes          : count     distribution
       0 -> 1        : 3        |                                      |
       2 -> 3        : 0        |                                      |
       4 -> 7        : 211      |**********                            |
       8 -> 15       : 0        |                                      |
      16 -> 31       : 0        |                                      |
      32 -> 63       : 0        |                                      |
      64 -> 127      : 1        |                                      |
     128 -> 255      : 800      |**************************************|

This output shows a multi-modal distribution, with the largest mode of 128->255 kbytes and a count of 800.

This is an efficient way to summarize data, as the summarization is performed in-kernel, and only the count column is passed to user space.

Examples in situ:
search /examples,
search /tools

11. print_linear_hist()

Syntax: table.print_linear_hist(val_type="value", section_header="Bucket ptr", section_print_fn=None)

Prints a table as a linear histogram in ASCII. This is intended to visualize small integer ranges, eg, 0 to 100.

Arguments:

  • val_type: optional, column header.
  • section_header: if the histogram has a secondary key, multiple tables will print and section_header can be used as a header description for each.
  • section_print_fn: if section_print_fn is not None, it will be passed the bucket value.

Example:

python
b = BPF(text="""
BPF_HISTOGRAM(dist);

int kprobe__blk_account_io_done(struct pt_regs *ctx, struct request *req)
{
	dist.increment(req->__data_len / 1024);
	return 0;
}
""")
[...]

b["dist"].print_linear_hist("kbytes")

Output:

text
kbytes        : count     distribution
        0          : 3        |******                                  |
        1          : 0        |                                        |
        2          : 0        |                                        |
        3          : 0        |                                        |
        4          : 19       |****************************************|
        5          : 0        |                                        |
        6          : 0        |                                        |
        7          : 0        |                                        |
        8          : 4        |********                                |
        9          : 0        |                                        |
        10         : 0        |                                        |
        11         : 0        |                                        |
        12         : 0        |                                        |
        13         : 0        |                                        |
        14         : 0        |                                        |
        15         : 0        |                                        |
        16         : 2        |****                                    |
[...]

This is an efficient way to summarize data, as the summarization is performed in-kernel, and only the values in the count column are passed to user space.

Examples in situ:
search /examples,
search /tools

12. open_ring_buffer()

Syntax: table.open_ring_buffer(callback, ctx=None)

This operates on a table as defined in BPF as BPF_RINGBUF_OUTPUT(), and associates the callback Python function callback to be called when data is available in the ringbuf ring buffer. This is part of the new (Linux 5.8+) recommended mechanism for transferring per-event data from kernel to user space. Unlike perf buffers, ringbuf sizes are specified within the BPF program, as part of the BPF_RINGBUF_OUTPUT macro. If the callback is not processing data fast enough, some submitted data may be lost. In this case, the events should be polled more frequently and/or the size of the ring buffer should be increased.

Example:

python
# process event
def print_event(ctx, data, size):
    event = ct.cast(data, ct.POINTER(Data)).contents
    [...]

# loop with callback to print_event
b["events"].open_ring_buffer(print_event)
while 1:
    try:
        b.ring_buffer_poll()
    except KeyboardInterrupt:
        exit()

Note that the data structure transferred will need to be declared in C in the BPF program. For example:

c
// define output data structure in C
struct data_t {
    u32 pid;
    u64 ts;
    char comm[TASK_COMM_LEN];
};
BPF_RINGBUF_OUTPUT(events, 8);
[...]

In Python, you can either let bcc generate the data structure from C declaration automatically (recommended):

python
def print_event(ctx, data, size):
    event = b["events"].event(data)
[...]

or define it manually:

python
# define output data structure in Python
TASK_COMM_LEN = 16    # linux/sched.h
class Data(ct.Structure):
    _fields_ = [("pid", ct.c_ulonglong),
                ("ts", ct.c_ulonglong),
                ("comm", ct.c_char * TASK_COMM_LEN)]

def print_event(ctx, data, size):
    event = ct.cast(data, ct.POINTER(Data)).contents
[...]

Examples in situ:
search /examples,

13. push()

Syntax: table.push(leaf, flags=0)

Push an element onto a Stack or Queue table. Raises an exception if the operation does not succeed.
Passing QueueStack.BPF_EXIST as a flag causes the Queue or Stack to discard the oldest element if it is full.

Examples in situ:
search /tests,

14. pop()

Syntax: leaf = table.pop()

Pop an element from a Stack or Queue table. Unlike peek(), pop()
removes the element from the table before returning it.
Raises a KeyError exception if the operation does not succeed.

Examples in situ:
search /tests,

15. peek()

Syntax: leaf = table.peek()

Peek the element at the head of a Stack or Queue table. Unlike pop(), peek()
does not remove the element from the table. Raises an exception if the operation does not succeed.

Examples in situ:
search /tests,

Helpers

Some helper methods provided by bcc. Note that since we're in Python, we can import any Python library and their methods, including, for example, the libraries: argparse, collections, ctypes, datetime, re, socket, struct, subprocess, sys, and time.

1. ksym()

Syntax: BPF.ksym(addr)

Translate a kernel memory address into a kernel function name, which is returned.

Example:

python
print("kernel function: " + b.ksym(addr))

Examples in situ:
search /examples,
search /tools

2. ksymname()

Syntax: BPF.ksymname(name)

Translate a kernel name into an address. This is the reverse of ksym. Returns -1 when the function name is unknown.

Example:

python
print("kernel address: %x" % b.ksymname("vfs_read"))

Examples in situ:
search /examples,
search /tools

3. sym()

Syntax: BPF.sym(addr, pid, show_module=False, show_offset=False)

Translate a memory address into a function name for a pid, which is returned. A pid of less than zero will access the kernel symbol cache. The show_module and show_offset parameters control whether the module in which the symbol lies should be displayed, and whether the instruction offset from the beginning of the symbol should be displayed. These extra parameters default to False.

Example:

python
print("function: " + b.sym(addr, pid))

Examples in situ:
search /examples,
search /tools

4. num_open_kprobes()

Syntax: BPF.num_open_kprobes()

Returns the number of open k[ret]probes. Can be useful for scenarios where event_re is used while attaching and detaching probes. Excludes perf_events readers.

Example:

python
b.attach_kprobe(event_re=pattern, fn_name="trace_count")
matched = b.num_open_kprobes()
if matched == 0:
    print("0 functions matched by \"%s\". Exiting." % args.pattern)
    exit()

Examples in situ:
search /examples,
search /tools

5. get_syscall_fnname()

Syntax: BPF.get_syscall_fnname(name : str)

Return the corresponding kernel function name of the syscall. This helper function will try different prefixes and use the right one to concatenate with the syscall name. Note that the return value may vary in different versions of linux kernel and sometimes it will causing trouble. (see #2590)

Example:

python
print("The function name of %s in kernel is %s" % ("clone", b.get_syscall_fnname("clone")))
# sys_clone or __x64_sys_clone or ...

Examples in situ:
search /examples,
search /tools

BPF Errors

See the "Understanding eBPF verifier messages" section in the kernel source under Documentation/networking/filter.txt.

1. Invalid mem access

This can be due to trying to read memory directly, instead of operating on memory on the BPF stack. All kernel memory reads must be passed via bpf_probe_read_kernel() to copy kernel memory into the BPF stack, which can be automatic by the bcc rewriter in some cases of simple dereferencing. bpf_probe_read_kernel() does all the required checks.

Example:

text
bpf: Permission denied
0: (bf) r6 = r1
1: (79) r7 = *(u64 *)(r6 +80)
2: (85) call 14
3: (bf) r8 = r0
[...]
23: (69) r1 = *(u16 *)(r7 +16)
R7 invalid mem access 'inv'

Traceback (most recent call last):
  File "./tcpaccept", line 179, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 172, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 612, in _trace_autoload
    fn = self.load_func(func_name, BPF.KPROBE)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 212, in load_func
    raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kretprobe__inet_csk_accept

2. Cannot call GPL only function from proprietary program

This error happens when a GPL-only helper is called from a non-GPL BPF program. To fix this error, do not use GPL-only helpers from a proprietary BPF program, or relicense the BPF program under a GPL-compatible license. Check which BPF helpers are GPL-only, and what licenses are considered GPL-compatible.

Example calling bpf_get_stackid(), a GPL-only BPF helper, from a proprietary program (#define BPF_LICENSE Proprietary):

text
bpf: Failed to load program: Invalid argument
[...]
8: (85) call bpf_get_stackid#27
cannot call GPL only function from proprietary program

Environment Variables

1. Kernel source directory

eBPF program compilation needs kernel sources or kernel headers with headers
compiled. In case your kernel sources are at a non-standard location where BCC
cannot find then, it's possible to provide BCC the absolute path of the location
by setting BCC_KERNEL_SOURCE to it.

2. Kernel version overriding

By default, BCC stores the LINUX_VERSION_CODE in the generated eBPF object
which is then passed along to the kernel when the eBPF program is loaded.
Sometimes this is quite inconvenient especially when the kernel is slightly
updated such as an LTS kernel release. Its extremely unlikely the slight
mismatch would cause any issues with the loaded eBPF program. By setting
BCC_LINUX_VERSION_CODE to the version of the kernel that's running, the check
for verifying the kernel version can be bypassed. This is needed for programs
that use kprobes. This needs to be encoded in the format: (VERSION * 65536) + (PATCHLEVEL * 256) + SUBLEVEL. For example, if the running kernel is 4.9.10,
then can set export BCC_LINUX_VERSION_CODE=264458 to override the kernel
version check successfully.


File: docs/special_filtering.md

Special Filtering

Some tools have special filtering capabitilies, the main use case is to trace
processes running in containers, but those mechanisms are generic and could
be used in other cases as well.

Filtering by cgroups

Some tools have an option to filter by cgroup by referencing a pinned BPF hash
map managed externally.

Examples of commands:

text
# ./opensnoop --cgroupmap /sys/fs/bpf/test01
# ./execsnoop --cgroupmap /sys/fs/bpf/test01
# ./tcpconnect --cgroupmap /sys/fs/bpf/test01
# ./tcpaccept --cgroupmap /sys/fs/bpf/test01
# ./tcptracer --cgroupmap /sys/fs/bpf/test01

The commands above will only display results from processes that belong to one
of the cgroups whose id, returned by bpf_get_current_cgroup_id(), is in the
pinned BPF hash map.

The BPF hash map can be created by:

text
# bpftool map create /sys/fs/bpf/test01 type hash key 8 value 8 entries 128 \
        name cgroupset flags 0

To get a shell in a new cgroup, you can use:

text
# systemd-run --pty --unit test bash

The shell will be running in the cgroup
/sys/fs/cgroup/unified/system.slice/test.service.

The cgroup id can be discovered using the name_to_handle_at() system call. In
the examples/cgroupid, you will find an example of program to get the cgroup
id.

text
# cd examples/cgroupid
# make
# ./cgroupid hex /sys/fs/cgroup/unified/system.slice/test.service

or, using Docker:

text
# cd examples/cgroupid
# docker build -t cgroupid .
# docker run --rm --privileged -v /sys/fs/cgroup:/sys/fs/cgroup \
	cgroupid cgroupid hex /sys/fs/cgroup/unified/system.slice/test.service

This prints the cgroup id as a hexadecimal string in the host endianness such
as 77 16 00 00 01 00 00 00.

text
# FILE=/sys/fs/bpf/test01
# CGROUPID_HEX="77 16 00 00 01 00 00 00"
# bpftool map update pinned $FILE key hex $CGROUPID_HEX value hex 00 00 00 00 00 00 00 00 any

Now that the shell started by systemd-run has its cgroup id in the BPF hash
map, bcc tools will display results from this shell. Cgroups can be added and
removed from the BPF hash map without restarting the bcc tool.

This feature is useful for integrating bcc tools in external projects.

Filtering by mount namespace

The BPF hash map can be created by:

text
# bpftool map create /sys/fs/bpf/mnt_ns_set type hash key 8 value 4 entries 128 \
        name mnt_ns_set flags 0

Execute the execsnoop tool filtering only the mount namespaces
in /sys/fs/bpf/mnt_ns_set:

text
# tools/execsnoop.py --mntnsmap /sys/fs/bpf/mnt_ns_set

Start a terminal in a new mount namespace:

text
# unshare -m bash

Update the hash map with the mount namespace ID of the terminal above:

text
FILE=/sys/fs/bpf/mnt_ns_set
if [ $(printf '\1' | od -dAn) -eq 1 ]; then
 HOST_ENDIAN_CMD=tac
else
  HOST_ENDIAN_CMD=cat
fi

NS_ID_HEX="$(printf '%016x' $(stat -Lc '%i' /proc/self/ns/mnt) | sed 's/.\{2\}/&\n/g' | $HOST_ENDIAN_CMD)"
bpftool map update pinned $FILE key hex $NS_ID_HEX value hex 00 00 00 00 any

Execute a command in this terminal:

text
# ping kinvolk.io

You'll see how on the execsnoop terminal you started above the call is logged:

text
# tools/execsnoop.py --mntnsmap /sys/fs/bpf/mnt_ns_set
[sudo] password for mvb:
PCOMM            PID    PPID   RET ARGS
ping             8096   7970     0 /bin/ping kinvolk.io

File: docs/tutorial.md

bcc Tutorial

This tutorial covers how to use bcc tools to quickly solve performance, troubleshooting, and networking issues. If you want to develop new bcc tools, see tutorial_bcc_python_developer.md for that tutorial.

It is assumed for this tutorial that bcc is already installed, and you can run tools like execsnoop successfully. See INSTALL.md. This uses enhancements added to the Linux 4.x series.

Observability

Some quick wins.

0. Before bcc

Before using bcc, you should start with the Linux basics. One reference is the Linux Performance Analysis in 60,000 Milliseconds post, which covers these commands:

  1. uptime
  2. dmesg | tail
  3. vmstat 1
  4. mpstat -P ALL 1
  5. pidstat 1
  6. iostat -xz 1
  7. free -m
  8. sar -n DEV 1
  9. sar -n TCP,ETCP 1
  10. top

1. General Performance

Here is a generic checklist for performance investigations with bcc, first as a list, then in detail:

  1. execsnoop
  2. opensnoop
  3. ext4slower (or btrfs*, xfs*, zfs*)
  4. biolatency
  5. biosnoop
  6. cachestat
  7. tcpconnect
  8. tcpaccept
  9. tcpretrans
  10. runqlat
  11. profile

These tools may be installed on your system under /usr/share/bcc/tools, or you can run them from the bcc github repo under /tools where they have a .py extension. Browse the 50+ tools available for more analysis options.

1.1 execsnoop

text
# ./execsnoop
PCOMM            PID    RET ARGS
supervise        9660     0 ./run
supervise        9661     0 ./run
mkdir            9662     0 /bin/mkdir -p ./main
run              9663     0 ./run
[...]

execsnoop prints one line of output for each new process. Check for short-lived processes. These can consume CPU resources, but not show up in most monitoring tools that periodically take snapshots of which processes are running.

It works by tracing exec(), not the fork(), so it will catch many types of new processes but not all (eg, it won't see an application launching working processes, that doesn't exec() anything else).

More examples.

1.2. opensnoop

text
# ./opensnoop
PID    COMM               FD ERR PATH
1565   redis-server        5   0 /proc/1565/stat
1565   redis-server        5   0 /proc/1565/stat
1565   redis-server        5   0 /proc/1565/stat
1603   snmpd               9   0 /proc/net/dev
1603   snmpd              11   0 /proc/net/if_inet6
1603   snmpd              -1   2 /sys/class/net/eth0/device/vendor
1603   snmpd              11   0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1603   snmpd              11   0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms
1603   snmpd              11   0 /proc/sys/net/ipv6/conf/eth0/forwarding
[...]

opensnoop prints one line of output for each open() syscall, including details.

Files that are opened can tell you a lot about how applications work: identifying their data files, config files, and log files. Sometimes applications can misbehave, and perform poorly, when they are constantly attempting to read files that do not exist. opensnoop gives you a quick look.

More examples.

1.3. ext4slower (or btrfs\*, xfs\*, zfs\*)

text
# ./ext4slower
Tracing ext4 operations slower than 10 ms
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
06:35:01 cron           16464  R 1249    0          16.05 common-auth
06:35:01 cron           16463  R 1249    0          16.04 common-auth
06:35:01 cron           16465  R 1249    0          16.03 common-auth
06:35:01 cron           16465  R 4096    0          10.62 login.defs
06:35:01 cron           16464  R 4096    0          10.61 login.defs

ext4slower traces the ext4 file system and times common operations, and then only prints those that exceed a threshold.

This is great for identifying or exonerating one type of performance issue: show individually slow disk i/O via the file system. Disks process I/O asynchronously, and it can be difficult to associate latency at that layer with the latency applications experience. Tracing higher up in the kernel stack, at the VFS -> file system interface, will more closely match what an application suffers. Use this tool to identify if file system latency exceeds a given threshold.

Similar tools exist in bcc for other file systems: btrfsslower, xfsslower, and zfsslower. There is also fileslower, which works at the VFS layer and traces everything (although at some higher overhead).

More examples.

1.4. biolatency

text
# ./biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
     usecs           : count     distribution
       0 -> 1        : 0        |                                      |
       2 -> 3        : 0        |                                      |
       4 -> 7        : 0        |                                      |
       8 -> 15       : 0        |                                      |
      16 -> 31       : 0        |                                      |
      32 -> 63       : 0        |                                      |
      64 -> 127      : 1        |                                      |
     128 -> 255      : 12       |********                              |
     256 -> 511      : 15       |**********                            |
     512 -> 1023     : 43       |*******************************       |
    1024 -> 2047     : 52       |**************************************|
    2048 -> 4095     : 47       |**********************************    |
    4096 -> 8191     : 52       |**************************************|
    8192 -> 16383    : 36       |**************************            |
   16384 -> 32767    : 15       |**********                            |
   32768 -> 65535    : 2        |*                                     |
   65536 -> 131071   : 2        |*                                     |

biolatency traces disk I/O latency (time from device issue to completion), and when the tool ends (Ctrl-C, or a given interval), it prints a histogram summary of the latency.

This is great for understanding disk I/O latency beyond the average times given by tools like iostat. I/O latency outliers will be visible at the end of the distribution, as well as multi-mode distributions.

More examples.

1.5. biosnoop

text
# ./biosnoop
TIME(s)        COMM           PID    DISK    T  SECTOR    BYTES   LAT(ms)
0.000004001    supervise      1950   xvda1   W  13092560  4096       0.74
0.000178002    supervise      1950   xvda1   W  13092432  4096       0.61
0.001469001    supervise      1956   xvda1   W  13092440  4096       1.24
0.001588002    supervise      1956   xvda1   W  13115128  4096       1.09
1.022346001    supervise      1950   xvda1   W  13115272  4096       0.98
1.022568002    supervise      1950   xvda1   W  13188496  4096       0.93
[...]

biosnoop prints a line of output for each disk I/O, with details including latency (time from device issue to completion).

This allows you to examine disk I/O in more detail, and look for time-ordered patterns (eg, reads queueing behind writes). Note that the output will be verbose if your system performs disk I/O at a high rate.

More examples.

1.6. cachestat

text
# ./cachestat
    HITS   MISSES  DIRTIES  READ_HIT% WRITE_HIT%   BUFFERS_MB  CACHED_MB
    1074       44       13      94.9%       2.9%            1        223
    2195      170        8      92.5%       6.8%            1        143
     182       53       56      53.6%       1.3%            1        143
   62480    40960    20480      40.6%      19.8%            1        223
       7        2        5      22.2%      22.2%            1        223
     348        0        0     100.0%       0.0%            1        223
[...]

cachestat prints a one line summary every second (or every custom interval) showing statistics from the file system cache.

Use this to identify a low cache hit ratio, and a high rate of misses: which gives one lead for performance tuning.

More examples.

1.7. tcpconnect

text
# ./tcpconnect
PID    COMM         IP SADDR            DADDR            DPORT
1479   telnet       4  127.0.0.1        127.0.0.1        23
1469   curl         4  10.201.219.236   54.245.105.25    80
1469   curl         4  10.201.219.236   54.67.101.145    80
1991   telnet       6  ::1              ::1              23
2015   ssh          6  fe80::2000:bff:fe82:3ac fe80::2000:bff:fe82:3ac 22
[...]

tcpconnect prints one line of output for every active TCP connection (eg, via connect()), with details including source and destination addresses.

Look for unexpected connections that may point to inefficiencies in application configuration, or an intruder.

More examples.

1.8. tcpaccept

text
# ./tcpaccept
PID    COMM         IP RADDR            LADDR            LPORT
907    sshd         4  192.168.56.1     192.168.56.102   22
907    sshd         4  127.0.0.1        127.0.0.1        22
5389   perl         6  1234:ab12:2040:5020:2299:0:5:0 1234:ab12:2040:5020:2299:0:5:0 7001
[...]

tcpaccept prints one line of output for every passive TCP connection (eg, via accept()), with details including source and destination addresses.

Look for unexpected connections that may point to inefficiencies in application configuration, or an intruder.

More examples.

1.9. tcpretrans

text
# ./tcpretrans
TIME     PID    IP LADDR:LPORT          T> RADDR:RPORT          STATE
01:55:05 0      4  10.153.223.157:22    R> 69.53.245.40:34619   ESTABLISHED
01:55:05 0      4  10.153.223.157:22    R> 69.53.245.40:34619   ESTABLISHED
01:55:17 0      4  10.153.223.157:22    R> 69.53.245.40:22957   ESTABLISHED
[...]

tcprerans prints one line of output for every TCP retransmit packet, with details including source and destination addresses, and kernel state of the TCP connection.

TCP retransmissions cause latency and throughput issues. For ESTABLISHED retransmits, look for patterns with networks. For SYN_SENT, this may point to target kernel CPU saturation and kernel packet drops.

More examples.

1.10. runqlat

text
# ./runqlat
Tracing run queue latency... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 233      |***********                             |
         2 -> 3          : 742      |************************************    |
         4 -> 7          : 203      |**********                              |
         8 -> 15         : 173      |********                                |
        16 -> 31         : 24       |*                                       |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 30       |*                                       |
       128 -> 255        : 6        |                                        |
       256 -> 511        : 3        |                                        |
       512 -> 1023       : 5        |                                        |
      1024 -> 2047       : 27       |*                                       |
      2048 -> 4095       : 30       |*                                       |
      4096 -> 8191       : 20       |                                        |
      8192 -> 16383      : 29       |*                                       |
     16384 -> 32767      : 809      |****************************************|
     32768 -> 65535      : 64       |***                                     |

runqlat times how long threads were waiting on the CPU run queues, and prints this as a histogram.

This can help quantify time lost waiting for a turn on CPU, during periods of CPU saturation.

More examples.

1.11. profile

text
# ./profile
Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end.
^C
    00007f31d76c3251 [unknown]
    47a2c1e752bf47f7 [unknown]
    -                sign-file (8877)
        1

    ffffffff813d0af8 __clear_user
    ffffffff813d5277 iov_iter_zero
    ffffffff814ec5f2 read_iter_zero
    ffffffff8120be9d __vfs_read
    ffffffff8120c385 vfs_read
    ffffffff8120d786 sys_read
    ffffffff817cc076 entry_SYSCALL_64_fastpath
    00007fc5652ad9b0 read
    -                dd (25036)
        4

    0000000000400542 func_a
    0000000000400598 main
    00007f12a133e830 __libc_start_main
    083e258d4c544155 [unknown]
    -                func_ab (13549)
        5

[...]

    ffffffff8105eb66 native_safe_halt
    ffffffff8103659e default_idle
    ffffffff81036d1f arch_cpu_idle
    ffffffff810bba5a default_idle_call
    ffffffff810bbd07 cpu_startup_entry
    ffffffff8104df55 start_secondary
    -                swapper/1 (0)
        75

profile is a CPU profiler, which takes samples of stack traces at timed intervals, and prints a summary of unique stack traces and a count of their occurrence.

Use this tool to understand the code paths that are consuming CPU resources.

More examples.

2. Observability with Generic Tools

In addition to the above tools for performance tuning, below is a checklist for bcc generic tools, first as a list, and in detail:

  1. trace
  2. argdist
  3. funccount

These generic tools may be useful to provide visibility to solve your specific problems.

2.1. trace

Example 1

Suppose you want to track file ownership change. There are three syscalls, chown, fchown and lchown which users can use to change file ownership. The corresponding syscall entry is SyS_[f|l]chown. The following command can be used to print out syscall parameters and the calling process user id. You can use id command to find the uid of a particular user.

text
$ trace.py \
  'p::SyS_chown "file = %s, to_uid = %d, to_gid = %d, from_uid = %d", arg1, arg2, arg3, $uid' \
  'p::SyS_fchown "fd = %d, to_uid = %d, to_gid = %d, from_uid = %d", arg1, arg2, arg3, $uid' \
  'p::SyS_lchown "file = %s, to_uid = %d, to_gid = %d, from_uid = %d", arg1, arg2, arg3, $uid'
PID    TID    COMM         FUNC             -
1269255 1269255 python3.6    SyS_lchown       file = /tmp/dotsync-usisgezu/tmp, to_uid = 128203, to_gid = 100, from_uid = 128203
1269441 1269441 zstd         SyS_chown        file = /tmp/dotsync-vic7ygj0/dotsync-package.zst, to_uid = 128203, to_gid = 100, from_uid = 128203
1269255 1269255 python3.6    SyS_lchown       file = /tmp/dotsync-a40zd7ev/tmp, to_uid = 128203, to_gid = 100, from_uid = 128203
1269442 1269442 zstd         SyS_chown        file = /tmp/dotsync-gzp413o_/dotsync-package.zst, to_uid = 128203, to_gid = 100, from_uid = 128203
1269255 1269255 python3.6    SyS_lchown       file = /tmp/dotsync-whx4fivm/tmp/.bash_profile, to_uid = 128203, to_gid = 100, from_uid = 128203

Example 2

Suppose you want to count nonvoluntary context switches (nvcsw) in your bpf based performance monitoring tools and you do not know what is the proper method. /proc//status already tells you the number (nonvoluntary_ctxt_switches) for a pid and you can use trace.py to do a quick experiment to verify your method. With kernel source code, the nvcsw is counted at file linux/kernel/sched/core.c function __schedule and under condition

text
!(!preempt && prev->state) // i.e., preempt || !prev->state

The __schedule function is marked as notrace, and the best place to evaluate the above condition seems in sched/sched_switch tracepoint called inside function __schedule and defined in linux/include/trace/events/sched.h. trace.py already has args being the pointer to the tracepoint TP_STRUCT__entry. The above condition in function __schedule can be represented as

text
args->prev_state == TASK_STATE_MAX || args->prev_state == 0

The below command can be used to count the involuntary context switches (per process or per pid) and compare to /proc//status or /proc//task/<task_id>/status for correctness, as in typical cases, involuntary context switches are not very common.

text
$ trace.py -p 1134138 't:sched:sched_switch (args->prev_state == TASK_STATE_MAX || args->prev_state == 0)'
PID    TID    COMM         FUNC
1134138 1134140 contention_test sched_switch
1134138 1134142 contention_test sched_switch
...
$ trace.py -L 1134140 't:sched:sched_switch (args->prev_state == TASK_STATE_MAX || args->prev_state == 0)'
PID    TID    COMM         FUNC
1134138 1134140 contention_test sched_switch
1134138 1134140 contention_test sched_switch
...

Example 3

This example is related to issue 1231 and 1516 where uprobe does not work at all in certain cases. First, you can do a strace as below

text
$ strace trace.py 'r:bash:readline "%s", retval'
...
perf_event_open(0x7ffd968212f0, -1, 0, -1, 0x8 /* PERF_FLAG_??? */) = -1 EIO (Input/output error)
...

The perf_event_open syscall returns -EIO. Digging into kernel uprobe related codes in /kernel/trace and /kernel/events directories to search EIO, the function uprobe_register is the most suspicious. Let us find whether this function is called or not and what is the return value if it is called. In one terminal using the following command to print out the return value of uprobe_register,

text
$ trace.py 'r::uprobe_register "ret = %d", retval'

In another terminal run the same bash uretprobe tracing example, and you should get

text
$ trace.py 'r::uprobe_register "ret = %d", retval'
PID    TID    COMM         FUNC             -
1041401 1041401 python2.7    uprobe_register  ret = -5

The -5 error code is EIO. This confirms that the following code in function uprobe_register is the most suspicious culprit.

text
if (!inode->i_mapping->a_ops->readpage && !shmem_mapping(inode->i_mapping))
        return -EIO;

The shmem_mapping function is defined as

text
bool shmem_mapping(struct address_space *mapping)
{
        return mapping->a_ops == &shmem_aops;
}

To confirm the theory, find what is inode->i_mapping->a_ops with the following command

text
$ trace.py -I 'linux/fs.h' 'p::uprobe_register(struct inode *inode) "a_ops = %llx", inode->i_mapping->a_ops'
PID    TID    COMM         FUNC             -
814288 814288 python2.7    uprobe_register  a_ops = ffffffff81a2adc0
^C$ grep ffffffff81a2adc0 /proc/kallsyms
ffffffff81a2adc0 R empty_aops

The kernel symbol empty_aops does not have readpage defined and hence the above suspicious condition is true. Further examining the kernel source code shows that overlayfs does not provide its own a_ops while some other file systems (e.g., ext4) define their own a_ops (e.g., ext4_da_aops), and ext4_da_aops defines readpage. Hence, uprobe works fine on ext4 while not on overlayfs.

More examples.

2.2. argdist

More examples.

2.3. funccount

More examples.

Networking

To do.