bpf-developer-tutorial Documentation

EBPF Developer Tutorial: Learning eBPF Step by Step with Examples

4,318 stars C C 37 files Β· ~125,797 tokens #bpf#ebpf#examples#libbpf#tutorial#xdp
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy bpf-developer-tutorial on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name bpf-developer-tutorial -p 9901:9901 --restart always eunomia-bpf/bpf-developer-tutorial:latest

Technical Documentation: eunomia-bpf/bpf-developer-tutorial

ℹ️ Provenance: Multi-Tier Ingestion Β· CodeWiki Reference

Overview (`README.md`)

eBPF Developer Tutorial: Learning eBPF Step by Step with Examples


GitHub
Gitee Mirror
δΈ­ζ–‡η‰ˆ

This is a development tutorial for eBPF based on CO-RE (Compile Once, Run Everywhere). It provides practical eBPF development practices from beginner to advanced, including basic concepts, code examples, and real-world applications. Unlike BCC, we use frameworks like libbpf, Cilium, libbpf-rs, and eunomia-bpf for development, with examples in languages such as C, Go, and Rust.

This tutorial does not cover complex concepts and scenario introductions. Its main purpose is to provide examples of eBPF tools (very short, starting with twenty lines of code!) to help eBPF application developers quickly grasp eBPF development methods and techniques. The tutorial content can be found in the directory, with each directory being an independent eBPF tool example.

The tutorial focuses on eBPF examples in observability, networking, security, and more.

δΈ­ζ–‡η‰ˆεœ¨θΏ™ι‡Œ

Table of Contents

Getting Started Examples

This section contains simple eBPF program examples and introductions. It primarily utilizes the eunomia-bpf framework to simplify development and introduces the basic usage and development process of eBPF.

Advanced Documents and Examples

We start to build complete eBPF projects mainly based on libbpf and combine them with various application scenarios for practical use.

In-Depth Topics

This section covers advanced topics related to eBPF, including using eBPF programs on Android, possible attacks and defenses using eBPF programs, and complex tracing. Combining the user-mode and kernel-mode aspects of eBPF can bring great power (as well as security risks).

GPU:

Scheduler:

Networking:

Tracing:

Security:

Features:

Other:

  • lesson 49-hid Fixing Broken HID Devices Without Kernel Patches
  • cgroup cgroup-based Policy Control

Android:

Continuously updating...

Why write this tutorial?

In the process of learning eBPF, we have been inspired and helped by the bcc Python developer tutorial. However, from the current perspective, using libbpf to develop eBPF applications is a relatively better choice.

This project is mainly based on libbpf frameworks.

  • We also provide a small tool called GPTtrace, which uses ChatGPT to automatically write eBPF programs and trace Linux systems through natural language descriptions. This tool allows you to interactively learn eBPF programs: GPTtrace
  • Feel free to raise any questions or issues related to eBPF learning, or bugs encountered in practice, in the issue or discussion section of this repository. We will do our best to help you!

Install deps and Compile

GitHub Templates: Easily build eBPF projects and development environments, compile and run eBPF programs online with one click

When starting a new eBPF project, are you confused about how to set up the environment and choose a programming language? Don't worry, we have prepared a series of GitHub templates for you to quickly start a brand new eBPF project. Just click the Use this template button on GitHub to get started.

These starter templates include the following features:

  • A Makefile to build the project with a single command
  • A Dockerfile to automatically create a containerized environment for your eBPF project and publish it to GitHub Packages
  • GitHub Actions to automate the build, test, and release processes
  • All dependencies required for eBPF development

By setting an existing repository as a template, you and others can quickly generate new repositories with the same basic structure, eliminating the need for manual creation and configuration. With GitHub template repositories, developers can focus on the core functionality and logic of their projects without wasting time on the setup and structure. For more information about template repositories, see the official documentation: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository

When you create a new repository using one of the eBPF project templates mentioned above, you can easily set up and launch an online development environment with GitHub Codespaces. Here are the steps to compile and run eBPF programs using GitHub Codespaces:

  1. Click the Code button in your new repository and select the Open with Codespaces option:

  2. GitHub will create a new Codespace for you, which may take a few minutes depending on your network speed and the size of the repository.

  3. Once your Codespace is launched and ready to use, you can open the terminal and navigate to your project directory.

  4. You can follow the instructions in the corresponding repository to compile and run eBPF programs:

With Codespaces, you can easily create, manage, and share cloud-based development environments, speeding up and making your development process more reliable. You can develop with Codespaces anywhere, on any device, just need a computer with a web browser. Additionally, GitHub Codespaces supports pre-configured environments, customized development containers, and customizable development experiences to meet your development needs.

After writing code in a codespace and making a commit, GitHub Actions will compile and automatically publish the container image. Then, you can use Docker to run this eBPF program anywhere with just one command, for example:

console
$ sudo docker run --rm -it --privileged ghcr.io/eunomia-bpf/libbpf-rs-template:latest
[sudo] password for xxx:
Tracing run queue latency higher than 10000 us
TIME     COMM             TID     LAT(us)
12:09:19 systemd-udevd    30786   18300
12:09:19 systemd-udevd    30796   21941
12:09:19 systemd-udevd    30793   10323
12:09:19 systemd-udevd    30795   14827
12:09:19 systemd-udevd    30790   17973
12:09:19 systemd-udevd    30793   12328
12:09:19 systemd-udevd    30796   28721

build

The example of local compilation is shown as follows:

shell
git clone https://github.com/eunomia-bpf/bpf-developer-tutorial.git
cd bpf-developer-tutorial
git submodule update --init --recursive # Synchronize submodule
cd src/24-hide
make

LICENSE

MIT

Overview (`src/0-introduce/README.md`)

eBPF Tutorial by Example 0: Introduction to Core Concepts and Tools

This is the first part of a comprehensive development tutorial for eBPF, designed to guide you through practical eBPF development, from beginner to advanced. It covers fundamental concepts, real-world code examples, and applications in modern systems. Rather than focusing on traditional tools like BCC, we will use modern frameworks such as libbpf, Cilium, libbpf-rs, and eunomia-bpf, with examples provided in C, Go, and Rust.

The primary goal of this tutorial is to provide clear and concise examples of eBPF tools (starting with as little as 20 lines of code!) to help developers quickly grasp essential eBPF development techniques. Each example is self-contained and can be found in the directory structure, with every directory representing an independent eBPF tool. You can also visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorial source code.

Introduction to eBPF: Secure and Efficient Kernel Extension

eBPF (extended Berkeley Packet Filter) is a groundbreaking technology that allows developers to run small programs directly in kernel space, safely and efficiently. Unlike traditional approaches that required modifying kernel source code or loading new modules, eBPF made it possible to customize and optimize network behavior dynamically, all without disrupting system operations. This flexibility and efficiency made eBPF a pivotal technology for overcoming the limitations of traditional networking stacks.

What Makes eBPF So Powerful?

  • Direct Kernel Interaction: eBPF programs execute within the kernel, interacting with system-level events such as network packets, system calls, or tracepoints.
  • Safe Execution: eBPF ensures safety through a verifier that checks the logic of the program before it runs, preventing potential kernel crashes or security breaches.
  • Minimal Overhead: eBPF achieves near-native execution speed by employing a Just-In-Time (JIT) compiler, which translates eBPF bytecode into optimized machine code for the specific architecture.

eBPF: Past, Present, and Future

Past: Programmable Networking Transformed

When eBPF was introduced in 2014, it revolutionized how developers approached networking by allowing small, programmable kernel-space applications to handle packet processing in real time. By hooking into key kernel points, eBPF enabled custom logic to be applied whenever a network packet arrived, leading to higher efficiency and flexibility. This allowed organizations to tailor networking behavior without the overhead of custom drivers or kernel modifications, creating an ideal solution for cloud-native and data-center environments.

Present: A Versatile Framework for Modern Computing Needs

eBPF has evolved into a versatile framework that extends beyond its original purpose of networking, now encompassing observability, tracing, security, and even system resource management. eBPF programs can dynamically hook into kernel events, giving developers precise control over system behavior and performance optimization without requiring kernel modifications or reboots. This makes eBPF an essential tool for system administrators and developers who aim to monitor, optimize, and secure their environments.

Here are some key areas where eBPF is widely used today:

  • Networking: eBPF offers real-time, high-speed packet filtering and processing within the kernel, allowing for the creation of custom protocol parsers and network policies without needing new drivers or system restarts. This enables highly efficient network management in cloud and data center environments.

  • Observability: eBPF enables developers to gather detailed insights into system behavior by collecting custom metrics and performing in-kernel data aggregation. By tapping into kernel tracepoints and function calls, eBPF helps identify performance issues and track down elusive bugs.

  • Tracing & Profiling: eBPF provides powerful tracing and profiling capabilities by attaching to kernel functions, tracepoints, and even user-space probes. This allows developers to gain deep insights into system and application behavior, enabling them to optimize performance and resolve complex system issues.

  • Security: eBPF plays a vital role in real-time security monitoring. It enables deep inspection of system calls, network traffic, and other kernel activities, helping to enforce dynamic security policies and detect anomalous behavior, providing an efficient way to safeguard infrastructure.

  • Scheduler Optimization: Since Linux kernel 6.12, eBPF-powered CPU schedulers (sched_ext) are now mainline, allowing custom scheduling policies to be implemented as BPF programs. This enables runtime-customizable scheduling that can optimize for different workloads without kernel modifications, and is already deployed on over 1 million machines in production.

  • HID (Human Interface Device) Driver Enhancements: Developers use eBPF to optimize HID drivers for devices like keyboards, mice, and touchscreens. By adding custom logic for handling input events, eBPF improves responsiveness in latency-sensitive applications.

Organizations across industries have adopted eBPF at scale:

  • Google: Uses eBPF for security auditing, packet processing, real-time performance monitoring, and is actively testing eBPF-based CPU schedulers across its vast infrastructure.
  • Netflix: Leverages eBPF for network traffic analysis, ensuring high availability and performance for streaming services.
  • Android: Every Android phone uses eBPF to optimize network usage, power consumption, and resource allocation, improving performance and battery life across billions of devices worldwide.
  • Meta: Runs eBPF-based schedulers in production (over 1 million machines with scx_layered) alongside its network load balancing (Katran).
  • S&P Global: Utilizes eBPF through Cilium for managing networking across multiple clouds and on-premises systems, ensuring scalability and security.
  • Shopify: Implements eBPF with Falco for intrusion detection, bolstering security on its e-commerce platform.
  • Cloudflare: Uses eBPF for network observability, security monitoring, and performance optimization, protecting millions of websites globally.

eBPF's ability to dynamically adjust system behavior and extend into user space makes it an essential technology for modern computing. Whether it's optimizing network traffic, improving security, or enhancing system performance, eBPF enables developers to address real-time requirements efficiently and safely.

In addition to its kernel-mode runtime, eBPF can also be extended beyond the kernel. For example, bpftime is a user-space eBPF runtime that allows for higher-performance tracing (10x faster than kernel uprobes), performance analysis, and plugin support in user-space applications without requiring manual recompilation or process restarts. eBPF programs can now even be offloaded to GPUs through bpftime integration, enabling dynamic instrumentation of GPU kernels with low overhead. This extension of eBPF beyond kernel space helps improve flexibility and performance across CPU, user-space, and GPU workloads.

Future: The Expanding Potential of eBPF

Looking forward, eBPF is expected to become an even more integral part of operating systems. The focus is on improving its flexibility, modularity, and ease of use, making it accessible for an even broader range of applications. Innovations in memory management, concurrency mechanisms, and better integration with user-space applications are on the horizon. Projects are already underway to compile significant parts of the Linux kernel to the BPF instruction set, potentially revolutionizing how kernel development and analysis are performed.

Advancements such as dynamic stacks, better observability tools for user space (e.g., Fast Uprobes and language-specific stack walkers), and safer program termination mechanisms will continue to strengthen eBPF’s reliability and expand its use cases. Additionally, new tools and libraries will simplify eBPF development, lowering the barrier to entry for both kernel and application developers.

Getting Started with the Tutorial

This tutorial provides practical eBPF development practices, covering topics from beginner to advanced levels. We focus on hands-on examples in areas like observability, networking, and security, using frameworks like libbpf, libbpf-rs, and eunomia-bpf, with examples in C, Go, and Rust.

Who Is This Tutorial For?

This tutorial is designed for developers who want to build custom kernel solutions, system administrators looking to enhance performance and security, and tech enthusiasts exploring cutting-edge kernel technologies. Whether you're debugging production issues or building the next generation of observability tools, this guide will help you get started with eBPF.

What Will You Learn?

You'll learn eBPF fundamentals and how they integrate with the Linux kernel. We'll cover practical skills for writing and deploying eBPF programs, from simple "Hello World" examples to advanced topics like security monitoring, tracing, and performance profiling. By the end, you'll be able to write your own eBPF tools and understand how to apply them to real-world problems.

Prerequisites

Before starting, make sure you have a Linux system with kernel version 4.8 or higher (we recommend 5.15+ or 6.9+ for full feature support, including BPF arena and tokens). You'll need basic C programming knowledge and familiarity with system concepts like processes and system calls. Most examples require root or sudo privileges to run. You should also install development tools like clang, llvm, and libelf-dev on your system.

Table of Contents

  1. Introduction to eBPF
    Basic concepts and the tools you need to get started.

  2. Beginner Examples
    Simple programs such as "Hello World" and basic tracing using kprobe and uprobe.

  3. Observability
    Examples focused on monitoring network traffic, file operations, and process behavior using eBPF.

  4. Networking
    Examples focused on modifying and optimizing network traffic, such as XDP, TC, and socket.

  5. Security
    Programs for hiding process and files, sending signals to kill process, and tracking process events for security.

  6. Advanced Use Cases
    Complex examples involving performance profiling, scheduler optimization, and eBPF in user space (e.g., bpftime).

  7. In-Depth Topics
    Exploring eBPF for Android, using eBPF for network acceleration, and securing systems through syscall modifications.

How to Use eBPF Programming

Writing eBPF programs from scratch can be complex. To simplify this, LLVM introduced the ability to compile high-level language code into eBPF bytecode in 2015. The eBPF community has since built libraries like libbpf to manage these programs. These libraries help load eBPF bytecode into the kernel and perform essential tasks. The Linux kernel source contains numerous eBPF examples in the samples/bpf/ directory.

A typical eBPF program involves two parts: kernel space code (*_kern.c) and user space code (*_user.c). The kernel space code defines the logic, while the user space code manages loading and interacting with the kernel. However, tools like libbpf-bootstrap and the Go eBPF library help simplify this process, allowing for one-time compilation and easier development.

Tools for eBPF Development

  • BCC: A Python-based toolchain that simplifies writing, compiling, and loading eBPF programs. It offers many pre-built tracing tools but has limitations with dependencies and compatibility.
  • eBPF Go Library: A Go library that decouples the process of obtaining eBPF bytecode from the loading and management of eBPF programs.
  • libbpf-bootstrap: A modern scaffold based on libbpf that provides an efficient workflow for writing eBPF programs, offering a simple one-time compilation process for reusable bytecode.
  • eunomia-bpf: A toolchain for writing eBPF programs with only kernel space code. It simplifies the development of eBPF programs by dynamically loading them.

These tools help reduce the complexity of developing eBPF programs, making the process more accessible to developers aiming to optimize system performance, security, and observability.

Some Tips on Learning eBPF Development

This article will not provide a more detailed introduction to the principles of eBPF, but here is a learning plan and reference materials that may be of value:

Introduction to eBPF (5-7h)

  • Google or other search engines: eBPF
  • Ask ChatGPT-like things: What is eBPF?

Recommended:

  • Read the introduction to ebpf: https://ebpf.io/ (30min)
  • Briefly understand the ebpf kernel-related documentation: https://docs.ebpf.io/ (Know where to queries for tech details, 30min)

Answer three questions:

  1. Understand what eBPF is? Why do we need it? Can't we use kernel modules?
  2. What functions does it have? What can it do in the Linux kernel? What are the types of eBPF programs and helpers (not all of them need to be known, but need to know where to find them)?
  3. What can it be used for? For example, in which scenarios can it be used? Networking, security, observability?

Understand how to develop eBPF programs (10-15h)

Understand and try eBPF development frameworks:

Other development frameworks: Go or Rust language, please search and try on your own (0-2h)

Have questions or things you want to know, whether or not they are related to this project, you can start discussing in the discussions of this project.

Answer some questions and try some experiments (2-5h):

  1. How to develop the simplest eBPF program?
  2. How to trace a kernel feature or function with eBPF? There are many ways, provide corresponding code examples; How to write a XDP network function or write a scheduler?
  3. What are the solutions for communication between user mode and kernel mode? How to send information from user mode to kernel mode? How to pass information from kernel mode to user mode? Provide code examples;
  4. Write your own eBPF program to implement a feature;
  5. In the entire lifecycle of an eBPF program, what does it do in user mode and kernel mode?

References

You can also visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorial source code. All content is open source. We will continue to share more content about eBPF development practices to help you better understand and master eBPF technology.".

Overview (`src/1-helloworld/README.md`)

eBPF Tutorial by Example 1: Hello World, Framework and Development

In this blog post, we will delve into the basic framework and development process of eBPF (Extended Berkeley Packet Filter). eBPF is a powerful network and performance analysis tool that runs on the Linux kernel, providing developers with the ability to dynamically load, update, and run user-defined code at kernel runtime. This enables developers to implement efficient, secure kernel-level network monitoring, performance analysis, and troubleshooting functionalities.

This article is the second part of the eBPF Tutorial by Example, where we will focus on how to write a simple eBPF program and demonstrate the entire development process through practical examples. Before reading this tutorial, it is recommended that you first learn the concepts of eBPF by studying the first tutorial.

When developing eBPF programs, there are multiple development frameworks to choose from, such as BCC (BPF Compiler Collection) libbpf, cilium/ebpf, eunomia-bpf, etc. Although these tools have different characteristics, their basic development process is similar. In the following content, we will delve into these processes and use the Hello World program as an example to guide readers in mastering the basic skills of eBPF development.

This tutorial will help you understand the basic structure of eBPF programs, the compilation and loading process, the interaction between user space and kernel space, as well as debugging and optimization techniques. By studying this tutorial, you will master the basic knowledge of eBPF development and lay a solid foundation for further learning and practice.

Preparation of eBPF Development Environment and Basic Development Process

Before starting to write eBPF programs, we need to prepare a suitable development environment and understand the basic development process of eBPF programs. This section will provide a detailed introduction to these subjects.

Installing the necessary software and tools

To develop eBPF programs, you need to install the following software and tools:

  • Linux kernel: Since eBPF is a kernel technology, you need to have a relatively new version of the Linux kernel (minimum version 4.8 and above, suggested version is 5.15+ or 6.2+) to support eBPF functionality.
    • If possible, install a new version of Ubuntu (e.g. 23.10) would be better.
  • LLVM and Clang: These tools are used to compile eBPF programs. Installing the latest version of LLVM and Clang ensures that you get the best eBPF support.

An eBPF program consists of two main parts: the kernel space part and the user space part. The kernel space part contains the actual logic of the eBPF program, while the user space part is responsible for loading, running, and monitoring the kernel space program.

Once you have chosen a suitable development framework, such as BCC (BPF Compiler Collection), libbpf, cilium/ebpf, or eunomia-bpf, you can begin developing the user space and kernel space programs. Taking the BCC tool as an example, we will introduce the basic development process of eBPF programs:

  1. Installing the BCC tool: Depending on your Linux distribution, follow the guidelines in the BCC documentation to install the BCC tool and its dependencies.
  2. Writing an eBPF program (C language): Use the C language to write a simple eBPF program, such as the Hello World program. This program can be executed in kernel space and perform specific tasks, such as counting network packets.
  3. Writing a user space program (Python or C, etc.): Use languages like Python or C to write a user space program that is responsible for loading, running, and interacting with the eBPF program. In this program, you need to use the API provided by BCC to load and manipulate the kernel space eBPF program.
  4. Compiling the eBPF program: Use the BCC tool to compile the eBPF program written in C language into bytecode that can be executed by the kernel. BCC dynamically compiles the eBPF program from source code at runtime.
  5. Loading and running the eBPF program: In the user space program, use the API provided by BCC to load the compiled eBPF program into kernel space and then run it.
  6. Interacting with the eBPF program: The user space program interacts with the eBPF program through the API provided by BCC, implementing data collection, analysis, and display functions. For example, you can use the BCC API to read map data in the eBPF program to obtain network packet statistics.
  7. Unloading the eBPF program: When the eBPF program is no longer needed, the user space program should unload it from the kernel space using the BCC API.
  8. Debugging and optimization: Use tools like bpftool to debug and optimize eBPF programs, improving program performance and stability.

Through the above process, you can develop, compile, run, and debug eBPF programs using the BCC tool. Note that the development process of other frameworks, such as libbpf, cilium/ebpf, and eunomia-bpf, is similar but slightly different. Therefore, when choosing a framework, please refer to the respective official documentation and examples.

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

Download and Install eunomia-bpf Development Tools

You can download and install eunomia-bpf using the following steps:

Download the ecli tool for running eBPF programs:

console
$ wget https://aka.pw/bpf-ecli -O ecli && chmod +x ./ecli
$ ./ecli -h
Usage: ecli [--help] [--version] [--json] [--no-cache] url-and-args

Download the compiler toolchain for compiling eBPF kernel code into config files or WASM modules:

console
$ wget https://github.com/eunomia-bpf/eunomia-bpf/releases/latest/download/ecc && chmod +x ./ecc
$ ./ecc -h
eunomia-bpf compiler
Usage: ecc [OPTIONS] <SOURCE_PATH> [EXPORT_EVENT_HEADER]
....

Note: If you are on the aarch64 platform, please use the ecc-aarch64 and ecli-aarch64.

You can also compile using the docker image:

console
$ docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest # Compile using docker. `pwd` should contain *.bpf.c files and *.h files.
export PATH=PATH:~/.eunomia/bin
Compiling bpf object...
Packing ebpf object and config into /src/package.json...

Hello World - minimal eBPF program

We will start with a simple eBPF program that prints a message in the kernel. We will use the eunomia-bpf compiler toolchain to compile it into a BPF bytecode file, and then load and run the program using the ecli tool. For the sake of the example, we can temporarily disregard the user space program.

c
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#define BPF_NO_GLOBAL_DATA
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

typedef unsigned int u32;
typedef int pid_t;
const pid_t pid_filter = 0;

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("tp/syscalls/sys_enter_write")
int handle_tp(void *ctx)
{
 pid_t pid = bpf_get_current_pid_tgid() >> 32;
 if (pid_filter && pid != pid_filter)
  return 0;
 bpf_printk("BPF triggered sys_enter_write from PID %d.\n", pid);
 return 0;
}

This program defines a handle_tp function and attaches it to the sys_enter_write tracepoint using the SEC macro. This means it gets executed every time the write system call is entered. The function retrieves the process ID of the current process and prints it to the kernel log using bpf_printk.

The SEC("tp/syscalls/sys_enter_write") macro tells the eBPF loader where to attach this function. The format is tp for tracepoint, followed by the subsystem (syscalls), and the specific event name (sys_enter_write). You can find available tracepoints by running sudo ls /sys/kernel/debug/tracing/events/syscalls/ on your system.

The BPF_NO_GLOBAL_DATA macro at the top is for compatibility with older kernels (before 5.2) that don't support global variables in eBPF. If you're running a modern kernel (5.15+), this isn't strictly necessary but doesn't hurt to include for portability.

The bpf_printk() function is your friend for debugging. It outputs to /sys/kernel/debug/tracing/trace_pipe, which is a simple way to see what your eBPF program is doing. However, it has some limitations you should know about. It only accepts up to 3 parameters, the trace_pipe is shared globally across all eBPF programs, and it can impact performance on high-frequency events. For production use, you'll want to use ring buffers or perf event arrays, which we'll cover in later tutorials.

The ctx parameter contains tracepoint-specific data, but since we don't need it for this simple example, we just declare it as void *. In more advanced programs, you would cast this to access the tracepoint's arguments. Finally, eBPF programs must return an integer - returning 0 is standard practice for tracepoints.

To compile and run this program, you can use the ecc tool and ecli command. First, on Ubuntu/Debian, execute the following command:

shell
sudo apt install clang llvm

Compile the program using ecc:

console
$ ./ecc minimal.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

Or compile using a docker image:

shell
docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Then run the compiled program using ecli:

console
$ sudo ./ecli run package.json
Running eBPF program...

After running this program, you can view the output of the eBPF program by checking the /sys/kernel/debug/tracing/trace_pipe file:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe | grep "BPF triggered sys_enter_write"
           <...>-3840345 [010] d... 3220701.101143: bpf_trace_printk: write system call from PID 3840345.
           <...>-3840345 [010] d... 3220701.101143: bpf_trace_printk: write system call from PID 3840345.

Once you stop the ecli process by pressing Ctrl+C, the corresponding output will also stop.

If you don't see any output, the tracing subsystem might not be enabled on your system. This is common on some Linux distributions. You can enable it by running:

console
$ sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/tracing_on'

If you're still not seeing output, make sure the program is actually loaded and running (you should see "Running eBPF program..." in the ecli terminal). Try triggering some write syscalls manually by running echo "test" > /tmp/test.txt in another terminal. You can also check if your eBPF program is loaded by running sudo bpftool prog list.

Basic Framework of eBPF Program

As mentioned above, the basic framework of an eBPF program includes:

  • Including header files: You need to include "vmlinux.h" and header files, among others.
  • Defining a license: You need to define a license, typically using "Dual BSD/GPL".
  • Defining a BPF function: You need to define a BPF function, for example, named handle_tp, which takes void *ctx as a parameter and returns int. This is usually written in the C language.
  • Using BPF helper functions: In the BPF function, you can use BPF helper functions such as bpf_get_current_pid_tgid() and bpf_printk().
  • Return value

Tracepoints

Tracepoints are a kernel static instrumentation technique, technically just trace functions placed in the kernel source code, which are essentially probe points with control conditions inserted into the source code, allowing post-processing with additional processing functions. For example, the most common static tracing method in the kernel is printk, which outputs log messages. For example, there are tracepoints at the start and end of system calls, scheduler events, file system operations, and disk I/O. Tracepoints were first introduced in Linux version 2.6.32 in 2009. Tracepoints are a stable API and their number is limited.

GitHub Templates: Build eBPF Projects and Development Environments Easily

When faced with creating an eBPF project, are you confused about how to set up the environment and choose a programming language? Don't worry, we have prepared a series of GitHub templates to help you quickly start a brand new eBPF project. Just click the Use this template button on GitHub to get started.

These starter templates include the following features:

  • A Makefile for building the project with one command.
  • A Dockerfile for automatically creating a containerized environment for your eBPF project and publishing it to Github Packages.- GitHub Actions, used for automating build, test, and release processes
  • All dependencies required for eBPF development

By setting an existing repository as a template, you and others can quickly generate new repositories with the same underlying structure, eliminating the tedious process of manual creation and configuration. With GitHub template repositories, developers can focus on the core functionality and logic of their projects without wasting time on setup and structure. For more information about template repositories, please refer to the official documentation: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository

Summary

The development and usage process of eBPF programs can be summarized in the following steps:

  • Define the interface and types of eBPF programs: This includes defining the interface functions of eBPF programs, defining and implementing eBPF kernel maps and shared memory (perf events), and defining and using eBPF kernel helper functions.
  • Write the code for eBPF programs: This includes writing the main logic of the eBPF program, implementing read and write operations on eBPF kernel maps, and using eBPF kernel helper functions.
  • Compile the eBPF program: This includes using an eBPF compiler (such as clang) to compile the eBPF program code into eBPF bytecode and generate an executable eBPF kernel module. ecc essentially calls the clang compiler to compile eBPF programs.
  • Load the eBPF program into the kernel: This includes loading the compiled eBPF kernel module into the Linux kernel and attaching the eBPF program to the specified kernel events.
  • Use the eBPF program: This includes monitoring the execution of the eBPF program and exchanging and sharing data using eBPF kernel maps and shared memory.
  • In practical development, there may be additional steps such as configuring compilation and loading parameters, managing eBPF kernel modules and kernel maps, and using other advanced features.

The execution of BPF programs occurs in the kernel space, so special tools and techniques are needed to write, compile, and debug them.

You can also visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials, all of which are open-source. We will continue to share more about eBPF development practices to help you better understand and master eBPF technology.

Overview (`src/10-hardirqs/README.md`)

eBPF Tutorial by Example 10: Capturing Interrupts with hardirqs or softirqs

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code at runtime in the kernel.

This article is the tenth part of the eBPF Tutorial by Example, focusing on capturing interrupt events using hardirqs or softirqs in eBPF.
hardirqs and softirqs are two different types of interrupt handlers in the Linux kernel. They are used to handle interrupt requests generated by hardware devices, as well as asynchronous events in the kernel. In eBPF, we can use the eBPF tools hardirqs and softirqs to capture and analyze information related to interrupt handling in the kernel.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/10-hardirqs

What are hardirqs and softirqs?

hardirqs are hardware interrupt handlers. When a hardware device generates an interrupt request, the kernel maps it to a specific interrupt vector and executes the associated hardware interrupt handler. Hardware interrupt handlers are commonly used to handle events in device drivers, such as completion of device data transfer or device errors.

softirqs are software interrupt handlers. They are a low-level asynchronous event handling mechanism in the kernel, used for handling high-priority tasks in the kernel. softirqs are commonly used to handle events in the network protocol stack, disk subsystem, and other kernel components. Compared to hardware interrupt handlers, software interrupt handlers have more flexibility and configurability.

Implementation Details

In eBPF, we can capture and analyze hardirqs and softirqs by attaching specific kprobes or tracepoints. To capture hardirqs and softirqs, eBPF programs need to be placed on relevant kernel functions. These functions include:

  • For hardirqs: irq_handler_entry and irq_handler_exit.
  • For softirqs: softirq_entry and softirq_exit.

When the kernel processes hardirqs or softirqs, these eBPF programs are executed to collect relevant information such as interrupt vectors, execution time of interrupt handlers, etc. The collected information can be used for analyzing performance issues and other interrupt handling related problems in the kernel.

To capture hardirqs and softirqs, the following steps can be followed:

  1. Define data structures and maps in eBPF programs for storing interrupt information.
  2. Write eBPF programs and attach them to the corresponding kernel functions to capture hardirqs or softirqs.
  3. In eBPF programs, collect relevant information about interrupt handlers and store this information in the maps.
  4. In user space applications, read the data from the maps to analyze and display the interrupt handling information.

By following the above approach, we can use hardirqs and softirqs in eBPF to capture and analyze interrupt events in the kernel, identifying potential performance issues and interrupt handling related problems.

Implementation of hardirqs Code

The main purpose of the hardirqs program is to obtain the name, execution count, and execution time of interrupt handlers and display the distribution of execution time in the form of a histogram. Let's analyze this code step by step.

The custom headers used below are part of the complete example rather than system headers. Build from the src/10-hardirqs directory, which contains hardirqs.h, bits.bpf.h, and maps.bpf.h alongside the source file.

text
/* Detailed source-code truncated for AI context efficiency. */

This code is an eBPF program used to capture and analyze the execution information of hardware interrupt handlers (hardirqs) in the kernel. The main purpose of the program is to obtain the name, execution count, and execution time of the interrupt handler, and display the distribution of execution time in the form of a histogram. Let's analyze this code step by step.

  1. Include necessary header files and define data structures:
c
#include <vmlinux.h>
    #include <bpf/bpf_core_read.h>
    #include <bpf/bpf_helpers.h>
    #include <bpf/bpf_tracing.h>
    #include "hardirqs.h"
    #include "bits.bpf.h"
    #include "maps.bpf.h"

This program includes the standard header files required for eBPF development, as well as custom header files for defining data structures and maps.

  1. Define global variables and maps:
c
#define MAX_ENTRIES 256

    const volatile bool filter_cg = false;
    const volatile bool targ_dist = false;
    const volatile bool targ_ns = false;
    const volatile bool do_count = false;

    ...

This program defines some global variables that are used to configure the behavior of the program. For example, filter_cg controls whether to filter cgroups, targ_dist controls whether to display the distribution of execution time, etc. Additionally, the program defines three maps for storing cgroup information, start timestamps, and interrupt handler information.

  1. Define two helper functions handle_entry and handle_exit:

These two functions are called at the entry and exit points of the interrupt handler. handle_entry records the start timestamp or updates the interrupt count, while handle_exit calculates the execution time of the interrupt handler and stores the result in the corresponding information map.

  1. Define the entry points of the eBPF program:
c
SEC("tp_btf/irq_handler_entry")
    int BPF_PROG(irq_handler_entry_btf, int irq, struct irqaction *action)
    {
    return handle_entry(irq, action);
    }

    SEC("tp_btf/irq_handler_exit")
    int BPF_PROG(irq_handler_exit_btf, int irq, struct irqaction *action)
    {
    return handle_exit(irq, action);
    }

    SEC("raw_tp/irq_handler_entry")
    int BPF_PROG(irq_handler_entry, int irq, struct irqaction *action)
    {
    return handle_entry(irq, action);
    }

    SEC("raw_tp/irq_handler_exit")
    int BPF_PROG(irq_handler_exit, int irq, struct irqaction *action)
    {
    return handle_exit(irq, action);
    }

Here, four entry points of the eBPF program are defined, which are used to capture the entry and exit events of the interrupt handler. tp_btf and raw_tp represent capturing events using BPF Type Format (BTF) and raw tracepoints, respectively. This ensures that the program can be ported and run on different kernel versions.

The code for Softirq is similar, and I won't elaborate on it here.

Run code.Translated content

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

To compile this program, use the ecc tool:

console
$ ecc hardirqs.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

Then run:

console
sudo ecli run ./package.json

Summary

In this chapter (eBPF Tutorial by Example Ten: Capturing Interrupt Events in eBPF with Hardirqs or Softirqs), we learned how to capture and analyze the execution information of hardware interrupt handlers (hardirqs) in the kernel using eBPF programs. We explained the example code in detail, including how to define data structures, mappings, eBPF program entry points, and how to call helper functions to record execution information at the entry and exit points of interrupt handlers.

By studying the content of this chapter, you should have mastered the methods of capturing interrupt events with hardirqs or softirqs in eBPF, as well as how to analyze these events to identify performance issues and other problems related to interrupt handling in the kernel. These skills are crucial for analyzing and optimizing the performance of the Linux kernel.

To better understand and practice eBPF programming, we recommend reading the official documentation of eunomia-bpf: https://github.com/eunomia-bpf/eunomia-bpf. In addition, we provide a complete tutorial and source code for you to view and learn from at https://github.com/eunomia-bpf/bpf-developer-tutorial.

The original link of this article: https://eunomia.dev/tutorials/10-hardirqs/

Overview (`src/11-bootstrap/README.md`)

eBPF Tutorial by Example 11: Develop User-Space Programs with libbpf and Trace exec() and exit()

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code during kernel runtime.

In this tutorial, we will learn how kernel-space and user-space eBPF programs work together. We will also learn how to use the native libbpf to develop user-space programs, package eBPF applications into executable files, and distribute them across different kernel versions.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/11-bootstrap

The libbpf Library and Why We Need to Use It

libbpf is a C language library that is distributed with the kernel version to assist in loading and running eBPF programs. It provides a set of C APIs for interacting with the eBPF system, allowing developers to write user-space programs more easily to load and manage eBPF programs. These user-space programs are typically used for system performance analysis, monitoring, or optimization.

There are several advantages to using the libbpf library:

  • It simplifies the process of loading, updating, and running eBPF programs.
  • It provides a set of easy-to-use APIs, allowing developers to focus on writing core logic instead of dealing with low-level details.
  • It ensures compatibility with the eBPF subsystem in the kernel, reducing maintenance costs.

At the same time, libbpf and BTF (BPF Type Format) are important components of the eBPF ecosystem. They play critical roles in achieving compatibility across different kernel versions. BTF is a metadata format used to describe type information in eBPF programs. The primary purpose of BTF is to provide a structured way to describe data structures in the kernel so that eBPF programs can access and manipulate them more easily.

The key roles of BTF in achieving compatibility across different kernel versions are as follows:

  • BTF allows eBPF programs to access detailed type information of kernel data structures without hardcoding specific kernel versions. This enables eBPF programs to adapt to different kernel versions, achieving compatibility across kernel versions.
  • By using BPF CO-RE (Compile Once, Run Everywhere) technology, eBPF programs can leverage BTF to parse the type information of kernel data structures during compilation, thereby generating eBPF programs that can run on different kernel versions.

By combining libbpf and BTF, eBPF programs can run on various kernel versions without the need for separate compilation for each kernel version. This greatly improves the portability and compatibility of the eBPF ecosystem and reduces the difficulty of development and maintenance.

What is Bootstrap

Bootstrap is a complete application that utilizes libbpf. It uses eBPF programs to trace the exec() system call in the kernel (handled by the SEC("tp/sched/sched_process_exec") handle_exec BPF program), which mainly corresponds to the creation of new processes (excluding the fork() part). In addition, it also traces the exit() system call of processes (handled by the SEC("tp/sched/sched_process_exit") handle_exit BPF program) to understand when each process exits.

These two BPF programs work together to capture interesting information about new processes, such as the file name of the binary and measure the lifecycle of processes. They also collect interesting statistics, such as exit codes or resource consumption, when a process exits. This is a good starting point to gain a deeper understanding of the inner workings of the kernel and observe how things actually operate.

Bootstrap also uses the argp API (part of libc) for command-line argument parsing, allowing users to configure the behavior of the application through command-line options. This provides flexibility and allows users to customize the program behavior according to their specific needs. While these functionalities can also be achieved using the eunomia-bpf tool, using libbpf here provides higher scalability in user space at the cost of additional complexity.

Bootstrap

Bootstrap consists of two parts: kernel space and user space. The kernel space part is an eBPF program that traces the exec() and exit() system calls. The user space part is a C language program that uses the libbpf library to load and run the kernel space program and process the data collected from the kernel space program.

Kernel-space eBPF Program bootstrap.bpf.c

text
/* Detailed source-code truncated for AI context efficiency. */

This code is a kernel-level eBPF program (bootstrap.bpf.c) used to trace exec() and exit() system calls. It captures process creation and exit events using an eBPF program and sends the relevant information to a user-space program for processing. Below is a detailed explanation of the code.

First, we include the necessary headers and define the license for the eBPF program. We also define two eBPF maps: exec_start and rb. exec_start is a hash type eBPF map used to store the timestamp when a process starts executing. rb is a ring buffer type eBPF map used to store captured event data and send it to the user-space program.

c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include "bootstrap.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, pid_t);
    __type(value, u64);
} exec_start SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

const volatile unsigned long long min_duration_ns = 0;

Next, we define an eBPF program named handle_exec which is triggered when a process executes the exec() system call. First, we retrieve the PID from the current process, record the timestamp when the process starts executing, and store it in the exec_start map.

c
SEC("tp/sched/sched_process_exec")
int handle_exec(struct trace_event_raw_sched_process_exec *ctx)
{
    // ...
    pid = bpf_get_current_pid_tgid() >> 32;
    ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&exec_start, &pid, &ts, BPF_ANY);

    // ...
}

Then, we reserve an event structure from the circular buffer map rb and fill in the relevant data, such as the process ID, parent process ID, and process name. Afterwards, we send this data to the user-mode program for processing.

c
// reserve sample from BPF ringbuf
    e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e)
        return 0;

    // fill out the sample with data
    task = (struct task_struct *)bpf_get_current_task();

    e->exit_event = false;
    e->pid = pid;
    e->ppid = BPF_CORE_READ(task, real_parent, tgid);
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

    fname_off = ctx->__data_loc_filename & 0xFFFF;
    bpf_probe_read_str(&e->filename, sizeof(e->filename), (void *)ctx + fname_off);

    // successfully submit it to user-space for post-processing
    bpf_ringbuf_submit(e, 0);
    return 0;

Finally, we define an eBPF program named handle_exit that will be triggered when a process executes the exit() system call. First, we retrieve the PID and TID (thread ID) from the current process. If the PID and TID are not equal, it means that this is a thread exit, and we will ignore this event.

c
SEC("tp/sched/sched_process_exit")
int handle_exit(struct trace_event_raw_sched_process_template* ctx)
{
    // ...
    id = bpf_get_current_pid_tgid();
    pid = id >> 32;
    tid = (u32)id;

    /* ignore thread exits */
    if (pid != tid)
        return 0;

    // ...
}

Next, we look up the timestamp of when the process started execution, which was previously stored in the exec_start map. If a timestamp is found, we calculate the process's lifetime duration and then remove the record from the exec_start map. If a timestamp is not found and a minimum duration is specified, we return directly.

c
// if we recorded start of the process, calculate lifetime duration
    start_ts = bpf_map_lookup_elem(&exec_start, &pid);
    if (start_ts)
        duration_ns = bpf_ktime_get_ns() - *start_ts;
    else if (min_duration_ns)
        return 0;
    bpf_map_delete_elem(&exec_start, &pid);

    // if process didn't live long enough, return early
    if (min_duration_ns && duration_ns < min_duration_ns)
        return 0;

Then, we reserve an event structure from the circular buffer map rb and fill in the relevant data, such as the process ID, parent process ID, process name, and process duration. Finally, we send this data to the user-mode program for processing.

c
/* reserve sample from BPF ringbuf */
    e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e)
        return 0;

    /* fill out the sample with data */
    task = (struct task_struct *)bpf_get_current_task();

    e->exit_event = true;
    e->duration_ns = duration_ns;```
e->pid = pid;
e->ppid = BPF_CORE_READ(task, real_parent, tgid);
e->exit_code = (BPF_CORE_READ(task, exit_code) >> 8) & 0xff;
bpf_get_current_comm(&e->comm, sizeof(e->comm));

/* send data to user-space for post-processing */
bpf_ringbuf_submit(e, 0);
return 0;
}

This way, when a process executes the exec() or exit() system calls, our eBPF program captures the corresponding events and sends detailed information to the user space program for further processing. This allows us to easily monitor process creation and termination and obtain detailed information about the processes.

In addition, in the bootstrap.h file, we also define the data structures for interaction with user space:

c
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
/* Copyright (c) 2020 Facebook */
#ifndef __BOOTSTRAP_H
#define __BOOTSTRAP_H

#define TASK_COMM_LEN 16
#define MAX_FILENAME_LEN 127

struct event {
    int pid;
    int ppid;
    unsigned exit_code;
    unsigned long long duration_ns;
    char comm[TASK_COMM_LEN];
    char filename[MAX_FILENAME_LEN];
    bool exit_event;
};

#endif /* __BOOTSTRAP_H */

User space, bootstrap.c

c
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
/* Copyright (c) 2020 Facebook */
#include <argp.h>
#include <signal.h>
#include <stdio.h>
#include <time.h>
#include <sys/resource.h>
#include 
#include "bootstrap.h"
#include "bootstrap.skel.h"

static struct env {
    bool verbose;
    long min_duration_ms;
} env;

const char *argp_program_version = "bootstrap 0.0";
const char *argp_program_bug_address = "";
const char argp_program_doc[] =
"BPF bootstrap demo application.\n"
"\n"
"It traces process start and exits and shows associated \n"
"information (filename, process duration, PID and PPID, etc).\n"
"\n"
"USAGE: ./bootstrap [-d <min-duration-ms>] [-v]\n";

static const struct argp_option opts[] = {
    { "verbose", 'v', NULL, 0, "Verbose debug output" },
    { "duration", 'd', "DURATION-MS", 0, "Minimum process duration (ms) to report" },
    {},
};

static error_t parse_arg(int key, char *arg, struct argp_state *state)
{
    switch (key) {
    case 'v':
        env.verbose = true;
        break;
    case 'd':
        errno = 0;
        env.min_duration_ms = strtol(arg, NULL, 10);
    if (errno || env.min_duration_ms <= 0) {
    fprintf(stderr, "Invalid duration: %s\n", arg);
    argp_usage(state);
}
break;
case ARGP_KEY_ARG:
    argp_usage(state);
    break;
default:
    return ARGP_ERR_UNKNOWN;
}
return 0;
}

static const struct argp argp = {
    .options = opts,
    .parser = parse_arg,
    .doc = argp_program_doc,
};

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    if (level == LIBBPF_DEBUG && !env.verbose)
        return 0;
    return vfprintf(stderr, format, args);
}

static volatile bool exiting = false;

static void sig_handler(int sig)
{
    exiting = true;
}

static int handle_event(void *ctx, void *data, size_t data_sz)
{
    const struct event *e = data;
    struct tm *tm;
    char ts[32];
    time_t t;

    time(&t);
    tm = localtime(&t);
    strftime(ts, sizeof(ts), "%H:%M:%S", tm);

    if (e->exit_event) {
        printf("%-8s %-5s %-16s %-7d %-7d [%u]",
               ts, "EXIT", e->comm, e->pid, e->ppid, e->exit_code);
        if (e->duration_ns)
            printf(" (%llums)", e->duration_ns / 1000000);
        printf("\n");
    } else {
        printf("%-8s %-5s %-16s %-7d %-7d %s\n",
               ts, "EXEC", e->comm, e->pid, e->ppid, e->filename);
    }

    return 0;
}

int main(int argc, char **argv)
{
    struct ring_buffer *rb = NULL;
    struct bootstrap_bpf *skel;
    int err;

    /* Parse command line arguments */
    err = argp_parse(&argp, argc, argv, 0, NULL, NULL);
    if (err)
        return err;

    /* Set up libbpf errors and debug info callback */
    libbpf_set_print(libbpf_print_fn);

    /* Cleaner handling of Ctrl-C */
    signal(SIGINT, sig_handler);
    signal(SIGTERM, sig_handler);

    /* Load and verify BPF application */
    skel = bootstrap_bpf__open();
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

    /* Parameterize BPF code with minimum duration parameter */
    skel->rodata->min_duration_ns = env.min_duration_ms * 1000000ULL;

    /* Load & verify BPF programs */
    err = bootstrap_bpf__load(skel);
    if (err) {
        fprintf(stderr, "Failed to load and verify BPF skeleton\n");
        goto cleanup;
    }

    /* Attach tracepoints */
    err = bootstrap_bpf__attach(skel);
    if (err) {
        fprintf(stderr, "Failed to attach BPF skeleton\n");
        goto cleanup;
    }

    /* Set up ring buffer polling */
    rb = ring_buffer__new(bpf_map__fd(skel->maps.rb), handle_event, NULL, NULL);
    if (!rb) {
        err = -1;
        fprintf(stderr, "Failed to create ring buffer\n");
        goto cleanup;
    }

    /* Process events */
    printf("%-8s %-5s %-16s %-7s %-7s %s\n",
        "TIME", "EVENT", "COMM", "PID", "PPID", "FILENAME/EXIT CODE");
    while (!exiting) {
        err = ring_buffer__poll(rb, 100 /* timeout, ms */);
        /* Ctrl-C will cause -EINTR */
        if (err == -EINTR) {
            err = 0;
            break;
        }
        if (err < 0) {
            printf("Error polling perf buffer: %d\n", err);
            break;
        }
    }

cleanup:
    /* Clean up */
    ring_buffer__free(rb);
    bootstrap_bpf__destroy(skel);

    return err < 0 ? -err : 0;
}

This user-level program is mainly used to load, verify, attach eBPF programs, and receive event data collected by eBPF programs and print it out. We will analyze some key parts.

First, we define an env structure to store command line arguments:

c
static struct env {
    bool verbose;
    long min_duration_ms;
} env;

Next, we use the argp library to parse command line arguments:

c
static const struct argp_option opts[] = {
    { "verbose", 'v', NULL, 0, "Verbose debug output" },
    { "duration", 'd', "DURATION-MS", 0, "Minimum process duration (ms) to report" },
    {},
};

static error_t parse_arg(int key, char *arg, struct argp_state *state)
{
    // ...
}

static const struct argp argp = {
    .options = opts,
    .parser = parse_arg,
    .doc = argp_program_doc,
};

In the main() function, we first parse the command line arguments, and then set the libbpf print callback function libbpf_print_fn to output debug information when needed:

c
err = argp_parse(&argp, argc, argv, 0, NULL, NULL);
if (err)
    return err;
libbpf_set_print(libbpf_print_fn);

Next, we open the eBPF skeleton file, pass the minimum duration parameter to the eBPF program, and load and attach the eBPF program:

c
skel = bootstrap_bpf__open();
if (!skel) {
    fprintf(stderr, "Failed to open and load BPF skeleton\n");
    return 1;
}

skel->rodata->min_duration_ns = env.min_duration_ms * 1000000ULL;

err = bootstrap_bpf__load(skel);
if (err) {
    fprintf(stderr, "Failed to load and verify BPF skeleton\n");
    goto cleanup;
}

err = bootstrap_bpf__attach(skel);
if (err) {
    fprintf(stderr, "Failed to attach BPF skeleton\n");
    goto cleanup;
}

Then, we create a ring buffer to receive event data sent by the eBPF program:

c
rb = ring_buffer__new(bpf_map__fd(skel->maps.rb), handle_event, NULL, NULL);
if (!rb) {
    err = -1;
    fprintf(stderr, "Failed to create ring buffer\n");
    goto cleanup;
}

The handle_event() function handles events received from the eBPF program. Depending on the event type (process execution or exit), it extracts and prints event information such as timestamp, process name, process ID, parent process ID, file name, or exit code.

Finally, we use the ring_buffer__poll() function to poll the ring buffer and process the received event data:

c
while (!exiting) {
    err = ring_buffer__poll(rb, 100 /* timeout, ms */);
    // ...
}

When the program receives the SIGINT or SIGTERM signal, it completes the final cleanup and exit operations, and closes and unloads the eBPF program:

c
cleanup:
 /* Clean up */
 ring_buffer__free(rb);
 bootstrap_bpf__destroy(skel);

 return err < 0 ? -err : 0;
}

Dependency Installation

Building the example requires clang, libelf, and zlib. The package names may vary in different distributions.

On Ubuntu/Debian, you need to execute the following command:

shell
sudo apt install clang libelf1 libelf-dev zlib1g-dev

On CentOS/Fedora, you need to execute the following command:

shell
sudo dnf install clang elfutils-libelf elfutils-libelf-devel zlib-devel

Compile and Run

Compile and run the above code:

console
$ git submodule update --init --recursive
$ make
  BPF      .output/bootstrap.bpf.o
  GEN-SKEL .output/bootstrap.skel.h
  CC       .output/bootstrap.o
  BINARY   bootstrap
$ sudo ./bootstrap 
[sudo] password for yunwei: 
TIME     EVENT COMM             PID     PPID    FILENAME/EXIT CODE
03:16:41 EXEC  sh               110688  80168   /bin/sh
03:16:41 EXEC  which            110689  110688  /usr/bin/which
03:16:41 EXIT  which            110689  110688  [0] (0ms)
03:16:41 EXIT  sh               110688  80168   [0] (0ms)".

The complete source code can be found at https://github.com/eunomia-bpf/bpf-developer-tutorial

Summary

Through this example, we have learned how to combine eBPF programs with user-space programs. This combination provides developers with a powerful toolkit for efficient data collection and processing across the kernel and user space. By using eBPF and libbpf, you can build more efficient, scalable, and secure monitoring and performance analysis tools.

In the following tutorials, we will continue to explore the advanced features of eBPF and share more about eBPF development practices. Through continuous learning and practice, you will have a better understanding and mastery of eBPF technology and apply it to solve real-world problems.

If you would like to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

Reference

The original link of this article: https://eunomia.dev/tutorials/11-bootstrap

Overview (`src/12-profile/README.md`)

eBPF Tutorial by Example 12: Using eBPF Program Profile for Performance Analysis

This tutorial will guide you on using eBPF programs for performance analysis with a Rust implementation. We will leverage the perf mechanism in the kernel to learn how to capture the execution time of functions and view performance data.

This implementation uses libbpf-rs, a Rust wrapper around libbpf, along with blazesym for symbol resolution. Perf is a performance analysis tool in the Linux kernel that allows users to measure and analyze the performance of kernel and user space programs, as well as obtain corresponding call stacks. It collects performance data using hardware counters and software events in the kernel.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/12-profile

eBPF Tool: profile Performance Analysis Example

The profile tool is implemented based on eBPF and utilizes the perf events in the Linux kernel for performance analysis. The profile tool periodically samples each processor to capture the execution of kernel and user space functions.

In the stack trace information, the tool displays the memory addresses of function calls, which represent the most primitive location information. Through symbol resolution, these addresses are converted to corresponding function names, allowing developers to directly identify which functions are being executed. Furthermore, when debug information is available, the tool can provide source code file names and specific line numbers, pinpointing the exact location in the code. This complete information chain from addresses to symbols to source code locations provides developers with a comprehensive perspective for performance analysis.

This detailed information helps developers quickly locate performance bottlenecks and optimize code. By analyzing which functions are frequently called and which code paths consume the most CPU time, developers can perform targeted optimizations. Additionally, this stack trace information can be converted into flame graph format, providing a visual representation of program execution hotspots that makes performance issues immediately apparent.

In this example, you can compile and run it with Rust and Cargo:

Prerequisites:

  • Rust and Cargo installed (see "The Cargo Book")
  • Clang and development libraries
console
$ git submodule update --init --recursive
$ sudo apt install clang libelf1 libelf-dev zlib1g-dev
$ make
$ sudo ./profile

Sample Output:

console
[1756723652.366364804] COMM: node (pid=285503) @ CPU 64
No Kernel Stack
Userspace:
0x0072e2a97f4be0: v8::internal::Scanner::Next() @ 0x15f47c0+0x420
0x0072e2a97d9051: v8::internal::ParserBase<v8::internal::PreParser>::ParseBlock(...) @ 0x15d8fc0+0x91
0x0072e2a97d6df0: v8::internal::ParserBase<v8::internal::PreParser>::ParseStatement(...) @ 0x15d6ce0+0x110
...

[1756723657.337170411] COMM: qemu-system-x86 (pid=4166437) @ CPU 70
Kernel:
0xffffffff95f403d5: _raw_spin_lock_irq @ 0xffffffff95f403b0+0x25
0xffffffff94e2b6d8: __flush_work @ 0xffffffff94e2b630+0xa8
0xffffffff94e2ba5c: flush_work @ 0xffffffff94e2ba40+0x1c
0xffffffff95852672: tty_buffer_flush_work @ 0xffffffff95852660+0x12
...
Userspace:
0x005849a7fbbd33: qemu_poll_ns @ 0xc63c4c+0xe7 /home/victoryang00/CXLMemSim/lib/qemu/build/../util/qemu-timer.c:347:1
0x005849a7fb64d7: os_host_main_loop_wait @ 0xc5e473+0x64 /home/victoryang00/CXLMemSim/lib/qemu/build/../util/main-loop.c:305:11
...

The tool provides detailed stack traces with symbol resolution, including function names, offsets, and source file locations when available.

Implementation Principle

The profile tool consists of two parts: the eBPF program in kernel space and the profile symbol handling program in user space. The profile symbol handling program is responsible for loading the eBPF program and processing the data outputted by the eBPF program.

Kernel Space Part

The implementation logic of the eBPF program in kernel space mainly relies on perf events to periodically sample the stack of the program, thereby capturing its execution flow.

c
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/* Copyright (c) 2022 Meta Platforms, Inc. */
#include "vmlinux.h"
#include 
#include 
#include 

#include "profile.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("perf_event")
int profile(void *ctx)
{
    int pid = bpf_get_current_pid_tgid() >> 32;
    int cpu_id = bpf_get_smp_processor_id();
    struct stacktrace_event *event;
    int cp;

    event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
    if (!event)
        return 1;

    event->pid = pid;
    event->cpu_id = cpu_id;
    event->timestamp = bpf_ktime_get_ns();  // Capture timestamp

    if (bpf_get_current_comm(event->comm, sizeof(event->comm)))
        event->comm[0] = 0;

    event->kstack_sz = bpf_get_stack(ctx, event->kstack, sizeof(event->kstack), 0);

    event->ustack_sz = 
        bpf_get_stack(ctx, event->ustack, sizeof(event->ustack), BPF_F_USER_STACK);

    bpf_ringbuf_submit(event, 0);

    return 0;
}

The header file profile.h defines the event structure:

c
/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
#ifndef __PROFILE_H_
#define __PROFILE_H_

#ifndef TASK_COMM_LEN
#define TASK_COMM_LEN 16
#endif

#ifndef MAX_STACK_DEPTH
#define MAX_STACK_DEPTH 128
#endif

typedef __u64 stack_trace_t[MAX_STACK_DEPTH];

struct stacktrace_event {
    __u32 pid;
    __u32 cpu_id;
    __u64 timestamp;  // Kernel timestamp in nanoseconds
    char comm[TASK_COMM_LEN];
    __s32 kstack_sz;
    __s32 ustack_sz;
    stack_trace_t kstack;
    stack_trace_t ustack;
};

#endif /* __PROFILE_H_ */

Next, we will focus on the key part of the kernel code.

  1. Define eBPF maps events:
c
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

Here, a eBPF maps of type BPF_MAP_TYPE_RINGBUF is defined. The Ring Buffer is a high-performance circular buffer used to transfer data between the kernel and user space. max_entries sets the maximum size of the Ring Buffer.

  1. Define perf_event eBPF program:
c
SEC("perf_event")
int profile(void *ctx)

Here, a eBPF program named profile is defined, which will be executed when a perf event is triggered.

  1. Get process ID and CPU ID:
c
int pid = bpf_get_current_pid_tgid() >> 32;
int cpu_id = bpf_get_smp_processor_id();

The function bpf_get_current_pid_tgid() returns the PID and TID of the current process. By right shifting 32 bits, we get the PID. The function bpf_get_smp_processor_id() returns the ID of the current CPU.

  1. Reserve space in the Ring Buffer:
c
event = bpf_ringbuf_reserve(&events, sizeof(*event), 0);
if (!event)
    return 1;

Use the bpf_ringbuf_reserve() function to reserve space in the Ring Buffer for storing the collected stack information. If the reservation fails, return an error.

  1. Get the current process name:
c
if (bpf_get_current_comm(event->comm, sizeof(event->comm)))
    event->comm[0] = 0;

Use the bpf_get_current_comm() function to get the current process name and store it in event->comm.

  1. Get kernel stack information:
c
event->kstack_sz = bpf_get_stack(ctx, event->kstack, sizeof(event->kstack), 0);

Use the bpf_get_stack() function to get kernel stack information. Store the result in event->kstack and the size in event->kstack_sz.

  1. Get user space stack information:
c
event->ustack_sz = bpf_get_stack(ctx, event->ustack, sizeof(event->ustack), BPF_F_USER_STACK);

Using the bpf_get_stack() function with the BPF_F_USER_STACK flag retrieves information about the user space stack. Store the result in event->ustack and its size in event->ustack_sz.

  1. Submit the event to the Ring Buffer:
c
bpf_ringbuf_submit(event, 0);

Finally, use the bpf_ringbuf_submit() function to submit the event to the Ring Buffer for the user space program to read and process.

This kernel mode eBPF program captures the program's execution flow by sampling the kernel stack and user space stack of the program periodically. These data are stored in the Ring Buffer for the user mode profile program to read.

User Mode Section (Rust Implementation)

The user-space portion is implemented in Rust using libbpf-rs and blazesym. The main components include:

Main Entry Point (src/main.rs):

text
/* Detailed source-code truncated for AI context efficiency. */

The perf_event_open function is a wrapper for the perf_event_open system call. It takes a pointer to a perf_event_attr structure to specify the type and attributes of the perf event. The pid parameter is used to specify the process ID to monitor (-1 for monitoring all processes), and the cpu parameter is used to specify the CPU to monitor. The group_fd parameter is used to group perf events, and we use -1 here to indicate no grouping is needed. The flags parameter is used to set some flags, and we use PERF_FLAG_FD_CLOEXEC to ensure file descriptors are closed when executing exec series system calls.

In the main function:

c
for (cpu = 0; cpu < num_cpus; cpu++) {
    // ...
}

This loop sets up perf events and attaches eBPF programs for each online CPU. Firstly, it checks if the current CPU is online and skips if it's not. Then, it uses the perf_event_open() function to set up perf events for the current CPU and stores the returned file descriptor in the pefds array. Finally, it attaches the eBPF program to the perf event using the bpf_program__attach_perf_event() function. The links array is used to store the BPF links for each CPU so that they can be destroyed when the program ends.By doing so, user-mode programs set perf events for each online CPU and attach eBPF programs to these perf events to monitor all online CPUs in the system.

Event Processing and Symbol Resolution (src/event.rs):

text
/* Detailed source-code truncated for AI context efficiency. */

Key Features of the Rust Implementation:

The Rust implementation provides strong type safety through its type system, effectively preventing memory safety issues that are common in C, such as buffer overflows and null pointer dereferences. This safety guarantee allows developers to focus on business logic without worrying about low-level memory management issues.

Symbol resolution is a core feature of performance analysis tools. This implementation integrates the blazesym library, which efficiently converts memory addresses into readable function names and source code locations. Blazesym supports DWARF debug information parsing, meaning that even optimized binaries can provide accurate source file paths and line number information. This capability is crucial for pinpointing the exact code locations of performance bottlenecks.

For error handling, Rust's Result type provides an explicit error handling mechanism. Every operation that might fail returns a Result type, forcing developers to handle potential error conditions. This design prevents unhandled errors from causing program crashes, improving the tool's stability and reliability.

The logging system uses the tracing crate, which provides structured logging capabilities. Through environment variables or command-line arguments, users can dynamically adjust the log level from WARN, INFO, DEBUG to TRACE, making it convenient to obtain appropriate levels of diagnostic information in different scenarios. This flexibility is very useful for debugging and troubleshooting.

The command-line interface is implemented through the clap library, providing intuitive argument parsing and help message generation. Users can adjust the sampling frequency with the -f parameter, switch to software events using --sw-event in environments like virtual machines that don't support hardware performance counters, filter specific processes with the -p parameter, and output in an extended format suitable for flame graph generation using the -E parameter.

This integrated approach combines the performance of eBPF with the safety and expressiveness of Rust, providing a robust profiling tool for system performance analysis.

Summary

Through this introductory tutorial on eBPF, we have learned how to use eBPF programs for performance analysis. In this process, we explained in detail how to create eBPF programs, monitor process performance, and retrieve data from the ring buffer for analyzing stack traces. We also learned how to use the perf_event_open() function to set up performance monitoring and attach BPF programs to performance events. In this tutorial, we also demonstrated how to write eBPF programs to capture the kernel and userspace stack information of processes in order to analyze program performance bottlenecks. With this example, you can understand the powerful features of eBPF in performance analysis.

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

The next tutorial will further explore advanced features of eBPF. We will continue to share more content about eBPF development practices to help you better understand and master eBPF technology. We hope these contents will be helpful for your learning and practice on the eBPF development journey.

The original link of this article: https://eunomia.dev/tutorials/12-profile

Overview (`src/13-tcpconnlat/README.md`)

eBPF Tutorial by Example 13: Statistics of TCP Connection Delay with libbpf

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting the kernel or changing the kernel source code.

This article is the thirteenth installment of the eBPF Tutorial by Example, mainly about how to use eBPF to statistics TCP connection delay and process data in user space using libbpf.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/13-tcpconnlat

Background

When developing backends, regardless of the programming language used, we often need to call databases such as MySQL and Redis, perform RPC remote calls, or call other RESTful APIs. The underlying implementation of these calls is usually based on the TCP protocol. This is because TCP protocol has advantages such as reliable connection, error retransmission, congestion control, etc., so TCP is more widely used in network transport layer protocols than UDP. However, TCP also has some drawbacks, such as longer connection establishment delay. Therefore, some alternative solutions have emerged, such as QUIC (Quick UDP Internet Connections).

Analyzing TCP connection delay is very useful for network performance analysis, optimization, and troubleshooting.

Overview of tcpconnlat Tool

The tcpconnlat tool can trace the functions in the kernel that perform active TCP connections (such as using the connect() system call), measure and display connection delay, i.e., the time from sending SYN to receiving response packets.

TCP Connection Principle

The process of establishing a TCP connection is often referred to as the "three-way handshake". Here are the steps of the entire process:

  1. Client sends SYN packet to the server: The client sends SYN through the connect() system call. This involves local system call and CPU time cost of software interrupts.
  2. SYN packet is transmitted to the server: This is a network transmission that depends on network latency.
  3. Server handles the SYN packet: The server kernel receives the packet through a software interrupt, then puts it into the listen queue and sends SYN/ACK response. This mainly involves CPU time cost.
  4. SYN/ACK packet is transmitted to the client: This is another network transmission.
  5. Client handles the SYN/ACK: The client kernel receives and handles the SYN/ACK packet, then sends ACK. This mainly involves software interrupt handling cost.
  6. ACK packet is transmitted to the server: This is the third network transmission.
  7. Server receives ACK: The server kernel receives and handles the ACK, then moves the corresponding connection from the listen queue to the established queue. This involves CPU time cost of a software interrupt.
  8. Wake up the server-side user process: The user process blocked by the accept() system call is awakened, and then the established connection is taken out from the established queue. This involves CPU cost of a context switch.

The complete flowchart is shown below:

From the client's perspective, under normal circumstances, the total time for a TCP connection is approximately the time consumed by one network round-trip. However, in some cases, it may cause an increase in network transmission time, an increase in CPU processing overhead, or even connection failure. When a long delay is detected, it can be analyzed in conjunction with other information.

eBPF Implementation of tcpconnlat

To understand the process of establishing a TCP connection, we need to understand two queues used by the Linux kernel when handling TCP connections:

  • Listen queue (SYN queue): Stores TCP connections that are in the process of performing three-way handshake. After the server receives the SYN packet, it stores the connection information in this queue.
  • Established queue (Accept queue): Stores TCP connections that have completed three-way handshake and are waiting for the application to call the accept() function. After the server receives the ACK packet, it creates a new connection and adds it to this queue.

With an understanding of the purpose of these two queues, we can begin to explore the specific implementation of tcpconnlat. The implementation of tcpconnlat can be divided into two parts: kernel space and user space, which include several main trace points: tcp_v4_connect, tcp_v6_connect, and tcp_rcv_state_process.

These trace points are mainly located in the TCP/IP network stack in the kernel. When executing the corresponding system call or kernel function, these trace points are activated, triggering the execution of eBPF programs. This allows us to capture and measure the entire process of establishing a TCP connection.

Let's take a look at the source code of these mounting points first:

c
SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
{
 return trace_connect(sk);
}

SEC("kprobe/tcp_v6_connect")
int BPF_KPROBE(tcp_v6_connect, struct sock *sk)
{
  return trace_connect(sk);
}

SEC("kprobe/tcp_rcv_state_process")
int BPF_KPROBE(tcp_rcv_state_process, struct sock *sk)
{
  return handle_tcp_rcv_state_process(ctx, sk);
}

This code snippet shows the definition of three kernel probes (kprobe). tcp_v4_connect and tcp_v6_connect are triggered when the corresponding IPv4 and IPv6 connections are initialized, invoking the trace_connect() function. On the other hand, tcp_rcv_state_process is triggered when the TCP connection state changes in the kernel, calling the handle_tcp_rcv_state_process() function.

The following section will be divided into two parts: one part analyzes the kernel part of these mount points, where we will delve into the kernel source code to explain how these functions work in detail. The other part analyzes the user part, focusing on how eBPF programs collect data from these mount points and interact with user-space programs.

Analysis of tcp_v4_connect function

The tcp_v4_connect function is the main way that the Linux kernel handles TCP IPv4 connection requests. When a user-space program creates a socket through the socket system call and then attempts to connect to a remote server through the connect system call, the tcp_v4_connect function is triggered.

text
/* Detailed source-code truncated for AI context efficiency. */

Reference link: https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_ipv4.c#L340

Next, let's analyze this function step by step:

First, this function takes three parameters: a socket pointer sk, a pointer to the socket address structure uaddr, and the length of the address addr_len.

c
int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)

The function starts by checking the parameters, making sure the address length is correct and the address family is IPv4. If these conditions are not met, the function returns an error.

Next, the function retrieves the destination address and, if a source routing option is set (an advanced IP feature that is typically not used), it also retrieves the next hop address for the source route.

c
nexthop = daddr = usin->sin_addr.s_addr;
inet_opt = rcu_dereference_protected(inet->inet_opt,
         lockdep_sock_is_held(sk));
if (inet_opt && inet_opt->opt.srr) {
  if (!daddr)
    return -EINVAL;
  nexthop = inet_opt->opt.faddr;
}

Then, using this information, the function looks for a route entry to the destination address. If a route entry cannot be found or the route entry points to a multicast or broadcast address, the function returns an error.

Next, it updates the source address, handles the state of some TCP timestamp options, and sets the destination port and address. After that, it updates some other socket and TCP options and sets the connection state to SYN-SENT.

Then, the function tries to add the socket to the connected sockets hash table using the inet_hash_connect function. If this step fails, it restores the socket state and returns an error.

If all the previous steps succeed, it then updates the route entry with the new source and destination ports. If this step fails, it cleans up resources and returns an error.

Next, it commits the destination information to the socket and selects a secure random value for the sequence offset for future segments.

Then, the function tries to establish the connection using TCP Fast Open (TFO), and if TFO is not available or the TFO attempt fails, it falls back to the regular TCP three-way handshake for connection.

Finally, if all the above steps succeed, the function returns success; otherwise, it cleans up all resources and returns an error.

In summary, the tcp_v4_connect function is a complex function that handles TCP connection requests. It handles many cases, including parameter checking, route lookup, source address selection, source routing, TCP option handling, TCP Fast Open, and more. Its main goal is to establish TCP connections as safely and efficiently as possible.

Kernel Code

text
/* Detailed source-code truncated for AI context efficiency. */

This eBPF (Extended Berkeley Packet Filter) program is mainly used to monitor and collect the time it takes to establish TCP connections, i.e., the time interval from initiating a TCP connection request (connect system call) to the completion of the connection establishment (SYN-ACK handshake process). This is very useful for monitoring network latency, service performance analysis, and other aspects.

First, two eBPF maps are defined: start and events. start is a hash table used to store the process information and timestamp of the initiating connection request, while events is a map of type PERF_EVENT_ARRAY used to transfer event data to user space.

c
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 4096);
  __type(key, struct sock *);
  __type(value, struct piddata);
} start SEC(".maps");

struct {
  __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
  __uint(key_size, sizeof(u32));
  __uint(value_size, sizeof(u32));
} events SEC(".maps");

In the kprobe handling functions trace_connect of tcp_v4_connect and tcp_v6_connect, the process information (process name, process ID, and current timestamp) of the initiating connection request is recorded and stored in the start map with the socket structure as the key.

c
static int trace_connect(struct sock *sk)
{
  u32 tgid = bpf_get_current_pid_tgid() >> 32;
  struct piddata piddata = {};

  if (targ_tgid && targ_tgid != tgid)
    return 0;

  bpf_get_current_comm(&piddata.comm, sizeof(piddata.comm));
  piddata.ts = bpf_ktime_get_ns();
  piddata.tgid = tgid;
  bpf_map_update_elem(&start, &sk, &piddata, 0);
  return 0;
}

When the TCP state machine processes the SYN-ACK packet, i.e., when the connection is established, the kprobe handling function handle_tcp_rcv_state_process of tcp_rcv_state_process is triggered. In this function, it first checks if the socket state is SYN-SENT. If it is, it looks up the process information for the socket in the start map. Then it calculates the time interval from the initiation of the connection to the present and sends this time interval, process information, and TCP connection details (source port, destination port, source IP, destination IP, etc.) as an event to user space using the bpf_perf_event_output function.

c
static int handle_tcp_rcv_state_process(void *ctx, struct sock *sk)
{
  struct piddata *piddatap;
  struct event event = {};
  s64 delta;
  u64 ts;

  if (BPF_CORE_READ(sk, __sk_common.skc_state) != TCP_SYN_SENT)
    return 0;

  piddatap = bpf_map_lookup_elem(&start, &sk);
  if (!piddatap)
    return 0;

  ts = bpf_ktime_get_ns();
  delta = (s64)(ts - piddatap->ts);
  if (delta < 0)
    goto cleanup;

  event.delta_us = delta / 1000U;
  if (targ_min_us && event.delta_us < targ_min_us)
    goto cleanup;
  __builtin_memcpy(&event.comm, piddatap->comm,
      sizeof(event.comm));
  event.ts_us = ts / 1000;
  event.tgid = piddatap->tgid;
  event.lport = BPF_CORE_READ(sk, __sk_common.skc_num);
  event.dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
  event.af = BPF_CORE_READ(sk, __sk_common.skc_family);
  if (event.af == AF_INET) {
    event.saddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_rcv_saddr);
    event.daddr_v4 = BPF_CORE_READ(sk, __sk_common.skc_daddr);
  } else {
    BPF_CORE_READ_INTO(&event.saddr_v6, sk,
        __sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
    BPF_CORE_READ_INTO(&event.daddr_v6, sk,
        __sk_common.skc_v6_daddr.in6_u.u6_addr32);
  }
  bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
      &event, sizeof(event));

cleanup:
  bpf_map_delete_elem(&start, &sk);
  return 0;
}

This program uses a while loop to repeatedly poll the perf event buffer. If there is an error during polling (e.g., due to a signal interruption), an error message will be printed. This polling process continues until an exit flag exiting is received.

Next, let's take a look at the handle_event function, which handles every eBPF event sent from the kernel to user space:

c
void handle_event(void* ctx, int cpu, void* data, __u32 data_sz) {
    const struct event* e = data;
    char src[INET6_ADDRSTRLEN];
    char dst[INET6_ADDRSTRLEN];
    union {
        struct in_addr x4;
        struct in6_addr x6;
    } s, d;
    static __u64 start_ts;

    if (env.timestamp) {
        if (start_ts == 0)
            start_ts = e->ts_us;
        printf("%-9.3f ", (e->ts_us - start_ts) / 1000000.0);
    }
    if (e->af == AF_INET) {
        s.x4.s_addr = e->saddr_v4;
        d.x4.s_addr = e->daddr_v4;
    } else if (e->af == AF_INET6) {
        memcpy(&s.x6.s6_addr, e->saddr_v6, sizeof(s.x6.s6_addr));
        memcpy(&d.x6.s6_addr, e->daddr_v6, sizeof(d.x6.s6_addr));
    } else {
        fprintf(stderr, "broken event: event->af=%d", e->af);
        return;
    }

    if (env.lport) {
        printf("%-6d %-12.12s %-2d %-16s %-6d %-16s %-5d %.2f\n", e->tgid,
               e->comm, e->af == AF_INET ? 4 : 6,
               inet_ntop(e->af, &s, src, sizeof(src)), e->lport,
               inet_ntop(e->af, &d, dst, sizeof(dst)), ntohs(e->dport),
               e->delta_us / 1000.0);
    } else {
        printf("%-6d %-12.12s %-2d %-16s %-16s %-5d %.2f\n", e->tgid, e->comm,
               e->af == AF_INET ? 4 : 6, inet_ntop(e->af, &s, src, sizeof(src)),
               inet_ntop(e->af, &d, dst, sizeof(dst)), ntohs(e->dport),
               e->delta_us / 1000.0);
    }
}

The handle_event function takes arguments including the CPU number, a pointer to the data, and the size of the data. The data is a event structure that contains information about TCP connections computed in the kernel space.

First, it compares the timestamp of the received event with the start timestamp (if available) to calculate the relative time of the event, and then prints it. Next, it converts the source address and destination address from network byte order to host byte order based on the IP address type (IPv4 or IPv6).

Finally, depending on whether the user chooses to display the local port, it prints the process ID, process name, IP version, source IP address, local port (if available), destination IP address, destination port, and connection establishment time. This connection establishment time is calculated in the eBPF program running in the kernel space and sent to the user space.

Compilation and Execution

console
$ make
...
  BPF      .output/tcpconnlat.bpf.o".GEN-SKEL .output/tcpconnlat.skel.h
  CC       .output/tcpconnlat.o
  BINARY   tcpconnlat
$ sudo ./tcpconnlat 
PID    COMM         IP SADDR            DADDR            DPORT LAT(ms)
222564 wget         4  192.168.88.15    110.242.68.3     80    25.29
222684 wget         4  192.168.88.15    167.179.101.42   443   246.76
222726 ssh          4  192.168.88.15    167.179.101.42   22    241.17
222774 ssh          4  192.168.88.15    1.15.149.151     22    25.31

Source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/13-tcpconnlat

References:

Summary

In this eBPF introductory tutorial, we learned how to use eBPF to track and measure the latency of TCP connections. We first explored how eBPF programs can attach to specific kernel functions in kernel-space and capture the start and end times of connection establishment to calculate latency.

We also learned how to use BPF maps to store and retrieve data in kernel-space, enabling data sharing among different parts of the eBPF program. Additionally, we discussed how to use perf events to send data from kernel-space to user-space for further processing and display.

In user-space, we introduced the usage of libbpf library APIs, such as perf_buffer__poll, to receive and process data sent from the kernel-space. We also demonstrated how to parse and print this data in a human-readable format.

If you are interested in learning more about eBPF and its practical applications, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial for more examples and complete tutorials.

In the upcoming tutorials, we will dive deeper into advanced features of eBPF, such as tracing the path of network packets and fine-grained system performance monitoring. We will continue to share more content on eBPF development practices to help you better understand and master eBPF technology. We hope these resources will be valuable in your learning and practical journey with eBPF.

The original link of this article: https://eunomia.dev/tutorials/13-tcpconnect

Overview (`src/14-tcpstates/README.md`)

eBPF Tutorial by Example 14: Recording TCP Connection Status and TCP RTT

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting the kernel or changing the kernel source code.

In this article of our eBPF Tutorial by Example series, we will introduce two sample programs: tcpstates and tcprtt. tcpstates is used to record the state changes of TCP connections, while tcprtt is used to record the Round-Trip Time (RTT) of TCP.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/14-tcpstates

`tcprtt` and `tcpstates`

Network quality is crucial in the current Internet environment. There are many factors that affect network quality, including hardware, network environment, and the quality of software programming. To help users better locate network issues, we introduce the tool tcprtt. tcprtt can monitor the Round-Trip Time of TCP connections, evaluate network quality, and help users identify potential problems.

When a TCP connection is established, tcprtt automatically selects the appropriate execution function based on the current system conditions. In the execution function, tcprtt collects various basic information of the TCP connection, such as source address, destination address, source port, destination port, and time elapsed, and updates this information to a histogram-like BPF map. After the execution is completed, tcprtt presents the collected information graphically to users through user-mode code.

tcpstates is a tool specifically designed to track and print changes in TCP connection status. It can display the duration of TCP connections in each state, measured in milliseconds. For example, for a single TCP session, tcpstates can print output similar to the following:

sh
SKADDR           C-PID C-COMM     LADDR           LPORT RADDR           RPORT OLDSTATE    -> NEWSTATE    MS
ffff9fd7e8192000 22384 curl       100.66.100.185  0     52.33.159.26    80    CLOSE       -> SYN_SENT    0.000
ffff9fd7e8192000 0     swapper/5  100.66.100.185  63446 52.33.159.26    80    SYN_SENT    -> ESTABLISHED 1.373
ffff9fd7e8192000 22384 curl       100.66.100.185  63446 52.33.159.26    80    ESTABLISHED -> FIN_WAIT1   176.042
ffff9fd7e8192000 0     swapper/5  100.66.100.185  63446 52.33.159.26    80    FIN_WAIT1   -> FIN_WAIT2   0.536
ffff9fd7e8192000 0     swapper/5  100.66.100.185  63446 52.33.159.26    80    FIN_WAIT2   -> CLOSE       0.006

In the above output, the most time is spent in the ESTABLISHED state, which indicates that the connection has been established and data transmission is in progress. The transition from this state to the FIN_WAIT1 state (the beginning of connection closure) took 176.042 milliseconds.

In our upcoming tutorials, we will delve deeper into these two tools, explaining their implementation principles, and hopefully, these contents will help you in your work with eBPF for network and performance analysis.

tcpstate eBPF code

Due to space constraints, here we mainly discuss and analyze the corresponding eBPF kernel-mode code implementation. The following is the eBPF code for tcpstate:

c
const volatile bool filter_by_sport = false;
const volatile bool filter_by_dport = false;
const volatile short target_family = 0;

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, MAX_ENTRIES);
    __type(key, __u16);
    __type(value, __u16);
} sports SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, MAX_ENTRIES);
    ...
```__type(key, __u16);
    __type(value, __u16);
} dports SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, MAX_ENTRIES);
    __type(key, struct sock *);
    __type(value, __u64);
} timestamps SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(__u32));
    __uint(value_size, sizeof(__u32));
} events SEC(".maps");

SEC("tracepoint/sock/inet_sock_set_state")
int handle_set_state(struct trace_event_raw_inet_sock_set_state *ctx)
{
    struct sock *sk = (struct sock *)ctx->skaddr;
    __u16 family = ctx->family;
    __u16 sport = ctx->sport;
    __u16 dport = ctx->dport;
    __u64 *tsp, delta_us, ts;
    struct event event = {};

    if (ctx->protocol != IPPROTO_TCP)
        return 0;

    if (target_family && target_family != family)
        return 0;

    if (filter_by_sport && !bpf_map_lookup_elem(&sports, &sport))
        return 0;

    if (filter_by_dport && !bpf_map_lookup_elem(&dports, &dport))
        return 0;

    tsp = bpf_map_lookup_elem(&timestamps, &sk);
    ts = bpf_ktime_get_ns();
    if (!tsp)
        delta_us = 0;
    else
        delta_us = (ts - *tsp) / 1000;

    event.skaddr = (__u64)sk;
    event.ts_us = ts / 1000;
    event.delta_us = delta_us;
    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.oldstate = ctx->oldstate;
    event.newstate = ctx->newstate;
    event.family = family;
    event.sport = sport;
    event.dport = dport;
    bpf_get_current_comm(&event.task, sizeof(event.task));

    if (family == AF_INET) {
        bpf_probe_read_kernel(&event.saddr, sizeof(event.saddr), &sk->__sk_common.skc_rcv_saddr);
        bpf_probe_read_kernel(&event.daddr, sizeof(event.daddr), &sk->__sk_common.skc_daddr);
    } else { /* family == AF_INET6 */
        bpf_probe_read_kernel(&event.saddr, sizeof(event.saddr), &sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32);
        bpf_probe_read_kernel(&event.daddr, sizeof(event.daddr), &sk->__sk_common.skc_v6_daddr.in6_u.u6_addr32);
    }
    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, &event, sizeof(event));

    if (ctx->newstate == TCP_CLOSE)
        bpf_map_delete_elem(&timestamps, &sk);
    else
        bpf_map_update_elem(&timestamps, &sk, &ts, BPF_ANY);

    return 0;
}

/* Detailed source-code truncated for AI context efficiency. */

c
static void handle_event(void* ctx, int cpu, void* data, __u32 data_sz) {
    char ts[32], saddr[26], daddr[26];
    struct event* e = data;
    struct tm* tm;
    int family;
    time_t t;

    if (emit_timestamp) {
        time(&t);
        tm = localtime(&t);
        strftime(ts, sizeof(ts), "%H:%M:%S", tm);
        printf("%8s ", ts);
    }

    inet_ntop(e->family, &e->saddr, saddr, sizeof(saddr));
    inet_ntop(e->family, &e->daddr, daddr, sizeof(daddr));
    if (wide_output) {
        family = e->family == AF_INET ? 4 : 6;
        printf(
            "%-16llx %-7d %-16s %-2d %-26s %-5d %-26s %-5d %-11s -> %-11s "
            "%.3f\n",
            e->skaddr, e->pid, e->task, family, saddr, e->sport, daddr,
            e->dport, tcp_states[e->oldstate], tcp_states[e->newstate],
            (double)e->delta_us / 1000);
    } else {
        printf(
            "%-16llx %-7d %-10.10s %-15s %-5d %-15s %-5d %-11s -> %-11s %.3f\n",
    ...

handle_event` is a callback function that is called by perf_event. It handles new events that arrive in the kernel.

In the handle_event function, we first use the inet_ntop function to convert the binary IP address to a human-readable format. Then, based on whether the wide format is needed or not, we print different information. This information includes the timestamp of the event, source IP address, source port, destination IP address, destination port, old state, new state, and the time spent in the old state.

This allows users to see the changes in TCP connection states and the duration of each state, helping them diagnose network issues.

In summary, the user-space part of the processing involves the following steps:

  1. Use libbpf to load and run the eBPF program.
  2. Set up a callback function to receive events sent by the kernel.
  3. Process the received events, convert them into a human-readable format, and print them.

The above is the main implementation logic of the user-space part of the tcpstates program. Through this chapter, you should have gained a deeper understanding of how to handle kernel events in user space. In the next chapter, we will introduce more knowledge about using eBPF for network monitoring.

tcprtt kernel eBPF code

In this section, we will analyze the kernel BPF code of the tcprtt eBPF program. tcprtt is a program used to measure TCP Round Trip Time (RTT) and stores the RTT information in a histogram.

c
/// @sample {"interval": 1000, "type" : "log2_hist"}
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, MAX_ENTRIES);
    __type(key, u64);
    __type(value, struct hist);
} hists SEC(".maps");

static struct hist zero;

SEC("fentry/tcp_rcv_established")
int BPF_PROG(tcp_rcv, struct sock *sk)
{
    const struct inet_sock *inet = (struct inet_sock *)(sk);
    struct tcp_sock *ts;
    struct hist *histp;
    u64 key, slot;
    u32 srtt;

    if (targ_sport && targ_sport != inet->inet_sport)
        return 0;
    if (targ_dport && targ_dport != sk->__sk_common.skc_dport)
        return 0;
    if (targ_saddr && targ_saddr != inet->inet_saddr)
        return 0;
    if (targ_daddr && targ_daddr != sk->__sk_common.skc_daddr)
        return 0;

    if (targ_laddr_hist)
        key = inet->inet_saddr;
    else if (targ_raddr_hist)
        key = inet->sk.__sk_common.skc_daddr;
    else
        key = 0;
    histp = bpf_map_lookup_or_try_init(&hists, &key, &zero);
    if (!histp)
        return 0;
    ts = (struct tcp_sock *)(sk);
    srtt = BPF_CORE_READ(ts, srtt_us) >> 3;
    if (targ_ms)
        srtt /= 1000U;
    slot = log2l(srtt);
    if (slot >= MAX_SLOTS)
        slot = MAX_SLOTS - 1;
    __sync_fetch_and_add(&histp->slots[slot], 1);
    if (targ_show_ext) {
        __sync_fetch_and_add(&histp->latency, srtt);
        __sync_fetch_and_add(&histp->cnt, 1);
    }
    return 0;
}

/* Detailed source-code truncated for AI context efficiency. */

c
void tcp_rcv_established(struct sock *sk, struct sk_buff *skb);

This function is the main function in the kernel for processing received TCP data and is called when a TCP connection is in the ESTABLISHED state. The processing logic of this function includes a fast path and a slow path. The fast path is disabled in the following cases:

  • We have advertised a zero window - zero window probing can only be handled correctly in the slow path.
  • Out-of-order data packets received.
  • Expecting to receive urgent data.
  • No remaining buffer space.
  • Received unexpected TCP flags/window values/header lengths (detected by checking TCP header against the expected flags).
  • Data is being transmitted in both directions. The fast path only supports pure senders or pure receivers (meaning the sequence number or acknowledgement value must remain unchanged).
  • Received unexpected TCP options.

When these conditions are not met, it enters a standard receive processing, which follows RFC 793 to handle all cases. The first three cases can be ensured by setting the correct expected flags, while the remaining cases require inline checks. When everything is normal, the fast processing path is invoked in the tcp_data_queue function.

Compilation and Execution

For tcpstates, you can compile and run the libbpf application with the following command:

console
$ make
...
  BPF      .output/tcpstates.bpf.o
  GEN-SKEL .output/tcpstates.skel.h
  CC       .output/tcpstates.o
  BINARY   tcpstates
$ sudo ./tcpstates 
SKADDR           PID     COMM       LADDR           LPORT RADDR           RPORT OLDSTATE    -> NEWSTATE    MS
ffff9bf61bb62bc0 164978  node       192.168.88.15   0     52.178.17.2     443   CLOSE       -> SYN_SENT    0.000
ffff9bf61bb62bc0 0       swapper/0  192.168.88.15   41596 52.178.17.2     443   SYN_SENT    -> ESTABLISHED 225.794".
"ffff9bf61bb62bc0 0       swapper/0  192.168.88.15   41596 52.178.17.2     443   ESTABLISHED -> CLOSE_WAIT  901.454
ffff9bf61bb62bc0 164978  node       192.168.88.15   41596 52.178.17.2     443   CLOSE_WAIT  -> LAST_ACK    0.793
ffff9bf61bb62bc0 164978  node       192.168.88.15   41596 52.178.17.2     443   LAST_ACK    -> LAST_ACK    0.086
ffff9bf61bb62bc0 228759  kworker/u6 192.168.88.15   41596 52.178.17.2     443   LAST_ACK    -> CLOSE       0.193
ffff9bf6d8ee88c0 229832  redis-serv 0.0.0.0         6379  0.0.0.0         0     CLOSE       -> LISTEN      0.000
ffff9bf6d8ee88c0 229832  redis-serv 0.0.0.0         6379  0.0.0.0         0     LISTEN      -> CLOSE       1.763
ffff9bf7109d6900 88750   node       127.0.0.1       39755 127.0.0.1       50966 ESTABLISHED -> FIN_WAIT1   0.000

For tcprtt, we can use eunomia-bpf to compile and run this example:

Compile:

shell
docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Or

console
$ ecc tcprtt.bpf.c tcprtt.h
Compiling bpf object...
Generating export types...
Packing ebpf object and config into package.json...

Run:

console
$ sudo ecli run package.json -h
A simple eBPF program

Usage: package.json [OPTIONS]

Options:
      --verbose                  Whether to show libbpf debug information
      --targ_laddr_hist          Set value of `bool` variable targ_laddr_hist
      --targ_raddr_hist          Set value of `bool` variable targ_raddr_hist
      --targ_show_ext            Set value of `bool` variable targ_show_ext
      --targ_sport <targ_sport>  Set value of `__u16` variable targ_sport
      --targ_dport <targ_dport>  Set value of `__u16` variable targ_dport
      --targ_saddr <targ_saddr>  Set value of `__u32` variable targ_saddr
      --targ_daddr <targ_daddr>  Set value of `__u32` variable targ_daddr
      --targ_ms                  Set value of `bool` variable targ_ms
  -h, --help                     Print help
  -V, --version                  Print version

Built with eunomia-bpf framework.".

/* Detailed source-code truncated for AI context efficiency. */

text
Complete source code:

- <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/14-tcpstates>

References:

- [tcpstates](https://github.com/iovisor/bcc/blob/master/tools/tcpstates_example.txt)
- [tcprtt](https://github.com/iovisor/bcc/blob/master/tools/tcprtt.py)
- [libbpf-tools/tcpstates](<https://github.com/iovisor/bcc/blob/master/libbpf-tools/tcpstates.bpf.c>)

## Summary

In this eBPF introductory tutorial, we learned how to use the tcpstates and tcprtt eBPF example programs to monitor and analyze the connection states and round-trip time of TCP. We understood the working principles and implementation methods of tcpstates and tcprtt, including how to store data using BPF maps, how to retrieve and process TCP connection information in eBPF programs, and how to parse and display the data collected by eBPF programs in user-space applications.

If you would like to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at <https://github.com/eunomia-bpf/bpf-developer-tutorial> or website <https://eunomia.dev/tutorials/> for more examples and complete tutorials. The upcoming tutorials will further explore advanced features of eBPF, and we will continue to share more content about eBPF development practices.

> The original link of this article: <https://eunomia.dev/tutorials/14-tcpstates>

---

### Overview (`src/15-javagc/README.md`)

# eBPF Tutorial by Example 15: Capturing User-Space Java GC Duration Using USDT

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without the need to restart the kernel or modify the kernel source code. This feature provides eBPF with high flexibility and performance, making it widely applicable in network and system performance analysis. Furthermore, eBPF also supports capturing user-space application behavior using User-Level Statically Defined Tracing (USDT).

In this article of our eBPF Tutorial by Example series, we will explore how to use eBPF and USDT to capture and analyze the duration of Java garbage collection (GC) events.

> The complete source code: <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/15-javagc>

## Introduction to USDT

USDT is a mechanism for inserting static tracepoints into applications, allowing developers to insert probes at critical points in the program for debugging and performance analysis purposes. These probes can be dynamically activated at runtime by tools such as DTrace, SystemTap, or eBPF, allowing access to the program's internal state and performance metrics without the need to restart the application or modify the program code. USDT is widely used in many open-source software applications such as MySQL, PostgreSQL, Ruby, Python, and Node.js.

### User-Level Tracing Mechanism: User-Level Dynamic Tracing and USDT

User-Level Dynamic Tracing allows us to instrument any user-level code by placing probes. For example, we can trace query requests in a MySQL server by placing a probe on the `dispatch_command()` function:

```bash
# ./uprobe 'p:cmd /opt/bin/mysqld:_Z16dispatch_command19enum_server_commandP3THDPcj +0(%dx):string'
Tracing uprobe cmd (p:cmd /opt/bin/mysqld:0x2dbd40 +0(%dx):string). Ctrl-C to end.
  mysqld-2855  [001] d... 19957757.590926: cmd: (0x6dbd40) arg1="show tables"
  mysqld-2855  [001] d... 19957759.703497: cmd: (0x6dbd40) arg1="SELECT * FROM numbers"
[...]

Here, we use the uprobe tool, which leverages Linux's built-in functionalities: ftrace (tracing framework) and uprobes (User-Level Dynamic Tracing, requires a relatively new Linux version, around 4.0 or later). Other tracing frameworks such as perf_events and SystemTap can also achieve this functionality.

Many other MySQL functions can be traced to obtain more information. We can list and count the number of these functions:

bash
# ./uprobe -l /opt/bin/mysqld | more
account_hash_get_key
add_collation
add_compiled_collation
add_plugin_noargs
adjust_time_range
[...]
# ./uprobe -l /opt/bin/mysqld | wc -l
21809

There are 21,000 functions here. We can also trace library functions or even individual instruction offsets.

User-Level Dynamic Tracing capability is very powerful and can solve numerous problems. However, using it also has some challenges: identifying the code to trace, handling function parameters, and dealing with code modifications.

User-Level Statically Defined Tracing (USDT) can address some of these challenges. USDT probes (or "markers" at the user level) are trace macros inserted at critical positions in the code, providing a stable and well-documented API. This makes the tracing work simpler.

With USDT, we can easily trace a probe called mysql:query__start instead of tracing the C++ symbol _Z16dispatch_command19enum_server_commandP3THDPcj, which is the dispatch_command() function. Of course, we can still trace dispatch_command() and the other 21,000 mysqld functions when needed, but only when USDT probes cannot solve the problem.In Linux, USDT (User Statically Defined Tracing) has actually existed in various forms for decades. It has recently gained attention again due to the popularity of Sun's DTrace tool, which has led to many common applications, including MySQL, PostgreSQL, Node.js, Java, etc., adding USDT support. SystemTap has developed a way to consume these DTrace probes.

You may be running a Linux application that already includes USDT probes, or you may need to recompile it (usually with --enable-dtrace). You can use readelf to check, for example, for Node.js:

bash
# readelf -n node
[...]
Notes at offset 0x00c43058 with length 0x00000494:
  Owner                 Data size   Description
  stapsdt              0x0000003c   NT_STAPSDT (SystemTap probe descriptors)
    Provider: node
    Name: gc__start
    Location: 0x0000000000bf44b4, Base: 0x0000000000f22464, Semaphore: 0x0000000001243028
    Arguments: 4@%esi 4@%edx 8@%rdi
[...]
  stapsdt              0x00000082       NT_STAPSDT (SystemTap probe descriptors)
    Provider: node
    Name: http__client__request
    Location: 0x0000000000bf48ff, Base: 0x0000000000f22464, Semaphore: 0x0000000001243024
    Arguments: 8@%rax 8@%rdx 8@-136(%rbp) -4@-140(%rbp) 8@-72(%rbp) 8@-80(%rbp) -4@-144(%rbp)
[...]

This is a Node.js recompiled with --enable-dtrace and installed with the systemtap-sdt-dev package that provides "dtrace" functionality to support USDT. Here are two probes displayed: node:gc__start (garbage collection start) and node:http__client__request.

At this point, you can use SystemTap or LTTng to trace these probes. However, built-in Linux tracers like ftrace and perf_events currently cannot do this (although perf_events support is under development).

Introduction to Java GC

Java, as a high-level programming language, has automatic garbage collection (GC) as one of its core features. The goal of Java GC is to automatically reclaim memory space that is no longer used by the program, thereby relieving programmers of the burden of memory management. However, the GC process may cause application pauses, which can impact program performance and response time. Therefore, monitoring and analyzing Java GC events are essential for understanding and optimizing the performance of Java applications.

In the following tutorial, we will demonstrate how to use eBPF and USDT to monitor and analyze the duration of Java GC events. We hope this content will be helpful to you in your work with eBPF for application performance analysis.

USDT in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime. bpftime is a user mode eBPF runtime based on LLVM JIT/AOT. It can run eBPF programs in user mode and is compatible with kernel mode eBPF, avoiding context switching between kernel mode and user mode, thereby improving the execution efficiency of eBPF programs by 10 times.

eBPF Implementation Mechanism

The eBPF program for Java GC is divided into two parts: kernel space and user space. We will introduce the implementation mechanisms of these two parts separately.

Kernel Space Program

text
/* Detailed source-code truncated for AI context efficiency. */

First, we define two maps:

  • data_map: This hashmap stores the start time of garbage collection for each process ID. The data_t structure contains the process ID, CPU ID, and timestamp.
  • perf_map: This is a perf event array used to send data back to the user-space program.

Then, we have four handler functions: gc_start, gc_end, and two USDT handler functions handle_mem_pool_gc_start and handle_mem_pool_gc_end. These functions are all annotated with the BPF SEC("usdt") macro to capture USDT events related to garbage collection in a Java process.

The gc_start function is called when garbage collection starts. It first gets the current CPU ID, process ID, and timestamp, and then stores this data in data_map.

The gc_end function is called when garbage collection ends. It performs similar operations as gc_start, but it also retrieves the start time from data_map and calculates the duration of garbage collection. If the duration exceeds a set threshold (time variable), it sends the data back to the user-space program.

handle_gc_start and handle_gc_end are handler functions for the garbage collection start and end events, respectively, and they call gc_start and gc_end, respectively.

handle_mem_pool_gc_start and handle_mem_pool_gc_end are handler functions for the garbage collection start and end events in the memory pool, and they also call gc_start and gc_end, respectively.Finally, we have a LICENSE array that declares the license of the BPF program, which is required for loading the BPF program.

User-space Program

The main goal of the user-space program is to load and run eBPF programs, as well as process data from the kernel-space program. This is achieved through the use of the libbpf library. Here, we are omitting some common code for loading and running eBPF programs and only showing the parts related to USDT.

The first function get_jvmso_path is used to obtain the path of the libjvm.so library for the running Java Virtual Machine (JVM). First, it opens the /proc/<pid>/maps file, which contains the memory mapping information of the process address space. Then, it searches for the line that contains libjvm.so in the file and copies the path of that line to the provided argument.

c
static int get_jvmso_path(char *path)
{
    char mode[16], line[128], buf[64];
    size_t seg_start, seg_end, seg_off;
    FILE *f;
    int i = 0;

    sprintf(buf, "/proc/%d/maps", env.pid);
    f = fopen(buf, "r");
    if (!f)
        return -1;

    while (fscanf(f, "%zx-%zx %s %zx %*s %*d%[^\n]\n",
            &seg_start, &seg_end, mode, &seg_off, line) == 5) {
        i = 0;
        while (isblank(line[i]))
            i++;
        if (strstr(line + i, "libjvm.so")) {
            break;
        }
    }

    strcpy(path, line + i);
    fclose(f);

    return 0;
}

Next, we see the attachment of the eBPF programs (handle_gc_start and handle_gc_end) to the relevant USDT probes in the Java process. Each program achieves this by calling the bpf_program__attach_usdt function, which takes as parameters the BPF program, the process ID, the binary path, and the provider and name of the probe. If the probe is successfully attached, bpf_program__attach_usdt will return a link object, which is stored in the skeleton's link member. If the attachment fails, the program will print an error message and perform cleanup.

c
skel->links.handle_mem_pool_gc_start = bpf_program__attach_usdt(skel->progs.handle_gc_start, env.pid,
                                    binary_path, "hotspot", "mem__pool__gc__begin", NULL);
    if (!skel->links.handle_mem_pool_gc_start) {
        err = errno;
        fprintf(stderr, "attach usdt mem__pool__gc__begin failed: %s\n", strerror(err));
        goto cleanup;
    }

    skel->links.handle_mem_pool_gc_end = bpf_program__attach_usdt(skel->progs.handle_gc_end, env.pid,
                                binary_path, "hotspot", "mem__pool__gc__end", NULL);
    if (!skel->links.handle_mem_pool_gc_end) {
        err = errno;
        fprintf(stderr, "attach usdt mem__pool__gc__end failed: %s\n", strerror(err));
        goto cleanup;
    }

    skel->links.handle_gc_start = bpf_program__attach_usdt(skel->progs.handle_gc_start, env.pid,
binary_path, "hotspot", "gc__begin", NULL);
    if (!skel->links.handle_gc_start) {
        err = errno;
        fprintf(stderr, "attach usdt gc__begin failed: %s\n", strerror(err));
        goto cleanup;
    }

    skel->links.handle_gc_end = bpf_program__attach_usdt(skel->progs.handle_gc_end, env.pid,
                binary_path, "hotspot", "gc__end", NULL);
    if (!skel->links.handle_gc_end) {
        err = errno;
        fprintf(stderr, "attach usdt gc__end failed: %s\n", strerror(err));
        goto cleanup;
    }

The last function handle_event is a callback function used to handle data received from the perf event array. This function is triggered by the perf event array and is called each time a new event is received. The function first converts the data to a data_t structure, then formats the current time as a string, and finally prints the timestamp, CPU ID, process ID, and duration of the garbage collection.

c
static void handle_event(void *ctx, int cpu, void *data, __u32 data_sz)
{
    struct data_t *e = (struct data_t *)data;
    struct tm *tm = NULL;
    char ts[16];
    time_t t;

    time(&t);
    tm = localtime(&t);
    strftime(ts, sizeof(ts), "%H:%M:%S", tm);
    printf("%-8s %-7d %-7d %-7lld\n", ts, e->cpu, e->pid, e->ts/1000);
}

Installing Dependencies

To build the example, you need clang, libelf, and zlib. The package names may vary with different distributions.

On Ubuntu/Debian, run the following command:

shell
sudo apt install clang libelf1 libelf-dev zlib1g-dev

On CentOS/Fedora, run the following command:

shell
sudo dnf install clang elfutils-libelf elfutils-libelf-devel zlib-devel

Compiling and Running

In the corresponding directory, run Make to compile and run the code:

console
$ make
$ sudo ./javagc -p 12345
Tracing javagc time... Hit Ctrl-C to end.
TIME     CPU     PID     GC TIME
10:00:01 10%     12345   50ms
10:00:02 12%     12345   55ms
10:00:03 9%      12345   47ms
10:00:04 13%     12345   52ms
10:00:05 11%     12345   50ms

Complete source code:

References:

Summary.Through this introductory eBPF tutorial, we have learned how to use eBPF and USDT for dynamic tracing and analysis of Java garbage collection (GC) events. We have understood how to set USDT tracepoints in user space applications and how to write eBPF programs to capture information from these tracepoints, thereby gaining a deeper understanding and optimizing the behavior and performance of Java GC.

Additionally, we have also introduced some basic knowledge and practical techniques related to Java GC, USDT, and eBPF. This knowledge and skills are valuable for developers who want to delve into the field of network and system performance analysis.

If you would like to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ to get more examples and the complete tutorial.

Overview (`src/16-memleak/README.md`)

eBPF Tutorial by Example 16: Monitoring Memory Leaks

eBPF (extended Berkeley Packet Filter) is a powerful network and performance analysis tool that is widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting the kernel or modifying its source code.

In this tutorial, we will explore how to write a Memleak program using eBPF to monitor memory leaks in programs.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/16-memleak

Background and Importance

Memory leaks are a common problem in computer programming and should not be underestimated. When memory leaks occur, programs gradually consume more memory resources without properly releasing them. Over time, this behavior can lead to a gradual depletion of system memory, significantly reducing the overall performance of the program and system.

There are many possible causes of memory leaks. It may be due to misconfiguration, such as a program incorrectly configuring dynamic allocation of certain resources. It may also be due to software bugs or incorrect memory management strategies, such as forgetting to release memory that is no longer needed during program execution. Additionally, if an application's memory usage is too high, system performance may significantly decrease due to paging/swapping, or it may even cause the application to be forcibly terminated by the system's OOM killer (Out of Memory Killer).

Challenges of Debugging Memory Leaks

Debugging memory leak issues is a complex and challenging task. This involves detailed examination of the program's configuration, memory allocation, and deallocation, often requiring specialized tools to aid in diagnosis. For example, there are tools that can associate malloc() function calls with specific detection tools, such as Valgrind memcheck, which can simulate the CPU to check all memory accesses, but may greatly slow down the application's execution speed. Another option is to use heap analyzers, such as libtcmalloc, which are relatively faster but may still decrease the application's execution speed by more than five times. Additionally, there are tools like gdb that can obtain core dumps of applications and perform post-processing analysis of memory usage. However, these tools often require pausing the application during core dump acquisition or calling the free() function after the application terminates.

Role of eBPF

In this context, the role of eBPF becomes particularly important. eBPF provides an efficient mechanism for monitoring and tracking system-level events, including memory allocation and deallocation. With eBPF, we can trace memory allocation and deallocation requests and collect the call stacks for each allocation. We can then analyze this information to identify call stacks that perform memory allocations but do not perform subsequent deallocations, helping us identify the source of memory leaks. The advantage of this approach is that it can be done in real-time within a running application without pausing the application or performing complex post-processing.

The memleak eBPF tool can trace and match memory allocation and deallocation requests, and collect the call stacks for each allocation. Subsequently, memleak can print a summary indicating which call stacks executed allocations but did not perform subsequent deallocations. For example, running the command:

console
# ./memleak -p $(pidof allocs)
Attaching to pid 5193, Ctrl+C to quit.
[11:16:33] Top 2 stacks with outstanding allocations:
        80 bytes in 5 allocations from stack
                 main+0x6d [allocs]
                 __libc_start_main+0xf0 [libc-2.21.so]

[11:16:34] Top 2 stacks with outstanding allocations:
        160 bytes in 10 allocations from stack
                 main+0x6d [allocs]
                 __libc_start_main+0xf0 [libc-2.21.so]

After running this command, we can see which stacks the allocated but not deallocated memory came from, as well as the size and quantity of these unreleased memory blocks.

Over time, it becomes evident that the main function of the allocs process is leaking memory, 16 bytes at a time. Fortunately, we don't need to inspect each allocation; we have a nice summary that tells us which stack is responsible for the significant leaks.

Implementation Principle of memleak

At a basic level, memleak operates by installing monitoring devices on the memory allocation and deallocation paths. It achieves this by inserting eBPF programs into memory allocation and deallocation functions. This means that when these functions are called, memleak will record important information, such as the caller's process ID (PID), the allocated memory address, and the size of the allocated memory. When the function for freeing memory is called, memleak will delete the corresponding memory allocation record in its internal map. This mechanism allows memleak to accurately trace which memory blocks have been allocated but not deallocated.For commonly used memory allocation functions in user space, such as malloc and calloc, memleak uses user space probing (uprobe) technology for monitoring. Uprobe is a dynamic tracing technology for user space applications, which can set breakpoints at any location at runtime without modifying the binary files, thus achieving tracing of specific function calls.

Uprobe in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime. bpftime is a user mode eBPF runtime based on LLVM JIT/AOT. It can run eBPF programs in user mode and is compatible with kernel mode eBPF, avoiding context switching between kernel mode and user mode, thereby improving the execution efficiency of eBPF programs by 10 times.

For kernel space memory allocation functions, such as kmalloc, memleak chooses to use tracepoints for monitoring. Tracepoint is a dynamic tracing technology provided in the Linux kernel, which can dynamically trace specific events in the kernel at runtime without recompiling the kernel or loading kernel modules.

Kernel Space eBPF Program Implementation

`memleak` Kernel Space eBPF Program Implementation

The kernel space eBPF program of memleak contains some key functions for tracking memory allocation and deallocation. Before delving into these functions, let's first take a look at some data structures defined by memleak, which are used in both its kernel space and user space programs.

c
#ifndef __MEMLEAK_H
#define __MEMLEAK_H

#define ALLOCS_MAX_ENTRIES 1000000
#define COMBINED_ALLOCS_MAX_ENTRIES 10240

struct alloc_info {
    __u64 size;            // Size of allocated memory
    __u64 timestamp_ns;    // Timestamp when allocation occurs, in nanoseconds
    int stack_id;          // Call stack ID when allocation occurs
};

union combined_alloc_info {
    struct {
        __u64 total_size : 40;        // Total size of all unreleased allocations
        __u64 number_of_allocs : 24;   // Total number of unreleased allocations
    };
    __u64 bits;    // Bitwise representation of the structure
};

#endif /* __MEMLEAK_H */

Here, two main data structures are defined: alloc_info and combined_alloc_info.

The alloc_info structure contains basic information about a memory allocation, including the allocated memory size size, the timestamp timestamp_ns when the allocation occurs, and the call stack ID stack_id that triggers the allocation.

The combined_alloc_info is a union that contains an embedded structure and a __u64 type bitwise representation bits. The embedded structure has two members: total_size and number_of_allocs, representing the total size and total count of unreleased allocations, respectively. The numbers 40 and 24 indicate the number of bits occupied by the total_size and number_of_allocs members, limiting their size. By using this limitation, storage space for the combined_alloc_info structure can be saved. Moreover, since total_size and number_of_allocs share the same unsigned long long type variable bits for storage, bitwise operations on the member variable bits can be used to access and modify total_size and number_of_allocs, avoiding the complexity of defining additional variables and functions in the program.

Next, memleak defines a series of eBPF maps for storing memory allocation information and analysis results. These maps are defined in the form of SEC(".maps"), indicating that they belong to the mapping section of the eBPF program.

c
const volatile size_t min_size = 0;
const volatile size_t max_size = -1;
const volatile size_t page_size = 4096;
const volatile __u64 sample_rate = 1;
const volatile bool trace_all = false;
const volatile __u64 stack_flags = 0;
const volatile bool wa_missing_free = false;

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, pid_t);
    __type(value, u64);
    __uint(max_entries, 10240);
} sizes SEC(".maps");

struct {
  //... (continued)__uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u64); /* address */
    __type(value, struct alloc_info);
    __uint(max_entries, ALLOCS_MAX_ENTRIES);
} allocs SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u64); /* stack id */
    __type(value, union combined_alloc_info);
    __uint(max_entries, COMBINED_ALLOCS_MAX_ENTRIES);
} combined_allocs SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u64);
    __type(value, u64);
    __uint(max_entries, 10240);
} memptrs SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_STACK_TRACE);
    __type(key, u32);
} stack_traces SEC(".maps");

static union combined_alloc_info initial_cinfo;

The code first defines some configurable parameters, such as min_size, max_size, page_size, sample_rate, trace_all, stack_flags, and wa_missing_free, representing the minimum allocation size, maximum allocation size, page size, sample rate, whether to trace all allocations, stack flags, and whether to work in missing free mode.

Then, five maps are defined:

  1. sizes: This is a hash-type map with the key as the process ID and the value as u64 type, storing the allocation size of each process.
  2. allocs: This is also a hash-type map with the key as the allocation address and the value as the alloc_info structure, storing detailed information about each memory allocation.
  3. combined_allocs: This is another hash-type map with the key as the stack ID and the value as the combined_alloc_info union, storing the total size and count of all unreleased allocations.
  4. memptrs: This is also a hash-type map with both the key and value as u64 type, used to pass memory pointers between user space and kernel space.
  5. stack_traces: This is a stack trace-type map with the key as u32 type, used to store stack IDs.

Taking the user-space memory allocation tracing as an example, it mainly hooks memory-related function calls such as malloc, free, calloc, realloc, mmap, and munmap to record data when these functions are called. In user space, memleak mainly uses uprobes technology for hooking.

Each function call is divided into "enter" and "exit" parts. The "enter" part records the function call parameters, such as the size of the allocation or the address being freed. The "exit" part is mainly used to obtain the return value of the function, such as the memory address obtained from the allocation.

Here, gen_alloc_enter, gen_alloc_exit, gen_free_enter are functions that implement the recording behavior, and they are used to record relevant information when allocation starts, allocation ends, and freeing starts, respectively.

The function prototype is as follows:

c
SEC("uprobe")
int BPF_KPROBE(malloc_enter, size_t size)
{
    // Record relevant information when allocation starts
    return gen_alloc_enter(size);
}

SEC("uretprobe")
int BPF_KRETPROBE(malloc_exit)
{
    // Record relevant information when allocation ends
    return gen_alloc_exit(ctx);
}

SEC("uprobe")
int BPF_KPROBE(free_enter, void *address)
{
    // Record relevant information when freeing starts
    return gen_free_enter(address);
}

malloc_enter and free_enter are probes mounted at the entry points of the malloc and free functions, respectively, to record data during function calls. malloc_exit is a probe mounted at the return point of the malloc function to record the return value of the function.

These functions are declared using the BPF_KPROBE and BPF_KRETPROBE macros, which are used to declare kprobes (kernel probes) and kretprobes (kernel return probes), respectively. Specifically, kprobe is triggered during function calls, while kretprobe is triggered during function returns.

The gen_alloc_enter function is called at the beginning of a memory allocation request. This function is mainly responsible for collecting some basic information when the function that allocates memory is called. Now, let's take a deep dive into the implementation of this function.

c
static int gen_alloc_enter(size_t size)
{
    if (size < min_size || size > max_size)
        return 0;

    if (sample_rate > 1) {
        if (bpf_ktime_get_ns() % sample_rate != 0)
            return 0;
    }

    const pid_t pid = bpf_get_current_pid_tgid() >> 32;
    bpf_map_update_elem(&sizes, &pid, &size, BPF_ANY);

    if (trace_all)
        bpf_printk("alloc entered, size = %lu\n", size);

    return 0;
}

SEC("uprobe")
int BPF_KPROBE(malloc_enter, size_t size)
{
    return gen_alloc_enter(size);
}

First, the gen_alloc_enter function takes a size parameter that represents the size of the requested memory allocation. If this value is not between min_size and max_size, the function will return directly without performing any further operations. This allows the tool to focus on tracing memory allocation requests within a specific range and filter out uninteresting allocation requests.

Next, the function checks the sampling rate sample_rate. If sample_rate is greater than 1, it means that we don't need to trace all memory allocation requests, but rather trace them periodically. Here, bpf_ktime_get_ns is used to get the current timestamp, and the modulus operation is used to determine whether to trace the current memory allocation request. This is a common sampling technique used to reduce performance overhead while providing a representative sample for analysis.

Then, the function uses the bpf_get_current_pid_tgid function to retrieve the current process's PID. Note that the PID here is actually a combination of the process ID and thread ID, and we shift it right by 32 bits to get the actual process ID.

The function then updates the sizes map, which uses the process ID as the key and the requested memory allocation size as the value. BPF_ANY indicates that if the key already exists, the value will be updated; otherwise, a new entry will be created.

Finally, if the trace_all flag is enabled, the function will print a message indicating that a memory allocation has occurred.

The BPF_KPROBE macro is used to intercept the execution of the malloc function with a BPF uprobe when the malloc_enter function is called, and it records the memory allocation size using gen_alloc_enter.
We have just analyzed the entry function gen_alloc_enter of memory allocation, now let's focus on the exit part of this process. Specifically, we will discuss the gen_alloc_exit2 function and how to obtain the returned memory address from the memory allocation call.

c
static int gen_alloc_exit2(void *ctx, u64 address)
{
    const pid_t pid = bpf_get_current_pid_tgid() >> 32;
    struct alloc_info info;

    const u64* size = bpf_map_lookup_elem(&sizes, &pid);
    if (!size)
        return 0; // missed alloc entry

    __builtin_memset(&info, 0, sizeof(info));

    info.size = *size;bpf_map_delete_elem(&sizes, &pid);

    if (address != 0) {
        info.timestamp_ns = bpf_ktime_get_ns();

        info.stack_id = bpf_get_stackid(ctx, &stack_traces, stack_flags);

        bpf_map_update_elem(&allocs, &address, &info, BPF_ANY);

        update_statistics_add(info.stack_id, info.size);
    }

    if (trace_all) {
        bpf_printk("alloc exited, size = %lu, result = %lx\n",
                info.size, address);
    }

    return 0;
}

static int gen_alloc_exit(struct pt_regs *ctx)
{
    return gen_alloc_exit2(ctx, PT_REGS_RC(ctx));
}

SEC("uretprobe")
int BPF_KRETPROBE(malloc_exit)
{
    return gen_alloc_exit(ctx);
}

gen_alloc_exit2 function is called when the memory allocation operation is completed. This function takes two parameters, one is the context ctx and the other is the memory address returned by the memory allocation function address.

First, it obtains the PID (Process ID) of the current thread and uses it as a key to look up the corresponding memory allocation size in the sizes map. If not found (i.e., no entry for the memory allocation operation), the function simply returns.

Then, it clears the content of the info structure and sets its size field to the memory allocation size found in the map. It also removes the corresponding element from the sizes map because the memory allocation operation has completed and this information is no longer needed.

Next, if address is not zero (indicating a successful memory allocation operation), the function further collects some additional information. First, it obtains the current timestamp as the completion time of the memory allocation and fetches the current stack trace. These pieces of information are stored in the info structure and subsequently updated in the allocs map.

Finally, the function calls update_statistics_add to update the statistics data and, if tracing of all memory allocation operations is enabled, it prints some information about the memory allocation operation.

Note that, gen_alloc_exit is a wrapper for gen_alloc_exit2, which passes PT_REGS_RC(ctx) as the address parameter to gen_alloc_exit2.

In our discussion, we just mentioned that update_statistics_add function is called in the gen_alloc_exit2 function to update the statistics data for memory allocations. Now let's take a closer look at the implementation of this function.

c
static void update_statistics_add(u64 stack_id, u64 sz)
{
    union combined_alloc_info *existing_cinfo;

    existing_cinfo = bpf_map_lookup_or_try_init(&combined_allocs, &stack_id, &initial_cinfo);
    if (!existing_cinfo)
        return;

    const union combined_alloc_info incremental_cinfo = {
        .total_size = sz,
        .number_of_allocs = 1
    };

    __sync_fetch_and_add(&existing_cinfo->bits, incremental_cinfo.bits);
}

The update_statistics_add function takes two parameters: the current stack ID stack_id and the size of the memory allocation sz. These two parameters are collected in the memory allocation event and used to update the statistics data for memory allocations.First, the function tries to find the element with the current stack ID as the key in the combined_allocs map. If it is not found, a new element is initialized with initial_cinfo (which is a default combined_alloc_info structure with all fields set to zero).

Next, the function creates an incremental_cinfo and sets its total_size to the current memory allocation size and number_of_allocs to 1. This is because each call to the update_statistics_add function represents a new memory allocation event, and the size of this event's memory allocation is sz.

Finally, the function atomically adds the value of incremental_cinfo to existing_cinfo using the __sync_fetch_and_add function. Note that this step is thread-safe, so even if multiple threads call the update_statistics_add function concurrently, each memory allocation event will be correctly recorded in the statistics.

In summary, the update_statistics_add function implements the logic for updating memory allocation statistics. By maintaining the total amount and number of memory allocations for each stack ID, we can gain insight into the memory allocation behavior of the program.

In our process of tracking memory allocation statistics, we not only need to count memory allocations but also consider memory releases. In the above code, we define a function called update_statistics_del that updates the statistics when memory is freed. The function gen_free_enter is executed when the process calls the free function.

The update_statistics_del function takes the stack ID and the size of the memory block to be freed as parameters. First, the function uses the current stack ID as the key to look up the corresponding combined_alloc_info structure in the combined_allocs map. If it is not found, an error message is output and the function returns. If it is found, a decremental_cinfo combined_alloc_info structure is constructed with its total_size set to the size of the memory to be freed and number_of_allocs set to 1. Then the __sync_fetch_and_sub function is used to atomically subtract the value of decremental_cinfo from existing_cinfo. Note that the number_of_allocs here is negative, indicating a decrease in memory allocation.

The gen_free_enter function takes the address to be freed as a parameter. It first converts the address to an unsigned 64-bit integer (u64). Then it looks up the alloc_info structure in the allocs map using the address as the key. If it is not found, the function returns 0. If it is found, the alloc_info structure is deleted from the allocs map, and the update_statistics_del function is called with the stack ID and size from info. If trace_all is true, an information message is output.

c
int BPF_KPROBE(free_enter, void *address)
{
    return gen_free_enter(address);
}

Next, let's look at the gen_free_enter function. It takes an address as a parameter, which is the result of memory allocation, i.e., the starting address of the memory to be freed. The function first uses this address as a key to search for the corresponding alloc_info structure in the allocs map. If it is not found, it simply returns because it means that this address has not been allocated. If it is found, the element is deleted, and the update_statistics_del function is called to update the statistics data. Finally, if global tracking is enabled, a message is also output, including this address and its size.

While tracking and profiling memory allocation, we also need to track kernel-mode memory allocation and deallocation. In the Linux kernel, the kmem_cache_alloc function and the kfree function are used for kernel-mode memory allocation and deallocation, respectively.

c
SEC("tracepoint/kmem/kfree")
int memleak__kfree(void *ctx)
{
    const void *ptr;

    if (has_kfree()) {
        struct trace_event_raw_kfree___x *args = ctx;
        ptr = BPF_CORE_READ(args, ptr);
    } else {
        struct trace_event_raw_kmem_free___x *args = ctx;
        ptr = BPF_CORE_READ(args, ptr);
    }

    return gen_free_enter(ptr);
}

The above code snippet defines a function memleak__kfree. This is a BPF program that will be executed when the kfree function is called in the kernel. First, the function checks if kfree exists. If it does, it reads the argument passed to the kfree function (i.e., the address of the memory block to be freed) and saves it in the variable ptr. Otherwise, it reads the argument passed to the kmem_free function (i.e., the address of the memory block to be freed) and saves it in the variable ptr. Then, the function calls the previously defined gen_free_enter function to handle the release of this memory block.

c
SEC("tracepoint/kmem/kmem_cache_alloc")
int memleak__kmem_cache_alloc(struct trace_event_raw_kmem_alloc *ctx)
{
    if (wa_missing_free)
        gen_free_enter(ctx->ptr);

    gen_alloc_enter(ctx->bytes_alloc);

    return gen_alloc_exit2(ctx, (u64)(ctx->ptr));
}

This code snippet defines a function memleak__kmem_cache_alloc. This is also a BPF program that will be executed when the kmem_cache_alloc function is called in the kernel. If the wa_missing_free flag is set, it calls the gen_free_enter function to handle possible missed release operations. Then, the function calls the gen_alloc_enter function to handle memory allocation and finally calls the gen_alloc_exit2 function to record the allocation result.

Both of these BPF programs use the SEC macro to define the corresponding tracepoints, so that they can be executed when the corresponding kernel functions are called. In the Linux kernel, a tracepoint is a static hook that can be inserted into the kernel to collect runtime kernel information. It is very useful for debugging and performance analysis.

In the process of understanding this code, pay attention to the use of the BPF_CORE_READ macro. This macro is used to read kernel data in BPF programs. In BPF programs, we cannot directly access kernel memory and need to use such macros to safely read data.

User-Space Program

After understanding the BPF kernel part, let's switch to the user-space program. The user-space program works closely with the BPF kernel program. It is responsible for loading BPF programs into the kernel, setting up and managing BPF maps, and handling data collected from BPF programs. The user-space program is longer, but here we can briefly refer to its mount point.

c
int attach_uprobes(struct memleak_bpf *skel)
{
    ATTACH_UPROBE_CHECKED(skel, malloc, malloc_enter);
        ATTACH_URETPROBE_CHECKED(skel, malloc, malloc_exit);
    ATTACH_UPROBE_CHECKED(skel, calloc, calloc_enter);
    ATTACH_URETPROBE_CHECKED(skel, calloc, calloc_exit);

    ATTACH_UPROBE_CHECKED(skel, realloc, realloc_enter);
    ATTACH_URETPROBE_CHECKED(skel, realloc, realloc_exit);

    ATTACH_UPROBE_CHECKED(skel, mmap, mmap_enter);
    ATTACH_URETPROBE_CHECKED(skel, mmap, mmap_exit);

    ATTACH_UPROBE_CHECKED(skel, posix_memalign, posix_memalign_enter);
    ATTACH_URETPROBE_CHECKED(skel, posix_memalign, posix_memalign_exit);

    ATTACH_UPROBE_CHECKED(skel, memalign, memalign_enter);
    ATTACH_URETPROBE_CHECKED(skel, memalign, memalign_exit);

    ATTACH_UPROBE_CHECKED(skel, free, free_enter);
    ATTACH_UPROBE_CHECKED(skel, munmap, munmap_enter);

    // the following probes are intentionally allowed to fail attachment

    // deprecated in libc.so bionic
    ATTACH_UPROBE(skel, valloc, valloc_enter);
    ATTACH_URETPROBE(skel, valloc, valloc_exit);

    // deprecated in libc.so bionic
    ATTACH_UPROBE(skel, pvalloc, pvalloc_enter);
    ATTACH_URETPROBE(skel, pvalloc, pvalloc_exit);

    // added in C11
    ATTACH_UPROBE(skel, aligned_alloc, aligned_alloc_enter);
    ATTACH_URETPROBE(skel, aligned_alloc, aligned_alloc_exit);

    return 0;
}

In this code snippet, we see a function called attach_uprobes that mounts uprobes (user space probes) onto memory allocation and deallocation functions. In Linux, uprobes are a kernel mechanism that allows setting breakpoints at arbitrary locations in user space programs, enabling precise observation and control over the behavior of user space programs.

Here, each memory-related function is traced using two uprobes: one at the entry (enter) of the function and one at the exit. Thus, every time these functions are called or return, a uprobes event is triggered, which in turn triggers the corresponding BPF program.

In the actual implementation, we use two macros, ATTACH_UPROBE and ATTACH_URETPROBE, to attach uprobes and uretprobes (function return probes), respectively. Each macro takes three arguments: the skeleton of the BPF program (skel), the name of the function to monitor, and the name of the BPF program to trigger.

These mount points include common memory allocation functions such as malloc, calloc, realloc, mmap, posix_memalign, memalign, free, and their corresponding exit points. Additionally, we also observe some possible allocation functions such as valloc, pvalloc, aligned_alloc, although they may not always exist.

The goal of these mount points is to capture all possible memory allocation and deallocation events, allowing our memory leak detection tool to obtain as comprehensive data as possible. This approach enables us to track not only memory allocation and deallocation but also their contextual information such as call stacks and invocation counts, helping us to pinpoint and fix memory leak issues.

Note that some memory allocation functions may not exist or may have been deprecated, such as valloc and pvalloc. Thus, their attachment may fail. In such cases, we allow for attachment failures, which do not prevent the program from executing. This is because we are more focused on mainstream and commonly used memory allocation functions, while these deprecated functions are often used less frequently in practical applications.

Complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/16-memleak

Reference: https://github.com/iovisor/bcc/blob/master/libbpf-tools/memleak.c

Compile and Run

This implementation uses the latest blazesym v0.2.0 library for symbol resolution, which provides improved performance and a modern C API through the capi package.

console
$ make
$ sudo ./memleak
using default object: libc.so.6
using page size: 4096
tracing kernel: true
Tracing outstanding memory allocs...  Hit Ctrl-C to end
[17:17:27] Top 10 stacks with outstanding allocations:
1236992 bytes in 302 allocations from stack
        0 [<ffffffff812c8f43>] <null sym>
        1 [<ffffffff812c8f43>] <null sym>
        2 [<ffffffff812a9d42>] <null sym>
        3 [<ffffffff812aa392>] <null sym>
        4 [<ffffffff810df0cb>] <null sym>
        5 [<ffffffff81edc3fd>] <null sym>
        6 [<ffffffff82000b62>] <null sym>
...

Testing memleak

This repository includes a test program (test_memleak.c) that intentionally leaks memory for testing purposes. You can build and run it to verify that memleak correctly detects memory leaks:

console
$ make test_memleak
$ ./test_memleak &
$ sudo ./memleak -p $(pidof test_memleak)
using default object: libc.so.6
using page size: 4096
tracing kernel: false
Tracing outstanding memory allocs...  Hit Ctrl-C to end
[19:31:49] Top 1 stacks with outstanding allocations:
10240 bytes in 5 allocations from stack
	0 [<00005a7ea0a34212>] leak_with_loop+0x1f
	1 [<00005a7ea0a3428e>] main+0x6e
	2 [<00007b4ea482a1ca>] <null sym>
	3 [<00007b4ea482a28b>] __libc_start_main+0x8b
	4 [<00005a7ea0a34105>] _start+0x25

As shown above, memleak successfully:

  • Detected 10240 bytes in 5 allocations (5 Γ— 2048 bytes) that were leaked
  • Identified the leaking function leak_with_loop+0x1f with the correct offset
  • Provided the complete call stack showing the leak originated from main function
  • Used the new blazesym v0.2.0 C API for fast and accurate symbol resolution

The test demonstrates that memleak with the updated blazesym library can effectively trace memory allocations and pinpoint the exact functions responsible for memory leaks.

Summary

Through this eBPF introductory tutorial, you have learned how to write a Memleak eBPF monitoring program to monitor memory leaks in real time. You have also learned about the application of eBPF in memory monitoring, how to write eBPF programs using the BPF API, create and use eBPF maps, and how to use eBPF tools to monitor and analyze memory leak issues. We have provided a detailed example to help you understand the execution flow and principles of eBPF code.

You can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

The original link of this article: https://eunomia.dev/tutorials/16-memleak

Overview (`src/17-biopattern/README.md`)

eBPF Tutorial by Example 17: Count Random/Sequential Disk I/O

eBPF (Extended Berkeley Packet Filter) is a new technology in the Linux kernel that allows users to execute custom programmes in kernel space without changing the kernel code. This provides system administrators and developers with powerful tools to gain insight into and monitor system behaviour for optimisation.

In this tutorial, we will explore how to use eBPF to write programs to count random and sequential disk I/O. Disk I/O is one of the key metrics of computer performance, especially in data-intensive applications.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/17-biopattern

Random/Sequential Disk I/O

As technology advances and data volumes explode, disk I/O becomes a critical bottleneck in system performance. The performance of an application depends heavily on how it interacts with the storage tier. Therefore, it becomes especially important to deeply understand and optimise disk I/O, especially random and sequential I/O.

  1. Random I/O: Random I/O occurs when an application reads or writes data from or to a non-sequential location on the disk. The main characteristic of this I/O mode is that the disk head needs to move frequently between locations, causing it to be typically slower than sequential I/O. Typical scenarios that generate random I/O include database queries, file system metadata operations, and concurrent tasks in virtualised environments.

  2. Sequential I/O: In contrast to random I/O, sequential I/O occurs when an application continuously reads or writes blocks of data to or from disk. The advantage of this I/O mode is that the disk head can move continuously in one direction, which greatly increases the speed at which data can be read and written. Video playback, downloading or uploading large files, and continuous logging are typical applications that generate sequential I/O.

To optimise storage performance, it is critical to understand both random and sequential disk I/O. For example, random I/O-sensitive applications typically perform far better on SSDs than on traditional hard drives because SSDs have virtually no addressing latency when dealing with random I/Os. Conversely, for applications with a lot of sequential I/O, it is much more critical to maximize the sequential read and write speed of the disk.

In the rest of this tutorial, we will discuss in detail how to use the eBPF tool to monitor and count both types of disk I/O in real time, which will not only help us better understand the I/O behaviour of the system, but will also provide us with strong data for further performance optimization.

Biopattern

Biopattern counts the percentage of random/sequential disk I/Os.

First of all, make sure that you have installed libbpf and the associated toolset correctly, you can find the source code here: bpf-developer-tutorial

Navigate to the biopattern source directory and compile it using the make command:

bash
cd ~/bpf-developer-tutorial/src/17-biopattern
make

After successful compilation, you should see the biopattern executable in the current directory. The basic runtime commands are as follows:

bash
sudo ./biopattern [interval] [count]

For example, to print the output once per second for 10 seconds, you can run:

console
$ sudo ./biopattern 1 10
Tracing block device I/O requested seeks... Hit Ctrl-C to end.
DISK     %RND  %SEQ    COUNT     KBYTES
sr0         0   100        3          0
sr1         0   100        8          0
sda         0   100        1          4
sda       100     0       26        136
sda         0   100        1          4

The output columns have the following meanings:

  • DISK: Name of the disk being tracked.
  • %RND: Percentage of random I/O.
  • %SEQ: percentage of sequential I/O.
  • COUNT: Number of I/O requests in the specified interval.
  • KBYTES: amount of data (in KB) read and written in the specified time interval.

From the above output, we can draw the following conclusions:

  • The sr0 and sr1 devices performed mostly sequential I/O during the observation period, but the amount of data was small.
  • The sda device performed only random I/O during some time periods and only sequential I/O during other time periods.

This information can help us understand the I/O pattern of the system so that we can target optimisation.

eBPF Biopattern Implementation Principles

First, let's look at the eBPF kernel state code at the heart of biopattern:

text
/* Detailed source-code truncated for AI context efficiency. */

Global variable definitions:

c
const volatile bool filter_dev = false; 
   const volatile __u32 targ_dev = 0;

These two global variables are used for device filtering. filter_dev determines whether device filtering is enabled or not, and targ_dev is the identifier of the target device we want to track.

BPF map definition:

c
struct { __uint(type, BPF_MAP_TYPE_HASH); 
            __uint(max_entries, 64); __type(key, u32); 
            __type(value, struct counter); 
    } counters SEC(".maps").

This part of the code defines a BPF map of type hash table. The key of the map is the identifier of the device, and the value is a counter struct, which is used to store the I/O statistics of the device.

The tracepoint function:

c
SEC("tracepoint/block/block_rq_complete")
    int handle__block_rq_complete(void *args)
    {
        struct counter *counterp, zero = {};
        sector_t sector;
        u32 nr_sector;
        u32 dev;

        if (has_block_rq_completion()) {
            struct trace_event_raw_block_rq_completion___x *ctx = args;
            sector = BPF_CORE_READ(ctx, sector);
            nr_sector = BPF_CORE_READ(ctx, nr_sector);
            dev = BPF_CORE_READ(ctx, dev);
        } else {
            struct trace_event_raw_block_rq_complete___x *ctx = args;
            sector = BPF_CORE_READ(ctx, sector);
            nr_sector = BPF_CORE_READ(ctx, nr_sector);
            dev = BPF_CORE_READ(ctx, dev);
        }

        if (filter_dev && targ_dev != dev)
            return 0;

        counterp = bpf_map_lookup_or_try_init(&counters, &dev, &zero);
        if (!counterp)
            return 0;
        if (counterp->last_sector) {
            if (counterp->last_sector == sector)
                __sync_fetch_and_add(&counterp->sequential, 1);
            else
                __sync_fetch_and_add(&counterp->random, 1);
            __sync_fetch_and_add(&counterp->bytes, nr_sector * 512);
        }
        counterp->last_sector = sector + nr_sector;
        return 0;
    }

In Linux, a trace point called block_rq_complete is triggered every time an I/O request for a block device completes. This provides an opportunity to capture these events with eBPF and further analyse the I/O patterns.

Main Logic Analysis:

  • Extracting I/O request information: get information about the I/O request from the incoming parameters. There are two possible context structures depending on the return value of has_block_rq_completion. This is because different versions of the Linux kernel may have different tracepoint definitions. In either case, we extract the sector number (sector), the number of sectors (nr_sector) and the device identifier (dev) from the context.

  • Device filtering: If device filtering is enabled (filter_dev is true ) and the current device is not the target device (targ_dev ), it is returned directly. This allows the user to track only specific devices, not all devices.

  • Statistics update:

    • Lookup or initialise statistics: use the bpf_map_lookup_or_try_init function to lookup or initialise statistics related to the current device. If there is no statistics for the current device in the map, it will be initialised using the zero structure.

  • Determine the I/O mode: Based on the sector number of the current I/O request and the previous I/O request, we can determine whether the current request is random or sequential. If the sector numbers of the two requests are the same, then it is sequential; otherwise, it is random. We then use the __sync_fetch_and_add function to update the corresponding statistics. This is an atomic operation that ensures data consistency in a concurrent environment.

  • Update the amount of data: we also update the total amount of data for the device, which is done by multiplying the number of sectors (nr_sector ) by 512 (the number of bytes per sector).

  • Update the sector number of the last I/O request: for the next comparison, we update the value of last_sector.

In some versions of the Linux kernel, the naming and structure of the tracepoint has changed due to the introduction of a new tracepoint, block_rq_error. This means that the structural name of the former block_rq_complete tracepoint has been changed from trace_event_raw_block_rq_complete to trace_event_raw_block_rq_completion, a change which may cause compatibility issues with eBPF programs on different versions of the kernel. This change may cause compatibility issues with eBPF programs on different versions of the kernel.

To address this issue, the biopattern utility introduces a mechanism to dynamically detect which trace point structure is currently used by the kernel, namely the has_block_rq_completion function.

  1. Define two trace point structures:
c
struct trace_event_raw_block_rq_complete___x {
        dev_t dev;
        sector_t sector;
        unsigned int nr_sector;
    } __attribute__((preserve_access_index));

    struct trace_event_raw_block_rq_completion___x {
        dev_t dev;
        sector_t sector;
        unsigned int nr_sector;
    } __attribute__((preserve_access_index));

Two tracepoint structures are defined here, corresponding to different versions of the kernel. Each structure contains a device identifier (dev ), sector number (sector ), and number of sectors (nr_sector ).

Dynamic detection of trackpoint structures:

c
static __always_inline bool has_block_rq_completion()
    {
        if (bpf_core_type_exists(struct trace_event_raw_block_rq_completion___x))
            return true;
        return false;
    }

The has_block_rq_completion function uses the bpf_core_type_exists function to detect the presence of the structure trace_event_raw_block_rq_completion___x in the current kernel. If it exists, the function returns true, indicating that the current kernel is using the new tracepoint structure; otherwise, it returns false, indicating that it is using the old structure. The two different definitions are handled separately in the corresponding eBPF code, which is a common solution for adapting to changes between kernel versions.

User State Code

The biopattern tool's userland code is responsible for reading statistics from the BPF mapping and presenting them to the user. In this way, system administrators can monitor the I/O patterns of each device in real time to better understand and optimise the I/O performance of the system.

  1. Main loop
c
/* main: poll */
    while (1) {
        sleep(env.interval);

        err = print_map(obj->maps.counters, partitions);
        if (err)
            break;

        if (exiting || --env.times == 0)
            break;
    }

This is the main loop of the biopattern utility, and its workflow is as follows:

  • Wait: use the sleep function to wait for the specified interval (env.interval ).
  • print_map: call print_map function to print the statistics in BPF map.
  • Exit condition: if an exit signal is received (exiting is true ) or if the specified number of runs is reached (env.times reaches 0), the loop exits.

Print mapping function:

text
/* Detailed source-code truncated for AI context efficiency. */

The print_map function is responsible for reading statistics from the BPF map and printing them to the console. The main logic is as follows:

  • Traverse the BPF map: Use the bpf_map_get_next_key and bpf_map_lookup_elem functions to traverse the BPF map and get the statistics for each device.
  • Calculate totals: Calculate the total number of random and sequential I/Os for each device.
  • Print statistics: If timestamp is enabled (env.timestamp is true ), the current time is printed first. Next, the device name, percentage of random I/O, percentage of sequential I/O, total I/O, and total data in KB are printed.
  • Cleaning up the BPF map: For the next count, use the bpf_map_get_next_key and bpf_map_delete_elem functions to clean up all entries in the BPF map.

Summary

In this tutorial, we have taken an in-depth look at how to use the eBPF tool biopattern to monitor and count random and sequential disk I/O in real-time. we started by understanding the importance of random and sequential disk I/O and their impact on system performance. We then describe in detail how biopattern works, including how to define and use BPF maps, how to deal with tracepoint variations in different versions of the Linux kernel, and how to capture and analyse disk I/O events in an eBPF program.

You can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/zh/tutorials/ for more examples and a complete tutorial.

The original link of this article: https://eunomia.dev/tutorials/17-biopattern

Overview (`src/18-further-reading/README.md`)

More Reference Materials： papers, projects

You may find more about eBPF in these places:

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/18-further-reading

This is also list of eBPF related papers I read in recent years, might be helpful for people who are interested in eBPF related research.

eBPF (extended Berkeley Packet Filter) is an emerging technology that allows safe execution of user-provided programs in the Linux kernel. It has gained widespread adoption in recent years for accelerating network processing, enhancing observability, and enabling programmable packet processing.

This document list some key research papers on eBPF over the past few years. The papers cover several aspects of eBPF, including accelerating distributed systems, storage, and networking, formally verifying the eBPF JIT compiler and verifier, applying eBPF for intrusion detection, and automatically generating hardware designs from eBPF programs.

Some key highlights:

  • eBPF enables executing custom functions in the kernel to accelerate distributed protocols, storage engines, and networking applications with improved throughput and lower latency compared to traditional userspace implementations.
  • Formal verification of eBPF components like JIT and verifier ensures correctness and reveals bugs in real-world implementations.
  • eBPF's programmability and efficiency make it suitable for building intrusion detection and network monitoring applications entirely in the kernel.
  • Automated synthesis of hardware designs from eBPF programs allows software developers to quickly generate optimized packet processing pipelines in network cards.

The papers demonstrate eBPF's versatility in accelerating systems, enhancing security, and simplifying network programming. As eBPF adoption grows, it is an important area of systems research with many open problems related to performance, safety, hardware integration, and ease of use.

If you have any suggestions or adding papers, please feel free to open an issue or PR. The list was created in 2023.10, New papers will be added in the future.

Check out our open-source projects at eunomia-bpf and eBPF tutorials at bpf-developer-tutorial. I'm also looking for a PhD position in the area of systems and networking in 2024/2025. My Github and email.

XRP: In-Kernel Storage Functions with eBPF

With the emergence of microsecond-scale NVMe storage devices, the Linux kernel storage stack overhead has become significant, almost doubling access times. We present XRP, a framework that allows applications to execute user-defined storage functions, such as index lookups or aggregations, from an eBPF hook in the NVMe driver, safely bypassing most of the kernel’s storage stack. To preserve file system semantics, XRP propagates a small amount of kernel state to its NVMe driver hook where the user-registered eBPF functions are called. We show how two key-value stores, BPF-KV, a simple B+-tree key-value store, and WiredTiger, a popular log-structured merge tree storage engine, can leverage XRP to significantly improve throughput and latency.

OSDI '22 Best Paper: https://www.usenix.org/conference/osdi22/presentation/zhong

Specification and verification in the field: Applying formal methods to BPF just-in-time compilers in the Linux kernel

This paper describes our experience applying formal methods to a critical component in the Linux kernel, the just-in-time compilers ("JITs") for the Berkeley Packet Filter (BPF) virtual machine. We verify these JITs using Jitterbug, the first framework to provide a precise specification of JIT correctness that is capable of ruling out real-world bugs, and an automated proof strategy that scales to practical implementations. Using Jitterbug, we have designed, implemented, and verified a new BPF JIT for 32-bit RISC-V, found and fixed 16 previously unknown bugs in five other deployed JITs, and developed new JIT optimizations; all of these changes have been upstreamed to the Linux kernel. The results show that it is possible to build a verified component within a large, unverified system with careful design of specification and proof strategy.

OSDI 20: https://www.usenix.org/conference/osdi20/presentation/nelson

Ξ»-IO: A Unified IO Stack for Computational Storage

The emerging computational storage device offers an opportunity for in-storage computing. It alleviates the overhead of data movement between the host and the device, and thus accelerates data-intensive applications. In this paper, we present Ξ»-IO, a unified IO stack managing both computation and storage resources across the host and the device. We propose a set of designs – interface, runtime, and scheduling – to tackle three critical issues. We implement Ξ»-IO in full-stack software and hardware environment, and evaluate it with synthetic and real applications against Linux IO, showing up to 5.12Γ— performance improvement.

FAST23: https://www.usenix.org/conference/fast23/presentation/yang-zhe

Extension Framework for File Systems in User space

User file systems offer numerous advantages over their in-kernel implementations, such as ease of development and better system reliability. However, they incur heavy performance penalty. We observe that existing user file system frameworks are highly general; they consist of a minimal interposition layer in the kernel that simply forwards all low-level requests to user space. While this design offers flexibility, it also severely degrades performance due to frequent kernel-user context switching.

This work introduces ExtFUSE, a framework for developing extensible user file systems that also allows applications to register "thin" specialized request handlers in the kernel to meet their specific operative needs, while retaining the complex functionality in user space. Our evaluation with two FUSE file systems shows that ExtFUSE can improve the performance of user file systems with less than a few hundred lines on average. ExtFUSE is available on GitHub.

ATC 19: https://www.usenix.org/conference/atc19/presentation/bijlani

Electrode: Accelerating Distributed Protocols with eBPF

Implementing distributed protocols under a standard Linux kernel networking stack enjoys the benefits of load-aware CPU scaling, high compatibility, and robust security and isolation. However, it suffers from low performance because of excessive user-kernel crossings and kernel networking stack traversing. We present Electrode with a set of eBPF-based performance optimizations designed for distributed protocols. These optimizations get executed in the kernel before the networking stack but achieve similar functionalities as were implemented in user space (e.g., message broadcasting, collecting quorum of acknowledgments), thus avoiding the overheads incurred by user-kernel crossings and kernel networking stack traversing. We show that when applied to a classic Multi-Paxos state machine replication protocol, Electrode improves its throughput by up to 128.4% and latency by up to 41.7%.

NSDI 23: https://www.usenix.org/conference/nsdi23/presentation/zhou

BMC: Accelerating Memcached using Safe In-kernel Caching and Pre-stack Processing

In-memory key-value stores are critical components that help scale large internet services by providing low-latency access to popular data. Memcached, one of the most popular key-value stores, suffers from performance limitations inherent to the Linux networking stack and fails to achieve high performance when using high-speed network interfaces. While the Linux network stack can be bypassed using DPDK based solutions, such approaches require a complete redesign of the software stack and induce high CPU utilization even when client load is low.

To overcome these limitations, we present BMC, an in-kernel cache for Memcached that serves requests before the execution of the standard network stack. Requests to the BMC cache are treated as part of the NIC interrupts, which allows performance to scale with the number of cores serving the NIC queues. To ensure safety, BMC is implemented using eBPF. Despite the safety constraints of eBPF, we show that it is possible to implement a complex cache service. Because BMC runs on commodity hardware and requires modification of neither the Linux kernel nor the Memcached application, it can be widely deployed on existing systems. BMC optimizes the processing time of Facebook-like small-size requests. On this target workload, our evaluations show that BMC improves throughput by up to 18x compared to the vanilla Memcached application and up to 6x compared to an optimized version of Memcached that uses the SO_REUSEPORT socket flag. In addition, our results also show that BMC has negligible overhead and does not deteriorate throughput when treating non-target workloads.

NSDI 21: https://www.usenix.org/conference/nsdi21/presentation/ghigoff

hXDP: Efficient Software Packet Processing on FPGA NICs

FPGA accelerators on the NIC enable the offloading of expensive packet processing tasks from the CPU. However, FPGAs have limited resources that may need to be shared among diverse applications, and programming them is difficult.

We present a solution to run Linux's eXpress Data Path programs written in eBPF on FPGAs, using only a fraction of the available hardware resources while matching the performance of high-end CPUs. The iterative execution model of eBPF is not a good fit for FPGA accelerators. Nonetheless, we show that many of the instructions of an eBPF program can be compressed, parallelized or completely removed, when targeting a purpose-built FPGA executor, thereby significantly improving performance. We leverage that to design hXDP, which includes (i) an optimizing-compiler that parallelizes and translates eBPF bytecode to an extended eBPF Instruction-set Architecture defined by us; a (ii) soft-processor to execute such instructions on FPGA; and (iii) an FPGA-based infrastructure to provide XDP's maps and helper functions as defined within the Linux kernel.

We implement hXDP on an FPGA NIC and evaluate it running real-world unmodified eBPF programs. Our implementation is clocked at 156.25MHz, uses about 15% of the FPGA resources, and can run dynamically loaded programs. Despite these modest requirements, it achieves the packet processing throughput of a high-end CPU core and provides a 10x lower packet forwarding latency.

OSDI 20: https://www.usenix.org/conference/osdi20/presentation/brunella

Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code

Microservices are becoming more complicated, posing new challenges for traditional performance monitoring solutions. On the one hand, the rapid evolution of microservices places a significant burden on the utilization and maintenance of existing distributed tracing frameworks. On the other hand, complex infrastructure increases the probability of network performance problems and creates more blind spots on the network side. In this paper, we present DeepFlow, a network-centric distributed tracing framework for troubleshooting microservices. DeepFlow provides out-of-the-box tracing via a network-centric tracing plane and implicit context propagation. In addition, it eliminates blind spots in network infrastructure, captures network metrics in a low-cost way, and enhances correlation between different components and layers. We demonstrate analytically and empirically that DeepFlow is capable of locating microservice performance anomalies with negligible overhead. DeepFlow has already identified over 71 critical performance anomalies for more than 26 companies and has been utilized by hundreds of individual developers. Our production evaluations demonstrate that DeepFlow is able to save users hours of instrumentation efforts and reduce troubleshooting time from several hours to just a few minutes.

SIGCOMM 23: https://dl.acm.org/doi/10.1145/3603269.3604823

Fast In-kernel Traffic Sketching in eBPF

The extended Berkeley Packet Filter (eBPF) is an infrastructure that allows to dynamically load and run micro-programs directly in the Linux kernel without recompiling it.

In this work, we study how to develop high-performance network measurements in eBPF. We take sketches as case-study, given their ability to support a wide-range of tasks while providing low-memory footprint and accuracy guarantees. We implemented NitroSketch, the state-of-the-art sketch for user-space networking and show that best practices in user-space networking cannot be directly applied to eBPF, because of its different performance characteristics. By applying our lesson learned we improve its performance by 40% compared to a naive implementation.

SIGCOMM 23: https://dl.acm.org/doi/abs/10.1145/3594255.3594256

SPRIGHT: extracting the server from serverless computing! high-performance eBPF-based event-driven, shared-memory processing

Serverless computing promises an efficient, low-cost compute capability in cloud environments. However, existing solutions, epitomized by open-source platforms such as Knative, include heavyweight components that undermine this goal of serverless computing. Additionally, such serverless platforms lack dataplane optimizations to achieve efficient, high-performance function chains that facilitate the popular microservices development paradigm. Their use of unnecessarily complex and duplicate capabilities for building function chains severely degrades performance. 'Cold-start' latency is another deterrent.

We describe SPRIGHT, a lightweight, high-performance, responsive serverless framework. SPRIGHT exploits shared memory processing and dramatically improves the scalability of the dataplane by avoiding unnecessary protocol processing and serialization-deserialization overheads. SPRIGHT extensively leverages event-driven processing with the extended Berkeley Packet Filter (eBPF). We creatively use eBPF's socket message mechanism to support shared memory processing, with overheads being strictly load-proportional. Compared to constantly-running, polling-based DPDK, SPRIGHT achieves the same dataplane performance with 10Γ— less CPU usage under realistic workloads. Additionally, eBPF benefits SPRIGHT, by replacing heavyweight serverless components, allowing us to keep functions 'warm' with negligible penalty.

Our preliminary experimental results show that SPRIGHT achieves an order of magnitude improvement in throughput and latency compared to Knative, while substantially reducing CPU usage, and obviates the need for 'cold-start'.

https://dl.acm.org/doi/10.1145/3544216.3544259

KEN: Kernel Extensions using Natural Language

The ability to modify and extend an operating system is an important feature for improving a system's security, reliability, and performance. The extended Berkeley Packet Filters (eBPF) ecosystem has emerged as the standard mechanism for extending the Linux kernel and has recently been ported to Windows. eBPF programs inject new logic into the kernel that the system will execute before or after existing logic. While the eBPF ecosystem provides a flexible mechanism for kernel extension, it is difficult for developers to write eBPF programs today. An eBPF developer must have deep knowledge of the internals of the operating system to determine where to place logic and cope with programming limitations on the control flow and data accesses of their eBPF program enforced by the eBPF verifier. This paper presents KEN, an alternative framework that alleviates the difficulty of writing an eBPF program by allowing Kernel Extensions to be written in Natural language. KEN uses recent advances in large language models (LLMs) to synthesize an eBPF program given a user's English language prompt. To ensure that LLM's output is semantically equivalent to the user's prompt, KEN employs a combination of LLM-empowered program comprehension, symbolic execution, and a series of feedback loops. KEN's key novelty is the combination of these techniques. In particular, the system uses symbolic execution in a novel structure that allows it to combine the results of program synthesis and program comprehension and build on the recent success that LLMs have shown for each of these tasks individually. To evaluate KEN, we developed a new corpus of natural language prompts for eBPF programs. We show that KEN produces correct eBPF programs on 80% which is an improvement of a factor of 2.67 compared to an LLM-empowered program synthesis baseline.

eBPF'24: https://dl.acm.org/doi/10.1145/3672197.3673434 and arxiv https://arxiv.org/abs/2312.05531

Programmable System Call Security with eBPF

System call filtering is a widely used security mechanism for protecting a shared OS kernel against untrusted user applications. However, existing system call filtering techniques either are too expensive due to the context switch overhead imposed by userspace agents, or lack sufficient programmability to express advanced policies. Seccomp, Linux's system call filtering module, is widely used by modern container technologies, mobile apps, and system management services. Despite the adoption of the classic BPF language (cBPF), security policies in Seccomp are mostly limited to static allow lists, primarily because cBPF does not support stateful policies. Consequently, many essential security features cannot be expressed precisely and/or require kernel modifications.
In this paper, we present a programmable system call filtering mechanism, which enables more advanced security policies to be expressed by leveraging the extended BPF language (eBPF). More specifically, we create a new Seccomp eBPF program type, exposing, modifying or creating new eBPF helper functions to safely manage filter state, access kernel and user state, and utilize synchronization primitives. Importantly, our system integrates with existing kernel privilege and capability mechanisms, enabling unprivileged users to install advanced filters safely. Our evaluation shows that our eBPF-based filtering can enhance existing policies (e.g., reducing the attack surface of early execution phase by up to 55.4% for temporal specialization), mitigate real-world vulnerabilities, and accelerate filters.

https://arxiv.org/abs/2302.10366

Cross Container Attacks: The Bewildered eBPF on Clouds

The extended Berkeley Packet Filter (eBPF) provides powerful and flexible kernel interfaces to extend the kernel functions for user space programs via running bytecode directly in the kernel space. It has been widely used by cloud services to enhance container security, network management, and system observability. However, we discover that the offensive eBPF that have been extensively discussed in Linux hosts can bring new attack surfaces to containers. With eBPF tracing features, attackers can break the container's isolation and attack the host, e.g., steal sensitive data, DoS, and even escape the container. In this paper, we study the eBPF-based cross container attacks and reveal their security impacts in real world services. With eBPF attacks, we successfully compromise five online Jupyter/Interactive Shell services and the Cloud Shell of Google Cloud Platform. Furthermore, we find that the Kubernetes services offered by three leading cloud vendors can be exploited to launch cross-node attacks after the attackers escape the container via eBPF. Specifically, in Alibaba's Kubernetes services, attackers can compromise the whole cluster by abusing their over-privileged cloud metrics or management Pods. Unfortunately, the eBPF attacks on containers are seldom known and can hardly be discovered by existing intrusion detection systems. Also, the existing eBPF permission model cannot confine the eBPF and ensure secure usage in shared-kernel container environments. To this end, we propose a new eBPF permission model to counter the eBPF attacks in containers.

https://www.usenix.org/conference/usenixsecurity23/presentation/he

Comparing Security in eBPF and WebAssembly

This paper examines the security of eBPF and WebAssembly (Wasm), two technologies that have gained widespread adoption in recent years, despite being designed for very different use cases and environments. While eBPF is a technology primarily used within operating system kernels such as Linux, Wasm is a binary instruction format designed for a stack-based virtual machine with use cases extending beyond the web. Recognizing the growth and expanding ambitions of eBPF, Wasm may provide instructive insights, given its design around securely executing arbitrary untrusted programs in complex and hostile environments such as web browsers and clouds. We analyze the security goals, community evolution, memory models, and execution models of both technologies, and conduct a comparative security assessment, exploring memory safety, control flow integrity, API access, and side-channels. Our results show that eBPF has a history of focusing on performance first and security second, while Wasm puts more emphasis on security at the cost of some runtime overheads. Considering language-based restrictions for eBPF and a security model for API access are fruitful directions for future work.

https://dl.acm.org/doi/abs/10.1145/3609021.3609306

More about can be found in the first workshop: https://conferences.sigcomm.org/sigcomm/2023/workshop-ebpf.html

A flow-based IDS using Machine Learning in eBPF

eBPF is a new technology which allows dynamically loading pieces of code into the Linux kernel. It can greatly speed up networking since it enables the kernel to process certain packets without the involvement of a userspace program. So far eBPF has been used for simple packet filtering applications such as firewalls or Denial of Service protection. We show that it is possible to develop a flow based network intrusion detection system based on machine learning entirely in eBPF. Our solution uses a decision tree and decides for each packet whether it is malicious or not, considering the entire previous context of the network flow. We achieve a performance increase of over 20% compared to the same solution implemented as a userspace program.

https://arxiv.org/abs/2102.09980

Femto-containers: lightweight virtualization and fault isolation for small software functions on low-power IoT microcontrollers

Low-power operating system runtimes used on IoT microcontrollers typically provide rudimentary APIs, basic connectivity and, sometimes, a (secure) firmware update mechanism. In contrast, on less constrained hardware, networked software has entered the age of serverless, microservices and agility. With a view to bridge this gap, in the paper we design Femto-Containers, a new middleware runtime which can be embedded on heterogeneous low-power IoT devices. Femto-Containers enable the secure deployment, execution and isolation of small virtual software functions on low-power IoT devices, over the network. We implement Femto-Containers, and provide integration in RIOT, a popular open source IoT operating system. We then evaluate the performance of our implementation, which was formally verified for fault-isolation, guaranteeing that RIOT is shielded from logic loaded and executed in a Femto-Container. Our experiments on various popular micro-controller architectures (Arm Cortex-M, ESP32 and RISC-V) show that Femto-Containers offer an attractive trade-off in terms of memory footprint overhead, energy consumption, and security.

https://dl.acm.org/doi/abs/10.1145/3528535.3565242

The original link of this article: https://eunomia.dev/tutorials/18-further-reading

Overview (`src/19-lsm-connect/README.md`)

eBPF Tutorial by Example 19: Security Detection and Defense using LSM

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting the kernel or modifying the kernel source code. This feature enables eBPF to provide high flexibility and performance, making it widely applicable in network and system performance analysis. The same applies to eBPF applications in security, and this article will introduce how to use the eBPF LSM (Linux Security Modules) mechanism to implement a simple security check program.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/19-lsm-connect

Background

LSM has been an official security framework in the Linux kernel since Linux 2.6, and security implementations based on it include SELinux and AppArmor. With the introduction of BPF LSM in Linux 5.7, system developers have been able to freely implement function-level security checks. This article provides an example of limiting access to a specific IPv4 address through the socket connect function using a BPF LSM program. (This demonstrates its high control precision.)

Overview of LSM

LSM (Linux Security Modules) is a framework in the Linux kernel that supports various computer security models. LSM predefines a set of hook points on critical paths related to Linux kernel security, decoupling the kernel from security modules. This allows different security modules to be loaded/unloaded in the kernel freely without modifying the existing kernel code, thus enabling them to provide security inspection features.

In the past, using LSM mainly involved configuring existing security modules like SELinux and AppArmor or writing custom kernel modules. However, with the introduction of the BPF LSM mechanism in Linux 5.7, everything changed. Now, developers can write custom security policies using eBPF and dynamically load them into the LSM mount points in the kernel without configuring or writing kernel modules.

Some of the hook points currently supported by LSM include:

  • File open, creation, deletion, and movement;
  • Filesystem mounting;
  • Operations on tasks and processes;
  • Operations on sockets (creating, binding sockets, sending and receiving messages, etc.);

For more hook points, refer to lsm_hooks.h.

Verifying BPF LSM Availability

First, please confirm that your kernel version is higher than 5.7. Next, you can use the following command to check if BPF LSM support is enabled:

console
$ cat /boot/config-$(uname -r) | grep BPF_LSM
CONFIG_BPF_LSM=y

If the output contains CONFIG_BPF_LSM=y, BPF LSM is supported. Provided that the above conditions are met, you can use the following command to check if the output includes the bpf option:

console
$ cat /sys/kernel/security/lsm
ndlock,lockdown,yama,integrity,apparmor

If the output does not include the bpf option (as in the example above), you can modify /etc/default/grub:

conf
GRUB_CMDLINE_LINUX="lsm=ndlock,lockdown,yama,integrity,apparmor,bpf"

Then, update the grub configuration using the update-grub2 command (the corresponding command may vary depending on the system), and restart the system.

Writing eBPF Programs

c
// lsm-connect.bpf.c
#include "vmlinux.h"
#include 
#include 
#include 

char LICENSE[] SEC("license") = "GPL";

#define EPERM 1
#define AF_INET 2

const __u32 blockme = 16843009; // 1.1.1.1 -> int

SEC("lsm/socket_connect")
int BPF_PROG(restrict_connect, struct socket *sock, struct sockaddr *address, int addrlen, int ret)
{
    // Satisfying "cannot override a denial" rule
    if (ret != 0)
    {
        return ret;
    }

    // Only IPv4 in this example
    if (address->sa_family != AF_INET)
    {
        return 0;
    }

    // Cast the address to an IPv4 socket address
    struct sockaddr_in *addr = (struct sockaddr_in *)address;

    // Where do you want to go?
    __u32 dest = addr->sin_addr.s_addr;
    bpf_printk("lsm: found connect to %d", dest);

    if (dest == blockme)
    {
        bpf_printk("lsm: blocking %d", dest);
        return -EPERM;
    }
    return 0;
}

This is eBPF code implemented in C on the kernel side. It blocks all connection operations through a socket to 1.1.1.1. The following information is included:

  • The SEC("lsm/socket_connect") macro indicates the expected mount point for this program.
  • The program is defined by the BPF_PROG macro (see tools/lib/bpf/bpf_tracing.h for details).
  • restrict_connect is the program name required by the BPF_PROG macro.
  • ret is the return value of the LSM check program (potential) before the current function on this mount point.

The overall idea of the program is not difficult to understand:

  • First, if the return value of other security check functions is non-zero (failed), there is no need to check further and the connection is rejected.
  • Next, it determines whether it is an IPv4 connection request and compares the address being connected to with 1.1.1.1.
  • If the requested address is 1.1.1.1, the connection is blocked; otherwise, the connection is allowed.

During the execution of the program, all connection operations through a socket will be output to /sys/kernel/debug/tracing/trace_pipe.

Compilation and Execution

Compile using a container:

console
docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Or compile using ecc:

console
$ ecc lsm-connect.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

And run using ecli:

shell
sudo ecli run package.json

Next, open another terminal and try to access 1.1.1.1:

console
$ ping 1.1.1.1
ping: connect: Operation not permitted
$ curl 1.1.1.1
curl: (7) Couldn't connect to server
$ wget 1.1.1.1
--2023-04-23 08:41:18--  (try: 2)  http://1.1.1.1/
Connecting to 1.1.1.1:80... failed: Operation not permitted.
Retrying.

At the same time, we can view the output of bpf_printk:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
            ping-7054    [000] d...1  6313.430872: bpf_trace_printk: lsm: found connect to 16843009
            ping-7054    [000] d...1  6313.430874: bpf_trace_printk: lsm: blocking 16843009
            curl-7058    [000] d...1  6316.346582: bpf_trace_printk: lsm: found connect to 16843009
            curl-7058    [000] d...1  6316.346584: bpf_trace_printk: lsm: blocking 16843009".```
wget-7061    [000] d...1  6318.800698: bpf_trace_printk: lsm: found connect to 16843009
wget-7061    [000] d...1  6318.800700: bpf_trace_printk: lsm: blocking 16843009

Complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/19-lsm-connect

Summary

This article introduces how to use BPF LSM to restrict access to a specific IPv4 address through a socket. We can enable the LSM BPF mount point by modifying the GRUB configuration file. In the eBPF program, we define functions using the BPF_PROG macro and specify the mount point using the SEC macro. In the implementation of the function, we follow the principle of "cannot override a denial" in the LSM security-checking module and restrict the socket connection request based on the destination address of the request.

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

References

The original link of this article: https://eunomia.dev/tutorials/19-lsm-connect

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code at runtime.

This article is the second part of the eBPF Tutorial by Example, focusing on using kprobe to capture the unlink system call in eBPF. The article will first explain the basic concepts and technical background of kprobes, and then introduce how to use kprobe to capture the unlink system call in eBPF.

Background of kprobes Technology

During the debugging process of the kernel or modules, developers often need to know whether certain functions are called, when they are called, whether the execution is correct, and what the input and return values of the functions are. A simple approach is to add log print information to the corresponding functions in the kernel code. However, this approach often requires recompiling the kernel or modules, restarting the device, etc., which is complex and may disrupt the original code execution process.

By using the kprobes technology, users can define their own callback functions and dynamically insert probes into almost all functions in the kernel or modules (some functions cannot be probed, such as the kprobes' own implementation functions, which will be explained in detail later). When the kernel execution flow reaches the specified probe function, it will invoke the callback function, allowing the user to collect the desired information. The kernel will then return to the normal execution flow. If the user has collected sufficient information and no longer needs to continue probing, the probes can be dynamically removed. Therefore, the kprobes technology has the advantages of minimal impact on the kernel execution flow and easy operation.

The kprobes technology includes three detection methods: kprobe, jprobe, and kretprobe. First, kprobe is the most basic detection method and serves as the basis for the other two. It allows probes to be placed at any position (including within a function). It provides three callback modes for probes: pre_handler, post_handler, and fault_handler. The pre_handler function is called before the probed instruction is executed, the post_handler is called after the probed instruction is completed (note that it is not the probed function), and the fault_handler is called when a memory access error occurs. The jprobe is based on kprobe and is used to obtain the input values of the probed function. Finally, as the name suggests, kretprobe is also based on kprobe and is used to obtain the return values of the probed function.

The kprobes technology is not only implemented through software but also requires support from the hardware architecture. This involves CPU exception handling and single-step debugging techniques. The former is used to make the program's execution flow enter the user-registered callback function, and the latter is used to single-step execute the probed instruction. Therefore, not all architectures support kprobes. Currently, kprobes technology supports various architectures, including i386, x86_64, ppc64, ia64, sparc64, arm, ppc, and mips (note that some architecture implementations may not be complete, see the kernel's Documentation/kprobes.txt for details).

Features and Usage Restrictions of kprobes:

  1. kprobes allows multiple kprobes to be registered at the same probe position, but jprobe currently does not support this. It is also not allowed to use other jprobe callback functions or the post_handler callback function of kprobe as probe points.

  2. In general, any function in the kernel can be probed, including interrupt handlers. However, the functions used to implement kprobes themselves in kernel/kprobes.c and arch/*/kernel/kprobes.c are not allowed to be probed. Additionally, do_page_fault and notifier_call_chain are also not allowed.

  3. If an inline function is used as a probe point, kprobes may not be able to guarantee that probe points are registered for all instances of that function. Since gcc may automatically optimize certain functions as inline functions, the desired probing effect may not be achieved.

  4. The callback function of a probe point may modify the runtime context of the probed function, such as by modifying the kernel's data structure or saving register information before triggering the prober in the struct pt_regs structure. Therefore, kprobes can be used to install bug fixes or inject fault testing code.

  5. kprobes avoids calling the callback function of another probe point again when processing the probe point function. For example, if a probe point is registered on the printk() function and the callback function may call printk() again, the callback for the printk probe point will not be triggered again. Only the nmissed field in the kprobe structure will be incremented.

  6. mutex locks and dynamic memory allocation are not used in the registration and removal process of kprobes.

  7. During the execution of kprobes callback functions, kernel preemption is disabled, and it may also be executed with interrupts disabled, which depends on the CPU architecture. Therefore, regardless of the situation, do not call functions that will give up the CPU in the callback function (such as semaphore, mutex lock, etc.);

  8. kretprobe is implemented by replacing the return address with the pre-defined trampoline address, so stack backtraces and gcc inline function __builtin_return_address() will return the address of the trampoline instead of the actual return address of the probed function;

  9. If the number of function calls and return calls of a function are unequal, registering kretprobe on such a function may not achieve the expected effect, for example, the do_exit() function will have problems, while the do_execve() function and do_fork() function will not;

  10. When entering and exiting a function, if the CPU is running on a stack that does not belong to the current task, registering kretprobe on that function may have unpredictable consequences. Therefore, kprobes does not support registering kretprobe for the __switch_to() function under the X86_64 architecture and will directly return -EINVAL.

kprobe Example

The complete code is as follows:

c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("kprobe/do_unlinkat")
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
{
    pid_t pid;
    const char *filename;

    pid = bpf_get_current_pid_tgid() >> 32;
    filename = BPF_CORE_READ(name, name);
    bpf_printk("KPROBE ENTRY pid = %d, filename = %s\n", pid, filename);
    return 0;
}

SEC("kretprobe/do_unlinkat")
int BPF_KRETPROBE(do_unlinkat_exit, long ret)
{
    pid_t pid;

    pid = bpf_get_current_pid_tgid() >> 32;
    bpf_printk("KPROBE EXIT: pid = %d, ret = %ld\n", pid, ret);
    return 0;
}

This code is a simple eBPF program used to monitor and capture the unlink system call executed in the Linux kernel. The unlink system call is used to delete a file. This eBPF program traces this system call by placing hooks at the entry and exit points of the do_unlinkat function using a kprobe (kernel probe).

First, we import necessary header files such as vmlinux.h, bpf_helpers.h, bpf_tracing.h, and bpf_core_read.h. Then, we define a license to allow the program to run in the kernel.

c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

The first kprobe hooks into the entry point of do_unlinkat. The BPF_KPROBE macro makes it easy to access the function's parameters - in this case dfd (file descriptor) and name (a pointer to the filename structure). We grab the current process ID and then use BPF_CORE_READ to safely read the filename from kernel memory.

c
SEC("kprobe/do_unlinkat")
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
{
    pid_t pid;
    const char *filename;

    pid = bpf_get_current_pid_tgid() >> 32;
    filename = BPF_CORE_READ(name, name);
    bpf_printk("KPROBE ENTRY pid = %d, filename = %s\n", pid, filename);
    return 0;
}

You might wonder why we can't just access name->name directly. The answer is that eBPF programs run in a restricted environment and need special helpers to safely read kernel memory. BPF_CORE_READ handles this safely and also provides CO-RE (Compile Once - Run Everywhere) support, meaning your program will work across different kernel versions even if the structure layout changes.

The kretprobe is the counterpart that triggers when the function returns. Here we can capture the return value to see if the unlink operation succeeded or failed. A return value of 0 means success, while negative values indicate errors.

c
SEC("kretprobe/do_unlinkat")
int BPF_KRETPROBE(do_unlinkat_exit, long ret)
{
    pid_t pid;

    pid = bpf_get_current_pid_tgid() >> 32;
    bpf_printk("KPROBE EXIT: pid = %d, ret = %ld\n", pid, ret);
    return 0;
}

By combining kprobe and kretprobe, you get the complete picture - you can see what file is being deleted and whether the operation succeeded. This pattern is useful for debugging, security monitoring, or building observability tools.

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

To compile this program, use the ecc tool:

console
$ ecc kprobe-link.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

Then run:

console
sudo ecli run package.json

In another window:

shell
touch test1
rm test1
touch test2
rm test2

You should see kprobe demo output similar to the following in the /sys/kernel/debug/tracing/trace_pipe file:

shell
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
              rm-9346    [005] d..3  4710.951696: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test1
              rm-9346    [005] d..4  4710.951819: bpf_trace_printk: KPROBE EXIT: ret = 0
              rm-9346    [005] d..3  4710.951852: bpf_trace_printk: KPROBE ENTRY pid = 9346, filename = test2
              rm-9346    [005] d..4  4710.951895: bpf_trace_printk: KPROBE EXIT: ret = 0

Summary

In this article's example, we learned how to use eBPF's kprobe and kretprobe to capture the unlink system call.

This article is the second part of the introductory eBPF development tutorial. The next article will explain how to use fentry to monitor and capture the unlink system call in eBPF.

If you'd like to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

Overview (`src/20-tc/README.md`)

eBPF Tutorial by Example 20: tc Traffic Control

Linux's Traffic Control (tc) subsystem has been present in the kernel for many years. Similar to the relationship between iptables and netfilter, tc includes a user-space tc program and a kernel-level traffic control framework. It is mainly used to control the sending and receiving of packets in terms of rate, sequence, and other aspects. Starting from Linux 4.1, tc has added some new attachment points and supports loading eBPF programs as filters onto these attachment points.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/20-tc

Overview of tc

From the protocol stack perspective, tc is located at the link layer. Its position has already completed the allocation of sk_buff and is later than xdp. In order to control the sending and receiving of packets, tc uses a queue structure to temporarily store and organize packets. In the tc subsystem, the corresponding data structure and algorithm control mechanism are abstracted as qdisc (Queueing discipline). It exposes two callback interfaces for enqueuing and dequeuing packets externally, and internally hides the implementation of queuing algorithms. In qdisc, we can implement complex tree structures based on filters and classes. Filters are mounted on qdisc or class to implement specific filtering logic, and the return value determines whether the packet belongs to a specific class.

When a packet reaches the top-level qdisc, its enqueue interface is called, and the mounted filters are executed one by one until a filter matches successfully. Then the packet is sent to the class pointed to by that filter and enters the qdisc processing process configured by that class. The tc framework provides the so-called classifier-action mechanism, that is, when a packet matches a specific filter, the action mounted by that filter is executed to process the packet, implementing a complete packet classification and processing mechanism.

The existing tc provides eBPF with the direct-action mode, which allows an eBPF program loaded as a filter to return values such as TC_ACT_OK as tc actions, instead of just returning a classid like traditional filters and handing over the packet processing to the action module. Now, eBPF programs can be mounted on specific qdiscs to perform packet classification and processing actions.

Writing eBPF Programs

c
#include <vmlinux.h>
#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#define TC_ACT_OK 0
#define ETH_P_IP 0x0800 /* Internet Protocol packet */

/// @tchook {"ifindex":1, "attach_point":"BPF_TC_INGRESS"}
/// @tcopts {"handle":1, "priority":1}
SEC("tc")
int tc_ingress(struct __sk_buff *ctx)
{
    void *data_end = (void *)(__u64)ctx->data_end;
    void *data = (void *)(__u64)ctx->data;
    struct ethhdr *l2;
    struct iphdr *l3;

    if (ctx->protocol != bpf_htons(ETH_P_IP))
        return TC_ACT_OK;

    l2 = data;
    if ((void *)(l2 + 1) > data_end)
        return TC_ACT_OK;

    l3 = (struct iphdr *)(l2 + 1);
    if ((void *)(l3 + 1) > data_end)
        return TC_ACT_OK;

    bpf_printk("Got IP packet: tot_len: %d, ttl: %d", bpf_ntohs(l3->tot_len), l3->ttl);
    return TC_ACT_OK;
}

char __license[] SEC("license") = "GPL";

This code defines an eBPF program that can capture and process packets through Linux TC (Transmission Control). In this program, we limit it to capture only IPv4 protocol packets, and then print out the total length and Time-To-Live (TTL) value of the packet using the bpf_printk function.

What needs to be noted is that we use some BPF library functions in the code, such as the functions bpf_htons and bpf_ntohs, which are used for conversion between network byte order and host byte order. In addition, we also use some comments to provide additional points and option information for TC. For example, at the beginning of this code, we use the following comments:

c
/// @tchook {"ifindex":1, "attach_point":"BPF_TC_INGRESS"}
/// @tcopts {"handle":1, "priority":1}

These comments tell TC to attach the eBPF program to the ingress attachment point of the network interface, and specify the values of the handle and priority options. You can refer to the introduction in patchwork for tc-related APIs in libbpf.

In summary, this code implements a simple eBPF program that captures packets and prints out their information.

Compilation and Execution

Compile using a container:

console
docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Or compile using ecc:

console
$ ecc tc.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

And run using ecli:

shell
sudo ecli run ./package.json

You can view the output of the program in the following way:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
            node-1254811 [007] ..s1 8737831.671074: 0: Got IP packet: tot_len: 79, ttl: 64
            sshd-1254728 [006] ..s1 8737831.674334: 0: Got IP packet: tot_len: 79, ttl: 64
            sshd-1254728 [006] ..s1 8737831.674349: 0: Got IP packet: tot_len: 72, ttl: 64
            node-1254811 [007] ..s1 8737831.674550: 0: Got IP packet: tot_len: 71, ttl: 64

Summary

This article introduces how to mount eBPF type filters to the TC traffic control subsystem to achieve queuing processing of link layer packets. Based on the solution provided by eunomia-bpf to pass parameters to libbpf through comments, we can mount our own tc BPF program to the target network device with specified options and use the sk_buff structure of the kernel to filter and process packets.

If you want to learn more about eBPF knowledge and practice, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

References

The original link of this article: https://eunomia.dev/tutorials/20-tc

Overview (`src/21-xdp/README.md`)

eBPF Tutorial by Example 21: Programmable Packet Processing with XDP

In this tutorial, we will introduce XDP (eXpress Data Path) and walk through a small example to help you get started. Later on, we will explore more advanced XDP applications, such as load balancers, firewalls, and other real-world use cases. Please give us a start on Github if you are interested in eBPF or XDP!

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/21-xdp

What is XDP?

XDP is a high-performance, programmable data path in the Linux kernel, designed for packet processing at the network interface level. By attaching eBPF programs directly to network device drivers, XDP can intercept and handle packets before they reach the kernel’s networking stack. This allows for extremely low-latency and efficient packet processing, making it ideal for tasks like DDoS defense, load balancing, and traffic filtering. In fact, XDP can achieve throughput as high as 24 million packets per second (Mpps) per core.

Why XDP?

XDP operates at a lower level than traditional Linux networking components, like cBPF (Classic BPF), by running inside the soft interrupt context of the network device driver. It can handle packets before they are even processed by the kernel's standard networking stack, bypassing the creation of the skb_buff structure, which represents network packets in Linux. This early-stage processing provides significant performance gains for simple but frequent operations like dropping malicious packets or load balancing across servers.

Compared to other packet processing mechanisms, XDP strikes a balance between performance and usability, leveraging the security and reliability of the Linux kernel while providing flexibility through programmable eBPF.

Overview of XDP vs. Other Approaches

Before XDP, several other solutions aimed to accelerate packet processing by bypassing the kernel entirely. One prominent example is DPDK (Data Plane Development Kit). DPDK allows user-space applications to take direct control of network devices, achieving very high performance. However, this approach comes with trade-offs:

  1. Lack of Kernel Integration: DPDK and other kernel-bypass solutions cannot utilize existing kernel networking features, requiring developers to reimplement many protocols and functions in user space.

  2. Security Boundaries: These bypass techniques break the kernel’s security model, making it harder to leverage security tools provided by the kernel.

  3. User-Kernel Transition Costs: When user-space packet processing needs to interact with traditional kernel networking (like socket-based applications), packets must be reinjected into the kernel, adding overhead and complexity.

  4. Dedicated CPU Usage: To handle high traffic, DPDK and similar solutions often require dedicating one or more CPU cores solely for packet processing, which limits the scalability and efficiency of general-purpose systems.

Another alternative to XDP is using kernel modules or hooks in the Linux networking stack. While this method integrates well with existing kernel features, it requires extensive kernel modifications and does not provide the same performance benefits, as it operates later in the packet processing pipeline.

The XDP + eBPF Advantage

XDP combined with eBPF offers a middle ground between kernel-bypass solutions like DPDK and kernel-integrated solutions. Here’s why XDP + eBPF stands out:

  • High Performance: By intercepting packets early at the NIC driver level, XDP achieves near-line rate performance for tasks like dropping, redirecting, or load balancing packets, all while keeping resource usage low.

  • Kernel Integration: Unlike DPDK, XDP works within the Linux kernel, allowing seamless interaction with the existing kernel network stack and tools (such as iptables, nftables, or sockets). There’s no need to reimplement networking protocols in user space.

  • Security: The eBPF virtual machine (VM) ensures that user-defined XDP programs are sandboxed and constrained, which means they cannot destabilize the kernel. The security model of eBPF prevents malicious or buggy code from harming the system, providing a safe environment for programmable packet processing.

  • No Dedicated CPUs Required: XDP allows packet processing without dedicating entire CPU cores solely to network tasks. This improves the overall efficiency of the system, allowing for more flexible resource allocation.

In summary, XDP + eBPF delivers a robust solution for programmable packet processing that combines high performance with the flexibility and safety of kernel integration. It eliminates the drawbacks of full kernel-bypass solutions while retaining the benefits of kernel security and functionality.

Projects and Use Cases with XDP

XDP is already being used in a number of high-profile projects that highlight its power and flexibility in real-world networking scenarios:

1. **Cilium**

  • Description: Cilium is an open-source networking, security, and observability tool designed for cloud-native environments, especially Kubernetes. It leverages XDP to implement high-performance packet filtering and load balancing.
  • Use Case: Cilium offloads packet filtering and security policies to XDP, enabling high-throughput and low-latency traffic management in containerized environments without sacrificing scalability.
  • Link: Cilium

2. **Katran**

  • Description: Katran is a layer 4 load balancer developed by Facebook, optimized for high scalability and performance. It uses XDP to handle packet forwarding with minimal overhead.
  • Use Case: Katran processes millions of packets per second to distribute traffic across backend servers efficiently, using XDP to achieve low-latency and high-performance load balancing in large-scale data centers.
  • Link: Katran GitHub

3. **XDP DDoS Protection at Cloudflare**

  • Description: Cloudflare has implemented XDP for real-time DDoS mitigation. By processing packets at the NIC level, Cloudflare can filter out attack traffic before it reaches the networking stack, minimizing the impact of DDoS attacks on their systems.
  • Use Case: Cloudflare leverages XDP to drop malicious packets early in the pipeline, protecting their infrastructure from large-scale DDoS attacks while maintaining high availability for legitimate traffic.
  • Link: Cloudflare Blog on XDP

These projects demonstrate the real-world capabilities of XDP for scalable and efficient packet processing across different domains, from security and load balancing to cloud-native networking.

Why Use XDP Over Other Methods?

Compared to traditional methods like iptables, nftables, or tc, XDP offers several clear advantages:

  • Speed and Low Overhead: Operating directly in the NIC driver, XDP bypasses much of the kernel’s overhead, enabling faster packet processing.

  • Customizability: XDP allows developers to create custom packet-processing programs with eBPF, providing more flexibility and granularity than legacy tools like iptables.

  • Resource Efficiency: XDP does not require dedicating entire CPU cores to packet processing, unlike user-space solutions like DPDK, making it a more efficient choice for high-performance networking.

Writing your first XDP Program

c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

/// @ifindex 1
/// @flags 0
/// @xdpopts {"old_prog_fd":0}
SEC("xdp")
int xdp_pass(struct xdp_md* ctx) {
    void* data = (void*)(long)ctx->data;
    void* data_end = (void*)(long)ctx->data_end;
    int pkt_sz = data_end - data;

    bpf_printk("packet size is %d", pkt_sz);
    return XDP_PASS;
}

char __license[] SEC("license") = "GPL";

This is a kernel-side eBPF code written in C. It captures all packets passing through the target network device using XDP, calculates their size, and outputs it to trace_pipe.

It's worth noting the following annotations in the code:

c
/// @ifindex 1
/// @flags 0
/// @xdpopts {"old_prog_fd":0}

This functionality is provided by eunomia-bpf, which allows these annotations to inform the eunomia-bpf loader about the desired target network device number, mounting flags, and options for this XDP program.

These variables are based on the API provided by libbpf. Detailed information about the interface can be viewed here.

The SEC("xdp") macro indicates the type of the BPF program, while ctx is the execution context of this BPF program for packet processing.

At the end of the program, we return XDP_PASS, signaling that our XDP program will deliver packets passing through the target network device to the kernel's network protocol stack as usual. For more on XDP actions, see XDP actions.

Compilation and Execution

To compile using a container:

console
docker run -it -v `pwd`/:/src/ ghcr.io/eunomia-bpf/ecc-`uname -m`:latest

Or compile with ecc:

console
$ ecc xdp.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...

Then, run with ecli:

console
sudo ecli run package.json

To view the program's output:

console
$ sudo cat /sys/kernel/tracing/trace_pipe
            node-1939    [000] d.s11  1601.190413: bpf_trace_printk: packet size is 177
            node-1939    [000] d.s11  1601.190479: bpf_trace_printk: packet size is 66
     ksoftirqd/1-19      [001] d.s.1  1601.237507: bpf_trace_printk: packet size is 66
            node-1939    [000] d.s11  1601.275860: bpf_trace_printk: packet size is 344

Conclusion

This article introduces how to use XDP to process packets passing through a specific network device. With eunomia-bpf's annotation-based approach for passing parameters to libbpf, we can mount our custom XDP BPF program onto the target device with specified options. This allows packet processing even before they enter the kernel's network protocol stack, achieving high-performance programmable packet processing.

For those interested in further exploring eBPF, visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and a comprehensive guide.

References

For more information, you can refer to:

The original link of this article: https://eunomia.dev/tutorials/21-xdp

Overview (`src/22-android/README.md`)

eBPF Tutorial by Example: Using eBPF Programs on Android

This article mainly documents the author's exploration process, results, and issues encountered while testing the level of support for CO-RE technology based on the libbpf library on high version Android kernels in the Android Studio Emulator.
The test was conducted by building a Debian environment in the Android Shell environment and attempting to build the eunomia-bpf toolchain and run its test cases based on this.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/22-android

Background

As of now (2023-04), Android has not provided good support for dynamic loading of eBPF programs. Both the compiler distribution scheme represented by bcc and the CO-RE scheme based on btf and libbpf rely heavily on Linux environment support and cannot run well on the Android system.[^WeiShu]

However, there have been some successful cases of trying eBPF on the Android platform. In addition to the solution provided by Google to modify Android.bp to build and mount eBPF programs with the entire system[^Google], some people have proposed building a Linux environment based on the Android kernel and running the eBPF toolchain using this approach, and have developed related tools.

Currently available information mostly focuses on the testing of bcc and bpftrace toolchains based on the adeb/eadb sandbox built on the Android kernel, with less testing work on the CO-RE scheme. There is more reference material available for using the bcc tool on Android, such as:

The main idea is to use chroot to run a Debian image on the Android kernel and build the entire bcc toolchain within it in order to use eBPF tools. The same principle applies to using bpftrace.

In fact, higher versions of the Android kernel already support the btf option, which means that the emerging CO-RE technology in the eBPF field should also be applicable to Linux systems based on the Android kernel. This article will test and run eunomia-bpf in the emulator environment based on this.

eunomia-bpf is an open-source project that combines libbpf and WebAssembly technology, aiming to simplify the writing, compilation, and deployment of eBPF programs. This project can be seen as a practical way of implementing CO-RE, with libbpf as its core dependency. It is believed that the testing work of eunomia-bpf can provide reference for other CO-RE schemes.

Test Environment

  • Android Emulator (Android Studio Flamingo | 2022.2.1)
  • AVD: Pixel 6
  • Android Image: Tiramisu Android 13.0 x86_64 (5.15.41-android13-8-00055-g4f5025129fe8-ab8949913)

Environment Setup[^SeeFlowerX]

  1. Obtain debianfs-amd64-full.tar.gz from the releases page of the eadb repository as the rootfs of the Linux environment. Also, get the assets directory from this project to build the environment.
  2. Configure and start the Android Virtual Device in the Android Studio Device Manager.
  3. Push debianfs-amd64-full.tar.gz and the assets directory to the AVD using the adb tool from the Android Studio SDK:
    • ./adb push debianfs-amd64-full.tar.gz /data/local/tmp/deb.tar.gz
    • ./adb push assets /data/local/tmp/assets
  4. Use adb to enter the Android shell environment and obtain root permissions:
    • ./adb shell
    • su
  5. Build and enter the debian environment in the Android shell:
    • mkdir -p /data/eadb
    • mv /data/local/tmp/assets/* /data/eadb
    • mv /data/local/tmp/deb.tar.gz /data/eadb/deb.tar.gz+ rm -r /data/local/tmp/assets
    • chmod +x /data/eadb/device-*
    • /data/eadb/device-unpack
    • /data/eadb/run /data/eadb/debian

At this point, the Linux environment required for testing eBPF has been set up. In addition, in the Android shell (before entering debian), you can use zcat /proc/config.gz in conjunction with grep to view kernel compilation options.

Currently, the debian environment packaged by eadb has a low version of libc and lacks many tool dependencies. Additionally, due to different kernel compilation options, some eBPF features may not be available.

Build Tools

Clone the eunomia-bpf repository into the local debian environment. For the specific build process, refer to the repository's build.md. In this test, I used the ecc compilation method to generate the package.json. Please refer to the repository page for the build and usage instructions for this tool.

During the build process, you may need to manually install tools such as curl, pkg-config, libssl-dev, etc.

Results

Some eBPF programs can be successfully executed on Android, but there are also some applications that cannot be executed successfully for various reasons.

Success Cases

[bootstrap](https://github.com/eunomia-bpf/eunomia-bpf/tree/master/examples/bpftools/bootstrap)

The output of running is as follows:

console
TIME     PID     PPID    EXIT_CODE  DURATION_NS  COMM    FILENAME  EXIT_EVENT
09:09:19  10217  479     0          0            sh      /system/bin/sh 0
09:09:19  10217  479     0          0            ps      /system/bin/ps 0
09:09:19  10217  479     0          54352100     ps                1
09:09:21  10219  479     0          0            sh      /system/bin/sh 0
09:09:21  10219  479     0          0            ps      /system/bin/ps 0
09:09:21  10219  479     0          44260900     ps                1

[tcpstates](https://github.com/eunomia-bpf/eunomia-bpf/tree/master/examples/bpftools/tcpstates)

After starting monitoring, download a web page using wget in the Linux environment:

console
TIME     SADDR   DADDR   SKADDR  TS_US   DELTA_US  PID     OLDSTATE  NEWSTATE  FAMILY  SPORT   DPORT   TASK
09:07:46  0x4007000200005000000000000f02000a 0x5000000000000f02000a8bc53f77 18446635827774444352 3315344998 0 10115 7 2 2 0 80 wget
09:07:46  0x40020002d98e50003d99f8090f02000a 0xd98e50003d99f8090f02000a8bc53f77 18446635827774444352 3315465870 120872 0 2 1 2 55694 80 swapper/0
09:07:46  0x40010002d98e50003d99f8090f02000a 0xd98e50003d99f8090f02000a8bc53f77 18446635827774444352 3315668799 202929 10115 1 4 2 55694 80 wget
09:07:46  0x40040002d98e50003d99f8090f02000a 0xd98e50003d99f8090f02000a8bc53f77 18446635827774444352 3315670037 1237 0 4 5 2 55694 80 swapper/0
09:07:46  0x40050002000050003d99f8090f02000a 0x50003d99f8090f02000a8bc53f77 18446635827774444352 3315670225 188 0 5 7 2 55694 80 swapper/0
09:07:47  0x400200020000bb01565811650f02000a 0xbb01565811650f02000a6aa0d9ac 18446635828348806592 3316433261 0 2546 2 7 2 49970 443 ChromiumNet
09:07:47  0x400200020000bb01db794a690f02000a 0xbb01db794a690f02000aea2afb8e 18446635827774427776 3316535591 0 1469 2 7 2 37386 443 ChromiumNet

Start the detection and open the Chrome browser in the Android Studio simulation interface to access the Baidu page:

text
/* Detailed source-code truncated for AI context efficiency. */

Note: some error messages may appear in the Android shell during the test:

console
libbpf: failed to determine tracepoint 'syscalls/sys_enter_open' perf event ID: No such file or directory
libbpf: prog 'tracepoint__syscalls__sys_enter_open': failed to create tracepoint 'syscalls/sys_enter_open' perf event: No such file or directory
libbpf: prog 'tracepoint__syscalls__sys_enter_open': failed to auto-attach: -2
failed to attach skeleton
Error: BpfError("load and attach ebpf program failed")

Later, after investigation, it was found that the kernel did not enable the CONFIG_FTRACE_SYSCALLS option, which resulted in the inability to use the tracepoint of syscalls.

Summary

The CONFIG_DEBUG_INFO_BTF option is enabled by default when viewing the kernel compilation options in the Android shell. Based on this, the examples provided by the eunomia-bpf project already have some successful cases, such as monitoring the execution of the exec family of functions and the status of TCP connections.

For some cases that cannot run, the reasons are mainly the following:

  1. The kernel compilation options do not support the relevant eBPF functionality;
  2. The Linux environment packaged by eadb is weak and lacks necessary dependencies;

Currently, using eBPF tools in the Android system still requires building a complete Linux runtime environment. However, the Android kernel itself has comprehensive support for eBPF. This test proves that higher versions of the Android kernel support BTF debugging information and CO-RE dependent eBPF programs.

The development of eBPF tools in the Android system requires the addition of official new features. Currently, it seems that using eBPF tools directly through an Android app requires a lot of effort. At the same time, since eBPF tools require root privileges, ordinary Android users will encounter more difficulties in using them.

If you want to learn more about eBPF knowledge and practice, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ to get more examples and complete tutorials.

Reference

The original link of this article: https://eunomia.dev/tutorials/22-android

Overview (`src/23-http/README.md`)

L7 Tracing with eBPF: HTTP and Beyond via Socket Filters and Syscall Tracepoints

In today's technology landscape, with the rise of microservices, cloud-native applications, and complex distributed systems, observability of systems has become a crucial factor in ensuring their health, performance, and security. Especially in a microservices architecture, application components may be distributed across multiple containers and servers, making traditional monitoring methods often insufficient to provide the depth and breadth needed to fully understand the behavior of the system. This is where observing seven-layer protocols such as HTTP, gRPC, MQTT, and more becomes particularly important.

Seven-layer protocols provide detailed insights into how applications interact with other services and components. In a microservices environment, understanding these interactions is vital, as they often serve as the root causes of performance bottlenecks, failures, and security issues. However, monitoring these protocols is not a straightforward task. Traditional network monitoring tools like tcpdump, while effective at capturing network traffic, often fall short when dealing with the complexity and dynamism of seven-layer protocols.

This is where eBPF (extended Berkeley Packet Filter) technology comes into play. eBPF allows developers and operators to delve deep into the kernel layer, observing and analyzing system behavior in real-time without the need to modify or insert instrumentation into application code. This presents a unique opportunity to handle application layer traffic more simply and efficiently, particularly in microservices environments.

In this tutorial, we will delve into the following:

  • Tracking seven-layer protocols such as HTTP and the challenges associated with them.
  • eBPF's socket filter and syscall tracing: How these two technologies assist in tracing HTTP network request data at different kernel layers, and the advantages and limitations of each.
  • eBPF practical tutorial: How to develop an eBPF program and utilize eBPF socket filter or syscall tracing to capture and analyze HTTP traffic.

As network traffic increases and applications grow in complexity, gaining a deeper understanding of seven-layer protocols becomes increasingly important. Through this tutorial, you will acquire the necessary knowledge and tools to more effectively monitor and analyze your network traffic, ultimately enhancing the performance of your applications and servers.

This article is part of the eBPF Developer Tutorial, and for more detailed content, you can visit here. The source code is available on the GitHub repository.

Challenges in Tracking HTTP, HTTP/2, and Other Seven-Layer Protocols

In the modern networking environment, seven-layer protocols extend beyond just HTTP. In fact, there are many seven-layer protocols such as HTTP/2, gRPC, MQTT, WebSocket, AMQP, and SMTP, each serving critical roles in various application scenarios. These protocols provide detailed insights into how applications interact with other services and components. However, tracking these protocols is not a simple task, especially within complex distributed systems.

  1. Diversity and Complexity: Each seven-layer protocol has its specific design and workings. For example, gRPC utilizes HTTP/2 as its transport protocol and supports multiple languages, while MQTT is a lightweight publish/subscribe messaging transport protocol designed for low-bandwidth and unreliable networks.

  2. Dynamism: Many seven-layer protocols are dynamic, meaning their behavior can change based on network conditions, application requirements, or other factors.

  3. Encryption and Security: With increased security awareness, many seven-layer protocols employ encryption technologies such as TLS/SSL. This introduces additional challenges for tracking and analysis, as decrypting traffic is required for in-depth examination.

  4. High-Performance Requirements: In high-traffic production environments, capturing and analyzing traffic for seven-layer protocols can impact system performance. Traditional network monitoring tools may struggle to handle a large number of concurrent sessions.

  5. Data Completeness and Continuity: Unlike tools like tcpdump, which capture individual packets, tracking seven-layer protocols requires capturing complete sessions, which may involve multiple packets. This necessitates tools capable of correctly reassembling and parsing these packets to provide a continuous session view.

  6. Code Intrusiveness: To gain deeper insights into the behavior of seven-layer protocols, developers may need to modify application code to add monitoring functionalities. This not only increases development and maintenance complexity but can also impact application performance.

As mentioned earlier, eBPF provides a powerful solution, allowing us to capture and analyze seven-layer protocol traffic in the kernel layer without modifying application code. This approach not only offers insights into system behavior but also ensures optimal performance and efficiency. This is why eBPF has become the preferred technology for modern observability tools, especially in production environments that demand high performance and low latency.

eBPF Socket Filter vs. Syscall Tracing: In-Depth Analysis and Comparison

**eBPF Socket Filter**

What Is It?
eBPF socket filter is an extension of the classic Berkeley Packet Filter (BPF) that allows for more advanced packet filtering directly within the kernel. It operates at the socket layer, enabling fine-grained control over which packets are processed by user-space applications.

Key Features:

  • Performance: By handling packets directly within the kernel, eBPF socket filters reduce the overhead of context switches between user and kernel spaces.
  • Flexibility: eBPF socket filters can be attached to any socket, providing a universal packet filtering mechanism for various protocols and socket types.
  • Programmability: Developers can write custom eBPF programs to define complex filtering logic beyond simple packet matching.

Use Cases:

  • Traffic Control: Restrict or prioritize traffic based on custom conditions.
  • Security: Discard malicious packets before they reach user-space applications.
  • Monitoring: Capture specific packets for analysis without affecting other traffic.

**eBPF Syscall Tracing**

What Is It?
System call tracing using eBPF allows monitoring and manipulation of system calls made by applications. System calls are the primary mechanism through which user-space applications interact with the kernel, making tracing them a valuable way to understand application behavior.

Key Features:

  • Granularity: eBPF allows tracing specific system calls, even specific parameters within those system calls.
  • Low Overhead: Compared to other tracing methods, eBPF syscall tracing is designed to have minimal performance impact.
  • Security: Kernel validates eBPF programs to ensure they do not compromise system stability.

How It Works:
eBPF syscall tracing typically involves attaching eBPF programs to tracepoints or kprobes related to the system calls being traced. When the traced system call is invoked, the eBPF program is executed, allowing data collection or even modification of system call parameters.

Comparison of eBPF Socket Filter and Syscall Tracing

Aspect eBPF Socket Filter eBPF Syscall Tracing
Operational Layer Socket layer, primarily dealing with network packets received from or sent to sockets. System call layer, monitoring and potentially altering the behavior of system calls made by applications.
Primary Use Cases Mainly used for filtering, monitoring, and manipulation of network packets. Used for performance analysis, security monitoring, and debugging of interactions with the network.
Granularity Focuses on individual network packets. Can monitor a wide range of system activities, including those unrelated to networking.
Tracking HTTP Traffic Can be used to filter and capture HTTP packets passed through sockets. Can trace system calls associated with networking operations, which may include HTTP traffic.

In summary, both eBPF socket filters and syscall tracing can be used to trace HTTP traffic, but socket filters are more direct and suitable for this purpose. However, if you are interested in the broader context of how an application interacts with the system (e.g., which system calls lead to HTTP traffic), syscall tracing can be highly valuable. In many advanced observability setups, both tools may be used simultaneously to provide a comprehensive view of system and network behavior.

Capturing HTTP Traffic with eBPF Socket Filter

eBPF code consists of user-space and kernel-space components, and here we primarily focus on the kernel-space code. Below is the main logic for capturing HTTP traffic in the kernel using eBPF socket filter technology, and the complete code is provided:

text
/* Detailed source-code truncated for AI context efficiency. */

When analyzing this eBPF program, we will explain it in detail according to the content of each code block and provide relevant background knowledge:

c
SEC("socket")
int socket_handler(struct __sk_buff *skb)
{
    // ...
}

This is the entry point of the eBPF program, defining a function named socket_handler that the kernel uses to handle incoming network packets. This function is located in an eBPF section named socket, indicating that it is intended for socket handling.

c
struct so_event *e;
__u8 verlen;
__u16 proto;
__u32 nhoff = ETH_HLEN;
__u32 ip_proto = 0;
__u32 tcp_hdr_len = 0;
__u16 tlen;
__u32 payload_offset = 0;
__u32 payload_length = 0;
__u8 hdr_len;

In this code block, several variables are defined to store information needed during packet processing. These variables include struct so_event *e for storing event information, verlen, proto, nhoff, ip_proto, tcp_hdr_len, tlen, payload_offset, payload_length, and hdr_len for storing packet information.

  • struct so_event *e;: This is a pointer to the so_event structure for storing captured event information. The specific definition of this structure is located elsewhere in the program.
  • __u8 verlen;, __u16 proto;, __u32 nhoff = ETH_HLEN;: These variables are used to store various pieces of information, such as protocol types, packet offsets, etc. nhoff is initialized to the length of the Ethernet frame header, typically 14 bytes, as Ethernet frame headers include destination MAC address, source MAC address, and frame type fields.
  • __u32 ip_proto = 0;: This variable is used to store the type of the IP protocol and is initialized to 0.
  • __u32 tcp_hdr_len = 0;: This variable is used to store the length of the TCP header and is initialized to 0.
  • __u16 tlen;: This variable is used to store the total length of the IP packet.
  • __u32 payload_offset = 0;, __u32 payload_length = 0;: These two variables are used to store the offset and length of the HTTP request payload.
  • __u8 hdr_len;: This variable is used to store the length of the IP header.
c
bpf_skb_load_bytes(skb, 12, &proto, 2);
proto = __bpf_ntohs(proto);
if (proto != ETH_P_IP)
    return 0;

Here, the code loads the Ethernet frame type field from the packet, which tells us the network layer protocol being used in the packet. It then uses the __bpf_ntohs function to convert the network byte order type field into host byte order. Next, the code checks if the type field is not equal to the Ethernet frame type for IPv4 (0x0800). If it's not equal, it means the packet is not an IPv4 packet, and the function returns 0, indicating that the packet should not be processed.

Key concepts to understand here:

  • Ethernet Frame: The Ethernet frame is a data link layer (Layer 2) protocol used for transmitting data frames within a local area network (LAN). Ethernet frames typically include destination MAC address, source MAC address, and frame type fields.
  • Network Byte Order: Network protocols often use big-endian byte order to represent data. Therefore, data received from the network needs to be converted into host byte order for proper interpretation on the host. Here, the type field from the network is converted to host byte order for further processing.
  • IPv4 Frame Type (ETH_P_IP): This represents the frame type field in the Ethernet frame, where 0x0800 indicates IPv4.
c
if (ip_is_fragment(skb, nhoff))
    return 0;

This part of the code checks if IP fragmentation is being handled. IP fragmentation is a mechanism for splitting larger IP packets into multiple smaller fragments for transmission. Here, if the packet is an IP fragment, the function returns 0, indicating that only complete packets will be processed.

c
static inline int ip_is_fragment(struct __sk_buff *skb, __u32 nhoff)
{
    __u16 frag_off;

    bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, frag_off), &frag_off, 2);
    frag_off = __bpf_ntohs(frag_off);
    return frag_off & (IP_MF | IP_OFFSET);
}

The above code is a helper function used to check if the incoming IPv4 packet is an IP fragment. IP fragmentation is a mechanism where, if the size of an IP packet exceeds the Maximum Transmission Unit (MTU) of the network, routers split it into smaller fragments for transmission across the network. The purpose of this function is to examine the fragment flags and fragment offset fields within the packet to determine if it is a fragment.

Here's an explanation of the code line by line:

  1. __u16 frag_off;: Defines a 16-bit unsigned integer variable frag_off to store the fragment offset field.
  2. bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, frag_off), &frag_off, 2);: This line of code uses the bpf_skb_load_bytes function to load the fragment offset field from the packet. nhoff is the offset of the IP header within the packet, and offsetof(struct iphdr, frag_off) calculates the offset of the fragment offset field within the IPv4 header.
  3. frag_off = __bpf_ntohs(frag_off);: Converts the loaded fragment offset field from network byte order (big-endian) to host byte order. Network protocols typically use big-endian to represent data, and the conversion to host byte order is done for further processing.
  4. return frag_off & (IP_MF | IP_OFFSET);: This line of code checks the value of the fragment offset field using a bitwise AND operation with two flag values:
    • IP_MF: Represents the "More Fragments" flag. If this flag is set to 1, it indicates that the packet is part of a fragmented sequence and more fragments are expected.
    • IP_OFFSET: Represents the fragment offset field. If the fragment offset field is non-zero, it indicates that the packet is part of a fragmented sequence and has a fragment offset value.
      If either of these flags is set to 1, the result is non-zero, indicating that the packet is an IP fragment. If both flags are 0, it means the packet is not fragmented.

It's important to note that the fragment offset field in the IP header is specified in units of 8 bytes, so the actual byte offset is obtained by left-shifting the value by 3 bits. Additionally, the "More Fragments" flag (IP_MF) in the IP header indicates whether there are more fragments in the sequence and is typically used in conjunction with the fragment offset field to indicate the status of fragmented packets.

c
bpf_skb_load_bytes(skb, ETH_HLEN, &

hdr_len, sizeof(hdr_len));
hdr_len &= 0x0f;
hdr_len *= 4;

In this part of the code, the length of the IP header is loaded from the packet. The IP header length field contains information about the length of the IP header in units of 4 bytes, and it needs to be converted to bytes. Here, it is converted by performing a bitwise AND operation with 0x0f and then multiplying it by 4.

Key concept:

  • IP Header: The IP header contains fundamental information about a packet, such as the source IP address, destination IP address, protocol type, total length, identification, flags, fragment offset, time to live (TTL), checksum, source port, and destination port.
c
if (hdr_len < sizeof(struct iphdr))
{
    return 0;
}

This code segment checks if the length of the IP header meets the minimum length requirement, typically 20 bytes. If the length of the IP header is less than 20 bytes, it indicates an incomplete or corrupted packet, and the function returns 0, indicating that the packet should not be processed.

Key concept:

  • struct iphdr: This is a structure defined in the Linux kernel, representing the format of an IPv4 header. It includes fields such as version, header length, service type, total length, identification, flags, fragment offset, time to live, protocol, header checksum, source IP address, and destination IP address, among others.
c
bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, protocol), &ip_proto, 1);
if (ip_proto != IPPROTO_TCP)
{
    return 0;
}

Here, the code loads the protocol field from the IP header to determine the transport layer protocol used in the packet. Then, it checks if the protocol field is not equal to the value for TCP (IPPROTO_TCP). If it's not TCP, it means the packet is not an HTTP request or response, and the function returns 0.

Key concept:

  • Transport Layer Protocol: The protocol field in the IP header indicates the transport layer protocol used in the packet, such as TCP, UDP, or ICMP.
c
tcp_hdr_len = nhoff + hdr_len;

This line of code calculates the offset of the TCP header. It adds the length of the Ethernet frame header (nhoff) to the length of the IP header (hdr_len) to obtain the starting position of the TCP header.

c
bpf_skb_load_bytes(skb, nhoff + 0, &verlen, 1);

This line of code loads the first byte of the TCP header from the packet, which contains information about the TCP header length. This length field is specified in units of 4 bytes and requires further conversion.

c
bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, tot_len), &tlen, sizeof(tlen));

This line of code loads the total length field of the IP header from the packet. The IP header's total length field represents the overall length of the IP packet, including both the IP header and the data portion.

c
__u8 doff;
bpf_skb_load_bytes(skb, tcp_hdr_len + offsetof(struct __tcphdr, ack_seq) + 4, &doff, sizeof(doff));
doff &= 0xf0;
doff >>= 4;
doff *= 4;

This piece of code is used to calculate the length of the TCP header. It loads the Data Offset field (also known as the Header Length field) from the TCP header, which represents the length of the TCP header in units of 4 bytes. The code clears the high four bits of the offset field, then shifts it right by 4 bits, and finally multiplies it by 4 to obtain the actual length of the TCP header.

Key points to understand:

  • TCP Header: The TCP header contains information related to the TCP protocol, such as source port, destination port, sequence number, acknowledgment number, flags (e.g., SYN, ACK, FIN), window size, and checksum.
c
payload_offset = ETH_HLEN + hdr_len + doff;
payload_length = __bpf_ntohs(tlen) - hdr_len - doff;

These two lines of code calculate the offset and length of the HTTP request payload. They add the lengths of the Ethernet frame header, IP header, and TCP header together to obtain the offset to the data portion of the HTTP request. Then, by subtracting the total length, IP header length, and TCP header length from the total length field, they calculate the length of the HTTP request data.

Key point:

  • HTTP Request Payload: The actual data portion included in an HTTP request, typically consisting of the HTTP request headers and request body.
c
char line_buffer[7];
if (payload_length < 7 || payload_offset < 0)
{
    return 0;
}
bpf_skb_load_bytes(skb, payload_offset, line_buffer, 7);
bpf_printk("%d len %d buffer: %s", payload_offset, payload_length, line_buffer);

This portion of the code loads the first 7 bytes of the HTTP request line and stores them in a character array named line_buffer. It then checks if the length of the HTTP request data is less than 7 bytes or if the offset is negative. If these conditions are met, it indicates an incomplete HTTP request, and the function returns 0. Finally, it uses the bpf_printk function to print the content of the HTTP request line to the kernel log for debugging and analysis.

c
if (bpf_strncmp(line_buffer, 3, "GET") != 0 &&
    bpf_strncmp(line_buffer, 4, "POST") != 0 &&
    bpf_strncmp(line_buffer, 3, "PUT") != 0 &&
    bpf_strncmp(line_buffer, 6, "DELETE") != 0 &&
    bpf_strncmp(line_buffer, 4, "HTTP") != 0)
{
    return 0;
}

Note: The bpf_strncmp function is a helper function available from kernel version 5.17. For earlier versions, you can manually write a function to compare strings.

This piece of code uses the bpf_strncmp function to compare the data in line_buffer with HTTP request methods (GET, POST, PUT, DELETE, HTTP). If there is no match, indicating that it is not an HTTP request, it returns 0, indicating that it should not be processed.

c
e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
if (!e)
    return 0;

This section of the code attempts to reserve a block of memory from the BPF ring buffer to store event information. If it cannot reserve the memory block, it returns 0. The BPF ring buffer is used to pass event data between the eBPF program and user space.

Key point:

  • BPF Ring Buffer: The BPF ring buffer is a mechanism for passing data between eBPF programs and user space. It can be used to store event information for further processing or analysis by user space applications.
c
e->ip_proto = ip_proto;
bpf_skb_load_bytes(skb, nhoff + hdr_len, &(e->ports), 4);
e->pkt_type = skb->pkt_type;
e->ifindex = skb->ifindex;

e->payload_length = payload_length;
bpf_skb_load_bytes(skb, payload_offset, e->payload, MAX_BUF_SIZE);

bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, saddr), &(e->src_addr), 4);
bpf_skb_load_bytes(skb, nhoff + offsetof(struct iphdr, daddr), &(e->dst_addr), 4);
bpf_ringbuf_submit(e, 0);

return skb->len;

Finally, this code segment stores the captured event information in the e structure and submits it to the BPF ring buffer. It includes information such as the captured IP protocol, source and destination ports, packet type, interface index, payload length, source IP address, and destination IP address. Finally, it returns the length of the packet, indicating that the packet was successfully processed.

This code is primarily used to store captured event information for further processing. The BPF ring buffer is used to pass this information to user space for additional handling or logging.

In summary, this eBPF program's main task is to capture HTTP requests. It accomplishes this by parsing the Ethernet frame, IP header, and TCP header of incoming packets to determine if they contain HTTP requests. Information about the requests is then stored in the so_event structure and submitted to the BPF ring buffer. This is an efficient method for capturing HTTP traffic at the kernel level and is suitable for applications such as network monitoring and security analysis.

Potential Limitations

The above code has some potential limitations, and one of the main limitations is that it cannot handle URLs that span multiple packets.

  • Cross-Packet URLs: The code checks the URL in an HTTP request by parsing a single data packet. If the URL of an HTTP request spans multiple packets, it will only examine the URL in the first packet. This can lead to missing or partially capturing long URLs that span multiple data packets.

To address this issue, a solution often involves reassembling multiple packets to reconstruct the complete HTTP request. This may require implementing packet caching and assembly logic within the eBPF program and waiting to collect all relevant packets until the HTTP request is detected. This adds complexity and may require additional memory to handle cases where URLs span multiple packets.

User-Space Code

The user-space code's main purpose is to create a raw socket and then attach the previously defined eBPF program in the kernel to that socket, allowing the eBPF program to capture and process network packets received on that socket. Here's an example of the user-space code:

c
/* Create raw socket for localhost interface */
sock = open_raw_sock(interface);
if (sock < 0) {
    err = -2;
    fprintf(stderr, "Failed to open raw socket\n");
    goto cleanup;
}

/* Attach BPF program to raw socket */
prog_fd = bpf_program__fd(skel->progs.socket_handler);
if (setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, &prog_fd, sizeof(prog_fd))) {
    err = -3;
    fprintf(stderr, "Failed to attach to raw socket\n");
    goto cleanup;
}
  1. sock = open_raw_sock(interface);: This line of code calls a custom function open_raw_sock, which is used to create a raw socket. Raw sockets allow a user-space application to handle network packets directly without going through the protocol stack. The interface parameter might specify the network interface from which to receive packets, determining where to capture packets from. If creating the socket fails, it returns a negative value, otherwise, it returns the file descriptor of the socket sock.
  2. If the value of sock is less than 0, indicating a failure to open the raw socket, it sets err to -2 and prints an error message on the standard error stream.
  3. prog_fd = bpf_program__fd(skel->progs.socket_handler);: This line of code retrieves the file descriptor of the socket filter program (socket_handler) previously defined in the eBPF program. It is necessary to attach this program to the socket. skel is a pointer to an eBPF program object, and it provides access to the program collection.
  4. setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, &prog_fd, sizeof(prog_fd)): This line of code uses the setsockopt system call to attach the eBPF program to the raw socket. It sets the SO_ATTACH_BPF option and passes the file descriptor of the eBPF program to the option, letting the kernel know which eBPF program to apply to this socket. If the attachment is successful, the socket starts capturing and processing network packets received on it.
  5. If setsockopt fails, it sets err to -3 and prints an error message on the standard error stream.

Compilation and Execution

The complete source code can be found at https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/23-http. To compile and run the code:

console
$ git submodule update --init --recursive
$ make
  BPF      .output/sockfilter.bpf.o
  GEN-SKEL .output/sockfilter.skel.h
  CC       .output/sockfilter.o
  BINARY   sockfilter
$ sudo ./sockfilter 
...

In another terminal, start a simple web server using Python:

console
python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
127.0.0.1 - - [18/Sep/2023 01:05:52] "GET / HTTP/1.1" 200 -

You can use curl to make requests:

c
$ curl http://0.0.0.0:8000/
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
....

In the eBPF program, you can see that it prints the content of HTTP requests:

console
127.0.0.1:34552(src) -> 127.0.0.1:8000(dst)
payload: GET / HTTP/1.1
Host: 0.0.0.0:8000
User-Agent: curl/7.88.1
...
127.0.0.1:8000(src) -> 127.0.0.1:34552(dst)
payload: HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.11.4
...

It captures both request and response content.

Capturing HTTP Traffic Using eBPF Syscall Tracepoints

eBPF provides a powerful mechanism for tracing system calls at the kernel level. In this example, we'll use eBPF to trace the accept and read system calls to capture HTTP traffic. Due to space limitations, we'll provide a brief overview of the code framework.

text
/* Detailed source-code truncated for AI context efficiency. */

This code briefly demonstrates how to use eBPF to trace system calls in the Linux kernel to capture HTTP traffic. Here's a detailed explanation of the hook locations and the flow, as well as the complete set of system calls that need to be hooked for comprehensive request tracing:

Hook Locations and Flow

  • The code uses eBPF Tracepoint functionality. Specifically, it defines a series of eBPF programs and binds them to specific system call Tracepoints to capture entry and exit events of these system calls.

  • First, it defines two eBPF hash maps (active_accept_args_map and active_read_args_map) to store system call parameters. These maps are used to track accept and read system calls.

  • Next, it defines multiple Tracepoint tracing programs, including:

    • sys_enter_accept: Defined at the entry of the accept system call, used to capture the arguments of the accept system call and store them in the hash map.
    • sys_exit_accept: Defined at the exit of the accept system call, used to process the result of the accept system call, including obtaining and storing the new socket file descriptor and related connection information.
    • sys_enter_read: Defined at the entry of the read system call, used to capture the arguments of the read system call and store them in the hash map.
    • sys_exit_read: Defined at the exit of the read system call, used to process the result of the read system call, including checking if the read data is HTTP traffic and sending events.

  • In sys_exit_accept and sys_exit_read, there is also some data processing and event sending logic, such as checking if the data is an HTTP connection, assembling event data, and using bpf_perf_event_output to send events to user space for further processing.

Complete Set of System Calls to Hook

To fully implement HTTP request tracing, the system calls that typically need to be hooked include:

  • socket: Used to capture socket creation for tracking new connections.
  • bind: Used to obtain port information where the socket is bound.
  • listen: Used to start listening for connection requests.
  • accept: Used to accept connection requests and obtain new socket file descriptors.
  • read: Used to capture received data and check if it contains HTTP requests.
  • write: Used to capture sent data and check if it contains HTTP responses.

The provided code already covers the tracing of accept and read system calls. To complete HTTP request tracing, additional system calls need to be hooked, and corresponding logic needs to be implemented to handle the parameters and results of these system calls.

The complete source code can be found at https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/23-http.

Summary

In today's complex technological landscape, system observability has become crucial, especially in the context of microservices and cloud-native applications. This article explores how to leverage eBPF technology for tracing the seven-layer protocols, along with the challenges and solutions that may arise in this process. Here's a summary of the content covered in this article:

  1. Introduction:

    • Modern applications often consist of multiple microservices and distributed components, making it essential to observe the behavior of the entire system.
    • Seven-layer protocols (such as HTTP, gRPC, MQTT, etc.) provide detailed insights into application interactions, but monitoring these protocols can be challenging.

  2. Role of eBPF Technology:

    • eBPF allows developers to dive deep into the kernel layer for real-time observation and analysis of system behavior without modifying or inserting application code.
    • eBPF technology offers a powerful tool for monitoring seven-layer protocols, especially in a microservices environment.

  3. Tracing Seven-Layer Protocols:

    • The article discusses the challenges of tracing seven-layer protocols, including their complexity and dynamism.
    • Traditional network monitoring tools struggle with the complexity of seven-layer protocols.

  4. Applications of eBPF:

    • eBPF provides two primary methods for tracing seven-layer protocols: socket filters and syscall tracing.
    • Both of these methods help capture network request data for protocols like HTTP and analyze them.

  5. eBPF Practical Tutorial:

    • The article provides a practical eBPF tutorial demonstrating how to capture and analyze HTTP traffic using eBPF socket filters or syscall tracing.
    • The tutorial covers the development of eBPF programs, the use of the eBPF toolchain, and the implementation of HTTP request tracing.

Through this article, readers can gain a deep understanding of how to use eBPF technology for tracing seven-layer protocols, particularly HTTP traffic. This knowledge will help enhance the monitoring and analysis of network traffic, thereby improving application performance and security. If you're interested in learning more about eBPF and its practical applications, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/tutorials/ for more examples and complete tutorials.

The original link of this article: https://eunomia.dev/tutorials/23-http

Overview (`src/24-hide/README.md`)

eBPF Practical Tutorial: Hiding Process or File Information

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows you to run, load, and update user-defined code without having to change the kernel source code or reboot the kernel. This capability allows eBPF to be used in a wide range of applications such as network and system performance analysis, packet filtering, and security policies.

In this tutorial, we will show how eBPF can be used to hide process or file information, a common technique in the field of network security and defence.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/24-hide

Background Knowledge and Implementation Mechanism

"Process hiding" enables a specific process to become invisible to the operating system's regular detection mechanisms. This technique can be used in both hacking and system defence scenarios. Specifically, each process on a Linux system has a subfolder named after its process ID in the /proc/ directory, which contains various information about the process. ps displays process information by looking in these folders. Therefore, if we can hide the /proc/ folder of a process, we can make that process invisible to ps commands and other detection methods.

The key to achieving process hiding is to manipulate the /proc/ directory. In Linux, the getdents64 system call can read the information of files in the directory. We can hide files by hooking into this system call and modifying the results it returns. To do this, you need to use eBPF's bpf_probe_write_user function, which can modify user-space memory, and therefore can be used to modify the results returned by getdents64.

In the following, we will describe in detail how to write eBPF programs in both kernel and user states to implement process hiding.

Kernel eBPF Program Implementation

Next, we will describe in detail how to write eBPF program to implement process hiding in kernel state. The first part of the eBPF programme is the start:

c
// SPDX-License-Identifier: BSD-3-Clause
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include "common.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

// Ringbuffer Map to pass messages from kernel to user
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

// Map to fold the dents buffer addresses
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, size_t);
    __type(value, long unsigned int);
} map_buffs SEC(".maps");

// Map used to enable searching through the
// data in a loop
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, size_t);
    __type(value, int);
} map_bytes_read SEC(".maps");

// Map with address of actual
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, size_t);
    __type(value, long unsigned int);
} map_to_patch SEC(".maps");

// Map to hold program tail calls
struct {
    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
    __uint(max_entries, 5);
    __type(key, __u32);
    __type(value, __u32);
} map_prog_array SEC(".maps");

The first thing we need to do is to understand the basic structure of the eBPF programme and the important components that are used. The first few lines reference several important header files, such as "vmlinux.h", "bpf_helpers.h", "bpf_tracing.h" and "bpf_core_read.h". These files provide the infrastructure needed for eBPF programming and some important functions or macros.

  • "vmlinux.h" is a header file containing the complete kernel data structures extracted from the vmlinux kernel binary. Using this header file, eBPF programs can access kernel data structures.
  • The "bpf_helpers.h" header file defines a series of macros that encapsulate the BPF helper functions used by eBPF programs. These BPF helper functions are the main way that eBPF programs interact with the kernel.
  • The "bpf_tracing.h" header file for tracing events contains a number of macros and functions designed to simplify the operation of tracepoints for eBPF programs.
  • The "bpf_core_read.h" header file provides a set of macros and functions for reading data from the kernel.

The program defines a series of map structures, which are the main data structures in an eBPF program, and are used to share data between the kernel and the user, or to store and transfer data within the eBPF program.

Among them, "rb" is a map of type Ringbuffer, which is used to pass messages from the kernel to the userland; Ringbuffer is a data structure that can efficiently pass large amounts of data between the kernel and the userland.

"map_buffs" is a map of type Hash which is used to store buffer addresses for directory entries.

"map_bytes_read" is another Hash-type map that is used to enable searching in data loops.

"map_to_patch" is another Hash type map that stores the address of the directory entry (dentry) that needs to be modified.

"map_prog_array" is a map of type Prog Array, which is used to store the tail calls of a programme.

The "target_ppid" and "pid_to_hide_len" and "pid_to_hide" in the program are a few important global variables that store the PID of the target parent process, the length of the PID that needs to be hidden, and the PID that needs to be hidden, respectively.

In the next part of the code, the program defines a structure called "linux_dirent64", which represents a Linux directory entry. The program then defines two functions, "handle_getdents_enter" and "handle_getdents_exit", which are called at the entry and exit of the getdents64 system call, respectively, and are used to implement operations on the directory entry.

c
// Optional Target Parent PID
const volatile int target_ppid = 0;

// These store the string representation
// of the PID to hide. This becomes the name
// of the folder in /proc/
const volatile int pid_to_hide_len = 0;
const volatile char pid_to_hide[MAX_PID_LEN];

// struct linux_dirent64 {
//     u64        d_ino;    /* 64-bit inode number */
//     u64        d_off;    /* 64-bit offset to next structure */
//     unsigned short d_reclen; /* Size of this dirent */
//     unsigned char  d_type;   /* File type */
//     char           d_name[]; /* Filename (null-terminated) */ }; 
// int getdents64(unsigned int fd, struct linux_dirent64 *dirp, unsigned int count);
SEC("tp/syscalls/sys_enter_getdents64")
int handle_getdents_enter(struct trace_event_raw_sys_enter *ctx)
{
    size_t pid_tgid = bpf_get_current_pid_tgid();
    // Check if we're a process thread of interest
    // if target_ppid is 0 then we target all pids
    if (target_ppid != 0) {
        struct task_struct *task = (struct task_struct *)bpf_get_current_task();
        int ppid = BPF_CORE_READ(task, real_parent, tgid);
        if (ppid != target_ppid) {
            return 0;
        }
    }
    int pid = pid_tgid >> 32;
    unsigned int fd = ctx->args[0];
    unsigned int buff_count = ctx->args[2];

    // Store params in map for exit function
    struct linux_dirent64 *dirp = (struct linux_dirent64 *)ctx->args[1];
    bpf_map_update_elem(&map_buffs, &pid_tgid, &dirp, BPF_ANY);

    return 0;
}

In this section of the code, we can see part of the implementation of the eBPF program that is responsible for the processing at the entry point of the getdents64 system call.

We start by declaring a few global variables. The target_ppid represents the PID of the target parent we want to focus on, and if this value is 0, then we will focus on all processes. pid_to_hide_len and pid_to_hide are used to store the length of the PID of the process we want to hide from, and the PID itself, respectively. This PID is translated into the name of a folder in the /proc/ directory, so the hidden process will not be visible in the /proc/ directory.

Next, we declare a structure called linux_dirent64. This structure represents a Linux directory entry and contains metadata such as the inode number, the offset of the next directory entry, the length of the current directory entry, the file type, and the filename.

Then there is the prototype for the getdents64 function. This function is a Linux system call that reads the contents of a directory. Our goal is to modify the directory entries during the execution of this function to enable process hiding.

The subsequent section is the concrete implementation of the eBPF program. We define a function called handle_getdents_enter at the entry point of the getdents64 system call. This function first gets the PID and thread group ID of the current process, and then checks to see if it is the process we are interested in. If we set target_ppid, then we only focus on processes whose parent has a PID of target_ppid. If target_ppid is 0, we focus on all processes.

After confirming that the current process is the one we are interested in, we save the arguments to the getdents64 system call into a map to be used when the system call returns. In particular, we focus on the second argument to the getdents64 system call, which is a pointer to the linux_dirent64 structure representing the contents of the directory to be read by the system call. We save this pointer, along with the current PID and thread group ID, as a key-value pair in the map_buffs map.

This completes the processing at the entry point of the getdents64 system call. When the system call returns, we will modify the directory entry in the handle_getdents_exit function to hide the process.

In the next snippet, we will implement the handling at the return of the getdents64 system call. Our main goal is to find the process we want to hide and modify the directory entry to hide it.

We start by defining a function called handle_getdents_exit that will be called when the getdents64 system call returns.

text
/* Detailed source-code truncated for AI context efficiency. */

In this function, we first get the PID and thread group ID of the current process, and then check to see if the system call has read the contents of the directory. If it didn't read the contents, we just return.

Then we get the address of the directory contents saved at the entry point of the getdents64 system call from the map_buffs map. If we haven't saved this address, then there's no need to do any further processing.

The next part is a bit more complicated, we use a loop to iteratively read the contents of the directory and check to see if we have the PID of the process we want to hide, and if we do, we use the bpf_tail_call function to jump to the handle_getdents_patch function to do the actual hiding.

text
/* Detailed source-code truncated for AI context efficiency. */

In the handle_getdents_patch function, we first check to see if we have found the PID of the process we want to hide, and then we read the contents of the directory entry and modify the d_reclen field so that it overwrites the next directory entry, thus hiding our target process.

In this process, we use the functions bpf_probe_read_user, bpf_probe_read_user_str, and bpf_probe_write_user to read and write user-space data. This is because in kernel space, we can't access user space data directly and must use these special functions.

After we finish the hiding operation, we send an event to a ring buffer called rb indicating that we have successfully hidden a process. We reserve space in the buffer with the bpf_ringbuf_reserve function, then fill that space with the event's data, and finally commit the event to the buffer with the bpf_ringbuf_submit function.

Finally, we clean up the data previously saved in the map and return.

This code is a good example of process hiding in an eBPF environment. Through this example, we can see the rich features provided by eBPF, such as system call tracing, map storage, user-space data access, tail calls, and so on. These features allow us to implement complex logic in kernel space without modifying the kernel code.

User-Style eBPF Programming

We perform the following operations in the userland eBPF program:

  1. Open the eBPF program.
  2. Set the PID of the process we want to hide.
  3. Verify and load the eBPF program.
  4. Wait for and process events sent by the eBPF program.

First, we open the eBPF application. This is done by calling the pidhide_bpf__open function. If this process fails, we simply return.

c
skel = pidhide_bpf__open();
    if (!skel)
    {
        fprintf(stderr, "Failed to open BPF program: %s\n", strerror(errno));
        return 1;
    }

Next, we set the PIDs of the processes we want to hide, which is done by saving the PIDs to the rodata area of the eBPF program. By default, we hide the current process.

c
char pid_to_hide[10];
    if (env.pid_to_hide == 0)
    {
        env.pid_to_hide = getpid();
    }
    sprintf(pid_to_hide, "%d", env.pid_to_hide);
    strncpy(skel->rodata->pid_to_hide, pid_to_hide, sizeof(skel->rodata->pid_to_hide));
    skel->rodata->pid_to_hide_len = strlen(pid_to_hide) + 1;
    skel->rodata->target_ppid = env.target_ppid;

We then validate and load the eBPF program. This is done by calling the pidhide_bpf__load function. If this process fails, we perform a cleanup operation.

c
err = pidhide_bpf__load(skel);
    if (err)
    {
        fprintf(stderr, "Failed to load and verify BPF skeleton\n");
        goto cleanup;
    }

Finally, we wait for and process events sent by the eBPF program. This process is achieved by calling the ring_buffer__poll function. During this process, we check the ring buffer every so often for new events. If there is, we call the handle_event function to handle the event.

c
printf("Successfully started!\n");
printf("Hiding PID %d\n", env.pid_to_hide);
while (!exiting)
{
    err = ring_buffer__poll(rb, 100 /* timeout, ms */);
    /* Ctrl-C will cause -EINTR */
    if (err == -EINTR)
    {
        err = 0;
        break;
    }
    if (err < 0)
    {
        printf("Error polling perf buffer: %d\n", err);
        break;
    }
}

In the handle_event function, we print the appropriate message based on the content of the event. The arguments to this function include a context, the data of the event, and the size of the data. We first convert the event data into an event structure, then determine if the event successfully hides a process based on the success field, and finally print the corresponding message.

and then print the corresponding message.

c
static int handle_event(void *ctx, void *data, size_t data_sz)
{
    const struct event *e = data;
    if (e->success)
        printf("Hid PID from program %d (%s)\n", e->pid, e->comm);
    else
        printf("Failed to hide PID from program %d (%s)\n", e->pid, e->comm);
    return 0;
}

This code shows how to use the eBPF programme to hide a process in the user state. We first open the eBPF application, then set the PID of the process we want to hide, then validate and load the eBPF application, and finally wait for and process the events sent by the eBPF application. This process makes use of some advanced features provided by eBPF, such as ring buffers and event handling, which allow us to easily interact with the kernel state eBPF program from the user state.

Full source code: https: //github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/24-hide

The techniques shown in this paper are for proof of concept only and are intended for learning purposes only, and are not to be used in scenarios that do not comply with legal or regulatory requirements.

Compile and Run

bash
make
sh
sudo ./pidhide --pid-to-hide 1534
console
$ ps -aux | grep 1534
yunwei      1534  0.0  0.0 244540  6848 ?        Ssl  6月02   0:00 /usr/libexec/gvfs-mtp-volume-monitor
yunwei     32065  0.0  0.0  17712  2580 pts/1    S+   05:43   0:00 grep --color=auto 1534
console
$ sudo ./pidhide --pid-to-hide 1534
Hiding PID 1534
Hid PID from program 31529 (ps)
Hid PID from program 31551 (ps)
Hid PID from program 31560 (ps)
Hid PID from program 31582 (ps)
Hid PID from program 31582 (ps)
Hid PID from program 31585 (bash)
Hid PID from program 31585 (bash)
Hid PID from program 31609 (bash)
Hid PID from program 31640 (ps)
Hid PID from program 31649 (ps)
console
$ ps -aux | grep 1534
root       31523  0.1  0.0  22004  5616 pts/2    S+   05:42   0:00 sudo ./pidhide -p 1534
root       31524  0.0  0.0  22004   812 pts/3    Ss   05:42   0:00 sudo ./pidhide -p 1534
root       31525  0.3  0.0   3808  2456 pts/3    S+   05:42   0:00 ./pidhide -p 1534
yunwei     31583  0.0  0.0  17712  2612 pts/1    S+   05:42   0:00 grep --color=auto 1534

Summary

You can also visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/zh/tutorials/ for more examples and the full tutorial.

The original link of this article: https://eunomia.dev/tutorials/24-hide

Overview (`src/25-signal/README.md`)

Using bpf_send_signal to Terminate Malicious Processes in eBPF

eBPF (Extended Berkeley Packet Filter) is a revolutionary technology in the Linux kernel that allows users to execute custom programs in kernel space without modifying the kernel source code or loading any kernel modules. This provides developers with great flexibility to observe, modify, and control the Linux system.

This article introduces how to use the bpf_send_signal feature of eBPF to intervene by sending signals to specified processes. For more tutorial documentation and complete source code, please refer to https://github.com/eunomia-bpf/bpf-developer-tutorial.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/25-signal

Use Cases

1. Performance Issues:

Optimizing the performance of applications is a core task for developers and system administrators in the modern software ecosystem. When applications, such as hhvm, run slowly or have abnormally high resource utilization, they can adversely affect the entire system. Therefore, pinpointing these performance bottlenecks and resolving them promptly is crucial.

2. Anomaly Detection and Response:

Any system running in a production environment may face various anomalies, from simple resource leaks to complex malware attacks. In these situations, the system needs to detect these anomalies quickly and accurately and take appropriate countermeasures.

3. Dynamic System Management:

With the rise of cloud computing and microservice architectures, dynamically adjusting resource configurations and application behaviors based on the current system state has become a key requirement. For example, auto-scaling based on traffic fluctuations or reducing CPU frequency when detecting system overheating.

Limitations of Existing Solutions

To meet the needs of the above use cases, traditional technical methods are as follows:

  • Install a bpf program that continuously monitors the system while polling a map.
  • When an event triggers specific conditions defined in the bpf program, it writes related data to this map.
  • Then, external analysis tools read data from this map and send signals to the target process based on the retrieved information.

Although this method is feasible in many scenarios, it has a major flaw: the time delay from when the event occurs to when the external tool responds can be relatively large. This delay can affect the speed of event response, making performance analysis results inaccurate or failing to respond promptly to malicious activity.

Advantages of the New Solution

To overcome the limitations of traditional methods, the Linux kernel offers the bpf_send_signal and bpf_send_signal_thread helper functions.

The main advantages of these functions include:

1. Real-time Response:

By sending signals directly from kernel space, avoiding extra overhead in user space, signals can be sent immediately after an event occurs, significantly reducing latency.

2. Accuracy:

Thanks to reduced latency, we can now obtain a more accurate snapshot of the system state, especially important for performance analysis and anomaly detection.

3. Flexibility:

These new helper functions provide developers with more flexibility. They can customize the signal sending logic according to different use cases and needs, allowing for more precise control and management of system behavior.

Kernel Code Analysis

In modern operating systems, a common security strategy is to monitor and control interactions between processes. Especially in Linux systems, the ptrace system call is a powerful tool that allows one process to observe and control the execution of another process, modifying its registers and memory. This makes it the primary mechanism for debugging and tracing tools like strace and gdb. However, malicious use of ptrace can also pose security risks.

The goal of this program is to monitor ptrace calls in kernel mode. When specific conditions are met, it sends a SIGKILL signal to terminate the calling process. Additionally, for debugging or auditing purposes, the program logs this intervention and sends related information to user space.

Code Analysis

1. Data Structure Definition (`signal.h`)

signal.h

c
// Simple message structure to get events from eBPF Programs
// in the kernel to user space
#define TASK_COMM_LEN 16
struct event {
    int pid;
    char comm[TASK_COMM_LEN];
    bool success;
};

This section defines a simple message structure used to pass events from eBPF programs in the kernel to user space. The structure includes the process ID, command name, and a boolean value indicating whether the signal was successfully sent.

2. eBPF Program (`signal.bpf.c`)

signal.bpf.c

c
// SPDX-License-Identifier: BSD-3-Clause
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include "common.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

// Ringbuffer Map to pass messages from kernel to user
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

// Optional Target Parent PID
const volatile int target_ppid = 0;

SEC("tp/syscalls/sys_enter_ptrace")
int bpf_dos(struct trace_event_raw_sys_enter *ctx)
{
    long ret = 0;
    size_t pid_tgid = bpf_get_current_pid_tgid();
    int pid = pid_tgid >> 32;

    // if target_ppid is 0 then we target all pids
    if (target_ppid != 0) {
        struct task_struct *task = (struct task_struct *)bpf_get_current_task();
        int ppid = BPF_CORE_READ(task, real_parent, tgid);
        if (ppid != target_ppid) {
            return 0;
        }
    }

    // Send signal. 9 == SIGKILL
    ret = bpf_send_signal(9);

    // Log event
    struct event *e;
    e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (e) {
        e->success = (ret == 0);
        e->pid = pid;
        bpf_get_current_comm(&e->comm, sizeof(e->comm));
        bpf_ringbuf_submit(e, 0);
    }

    return 0;
}

  • License Declaration

    The program's license is declared as "Dual BSD/GPL". This is to meet the Linux kernel's licensing requirements for eBPF programs.

  • Ringbuffer Map

    This is a ring buffer type map that allows messages generated by the eBPF program in kernel space to be efficiently read by user space programs.

  • Target Parent Process ID

    target_ppid is an optional parent process ID used to limit which processes are affected. If set to a non-zero value, only processes that match it will be targeted.

  • Main Function bpf_dos

    • Process Check
      The program first retrieves the current process's ID. If target_ppid is set, it also retrieves the current process's parent process ID and compares them. If they don't match, it returns immediately.

    • Sending Signal
      It uses bpf_send_signal(9) to send a SIGKILL signal. This terminates the process calling ptrace.

    • Logging the Event
      The event is logged using the ring buffer map. This includes whether the signal was successfully sent, the process ID, and the process's command name.

In summary, this eBPF program provides a method that allows system administrators or security teams to monitor and intervene ptrace calls at the kernel level, offering an additional layer against potential malicious activities or misoperations.

Compilation and Execution

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

Compilation:

bash
./ecc signal.bpf.c signal.h

Usage:

console
$ sudo ./ecli package.json
TIME     PID    COMM   SUCCESS

This program will send a SIG_KILL signal to any program attempting to use the ptrace system call, such as strace. Once the eBPF program starts running, you can test it by running the following command:

bash
$ strace /bin/whoami
Killed

The original console will output:

txt
INFO [bpf_loader_lib::skeleton] Running ebpf program...
TIME     PID    COMM   SUCCESS 
13:54:45  8857  strace true

The complete source code can be found at: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/25-signal

Conclusion

Through this example, we delved into how to combine eBPF programs with user-space programs to monitor and intervene in system calls. eBPF provides a mechanism for executing programs in kernel space. This technology is not limited to monitoring but can also be used for performance optimization, security defense, system diagnostics, and various other scenarios. For developers, it offers a powerful and flexible tool for performance tuning and troubleshooting in Linux systems.

Lastly, if you are interested in eBPF technology and wish to further understand and practice, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial and tutorial website https://eunomia.dev/zh/tutorials/.

References

The original link of this article: https://eunomia.dev/tutorials/25-signal

Overview (`src/26-sudo/README.md`)

eBPF Tutorial: Privilege Escalation via File Content Manipulation

eBPF's power extends far beyond simple tracingβ€”it can modify data flowing through the kernel in real-time. While this capability enables innovative solutions for performance optimization and security monitoring, it also opens doors to sophisticated attack vectors that traditional security tools might miss. This tutorial demonstrates one such technique: using eBPF to grant unprivileged users root access by manipulating what sudo sees when reading /etc/sudoers.

This example reveals how attackers could abuse eBPF's bpf_probe_write_user helper to bypass Linux's permission model entirely, without leaving traces in log files or modifying actual system files. Understanding these attack patterns is crucial for defenders building eBPF-aware security monitoring.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/26-sudo

The Attack Vector: Intercepting File Reads

Traditional privilege escalation attacks modify /etc/sudoers directly, leaving obvious traces in file timestamps, audit logs, and integrity monitoring systems. This eBPF-based approach is far more subtleβ€”it intercepts sudo's read operation and replaces the file content in memory before sudo processes it. The actual file on disk remains unchanged, defeating most file integrity monitors.

The attack works by exploiting a critical window: when sudo reads /etc/sudoers into a buffer, the data briefly exists in userspace memory. eBPF programs can access and modify this userspace memory using bpf_probe_write_user, effectively lying to sudo about what permissions exist without ever touching the real file.

Here's the attack flow: when any process opens /etc/sudoers, we record its file descriptor. When that same process reads from the file, we capture the buffer address. After the read completes, we overwrite the first line with ALL=(ALL:ALL) NOPASSWD:ALL #, making sudo believe the target user has full root privileges. The trailing # comments out whatever was originally on that line, preventing parse errors.

Implementation: Hooking the System Call Path

Let's examine how this attack is implemented in eBPF. The complete kernel-side code coordinates four system call hooks to track file operations and inject malicious content.

text
/* Detailed source-code truncated for AI context efficiency. */

The program uses a multi-stage approach. First, handle_openat_enter acts as a filterβ€”it checks that the process is sudo, that it's opening /etc/sudoers, and optionally that it matches a specific UID or parent PID. This filtering is critical because we don't want to affect every file operation on the system, only the specific case where sudo reads its configuration.

When sudo opens /etc/sudoers, the kernel returns a file descriptor. We catch this in handle_openat_exit and store the file descriptor in map_fds. This map links the process (identified by pid_tgid) to its sudoers file descriptor, so we know which reads to intercept.

The next hook, handle_read_enter, triggers when sudo calls read() on that file descriptor. The crucial detail here is capturing the buffer addressβ€”that's where the kernel will copy the file contents, and that's what we'll overwrite. We store this address in map_buff_addrs.

The attack executes in handle_read_exit. After the kernel completes the read operation and fills the buffer with the real sudoers content, we use bpf_probe_write_user to overwrite it. We replace the first line with our payload ( ALL=(ALL:ALL) NOPASSWD:ALL #) and fill the rest of the buffer with # characters to comment out the original content. From sudo's perspective, it read a legitimate sudoers file that grants our user full privileges.

Finally, handle_close_exit cleans up our tracking maps when sudo closes the file, preventing memory leaks.

Userspace Loader and Configuration

The userspace component is straightforwardβ€”it configures the attack parameters and loads the eBPF program. The critical part is setting up the payload string that will be injected into sudo's memory. This string is stored in the eBPF program's read-only data section, making it visible to the kernel at verification time but modifiable before loading.

The loader accepts command-line arguments to specify the username to grant privileges, optionally restrict the attack to a specific user or process tree, and then loads the eBPF program with these parameters baked into the bytecode. When sudo next runs, the attack executes automatically without any further userspace interaction needed.

Security Implications and Detection

This attack demonstrates why eBPF requires CAP_BPF or CAP_SYS_ADMIN capabilitiesβ€”these programs can fundamentally alter system behavior. An attacker who gains root access even briefly could load this eBPF program and maintain persistent access even after their initial foothold is removed.

Detection is challenging. The file on disk remains unchanged, so traditional file integrity monitoring fails. The attack happens entirely in kernel space during normal system call execution, leaving no unusual process behavior. However, defenders can look for loaded eBPF programs with write capabilities (bpftool prog list), monitor for bpf() system calls, or use eBPF-aware security tools that can inspect loaded programs.

Modern security platforms like Falco and Tetragon can detect suspicious eBPF activity by monitoring program loading and examining attached hooks. The key is maintaining visibility into the eBPF subsystem itself.

Compilation and Execution

Compile the program by running make in the tutorial directory:

bash
cd src/26-sudo
make

To test the attack (in a safe VM environment), run as root and specify a target username:

bash
sudo ./sudoadd --username lowpriv-user

This will intercept sudo operations and grant lowpriv-user root access without modifying /etc/sudoers. When lowpriv-user runs sudo, they'll be able to execute commands as root without entering a password. Other programs reading /etc/sudoers (like cat or text editors) will still see the original, unmodified file.

The --restrict flag limits the attack to only work when executed by the specified user, and --target-ppid can scope the attack to a specific process tree.

Summary

This tutorial showed how eBPF's memory manipulation capabilities can subvert Linux's security model by intercepting and modifying data flowing through the kernel. While powerful for legitimate debugging and monitoring, these same features enable sophisticated attacks that bypass traditional security controls. The key takeaway for defenders is that eBPF programs themselves must be treated as a critical part of your attack surfaceβ€”monitoring what eBPF programs are loaded and what capabilities they use is essential for modern Linux security.

If you'd like to dive deeper into eBPF, check out our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or visit our website at https://eunomia.dev/tutorials/.

References

Overview (`src/27-replace/README.md`)

eBPF Tutorial: Transparent Text Replacement in File Reads

When you read a file in Linux, you trust that what you see matches what's stored on disk. But what if the kernel itself was lying to you? This tutorial demonstrates how eBPF programs can intercept file read operations and silently replace text before applications ever see itβ€”creating a powerful capability for both defensive security monitoring and offensive rootkit techniques.

Unlike traditional file modification that leaves traces in timestamps and audit logs, this approach manipulates data in-flight during the read system call. The file on disk remains untouched, yet every program reading it sees modified content. This technique has legitimate uses in security research, honeypot deployment, and anti-malware deception, but also reveals how rootkits can hide their presence from system administrators.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/27-replace

Use Cases: From Security to Deception

Text replacement in file reads serves several purposes across the security spectrum. For defenders, it enables honeypot systems that present fake credentials to attackers, or deception layers that make malware believe it's succeeded when it hasn't. Security researchers use it to study malware behavior by feeding controlled data to suspicious processes.

On the offensive side, rootkits use this exact technique to hide their presence. The classic example is hiding kernel modules from lsmod by replacing their names in /proc/modules with whitespace or other module names. Malware can spoof MAC addresses by modifying reads from /sys/class/net/*/address, defeating sandbox detection that looks for virtual machine identifiers.

The key insight is that this operates at the system call boundaryβ€”after the kernel reads the file but before the userspace process sees the data. No matter how many times you cat the file or open it in different editors, you'll always see the modified version, because the eBPF program intercepts every read operation.

Architecture: Multi-Stage Text Scanning and Replacement

This implementation is more sophisticated than simple string replacement. The challenge is working within eBPF's constraints: limited stack size, no unbounded loops, and strict verifier checks. To handle arbitrarily large files and multiple matches, the program uses a three-stage approach with tail calls to chain eBPF programs together.

The first stage (find_possible_addrs) scans through the read buffer looking for characters that match the first character of our search string. It can't do full string matching yet due to complexity limits, so it just marks potential locations. These addresses are stored in map_name_addrs for the next stage.

The second stage (check_possible_addresses) is tail-called from the first. It examines each potential match location and performs full string comparison using bpf_strncmp. This verifies whether we actually found our target text. Confirmed matches go into map_to_replace_addrs.

The third stage (overwrite_addresses) loops through confirmed match locations and uses bpf_probe_write_user to overwrite the text with the replacement string. Because both strings must be the same length (to avoid shifting memory and corrupting the buffer), users must pad their replacement text to match.

This pipeline handles the verifier's complexity limits by splitting the work across multiple programs, each staying under the instruction count threshold. Tail calls provide the glue, allowing one program to pass control to the next with the same context.

Implementation Details

Let's examine the complete eBPF code that implements this three-stage pipeline:

text
/* Detailed source-code truncated for AI context efficiency. */

The program starts with the familiar pattern of tracking file opens. When a process opens our target file (specified via the filename constant), we record its file descriptor in map_fds. This lets us identify reads from that specific file later.

The interesting part begins in handle_read_enter, where we capture the buffer address that userspace passed to the read() system call. This address is where the kernel will write the file contents, and crucially, it's also where we can modify them before the userspace process looks at the data.

The main logic lives in find_possible_addrs, attached to sys_exit_read. After the kernel completes the read operation, we scan through the buffer looking for potential matches. The constraint here is that we can't do unbounded loopsβ€”the verifier would reject that. So we read in chunks of LOCAL_BUFF_SIZE bytes and scan for the first character of our search string. Each potential match address goes into map_name_addrs.

Once we've scanned the buffer, we use a tail call to jump into check_possible_addresses. This program iterates through the potential matches and performs full string comparison using bpf_strncmp (available in kernel 5.17+). Confirmed matches move to map_to_replace_addrs. If we found any matches, we tail-call once more into overwrite_addresses.

The final stage, overwrite_addresses, performs the actual modification using bpf_probe_write_user. It loops through confirmed match locations and overwrites each one with the replacement text. The requirement that both strings have the same length prevents buffer corruptionβ€”we're doing in-place replacement without shifting any memory.

Tail Calls and Verifier Constraints

The use of tail calls (bpf_tail_call) is critical here. eBPF programs face strict complexity limitsβ€”the verifier analyzes every possible execution path to ensure the program terminates and doesn't access invalid memory. A single program that does scanning, matching, and replacement would exceed these limits.

Tail calls provide a way to chain programs while bypassing the cumulative instruction count. When find_possible_addrs calls bpf_tail_call(ctx, &map_prog_array, PROG_01), it's essentially jumping to a different program (check_possible_addresses) with the same context. The current program's execution ends, and the new program starts with a fresh instruction count budget.

The userspace loader must populate map_prog_array with file descriptors for the tail-called programs before attaching anything. This is done in the userspace code using bpf_map_update_elem, mapping index PROG_01 to the check_possible_addresses program and PROG_02 to overwrite_addresses.

This architecture demonstrates a key eBPF development pattern: when you hit verifier limits, split your logic into multiple programs and use tail calls to coordinate them.

Practical Examples and Security Implications

Let's look at real-world use cases. Hiding kernel modules from detection:

bash
./replace -f /proc/modules -i 'joydev' -r 'cryptd'

When any process reads /proc/modules, they'll see cryptd where joydev actually appears. The module is still loaded and functioning, but tools like lsmod can't see it. This is a classic rootkit technique.

Spoofing MAC addresses for anti-sandbox evasion:

bash
./replace -f /sys/class/net/eth0/address -i '00:15:5d:01:ca:05' -r '00:00:00:00:00:00'

Malware often checks for virtualization by looking at MAC address prefixes (0x00:15:5d indicates Hyper-V). By replacing the actual MAC address with zeros, the malware's virtualization detection fails, making sandbox analysis easier.

The defensive flip side is using this for honeypot systems. You can present fake credentials in configuration files, or make malware believe it successfully compromised a system when it hasn't. The file content on disk remains secure, but attackers reading it see false information.

Compilation and Execution

Compile the program:

bash
cd src/27-replace
make

Run with specified file and text replacement:

bash
sudo ./replace --filename /path/to/file --input foo --replace bar

Both input and replace must be the same length to avoid buffer corruption. To include newlines in bash, use $'\n':

bash
./replace -f /proc/modules -i 'joydev' -r $'aaaa\n'

The program intercepts all reads of the specified file and replaces matching text transparently. Press Ctrl-C to stop.

Summary

This tutorial demonstrated how eBPF programs can intercept file read operations and modify data before userspace sees it, without altering the actual file. We explored the three-stage architecture using tail calls to work within verifier constraints, the use of bpf_probe_write_user for memory manipulation, and practical applications ranging from rootkit techniques to defensive honeypot deployment. Understanding these patterns is crucial for both offensive security research and building detection mechanisms that account for eBPF-based attacks.

If you'd like to dive deeper into eBPF, check out our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or visit our website at https://eunomia.dev/tutorials/.

References

Overview (`src/28-detach/README.md`)

Running eBPF After Application Exits: The Lifecycle of eBPF Programs

eBPF (Extended Berkeley Packet Filter) is a revolutionary technology in the Linux kernel that allows users to execute custom programs in kernel space without modifying the kernel source code or loading any kernel modules. This provides developers with great flexibility to observe, modify, and control the Linux system.

This article introduces the Lifecycle of eBPF Programs, how to run eBPF programs after user-space application exits, and how to use pin to share eBPF objects between processes. This article is part of the eBPF Developer Tutorial, more details can be found in https://github.com/eunomia-bpf/bpf-developer-tutorial and https://eunomia.dev/tutorials

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/28-detach

By using the detach method to run eBPF programs, the user space loader can exit without stopping the eBPF program. Another common use case for pinning is sharing eBPF objects between processes. For example, one could create a Map from Go, pin it, and inspect it using bpftool map dump pinned /sys/fs/bpf/my_map.

The Lifecycle of eBPF Programs

File descriptors and reference counters are used to manage BPF objects (progs, maps, and debug info). When a map is created, the kernel initializes its reference counter to 1 and returns a file descriptor to the user space process. If the process exits or crashes, the file descriptor is closed and the reference counter of the map is decremented. After the RCU grace period, the map is freed from memory.

BPF programs that use BPF maps are loaded in two phases. The maps are created and their file descriptors are stored in the program's 'imm' field. The kernel increments the reference counters of the maps used by the program and initializes the program's reference counter to 1. Even if the user space process closes the file descriptors associated with the maps, the maps will not disappear because the program is still "using" them. When the file descriptor of the program is closed and its reference counter reaches zero, the destruction logic decrements the reference counters of all maps used by the program. This allows the same map to be used by multiple programs at once.

When a program is attached to a hook, its reference counter is incremented. The user space process that created the maps and program can then exit, and the maps and program will remain alive as long as their reference counters are greater than zero. This is the lifecycle of a BPF object.

Not all attachment points are the same. XDP, tc's clsact, and cgroup-based hooks are global, meaning that programs will stay attached to them as long as the corresponding objects are alive. On the other hand, programs attached to kprobe, uprobe, tracepoint, perf_event, raw_tracepoint, socket filters, and so_reuseport hooks are local to the process. If the process crashes or closes the file descriptors associated with these hooks, the kernel will detach the BPF program and decrement its reference counter.

The file descriptor based interface provides auto-cleanup, meaning that if anything goes wrong with the user space process, the kernel will automatically clean up all BPF objects. This interface is useful for networking as well. The use of BPFFS (BPF File System) allows a process to pin a BPF program or map, which increments their reference counters and keeps them alive even if they are not attached or used by any program. This is useful when an admin wants to examine a map even when the associated program is not running.

Detach and replace are important aspects of the lifetime of a BPF program. The detach hook prevents the execution of a previously attached program from any future events, while the replace feature allows a program to be replaced in cgroup-based hooks. There is a window where the old and new programs can be executing on different CPUs, but the kernel guarantees that one of them will be processing events. Some BPF developers use a scheme where the new program is loaded with the same maps as the old program to ensure safe replacement.

Overall, understanding the lifetime of BPF programs and maps is crucial for users to use BPF safely and without surprises. The use of file descriptors, reference counters, and BPFFS helps manage the lifecycle of BPF objects, ensuring their proper creation, attachment, detachment, and replacement.

eBPF in Kubernetes: Deploy eBPF Programs via Remote Procedure Call

In a Kubernetes environment, deploying eBPF programs often necessitates a higher level of system privileges. Typically, these applications require at least CAP_BPF permissions, and depending on the program type, they may need even more. This requirement poses a challenge in a multi-tenant Kubernetes environment where granting extensive privileges can be a security risk.

Using Pin to Mitigate Privilege Requirements

One way to address the privilege issue is through the use of pinning eBPF maps. Pinning allows eBPF objects to persist beyond the life of the process that created them, making them accessible to other processes. This method can be particularly useful in Kubernetes, where different containers might need to interact with the same eBPF objects.

For example, an eBPF map can be created and pinned by a privileged initializer container. Subsequent containers, which may run with fewer privileges, can then interact with the pinned eBPF objects. This approach limits the need for elevated privileges to the initialization phase, thereby enhancing overall security.

The Role of bpfman in eBPF Lifecycle Management

The bpfman project can play a crucial role in this context. bpfman, or BPF Daemon, is designed to manage the lifecycle of eBPF programs and maps in a more controlled and secure manner. It acts as a mediator between user space and kernel space, providing a mechanism to load and manage eBPF programs without granting extensive privileges to each individual container or application.

In Kubernetes, bpfman could be deployed as a privileged service, responsible for loading and managing eBPF programs across different nodes in the cluster. It can handle the intricacies of eBPF lifecycle management, such as loading, unloading, updating eBPF programs, and managing their state. This centralized approach simplifies the deployment and management of eBPF programs in a Kubernetes cluster, while adhering to security best practices.

Use Detach to Replace by Any Program with eBPF After it Exits

In libbpf, the bpf_object__pin_maps function can be used to pin the maps in the BPF object, the programs and links has similar API.

Here we use similar programs as textreplace in the previous section to demonstrate the detach method, the pin eBPF code is like:

c
int pin_program(struct bpf_program *prog, const char* path)
{
    int err;
    err = bpf_program__pin(prog, path);
        if (err) {
            fprintf(stdout, "could not pin prog %s: %d\n", path, err);
            return err;
        }
    return err;
}

int pin_map(struct bpf_map *map, const char* path)
{
    int err;
    err = bpf_map__pin(map, path);
        if (err) {
            fprintf(stdout, "could not pin map %s: %d\n", path, err);
            return err;
        }
    return err;
}

int pin_link(struct bpf_link *link, const char* path)
{
    int err;
    err = bpf_link__pin(link, path);
        if (err) {
            fprintf(stdout, "could not pin link %s: %d\n", path, err);
            return err;
        }
    return err;
}

Running

Here, we still use the example of string replacement used in the previous application to demonstrate potential security risks. By using --detach to run the program, the user space loader can exit without stopping the eBPF program.

The code of This example can be found in https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/28-detach

Compilation:

bash
make

Before running, please make sure that the BPF file system has been mounted:

bash
sudo mount bpffs -t bpf /sys/fs/bpf
mkdir /sys/fs/bpf/textreplace

Then, you can run text-replace2 with detach:

bash
./textreplace2 -f /proc/modules -i 'joydev' -r 'cryptd' -d

This will create some eBPF link files under /sys/fs/bpf/textreplace. Once the loader is successfully running, you can check the log by running the following command:

bash
sudo cat /sys/kernel/debug/tracing/trace_pipe
# Confirm that the link files exist
sudo ls -l /sys/fs/bpf/textreplace

Finally, to stop, simply delete the link files:

bash
sudo rm -r /sys/fs/bpf/textreplace

References

You can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/zh/tutorials/ for more examples and a complete tutorial.

The original link of this article: https://eunomia.dev/tutorials/28-detach

Overview (`src/29-sockops/README.md`)

eBPF Development Practices: Accelerating Network Request Forwarding with Sockops

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows running, loading, and updating user-defined code without the need to modify the kernel source code or reboot the kernel. This capability makes eBPF widely used in various areas such as network and system performance analysis, packet filtering, security policies, etc.

This tutorial will focus on the application of eBPF in the networking domain, specifically how to use sockops-type eBPF programs to accelerate the forwarding of local network requests. This application is often valuable in scenarios where software load balancers are used for request forwarding, such as using tools like Nginx or HAProxy.

In many workloads, such as inter-service communication in a microservices architecture, the performance overhead of network requests made through the loopback interface can significantly impact the overall application performance. Since these requests have to go through the local network stack, their processing performance can become a bottleneck, especially in high-concurrency scenarios. To address this issue, sockops-type eBPF programs can be used to accelerate local request forwarding, providing functionality similar to direct memory access (DMA). Sockops programs can manage sockets in the kernel space and directly forward packets between sockets on the local machine, reducing the CPU time required for packet forwarding in the TCP/IP stack.

This tutorial will demonstrate how to use sockops-type eBPF programs to accelerate network request forwarding through a specific example. To help you understand how to use sockops programs, we will step by step introduce the code of the example program and discuss the working principle of each part.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/29-sockops

Leveraging eBPF Sockops for Performance Optimization

Network connections are essentially communication between sockets, and eBPF provides a bpf_msg_redirect_hash function that allows packets sent by an application to be directly forwarded to the corresponding socket on the recipient side, greatly accelerating the packet processing flow in the kernel.

Here, the sock_map is a key component that stores socket rules, i.e., it selects an existing socket connection from the sock_map based on the current packet information. Therefore, it is necessary to save the socket information to the sock_map at the hook of the sockops or elsewhere and provide a rule (usually a four-tuple) to find the socket based on the key.

The Merbridge project has achieved acceleration for Istio by replacing iptables with eBPF. After using Merbridge (eBPF) optimization, the inbound and outbound traffic bypasses many kernel modules, significantly improving performance, as shown in the following diagram:

Example Program

This example program redirects traffic from the sender’s socket (outgoing) to the recipient’s socket (incoming), bypassing the TCP/IP kernel network stack. In this example, we assume that the sender and recipient are both running on the same machine. This example program has two parts that share a map definition:

bpf_sockmap.h

c
#include "vmlinux.h"
#include <bpf/bpf_endian.h>
#include <bpf/bpf_helpers.h>

#define LOCALHOST_IPV4 16777343

struct sock_key {
    __u32 sip;
    __u32 dip;
    __u32 sport;
    __u32 dport;
    __u32 family;
};

struct {
 __uint(type, BPF_MAP_TYPE_SOCKHASH);
 __uint(max_entries, 65535);
 __type(key, struct sock_key);
 __type(value, int);
} sock_ops_map SEC(".maps");

The BPF program in this example is divided into two parts: bpf_redirect.bpf.c and bpf_contrack.bpf.c.

  • The BPF code in bpf_contrack.bpf.c defines a socket operation (sockops) program, whose main function is to create an entry in the sock_ops_map BPF map in which it stores the five-tuple (source address, destination address, source port, destination port, protocol) for each new TCP connection established on the local machine (using localhost). This BPF map is defined as type BPF_MAP_TYPE_SOCKHASH and can store sockets and their corresponding five-tuple. This allows the five-tuple information of each local TCP connection to be found in the BPF map whenever the connection is created.

  • The BPF code in bpf_redirect.bpf.c defines a sk_msg handler that is called when a message arrives on a local socket. The sk_msg program checks if the message is from a local address, and if so, it retrieves the five-tuple (source address, destination address, source port, destination port, protocol) from the message and looks up the corresponding socket in the sock_ops_map using the obtained key. Then, it redirects the message to the socket found in the sock_ops_map, thus bypassing the kernel network stack and directly delivering the message from the sender's socket to the receiver's socket.

For example, let's assume that there are two processes running locally, process A binds to port 8000, and process B binds to port 9000. Process A sends a message to process B.

  1. When the TCP connection is first established between process A and process B, the sockops program in bpf_contrack.bpf.c is triggered, and it creates an entry in the sock_ops_map BPF map for the five-tuple {127.0.0.1, 127.0.0.1, 8000, 9000, TCP}, with the value being the socket of process A.

  2. When process A sends a message, the sk_msg program in bpf_redirect.bpf.c is triggered, and it redirects the message from process A's socket to the socket stored in the sock_ops_map based on the obtained five-tuple information (source address, destination address, source port, destination port, protocol). As a result, the message is directly delivered from process A to process B, bypassing the kernel network stack.

This example program uses BPF to efficiently redirect messages from the sender's socket to the recipient's socket during local communication, bypassing the kernel network stack to improve transmission efficiency.

bpf_redirect.bpf.c

c
#include "bpf_sockmap.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("sk_msg")
int bpf_redir(struct sk_msg_md *msg)
{
    if(msg->remote_ip4 != LOCALHOST_IPV4 || msg->local_ip4!= LOCALHOST_IPV4) 
        return SK_PASS;
    
    struct sock_key key = {
        .sip = msg->remote_ip4,
        .dip = msg->local_ip4,
        .dport = bpf_htonl(msg->local_port), /* convert to network byte order */
        .sport = msg->remote_port,
        .family = msg->family,
    };
    return bpf_msg_redirect_hash(msg, &sock_ops_map, &key, BPF_F_INGRESS);
}

bpf_contrack.bpf.c

c
#include "bpf_sockmap.h"

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("sockops")
int bpf_sockops_handler(struct bpf_sock_ops *skops){
    u32 family, op;

    family = skops->family;
    op = skops->op;
    if (op != BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB
        && op != BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB) {
        return BPF_OK;
    }

    if(skops->remote_ip4 != LOCALHOST_IPV4 || skops->local_ip4!= LOCALHOST_IPV4) {
        return BPF_OK;
    }

    struct sock_key key = {
        .dip = skops->remote_ip4,
        .sip = skops->local_ip4,
        .sport = bpf_htonl(skops->local_port),  /* convert to network byte order */
        .dport = skops->remote_port,
        .family = skops->family,
    };

    bpf_printk(">>> new connection: OP:%d, PORT:%d --> %d\n", op, bpf_ntohl(key.sport), bpf_ntohl(key.dport));

    bpf_sock_hash_update(skops, &sock_ops_map, &key, BPF_NOEXIST);
    return BPF_OK;
}

Compiling the eBPF Program

Here, we use libbpf to compile the eBPF program. The complete source code and project can be found at https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/29-sockops.

shell
# Compile the bpf program with libbpf
make

Loading the eBPF Program

We have created a script to load the eBPF program, which will automatically load both eBPF programs and create a BPF map:

shell
sudo ./load.sh

This script actually performs the following operations:

sh
#!/bin/bash
set -x
set -e

sudo mount -t bpf bpf /sys/fs/bpf/

# check if old program already loaded
if [ -e "/sys/fs/bpf/bpf_sockops" ]; then
    echo ">>> bpf_sockops already loaded, uninstalling..."
    ./unload.sh
    echo ">>> old program already deleted..."
fi

# load and attach sock_ops program
sudo bpftool prog load bpf_contrack.bpf.o /sys/fs/bpf/bpf_sockops type sockops pinmaps /sys/fs/bpf/
sudo bpftool cgroup attach "/sys/fs/cgroup/" sock_ops pinned "/sys/fs/bpf/bpf_sockops"

# load and attach sk_msg program 
sudo bpftool prog load bpf_redirect.bpf.o "/sys/fs/bpf/bpf_redir" map name sock_ops_map pinned "/sys/fs/bpf/sock_ops_map"
sudo bpftool prog attach pinned /sys/fs/bpf/bpf_redir msg_verdict pinned /sys/fs/bpf/sock_ops_map

This is a script for loading BPF programs. Its main function is to load and attach BPF programs to the kernel system, and store the associated BPF maps in the BPF file system so that the BPF programs can access and operate on these maps.

Let's take a detailed look at what each line of the script does.

  1. sudo mount -t bpf bpf /sys/fs/bpf/ mounts the BPF file system, enabling access to and operation on BPF programs and related maps by the system.
  2. The condition check [ -e "/sys/fs/bpf/bpf_sockops" ] checks whether the /sys/fs/bpf/bpf_sockops file already exists. If it does exist, it means that the bpf_sockops program has already been loaded into the system, so it will be uninstalled using the ./unload.sh script.
  3. sudo bpftool prog load bpf_contrack.bpf.o /sys/fs/bpf/bpf_sockops type sockops pinmaps /sys/fs/bpf/ loads the BPF object file bpf_contrack.bpf.o compiled from the bpf_contrack.bpf.c into the BPF file system, storing it in /sys/fs/bpf/bpf_sockops, and specifying its type as sockops. pinmaps /sys/fs/bpf/ specifies that the BPF maps associated with the loaded BPF program will be stored under /sys/fs/bpf/.
  4. sudo bpftool cgroup attach "/sys/fs/cgroup/" sock_ops pinned "/sys/fs/bpf/bpf_sockops" attaches the bpf_sockops program that has been loaded into the BPF file system to the cgroup (the path is "/sys/fs/cgroup/"). After the attachment, all socket operations belonging to this cgroup will be affected by the bpf_sockops program.
  5. sudo bpftool prog load bpf_redirect.bpf.o "/sys/fs/bpf/bpf_redir" map name sock_ops_map pinned "/sys/fs/bpf/sock_ops_map" loads the BPF object file bpf_redirect.bpf.o compiled from bpf_redirect.bpf.c into the BPF file system, storing it in /sys/fs/bpf/bpf_redir, and specifying the associated map as sock_ops_map, which is located in /sys/fs/bpf/sock_ops_map.
  6. sudo bpftool prog attach pinned /sys/fs/bpf/bpf_redir msg_verdict pinned /sys/fs/bpf/sock_ops_map attaches the already loaded bpf_redir program to the sock_ops_map using the msg_verdict attachment type, which means that when the socket associated with this map receives a message, the bpf_redir program will be called to handle it.

In summary, the main function of this script is to load the two BPF programs used to process local socket traffic into the system and attach them to the correct locations so that they can be correctly called, ensuring that they can access and manipulate the associated BPF maps.

You can use the bpftool utility to check if these two eBPF programs have been loaded.

console
$ sudo bpftool prog show
63: sock_ops  name bpf_sockmap  tag 275467be1d69253d  gpl
 loaded_at 2019-01-24T13:07:17+0200  uid 0
 xlated 1232B  jited 750B  memlock 4096B  map_ids 58
64: sk_msg  name bpf_redir  tag bc78074aa9dd96f4  gpl
 loaded_at 2019-01-24T13:07:17+0200  uid 0
 xlated 304B  jited 233B  memlock 4096B  map_ids 58

Test with iperf3 or curl

See https://iperf.fr/ to install iperf3.

Running the iperf3 Server:

shell
iperf3 -s -p 5001

Running the iperf3 Client:

shell
iperf3 -c 127.0.0.1 -t 10 -l 64k -p 5001

Or you can use curl and python:

sh
python3 -m http.server
curl http://0.0.0.0:8000/

Collecting Traces

Check the sock_ops trace for local connection establishments.

console
$ ./trace_bpf_output.sh # which is basically sudo cat /sys/kernel/debug/tracing/trace_pipe
iperf3-9516  [001] .... 22500.634108: 0: <<< ipv4 op = 4, port 18583 --> 4135
iperf3-9516  [001] ..s1 22500.634137: 0: <<< ipv4 op = 5, port 4135 --> 18583
iperf3-9516  [001] .... 22500.634523: 0: <<< ipv4 op = 4, port 19095 --> 4135
iperf3-9516  [001] ..s1 22500.634536: 0: <<< ipv4 op = 5, port 4135 --> 19095

When the connection is established between iperf3 -c and the server, you should see the events above for socket establishment. If you don't see any events, then the eBPF programs may not have been attached correctly.

Furthermore, when sk_msg takes effect, you should observe that when capturing local traffic on the loopback interface using tcpdump, only the three-way handshake and four-way termination traffic are captured, and the actual data flow of iperf is not captured. If the iperf data flow is captured, then the eBPF programs may not have been attached correctly.

console
$ ./trace_lo_traffic.sh # tcpdump -i lo port 5001

# Three-way handshake
13:24:07.181804 IP localhost.46506 > localhost.5001: Flags [S], seq 620239881, win 65495, options [mss 65495,sackOK,TS val 1982813394 ecr 0,nop,wscale 7], length 0
13:24:07.181815 IP localhost.5001 > localhost.46506: Flags [S.], seq 1084484879, ack 620239882, win 65483, options [mss 65495,sackOK,TS val 1982813394 ecr 1982813394,nop,wscale 7], length 0
13:24:07.181832 IP localhost.46506 > localhost.5001: Flags [.], ack 1, win 512, options [nop,nop,TS val 1982813394 ecr 1982813394], length 0

# Four-way termination
13:24:12.475649 IP localhost.46506 > localhost.5001: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 1982818688 ecr 1982813394], length 0
13:24:12.479621 IP localhost.5001 > localhost.46506: Flags [.], ack 2, win 512, options [nop,nop,TS val 1982818692 ecr 1982818688], length 0
13:24:12.481265 IP localhost.5001 > localhost.46506: Flags [F.], seq 1, ack 2, win 512, options [nop,nop,TS val 1982818694 ecr 1982818688], length 0
13:24:12.481270 IP localhost.46506 > localhost.5001: Flags [.], ack 2, win 512, options [nop,nop,TS val 1982818694 ecr 1982818694], length 0

Unloading the eBPF Program

shell
sudo ./unload.sh

References

Finally, if you are interested in eBPF technology and want to learn more and practice further, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial and the tutorial website at https://eunomia.dev/.

The original link of this article: https://eunomia.dev/tutorials/29-sockops

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and execute user-defined code at runtime in the kernel.

This article is the third part of the eBPF Tutorial by Example, focusing on capturing unlink system calls using fentry in eBPF.

What is fentry and Why Use It?

fentry (function entry) and fexit (function exit) are the modern way to trace kernel functions in eBPF. They were introduced in kernel 5.5 for x86 processors and 6.0 for ARM processors. Think of them as the faster, more efficient successors to kprobes.

The big advantage of fentry/fexit over kprobes is performance and convenience. With fentry, you can directly access function parameters just like in regular C code - no need for special helpers like BPF_CORE_READ. This makes your code simpler and faster. fexit gives you an extra bonus - you can access both the input parameters and the return value at the same time, while kretprobe only gives you the return value.

The performance difference is real. fentry/fexit programs run about 10x faster than kprobes because they use a BPF trampoline mechanism instead of the older breakpoint-based approach. If you're building production monitoring tools that run on every function call, this matters a lot.

Note that if you're on ARM, you'll need kernel 6.0 or newer. For x86, kernel 5.5+ works fine. If you're stuck on an older kernel, you'll need to use kprobes instead.

c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("fentry/do_unlinkat")
int BPF_PROG(do_unlinkat, int dfd, struct filename *name)
{
    pid_t pid;

    pid = bpf_get_current_pid_tgid() >> 32;
    bpf_printk("fentry: pid = %d, filename = %s\n", pid, name->name);
    return 0;
}

SEC("fexit/do_unlinkat")
int BPF_PROG(do_unlinkat_exit, int dfd, struct filename *name, long ret)
{
    pid_t pid;

    pid = bpf_get_current_pid_tgid() >> 32;
    bpf_printk("fexit: pid = %d, filename = %s, ret = %ld\n", pid, name->name, ret);
    return 0;
}

Let's break down how this program works. The fentry probe attaches to the entry of do_unlinkat and can access the function's parameters directly. Notice how we access name->name without any special helpers - this is one of the benefits of using fentry instead of kprobes.

The fexit probe is even more interesting. It gets triggered when the function returns, and it can access both the original parameters (dfd and name) and the return value (ret). This gives you complete visibility into what the function did. If ret is 0, the file was successfully deleted. If it's negative, something went wrong and you can see the error code.

The BPF_PROG macro is similar to BPF_KPROBE from the previous tutorial, but it's designed for fentry/fexit. It handles the parameter unwrapping automatically so you can focus on your logic.

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

To compile and run the above code:

console
$ ecc fentry-link.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Running eBPF program...

In another window:

shell
touch test_file
rm test_file
touch test_file2
rm test_file2

After running this program, you can view the output of the eBPF program by examining the /sys/kernel/debug/tracing/trace_pipe file:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
              rm-9290    [004] d..2  4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file
              rm-9290    [004] d..2  4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file, ret = 0
              rm-9290    [004] d..2  4637.798698: bpf_trace_printk: fentry: pid = 9290, filename = test_file2
              rm-9290    [004] d..2  4637.798843: bpf_trace_printk: fexit: pid = 9290, filename = test_file2, ret = 0

Troubleshooting

If you encounter errors when running this example, here are some common issues and solutions:

Error: "failed to attach: ERROR: strerror_r(-524)=22"

This error (error code -524 = ENOTSUPP) typically means your kernel doesn't support fentry/fexit. Here's how to troubleshoot:

1. Check your kernel version:

console
$ uname -r

You need:

  • Kernel 5.5 or newer for x86/x86_64 processors
  • Kernel 6.0 or newer for ARM/ARM64 processors

If your kernel is too old, you have two options:

2. Verify BTF (BPF Type Format) support:

BTF is required for fentry/fexit to work. Check if it's enabled:

console
$ cat /boot/config-$(uname -r) | grep CONFIG_DEBUG_INFO_BTF
CONFIG_DEBUG_INFO_BTF=y

If BTF is not enabled, you'll need to either:

  • Use a kernel with BTF support enabled
  • Use the kprobe example as an alternative

3. Check if the kernel function exists:

The function do_unlinkat may have a different name or may not be exported in some kernel versions. You can check available functions:

console
$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep unlink

If do_unlinkat is not listed, the function may not be available for tracing on your kernel.

4. Verify your kernel configuration:

Ensure your kernel was compiled with the necessary eBPF features:

console
$ cat /boot/config-$(uname -r) | grep BPF

Look for these important settings:

  • CONFIG_BPF=y
  • CONFIG_BPF_SYSCALL=y
  • CONFIG_DEBUG_INFO_BTF=y
  • CONFIG_BPF_JIT=y

If you're still experiencing issues after checking these items, please report your kernel version and OS distribution by running:

console
$ uname -a
$ cat /etc/os-release

Summary

This program is an eBPF program that captures the do_unlinkat and do_unlinkat_exit functions using fentry and fexit, and uses bpf_get_current_pid_tgid and bpf_printk functions to obtain the ID, filename, and return value of the process calling do_unlinkat, and print them in the kernel log.

To compile this program, you can use the ecc tool, and to run it, you can use the ecli command, and view the output of the eBPF program by checking the /sys/kernel/debug/tracing/trace_pipe file.

If you'd like to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and complete tutorials.

Overview (`src/30-sslsniff/README.md`)

eBPF Practical Tutorial: Capturing SSL/TLS Plain Text Data Using uprobe

With the widespread use of TLS in modern network environments, tracing microservices RPC messages has become increasingly challenging. Traditional traffic sniffing techniques often face limitations in accessing only encrypted data, preventing a genuine observation of the original communication content. This restriction poses significant obstacles to system debugging and analysis.

However, a new solution is now available. Through the use of eBPF technology and its capability to perform probing in user space, a method has emerged to regain plain text data, allowing us to intuitively view the pre-encrypted communication content. Nevertheless, each application might utilize different libraries, and each library comes in multiple versions, introducing complexity to the tracking process.

In this tutorial, we will guide you through an eBPF tracing technique that spans across various user-space SSL/TLS libraries. This technique not only allows simultaneous tracing of user-space libraries like GnuTLS and OpenSSL but also significantly reduces maintenance efforts for new library versions compared to previous methods.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/30-sslsniff

Background Knowledge

Before delving into the main topic of this tutorial, we need to grasp some core concepts that will serve as the foundation for our subsequent discussions.

SSL and TLS

SSL (Secure Sockets Layer): Developed by Netscape in the early 1990s, SSL provides data encryption for communication between two machines on a network. However, due to known security vulnerabilities, SSL has been succeeded by its successor, TLS.

TLS (Transport Layer Security): TLS is the successor to SSL, aiming to provide stronger and more secure data encryption methods. TLS operates through a handshake process during which a client and a server select an encryption algorithm and corresponding keys. Once the handshake is complete, data transmission begins, with all data being encrypted using the chosen algorithm and keys.

Operation Principles of TLS

Transport Layer Security (TLS) is a cryptographic protocol designed to provide security for communication over computer networks. Its primary goal is to provide security, including privacy (confidentiality), integrity, and authenticity, for two or more communicating computer applications over a network using cryptography, such as certificates. TLS consists of two sub-layers: the TLS Record Protocol and the TLS Handshake Protocol.

Handshake Process

When a client connects to a TLS-enabled server and requests a secure connection, the handshake process begins. The handshake allows the client and server to establish security parameters for the connection using asymmetric cryptography. The complete process is as follows:

  1. Initial Handshake: The client connects to the TLS-enabled server, requests a secure connection, and provides a list of supported cipher suites (encryption algorithms and hash functions).
  2. Selecting Cipher Suite: From the provided list, the server chooses a cipher suite and hash function it also supports and notifies the client of the decision.
  3. Providing Digital Certificate: Usually, the server then provides identity authentication in the form of a digital certificate. This certificate includes the server's name, trusted certificate authorities (guaranteeing the certificate's authenticity), and the server's public encryption key.
  4. Certificate Verification: The client verifies the certificate's validity before proceeding.
  5. Generating Session Key: To create a session key for a secure connection, the client has two methods:
    • Encrypt a random number (PreMasterSecret) with the server's public key and send the result to the server (only the server can decrypt it with its private key); both parties then use this random number to generate a unique session key for encrypting and decrypting data during the session.
    • Use Diffie-Hellman key exchange (or its variant, Elliptic Curve DH) to securely generate a random and unique session key for encryption and decryption. This key has the additional property of forward secrecy: even if the server's private key is exposed in the future, it can't be used to decrypt the current session, even if a third party intercepts and records the session.

Once these steps are successfully completed, the handshake process concludes, and the encrypted connection begins. This connection uses the session key for encryption and decryption until the connection is closed. If any of the above steps fail, the TLS handshake fails, and the connection won't be established.

TLS in the OSI Model

TLS and SSL don't perfectly align with any single layer of the OSI model or the TCP/IP model. TLS "runs over some reliable transport protocol (such as TCP)," which means it sits above the transport layer. It provides encryption for higher layers, typically the presentation layer. However, applications using TLS often consider it the transport layer, even though applications using TLS must actively control the initiation of TLS handshakes and the handling of exchanged authentication certificates.

eBPF and uprobes

eBPF (Extended Berkeley Packet Filter): It's a kernel technology that allows users to run predefined programs in the kernel space without modifying kernel source code or reloading modules. It creates a bridge that enables interaction between user space and kernel space, providing unprecedented capabilities for tasks like system monitoring, performance analysis, and network traffic analysis.

uprobes are a significant feature of eBPF, allowing dynamic insertion of probe points in user space applications, particularly useful for tracking function calls in SSL/TLS libraries. Uprobe in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime。bpftime is a user mode eBPF runtime based on LLVM JIT/AOT. It can run eBPF programs in user mode and is compatible with kernel mode eBPF, avoiding context switching between kernel mode and user mode, thereby improving the execution efficiency of eBPF programs. bpftime can have a performance overhead that is one order of magnitude smaller than that of kernel mode eBPF.

User-Space Libraries

The implementation of the SSL/TLS protocol heavily relies on user-space libraries. Here are some common ones:

  • OpenSSL: An open-source, feature-rich cryptographic library widely used in many open-source and commercial projects.
  • BoringSSL: A fork of OpenSSL maintained by Google, focusing on simplification and optimization for Google's needs.
  • GnuTLS: Part of the GNU project, offering an implementation of SSL, TLS, and DTLS protocols. GnuTLS differs from OpenSSL and BoringSSL in API design, module structure, and licensing.

OpenSSL API Analysis

OpenSSL is a widely used open-source library providing a complete implementation of the SSL and TLS protocols, ensuring data transmission security in various applications. Among its functions, SSL_read() and SSL_write() are two core API functions for reading from and writing to TLS/SSL connections. In this section, we'll delve into these functions to help you understand their mechanisms.

1. SSL_read Function

When we want to read data from an established SSL connection, we can use the SSL_read or SSL_read_ex function. The function prototype is as follows:

c
int SSL_read_ex(SSL *ssl, void *buf, size_t num, size_t *readbytes);
int SSL_read(SSL *ssl, void *buf, int num);

SSL_read and SSL_read_ex attempt to read up to num bytes of data from the specified ssl into the buffer buf. Upon success, SSL_read_ex stores the actual number of read bytes in *readbytes.

2. Function SSL_write

When we want to write data into an established SSL connection, we can use the SSL_write or SSL_write_ex functions.

Function prototype:

c
int SSL_write_ex(SSL *s, const void *buf, size_t num, size_t *written);
int SSL_write(SSL *ssl, const void *buf, int num);

SSL_write and SSL_write_ex will write up to num bytes of data from the buffer buf into the specified ssl connection. Upon success, SSL_write_ex will store the actual number of written bytes in *written.

Writing eBPF Kernel Code

In our example, we use eBPF to hook the ssl_read and ssl_write functions to perform custom actions when data is read from or written to an SSL connection.

Data Structures

Firstly, we define a data structure probe_SSL_data_t to transfer data between kernel and user space:

c
#define MAX_BUF_SIZE 8192
#define TASK_COMM_LEN 16

struct probe_SSL_data_t {
    __u64 timestamp_ns;  // Timestamp (nanoseconds)
    __u64 delta_ns;      // Function execution time
    __u32 pid;           // Process ID
    __u32 tid;           // Thread ID
    __u32 uid;           // User ID
    __u32 len;           // Length of read/write data
    int buf_filled;      // Whether buffer is filled completely
    int rw;              // Read or Write (0 for read, 1 for write)
    char comm[TASK_COMM_LEN]; // Process name
    __u8 buf[MAX_BUF_SIZE];  // Data buffer
    int is_handshake;    // Whether it's handshake data
};

Hook Functions

Our goal is to hook into the SSL_read and SSL_write functions. We define a function SSL_exit to handle the return values of these two functions. This function determines whether to trace and collect data based on the current process and thread IDs.

text
/* Detailed source-code truncated for AI context efficiency. */

The rw parameter here indicates whether it's a read or write operation. 0 represents read, and 1 represents write.

Data Collection Process

  1. Obtain the ID of the current process and thread, along with the ID of the current user.
  2. Use trace_allowed to determine if tracing is allowed for this process.
  3. Get the start time to calculate the execution time of the function.
  4. Attempt to retrieve relevant data from the bufs and start_ns maps.
  5. If data retrieval is successful, create or locate a probe_SSL_data_t structure to populate the data.
  6. Copy the data from user space to the buffer, ensuring it doesn't exceed the designated size.
  7. Finally, send the data to user space.

Note: We use two user-level return probes uretprobe to respectively hook the returns of SSL_read and SSL_write:

c
SEC("uretprobe/SSL_read")
int BPF_URETPROBE(probe_SSL_read_exit) {
    return (SSL_exit(ctx, 0));  // 0 indicates read operation
}

SEC("uretprobe/SSL_write")
int BPF_URETPROBE(probe_SSL_write_exit) {
    return (SSL_exit(ctx, 1));  // 1 indicates write operation
}

Hooking into the Handshake Process

In SSL/TLS, the handshake is a special process used to establish a secure connection between a client and a server. To analyze this process, we hook into the do_handshake function to track the start and end of the handshake.

Entering the Handshake

We use a uprobe to set a probe for the do_handshake function:

c
SEC("uprobe/do_handshake")
int BPF_UPROBE(probe_SSL_do_handshake_enter, void *ssl) {
    u64 pid_tgid = bpf_get_current_pid_tgid();
    u32 pid = pid_tgid >> 32;
    u32 tid = (u32)pid_tgid;
    u64 ts = bpf_ktime_get_ns();
    u32 uid = bpf_get_current_uid_gid();

    if (!trace_allowed(uid, pid)) {
        return 0;
    }

    /* store arg info for later lookup */
    bpf_map_update_elem(&start_ns, &tid, &ts, BPF_ANY);
    return 0;
}

The main functionality of this code is as follows:

  1. Obtain the current pid, tid, ts, and uid.
  2. Use trace_allowed to verify if the process is allowed to be traced.
  3. Store the current timestamp in the start_ns map, which will be used to calculate the duration of the handshake process later.

Exiting the Handshake

Similarly, we've set a uretprobe for the return of do_handshake:

c
SEC("uretprobe/do_handshake")
int BPF_URETPROBE(handle_do_handshake_exit) {
    // Code to execute upon exiting the do_handshake function.
    return 0;
}

In this context, the uretprobe will execute the provided code when the do_handshake function exits.

c
SEC("uretprobe/do_handshake")
int BPF_URETPROBE(probe_SSL_do_handshake_exit) {
    u32 zero = 0;
    u64 pid_tgid = bpf_get_current_pid_tgid();
    u32 pid = pid_tgid >> 32;
    u32 tid = (u32)pid_tgid;
    u32 uid = bpf_get_current_uid_gid();
    u64 ts = bpf_ktime_get_ns();
    int ret = 0;

    /* use kernel terminology here for tgid/pid: */
    u32 tgid = pid_tgid >> 32;

    /* store arg info for later lookup */
    if (!trace_allowed(tgid, pid)) {
        return 0;
    }

    u64 *tsp = bpf_map_lookup_elem(&start_ns, &tid);
    if (tsp == 0)
        return 0;

    ret = PT_REGS_RC(ctx);
    if (ret <= 0)  // handshake failed
        return 0;

    struct probe_SSL_data_t *data = bpf_map_lookup_elem(&ssl_data, &zero);
    if (!data)
        return 0;

    data->timestamp_ns = ts;
    data->delta_ns = ts - *tsp;
    data->pid = pid;
    data->tid = tid;
    data->uid = uid;
    data->len = ret;
    data->buf_filled = 0;
    data->rw = 2;
    data->is_handshake = true;
    bpf_get_current_comm(&data->comm, sizeof(data->comm));
    bpf_map_delete_elem(&start_ns, &tid);

    bpf_perf_event_output(ctx, &perf_SSL_events, BPF_F_CURRENT_CPU, data,
                            EVENT_SIZE(0));
    return 0;
}

Logic of this Function:

  1. Obtain the current pid, tid, ts, and uid.
  2. Use trace_allowed to recheck if tracing is allowed.
  3. Look up the timestamp in the start_ns map for calculating handshake duration.
  4. Use PT_REGS_RC(ctx) to get the return value of do_handshake and determine if the handshake was successful.
  5. Find or initialize the probe_SSL_data_t data structure associated with the current thread.
  6. Update the data structure's fields, including timestamp, duration, process information, etc.
  7. Use bpf_perf_event_output to send the data to user space.

Our eBPF code not only tracks data transmission for ssl_read and ssl_write but also focuses on the SSL/TLS handshake process. This information is crucial for a deeper understanding and optimization of the performance of secure connections.

Through these hook functions, we can obtain data regarding the success of the handshake, the time taken for the handshake, and related process information. This provides us with insights into the behavior of the system's SSL/TLS, enabling us to perform more in-depth analysis and optimization when necessary.

User-Space Assisted Code Analysis and Interpretation

In the eBPF ecosystem, user-space and kernel-space code often work in collaboration. Kernel-space code is responsible for data collection, while user-space code manages, processes, and handles this data. In this section, we will explain how the above user-space code collaborates with eBPF to trace SSL/TLS interactions.

1. Supported Library Attachment

In the provided code snippet, based on the setting of the env environment variable, the program can choose to attach to three common encryption libraries (OpenSSL, GnuTLS, and NSS). This means that we can trace calls to multiple libraries within the same tool.

To achieve this functionality, the find_library_path function is first used to determine the library's path. Then, depending on the library type, the corresponding attach_ function is called to attach the eBPF program to the library function. If find_library_path cannot locate a library (for example, because it is not installed on the system), the tool prints a warning to stderr and skips probing that provider instead of attaching against an invalid path. Each successful attachment is tracked separately so that multiple libraries can share the same eBPF program without losing link handles. Incompatible provider symbols produce a warning, and the tool exits if no probe could be attached.

c
attach_provider(obj, env.openssl, "OpenSSL", "libssl.so", attach_openssl);
    attach_provider(obj, env.gnutls, "GnuTLS", "libgnutls.so", attach_gnutls);
    attach_provider(obj, env.nss, "NSS", "libnspr4.so", attach_nss);

This section primarily covers the attachment logic for the OpenSSL, GnuTLS, and NSS libraries. NSS is a set of security libraries designed for organizations, supporting the creation of secure client and server applications. Originally developed by Netscape, they are now maintained by Mozilla. The other two libraries have been introduced earlier and are not reiterated here.

2. Detailed Attachment Logic

The specific attach functions are as follows:

text
/* Detailed source-code truncated for AI context efficiency. */

We further examine the attach_ function and can see that they both use the ATTACH_UPROBE_CHECKED and ATTACH_URETPROBE_CHECKED macros to implement specific mounting logic. These two macros are used respectively for setting uprobe (function entry) and uretprobe (function return).

Considering that different libraries have different API function names (for example, OpenSSL uses SSL_write, while GnuTLS uses gnutls_record_send), we need to write a separate attach_ function for each library.

For instance, in the attach_openssl function, we set up probes for both SSL_write and SSL_read. If users also want to track handshake latency (env.latency) and the handshake process (env.handshake), we set up a probe for SSL_do_handshake.

In the eBPF ecosystem, perf_buffer is an efficient mechanism used to transfer data from kernel space to user space. This is particularly useful for kernel-space eBPF programs as they can't directly interact with user space. With perf_buffer, we can collect data in kernel-space eBPF programs and then asynchronously read this data in user space. We use the perf_buffer__poll function to read data reported in kernel space, as shown below:

c
while (!exiting) {
    err = perf_buffer__poll(pb, PERF_POLL_TIMEOUT_MS);
    if (err < 0 && err != -EINTR) {
        warn("error polling perf buffer: %s\n", strerror(-err));
        goto cleanup;
    }
    err = 0;
}

Finally, in the print_event function, we print the data to standard output:

c
// Function to print the event from the perf buffer
void print_event(struct probe_SSL_data_t *event, const char *evt) {
    ...
    if (buf_size != 0) {
        if (env.hexdump) {
            // 2 characters for each byte + null terminator
            char hex_data[MAX_BUF_SIZE * 2 + 1] = {0};
            buf_to_hex((uint8_t *)buf, buf_size, hex_data);

            printf("\n%s\n", s_mark);
            for (size_t i = 0; i < strlen(hex_data); i += 32) {
                printf("%.32s\n", hex_data + i);
            }
            printf("%s\n\n", e_mark);
        } else {
            printf("\n%s\n%s\n%s\n\n", s_mark, buf, e_mark);
        }
    }
}

You can find the complete source code here: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/30-sslsniff

Compilation and Execution

To start using sslsniff, you need to first compile it:

sh
make

Once done, follow these steps:

**Start sslsniff**

In a terminal, execute the following command to start sslsniff:

sh
sudo ./sslsniff

**Execute CURL command**

In another terminal, execute:

console
curl https://example.com

Under normal circumstances, you will see output similar to the following:

html
<!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        ...
    <body>
    <div>
        ...
    </div>
    </body>
    </html>

**sslsniff Output**

After executing the curl command, sslsniff will display the following content:

txt
READ/RECV    0.132786160        curl             47458   1256
    ----- DATA -----
    <!doctype html>
    ...
    <div>
        <h1>Example Domain</h1>
        ...
    </div>
    </body>
    </html>

    ----- END DATA -----

Note: The displayed HTML content may vary depending on the specific content of the example.com page.

Displaying Latency and Handshake Process

To view latency and handshake process, execute the following command:

console
$ sudo ./sslsniff -l --handshake
OpenSSL path: /lib/x86_64-linux-gnu/libssl.so.3
GnuTLS path: /lib/x86_64-linux-gnu/libgnutls.so.30
NSS path: /lib/x86_64-linux-gnu/libnspr4.so
FUNC         TIME(s)            COMM             PID     LEN     LAT(ms)
HANDSHAKE    0.000000000        curl             6460    1      1.384  WRITE/SEND   0.000115400        curl             6460    24     0.014

Hexadecimal Output

To display data in hexadecimal format, execute the following command:

console
$ sudo ./sslsniff --hexdump
WRITE/SEND   0.000000000        curl             16104   24
----- DATA -----
505249202a20485454502f322e300d0a
0d0a534d0d0a0d0a
----- END DATA -----

...

Summary

eBPF is a very powerful technology that can help us gain deeper insights into how a system works. This tutorial is a simple example demonstrating how to use eBPF to monitor SSL/TLS communication. If you're interested in eBPF technology and want to learn more and practice further, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial and tutorial website at https://eunomia.dev/zh/tutorials/.

References:

The original link of this article: https://eunomia.dev/tutorials/30-sslsniff

Overview (`src/31-goroutine/README.md`)

eBPF Practical Tutorial: Using eBPF to Trace Go Routine States

Go, the popular programming language created by Google, is known for its powerful concurrency model. One of the key features that makes Go stand out is the use of goroutinesβ€”lightweight, managed threads that make it easy to write concurrent programs. However, understanding and tracing the execution states of these goroutines in real time can be challenging, especially when debugging complex systems.

Enter eBPF (Extended Berkeley Packet Filter), a technology originally designed for network packet filtering, but which has since evolved into a powerful tool for tracing and monitoring system behavior. By leveraging eBPF, we can tap into the kernel and gather insights about the runtime behavior of Go programs, including the states of goroutines. This blog post explores how to use eBPF to trace the state transitions of goroutines in a Go program.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/31-goroutine

Background: Goroutines and eBPF

Goroutines

Goroutines are a core feature of Go, providing a simple and efficient way to handle concurrency. Unlike traditional threads, goroutines are managed by the Go runtime rather than the operating system, making them much more lightweight. Goroutines can switch states, such as:

  • RUNNABLE: The goroutine is ready to run.
  • RUNNING: The goroutine is currently executing.
  • WAITING: The goroutine is waiting for some event (e.g., I/O, timers).
  • DEAD: The goroutine has finished executing and is terminated.

Understanding these states and how goroutines transition between them is crucial for diagnosing performance issues and ensuring that your Go programs are running efficiently.

eBPF

eBPF is a powerful technology that allows developers to run custom programs inside the Linux kernel without changing the kernel source code or loading kernel modules. Initially designed for packet filtering, eBPF has grown into a versatile tool used for performance monitoring, security, and debugging.

By writing eBPF programs, developers can trace various system events, including system calls, network events, and process execution. In this blog, we'll focus on how eBPF can be used to trace the state transitions of goroutines in a Go program.

The eBPF Kernel Code

Let's dive into the eBPF kernel code that makes this tracing possible.

c
#include <vmlinux.h>
#include "goroutine.h"
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#define GOID_OFFSET 0x98

struct {
  __uint(type, BPF_MAP_TYPE_RINGBUF);
  __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("uprobe/./go-server-http/main:runtime.casgstatus")
int uprobe_runtime_casgstatus(struct pt_regs *ctx) {
  int newval = ctx->cx;
  void *gp = ctx->ax;
  struct goroutine_execute_data *data;
  u64 goid;
  if (bpf_probe_read_user(&goid, sizeof(goid), gp + GOID_OFFSET) == 0) {
    data = bpf_ringbuf_reserve(&rb, sizeof(*data), 0);
    if (data) {
      u64 pid_tgid = bpf_get_current_pid_tgid();
      data->pid = pid_tgid;
      data->tgid = pid_tgid >> 32;
      data->goid = goid;
      data->state = newval;
      bpf_ringbuf_submit(data, 0);
    }
  }
  return 0;
}

char LICENSE[] SEC("license") = "GPL";
  1. Header Files: The code begins by including necessary header files, such as vmlinux.h, which provides kernel definitions, and bpf_helpers.h, which offers helper functions for eBPF programs.
  2. GOID_OFFSET: The offset of the goid field is hardcoded to 0x98, which is specific to the Go version and the program being traced. This offset may vary between different Go versions or programs.
  3. Ring Buffer Map: A BPF ring buffer map is defined to store the goroutine execution data. This buffer allows the kernel to pass information to user space efficiently.
  4. Uprobe: The core of this eBPF program is an uprobes (user-level probe) attached to the runtime.casgstatus function in the Go program. This function is responsible for changing the state of a goroutine, making it an ideal place to intercept and trace state transitions.
  5. Reading Goroutine ID: The bpf_probe_read_user function reads the goroutine ID (goid) from the user space memory, using the predefined offset.
  6. Submitting Data: If the goroutine ID is successfully read, the data is stored in the ring buffer along with the process ID, thread group ID, and the new state of the goroutine. This data is then submitted to the user space for analysis.

Running the Program

Run the following commands from src/31-goroutine. The eBPF program attaches to
./go-server-http/main, so the target binary is generated locally rather than
stored in Git. Use Go 1.17.13 to match the checked-in GOID_OFFSET value; verify
the offset before tracing a binary built with another Go version.

  1. Build and start the target program: Keep the HTTP server running, then use a second terminal for the remaining commands.

    bash
    make -C go-server-http
sudo ./go-server-http/main

  2. Compile the eBPF Code: Compile the eBPF program using a compiler like ecc (eBPF Compiler Collection) and generate a package that can be loaded by an eBPF loader.

    bash
    ecc goroutine.bpf.c goroutine.h

  3. Run the eBPF Program: Use an eBPF loader to run the compiled eBPF program.

    bash
    ecli-rs run package.json

  4. Output: The program will output the state transitions of goroutines along with their goid, pid, and tgid. Here’s an example of the output:

    console
    TIME     STATE       GOID   PID    TGID   
21:00:47 DEAD(6)     0      2542844 2542844
21:00:47 RUNNABLE(1) 0      2542844 2542844
21:00:47 RUNNING(2)  1      2542844 2542844
21:00:47 WAITING(4)  2      2542847 2542844

You can find the complete code in https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/31-goroutine

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ to get more examples and complete tutorials.

Uprobe in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime. bpftime is a user mode eBPF runtime based on LLVM JIT/AOT. It can run eBPF programs in user mode, compatible with kernel mode eBPF and can be faster for uprobe.

Conclusion

Tracing goroutine states using eBPF provides deep insights into the execution of Go programs, especially in production environments where traditional debugging tools may fall short. By leveraging eBPF, developers can monitor and diagnose performance issues, ensuring their Go applications run efficiently.

Keep in mind that the offsets used in this eBPF program are specific to the Go version and the program being traced. As Go evolves, these offsets may change, requiring updates to the eBPF code.

In future explorations, we can extend this approach to trace other aspects of Go programs or even other languages, demonstrating the versatility and power of eBPF in modern software development.

Overview (`src/32-wallclock-profiler/README.md`)

eBPF Tutorial by Example 32: Wall Clock Profiling with Combined On-CPU and Off-CPU Analysis

Performance bottlenecks can hide in two very different places. Your code might be burning CPU cycles in hot loops, or it might be sitting idle waiting for I/O, network responses, or lock contention. Traditional profilers often focus on just one side of this story. But what if you could see both at once?

This tutorial introduces a complete wall clock profiling solution that combines on-CPU and off-CPU analysis using eBPF. We'll show you how to capture the full picture of where your application spends its time, using two complementary eBPF programs that work together to account for every microsecond of execution. Whether your performance problems come from computation or waiting, you'll be able to spot them in a unified flame graph view.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/32-wallclock-profiler

Understanding Wall Clock Profiling

Wall clock time is the actual elapsed time from start to finish, like checking a stopwatch. For any running process, this time gets divided into two categories. On-CPU time is when your code actively executes on a processor, doing real work. Off-CPU time is when your process exists but isn't running, waiting for something like disk I/O, network packets, or acquiring a lock.

Traditional CPU profilers only show you the on-CPU story. They sample the stack at regular intervals when your code runs, building a picture of which functions consume CPU cycles. But these profilers are blind to off-CPU time. When your thread blocks on a system call or waits for a mutex, the profiler stops seeing it. This creates a massive blind spot for applications that spend significant time waiting.

Off-CPU profilers flip the problem around. They track when threads go to sleep and wake up, measuring blocked time and capturing stack traces at blocking points. This reveals I/O bottlenecks and lock contention. But they miss pure computation problems.

The tools in this tutorial solve both problems by running two eBPF programs simultaneously. The oncputime tool samples on-CPU execution using perf events. The offcputime tool hooks into the kernel scheduler to catch blocking operations. A Python script combines the results, normalizing the time scales so you can see CPU-intensive code paths (marked red) and blocking operations (marked blue) in the same flame graph. This complete view shows where every microsecond goes.

Here's an example flame graph showing combined on-CPU and off-CPU profiling results:

In this visualization, you can clearly see the distinction between CPU-intensive work (shown in red/warm colors marked with _[c]) and blocking operations (shown in blue/cool colors marked with _[o]). The relative widths immediately reveal where your application spends its wall clock time.

The Tools: oncputime and offcputime

This tutorial provides two complementary profiling tools. The oncputime tool samples your process at regular intervals using perf events, capturing stack traces when code actively runs on the CPU. At a default rate of 49 Hz, it wakes up roughly every 20 milliseconds to record where your program is executing. Higher sample counts in the output indicate more CPU time spent in those code paths.

The offcputime tool takes a different approach. It hooks into the kernel scheduler's context switch mechanism, specifically the sched_switch tracepoint. When your thread goes off-CPU, the tool records a timestamp and captures the stack trace showing why it blocked. When the thread returns to running, it calculates how long the thread was sleeping. This directly measures I/O waits, lock contention, and other blocking operations in microseconds.

Both tools use BPF stack maps to efficiently capture kernel and user space call chains with minimal overhead. They aggregate results by unique stack traces, so repeated execution of the same code path gets summed together. The tools can filter by process ID, thread ID, and various other criteria to focus analysis on specific parts of your application.

Implementation: Kernel-Space eBPF Programs

Let's examine how these tools work at the eBPF level. We'll start with the on-CPU profiler, then look at the off-CPU profiler, and see how they complement each other.

On-CPU Profiling with oncputime

The on-CPU profiler uses perf events to sample execution at regular time intervals. Here's the complete eBPF program:

text
/* Detailed source-code truncated for AI context efficiency. */

The program starts by defining several BPF maps. The stackmap is a special map type for storing stack traces. When you call bpf_get_stackid(), the kernel walks the stack and stores the instruction pointers in this map, returning an ID you can use to look it up later. The counts map aggregates samples by a composite key that includes both the process ID and the stack IDs. The pids and tids maps act as filters, letting you restrict profiling to specific processes or threads.

The main logic lives in the do_perf_event() function, which runs every time a perf event fires. The user space program sets up these perf events at a specific frequency (default 49 Hz), one per CPU core. When a CPU triggers its timer, this function executes on whatever process happens to be running at that moment. It first extracts the process and thread IDs from the current task, then applies any configured filters. If the current thread should be sampled, it builds a key structure that includes the process name and stack traces.

The two calls to bpf_get_stackid() capture different pieces of the execution context. The first call without flags gets the kernel stack, showing what kernel functions were active. The second call with BPF_F_USER_STACK gets the user space stack, showing your application's function calls. These stack IDs go into the key, and the program increments a counter for that unique combination. Over time, hot code paths get sampled more frequently, building up higher counts.

Off-CPU Profiling with offcputime

The off-CPU profiler hooks into the scheduler to measure blocking time. Here's the complete eBPF program:

text
/* Detailed source-code truncated for AI context efficiency. */

The off-CPU profiler is more complex because it needs to track timing across multiple events. The start map stores timestamps and stack information for threads that go off-CPU. When a thread blocks, we record when it happened and why (the stack trace). When that same thread returns to running, we calculate how long it was blocked.

The scheduler switch happens many times per second on a busy system, so performance matters. The allow_record() function quickly filters out threads we don't care about before doing expensive operations. If a thread passes the filter, the program captures the current timestamp using bpf_ktime_get_ns() and records the stack traces showing where the thread blocked.

The key insight is in the two-stage approach. The prev task (the thread going off-CPU) gets its blocking point recorded with a timestamp. When the scheduler later switches to the next task (a thread waking up), we look up whether we previously recorded this thread going to sleep. If we find a record, we calculate the delta between now and when it went to sleep. This delta is the off-CPU time in nanoseconds, which we convert to microseconds and add to the accumulated total for that stack trace.

User-Space Programs: Loading and Processing

Both tools follow a similar pattern in user space. They use libbpf to load the compiled eBPF object file and attach it to the appropriate event. For oncputime, this means setting up perf events at the desired sampling frequency. For offcputime, it means attaching to the scheduler tracepoint. The user space programs then periodically read the BPF maps, resolve the stack IDs to actual function names using symbol tables, and format the output.

The symbol resolution is handled by the blazesym library, which parses DWARF debug information from binaries. When you see a stack trace with function names and line numbers, that's blazesym converting raw instruction pointer addresses into human-readable form. The user space programs output in "folded" format, where each line contains a semicolon-separated stack trace followed by a count or time value. This format feeds directly into flame graph generation tools.

Combining On-CPU and Off-CPU Profiles

The real power comes from running both tools together and merging their results. The wallclock_profiler.py script orchestrates this process. It launches both profilers simultaneously on the target process, waits for them to complete, and then combines their outputs.

The challenge is that the two tools measure different things in different units. The on-CPU profiler counts samples (49 per second by default), while the off-CPU profiler measures microseconds. To create a unified view, the script normalizes the off-CPU time to equivalent sample counts. If sampling at 49 Hz, each sample represents about 20,408 microseconds of potential execution time. The script divides off-CPU microseconds by this value to get equivalent samples.

After normalization, the script adds annotations to distinguish the two types of time. On-CPU stack traces get a _[c] suffix (for compute), while off-CPU stacks get _[o] (for off-CPU or blocking). A custom color palette in the flame graph tool renders these different colors, red for CPU time and blue for blocking time. The result is a single flame graph where you can see both types of activity and their relative magnitudes.

The script also handles multi-threaded applications by profiling each thread separately. It detects threads at startup, launches parallel profiling sessions for each one, and generates individual flame graphs showing per-thread behavior. This helps identify which threads are busy versus idle, and whether your parallelism is effective.

Compilation and Execution

Building the tools requires a standard eBPF development environment. The tutorial repository includes all dependencies in the src/third_party/ directory. To build:

bash
cd src/32-wallclock-profiler
make

The Makefile compiles the eBPF C code with clang, generates skeletons with bpftool, builds the blazesym symbol resolver, and links everything with libbpf to create the final executables.

To use the individual tools:

bash
# Profile on-CPU execution for 30 seconds
sudo ./oncputime -p <PID> -F 99 30

# Profile off-CPU blocking for 30 seconds
sudo ./offcputime -p <PID> -m 1000 30

# Use the combined profiler (recommended)
sudo python3 wallclock_profiler.py <PID> -d 30 -f 99

Let's try profiling a test program that does both CPU work and blocking I/O:

bash
# Build and run the test program
cd tests
make
./test_combined &
TEST_PID=$!

# Profile it with the combined profiler
cd ..
sudo python3 wallclock_profiler.py $TEST_PID -d 30

# This generates:
# - combined_profile_pid<PID>_<timestamp>.folded (raw data)
# - combined_profile_pid<PID>_<timestamp>.svg (flame graph)
# - combined_profile_pid<PID>_<timestamp>_single_thread_analysis.txt (time breakdown)

The output flame graph will show red frames for the cpu_work() function consuming CPU time, and blue frames for the blocking_work() function spending time in sleep. The relative widths show how much wall clock time each consumes.

For multi-threaded applications, the profiler creates a directory with per-thread results:

bash
# Profile a multi-threaded application
sudo python3 wallclock_profiler.py <PID> -d 30

# Output in multithread_combined_profile_pid<PID>_<timestamp>/
# - thread_<TID>_main.svg (main thread flame graph)
# - thread_<TID>_<role>.svg (worker thread flame graphs)
# - *_thread_analysis.txt (time analysis for all threads)

The analysis files show time accounting, letting you verify that on-CPU plus off-CPU time adds up correctly to the wall clock profiling duration. Coverage percentages help identify if threads are mostly idle or if you're missing data.

Interpreting the Results

When you open the flame graph SVG in a browser, each horizontal box represents a function in a stack trace. The width shows how much time was spent there. Boxes stacked vertically show the call chain, with lower boxes calling higher ones. Red boxes indicate on-CPU time, blue boxes show off-CPU time.

Look for wide red sections to find CPU bottlenecks. These are functions burning through cycles in tight loops or expensive algorithms. Wide blue sections indicate blocking operations. Common patterns include file I/O (read/write system calls), network operations (recv/send), and lock contention (futex calls).

The flame graph is interactive. Click any box to zoom in and see details about that subtree. The search function lets you highlight all frames matching a pattern, useful for finding specific functions or libraries. Hovering shows the full function name and exact sample count or time value.

Pay attention to the relative proportions. An application that's 90% blue is I/O bound and probably won't benefit much from CPU optimization. One that's mostly red is CPU bound. Applications split evenly between red and blue might benefit from overlapping computation and I/O, such as using asynchronous I/O or threading.

For multi-threaded profiles, compare the per-thread flame graphs. Ideally, worker threads should show similar patterns if the workload is balanced. If one thread is mostly red while others are mostly blue, you might have load imbalance. If all threads show lots of blue time in futex waits with similar stacks, that's lock contention.

Wall-clock profiling builds on decades of research in performance analysis that distinguishes on-CPU computation from off-CPU waiting. Curtsinger and Berger's Coz (ASPLOS'15) introduced causal profiling, which experimentally determines which code regions, if optimized, actually reduce end-to-end latencyβ€”addressing the fundamental question of where optimization effort pays off. Zhou et al.'s wPerf (OSDI'18) presented a generic off-CPU analysis framework that identifies critical waiting events (locks, I/O) bounding throughput with low overhead, while the more recent work by Ahn et al. (OSDI'24) unified on- and off-CPU analysis through blocked-sample profiling that captures both running and blocked thread states. The visualization techniques we employ draw from Gregg's flame graph methodology (CACM'16, USENIX ATC'17), which transforms stack-trace aggregations into intuitive hierarchical diagrams; his off-CPU flame graphs specifically highlight blocking patterns by rendering sleep stacks in contrasting colors. Timing accuracy itself poses challenges, as Najafi et al. (HotOS'21) argue that modern systems research increasingly depends on precise wall-clock measurements, and earlier work on time-sensitive Linux (Goel et al., OSDI'02) explored kernel techniques for low-latency timing under load. Practical eBPF-based profiling has been demonstrated in production contexts, including Java profiling with off-CPU "offwaketime" analysis (ICPE'19) and comprehensive workflows outlined in recent eBPF performance tutorials (Gregg, SIGCOMM'24). Together, these techniques and tools provide the foundation for understanding where applications spend time and how to optimize holistically across both compute and blocking dimensions.

Summary

Wall clock profiling with eBPF gives you complete visibility into application performance by combining on-CPU and off-CPU analysis. The on-CPU profiler samples execution to find hot code paths that consume CPU cycles. The off-CPU profiler hooks into the scheduler to measure blocking time and identify I/O bottlenecks or lock contention. Together, they account for every microsecond of wall clock time, showing where your application actually spends its life.

The tools use eBPF's low-overhead instrumentation to collect this data with minimal impact on the target application. Stack trace capture and aggregation happen in the kernel, avoiding expensive context switches. The user space programs only need to periodically read accumulated results and resolve symbols, making the overhead negligible even for production use.

By visualizing both types of time in a single flame graph with color coding, you can quickly identify whether problems are computational or blocking in nature. This guides optimization efforts more effectively than traditional profiling approaches that only show one side of the picture. Multi-threaded profiling support reveals parallelism issues and thread-level bottlenecks.

If you'd like to dive deeper into eBPF, check out our tutorial repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or visit our website at https://eunomia.dev/tutorials/.

Reference

The original link of this article: https://eunomia.dev/tutorials/32-wallclock-profiler

Overview (`src/33-funclatency/README.md`)

Measuring Function Latency with eBPF

In modern software systems, understanding the performance characteristics of functionsβ€”especially those critical to the operation of your applicationβ€”is paramount. One key metric in performance analysis is function latency, which is the time taken by a function to execute from start to finish. By analyzing function latency, developers can identify bottlenecks, optimize performance, and ensure that their systems operate efficiently under various conditions.

This blog post will dive into how to measure function latency using eBPF, an incredibly powerful tool for tracing and monitoring both kernel and user-space programs.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/33-funclatency

What is eBPF?

eBPF (Extended Berkeley Packet Filter) is a revolutionary technology that allows developers to write small programs that run in the Linux kernel. Originally designed for packet filtering, eBPF has evolved into a versatile tool for tracing, monitoring, and profiling system behavior. With eBPF, you can instrument almost any part of the Linux kernel or user-space programs to collect performance data, enforce security policies, or even debug systems in real timeβ€”all without the need to modify the kernel source code or restart the system.

eBPF programs are executed in a sandboxed environment within the kernel, ensuring safety and stability. These programs can attach to various hooks within the kernel, such as system calls, network events, and tracepoints, or even user-space functions using uprobes (user-level probes). The data collected by eBPF programs can then be exported to user space for analysis, making it an invaluable tool for system observability. Uprobe in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime.

Why is Function Latency Important?

Function latency is a critical metric in performance analysis for both kernel and user-space applications. It provides insights into how long a particular function takes to execute, which is crucial for:

  • Identifying Performance Bottlenecks: High function latency may indicate inefficiencies or issues within the code that need optimization.
  • Ensuring System Responsiveness: In real-time systems or latency-sensitive applications, understanding and minimizing function latency is essential to maintain responsiveness.
  • Profiling and Benchmarking: By measuring the latency of various functions, developers can benchmark their systems and compare the performance of different implementations or configurations.
  • Debugging and Diagnostics: When a system exhibits unexpected behavior or performance degradation, measuring function latency can help pinpoint the source of the problem.

Both kernel-space (e.g., system calls, file operations) and user-space (e.g., library functions) functions can be profiled for latency, providing a comprehensive view of system performance.

eBPF Kernel Code for Function Latency

Below is an eBPF program designed to measure the latency of a function by hooking into its entry and exit points. The program uses kprobes and kretprobes (for kernel functions) or uprobes and uretprobes (for user-space functions) to capture the start and end times of the function execution.

text
/* Detailed source-code truncated for AI context efficiency. */

Explanation of the Code

  1. Header Files: The code begins by including the necessary headers like vmlinux.h (which provides kernel definitions) and bpf_helpers.h (which offers helper functions for eBPF programs).

  2. Global Variables: targ_tgid is a target process ID (or thread group ID), and units determines the time unit for latency measurement (e.g., microseconds or milliseconds).

  3. BPF Maps: A hash map (starts) is defined to store the start time of function executions for each process ID. Another array (hist) is used to store the latency distribution.

  4. Entry Function: The entry() function captures the current timestamp when the function is entered and stores it in the starts map keyed by the process ID.

  5. Exit Function: The exit() function calculates the latency by subtracting the stored start time from the current time. The result is then categorized into a histogram slot, which is incremented to record the occurrence of that latency range.

  6. Probes: The kprobe and kretprobe are used to attach to the entry and exit points of the function, respectively. These probes trigger the entry() and exit() functions to measure the latency.

  7. License: The program is licensed under GPL to ensure compliance with kernel licensing requirements.

Running the Function Latency Tool

User-Space Function Latency

To trace the latency of a user-space function, such as the read function in the libc library, you can run the following command:

console
# ./funclatency /usr/lib/x86_64-linux-gnu/libc.so.6:read    
tracing /usr/lib/x86_64-linux-gnu/libc.so.6:read...
tracing func read in /usr/lib/x86_64-linux-gnu/libc.so.6...
Tracing /usr/lib/x86_64-linux-gnu/libc.so.6:read.  Hit Ctrl-C to exit
^C
     nsec                : count    distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
       128 -> 255        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      65536 -> 131071     : 651      |****************************************+|
    131072 -> 262143     : 107      |******                                  |
    262144 -> 524287     : 36       |**                                      |
    524288 -> 1048575    : 8        |                                        |
   8388608 -> 16777215   : 2        |                                        |
Exiting trace of /usr/lib/x86_64-linux-gnu/libc.so.6:read

Kernel-Space Function Latency

To trace the latency of a kernel-space function, such as vfs_read, run the following command:

console
# sudo ./funclatency -u vfs_read
Tracing vfs_read.  Hit Ctrl-C to exit
^C
     usec                : count    distribution
         0 -> 1          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 3397     |****************************************|
        32 -> 63         : 2175     |*************************               |
        64 -> 127        : 184      |**                                      |
       1024 -> 2047       : 0        |                                        |
       4096 -> 8191       : 5        |                                        |
   2097152 -> 4194303    : 2        |                                        |
Exiting trace of vfs_read

These commands trace the execution of the specified function, either in user-space or kernel-space, and print a histogram of the observed latencies, showing the distribution of function execution times.

You can find the source code in https://github.com/eunomia-bpf/bpf-developer-tutorial/blob/main/src/33-funclatency

Conclusion

Measuring function latency with eBPF offers deep insights into the performance of both user-space and kernel-space code. By understanding function latency, developers can identify performance bottlenecks, improve system responsiveness, and ensure the smooth operation of their applications.

This

blog post covered the basics of using eBPF to trace function latency, including an overview of the eBPF kernel code used to perform the tracing. The examples provided demonstrated how to run the tool to trace both user-space and kernel-space functions.

For those interested in learning more about eBPF, including more advanced examples and tutorials, please visit our https://github.com/eunomia-bpf/bpf-developer-tutorial or our website https://eunomia.dev/tutorials/.

If you are looking for a production-ready tool for function latency measurement, you might want to check out the full implementation available in the BCC repository.

Overview (`src/34-syscall/README.md`)

eBPF Development Practice: Modifying System Call Arguments with eBPF

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows user-defined code to be run, loaded, and updated without the need to modify kernel source code or reboot the kernel. This functionality has enabled a wide range of applications for eBPF, such as network and system performance analysis, packet filtering, and security policies.

In this tutorial, we will explore how to use eBPF to modify the arguments of a running system call. This technique can be used for security auditing, system monitoring, or even malicious behavior. However, it is important to note that modifying system call arguments can have negative implications for system stability and security, so caution must be exercised. To implement this functionality, we will use the bpf_probe_write_user feature of eBPF, which allows us to modify memory in the user space and therefore modify system call arguments before the kernel reads them from user space.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/34-syscall

Modifying the File Name of the `open` System Call

This functionality is used to modify the arguments of the openat system call to open a different file. This technique can be useful for:

  1. File Access Auditing: In environments with strict legal and data security requirements, auditors may need to record access to sensitive files. By modifying the openat system call arguments, all attempts to access a specific sensitive file can be redirected to a backup file or a log file.
  2. Secure Sandbox: In the early stages of development, it may be desirable to monitor the files accessed by an application. By changing the openat calls, the application can be run in a secure sandbox environment where all file operations are redirected to an isolated file system path.
  3. Sensitive Data Protection: For files containing sensitive information, such as a configuration file that contains database passwords, a eBPF-based system can redirect those calls to an encrypted or temporary location to enhance data security.

If leveraged by malicious software, this technique can be used to redirect file operations resulting in data leaks or compromise data integrity. For example, when a program is writing to a log file, an attacker could redirect the data to a controlled file, disrupting the audit trail.

Kernel code (partial code, see complete code on Github bpf-developer-tutorial):

c
SEC("tracepoint/syscalls/sys_enter_openat")
int tracepoint__syscalls__sys_enter_openat(struct trace_event_raw_sys_enter *ctx)
{
    u64 pid = bpf_get_current_pid_tgid() >> 32;
    /* use kernel terminology here for tgid/pid: */
    if (target_pid && pid != target_pid) {
        return 0;
    }
    /* store arg info for later lookup */
    // since we can manually specify the attach process in userspace,
    // we don't need to check the process allowed here

    struct args_t args = {};
    args.fname = (const char *)ctx->args[1];
    args.flags = (int)ctx->args[2];
    if (rewrite) {
        bpf_probe_write_user((char*)ctx->args[1], "hijacked", 9);
    }
    bpf_map_update_elem(&start, &pid, &args, 0);
    return 0;
}

Analysis of the kernel code:

  • bpf_get_current_pid_tgid() retrieves the current process ID.
  • If target_pid is specified and does not match the current process ID, the function returns 0 and does not execute further.
  • We create an args_t structure to store the file name and flags.
  • We use bpf_probe_write_user to modify the file name in the user space memory to "hijacked".

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf.

Compile the code:

bash
./ecc open_modify.bpf.c open_modify.h

Build a simple victim program using make for testing:

c
int main()
{
    char filename[100] = "my_test.txt";
    // print pid
    int pid = getpid();
    std::cout << "current pid: " << pid << std::endl;
    system("echo \"hello\" > my_test.txt");
    system("echo \"world\" >> hijacked");
    while (true) {
        std::cout << "Opening my_test.txt" << std::endl;

        int fd = open(filename, O_RDONLY);
        assert(fd != -1);

        std::cout << "test.txt opened, fd=" << fd << std::endl;
        usleep(1000 * 300);
        // print the file content
        char buf[100] = {0};
        int ret = read(fd, buf, 5);
        std::cout << "read " << ret << " bytes: " << buf << std::endl;
        std::cout << "Closing test.txt..." << std::endl;
        close(fd);
        std::cout << "test.txt closed" << std::endl;
    }
    return 0;
}

Compile and run the test code:

sh
$ ./victim
test.txt opened, fd=3
read 5 bytes: hello
Closing test.txt...
test.txt closed

Use the following command to specify the target process ID to modify the openat system call arguments:

bash
sudo ./ecli run package.json --rewrite --target_pid=$(pidof victim)

You will see that the output changes to "world". Instead of opening the "my_test.txt" file, it opens the "hijacked" file:

console
test.txt opened, fd=3
read 5 bytes: hello
Closing test.txt...
test.txt closed
Opening my_test.txt
test.txt opened, fd=3
read 5 bytes: world
Closing test.txt...
test.txt closed
Opening my_test.txt
test.txt opened, fd=3
read 5 bytes: world

The complete code with test cases can be found in the https://github.com/eunomia-bpf/bpf-developer-tutorial repository.

Modifying the Process Name of bash `execve`

This functionality is used to modify the program name when the execve system call is made. In certain auditing or monitoring scenarios, this may be used to track the behavior of specific processes or modify their behavior. However, such modifications can lead to confusion and make it difficult for users or administrators to determine the actual program being executed by the system. The most serious risk is that if malicious users are able to control the eBPF program, they could redirect legitimate system commands to malicious software, resulting in a significant security threat.

text
/* Detailed source-code truncated for AI context efficiency. */

Analysis of the kernel code:

  • Execute bpf_get_current_pid_tgid to get the current process ID and thread group ID.
  • If target_ppid is set, the code checks if the current process's parent process ID matches.
  • Read the program name from the first argument of execve.
  • Use bpf_probe_write_user to overwrite the argument with a hijacked binary path.

This approach poses a risk as it can be leveraged to hijack the behavior of software, resulting in the execution of malicious code on the system. Using ecc and ecli to compile and run:

bash
./ecc exechijack.bpf.c exechijack.h
sudo ./ecli run package.json

Conclusion

eBPF provides powerful capabilities for real-time monitoring and intervention in running systems. When used in conjunction with appropriate governance and security policies, this can bring many benefits such as enhanced security, performance optimization, and operational convenience. However, this technology must be used with great care as incorrect operations or misuse can result in system disruption or serious security incidents. In practice, it should be ensured that only authorized users and programs can deploy and manage eBPF programs, and their behavior should be validated in isolated test environments before they are applied in production.

Overview (`src/35-user-ringbuf/README.md`)

eBPF Development Practices: Asynchronously Send to Kernel with User Ring Buffer

eBPF, or Extended Berkeley Packet Filter, is a revolutionary technology in the Linux kernel that allows developers to run custom "micro programs" in kernel mode, thereby changing system behavior or collecting fine-grained performance data without modifying kernel code.

One unique aspect of eBPF is that it not only allows programs to run in kernel mode to access low-level system states and resources, but it can also communicate with user mode programs through special data structures. One important concept in this regard is the ring buffer between kernel mode and user mode. In many real-time or high-performance applications, the ring buffer is a commonly used data structure. Due to its FIFO (first in, first out) characteristics, data can flow continuously and linearly between the producer and the consumer, avoiding frequent IO operations and unnecessary memory reallocation overhead.

In eBPF, two types of ring buffers are provided: user ring buffer and kernel ring buffer, to achieve efficient data communication between user mode and kernel mode. This article is part of the eBPF developer tutorial. More detailed content can be found here: https://eunomia.dev/tutorials/

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/35-user-ringbuf

User mode and kernel mode ring buffersβ€”user ring buffer and kernel ring buffer

Around the two main run levels of kernel mode and user mode, eBPF provides two corresponding ring buffer data structures: User ring buffer and Kernel ring buffer.

Kernel ring buffer is implemented by eBPF and is specially designed for the Linux kernel to track and record kernel logs, performance statistics, etc. It is the core of data transfer from kernel mode to user mode and can send data from kernel mode to user mode. Kernel ring buffer was introduced in the 5.7 version of the kernel and is now widely used in the kernel logging system, performance analysis tools, etc.

For scenarios where the kernel sends to user mode, such as sending kernel monitoring events, asynchronous notifications, status update notifications, etc., the ring buffer data structure can handle them. For example, when we need to monitor the status of a large number of ports of network service programs, the opening, closing, errors, and other status updates of these ports need to be real-time transferred to the user space for processing. Linux kernel's logging system, performance analysis tools, etc., also need to frequently send large amounts of data to user space to support user-friendly display and analysis of these data. In these scenarios, the ring buffer shows extremely high efficiency in sending data from the kernel to the user.

User ring buffer is a new type of Map type based on the ring buffer, it provides the semantics of a single user space producer/single kernel consumer. The advantage of this ring buffer is that it provides excellent support for asynchronous message passing, avoiding unnecessary synchronization operations, optimizing data transfer from the kernel to user space, and reducing the system overhead of system calls. User ring buffer was introduced in the 6.1 version of the kernel and its current use cases are relatively limited.

bpftime is a user space eBPF runtime that allows existing eBPF applications to run in unprivileged user space using the same libraries and toolchain. It provides Uprobe and Syscall tracing points for eBPF, which significantly improves performance compared to kernel Uprobe and does not require manual code detection or process restart. The runtime supports process eBPF mapping in user space shared memory, and is also compatible with kernel eBPF mapping, allowing seamless operation with the kernel eBPF infrastructure. It includes a high-performance LLVM JIT for various architectures, a lightweight JIT for x86, and an interpreter. GitHub address: https://github.com/eunomia-bpf/bpftime

In bpftime, we use the user ring buffer to implement data transmission from user mode eBPF to kernel mode eBPF, and update the maps corresponding to kernel mode eBPF, so that kernel mode and user mode eBPF can work together. The asynchronous characteristics of user ring buffer can avoid unnecessary synchronization operations of system calls, thereby improving the efficiency of data transmission between kernel mode and user mode.

The bi-directional ring buffer of eBPF also has similarities to io_uring in some respects, but their design intentions and use cases are different:

  • Design focus: io_uring primarily focuses on improving the performance and efficiency of asynchronous I/O operations, while eBPF's ring buffer focuses more on data communication and event transmission between the kernel and user space.
  • Application range: io_uring is mainly used in file I/O and network I/O scenarios, while eBPF's ring buffer is more widespread, not limited to I/O operations, but also including system call tracing, network packet processing, etc.
  • Flexibility and extensibility: eBPF provides higher flexibility and extensibility, allowing users to define complex data processing logic and execute it in kernel mode.

Following is a code example where we will show in detail how to use user ring buffer to transmit data from user mode to the kernel, and how to respond accordingly with kernel ring buffer to transmit data from kernel mode to user mode.

I. Implementation: Using Ring Buffer to Transfer Data Between User Mode and Kernel Mode

With the help of the new BPF MAP, we can implement the transfer of data between user mode and kernel mode through the ring buffer. In this example, we will detail how to create a "user ring buffer" in user space and write data to it and then consume this data in kernel space with the bpf_user_ringbuf_drain function. At the same time, we will use the "kernel ring buffer" to feed back data from kernel space to user space. To do this, we need to create and operate these two ring buffers separately in user space and kernel space.

The complete code can be found at https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/35-user-ringbuf.

Create Ring Buffer

In kernel mode, we created a user_ringbuf of type BPF_MAP_TYPE_USER_RINGBUF and a kernel_ringbuf of type BPF_MAP_TYPE_RINGBUF. In user mode, we created an instance of the struct ring_buffer_user structure and managed this user ring buffer through the ring_buffer_user__new function and corresponding operations.

c
/* Set up ring buffer polling */
    rb = ring_buffer__new(bpf_map__fd(skel->maps.kernel_ringbuf), handle_event, NULL, NULL);
    if (!rb)
    {
        err = -1;
        fprintf(stderr, "Failed to create ring buffer\n");
        goto cleanup;
    }
    user_ringbuf = user_ring_buffer__new(bpf_map__fd(skel->maps.user_ringbuf), NULL);

Writing Kernel Mode Programs

We define a kill_exit tracepoint program that will read user data from user_ringbuf with the bpf_user_ringbuf_drain function whenever a process exits. Then, it creates a new record in kernel_ringbuf with the bpf_ringbuf_reserve function and writes relevant information. Finally, the record is submitted with the bpf_ringbuf_submit function so that it can be read by user mode.

c
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include "user_ringbuf.h"

char _license[] SEC("license") = "GPL";

struct
{
    __uint(type, BPF_MAP_TYPE_USER_RINGBUF);
    __uint(max_entries, 256 * 1024);
} user_ringbuf SEC(".maps");

struct
{
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} kernel_ringbuf SEC(".maps");

int read = 0;

static long
do_nothing_cb(struct bpf_dynptr *dynptr, void *context)
{
    struct event *e;
    pid_t pid;
    /* get PID and TID of exiting thread/process */
    pid = bpf_get_current_pid_tgid() >> 32;

    /* reserve sample from BPF ringbuf */
    e = bpf_ringbuf_reserve(&kernel_ringbuf, sizeof(*e), 0);
    if (!e)
        return 0;

    e->pid = pid;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

    /* send data to user-space for post-processing */
    bpf_ringbuf_submit(e, 0);
    __sync_fetch_and_add(&read, 1);
    return 0;
}

SEC("tracepoint/syscalls/sys_exit_kill")
int kill_exit(struct trace_event_raw_sys_exit *ctx)
{
    long num_samples;
    int err = 0;
    
    // receive data from userspace
    num_samples = bpf_user_ringbuf_drain(&user_ringbuf, do_nothing_cb, NULL, 0);

    return 0;
}

Writing User Mode Programs

In user mode, we reserved a section of space in the ring buffer with the ring_buffer_user__reserve function. This space is used to write the information we want to pass to the kernel. Then, the data is submitted using the ring_buffer_user__submit function, after which this data can be read and processed in kernel mode.

c
static int write_samples(struct user_ring_buffer *ringbuf)
{
    int i, err = 0;
    struct user_sample *entry;

    entry = user_ring_buffer__reserve(ringbuf, sizeof(*entry));
    if (!entry)
    {
        err = -errno;
        goto done;
    }

    entry->i = getpid();
    strcpy(entry->comm, "hello");

    int read = snprintf(entry->comm, sizeof(entry->comm), "%u", i);
    if (read <= 0)
    {
        /* Assert on the error path to avoid spamming logs with
         * mostly success messages.
         */
        err = read;
        user_ring_buffer__discard(ringbuf, entry);
        goto done;
    }

    user_ring_buffer__submit(ringbuf, entry);

done:
    drain_current_samples();

    return err;
}

Initialization of the Ring Buffer and Poll

Finally, initialize the ring buffer and periodically poll, so we can know in real-time the consumption of data in kernel mode. We can also write to the user_ringbuf in user mode, then read and process it in kernel mode.

c
write_samples(user_ringbuf);

    /* Process events */
    printf("%-8s %-5s %-16s %-7s %-7s %s\n",
           "TIME", "EVENT", "COMM", "PID", "PPID", "FILENAME/EXIT CODE");
    while (!exiting)
    {
        err = ring_buffer__poll(rb, 100 /* timeout, ms */);
        /* Ctrl-C will cause -EINTR */
        if (err == -EINTR)
        {
            err = 0;
            break;
        }
        if (err < 0)
        {
            printf("Error polling perf buffer: %d\n", err);
            break;
        }
    }

Through the above steps, we have implemented two-way data transmission between user mode and kernel mode.

II. Compile and Run the Code

To compile and run the above code, we can run the following command:

sh
make

For information on how to install dependencies, refer to: https://eunomia.dev/tutorials/11-bootstrap/

The execution result displays how to use the user ring buffer and kernel ringbuffer for efficient data transmission between user mode and kernel mode:

console
$ sudo ./user_ringbuf
Draining current samples...
TIME     EVENT COMM             PID   
16:31:37 SIGN  node             1707   
Draining current samples...
16:31:38 SIGN  node             1981   
Draining current samples...
16:31:38 SIGN  node             1707   
Draining current samples...
16:31:38 SIGN  node             1707   
Draining current samples...

Conclusion

In this article, we discussed how to use eBPF's user ring buffer and kernel ring buffer for data transmission between user mode and kernel mode. Through this method, we can effectively deliver user data to the kernel or feed back kernel-generated data to the user, thus implementing two-way communication between the kernel and user modes.

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or our website at https://eunomia.dev/zh/tutorials/ for more examples and complete tutorials.

References:

  1. https://lwn.net/Articles/907056/

Original URL: https://eunomia.dev/zh/tutorials/35-user-ringbuf/ Please indicate the source when reprinting.

Overview (`src/36-userspace-ebpf/README.md`)

Userspace eBPF Runtimes: Overview and Applications

Yusheng Zheng

In this blog post, we'll dive into the world of eBPF in userspace. While many are familiar with kernel-based eBPF, userspace eBPF runtimes have been making significant strides and offer compelling use cases. We will also compare userspace eBPF runtimes with Wasm runtimes, another popular technology in the cloud-native and edge computing landscape. Among these, we're excited to introduce bpftime. Powered by an LLVM JIT/AOT backend, our benchmarks suggest that bpftime stands out as one of the fastest userspace eBPF runtimes available.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/36-userspace-ebpf

Introduction to eBPF

What is eBPF?

eBPF, which stands for "extended Berkeley Packet Filter," is a revolutionary technology that facilitates the dynamic tracing and monitoring of kernel operations without modifying kernel source code or rebooting the system. Originally designed for network packet filtering, eBPF has evolved to support a wide range of applications, from performance analysis to security, making it a versatile tool in a system administrator's arsenal.

The story of eBPF begins with the Berkeley Packet Filter (BPF), introduced in the early 1990s as a way to filter and capture network packets efficiently. Over the years, BPF proved to be an invaluable asset, but there was room for improvement. eBPF emerged as an advanced iteration of BPF, equipped with a richer instruction set and the capability to interact with kernel data structures directly.

The Linux kernel adopted eBPF around 2014, and since then, its popularity and adoption have skyrocketed. Key contributors to the Linux kernel worked diligently to evolve eBPF from a simple packet filter to a generic and powerful bytecode engine.

Its significance in modern computing and network solutions

In today's complex computing environments, the need for real-time data and insights is paramount. eBPF shines in this regard, allowing developers and administrators to introspect and modify system behaviors on the fly.

Given its dynamic nature, eBPF has become a cornerstone of modern networking solutions. It enables fine-grained traffic control, load balancing, and security enforcement at the kernel level, ensuring optimal performance and security. Furthermore, in the realm of observability, eBPF provides granular insights into system calls, hardware events, and more, facilitating proactive problem detection and resolution.

eBPF: from kernel runtime to userspace runtime

While the initial design of eBPF was deeply embedded within the kernel, the demand for similar functionality in userspace applications led to the evolution of userspace eBPF runtimes. These runtimes allow developers to leverage eBPF's capabilities outside the kernel, expanding its utility and applicability. Userspace eBPF runtimes make it feasible to apply eBPF's prowess to a broader set of applications, from custom network protocols to novel security solutions, further cementing eBPF's role as a transformative technology in the computing landscape.

Userspace eBPF Runtimes and Their Role

What is a userspace eBPF runtime?

A userspace eBPF runtime provides a platform outside of the kernel to run eBPF programs. While one of eBPF's standout attributes is its capability to execute code within the kernel space, offering rapid observability and data aggregation, there are scenarios where having a userspace alternative becomes valuable. These userspace runtimes extend the reach of eBPF's versatility to areas beyond kernel integrations and often serve as experimental grounds, debugging tools, or frameworks for specific use cases.

Introduction to specific runtimes

**ubpf**

uBPF was among the early attempts to bring eBPF to the userspace. Conceived primarily as a proof-of-concept, it served as a user-space interpretation of an eBPF interpreter combined with an x86_64 and arm64 JIT. Despite its origins as an early prototype, uBPF garnered attention and was utilized as a foundation for high-performance networking projects such as DPDK and Oko. Its non-GPL licensing (Apache) makes it favorable for a wide range of projects, inclusive of proprietary ones. However, as of recent, uBPF is catching up with kernel developments, particularly with contributions from Microsoft for its eBPF Windows implementation. However, develop ubpf and rbpf programs may require a specific toolchain, which may be a barrier for some users. ubpf only have a limited hashmap implementation, which may not be enough for some users.

**rbpf**

rbpf is heavily influenced by uBPF but with an emphasis on Rust, a language renowned for its memory safety guarantees. The creation of rbpf was driven by a desire to explore the intersections of eBPF and Rust. While not as widespread in adoption, notable users of rbpf include the Solana team, employing it for blockchain tools with eBPF-driven smart contracts. One of rbpf's advantages lies in its licensing (MIT), allowing for broad reuse across various projects. rbpf also lacks eBPF map support, and only has JIT support for x86_64.

**bpftime**

Built atop LLVM JIT/AOT, bpftime is a cutting-edge, high-performance eBPF runtime designed exclusively for userspace operations. It stands out with its rapid Uprobe capabilities and Syscall hooks, notably outperforming the kernel Uprobe by a tenfold margin. Additionally, bpftime offers programmatic syscall hooking, shared memory maps, and compatibility with familiar toolchains like libbpf and clang. Its design addresses some kernel eBPF limitations and outpaces plugin systems like the Wasm runtime in certain aspects.

Why is Having a Userspace Version of eBPF Interesting?

eBPF, while renowned for its kernel-space operations, has observed a growing interest in its userspace adaptations. Here's why migrating eBPF to userspace is capturing the attention of technologists:

Enhanced Performance

In kernel operations, the Uprobe component of eBPF is often beleaguered by performance inefficiencies, primarily due to the overheads introduced by context switches. In latency-sensitive applications, these inefficiencies can be detrimental, affecting real-time monitoring and data processing. By transitioning to userspace, eBPF can bypass these context switch related delays, leading to a more optimized performance. Runtimes like bpftime exemplify this, offering substantial performance improvements compared to their kernel counterparts.

Flexibility and Integration

Userspace eBPF runtimes champion flexibility. Unlike some alternatives, such as the Wasm runtime, which might necessitate manual integrations, userspace eBPF provides the boon of automatic instrumentation. This means they can be seamlessly introduced into running processes without the need for cumbersome restarts or recompilations, ensuring smoother operational flows.

Augmented Security

Operating in kernel mode, eBPF programs require root access, which can inadvertently expand the attack surface, making systems susceptible to vulnerabilities like container escapes or even potential kernel exploits. Userspace runtimes, however, operate outside this high-risk zone. By functioning in userspace, they demand fewer privileges, inherently reducing the potential avenues for security breaches.

Debugging and Licensing Flexibility

One of the innate advantages of userspace eBPF runtimes is the ease with which developers can debug their code. The accessibility to integrate breakpoints in a userspace interpreter is a marked advantage over the relatively constrained debugging capabilities in kernel eBPF. Additionally, the licensing flexibility of userspace eBPF runtimes, typically offered under licenses like Apache or MIT, ensures they can be paired with a diverse range of projects, including proprietary ones, sidestepping the GPL constraints associated with kernel code.

Use Cases: Existing eBPF Userspace Applications

Userspace eBPF is being utilized in a number of notable projects, each harnessing the unique capabilities of eBPF to enhance their functionalities. Here's how Userspace eBPF is currently utilized in various applications:

  1. Oko:

    Oko is an extension of Open vSwitch-DPDK that provides runtime extension with BPF programs. It enables the use of BPF programs to process packets in userspace, providing flexible packet processing and facilitating the integration of Open vSwitch with other systems.

  2. DPDK eBPF Support:

    The DPDK (Data Plane Development Kit) eBPF support facilitates fast packet processing by enabling the use of eBPF programs in userspace, which can be loaded and run to analyze network packets. This enhances the flexibility and programmability of network applications without requiring kernel modifications.

  3. Solana:

    Solana utilizes eBPF to implement a JIT (Just-In-Time) compiler, which is essential for executing smart contracts on its blockchain network. The use of eBPF ensures safety, performance, and architecture agnosticism, thus allowing efficient execution of smart contracts across validator nodes on the Solana blockchain.

  4. eBPF for Windows (Work-In-Progress):

    This project is aimed at bringing the eBPF toolchains and APIs familiar in the Linux ecosystem to Windows, allowing existing eBPF toolchains to be utilized on top of Windows. This demonstrates a promising endeavor to extend the capabilities of eBPF beyond Linux, although it's still a work in progress.

The benefits of using eBPF in these applications include:

  • Flexibility: eBPF provides a flexible framework for running programs in the kernel or userspace, enabling developers to extend the functionality of existing systems without modifying their core code.
  • Performance: By allowing JIT compilation and efficient packet processing, eBPF can significantly enhance the performance of network applications and blockchain smart contract execution.
  • Safety and Security: The eBPF framework provides mechanisms for verifying the safety properties of programs before execution, thus ensuring the integrity and security of the systems it is integrated with.
  • Cross-platform Capability: The architecture-agnostic nature of eBPF instruction set enables cross-platform compatibility, as seen in projects like Solana and the work-in-progress eBPF for Windows.

These attributes make eBPF a powerful tool for augmenting a variety of applications, ranging from network processing to blockchain smart contract execution, and beyond. There are also some papers that discuss the use of eBPF in userspace:

  1. RapidPatch: Firmware Hotpatching for Real-Time Embedded Devices:

    This paper introduces a new hotpatching framework named RapidPatch, which is designed to facilitate the propagation of patches by installing generic patches on heterogeneous embedded devices without disrupting other tasks running on them.

    Furthermore, RapidPatch proposes two types of eBPF patches for different types of vulnerabilities and develops an eBPF patch verifier to ensure patch safety.

  2. Femto-Containers: Lightweight Virtualization and Fault Isolation For Small Software Functions on Low-Power IoT Microcontrollers:

    This paper presents Femto-Containers, a novel framework that enables the secure deployment, execution, and isolation of small virtual software functions on low-power IoT devices over a network.

    The framework is implemented and provided in RIOT, a popular open source IoT operating system, with an emphasis on secure deployment, execution, and isolation of small virtual software functions on low-power IoT devices, over the network.

    The paper discusses the implementation of a Femto-Container hosting engine integrated within a common low-power IoT operating system (RIOT), enhancing it with the ability to start, update, or terminate Femto-Containers on demand, securely over a standard IPv6/6LoWPAN network.

These papers delve into pertinent advancements concerning firmware patching and lightweight virtualization, demonstrating innovations that address critical challenges in the domains of real-time embedded systems and low-power IoT microcontrollers respectively.

Userspace eBPF Runtime vs Wasm Runtime

In the evolving landscape of cloud-native and edge computing, both eBPF (extended Berkeley Packet Filter) and Wasm (WebAssembly) have emerged as powerful tools. However, they come with their own set of design principles and trade-offs.

A Comparison of eBPF and Wasm

eBPF:

  • Philosophy: eBPF prioritizes performance, often making it the choice for real-time kernel operations and high-throughput networking tasks.
  • Security: While performance takes the forefront, security in eBPF is ensured through the use of a verifier, ensuring that all programs are safe to run without causing kernel panics or infinite loops.

Wasm:

  • Philosophy: Originally designed for the web, Wasm places a higher emphasis on portability and security. It was conceived to execute code nearly as fast as running native machine code and ensures safety in hostile environments like web browsers.
  • Security: The primary security model for Wasm revolves around Software Fault Isolation (SFI). This model guarantees safe execution by enforcing sandboxing, even though this can introduce some runtime overheads.

For both technologies, reliance on underlying libraries for complex operations is paramount. For instance, Wasm leans on libraries like Wasi-nn for neural network operations. However, when interfacing with such external APIs, especially in Wasm's context, there's a need for additional validation and runtime checks, sometimes leading to substantial performance costs. eBPF, when embedded within the host, capitalizes on its verifier to ensure code safety, offering a more performance-centric approach.

On the language support front, while eBPF's niche and specialized nature mean limited language support, Wasm boasts a broader language portfolio due to its origin and design for the web.

bpftime Quick Start

With bpftime, you can build eBPF applications using familiar tools like clang and libbpf, and execute them in userspace. For instance, the malloc eBPF program traces malloc calls using uprobe and aggregates the counts using a hash map.

You can refer to documents/build-and-test.md for how to build the project, or using the container images from GitHub packages.

To get started, you can build and run a libbpf based eBPF program starts with bpftime cli:

console
make -C example/malloc # Build the eBPF program example
bpftime load ./example/malloc/malloc

In another shell, Run the target program with eBPF inside:

console
$ bpftime start ./example/malloc/victim
Hello malloc!
malloc called from pid 250215
continue malloc...
malloc called from pid 250215

You can also dynamically attach the eBPF program with a running process:

console
$ ./example/malloc/victim & echo $! # The pid is 101771
[1] 101771
101771
continue malloc...
continue malloc...

And attach to it:

console
$ sudo bpftime attach 101771 # You may need to run make install in root
Inject: "/root/.bpftime/libbpftime-agent.so"
Successfully injected. ID: 1

You can see the output from original program:

console
$ bpftime load ./example/malloc/malloc
...
12:44:35 
        pid=247299      malloc calls: 10
        pid=247322      malloc calls: 10

Alternatively, you can also run our sample eBPF program directly in the kernel eBPF, to see the similar output:

console
$ sudo example/malloc/malloc
15:38:05
        pid=30415       malloc calls: 1079
        pid=30393       malloc calls: 203
        pid=29882       malloc calls: 1076
        pid=34809       malloc calls: 8

See documents/usage.md for more details.

Conclusion

Userspace eBPF runtimes are an exciting development that expands the capabilities of eBPF beyond the kernel. As highlighted in this post, they offer compelling benefits like enhanced performance, flexibility, and security compared to kernel-based eBPF. Runtimes like bpftime demonstrate the potential for substantial speedups, even outperforming alternatives like Wasm runtimes in certain dimensions like low-level performance.

With innovative frameworks like RapidPatch and Femto-Containers utilizing userspace eBPF for patching and lightweight virtualization respectively, we are witnessing pioneering use cases that address critical challenges in embedded systems and IoT domains. As eBPF continues its evolution in userspace, we can expect even more creative applications that augment everything from smart contracts to network protocols.

While alternatives like Wasm certainly have their place with a strong emphasis on web portability and security, eBPF's specialized nature gives it an edge for performance-critical tasks. Ultimately, the choice between the two depends on the specific use case and priorities. As they continue to evolve, userspace eBPF runtimes are cementing their position as an indispensable part of the cloud-native technology stack, offering an unparalleled combination of safety, efficiency and innovation.

We encourage our readers to dive deep into the world of userspace eBPF, starting with our bpftime GitHub repository: https://github.com/eunomia-bpf/bpftime Contributions, feedback, or simply using the tool can further the cause and provide invaluable insights to the community.

If you use our project in research, please cite our repo.

reference

  1. bpftime: https://github.com/eunomia-bpf/bpftime
  2. ubpf: https://github.com/iovisor/ubpf
  3. rbpf: https://github.com/qmonnet/rbpf
  4. Oko: https://github.com/Orange-OpenSource/Oko
  5. RapidPatch: Firmware Hotpatching for Real-Time Embedded Devices: https://www.usenix.org/conference/usenixsecurity22/presentation/he-yi
  6. DPDK eBPF Support: https://doc.dpdk.org/guides/prog_guide/bpf_lib.html
  7. Solana: https://solana.com/
  8. eBPF for Windows (Work-In-Progress): https://github.com/microsoft/ebpf-for-windows
  9. Femto-Containers: Lightweight Virtualization and Fault Isolation For Small Software Functions on Low-Power IoT Microcontrollers: https://arxiv.org/abs/2210.03432

Overview (`src/37-uprobe-rust/README.md`)

eBPF Practice: Tracing User Space Rust Applications with Uprobe

eBPF, or Extended Berkeley Packet Filter, is a revolutionary technology in the Linux kernel that allows developers to run custom "micro-programs" in kernel mode, thus changing system behavior or collecting granular performance data without modifying the kernel code.

This article discusses how to trace user space Rust applications with Uprobe and eBPF, including how to obtain symbol names and attach them, get function parameters, get return values, etc. This article is part of the eBPF developer tutorial, more detailed content can be found here: https://eunomia.dev/tutorials/

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/37-uprobe-rust

Uprobe

Uprobe is a user space probe. Uprobe probes allow dynamic instrumentation in user space programs, with instrumentation locations including: function entry points, specific offsets, and function return points. When we define a Uprobe, the kernel creates a fast breakpoint instruction (the int3 instruction on x86 machines) at the attached instruction. When the program executes this instruction, the kernel triggers an event, the program falls into kernel mode, and the probe function is called in a callback manner. After the probe function is executed, it returns to user mode to continue executing subsequent instructions.

Uprobe is useful for parsing traffic in user space that cannot be parsed by kernel probes, such as http2 traffic, https traffic, and can also analyze runtime program, business logic, etc. For more information about Uprobe, you can refer to:

Running Uprobe in kernel mode eBPF might also produce significant performance overhead, in which case you might consider using user space eBPF runtime, such as bpftime. bpftime is a user-space eBPF runtime based on LLVM JIT/AOT. It can run eBPF Uprobe programs in user mode and is compatible with kernel mode eBPF. Because it avoids context switching between user and kernel modes, bpftime's Uprobe overheads are about 10 times less than the kernel's, and it also more easy to extend.

Rust

Rust is an open-source systems programming language that focuses on safety, speed, and concurrency. It was developed by Graydon Hoare at the Mozilla Research Center in 2010 and released its first stable version in 2015. The design philosophy of Rust language is to provide the performance advantages of C++ while greatly reducing memory safety vulnerabilities. Rust is gradually popular in the field of systems programming, especially in applications that require high performance, security, and reliability, such as operating systems, file systems, game engines, network services, etc. Many large technology companies, including Mozilla, Google, Microsoft, and Amazon, are using or supporting the Rust language.

You can refer to the official Rust website for more information about Rust language and install the Rust toolchain.

Simplest example: Symbol name mangling

Let's start with a simple example, tracing the main function of a Rust program with Uprobe, with the code as follows:

rust
pub fn hello() -> i32 {
    println!("Hello, world!");
    0
}

fn main() {
    hello();
}

Build and try to get the symbol:

console
$ cd helloworld
$ cargo build
$ nm -C target/debug/helloworld | grep 'helloworld::main'
0000000000013ec0 t helloworld::main

Rustc uses symbol name mangling to encode unique linker names. Rust 1.97 uses v0 mangling by default, while nm -C prints the stable demangled name. Resolve the raw symbol through its address before passing it to bpftrace:

console
$ main_address=$(nm -C target/debug/helloworld | awk '$2 ~ /^[tT]$/ && $3 ~ /^helloworld::main(::h[[:xdigit:]]+)?$/ && !matched { found = $1; matched = 1 } END { if (matched) print found }')
$ main_symbol=$(nm target/debug/helloworld | awk -v address="$main_address" '$1 == address && $2 ~ /^[tT]$/ && !matched { found = $3; matched = 1 } END { if (matched) print found }')
$ echo "$main_symbol"
_RNvCsiXtUZV4ocyZ_10helloworld4main

The -C symbol-mangling-version rustc option can select a mangling scheme, but resolving the current binary avoids depending on a particular scheme. We can now trace main:

console
$ sudo bpftrace -e "uprobe:target/debug/helloworld:${main_symbol} { printf(\"Function hello-world called\n\"); }"
Attaching 1 probe...
Function hello-world called

Tracing function calls with multiple invocations and getting return values

For a more complex example, which includes multiple calls and retrieving return values:

rust
use std::env;

pub fn hello(i: i32, len: usize) -> i32 {
    println!("Hello, world! {} in {}", i, len);
    i + len as i32
}

fn main() {
    let args: Vec<String> = env::args().collect();

    // Skip the first argument, which is the path to the binary, and iterate over the rest
    for arg in args.iter().skip(1) {
        match arg.parse::<i32>() {
            Ok(i) => {
                let ret = hello(i, args.len());
                println!("return value: {}", ret);
            }
            Err(_) => {
                eprintln!("Error: Argument '{}' is not a valid integer", arg);
            }
        }
    }
}

First, we need to build in debug mode because the hello function gets inlined in release mode and won't have its own symbol:

console
$ cd args
$ cargo build
$ nm -C target/debug/helloworld | grep 'helloworld::hello'
0000000000016f90 t helloworld::hello

Note that in release mode (cargo build --release), only the main function symbol appears because hello gets inlined during optimization.

Rust 1.97 uses v0 symbol mangling by default. bpftrace needs the raw symbol, so resolve it from the address of the stable demangled name:

console
$ hello_address=$(nm -C target/debug/helloworld | awk '$2 ~ /^[tT]$/ && $3 ~ /^helloworld::hello(::h[[:xdigit:]]+)?$/ && !matched { found = $1; matched = 1 } END { if (matched) print found }')
$ hello_symbol=$(nm target/debug/helloworld | awk -v address="$hello_address" '$1 == address && $2 ~ /^[tT]$/ && !matched { found = $3; matched = 1 } END { if (matched) print found }')
$ echo "$hello_symbol"
_RNvCsiXtUZV4ocyZ_10helloworld5hello
$ sudo bpftrace -e "uprobe:target/debug/helloworld:${hello_symbol} { printf(\"Function hello called\n\"); }"
Attaching 1 probe...
Function hello called
Function hello called
Function hello called
Function hello called

When we run the program with multiple arguments, bpftrace correctly catches all calls to the hello function:

console
$ ./target/debug/helloworld 1 2 3 4
Hello, world! 1 in 5
return value: 6
Hello, world! 2 in 5
return value: 7
Hello, world! 3 in 5
return value: 8
Hello, world! 4 in 5
return value: 9

We can also get the return value using Uretprobe and the same resolved symbol:

console
$ sudo bpftrace -e "uretprobe:target/debug/helloworld:${hello_symbol} { printf(\"Function hello returned: %d\n\", retval); }"
Attaching 1 probe...
Function hello returned: 6
Function hello returned: 7
Function hello returned: 8
Function hello returned: 9

Resolving the raw symbol through its demangled name works with both legacy and v0 Rust mangling.

References

Overview (`src/38-btf-uprobe/README.md`)

Expanding eBPF Compile Once, Run Everywhere(CO-RE) to Userspace Compatibility

Yusheng

eBPF, short for extended Berkeley Packet Filter, is a powerful and versatile technology used in modern Linux systems. It allows for the running of sandboxed programs in a virtual machine-like environment within the kernel, providing a safe way to extend the capabilities of the kernel without the risk of crashing the system or compromising security.

Co-RE, standing for 'Compile Once, Run Everywhere', tackles the critical issue of eBPF program compatibility across diverse kernel versions. This feature allows eBPF programs to run on various kernel versions without the need for recompilation, simplifying deployment and maintenance.

With eBPF Uprobe, you can also trace userspace applications and access their internal data structures. However, the CO-RE is not designed for userspace applications. This blog will introduce how to leverage CO-RE for user-space applications, ensuring eBPF Uprobe programs remain compatible across different application versions without the need for multiple compilations.

This approach may be particularly beneficial for tracing applications like OpenSSL, where maintaining separate eBPF programs for each version is impractical. With userspace eBPF runtimes like bpftime, you can also expand the CO-RE to more usecases, including extensions, networking, and dynamic patching, providing versatile and efficient solutions.

To implement the Co-RE feature of eBPF in user-space applications, we also need to utilize the BPF Type Format (BTF) to overcome some of the limitations of traditional eBPF programs. The key to this approach lies in providing user-space programs with similar type information and compatibility support as the kernel, thereby enabling eBPF programs to more flexibly handle different versions of user-space applications and libraries.

This article is part of the eBPF Developer Tutorial, and for more detailed content, you can visit https://eunomia.dev/tutorials/.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe

Why we need CO-RE?

  • Kernel Dependencies: Traditional eBPF programs are tightly coupled with the specific Linux kernel version they are compiled for. This is because they rely on specific internal data structures and kernel APIs which can change between kernel versions.
  • Portability Issues: If you wanted to run an eBPF program on different Linux systems with different kernel versions, you'd traditionally have to recompile the eBPF program for each kernel version, which is a cumbersome and inefficient process.

The Co-RE Solution

  • Abstracting Kernel Dependencies: Co-RE enables eBPF programs to be more portable by abstracting away specific kernel dependencies. This is achieved through the use of BPF Type Format (BTF) and relocations.
  • BPF Type Format (BTF): BTF provides rich type information about data structures and functions in the kernel. This metadata allows eBPF programs to understand the layout of kernel structures at runtime.
  • Relocations: eBPF programs compiled with Co-RE support contain relocations that are resolved at load time. These relocations adjust the program's references to kernel data structures and functions according to the actual layout and addresses in the running kernel.

Advantages of Co-RE

  1. Write Once, Run Anywhere: eBPF programs compiled with Co-RE can run on different kernel versions without the need for recompilation. This greatly simplifies the deployment and maintenance of eBPF programs in diverse environments.
  2. Safety and Stability: Co-RE maintains the safety guarantees of eBPF, ensuring that programs do not crash the kernel and adhere to security constraints.
  3. Ease of Development: Developers don't need to worry about the specifics of each kernel version, which simplifies the development of eBPF programs.

Problem: userspace application CO-RE

The eBPF also supports tracing userspace applications. Uprobe is a user-space probe that allows dynamic instrumentation in user-space programs. The probe locations include function entry, specific offsets, and function returns.

The BTF is designed for the kernel and generated from vmlinux, it can help the eBPF program to be easily compatible with different kernel versions.

The userspace application, however, also need CO-RE. For example, the SSL/TLS uprobe is widely used to capture the plaintext data from the encrypted traffic. It is implemented with the userspace library, such as OpenSSL, GnuTLS, NSS, etc. The userspace application and libraries also has different versions, it would be complex if we need to compile and maintain the eBPF program for each version.

Let's see what will happen if CO-RE is not enabled for userspace applications, and how the BTF from userspace applications can solve this.

No BTF for userspace program

This is a simple uprobe example, it can capture the function call and arguments of the add_test function in the userspace program. You can add #define BPF_NO_PRESERVE_ACCESS_INDEX in the uprobe.bpf.c to make sure the eBPF program can be compiled without BTF for struct data.

c
#define BPF_NO_GLOBAL_DATA
#define BPF_NO_PRESERVE_ACCESS_INDEX
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct data {
        int a;
        int c;
        int d;
};

SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int a = 0, c = 0;
    bpf_probe_read_user(&a, sizeof(a), &d->a);
    bpf_probe_read_user(&c, sizeof(c), &d->c);
    bpf_printk("add_test(&d) %d + %d = %d\n", a, c,  a + c);
    return a + c;
}

char LICENSE[] SEC("license") = "Dual BSD/GPL";

Then, we have two different versions of the userspace program, examples/btf-base and examples/btf-base-new. The struct data is different in the two versions.

examples/btf-base:

c
// use a different struct
struct data {
        int a;
        int c;
        int d;
};

int add_test(struct data *d) {
    return d->a + d->c;
}

int main(int argc, char **argv) {
    struct data d = {1, 3, 4};
    printf("add_test(&d) = %d\n", add_test(&d));
    return 0;
}

examples/btf-base-new:

c
struct data {
        int a;
        int b;
        int c;
        int d;
};

int add_test(struct data *d) {
    return d->a + d->c;
}

int main(int argc, char **argv) {
    struct data d = {1, 2, 3, 4};
    printf("add_test(&d) = %d\n", add_test(&d));
    return 0;
}

We can use pahole and clang to generate the btf for each version of userspace applications. The pahole tool can simply generate BTF from the debug info: https://linux.die.net/man/1/pahole

make examples and generate btf for them:

sh
make -C example # it's like: pahole --btf_encode_detached base.btf btf-base.o

The we execute the eBPF program with the userspace program. for btf-base:

sh
sudo ./uprobe examples/btf-base

And also the userspace program:

console
$ examples/btf-base
add_test(&d) = 4

We will see:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
           <...>-25458   [000] ...11 27694.081465: bpf_trace_printk: add_test(&d) 1 + 3 = 4

For btf-base-new:

sh
sudo ./uprobe examples/btf-base-new

And also the userspace program:

console
$ examples/btf-base-new
add_test(&d) = 4

But we will see:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe\
           <...>-25809   [001] ...11 27828.314224: bpf_trace_printk: add_test(&d) 1 + 2 = 3

The result is different, because the struct data is different in the two versions. The eBPF program can't be compatible with different versions of the userspace program, so we cannot get the correct information.

Use BTF for userspace program

Comment the #define BPF_NO_PRESERVE_ACCESS_INDEX in the uprobe.bpf.c to make sure the eBPF program can be compiled with BTF for struct data.

c
#define BPF_NO_GLOBAL_DATA
// #define BPF_NO_PRESERVE_ACCESS_INDEX
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif

struct data {
        int a;
        int c;
        int d;
};

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif


SEC("uprobe/examples/btf-base:add_test")
int BPF_UPROBE(add_test, struct data *d)
{
    int a = 0, c = 0;
    bpf_probe_read_user(&a, sizeof(a), &d->a);
    bpf_probe_read_user(&c, sizeof(c), &d->c);
    bpf_printk("add_test(&d) %d + %d = %d\n", a, c,  a + c);
    return a + c;
}

char LICENSE[] SEC("license") = "Dual BSD/GPL";

The record of struct data is preserved in the eBPF program. Then, we can use the btf-base.btf to compile the eBPF program.

Merge user btf with kernel btf, so we have a complete btf for the kernel and userspace:

sh
./merge-btf /sys/kernel/btf/vmlinux examples/base.btf target-base.btf

Then we execute the eBPF program with the userspace program. for btf-base:

console
$ sudo ./uprobe examples/btf-base target-base.btf
...
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:1 @ offset 4)
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 4
...

Execute the userspace program and get result:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
[sudo] password for yunwei37: 
           <...>-26740   [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4

Also, we do the same for another version of the userspace program btf-base-new:

console
$ ./merge-btf /sys/kernel/btf/vmlinux examples/base-new.btf target-base-new.btf
$ sudo ./uprobe examples/btf-base-new target-base-new.btf
....
libbpf: sec 'uprobe/examples/btf-base:add_test': found 3 CO-RE relocations
libbpf: CO-RE relocating [2] struct pt_regs: found target candidate [357] struct pt_regs in [vmlinux]
libbpf: prog 'add_test': relo #0: <byte_off> [2] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: matching candidate #0 <byte_off> [357] struct pt_regs.di (0:14 @ offset 112)
libbpf: prog 'add_test': relo #0: patched insn #0 (LDX/ST/STX) off 112 -> 112
libbpf: CO-RE relocating [7] struct data: found target candidate [133110] struct data in [vmlinux]
libbpf: prog 'add_test': relo #1: <byte_off> [7] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: matching candidate #0 <byte_off> [133110] struct data.a (0:0 @ offset 0)
libbpf: prog 'add_test': relo #1: patched insn #4 (ALU/ALU64) imm 0 -> 0
libbpf: prog 'add_test': relo #2: <byte_off> [7] struct data.c (0:1 @ offset 4)
libbpf: prog 'add_test': relo #2: matching candidate #0 <byte_off> [133110] struct data.c (0:2 @ offset 8)
libbpf: prog 'add_test': relo #2: patched insn #11 (ALU/ALU64) imm 4 -> 8
libbpf: elf: symbol address match for 'add_test' in 'examples/btf-base-new': 0x1140
Successfully started! Press Ctrl+C to stop.

The result is correct:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
[sudo] password for yunwei37: 
           <...>-26740   [001] ...11 28180.156220: bpf_trace_printk: add_test(&d) 1 + 3 = 4

For complete source code, you can visit https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/38-btf-uprobe for more details.

The eBPF uprobe tracing program almost doesn't need any modifications. We just need to load the BTF containing the offsets of kernel and user-space structures. This is the same usage as enabling CO-RE on older kernel versions without BTF information:

c
LIBBPF_OPTS(bpf_object_open_opts , opts,
    );
    LIBBPF_OPTS(bpf_uprobe_opts, uprobe_opts);
    if (argc != 3 && argc != 2) {
        fprintf(stderr, "Usage: %s <example-name> [<external-btf>]\n", argv[0]);
        return 1;
    }
    if (argc == 3)
        opts.btf_custom_path = argv[2];

    /* Set up libbpf errors and debug info callback */
    libbpf_set_print(libbpf_print_fn);

    /* Cleaner handling of Ctrl-C */
    signal(SIGINT, sig_handler);
    signal(SIGTERM, sig_handler);

    /* Load and verify BPF application */
    skel = uprobe_bpf__open_opts(&opts);
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

In fact, the BTF implementation for relocation requires two parts: the compile-time BTF information carried by the BPF program, and the BTF information of the kernel when loading the eBPF program. When actually loading the eBPF program, libbpf will modify potentially incorrect eBPF instructions based on the accurate BTF information of the current kernel, ensuring compatibility across different kernel versions.

Interestingly, libbpf does not differentiate whether these BTF information come from user-space programs or the kernel. Therefore, by merging the user-space BTF information with kernel BTF and provide them to libbpf, the problem is solved.

And also, since the relocation is happened in userspace loader(like libbpf), both kernel eBPF runtime and userspace eBPF runtimes(Such as bpftime) can benefit from the CO-RE. bpftime (https://github.com/eunomia-bpf/bpftime) is an open-source user-space eBPF runtime based on LLVM JIT/AOT. It enables the execution of eBPF programs in user space, compatible with kernel-space eBPF. While supporting uprobes, syscall trace, and general plugin extensions, it avoids the context switching between kernel and user spaces, thereby enhancing the execution efficiency of uprobe programs. With the support of libbpf and BTF, bpftime can also dynamically extend user-space applications, achieving compatibility across different versions of user-space programs.

For more details about BTF relocation, you may refer to https://nakryiko.com/posts/bpf-core-reference-guide/

Conclusion

  • Flexibility and Compatibility: The use of BTF in user-space eBPF programs greatly enhances their flexibility and compatibility across different versions of user-space applications and libraries.
  • Reduced Complexity: This approach significantly reduces the complexity involved in maintaining eBPF programs for different versions of user-space applications, as it eliminates the need for multiple program versions.
  • Potential for Broader Application: While your example focused on SSL/TLS monitoring, this methodology may has broader applications in performance monitoring, security, and debugging of user-space applications.

This example showcases a significant advancement in the practical application of eBPF, extending its powerful features to more dynamically handle user-space applications in a Linux environment. It's a compelling solution for software engineers and system administrators dealing with the complexities of modern Linux systems.

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ to get more examples and complete tutorials.

Overview (`src/39-nginx/README.md`)

Using eBPF to Trace Nginx Requests

Nginx is one of the most popular web servers and reverse proxies in the world, known for its high performance, stability, and low resource consumption. It is widely used for serving static content, load balancing, and acting as a reverse proxy for dynamic applications. To maintain its performance edge, it's crucial to monitor and optimize Nginx's operations, especially when handling a large number of requests. One powerful way to gain insights into Nginx's performance is by using eBPF (Extended Berkeley Packet Filter).

eBPF is a revolutionary technology that allows developers to run custom programs in the Linux kernel. Originally designed for network packet filtering, eBPF has evolved into a versatile tool for tracing, monitoring, and profiling system behavior in both kernel and user space. By leveraging eBPF, you can trace Nginx's critical functions, measure latency, and identify bottlenecks without modifying the source code or restarting the service.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/39-nginx

Background: Nginx and eBPF

Nginx

Nginx operates on an event-driven architecture, making it highly efficient and capable of handling thousands of simultaneous connections with minimal resources. This efficiency is achieved through various performance-critical functions involved in request processing, response generation, and event handling. Understanding how these functions behave under different loads is key to optimizing Nginx for your specific use case.

eBPF

eBPF programs run in a secure, sandboxed environment within the Linux kernel. These programs can attach to various hooks, such as system calls, tracepoints, and even user-space functions via uprobes (user-level probes). This capability allows you to collect detailed performance data and enforce policies in real time, making eBPF an invaluable tool for system observability.

One common use case of eBPF is tracing function execution to measure latency, which is particularly useful for understanding how long specific Nginx functions take to execute. This information can help in diagnosing performance issues, optimizing resource usage, and improving the overall efficiency of your Nginx deployment.

Uprobes

Uprobes are a type of probe that can be used to trace functions in user-space applications, such as Nginx. They work by attaching to specific user-space function entry and exit points, allowing you to capture precise timing information. However, it’s important to note that using uprobes in the kernel mode eBPF runtime may cause some performance overhead. To mitigate this, you can consider using a user-mode eBPF runtime like bpftime, which is based on LLVM JIT/AOT. This runtime can run eBPF programs in user space, offering compatibility with kernel mode eBPF while potentially reducing overhead.

Performance-Critical Functions in Nginx

Here are some key Nginx functions that are performance-critical and can be monitored using eBPF:

  • ngx_http_process_request: Processes incoming HTTP requests. Monitoring this function helps track the start of request handling.
  • ngx_http_upstream_send_request: Handles sending requests to upstream servers when Nginx is acting as a reverse proxy.
  • ngx_http_finalize_request: Finalizes HTTP request processing, including sending the response. Tracing this can measure total request handling time.
  • ngx_event_process_posted: Processes queued events as part of the Nginx event loop.
  • ngx_handle_read_event: Handles read events from sockets, crucial for monitoring network I/O performance.
  • ngx_writev_chain: Sends responses back to the client, typically used in conjunction with the write event loop.

Using bpftrace to Trace Nginx Functions

To monitor these functions, we can use bpftrace, a high-level tracing language for eBPF. Below is a script that traces the execution time of several critical Nginx functions:

bt
#!/usr/sbin/bpftrace

// Monitor the start of HTTP request processing
uprobe:/usr/sbin/nginx:ngx_http_process_request
{
    printf("HTTP request processing started (tid: %d)\n", tid);
    @start[tid] = nsecs;
}

// Monitor when an HTTP request is finalized
uretprobe:/usr/sbin/nginx:ngx_http_finalize_request
/@start[tid]/
{
    $elapsed = nsecs - @start[tid];
    printf("HTTP request processed in %d ns (tid: %d)\n", $elapsed, tid);
    delete(@start[tid]);
}

// Monitor the start of sending a request to an upstream server
uprobe:/usr/sbin/nginx:ngx_http_upstream_send_request
{
    printf("Upstream request sending started (tid: %d)\n", tid);
    @upstream_start[tid] = nsecs;
}

// Monitor when the upstream request is sent
uretprobe:/usr/sbin/nginx:ngx_http_upstream_send_request
/@upstream_start[tid]/
{
    $elapsed = nsecs - @upstream_start[tid];
    printf("Upstream request sent in %d ns (tid: %d)\n", $elapsed, tid);
    delete(@upstream_start[tid]);
}

// Monitor the start of event processing
uprobe:/usr/sbin/nginx:ngx_event_process_posted
{
    printf("Event processing started (tid: %d)\n", tid);
    @event_start[tid] = nsecs;
}

// Monitor when event processing is completed
uretprobe:/usr/sbin/nginx:ngx_event_process_posted
/@event_start[tid]/
{
    $elapsed = nsecs - @event_start[tid];
    printf("Event processed in %d ns (tid: %d)\n", $elapsed, tid);
    delete(@event_start[tid]);
}

Running the Program

To run the above script, start Nginx and use a tool like curl to generate HTTP requests:

bt
# bpftrace /home/yunwei37/bpf-developer-tutorial/src/39-nginx/trace.bt
Attaching 4 probes...
Event processing started (tid: 1071)
Event processed in 166396 ns (tid: 1071)
Event processing started (tid: 1071)
Event processed in 87998 ns (tid: 1071)
HTTP request processing started (tid: 1071)
HTTP request processed in 1083969 ns (tid: 1071)
Event processing started (tid: 1071)
Event processed in 92597 ns (tid: 1071)

The script monitors the start and end times of various Nginx functions and prints the elapsed time for each. This data can be used to analyze and optimize the performance of your Nginx server.

Testing Function Latency in Nginx

For a more detailed analysis of function latency, you can use the funclatency tool, which measures the latency distribution of Nginx functions. Here’s how to test the latency of the ngx_http_process_request function:

console
# sudo ./funclatency /usr/sbin/nginx:ngx_http_process_request
tracing /usr/sbin/nginx:ngx_http_process_request...
tracing func ngx_http_process_request in /usr/sbin/nginx...
Tracing /usr/sbin/nginx:ngx_http_process_request.  Hit Ctrl-C to exit
^C
     nsec                : count    distribution
         0 -> 1          : 0        |                                        |
   524288 -> 1048575    : 16546    |****************************************|
   1048576 -> 2097151    : 2296     |*****                                   |
   2097152 -> 4194303    : 1264     |***                                     |
   4194304 -> 8388607    : 293      |                                        |
   8388608 -> 16777215   : 37       |                                        |
Exiting trace of /usr/sbin/nginx:ngx_http_process_request

Summary of Results

The results above show the distribution of latency for the ngx_http_process_request function. The majority of requests were processed within 524,288 to 1,048,575 nanoseconds, with a smaller percentage taking longer. This information can be crucial in identifying performance bottlenecks and optimizing request handling in Nginx.

By using funclatency, you can:

  • Identify Performance Bottlenecks: Understand which functions are taking the most time to execute and focus your optimization efforts there.
  • Monitor System Performance: Regularly monitor function latency to ensure your Nginx server is performing optimally, especially under heavy load.
  • Optimize Nginx Configuration: Use the insights gained from latency measurements to tweak Nginx settings or modify your application to improve overall performance.

You can find the funclatency tool in the bpf-developer-tutorial repository.

Conclusion

Tracing Nginx requests with eBPF provides valuable insights into the performance of your web server, allowing you to monitor, analyze, and optimize its operation. By using tools like bpftrace and funclatency, you can measure function execution times, identify bottlenecks, and make data-driven decisions to improve your Nginx deployment.

For those interested in learning more about eBPF, including more advanced examples and tutorials, please visit our https://github.com/eunomia-bpf/bpf-developer-tutorial or check out our https://eunomia.dev/tutorials/.

Overview (`src/4-opensnoop/README.md`)

eBPF Tutorial by Example 4: Capturing Opening Files and Filter with Global Variables

eBPF (Extended Berkeley Packet Filter) is a kernel execution environment that allows users to run secure and efficient programs in the kernel. It is commonly used for network filtering, performance analysis, security monitoring, and other scenarios. The power of eBPF lies in its ability to capture and modify network packets or system calls at runtime in the kernel, enabling monitoring and adjustment of the operating system's behavior.

This article is the fourth part of the eBPF Tutorial by Example, mainly focusing on how to capture the system call collection of process opening files and filtering process PIDs using global variables in eBPF.

In Linux system, the interaction between processes and files is achieved through system calls. System calls serve as the interface between user space programs and kernel space programs, allowing user programs to request specific operations from the kernel. In this tutorial, we focus on the sys_openat system call, which is used to open files.

When a process opens a file, it issues a sys_openat system call to the kernel and passes relevant parameters (such as file path, open mode, etc.). The kernel handles this request and returns a file descriptor, which serves as a reference for subsequent file operations. By capturing the sys_openat system call, we can understand when and how a process opens a file.

Capturing the System Call Collection of Process Opening Files in eBPF

First, we need to write an eBPF program to capture the system call of a process opening a file. The specific implementation is as follows:

c
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>

/// @description "Process ID to trace"
const volatile int pid_target = 0;

SEC("tracepoint/syscalls/sys_enter_openat")
int tracepoint__syscalls__sys_enter_openat(struct trace_event_raw_sys_enter* ctx)
{
    u64 id = bpf_get_current_pid_tgid();
    u32 pid = id >> 32;

    if (pid_target && pid_target != pid)
        return false;

    // Use bpf_printk to print the process information
    bpf_printk("Process ID: %d enter sys openat\n", pid);
    return 0;
}

/// "Trace open family syscalls."
char LICENSE[] SEC("license") = "GPL";

Let's look at the key parts of this program. The global variable pid_target is declared with const volatile. The const means the eBPF program can't modify it (it's read-only from the kernel's perspective), while volatile tells the compiler that user-space can modify it before loading the program. This combination lets you pass runtime configuration from user-space to your eBPF program.

When pid_target is 0, the program captures openat calls from all processes. If you set it to a specific PID, it only monitors that process. This filtering happens in the kernel, which is much more efficient than filtering in user-space - you're not wasting resources sending events you don't care about.

The filtering logic is straightforward. We get the current process ID using bpf_get_current_pid_tgid() and shift it right by 32 bits to extract just the PID (the function returns both PID and TID packed into a 64-bit value). Then we check if it matches our target. If not, we return early and skip the event.

The annotation /// @description "Process ID to trace" above the global variable is special - eunomia-bpf uses it to automatically generate command-line help text. This makes your tool more user-friendly without extra code.

This eBPF program can be loaded into the kernel and executed using tools like libbpf or eunomia-bpf. It captures the sys_openat system call of the specified process (or all processes) and outputs relevant information in user-space.

We use eunomia-bpf to compile and run this example. You can install it from https://github.com/eunomia-bpf/eunomia-bpf. The complete code is at https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/4-opensnoop.

Compile and run the above code:

console
$ ecc opensnoop.bpf.c
Compiling bpf object...
Packing ebpf object and config into package.json...
$ sudo ecli run package.json
Running eBPF program...

After running this program, you can view the output of the eBPF program by viewing the /sys/kernel/debug/tracing/trace_pipe file:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
           <...>-3840345 [010] d... 3220701.101179: bpf_trace_printk: Process ID: 3840345 enter sys openat
           <...>-3840345 [010] d... 3220702.158000: bpf_trace_printk: Process ID: 3840345 enter sys openat

At this point, we are able to capture the sys_openat system call for opening files by processes.

How Global Variables Work in eBPF

Global variables in eBPF are stored in the data section of your compiled program. When you load the eBPF program into the kernel, these variables get their initial values. The neat part is that user-space can modify these values before the program starts running, effectively passing configuration parameters into your kernel code.

In our example, the global variable pid_target is used to filter process PIDs. User space programs can set the value of this variable to capture only the sys_openat system calls related to the specified PID in the eBPF program.

The principle of using global variables is that they are defined and stored in the data section of eBPF programs. When the eBPF program is loaded into the kernel and executed, these global variables are retained in the kernel and can be accessed through BPF system calls. User space programs can use certain features of BPF system calls, such as bpf_obj_get_info_by_fd and bpf_obj_get_info, to obtain information about the eBPF object, including the position and value of global variables.

You can view the help information for opensnoop by executing the command ecli -h:

console
$ ecli package.json -h
Usage: opensnoop_bpf [--help] [--version] [--verbose] [--pid_target VAR]

Trace open family syscalls.

Optional arguments:
  -h, --help    shows help message and exits 
  -v, --version prints version information and exits 
  --verbose     prints libbpf debug information 
  --pid_target  Process ID to trace 

Built with eunomia-bpf framework.
See https://github.com/eunomia-bpf/eunomia-bpf for more information.

You can specify the PID of the process to capture by using the --pid_target option, for example:

console
$ sudo ./ecli run package.json  --pid_target 618
Running eBPF program...

After running this program, you can view the output of the eBPF program by viewing the /sys/kernel/debug/tracing/trace_pipe file:

console
$ sudo cat /sys/kernel/debug/tracing/trace_pipe".\-3840345 [010] d... 3220701.101179: bpf_trace_printk: Process ID: 618 enter sys openat
\-3840345 [010] d... 3220702.158000: bpf_trace_printk: Process ID: 618 enter sys openat

Summary

This article introduces how to use eBPF programs to capture the system calls for process file opening. In an eBPF program, we can capture the system calls for process file opening by defining functions tracepoint__syscalls__sys_enter_open and tracepoint__syscalls__sys_enter_openat and attaching them to the tracepoints sys_enter_open and sys_enter_openat using the SEC macro. We can use the bpf_get_current_pid_tgid function to get the process ID that calls the open or openat system call, and print it out in the kernel log using the bpf_printk function. In an eBPF program, we can also filter the output by defining a global variable pid_target to specify the pid of the process to be captured, only outputting the information of the specified process.

By learning this tutorial, you should have a deeper understanding of how to capture and filter system calls for specific processes in eBPF. This method has widespread applications in system monitoring, performance analysis, and security auditing.

If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository at https://github.com/eunomia-bpf/bpf-developer-tutorial or website https://eunomia.dev/tutorials/ for more examples and a complete tutorial.

Overview (`src/40-mysql/README.md`)

Using eBPF to Trace MySQL Queries

MySQL is one of the most widely used relational database management systems in the world. Whether you are running a small application or a large-scale enterprise system, understanding the performance characteristics of your MySQL database can be crucial. In particular, knowing how long SQL queries take to execute and which queries are consuming the most time can help in diagnosing performance issues and optimizing your database for better efficiency.

This is where eBPF (Extended Berkeley Packet Filter) comes into play. eBPF is a powerful technology that allows you to write programs that can run in the Linux kernel, enabling you to trace, monitor, and analyze various aspects of system behavior, including the performance of applications like MySQL. In this blog, we'll explore how to use eBPF to trace MySQL queries, measure their execution time, and gain valuable insights into your database's performance.

The complete source code: https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/40-mysql

Background: MySQL and eBPF

MySQL

MySQL is a relational database management system (RDBMS) that uses Structured Query Language (SQL) to manage and query data. It is widely used for a variety of applications, from web applications to data warehousing. MySQL's performance can be critical to the overall performance of your application, especially when dealing with large datasets or complex queries.

eBPF

eBPF is a technology that allows for the execution of custom programs in the Linux kernel without the need to modify the kernel source code or load kernel modules. Initially designed for network packet filtering, eBPF has evolved into a versatile tool for performance monitoring, security, and debugging. eBPF programs can be attached to various kernel and user-space events, making it possible to trace the execution of functions, system calls, and more.

Using eBPF, we can trace the execution of MySQL functions, such as dispatch_command, which is responsible for handling SQL queries. By tracing this function, we can capture the start and end times of query execution, measure the latency, and log the executed queries.

Tracing MySQL Queries with eBPF

To trace MySQL queries using eBPF, we can write a script using bpftrace, a high-level tracing language for eBPF. Below is a script that traces the dispatch_command function in MySQL to log executed queries and measure their execution time:

bt
#!/usr/bin/env bpftrace

// Trace the dispatch_command function in MySQL
uprobe:/usr/sbin/mysqld:dispatch_command
{
    // Store the start time of the command execution in the map
    @start_times[tid] = nsecs;
    
    // Print the process ID and command string
    printf("MySQL command executed by PID %d: ", pid);
    
    // The third argument to dispatch_command is the SQL query string
    printf("%s\n", str(arg3));
}

uretprobe:/usr/sbin/mysqld:dispatch_command
{
    // Retrieve the start time from the map
    $start = @start_times[tid];
    
    // Calculate the latency in milliseconds
    $delta = (nsecs - $start) / 1000000;
    
    // Print the latency
    printf("Latency: %u ms\n", $delta);
    
    // Delete the entry from the map to avoid memory leaks
    delete(@start_times[tid]);
}

Explanation of the Script

  1. Tracing the dispatch_command Function:
    • The script attaches an uprobe to the dispatch_command function in MySQL. This function is called whenever MySQL needs to execute a SQL query. Uprobe in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as bpftime.
    • The uprobe captures the start time of the function execution and logs the SQL query being executed.

Explore Related Technologies & Topics

View All 9,111+ Repositories β†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#bpf Browse bpf catalog β†’ #ebpf Browse ebpf catalog β†’ #examples Browse examples catalog β†’ #libbpf Browse libbpf catalog β†’ #tutorial Browse tutorial catalog β†’ #xdp Browse xdp catalog β†’ ⚑ MCP Servers Model Context Protocol β†’ 🧠 Agent Skills Prompt & rule files β†’ πŸ”Œ OpenAPI Specs REST & Swagger schemas β†’