tracee Documentation

Technical Documentation: aquasecurity/tracee

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`Readme.md`) Copy Section

Before moving on, please consider giving us a GitHub star ⭐️. Thank you!

About Tracee Copy Chapter

Tracee is a runtime security and observability tool that helps you understand how your system and applications behave.
It is using eBPF technology to tap into your system and expose that information as events that you can consume.
Events range from factual system activity events to sophisticated security events that detect suspicious behavioral patterns.

To learn more about Tracee, check out the documentation.

Quickstart Copy Chapter

To quickly try Tracee use one of the following snippets. For a more complete installation guide, check out the Installation section.
Tracee should run on most common Linux distributions and kernels. For compatibility information see the Prerequisites page. Mac users, please read this FAQ.

Using Docker Copy Section

docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest

For a complete walkthrough please see the Docker getting started guide.

On Kubernetes Copy Section

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace

kubectl logs --follow --namespace tracee daemonset/tracee

For a complete walkthrough please see the Kubernetes getting started guide.

Contributing Copy Chapter

Join the community, and talk to us about any matter in the GitHub Discussions or Slack.
If you run into any trouble using Tracee or you would like to give use user feedback, please create an issue.

Find more information on contribution documentation.

More about Aqua Security Copy Chapter

Tracee is an Aqua Security open source project.
Learn about our open source work and portfolio here.

Overview (`brand/readme.md`) Copy Section

This directory contains media assets, such as the Tracee logo. Assets under this directory are provided under the Creative Commons - BY 4.0 License. For more details, see here: https://creativecommons.org/licenses/by/4.0/

Overview (`common/README.md`) Copy Section

Common Module

The common module provides shared utilities and types that can be used by both Tracee itself and Go signatures. This module is designed to be lightweight, self-contained, and independent of Tracee's core business logic.

Purpose Copy Chapter

This module serves as a foundation layer that provides:

• Common data structures and types
• Utility functions for basic operations
• Shared constants and definitions
• Helper functions that don't contain Tracee-specific logic

Design Principles Copy Chapter

Self-Contained Copy Section

The common module should be self-contained and not depend on other Tracee modules or packages. External dependencies are limited to well-established, minimal libraries that provide essential functionality.

No Business Logic Copy Section

This module must not contain any business logic specific to Tracee's core functionality. It should only provide generic utilities and types that could potentially be useful in other contexts.

Minimal Dependencies Copy Section

• Primarily uses Go standard library packages
• External dependencies are limited to well-established, minimal libraries (e.g., optimized hashing, LRU caches, structured logging, system interfaces)
• No dependencies on other Tracee modules or packages

Usage Copy Chapter

In Consumer code Copy Section

import "github.com/aquasecurity/tracee/common"

Packages Copy Chapter

Data Structures & Utilities Copy Section

• bitwise - Bit manipulation utilities for flag operations and bitwise calculations
• bucketcache - Bucket-based caching system for efficient memory-aware data storage
• changelog - Time-ordered generic changelog data structure for tracking changes over time
• counter - Thread-safe atomic counter with overflow/underflow protection
• murmur - Generic MurMur hash functions for efficient hashing and data integrity verification
• interfaces - Generic interfaces for common patterns (cloning, iteration)
• set - Generic set data structures for efficient collection operations
• stringutil - String manipulation utilities for trimming and processing text data

System Integration Copy Section

• capabilities - Linux capabilities handling and management for process permissions
• cgroup - Control group utilities for container and process resource management
• environment - System environment detection (kernel version, OS info, CPU/memory details)
• mount - Mount point utilities for filesystem and container mount management
• proc - /proc filesystem utilities for process and system information extraction
• system - System monitoring utilities for resource and performance tracking

File & Binary Analysis Copy Section

• digest - File content hashing (SHA256) with caching for efficient content verification and integrity checking
• elf - ELF file analysis, magic number validation, and symbol extraction utilities
• fileutil - File operation utilities with atomic operations, directory management, and protected memory-mapped file reading
• sharedobjs - Shared object and library handling for dynamic loading and symbol resolution

I/O & Utilities Copy Section

• errfmt - Error formatting utilities for consistent error handling and display
• logger - Structured logging interface with filtering and formatting capabilities
• parsers - Comprehensive parsing utilities including:
  • System call argument parsers for Linux syscalls (flags, capabilities, socket types, etc.)
  • Network address parsers for sockaddr structures (IPv4, IPv6, Unix domain sockets)
  • File operation parsers (read/write detection, memory path identification)
• timeutil - Time-related utilities for timestamp handling and time operations

What Belongs Here Copy Chapter

✅ Appropriate content:

• Generic data structures (maps, slices utilities)
• String manipulation helpers
• Type conversion utilities
• Mathematical helpers
• Generic algorithms
• Common constants that aren't Tracee-specific
• Basic validation functions

❌ Inappropriate content:

• eBPF-related logic
• Event processing functions
• Policy enforcement code
• Container runtime integrations
• Signature detection logic
• Tracee-specific configuration

Contributing Copy Chapter

When adding to this module, ensure that:

1. Generic functionality - Code should be useful beyond Tracee's specific use case
2. Self-contained - No imports from other Tracee packages
3. Well documented - All public functions and types should have clear documentation
4. Tested - Include unit tests for new functionality
5. Package organization - New code should be placed in internal packages (e.g., common/hash/hash.go) rather than directly under common (e.g., common/hash.go)

Overview (`detectors/README.md`) Copy Section

Tracee Detectors

This module contains event detector implementations for Tracee's detector API.

Detector Types Copy Chapter

Go Detectors (Built-in) Copy Section

Go detectors in this module use automatic registration via init() functions:

package detectors

func init() {
    register(&MyDetector{})
}

type MyDetector struct {
    // detector fields
}

// Implement detection.EventDetector interface methods

No manual registration list is needed - detectors are automatically discovered during package initialization.

YAML Detectors Copy Section

YAML detectors provide a declarative way to define threat detection and derived events without writing Go code.

Example YAML detectors are provided in the repository's examples/detectors/yaml/ directory for reference. These are pure reference files meant to be copied and customized by users.

See the YAML Detectors documentation for syntax, examples, and usage details.

Module Structure Copy Chapter

• registry.go - Auto-registration system for Go detectors
• example_detector.go - Comprehensive educational example (see below)
• *.go - Built-in Go detector implementations

Usage Copy Chapter

Tracee's main package imports this module and calls GetAllDetectors() to retrieve all registered detectors:

import "github.com/aquasecurity/tracee/detectors"

allDetectors := detectors.GetAllDetectors()
// Register detectors with the detector engine

Example Detector Copy Chapter

Comprehensive Example: `example_detector.go` Copy Section

A fully-featured Go detector demonstrating all detector API capabilities:

• DataStore API usage (ContainerStore, SystemStore)
• Event filtering patterns (scope filters, data filters, version constraints)
• Conditional field population and enrichment
• Proper error handling with sentinel errors
• Auto-population features (Threat, DetectedFrom, ProcessAncestry)

Usage:

1. For reference: Read the code to learn detector patterns
2. For testing: Build Tracee with the example included:

# Build Tracee with example detectors included
make tracee-with-examples

# Or manually override GO_TAGS_EBPF
make tracee GO_TAGS_EBPF="core,ebpf,lsmsupport,detectorexamples"

# Run tracee and filter for the example detection
sudo ./dist/tracee -e example_detection --output json

Note: This detector is excluded from default builds using the detectorexamples build tag to prevent noise (it triggers on all execve events).

Adding a New Detector Copy Chapter

Go Detector Copy Section

1. Create a new file (e.g., my_detector.go)
2. Implement the detection.EventDetector interface
3. Add init() function calling register(&MyDetector{})
4. The detector will automatically be included when the package is imported

YAML Detector Copy Section

See the YAML Detectors documentation for instructions on creating and deploying YAML detectors.

Overview (`scripts/README.md`) Copy Section

Tracee Scripts

This directory contains utility scripts for developing, building, testing, and debugging Tracee.

Directory Structure Copy Chapter

`debugging/` Copy Section

Tools for debugging and analyzing Tracee components:

• inspect_bpf_prog.sh - Inspect and measure eBPF programs in BPF object files. Supports program size measurement, assembly dumping, and comparing between builds.

`installation/` Copy Section

Scripts for installing dependencies and preparing environments:

• install-deps-ubuntu.sh - Install build dependencies on Ubuntu
• install-deps-centos.sh - Install build dependencies on CentOS
• install-deps-alpine.sh - Install build dependencies on Alpine Linux
• install-clang.sh - Install specific LLVM/Clang versions
• prepare-ami.sh - Prepare AWS AMI for CI/CD
• pull-test-images.sh - Pull container images for testing

Core Scripts Copy Section

• btfhub.sh - Manage BTFhub integration for CO-RE support
• checkpatch.sh - Run Linux kernel style checks on patches
• system-info.sh - Display system information for debugging
• sync_system_time.sh - Synchronize system time (useful in VMs)
• tracee_start.sh / tracee_stop.sh - Start/stop Tracee with common configurations
• run_test_scripts.sh - Run integration test scripts
• verify_man_md_sync.sh - Verify man pages and markdown docs are in sync

Library Scripts Copy Section

Common utilities sourced by other scripts:

General Purpose Libraries:

• lib.sh - Main library that imports all other libs
• lib_log.sh - Logging functions (info, error, warn, etc.)
• lib_print.sh - Formatted output utilities
• lib_git.sh - Git operations
• lib_test.sh - Test framework utilities
• lib_testing.sh - Additional test helpers
• lib_misc.sh - Miscellaneous utilities
• lib_internal.sh - Internal helper functions

Tracee-Specific Library:

• tracee_common.sh - Common functions for tracee_start.sh and tracee_stop.sh

Usage Copy Chapter

Most scripts include a --help flag for detailed usage information:

./script-name.sh --help

Contributing Copy Chapter

When adding new scripts:

1. Follow the existing structure and naming conventions
2. Add a shebang line (#!/bin/bash) and set executable permissions
3. Source lib.sh for consistent logging and utilities
4. Include usage/help documentation
5. Update this README with a brief description
6. Consider adding the script to the appropriate subdirectory

For script style guidelines, see .cursor/skills/shell-style-guide/SKILL.md.

Overview (`cmd/tracee-bench/readme.md`) Copy Section

Documentation

tracee-bench is a promQL based tool to query tracee runtime performance metrics.
It can be used to benchmark tracee's event pipeline's (see here) performance on your environment.

Enabling Prometheus Copy Chapter

In order to use prometheus with tracee see this documentation.
A simple script for running a prometheus container scraping tracee is available in this repository in prometheus.sh.

Metrics tracked Copy Chapter

tracee-bench tracks three important stats about tracee's performance in your environment:1

1. Avg rate of events emitted per second
2. Avg rate of events lost per second
3. Overall events lost

Ideal performance of tracee should have a stable throughput of events emitted with minimal event loss. If heavy event loss occurs, consider tuning tracee either through filtering or allocating additional CPU (if running on kubernetes).

Overview (`deploy/grafana/README.md`) Copy Section

Using the Dashboard

A tutorial for importing this dashboard (and general usage of Tracee with Prometheus and Grafana) can be found here.

Overview (`tests/detectors/README.md`) Copy Section

Detector Testing Framework

A lightweight framework for testing detector integration with the Tracee engine.

When to Use Copy Chapter

For most detectors, direct testing is sufficient. Use TestHarness when you need to test:

• Event ID allocation and registration
• Policy filtering behavior
• Auto-population of Threat/DetectedFrom fields
• Detector chaining (output of detector A feeds detector B)

Quick Start Copy Chapter

TestHarness (Engine Integration) Copy Section

func TestMyDetector(t *testing.T) {
    // Create harness with base events your detector consumes
    harness := detectors.NewTestHarness(t, events.Execve)

    // Register your detector
    detector := &MyDetector{}
    require.NoError(t, harness.RegisterDetector(detector))

    // Create input event and dispatch
    input := &v1beta1.Event{
        Id:   v1beta1.EventId(events.Execve),
        Name: "execve",
        Data: []*v1beta1.EventValue{
            v1beta1.NewStringValue("pathname", "/bin/nc"),
        },
    }

    outputs := harness.DispatchEvent(input)

    // Verify outputs
    require.Len(t, outputs, 1)
    harness.AssertThreatPopulated(outputs[0])
    harness.AssertDetectedFromPopulated(outputs[0], "execve")
}

YAML Detector Testing Copy Section

func TestYAMLDetector(t *testing.T) {
    yaml := `
id: test-detector
produced_event:
  name: suspicious_exec
requirements:
  events:
    - name: sched_process_exec
      data_filters:
        - pathname=/bin/nc
output:
  fields:
    - name: binary_path
      expression: getEventData("pathname")
`
    harness := detectors.NewYAMLTestHarness(t, yaml, events.SchedProcessExec)

    input := &v1beta1.Event{
        Id:   v1beta1.EventId(events.SchedProcessExec),
        Name: "sched_process_exec",
        Data: []*v1beta1.EventValue{
            v1beta1.NewStringValue("pathname", "/bin/nc"),
        },
    }

    outputs := harness.DispatchEvent(input)
    require.Len(t, outputs, 1)
}

See Also Copy Chapter

• Direct detector tests: detectors/*_test.go - examples of testing detector logic directly
• Detector implementations: detectors/*.go - Go detector examples
• YAML detectors: pkg/detectors/yaml/*.yaml - YAML detector examples

Overview (`tests/integration/syscaller/README.md`) Copy Section

syscaller

A tiny, low-noise, self-isolating syscall-event trigger for the tracee integration tests.

Some events are hard to produce from a normal program — either the syscall is destructive
(reboot, delete_module), or the interesting event is derived from a syscall that must actually
succeed (openat → security_file_open). syscaller sets its own comm (so a policy can scope by
comm=) and then issues exactly the syscalls a test asks for, with per-syscall argument strategies.

Why it works Copy Chapter

A tracee syscall-name event fires at raw_tracepoint/sys_enter, on entry, before the syscall
runs and with no success check (pkg/ebpf/c/tracee.bpf.c). So calling a syscall triggers its
event even when the call then fails harmlessly on invalid arguments. That is what lets syscaller
trigger destructive syscall events safely: it calls them with poison arguments that fail at
validation (EINVAL/EFAULT/EBADF), so the event fires but the destructive effect never lands.

Why Zig Copy Chapter

The tool this replaced was written in Go, and the Go runtime is noisy: just starting it costs ~900
syscalls — a storm of clone/futex/mmap/rt_sig from the scheduler and GC — versus about 12 for
a single trigger here. Worse, PR_SET_NAME sets the comm per thread, so the runtime's extra
threads kept their own comm and leaked into any comm-scoped policy.

This tool sidesteps all of that. It targets the raw std.os.linux interface — no libc, no threads,
no runtime — so it sits at the syscall-noise floor, and a comm scope sees only the syscalls the test
asked for. As a bonus, one Zig toolchain cross-compiles amd64 and arm64, so the build no longer drags
in the CGO/eBPF environment the Go tool needed (it used to import pkg/events).

CLI Copy Chapter

syscaller [--unshare] [--cgroup] [--fork-each] <comm> <spec>...
  <spec>       = <n>            issue syscall n with ZERO args (fires the syscall-name event) [legacy]
               = <n>:<strategy> strategy-driven args; strategy = zero | invalid | file        [new]
  --unshare    run in throwaway user/mount/pid/net/uts/ipc/cgroup namespaces (host-safe)
  --fork-each  run EVERY spec in its own fork+watchdog child (contains side effects)

<n> is the arch-native syscall number. The Go test side resolves an event name to its number
via the build-tagged pkg/events/core_{amd64,arm64}.go tables (GetDefinitionByName().GetID()), so
the tool itself does no translation. The bare <n> form is unchanged from the old Go tool, so the
existing event_filters useSyscaller cases (formatCmdEvents) keep working with no change.

Strategies Copy Chapter

The scratch file used by file is created before the comm change, so its setup syscalls land
under the original comm, not the scoped one, and it is removed on exit.

Isolation (`--unshare`) Copy Chapter

--unshare runs the workload inside a throwaway set of namespaces — user, mount, pid, net, uts, ipc
and cgroup — created just for this run. It maps the caller to uid/gid 0 in the new user namespace
(with deny on setgroups), then forks so the child actually enters the new pid namespace.

The point: that mapped-root has no privilege over the host, so even a syscall whose arguments didn't
fully neuter it can't do damage. A stray mount, kill or network op is confined, and the uts/ipc/
cgroup namespaces also contain sethostname/setdomainname, IPC and cgroup ops (which matters once
the sweep triggers the whole table). It is defense-in-depth on top of the poison-argument safety —
reach for it whenever you trigger a destructive syscall with invalid.

tracee is a single host-wide, scope-filtered collector, so a self-unshared process is still
captured. Scope proof tests by comm / mntns / pidns — not container=new, which keys on a
runtime cgroup + enricher that a bare unshare does not have.

Execution model Copy Chapter

By default zero and file specs run inline while an invalid spec runs in a forked child with a
watchdog: the child issues the syscall and exits; the parent reaps it and SIGKILLs it if it does
not return within a short budget. This makes the tool robust to a syscall that (despite bad args)
blocks or ends the process — including exit-class syscalls, where only the child exits.

--fork-each extends the fork+watchdog to every spec. That is what lets the coverage sweep
trigger the whole syscall table safely: a syscall with a process-local side effect (seccomp strict
mode, close, umask, setsid, ptrace→TRACEME) is contained to a throwaway child and cannot
poison the specs that follow.

Safety taxonomy (which strategy per syscall) Copy Chapter

Almost every syscall event just needs zero (the default): it fires at sys_enter before the call
runs, and zero args also defuse most "dangerous" syscalls (a NULL pointer → EFAULT, a zero
count/flags → EINVAL, a zero fd is just stdin). The exceptions live in the Go-side
syscallerStrategy / syscallerSkip maps (syscaller_trigger_test.go), which are the curated,
extensible source of per-syscall knowledge:

• invalid — destructive syscalls (reboot, kexec_load/kexec_file_load,
  init_module/finit_module/delete_module, mount, pivot_root, swapon/swapoff,
  unlinkat), plus ones that would end or hang the tool if run inline: exit-class (exit,
  exit_group), ptrace (zero args = PTRACE_TRACEME, a real side effect), and pause (blocks
  forever — the watchdog kills it).
• file — openat (and x86_64-only open), to drive security_file_open.
• skip — cannot be triggered as a benign, capturable event: execve/execveat (replace the
  image), rt_sigreturn (only valid from a signal frame), and the tool's own fork primitives
  clone/clone3/fork/vfork. runSyscaller refuses these.

Unlisted syscalls already work via zero, so the tool needs no edit when tracee adds syscalls: the
Go side resolves any event name through tracee's own events.Core table.

Staying aligned with tracee Copy Chapter

The syscaller does not keep its own syscall list — coverage tracks tracee automatically. Two tests
keep it honest:

• Test_SyscallerTableDrift (no root): asserts every syscallerStrategy key is a real syscall
  event on this arch (so a tracee rename/removal or a typo fails fast), that syscallerSkip is a
  consistent deny-list, and that no name is in both. Cheap; runs in the normal suite.
• Test_SyscallerSweep (root; part of the integration suite): derives the corpus from
  events.Core (every syscall event minus the skip set) and triggers the whole table at once under
  --unshare --fork-each, asserting each is captured under the tool's comm. When tracee gains
  syscalls (e.g. a PR adding new syscall definitions) this covers them with no edit here;
  anything genuinely un-triggerable this way is added to syscallerSkip — the failure lists exactly
  what was missed.

Running the sweep Copy Section

The sweep is an ordinary integration test, so it runs automatically wherever the suite runs —
including PR CI (make test-integration → go test ./tests/integration/...). The cost is one extra
short-lived tracee: the whole syscall table is triggered in a single run. Like every integration test
it needs root and eBPF, so it skips on an unprivileged box.

To run just it locally:

sudo make test-integration TEST=Test_SyscallerSweep

When it fails it lists the syscall events it could not capture; add any that are genuinely
un-triggerable this way to syscallerSkip.

Build Copy Chapter

The Makefile builds it as part of the integration flow (it is a prerequisite of
test-integration):

make dist/syscaller

This requires Zig 0.16 (scripts/installation/install-zig.sh, pinned + checksum-verified; the
Makefile's .checkver_zig enforces the exact minor). For local development:

zig build            # -> zig-out/bin/syscaller
zig build test       # pure arg/strategy unit tests (no root, no syscalls issued)

Using it from Go tests Copy Chapter

• Legacy: formatCmdEvents (in event_filters_test.go) turns a <comm> + expected event names into
  dist/syscaller <comm> <n>... — unchanged.
• New: runSyscaller / syscallerStrategy (in syscaller_trigger_test.go) emit <n>:<strategy> and
  add --unshare/--fork-each. Test_SyscallerTrigger is the end-to-end proof (a destructive batch
  via --unshare+invalid; openat via file driving security_file_open); Test_SyscallerSweep
  is the opt-in full-table coverage sweep; Test_SyscallerTableDrift is the no-root table guard.

Source Copy Chapter

• src/main.zig — the whole tool.
• build.zig — zig build / zig build test (the Makefile builds src/main.zig directly).

Index (`docs/index.md`) Copy Section

hide:

• toc

👋 Welcome to Tracee Documentation! To help you get around, please notice the different sections at the top global menu:

• You are currently in the Getting Started section where you can find general information and help with first steps.
• In the Tutorials section you can find step-by-step guides that help you accomplish specific tasks.
• In the Docs section you can find the complete reference documentation for all of the different features and settings that Tracee has to offer.
• In the Contributing section you can find technical developer documentation and contribution guidelines.

Before moving on, please consider giving us a GitHub star ⭐️. Thank you!

About Tracee Copy Chapter

Tracee is a runtime security and observability tool that helps you understand how your system and applications behave using eBPF technology. It provides deep visibility into Linux systems by monitoring system calls, network activity, and file operations in real-time.

What Tracee Does Copy Section

• 🔍 System Monitoring: Tracks system calls, process execution, file operations, and network activity
• 🛡️ Security Detection: Identifies suspicious behavior patterns and potential security threats
• 📊 Observability: Provides detailed insights into application and system behavior
• 🚨 Real-time Alerts: Generates events for immediate threat detection and response

Key Features Copy Section

• Zero Code Changes: Monitor existing applications without modification
• Low Overhead: Minimal performance impact using efficient eBPF programs
• Container Aware: Native support for containerized environments and Kubernetes
• Flexible Policies: Customize what to monitor and how to respond to events
• Rich Event Data: Detailed context including process lineage, file paths, and network connections

Use Cases Copy Section

• Security Monitoring: Detect malware, privilege escalation, and suspicious activity
• Compliance: Monitor file access, data exfiltration, and system changes
• Troubleshooting: Debug application issues and system behavior
• Forensics: Investigate security incidents with detailed audit trails

Quickstart Copy Chapter

To quickly try Tracee use one of the following snippets. For a more complete installation guide, check out the Installation section.
Tracee should run on most common Linux distributions and kernels. For compatibility information see the Prerequisites page. Mac users, please read this FAQ.

Using Docker Copy Section

docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest

For a complete walkthrough please see the Docker getting started guide.

On Kubernetes Copy Section

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace

kubectl logs --follow --namespace tracee daemonset/tracee

For a complete walkthrough please see the Kubernetes getting started guide.

Next Steps Copy Chapter

After trying the quickstart, here's how to dive deeper:

🎯 For Security Analysts Copy Section

• Events Documentation: Learn about security events and signatures
• Policies Guide: Create custom detection rules
• Kubernetes Tutorial: Monitor containerized workloads

🛠️ For DevOps Engineers Copy Section

• Installation Guide: Production deployment options
• Configuration: Customize Tracee for your environment
• Outputs: Integrate with your monitoring stack

👨‍💻 For Developers Copy Section

• Custom Events: Create your own detection logic
• Contributing Guide: Help improve Tracee
• Building from Source: Development setup

Contributing Copy Chapter

Join the community, and talk to us about any matter in the GitHub Discussions or Slack.
If you run into any trouble using Tracee or you would like to give us user feedback, please create an issue.

Find more information on contribution documentation.

More about Aqua Security Copy Chapter

Tracee is an Aqua Security open source project.
Learn about our open source work and portfolio here.

Contributing Overview (`docs/contributing/overview.md`) Copy Section

Contributing

Thank you for taking interest in contributing to Tracee! This document covers our working practices and conventions.

Issues and Discussions Copy Chapter

We encourage open discussion and collaboration using both GitHub Issues and Discussions.

• Discussions are a free-style conversational tool.
• Issues are a project management tool, we use them to keep track of who's working on what and plan ahead.

If you have a suggestion, question, or a general comment - please use Discussions. If there's a clear work item (including bugs) - you can open an Issue.

Discussions Copy Chapter

• We have the following discussion topics:

  1. Announcements: One way communication from the team to the community. Consider this like our mini blog.
  2. Questions and Help: For help and support. Consider this similar to StackOverflow.
  3. Development: For discussing potential features, and collaborating on their design.

Issues Copy Chapter

1. Actionable and Assignable: Every issue must be clear, actionable, and assignable to a specific person. Break down large issues into smaller, more manageable tasks.

2. Clear and Definitive Descriptions: Be precise in your issue descriptions:

   • Bug Reports: Include the following:
     • Detailed steps to reproduce the bug.
     • The complete error message.
     • Your operating system, Tracee version, and any other relevant environment details.
   • Feature Requests: Define:
     • A clear scope for the feature.
     • Specific acceptance criteria that will be used to determine if the feature is complete.

3. Issue Labels:

   • area/... (e.g., ebpf, events): Area of Tracee affected.
   • kind/... (e.g., bug, feature, documentation): Type of issue or pull request.
   • priority/...: Urgency of the issue.
   • milestone/... (e.g., milestone/v0.x.y): Target release; add when you want the item considered for a specific version (issues and pull requests).
   • good-first-issue: Suitable for first-time contributors.

4. Issue Assignment: Self-assign issues or request assignment. Don't work on an issue assigned to someone else without their consent.

Submitting Changes Copy Chapter

External contributors must fork the repository and submit changes as pull
requests from their fork. Branch creation in this repository is restricted.

Pull Requests Copy Chapter

1. Every Pull Request should have an associated Issue unless it is a trivial fix.
2. When adding a flag option or other UX related change, make sure the design is explicitly described in the associated issue, and a maintainer approved it.
3. Pull requests require all review threads resolved and at least one approving
   review. When the change touches code-owner areas, at least two approving
   reviews are required.
4. Code related information should be in the commit message, review related information should be in the PR description.
5. For changes that span different areas please try to make each change self contained and independent.

Commits Copy Chapter

All commits must be signed (GPG, SSH, or S/MIME). See
GitHub's signing guide
to set up commit signing.

Commit messages must follow the
Conventional Commits
specification:

type(optional scope): description

optional body

Valid types: feat, fix, docs, style, refactor, perf, test,
build, ci, chore, revert.

The commit subject should be max 50 characters, written in imperative mood.
The optional body (separated by an empty line) may explain why the change
was made. Wrap at 72 characters.

Breaking Changes Copy Section

Breaking changes must include ! after the type/scope and a BREAKING CHANGE
footer in the commit body explaining what changed:

feat(ebpf)!: remove deprecated event field

BREAKING CHANGE: The `oldField` field has been removed from the
`anti_debugging` event. Use `newField` instead.

The BREAKING CHANGE footer is required so that the change is clearly
documented for downstream users and automated tooling.

Development and Testing Copy Chapter

Development Images Copy Section

For testing the latest changes without building from source, Tracee provides daily development images:

# Get the latest development build
docker pull aquasec/tracee:dev

# Test your changes quickly
docker run --rm -it --pid=host --privileged aquasec/tracee:dev --version

Development images are built daily from the main branch and include the latest features and fixes. See Building Documentation for complete details.

Development Workflows Copy Section

Quick Start for Development:

# 1. Check your environment
make env

# 2. Build everything
make all

# 3. Run tests
make test-unit

# 4. Check code quality before submitting PR
make check-pr

Common Development Tasks:

# Development builds
make tracee                # Build the main tracee binary
make bpf                   # Build just the eBPF object
make all                   # Build all components

# Testing
make test-unit             # Run unit tests with coverage
make test-integration      # Run integration tests

# Performance and stress testing
make evt                   # Build the event generator tool
make evt-trigger-runner    # Build stress testing container

For detailed information about code quality checks, dependencies, and troubleshooting, see our Code Quality Guide.

For performance testing and stress testing with the evt tool, see evt - Event Generator and Stress Testing Tool.

For complete build options and development environment setup, run make help or see Building Documentation.

Code Copy Chapter

1. Follow Golang's code review standards: https://github.com/golang/go/wiki/CodeReviewComments
2. Follow gofmt + govet + goimports formatting.
3. Tests should be included alongside code changes wherever applicable, except for parts that are harder to test and are not currently tested (e.g. eBPF). When modifying already tested code, your changes must be represented in the existing tests.

Contributing Code Copy Chapter

Adding New Event Tracing Copy Section

If you're looking to add new event tracing capabilities to Tracee, see our comprehensive guide: Adding New Event Tracing.

This guide covers:

• Event definition in Go code
• eBPF probe configuration
• Implementation patterns and best practices
• Testing and troubleshooting

Docs Overview (`docs/docs/overview.md`) Copy Section

Docs

In this section you can find the complete reference documentation for all of the different features and settings that Tracee has to offer.

Why Choose Tracee? Copy Chapter

Tracee stands out from other security and observability tools through several key differentiators:

🎯 Everything is an Event Copy Section

Unlike tools that separate raw data from detections, Tracee presents everything as events - from low-level system calls to high-level security detections. This unified approach lets you:

• Combine different event types in the same policy
• Build complex detection logic across multiple data sources
• Maintain a consistent view of your system's behavior

📊 Rich Event Coverage with Minimal Effort Copy Section

Tracee provides extensive built-in events out of the box:

• 400+ system calls for comprehensive system monitoring
• Network events including DNS, HTTP, and packet analysis
• Security events with pre-built threat detection signatures
• Container events with native Kubernetes integration

🛠️ Simplicity without Compromise Copy Section

Create powerful policies with just a few lines of YAML:

• Intuitive syntax for scoping and filtering
• Flexible targeting from global to container-specific
• Easy deployment across development and production environments

🔍 Forensic Capabilities Copy Section

Go beyond detection with artifact collection:

• Network traffic capture for detailed analysis
• Binary collection for malware investigation
• Memory dumps for advanced forensics
• File artifacts for compliance and auditing

🔗 Unified Architecture Copy Section

Everything works together seamlessly:

• Events flow through the same processing pipeline
• Policies can reference any combination of event types
• Custom signatures integrate naturally with built-in events
• Single configuration controls the entire system

🔒 Clear Security Model Copy Section

Tracee provides transparency about its capabilities and limitations:

• Strong protection against userspace adversaries through eBPF's tamper-resistant architecture
• Best-effort detection of kernel-level threats with rootkit detection signatures
• Clear boundaries documented to set proper expectations
• See the Security Model for detailed information about Tracee's adversary model and security guarantees

👈 Please use the side-navigation on the left in order to browse the different topics.

Traceectl Index (`docs/traceectl/index.md`) Copy Section

traceectl - Client Service for Tracee

traceectl is a command-line interface (CLI) tool designed as a client service for Tracee, Aqua Security's open-source runtime security solution. Tracee provides real-time, powerful observability for Linux environments by monitoring system calls, events, and more. traceectl is built to simplify interactions with Tracee, making it easier for users to manage, monitor, and gather security insights.

Overview Copy Chapter

traceectl acts as a controller for Tracee, allowing users to:

• Stream Events: Continuously stream security events from Tracee, with options to format the output as JSON, tables, or custom templates.
• List Available Events: Display the available events that Tracee can capture, providing essential insights into runtime activities.
• Query Metrics: Access various metrics related to Tracee, including event counts, errors, and more.

Installation and Usage Copy Chapter

To get started with traceectl, go over the Installation and Usage page

Tutorials Overview (`docs/tutorials/overview.md`) Copy Section

Tutorials

In this section you can find step-by-step guides that help you accomplish specific tasks.

👈 Please use the side-navigation on the left in order to browse the different topics.

If there are any specific tutorials that you would like to see please let us know on the Aqua Open Source Slack.

Docs Detectors Index (`docs/docs/detectors/index.md`) Copy Section

Detector Documentation

Detectors are the modern way to write custom threat detection and event derivation logic in Tracee.

What are Detectors? Copy Chapter

Detectors analyze runtime events to identify security threats and derive higher-level events from raw eBPF data.

Two ways to create detectors:

1. YAML Detectors: Declarative configuration - no coding required
2. Go Detectors: Full programmatic control with direct API access

Both provide:

• Declarative filtering: Engine-level event filtering by data, scope, and version
• Auto-enrichment: Automatic population of threat metadata and process ancestry
• Rich context: Access to process trees, containers, and more
• Built-in observability: Prometheus metrics and structured logging

Documentation Guide Copy Chapter

🚀 For Newcomers Copy Section

Start here if you're new to Tracee detectors:

[YAML Detectors Guide](yaml-detectors.md) Copy Section

Learn how to create detectors using declarative YAML configuration:

• Quick start with examples
• Complete schema reference
• Event filtering and data extraction
• Threat metadata and auto-population
• Deployment and best practices
• Troubleshooting guide

Start here if you want to create detectors without writing Go code.

Go Detectors Copy Section

Complete guide to writing Go detectors, from quick start to advanced features:

[Quick Start Guide](quickstart.md)

Get your first detector running in 30 minutes

Step-by-step tutorial to build a working detector:

• Create a detector file
• Build and run Tracee
• Trigger and observe detections
• Understand auto-registration, filtering, and enrichment

Perfect for: First-time detector developers, hands-on learners

📚 For Reference Copy Section

Comprehensive API documentation:

[Detector API Reference](api-reference.md)

Complete detector API specification

Everything you need to know about the detector API:

• Core interfaces and structures
• Event requirements and filtering
• Auto-population features
• Advanced features and lifecycle management
• Testing patterns and best practices
• Migration from old signatures
• Troubleshooting common issues

Perfect for: Understanding all capabilities, deep dives, migrating from signatures

[DataStore API Reference](datastore-api.md)

Complete datastore API specification

Query system state from your detectors:

• ProcessStore (process information and ancestry)
• ContainerStore (container and Kubernetes metadata)
• SystemStore, SyscallStore, KernelSymbolStore, DNSStore
• Health monitoring and error handling

Perfect for: Detectors that need to query system state and context

Documentation at a Glance Copy Chapter

Total reading time to first detector:

• YAML: ~15 minutes
• Go: ~30 minutes (Quick Start only)

Resources Copy Chapter

Examples Copy Section

• YAML Examples: examples/detectors/yaml/ directory - ready-to-use YAML detector examples
• Go Quick Start: Quick Start Guide - complete annotated walkthrough
• Go Comprehensive Example: detectors/example_detector.go - demonstrates all detector API features including DataStore usage, filtering patterns, and enrichment (build with make tracee-with-examples)
• Production Detectors: Browse detectors/ directory for real implementations
• Migration Examples: API Reference - signature → detector

API Definitions (Go Detectors) Copy Section

• Detector Interfaces: api/v1beta1/detection/detector.go
• DataStore Interfaces: api/v1beta1/datastores/interfaces.go
• Event Protobuf: api/v1beta1/event.proto

YAML Schema Copy Section

• Schema Reference: See YAML Detectors Guide for complete schema
• Examples: examples/detectors/yaml/ directory

Migration from Signatures Copy Section

Existing signatures can be migrated to the new detector API. See Migration Guide for:

• Side-by-side API comparison
• Step-by-step migration instructions
• Pattern translations and examples
• Complete migration checklist

Community and Support Copy Chapter

• Tracee GitHub: https://github.com/aquasecurity/tracee
• Issues: Report bugs and request features
• Discussions: Ask questions and share detectors
• Documentation: https://aquasecurity.github.io/tracee/

Ready to start?

• YAML detectors: Jump to the YAML Detectors Guide
• Go detectors: Jump to the Quick Start Guide or dive into the API Reference

Docs Detectors Quickstart (`docs/docs/detectors/quickstart.md`) Copy Section

Detector Quick Start Guide

Get your first Tracee detector running in 30 minutes. This hands-on tutorial assumes basic Go knowledge but no prior Tracee experience.

What You'll Build Copy Chapter

A detector that identifies when processes access sensitive system files like /etc/shadow or /etc/sudoers, with:

• Automatic threat metadata enrichment
• Full process ancestry (parent → grandparent → ...)
• Container context (if applicable)

Prerequisites Copy Chapter

# Clone Tracee
git clone https://github.com/aquasecurity/tracee
cd tracee

# Verify build environment
make env

# Required: Linux kernel 4.18+, Go 1.24+, Clang 12+

Step 1: Create Your Detector File Copy Chapter

Create detectors/sensitive_file_access.go:

{% raw %}

/* Detailed source-code truncated for AI context efficiency. */

{% endraw %}

Helper Functions: For cleaner code, use detection.Detected() for simple detections with no extra data, or detection.DetectedWithData(data) when including additional event values.

Step 2: Build Tracee Copy Chapter

# Build with your detector included
make tracee

# The detector is compiled in - no plugins or manual registration needed!

Step 3: Run Tracee Copy Chapter

# Start Tracee with process tree and JSON output
sudo ./dist/tracee --stores process --output json | jq

Step 4: Trigger Your Detector Copy Chapter

In a third terminal, access a sensitive file:

sudo cat /etc/shadow

Step 5: See the Detection Copy Chapter

You should see output like:

{
  "id": 5001,
  "name": "sensitive_file_access",
  "version": "1.0.0",
  "timestamp": "2025-12-16T10:30:45.123Z",
  "threat": {
    "name": "Sensitive File Access",
    "description": "A process attempted to access a sensitive system file",
    "severity": "MEDIUM"
  },
  "data": {
    "file_path": "/etc/shadow",
    "executable": "/bin/cat"
  },
  "detected_from": {
    "event_id": 257,
    "event_name": "security_file_open"
  },
  "workload": {
    "process": {
      "entity_id": 12345,
      "pid": 67890,
      "executable": {"path": "/bin/cat"},
      "ancestors": [
        {"entity_id": 12344, "pid": 67889, "executable": {"path": "/bin/bash"}},
        {"entity_id": 12343, "pid": 67888, "thread": {"name": "sshd"}},
        {"entity_id": 1, "pid": 1, "thread": {"name": "systemd"}}
      ]
    }
  }
}

What Just Happened? Copy Chapter

Let's break down the magic:

1. Auto-Registration Copy Section

{% raw %}

func init() {
	register(&SensitiveFileAccess{})
}

{% endraw %}

Your detector registered itself automatically. No manual list maintenance!

2. Engine Filtering Copy Section

{% raw %}

DataFilters: []string{
	"pathname=/etc/shadow",
	"pathname=/etc/sudoers",
},

{% endraw %}

Tracee's engine filtered millions of security_file_open events. Only those matching your paths reached OnEvent(). This happens in the engine - super efficient!

3. Type-Safe Extraction Copy Section

{% raw %}

pathname, found := v1beta1.GetData[string](event, "pathname")

{% endraw %}

Generic type parameter ensures compile-time type safety. No runtime casting errors!

4. Auto-Enrichment Copy Section

{% raw %}

AutoPopulate: detection.AutoPopulateFields{
	Threat:          true,
	DetectedFrom:    true,
	ProcessAncestry: true,
}

{% endraw %}

The engine automatically:

• ✅ Copied ThreatMetadata to output.Threat
• ✅ Set output.DetectedFrom pointing to the triggering event
• ✅ Queried the process tree and populated 5 ancestor levels
• ✅ Preserved timestamp, workload, and policies from input event

You just returned simple data fields - the engine did the heavy lifting!

Common Patterns Copy Chapter

Multiple Outputs Copy Section

A single input can produce multiple detections:

{% raw %}

func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	var outputs []detection.DetectorOutput
	
	// Check condition 1
	if isSuspicious {
		outputs = append(outputs, detection.DetectorOutput{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("reason", "suspicious pattern"),
			},
		})
	}
	
	// Check condition 2
	if isMalicious {
		outputs = append(outputs, detection.DetectorOutput{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("reason", "known malware signature"),
			},
		})
	}
	
	return outputs, nil
}

{% endraw %}

Custom Threat Severity Copy Section

Override the default threat metadata per detection:

{% raw %}

return []detection.DetectorOutput{{
	Data: []*v1beta1.EventValue{
		v1beta1.NewStringValue("file", pathname),
	},
	Threat: &v1beta1.Threat{
		Name:        "Critical System File Access",
		Severity:    v1beta1.Severity_CRITICAL,  // Override default MEDIUM
		Description: "Root accessed critical system file",
	},
}}, nil

{% endraw %}

Using DataStores Copy Section

Access system state like process information:

{% raw %}

func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	entityId := event.GetWorkload().GetProcess().GetEntityId()
	
	// Query process store for process details
	proc, err := d.dataStores.Processes().GetProcess(entityId)
	if errors.Is(err, datastores.ErrNotFound) {
		// Process not in tree yet
		return nil, nil
	}
	if err != nil {
		return nil, err
	}
	
	// Use process info in detection logic
	if proc.GetInterpreter().GetPath() == "/bin/bash" {
		return []detection.DetectorOutput{{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("shell", "bash"),
			},
		}}, nil
	}
	
	return nil, nil
}

{% endraw %}

Filtering by Container Copy Section

Only detect events from containers:

{% raw %}

Requirements: detection.DetectorRequirements{
	Events: []detection.EventRequirement{
		{
			Name: "security_file_open",
			ScopeFilters: []string{"container=started"},  // Containers only
		},
	},
}

{% endraw %}

Or only from host:

{% raw %}

ScopeFilters: []string{"not-container"},  // Host only

{% endraw %}

Accessing Container Information Copy Section

Detectors can access container information using the container enrichment requirement.

Container enrichment is required for both approaches below - it enables Tracee to query container runtimes and populate metadata.

Option 1: Use Pre-Populated Event Fields

{% raw %}

Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    Enrichments: []detection.EnrichmentRequirement{
        {
            Name:       detection.EnrichmentContainer,
            Dependency: detection.DependencyRequired,
        },
    },
}

func (d *MyDetector) OnEvent(event *v1beta1.Event) {
    // Container fields are pre-populated in the event
    containerName := event.GetWorkload().GetContainer().GetName()
    containerImage := event.GetWorkload().GetContainer().GetImage().GetName()

    if strings.Contains(containerImage, "malicious") {
        // detect...
    }
}

{% endraw %}

Pros:

• Simple, direct field access

Cons:

• Cannot handle missing/not-found containers gracefully

Option 2: Query Container Datastore

{% raw %}

Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    Enrichments: []detection.EnrichmentRequirement{
        {
            Name:       detection.EnrichmentContainer,
            Dependency: detection.DependencyRequired,
        },
    },
    DataStores: []detection.DataStoreRequirement{
        {
            Name:       detection.DataStoreContainer,
            Dependency: detection.DependencyRequired,
        },
    },
}

func (d *MyDetector) OnEvent(event *v1beta1.Event) {
    // Container ID is always available
    containerID := event.GetWorkload().GetContainer().GetId()
    if containerID == "" {
        return nil, nil  // Not a container event
    }

    // Explicitly query datastore for metadata
    containerInfo, err := d.dataStores.Containers().GetContainer(containerID)
    if errors.Is(err, datastores.ErrNotFound) {
        // Handle not-found case (e.g., short-lived container)
        return nil, nil
    }
    if err != nil {
        return nil, err
    }

    if strings.Contains(containerInfo.Image, "malicious") {
        // detect...
    }
}

{% endraw %}

Pros:

• Flexible error handling, access to additional datastore methods

Cons:

• Slightly more verbose

Both approaches require --enrichment container to be enabled. The enrichment flag tells Tracee to query container runtimes and populate both Event fields and the datastore.

User must run: tracee --enrichment container --detector my-detector

Debugging Tips Copy Chapter

Enable Debug Logging Copy Section

# Run with debug logs
sudo ./dist/tracee --logging debug --stores process

Check Detector Registration Copy Section

# List detector events
sudo ./dist/tracee list | grep detectors

Verify Event Filtering Copy Section

Check that your event name is correct:

# List available events
sudo ./dist/tracee list | grep security_file_open

Test Without Filters Copy Section

Temporarily remove DataFilters to see all events:

{% raw %}

Requirements: detection.DetectorRequirements{
	Events: []detection.EventRequirement{
		{
			Name: "security_file_open",
			// DataFilters: []string{...},  // Commented out for testing
		},
	},
}

{% endraw %}

Add logging in OnEvent():

{% raw %}

func (d *SensitiveFileAccess) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	d.logger.Debugw("Event received", "event", event.Name)
	// ... rest of logic
}

{% endraw %}

Next Steps Copy Chapter

Congratulations! You've written your first Tracee detector. Now explore:

📚 Deep Dive into APIs Copy Section

• API Reference: Complete detector API documentation
• DataStore API: Query process trees, containers, DNS, and more

🔄 Migration and Troubleshooting Copy Section

• Migration Guide: Migrating from old signatures
• Troubleshooting: Common issues and solutions

🔍 Study Real Examples Copy Section

• Browse detectors/ directory for production implementations
• See api/v1beta1/detection/detector.go for interface definitions

🧪 Write Tests Copy Section

• Add unit tests in detectors/sensitive_file_access_test.go
• See Testing Guide for patterns

Troubleshooting Copy Chapter

Detector Not Running Copy Section

Problem: Your detector code changed but behavior didn't

Solution: Rebuild Tracee - detectors are compiled in, not dynamically loaded

make tracee

No Events Received Copy Section

Problem: OnEvent() never called

Possible causes:

1. Wrong event name - check tracee list
2. Filters too restrictive - temporarily remove them
3. Event not enabled - Tracee enables events based on requirements automatically

Process Ancestry Empty Copy Section

Problem: workload.process.ancestors is null

Solution: Enable process tree:

sudo ./dist/tracee --stores process

Build Errors Copy Section

Problem: Compilation fails

Common fixes:

# Update dependencies
make go-tidy

# Clean build
make clean
make tracee

Quick Reference Copy Chapter

Detector Skeleton Copy Section

{% raw %}

package detectors

import (
	"context"

	"github.com/aquasecurity/tracee/api/v1beta1"
	"github.com/aquasecurity/tracee/api/v1beta1/datastores"
	"github.com/aquasecurity/tracee/api/v1beta1/detection"
)

func init() {
	register(&MyDetector{})
}

type MyDetector struct {
	logger     detection.Logger
	dataStores datastores.Registry
}

func (d *MyDetector) GetDefinition() detection.DetectorDefinition {
	return detection.DetectorDefinition{
		ID: "TRC-XXX",
		Requirements: detection.DetectorRequirements{
			Events: []detection.EventRequirement{{Name: "event_name"}},
		},
		ProducedEvent: v1beta1.EventDefinition{
			Name:    "my_detection",
			Version: &v1beta1.Version{Major: 1},
		},
		AutoPopulate: detection.AutoPopulateFields{
			Threat:          true,
			DetectedFrom:    true,
			ProcessAncestry: true,
		},
	}
}

func (d *MyDetector) Init(params detection.DetectorParams) error {
	d.logger = params.Logger
	d.dataStores = params.DataStores
	return nil
}

func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	// Your detection logic
	return []detection.DetectorOutput{{
		Data: []*v1beta1.EventValue{},
	}}, nil
}

{% endraw %}

Essential Commands Copy Section

# Build
make tracee

# Run with process tree
sudo ./dist/tracee --stores process

# View detections in JSON
sudo ./dist/tracee --output json | jq

# List detector events
sudo ./dist/tracee list | grep detectors

# Debug mode
sudo ./dist/tracee --logging debug

Next Steps Copy Chapter

Learn More Advanced Patterns Copy Section

For a comprehensive example demonstrating all detector API features, see detectors/example_detector.go:

• DataStore API usage (ContainerStore, SystemStore)
• Event filtering patterns (scope, data, version constraints)
• Conditional field population
• Advanced error handling
• Proper enrichment patterns

To build and test it:

# Build Tracee with example detectors
make tracee-with-examples

# Run tracee and filter for the example detection
sudo ./dist/tracee -e example_detection --output json

Note: The example detector is excluded from default builds (via build tag) to prevent noise in production environments.

Ready for more? Continue to the API Reference for comprehensive documentation, including migration guides and troubleshooting.

Docs Events Index (`docs/docs/events/index.md`) Copy Section

Events

Events are the core of how Tracee works. Whether you're monitoring system calls, network activity, or security threats, Tracee treats everything as events that you can filter, combine, and act upon in your policies.

Event Categories Copy Section

Tracee provides rich built-in events across six main categories:

• syscalls - System call monitoring
• network - Network activity and protocol analysis
• security - Security-focused detections and signatures
• lsm - Linux Security Module hooks
• containers - Container lifecycle and metadata
• misc - Additional system events and utilities

This section documents all of the different events that Tracee exposes.

Configuring Tracee Events Copy Chapter

Events are defined in the Policy YAML manifest.

Tracing the execve events in a policy:

apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-policy
  annotations:
    description: traces execve events
spec:
  scope:
    - global
  rules:
    - event: execve

If no event is passed with [filters] or [policies], tracee will start with a set of default events.

Please head over to the Tracee usage documentation for more information on configuring events.

Event Sets Copy Section

Event sets are predefined groups of related events. Instead of listing events individually, you can use sets like syscalls (all system calls), fs (file system operations), or net (network events).

Discover available sets:

tracee list --wide

Use in policies:

rules:
  - event: tag=syscalls     # All system calls
  - event: tag=fs           # File system events

Use on command line:

tracee --events syscalls,net

!!! tip "Tracing All System Calls"
Use --events syscalls to trace all system calls without listing them individually.

Related Topics Copy Chapter

• Policies Documentation - Learn how to create policies that use events
• Custom Signatures - Build your own detection logic using Go
• Troubleshooting - Solutions for event-related issues

Video Content Copy Chapter

If you are curious to learn more about the Tracee Events architecture and related decision making, then have a look at the following video Q&A:

Everything is an Event in Tracee

Docs Install Index (`docs/docs/install/index.md`) Copy Section

Installing Tracee

This guide walks you through installing and setting up Tracee in your environment.

📋 Before You Start Copy Chapter

System Requirements Copy Section

Before installing Tracee, ensure your system meets the necessary requirements:

• Prerequisites - Operating system, kernel version, BTF, and capability requirements
• Kernel Symbols - Kernel symbol table details and configuration
• OS Requirements - OS release file and kernel config requirements
• Capabilities - Running Tracee with proper Linux capabilities
• LSM BPF Support - Linux Security Module BPF support (optional)
• Mac Users FAQ - Running Tracee on macOS (spoiler: you'll need a Linux VM or container)

!!! tip "Quick Compatibility Check"
Tracee requires Linux kernel 5.4+ (or 4.18 for RHEL 8) with BTF support. Check /sys/kernel/btf/vmlinux exists and /proc/kallsyms is available.

🚀 Installation Methods Copy Chapter

Download Options Copy Section

Tracee is available through multiple distribution channels:

1. Binary releases: GitHub Releases
2. Container images: Docker Hub - aquasec/tracee
3. Helm charts: Aqua Security's Helm repository - https://aquasecurity.github.io/helm-charts/

Tracee may also be available in various community-managed package managers.

Quick Start Guides Copy Section

Choose your deployment method:

• Docker Installation - Run Tracee as a Docker container (fastest way to get started)
• Kubernetes Installation - Deploy Tracee on Kubernetes clusters with Helm

⚙️ Post-Installation Setup Copy Chapter

After installing Tracee, configure it for your environment:

Configuration Copy Section

• Configuration Guide - Complete configuration reference for CLI and Kubernetes deployments
  • Kubernetes Configuration - Configure Tracee in Kubernetes
  • Container Enrichment - Configure Tracee to detect and enrich events from Docker, Containerd, CRI-O, or Podman

Monitoring & Observability Copy Section

• Monitoring Configuration - Enable Prometheus metrics and health check endpoints for monitoring Tracee

🔍 Next Steps Copy Chapter

After installation:

1. Learn about policies: Read the Policies Guide to define what to monitor
2. Explore events: Check out Events Documentation to understand available events
3. Configure outputs: Set up Output formats for your monitoring stack

💡 Troubleshooting Copy Chapter

Having issues during installation or setup? Check our Troubleshooting Guide for common solutions.

For help and support, visit:

• GitHub Discussions
• Slack Community

Docs Outputs Index (`docs/docs/outputs/index.md`) Copy Section

Tracee Output

Tracee can generate thousands of events per day depending on your system and policies. The output system provides flexible ways to format, filter, route, and store these events for analysis and integration with your monitoring stack.

Overview Copy Chapter

The output system consists of several components that work together to control how Tracee events are handled:

• Formats: Choose how events are serialized (JSON, table, custom templates)
• Destinations: Route events to files, webhooks, or forward to log aggregators
• Streams: Create filtered event pipelines with independent routing
• Options: Enrich events with additional context (stack traces, environment variables, etc.)
• Logging: Configure Tracee's diagnostic logs separate from event output

Configuration Copy Chapter

Output is configured using the --output flag or the output: section in your configuration file. For complete details, see the output flag reference.

Basic CLI example:

tracee --output destinations.stdout_json.format=json

Configuration file example:

output:
  destinations:
    - name: json_file
      type: file
      format: json
      path: /var/log/tracee/events.json
  enrichment:
    decoded-data: true

Output Components Copy Chapter

[Destinations](../flags/output.1.md) Copy Section

Route events to different outputs. Tracee supports three destination types:

• File: Write to files or stdout/stderr
• Webhook: Send events to HTTP endpoints
• Forward: Stream to FluentBit/Fluentd receivers

Each destination is configured with a type, format, and path or URL.

[Formats](./output-formats.md) Copy Section

Control how events are serialized and presented:

• JSON: Machine-readable format for log aggregation and SIEM integration
• Table: Human-readable terminal output for debugging and development
• Go Templates: Custom formatting using Go template syntax

[Streams](./streams.md) Copy Section

Create multiple output pipelines with independent filtering and routing. Streams allow you to:

• Route different events to different destinations
• Apply policy-based filtering
• Configure separate buffering strategies

[Event Structure](./event-structure.md) Copy Section

Understand the structure and fields available in Tracee events for parsing and analysis.

[Sorting Events](./sorting-events.md) Copy Section

Learn about event ordering guarantees and how to enable chronological sorting when needed.

[Logging](./logging.md) Copy Section

Configure Tracee's diagnostic logs (separate from event output) for troubleshooting and monitoring Tracee itself.

Quick Start Examples Copy Chapter

JSON output to file:

tracee --output destinations.file_out.format=json --output destinations.file_out.path=/var/log/tracee.json

Default table output with parsed arguments:

tracee --enrichment decoded-data

Send events to webhook:

tracee --output destinations.webhook1.type=webhook --output destinations.webhook1.url=http://my-webhook:8080

Multiple destinations with streams:

output:
  destinations:
    - name: all_events
      type: file
      format: json
      path: /var/log/tracee/all.json
    - name: security_events
      type: webhook
      url: https://siem.example.com/events
  streams:
    - name: security_stream
      destinations:
        - security_events
      filters:
        events:
          - security_file_open
          - security_socket_connect

Additional Resources Copy Chapter

• Output flag reference: Complete CLI flag documentation
• Example configurations: Sample configuration files
• Configuration guide: General configuration documentation

Docs Policies Index (`docs/docs/policies/index.md`) Copy Section

Policies

Policies allow users to specify which events to trace in which workloads. The policy scope defines which workloads this policy is limited to. The policy can define multiple rules that specify the events to trace. Policies are used both for the Tracee CLI and for the Tracee Kubernetes installation. This makes it easier to share policies across use cases and environments.

It is possible to load up to 64 policies into Tracee.

Policy Formats Copy Chapter

Tracee supports two policy formats: Kubernetes CRD format and Plain format (YAML or JSON). Both formats are fully interchangeable and produce identical results. Tracee automatically detects the format when loading policies.

Kubernetes CRD Format Copy Section

The Kubernetes CRD format follows the standard Kubernetes Custom Resource Definition structure. Use this format for Kubernetes deployments or when managing policies as Kubernetes resources.

Example:

apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: overview-policy
  annotations:
    description: sample overview policy
spec:
  scope:
    - global
  rules:
    - event: dropped_executable
    - event: security_file_open
      filters:
        - data.pathname=/tmp/*

Plain Format Copy Section

The plain format provides a simpler, more concise syntax. Use this format for local development, testing, or non-Kubernetes environments.

Example:

type: policy
name: overview-policy
description: sample overview policy
scope:
  - global
rules:
  - event: dropped_executable
  - event: security_file_open
    filters:
      - data.pathname=/tmp/*

Both formats support the same functionality. Tracee automatically detects the format by checking for type: policy (plain format) or apiVersion and kind fields (K8s CRD format). You can mix both formats in the same directory - Tracee will detect and load them correctly.

!!! Note
Each event type can only be defined once in a policy

There are many ways to fine tune the scope and filters. For further information on the details, have a look at the respective sections:

• Specify the Policy scope
• Filter events in the rules section

While specifying event filters is optional, policies must have the name, description, scope and rules fields.

Related Topics Copy Chapter

• Events Documentation - Learn about available events to use in policies
• Policy Usage Guide - How to use policies with Tracee CLI
• Kubernetes Policies - How to deploy policies in Kubernetes
• Troubleshooting - Common policy-related issues and solutions

Docs Events Custom Overview (`docs/docs/events/custom/overview.md`) Copy Section

Custom Events

Tracee comes with many built-in events, but you can extend its capabilities by creating custom events tailored to your specific needs.

Modern Approach: Detectors Copy Chapter

Recommended: The modern way to create custom threat detections and derived events is using the EventDetector API.

📖 See the Detector Documentation for complete guide and examples.

Key benefits:

• Type-safe protobuf access
• Rich data extraction helpers
• System state access (process trees, containers, DNS)
• Declarative filtering and auto-enrichment
• Built-in metrics and observability
• No plugin complexity

Legacy Approach: Signatures (Plugin System) Copy Chapter

The older signature system using .so plugins is still supported for backward compatibility, but we recommend migrating to detectors.

Refer to the Go documentation for instructions on the legacy plugin-based approach.

Loading Signatures Copy Section

Once you've created a signature plugin, load it using the signatures-dir flag:

tracee --signatures-dir=/tmp/myevents

!!! Tip
Tracee also uses the custom events to add a few events, if you pass your own directory
for signatures-dir you will not load the tracee signatures,
to avoid such problems, you can either place your own events under the same directory of the tracee custom events,
or pass multiple directories for example:
bash tracee --signatures-dir=/tmp/myevents --signatures-dir=./dist/signatures

Migrating from Signatures to Detectors Copy Section

The Detector API Reference includes complete migration instructions with:

• Step-by-step migration guide
• Before/after code examples
• Pattern translations
• Migration checklist

Choose Your Approach Copy Chapter

Recommendation: Use detectors for all new development. Migrate existing signatures over time.

👈 Please use the side-navigation on the left in order to browse the different topics.

Docs Install Config Index (`docs/docs/install/config/index.md`) Copy Section

Configuring Tracee

Tracee has many different options and settings that control how Tracee operates.

Using Configuration Files Copy Chapter

The --config flag allows you to specify global configuration options for Tracee by providing a configuration file in YAML or JSON format.

The configuration file can include settings for output formats, enrichment, buffers, logging, server options, and more.

To use the --config flag, provide the path to the configuration file:

tracee --config /path/to/tracee-config.yaml

A complete example config file with all available options can be found here. Most of the options are documented in different sections in the documentation.

!!! note
The YAML examples below can be translated to JSON as well. For Kubernetes-specific configuration, see the Kubernetes Configuration guide.

!!! warning
The following options are not supported in the configuration file and must be provided exclusively via the CLI: --config, --policy, --scope, --events. Policies can also be supplied through Kubernetes CRDs when running in a cluster (see Policy Configuration).

Common configurations Copy Chapter

Output Copy Section

• --output (-o): Controls how and where the output is printed.

  NOTE: You can view more in the output section.

  YAML:

  yaml Copy

  output:
    - json

Server Copy Section

• --server: Sets options for the HTTP and/or gRPC servers.

  CLI Examples:

  bash Copy

  # Address configuration
  tracee --server http-address=:3366
  tracee --server grpc-address=unix:/var/run/tracee.sock
  
  # HTTP features (boolean flags)
  tracee --server metrics --server healthz --server pprof

  YAML:

  yaml Copy

  server:
    http-address: ":3366"
    grpc-address: "unix:/var/run/tracee.sock"
    metrics: true
    healthz: true
    pprof: true
    pyroscope: true

Stores (Process Tree and DNS Cache) Copy Section

• --stores: Controls process tree and DNS cache options.

  NOTE: You can view more in the Stores flag documentation for CLI options. For information about accessing stores in detectors, see the DataStore API documentation.

  YAML:

  yaml Copy

  stores:
    process:
      enabled: true
      source: both
    dns:
      enabled: true

Log Copy Section

• --logging (-l): Controls Tracee logging options, like verbosity,
  filters, destination file and others.

  NOTE: You can view more in the Tracee Logs section.

  YAML:

  yaml Copy

  log:
    - level: debug

Containers Copy Section

• To enable container enrichment, include the flag: --enrichment container. To disable it, simply omit the flag. Note: Setting any container sub-option (e.g., container.docker.socket=/path) automatically enables container, so --enrichment container is not needed.

  YAML:

  yaml Copy

  enrichment:
    container:
      enabled: true

  NOTE: You can view more in the enrichment section.

• --enrichment: Configures enrichment options including container enrichment and runtime sockets. For example, to configure runtime sockets:

  YAML:

  yaml Copy

  enrichment:
    container:
      enabled: true
      cgroupfs:
        path: /host/sys/fs/cgroup
        force: false
      docker-socket: /var/run/docker.sock
      containerd-socket: /var/run/containerd/containerd.sock
      crio-socket: /var/run/crio/crio.sock
      podman-socket: /var/run/podman/podman.sock
    fd-paths: true
    executable-hash:
      enabled: true
      mode: sha256
    user-stack: true

Capabilities Copy Section

• --capabilities (-C): Define specific capabilities for Tracee to run with. This allows you to either bypass, add, or drop certain capabilities based on your security and operational needs.

  NOTE: You can view more in the Tracee Capabilities section

  YAML:

  yaml Copy

  capabilities:
    - add: 
        - CAP_SYS_ADMIN
    - drop: 
        - CAP_NET_RAW

  NOTE: Capabilities are Linux-specific permissions that control which privileged operations a program can perform.

Buffer and Cache Copy Section

• --buffers: Configures the buffer sizes for kernel and user buffers.

  Buffer sizes for perf ring buffers (kernel.events, kernel.artifacts, kernel.control-plane) are specified in pages. The pipeline buffer size is specified in event objects. The default size for perf ring buffers is 1024 pages. The default size for the pipeline buffer is 1000 event objects.

  YAML:

  yaml Copy

  buffers:
      kernel:
          events: 2048
          artifacts: 1024
          control-plane: 512
      pipeline: 20000

  CLI:

  bash Copy

  --buffers kernel.events=2048 --buffers kernel.artifacts=1024 --buffers kernel.control-plane=512 --buffers pipeline=20000

  NOTE: You can view more in the buffers section.

See Also Copy Chapter

• Kubernetes Configuration - Kubernetes-specific configuration
• Container Enrichment - Configure container runtime integration
• Monitoring - Configure Prometheus metrics and health checks
• Flag Reference - Complete configuration file reference
• Output Configuration - Configure event output and formatting
• Policy Configuration - Define what to monitor

Contributing Adding Events (`docs/contributing/adding-events.md`) Copy Section

Adding New Events

This guide walks you through the process of adding new event monitoring capabilities to Tracee. Adding new events involves modifications to several files across the codebase.

Overview Copy Chapter

Adding a new event to Tracee requires changes in several main areas:

1. eBPF Implementation - Implement the actual event handling logic
2. Probe Configuration - Configure how the event attaches to kernel functions
3. Event Definition - Define the event and its metadata in Go code
4. Protobuf Schema - Add event to protobuf definitions for gRPC API
5. gRPC Translation - Map event to protobuf enums for external APIs
6. Event Documentation - Create documentation file for the event

Step-by-Step Process Copy Chapter

1. Implement eBPF Program in `pkg/ebpf/c/tracee.bpf.c` Copy Section

Choose a kernel function to monitor:

• System calls: sys_openat, sys_execve
• LSM hooks: security_file_open, security_bprm_check
• VFS operations: vfs_read, vfs_write
• Network: tcp_connect, inet_csk_accept

Find functions: cat /proc/kallsyms | grep function_name

Implement the eBPF program:

SEC("kprobe/your_kernel_function")          // Must match probe configuration
int BPF_KPROBE(trace_your_event)            // This name will be used in probe config
{
    // Your event implementation follows this pattern:
    // 1. Initialize program data
    // 2. Apply scope filtering
    // 3. Extract data fields
    // 4. Apply data field-based filtering
    // 5. Submit event

    program_data_t p = {};
    if (!init_program_data(&p, ctx, YOUR_EVENT_ID))
        return 0;

    if (!evaluate_scope_filters(&p))
        return 0;

    // Extract data fields from kernel function
    void *arg1 = (void *)PT_REGS_PARM1(ctx);
    int arg2 = PT_REGS_PARM2(ctx);

    // Apply data field-based filtering if needed
    // ... your filtering logic ...

    // Save data fields to event buffer
    save_str_to_buf(&p.event->args_buf, arg1, 0);
    save_to_submit_buf(&p.event->args_buf, &arg2, sizeof(arg2), 1);

    events_perf_submit(&p, 0);
    return 0;
}

2. Configure Probes in `pkg/ebpf/probes/probe_group.go` Copy Section

Add the probe configuration that matches your eBPF function names:

// In pkg/ebpf/probes/probe_group.go, add to allProbes map in NewDefaultProbeGroup():
YourNewEvent: NewTraceProbe(KProbe, "your_kernel_function", "trace_your_event"),

Attachment types:

• KProbe - Attaches to kernel function entry (most common)
• KretProbe - Attaches to kernel function exit (for return values)
• Tracepoint - Attaches to predefined kernel tracepoints (stable interface)
• SyscallEnter/SyscallExit - Specialized for system call entry/exit

3. Define the Event in `pkg/events/core.go` Copy Section

Choose an event ID from the appropriate range and add it to both eBPF and Go code:

Event ID Ranges:

• Common events: 1-699 (most events)
• Network events: 700-1999
• User-space network: 2000-3999
• Capture meta-events: 4000-4999
• Signal meta-events: 5000+

In eBPF (pkg/ebpf/c/tracee.bpf.c):

#define YOUR_EVENT_ID 42

In Go (pkg/events/core.go):
{% raw %}

// Add to appropriate const block (around line 100-200):
YourNewEvent = 42    // Must match eBPF ID

// Add to Core variable (around line 500+):
YourNewEvent: {
    id:      YourNewEvent,
    id32Bit: Sys32Undefined,
    name:    "your_event_name",
    version: NewVersion(1, 0, 0),
    sets:    []string{"fs"}, // Choose: syscalls, fs, net, security, proc, default
    fields: []DataField{
        {DecodeAs: data.STR_T, ArgMeta: trace.ArgMeta{Type: "const char*", Name: "pathname"}},
        {DecodeAs: data.INT_T, ArgMeta: trace.ArgMeta{Type: "int", Name: "flags"}},
        // Add all event data fields
    },
    dependencies: Dependencies{
        probes: []Probe{{handle: probes.YourNewEvent, required: true}},
    },
},

{% endraw %}

4. Add Event to Protobuf Schema Copy Section

Add event ID to protobuf enum in api/v1beta1/event.proto:

enum EventId {
    // ... existing events ...
    your_new_event = 42;  // Must match the ID from Go/eBPF
    // ... more events ...
}

Important notes:

• Event IDs in protobuf must match exactly with Go constants and eBPF defines
• Follow the existing naming convention: lowercase with underscores
• Add the event in the appropriate section (syscalls, common events, etc.)

5. Add Event to gRPC Translation Table Copy Section

Update the translation table in pkg/server/grpc/tracee.go:

// Add to EventTranslationTable map in appropriate section:
events.YourNewEvent: pb.EventId_your_new_event,

This table maps internal Go event IDs to external protobuf event IDs for gRPC communication.

6. Generate Protobuf Files Copy Section

After modifying any .proto files, regenerate the Go code:

# Generate protobuf Go files from proto definitions
make -f builder/Makefile.protoc

This command uses protoc to generate:

• Go structs from protobuf messages
• gRPC service definitions
• JSON marshaling/unmarshaling code

Verify the generated files:

• api/v1beta1/*.pb.go - Generated protobuf structs
• api/v1beta1/*_grpc.pb.go - Generated gRPC service code

7. Handle Complex Data Types (If Needed) Copy Section

If your event uses complex data types, you may need to add marshalling logic in pkg/server/grpc/event_data.go:

// Add case in parseArgument function for custom types
case *YourCustomType:
    return &pb.EventValue{
        Name: arg.Name,
        Value: &pb.EventValue_YourCustomField{
            YourCustomField: convertYourCustomType(v),
        },
    }, nil

For new data types:

1. Add the type definition to api/v1beta1/event_data.proto
2. Add the field to the EventValue oneof
3. Implement conversion logic in event_data.go
4. Run make -f builder/Makefile.protoc to regenerate files

Testing Your New Event Copy Chapter

8. Build and Test Compilation Copy Section

# Build Tracee with your changes
make tracee

# Verify no compilation errors
echo $?  # Should be 0

9. Test Event Functionality Copy Section

# Test that your event can be selected
sudo ./dist/tracee --events your_event_name --output json

# Test with policies
sudo ./dist/tracee --config-file your_test_policy.yaml

10. Unit Tests Copy Section

Add unit tests for your event definition and run them:

# Run unit tests
make test-unit

# Run integration tests
make test-integration

// In appropriate _test.go file
func TestYourNewEvent(t *testing.T) {
    // Test event definition
    // Test data field extraction
    // Test filtering behavior
}

11. Integration Tests Copy Section

Consider adding integration tests that actually trigger your event and verify it's captured correctly.

12. Create Event Documentation Copy Section

After everything is working, create proper documentation for your event to help users understand its purpose, usage, and configuration.

For comprehensive documentation standards and templates, see the Event Documentation Guide.

The documentation guide covers all formatting requirements, accuracy standards, and provides a complete template to ensure your documentation matches Tracee's quality standards.

Common Patterns Copy Chapter

• System calls: Hook sys_* functions, use SyscallEnter/SyscallExit probes
• Security events: Hook security_* functions, use KProbe attachments
• File operations: Hook VFS functions (e.g., vfs_read, vfs_write)
• Network events: Hook network functions, extract connection info

Best Practices Copy Chapter

• Keep eBPF programs lightweight - apply filtering early, minimize processing
• Use descriptive names - follow existing naming conventions
• Document data fields clearly - explain purpose and format of each field
• Choose appropriate event sets - helps users discover and select your event
• Test thoroughly - verify event triggers correctly and data is accurate

Troubleshooting Copy Chapter

Common Issues Copy Section

Compilation Errors:

• Verify event IDs don't conflict
• Check eBPF program section names match probe configuration
• Ensure all required headers are included
• Run make -f builder/Makefile.protoc after modifying proto files

Protobuf/gRPC Issues:

• Event ID mismatches between Go, eBPF, and protobuf
• Missing entry in EventTranslationTable
• Protobuf files not regenerated after schema changes
• Complex data types not handled in event_data.go

Runtime Issues:

• Check kernel compatibility for attachment points
• Verify probe symbols exist on target kernel
• Test with bpftrace first for complex kernel function monitoring

Event Not Triggering:

• Verify the kernel function is actually called for your test case
• Check filtering logic in eBPF program
• Use bpf_printk() for debugging eBPF code

Debugging Tools Copy Section

# Check eBPF program loading
bpftool prog list

# Monitor eBPF logs
cat /sys/kernel/debug/tracing/trace_pipe

# Test kernel symbol availability
cat /proc/kallsyms | grep your_symbol

Getting Help Copy Chapter

• GitHub Issues: Ask questions about event implementation
• Discussions: Discuss design decisions for complex events
• Code Review: Submit WIP PRs for feedback on approach

Related Resources Copy Chapter

• eBPF Programming Guide
• Linux Tracepoints
• BPF LSM Documentation
• Tracee Documentation

Contributing Checkpatch (`docs/contributing/checkpatch.md`) Copy Section

Code Quality Guide

This guide covers all code quality tools and checks for Tracee development. The checkpatch script (scripts/checkpatch.sh) powers the make check-pr command and provides comprehensive PR verification for all developers.

Purpose Copy Chapter

This script now powers the official make check-pr command and includes all necessary checks for PR submission:

• Documentation verification: Man page synchronization checks
• Code analysis: Comprehensive formatting, linting, and static analysis
• Unit tests: Go and script unit tests
• PR formatting: Commit message formatting for PR descriptions
• Enhanced UX: Better error messages, tool installation guidance, and flexible options

Usage Copy Chapter

Quick Start Copy Section

# Official way - use make targets (recommended)
make check-pr                   # Full comprehensive checks
make check-pr-fast              # Quick checks (skip static analysis + unit tests)
make check-pr-skip-docs         # Skip documentation verification
make check-pr-skip-tests        # Skip unit tests

# Direct script usage for advanced options
./scripts/checkpatch.sh                   # Check HEAD
./scripts/checkpatch.sh abc123def         # Check specific commit
./scripts/checkpatch.sh HEAD~1            # Check previous commit
./scripts/checkpatch.sh --skip-unit-tests # Skip time-consuming tests
./scripts/checkpatch.sh --help            # Show all options

All Available Commands Copy Section

Primary Make Targets (Recommended):

make check-pr                   # Full comprehensive checks (recommended for PR submission)
make check-pr-fast              # Quick checks (skip static analysis + unit tests)
make check-pr-skip-docs         # Skip documentation verification
make check-pr-skip-tests        # Skip unit tests
make format-pr                  # Show what formatting changes are needed
make fix-fmt                   # Automatically fix code formatting issues

Individual Check Commands:

make -f builder/Makefile.checkers fmt-check     # Check code formatting only
make -f builder/Makefile.checkers fmt-fix       # Fix code formatting automatically
make -f builder/Makefile.checkers code-check    # Static code analysis only

Direct Script Usage (Advanced):

./scripts/checkpatch.sh [OPTIONS] [commit-ref]  # Direct script access
make check-pr ARGS="--fast --skip-docs HEAD~1"  # Pass arguments through make

Test Categories Copy Chapter

1. Documentation Verification Copy Section

• Ensures .1.md and .1 man page files are synchronized
• Runs scripts/verify_man_md_sync.sh
• Compares changes against origin/main

2. Code Analysis Copy Section

• Go Formatting: Checks gofmt compliance
• Linting: Runs revive linter via make check-lint
• Code Style: Validates formatting via make check-fmt
• Go Vet: Static analysis via make check-vet
• StaticCheck: Advanced static analysis via make check-staticcheck
• Error Check: Unhandled error detection via make check-err
• Vulnerability Check: Known vulnerability detection via make check-vulncheck

3. Unit Tests Copy Section

• Go Unit Tests: Runs make test-unit
• Script Unit Tests: Runs make run-scripts-test-unit

4. PR Formatting Copy Section

• Commit Display: Shows commits in PR-ready format
• Description Generation: Extracts commit bodies for PR descriptions

Dependencies Copy Chapter

Required Copy Section

• go (Go compiler and tools)
• make (Build system)
• git (Version control)

Optional Tools Copy Section

The script will warn if these tools are missing but continue with available checks:

• revive (v1.7.0): go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d
• staticcheck (2025.1): go install honnef.co/go/tools/cmd/staticcheck@5af2e5fc3b08ba46027eb48ebddeba34dc0bd02c
• errcheck (v1.9.0): go install github.com/kisielk/errcheck@11c27a7ce69d583465d80d808817d22d6653ee34
• govulncheck (v1.1.4): go install golang.org/x/vuln/cmd/govulncheck@d1f380186385b4f64e00313f31743df8e4b89a77

Installation of Optional Tools Copy Chapter

To install all optional tools at once:

# Install linting and static analysis tools
go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d                   # v1.7.0
go install honnef.co/go/tools/cmd/staticcheck@5af2e5fc3b08ba46027eb48ebddeba34dc0bd02c          # 2025.1
go install github.com/kisielk/errcheck@11c27a7ce69d583465d80d808817d22d6653ee34                 # v1.9.0
go install golang.org/x/vuln/cmd/govulncheck@d1f380186385b4f64e00313f31743df8e4b89a77           # v1.1.4

# Ensure tools are in PATH
export PATH=$PATH:$(go env GOPATH)/bin

Exit Codes Copy Chapter

• 0: All tests passed successfully
• 1: One or more tests failed or an error occurred

Integration with Development Workflow Copy Chapter

Pre-commit Hook Copy Section

You can integrate this script into your git pre-commit hooks:

#!/bin/sh
# .git/hooks/pre-commit
make check-pr-fast || exit 1

IDE Integration Copy Section

Many IDEs can be configured to run external scripts. Configure your IDE to run make check-pr-fast or ./scripts/checkpatch.sh as a build or test step.

Recommended Development Workflow Copy Section

1. Make your changes
2. Quick validation: Run make check-pr-fast for immediate feedback
3. Fix any issues: Use make fix-fmt for formatting or address other issues manually
4. Full validation: Run make check-pr before PR submission
5. Commit and push with confidence

Integration Options:

• Pre-commit hook: Add make check-pr-fast || exit 1 to .git/hooks/pre-commit
• IDE integration: Configure your IDE to run make check-pr-fast as a build step

Troubleshooting Copy Chapter

Common Issues Copy Section

Missing Tools Warning

⚠ revive not found. Install with: go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d # v1.7.0

Solution: Install the missing tool or ignore if you don't need that specific check.

Documentation Mismatch

✗ Documentation verification failed
- tracee.1.md change requires corresponding tracee.1 change

Solution: Run make -f builder/Makefile.man to regenerate man pages.

Formatting Issues

✗ Go formatting issues found:

Solution: Run make fix-fmt or make -f builder/Makefile.checkers fmt-fix to fix formatting automatically.

Protocol Buffer Issues

✗ Protobuf files need regeneration

Solution: Run make -f builder/Makefile.protoc to regenerate Go code from .proto files.

Performance Tips Copy Section

• The script fetches origin/main with --depth=1 for efficiency
• Dependencies are checked once at the beginning
• Tests run in the order of fastest to slowest (docs → code analysis → unit tests)

Extending the Script Copy Chapter

To add additional test categories:

1. Create a new test function following the pattern of existing functions
2. Add it to the main execution in the main() function
3. Update the help text and documentation

Comparison with CI Copy Chapter

This script covers the basic PR tests but does not include:

• Integration tests (require elevated privileges)
• Performance tests (resource intensive)
• Multi-kernel testing (requires special infrastructure)
• Build verification for other tools

For the complete test suite, the CI pipeline in GitHub Actions is still required.

Contributing Event Documentation Guide (`docs/contributing/event-documentation-guide.md`) Copy Section

Event Documentation Guide

This guide provides the standard format and requirements for documenting Tracee events in the official documentation. Following these standards ensures consistency and quality across all event documentation.

Note: This guide complements the Adding New Events guide. Complete the technical implementation first, then use this guide for creating proper documentation.

Why Documentation Standards Matter Copy Chapter

Consistent event documentation helps users:

• Understand event purpose and when to use specific events
• Configure events correctly with accurate field names and types
• Troubleshoot issues with clear dependency information
• Discover related events for comprehensive monitoring

Required Sections Copy Chapter

Every event documentation MUST include these sections in this exact order:

1. **YAML Frontmatter** Copy Section

---
title: TRACEE-EVENT-NAME
section: 1
header: Tracee Event Manual
---

2. **NAME** Copy Section

• Format: **event_name** - brief one-line description
• Keep description concise and accurate

3. **DESCRIPTION** Copy Section

• First paragraph: When the event is triggered and what it captures
• Second paragraph: Additional context about the event's purpose and importance
• Keep descriptions focused on what/when/how rather than why (save for USE CASES)
• Avoid excessive technical jargon

4. **EVENT SETS** Copy Section

• List the event sets this event belongs to (from pkg/events/core.go)
• Use none if the event doesn't belong to any sets
• Example: **proc**, **proc_life** or **none**

5. **DATA FIELDS** Copy Section

• List ALL data fields exactly as defined in pkg/events/core.go
• Format: **field_name** (*type*)
• Follow with : Description of the field
• Types must match exactly (e.g., int32, uint64, string, time.Time, SockAddr)
• Field names must match exactly from the source code
• If no fields: "This event currently does not capture specific arguments, but provides timing and context information about when [description]."

6. **DEPENDENCIES** Copy Section

• CRITICAL: Extract probe information from pkg/ebpf/probes/probe_group.go
• Use kernel function names/tracepoint names, NOT internal Tracee probe handles
• Format headers:
  • **Kernel Probe:** (singular) for kprobes
  • **Kernel Tracepoint:** (singular) for raw tracepoints
• Format entries: - kernel_function_name (required): Description for single probes
• Combine kprobe+kretprobe: When both entry and return probes exist for the same function, combine as:
  • - function_name (kprobe + kretprobe, required): Description
• For combined probes, include the probe types to clarify the instrumentation method
• For multiple dependencies, list each type separately
• Include tail calls if present: **Tail Calls:** - call_name: Description

7. **USE CASES** Copy Section

• Provide 3-5 specific, actionable use cases
• Format: - **Category**: Specific use case description
• Focus on practical applications for users
• Be specific and practical rather than generic

8. **RELATED EVENTS** Copy Section

• List 3-6 related events that users might find relevant
• Format: - **event_name**: Brief description
• Include complementary events, lifecycle events, and related security events

Signature Events Additional Requirements Copy Chapter

Signature events require these additional sections:

**SIGNATURE METADATA** (after DESCRIPTION, before EVENT SETS) Copy Section

- **ID**: TRC-XXX (unique signature identifier)
- **Version**: 1 (signature version number)
- **Severity**: X (threat level number + description)
- **Category**: category-name (e.g., defense-evasion, privilege-escalation)
- **Technique**: Descriptive technique name
- **MITRE ATT&CK**: TXXXX (MITRE ATT&CK technique ID)
- **Tags**: comma-separated tags (optional, e.g., linux, container)

**DETECTION LOGIC** (after USE CASES, before RELATED EVENTS) Copy Section

• Explain how the signature detection works
• List what the signature monitors for (numbered list preferred)
• Include detection criteria and logic flow
• Format example:

The signature monitors for:

1. **Event type** - specific condition or behavior
2. **Analysis step** - how events are processed
3. **Context evaluation** - reducing false positives

Event Type Identification Copy Chapter

Before documenting, determine if the event is a regular event or a signature event:

• Regular events: Core system events (syscalls, kernel events, LSM hooks, network events)
• Signature events: Security detection rules (found in docs/events/builtin/man/security/)

Signature events require additional sections (SIGNATURE METADATA, DETECTION LOGIC) as detailed above.

Data Accuracy Requirements Copy Chapter

Event Definition Source Copy Section

• PRIMARY SOURCE: pkg/events/core.go

• Find the event struct by searching for the event name

• Extract the following with 100% accuracy:

  • sets: []string{} → EVENT SETS section
  • fields: []DataField{} → DATA FIELDS section (names and types)
  • dependencies.probes → DEPENDENCIES section

Probe Definition Source Copy Section

• PRIMARY SOURCE: pkg/ebpf/probes/probe_group.go

• Map internal probe handles to actual kernel attachment points

• Examples:

  • SysEnter: NewTraceProbe(RawTracepoint, "raw_syscalls:sys_enter", ...) → raw_syscalls:sys_enter
  • DoExit: NewTraceProbe(KProbe, "do_exit", ...) → do_exit
  • SecurityFileOpen: NewTraceProbe(KProbe, "security_file_open", ...) → security_file_open

Probe Type Mapping Copy Section

• KProbe → Use **Kernel Probe:** header
• RawTracepoint → Use **Kernel Tracepoint:** header
• Multiple types → Use separate headers for each type

Quality Improvements Copy Chapter

Avoid Redundancy Copy Section

• Problem: Having "This event is useful for:" bullets in DESCRIPTION AND a separate USE CASES section creates redundancy
• Solution: Keep DESCRIPTION focused on what/when/how the event works, save why/for what explanations for USE CASES section

Simplify Dependencies Copy Section

• Problem: Separate entries for kprobe + kretprobe on same function is verbose

• Solution: Combine as function_name (kprobe + kretprobe, required): Description

• Example: Instead of:

  text Copy

  - vfs_write (required): VFS write function entry
  - vfs_write (required): VFS write function return

  Use: - vfs_write (kprobe + kretprobe, required): VFS write function

Content Guidelines Copy Chapter

Writing Style Copy Section

• Use clear, technical language appropriate for system administrators and security professionals
• Avoid marketing language or excessive enthusiasm
• Keep descriptions factual and implementation-focused
• Use present tense for event triggers ("Triggered when...")
• Keep documentation concise - aim for 50-70 lines total per event

Description Structure Copy Section

1. Trigger condition: "Triggered when..."
2. Context and purpose: What the event captures and why it's important

Use Case Categories Copy Section

Common categories to consider:

• Security monitoring/analysis
• Performance analysis
• Debugging/troubleshooting
• Compliance auditing
• Process/network/file monitoring
• Threat detection/hunting
• Forensic analysis

Formatting Requirements Copy Chapter

Consistency Rules Copy Section

• Use **bold** for field names, event names, and section emphasis
• Use *italics* for data types
• Use - for bullet points (not * )
• Use : (colon + space) to separate field names from descriptions
• Use singular headers for dependencies ("Kernel Probe:" not "Kernel Probes:")

File Naming and Location Copy Section

• Filename: event_name.md (use underscores, lowercase)

• Location based on event category:

  • System events: docs/docs/events/builtin/man/misc/
  • LSM events: docs/docs/events/builtin/man/lsm/
  • Network events: docs/docs/events/builtin/man/net/
  • Container events: docs/docs/events/builtin/man/containers/

Verification Checklist Copy Chapter

Before submitting event documentation:

• All data field names match exactly from pkg/events/core.go
• All data field types match exactly from pkg/events/core.go
• Event sets match exactly from pkg/events/core.go
• Dependencies use kernel function/tracepoint names from pkg/ebpf/probes/probe_group.go
• Probe types correctly mapped to section headers
• No internal probe handles used in dependencies
• All required sections present in correct order
• No prohibited verbose sections included
• Writing style is clear and technical
• Use cases are specific and actionable
• Related events are relevant and helpful

Example Templates Copy Chapter

Regular Event Template Copy Section

Template Start - Copy everything between the horizontal lines for regular events

---
title: TRACEE-EVENT-NAME
section: 1
header: Tracee Event Manual
---

# NAME

**event_name** - brief description

# DESCRIPTION

Triggered when [condition]. This event [captures/provides/monitors] [what].

[Additional context about purpose and implementation].

# EVENT SETS

**set1**, **set2** (or **none**)

# DATA FIELDS

**field_name** (*type*)
: Description of the field

**another_field** (*type*)
: Description of the field

# DEPENDENCIES

**Kernel Probe:**

- kernel_function_name (required): Description

**Kernel Tracepoint:**

- tracepoint:name (required): Description

# USE CASES

- **Security monitoring**: Specific use case

- **Performance analysis**: Specific use case

- **Debugging**: Specific use case

- **Compliance**: Specific use case

- **Monitoring**: Specific use case

# RELATED EVENTS

- **related_event_1**: Brief description
- **related_event_2**: Brief description
- **related_event_3**: Brief description

Template End - Copy everything above this line

Signature Event Template Copy Section

Template Start - Copy everything between the horizontal lines for signature events

---
title: TRACEE-SIGNATURE-NAME
section: 1
header: Tracee Event Manual
---

# NAME

**signature_name** - brief description of security detection

# DESCRIPTION

Triggered when [security condition]. This security signature [detects/identifies/monitors] [threat/behavior].

[Additional context about threat landscape and detection rationale].

# SIGNATURE METADATA

- **ID**: TRC-XXX
- **Version**: 1
- **Severity**: X (threat level description)
- **Category**: category-name
- **Technique**: Descriptive technique name
- **MITRE ATT&CK**: TXXXX
- **Tags**: tag1, tag2 (optional)

# EVENT SETS

**signatures**, **category_set**

# DATA FIELDS

**field_name** (*type*)
: Description of the field

**detection_context** (*object*)
: Context information about the detection

# DEPENDENCIES

**System Events:**

- base_event (required): System event being analyzed for detection

# USE CASES

- **Threat detection**: Specific threat hunting scenario
- **Security monitoring**: Specific security use case
- **Incident response**: Specific response scenario

# DETECTION LOGIC

The signature monitors for:

1. **Base event** - specific condition or behavior
2. **Analysis criteria** - how events are analyzed
3. **Context evaluation** - reducing false positives

# RELATED EVENTS

- **related_signature**: Brief description
- **base_event**: System event being monitored
- **related_detection**: Similar detection mechanism

Template End - Copy everything above this line

Documentation Workflow Copy Chapter

Complete Process Copy Section

1. Implement the event following the Adding New Events guide
2. Create markdown documentation following this guide's standards
3. Verify accuracy by cross-referencing with pkg/events/core.go
4. Generate man pages using the specialized build system
5. Add to navigation by updating mkdocs.yml
6. Test locally with MkDocs to ensure proper rendering

Generating Man Pages Copy Section

After creating or modifying event documentation, you must generate the corresponding man pages:

# From the Tracee root directory
make -f builder/Makefile.man

This command:

• Builds a Docker container with pandoc and required tools
• Converts markdown files to proper man page format
• Updates man pages in the appropriate directories

Critical: Always run this step after creating or modifying event documentation to keep man pages synchronized with markdown files.

This guide ensures all Tracee event documentation maintains consistency, accuracy, and usability for users and contributors.

Contributing Evt Tool (`docs/contributing/evt-tool.md`) Copy Section

evt - Event Generator and Stress Testing Tool

evt is a testing tool designed to generate events and stress test Tracee's event processing capabilities. It provides two main functionalities: triggering individual events and running comprehensive stress tests.

Overview Copy Chapter

The evt tool helps with:

• Event Generation: Trigger specific Linux events for testing Tracee's detection capabilities
• Performance Testing: Stress test Tracee with high volumes of concurrent events
• Development: Validate event detection and measure Tracee's performance under load

Building Copy Chapter

Build the evt binary:

make evt

Build the trigger runner container for stress testing:

make evt-trigger-runner

Or with a custom image name:

EVT_TRIGGER_RUNNER_IMAGE=my-runner:dev make evt-trigger-runner

Commands Copy Chapter

evt trigger Copy Section

Trigger a specific event type to test Tracee's detection.

Usage:

evt trigger --event <event_name> [flags]

Examples:

# Trigger a single event
evt trigger --event security_file_open

# Trigger multiple operations
evt trigger --event ptrace --ops 100

# Parallel execution
evt trigger --event security_file_open --parallel 4 --ops 1000

# Show Tracee bypass flags for an event
evt trigger --event security_file_open --bypass-flags

Flags:

• --event, -e <name>: Event type to trigger (required)
• --ops, -o <number>: Number of operations to perform
• --parallel, -p <number>: Number of parallel workers (total ops = workers × ops)
• --sleep, -s <duration>: Sleep duration between operations
• --bypass-flags, -b: Print Tracee bypass flags for the event
• --wait-signal, -w: Wait for SIGUSR1 before starting
• --signal-timeout <duration>: Timeout for waiting for signal

evt stress Copy Section

Run comprehensive stress tests with multiple event types in isolated containers.

Usage:

evt stress --events <event_specs> [flags]

Event Specification Format:

event[:instances=N:ops=N:sleep=dur]

• event: Event name to trigger
• instances: Number of parallel workers
• ops: Operations per worker
• sleep: Sleep between operations

Examples:

# Basic stress test with default settings
evt stress --events security_file_open --events ptrace

# Multiple events with custom configurations
evt stress \
  --events security_file_open:instances=10:ops=1000:sleep=1ms \
  --events ptrace:instances=2:ops=100:sleep=100ms

# Performance testing with profiling
evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --pprof --metrics

# Keep Tracee running after test for analysis
evt stress --events ptrace --keep-tracee --pyroscope

# Manual Tracee control
evt stress --events security_file_open --auto-tracee=false

# Custom cooldowns for timing control
evt stress \
  --events security_file_open:instances=10:ops=1000 \
  --tracee-init-cooldown 10s \
  --stress-end-cooldown 30s

Flags:

Event Configuration:

• --events, -e <spec>: Events to stress test (required, repeatable)
  • Format: event[:instances=N:ops=N:sleep=dur]
  • Example: security_file_open:instances=10:ops=1000:sleep=1ms

Container Configuration:

• --image <name:tag>: Trigger runner container image

Tracee Management:

• --auto-tracee: Automatically manage Tracee lifecycle (start and stop)
• --keep-tracee: Keep Tracee running after test (requires --auto-tracee=true, useful for profiling analysis)
• --tracee-binary : Path to Tracee binary (used when --auto-tracee=true)
• --tracee-output <format:path>: Tracee output format and path

Profiling:

• --metrics: Enable Tracee metrics endpoint
• --pprof: Enable Tracee pprof profiling endpoint
• --pyroscope: Enable Tracee pyroscope continuous profiling

Execution Control:

• --wait-before-trigger: Wait for user input before triggering events (useful to start profiling/scraping tools)
• --signal-timeout <duration>: Timeout for containers waiting for signal
• --tracee-init-cooldown <duration>: Cooldown after Tracee starts for stabilization before triggering events
• --stress-end-cooldown <duration>: Cooldown after stress completes for stabilization before cleanup
• --dry-run: Show what would be executed without running

Stress Testing Architecture Copy Chapter

The stress testing system uses a containerized architecture to run triggers in isolation:

1. Trigger Runner Container: Contains the evt binary and trigger scripts
2. Detached Execution: Containers run in the background with full privileges
3. Coordinated Start: All containers wait for SIGUSR1 before triggering events
4. Automatic Cleanup: Containers are removed after completion or interruption

Workflow Copy Section

1. Phase 1: Validate prerequisites (Docker, container image, Tracee binary)
2. Phase 2: Start trigger containers (detached mode, waiting for signal)
3. Phase 3: Start Tracee with container scope filters (optional)
4. Phase 4: Signal containers to start triggering events simultaneously
5. Phase 5: Monitor containers until completion
6. Cleanup: Stop Tracee (if auto-managed) and remove containers

Tracee Lifecycle Management Copy Section

Auto-managed (default):

evt stress --events security_file_open
# Tracee is started and stopped automatically

Keep Tracee running:

evt stress --events ptrace --keep-tracee --pyroscope
# Tracee stays running for profiling analysis
# You stop it manually later

Manual control:

# Terminal 1: Start Tracee yourself
sudo ./dist/tracee --events security_file_open --server pyroscope

# Terminal 2: Run stress test
evt stress --events security_file_open --auto-tracee=false

Container Image Copy Chapter

The evt-trigger-runner container image contains:

• evt binary for executing triggers
• Trigger scripts for various event types
• Runtime dependencies (bash, strace, bpftrace, etc.)

Build the image:

make evt-trigger-runner

Build with custom name:

EVT_TRIGGER_RUNNER_IMAGE=my-runner:test make evt-trigger-runner

Use custom image:

evt stress --image my-runner:test --events security_file_open

Available Triggers Copy Chapter

The following event triggers are available (located in cmd/evt/cmd/trigger/triggers/):

• security_file_open - File open operations
• ptrace - Process tracing events
• security_bpf_prog - BPF program operations
• security_socket_connect - Socket connection events
• security_socket_bind - Socket bind operations
• security_socket_create - Socket creation events
• sched_process_exec - Process execution events
• sched_process_fork - Process fork events
• sched_process_exit - Process exit events
• And many more...

See cmd/evt/cmd/trigger/triggers/ for the complete list.

Performance Testing Best Practices Copy Chapter

1. Disable Output for Maximum Performance Copy Section

For optimal performance during stress testing, avoid writing event logs:

# Maximum performance (no logging)
evt stress --events security_file_open:instances=20:ops=10000

# Enable logging if needed
evt stress --events ptrace --tracee-output json:/tmp/events.json

2. Use Profiling for Analysis Copy Section

Enable profiling endpoints to analyze Tracee's performance:

evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --pprof --metrics \
  --keep-tracee

Then access:

• Pyroscope UI: http://localhost:4040/?query=tracee.cpu
• Pprof: http://localhost:3366/debug/pprof
• Metrics: http://localhost:3366/metrics

3. Control Timing with Cooldowns Copy Section

Use cooldown periods to ensure accurate measurements:

# Longer initialization cooldown for complex setups
evt stress \
  --events security_file_open:instances=20:ops=10000 \
  --tracee-init-cooldown 10s

# End cooldown to let Tracee finish processing before cleanup
evt stress \
  --events ptrace:instances=10:ops=5000 \
  --stress-end-cooldown 30s \
  --keep-tracee

When to adjust cooldowns:

• Init cooldown: Provides time for Tracee to stabilize after starting (e.g., loading signatures, initializing eBPF). Increase for complex configurations.
• End cooldown: Provides time for system stabilization after stress completes. Increase for complex workloads or when collecting final metrics/profiles.

4. Scale Gradually Copy Section

Start with low load and increase gradually:

# Low load
evt stress --events security_file_open:instances=1:ops=100

# Medium load
evt stress --events security_file_open:instances=10:ops=1000

# High load
evt stress --events security_file_open:instances=20:ops=10000

5. Multiple Event Types Copy Section

Test with multiple event types for realistic scenarios:

evt stress \
  --events security_file_open:instances=10:ops=5000 \
  --events ptrace:instances=5:ops=1000 \
  --events security_socket_connect:instances=8:ops=2000

Troubleshooting Copy Chapter

Container Image Not Found Copy Section

Error: container image "evt-trigger-runner:latest" not found

Solution:

make evt-trigger-runner

Tracee Binary Not Found Copy Section

Error: tracee binary not found at "./dist/tracee"

Solution:

make tracee

Docker Not Available Copy Section

Error: docker not available

Solution: Ensure Docker is installed and running.

Monitoring Interrupted Copy Section

When you press Ctrl+C during monitoring:

Monitoring interrupted (containers will be cleaned up)

This is expected behavior. Containers are automatically cleaned up.

Integration with Performance Dashboard Copy Chapter

The stress testing tool integrates with Tracee's performance dashboard:

# Start performance dashboard
make -f builder/Makefile.performance dashboard-start

# Run stress test with profiling
evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --metrics \
  --keep-tracee

# Analyze results in dashboard
# Grafana: http://localhost:3000
# Pyroscope: http://localhost:4040

# Stop dashboard when done
make -f builder/Makefile.performance dashboard-stop

See Also Copy Chapter

• Performance Considerations - Tracee profiling and benchmarking
• Testing Coverage - Tracee testing framework
• Building - Building Tracee from source

Contributing Guidelines (`docs/contributing/guidelines.md`) Copy Section

Style Guide for Contributors

For those willing to contribute to Tracee, this repository has code formatting
guidelines being enforced.

Contributing to Documentation Copy Chapter

Our documentation aims to follow the Diátaxis documentation framework.

To contribute to the documentation:

1. Clone the Tracee GitHub repository.
2. Make changes in the /docs directory in the root folder.
3. Remember to edit the index in mkdocs.yml if you add or move existing files or directories.
4. Test your changes to the documentation.

Test Documentation Changes Copy Section

You can test your changes to the documentation by building and running a docker container.
Prerequisites: Docker installed and running locally.
Build the container image:

make -f ./builder/Makefile.mkdocs mkdocs-build

Serve the container image:

make -f ./builder/Makefile.mkdocs mkdocs-serve

Open localhost:8000/tracee

Now you should see the documentation.
Check that everything is as you intended.

And finally Submit a PR about the changes.

Contributing to Tracee Code Copy Chapter

We welcome contributions to Tracee's codebase! Before submitting your changes, please familiarize yourself with these guidelines.
To contribute to the code:

Before You Commit Copy Section

Before submitting code changes, ensure you follow these essential steps:

1. Generated Files: If you've modified certain types of files, regenerate the corresponding outputs:

   • Man pages: Run make -f builder/Makefile.man if you changed core code or documentation
   • Protocol buffers: Run make -f builder/Makefile.protoc if you modified .proto files

2. Code Quality: Run comprehensive code quality checks:

   bash Copy

   make check-pr

   This verifies formatting, runs linting, performs static analysis, and validates your changes meet project standards.

3. Development Environment: For consistent results, use a supported development environment when running checks.

For detailed information about these tools, troubleshooting, and advanced options, see our comprehensive Code Quality Guide.

Performance Considerations Copy Section

Performance is a critical aspect of Tracee.

To ensure your contributions maintain optimal performance, follow the guidelines in Performance Considerations page.

Kubernetes Considerations Copy Section

If your contribution impacts Tracee's behavior within a Kubernetes cluster, follow the guidelines in Kubernetes Considerations.

Contributing Kubernetes (`docs/contributing/kubernetes.md`) Copy Section

Kubernetes Considerations for Tracee Contributors

1. Set up a Local Kubernetes Cluster

   We recommend using a local Kubernetes cluster for development and testing. Popular options include:

   • kind: Kubernetes IN Docker
   • minikube: Runs a single-node cluster in a VM
   • MicroK8s: Lightweight, snap-based Kubernetes

   Tracee's Makefile provides convenient targets for setting up MicroK8s:

   bash Copy

   make -f builder/Makefile.k8s help

   text Copy

   To build the operator docker container:
   
       $ make -f builder/Makefile.k8s build
   
   To generate the kubernetes manifests:
   
       $ make -f builder/Makefile.k8s manifests
   
   To generate operator code:
   
       $ make -f builder/Makefile.k8s generate
   
   Or simply:
   
       $ make -f builder/Makefile.k8s

2. Deploy Tracee

   Deploy Tracee to your chosen local cluster. The deployment method will vary depending on your needs:

   • DaemonSet: For system-wide tracing, deploy Tracee as a DaemonSet.
   • Tracee Operator: Use the Tracee operator for simplified management.
   • Other Methods: Explore alternative methods like sidecar containers based on your use case.

3. Test Your Changes

   Test your modifications with realistic scenarios within the Kubernetes environment. Consider these examples:

   • Application Monitoring: Deploy sample applications and observe Tracee's event capture for expected behavior.
   • Network Policies: Configure network policies to restrict pod communication and verify Tracee captures relevant network events.
   • Resource Constraints: Apply resource limits to pods and ensure Tracee functions correctly under constrained conditions.
   • Security Policies: Implement Pod Security Policies and/or Security Contexts to validate Tracee's compliance and event capture in secure environments.

4. Monitor Tracee's Performance

   Utilize Kubernetes' observability tools to monitor Tracee:

   • Resource Usage: Track pod resource consumption (CPU, memory) using kubectl top or monitoring dashboards.
   • Logs: Analyze Tracee logs for errors, warnings, or unexpected behavior using kubectl logs.
   • Kubernetes Metrics: If available, leverage Kubernetes metrics to gain insights into Tracee's performance.

5. Clean Up

   After testing, remove deployed Tracee components and test resources to maintain a clean cluster environment.

Kubernetes-Specific Features Copy Chapter

If your contribution involves Kubernetes-specific features within Tracee (e.g., capturing Kubernetes events or metadata), ensure you test these functionalities thoroughly within the cluster environment. Pay close attention to event accuracy and any integration with Kubernetes APIs.

Contributing Performance (`docs/contributing/performance.md`) Copy Section

Performance Considerations

1. Profiling Tracee for Performance test - Tracee integrates with Pyroscope and Pprof for continuous profiling. When running Tracee locally for development or testing, use the --server pyroscope --server pprof command-line options.

   bash Copy

   sudo ./dist/tracee --server pyroscope --server pprof

   This enables profiling data to be sent to a local server. The Tracee repository includes a convenient way to deploy a performance dashboard for analyzing this data. Run the following for more details:

   bash Copy

   make -f builder/Makefile.performance help

2. Performance Dashboard: The provided performance dashboard allows visualization of host metrics, CPU flame graphs, and other performance-related data. Follow these steps to deploy locally and see instructions on using the dashboard:

   bash Copy

   make -f builder/Makefile.performance dashboard-start

   bash Copy

   make -f builder/Makefile.performance dashboard-stop

3. Benchmarking and Stress Testing: Before submitting significant code changes, use the evt tool to run stress tests and assess their impact on Tracee's performance:

   bash Copy

   # Build evt and trigger runner container
   make evt evt-trigger-runner
   
   # Run stress test with profiling
   evt stress \
     --events security_file_open:instances=16:ops=100000 \
     --pyroscope --pprof --metrics \
     --keep-tracee

   For comprehensive documentation on the evt tool, see evt - Event Generator and Stress Testing Tool.

4. Common Performance Pitfalls:

   • Excessive eBPF Events: Be mindful of the number and frequency of eBPF events being generated. Overly frequent events can lead to performance overhead. (Provide Tracee-specific examples or best practices to avoid this, such as filtering events effectively or using appropriate sampling rates.)
   • Inefficient eBPF Programs: Optimize your eBPF programs for minimal overhead. (Provide Tracee-specific guidance. Are there common patterns to avoid within Tracee's eBPF context?)
   • Resource Consumption: Consider the CPU and memory usage of Tracee itself. Avoid unnecessary allocations or computations.

By adhering to these practices, you can contribute to Tracee's performance and help ensure its efficiency. Remember that performance is an ongoing effort, so continuous profiling, benchmarking, and optimization are essential.

Contributing Setup Development Machine With Vagrant (`docs/contributing/setup-development-machine-with-vagrant.md`) Copy Section

Setup Development Machine with Vagrant

HashiCorp Vagrant leverages a declarative configuration file, which describes
all software requirements, packages, operating system configuration, and users
to provide the same development environment for everyone.

The Vagrantfile describes a cross-platform QEMU-based virtual machine optimized for building Tracee from the Getting Started guides. It supports both Linux and macOS hosts with hardware acceleration for optimal performance.

This allows developers involved in the project to check out the code, run vagrant up, and be on their way with a fully configured cloud-image/ubuntu-24.04 development environment.

Features Copy Chapter

• Cross-platform support: Linux (KVM) and macOS (HVF) with automatic hardware acceleration detection
• Optimized shared folders: Platform-specific 9p filesystem implementation for best performance
• Flexible configuration: Easily customize CPU, memory, and acceleration settings
• Automatic error handling: Comprehensive validation and user-friendly error messages

Prerequisites Copy Chapter

• HashiCorp Vagrant
• QEMU with hardware acceleration support:
  • Linux: KVM acceleration (recommended for optimal performance)
  • macOS: HVF (Hypervisor Framework) acceleration on Intel/Apple Silicon
  • Fallback: TCG software emulation (slower but works on all systems)
• vagrant-qemu plugin: Install with vagrant plugin install vagrant-qemu

Clone the Tracee Repository Copy Chapter

Clone the Tracee repository to your local machine. This repository contains the Vagrantfile.

git clone https://github.com/aquasecurity/tracee.git

Navigate to the Tracee Directory Copy Chapter

Open a terminal and navigate to the directory containing the Vagrantfile within the cloned Tracee repository (tracee/)

cd tracee

Configure VM Type (Optional) Copy Chapter

The VM can be provisioned for either a dev or test environment. The dev environment includes additional tools like MicroK8s, kubectl, and Helm.

• Development Environment: Full development environment (Default)

  bash Copy

  export VM_TYPE=dev

• Testing Environment: Smaller vagrant machine without k8s cumbersome to avoid conflicts with specific tests.

  bash Copy

  export VM_TYPE=test

Important: There are two ways to use a custom VM_TYPE:

Option 1: Export the environment variable (recommended)

# Export once, then use normal vagrant commands
export VM_TYPE=test
vagrant up
vagrant ssh
vagrant halt
vagrant destroy

Option 2: Prefix each command (if not exported)

# Include VM_TYPE in every command if not exported
VM_TYPE=test vagrant up
VM_TYPE=test vagrant ssh
VM_TYPE=test vagrant halt
VM_TYPE=test vagrant destroy

Without setting VM_TYPE, vagrant defaults to the dev environment. If you use the prefix method (Option 2) and forget to include VM_TYPE in a command, vagrant may operate on the wrong VM or give "VM not found" errors.

💡 Tip: Use export VM_TYPE=test at the beginning of your session to avoid repeating it in every command.

Configure VM Settings (Optional) Copy Chapter

Customize the VM's configuration by setting the following environment variables:

Resource Allocation Copy Section

• VM_CPUS: Number of virtual processors. Defaults to half of the host's processors. Example:

  bash Copy

  export VM_CPUS=4

• VM_MEM: Memory in gigabytes. Defaults to 8GB. Example:

  bash Copy

  export VM_MEM=16

Hardware Acceleration Copy Section

• VM_ACCEL: Force specific acceleration type. Options vary by host OS:

  Linux:

  bash Copy

  export VM_ACCEL=kvm   # KVM acceleration (fast)
  export VM_ACCEL=tcg   # Software emulation (slow)

  macOS:

  bash Copy

  export VM_ACCEL=hvf   # HVF acceleration (fast)
  export VM_ACCEL=tcg   # Software emulation (slow)

  If not specified, the optimal acceleration method is automatically detected.

Start the VM Copy Chapter

Basic Usage Copy Section

Run the following command to start the VM with default settings:

vagrant up

Custom Configuration Examples Copy Section

You can combine environment variables for custom configurations:

# Test VM with custom resources
VM_TYPE=test VM_CPUS=4 VM_MEM=4 vagrant up

# Force software emulation (useful for troubleshooting)
VM_ACCEL=tcg vagrant up

# High-performance development VM
VM_TYPE=dev VM_CPUS=8 VM_MEM=16 vagrant up

Changing Acceleration After VM Creation Copy Section

To change hardware acceleration, you must halt and restart the VM:

# Switch to software emulation
vagrant halt && VM_ACCEL=tcg vagrant up

# Switch back to hardware acceleration (Linux)
vagrant halt && VM_ACCEL=kvm vagrant up

# Switch back to hardware acceleration (macOS)
vagrant halt && VM_ACCEL=hvf vagrant up

Note: vagrant reload does NOT apply acceleration changes.

Vagrant will download the cloud-image/ubuntu-24.04 base box, provision the VM, and install all required dependencies including clang19. This process may take some time on first run.

Accessing the VM Copy Chapter

Once the VM is up and running, you can access it via SSH:

vagrant ssh

This will place you in the /vagrant directory inside the VM, which is synced with the Tracee directory on your host machine.

Shared Folder Implementation Copy Section

The VM uses a sophisticated shared folder system that varies by host OS:

• Linux: Direct 9p mount with UID/GID mapping for optimal performance
• macOS: 9p + bindfs overlay for proper ownership mapping

Performance Note for macOS users: File operations may be slower due to the bindfs overlay. For large Git operations or extensive file processing, consider running them outside the VM.

Build and Run Tracee Copy Chapter

You can now build Tracee within the VM using the provided Makefile. Consult the Tracee documentation for specific build instructions.
Building Tracee Documentation

Stopping the VM Copy Chapter

To stop the VM, use:

vagrant halt

Destroying the VM Copy Chapter

To completely remove the VM, use:

vagrant destroy

Troubleshooting Copy Chapter

Hardware Acceleration Issues Copy Section

KVM not available (Linux):

# Check if KVM is supported
ls /dev/kvm
# If missing, enable virtualization in BIOS/UEFI
# Install KVM packages:
sudo apt install qemu-kvm libvirt-daemon-system

HVF not available (macOS):

# Check if HVF is supported
sysctl -n kern.hv_support
# Ensure you're on macOS 10.10+ with Intel CPU or Apple Silicon

Fallback to software emulation:
If hardware acceleration is not available, the VM automatically falls back to TCG software emulation. This is slower but functional.

Shared Folder Issues Copy Section

Files not visible in /vagrant:

# Check mount status inside VM
mount | grep vagrant

# For Linux hosts - check direct 9p mount
mount | grep 9p

# For macOS hosts - check bindfs mount
mount | grep bindfs

Permission issues (macOS):

# Inside VM, check ownership
ls -la /vagrant

# Should show vagrant:vagrant (1000:1000)
# If not, check systemd mount services (macOS hosts):
sudo systemctl status mnt-shared.mount
sudo systemctl status vagrant.mount

Performance Issues Copy Section

Slow file operations (macOS):

• This is expected due to bindfs overlay
• Consider running large operations outside VM
• Use Git inside VM for better performance with large repositories

VM performance is slow:

• Ensure hardware acceleration is enabled
• Check: vagrant ssh then cat /proc/cpuinfo | grep flags for virtualization features
• Increase VM resources: VM_CPUS=8 VM_MEM=16 vagrant up

Network and Port Issues Copy Section

Port conflicts:

# Check actual forwarded ports
vagrant ssh-config
ps aux | grep qemu-system | grep hostfwd

SSH connection issues:

# Force VM restart
vagrant halt --force && vagrant up

# Check VM status
vagrant status

General Issues Copy Section

VM stuck or unresponsive:

# Force halt and restart
vagrant halt --force
vagrant up

Provisioning failures:

# Re-run provisioning
vagrant provision

# Or destroy and recreate
vagrant destroy -f && vagrant up

QEMU/plugin issues:

# Ensure vagrant-qemu plugin is installed
vagrant plugin list | grep qemu

# Reinstall if needed
vagrant plugin uninstall vagrant-qemu
vagrant plugin install vagrant-qemu

Contributing Testing Coverage (`docs/contributing/testing-coverage.md`) Copy Section

Code Coverage in Tracee

This document explains how code coverage is configured and used in Tracee.

Overview Copy Chapter

Tracee uses Go's built-in coverage tools combined with Codecov for comprehensive test coverage reporting. Coverage is tracked for multiple test types to ensure quality across all components.

Coverage Types Copy Chapter

1. Unit Tests Coverage Copy Section

• Target: Maintain current baseline, 60% for new code
• Scope: Core Go logic, utilities, and non-eBPF components
• Files: cmd/, pkg/, signatures/
• Command: make test-unit

2. Integration Tests Coverage Copy Section

• Scope: eBPF + Go integration, full system testing
• Command: make test-integration

Local Development Copy Chapter

View Coverage Summary Copy Section

make coverage

Generate HTML Coverage Reports Copy Section

make coverage-html
# Opens coverage.html and types/coverage.html

Run Individual Coverage Commands Copy Section

# Unit tests only
go test ./... -coverprofile=coverage.txt -covermode=atomic

# View coverage percentage
go tool cover -func=coverage.txt

# Generate HTML report
go tool cover -html=coverage.txt -o coverage.html

CI/CD Integration Copy Chapter

Coverage is automatically collected and reported on:

• Every Pull Request
• Every push to main branch
• Integration tests (when run)

Codecov Integration Copy Section

Coverage reports are uploaded to Codecov with the following flags:

• unit: Unit test coverage
• integration: Integration test coverage

Coverage Configuration Copy Section

Coverage behavior is configured in codecov.yml:

• Project coverage target: Auto (maintains current baseline)
• Patch coverage target: 60% for new code
• Automatic PR comments with coverage diff
• Excludes test files, generated code, and vendor dependencies

Coverage Files Copy Chapter

The following coverage files are generated (and gitignored):

• coverage.txt: Main unit test coverage
• integration-coverage.txt: Integration test coverage
• coverage.html: HTML report for local viewing

Best Practices Copy Chapter

1. Focus on Critical Paths: Prioritize coverage for core event processing, filtering, and detection logic
2. Test Edge Cases: Include error handling and boundary conditions
3. Integration Coverage: Ensure eBPF + Go interactions are tested
4. Meaningful Tests: Aim for tests that verify behavior, not just coverage numbers

Excluded from Coverage Copy Chapter

• Generated files (*.pb.go)
• Test files (*_test.go)
• Mock files (mock_*.go, *_mock.go)
• Vendor dependencies (vendor/, 3rdparty/)
• Documentation and build scripts
• Test utilities and test data

Troubleshooting Copy Chapter

Coverage Not Generated Copy Section

• Ensure you're running tests with -coverprofile flag
• Check that -covermode=atomic is set (required for concurrent programs)

Low Coverage Warnings Copy Section

• Review which functions/lines are not covered
• Consider if uncovered code represents important paths
• Add tests for critical uncovered functionality

Codecov Upload Failures Copy Section

• Verify CODECOV_TOKEN is set in repository secrets
• Check that coverage files exist before upload
• Review GitHub Actions logs for specific error messages

Docs Forensics (`docs/docs/forensics.md`) Copy Section

Getting Started with Forensics

Tracee has a unique feature that lets you capture interesting artifacts from
running applications, using the --artifacts flag.

./dist/tracee man artifacts

sudo ./dist/tracee --artifacts xxx

!!! Tip
All captured artifacts are saved in Tracee's "output directory", which can
be configured using --artifacts dir.path=/path/to/dir. You may also use
--artifacts dir.clear if you want contents of the destination directory
to be cleared every time you execute tracee.

Artifacts Types Copy Chapter

Tracee can capture the following types of artifacts:

1. I/O Files

   Anytime a file is being written to and/or read from, the contents of the
   file will be captured. I/O files can be filtered using 3 optional filters:

   1. path - prefix of the file written/read. Up to 3 path filters can be
      provided per capture type.
   2. type - file's type can be pipe, socket, elf or regular.
   3. fd - standard FD, one of the following: stdin, stdout and stderr.

   write example

   console Copy

   sudo ./dist/tracee \
      --output json \
      --scope comm=bash \
      --scope follow \
      --enrichment decoded-data \
      --artifacts dir.path=/tmp/tracee \
      --artifacts file-write.filters=path=/tmp/*

   console Copy

   echo write testing 123 > /tmp/testing.txt

   text Copy

/* Detailed source-code truncated for AI context efficiency. */

!!! Note
        You can read captured files written at `/tmp/tracee/out`:
        ```console
        sudo cat /tmp/tracee/out/host/write.dev-271581185.inode-1966101
        ```

        ```text
        write testing 123
        ```
    
    ***read example***

    ```console
    sudo ./dist/tracee \
       --output json \
       --scope comm=bash \
       --scope follow \
       --enrichment decoded-data \
       --artifacts dir.path=/tmp/tracee \
       --artifacts file-read.filters=type=pipe \
       --artifacts file-read.filters=fd=stdin
    ```

    ```console
    echo read testing 123 | cat
    ```
   
    ```
/* Detailed source-code truncated for AI context efficiency. */

!!! Note
You can read captured files read at /tmp/tracee/out:
console sudo cat /tmp/tracee/out/host/read.dev-12.inode-176203

```text
read testing 123
```

1. Executed Files

   Anytime a binary is executed, the binary file will be captured. If the
   same binary is executed multiple times, it will be captured just once.

   console Copy

   sudo ./dist/tracee \
      --output json \
      --scope comm=bash \
      --scope follow \
      --enrichment decoded-data \
      --artifacts dir.path=/tmp/tracee \
      --artifacts executable

   console Copy

   /bin/ls

   text Copy

/* Detailed source-code truncated for AI context efficiency. */

!!! Note
        You will have a copy of each executed file written at `/tmp/tracee/out`:
        ```console
        ldd /bin/ls
        ```

        ```text
        linux-vdso.so.1 (0x00007ffca632c000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f9a930d5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a92ead000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9a92e16000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9a93136000)
        ```

        ```console
        ldd /tmp/tracee/out/host/exec.1657322052835478987.ls
        ```

        ```text
        linux-vdso.so.1 (0x00007ffe337fb000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007feeb1fa5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feeb1d7d000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007feeb1ce6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007feeb2006000)

        ```console
        sudo chmod +x /tmp/tracee/out/host/exec.1657322052835478987.ls
        /tmp/tracee/out/host/exec.1657322052835478987.ls
        ```

1. **Memory Files**

    Anytime a **memory unpacker** is detected, the suspicious **memory region**
    will be captured. This is triggered when memory protection changes from
    **Write+Execute** to **Write**.

    ```console
    sudo ./dist/tracee \
       --output none \
       --scope comm=bash \
       --scope follow \
       --artifacts dir.path=/tmp/tracee \
       --artifacts memory-regions
    ```

    !!! Note
        You may opt not to have any output from **tracee** with `--output none`
        command flag is given. This makes tracee to work in capture mode only.

1. **Network PCAP Files**

    Anytime a **network packet** is delivered to a process, traced by tracee,
    this packet might be captured into one or multiple pcap files.

    !!! Attention
        The default behavior when capturing network traffic is to capture
        ALL traffic, despite given event filters. If you want to make
        capture feature to follow the given event filters, like for example
        capturing DNS events only, then you have to provide `--artifacts
        network.pcap.options=filtered` argument in the command line. Then only
        net_packet_XXX events will be captured (IPv4, IPv6, TCP, UDP,
        ICMP, ICMPv6, DNS, HTTP, etc).

    A good way to test this behavior is to execute:

    ```console
    sudo ./dist/tracee \
        --events net_packet_ipv4 \
        --artifacts network \
        --artifacts network.pcap.options=filtered
    ```

    and observe a single **pcap file** for all ipv4 packets created:

    ```console
    find /tmp/tracee/out/pcap/
    ```

    ```text
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    ```

    You can select only dns packets, for example:

    ```console
    sudo ./dist/tracee \
        --events net_packet_dns \
        --artifacts network \
        --artifacts network.pcap.options=filtered
    ```

    and the file `/tmp/tracee/out/pcap/single.pcap` would only contain DNS
    related packets:

    ```console
    find /tmp/tracee/out/pcap/
    ```

    ```text
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    ```

    ```console
    sudo tcpdump -n -r /tmp/tracee/out/pcap/single.pcap | head -2
    ```

    ```text
    reading from file /tmp/tracee/out/pcap/single.pcap, link-type NULL (BSD loopback), snapshot length 262144
    16:53:48.870629 IP 127.0.0.1.55569 > 127.0.0.53.53: 33361+ [1au] A? www.uol.com.br. (43)
    16:53:48.870690 IP 127.0.0.1.55569 > 127.0.0.53.53: 25943+ [1au] AAAA? www.uol.com.br. (43)
    ```

    A great thing is that you may have multiple pcap files, divided by:

    1. single: a single pcap file containing all packets (the default)
    1. process: one file per process executed, ordered by host and container
    1. container: one file for the host and one pcap file per container
    1. per-command: one file per command executed (even if multiple times)

    and you can even have multiple ways at the same time. Example: a ping
    command is executed inside a container. You want to summarize captured
    traffic per container and per command. You will find the same captured
    data for that ping command inside `commands/container_id/ping.pcap` and
    inside `containers/container_id.pcap`.

    ```console
    sudo ./dist/tracee \
        --events net_packet_icmp \
        --artifacts network \
        --artifacts network.pcap.options=filtered \
        --artifacts network.pcap.split=process,container,command
    ```

    ```console
    cd /tmp/tracee/out
    find pcap
    ```

    ```text
    pcap
    pcap/commands
    pcap/commands/b86533d11f3
    pcap/commands/b86533d11f3/ping.pcap
    pcap/commands/host
    pcap/commands/host/sshd.pcap
    pcap/commands/host/zerotier-one.pcap
    pcap/commands/host/node.pcap
    pcap/commands/fd95a035ce5
    pcap/commands/fd95a035ce5/ping.pcap
    pcap/processes
    pcap/processes/b86533d11f3
    pcap/processes/b86533d11f3/ping_1261180_1663772450241192.pcap
    pcap/processes/host
    pcap/processes/host/node_186708_1573567360495399.pcap
    pcap/processes/host/node_1196826_1662656211119567.pcap
    pcap/processes/host/zerotier-one_7882_137007714376.pcap
    pcap/processes/host/sshd_1196773_1662654999660718.pcap
    pcap/processes/fd95a035ce5
    pcap/processes/fd95a035ce5/ping_1261163_1663769383806467.pcap
    pcap/containers
    pcap/containers/host.pcap
    pcap/containers/b86533d11f3.pcap
    pcap/containers/fd95a035ce5.pcap
    ```

    you can see the packets by executing tcpdump on any pcap file:

    ```console
    tcpdump -r pcap/containers/b86533d11f3.pcap
    ```

    ```text
    reading from file pcap/containers/b86533d11f3.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    ```

    ```console
    tcpdump -r pcap/commands/b86533d11f3/ping.pcap
    ```

    ```text
    reading from file pcap/commands/b86533d11f3/ping.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    ```

    !!! Note
        Note that the same packets were written to 2 different pcap files: the
        pcap file describing the container `b86533d11f3` (because it was
        executing a single process: ping) and the pcap file describing ANY ping
        command executed in that container (commands/b86533d11f3/ping.pcap).

    The format for the pcap filenames inside `output_dir` is the following:

    1. **single**:  
       ./pcap/single.pcap
    1. **processes**:  
       ./pcap/processes/`container_id`/`process_comm`/`host_tid`/`task_starttime`.pcap
    1. **containers**:  
       ./pcap/containers/`container_id`.pcap
    1. **commands**:  
       ./pcap/commands/`container_id`/`process_comm`.pcap

    !!! Attention
        By default, all pcap files will contain packets with headers only. That
        might too little for introspection, since sometimes one might be
        interested in a few bytes of the captured packet (or event it all). Next
        item shows how to capture a specific packet payload size.

    In order to capture a specific payload size you may specify:

    ```console
    sudo ./dist/tracee \
        --events net_packet_tcp \
        --artifacts network \
        --artifacts network.pcap.options=filtered \
        --artifacts network.pcap.split=single,command \
        --artifacts network.pcap.snaplen=default
    ```

    To capture packet headers + 96 bytes of payload. Or replace `default` by:

    1. headers: capture up to L4 headers only
    1. max: full sized packets into pcap. WARNING: big pcap files.
    1. **256b**, **512b**, **1024b**, ... (any number plus "b")
    1. **16kb**, **32kb**, **64kb**,  ... (any number plus "kb")

    > when specifying a payload size, it refers to the payload AFTER the layer4
    > headers (and not the entire packet length).

1. **Loaded Kernel Modules**

    Anytime a **kernel module** is loaded, the binary file will be captured.
    If the same binary is loaded multiple times, it will be captured just once.

    ```console
    sudo ./dist/tracee \
        --output none \
        --scope comm=bash \
        --scope follow \
        --artifacts dir.clear \
        --artifacts kernel-modules
    ```

    Captured module will be found in tracee destination directory, just like
    any other captured file would be:

    ```console
    sudo ls /tmp/tracee/out/host
    ```

    ```text
    module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    ```

    AND, the captured module is an exact copy of the loaded module:

    ```console
    sudo rmmod lkm_example
    sudo insmod /tmp/tracee/out/host/module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    ```

    ```console
    lsmod | grep example

    ```text
    lkm_example            16384  0
    ```

    ```console
    sudo rmmod lkm_example
    ```

    you can even load/unload it.

    !!! Note
        Example kernel module taken from [this blog]

[this blog]: https://blog.sourcerer.io/writing-a-simple-linux-kernel-module-d9dc3762c234

1. **BPF programs**

    Wheneber a **BPF program** is loaded, the BPF bytecode will be captured.
    This captured bytecode represents the BPF program as it was loaded by the loading process.
    It is basically the BPF section of the compiled ELF that loads the BPF program, which contains the program instructions themselves.
    You can disassemble the bytecode with the help of `binutils-bpf` package and the following command line:
    `objdump -D -b binary -m bpf `

     ```text
     $ sudo ./dist/tracee \
        --output none \
        --scope comm=bash \
        --scope follow \
        --artifacts dir.clear \
        --artifacts bpf-programs
     ```

    Captured bpf bytecode will be found in tracee destination directory, just like
    any other captured file would be:

     ```text
     $ sudo ls /tmp/tracee/out/host
       bpf.name-test_prog.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
     ```
   The hex value after the last "." is the hash of the bpf bytecode.

---

### Docs Security Model (`docs/docs/security-model.md`)

# Tracee's Security Model

## Introduction

Understanding Tracee's security model is essential for setting proper expectations about what Tracee can and cannot protect against. This document describes Tracee's adversary model, security boundaries, detection capabilities, and limitations to help you make informed decisions about deploying and using Tracee in your security infrastructure.

Tracee is an eBPF-based runtime security tool designed to detect threats and suspicious behavior by monitoring kernel events. However, like any security tool, it operates under specific assumptions and has inherent limitations based on its architecture and the adversary it targets.

## Core Security Assumptions

Tracee's security model is built on one fundamental assumption:

**The kernel is trusted when Tracee starts.**

This means:

- The kernel has not been compromised before Tracee loads its eBPF programs
- No kernel rootkits or malicious kernel modules are already present
- The kernel's integrity is intact at Tracee startup time

If this assumption is violated (i.e., the kernel is already compromised), Tracee's detection capabilities cannot be guaranteed. A compromised kernel has complete control over the system and can potentially:

- Interfere with eBPF program execution
- Hide events from Tracee
- Manipulate Tracee's data structures
- Disable Tracee entirely

## Two-Tier Adversary Model

Tracee's security guarantees differ significantly depending on whether the adversary operates in userspace or kernel space.

### Userspace Adversaries: Strong Security Boundary

**Target Profile:** Root-privileged adversaries operating in userspace, including containerized environments.

**Tracee's Capabilities:**

Tracee provides strong security boundaries against userspace adversaries through eBPF's architectural guarantees:

**Tamper-Resistant Monitoring**: eBPF programs run in the kernel with their own memory protection. While Tracee's eBPF programs are active and attached:

- Userspace processes cannot bypass the monitoring or evade the attached hooks through normal system operations
- The eBPF bytecode itself cannot be modified once loaded and verified
- Events are captured at the kernel level before userspace can manipulate them
- **However**: Root users with appropriate capabilities (CAP_BPF, CAP_SYS_ADMIN) can administratively disable the monitoring by detaching eBPF programs, killing the Tracee process, or accessing eBPF maps. Tracee's effectiveness assumes the monitoring infrastructure remains intact and that Tracee itself is protected by proper access controls

**Complete Visibility**: Tracee can observe all system calls, LSM hooks, and kernel events from userspace processes, providing comprehensive monitoring of:

- Process execution and lifecycle
- File system operations
- Network activity
- Container escape attempts
- Privilege escalation attempts
- Suspicious behaviors and exploitation techniques

**Security Guarantee:** Userspace adversaries with root privileges cannot evade Tracee's monitoring or bypass its detection mechanisms through userspace operations alone (assuming Tracee itself remains protected and operational).

### Kernel-Level Adversaries: Best-Effort Detection

**Target Profile:** Adversaries with kernel-level access, including rootkits, malicious kernel modules, and kernel zero-day exploits.

**Tracee's Capabilities:**

Against kernel-level adversaries, Tracee provides **best-effort detection** but cannot guarantee protection:

**Rootkits Loaded Before Tracee**: Cannot be reliably detected. If a rootkit compromises the kernel before Tracee starts, it may already have mechanisms to hide from detection or interfere with Tracee's operation.

**Rootkits Loaded After Tracee**: May be detected using known signatures and heuristics. Tracee includes detection capabilities for:

- System call table hooking
- Kernel module loading
- Hidden kernel modules
- Ftrace hooks
- /proc filesystem hooks
- Suspicious kernel modifications

**Kernel Zero-Day Exploits**: Cannot be prevented or reliably detected. An attacker exploiting unknown kernel vulnerabilities can potentially:

- Bypass Tracee's monitoring
- Disable eBPF programs
- Gain complete system control
- Hide their activities from detection

**Limitations:**

- Detection is signature-based and heuristic-driven
- Novel rootkit techniques may evade detection
- Sophisticated kernel-level malware may disable or circumvent Tracee
- Cannot protect against attackers who control the kernel

**Security Guarantee:** Tracee attempts to detect kernel-level threats but cannot guarantee detection or prevention against sophisticated kernel adversaries.

## Security Boundaries and Attack Vectors

### Protected by eBPF Security Model

The following attack vectors are effectively monitored by Tracee due to eBPF's security guarantees:

- **System Calls**: All syscall invocations are observable
- **LSM Hooks**: Linux Security Module hooks provide tamper-resistant checkpoints
- **Network Events**: Packet-level visibility for monitored processes
- **File Operations**: Comprehensive file system activity monitoring
- **Process Lifecycle**: Complete process tree visibility
- **Container Operations**: Container creation, modification, and escape attempts

### Limited or No Protection

The following attack vectors have limited or no protection guarantees:

- **Kernel Memory Manipulation**: Direct kernel memory writes can bypass detection
- **Hardware-Level Attacks**: CPU vulnerabilities, firmware-level compromises
- **Pre-Boot Compromises**: Bootkit, firmware rootkits installed before OS loads
- **Kernel Zero-Days**: Unknown vulnerabilities in the kernel itself
- **eBPF Verifier Bypasses**: Theoretical vulnerabilities in the eBPF verifier (very rare)

## Race Conditions and TOCTOU

### The Problem

When Tracee reads information from user programs, it is subject to a **race condition** where the user program might be able to change the arguments after Tracee reads them. This is known as a Time-of-Check-Time-of-Use (TOCTOU) vulnerability.

Consider this scenario:

A program invokes:

```c
execve("/bin/ls", NULL, 0)

Tracee picks this up and reports it. However, the program could change the first argument from /bin/ls to /bin/bash after Tracee reads it but before the kernel executes it. This means Tracee would report /bin/ls while the kernel actually executes /bin/bash.

The Solution: LSM Events Copy Section

To mitigate this race condition, Tracee provides LSM (Linux Security Module) based events. These events occur at security check time within the kernel, providing more reliable data that's harder for user-space programs to manipulate.

How to Use LSM Events

For critical security monitoring, you should cross-reference regular syscall events with their corresponding LSM events:

Example: Process Execution Monitoring

Instead of relying solely on the execve syscall event, also monitor:

• security_bprm_check - LSM hook that validates executables before execution

By comparing both events, you can detect potential tampering:

• If the execve event shows /bin/ls
• But security_bprm_check shows /bin/bash
• This indicates a TOCTOU attack attempt

When to Use LSM Events

Use LSM-based events for:

• Security-critical monitoring where argument integrity is essential
• Detecting evasion techniques that exploit TOCTOU vulnerabilities
• Compliance requirements that demand tamper-resistant auditing

Best Practices Copy Section

1. Defense in Depth: Use both syscall and LSM events for critical monitoring
2. Cross-Reference: Compare data between event types to detect anomalies
3. Document Assumptions: If using only syscall events, document the TOCTOU risk
4. Test Detection: Verify your policies catch TOCTOU evasion attempts

Attack Scenarios Copy Chapter

Scenario 1: Container Escape Attempt (Detectable) Copy Section

Attack: An adversary with root privileges inside a container attempts to escape using a known technique (e.g., cgroup release_agent modification).

Tracee's Response:

• ✅ Detects the suspicious cgroup modification
• ✅ Triggers cgroup_release_agent_modification signature
• ✅ Provides full context: container ID, process tree, file modifications
• ✅ Alerts security teams for immediate response

Outcome: Attack detected with full forensic context for response.

Scenario 2: Rootkit Loaded Before Tracee (Not Detectable) Copy Section

Attack: A kernel rootkit is installed on the system before Tracee starts. The rootkit hooks system calls to hide malicious processes.

Tracee's Response:

• ❌ Cannot guarantee detection
• ⚠️ May detect some symptoms if rootkit is imperfect
• ⚠️ Initial system scan may identify some inconsistencies

Outcome: Rootkit may remain undetected. Mitigation: Deploy Tracee early in the boot process, ideally as part of the init system or container runtime startup.

Scenario 3: Rootkit Loaded After Tracee (Potentially Detectable) Copy Section

Attack: An adversary attempts to load a kernel module containing a rootkit after Tracee is running.

Tracee's Response:

• ✅ Detects kernel module loading event
• ✅ Triggers kernel_module_loading signature
• ✅ May detect system call table hooking via hooked_syscall event
• ✅ May identify hidden kernel modules via hidden_kernel_module event
• ⚠️ Sophisticated rootkits may still evade detection using novel techniques

Outcome: Good chance of detection for known techniques, but not guaranteed against advanced adversaries.

Scenario 4: Privilege Escalation via Exploit (Detectable) Copy Section

Attack: An adversary exploits a userspace vulnerability to escalate privileges (e.g., dirty pipe, container escape via exposed socket).

Tracee's Response:

• ✅ Detects suspicious capability usage
• ✅ Identifies unusual process behaviors
• ✅ Monitors credential changes
• ✅ Tracks file and memory modifications

Outcome: Attack detected with high confidence.

Scenario 5: Evading Active Monitoring (Not Possible Without Disabling Tracee) Copy Section

Attack: An adversary with root privileges attempts to evade Tracee's monitoring while performing malicious activities, without administratively disabling Tracee first.

Tracee's Response:

• ✅ While eBPF programs are attached and active, userspace operations cannot bypass the monitoring hooks
• ✅ System calls, file operations, network activity, and process events are captured at the kernel level
• ✅ Attempts to manipulate data after capture don't affect what Tracee already observed
• ⚠️ However, an adversary with CAP_BPF or CAP_SYS_ADMIN capabilities can administratively disable Tracee by:
  • Detaching eBPF programs from their hooks
  • Killing the Tracee userspace process
  • Unloading eBPF programs or modifying eBPF maps
• ⚠️ Such administrative actions to disable monitoring may themselves be detectable if other security monitoring is in place

Outcome: Evasion of active monitoring is not possible through normal userspace operations. However, an adversary with sufficient capabilities can disable the monitoring infrastructure itself, which is why protecting Tracee's operational integrity is critical.

Detection Capabilities Overview Copy Chapter

Mitigation Strategies Copy Chapter

Maximize Protection Copy Section

1. Deploy Early: Start Tracee as early as possible in the boot sequence to establish trust before potential compromises
2. Use LSM Events: Prefer LSM-based events for security-critical monitoring to avoid TOCTOU vulnerabilities
3. Enable All Rootkit Signatures: Activate all kernel integrity monitoring signatures (hooked_syscall, hidden_kernel_module, etc.)
4. Layered Security: Combine Tracee with other security controls:
   • Secure boot to verify kernel integrity
   • Kernel module signing to prevent unauthorized module loads
   • SELinux/AppArmor for mandatory access control
   • Network segmentation
   • Regular vulnerability scanning

Acknowledge Limitations Copy Section

1. Understand Boundaries: Be aware that kernel-level compromises can bypass Tracee
2. Verify Kernel Integrity: Use external tools to verify kernel integrity before trusting Tracee's output
3. Monitor Tracee Itself: Watch for signs that Tracee may have been tampered with (e.g., unexpected crashes, missing events)
4. Regular Updates: Keep Tracee, the kernel, and eBPF subsystem updated to benefit from latest security fixes
5. Incident Response Plan: Have procedures for handling scenarios where Tracee's integrity may be compromised

Operational Best Practices Copy Section

1. Start with Known-Good State: Deploy Tracee on freshly provisioned or verified clean systems
2. Baseline Normal Behavior: Establish baselines before deploying to production
3. Alert Tuning: Configure appropriate alert thresholds to balance detection and false positives
4. Regular Review: Periodically review detected events and tune signatures
5. Forensic Capabilities: Enable artifact capture for post-incident analysis

Related Documentation Copy Chapter

• Capabilities - Learn about running Tracee with least privileges
• Rootkit Detection Events - Detailed documentation on kernel integrity monitoring
• Policies - Create policies that implement defense-in-depth
• Forensics - Capture artifacts for investigation

Summary Copy Chapter

Tracee provides strong security guarantees against userspace adversaries through eBPF's tamper-resistant architecture. However, it offers only best-effort detection against kernel-level adversaries and cannot protect against attackers who control the kernel before Tracee starts or exploit kernel zero-day vulnerabilities.

By understanding these boundaries and implementing layered security controls, you can maximize Tracee's effectiveness as part of a comprehensive security strategy.

Docs Troubleshooting (`docs/docs/troubleshooting.md`) Copy Section

Troubleshooting

This guide helps you diagnose and resolve common issues when running Tracee.

Installation Issues Copy Chapter

BTF Not Available Copy Section

Problem: Error message about missing BTF support

ERRO error loading eBPF program: BTF is required

Solution:

1. Check if BTF is available: ls /sys/kernel/btf/vmlinux
2. If missing, upgrade to a kernel version that includes BTF support (most modern distributions)
3. See Prerequisites for more details

Permission Denied Copy Section

Problem: Tracee fails to start with permission errors

ERRO permission denied loading eBPF program

Solutions:

1. Run as root: sudo tracee
2. Use required capabilities:
   bash Copy

   # Docker - minimal required capabilities
   docker run \
     --cap-add SYS_RESOURCE \
     --cap-add SYS_ADMIN \
     --cap-add SYS_PTRACE \
     --cap-add NET_ADMIN \
     --cap-add SYSLOG \
     tracee
   
   # Kubernetes
   securityContext:
     capabilities:
       add: 
         - "SYS_RESOURCE"
         - "SYS_ADMIN" 
         - "SYS_PTRACE"
         - "NET_ADMIN"
         - "SYSLOG"

3. For recent kernels (>=5.8), you may use CAP_BPF + CAP_PERFMON instead of CAP_SYS_ADMIN
4. See Prerequisites for complete details and justifications

Kernel Version Incompatibility Copy Section

Problem: Unsupported kernel version errors

Solution:

• Check supported kernels: Prerequisites
• Tracee requires kernel 5.4 or newer (4.18 for RHEL 8)
• Consider upgrading to a supported kernel version

Runtime Issues Copy Chapter

No Events Generated Copy Section

Problem: Tracee starts but produces no events

Troubleshooting steps:

1. Check scope filters:

   bash Copy

   # Test with minimal configuration
   tracee --events execve

2. Verify events are enabled:

   bash Copy

   # List enabled events
   tracee --events help

3. Check policy configuration:

   yaml Copy

   # Ensure policy scope matches your workload
   scope:
     - container  # Or specific process filters if needed

High CPU Usage Copy Section

Problem: Tracee consuming excessive CPU

Solutions:

1. Reduce event scope:

   yaml Copy

   scope:
     - container  # Limit to containers only

2. Filter events:

   yaml Copy

   rules:
     - event: execve
       filters:
         - uid!=0  # Example: exclude root processes

3. Use specific events instead of sets:

   yaml Copy

   rules:
     - event: execve        # Specific event
     # - event: tag=syscalls    # Avoid broad tags

Container Issues Copy Chapter

Container Events Not Captured Copy Section

Problem: Missing container-related events

Solutions:

1. Mount container runtime socket:
   bash Copy

   # Docker
   docker run -v /var/run/docker.sock:/var/run/docker.sock tracee
   
   # Containerd
   docker run -v /run/containerd/containerd.sock:/run/containerd/containerd.sock tracee

Performance Issues Copy Chapter

Events Being Dropped Copy Section

Problem: Warning about dropped events

WARN events dropped due to buffer overflow

Solutions:

1. Increase buffer size:

   bash Copy

   tracee --buffers kernel-events=1024

2. Reduce event frequency:

   yaml Copy

   rules:
     - event: openat
       filters:
         - data.pathname!=/tmp/*  # Filter noisy paths

Slow Event Processing Copy Section

Problem: Events arrive with significant delay

Solutions:

1. Check system load: Use top, htop to verify system isn't overloaded
2. Optimize event selection: Use specific events instead of broad event sets

Output Issues Copy Chapter

JSON Parsing Errors Copy Section

Problem: Invalid JSON output

Solutions:

1. Use proper output format:

   bash Copy

   tracee --output json --enrichment decoded-data

2. Check for mixed output:

   bash Copy

   # Separate logs from events
   tracee --output json --logging file=/var/log/tracee.log

Missing Event Fields Copy Section

Problem: Expected fields not present in events

Solutions:

1. Enable argument parsing:

   bash Copy

   tracee --enrichment decoded-data

2. Check event definition: Some events may not include all expected fields

Debugging Copy Chapter

Enable Debug Logging Copy Section

# Enable debug logs
tracee --logging level=debug

# Or via environment
TRACEE_LOG_LEVEL=debug tracee

Capture System Information Copy Section

# System info for bug reports
tracee --version
uname -a
cat /etc/os-release
ls -la /sys/kernel/btf/vmlinux

Test Minimal Configuration Copy Section

# Minimal test configuration
sudo tracee --events execve

Getting Help Copy Chapter

If you continue experiencing issues:

1. Search existing issues: GitHub Issues
2. Check discussions: GitHub Discussions
3. Join Slack: Aqua Community Slack

When reporting issues, include:

• Tracee version (tracee --version)
• Operating system and kernel version
• Container runtime (if applicable)
• Complete error messages
• Minimal reproduction steps

Traceectl Usage (`docs/traceectl/usage.md`) Copy Section

traceectl Installation and Usage Guide

Installation Copy Chapter

To use traceectl, you first need to compile and install the tool. Follow these steps to get started:

1. **Clone the Repository** Copy Section

Begin by cloning the Tracee repository to your local machine and navigating to traceectl:

git clone https://github.com/aquasecurity/tracee.git
cd cmd/traceectl

2. **Build and Install** Copy Section

Compile and install traceectl using the following commands:

make

Configuring Tracee for traceectl Copy Chapter

To use traceectl effectively, you need to configure Tracee so that it can communicate with traceectl over a Unix socket. This can be done by running Tracee with the correct gRPC settings:

1. **Run Tracee with gRPC Unix Socket** Copy Section

Use the following command to start Tracee with gRPC support over a Unix socket:

tracee --server grpc-address=unix:/var/run/tracee.sock

This command sets up Tracee to listen for incoming connections from traceectl at the specified Unix socket path (/var/run/tracee.sock). Ensure that this socket path is accessible and not blocked by permissions or other constraints.

2. **Output Flag Configuration** Copy Section

The --output flag in Tracee allows you to configure how data from Tracee is presented. Among the available options, you can specify none for minimal output, which can be useful for scenarios where bandwidth or latency considerations are critical. For example:

tracee --output none

Why Use `--output none`?

• Reduced Bandwidth Usage: By suppressing output, you can minimize the data transferred over the Unix socket, which is especially helpful in environments with limited resources.
• Lower Latency: With no data formatting or transmission overhead, the interaction between traceectl and Tracee becomes faster.

Use this mode for performance testing, silent monitoring, or when integrating traceectl with other systems that handle data processing separately.

This command sets up Tracee to listen for incoming connections from traceectl at the specified Unix socket path (/var/run/tracee.sock).
Ensure that this socket path is accessible and not blocked by permissions or other constraints.

Usage Copy Chapter

Once traceectl is installed and Tracee is running, you can use various commands to interact with Tracee. Below are the main commands provided by traceectl:

• Stream Events: traceectl stream

• Events management: traceectl event

• Retrieve Metrics: traceectl metrics

• Check Version: traceectl version

For more info about the traceectl command please refer to the appoint command documentation

Flags Copy Chapter

For more info about the traceectl flags see:

• output
• format
• server
• policy

Summary Copy Chapter

• Install traceectl by cloning the repository, building, and installing it with make.
• Use traceectl to interact with Tracee via commands like stream, event, metrics, and version.

Tutorials Additional Resources (`docs/tutorials/additional-resources.md`) Copy Section

Additional Resources

This page links to external resources, created by Aqua Security and the community. If you create content on Tracee, we would highly appreciate if you would contribute the link to this page.

Blog Post Copy Chapter

• The Story of Tracee: The Path to Runtime Security Tool
• Detecting eBPF Malware with Tracee

Useful Videos Copy Chapter

• Tracee Livestream: Everything is an Event in Tracee
• Tracee Video Series on the Aqua Open Source YouTube channel

Have a look at the recordings on the Aqua Open Source YouTube channel.

Conference recording Copy Chapter

• eBPF Warfare - Detecting Kernel & eBPF Rootkits with Tracee
• Verifiable GitHub Actions with eBPF - Jose Donizetti, Aqua
• Tracee Features and Internals: High Throughput of eBPF Events for Execution Patterns Detections
• BlackHat Arsenal 2022: Detecting Linux kernel rootkits with Aqua Tracee
• Tracee presentation at BlackHat USA 2021

Tutorials Deploy Grafana Dashboard (`docs/tutorials/deploy-grafana-dashboard.md`) Copy Section

Access Tracee Metrics in Prometheus and Grafana

Tracee exposes a metrics endpoint.
This makes it possible to monitor Tracee like other cloud native workloads.

In this tutorial, we will showcase how to access Tracee metrics through Prometheus and Grafana running through docker containers.
The tutorial can easily adapted to Kubernetes workloads with the configuration provided.

Tutorial Overview Copy Chapter

Grafana is a visualization tools for exported metrics and logs, most commonly
used alongside Prometheus.

Since version 0.7.0, tracee exports useful runtime metrics to Prometheus.

By using Grafana and the new metrics from tracee, we can deploy a simple
dashboard which tracks the tracee instance performance and outputs.

There are two options for accessing Tracee metrics:

• Running the Tracee Docker Container Image -- Shown in this tutorial
• Running the Tracee Helm Chart -- Detailed as part of the Promtail-Tracee tutorial

Tracee Docker Container Image Copy Chapter

These metrics exports are enabled by default in all docker images and can be
enabled using the --server metrics flag.

[tracee]: https://github.com/aquasecurity/tracee/tree/{{ git.tag }}/cmd/tracee

Prerequisites Copy Section

The following tools must be available for use, they can all be installed either
through docker or installed/built on your machine. Note that you need to be on a Linux machine to follow the Docker tutorial.
Alternative, on a MacBook it is possible to use Vagrant with Parallels as detailed in the following tutorial:
Running Tracee on Mac with Parallels and Vagrant

• Tracee
• Prometheus
• Grafana

Run Tracee with Metrics Enabled -- The Docker Command Copy Section

Tracee can be most easily deployed with metrics enabled by default and port
forwarded through the following commands:

docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  -p 3366:3366 \
  aquasec/tracee:latest \
  --server metrics

Of course, the forwarded metrics ports can be changed, but you should note that
some of the later instructions depend on these ports.

If running Tracee locally through built binaries, the metrics address may be
overrides with the --listen-addr flag.

Run Prometheus and Configure it to Scrape Tracee Copy Section

Install Prometheus or pull it's Docker image. Then create the following
configuration file, call it prometheus.yml to scrape Tracee:

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Tracee.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'tracee'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    metrics_path: /metrics

    #Scrape tracee's default metrics hosts.
    #If forwarding different ports make sure to change these addresses.
    static_configs:
      - targets: ['localhost:3366']

Note that the localhost:3366 might have to be replaced with your IP address. The IP address can be found with the following command:

/sbin/ifconfig

We can then start Prometheus with the following command:

prometheus --config.file=/path/to/prometheus.yml

Or alternatively with docker:

docker run \
    -p 9090:9090 \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

Ensure that prometheus can scrape the Tracee target: Open the Prometheus UI at http://localhost:9090
Under Status < Targets will be the scrape targets listed.

If successful, move to the next step, otherwise consult with the Prometheus documentation.

Run Grafana to display Tracee's Prometheus Metrics Copy Chapter

After successfully deploying Tracee and Prometheus we may now run Grafana to
visualize it's metrics.

Install Grafana using the Grafana container image:

docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise

Next, you should be able to access the Grafana UI throug: http://localhost:3000.

After entering the website, logging in with username and password admin (and
changing your password if you wish), you should see the homepage:

Add your data source by hovering the configuration tab (the gear icon),
selecting "Data Sources" and pressing "Add Data Source" at the top left. Create
a Prometheus Data Source and point it's URL to the relevant location (usually
http://localhost:9090)

Note that you might have to use your local IP address again instead of localhost.

You may now either create your own Dashboard or import our default dashboard.

Import Tracee's Default Dashboard Copy Section

First download our Grafana Dashboard's json here.

After adding the data source hover on the plus icon in the sidebar and select
"Import". Press "Upload JSON File" at the top of the page and select the
downloaded json from your file browser. Change the name and Dashboard UID if
you wish and press "Import" to finish.

Finally you will be redirected to the dashboard 🥳

Tutorials K8s Policies (`docs/tutorials/k8s-policies.md`) Copy Section

Working with Tracee Policies on Kubernetes

Prerequisites Copy Chapter

Before you begin, ensure that you have the following:

• A Kubernetes cluster up and running
• kubectl command-line tool installed and configured to work with your cluster
• Helm v3 or later installed on your local machine

Install Tracee via Helm Copy Chapter

To install Tracee using Helm, follow these steps:

Add the Aqua Security Helm repository:

helm repo add aqua https://aquasecurity.github.io/helm-charts/

Install Tracee with the default settings:

helm install tracee aqua/tracee \
    --namespace tracee-system --create-namespace \
    --set hostPID=true

This command installs Tracee in the tracee-system namespace, enabling the use of the host's PID namespace.

Add a new Tracee policy Copy Chapter

By default, Tracee comes with a policy for signature events. In this step, you will learn how to add a new policy to suit your requirements.

The tracee-policies configmap should have all policies Tracee will load when booting. Let's take a look at the default policy:

kubectl get configmap -n tracee-system

NAME               DATA   AGE
tracee-config      1      58m
tracee-policies    2      58m

Let's take a look at the default policy:

kubectl describe configmap/tracee-policies -ntracee-system

Name:         tracee-policies
Namespace:    tracee-system
Data
====
signatures.yaml:
----
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
	name: signature-events
	annotations:
		description: traces all signature events
spec:
	scope:
	  - global
	rules:
	  - event: stdio_over_socket
	  - event: k8s_api_connection
	  - event: aslr_inspection
	  - event: proc_mem_code_injection
	  - event: docker_abuse
	  - event: scheduled_task_mod
	  - event: ld_preload
	  - event: cgroup_notify_on_release
	  - event: default_loader_mod
	  - event: sudoers_modification
	  - event: sched_debug_recon
	  - event: system_request_key_mod
	  - event: cgroup_release_agent
	  - event: rcd_modification
	  - event: core_pattern_modification
	  - event: proc_kcore_read
	  - event: proc_mem_access
	  - event: hidden_file_created
	  - event: anti_debugging
	  - event: ptrace_code_injection
	  - event: process_vm_write_inject
	  - event: disk_mount
	  - event: dynamic_code_loading
	  - event: fileless_execution
	  - event: illegitimate_shell
	  - event: kernel_module_loading
	  - event: k8s_cert_theft
	  - event: proc_fops_hooking
	  - event: syscall_hooking
	  - event: dropped_executable

But let's suppose we also need Tracee to trace all execve events. For this, we need to change the configmap tracee-policies by adding a new policy.

Let's edit the tracee-policies ConfigMap using kubectl:

kubectl edit cm/tracee-policies -n tracee-system

The ConfigMap will open in your default text editor. Locate the data section.

To add a new policy for tracking execve events, add the following YAML block before the signatures.yaml section, maintaining proper indentation:

data:
  events.yaml: |-
    apiVersion: tracee.aquasec.com/v1beta1
    kind: Policy
    metadata:
        name: execve-event
        annotations:
          description: traces all execve events
    spec:
        scope:
          - global
        rules:
          - event: execve
  signatures.yaml: |-
  ...

Save and close the file. The changes will be applied to the configmap.

!!! note
If you are having a problem editing the configmap, you can apply it directly with:
console kubectl apply -f https://gist.githubusercontent.com/josedonizetti/3df19a61d39840441ea5be448d6c9354/raw/c50b9b66d7996bb27b6fac301d24d6390e356f8c/tracee-policies-configmap.yaml

Step 3: Restart Tracee Daemonset
After modifying the Tracee policies, you need to restart the Tracee daemonset for the changes to take effect.

Restart the Tracee daemonset using the following command:

kubectl rollout restart ds/tracee -n tracee-system

Wait for the daemonset to restart and stabilize. You can monitor the progress using the following command:

kubectl rollout status ds/tracee -n tracee-system

Then check for execve events:

kubectl logs -f ds/tracee -n tracee-system | grep execve

{"timestamp":1684688250477166817,"threadStartTime":1684688250477064221,"processorId":7,"processId":35694,"cgroupId":1386180,"threadId":35694,"parentProcessId":1033,"hostProcessId":3242201,"hostThreadId":3242201,"hostParentProcessId":3205483,"userId":0,"mountNamespace":4026532829,"pidNamespace":4026532833,"processName":"cri-dockerd","hostName":"minikube","container":{},"kubernetes":{},"eventId":"59","eventName":"execve","matchedPolicies":["execve_event"],"argsNum":2,"returnValue":0,"syscall":"execve","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":"/opt/cni/bin/bridge"},{"name":"argv","type":"const char*const*","value":["/opt/cni/bin/bridge"]},{"name":"envp","type":"const char*const*","value":null}]}

Once the daemonset is up and running, the modified policies will be applied.

Congratulations! You have successfully installed Tracee via Helm, modified the default policies to add an execve event policy.

Note: Modifying the policies may have security implications, so it is important to carefully consider the events you enable based on your specific requirements and security considerations.

Feel free to reach out if you have any further questions or need additional assistance!

Tutorials Promtail (`docs/tutorials/promtail.md`) Copy Section

Using Promtail, Loki and Grafana to access Tracee Logs

By default, Tracee is emitting events to stdout. Users can then configure logging solutions to collect, store, and manage Tracee logs.

This tutorial will showcase how to install and configure Promtail, Loki, Grafana and Prometheus to then access Tracee logs from the cluster in Grafana.

If you prefer the video tutorial, check out the tutorial below on the Aqua Open Source YouTube channel:

Grafana Loki to access Tracee logs

Prerequisites Copy Chapter

Please make sure to have the following tools installed in your CLI:

• Kubectl installed and connected to a Kubernetes cluster (any cluster will work for this purpose)
• The Helm CLI installed

Additionally, you might have the following Obserability Stack already installed in your cluster, if not we will detailed how to set it up further below in this guide:

• Prometheus
• Loki and Promtail
• Grafana

Alternatively, this tutorial showcases after the Tracee Installation section how to get an observability stack running with the above tools.

Installing the Tracee Helm Chart and accessing logs Copy Chapter

Right now, we cannot access any logs from our cluster since we do not have any application that actively produces logs.
Thus, we will install Tracee inside our cluster through the Tracee Helm Chart.

Add the Tracee Helm Chart:

helm repo add aqua https://aquasecurity.github.io/helm-charts/

Update the repository list on Helm:

helm repo update

Install the Tracee Helm Chart inside your Kubernetes cluster:

helm install tracee aqua/tracee \
        --namespace tracee-system --create-namespace \
        --set hostPID=true

Now, ensure that Tracee is running inside the tracee-system namespace:

kubectl get all -n tracee-system

Similar to Promtail, also for Tracee one pod should run on each node of the Kubernetes cluster.

Accessing Tracee Logs Copy Section

Generally, it is possible to access logs from the Tracee pods directly through kubectl:

kubectl logs -f daemonset/tracee -n tracee-system

Next, open the Grafana Dashboard, on the left, go to "Explore". There, you should be able to select Loki as a Datasource.

Now, you can write log queries in LogQL to access the logs that are stored in the Tracee pods:

Installation of Observability Tools Copy Chapter

We need to install an observability stack to access the logs of the pods inside our cluster. This will consist of:

• Grafana (for Dashboards and querying logs)
• Promtail for collecting logs from the pods on each node
• Loki, which is feeding the logs into Grafana

And since it is easier to install Grafana together with Prometheus, we are also going to install Prometheus.

If you are completely new to Loki, have a look at the following presentation: Learning the tricks of Grafana Loki for distributed logging at scale in a Kubernetes environment

Grafana and Prometheus Copy Section

First, we are going to install the kube-prometheus-stack chart with Prometheus and Grafana.

For this, we will need to specify some custom values that we will pass into the Helm Chart.

Create a new file called grafana-config.yaml with the following content:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

grafana:
  sidecar:
    datasources:
      defaultDatasourceEnabled: true
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki-loki-distributed-query-frontend.monitoring:3100

Next, we can install the kube-prometheus-stack chart into our cluster with the following commands:

Create a namespace for all the monitoring tools

kubectl create ns monitoring

Add the kube-prometheus-stack Helm Chart to your Helm repository list:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Ensure you have the latest version of all your repositories:

helm repo update

Install the kube-prometheus-stack Helm Chart:

helm upgrade --install prom prometheus-community/kube-prometheus-stack -n monitoring --values grafana-config.yaml

Lastly, confirm that all the pods have been created properly by querying the namespace:

kubectl get all -n monitoring

Promtail and Loki Copy Section

Next, we need to install Promtail and Loki inside the cluster to actually access logs.

For this, first add the Grafana Helm Chart repository to you repository list:

helm repo add grafana https://grafana.github.io/helm-charts

Update your Helm repository list:

helm repo update

Next, create a file with the Helm Chart configuration for Prometail in a promtail-config.yaml:

config:
  serverPort: 8080
  clients:
    - url: http://loki-loki-distributed-gateway/loki/api/v1/push

Now we can install the Promtail Helm Chart inside our cluster:

helm upgrade --install promtail grafana/promtail --values promtail-config.yaml -n monitoring

Make sure that Promtail is running the same number of pods as there are nodes on the cluster since Promtail has to run one pod per node:

k get pods -n monitoring

For instance, if the cluster consists of three nodes, then there should be three Promtails pods inside of the monitoring namespace.

Now, we can install Loki. Loki's job is to collect the logs from Promtail and forward them to Grafana.

helm upgrade --install loki grafana/loki-distributed -n monitoring

Loki will install a variety of components inside your cluster, so don't be alarmed by the number of pods it is starting, namely:

loki-loki-distributed-distributor-5db986bb4f-x444n
loki-loki-distributed-query-frontend-bd6845f89-z6nx6
loki-loki-distributed-gateway-79d4d4ff5d-sxx58
loki-loki-distributed-querier-0
loki-loki-distributed-ingester-0

At this point, the following pods should be running inside the Kubernetes cluster:

NAME                                                     READY   STATUS    RESTARTS      AGE
prom-prometheus-node-exporter-l4cm4                      1/1     Running   0             22m
prom-kube-prometheus-stack-operator-84cf966ff5-96xdp     1/1     Running   0             22m
prom-kube-state-metrics-dc769cd87-fmrsk                  1/1     Running   0             22m
prom-grafana-6fdb45b4d5-2zxw7                            3/3     Running   0             22m
alertmanager-prom-kube-prometheus-stack-alertmanager-0   2/2     Running   1 (22m ago)   22m
prometheus-prom-kube-prometheus-stack-prometheus-0       2/2     Running   0             22m
promtail-sxfl5                                           1/1     Running   0             4m7s
loki-loki-distributed-gateway-79d4d4ff5d-sxx58           1/1     Running   0             72s
loki-loki-distributed-distributor-5db986bb4f-x444n       1/1     Running   0             72s
loki-loki-distributed-query-frontend-bd6845f89-z6nx6     1/1     Running   0             72s
loki-loki-distributed-querier-0                          1/1     Running   0             72s
loki-loki-distributed-ingester-0                         1/1     Running   0             72s

Since everything is running properly, we need to ensure that we can access Loki as a data source inside of Grafana.

For this, port-forward to Grafana:

kubectl port-forward service/prom-grafana -n monitoring 3000:80

and open the Grafana UI on localhost:3000.

Here, you will need the username and the password:
username: admin
password: prom-operator

The password name is dependent on how you called the Helm Chart installation of the kube-prometheus-stack chart e.g. in our case, it was "prom".

Now navigate on Grafana to: Explore
Here select Loki as a data source.

Tutorials Tracee Vagrant (`docs/tutorials/tracee-vagrant.md`) Copy Section

Using the Tracee container image on MacOS with Parallels and Vagrant

There are a few more steps involved in running Tracee through a container image on arm64 (M1).

Prerequisites:

• Vagrant CLI installed
• Parallels Pro installed

First, clone the Tracee Git repository and move into the root directory:

git clone [email protected]:aquasecurity/tracee.git

cd tracee

Next, use Vagrant to start a Parallels VM:

vagrant up

This will use the Vagrantfile in the root of the Tracee directory.

Lastly, ssh into the created VM:

vagrant ssh

Now, it is possible to run the Tracee Container image:

docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest

To learn how to install Tracee in a production environment, check out the Kubernetes guide.

Tutorials Verify Tracee Signature (`docs/tutorials/verify-tracee-signature.md`) Copy Section

Cosign: verify tracee signature

Prerequisites Copy Chapter

Before you begin, ensure that you have the following installed:

• cosign

Verify tracee signature Copy Chapter

Tracee images are signed with cosign keyless. To verify the signature we can run the command:

cosign verify aquasec/tracee:tag-name  --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-identity-regexp https://github.com/aquasecurity/tracee | jq

Note that all of the tag-names can be found on the Tracee Docker Hub Registry.

The output should look similar to the following:

Contributing Building Building (`docs/contributing/building/building.md`) Copy Section

Building from the source

!!! Note
Consider also visiting the following instructions:
1. docker container as building environment
2. building tracee container images
3. using development images for testing latest changes

1. Supported Architectures

   1. x86_64 (amd64)
   2. aarch64 (arm64)

2. Building dependencies

   1. clang && llvm (19)
   2. golang (1.26.5 toolchain)
   3. libelf and libelf-dev
      (or elfutils-libelf and elfutils-libelf-devel)
   4. zlib1g and zlib1g-dev
      (or zlib and zlib-devel)
   5. libzstd-dev for static build (libelf linkage)
   6. clang-format-19 (specific version) for fix-fmt

   You might take a look at the following files to understand how to have a
   building environment:

   1. .github/actions/build-dependencies/action.yaml
   2. builder/Dockerfile.ubuntu-tracee-make
   3. builder/Dockerfile.alpine-tracee-make

   Those are very good examples for you to replicate a working environment.

3. Clone tracee repository

   bash Copy

   git clone [https://github.com/aquasecurity/tracee/](https://github.com/aquasecurity/tracee/)

4. All makefiles have a help target to give you needed instructions

   bash Copy

   make help

   text Copy

/* Detailed source-code truncated for AI context efficiency. */

5. Build **all** targets at once

    ```bash
    make all
    ```

6. Build a **static binary** by setting `STATIC=1`

    ```bash
    STATIC=1 make all
    ```

7. Build a **static binary** with [BTFHUB Support](https://github.com/aquasecurity/btfhub)

    ```bash
    BTFHUB=1 STATIC=1 make all
    ```

    !!! Note
        BTFHUB support will embed several very small files (BTF files) into your
        final binary. Those files will allow **tracee** binary to be executed
        in kernels that doesn't have embedded BTF information (the ones described
        at the BTFHUB repository)

    !!! Attention
        compiling with STATIC=1 won't allow you to use golang based
        signatures as plugins, only as built-ins:
        >```text
        >2021/12/13 13:27:21 error opening plugin /tracee/dist/signatures/builtin.so:
        >plugin.Open("/tracee/dist/signatures/builtin.so"): Dynamic loading not supported
        >```

8. Build a **debuggable binary** with DWARF debug symbols by setting `DEBUG=1`

    ```bash
    DEBUG=1 make
    ```

9. Build enabling BPF metrics by setting `METRICS=1`

    BPF metrics are only available if the BPF object is built with `METRICS` debug flag defined.

    ```bash
    METRICS=1 make
    ```

## Development Images

See the development images, usage examples, and stability guidance in the container images guide: [Development Images](./containers.md#development-images).

---

### Contributing Building Containers (`docs/contributing/building/containers.md`)

# Creating Tracee Container Images

> These instructions are meant to describe how to build the official tracee
> container image, instead of just downloading it from the
> [Docker Hub](https://hub.docker.com/r/aquasec/tracee).
>
> If you would like to have a local building and execution environment,
> [read this](./environment.md) instead.

## Using Tracee Container Image from Docker Hub

Before moving on to how to build Tracee container, it is important to know the
published container images and their tag meanings. Here is the current list of
docker container images being published during a release (or a snapshot
release):

1. **SNAPSHOT (development) container images:**

     Daily development images built from the latest `main` branch. Tags:

     - **aquasec/tracee:dev** (multi-arch)
     - **aquasec/tracee:x86_64-dev** (amd64 only)
     - **aquasec/tracee:aarch64-dev** (arm64 only)
     - **aquasec/tracee:dev-YYYYMMDD-HHMMSSUTC** (timestamped)

2. **RELEASE (official versions) container images:**

     Preferable alias for latest released image:

     - **aquasec/tracee:latest**

     And the container images for each released version of Tracee:

     - **aquasec/tracee:VERSION**

## Generating Tracee Container Images

1. **tracee:latest**

    Contains an executable binary with an embedded and CO-RE enabled eBPF object
    that makes it portable against multiple Linux and kernel versions.

    ```console
    make -f builder/Makefile.tracee-container build-tracee
    ```

    !!! Note
        `BTFHUB=1` adds support to some [older kernels](https://github.com/aquasecurity/btfhub/blob/main/docs/supported-distros.md).

        ```console
        BTFHUB=1 make -f builder/Makefile.tracee-container build-tracee
        ```

## Running Generated Tracee Container Image

Tracee container is supposed to be executed through docker cmdline directly,
from the official built images. Nevertheless, during the image building process,
it may be useful to execute the recently generated container image with correct
arguments, mostly to see if the image is working.

User may execute built containers through `Makefile.tracee-container` file with
the "run" targets:

1. To run recently generated **tracee:latest** container:

    ```console
    make -f builder/Makefile.tracee-container run-tracee
    ```

    !!! note
        Tracee arguments are passed through the `ARG` variable:
        ```console
        make -f builder/Makefile.tracee-container run-tracee ARG="--help"
        ```

## Development Images

For contributors who want to test the latest changes without building from source, use the daily development images from Docker Hub.

### Available Images

```bash
# Multi-architecture (recommended)
docker pull aquasec/tracee:dev

# Architecture-specific
docker pull aquasec/tracee:x86_64-dev
docker pull aquasec/tracee:aarch64-dev

# Timestamped build
docker pull aquasec/tracee:dev-20240115-050123UTC

Image Details Copy Section

• Build Schedule: Daily at 05:00 UTC
• Source: Latest main branch
• Security: Scanned for critical vulnerabilities before publishing
• Architectures: x86_64 and ARM64

Usage Examples Copy Section

# Quick test with development image
docker run --rm -it --pid=host --privileged \
  aquasec/tracee:dev --events syscalls --output format:table

# Use in development workflows
docker run --rm --pid=host --privileged \
  -v $(pwd)/policy.yaml:/policy.yaml \
  aquasec/tracee:dev --policy /policy.yaml

Guidance Copy Section

Use development images for testing unreleased features and validating fixes. Do not use them in production environments.

To inspect build metadata:

docker inspect aquasec/tracee:dev | jq '.[0].Config.Labels'
docker run --rm aquasec/tracee:dev --version

For stable release images, see the Installation Guide.

Contributing Building Environment (`docs/contributing/building/environment.md`) Copy Section

Creating a local building environment

These instructions are meant to describe how to create a local building and
execution environment. If you would like to build tracee container(s)
image(s), read this instead.

!!! Note
A building environment will let you build and execute tracee inside a docker
container, containing all needed tools to build and execute it. If you're
using an OSX environment, for example, you can install gmake (brew install gmake) and configure such environment by using Docker.

!!! Attention
If you want to build tracee on your local machine
read this.

Quick steps Copy Chapter

1. Build tracee environment:

   bash Copy

   make -f builder/Makefile.tracee-make alpine-prepare
   make -f builder/Makefile.tracee-make alpine-shell

2. Build and execute tracee:

   bash Copy

   make clean
   make tracee
   sudo ./dist/tracee \
       --enrichment decoded-data \
       --scope comm=bash \
       --scope follow

Now, in your host's shell, execute a command. You will see all events
(except scheduler ones) being printed, in "table format", to stdout.

How to build and use the environment Copy Chapter

In order to have a controlled building environment for tracee, tracee provides
a Makefile.tracee-make file that allows you to create and use a docker container environment to build & test tracee.

There are Two different environments that are maintained for building tracee:

• Alpine
• Ubuntu

The reason for that is that Alpine Linux is based in the musl C standard library,
while the Ubuntu Linux uses glibc.

By supporting both building environments we can always be sure that the project builds (and executes) correctly in both environments.

!!! Attention
Locally created containers, called alpine-tracee-make or
ubuntu-tracee-make, share the host source code directory. This means
that, if you build tracee binary using alpine distribution, the binary
might not be compatible to the Linux distribution from your host OS.

Creating a builder environment Copy Section

• To create an alpine-tracee-make container:

  bash Copy

  make -f builder/Makefile.tracee-make alpine-prepare

• To create an ubuntu-tracee-make container:

  bash Copy

  make -f builder/Makefile.tracee-make ubuntu-prepare

Executing a builder environment Copy Section

• To execute an alpine-tracee-make shell:

  bash Copy

  make -f builder/Makefile.tracee-make alpine-shell

• To execute an ubuntu-tracee-make shell:

  bash Copy

  make -f builder/Makefile.tracee-make ubuntu-shell

Using build environment as a **make** replacement Copy Section

Instead of executing a builder shell, you may use alpine-make, or
ubuntu-make, as a replacement for the make command:

1. Create builder environment as described:
   Creating a builder environment

2. Compile tracee using ubuntu-make

   • Build tracee binary:

     bash Copy

     make -f builder/Makefile.tracee-make ubuntu-make ARG="tracee"

   • Show available ubuntu-make commands:

     bash Copy

     make -f builder/Makefile.tracee-make ubuntu-make ARG="help"

   • Remove tracee binary

     bash Copy

     make -f builder/Makefile.tracee-make ubuntu-make ARG="clean"

   • Build binaries for all

     bash Copy

     make -f builder/Makefile.tracee-make ubuntu-make ARG="all"

3. Run tracee binary

   bash Copy

   sudo ./dist/tracee

Note: the generated binary must be compatible to your host (depending on glibc version).

If you don't want to depend on the host's library versions, or you are using the alpine-make container as a replacement for make, then it's necessary to set the STATIC variable to 1 so you can run compiled binaries on your host machine:

1. Compile tracee

   bash Copy

   make -f builder/Makefile.tracee-make alpine-prepare
   STATIC=1 make -f builder/Makefile.tracee-make alpine-make ARG="all"

2. Verify the executable is static

   • Note: ldd prints the shared libraries required by an executable file
   bash Copy

   ldd dist/tracee

   text Copy

   not a dynamic executable

3. Execute the static binary from your host

   bash Copy

   sudo ./dist/tracee

Docs Detectors Api Reference (`docs/docs/detectors/api-reference.md`) Copy Section

Detector API Reference

Complete reference documentation for the Tracee Detector API. This guide documents all interfaces, structures, and features for writing custom detectors.

New to detectors? Start with the Quick Start Guide first.

Table of Contents Copy Chapter

1. Core Interfaces
2. DetectorDefinition
3. Detector Requirements
4. DetectorOutput
5. Auto-Population
6. Lifecycle Management
7. Testing
8. Best Practices

Core Interfaces Copy Chapter

EventDetector Interface Copy Section

All detectors must implement this interface:

{% raw %}

type EventDetector interface {
    // GetDefinition returns the detector's metadata and requirements
    // Called once during registration - result is cached
    GetDefinition() DetectorDefinition

    // Init is called once at startup with shared resources
    // Use this to store logger, datastores, and initialize state
    Init(params DetectorParams) error

    // OnEvent processes each matching event
    // Return one or more DetectorOutput, nil for no detection, or error
    OnEvent(ctx context.Context, event *v1beta1.Event) ([]DetectorOutput, error)
}

{% endraw %}

DetectorParams Copy Section

Provided to Init() with shared resources:

{% raw %}

type DetectorParams struct {
    Logger     Logger                 // Structured logger (zap-based)
    DataStores datastores.Registry    // Access to all datastores
    Config     DetectorConfig         // Detector-specific configuration
}

{% endraw %}

Example:
{% raw %}

func (d *MyDetector) Init(params detection.DetectorParams) error {
    d.logger = params.Logger
    d.dataStores = params.DataStores

    // Check required datastore availability
    if !params.DataStores.IsAvailable("process") {
        return errors.New("process store required but unavailable")
    }

    d.logger.Debugw("Detector initialized", "id", d.GetDefinition().ID)
    return nil
}

{% endraw %}

DetectorDefinition Copy Chapter

Complete Structure Copy Section

{% raw %}

type DetectorDefinition struct {
    // Unique identifier (required)
    ID string

    // Event and datastore requirements (required)
    Requirements DetectorRequirements

    // Output event specification (required)
    ProducedEvent v1beta1.EventDefinition

    // Default threat metadata (optional, recommended for threats)
    ThreatMetadata *v1beta1.Threat

    // Auto-population configuration (optional, recommended)
    AutoPopulate AutoPopulateFields
}

{% endraw %}

Detector ID Conventions Copy Section

Choose an appropriate ID format:

Threat Detectors (TRC-XXX):
{% raw %}

ID: "TRC-001"  // Sensitive file access
ID: "TRC-014"  // Process injection detection

{% endraw %}

Derived Events (DRV-XXX):
{% raw %}

ID: "DRV-001"  // Container lifecycle events
ID: "DRV-003"  // Hooked syscall detector

{% endraw %}

Custom/Vendor (VENDOR-XXX):
{% raw %}

ID: "ACME-001"  // Custom company detection

{% endraw %}

ProducedEvent: Event Definition Copy Section

Defines the event your detector produces:

{% raw %}

ProducedEvent: v1beta1.EventDefinition{
    Name:        "my_detection",           // Required: unique snake_case name
    Description: "Detailed description",    // Required: clear explanation
    Version:     &v1beta1.Version{         // Required: semantic version
        Major: 1,
        Minor: 0,
        Patch: 0,
    },
    Fields: []*v1beta1.EventField{         // Required: output fields schema
        {
            Name: "field_name",
            Type: "const char*",            // Type for display/docs
        },
    },
    Tags: []string{"security", "file"},    // Optional: categorization
}

{% endraw %}

Field Types (documentation only):

• const char* - String
• int, u32, u64 - Integers
• bool - Boolean
• Custom types for complex data

Example with all fields:
{% raw %}

ProducedEvent: v1beta1.EventDefinition{
    Name:        "suspicious_shell_execution",
    Description: "Detects suspicious shell command execution patterns",
    Version:     &v1beta1.Version{Major: 1, Minor: 2, Patch: 0},
    Fields: []*v1beta1.EventField{
        {Name: "command", Type: "const char*"},
        {Name: "shell_type", Type: "const char*"},
        {Name: "risk_score", Type: "int"},
        {Name: "indicators", Type: "const char*[]"},
    },
    Tags: []string{"execution", "defense-evasion"},
}

{% endraw %}

ThreatMetadata: Threat Template Copy Section

Default threat information, copied to outputs when AutoPopulate.Threat=true:

{% raw %}

ThreatMetadata: &v1beta1.Threat{
    Name:        "Short threat name",
    Description: "Detailed threat description",
    Severity:    v1beta1.Severity_MEDIUM,
    Signature: &v1beta1.ThreatSignature{
        ID:   "MITRE T1055",
        Name: "Process Injection",
    },
    Mitre: &v1beta1.MitreAttack{
        Tactic:     []string{"Defense Evasion", "Privilege Escalation"},
        Technique:  []string{"T1055"},
        SubTechnique: []string{"T1055.001"},
    },
}

{% endraw %}

Severity Levels:

• Severity_INFO - Informational events
• Severity_LOW - Low-risk threats
• Severity_MEDIUM - Moderate threats (default for most detections)
• Severity_HIGH - Serious threats requiring immediate attention
• Severity_CRITICAL - Critical threats (data exfiltration, rootkits)

Override per detection:
{% raw %}

return []detection.DetectorOutput{{
    Data: data,
    Threat: &v1beta1.Threat{
        Name:     "Critical System Compromise",
        Severity: v1beta1.Severity_CRITICAL,  // Override default MEDIUM
    },
}}, nil

{% endraw %}

Detector Requirements Copy Chapter

DetectorRequirements Structure Copy Section

Declare what your detector needs:

{% raw %}

type DetectorRequirements struct {
    // Events lists the events this detector needs to receive
    Events []EventRequirement

    // DataStores lists required datastores with their dependency types
    DataStores []DataStoreRequirement

    // Enrichments lists required event enrichment options
    Enrichments []EnrichmentRequirement

    // Architectures lists supported CPU architectures (empty = all)
    Architectures []string

    // MinTraceeVersion specifies minimum Tracee version (optional, inclusive)
    MinTraceeVersion *v1beta1.Version

    // MaxTraceeVersion specifies maximum Tracee version (optional, exclusive)
    MaxTraceeVersion *v1beta1.Version
}

{% endraw %}

EventRequirement Structure Copy Section

Declare which events your detector needs:

{% raw %}

type EventRequirement struct {
    Name             string            // Event name (required)
    DataFilters      []string          // Filter by event data (optional)
    ScopeFilters     []string          // Filter by origin (optional)
    MinVersion       *v1beta1.Version  // Minimum event version (optional)
    MaxVersion       *v1beta1.Version  // Maximum event version (optional, exclusive)
    Dependency       DependencyType    // Required vs optional (optional, default Required)
}

{% endraw %}

Basic Requirements Copy Section

{% raw %}

Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
        {Name: "security_inode_unlink"},
    },
}

{% endraw %}

DataFilters: Filtering Event Data Copy Section

Engine-level filtering - only matching events reach your detector:

{% raw %}

{
    Name: "security_file_open",
    DataFilters: []string{
        "pathname=/etc/shadow",          // Exact match
        "pathname=/etc/sudoers",          // OR condition
    },
}

{% endraw %}

Filter operators:
{% raw %}

"field=value"       // Exact match
"field=/path/*"     // Glob pattern (*, ?)
"field!=value"      // Not equal
"field>100"         // Greater than (numeric)
"field<100"         // Less than (numeric)
"field>=100"        // Greater or equal
"field<=100"        // Less or equal

{% endraw %}

Multiple filters (AND logic):
{% raw %}

DataFilters: []string{
    "pathname=/tmp/*",       // Must match /tmp/*
    "flags>0",               // AND flags > 0
}

{% endraw %}

Common patterns:
{% raw %}

// Sensitive paths
DataFilters: []string{
    "pathname=/etc/shadow",
    "pathname=/etc/passwd",
    "pathname=/root/.ssh/*",
}

// Specific flags
DataFilters: []string{
    "flags=0x80000",  // O_CLOEXEC
}

// Suspicious syscalls
DataFilters: []string{
    "syscall=ptrace",
    "syscall=process_vm_writev",
}

{% endraw %}

ScopeFilters: Filtering by Origin Copy Section

Filter by where events originated:

{% raw %}

{
    Name: "security_file_open",
    ScopeFilters: []string{
        "container=started",  // Only containers
    },
}

{% endraw %}

Scope options:
{% raw %}

"container=started"      // Only from containers
"not-container"          // Only from host (not containers)
"pid=1000"              // Specific PID
"pid!=1"                // Exclude init process

{% endraw %}

Event Version Constraints Copy Section

Require minimum event version:

{% raw %}

{
    Name: "security_bprm_check",
    MinVersion: &v1beta1.Version{
        Major: 2,
        Minor: 1,
    },
}

{% endraw %}

Required vs Optional Events Copy Section

{% raw %}

type DependencyType int

const (
    DependencyRequired DependencyType = iota  // Detector fails if event unavailable
    DependencyOptional                         // Detector works without event
)

{% endraw %}

Example:
{% raw %}

Events: []detection.EventRequirement{
    {
        Name:       "security_file_open",
        Dependency: detection.DependencyRequired,  // Must have
    },
    {
        Name:       "hooked_syscalls",
        Dependency: detection.DependencyOptional,  // Nice to have
    },
}

{% endraw %}

DataStore Requirements Copy Section

Declare datastore dependencies:

{% raw %}

type DataStoreRequirement struct {
    Name       string          // Datastore name
    Dependency DependencyType  // Required vs optional
}

{% endraw %}

Available datastores:

• process - Process tree and ancestry
• container - Container metadata
• dns - DNS cache
• system - System information
• syscall - Syscall mappings
• symbol - Kernel symbols

Example:
{% raw %}

Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    DataStores: []detection.DataStoreRequirement{
        {Name: detection.DataStoreProcess, Dependency: detection.DependencyRequired},
        {Name: detection.DataStoreContainer, Dependency: detection.DependencyOptional},
    },
}

{% endraw %}

Enrichment Requirements Copy Section

Request specific data enrichments on input events:

{% raw %}

type EnrichmentRequirement struct {
    Name       string          // Enrichment option name
    Dependency DependencyType  // Required vs optional
    Config     string          // Enrichment-specific config (optional)
}

{% endraw %}

Available enrichments:

• environment - Execution environment variables
• executable-hash - Executable file hashes (Config: "inode", "dev-inode", "digest-inode")
• container - Container metadata fields in Event struct (Name, Image, Pod info)

Example:
{% raw %}

DetectorDefinition{
    ID: "TRC-001",
    Requirements: detection.DetectorRequirements{
        Enrichments: []detection.EnrichmentRequirement{
            {
                Name:       detection.EnrichmentExecutableHash,
                Config:     detection.ExecutableHashConfigDigestInode,
                Dependency: detection.DependencyRequired,
            },
        },
    },
}

{% endraw %}

Container enrichment example:

{% raw %}

DetectorDefinition{
    ID: "TRC-CONTAINER-001",
    Requirements: detection.DetectorRequirements{
        Events: []detection.EventRequirement{
            {Name: "security_file_open"},
        },
        Enrichments: []detection.EnrichmentRequirement{
            {
                Name:       detection.EnrichmentContainer,
                Dependency: detection.DependencyRequired,
            },
        },
    },
}

{% endraw %}

This detector requires container fields to be pre-populated in the Event struct. Without --enrichment container, registration will fail.

Architecture Filtering Copy Section

Restrict detector to specific CPU architectures (rarely needed):

{% raw %}

DetectorDefinition{
    ID: "TRC-X86-001",
    Requirements: detection.DetectorRequirements{
        Architectures: []string{"amd64"},  // Only x86-64
        // ... rest of requirements
    },
}

{% endraw %}

Supported architectures:

• amd64 (x86-64)
• arm64 (AArch64)