# Technical Documentation: aquasecurity/tracee

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/aquasecurity/tracee)

### Overview (`Readme.md`)

[installation]:https://aquasecurity.github.io/tracee/latest/docs/install/
[docker-guide]:https://aquasecurity.github.io/tracee/latest/docs/install/docker/
[kubernetes-guide]:https://aquasecurity.github.io/tracee/latest/docs/install/kubernetes/
[prereqs]:https://aquasecurity.github.io/tracee/latest/docs/install/prerequisites/
[macfaq]:https://aquasecurity.github.io/tracee/latest/docs/advanced/mac/

Before moving on, please consider giving us a GitHub star ⭐️. Thank you!

## About Tracee

Tracee is a runtime security and observability tool that helps you understand how your system and applications behave.  
It is using [eBPF technology](https://ebpf.io/what-is-ebpf/) to tap into your system and expose that information as events that you can consume.  
Events range from factual system activity events to sophisticated security events that detect suspicious behavioral patterns.

To learn more about Tracee, check out the [documentation](https://aquasecurity.github.io/tracee/).

## Quickstart

To quickly try Tracee use one of the following snippets. For a more complete installation guide, check out the [Installation section][installation].  
Tracee should run on most common Linux distributions and kernels. For compatibility information see the [Prerequisites][prereqs] page.  Mac users, please read [this FAQ][macfaq].

### Using Docker

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest
```

For a complete walkthrough please see the [Docker getting started guide][docker-guide].

### On Kubernetes

```shell
helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace
```

```shell
kubectl logs --follow --namespace tracee daemonset/tracee
```

For a complete walkthrough please see the [Kubernetes getting started guide][kubernetes-guide].

## Contributing
  
Join the community, and talk to us about any matter in the [GitHub Discussions](https://github.com/aquasecurity/tracee/discussions) or [Slack](https://slack.aquasec.com).  
If you run into any trouble using Tracee or you would like to give use user feedback, please [create an issue.](https://github.com/aquasecurity/tracee/issues)

Find more information on [contribution documentation](https://aquasecurity.github.io/tracee/latest/contributing/overview/).

## More about Aqua Security

Tracee is an [Aqua Security](https://aquasec.com) open source project.  
Learn about our open source work and portfolio [here](https://www.aquasec.com/products/open-source-projects/).

---

### Overview (`brand/readme.md`)

This directory contains media assets, such as the Tracee logo. Assets under this directory are provided under the Creative Commons - BY 4.0 License. For more details, see here: https://creativecommons.org/licenses/by/4.0/

---

### Overview (`common/README.md`)

# Common Module

The `common` module provides shared utilities and types that can be used by both Tracee itself and Go signatures. This module is designed to be lightweight, self-contained, and independent of Tracee's core business logic.

## Purpose

This module serves as a foundation layer that provides:
- Common data structures and types
- Utility functions for basic operations
- Shared constants and definitions
- Helper functions that don't contain Tracee-specific logic

## Design Principles

### Self-Contained
The common module should be self-contained and not depend on other Tracee modules or packages. External dependencies are limited to well-established, minimal libraries that provide essential functionality.

### No Business Logic
This module **must not** contain any business logic specific to Tracee's core functionality. It should only provide generic utilities and types that could potentially be useful in other contexts.

### Minimal Dependencies
- Primarily uses Go standard library packages
- External dependencies are limited to well-established, minimal libraries (e.g., optimized hashing, LRU caches, structured logging, system interfaces)
- No dependencies on other Tracee modules or packages

## Usage

### In Consumer code
```go
import "github.com/aquasecurity/tracee/common"
```

## Packages

### Data Structures & Utilities
- **`bitwise`** - Bit manipulation utilities for flag operations and bitwise calculations
- **`bucketcache`** - Bucket-based caching system for efficient memory-aware data storage
- **`changelog`** - Time-ordered generic changelog data structure for tracking changes over time
- **`counter`** - Thread-safe atomic counter with overflow/underflow protection
- **`murmur`** - Generic MurMur hash functions for efficient hashing and data integrity verification
- **`interfaces`** - Generic interfaces for common patterns (cloning, iteration)
- **`set`** - Generic set data structures for efficient collection operations
- **`stringutil`** - String manipulation utilities for trimming and processing text data


### System Integration
- **`capabilities`** - Linux capabilities handling and management for process permissions
- **`cgroup`** - Control group utilities for container and process resource management
- **`environment`** - System environment detection (kernel version, OS info, CPU/memory details)
- **`mount`** - Mount point utilities for filesystem and container mount management
- **`proc`** - `/proc` filesystem utilities for process and system information extraction
- **`system`** - System monitoring utilities for resource and performance tracking

### File & Binary Analysis
- **`digest`** - File content hashing (SHA256) with caching for efficient content verification and integrity checking
- **`elf`** - ELF file analysis, magic number validation, and symbol extraction utilities
- **`fileutil`** - File operation utilities with atomic operations, directory management, and protected memory-mapped file reading
- **`sharedobjs`** - Shared object and library handling for dynamic loading and symbol resolution

### I/O & Utilities
- **`errfmt`** - Error formatting utilities for consistent error handling and display
- **`logger`** - Structured logging interface with filtering and formatting capabilities
- **`parsers`** - Comprehensive parsing utilities including:
  - System call argument parsers for Linux syscalls (flags, capabilities, socket types, etc.)
  - Network address parsers for sockaddr structures (IPv4, IPv6, Unix domain sockets)
  - File operation parsers (read/write detection, memory path identification)
- **`timeutil`** - Time-related utilities for timestamp handling and time operations

## What Belongs Here

✅ **Appropriate content:**
- Generic data structures (maps, slices utilities)
- String manipulation helpers
- Type conversion utilities  
- Mathematical helpers
- Generic algorithms
- Common constants that aren't Tracee-specific
- Basic validation functions

❌ **Inappropriate content:**
- eBPF-related logic
- Event processing functions
- Policy enforcement code
- Container runtime integrations
- Signature detection logic
- Tracee-specific configuration

## Contributing

When adding to this module, ensure that:

1. **Generic functionality** - Code should be useful beyond Tracee's specific use case
2. **Self-contained** - No imports from other Tracee packages
3. **Well documented** - All public functions and types should have clear documentation
4. **Tested** - Include unit tests for new functionality
5. **Package organization** - New code should be placed in internal packages (e.g., `common/hash/hash.go`) rather than directly under common (e.g., `common/hash.go`)

---

### Overview (`detectors/README.md`)

# Tracee Detectors

This module contains event detector implementations for Tracee's detector API.

## Detector Types

### Go Detectors (Built-in)

Go detectors in this module use automatic registration via `init()` functions:

```go
package detectors

func init() {
    register(&MyDetector{})
}

type MyDetector struct {
    // detector fields
}

// Implement detection.EventDetector interface methods
```

No manual registration list is needed - detectors are automatically discovered during package initialization.

### YAML Detectors

YAML detectors provide a declarative way to define threat detection and derived events without writing Go code.

**Example YAML detectors** are provided in the repository's `examples/detectors/yaml/` directory for reference. These are pure reference files meant to be copied and customized by users.

See the [YAML Detectors documentation](https://aquasecurity.github.io/tracee/latest/detectors/yaml-detectors/) for syntax, examples, and usage details.

## Module Structure

- `registry.go` - Auto-registration system for Go detectors
- `example_detector.go` - Comprehensive educational example (see below)
- `*.go` - Built-in Go detector implementations

## Usage

Tracee's main package imports this module and calls `GetAllDetectors()` to retrieve all registered detectors:

```go
import "github.com/aquasecurity/tracee/detectors"

allDetectors := detectors.GetAllDetectors()
// Register detectors with the detector engine
```

## Example Detector

### Comprehensive Example: `example_detector.go`

A fully-featured Go detector demonstrating all detector API capabilities:

- DataStore API usage (ContainerStore, SystemStore)
- Event filtering patterns (scope filters, data filters, version constraints)
- Conditional field population and enrichment
- Proper error handling with sentinel errors
- Auto-population features (Threat, DetectedFrom, ProcessAncestry)

**Usage:**

1. **For reference**: Read the code to learn detector patterns
2. **For testing**: Build Tracee with the example included:

```bash
# Build Tracee with example detectors included
make tracee-with-examples

# Or manually override GO_TAGS_EBPF
make tracee GO_TAGS_EBPF="core,ebpf,lsmsupport,detectorexamples"

# Run tracee and filter for the example detection
sudo ./dist/tracee -e example_detection --output json
```

**Note**: This detector is excluded from default builds using the `detectorexamples` build tag to prevent noise (it triggers on all `execve` events).

## Adding a New Detector

### Go Detector

1. Create a new file (e.g., `my_detector.go`)
2. Implement the `detection.EventDetector` interface
3. Add `init()` function calling `register(&MyDetector{})`
4. The detector will automatically be included when the package is imported

### YAML Detector

See the [YAML Detectors documentation](https://aquasecurity.github.io/tracee/latest/detectors/yaml-detectors/) for instructions on creating and deploying YAML detectors.

---

### Overview (`scripts/README.md`)

# Tracee Scripts

This directory contains utility scripts for developing, building, testing, and debugging Tracee.

## Directory Structure

### `debugging/`

Tools for debugging and analyzing Tracee components:

- **[inspect_bpf_prog.sh](debugging/inspect_bpf_prog.md)** - Inspect and measure eBPF programs in BPF object files. Supports program size measurement, assembly dumping, and comparing between builds.

### `installation/`

Scripts for installing dependencies and preparing environments:

- `install-deps-ubuntu.sh` - Install build dependencies on Ubuntu
- `install-deps-centos.sh` - Install build dependencies on CentOS
- `install-deps-alpine.sh` - Install build dependencies on Alpine Linux
- `install-clang.sh` - Install specific LLVM/Clang versions
- `prepare-ami.sh` - Prepare AWS AMI for CI/CD
- `pull-test-images.sh` - Pull container images for testing

### Core Scripts

- `btfhub.sh` - Manage BTFhub integration for CO-RE support
- `checkpatch.sh` - Run Linux kernel style checks on patches
- `system-info.sh` - Display system information for debugging
- `sync_system_time.sh` - Synchronize system time (useful in VMs)
- `tracee_start.sh` / `tracee_stop.sh` - Start/stop Tracee with common configurations
- `run_test_scripts.sh` - Run integration test scripts
- `verify_man_md_sync.sh` - Verify man pages and markdown docs are in sync

### Library Scripts

Common utilities sourced by other scripts:

**General Purpose Libraries:**
- `lib.sh` - Main library that imports all other libs
- `lib_log.sh` - Logging functions (info, error, warn, etc.)
- `lib_print.sh` - Formatted output utilities
- `lib_git.sh` - Git operations
- `lib_test.sh` - Test framework utilities
- `lib_testing.sh` - Additional test helpers
- `lib_misc.sh` - Miscellaneous utilities
- `lib_internal.sh` - Internal helper functions

**Tracee-Specific Library:**
- `tracee_common.sh` - Common functions for `tracee_start.sh` and `tracee_stop.sh`

## Usage

Most scripts include a `--help` flag for detailed usage information:

```bash
./script-name.sh --help
```

## Contributing

When adding new scripts:

1. Follow the existing structure and naming conventions
2. Add a shebang line (`#!/bin/bash`) and set executable permissions
3. Source `lib.sh` for consistent logging and utilities
4. Include usage/help documentation
5. Update this README with a brief description
6. Consider adding the script to the appropriate subdirectory

For script style guidelines, see [.cursor/skills/shell-style-guide/SKILL.md](../.cursor/skills/shell-style-guide/SKILL.md).

---

### Overview (`cmd/tracee-bench/readme.md`)

# Documentation

`tracee-bench` is a promQL based tool to query `tracee` runtime performance metrics.
It can be used to benchmark tracee's event pipeline's (see [here](https://aquasecurity.github.io/tracee/dev/architecture/)) performance on your environment.

## Enabling Prometheus

In order to use prometheus with tracee see [this](https://aquasecurity.github.io/tracee/dev/integrating/prometheus/) documentation.
A simple script for running a prometheus container scraping tracee is available in this repository in `prometheus.sh`.

## Metrics tracked

`tracee-bench` tracks three important stats about tracee's performance in your environment:1
1. Avg rate of events emitted per second
2. Avg rate of events lost per second
3. Overall events lost

Ideal performance of tracee should have a stable throughput of events emitted with minimal event loss. If heavy event loss occurs, consider tuning tracee either through [filtering](https://aquasecurity.github.io/tracee/dev/tracing/event-filtering/) or allocating additional CPU (if running on kubernetes).

---

### Overview (`deploy/grafana/README.md`)

# Using the Dashboard

A tutorial for importing this dashboard (and general usage of Tracee with Prometheus and Grafana) can be found [here](https://aquasecurity.github.io/tracee/latest/tutorials/deploy-grafana-dashboard/).

---

### Overview (`tests/detectors/README.md`)

# Detector Testing Framework

A lightweight framework for testing detector integration with the Tracee engine.

## When to Use

| Approach | Use Case |
|----------|----------|
| **Direct testing** | Unit test detector logic via `OnEvent()` directly (see `detectors/*_test.go`) |
| **TestHarness** | Integration testing with engine (policy filtering, auto-population, detector chains) |

For most detectors, direct testing is sufficient. Use TestHarness when you need to test:
- Event ID allocation and registration
- Policy filtering behavior
- Auto-population of Threat/DetectedFrom fields
- Detector chaining (output of detector A feeds detector B)

## Quick Start

### TestHarness (Engine Integration)

```go
func TestMyDetector(t *testing.T) {
    // Create harness with base events your detector consumes
    harness := detectors.NewTestHarness(t, events.Execve)

    // Register your detector
    detector := &MyDetector{}
    require.NoError(t, harness.RegisterDetector(detector))

    // Create input event and dispatch
    input := &v1beta1.Event{
        Id:   v1beta1.EventId(events.Execve),
        Name: "execve",
        Data: []*v1beta1.EventValue{
            v1beta1.NewStringValue("pathname", "/bin/nc"),
        },
    }

    outputs := harness.DispatchEvent(input)

    // Verify outputs
    require.Len(t, outputs, 1)
    harness.AssertThreatPopulated(outputs[0])
    harness.AssertDetectedFromPopulated(outputs[0], "execve")
}
```

### YAML Detector Testing

```go
func TestYAMLDetector(t *testing.T) {
    yaml := `
id: test-detector
produced_event:
  name: suspicious_exec
requirements:
  events:
    - name: sched_process_exec
      data_filters:
        - pathname=/bin/nc
output:
  fields:
    - name: binary_path
      expression: getEventData("pathname")
`
    harness := detectors.NewYAMLTestHarness(t, yaml, events.SchedProcessExec)

    input := &v1beta1.Event{
        Id:   v1beta1.EventId(events.SchedProcessExec),
        Name: "sched_process_exec",
        Data: []*v1beta1.EventValue{
            v1beta1.NewStringValue("pathname", "/bin/nc"),
        },
    }

    outputs := harness.DispatchEvent(input)
    require.Len(t, outputs, 1)
}
```

## See Also

- **Direct detector tests**: `detectors/*_test.go` - examples of testing detector logic directly
- **Detector implementations**: `detectors/*.go` - Go detector examples
- **YAML detectors**: `pkg/detectors/yaml/*.yaml` - YAML detector examples

---

### Overview (`tests/integration/syscaller/README.md`)

# syscaller

A tiny, low-noise, self-isolating **syscall-event trigger** for the tracee integration tests.

Some events are hard to produce from a normal program — either the syscall is destructive
(`reboot`, `delete_module`), or the interesting event is *derived* from a syscall that must actually
succeed (`openat` → `security_file_open`). `syscaller` sets its own `comm` (so a policy can scope by
`comm=`) and then issues exactly the syscalls a test asks for, with per-syscall argument strategies.

## Why it works

A tracee syscall-name event fires at `raw_tracepoint/sys_enter`, **on entry, before the syscall
runs and with no success check** (`pkg/ebpf/c/tracee.bpf.c`). So *calling* a syscall triggers its
event even when the call then fails harmlessly on invalid arguments. That is what lets `syscaller`
trigger destructive syscall events safely: it calls them with poison arguments that fail at
validation (`EINVAL`/`EFAULT`/`EBADF`), so the event fires but the destructive effect never lands.

## Why Zig

The tool this replaced was written in Go, and the Go runtime is noisy: just starting it costs ~900
syscalls — a storm of `clone`/`futex`/`mmap`/`rt_sig` from the scheduler and GC — versus about 12 for
a single trigger here. Worse, `PR_SET_NAME` sets the comm per *thread*, so the runtime's extra
threads kept their own comm and leaked into any `comm`-scoped policy.

This tool sidesteps all of that. It targets the raw `std.os.linux` interface — no libc, no threads,
no runtime — so it sits at the syscall-noise floor, and a comm scope sees only the syscalls the test
asked for. As a bonus, one Zig toolchain cross-compiles amd64 and arm64, so the build no longer drags
in the CGO/eBPF environment the Go tool needed (it used to `import pkg/events`).

## CLI

```
syscaller [--unshare] [--cgroup] [--fork-each] <comm> <spec>...
  <spec>       = <n>            issue syscall n with ZERO args (fires the syscall-name event) [legacy]
               = <n>:<strategy> strategy-driven args; strategy = zero | invalid | file        [new]
  --unshare    run in throwaway user/mount/pid/net/uts/ipc/cgroup namespaces (host-safe)
  --fork-each  run EVERY spec in its own fork+watchdog child (contains side effects)
```

`<n>` is the **arch-native syscall number**. The Go test side resolves an event *name* to its number
via the build-tagged `pkg/events/core_{amd64,arm64}.go` tables (`GetDefinitionByName().GetID()`), so
the tool itself does no translation. The bare `<n>` form is unchanged from the old Go tool, so the
existing `event_filters` `useSyscaller` cases (`formatCmdEvents`) keep working with no change.

## Strategies

| strategy  | args                                             | use for |
|-----------|--------------------------------------------------|---------|
| `zero`    | all-zero                                          | any syscall-name event (default) |
| `file`    | `openat`-shaped: `(AT_FDCWD, <scratch file>, O_RDONLY)` | driving *derived* events from a real, successful op (`openat` → `security_file_open`) |
| `invalid` | all-poison (`0xFFFF…`)                            | **destructive** syscalls — fires the event, fails at validation |

The scratch file used by `file` is created **before** the comm change, so its setup syscalls land
under the original comm, not the scoped one, and it is removed on exit.

## Isolation (`--unshare`)

`--unshare` runs the workload inside a throwaway set of namespaces — user, mount, pid, net, uts, ipc
and cgroup — created just for this run. It maps the caller to uid/gid 0 in the new *user* namespace
(with `deny` on setgroups), then forks so the child actually enters the new pid namespace.

The point: that mapped-root has no privilege over the host, so even a syscall whose arguments didn't
fully neuter it can't do damage. A stray mount, kill or network op is confined, and the uts/ipc/
cgroup namespaces also contain `sethostname`/`setdomainname`, IPC and cgroup ops (which matters once
the sweep triggers the whole table). It is defense-in-depth on top of the poison-argument safety —
reach for it whenever you trigger a destructive syscall with `invalid`.

tracee is a single host-wide, scope-filtered collector, so a self-unshared process **is** still
captured. Scope proof tests by `comm` / `mntns` / `pidns` — **not** `container=new`, which keys on a
runtime cgroup + enricher that a bare `unshare` does not have.

## Execution model

By default `zero` and `file` specs run inline while an `invalid` spec runs in a **forked child with a
watchdog**: the child issues the syscall and exits; the parent reaps it and `SIGKILL`s it if it does
not return within a short budget. This makes the tool robust to a syscall that (despite bad args)
blocks or ends the process — including exit-class syscalls, where only the child exits.

`--fork-each` extends the fork+watchdog to **every** spec. That is what lets the coverage sweep
trigger the whole syscall table safely: a syscall with a process-local side effect (`seccomp` strict
mode, `close`, `umask`, `setsid`, `ptrace`→`TRACEME`) is contained to a throwaway child and cannot
poison the specs that follow.

## Safety taxonomy (which strategy per syscall)

Almost every syscall event just needs `zero` (the default): it fires at `sys_enter` before the call
runs, and zero args also defuse most "dangerous" syscalls (a NULL pointer → `EFAULT`, a zero
count/flags → `EINVAL`, a zero fd is just stdin). The **exceptions** live in the Go-side
`syscallerStrategy` / `syscallerSkip` maps (`syscaller_trigger_test.go`), which are the curated,
extensible source of per-syscall knowledge:

- **`invalid`** — destructive syscalls (`reboot`, `kexec_load`/`kexec_file_load`,
  `init_module`/`finit_module`/`delete_module`, `mount`, `pivot_root`, `swapon`/`swapoff`,
  `unlinkat`), plus ones that would end or hang the tool if run inline: exit-class (`exit`,
  `exit_group`), `ptrace` (zero args = `PTRACE_TRACEME`, a real side effect), and `pause` (blocks
  forever — the watchdog kills it).
- **`file`** — `openat` (and x86_64-only `open`), to drive `security_file_open`.
- **skip** — cannot be triggered as a benign, capturable event: `execve`/`execveat` (replace the
  image), `rt_sigreturn` (only valid from a signal frame), and the tool's own fork primitives
  `clone`/`clone3`/`fork`/`vfork`. `runSyscaller` refuses these.

Unlisted syscalls already work via `zero`, so the tool needs no edit when tracee adds syscalls: the
Go side resolves any event name through tracee's own `events.Core` table.

## Staying aligned with tracee

The syscaller does not keep its own syscall list — coverage tracks tracee automatically. Two tests
keep it honest:

- **`Test_SyscallerTableDrift`** (no root): asserts every `syscallerStrategy` key is a real syscall
  event on this arch (so a tracee rename/removal or a typo fails fast), that `syscallerSkip` is a
  consistent deny-list, and that no name is in both. Cheap; runs in the normal suite.
- **`Test_SyscallerSweep`** (root; part of the integration suite): derives the corpus from
  `events.Core` (every syscall event minus the skip set) and triggers the whole table at once under
  `--unshare --fork-each`, asserting each is captured under the tool's comm. When tracee gains
  syscalls (e.g. a PR adding new syscall definitions) this covers them with **no edit here**;
  anything genuinely un-triggerable this way is added to `syscallerSkip` — the failure lists exactly
  what was missed.

### Running the sweep

The sweep is an ordinary integration test, so it runs automatically wherever the suite runs —
including PR CI (`make test-integration` → `go test ./tests/integration/...`). The cost is one extra
short-lived tracee: the whole syscall table is triggered in a single run. Like every integration test
it needs root and eBPF, so it skips on an unprivileged box.

To run just it locally:

```
sudo make test-integration TEST=Test_SyscallerSweep
```

When it fails it lists the syscall events it could not capture; add any that are genuinely
un-triggerable this way to `syscallerSkip`.

## Build

The Makefile builds it as part of the integration flow (it is a prerequisite of
`test-integration`):

```
make dist/syscaller
```

This requires **Zig 0.16** (`scripts/installation/install-zig.sh`, pinned + checksum-verified; the
Makefile's `.checkver_zig` enforces the exact minor). For local development:

```
zig build            # -> zig-out/bin/syscaller
zig build test       # pure arg/strategy unit tests (no root, no syscalls issued)
```

## Using it from Go tests

- Legacy: `formatCmdEvents` (in `event_filters_test.go`) turns a `<comm>` + expected event names into
  `dist/syscaller <comm> <n>...` — unchanged.
- New: `runSyscaller` / `syscallerStrategy` (in `syscaller_trigger_test.go`) emit `<n>:<strategy>` and
  add `--unshare`/`--fork-each`. `Test_SyscallerTrigger` is the end-to-end proof (a destructive batch
  via `--unshare`+`invalid`; `openat` via `file` driving `security_file_open`); `Test_SyscallerSweep`
  is the opt-in full-table coverage sweep; `Test_SyscallerTableDrift` is the no-root table guard.

## Source

- `src/main.zig` — the whole tool.
- `build.zig` — `zig build` / `zig build test` (the Makefile builds `src/main.zig` directly).

---

### Index (`docs/index.md`)

---
hide:
- toc
---


👋 Welcome to Tracee Documentation! To help you get around, please notice the different sections at the top global menu:

- You are currently in the [Getting Started](index.md) section where you can find general information and help with first steps.
- In the [Tutorials](tutorials/overview.md) section you can find step-by-step guides that help you accomplish specific tasks.
- In the [Docs](docs/overview.md) section you can find the complete reference documentation for all of the different features and settings that Tracee has to offer.
- In the [Contributing](contributing/overview.md) section you can find technical developer documentation and contribution guidelines.


[installation]:./docs/install/index.md
[docker-guide]:./docs/install/docker.md
[kubernetes-guide]:./docs/install/kubernetes.md
[prereqs]:./docs/install/prerequisites.md
[macfaq]:./docs/install/mac-faq.md


Before moving on, please consider giving us a GitHub star ⭐️. Thank you!
 
## About Tracee

Tracee is a runtime security and observability tool that helps you understand how your system and applications behave using [eBPF technology](https://ebpf.io/what-is-ebpf/). It provides deep visibility into Linux systems by monitoring system calls, network activity, and file operations in real-time.

### What Tracee Does

- **🔍 System Monitoring**: Tracks system calls, process execution, file operations, and network activity
- **🛡️ Security Detection**: Identifies suspicious behavior patterns and potential security threats
- **📊 Observability**: Provides detailed insights into application and system behavior
- **🚨 Real-time Alerts**: Generates events for immediate threat detection and response

### Key Features

- **Zero Code Changes**: Monitor existing applications without modification
- **Low Overhead**: Minimal performance impact using efficient eBPF programs
- **Container Aware**: Native support for containerized environments and Kubernetes
- **Flexible Policies**: Customize what to monitor and how to respond to events
- **Rich Event Data**: Detailed context including process lineage, file paths, and network connections

### Use Cases

- **Security Monitoring**: Detect malware, privilege escalation, and suspicious activity
- **Compliance**: Monitor file access, data exfiltration, and system changes
- **Troubleshooting**: Debug application issues and system behavior
- **Forensics**: Investigate security incidents with detailed audit trails

## Quickstart

To quickly try Tracee use one of the following snippets. For a more complete installation guide, check out the [Installation section][installation].  
Tracee should run on most common Linux distributions and kernels. For compatibility information see the [Prerequisites][prereqs] page. Mac users, please read [this FAQ][macfaq].

### Using Docker

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest
```

For a complete walkthrough please see the [Docker getting started guide][docker-guide].

### On Kubernetes

```shell
helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace
```

```shell
kubectl logs --follow --namespace tracee daemonset/tracee
```

For a complete walkthrough please see the [Kubernetes getting started guide][kubernetes-guide].

## Next Steps

After trying the quickstart, here's how to dive deeper:

### 🎯 For Security Analysts

- **[Events Documentation](docs/events/index.md)**: Learn about security events and signatures
- **[Policies Guide](docs/policies/index.md)**: Create custom detection rules
- **[Kubernetes Tutorial](tutorials/k8s-policies.md)**: Monitor containerized workloads

### 🛠️ For DevOps Engineers

- **[Installation Guide](docs/install/index.md)**: Production deployment options
- **[Configuration](docs/install/config/index.md)**: Customize Tracee for your environment
- **[Outputs](docs/outputs/index.md)**: Integrate with your monitoring stack

### 👨‍💻 For Developers

- **[Custom Events](docs/events/custom/overview.md)**: Create your own detection logic
- **[Contributing Guide](contributing/overview.md)**: Help improve Tracee
- **[Building from Source](contributing/building/building.md)**: Development setup

## Contributing
  
Join the community, and talk to us about any matter in the [GitHub Discussions](https://github.com/aquasecurity/tracee/discussions) or [Slack](https://slack.aquasec.com).  
If you run into any trouble using Tracee or you would like to give us user feedback, please [create an issue.](https://github.com/aquasecurity/tracee/issues)

Find more information on [contribution documentation](contributing/overview.md).

## More about Aqua Security

Tracee is an [Aqua Security](https://aquasec.com) open source project.  
Learn about our open source work and portfolio [here](https://www.aquasec.com/products/open-source-projects/).

---

### Contributing Overview (`docs/contributing/overview.md`)

# Contributing

Thank you for taking interest in contributing to Tracee! This document covers our working practices and conventions.

## Issues and Discussions

We encourage open discussion and collaboration using both GitHub Issues and Discussions.  

- [Discussions](https://github.com/aquasecurity/tracee/discussions) are a free-style conversational tool.
- [Issues](https://github.com/aquasecurity/tracee/issues) are a project management tool, we use them to keep track of who's working on what and plan ahead.

If you have a suggestion, question, or a general comment - please use Discussions. If there's a clear work item (including bugs) - you can open an Issue.

## Discussions

- We have the following discussion topics:

    1. [Announcements](https://github.com/aquasecurity/tracee/discussions/categories/announcements): One way communication from the team to the community. Consider this like our mini blog.
    1. [Questions and Help](https://github.com/aquasecurity/tracee/discussions/categories/questions-and-help): For help and support. Consider this similar to StackOverflow.
    1. [Development](https://github.com/aquasecurity/tracee/discussions/categories/development): For discussing potential features, and collaborating on their design.

## Issues

1. **Actionable and Assignable:** Every issue must be clear, actionable, and assignable to a specific person.  Break down large issues into smaller, more manageable tasks.
2. **Clear and Definitive Descriptions:** Be precise in your issue descriptions:

    - **Bug Reports:** Include the following:
        - Detailed steps to reproduce the bug.
        - The complete error message.
        - Your operating system, Tracee version, and any other relevant environment details.
    - **Feature Requests:** Define:
        - A clear scope for the feature.
        - Specific acceptance criteria that will be used to determine if the feature is complete.
3. **Issue Labels:**

    - `area/...` (e.g., `ebpf`, `events`): Area of Tracee affected.
    - `kind/...` (e.g., `bug`, `feature`, `documentation`): Type of issue or pull request.
    - `priority/...`: Urgency of the issue.
    - `milestone/...` (e.g., `milestone/v0.x.y`): Target release; add when you want the item considered for a specific version (issues and pull requests).
    - `good-first-issue`: Suitable for first-time contributors.

4. **Issue Assignment:** Self-assign issues or request assignment. Don't work on an issue assigned to someone else without their consent.

## Submitting Changes

External contributors must fork the repository and submit changes as pull
requests from their fork. Branch creation in this repository is restricted.

## Pull Requests

1. Every Pull Request should have an associated Issue unless it is a trivial fix.
2. When adding a flag option or other UX related change, make sure the design is explicitly described in the associated issue, and a maintainer approved it.
3. Pull requests require all review threads resolved and at least one approving
   review. When the change touches code-owner areas, at least two approving
   reviews are required.
4. Code related information should be in the commit message, review related information should be in the PR description.
5. For changes that span different areas please try to make each change self contained and independent.

## Commits

All commits must be **signed** (GPG, SSH, or S/MIME). See
[GitHub's signing guide](https://docs.github.com/en/authentication/managing-commit-signature-verification)
to set up commit signing.

Commit messages must follow the
[Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
specification:

```
type(optional scope): description

optional body
```

Valid types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`,
`build`, `ci`, `chore`, `revert`.

The commit subject should be max 50 characters, written in imperative mood.
The optional body (separated by an empty line) may explain **why** the change
was made. Wrap at 72 characters.

### Breaking Changes

Breaking changes must include `!` after the type/scope and a `BREAKING CHANGE`
footer in the commit body explaining what changed:

```
feat(ebpf)!: remove deprecated event field

BREAKING CHANGE: The `oldField` field has been removed from the
`anti_debugging` event. Use `newField` instead.
```

The `BREAKING CHANGE` footer is required so that the change is clearly
documented for downstream users and automated tooling.

## Development and Testing

### Development Images

For testing the latest changes without building from source, Tracee provides daily development images:

```bash
# Get the latest development build
docker pull aquasec/tracee:dev

# Test your changes quickly
docker run --rm -it --pid=host --privileged aquasec/tracee:dev --version
```

Development images are built daily from the `main` branch and include the latest features and fixes. See [Building Documentation](building/building.md#development-images) for complete details.

### Development Workflows

**Quick Start for Development:**
```bash
# 1. Check your environment
make env

# 2. Build everything
make all

# 3. Run tests
make test-unit

# 4. Check code quality before submitting PR
make check-pr
```

**Common Development Tasks:**
```bash
# Development builds
make tracee                # Build the main tracee binary
make bpf                   # Build just the eBPF object
make all                   # Build all components

# Testing
make test-unit             # Run unit tests with coverage
make test-integration      # Run integration tests

# Performance and stress testing
make evt                   # Build the event generator tool
make evt-trigger-runner    # Build stress testing container
```

For detailed information about code quality checks, dependencies, and troubleshooting, see our [Code Quality Guide](checkpatch.md).

For performance testing and stress testing with the `evt` tool, see [evt - Event Generator and Stress Testing Tool](evt-tool.md).

For complete build options and development environment setup, run `make help` or see [Building Documentation](building/building.md).

## Code

1. Follow Golang's code review standards: [https://github.com/golang/go/wiki/CodeReviewComments](https://github.com/golang/go/wiki/CodeReviewComments)
2. Follow `gofmt` + `govet` + `goimports` formatting.
3. Tests should be included alongside code changes wherever applicable, except for parts that are harder to test and are not currently tested (e.g. eBPF). When modifying already tested code, your changes must be represented in the existing tests.

## Contributing Code

### Adding New Event Tracing

If you're looking to add new event tracing capabilities to Tracee, see our comprehensive guide: [Adding New Event Tracing](adding-events.md).

This guide covers:

- Event definition in Go code
- eBPF probe configuration
- Implementation patterns and best practices
- Testing and troubleshooting

---

### Docs Overview (`docs/docs/overview.md`)

# Docs

In this section you can find the complete reference documentation for all of the different features and settings that Tracee has to offer.

## Why Choose Tracee?

Tracee stands out from other security and observability tools through several key differentiators:

### 🎯 Everything is an Event
Unlike tools that separate raw data from detections, Tracee presents **everything as events** - from low-level system calls to high-level security detections. This unified approach lets you:

- Combine different event types in the same policy
- Build complex detection logic across multiple data sources
- Maintain a consistent view of your system's behavior

### 📊 Rich Event Coverage with Minimal Effort
Tracee provides extensive built-in events out of the box:

- **400+ system calls** for comprehensive system monitoring
- **Network events** including DNS, HTTP, and packet analysis
- **Security events** with pre-built threat detection signatures
- **Container events** with native Kubernetes integration

### 🛠️ Simplicity without Compromise
Create powerful policies with just a few lines of YAML:

- **Intuitive syntax** for scoping and filtering
- **Flexible targeting** from global to container-specific
- **Easy deployment** across development and production environments

### 🔍 Forensic Capabilities
Go beyond detection with artifact collection:

- **Network traffic capture** for detailed analysis
- **Binary collection** for malware investigation
- **Memory dumps** for advanced forensics
- **File artifacts** for compliance and auditing

### 🔗 Unified Architecture
Everything works together seamlessly:

- Events flow through the same processing pipeline
- Policies can reference any combination of event types
- Custom signatures integrate naturally with built-in events
- Single configuration controls the entire system

### 🔒 Clear Security Model
Tracee provides transparency about its capabilities and limitations:

- **Strong protection** against userspace adversaries through eBPF's tamper-resistant architecture
- **Best-effort detection** of kernel-level threats with rootkit detection signatures
- **Clear boundaries** documented to set proper expectations
- See the [Security Model](security-model.md) for detailed information about Tracee's adversary model and security guarantees

---

👈 Please use the side-navigation on the left in order to browse the different topics.

---

### Traceectl Index (`docs/traceectl/index.md`)

# traceectl - Client Service for Tracee

**traceectl** is a command-line interface (CLI) tool designed as a client service for [Tracee](https://github.com/aquasecurity/tracee), Aqua Security's open-source runtime security solution. Tracee provides real-time, powerful observability for Linux environments by monitoring system calls, events, and more. traceectl is built to simplify interactions with Tracee, making it easier for users to manage, monitor, and gather security insights.

## Overview

traceectl acts as a controller for Tracee, allowing users to:

- **Stream Events**: Continuously stream security events from Tracee, with options to format the output as JSON, tables, or custom templates.
- **List Available Events**: Display the available events that Tracee can capture, providing essential insights into runtime activities.
- **Query Metrics**: Access various metrics related to Tracee, including event counts, errors, and more.

## Installation and Usage

To get started with traceectl, go over the [Installation and Usage page](./usage.md)

---

### Tutorials Overview (`docs/tutorials/overview.md`)

# Tutorials

In this section you can find step-by-step guides that help you accomplish specific tasks.

👈 Please use the side-navigation on the left in order to browse the different topics.

If there are any specific tutorials that you would like to see please let us know on the [Aqua Open Source Slack](https://slack.aquasec.com/).

---

### Docs Detectors Index (`docs/docs/detectors/index.md`)

# Detector Documentation

Detectors are the modern way to write custom threat detection and event derivation logic in Tracee.

## What are Detectors?

Detectors analyze runtime events to identify security threats and derive higher-level events from raw eBPF data.

**Two ways to create detectors:**

1. **YAML Detectors**: Declarative configuration - no coding required
2. **Go Detectors**: Full programmatic control with direct API access

Both provide:

- **Declarative filtering**: Engine-level event filtering by data, scope, and version
- **Auto-enrichment**: Automatic population of threat metadata and process ancestry
- **Rich context**: Access to process trees, containers, and more
- **Built-in observability**: Prometheus metrics and structured logging

## Documentation Guide

### 🚀 For Newcomers

**Start here if you're new to Tracee detectors:**

### [YAML Detectors Guide](yaml-detectors.md)

Learn how to create detectors using declarative YAML configuration:

- Quick start with examples
- Complete schema reference
- Event filtering and data extraction
- Threat metadata and auto-population
- Deployment and best practices
- Troubleshooting guide

**Start here** if you want to create detectors without writing Go code.

---

### Go Detectors

Complete guide to writing Go detectors, from quick start to advanced features:

#### [Quick Start Guide](quickstart.md)
**Get your first detector running in 30 minutes**

Step-by-step tutorial to build a working detector:

- Create a detector file
- Build and run Tracee
- Trigger and observe detections
- Understand auto-registration, filtering, and enrichment

Perfect for: First-time detector developers, hands-on learners

---

### 📚 For Reference

**Comprehensive API documentation:**

#### [Detector API Reference](api-reference.md)
**Complete detector API specification**

Everything you need to know about the detector API:

- Core interfaces and structures
- Event requirements and filtering
- Auto-population features
- Advanced features and lifecycle management
- Testing patterns and best practices
- Migration from old signatures
- Troubleshooting common issues

Perfect for: Understanding all capabilities, deep dives, migrating from signatures

#### [DataStore API Reference](datastore-api.md)
**Complete datastore API specification**

Query system state from your detectors:

- ProcessStore (process information and ancestry)
- ContainerStore (container and Kubernetes metadata)
- SystemStore, SyscallStore, KernelSymbolStore, DNSStore
- Health monitoring and error handling

Perfect for: Detectors that need to query system state and context

---

## Documentation at a Glance

| Document | Purpose | Audience |
|----------|---------|----------|
| [YAML Detectors](yaml-detectors.md) | Declarative detector creation without code | Security analysts, operators |
| [Quick Start](quickstart.md) | Hands-on tutorial, first Go detector | Go developers |
| [API Reference](api-reference.md) | Complete detector API + migration + troubleshooting | Go developers |
| [DataStore API](datastore-api.md) | Complete datastore API docs | Advanced Go developers |

**Total reading time to first detector**:

- **YAML**: ~15 minutes
- **Go**: ~30 minutes (Quick Start only)

## Resources

### Examples
- **YAML Examples**: `examples/detectors/yaml/` directory - ready-to-use YAML detector examples
- **Go Quick Start**: [Quick Start Guide](quickstart.md) - complete annotated walkthrough
- **Go Comprehensive Example**: `detectors/example_detector.go` - demonstrates all detector API features including DataStore usage, filtering patterns, and enrichment (build with `make tracee-with-examples`)
- **Production Detectors**: Browse `detectors/` directory for real implementations
- **Migration Examples**: [API Reference](api-reference.md#migration-from-signatures) - signature → detector

### API Definitions (Go Detectors)
- **Detector Interfaces**: `api/v1beta1/detection/detector.go`
- **DataStore Interfaces**: `api/v1beta1/datastores/interfaces.go`
- **Event Protobuf**: `api/v1beta1/event.proto`

### YAML Schema
- **Schema Reference**: See [YAML Detectors Guide](yaml-detectors.md) for complete schema
- **Examples**: `examples/detectors/yaml/` directory

### Migration from Signatures
Existing signatures can be migrated to the new detector API. See [Migration Guide](api-reference.md#migration-from-signatures) for:

- Side-by-side API comparison
- Step-by-step migration instructions
- Pattern translations and examples
- Complete migration checklist

## Community and Support

- **Tracee GitHub**: https://github.com/aquasecurity/tracee
- **Issues**: Report bugs and request features
- **Discussions**: Ask questions and share detectors
- **Documentation**: https://aquasecurity.github.io/tracee/

---

**Ready to start?**
- **YAML detectors**: Jump to the [YAML Detectors Guide](yaml-detectors.md)
- **Go detectors**: Jump to the [Quick Start Guide](quickstart.md) or dive into the [API Reference](api-reference.md)

---

### Docs Detectors Quickstart (`docs/docs/detectors/quickstart.md`)

# Detector Quick Start Guide

Get your first Tracee detector running in 30 minutes. This hands-on tutorial assumes basic Go knowledge but no prior Tracee experience.

## What You'll Build

A detector that identifies when processes access sensitive system files like `/etc/shadow` or `/etc/sudoers`, with:

- Automatic threat metadata enrichment
- Full process ancestry (parent → grandparent → ...)
- Container context (if applicable)

## Prerequisites

```bash
# Clone Tracee
git clone https://github.com/aquasecurity/tracee
cd tracee

# Verify build environment
make env

# Required: Linux kernel 4.18+, Go 1.24+, Clang 12+
```

## Step 1: Create Your Detector File

Create `detectors/sensitive_file_access.go`:

{% raw %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endraw %}

**Helper Functions**: For cleaner code, use `detection.Detected()` for simple detections with no extra data, or `detection.DetectedWithData(data)` when including additional event values.

## Step 2: Build Tracee

```bash
# Build with your detector included
make tracee

# The detector is compiled in - no plugins or manual registration needed!
```

## Step 3: Run Tracee

```bash
# Start Tracee with process tree and JSON output
sudo ./dist/tracee --stores process --output json | jq
```

## Step 4: Trigger Your Detector

In a third terminal, access a sensitive file:

```bash
sudo cat /etc/shadow
```

## Step 5: See the Detection

You should see output like:

```json
{
  "id": 5001,
  "name": "sensitive_file_access",
  "version": "1.0.0",
  "timestamp": "2025-12-16T10:30:45.123Z",
  "threat": {
    "name": "Sensitive File Access",
    "description": "A process attempted to access a sensitive system file",
    "severity": "MEDIUM"
  },
  "data": {
    "file_path": "/etc/shadow",
    "executable": "/bin/cat"
  },
  "detected_from": {
    "event_id": 257,
    "event_name": "security_file_open"
  },
  "workload": {
    "process": {
      "entity_id": 12345,
      "pid": 67890,
      "executable": {"path": "/bin/cat"},
      "ancestors": [
        {"entity_id": 12344, "pid": 67889, "executable": {"path": "/bin/bash"}},
        {"entity_id": 12343, "pid": 67888, "thread": {"name": "sshd"}},
        {"entity_id": 1, "pid": 1, "thread": {"name": "systemd"}}
      ]
    }
  }
}
```

## What Just Happened?

Let's break down the magic:

### 1. Auto-Registration

{% raw %}
```go
func init() {
	register(&SensitiveFileAccess{})
}
```
{% endraw %}

Your detector registered itself automatically. No manual list maintenance!

### 2. Engine Filtering

{% raw %}
```go
DataFilters: []string{
	"pathname=/etc/shadow",
	"pathname=/etc/sudoers",
},
```
{% endraw %}

Tracee's engine filtered millions of `security_file_open` events. Only those matching your paths reached `OnEvent()`. This happens in the engine - super efficient!

### 3. Type-Safe Extraction

{% raw %}
```go
pathname, found := v1beta1.GetData[string](event, "pathname")
```
{% endraw %}

Generic type parameter ensures compile-time type safety. No runtime casting errors!

### 4. Auto-Enrichment

{% raw %}
```go
AutoPopulate: detection.AutoPopulateFields{
	Threat:          true,
	DetectedFrom:    true,
	ProcessAncestry: true,
}
```
{% endraw %}

The engine automatically:

- ✅ Copied `ThreatMetadata` to `output.Threat`
- ✅ Set `output.DetectedFrom` pointing to the triggering event
- ✅ Queried the process tree and populated 5 ancestor levels
- ✅ Preserved timestamp, workload, and policies from input event

You just returned simple data fields - the engine did the heavy lifting!

## Common Patterns

### Multiple Outputs

A single input can produce multiple detections:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	var outputs []detection.DetectorOutput
	
	// Check condition 1
	if isSuspicious {
		outputs = append(outputs, detection.DetectorOutput{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("reason", "suspicious pattern"),
			},
		})
	}
	
	// Check condition 2
	if isMalicious {
		outputs = append(outputs, detection.DetectorOutput{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("reason", "known malware signature"),
			},
		})
	}
	
	return outputs, nil
}
```
{% endraw %}

### Custom Threat Severity

Override the default threat metadata per detection:

{% raw %}
```go
return []detection.DetectorOutput{{
	Data: []*v1beta1.EventValue{
		v1beta1.NewStringValue("file", pathname),
	},
	Threat: &v1beta1.Threat{
		Name:        "Critical System File Access",
		Severity:    v1beta1.Severity_CRITICAL,  // Override default MEDIUM
		Description: "Root accessed critical system file",
	},
}}, nil
```
{% endraw %}

### Using DataStores

Access system state like process information:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	entityId := event.GetWorkload().GetProcess().GetEntityId()
	
	// Query process store for process details
	proc, err := d.dataStores.Processes().GetProcess(entityId)
	if errors.Is(err, datastores.ErrNotFound) {
		// Process not in tree yet
		return nil, nil
	}
	if err != nil {
		return nil, err
	}
	
	// Use process info in detection logic
	if proc.GetInterpreter().GetPath() == "/bin/bash" {
		return []detection.DetectorOutput{{
			Data: []*v1beta1.EventValue{
				v1beta1.NewStringValue("shell", "bash"),
			},
		}}, nil
	}
	
	return nil, nil
}
```
{% endraw %}

### Filtering by Container

Only detect events from containers:

{% raw %}
```go
Requirements: detection.DetectorRequirements{
	Events: []detection.EventRequirement{
		{
			Name: "security_file_open",
			ScopeFilters: []string{"container=started"},  // Containers only
		},
	},
}
```
{% endraw %}

Or only from host:

{% raw %}
```go
ScopeFilters: []string{"not-container"},  // Host only
```
{% endraw %}

### Accessing Container Information

Detectors can access container information using the `container` enrichment requirement.

**Container enrichment is required** for both approaches below - it enables Tracee to query container runtimes and populate metadata.

#### Option 1: Use Pre-Populated Event Fields

{% raw %}
```go
Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    Enrichments: []detection.EnrichmentRequirement{
        {
            Name:       detection.EnrichmentContainer,
            Dependency: detection.DependencyRequired,
        },
    },
}

func (d *MyDetector) OnEvent(event *v1beta1.Event) {
    // Container fields are pre-populated in the event
    containerName := event.GetWorkload().GetContainer().GetName()
    containerImage := event.GetWorkload().GetContainer().GetImage().GetName()

    if strings.Contains(containerImage, "malicious") {
        // detect...
    }
}
```
{% endraw %}

**Pros**:

- Simple, direct field access

**Cons**:

- Cannot handle missing/not-found containers gracefully

#### Option 2: Query Container Datastore

{% raw %}
```go
Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    Enrichments: []detection.EnrichmentRequirement{
        {
            Name:       detection.EnrichmentContainer,
            Dependency: detection.DependencyRequired,
        },
    },
    DataStores: []detection.DataStoreRequirement{
        {
            Name:       detection.DataStoreContainer,
            Dependency: detection.DependencyRequired,
        },
    },
}

func (d *MyDetector) OnEvent(event *v1beta1.Event) {
    // Container ID is always available
    containerID := event.GetWorkload().GetContainer().GetId()
    if containerID == "" {
        return nil, nil  // Not a container event
    }

    // Explicitly query datastore for metadata
    containerInfo, err := d.dataStores.Containers().GetContainer(containerID)
    if errors.Is(err, datastores.ErrNotFound) {
        // Handle not-found case (e.g., short-lived container)
        return nil, nil
    }
    if err != nil {
        return nil, err
    }

    if strings.Contains(containerInfo.Image, "malicious") {
        // detect...
    }
}
```
{% endraw %}

**Pros**:

- Flexible error handling, access to additional datastore methods

**Cons**:

- Slightly more verbose

**Both approaches require** `--enrichment container` to be enabled. The enrichment flag tells Tracee to query container runtimes and populate both Event fields and the datastore.

User must run: `tracee --enrichment container --detector my-detector`

## Debugging Tips

### Enable Debug Logging

```bash
# Run with debug logs
sudo ./dist/tracee --logging debug --stores process
```

### Check Detector Registration

```bash
# List detector events
sudo ./dist/tracee list | grep detectors
```

### Verify Event Filtering

Check that your event name is correct:

```bash
# List available events
sudo ./dist/tracee list | grep security_file_open
```

### Test Without Filters

Temporarily remove `DataFilters` to see all events:

{% raw %}
```go
Requirements: detection.DetectorRequirements{
	Events: []detection.EventRequirement{
		{
			Name: "security_file_open",
			// DataFilters: []string{...},  // Commented out for testing
		},
	},
}
```
{% endraw %}

Add logging in `OnEvent()`:

{% raw %}
```go
func (d *SensitiveFileAccess) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	d.logger.Debugw("Event received", "event", event.Name)
	// ... rest of logic
}
```
{% endraw %}

## Next Steps

Congratulations! You've written your first Tracee detector. Now explore:

### 📚 Deep Dive into APIs
- **[API Reference](api-reference.md)**: Complete detector API documentation
- **[DataStore API](datastore-api.md)**: Query process trees, containers, DNS, and more

### 🔄 Migration and Troubleshooting
- **[Migration Guide](api-reference.md#migration-from-signatures)**: Migrating from old signatures
- **[Troubleshooting](api-reference.md#troubleshooting)**: Common issues and solutions

### 🔍 Study Real Examples
- Browse `detectors/` directory for production implementations
- See `api/v1beta1/detection/detector.go` for interface definitions

### 🧪 Write Tests
- Add unit tests in `detectors/sensitive_file_access_test.go`
- See [Testing Guide](api-reference.md#testing) for patterns

## Troubleshooting

### Detector Not Running

**Problem**: Your detector code changed but behavior didn't

**Solution**: Rebuild Tracee - detectors are compiled in, not dynamically loaded
```bash
make tracee
```

### No Events Received

**Problem**: `OnEvent()` never called

**Possible causes**:

1. Wrong event name - check `tracee list`
2. Filters too restrictive - temporarily remove them
3. Event not enabled - Tracee enables events based on requirements automatically

### Process Ancestry Empty

**Problem**: `workload.process.ancestors` is null

**Solution**: Enable process tree:
```bash
sudo ./dist/tracee --stores process
```

### Build Errors

**Problem**: Compilation fails

**Common fixes**:
```bash
# Update dependencies
make go-tidy

# Clean build
make clean
make tracee
```

## Quick Reference

### Detector Skeleton

{% raw %}
```go
package detectors

import (
	"context"

	"github.com/aquasecurity/tracee/api/v1beta1"
	"github.com/aquasecurity/tracee/api/v1beta1/datastores"
	"github.com/aquasecurity/tracee/api/v1beta1/detection"
)

func init() {
	register(&MyDetector{})
}

type MyDetector struct {
	logger     detection.Logger
	dataStores datastores.Registry
}

func (d *MyDetector) GetDefinition() detection.DetectorDefinition {
	return detection.DetectorDefinition{
		ID: "TRC-XXX",
		Requirements: detection.DetectorRequirements{
			Events: []detection.EventRequirement{{Name: "event_name"}},
		},
		ProducedEvent: v1beta1.EventDefinition{
			Name:    "my_detection",
			Version: &v1beta1.Version{Major: 1},
		},
		AutoPopulate: detection.AutoPopulateFields{
			Threat:          true,
			DetectedFrom:    true,
			ProcessAncestry: true,
		},
	}
}

func (d *MyDetector) Init(params detection.DetectorParams) error {
	d.logger = params.Logger
	d.dataStores = params.DataStores
	return nil
}

func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
	// Your detection logic
	return []detection.DetectorOutput{{
		Data: []*v1beta1.EventValue{},
	}}, nil
}
```
{% endraw %}

### Essential Commands

```bash
# Build
make tracee

# Run with process tree
sudo ./dist/tracee --stores process

# View detections in JSON
sudo ./dist/tracee --output json | jq

# List detector events
sudo ./dist/tracee list | grep detectors

# Debug mode
sudo ./dist/tracee --logging debug
```

---

## Next Steps

### Learn More Advanced Patterns

For a comprehensive example demonstrating all detector API features, see `detectors/example_detector.go`:

- DataStore API usage (ContainerStore, SystemStore)
- Event filtering patterns (scope, data, version constraints)
- Conditional field population
- Advanced error handling
- Proper enrichment patterns

To build and test it:

```bash
# Build Tracee with example detectors
make tracee-with-examples

# Run tracee and filter for the example detection
sudo ./dist/tracee -e example_detection --output json
```

**Note**: The example detector is excluded from default builds (via build tag) to prevent noise in production environments.

---

**Ready for more?** Continue to the [API Reference](api-reference.md) for comprehensive documentation, including migration guides and troubleshooting.

---

### Docs Events Index (`docs/docs/events/index.md`)

# Events

Events are the core of how Tracee works. Whether you're monitoring system calls, network activity, or security threats, Tracee treats everything as events that you can filter, combine, and act upon in your policies.

### Event Categories

Tracee provides rich built-in events across six main categories:

* **syscalls** - System call monitoring
* **network** - Network activity and protocol analysis
* **security** - Security-focused detections and signatures
* **lsm** - Linux Security Module hooks
* **containers** - Container lifecycle and metadata
* **misc** - Additional system events and utilities

This section documents all of the different events that Tracee exposes.

## Configuring Tracee Events

Events are defined in the [Policy](../policies/index.md) YAML manifest. 

Tracing the `execve` events in a [policy](../policies/index.md):

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-policy
  annotations:
    description: traces execve events
spec:
  scope:
    - global
  rules:
    - event: execve
```

If no event is passed with [filters] or [policies], tracee will start with a set of default events.

Please head over to the [Tracee usage](../policies/usage/kubernetes.md) documentation for more information on configuring events.

### Event Sets

Event sets are predefined groups of related events. Instead of listing events individually, you can use sets like `syscalls` (all system calls), `fs` (file system operations), or `net` (network events).

**Discover available sets:**
```bash
tracee list --wide
```

**Use in policies:**
```yaml
rules:
  - event: tag=syscalls     # All system calls
  - event: tag=fs           # File system events
```

**Use on command line:**
```bash
tracee --events syscalls,net
```

!!! tip "Tracing All System Calls"
    Use `--events syscalls` to trace all system calls without listing them individually.

## Related Topics

* [Policies Documentation](../policies/index.md) - Learn how to create policies that use events
* [Custom Signatures](./custom/overview.md) - Build your own detection logic using Go
* [Troubleshooting](../troubleshooting.md) - Solutions for event-related issues

## Video Content

If you are curious to learn more about the Tracee Events architecture and related decision making, then have a look at the following video Q&A:

Everything is an Event in Tracee
  [](https://www.youtube.com/live/keqVe4d71uk?si=OTbVxgWsFBtdqEMW)

---

### Docs Install Index (`docs/docs/install/index.md`)

# Installing Tracee

This guide walks you through installing and setting up Tracee in your environment.

## 📋 Before You Start

### System Requirements

Before installing Tracee, ensure your system meets the necessary requirements:

- **[Prerequisites](./prerequisites.md)** - Operating system, kernel version, BTF, and capability requirements
- **[Kernel Symbols](./kernel-symbols.md)** - Kernel symbol table details and configuration
- **[OS Requirements](./os-requirements.md)** - OS release file and kernel config requirements
- **[Capabilities](./capabilities.md)** - Running Tracee with proper Linux capabilities
- **[LSM BPF Support](./lsm-support.md)** - Linux Security Module BPF support (optional)
- **[Mac Users FAQ](./mac-faq.md)** - Running Tracee on macOS (spoiler: you'll need a Linux VM or container)

!!! tip "Quick Compatibility Check"
    Tracee requires Linux kernel 5.4+ (or 4.18 for RHEL 8) with BTF support. Check `/sys/kernel/btf/vmlinux` exists and `/proc/kallsyms` is available.

## 🚀 Installation Methods

### Download Options

Tracee is available through multiple distribution channels:

1. **Binary releases**: [GitHub Releases](https://github.com/aquasecurity/tracee/releases)
2. **Container images**: [Docker Hub - aquasec/tracee](https://hub.docker.com/r/aquasec/tracee)
3. **Helm charts**: Aqua Security's Helm repository - `https://aquasecurity.github.io/helm-charts/`

Tracee may also be available in various community-managed package managers.

### Quick Start Guides

Choose your deployment method:

- **[Docker Installation](./docker.md)** - Run Tracee as a Docker container (fastest way to get started)
- **[Kubernetes Installation](./kubernetes.md)** - Deploy Tracee on Kubernetes clusters with Helm

## ⚙️ Post-Installation Setup

After installing Tracee, configure it for your environment:

### Configuration

- **[Configuration Guide](./config/index.md)** - Complete configuration reference for CLI and Kubernetes deployments
  - [Kubernetes Configuration](./config/kubernetes.md) - Configure Tracee in Kubernetes
  - [Container Enrichment](./container-engines.md) - Configure Tracee to detect and enrich events from Docker, Containerd, CRI-O, or Podman

### Monitoring & Observability

- **[Monitoring Configuration](./config/monitoring.md)** - Enable Prometheus metrics and health check endpoints for monitoring Tracee

## 🔍 Next Steps

After installation:

1. **Learn about policies**: Read the [Policies Guide](../policies/index.md) to define what to monitor
2. **Explore events**: Check out [Events Documentation](../events/index.md) to understand available events
3. **Configure outputs**: Set up [Output formats](../outputs/index.md) for your monitoring stack

## 💡 Troubleshooting

Having issues during installation or setup? Check our [Troubleshooting Guide](../troubleshooting.md) for common solutions.

For help and support, visit:

- [GitHub Discussions](https://github.com/aquasecurity/tracee/discussions)
- [Slack Community](https://slack.aquasec.com)

---

### Docs Outputs Index (`docs/docs/outputs/index.md`)

# Tracee Output

Tracee can generate thousands of events per day depending on your system and policies. The output system provides flexible ways to format, filter, route, and store these events for analysis and integration with your monitoring stack.

## Overview

The output system consists of several components that work together to control how Tracee events are handled:

- **Formats**: Choose how events are serialized (JSON, table, custom templates)
- **Destinations**: Route events to files, webhooks, or forward to log aggregators
- **Streams**: Create filtered event pipelines with independent routing
- **Options**: Enrich events with additional context (stack traces, environment variables, etc.)
- **Logging**: Configure Tracee's diagnostic logs separate from event output

## Configuration

Output is configured using the `--output` flag or the `output:` section in your configuration file. For complete details, see the [output flag reference](../flags/output.1.md).

**Basic CLI example:**
```console
tracee --output destinations.stdout_json.format=json
```

**Configuration file example:**
```yaml
output:
  destinations:
    - name: json_file
      type: file
      format: json
      path: /var/log/tracee/events.json
  enrichment:
    decoded-data: true
```

## Output Components

### [Destinations](../flags/output.1.md)

Route events to different outputs. Tracee supports three destination types:

- **File**: Write to files or stdout/stderr
- **Webhook**: Send events to HTTP endpoints
- **Forward**: Stream to FluentBit/Fluentd receivers

Each destination is configured with a type, format, and path or URL.

### [Formats](./output-formats.md)

Control how events are serialized and presented:

- **JSON**: Machine-readable format for log aggregation and SIEM integration
- **Table**: Human-readable terminal output for debugging and development
- **Go Templates**: Custom formatting using Go template syntax

### [Streams](./streams.md)

Create multiple output pipelines with independent filtering and routing. Streams allow you to:

- Route different events to different destinations
- Apply policy-based filtering
- Configure separate buffering strategies

### [Event Structure](./event-structure.md)

Understand the structure and fields available in Tracee events for parsing and analysis.

### [Sorting Events](./sorting-events.md)

Learn about event ordering guarantees and how to enable chronological sorting when needed.

### [Logging](./logging.md)

Configure Tracee's diagnostic logs (separate from event output) for troubleshooting and monitoring Tracee itself.

## Quick Start Examples

**JSON output to file:**
```console
tracee --output destinations.file_out.format=json --output destinations.file_out.path=/var/log/tracee.json
```

**Default table output with parsed arguments:**
```console
tracee --enrichment decoded-data
```

**Send events to webhook:**
```console
tracee --output destinations.webhook1.type=webhook --output destinations.webhook1.url=http://my-webhook:8080
```

**Multiple destinations with streams:**
```yaml
output:
  destinations:
    - name: all_events
      type: file
      format: json
      path: /var/log/tracee/all.json
    - name: security_events
      type: webhook
      url: https://siem.example.com/events
  streams:
    - name: security_stream
      destinations:
        - security_events
      filters:
        events:
          - security_file_open
          - security_socket_connect
```

## Additional Resources

- [Output flag reference](../flags/output.1.md): Complete CLI flag documentation
- [Example configurations](https://github.com/aquasecurity/tracee/blob/main/examples/config/global_config.yaml): Sample configuration files
- [Configuration guide](../install/config/index.md): General configuration documentation

---

### Docs Policies Index (`docs/docs/policies/index.md`)

# Policies

Policies allow users to specify which [events](../events/index.md) to trace in which workloads. The policy `scope` defines which workloads this policy is limited to. The policy can define multiple `rules` that specify the events to trace. Policies are used both for the [Tracee CLI](./usage/cli.md) and for the [Tracee Kubernetes](./usage/kubernetes.md) installation. This makes it easier to share policies across use cases and environments.

It is possible to load up to 64 policies into Tracee.

## Policy Formats

Tracee supports two policy formats: **Kubernetes CRD format** and **Plain format** (YAML or JSON). Both formats are fully interchangeable and produce identical results. Tracee automatically detects the format when loading policies.

### Kubernetes CRD Format

The Kubernetes CRD format follows the standard Kubernetes Custom Resource Definition structure. Use this format for Kubernetes deployments or when managing policies as Kubernetes resources.

**Example:**

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: overview-policy
  annotations:
    description: sample overview policy
spec:
  scope:
    - global
  rules:
    - event: dropped_executable
    - event: security_file_open
      filters:
        - data.pathname=/tmp/*
```

### Plain Format

The plain format provides a simpler, more concise syntax. Use this format for local development, testing, or non-Kubernetes environments.

**Example:**

```yaml
type: policy
name: overview-policy
description: sample overview policy
scope:
  - global
rules:
  - event: dropped_executable
  - event: security_file_open
    filters:
      - data.pathname=/tmp/*
```

Both formats support the same functionality. Tracee automatically detects the format by checking for `type: policy` (plain format) or `apiVersion` and `kind` fields (K8s CRD format). You can mix both formats in the same directory - Tracee will detect and load them correctly.

!!! Note
    Each event type can only be defined once in a policy

There are many ways to fine tune the scope and filters. For further information on the details, have a look at the respective sections: 

* [Specify the Policy scope](./scopes.md)
* [Filter events in the rules section](./rules.md)

While specifying event filters is optional, policies must have the `name`, `description`, `scope` and `rules` fields.

## Related Topics

* [Events Documentation](../events/index.md) - Learn about available events to use in policies
* [Policy Usage Guide](./usage/cli.md) - How to use policies with Tracee CLI
* [Kubernetes Policies](./usage/kubernetes.md) - How to deploy policies in Kubernetes
* [Troubleshooting](../troubleshooting.md) - Common policy-related issues and solutions

---

### Docs Events Custom Overview (`docs/docs/events/custom/overview.md`)

# Custom Events

Tracee comes with many built-in events, but you can extend its capabilities by creating custom events tailored to your specific needs.

## Modern Approach: Detectors

**Recommended**: The modern way to create custom threat detections and derived events is using the **EventDetector API**.

**📖 See the [Detector Documentation](../../detectors/index.md) for complete guide and examples.**

Key benefits:

- Type-safe protobuf access
- Rich data extraction helpers
- System state access (process trees, containers, DNS)
- Declarative filtering and auto-enrichment
- Built-in metrics and observability
- No plugin complexity

## Legacy Approach: Signatures (Plugin System)

The older signature system using `.so` plugins is still supported for backward compatibility, but we recommend migrating to detectors.

Refer to the [Go](./golang.md) documentation for instructions on the legacy plugin-based approach.

### Loading Signatures

Once you've created a signature plugin, load it using the `signatures-dir` flag:

```bash
tracee --signatures-dir=/tmp/myevents
```

!!! Tip
    Tracee also uses the custom events to add a few events, if you pass your own directory
    for `signatures-dir` you will not load the tracee [signatures](../builtin/security-events.md),
    to avoid such problems, you can either place your own events under the same directory of the tracee custom events,
    or pass multiple directories for example:
    ```bash
    tracee --signatures-dir=/tmp/myevents --signatures-dir=./dist/signatures
    ```

### Migrating from Signatures to Detectors

The [Detector API Reference](../../detectors/api-reference.md#migration-from-signatures) includes complete migration instructions with:

- Step-by-step migration guide
- Before/after code examples
- Pattern translations
- Migration checklist

---

## Choose Your Approach

| Feature | Detectors (Modern) | Signatures (Legacy) |
|---------|-------------------|---------------------|
| Type Safety | ✅ Compile-time | ❌ Runtime casting |
| Data Access | ✅ Type-safe helpers | ❌ Manual parsing |
| System State | ✅ Full datastore access | ❌ Limited |
| Event Filtering | ✅ Declarative | ❌ Manual in code |
| Auto-Enrichment | ✅ Process ancestry, threat metadata | ❌ Manual |
| Deployment | ✅ Compiled-in | ❌ Separate .so files |
| Testing | ✅ Direct function calls | ❌ Callback mocking |
| Observability | ✅ Built-in metrics | ❌ Manual |
| Documentation | ✅ [Complete guide](../../detectors/index.md) | ✅ [Legacy docs](./golang.md) |

**Recommendation**: Use detectors for all new development. Migrate existing signatures over time.

👈 Please use the side-navigation on the left in order to browse the different topics.

---

### Docs Install Config Index (`docs/docs/install/config/index.md`)

# Configuring Tracee

Tracee has many different options and settings that control how Tracee operates.

## Using Configuration Files

The `--config` flag allows you to specify global configuration options for Tracee by providing a configuration file in [YAML](https://github.com/aquasecurity/tracee/blob/main/examples/config/global_config.yaml) or [JSON](https://github.com/aquasecurity/tracee/blob/main/examples/config/global_config.json) format.

The configuration file can include settings for output formats, enrichment, buffers, logging, server options, and more.

To use the `--config` flag, provide the path to the configuration file:

```console
tracee --config /path/to/tracee-config.yaml
```

A complete example config file with all available options can be found [here](https://github.com/aquasecurity/tracee/blob/main/examples/config/global_config.yaml). Most of the options are documented in different sections in the documentation.

!!! note
    The YAML examples below can be translated to JSON as well. For Kubernetes-specific configuration, see the [Kubernetes Configuration](./kubernetes.md) guide.

!!! warning
    The following options are **not** supported in the configuration file and must be provided exclusively via the CLI: `--config`, `--policy`, `--scope`, `--events`. Policies can also be supplied through Kubernetes CRDs when running in a cluster (see [Policy Configuration](../../policies/index.md)).

## Common configurations


### Output

- **`--output` (`-o`)**: Controls how and where the output is printed.

  __NOTE__: You can view more in the [output section](../../outputs/index.md).

  YAML:
  ```yaml
  output:
    - json
  ```


### Server

- __`--server`__: Sets options for the HTTP and/or gRPC servers.

  CLI Examples:
  ```bash
  # Address configuration
  tracee --server http-address=:3366
  tracee --server grpc-address=unix:/var/run/tracee.sock

  # HTTP features (boolean flags)
  tracee --server metrics --server healthz --server pprof
  ```

  YAML:
  ```yaml
  server:
    http-address: ":3366"
    grpc-address: "unix:/var/run/tracee.sock"
    metrics: true
    healthz: true
    pprof: true
    pyroscope: true
  ```

### Stores (Process Tree and DNS Cache)

- **`--stores`**: Controls process tree and DNS cache options.


  __NOTE__: You can view more in the [Stores flag documentation](../../flags/stores.1.md) for CLI options. For information about accessing stores in detectors, see the [DataStore API documentation](../../detectors/datastore-api.md).

  YAML:
  ```yaml
  stores:
    process:
      enabled: true
      source: both
    dns:
      enabled: true
  ```

### Log

- **`--logging` (`-l`)**: Controls Tracee logging options, like verbosity,
filters, destination file and others.


  __NOTE__: You can view more in the [Tracee Logs section](../../outputs/logging.md).

  YAML:
  ```yaml
  log:
    - level: debug
  ```

### Containers

- To enable container enrichment, include the flag: **`--enrichment container`**. To disable it, simply omit the flag. Note: Setting any container sub-option (e.g., `container.docker.socket=/path`) automatically enables container, so `--enrichment container` is not needed.

  YAML:
  ```yaml
  enrichment:
    container:
      enabled: true
  ```

  __NOTE__: You can view more in the [enrichment section](../../flags/enrichment.1.md).

- **`--enrichment`**: Configures enrichment options including container enrichment and runtime sockets. For example, to configure runtime sockets:

  YAML:
  ```yaml
  enrichment:
    container:
      enabled: true
      cgroupfs:
        path: /host/sys/fs/cgroup
        force: false
      docker-socket: /var/run/docker.sock
      containerd-socket: /var/run/containerd/containerd.sock
      crio-socket: /var/run/crio/crio.sock
      podman-socket: /var/run/podman/podman.sock
    fd-paths: true
    executable-hash:
      enabled: true
      mode: sha256
    user-stack: true
  ```

### Capabilities

- **`--capabilities` (`-C`)**: Define specific capabilities for Tracee to run with. This allows you to either bypass, add, or drop certain capabilities based on your security and operational needs.

    
  __NOTE__: You can view more in the [Tracee Capabilities section](../../flags/capabilities.1.md)

  YAML:
  ```yaml
  capabilities:
    - add: 
        - CAP_SYS_ADMIN
    - drop: 
        - CAP_NET_RAW
  ```

  __NOTE__: Capabilities are Linux-specific permissions that control which privileged operations a program can perform.



### Buffer and Cache

- **`--buffers`**: Configures the buffer sizes for kernel and user buffers.

  Buffer sizes for perf ring buffers (kernel.events, kernel.artifacts, kernel.control-plane) are specified in pages. The pipeline buffer size is specified in event objects. The default size for perf ring buffers is 1024 pages. The default size for the pipeline buffer is 1000 event objects.

  YAML:
  ```yaml
  buffers:
      kernel:
          events: 2048
          artifacts: 1024
          control-plane: 512
      pipeline: 20000
  ```

  CLI:
  ```bash
  --buffers kernel.events=2048 --buffers kernel.artifacts=1024 --buffers kernel.control-plane=512 --buffers pipeline=20000
  ```

  __NOTE__: You can view more in the [buffers section](../../flags/buffers.1.md).

## See Also

- [Kubernetes Configuration](./kubernetes.md) - Kubernetes-specific configuration
- [Container Enrichment](../container-engines.md) - Configure container runtime integration
- [Monitoring](./monitoring.md) - Configure Prometheus metrics and health checks
- [Flag Reference](../../flags/config.1.md) - Complete configuration file reference
- [Output Configuration](../../outputs/index.md) - Configure event output and formatting
- [Policy Configuration](../../policies/index.md) - Define what to monitor

---

### Contributing Adding Events (`docs/contributing/adding-events.md`)

# Adding New Events

This guide walks you through the process of adding new event monitoring capabilities to Tracee. Adding new events involves modifications to several files across the codebase.

## Overview

Adding a new event to Tracee requires changes in several main areas:

1. **eBPF Implementation** - Implement the actual event handling logic
2. **Probe Configuration** - Configure how the event attaches to kernel functions
3. **Event Definition** - Define the event and its metadata in Go code
4. **Protobuf Schema** - Add event to protobuf definitions for gRPC API
5. **gRPC Translation** - Map event to protobuf enums for external APIs
6. **Event Documentation** - Create documentation file for the event

## Step-by-Step Process

### 1. Implement eBPF Program in `pkg/ebpf/c/tracee.bpf.c`

**Choose a kernel function to monitor:**

- **System calls**: `sys_openat`, `sys_execve`
- **LSM hooks**: `security_file_open`, `security_bprm_check`
- **VFS operations**: `vfs_read`, `vfs_write`
- **Network**: `tcp_connect`, `inet_csk_accept`

**Find functions:** `cat /proc/kallsyms | grep function_name`

**Implement the eBPF program:**

```c
SEC("kprobe/your_kernel_function")          // Must match probe configuration
int BPF_KPROBE(trace_your_event)            // This name will be used in probe config
{
    // Your event implementation follows this pattern:
    // 1. Initialize program data
    // 2. Apply scope filtering
    // 3. Extract data fields
    // 4. Apply data field-based filtering
    // 5. Submit event

    program_data_t p = {};
    if (!init_program_data(&p, ctx, YOUR_EVENT_ID))
        return 0;

    if (!evaluate_scope_filters(&p))
        return 0;

    // Extract data fields from kernel function
    void *arg1 = (void *)PT_REGS_PARM1(ctx);
    int arg2 = PT_REGS_PARM2(ctx);

    // Apply data field-based filtering if needed
    // ... your filtering logic ...

    // Save data fields to event buffer
    save_str_to_buf(&p.event->args_buf, arg1, 0);
    save_to_submit_buf(&p.event->args_buf, &arg2, sizeof(arg2), 1);

    events_perf_submit(&p, 0);
    return 0;
}
```

### 2. Configure Probes in `pkg/ebpf/probes/probe_group.go`

Add the probe configuration that matches your eBPF function names:

```go
// In pkg/ebpf/probes/probe_group.go, add to allProbes map in NewDefaultProbeGroup():
YourNewEvent: NewTraceProbe(KProbe, "your_kernel_function", "trace_your_event"),
```

**Attachment types:**

- `KProbe` - Attaches to kernel function entry (most common)
- `KretProbe` - Attaches to kernel function exit (for return values)
- `Tracepoint` - Attaches to predefined kernel tracepoints (stable interface)
- `SyscallEnter`/`SyscallExit` - Specialized for system call entry/exit

### 3. Define the Event in `pkg/events/core.go`

**Choose an event ID** from the appropriate range and add it to both eBPF and Go code:

**Event ID Ranges:**

- Common events: 1-699 (most events)
- Network events: 700-1999
- User-space network: 2000-3999
- Capture meta-events: 4000-4999
- Signal meta-events: 5000+

**In eBPF (`pkg/ebpf/c/tracee.bpf.c`):**
```c
#define YOUR_EVENT_ID 42
```

**In Go (`pkg/events/core.go`):**
{% raw %}
```go
// Add to appropriate const block (around line 100-200):
YourNewEvent = 42    // Must match eBPF ID

// Add to Core variable (around line 500+):
YourNewEvent: {
    id:      YourNewEvent,
    id32Bit: Sys32Undefined,
    name:    "your_event_name",
    version: NewVersion(1, 0, 0),
    sets:    []string{"fs"}, // Choose: syscalls, fs, net, security, proc, default
    fields: []DataField{
        {DecodeAs: data.STR_T, ArgMeta: trace.ArgMeta{Type: "const char*", Name: "pathname"}},
        {DecodeAs: data.INT_T, ArgMeta: trace.ArgMeta{Type: "int", Name: "flags"}},
        // Add all event data fields
    },
    dependencies: Dependencies{
        probes: []Probe{{handle: probes.YourNewEvent, required: true}},
    },
},
```
{% endraw %}

### 4. Add Event to Protobuf Schema

**Add event ID to protobuf enum** in `api/v1beta1/event.proto`:

```proto
enum EventId {
    // ... existing events ...
    your_new_event = 42;  // Must match the ID from Go/eBPF
    // ... more events ...
}
```

**Important notes:**
- Event IDs in protobuf must match exactly with Go constants and eBPF defines
- Follow the existing naming convention: lowercase with underscores
- Add the event in the appropriate section (syscalls, common events, etc.)

### 5. Add Event to gRPC Translation Table

**Update the translation table** in `pkg/server/grpc/tracee.go`:

```go
// Add to EventTranslationTable map in appropriate section:
events.YourNewEvent: pb.EventId_your_new_event,
```

This table maps internal Go event IDs to external protobuf event IDs for gRPC communication.

### 6. Generate Protobuf Files

**After modifying any `.proto` files, regenerate the Go code:**

```bash
# Generate protobuf Go files from proto definitions
make -f builder/Makefile.protoc
```

This command uses `protoc` to generate:
- Go structs from protobuf messages
- gRPC service definitions
- JSON marshaling/unmarshaling code

**Verify the generated files:**
- `api/v1beta1/*.pb.go` - Generated protobuf structs
- `api/v1beta1/*_grpc.pb.go` - Generated gRPC service code

### 7. Handle Complex Data Types (If Needed)

**If your event uses complex data types**, you may need to add marshalling logic in `pkg/server/grpc/event_data.go`:

```go
// Add case in parseArgument function for custom types
case *YourCustomType:
    return &pb.EventValue{
        Name: arg.Name,
        Value: &pb.EventValue_YourCustomField{
            YourCustomField: convertYourCustomType(v),
        },
    }, nil
```

**For new data types:**
1. Add the type definition to `api/v1beta1/event_data.proto`
2. Add the field to the `EventValue` oneof
3. Implement conversion logic in `event_data.go`
4. Run `make -f builder/Makefile.protoc` to regenerate files

## Testing Your New Event

### 8. Build and Test Compilation

```bash
# Build Tracee with your changes
make tracee

# Verify no compilation errors
echo $?  # Should be 0
```

### 9. Test Event Functionality

```bash
# Test that your event can be selected
sudo ./dist/tracee --events your_event_name --output json

# Test with policies
sudo ./dist/tracee --config-file your_test_policy.yaml
```

### 10. Unit Tests

Add unit tests for your event definition and run them:

```bash
# Run unit tests
make test-unit

# Run integration tests
make test-integration
```

```go
// In appropriate _test.go file
func TestYourNewEvent(t *testing.T) {
    // Test event definition
    // Test data field extraction
    // Test filtering behavior
}
```

### 11. Integration Tests

Consider adding integration tests that actually trigger your event and verify it's captured correctly.

### 12. Create Event Documentation

After everything is working, create proper documentation for your event to help users understand its purpose, usage, and configuration.

**For comprehensive documentation standards and templates, see the [Event Documentation Guide](event-documentation-guide.md).**

The documentation guide covers all formatting requirements, accuracy standards, and provides a complete template to ensure your documentation matches Tracee's quality standards.

## Common Patterns

- **System calls**: Hook `sys_*` functions, use `SyscallEnter`/`SyscallExit` probes
- **Security events**: Hook `security_*` functions, use `KProbe` attachments
- **File operations**: Hook VFS functions (e.g., `vfs_read`, `vfs_write`)
- **Network events**: Hook network functions, extract connection info

## Best Practices

- **Keep eBPF programs lightweight** - apply filtering early, minimize processing
- **Use descriptive names** - follow existing naming conventions
- **Document data fields clearly** - explain purpose and format of each field
- **Choose appropriate event sets** - helps users discover and select your event
- **Test thoroughly** - verify event triggers correctly and data is accurate

## Troubleshooting

### Common Issues

**Compilation Errors:**
- Verify event IDs don't conflict
- Check eBPF program section names match probe configuration
- Ensure all required headers are included
- Run `make -f builder/Makefile.protoc` after modifying proto files

**Protobuf/gRPC Issues:**
- Event ID mismatches between Go, eBPF, and protobuf
- Missing entry in EventTranslationTable
- Protobuf files not regenerated after schema changes
- Complex data types not handled in event_data.go

**Runtime Issues:**
- Check kernel compatibility for attachment points
- Verify probe symbols exist on target kernel
- Test with `bpftrace` first for complex kernel function monitoring

**Event Not Triggering:**
- Verify the kernel function is actually called for your test case
- Check filtering logic in eBPF program
- Use `bpf_printk()` for debugging eBPF code

### Debugging Tools

```bash
# Check eBPF program loading
bpftool prog list

# Monitor eBPF logs
cat /sys/kernel/debug/tracing/trace_pipe

# Test kernel symbol availability
cat /proc/kallsyms | grep your_symbol
```

## Getting Help

- **GitHub Issues**: Ask questions about event implementation
- **Discussions**: Discuss design decisions for complex events
- **Code Review**: Submit WIP PRs for feedback on approach

## Related Resources

- [eBPF Programming Guide](https://ebpf.io/)
- [Linux Tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html)
- [BPF LSM Documentation](https://www.kernel.org/doc/html/latest/bpf/prog_lsm.html)
- [Tracee Documentation](../docs/overview.md)

---

### Contributing Checkpatch (`docs/contributing/checkpatch.md`)

# Code Quality Guide

This guide covers all code quality tools and checks for Tracee development. The checkpatch script (`scripts/checkpatch.sh`) powers the `make check-pr` command and provides comprehensive PR verification for all developers.

## Purpose

This script now **powers the official `make check-pr` command** and includes all necessary checks for PR submission:

- **Documentation verification**: Man page synchronization checks
- **Code analysis**: Comprehensive formatting, linting, and static analysis
- **Unit tests**: Go and script unit tests
- **PR formatting**: Commit message formatting for PR descriptions
- **Enhanced UX**: Better error messages, tool installation guidance, and flexible options

## Usage

### Quick Start

```bash
# Official way - use make targets (recommended)
make check-pr                   # Full comprehensive checks
make check-pr-fast              # Quick checks (skip static analysis + unit tests)
make check-pr-skip-docs         # Skip documentation verification
make check-pr-skip-tests        # Skip unit tests

# Direct script usage for advanced options
./scripts/checkpatch.sh                   # Check HEAD
./scripts/checkpatch.sh abc123def         # Check specific commit
./scripts/checkpatch.sh HEAD~1            # Check previous commit
./scripts/checkpatch.sh --skip-unit-tests # Skip time-consuming tests
./scripts/checkpatch.sh --help            # Show all options
```

### All Available Commands

**Primary Make Targets (Recommended):**
```bash
make check-pr                   # Full comprehensive checks (recommended for PR submission)
make check-pr-fast              # Quick checks (skip static analysis + unit tests)
make check-pr-skip-docs         # Skip documentation verification
make check-pr-skip-tests        # Skip unit tests
make format-pr                  # Show what formatting changes are needed
make fix-fmt                   # Automatically fix code formatting issues
```

**Individual Check Commands:**
```bash
make -f builder/Makefile.checkers fmt-check     # Check code formatting only
make -f builder/Makefile.checkers fmt-fix       # Fix code formatting automatically
make -f builder/Makefile.checkers code-check    # Static code analysis only
```

**Direct Script Usage (Advanced):**
```bash
./scripts/checkpatch.sh [OPTIONS] [commit-ref]  # Direct script access
make check-pr ARGS="--fast --skip-docs HEAD~1"  # Pass arguments through make
```

## Test Categories

### 1. Documentation Verification

- Ensures `.1.md` and `.1` man page files are synchronized
- Runs `scripts/verify_man_md_sync.sh`
- Compares changes against `origin/main`

### 2. Code Analysis

- **Go Formatting**: Checks `gofmt` compliance
- **Linting**: Runs `revive` linter via `make check-lint`
- **Code Style**: Validates formatting via `make check-fmt`
- **Go Vet**: Static analysis via `make check-vet`
- **StaticCheck**: Advanced static analysis via `make check-staticcheck`
- **Error Check**: Unhandled error detection via `make check-err`
- **Vulnerability Check**: Known vulnerability detection via `make check-vulncheck`

### 3. Unit Tests

- **Go Unit Tests**: Runs `make test-unit`
- **Script Unit Tests**: Runs `make run-scripts-test-unit`

### 4. PR Formatting

- **Commit Display**: Shows commits in PR-ready format
- **Description Generation**: Extracts commit bodies for PR descriptions

## Dependencies

### Required

- `go` (Go compiler and tools)
- `make` (Build system)
- `git` (Version control)

### Optional Tools
The script will warn if these tools are missing but continue with available checks:

- **revive** (v1.7.0): `go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d`
- **staticcheck** (2025.1): `go install honnef.co/go/tools/cmd/staticcheck@5af2e5fc3b08ba46027eb48ebddeba34dc0bd02c`
- **errcheck** (v1.9.0): `go install github.com/kisielk/errcheck@11c27a7ce69d583465d80d808817d22d6653ee34`
- **govulncheck** (v1.1.4): `go install golang.org/x/vuln/cmd/govulncheck@d1f380186385b4f64e00313f31743df8e4b89a77`

## Installation of Optional Tools

To install all optional tools at once:

```bash
# Install linting and static analysis tools
go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d                   # v1.7.0
go install honnef.co/go/tools/cmd/staticcheck@5af2e5fc3b08ba46027eb48ebddeba34dc0bd02c          # 2025.1
go install github.com/kisielk/errcheck@11c27a7ce69d583465d80d808817d22d6653ee34                 # v1.9.0
go install golang.org/x/vuln/cmd/govulncheck@d1f380186385b4f64e00313f31743df8e4b89a77           # v1.1.4

# Ensure tools are in PATH
export PATH=$PATH:$(go env GOPATH)/bin
```

## Exit Codes

- `0`: All tests passed successfully
- `1`: One or more tests failed or an error occurred

## Integration with Development Workflow

### Pre-commit Hook
You can integrate this script into your git pre-commit hooks:

```bash
#!/bin/sh
# .git/hooks/pre-commit
make check-pr-fast || exit 1
```

### IDE Integration
Many IDEs can be configured to run external scripts. Configure your IDE to run `make check-pr-fast` or `./scripts/checkpatch.sh` as a build or test step.

### Recommended Development Workflow

1. **Make your changes**
2. **Quick validation**: Run `make check-pr-fast` for immediate feedback
3. **Fix any issues**: Use `make fix-fmt` for formatting or address other issues manually
4. **Full validation**: Run `make check-pr` before PR submission
5. **Commit and push** with confidence

**Integration Options:**

- **Pre-commit hook**: Add `make check-pr-fast || exit 1` to `.git/hooks/pre-commit`
- **IDE integration**: Configure your IDE to run `make check-pr-fast` as a build step

## Troubleshooting

### Common Issues

**Missing Tools Warning**
```
⚠ revive not found. Install with: go install github.com/mgechev/revive@8ece20b0789c517bd3a6742db0daa4dd5928146d # v1.7.0
```
*Solution*: Install the missing tool or ignore if you don't need that specific check.

**Documentation Mismatch**
```
✗ Documentation verification failed
- tracee.1.md change requires corresponding tracee.1 change
```
*Solution*: Run `make -f builder/Makefile.man` to regenerate man pages.

**Formatting Issues**
```
✗ Go formatting issues found:
```
*Solution*: Run `make fix-fmt` or `make -f builder/Makefile.checkers fmt-fix` to fix formatting automatically.

**Protocol Buffer Issues**
```
✗ Protobuf files need regeneration
```
*Solution*: Run `make -f builder/Makefile.protoc` to regenerate Go code from `.proto` files.

### Performance Tips

- The script fetches `origin/main` with `--depth=1` for efficiency
- Dependencies are checked once at the beginning
- Tests run in the order of fastest to slowest (docs → code analysis → unit tests)

## Extending the Script

To add additional test categories:

1. Create a new test function following the pattern of existing functions
2. Add it to the main execution in the `main()` function
3. Update the help text and documentation

## Comparison with CI

This script covers the basic PR tests but does not include:

- Integration tests (require elevated privileges)
- Performance tests (resource intensive)
- Multi-kernel testing (requires special infrastructure)
- Build verification for other tools

For the complete test suite, the CI pipeline in GitHub Actions is still required.

---

### Contributing Event Documentation Guide (`docs/contributing/event-documentation-guide.md`)

# Event Documentation Guide

This guide provides the standard format and requirements for documenting Tracee events in the official documentation. Following these standards ensures consistency and quality across all event documentation.

> **Note**: This guide complements the [Adding New Events](adding-events.md) guide. Complete the technical implementation first, then use this guide for creating proper documentation.

## Why Documentation Standards Matter

Consistent event documentation helps users:

- **Understand event purpose** and when to use specific events
- **Configure events correctly** with accurate field names and types
- **Troubleshoot issues** with clear dependency information
- **Discover related events** for comprehensive monitoring

## Required Sections

Every event documentation MUST include these sections in this exact order:

### 1. **YAML Frontmatter**
```yaml
---
title: TRACEE-EVENT-NAME
section: 1
header: Tracee Event Manual
---
```

### 2. **NAME**
- Format: `**event_name** - brief one-line description`
- Keep description concise and accurate

### 3. **DESCRIPTION**
- First paragraph: When the event is triggered and what it captures
- Second paragraph: Additional context about the event's purpose and importance
- Keep descriptions focused on **what/when/how** rather than **why** (save for USE CASES)
- Avoid excessive technical jargon

### 4. **EVENT SETS**
- List the event sets this event belongs to (from `pkg/events/core.go`)
- Use **none** if the event doesn't belong to any sets
- Example: `**proc**, **proc_life**` or `**none**`

### 5. **DATA FIELDS**
- List ALL data fields exactly as defined in `pkg/events/core.go`
- Format: `**field_name** (*type*)`
- Follow with `: Description of the field`
- Types must match exactly (e.g., `int32`, `uint64`, `string`, `time.Time`, `SockAddr`)
- Field names must match exactly from the source code
- If no fields: "This event currently does not capture specific arguments, but provides timing and context information about when [description]."

### 6. **DEPENDENCIES**
- **CRITICAL**: Extract probe information from `pkg/ebpf/probes/probe_group.go`
- Use kernel function names/tracepoint names, NOT internal Tracee probe handles
- Format headers:
  - `**Kernel Probe:**` (singular) for kprobes
  - `**Kernel Tracepoint:**` (singular) for raw tracepoints
- Format entries: `- kernel_function_name (required): Description` for single probes
- **Combine kprobe+kretprobe**: When both entry and return probes exist for the same function, combine as:
  - `- function_name (kprobe + kretprobe, required): Description`
- For combined probes, include the probe types to clarify the instrumentation method
- For multiple dependencies, list each type separately
- Include tail calls if present: `**Tail Calls:** - call_name: Description`

### 7. **USE CASES**
- Provide 3-5 specific, actionable use cases
- Format: `- **Category**: Specific use case description`
- Focus on practical applications for users
- Be specific and practical rather than generic

### 8. **RELATED EVENTS**
- List 3-6 related events that users might find relevant
- Format: `- **event_name**: Brief description`
- Include complementary events, lifecycle events, and related security events

## Signature Events Additional Requirements

Signature events require these additional sections:

### **SIGNATURE METADATA** (after DESCRIPTION, before EVENT SETS)
```markdown
- **ID**: TRC-XXX (unique signature identifier)
- **Version**: 1 (signature version number)
- **Severity**: X (threat level number + description)
- **Category**: category-name (e.g., defense-evasion, privilege-escalation)
- **Technique**: Descriptive technique name
- **MITRE ATT&CK**: TXXXX (MITRE ATT&CK technique ID)
- **Tags**: comma-separated tags (optional, e.g., linux, container)
```

### **DETECTION LOGIC** (after USE CASES, before RELATED EVENTS)
- Explain how the signature detection works
- List what the signature monitors for (numbered list preferred)
- Include detection criteria and logic flow
- Format example:
```
The signature monitors for:

1. **Event type** - specific condition or behavior
2. **Analysis step** - how events are processed
3. **Context evaluation** - reducing false positives
```

## Event Type Identification

Before documenting, determine if the event is a **regular event** or a **signature event**:

- **Regular events**: Core system events (syscalls, kernel events, LSM hooks, network events)
- **Signature events**: Security detection rules (found in `docs/events/builtin/man/security/`)

Signature events require additional sections (SIGNATURE METADATA, DETECTION LOGIC) as detailed above.

## Data Accuracy Requirements

### Event Definition Source
- **PRIMARY SOURCE**: `pkg/events/core.go`
- Find the event struct by searching for the event name
- Extract the following with 100% accuracy:

  - `sets: []string{}` → EVENT SETS section
  - `fields: []DataField{}` → DATA FIELDS section (names and types)
  - `dependencies.probes` → DEPENDENCIES section

### Probe Definition Source
- **PRIMARY SOURCE**: `pkg/ebpf/probes/probe_group.go`
- Map internal probe handles to actual kernel attachment points
- Examples:

  - `SysEnter: NewTraceProbe(RawTracepoint, "raw_syscalls:sys_enter", ...)` → `raw_syscalls:sys_enter`
  - `DoExit: NewTraceProbe(KProbe, "do_exit", ...)` → `do_exit`
  - `SecurityFileOpen: NewTraceProbe(KProbe, "security_file_open", ...)` → `security_file_open`

### Probe Type Mapping

- `KProbe` → Use `**Kernel Probe:**` header
- `RawTracepoint` → Use `**Kernel Tracepoint:**` header
- Multiple types → Use separate headers for each type

## Quality Improvements

### Avoid Redundancy

- **Problem**: Having "This event is useful for:" bullets in DESCRIPTION AND a separate USE CASES section creates redundancy
- **Solution**: Keep DESCRIPTION focused on **what/when/how** the event works, save **why/for what** explanations for USE CASES section

### Simplify Dependencies

- **Problem**: Separate entries for kprobe + kretprobe on same function is verbose
- **Solution**: Combine as `function_name (kprobe + kretprobe, required): Description`
- **Example**: Instead of:

  ```
  - vfs_write (required): VFS write function entry
  - vfs_write (required): VFS write function return
  ```
  Use: `- vfs_write (kprobe + kretprobe, required): VFS write function`

## Content Guidelines

### Writing Style

- Use clear, technical language appropriate for system administrators and security professionals
- Avoid marketing language or excessive enthusiasm
- Keep descriptions factual and implementation-focused
- Use present tense for event triggers ("Triggered when...")
- Keep documentation **concise** - aim for 50-70 lines total per event

### Description Structure

1. **Trigger condition**: "Triggered when..."
2. **Context and purpose**: What the event captures and why it's important

### Use Case Categories
Common categories to consider:

- Security monitoring/analysis
- Performance analysis
- Debugging/troubleshooting
- Compliance auditing
- Process/network/file monitoring
- Threat detection/hunting
- Forensic analysis

## Formatting Requirements

### Consistency Rules

- Use `**bold**` for field names, event names, and section emphasis
- Use `*italics*` for data types
- Use `- ` for bullet points (not `* `)
- Use `: ` (colon + space) to separate field names from descriptions
- Use singular headers for dependencies ("Kernel Probe:" not "Kernel Probes:")

### File Naming and Location

- Filename: `event_name.md` (use underscores, lowercase)
- Location based on event category:

  - System events: `docs/docs/events/builtin/man/misc/`
  - LSM events: `docs/docs/events/builtin/man/lsm/`
  - Network events: `docs/docs/events/builtin/man/net/`
  - Container events: `docs/docs/events/builtin/man/containers/`

## Verification Checklist

Before submitting event documentation:

- [ ] All data field names match exactly from `pkg/events/core.go`
- [ ] All data field types match exactly from `pkg/events/core.go`
- [ ] Event sets match exactly from `pkg/events/core.go`
- [ ] Dependencies use kernel function/tracepoint names from `pkg/ebpf/probes/probe_group.go`
- [ ] Probe types correctly mapped to section headers
- [ ] No internal probe handles used in dependencies
- [ ] All required sections present in correct order
- [ ] No prohibited verbose sections included
- [ ] Writing style is clear and technical
- [ ] Use cases are specific and actionable
- [ ] Related events are relevant and helpful

## Example Templates

### Regular Event Template

> **Template Start** - Copy everything between the horizontal lines for regular events

---

```markdown
---
title: TRACEE-EVENT-NAME
section: 1
header: Tracee Event Manual
---

# NAME

**event_name** - brief description

# DESCRIPTION

Triggered when [condition]. This event [captures/provides/monitors] [what].

[Additional context about purpose and implementation].

# EVENT SETS

**set1**, **set2** (or **none**)

# DATA FIELDS

**field_name** (*type*)
: Description of the field

**another_field** (*type*)
: Description of the field

# DEPENDENCIES

**Kernel Probe:**

- kernel_function_name (required): Description

**Kernel Tracepoint:**

- tracepoint:name (required): Description

# USE CASES

- **Security monitoring**: Specific use case

- **Performance analysis**: Specific use case

- **Debugging**: Specific use case

- **Compliance**: Specific use case

- **Monitoring**: Specific use case

# RELATED EVENTS

- **related_event_1**: Brief description
- **related_event_2**: Brief description
- **related_event_3**: Brief description
```

---

> **Template End** - Copy everything above this line

### Signature Event Template

> **Template Start** - Copy everything between the horizontal lines for signature events

---

```markdown
---
title: TRACEE-SIGNATURE-NAME
section: 1
header: Tracee Event Manual
---

# NAME

**signature_name** - brief description of security detection

# DESCRIPTION

Triggered when [security condition]. This security signature [detects/identifies/monitors] [threat/behavior].

[Additional context about threat landscape and detection rationale].

# SIGNATURE METADATA

- **ID**: TRC-XXX
- **Version**: 1
- **Severity**: X (threat level description)
- **Category**: category-name
- **Technique**: Descriptive technique name
- **MITRE ATT&CK**: TXXXX
- **Tags**: tag1, tag2 (optional)

# EVENT SETS

**signatures**, **category_set**

# DATA FIELDS

**field_name** (*type*)
: Description of the field

**detection_context** (*object*)
: Context information about the detection

# DEPENDENCIES

**System Events:**

- base_event (required): System event being analyzed for detection

# USE CASES

- **Threat detection**: Specific threat hunting scenario
- **Security monitoring**: Specific security use case
- **Incident response**: Specific response scenario

# DETECTION LOGIC

The signature monitors for:

1. **Base event** - specific condition or behavior
2. **Analysis criteria** - how events are analyzed
3. **Context evaluation** - reducing false positives

# RELATED EVENTS

- **related_signature**: Brief description
- **base_event**: System event being monitored
- **related_detection**: Similar detection mechanism
```

---

> **Template End** - Copy everything above this line

## Documentation Workflow

### Complete Process

1. **Implement the event** following the [Adding New Events](adding-events.md) guide
2. **Create markdown documentation** following this guide's standards
3. **Verify accuracy** by cross-referencing with `pkg/events/core.go`
4. **Generate man pages** using the specialized build system
5. **Add to navigation** by updating `mkdocs.yml`
6. **Test locally** with MkDocs to ensure proper rendering

### Generating Man Pages

After creating or modifying event documentation, you **must** generate the corresponding man pages:

```bash
# From the Tracee root directory
make -f builder/Makefile.man
```

This command:
- **Builds a Docker container** with pandoc and required tools
- **Converts markdown files** to proper man page format
- **Updates man pages** in the appropriate directories

> **Critical**: Always run this step after creating or modifying event documentation to keep man pages synchronized with markdown files.

---

This guide ensures all Tracee event documentation maintains consistency, accuracy, and usability for users and contributors.

---

### Contributing Evt Tool (`docs/contributing/evt-tool.md`)

# evt - Event Generator and Stress Testing Tool

`evt` is a testing tool designed to generate events and stress test Tracee's event processing capabilities. It provides two main functionalities: triggering individual events and running comprehensive stress tests.

## Overview

The `evt` tool helps with:

- **Event Generation**: Trigger specific Linux events for testing Tracee's detection capabilities
- **Performance Testing**: Stress test Tracee with high volumes of concurrent events
- **Development**: Validate event detection and measure Tracee's performance under load

## Building

Build the `evt` binary:

```bash
make evt
```

Build the trigger runner container for stress testing:

```bash
make evt-trigger-runner
```

Or with a custom image name:

```bash
EVT_TRIGGER_RUNNER_IMAGE=my-runner:dev make evt-trigger-runner
```

## Commands

### evt trigger

Trigger a specific event type to test Tracee's detection.

**Usage:**

```bash
evt trigger --event <event_name> [flags]
```

**Examples:**

```bash
# Trigger a single event
evt trigger --event security_file_open

# Trigger multiple operations
evt trigger --event ptrace --ops 100

# Parallel execution
evt trigger --event security_file_open --parallel 4 --ops 1000

# Show Tracee bypass flags for an event
evt trigger --event security_file_open --bypass-flags
```

**Flags:**

- `--event, -e <name>`: Event type to trigger (required)
- `--ops, -o <number>`: Number of operations to perform
- `--parallel, -p <number>`: Number of parallel workers (total ops = workers × ops)
- `--sleep, -s <duration>`: Sleep duration between operations
- `--bypass-flags, -b`: Print Tracee bypass flags for the event
- `--wait-signal, -w`: Wait for SIGUSR1 before starting
- `--signal-timeout <duration>`: Timeout for waiting for signal

### evt stress

Run comprehensive stress tests with multiple event types in isolated containers.

**Usage:**

```bash
evt stress --events <event_specs> [flags]
```

**Event Specification Format:**

```
event[:instances=N:ops=N:sleep=dur]
```

- `event`: Event name to trigger
- `instances`: Number of parallel workers
- `ops`: Operations per worker
- `sleep`: Sleep between operations

**Examples:**

```bash
# Basic stress test with default settings
evt stress --events security_file_open --events ptrace

# Multiple events with custom configurations
evt stress \
  --events security_file_open:instances=10:ops=1000:sleep=1ms \
  --events ptrace:instances=2:ops=100:sleep=100ms

# Performance testing with profiling
evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --pprof --metrics

# Keep Tracee running after test for analysis
evt stress --events ptrace --keep-tracee --pyroscope

# Manual Tracee control
evt stress --events security_file_open --auto-tracee=false

# Custom cooldowns for timing control
evt stress \
  --events security_file_open:instances=10:ops=1000 \
  --tracee-init-cooldown 10s \
  --stress-end-cooldown 30s
```

**Flags:**

**Event Configuration:**
- `--events, -e <spec>`: Events to stress test (required, repeatable)
  - Format: `event[:instances=N:ops=N:sleep=dur]`
  - Example: `security_file_open:instances=10:ops=1000:sleep=1ms`

**Container Configuration:**
- `--image <name:tag>`: Trigger runner container image

**Tracee Management:**
- `--auto-tracee`: Automatically manage Tracee lifecycle (start and stop)
- `--keep-tracee`: Keep Tracee running after test (requires `--auto-tracee=true`, useful for profiling analysis)
- `--tracee-binary <path>`: Path to Tracee binary (used when `--auto-tracee=true`)
- `--tracee-output <format:path>`: Tracee output format and path

**Profiling:**
- `--metrics`: Enable Tracee metrics endpoint
- `--pprof`: Enable Tracee pprof profiling endpoint
- `--pyroscope`: Enable Tracee pyroscope continuous profiling

**Execution Control:**
- `--wait-before-trigger`: Wait for user input before triggering events (useful to start profiling/scraping tools)
- `--signal-timeout <duration>`: Timeout for containers waiting for signal
- `--tracee-init-cooldown <duration>`: Cooldown after Tracee starts for stabilization before triggering events
- `--stress-end-cooldown <duration>`: Cooldown after stress completes for stabilization before cleanup
- `--dry-run`: Show what would be executed without running

## Stress Testing Architecture

The stress testing system uses a containerized architecture to run triggers in isolation:

1. **Trigger Runner Container**: Contains the `evt` binary and trigger scripts
2. **Detached Execution**: Containers run in the background with full privileges
3. **Coordinated Start**: All containers wait for SIGUSR1 before triggering events
4. **Automatic Cleanup**: Containers are removed after completion or interruption

### Workflow

1. **Phase 1**: Validate prerequisites (Docker, container image, Tracee binary)
2. **Phase 2**: Start trigger containers (detached mode, waiting for signal)
3. **Phase 3**: Start Tracee with container scope filters (optional)
4. **Phase 4**: Signal containers to start triggering events simultaneously
5. **Phase 5**: Monitor containers until completion
6. **Cleanup**: Stop Tracee (if auto-managed) and remove containers

### Tracee Lifecycle Management

**Auto-managed (default):**
```bash
evt stress --events security_file_open
# Tracee is started and stopped automatically
```

**Keep Tracee running:**
```bash
evt stress --events ptrace --keep-tracee --pyroscope
# Tracee stays running for profiling analysis
# You stop it manually later
```

**Manual control:**
```bash
# Terminal 1: Start Tracee yourself
sudo ./dist/tracee --events security_file_open --server pyroscope

# Terminal 2: Run stress test
evt stress --events security_file_open --auto-tracee=false
```

## Container Image

The `evt-trigger-runner` container image contains:

- `evt` binary for executing triggers
- Trigger scripts for various event types
- Runtime dependencies (bash, strace, bpftrace, etc.)

**Build the image:**

```bash
make evt-trigger-runner
```

**Build with custom name:**

```bash
EVT_TRIGGER_RUNNER_IMAGE=my-runner:test make evt-trigger-runner
```

**Use custom image:**

```bash
evt stress --image my-runner:test --events security_file_open
```

## Available Triggers

The following event triggers are available (located in `cmd/evt/cmd/trigger/triggers/`):

- `security_file_open` - File open operations
- `ptrace` - Process tracing events
- `security_bpf_prog` - BPF program operations
- `security_socket_connect` - Socket connection events
- `security_socket_bind` - Socket bind operations
- `security_socket_create` - Socket creation events
- `sched_process_exec` - Process execution events
- `sched_process_fork` - Process fork events
- `sched_process_exit` - Process exit events
- And many more...

See `cmd/evt/cmd/trigger/triggers/` for the complete list.

## Performance Testing Best Practices

### 1. Disable Output for Maximum Performance

For optimal performance during stress testing, avoid writing event logs:

```bash
# Maximum performance (no logging)
evt stress --events security_file_open:instances=20:ops=10000

# Enable logging if needed
evt stress --events ptrace --tracee-output json:/tmp/events.json
```

### 2. Use Profiling for Analysis

Enable profiling endpoints to analyze Tracee's performance:

```bash
evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --pprof --metrics \
  --keep-tracee
```

Then access:
- Pyroscope UI: http://localhost:4040/?query=tracee.cpu
- Pprof: http://localhost:3366/debug/pprof
- Metrics: http://localhost:3366/metrics

### 3. Control Timing with Cooldowns

Use cooldown periods to ensure accurate measurements:

```bash
# Longer initialization cooldown for complex setups
evt stress \
  --events security_file_open:instances=20:ops=10000 \
  --tracee-init-cooldown 10s

# End cooldown to let Tracee finish processing before cleanup
evt stress \
  --events ptrace:instances=10:ops=5000 \
  --stress-end-cooldown 30s \
  --keep-tracee
```

**When to adjust cooldowns:**
- **Init cooldown**: Provides time for Tracee to stabilize after starting (e.g., loading signatures, initializing eBPF). Increase for complex configurations.
- **End cooldown**: Provides time for system stabilization after stress completes. Increase for complex workloads or when collecting final metrics/profiles.

### 4. Scale Gradually

Start with low load and increase gradually:

```bash
# Low load
evt stress --events security_file_open:instances=1:ops=100

# Medium load
evt stress --events security_file_open:instances=10:ops=1000

# High load
evt stress --events security_file_open:instances=20:ops=10000
```

### 5. Multiple Event Types

Test with multiple event types for realistic scenarios:

```bash
evt stress \
  --events security_file_open:instances=10:ops=5000 \
  --events ptrace:instances=5:ops=1000 \
  --events security_socket_connect:instances=8:ops=2000
```

## Troubleshooting

### Container Image Not Found

```
Error: container image "evt-trigger-runner:latest" not found
```

**Solution:**
```bash
make evt-trigger-runner
```

### Tracee Binary Not Found

```
Error: tracee binary not found at "./dist/tracee"
```

**Solution:**
```bash
make tracee
```

### Docker Not Available

```
Error: docker not available
```

**Solution:** Ensure Docker is installed and running.

### Monitoring Interrupted

When you press Ctrl+C during monitoring:
```
Monitoring interrupted (containers will be cleaned up)
```

This is expected behavior. Containers are automatically cleaned up.

## Integration with Performance Dashboard

The stress testing tool integrates with Tracee's performance dashboard:

```bash
# Start performance dashboard
make -f builder/Makefile.performance dashboard-start

# Run stress test with profiling
evt stress \
  --events security_file_open:instances=16:ops=100000 \
  --pyroscope --metrics \
  --keep-tracee

# Analyze results in dashboard
# Grafana: http://localhost:3000
# Pyroscope: http://localhost:4040

# Stop dashboard when done
make -f builder/Makefile.performance dashboard-stop
```

## See Also

- [Performance Considerations](performance.md) - Tracee profiling and benchmarking
- [Testing Coverage](testing-coverage.md) - Tracee testing framework
- [Building](building/building.md) - Building Tracee from source

---

### Contributing Guidelines (`docs/contributing/guidelines.md`)

# Style Guide for Contributors

For those willing to contribute to Tracee, this repository has code formatting
guidelines being enforced.

## Contributing to Documentation

Our documentation aims to follow the [Diátaxis documentation framework](https://diataxis.fr/).

To contribute to the documentation:

1. Clone the Tracee GitHub repository.
2. Make changes in the `/docs` directory in the root folder.
3. Remember to edit the index in `mkdocs.yml` if you add or move existing files or directories.
4. Test your changes to the documentation.

### Test Documentation Changes

You can test your changes to the documentation by building and running a docker container.
Prerequisites: Docker installed and running locally.
Build the container image:

```bash
make -f ./builder/Makefile.mkdocs mkdocs-build
```

Serve the container image:

```bash
make -f ./builder/Makefile.mkdocs mkdocs-serve
```

Open `localhost:8000/tracee`

Now you should see the documentation.
Check that everything is as you intended.

And finally Submit a PR about the changes.

## Contributing to Tracee Code

We welcome contributions to Tracee's codebase! Before submitting your changes, please familiarize yourself with these guidelines.
To contribute to the code:

### Before You Commit

Before submitting code changes, ensure you follow these essential steps:

1. **Generated Files**: If you've modified certain types of files, regenerate the corresponding outputs:
   - **Man pages**: Run `make -f builder/Makefile.man` if you changed core code or documentation
   - **Protocol buffers**: Run `make -f builder/Makefile.protoc` if you modified `.proto` files

2. **Code Quality**: Run comprehensive code quality checks:
   ```bash
   make check-pr
   ```

   This verifies formatting, runs linting, performs static analysis, and validates your changes meet project standards.

3. **Development Environment**: For consistent results, use a supported [development environment](./building/environment.md) when running checks.

For detailed information about these tools, troubleshooting, and advanced options, see our comprehensive [Code Quality Guide](checkpatch.md).

### Performance Considerations

Performance is a critical aspect of Tracee.

To ensure your contributions maintain optimal performance, follow the guidelines in [Performance Considerations](./performance.md) page.

### Kubernetes Considerations

If your contribution impacts Tracee's behavior within a Kubernetes cluster, follow the guidelines in [Kubernetes Considerations](./kubernetes.md).

---

### Contributing Kubernetes (`docs/contributing/kubernetes.md`)

# Kubernetes Considerations for Tracee Contributors

1. **Set up a Local Kubernetes Cluster**

    We recommend using a local Kubernetes cluster for development and testing. Popular options include:

    * **kind:** Kubernetes IN Docker
    * **minikube:** Runs a single-node cluster in a VM
    * **MicroK8s:** Lightweight, snap-based Kubernetes

    Tracee's Makefile provides convenient targets for setting up MicroK8s:

    ```bash
    make -f builder/Makefile.k8s help
    ```

    ```text
    To build the operator docker container:

        $ make -f builder/Makefile.k8s build

    To generate the kubernetes manifests:

        $ make -f builder/Makefile.k8s manifests

    To generate operator code:

        $ make -f builder/Makefile.k8s generate

    Or simply:

        $ make -f builder/Makefile.k8s
    ```

2. **Deploy Tracee**

    Deploy Tracee to your chosen local cluster. The deployment method will vary depending on your needs:

    * **DaemonSet:** For system-wide tracing, deploy Tracee as a DaemonSet.
    * **Tracee Operator:**  Use the Tracee operator for simplified management.
    * **Other Methods:** Explore alternative methods like sidecar containers based on your use case.

3. **Test Your Changes**

    Test your modifications with realistic scenarios within the Kubernetes environment.  Consider these examples:

    * **Application Monitoring:** Deploy sample applications and observe Tracee's event capture for expected behavior.
    * **Network Policies:**  Configure network policies to restrict pod communication and verify Tracee captures relevant network events.
    * **Resource Constraints:**  Apply resource limits to pods and ensure Tracee functions correctly under constrained conditions.
    * **Security Policies:**  Implement Pod Security Policies and/or Security Contexts to validate Tracee's compliance and event capture in secure environments.

4. **Monitor Tracee's Performance**

    Utilize Kubernetes' observability tools to monitor Tracee:

    * **Resource Usage:** Track pod resource consumption (CPU, memory) using `kubectl top` or monitoring dashboards.
    * **Logs:** Analyze Tracee logs for errors, warnings, or unexpected behavior using `kubectl logs`.
    * **Kubernetes Metrics:**  If available, leverage Kubernetes metrics to gain insights into Tracee's performance.

5. **Clean Up**

    After testing, remove deployed Tracee components and test resources to maintain a clean cluster environment.

## Kubernetes-Specific Features

If your contribution involves Kubernetes-specific features within Tracee (e.g., capturing Kubernetes events or metadata), ensure you test these functionalities thoroughly within the cluster environment. Pay close attention to event accuracy and any integration with Kubernetes APIs.

---

### Contributing Performance (`docs/contributing/performance.md`)

# Performance Considerations

1. **Profiling Tracee for Performance test** - Tracee integrates with Pyroscope and Pprof for continuous profiling. When running Tracee locally for development or testing, use the `--server pyroscope --server pprof` command-line options.

    ```bash
    sudo ./dist/tracee --server pyroscope --server pprof
    ```

    This enables profiling data to be sent to a local server. The Tracee repository includes a convenient way to deploy a performance dashboard for analyzing this data. Run the following for more details:

    ```bash
    make -f builder/Makefile.performance help
    ```

2. **Performance Dashboard:** The provided performance dashboard allows visualization of host metrics, CPU flame graphs, and other performance-related data. Follow these steps to deploy locally and see instructions on using the dashboard:

    ```bash
    make -f builder/Makefile.performance dashboard-start
    ```

    ```bash
    make -f builder/Makefile.performance dashboard-stop
    ```

3. **Benchmarking and Stress Testing:** Before submitting significant code changes, use the `evt` tool to run stress tests and assess their impact on Tracee's performance:

    ```bash
    # Build evt and trigger runner container
    make evt evt-trigger-runner
    
    # Run stress test with profiling
    evt stress \
      --events security_file_open:instances=16:ops=100000 \
      --pyroscope --pprof --metrics \
      --keep-tracee
    ```
    
    For comprehensive documentation on the `evt` tool, see [evt - Event Generator and Stress Testing Tool](evt-tool.md).

4. **Common Performance Pitfalls:**

    - **Excessive eBPF Events:** Be mindful of the number and frequency of eBPF events being generated. Overly frequent events can lead to performance overhead. (Provide Tracee-specific examples or best practices to avoid this, such as filtering events effectively or using appropriate sampling rates.)
    - **Inefficient eBPF Programs:** Optimize your eBPF programs for minimal overhead. (Provide Tracee-specific guidance. Are there common patterns to avoid within Tracee's eBPF context?)
    - **Resource Consumption:** Consider the CPU and memory usage of Tracee itself. Avoid unnecessary allocations or computations.

By adhering to these practices, you can contribute to Tracee's performance and help ensure its efficiency. Remember that performance is an ongoing effort, so continuous profiling, benchmarking, and optimization are essential.

---

### Contributing Setup Development Machine With Vagrant (`docs/contributing/setup-development-machine-with-vagrant.md`)

# Setup Development Machine with Vagrant

[HashiCorp Vagrant] leverages a declarative configuration file, which describes
all software requirements, packages, operating system configuration, and users
to provide the same development environment for everyone.

The [Vagrantfile](https://github.com/aquasecurity/tracee/blob/main/Vagrantfile) describes a cross-platform QEMU-based virtual machine optimized for building Tracee from the [Getting Started](../index.md) guides. It supports both Linux and macOS hosts with hardware acceleration for optimal performance.

This allows developers involved in the project to check out the code, run `vagrant up`, and be on their way with a fully configured [cloud-image/ubuntu-24.04] development environment.

## Features

- **Cross-platform support**: Linux (KVM) and macOS (HVF) with automatic hardware acceleration detection
- **Optimized shared folders**: Platform-specific 9p filesystem implementation for best performance
- **Flexible configuration**: Easily customize CPU, memory, and acceleration settings
- **Automatic error handling**: Comprehensive validation and user-friendly error messages

## Prerequisites

- [HashiCorp Vagrant]
- [QEMU] with hardware acceleration support:
  - **Linux**: KVM acceleration (recommended for optimal performance)
  - **macOS**: HVF (Hypervisor Framework) acceleration on Intel/Apple Silicon
  - **Fallback**: TCG software emulation (slower but works on all systems)
- [vagrant-qemu plugin]: Install with `vagrant plugin install vagrant-qemu`

## Clone the Tracee Repository

Clone the Tracee repository to your local machine. This repository contains the Vagrantfile.

```bash
git clone https://github.com/aquasecurity/tracee.git
```

## Navigate to the Tracee Directory

Open a terminal and navigate to the directory containing the `Vagrantfile` within the cloned Tracee repository (`tracee/`)

```bash
cd tracee
```

## Configure VM Type (Optional)

The VM can be provisioned for either a `dev` or `test` environment. The `dev` environment includes additional tools like MicroK8s, kubectl, and Helm.

- **Development Environment:** Full development environment (Default)

  ```bash
  export VM_TYPE=dev
  ```

- **Testing Environment:**  Smaller vagrant machine without k8s cumbersome to avoid conflicts with specific tests.

  ```bash
  export VM_TYPE=test
  ```

**Important:** There are two ways to use a custom `VM_TYPE`:

**Option 1: Export the environment variable (recommended)**
```bash
# Export once, then use normal vagrant commands
export VM_TYPE=test
vagrant up
vagrant ssh
vagrant halt
vagrant destroy
```

**Option 2: Prefix each command (if not exported)**
```bash
# Include VM_TYPE in every command if not exported
VM_TYPE=test vagrant up
VM_TYPE=test vagrant ssh
VM_TYPE=test vagrant halt
VM_TYPE=test vagrant destroy
```

Without setting `VM_TYPE`, vagrant defaults to the `dev` environment. If you use the prefix method (Option 2) and forget to include `VM_TYPE` in a command, vagrant may operate on the wrong VM or give "VM not found" errors.

> **💡 Tip**: Use `export VM_TYPE=test` at the beginning of your session to avoid repeating it in every command.

## Configure VM Settings (Optional)

Customize the VM's configuration by setting the following environment variables:

### Resource Allocation

- `VM_CPUS`: Number of virtual processors. Defaults to half of the host's processors. Example:

  ```bash
  export VM_CPUS=4
  ```

- `VM_MEM`: Memory in gigabytes. Defaults to 8GB. Example:

  ```bash
  export VM_MEM=16
  ```

### Hardware Acceleration

- `VM_ACCEL`: Force specific acceleration type. Options vary by host OS:

  **Linux:**
  ```bash
  export VM_ACCEL=kvm   # KVM acceleration (fast)
  export VM_ACCEL=tcg   # Software emulation (slow)
  ```

  **macOS:**
  ```bash
  export VM_ACCEL=hvf   # HVF acceleration (fast)
  export VM_ACCEL=tcg   # Software emulation (slow)
  ```

  If not specified, the optimal acceleration method is automatically detected.

## Start the VM

### Basic Usage

Run the following command to start the VM with default settings:

```bash
vagrant up
```

### Custom Configuration Examples

You can combine environment variables for custom configurations:

```bash
# Test VM with custom resources
VM_TYPE=test VM_CPUS=4 VM_MEM=4 vagrant up

# Force software emulation (useful for troubleshooting)
VM_ACCEL=tcg vagrant up

# High-performance development VM
VM_TYPE=dev VM_CPUS=8 VM_MEM=16 vagrant up
```

### Changing Acceleration After VM Creation

To change hardware acceleration, you must halt and restart the VM:

```bash
# Switch to software emulation
vagrant halt && VM_ACCEL=tcg vagrant up

# Switch back to hardware acceleration (Linux)
vagrant halt && VM_ACCEL=kvm vagrant up

# Switch back to hardware acceleration (macOS)
vagrant halt && VM_ACCEL=hvf vagrant up
```

**Note:** `vagrant reload` does NOT apply acceleration changes.

Vagrant will download the [cloud-image/ubuntu-24.04] base box, provision the VM, and install all required dependencies including clang19. This process may take some time on first run.

## Accessing the VM

Once the VM is up and running, you can access it via SSH:

```bash
vagrant ssh
```

This will place you in the `/vagrant` directory inside the VM, which is synced with the Tracee directory on your host machine.

### Shared Folder Implementation

The VM uses a sophisticated shared folder system that varies by host OS:

- **Linux**: Direct 9p mount with UID/GID mapping for optimal performance
- **macOS**: 9p + bindfs overlay for proper ownership mapping

**Performance Note for macOS users:** File operations may be slower due to the bindfs overlay. For large Git operations or extensive file processing, consider running them outside the VM.

## Build and Run Tracee

You can now build Tracee within the VM using the provided Makefile. Consult the Tracee documentation for specific build instructions.
[Building Tracee Documentation](./building/building.md)

## Stopping the VM

To stop the VM, use:

  ```bash
  vagrant halt
  ```

## Destroying the VM

To completely remove the VM, use:

  ```bash
  vagrant destroy
  ```

## Troubleshooting

### Hardware Acceleration Issues

**KVM not available (Linux):**
```bash
# Check if KVM is supported
ls /dev/kvm
# If missing, enable virtualization in BIOS/UEFI
# Install KVM packages:
sudo apt install qemu-kvm libvirt-daemon-system
```

**HVF not available (macOS):**
```bash
# Check if HVF is supported
sysctl -n kern.hv_support
# Ensure you're on macOS 10.10+ with Intel CPU or Apple Silicon
```

**Fallback to software emulation:**
If hardware acceleration is not available, the VM automatically falls back to TCG software emulation. This is slower but functional.

### Shared Folder Issues

**Files not visible in `/vagrant`:**
```bash
# Check mount status inside VM
mount | grep vagrant

# For Linux hosts - check direct 9p mount
mount | grep 9p

# For macOS hosts - check bindfs mount
mount | grep bindfs
```

**Permission issues (macOS):**
```bash
# Inside VM, check ownership
ls -la /vagrant

# Should show vagrant:vagrant (1000:1000)
# If not, check systemd mount services (macOS hosts):
sudo systemctl status mnt-shared.mount
sudo systemctl status vagrant.mount
```

### Performance Issues

**Slow file operations (macOS):**
- This is expected due to bindfs overlay
- Consider running large operations outside VM
- Use Git inside VM for better performance with large repositories

**VM performance is slow:**
- Ensure hardware acceleration is enabled
- Check: `vagrant ssh` then `cat /proc/cpuinfo | grep flags` for virtualization features
- Increase VM resources: `VM_CPUS=8 VM_MEM=16 vagrant up`

### Network and Port Issues

**Port conflicts:**
```bash
# Check actual forwarded ports
vagrant ssh-config
ps aux | grep qemu-system | grep hostfwd
```

**SSH connection issues:**
```bash
# Force VM restart
vagrant halt --force && vagrant up

# Check VM status
vagrant status
```

### General Issues

**VM stuck or unresponsive:**
```bash
# Force halt and restart
vagrant halt --force
vagrant up
```

**Provisioning failures:**
```bash
# Re-run provisioning
vagrant provision

# Or destroy and recreate
vagrant destroy -f && vagrant up
```

**QEMU/plugin issues:**
```bash
# Ensure vagrant-qemu plugin is installed
vagrant plugin list | grep qemu

# Reinstall if needed
vagrant plugin uninstall vagrant-qemu
vagrant plugin install vagrant-qemu
```

[HashiCorp Vagrant]: https://www.vagrantup.com/
[cloud-image/ubuntu-24.04]: https://portal.cloud.hashicorp.com/vagrant/discover/cloud-image
[QEMU]: https://www.qemu.org/
[vagrant-qemu plugin]: https://github.com/ppggff/vagrant-qemu
[Hypervisor]: https://en.wikipedia.org/wiki/Hypervisor

---

### Contributing Testing Coverage (`docs/contributing/testing-coverage.md`)

# Code Coverage in Tracee

This document explains how code coverage is configured and used in Tracee.

## Overview

Tracee uses Go's built-in coverage tools combined with Codecov for comprehensive test coverage reporting. Coverage is tracked for multiple test types to ensure quality across all components.

## Coverage Types

### 1. Unit Tests Coverage

- **Target**: Maintain current baseline, 60% for new code
- **Scope**: Core Go logic, utilities, and non-eBPF components
- **Files**: `cmd/`, `pkg/`, `signatures/`
- **Command**: `make test-unit`

### 2. Integration Tests Coverage

- **Scope**: eBPF + Go integration, full system testing
- **Command**: `make test-integration`

## Local Development

### View Coverage Summary
```bash
make coverage
```

### Generate HTML Coverage Reports
```bash
make coverage-html
# Opens coverage.html and types/coverage.html
```

### Run Individual Coverage Commands
```bash
# Unit tests only
go test ./... -coverprofile=coverage.txt -covermode=atomic

# View coverage percentage
go tool cover -func=coverage.txt

# Generate HTML report
go tool cover -html=coverage.txt -o coverage.html
```

## CI/CD Integration

Coverage is automatically collected and reported on:
- Every Pull Request
- Every push to main branch
- Integration tests (when run)

### Codecov Integration

Coverage reports are uploaded to [Codecov](https://codecov.io) with the following flags:

- `unit`: Unit test coverage
- `integration`: Integration test coverage

### Coverage Configuration

Coverage behavior is configured in `codecov.yml`:

- Project coverage target: Auto (maintains current baseline)
- Patch coverage target: 60% for new code
- Automatic PR comments with coverage diff
- Excludes test files, generated code, and vendor dependencies

## Coverage Files

The following coverage files are generated (and gitignored):

- `coverage.txt`: Main unit test coverage
- `integration-coverage.txt`: Integration test coverage
- `coverage.html`: HTML report for local viewing

## Best Practices

1. **Focus on Critical Paths**: Prioritize coverage for core event processing, filtering, and detection logic
2. **Test Edge Cases**: Include error handling and boundary conditions
3. **Integration Coverage**: Ensure eBPF + Go interactions are tested
4. **Meaningful Tests**: Aim for tests that verify behavior, not just coverage numbers

## Excluded from Coverage

- Generated files (`*.pb.go`)
- Test files (`*_test.go`)
- Mock files (`mock_*.go`, `*_mock.go`)
- Vendor dependencies (`vendor/`, `3rdparty/`)
- Documentation and build scripts
- Test utilities and test data

## Troubleshooting

### Coverage Not Generated

- Ensure you're running tests with `-coverprofile` flag
- Check that `-covermode=atomic` is set (required for concurrent programs)

### Low Coverage Warnings

- Review which functions/lines are not covered
- Consider if uncovered code represents important paths
- Add tests for critical uncovered functionality

### Codecov Upload Failures

- Verify `CODECOV_TOKEN` is set in repository secrets
- Check that coverage files exist before upload
- Review GitHub Actions logs for specific error messages

---

### Docs Forensics (`docs/docs/forensics.md`)

# Getting Started with Forensics

Tracee has a unique feature that lets you capture interesting artifacts from
running applications, using the `--artifacts` flag.

```console
./dist/tracee man artifacts
```

```console
sudo ./dist/tracee --artifacts xxx
```

!!! Tip
    All captured artifacts are saved in Tracee's "output directory", which can
    be configured using `--artifacts dir.path=/path/to/dir`. You may also use
    `--artifacts dir.clear` if you want contents of the destination directory
    to be cleared every time you execute tracee.

## Artifacts Types

Tracee can capture the following types of artifacts:

1. **I/O Files**

    Anytime a file is being written to and/or read from, the contents of the
    file will be captured. I/O files can be filtered using 3 optional filters:
    1. path - prefix of the file written/read. Up to 3 path filters can be
       provided per capture type.
    2. type - file's type can be `pipe`, `socket`, `elf` or `regular`.
    3. fd - standard FD, one of the following: `stdin`, `stdout` and `stderr`.

    ***write example***
    ```console
    sudo ./dist/tracee \
       --output json \
       --scope comm=bash \
       --scope follow \
       --enrichment decoded-data \
       --artifacts dir.path=/tmp/tracee \
       --artifacts file-write.filters=path=/tmp/*
    ```

    ```console
    echo write testing 123 > /tmp/testing.txt
    ```

    ```
/* Detailed source-code truncated for AI context efficiency. */
```

    !!! Note
        You can read captured files written at `/tmp/tracee/out`:
        ```console
        sudo cat /tmp/tracee/out/host/write.dev-271581185.inode-1966101
        ```

        ```text
        write testing 123
        ```
    
    ***read example***

    ```console
    sudo ./dist/tracee \
       --output json \
       --scope comm=bash \
       --scope follow \
       --enrichment decoded-data \
       --artifacts dir.path=/tmp/tracee \
       --artifacts file-read.filters=type=pipe \
       --artifacts file-read.filters=fd=stdin
    ```

    ```console
    echo read testing 123 | cat
    ```
   
    ```
/* Detailed source-code truncated for AI context efficiency. */
```

!!! Note
    You can read captured files read at `/tmp/tracee/out`:
    ```console
    sudo cat /tmp/tracee/out/host/read.dev-12.inode-176203
    ```

    ```text
    read testing 123
    ```

1. **Executed Files**

    Anytime a **binary is executed**, the binary file will be captured. If the
    same binary is executed multiple times, it will be captured just once.

    ```console
    sudo ./dist/tracee \
       --output json \
       --scope comm=bash \
       --scope follow \
       --enrichment decoded-data \
       --artifacts dir.path=/tmp/tracee \
       --artifacts executable
    ```

    ```console
    /bin/ls
    ```

    ```
/* Detailed source-code truncated for AI context efficiency. */
```

    !!! Note
        You will have a copy of each executed file written at `/tmp/tracee/out`:
        ```console
        ldd /bin/ls
        ```

        ```text
        linux-vdso.so.1 (0x00007ffca632c000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f9a930d5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a92ead000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9a92e16000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9a93136000)
        ```

        ```console
        ldd /tmp/tracee/out/host/exec.1657322052835478987.ls
        ```

        ```text
        linux-vdso.so.1 (0x00007ffe337fb000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007feeb1fa5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feeb1d7d000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007feeb1ce6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007feeb2006000)

        ```console
        sudo chmod +x /tmp/tracee/out/host/exec.1657322052835478987.ls
        /tmp/tracee/out/host/exec.1657322052835478987.ls
        ```

1. **Memory Files**

    Anytime a **memory unpacker** is detected, the suspicious **memory region**
    will be captured. This is triggered when memory protection changes from
    **Write+Execute** to **Write**.

    ```console
    sudo ./dist/tracee \
       --output none \
       --scope comm=bash \
       --scope follow \
       --artifacts dir.path=/tmp/tracee \
       --artifacts memory-regions
    ```

    !!! Note
        You may opt not to have any output from **tracee** with `--output none`
        command flag is given. This makes tracee to work in capture mode only.

1. **Network PCAP Files**

    Anytime a **network packet** is delivered to a process, traced by tracee,
    this packet might be captured into one or multiple pcap files.

    !!! Attention
        The default behavior when capturing network traffic is to capture
        ALL traffic, despite given event filters. If you want to make
        capture feature to follow the given event filters, like for example
        capturing DNS events only, then you have to provide `--artifacts
        network.pcap.options=filtered` argument in the command line. Then only
        net_packet_XXX events will be captured (IPv4, IPv6, TCP, UDP,
        ICMP, ICMPv6, DNS, HTTP, etc).

    A good way to test this behavior is to execute:

    ```console
    sudo ./dist/tracee \
        --events net_packet_ipv4 \
        --artifacts network \
        --artifacts network.pcap.options=filtered
    ```

    and observe a single **pcap file** for all ipv4 packets created:

    ```console
    find /tmp/tracee/out/pcap/
    ```

    ```text
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    ```

    You can select only dns packets, for example:

    ```console
    sudo ./dist/tracee \
        --events net_packet_dns \
        --artifacts network \
        --artifacts network.pcap.options=filtered
    ```

    and the file `/tmp/tracee/out/pcap/single.pcap` would only contain DNS
    related packets:

    ```console
    find /tmp/tracee/out/pcap/
    ```

    ```text
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    ```

    ```console
    sudo tcpdump -n -r /tmp/tracee/out/pcap/single.pcap | head -2
    ```

    ```text
    reading from file /tmp/tracee/out/pcap/single.pcap, link-type NULL (BSD loopback), snapshot length 262144
    16:53:48.870629 IP 127.0.0.1.55569 > 127.0.0.53.53: 33361+ [1au] A? www.uol.com.br. (43)
    16:53:48.870690 IP 127.0.0.1.55569 > 127.0.0.53.53: 25943+ [1au] AAAA? www.uol.com.br. (43)
    ```

    A great thing is that you may have multiple pcap files, divided by:

    1. single: a single pcap file containing all packets (the default)
    1. process: one file per process executed, ordered by host and container
    1. container: one file for the host and one pcap file per container
    1. per-command: one file per command executed (even if multiple times)

    and you can even have multiple ways at the same time. Example: a ping
    command is executed inside a container. You want to summarize captured
    traffic per container and per command. You will find the same captured
    data for that ping command inside `commands/container_id/ping.pcap` and
    inside `containers/container_id.pcap`.

    ```console
    sudo ./dist/tracee \
        --events net_packet_icmp \
        --artifacts network \
        --artifacts network.pcap.options=filtered \
        --artifacts network.pcap.split=process,container,command
    ```

    ```console
    cd /tmp/tracee/out
    find pcap
    ```

    ```text
    pcap
    pcap/commands
    pcap/commands/b86533d11f3
    pcap/commands/b86533d11f3/ping.pcap
    pcap/commands/host
    pcap/commands/host/sshd.pcap
    pcap/commands/host/zerotier-one.pcap
    pcap/commands/host/node.pcap
    pcap/commands/fd95a035ce5
    pcap/commands/fd95a035ce5/ping.pcap
    pcap/processes
    pcap/processes/b86533d11f3
    pcap/processes/b86533d11f3/ping_1261180_1663772450241192.pcap
    pcap/processes/host
    pcap/processes/host/node_186708_1573567360495399.pcap
    pcap/processes/host/node_1196826_1662656211119567.pcap
    pcap/processes/host/zerotier-one_7882_137007714376.pcap
    pcap/processes/host/sshd_1196773_1662654999660718.pcap
    pcap/processes/fd95a035ce5
    pcap/processes/fd95a035ce5/ping_1261163_1663769383806467.pcap
    pcap/containers
    pcap/containers/host.pcap
    pcap/containers/b86533d11f3.pcap
    pcap/containers/fd95a035ce5.pcap
    ```

    you can see the packets by executing tcpdump on any pcap file:

    ```console
    tcpdump -r pcap/containers/b86533d11f3.pcap
    ```

    ```text
    reading from file pcap/containers/b86533d11f3.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    ```

    ```console
    tcpdump -r pcap/commands/b86533d11f3/ping.pcap
    ```

    ```text
    reading from file pcap/commands/b86533d11f3/ping.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    ```

    !!! Note
        Note that the same packets were written to 2 different pcap files: the
        pcap file describing the container `b86533d11f3` (because it was
        executing a single process: ping) and the pcap file describing ANY ping
        command executed in that container (commands/b86533d11f3/ping.pcap).

    The format for the pcap filenames inside `output_dir` is the following:

    1. **single**:  
       ./pcap/single.pcap
    1. **processes**:  
       ./pcap/processes/`container_id`/`process_comm`/`host_tid`/`task_starttime`.pcap
    1. **containers**:  
       ./pcap/containers/`container_id`.pcap
    1. **commands**:  
       ./pcap/commands/`container_id`/`process_comm`.pcap

    !!! Attention
        By default, all pcap files will contain packets with headers only. That
        might too little for introspection, since sometimes one might be
        interested in a few bytes of the captured packet (or event it all). Next
        item shows how to capture a specific packet payload size.

    In order to capture a specific payload size you may specify:

    ```console
    sudo ./dist/tracee \
        --events net_packet_tcp \
        --artifacts network \
        --artifacts network.pcap.options=filtered \
        --artifacts network.pcap.split=single,command \
        --artifacts network.pcap.snaplen=default
    ```

    To capture packet headers + 96 bytes of payload. Or replace `default` by:

    1. headers: capture up to L4 headers only
    1. max: full sized packets into pcap. WARNING: big pcap files.
    1. **256b**, **512b**, **1024b**, ... (any number plus "b")
    1. **16kb**, **32kb**, **64kb**,  ... (any number plus "kb")

    > when specifying a payload size, it refers to the payload AFTER the layer4
    > headers (and not the entire packet length).

1. **Loaded Kernel Modules**

    Anytime a **kernel module** is loaded, the binary file will be captured.
    If the same binary is loaded multiple times, it will be captured just once.

    ```console
    sudo ./dist/tracee \
        --output none \
        --scope comm=bash \
        --scope follow \
        --artifacts dir.clear \
        --artifacts kernel-modules
    ```

    Captured module will be found in tracee destination directory, just like
    any other captured file would be:

    ```console
    sudo ls /tmp/tracee/out/host
    ```

    ```text
    module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    ```

    AND, the captured module is an exact copy of the loaded module:

    ```console
    sudo rmmod lkm_example
    sudo insmod /tmp/tracee/out/host/module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    ```

    ```console
    lsmod | grep example

    ```text
    lkm_example            16384  0
    ```

    ```console
    sudo rmmod lkm_example
    ```

    you can even load/unload it.

    !!! Note
        Example kernel module taken from [this blog]

[this blog]: https://blog.sourcerer.io/writing-a-simple-linux-kernel-module-d9dc3762c234

1. **BPF programs**

    Wheneber a **BPF program** is loaded, the BPF bytecode will be captured.
    This captured bytecode represents the BPF program as it was loaded by the loading process.
    It is basically the BPF section of the compiled ELF that loads the BPF program, which contains the program instructions themselves.
    You can disassemble the bytecode with the help of `binutils-bpf` package and the following command line:
    `objdump -D -b binary -m bpf <path>`

     ```text
     $ sudo ./dist/tracee \
        --output none \
        --scope comm=bash \
        --scope follow \
        --artifacts dir.clear \
        --artifacts bpf-programs
     ```

    Captured bpf bytecode will be found in tracee destination directory, just like
    any other captured file would be:

     ```text
     $ sudo ls /tmp/tracee/out/host
       bpf.name-test_prog.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
     ```
   The hex value after the last "." is the hash of the bpf bytecode.

---

### Docs Security Model (`docs/docs/security-model.md`)

# Tracee's Security Model

## Introduction

Understanding Tracee's security model is essential for setting proper expectations about what Tracee can and cannot protect against. This document describes Tracee's adversary model, security boundaries, detection capabilities, and limitations to help you make informed decisions about deploying and using Tracee in your security infrastructure.

Tracee is an eBPF-based runtime security tool designed to detect threats and suspicious behavior by monitoring kernel events. However, like any security tool, it operates under specific assumptions and has inherent limitations based on its architecture and the adversary it targets.

## Core Security Assumptions

Tracee's security model is built on one fundamental assumption:

**The kernel is trusted when Tracee starts.**

This means:

- The kernel has not been compromised before Tracee loads its eBPF programs
- No kernel rootkits or malicious kernel modules are already present
- The kernel's integrity is intact at Tracee startup time

If this assumption is violated (i.e., the kernel is already compromised), Tracee's detection capabilities cannot be guaranteed. A compromised kernel has complete control over the system and can potentially:

- Interfere with eBPF program execution
- Hide events from Tracee
- Manipulate Tracee's data structures
- Disable Tracee entirely

## Two-Tier Adversary Model

Tracee's security guarantees differ significantly depending on whether the adversary operates in userspace or kernel space.

### Userspace Adversaries: Strong Security Boundary

**Target Profile:** Root-privileged adversaries operating in userspace, including containerized environments.

**Tracee's Capabilities:**

Tracee provides strong security boundaries against userspace adversaries through eBPF's architectural guarantees:

**Tamper-Resistant Monitoring**: eBPF programs run in the kernel with their own memory protection. While Tracee's eBPF programs are active and attached:

- Userspace processes cannot bypass the monitoring or evade the attached hooks through normal system operations
- The eBPF bytecode itself cannot be modified once loaded and verified
- Events are captured at the kernel level before userspace can manipulate them
- **However**: Root users with appropriate capabilities (CAP_BPF, CAP_SYS_ADMIN) can administratively disable the monitoring by detaching eBPF programs, killing the Tracee process, or accessing eBPF maps. Tracee's effectiveness assumes the monitoring infrastructure remains intact and that Tracee itself is protected by proper access controls

**Complete Visibility**: Tracee can observe all system calls, LSM hooks, and kernel events from userspace processes, providing comprehensive monitoring of:

- Process execution and lifecycle
- File system operations
- Network activity
- Container escape attempts
- Privilege escalation attempts
- Suspicious behaviors and exploitation techniques

**Security Guarantee:** Userspace adversaries with root privileges cannot evade Tracee's monitoring or bypass its detection mechanisms through userspace operations alone (assuming Tracee itself remains protected and operational).

### Kernel-Level Adversaries: Best-Effort Detection

**Target Profile:** Adversaries with kernel-level access, including rootkits, malicious kernel modules, and kernel zero-day exploits.

**Tracee's Capabilities:**

Against kernel-level adversaries, Tracee provides **best-effort detection** but cannot guarantee protection:

**Rootkits Loaded Before Tracee**: Cannot be reliably detected. If a rootkit compromises the kernel before Tracee starts, it may already have mechanisms to hide from detection or interfere with Tracee's operation.

**Rootkits Loaded After Tracee**: May be detected using known signatures and heuristics. Tracee includes detection capabilities for:

- System call table hooking
- Kernel module loading
- Hidden kernel modules
- Ftrace hooks
- /proc filesystem hooks
- Suspicious kernel modifications

**Kernel Zero-Day Exploits**: Cannot be prevented or reliably detected. An attacker exploiting unknown kernel vulnerabilities can potentially:

- Bypass Tracee's monitoring
- Disable eBPF programs
- Gain complete system control
- Hide their activities from detection

**Limitations:**

- Detection is signature-based and heuristic-driven
- Novel rootkit techniques may evade detection
- Sophisticated kernel-level malware may disable or circumvent Tracee
- Cannot protect against attackers who control the kernel

**Security Guarantee:** Tracee attempts to detect kernel-level threats but cannot guarantee detection or prevention against sophisticated kernel adversaries.

## Security Boundaries and Attack Vectors

### Protected by eBPF Security Model

The following attack vectors are effectively monitored by Tracee due to eBPF's security guarantees:

- **System Calls**: All syscall invocations are observable
- **LSM Hooks**: Linux Security Module hooks provide tamper-resistant checkpoints
- **Network Events**: Packet-level visibility for monitored processes
- **File Operations**: Comprehensive file system activity monitoring
- **Process Lifecycle**: Complete process tree visibility
- **Container Operations**: Container creation, modification, and escape attempts

### Limited or No Protection

The following attack vectors have limited or no protection guarantees:

- **Kernel Memory Manipulation**: Direct kernel memory writes can bypass detection
- **Hardware-Level Attacks**: CPU vulnerabilities, firmware-level compromises
- **Pre-Boot Compromises**: Bootkit, firmware rootkits installed before OS loads
- **Kernel Zero-Days**: Unknown vulnerabilities in the kernel itself
- **eBPF Verifier Bypasses**: Theoretical vulnerabilities in the eBPF verifier (very rare)

## Race Conditions and TOCTOU

### The Problem

When Tracee reads information from user programs, it is subject to a **race condition** where the user program might be able to change the arguments after Tracee reads them. This is known as a Time-of-Check-Time-of-Use (TOCTOU) vulnerability.

Consider this scenario:

A program invokes:

```c
execve("/bin/ls", NULL, 0)
```

Tracee picks this up and reports it. However, the program could change the first argument from `/bin/ls` to `/bin/bash` after Tracee reads it but before the kernel executes it. This means Tracee would report `/bin/ls` while the kernel actually executes `/bin/bash`.

### The Solution: LSM Events

To mitigate this race condition, Tracee provides **LSM (Linux Security Module)** based events. These events occur at security check time within the kernel, providing more reliable data that's harder for user-space programs to manipulate.

#### How to Use LSM Events

For critical security monitoring, you should cross-reference regular syscall events with their corresponding LSM events:

**Example: Process Execution Monitoring**

Instead of relying solely on the `execve` syscall event, also monitor:

- `security_bprm_check` - LSM hook that validates executables before execution

By comparing both events, you can detect potential tampering:

- If the `execve` event shows `/bin/ls`
- But `security_bprm_check` shows `/bin/bash`
- This indicates a TOCTOU attack attempt

#### When to Use LSM Events

Use LSM-based events for:

- **Security-critical monitoring** where argument integrity is essential
- **Detecting evasion techniques** that exploit TOCTOU vulnerabilities
- **Compliance requirements** that demand tamper-resistant auditing

### Best Practices

1. **Defense in Depth**: Use both syscall and LSM events for critical monitoring
2. **Cross-Reference**: Compare data between event types to detect anomalies
3. **Document Assumptions**: If using only syscall events, document the TOCTOU risk
4. **Test Detection**: Verify your policies catch TOCTOU evasion attempts

## Attack Scenarios

### Scenario 1: Container Escape Attempt (Detectable)

**Attack**: An adversary with root privileges inside a container attempts to escape using a known technique (e.g., cgroup release_agent modification).

**Tracee's Response**:

- ✅ Detects the suspicious cgroup modification
- ✅ Triggers `cgroup_release_agent_modification` signature
- ✅ Provides full context: container ID, process tree, file modifications
- ✅ Alerts security teams for immediate response

**Outcome**: Attack detected with full forensic context for response.

### Scenario 2: Rootkit Loaded Before Tracee (Not Detectable)

**Attack**: A kernel rootkit is installed on the system before Tracee starts. The rootkit hooks system calls to hide malicious processes.

**Tracee's Response**:

- ❌ Cannot guarantee detection
- ⚠️ May detect some symptoms if rootkit is imperfect
- ⚠️ Initial system scan may identify some inconsistencies

**Outcome**: Rootkit may remain undetected. **Mitigation**: Deploy Tracee early in the boot process, ideally as part of the init system or container runtime startup.

### Scenario 3: Rootkit Loaded After Tracee (Potentially Detectable)

**Attack**: An adversary attempts to load a kernel module containing a rootkit after Tracee is running.

**Tracee's Response**:

- ✅ Detects kernel module loading event
- ✅ Triggers `kernel_module_loading` signature
- ✅ May detect system call table hooking via `hooked_syscall` event
- ✅ May identify hidden kernel modules via `hidden_kernel_module` event
- ⚠️ Sophisticated rootkits may still evade detection using novel techniques

**Outcome**: Good chance of detection for known techniques, but not guaranteed against advanced adversaries.

### Scenario 4: Privilege Escalation via Exploit (Detectable)

**Attack**: An adversary exploits a userspace vulnerability to escalate privileges (e.g., dirty pipe, container escape via exposed socket).

**Tracee's Response**:

- ✅ Detects suspicious capability usage
- ✅ Identifies unusual process behaviors
- ✅ Monitors credential changes
- ✅ Tracks file and memory modifications

**Outcome**: Attack detected with high confidence.

### Scenario 5: Evading Active Monitoring (Not Possible Without Disabling Tracee)

**Attack**: An adversary with root privileges attempts to evade Tracee's monitoring while performing malicious activities, without administratively disabling Tracee first.

**Tracee's Response**:

- ✅ While eBPF programs are attached and active, userspace operations cannot bypass the monitoring hooks
- ✅ System calls, file operations, network activity, and process events are captured at the kernel level
- ✅ Attempts to manipulate data after capture don't affect what Tracee already observed
- ⚠️ However, an adversary with CAP_BPF or CAP_SYS_ADMIN capabilities can administratively disable Tracee by:
  - Detaching eBPF programs from their hooks
  - Killing the Tracee userspace process
  - Unloading eBPF programs or modifying eBPF maps
- ⚠️ Such administrative actions to disable monitoring may themselves be detectable if other security monitoring is in place

**Outcome**: Evasion of active monitoring is not possible through normal userspace operations. However, an adversary with sufficient capabilities can disable the monitoring infrastructure itself, which is why protecting Tracee's operational integrity is critical.

## Detection Capabilities Overview

| Threat Category | Detection Level | Notes |
|-----------------|----------------|-------|
| Container Escapes | High | Comprehensive monitoring of container boundaries |
| Privilege Escalation | High | Capability and credential monitoring |
| Suspicious Process Behavior | High | Process tree and execution pattern analysis |
| Malicious File Operations | High | Complete file system activity visibility |
| Network Attacks | Medium-High | Packet-level visibility for monitored processes |
| Kernel Module Loading | High | Immediate detection of module loads |
| Rootkits (pre-Tracee) | Low | Cannot reliably detect pre-existing compromises |
| Rootkits (post-Tracee) | Medium | Signature-based detection of known techniques |
| Kernel Zero-Days | Low | Cannot detect unknown kernel vulnerabilities |
| Hardware Attacks | None | Outside Tracee's scope |

## Mitigation Strategies

### Maximize Protection

1. **Deploy Early**: Start Tracee as early as possible in the boot sequence to establish trust before potential compromises
2. **Use LSM Events**: Prefer LSM-based events for security-critical monitoring to avoid TOCTOU vulnerabilities
3. **Enable All Rootkit Signatures**: Activate all kernel integrity monitoring signatures (hooked_syscall, hidden_kernel_module, etc.)
4. **Layered Security**: Combine Tracee with other security controls:
   - Secure boot to verify kernel integrity
   - Kernel module signing to prevent unauthorized module loads
   - SELinux/AppArmor for mandatory access control
   - Network segmentation
   - Regular vulnerability scanning

### Acknowledge Limitations

1. **Understand Boundaries**: Be aware that kernel-level compromises can bypass Tracee
2. **Verify Kernel Integrity**: Use external tools to verify kernel integrity before trusting Tracee's output
3. **Monitor Tracee Itself**: Watch for signs that Tracee may have been tampered with (e.g., unexpected crashes, missing events)
4. **Regular Updates**: Keep Tracee, the kernel, and eBPF subsystem updated to benefit from latest security fixes
5. **Incident Response Plan**: Have procedures for handling scenarios where Tracee's integrity may be compromised

### Operational Best Practices

1. **Start with Known-Good State**: Deploy Tracee on freshly provisioned or verified clean systems
2. **Baseline Normal Behavior**: Establish baselines before deploying to production
3. **Alert Tuning**: Configure appropriate alert thresholds to balance detection and false positives
4. **Regular Review**: Periodically review detected events and tune signatures
5. **Forensic Capabilities**: Enable artifact capture for post-incident analysis

## Related Documentation

- [Capabilities](install/capabilities.md) - Learn about running Tracee with least privileges
- [Rootkit Detection Events](events/builtin/security-events.md) - Detailed documentation on kernel integrity monitoring
- [Policies](policies/index.md) - Create policies that implement defense-in-depth
- [Forensics](forensics.md) - Capture artifacts for investigation

## Summary

Tracee provides **strong security guarantees against userspace adversaries** through eBPF's tamper-resistant architecture. However, it offers only **best-effort detection against kernel-level adversaries** and cannot protect against attackers who control the kernel before Tracee starts or exploit kernel zero-day vulnerabilities.

By understanding these boundaries and implementing layered security controls, you can maximize Tracee's effectiveness as part of a comprehensive security strategy.

---

### Docs Troubleshooting (`docs/docs/troubleshooting.md`)

# Troubleshooting

This guide helps you diagnose and resolve common issues when running Tracee.

## Installation Issues

### BTF Not Available

**Problem**: Error message about missing BTF support
```
ERRO error loading eBPF program: BTF is required
```

**Solution**: 

1. Check if BTF is available: `ls /sys/kernel/btf/vmlinux`
2. If missing, upgrade to a kernel version that includes BTF support (most modern distributions)
3. See [Prerequisites](install/prerequisites.md#btf) for more details

### Permission Denied

**Problem**: Tracee fails to start with permission errors
```
ERRO permission denied loading eBPF program
```

**Solutions**:

1. **Run as root**: `sudo tracee`
2. **Use required capabilities**:
   ```bash
   # Docker - minimal required capabilities
   docker run \
     --cap-add SYS_RESOURCE \
     --cap-add SYS_ADMIN \
     --cap-add SYS_PTRACE \
     --cap-add NET_ADMIN \
     --cap-add SYSLOG \
     tracee
   
   # Kubernetes
   securityContext:
     capabilities:
       add: 
         - "SYS_RESOURCE"
         - "SYS_ADMIN" 
         - "SYS_PTRACE"
         - "NET_ADMIN"
         - "SYSLOG"
   ```
3. **For recent kernels (>=5.8)**, you may use `CAP_BPF` + `CAP_PERFMON` instead of `CAP_SYS_ADMIN`
4. See [Prerequisites](install/prerequisites.md#process-capabilities) for complete details and justifications

### Kernel Version Incompatibility

**Problem**: Unsupported kernel version errors

**Solution**: 

- Check supported kernels: [Prerequisites](install/prerequisites.md#kernel-version)
- Tracee requires kernel 5.4 or newer (4.18 for RHEL 8)
- Consider upgrading to a supported kernel version

## Runtime Issues

### No Events Generated

**Problem**: Tracee starts but produces no events

**Troubleshooting steps**:

1. **Check scope filters**:
   ```bash
   # Test with minimal configuration
   tracee --events execve
   ```

2. **Verify events are enabled**:
   ```bash
   # List enabled events
   tracee --events help
   ```

3. **Check policy configuration**:
   ```yaml
   # Ensure policy scope matches your workload
   scope:
     - container  # Or specific process filters if needed
   ```

### High CPU Usage

**Problem**: Tracee consuming excessive CPU

**Solutions**:

1. **Reduce event scope**:
   ```yaml
   scope:
     - container  # Limit to containers only
   ```

2. **Filter events**:
   ```yaml
   rules:
     - event: execve
       filters:
         - uid!=0  # Example: exclude root processes
   ```

3. **Use specific events instead of sets**:
   ```yaml
   rules:
     - event: execve        # Specific event
     # - event: tag=syscalls    # Avoid broad tags
   ```

## Container Issues

### Container Events Not Captured

**Problem**: Missing container-related events

**Solutions**:

1. **Mount container runtime socket**:
   ```bash
   # Docker
   docker run -v /var/run/docker.sock:/var/run/docker.sock tracee
   
   # Containerd
   docker run -v /run/containerd/containerd.sock:/run/containerd/containerd.sock tracee
   ```

## Performance Issues

### Events Being Dropped

**Problem**: Warning about dropped events
```
WARN events dropped due to buffer overflow
```

**Solutions**:

1. **Increase buffer size**:
   ```bash
   tracee --buffers kernel-events=1024
   ```

2. **Reduce event frequency**:
   ```yaml
   rules:
     - event: openat
       filters:
         - data.pathname!=/tmp/*  # Filter noisy paths
   ```

### Slow Event Processing

**Problem**: Events arrive with significant delay

**Solutions**:

1. **Check system load**: Use `top`, `htop` to verify system isn't overloaded
2. **Optimize event selection**: Use specific events instead of broad event sets

## Output Issues

### JSON Parsing Errors

**Problem**: Invalid JSON output

**Solutions**:

1. **Use proper output format**:
   ```bash
   tracee --output json --enrichment decoded-data
   ```

2. **Check for mixed output**:
   ```bash
   # Separate logs from events
   tracee --output json --logging file=/var/log/tracee.log
   ```

### Missing Event Fields

**Problem**: Expected fields not present in events

**Solutions**:

1. **Enable argument parsing**:
   ```bash
   tracee --enrichment decoded-data
   ```

2. **Check event definition**: Some events may not include all expected fields

## Debugging

### Enable Debug Logging

```bash
# Enable debug logs
tracee --logging level=debug

# Or via environment
TRACEE_LOG_LEVEL=debug tracee
```

### Capture System Information

```bash
# System info for bug reports
tracee --version
uname -a
cat /etc/os-release
ls -la /sys/kernel/btf/vmlinux
```

### Test Minimal Configuration

```bash
# Minimal test configuration
sudo tracee --events execve
```

## Getting Help

If you continue experiencing issues:

1. **Search existing issues**: [GitHub Issues](https://github.com/aquasecurity/tracee/issues)
2. **Check discussions**: [GitHub Discussions](https://github.com/aquasecurity/tracee/discussions)
3. **Join Slack**: [Aqua Community Slack](https://slack.aquasec.com)

When reporting issues, include:

- Tracee version (`tracee --version`)
- Operating system and kernel version
- Container runtime (if applicable)
- Complete error messages
- Minimal reproduction steps

---

### Traceectl Usage (`docs/traceectl/usage.md`)

# traceectl Installation and Usage Guide

## Installation

To use **traceectl**, you first need to compile and install the tool. Follow these steps to get started:

### 1. **Clone the Repository**

   Begin by cloning the Tracee repository to your local machine and navigating to traceectl:

   ``` bash
   git clone https://github.com/aquasecurity/tracee.git
   cd cmd/traceectl
   ```

### 2. **Build and Install**

   Compile and install traceectl using the following commands:

   ``` bash
   make
   ```

## Configuring Tracee for traceectl

To use traceectl effectively, you need to configure Tracee so that it can communicate with traceectl over a Unix socket. This can be done by running Tracee with the correct gRPC settings:

### 1. **Run Tracee with gRPC Unix Socket**

Use the following command to start Tracee with gRPC support over a Unix socket:

``` bash
tracee --server grpc-address=unix:/var/run/tracee.sock
```

This command sets up Tracee to listen for incoming connections from traceectl at the specified Unix socket path (`/var/run/tracee.sock`). Ensure that this socket path is accessible and not blocked by permissions or other constraints.

### 2. **Output Flag Configuration**

The `--output` flag in Tracee allows you to configure how data from Tracee is presented. Among the available options, you can specify `none` for minimal output, which can be useful for scenarios where bandwidth or latency considerations are critical. For example:

``` bash
tracee --output none
```

#### Why Use `--output none`?

- **Reduced Bandwidth Usage:** By suppressing output, you can minimize the data transferred over the Unix socket, which is especially helpful in environments with limited resources.
- **Lower Latency:** With no data formatting or transmission overhead, the interaction between traceectl and Tracee becomes faster.

Use this mode for performance testing, silent monitoring, or when integrating traceectl with other systems that handle data processing separately.

This command sets up Tracee to listen for incoming connections from traceectl at the specified Unix socket path (`/var/run/tracee.sock`).
Ensure that this socket path is accessible and not blocked by permissions or other constraints.

## Usage

Once traceectl is installed and Tracee is running, you can use various commands to interact with Tracee. Below are the main commands provided by traceectl:

- Stream Events: traceectl stream

- Events management: traceectl event

- Retrieve Metrics: traceectl metrics

- Check Version: traceectl version

For more info about the traceectl command please refer to the appoint command documentation

## Flags

For more info about the traceectl flags see:

- [output](./flags/output.md)
- [format](./flags/format.md)
- [server](./flags/server.md)
- [policy](./flags/policy.md)

## Summary

- **Install traceectl** by cloning the repository, building, and installing it with `make`.
- **Use traceectl** to interact with Tracee via commands like `stream`, `event`, `metrics`, and `version`.

---

### Tutorials Additional Resources (`docs/tutorials/additional-resources.md`)

# Additional Resources

This page links to external resources, created by Aqua Security and the community. If you create content on Tracee, we would highly appreciate if you would contribute the link to this page.

## Blog Post
* [The Story of Tracee: The Path to Runtime Security Tool](https://blog.aquasec.com/open-source-container-runtime-security)
* [Detecting eBPF Malware with Tracee](https://blog.aquasec.com/detecting-ebpf-malware-with-tracee)

## Useful Videos

* [Tracee Livestream: Everything is an Event in Tracee](https://www.youtube.com/live/keqVe4d71uk?feature=share)
* [Tracee Video Series on the Aqua Open Source YouTube channel](https://www.youtube.com/watch?v=ddkTX9vAHqE&list=PLRdPssLrT8d2Jp71pkgDTjq2fm_8P7Nqk)

Have a look at the recordings on the [Aqua Open Source YouTube channel.](https://www.youtube.com/channel/UCZd5NF4XJRaU-yfextsY-pw)

## Conference recording

* [eBPF Warfare - Detecting Kernel & eBPF Rootkits with Tracee](https://youtu.be/0MjVan5lQd0)
* [Verifiable GitHub Actions with eBPF - Jose Donizetti, Aqua](https://www.youtube.com/watch?v=6vNt9JMU9p4) 
* [Tracee Features and Internals: High Throughput of eBPF Events for Execution Patterns Detections](https://youtu.be/xdEVGT5vAIM)
* [BlackHat Arsenal 2022: Detecting Linux kernel rootkits with Aqua Tracee](https://youtu.be/EATX8g3sh-0)
* [Tracee presentation at BlackHat USA 2021](https://youtu.be/LXP2tdBPNj8)

---

### Tutorials Deploy Grafana Dashboard (`docs/tutorials/deploy-grafana-dashboard.md`)

# Access Tracee Metrics in Prometheus and Grafana

Tracee exposes a metrics endpoint. 
This makes it possible to monitor Tracee like other cloud native workloads.

In this tutorial, we will showcase how to access Tracee metrics through Prometheus and Grafana running through docker containers.
The tutorial can easily adapted to Kubernetes workloads with the configuration provided.

## Tutorial Overview

Grafana is a visualization tools for exported metrics and logs, most commonly
used alongside Prometheus.

Since version 0.7.0, tracee exports useful runtime metrics to Prometheus.

By using Grafana and the new metrics from tracee, we can deploy a simple
dashboard which tracks the tracee instance performance and outputs.

There are two options for accessing Tracee metrics:

* Running the Tracee Docker Container Image -- Shown in this tutorial
* Running the Tracee Helm Chart -- Detailed as part of the [Promtail-Tracee](./promtail.md) tutorial



## Tracee Docker Container Image

These metrics exports are enabled by default in all docker images and can be
enabled using the `--server metrics` flag.

[tracee]: https://github.com/aquasecurity/tracee/tree/{{ git.tag }}/cmd/tracee

### Prerequisites

The following tools must be available for use, they can all be installed either
through docker or installed/built on your machine. Note that you need to be on a Linux machine to follow the Docker tutorial.
Alternative, on a MacBook it is possible to use Vagrant with Parallels as detailed in the following tutorial:
[Running Tracee on Mac with Parallels and Vagrant](./tracee-vagrant.md)

- [Tracee](https://github.com/aquasecurity/tracee/)
- [Prometheus](https://prometheus.io/download/)
- [Grafana](https://grafana.com/docs/grafana/latest/getting-started/getting-started)

### Run Tracee with Metrics Enabled -- The Docker Command

Tracee can be most easily deployed with metrics enabled by default and port
forwarded through the following commands:

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  -p 3366:3366 \
  aquasec/tracee:latest \
  --server metrics
```

Of course, the forwarded metrics ports can be changed, but you should note that
some of the later instructions depend on these ports.

If running Tracee locally through built binaries, the metrics address may be
overrides with the `--listen-addr` flag.

### Run Prometheus and Configure it to Scrape Tracee

Install Prometheus or pull it's Docker image. Then create the following
configuration file, call it `prometheus.yml` to scrape Tracee:

```yaml
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Tracee.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'tracee'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    metrics_path: /metrics

    #Scrape tracee's default metrics hosts.
    #If forwarding different ports make sure to change these addresses.
    static_configs:
      - targets: ['localhost:3366']
```

Note that the localhost:3366 might have to be replaced with your IP address. The IP address can be found with the following command:
```
/sbin/ifconfig
```

We can then start Prometheus with the following command:

```console
prometheus --config.file=/path/to/prometheus.yml
```

Or alternatively with docker:

```console
docker run \
    -p 9090:9090 \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus
```

Ensure that prometheus can scrape the Tracee target: Open the Prometheus UI at `http://localhost:9090`
Under Status < Targets will be the scrape targets listed.

If successful, move to the next step, otherwise consult with the Prometheus documentation.

## Run Grafana to display Tracee's Prometheus Metrics

After successfully deploying Tracee and Prometheus we may now run Grafana to
visualize it's metrics.

Install Grafana using the Grafana container image:

```console
docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise
```

Next, you should be able to access the Grafana UI throug: `http://localhost:3000`.

After entering the website, logging in with username and password `admin` (and
changing your password if you wish), you should see the homepage:



Add your data source by hovering the configuration tab (the gear icon),
selecting "Data Sources" and pressing "Add Data Source" at the top left. Create
a Prometheus Data Source and point it's URL to the relevant location (usually
http://localhost:9090)

Note that you might have to use your local IP address again instead of `localhost`.

You may now either create your own Dashboard or import our default dashboard.

### Import Tracee's Default Dashboard

First download our Grafana Dashboard's json [here].

[here]: https://github.com/aquasecurity/tracee/tree/main/deploy/grafana/tracee.json

After adding the data source hover on the plus icon in the sidebar and select
"Import". Press "Upload JSON File" at the top of the page and select the
downloaded json from your file browser. Change the name and Dashboard UID if
you wish and press "Import" to finish. 

Finally you will be redirected to the dashboard 🥳

---

### Tutorials K8s Policies (`docs/tutorials/k8s-policies.md`)

# Working with Tracee Policies on Kubernetes


## Prerequisites

Before you begin, ensure that you have the following:

- A Kubernetes cluster up and running
- kubectl command-line tool installed and configured to work with your cluster
- Helm v3 or later installed on your local machine

## Install Tracee via Helm

To install Tracee using Helm, follow these steps:

Add the Aqua Security Helm repository:

```console
helm repo add aqua https://aquasecurity.github.io/helm-charts/
```

Install Tracee with the default settings:

```console
helm install tracee aqua/tracee \
    --namespace tracee-system --create-namespace \
    --set hostPID=true
```

This command installs Tracee in the tracee-system namespace, enabling the use of the host's PID namespace.

## Add a new Tracee policy

By default, Tracee comes with a policy for signature events. In this step, you will learn how to add a new policy to suit your requirements.


The `tracee-policies` configmap should have all policies Tracee will load when booting. Let's take a look at the default policy:

```console
kubectl get configmap -n tracee-system

NAME               DATA   AGE
tracee-config      1      58m
tracee-policies    2      58m
```

Let's take a look at the default policy:

```console
kubectl describe configmap/tracee-policies -ntracee-system
```
```yaml
Name:         tracee-policies
Namespace:    tracee-system
Data
====
signatures.yaml:
----
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
	name: signature-events
	annotations:
		description: traces all signature events
spec:
	scope:
	  - global
	rules:
	  - event: stdio_over_socket
	  - event: k8s_api_connection
	  - event: aslr_inspection
	  - event: proc_mem_code_injection
	  - event: docker_abuse
	  - event: scheduled_task_mod
	  - event: ld_preload
	  - event: cgroup_notify_on_release
	  - event: default_loader_mod
	  - event: sudoers_modification
	  - event: sched_debug_recon
	  - event: system_request_key_mod
	  - event: cgroup_release_agent
	  - event: rcd_modification
	  - event: core_pattern_modification
	  - event: proc_kcore_read
	  - event: proc_mem_access
	  - event: hidden_file_created
	  - event: anti_debugging
	  - event: ptrace_code_injection
	  - event: process_vm_write_inject
	  - event: disk_mount
	  - event: dynamic_code_loading
	  - event: fileless_execution
	  - event: illegitimate_shell
	  - event: kernel_module_loading
	  - event: k8s_cert_theft
	  - event: proc_fops_hooking
	  - event: syscall_hooking
	  - event: dropped_executable
```

But let's suppose we also need Tracee to trace all `execve` events. For this, we need to change the configmap `tracee-policies` by adding a new policy.

Let's edit the tracee-policies ConfigMap using kubectl:

```console
kubectl edit cm/tracee-policies -n tracee-system
```

The ConfigMap will open in your default text editor. Locate the data section.

To add a new policy for tracking execve events, add the following YAML block before the signatures.yaml section, maintaining proper indentation:

```yaml
data:
  events.yaml: |-
    apiVersion: tracee.aquasec.com/v1beta1
    kind: Policy
    metadata:
        name: execve-event
        annotations:
          description: traces all execve events
    spec:
        scope:
          - global
        rules:
          - event: execve
  signatures.yaml: |-
  ...
```
Save and close the file. The changes will be applied to the configmap.

!!! note
	If you are having a problem editing the configmap, you can apply it directly with:
	```console
	kubectl apply -f https://gist.githubusercontent.com/josedonizetti/3df19a61d39840441ea5be448d6c9354/raw/c50b9b66d7996bb27b6fac301d24d6390e356f8c/tracee-policies-configmap.yaml
	```

Step 3: Restart Tracee Daemonset
After modifying the Tracee policies, you need to restart the Tracee daemonset for the changes to take effect.

Restart the Tracee daemonset using the following command:

```console
kubectl rollout restart ds/tracee -n tracee-system
```

Wait for the daemonset to restart and stabilize. You can monitor the progress using the following command:

```console
kubectl rollout status ds/tracee -n tracee-system
```

Then check for `execve` events:

```console
kubectl logs -f ds/tracee -n tracee-system | grep execve
```

```json
{"timestamp":1684688250477166817,"threadStartTime":1684688250477064221,"processorId":7,"processId":35694,"cgroupId":1386180,"threadId":35694,"parentProcessId":1033,"hostProcessId":3242201,"hostThreadId":3242201,"hostParentProcessId":3205483,"userId":0,"mountNamespace":4026532829,"pidNamespace":4026532833,"processName":"cri-dockerd","hostName":"minikube","container":{},"kubernetes":{},"eventId":"59","eventName":"execve","matchedPolicies":["execve_event"],"argsNum":2,"returnValue":0,"syscall":"execve","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":"/opt/cni/bin/bridge"},{"name":"argv","type":"const char*const*","value":["/opt/cni/bin/bridge"]},{"name":"envp","type":"const char*const*","value":null}]}
```

Once the daemonset is up and running, the modified policies will be applied.

Congratulations! You have successfully installed Tracee via Helm, modified the default policies to add an `execve` event policy.

Note: Modifying the policies may have security implications, so it is important to carefully consider the events you enable based on your specific requirements and security considerations.

Feel free to reach out if you have any further questions or need additional assistance!

---

### Tutorials Promtail (`docs/tutorials/promtail.md`)

# Using Promtail, Loki and Grafana to access Tracee Logs

By default, Tracee is emitting events to stdout. Users can then configure logging solutions to collect, store, and manage Tracee logs. 

This tutorial will showcase how to install and configure Promtail, Loki, Grafana and Prometheus to then access Tracee logs from the cluster in Grafana.

If you prefer the video tutorial, check out the tutorial below on the Aqua Open Source YouTube channel:

 Grafana Loki to access Tracee logs 
  [](https://youtu.be/mMC9-yzbgpE?si=6C0emOEJJ5K4ACqB)


## Prerequisites

Please make sure to have the following tools installed in your CLI:

* Kubectl installed and connected to a Kubernetes cluster (any cluster will work for this purpose)
* The [Helm CLI](https://helm.sh/docs/) installed

Additionally, you might have the following Obserability Stack already installed in your cluster, if not we will detailed how to set it up further below in this guide: 

* [Prometheus](https://prometheus.io/)
* [Loki and Promtail](https://grafana.com/oss/loki/)
* [Grafana](https://grafana.com/oss/)

Alternatively, this tutorial showcases after the Tracee Installation section how to get an observability stack running with the above tools.

## Installing the Tracee Helm Chart and accessing logs

Right now, we cannot access any logs from our cluster since we do not have any application that actively produces logs.
Thus, we will install Tracee inside our cluster through the Tracee Helm Chart.

Add the Tracee Helm Chart:

```console
helm repo add aqua https://aquasecurity.github.io/helm-charts/
```

Update the repository list on Helm:

```console
helm repo update
```

Install the Tracee Helm Chart inside your Kubernetes cluster:

```console
helm install tracee aqua/tracee \
        --namespace tracee-system --create-namespace \
        --set hostPID=true
```

Now, ensure that Tracee is running inside the `tracee-system` namespace:

```console
kubectl get all -n tracee-system
```

Similar to Promtail, also for Tracee one pod should run on each node of the Kubernetes cluster.

### Accessing Tracee Logs

Generally, it is possible to access logs from the Tracee pods directly through kubectl:

```console
kubectl logs -f daemonset/tracee -n tracee-system
```

Next, open the Grafana Dashboard, on the left, go to "Explore". There, you should be able to select Loki as a Datasource.

Now, you can write log queries in LogQL to access the logs that are stored in the Tracee pods:



## Installation of Observability Tools

We need to install an observability stack to access the logs of the pods inside our cluster. This will consist of:

- Grafana (for Dashboards and querying logs)
- Promtail for collecting logs from the pods on each node
- Loki, which is feeding the logs into Grafana

And since it is easier to install Grafana together with Prometheus, we are also going to install Prometheus.

If you are completely new to Loki, have a look at the following presentation:  [Learning the tricks of Grafana Loki for distributed logging at scale in a Kubernetes environment](https://youtu.be/jmtYUiBd_z0) 

### Grafana and Prometheus

First, we are going to install the kube-prometheus-stack chart with Prometheus and Grafana.

For this, we will need to specify some custom values that we will pass into the Helm Chart. 

Create a new file called `grafana-config.yaml` with the following content:

```yaml
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

grafana:
  sidecar:
    datasources:
      defaultDatasourceEnabled: true
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki-loki-distributed-query-frontend.monitoring:3100
```

Next, we can install the kube-prometheus-stack chart into our cluster with the following commands:

Create a namespace for all the monitoring tools

```console
kubectl create ns monitoring
```

Add the kube-prometheus-stack Helm Chart to your Helm repository list:

```console
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
```

Ensure you have the latest version of all your repositories:

```console
helm repo update
```

Install the kube-prometheus-stack Helm Chart:

```console
helm upgrade --install prom prometheus-community/kube-prometheus-stack -n monitoring --values grafana-config.yaml
```

Lastly, confirm that all the pods have been created properly by querying the namespace:

```console
kubectl get all -n monitoring
```

### Promtail and Loki

Next, we need to install Promtail and Loki inside the cluster to actually access logs.

For this, first add the Grafana Helm Chart repository to you repository list:

```console
helm repo add grafana https://grafana.github.io/helm-charts
```

Update your Helm repository list:

```console
helm repo update
```

Next, create a file with the Helm Chart configuration for Prometail in a promtail-config.yaml:

```yaml
config:
  serverPort: 8080
  clients:
    - url: http://loki-loki-distributed-gateway/loki/api/v1/push
```

Now we can install the Promtail Helm Chart inside our cluster:

```console
helm upgrade --install promtail grafana/promtail --values promtail-config.yaml -n monitoring
```

Make sure that Promtail is running the same number of pods as there are nodes on the cluster since Promtail has to run one pod per node:

```console
k get pods -n monitoring
```

For instance, if the cluster consists of three nodes, then there should be three Promtails pods inside of the monitoring namespace.

Now, we can install Loki. Loki's job is to collect the logs from Promtail and forward them to Grafana.

```console
helm upgrade --install loki grafana/loki-distributed -n monitoring
```

Loki will install a variety of components inside your cluster, so don't be alarmed by the number of pods it is starting, namely:

```text
loki-loki-distributed-distributor-5db986bb4f-x444n
loki-loki-distributed-query-frontend-bd6845f89-z6nx6
loki-loki-distributed-gateway-79d4d4ff5d-sxx58
loki-loki-distributed-querier-0
loki-loki-distributed-ingester-0    
```

At this point, the following pods should be running inside the Kubernetes cluster:

```text
NAME                                                     READY   STATUS    RESTARTS      AGE
prom-prometheus-node-exporter-l4cm4                      1/1     Running   0             22m
prom-kube-prometheus-stack-operator-84cf966ff5-96xdp     1/1     Running   0             22m
prom-kube-state-metrics-dc769cd87-fmrsk                  1/1     Running   0             22m
prom-grafana-6fdb45b4d5-2zxw7                            3/3     Running   0             22m
alertmanager-prom-kube-prometheus-stack-alertmanager-0   2/2     Running   1 (22m ago)   22m
prometheus-prom-kube-prometheus-stack-prometheus-0       2/2     Running   0             22m
promtail-sxfl5                                           1/1     Running   0             4m7s
loki-loki-distributed-gateway-79d4d4ff5d-sxx58           1/1     Running   0             72s
loki-loki-distributed-distributor-5db986bb4f-x444n       1/1     Running   0             72s
loki-loki-distributed-query-frontend-bd6845f89-z6nx6     1/1     Running   0             72s
loki-loki-distributed-querier-0                          1/1     Running   0             72s
loki-loki-distributed-ingester-0                         1/1     Running   0             72s
```

Since everything is running properly, we need to ensure that we can access Loki as a data source inside of Grafana.

For this, port-forward to Grafana:

```console
kubectl port-forward service/prom-grafana -n monitoring 3000:80
```

and open the Grafana UI on localhost:3000.

Here, you will need the username and the password:
username: admin
password: prom-operator

The password name is dependent on how you called the Helm Chart installation of the kube-prometheus-stack chart e.g. in our case, it was "prom".

Now navigate on Grafana to: Explore 
Here select Loki as a data source.

---

### Tutorials Tracee Vagrant (`docs/tutorials/tracee-vagrant.md`)

# Using the Tracee container image on MacOS with Parallels and Vagrant

There are a few more steps involved in running Tracee through a container image on arm64 (M1).

Prerequisites:

* [Vagrant CLI](https://developer.hashicorp.com/vagrant/downloads) installed
* [Parallels Pro](https://www.parallels.com/uk/products/desktop/pro/) installed

First, clone the Tracee Git repository and move into the root directory:

```console
git clone git@github.com:aquasecurity/tracee.git

cd tracee
```

Next, use Vagrant to start a Parallels VM:

```console
vagrant up
```

This will use the [Vagrantfile](https://github.com/aquasecurity/tracee/blob/main/Vagrantfile) in the root of the Tracee directory.

Lastly, ssh into the created VM:

```console
vagrant ssh
```

Now, it is possible to run the Tracee Container image:

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest
```

To learn how to install Tracee in a production environment, [check out the Kubernetes guide](../contributing/kubernetes.md).

---

### Tutorials Verify Tracee Signature (`docs/tutorials/verify-tracee-signature.md`)

# Cosign: verify tracee signature


## Prerequisites

Before you begin, ensure that you have the following installed:

- [cosign](https://docs.sigstore.dev/cosign/installation/)

## Verify tracee signature

Tracee images are signed with cosign keyless. To verify the signature we can run the command:

```console
cosign verify aquasec/tracee:tag-name  --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-identity-regexp https://github.com/aquasecurity/tracee | jq
```

Note that all of the tag-names can be found on the [Tracee Docker Hub Registry](https://hub.docker.com/r/aquasec/tracee/tags).

The output should look similar to the following:

---

### Contributing Building Building (`docs/contributing/building/building.md`)

# Building from the source

!!! Note
    Consider also visiting the following instructions:  
    1. docker container as [building environment](./environment.md)  
    2. building tracee [container images](./containers.md)
    3. using [development images](./containers.md#development-images) for testing latest changes  

1. Supported **Architectures**

    1. x86_64 (amd64)
    1. aarch64 (arm64)

2. Building **dependencies**

    1. `clang` && `llvm` (19)
    2. `golang` (1.26.5 toolchain)
    3. `libelf` and `libelf-dev`
       (or elfutils-libelf and elfutils-libelf-devel)
    4. `zlib1g` and `zlib1g-dev`
       (or zlib and zlib-devel)
    5. `libzstd-dev` for static build (libelf linkage)
    6. `clang-format-19` (specific version) for `fix-fmt`

    > You might take a look at the following files to understand how to have a
    > building environment:
    >
    > 1. [.github/actions/build-dependencies/action.yaml](https://github.com/aquasecurity/tracee/blob/main/.github/actions/build-dependencies/action.yaml)
    > 1. [builder/Dockerfile.ubuntu-tracee-make](https://github.com/aquasecurity/tracee/blob/main/builder/Dockerfile.ubuntu-tracee-make)
    > 1. [builder/Dockerfile.alpine-tracee-make](https://github.com/aquasecurity/tracee/blob/main/builder/Dockerfile.alpine-tracee-make)
    >
    > Those are very good examples for you to replicate a working environment.

3. **Clone** [tracee repository](https://github.com/aquasecurity/tracee/)

    ```bash
    git clone [https://github.com/aquasecurity/tracee/](https://github.com/aquasecurity/tracee/)
    ```

4. All makefiles have a **help** target to give you needed instructions

    ```bash
    make help
    ```

    ```
/* Detailed source-code truncated for AI context efficiency. */
```

5. Build **all** targets at once

    ```bash
    make all
    ```

6. Build a **static binary** by setting `STATIC=1`

    ```bash
    STATIC=1 make all
    ```

7. Build a **static binary** with [BTFHUB Support](https://github.com/aquasecurity/btfhub)

    ```bash
    BTFHUB=1 STATIC=1 make all
    ```

    !!! Note
        BTFHUB support will embed several very small files (BTF files) into your
        final binary. Those files will allow **tracee** binary to be executed
        in kernels that doesn't have embedded BTF information (the ones described
        at the BTFHUB repository)

    !!! Attention
        compiling with STATIC=1 won't allow you to use golang based
        signatures as plugins, only as built-ins:
        >```text
        >2021/12/13 13:27:21 error opening plugin /tracee/dist/signatures/builtin.so:
        >plugin.Open("/tracee/dist/signatures/builtin.so"): Dynamic loading not supported
        >```

8. Build a **debuggable binary** with DWARF debug symbols by setting `DEBUG=1`

    ```bash
    DEBUG=1 make
    ```

9. Build enabling BPF metrics by setting `METRICS=1`

    BPF metrics are only available if the BPF object is built with `METRICS` debug flag defined.

    ```bash
    METRICS=1 make
    ```

## Development Images

See the development images, usage examples, and stability guidance in the container images guide: [Development Images](./containers.md#development-images).

---

### Contributing Building Containers (`docs/contributing/building/containers.md`)

# Creating Tracee Container Images

> These instructions are meant to describe how to build the official tracee
> container image, instead of just downloading it from the
> [Docker Hub](https://hub.docker.com/r/aquasec/tracee).
>
> If you would like to have a local building and execution environment,
> [read this](./environment.md) instead.

## Using Tracee Container Image from Docker Hub

Before moving on to how to build Tracee container, it is important to know the
published container images and their tag meanings. Here is the current list of
docker container images being published during a release (or a snapshot
release):

1. **SNAPSHOT (development) container images:**

     Daily development images built from the latest `main` branch. Tags:

     - **aquasec/tracee:dev** (multi-arch)
     - **aquasec/tracee:x86_64-dev** (amd64 only)
     - **aquasec/tracee:aarch64-dev** (arm64 only)
     - **aquasec/tracee:dev-YYYYMMDD-HHMMSSUTC** (timestamped)

2. **RELEASE (official versions) container images:**

     Preferable alias for latest released image:

     - **aquasec/tracee:latest**

     And the container images for each released version of Tracee:

     - **aquasec/tracee:VERSION**

## Generating Tracee Container Images

1. **tracee:latest**

    Contains an executable binary with an embedded and CO-RE enabled eBPF object
    that makes it portable against multiple Linux and kernel versions.

    ```console
    make -f builder/Makefile.tracee-container build-tracee
    ```

    !!! Note
        `BTFHUB=1` adds support to some [older kernels](https://github.com/aquasecurity/btfhub/blob/main/docs/supported-distros.md).

        ```console
        BTFHUB=1 make -f builder/Makefile.tracee-container build-tracee
        ```

## Running Generated Tracee Container Image

Tracee container is supposed to be executed through docker cmdline directly,
from the official built images. Nevertheless, during the image building process,
it may be useful to execute the recently generated container image with correct
arguments, mostly to see if the image is working.

User may execute built containers through `Makefile.tracee-container` file with
the "run" targets:

1. To run recently generated **tracee:latest** container:

    ```console
    make -f builder/Makefile.tracee-container run-tracee
    ```

    !!! note
        Tracee arguments are passed through the `ARG` variable:
        ```console
        make -f builder/Makefile.tracee-container run-tracee ARG="--help"
        ```

## Development Images

For contributors who want to test the latest changes without building from source, use the daily development images from Docker Hub.

### Available Images

```bash
# Multi-architecture (recommended)
docker pull aquasec/tracee:dev

# Architecture-specific
docker pull aquasec/tracee:x86_64-dev
docker pull aquasec/tracee:aarch64-dev

# Timestamped build
docker pull aquasec/tracee:dev-20240115-050123UTC
```

### Image Details

- Build Schedule: Daily at 05:00 UTC
- Source: Latest `main` branch
- Security: Scanned for critical vulnerabilities before publishing
- Architectures: x86_64 and ARM64

### Usage Examples

```bash
# Quick test with development image
docker run --rm -it --pid=host --privileged \
  aquasec/tracee:dev --events syscalls --output format:table

# Use in development workflows
docker run --rm --pid=host --privileged \
  -v $(pwd)/policy.yaml:/policy.yaml \
  aquasec/tracee:dev --policy /policy.yaml
```

### Guidance

Use development images for testing unreleased features and validating fixes. Do not use them in production environments.

To inspect build metadata:

```bash
docker inspect aquasec/tracee:dev | jq '.[0].Config.Labels'
docker run --rm aquasec/tracee:dev --version
```

For stable release images, see the [Installation Guide](../../docs/install/index.md).

---

### Contributing Building Environment (`docs/contributing/building/environment.md`)

# Creating a local building environment

> These instructions are meant to describe how to create a local building and
> execution environment. If you would like to build tracee container(s)
> image(s), [read this](./containers.md) instead.

!!! Note
    A building environment will let you build and execute tracee inside a docker
    container, containing all needed tools to build and execute it. If you're
    using an OSX environment, for example, you can install gmake (`brew install
    gmake`) and configure such environment by using Docker.

!!! Attention
    If you want to build tracee on your local machine
    [read this](./building.md).

## Quick steps

1. Build tracee environment:

    ```bash
    make -f builder/Makefile.tracee-make alpine-prepare
    make -f builder/Makefile.tracee-make alpine-shell
    ```

2. Build and execute tracee:

    ```bash
    make clean
    make tracee
    sudo ./dist/tracee \
        --enrichment decoded-data \
        --scope comm=bash \
        --scope follow
    ```

Now, in your host's shell, execute a command. You will see all events
(except scheduler ones) being printed, in "table format", to stdout.

## How to build and use the environment

In order to have a controlled building environment for tracee, tracee provides
a `Makefile.tracee-make` file that allows you to create and use a docker container environment to build & test **tracee**.

There are Two different environments that are maintained for building tracee:

* **Alpine**
* **Ubuntu**

The reason for that is that `Alpine Linux` is based in the [musl](https://en.wikipedia.org/wiki/Musl) C standard library,
while the `Ubuntu Linux` uses [glibc](https://en.wikipedia.org/wiki/Glibc).

By supporting both building environments we can always be sure that the project builds (and executes) correctly in both environments.

!!! Attention
    Locally created containers, called `alpine-tracee-make` or
    `ubuntu-tracee-make`, share the host source code directory. This means
    that, if you build tracee binary using `alpine` distribution, the binary
    might not be compatible to the Linux distribution from your host OS.

### Creating a builder environment

* To create an **alpine-tracee-make** container:

    ```bash
    make -f builder/Makefile.tracee-make alpine-prepare
    ```

* To create an **ubuntu-tracee-make** container:

    ```bash
    make -f builder/Makefile.tracee-make ubuntu-prepare
    ```

### Executing a builder environment

* To execute an **alpine-tracee-make** shell:

    ```bash
    make -f builder/Makefile.tracee-make alpine-shell
    ```

* To execute an **ubuntu-tracee-make** shell:

    ```bash
    make -f builder/Makefile.tracee-make ubuntu-shell
    ```

### Using build environment as a **make** replacement

Instead of executing a builder shell, you may use `alpine-make`, or
`ubuntu-make`, as a replacement for the `make` command:

1. Create builder environment as described:
    [Creating a builder environment](#creating-a-builder-environment)
2. Compile tracee using `ubuntu-make`

    * Build tracee binary:

        ```bash
        make -f builder/Makefile.tracee-make ubuntu-make ARG="tracee"
        ```

    * Show available `ubuntu-make` commands:

        ```bash
        make -f builder/Makefile.tracee-make ubuntu-make ARG="help"
        ```

    * Remove tracee binary

        ```bash
        make -f builder/Makefile.tracee-make ubuntu-make ARG="clean"
        ```

    * Build binaries for all

        ```bash
        make -f builder/Makefile.tracee-make ubuntu-make ARG="all"
        ```

3. Run tracee binary

    ```bash
    sudo ./dist/tracee
    ```

> **Note**: the generated binary must be compatible to your host (depending on glibc version).

If you don't want to depend on the host's library versions, or you are using the `alpine-make` container as a replacement for `make`, then it's necessary to set the `STATIC` variable to `1` so you can run compiled binaries on your host machine:

1. Compile tracee

    ```bash
    make -f builder/Makefile.tracee-make alpine-prepare
    STATIC=1 make -f builder/Makefile.tracee-make alpine-make ARG="all"
    ```

2. Verify the executable is static

    * Note: ldd prints the shared libraries required by an executable file

    ```bash
    ldd dist/tracee
    ```

    ```text
    not a dynamic executable
    ```

3. Execute the static binary from your host

    ```bash
    sudo ./dist/tracee
    ```

---

### Docs Detectors Api Reference (`docs/docs/detectors/api-reference.md`)

# Detector API Reference

Complete reference documentation for the Tracee Detector API. This guide documents all interfaces, structures, and features for writing custom detectors.

**New to detectors?** Start with the [Quick Start Guide](quickstart.md) first.

## Table of Contents

1. [Core Interfaces](#core-interfaces)
2. [DetectorDefinition](#detectordefinition)
3. [Detector Requirements](#detector-requirements)
4. [DetectorOutput](#detectoroutput)
5. [Auto-Population](#auto-population)
6. [Lifecycle Management](#lifecycle-management)
7. [Testing](#testing)
8. [Best Practices](#best-practices)

---

## Core Interfaces

### EventDetector Interface

All detectors must implement this interface:

{% raw %}
```go
type EventDetector interface {
    // GetDefinition returns the detector's metadata and requirements
    // Called once during registration - result is cached
    GetDefinition() DetectorDefinition

    // Init is called once at startup with shared resources
    // Use this to store logger, datastores, and initialize state
    Init(params DetectorParams) error

    // OnEvent processes each matching event
    // Return one or more DetectorOutput, nil for no detection, or error
    OnEvent(ctx context.Context, event *v1beta1.Event) ([]DetectorOutput, error)
}
```
{% endraw %}

### DetectorParams

Provided to `Init()` with shared resources:

{% raw %}
```go
type DetectorParams struct {
    Logger     Logger                 // Structured logger (zap-based)
    DataStores datastores.Registry    // Access to all datastores
    Config     DetectorConfig         // Detector-specific configuration
}
```
{% endraw %}

**Example**:
{% raw %}
```go
func (d *MyDetector) Init(params detection.DetectorParams) error {
    d.logger = params.Logger
    d.dataStores = params.DataStores

    // Check required datastore availability
    if !params.DataStores.IsAvailable("process") {
        return errors.New("process store required but unavailable")
    }

    d.logger.Debugw("Detector initialized", "id", d.GetDefinition().ID)
    return nil
}
```
{% endraw %}

---

## DetectorDefinition

### Complete Structure

{% raw %}
```go
type DetectorDefinition struct {
    // Unique identifier (required)
    ID string

    // Event and datastore requirements (required)
    Requirements DetectorRequirements

    // Output event specification (required)
    ProducedEvent v1beta1.EventDefinition

    // Default threat metadata (optional, recommended for threats)
    ThreatMetadata *v1beta1.Threat

    // Auto-population configuration (optional, recommended)
    AutoPopulate AutoPopulateFields
}
```
{% endraw %}

### Detector ID Conventions

Choose an appropriate ID format:

**Threat Detectors** (TRC-XXX):
{% raw %}
```go
ID: "TRC-001"  // Sensitive file access
ID: "TRC-014"  // Process injection detection
```
{% endraw %}

**Derived Events** (DRV-XXX):
{% raw %}
```go
ID: "DRV-001"  // Container lifecycle events
ID: "DRV-003"  // Hooked syscall detector
```
{% endraw %}

**Custom/Vendor** (VENDOR-XXX):
{% raw %}
```go
ID: "ACME-001"  // Custom company detection
```
{% endraw %}

### ProducedEvent: Event Definition

Defines the event your detector produces:

{% raw %}
```go
ProducedEvent: v1beta1.EventDefinition{
    Name:        "my_detection",           // Required: unique snake_case name
    Description: "Detailed description",    // Required: clear explanation
    Version:     &v1beta1.Version{         // Required: semantic version
        Major: 1,
        Minor: 0,
        Patch: 0,
    },
    Fields: []*v1beta1.EventField{         // Required: output fields schema
        {
            Name: "field_name",
            Type: "const char*",            // Type for display/docs
        },
    },
    Tags: []string{"security", "file"},    // Optional: categorization
}
```
{% endraw %}

**Field Types** (documentation only):

- `const char*` - String
- `int`, `u32`, `u64` - Integers
- `bool` - Boolean
- Custom types for complex data

**Example with all fields**:
{% raw %}
```go
ProducedEvent: v1beta1.EventDefinition{
    Name:        "suspicious_shell_execution",
    Description: "Detects suspicious shell command execution patterns",
    Version:     &v1beta1.Version{Major: 1, Minor: 2, Patch: 0},
    Fields: []*v1beta1.EventField{
        {Name: "command", Type: "const char*"},
        {Name: "shell_type", Type: "const char*"},
        {Name: "risk_score", Type: "int"},
        {Name: "indicators", Type: "const char*[]"},
    },
    Tags: []string{"execution", "defense-evasion"},
}
```
{% endraw %}

### ThreatMetadata: Threat Template

Default threat information, copied to outputs when `AutoPopulate.Threat=true`:

{% raw %}
```go
ThreatMetadata: &v1beta1.Threat{
    Name:        "Short threat name",
    Description: "Detailed threat description",
    Severity:    v1beta1.Severity_MEDIUM,
    Signature: &v1beta1.ThreatSignature{
        ID:   "MITRE T1055",
        Name: "Process Injection",
    },
    Mitre: &v1beta1.MitreAttack{
        Tactic:     []string{"Defense Evasion", "Privilege Escalation"},
        Technique:  []string{"T1055"},
        SubTechnique: []string{"T1055.001"},
    },
}
```
{% endraw %}

**Severity Levels**:

- `Severity_INFO` - Informational events
- `Severity_LOW` - Low-risk threats
- `Severity_MEDIUM` - Moderate threats (default for most detections)
- `Severity_HIGH` - Serious threats requiring immediate attention
- `Severity_CRITICAL` - Critical threats (data exfiltration, rootkits)

**Override per detection**:
{% raw %}
```go
return []detection.DetectorOutput{{
    Data: data,
    Threat: &v1beta1.Threat{
        Name:     "Critical System Compromise",
        Severity: v1beta1.Severity_CRITICAL,  // Override default MEDIUM
    },
}}, nil
```
{% endraw %}

---

## Detector Requirements

### DetectorRequirements Structure

Declare what your detector needs:

{% raw %}
```go
type DetectorRequirements struct {
    // Events lists the events this detector needs to receive
    Events []EventRequirement

    // DataStores lists required datastores with their dependency types
    DataStores []DataStoreRequirement

    // Enrichments lists required event enrichment options
    Enrichments []EnrichmentRequirement

    // Architectures lists supported CPU architectures (empty = all)
    Architectures []string

    // MinTraceeVersion specifies minimum Tracee version (optional, inclusive)
    MinTraceeVersion *v1beta1.Version

    // MaxTraceeVersion specifies maximum Tracee version (optional, exclusive)
    MaxTraceeVersion *v1beta1.Version
}
```
{% endraw %}

### EventRequirement Structure

Declare which events your detector needs:

{% raw %}
```go
type EventRequirement struct {
    Name             string            // Event name (required)
    DataFilters      []string          // Filter by event data (optional)
    ScopeFilters     []string          // Filter by origin (optional)
    MinVersion       *v1beta1.Version  // Minimum event version (optional)
    MaxVersion       *v1beta1.Version  // Maximum event version (optional, exclusive)
    Dependency       DependencyType    // Required vs optional (optional, default Required)
}
```
{% endraw %}

### Basic Requirements

{% raw %}
```go
Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
        {Name: "security_inode_unlink"},
    },
}
```
{% endraw %}

### DataFilters: Filtering Event Data

Engine-level filtering - only matching events reach your detector:

{% raw %}
```go
{
    Name: "security_file_open",
    DataFilters: []string{
        "pathname=/etc/shadow",          // Exact match
        "pathname=/etc/sudoers",          // OR condition
    },
}
```
{% endraw %}

**Filter operators**:
{% raw %}
```go
"field=value"       // Exact match
"field=/path/*"     // Glob pattern (*, ?)
"field!=value"      // Not equal
"field>100"         // Greater than (numeric)
"field<100"         // Less than (numeric)
"field>=100"        // Greater or equal
"field<=100"        // Less or equal
```
{% endraw %}

**Multiple filters** (AND logic):
{% raw %}
```go
DataFilters: []string{
    "pathname=/tmp/*",       // Must match /tmp/*
    "flags>0",               // AND flags > 0
}
```
{% endraw %}

**Common patterns**:
{% raw %}
```go
// Sensitive paths
DataFilters: []string{
    "pathname=/etc/shadow",
    "pathname=/etc/passwd",
    "pathname=/root/.ssh/*",
}

// Specific flags
DataFilters: []string{
    "flags=0x80000",  // O_CLOEXEC
}

// Suspicious syscalls
DataFilters: []string{
    "syscall=ptrace",
    "syscall=process_vm_writev",
}
```
{% endraw %}

### ScopeFilters: Filtering by Origin

Filter by where events originated:

{% raw %}
```go
{
    Name: "security_file_open",
    ScopeFilters: []string{
        "container=started",  // Only containers
    },
}
```
{% endraw %}

**Scope options**:
{% raw %}
```go
"container=started"      // Only from containers
"not-container"          // Only from host (not containers)
"pid=1000"              // Specific PID
"pid!=1"                // Exclude init process
```
{% endraw %}

### Event Version Constraints

Require minimum event version:

{% raw %}
```go
{
    Name: "security_bprm_check",
    MinVersion: &v1beta1.Version{
        Major: 2,
        Minor: 1,
    },
}
```
{% endraw %}

### Required vs Optional Events

{% raw %}
```go
type DependencyType int

const (
    DependencyRequired DependencyType = iota  // Detector fails if event unavailable
    DependencyOptional                         // Detector works without event
)
```
{% endraw %}

**Example**:
{% raw %}
```go
Events: []detection.EventRequirement{
    {
        Name:       "security_file_open",
        Dependency: detection.DependencyRequired,  // Must have
    },
    {
        Name:       "hooked_syscalls",
        Dependency: detection.DependencyOptional,  // Nice to have
    },
}
```
{% endraw %}

### DataStore Requirements

Declare datastore dependencies:

{% raw %}
```go
type DataStoreRequirement struct {
    Name       string          // Datastore name
    Dependency DependencyType  // Required vs optional
}
```
{% endraw %}

**Available datastores**:

- `process` - Process tree and ancestry
- `container` - Container metadata
- `dns` - DNS cache
- `system` - System information
- `syscall` - Syscall mappings
- `symbol` - Kernel symbols

**Example**:
{% raw %}
```go
Requirements: detection.DetectorRequirements{
    Events: []detection.EventRequirement{
        {Name: "security_file_open"},
    },
    DataStores: []detection.DataStoreRequirement{
        {Name: detection.DataStoreProcess, Dependency: detection.DependencyRequired},
        {Name: detection.DataStoreContainer, Dependency: detection.DependencyOptional},
    },
}
```
{% endraw %}

### Enrichment Requirements

Request specific data enrichments on input events:

{% raw %}
```go
type EnrichmentRequirement struct {
    Name       string          // Enrichment option name
    Dependency DependencyType  // Required vs optional
    Config     string          // Enrichment-specific config (optional)
}
```
{% endraw %}

**Available enrichments**:

- `environment` - Execution environment variables
- `executable-hash` - Executable file hashes (Config: "inode", "dev-inode", "digest-inode")
- `container` - Container metadata fields in Event struct (Name, Image, Pod info)

**Example**:
{% raw %}
```go
DetectorDefinition{
    ID: "TRC-001",
    Requirements: detection.DetectorRequirements{
        Enrichments: []detection.EnrichmentRequirement{
            {
                Name:       detection.EnrichmentExecutableHash,
                Config:     detection.ExecutableHashConfigDigestInode,
                Dependency: detection.DependencyRequired,
            },
        },
    },
}
```
{% endraw %}

**Container enrichment example**:

{% raw %}
```go
DetectorDefinition{
    ID: "TRC-CONTAINER-001",
    Requirements: detection.DetectorRequirements{
        Events: []detection.EventRequirement{
            {Name: "security_file_open"},
        },
        Enrichments: []detection.EnrichmentRequirement{
            {
                Name:       detection.EnrichmentContainer,
                Dependency: detection.DependencyRequired,
            },
        },
    },
}
```
{% endraw %}

This detector requires container fields to be pre-populated in the Event struct. Without `--enrichment container`, registration will fail.

### Architecture Filtering

Restrict detector to specific CPU architectures (rarely needed):

{% raw %}
```go
DetectorDefinition{
    ID: "TRC-X86-001",
    Requirements: detection.DetectorRequirements{
        Architectures: []string{"amd64"},  // Only x86-64
        // ... rest of requirements
    },
}
```
{% endraw %}

**Supported architectures**:

- `amd64` (x86-64)
- `arm64` (AArch64)

**Default**: Empty slice (or omit field) means the detector supports all architectures.

**When to use**: Only when your detector uses architecture-specific logic or syscalls that don't exist on all platforms.

### Tracee Version Constraints

Require minimum/maximum Tracee version:

{% raw %}
```go
DetectorDefinition{
    ID: "TRC-NEW-001",
    Requirements: detection.DetectorRequirements{
        MinTraceeVersion: &v1beta1.Version{
            Major: 0,
            Minor: 20,
            Patch: 0,
        },
        // MaxTraceeVersion is optional (exclusive)
    },
}
```
{% endraw %}

**When to use**: Your detector relies on features introduced in specific Tracee versions.

### Complete Requirements Example

Putting it all together - a detector with all requirement types:

{% raw %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endraw %}

---

## DetectorOutput

### Structure

What your `OnEvent()` returns:

{% raw %}
```go
type DetectorOutput struct {
    // Event data fields (required) - corresponds to ProducedEvent.Fields
    Data []*v1beta1.EventValue

    // Override auto-population settings for this output (optional)
    // If nil, uses Definition.AutoPopulate
    AutoPopulate *AutoPopulateFields

    // Override threat metadata (optional) - overrides Definition.ThreatMetadata
    Threat *v1beta1.Threat

    // Override ancestry depth (optional) - overrides AutoPopulate.ProcessAncestry
    AncestryDepth *uint32
}
```
{% endraw %}

### Helper Functions

For cleaner, self-documenting code, use these helpers instead of constructing `DetectorOutput` directly:

{% raw %}
```go
// Simple detection - all info in triggering event, auto-populated by engine
return detection.Detected(), nil

// Detection with additional data
return detection.DetectedWithData([]*v1beta1.EventValue{
    v1beta1.NewStringValue("env_var", envValue),
}), nil

// Complex cases - use struct literal for multiple fields
return []detection.DetectorOutput{{
    Data:          data,
    Threat:        customThreat,
    AncestryDepth: &depth,
}}, nil
```
{% endraw %}

### Creating Event Data

Use helper functions to create type-safe event values:

{% raw %}
```go
// String values
v1beta1.NewStringValue("field_name", "value")

// Integer values
v1beta1.NewInt32Value("count", int32(42))
v1beta1.NewInt64Value("count", int64(42))
v1beta1.NewUInt32Value("flags", uint32(0x1234))
v1beta1.NewUInt64Value("flags", uint64(0x1234))

// Boolean values
v1beta1.NewBoolValue("is_suspicious", true)

// Byte array values
v1beta1.NewBytesValue("data", []byte{0x01, 0x02})

// Complex example with multiple data fields
return detection.DetectedWithData([]*v1beta1.EventValue{
    v1beta1.NewStringValue("command", "/bin/bash -c 'curl evil.com'"),
    v1beta1.NewStringValue("shell", "bash"),
    v1beta1.NewInt32Value("risk_score", int32(85)),
    v1beta1.NewBoolValue("known_ioc", true),
}), nil
```
{% endraw %}

### Multiple Outputs

Return multiple detections from a single input:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    var outputs []detection.DetectorOutput

    if condition1 {
        outputs = append(outputs, detection.DetectorOutput{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("type", "pattern_a"),
            },
        })
    }

    if condition2 {
        outputs = append(outputs, detection.DetectorOutput{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("type", "pattern_b"),
            },
            Threat: &v1beta1.Threat{
                Severity: v1beta1.Severity_HIGH,  // Different severity
            },
        })
    }

    return outputs, nil
}
```
{% endraw %}

### No Detection

Return `nil, nil` when no detection occurred:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    value, found := v1beta1.GetData[string](event, "field")
    if !found || !d.isSuspicious(value) {
        return nil, nil  // No detection
    }

    return []detection.DetectorOutput{{...}}, nil
}
```
{% endraw %}

---

## Auto-Population

Declaratively specify automatic field enrichment:

{% raw %}
```go
type AutoPopulateFields struct {
    Threat          bool  // Copy ThreatMetadata to output
    DetectedFrom    bool  // Link to triggering event
    ProcessAncestry bool  // Fetch 5 levels of process ancestry
}
```
{% endraw %}

### Threat: Automatic Threat Metadata

{% raw %}
```go
AutoPopulate: detection.AutoPopulateFields{
    Threat: true,
}
```
{% endraw %}

**Behavior**:

- If `output.Threat` is nil, engine copies `Definition.ThreatMetadata`
- If `output.Threat` is set, that takes precedence
- `Definition.ThreatMetadata` acts as a template/default

**Example**:
{% raw %}
```go
// In definition
ThreatMetadata: &v1beta1.Threat{
    Name:     "Suspicious File Access",
    Severity: v1beta1.Severity_MEDIUM,
}

// In OnEvent - uses default
return []detection.DetectorOutput{{
    Data: data,
    // Threat: nil - engine copies ThreatMetadata above
}}, nil

// In OnEvent - overrides default
return []detection.DetectorOutput{{
    Data: data,
    Threat: &v1beta1.Threat{
        Name:     "Critical System File Access",
        Severity: v1beta1.Severity_CRITICAL,  // Higher severity
    },
}}, nil
```
{% endraw %}

### DetectedFrom: Provenance Tracking

{% raw %}
```go
AutoPopulate: detection.AutoPopulateFields{
    DetectedFrom: true,
}
```
{% endraw %}

**Behavior**: Engine sets `output.DetectedFrom` with:

- `event_id` - Triggering event's ID
- `event_name` - Triggering event's name
- `stack_addresses` - Stack trace (if available)

**Output structure**:
```json
{
  "detected_from": {
    "event_id": 257,
    "event_name": "security_file_open",
    "stack_addresses": [...]
  }
}
```

**Use cases**:
- Debugging: Understand why detector fired
- Forensics: Reconstruct detection chains
- Correlation: Link detections to root causes

#### Detection Chain Preservation

When detectors form chains (detector consuming another detector's output), the engine automatically preserves the complete provenance chain via `DetectedFrom.parent`:

**Example 3-level chain:**

```json
{
  "name": "critical_threat_in_production",
  "detected_from": {
    "id": "7001",
    "name": "suspicious_container_exec",
    "data": [{"name": "binary", "value": "nc"}],
    "parent": {
      "id": "7000",
      "name": "suspicious_binary_exec",
      "data": [{"name": "pathname", "value": "/usr/bin/nc"}],
      "parent": {
        "id": "700",
        "name": "sched_process_exec",
        "data": [{"name": "pathname", "value": "/usr/bin/nc"}]
      }
    }
  }
}
```

**Traversal helpers:**

{% raw %}
```go
// Get full chain (immediate parent to root)
chain := v1beta1.GetDetectionChain(event)

// Get original triggering event
root := v1beta1.GetRootDetection(event)

// Get chain depth
depth := v1beta1.GetChainDepth(event)  // 3 for example above
```
{% endraw %}

This chaining is automatic - detectors don't need to do anything special.

### ProcessAncestry: Automatic Ancestry Enrichment

{% raw %}
```go
AutoPopulate: detection.AutoPopulateFields{
    ProcessAncestry: true,  // Default: 5 levels
}
```
{% endraw %}

**Requirements**:

- Tracee must run with `--stores process`
- Process must be in the process tree
- Default depth: 5 levels (parent → grandparent → great-grandparent → ...)

**Ancestry structure**:
```json
{
  "workload": {
    "process": {
      "entity_id": 12345,
      "pid": 67890,
      "executable": {"path": "/bin/cat"},
      "ancestors": [
        {"entity_id": 12344, "pid": 67889, "executable": {"path": "/bin/bash"}},
        {"entity_id": 12343, "pid": 67888, "thread": {"name": "sshd"}},
        {"entity_id": 1, "pid": 1, "thread": {"name": "systemd"}}
      ]
    }
  }
}
```

**Per-detection ancestry depth control**:
{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    severity := d.analyzeThreat(event)

    if severity == "critical" {
        // Deep ancestry for forensics
        depth := uint32(10)
        return []detection.DetectorOutput{{
            Data:          data,
            AncestryDepth: &depth,
        }}, nil
    }

    if severity == "low" {
        // Disable ancestry for performance
        depth := uint32(0)
        return []detection.DetectorOutput{{
            Data:          data,
            AncestryDepth: &depth,
        }}, nil
    }

    // Use default (ProcessAncestry=true → 5 levels)
    return []detection.DetectorOutput{{
        Data: data,
        // AncestryDepth: nil
    }}, nil
}
```
{% endraw %}

### Complete Auto-Population Example

{% raw %}
```go
DetectorDefinition{
    ID: "TRC-001",

    ThreatMetadata: &v1beta1.Threat{
        Name:     "Sensitive File Access",
        Severity: v1beta1.Severity_MEDIUM,
    },

    AutoPopulate: detection.AutoPopulateFields{
        Threat:          true,  // Copy ThreatMetadata
        DetectedFrom:    true,  // Add provenance
        ProcessAncestry: true,  // Add 5-level ancestry
    },
}

// OnEvent just returns data - engine enriches everything else
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    return []detection.DetectorOutput{{
        Data: []*v1beta1.EventValue{
            v1beta1.NewStringValue("file", "/etc/shadow"),
        },
        // Engine automatically adds:
        // - Threat (from ThreatMetadata)
        // - DetectedFrom (provenance)
        // - Workload.Process.Ancestors (5 levels)
    }}, nil
}
```
{% endraw %}

### Overriding Auto-Population Per Output

You can override auto-population settings for specific outputs using the `AutoPopulate` field in `DetectorOutput`:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    severity := d.analyzeThreat(event)

    if severity == "critical" {
        // Enable all enrichment for critical threats
        return []detection.DetectorOutput{{
            Data: data,
            AutoPopulate: &detection.AutoPopulateFields{
                Threat:          true,
                DetectedFrom:    true,
                ProcessAncestry: true,
            },
        }}, nil
    }

    if severity == "low" {
        // Disable enrichment for low-severity detections (performance)
        return []detection.DetectorOutput{{
            Data: data,
            AutoPopulate: &detection.AutoPopulateFields{
                Threat:          true,
                DetectedFrom:    false,  // Skip provenance
                ProcessAncestry: false,  // Skip ancestry
            },
        }}, nil
    }

    // Use default from Definition.AutoPopulate
    return []detection.DetectorOutput{{
        Data: data,
        // AutoPopulate: nil - uses Definition.AutoPopulate
    }}, nil
}
```
{% endraw %}

**Use cases for per-output overrides**:

- Vary enrichment based on threat severity
- Skip expensive enrichment for low-confidence detections
- Add extra context for high-priority alerts

---

## Lifecycle Management

### Init() Best Practices

Called once at startup - use for initialization:

{% raw %}
```go
func (d *MyDetector) Init(params detection.DetectorParams) error {
    // 1. Store logger (always do this)
    d.logger = params.Logger

    // 2. Store datastores if needed
    d.dataStores = params.DataStores

    // 3. Validate required datastores
    if !params.DataStores.IsAvailable("process") {
        return errors.New("process store required but unavailable")
    }

    // 4. Initialize detector state
    d.cache = make(map[string]int)

    // 5. Log initialization
    d.logger.Debugw("Detector initialized",
        "id", d.GetDefinition().ID,
        "version", d.GetDefinition().ProducedEvent.Version,
    )

    return nil
}
```
{% endraw %}

**Don't do in Init()**:

- Heavy computation (blocks startup)
- Network requests
- File I/O (except small config files)

### OnEvent() Guidelines

Called for every matching event - must be fast:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // 1. Respect context cancellation
    select {
    case <-ctx.Done():
        return nil, ctx.Err()
    default:
    }

    // 2. Extract required data early
    value, found := v1beta1.GetData[string](event, "field")
    if !found {
        return nil, nil  // Skip silently
    }

    // 3. Fast-path rejection
    if !d.couldBeSuspicious(value) {
        return nil, nil
    }

    // 4. Detailed analysis
    if d.isSuspicious(value) {
        return []detection.DetectorOutput{{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("field", value),
            },
        }}, nil
    }

    return nil, nil
}
```
{% endraw %}

**Performance tips**:

- Check cheapest conditions first
- Use datastores judiciously (they're fast but not free)
- Avoid allocations in hot path
- Consider caching expensive computations

### Error Handling Guidelines

**Transient errors** - Return error, engine may retry:
{% raw %}
```go
proc, err := d.dataStores.Processes().GetProcess(entityId)
if err != nil && !errors.Is(err, datastores.ErrNotFound) {
    return nil, fmt.Errorf("failed to get process: %w", err)
}
```
{% endraw %}

**Expected conditions** - Return nil detection:
{% raw %}
```go
// Data not found - this is normal
if errors.Is(err, datastores.ErrNotFound) {
    return nil, nil
}

// Optional field missing - expected
value, found := v1beta1.GetData[string](event, "optional_field")
if !found {
    return nil, nil
}
```
{% endraw %}

**Critical errors** - Log and return error:
{% raw %}
```go
if err := d.criticalOperation(); err != nil {
    d.logger.Errorw("Critical operation failed",
        "error", err,
        "event", event.Name,
    )
    return nil, fmt.Errorf("critical failure: %w", err)
}
```
{% endraw %}

**Error wrapping**:
{% raw %}
```go
// Good - provides context
return nil, fmt.Errorf("failed to analyze process %d: %w", pid, err)

// Bad - loses context
return nil, err
```
{% endraw %}

---

## Testing

### Unit Testing Patterns

Test detectors with mock events:

{% raw %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endraw %}

### Mock Logger and DataStores

Create mock implementations for testing:

{% raw %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endraw %}

### Test Helpers

Use built-in type-safe helpers for data extraction:

{% raw %}
```go
// Type-safe data extraction with assertion
pathname, found := v1beta1.GetData[string](event, "pathname")
require.True(t, found, "pathname field should exist")
assert.Equal(t, "/etc/shadow", pathname)
```
{% endraw %}

---

## Best Practices

### Detector Design Principles

**1. Single Responsibility**
Each detector should focus on one threat pattern or derived event type.

**Good**:
{% raw %}
```go
// Focused on one threat
type ContainerEscapeDetector struct {}

// Focused on one derived event type
type ContainerLifecycleDetector struct {}
```
{% endraw %}

**Bad**:
{% raw %}
```go
// Too broad
type AllContainerThreatsDetector struct {}  // Don't do this!
```
{% endraw %}

**2. Fail Fast**
Reject events as early as possible:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // 1. Check required fields first
    pathname, found := v1beta1.GetData[string](event, "pathname")
    if !found {
        return nil, nil
    }

    // 2. Quick pattern checks
    if !strings.HasPrefix(pathname, "/etc/") {
        return nil, nil
    }

    // 3. Expensive checks last
    if d.isKnownMaliciousPattern(pathname) {
        // Create detection
    }

    return nil, nil
}
```
{% endraw %}

**3. Leverage Engine Filtering**
Use `DataFilters` instead of logic in `OnEvent()`:

**Good**:
{% raw %}
```go
DataFilters: []string{
    "pathname=/etc/*",
    "flags>0",
}

func (d *MyDetector) OnEvent(...) {
    // Events already filtered - just detection logic
}
```
{% endraw %}

**Bad**:
{% raw %}
```go
func (d *MyDetector) OnEvent(...) {
    pathname, _ := v1beta1.GetData[string](event, "pathname")
    if !strings.HasPrefix(pathname, "/etc/") {  // Should be in DataFilters!
        return nil, nil
    }
}
```
{% endraw %}

**4. Use Auto-Population**
Let the engine handle enrichment:

**Good**:
{% raw %}
```go
AutoPopulate: detection.AutoPopulateFields{
    Threat:          true,
    DetectedFrom:    true,
    ProcessAncestry: true,
}

func (d *MyDetector) OnEvent(...) {
    return []detection.DetectorOutput{{
        Data: simpleData,  // Just the data fields
    }}, nil
}
```
{% endraw %}

**Bad**:
{% raw %}
```go
func (d *MyDetector) OnEvent(...) {
    // Don't manually query ancestry if you can auto-populate!
    ancestry, _ := d.dataStores.Processes().GetAncestry(entityId, 5)
    // Manual enrichment is error-prone and slower
}
```
{% endraw %}

**5. Graceful Degradation**
Handle missing datastores gracefully:

{% raw %}
```go
// In Init - validate required stores
if !params.DataStores.IsAvailable("process") {
    return errors.New("process store required")
}

// In OnEvent - handle optional stores
if params.DataStores.IsAvailable("container") {
    containerInfo, _ := d.dataStores.Containers().GetContainer(id)
    // Use if available
}
```
{% endraw %}

### Performance Considerations

**1. Minimize DataStore Queries**
{% raw %}
```go
// Good - single query
proc, err := d.dataStores.Processes().GetProcess(entityId)

// Bad - unnecessary queries
proc, _ := d.dataStores.Processes().GetProcess(entityId)
ancestors, _ := d.dataStores.Processes().GetAncestry(entityId, 5)
// Could use AutoPopulate.ProcessAncestry instead!
```
{% endraw %}

**2. Cache Expensive Computations**
{% raw %}
```go
type MyDetector struct {
    mu    sync.RWMutex
    cache map[string]result
}

func (d *MyDetector) expensiveCheck(key string) result {
    // Check cache (read lock)
    d.mu.RLock()
    if cached, ok := d.cache[key]; ok {
        d.mu.RUnlock()
        return cached
    }
    d.mu.RUnlock()

    // Compute (write lock)
    d.mu.Lock()
    defer d.mu.Unlock()

    // Double-check after acquiring write lock
    if cached, ok := d.cache[key]; ok {
        return cached
    }

    result := d.compute(key)
    d.cache[key] = result
    return result
}
```
{% endraw %}

**3. Batch Operations**
If checking multiple conditions, batch them:

{% raw %}
```go
// Good - batch check
if d.isSuspiciousPath(pathname) &&
   d.isSuspiciousFlags(flags) &&
   d.isSuspiciousProcess(proc) {
    // All checks passed
}

// Bad - create intermediary slices
var checks []bool
checks = append(checks, d.isSuspiciousPath(pathname))
checks = append(checks, d.isSuspiciousFlags(flags))
// Unnecessary allocations
```
{% endraw %}

**4. Use Appropriate Data Structures**
{% raw %}
```go
// Good - fast lookup
type MyDetector struct {
    suspiciousPaths map[string]bool  // O(1) lookup
}

// Bad - linear search
type MyDetector struct {
    suspiciousPaths []string  // O(n) lookup
}
```
{% endraw %}

### Code Organization

**File structure**:
```
detectors/
├── my_detector.go           # Detector implementation
├── my_detector_test.go      # Unit tests
└── my_detector_patterns.go  # Optional: complex pattern logic
```

**Detector file template**:
{% raw %}
```go
package detectors

// 1. Imports
import (
    "context"
    "github.com/aquasecurity/tracee/api/v1beta1"
    "github.com/aquasecurity/tracee/api/v1beta1/detection"
)

// 2. Auto-registration
func init() {
    register(&MyDetector{})
}

// 3. Detector struct
type MyDetector struct {
    logger     detection.Logger
    dataStores datastores.Registry
    // ... other state
}

// 4. GetDefinition
func (d *MyDetector) GetDefinition() detection.DetectorDefinition {
    // ...
}

// 5. Init
func (d *MyDetector) Init(params detection.DetectorParams) error {
    // ...
}

// 6. OnEvent
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // ...
}

// 7. Private helper methods
func (d *MyDetector) helperMethod() {
    // ...
}
```
{% endraw %}

---

## Data Access Helpers

### Type-Safe Event Data Extraction

Use generic `GetData[T]` for compile-time type safety:

{% raw %}
```go
// String extraction
pathname, found := v1beta1.GetData[string](event, "pathname")
if !found {
    return nil, nil
}

// Integer extraction
flags, found := v1beta1.GetData[int64](event, "flags")

// Unsigned integer
uid, found := v1beta1.GetData[uint64](event, "uid")

// Boolean
success, found := v1beta1.GetData[bool](event, "success")

// Byte array
data, found := v1beta1.GetData[[]byte](event, "data")
```
{% endraw %}

### Null-Safe Accessors

Use protobuf's generated `Get*()` methods - they're nil-safe:

{% raw %}
```go
// Safe - returns empty string if any field is nil
processPath := event.GetWorkload().GetProcess().GetExecutable().GetPath()

// Safe - returns 0 if nil
entityId := event.GetWorkload().GetProcess().GetEntityId()

// Safe - returns empty slice if nil
ancestors := event.GetWorkload().GetProcess().GetAncestors()
```
{% endraw %}

**Never do this**:
{% raw %}
```go
// UNSAFE - panics if any field is nil!
if event.Workload != nil && event.Workload.Process != nil {
    path := event.Workload.Process.Executable.Path  // Can still panic!
}
```
{% endraw %}

### Common Access Patterns

{% raw %}
```go
// Process information
entityId := event.GetWorkload().GetProcess().GetEntityId()
pid := event.GetWorkload().GetProcess().GetPid()
executable := event.GetWorkload().GetProcess().GetExecutable().GetPath()
commandLine := event.GetWorkload().GetProcess().GetCommandLine()

// Container information (if in container)
containerId := event.GetWorkload().GetContainer().GetId()
containerName := event.GetWorkload().GetContainer().GetName()
containerImage := event.GetWorkload().GetContainer().GetImage().GetName()

// Kubernetes information (if available)
namespace := event.GetWorkload().GetK8s().GetNamespace()
podName := event.GetWorkload().GetK8s().GetPod().GetName()

// User information
uid := event.GetWorkload().GetProcess().GetRealUser().GetId()
username := event.GetWorkload().GetProcess().GetRealUser().GetName()
```
{% endraw %}

---

### Next Steps

- **[Quick Start](quickstart.md)**: Write your first detector
- **[DataStore API](datastore-api.md)**: Complete datastore reference
- **Real Examples**: Browse `detectors/` directory

### Getting Help

- **GitHub Issues**: https://github.com/aquasecurity/tracee/issues
- **Discussions**: https://github.com/aquasecurity/tracee/discussions
- **API Definitions**: `api/v1beta1/detection/detector.go`

---

## Migration from Signatures

This section helps you migrate existing signatures to the new detector API.

### Quick Comparison

| Feature | Old Signature API | New Detector API |
|---------|-------------------|------------------|
| Package | `package main` (plugin) | `package detectors` (compiled-in) |
| Interface | `detect.Signature` | `detection.EventDetector` |
| Event Access | `protocol.Event` (runtime casting) | `*v1beta1.Event` (direct protobuf) |
| Data Extraction | Manual loop through `Args` | `GetData[T]` (type-safe) |
| Event Filtering | `GetSelectedEvents()` (name only) | `EventRequirement` (data + scope filters) |
| Output | Async `ctx.Callback()` | Synchronous return `[]detection.DetectorOutput` |
| Metadata | `GetMetadata()` separate | `GetDefinition()` unified |
| State Management | Manual | LRU caches + datastore access |
| Context Access | Limited | Full datastore access (process, container, etc.) |
| Auto-enrichment | Manual | Declarative (`AutoPopulateFields`) |
| Testing | Callback mocking | Direct function calls |
| Registration | `ExportedSignatures` list | `init()` auto-registration |

### Step-by-Step Migration

**Before** (old signature):

{% raw %}
```go
package main

import (
	"github.com/aquasecurity/tracee/types/detect"
	"github.com/aquasecurity/tracee/types/protocol"
)

type MySignature struct {
	cb detect.SignatureHandler
}

func (s *MySignature) GetMetadata() detect.SignatureMetadata {
	return detect.SignatureMetadata{
		ID:          "TRC-001",
		Name:        "Sensitive File Access",
		Description: "Detects access to sensitive files",
		Version:     "1.0.0",
		Severity:    detect.SeverityMedium,
	}
}

func (s *MySignature) GetSelectedEvents() []detect.SignatureEventSelector {
	return []detect.SignatureEventSelector{
		{Source: "tracee", Name: "security_file_open"},
	}
}

func (s *MySignature) Init(ctx detect.SignatureContext) error {
	s.cb = ctx.Callback
	return nil
}

func (s *MySignature) OnEvent(event protocol.Event) error {
	traceEvent, ok := event.Payload.(trace.Event)
	if !ok {
		return fmt.Errorf("unexpected event type")
	}

	var pathname string
	for _, arg := range traceEvent.Args {
		if arg.Name == "pathname" {
			pathname = arg.Value.(string)
			break
		}
	}

	if !strings.HasPrefix(pathname, "/etc/") {
		return nil
	}

	s.cb(&detect.Finding{
		SigMetadata: s.GetMetadata(),
		Event:       event,
		Data: map[string]interface{}{
			"file": pathname,
		},
	})

	return nil
}

func (s *MySignature) Close() {}
```
{% endraw %}

**After** (new detector):

{% raw %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endraw %}

### Migration Checklist

- [ ] Change package from `main` to `detectors`
- [ ] Add `init() { register(&YourDetector{}) }`
- [ ] Update imports to `api/v1beta1` packages
- [ ] Replace `detect.Signature` with `detection.EventDetector`
- [ ] Combine `GetMetadata()` + `GetSelectedEvents()` → `GetDefinition()`
- [ ] Move data filters from `OnEvent()` to `EventRequirement.DataFilters`
- [ ] Replace `protocol.Event` casting with `*v1beta1.Event` parameter
- [ ] Replace manual `Args` loop with `GetData[T]()`
- [ ] Replace `ctx.Callback()` with synchronous `return []detection.DetectorOutput{}`
- [ ] Return `DetectorOutput` with `Data` fields instead of full `v1beta1.Event`
- [ ] Add `AutoPopulateFields` for automatic enrichment
- [ ] Update tests to call `OnEvent()` directly and verify `DetectorOutput`
- [ ] Remove `Close()` if empty (optional interface)

### Common Pattern Translations

**Pattern: Event Selection**
{% raw %}
```go
// Before
GetSelectedEvents() []detect.SignatureEventSelector {
	return []detect.SignatureEventSelector{
		{Source: "tracee", Name: "openat"},
	}
}

// After
Requirements: detection.DetectorRequirements{
	Events: []detection.EventRequirement{
		{Name: "openat"},
	},
},
```
{% endraw %}

**Pattern: Data Extraction**
{% raw %}
```go
// Before
var pathname string
for _, arg := range traceEvent.Args {
	if arg.Name == "pathname" {
		pathname = arg.Value.(string)  // Runtime casting
		break
	}
}

// After
pathname, found := v1beta1.GetData[string](event, "pathname")  // Compile-time safety
if !found {
	return nil, nil
}
```
{% endraw %}

**Pattern: Filtering**
{% raw %}
```go
// Before (in OnEvent)
if !strings.HasPrefix(pathname, "/etc/") {
	return nil  // Manual filter
}

// After (in EventRequirement)
DataFilters: []string{"pathname=/etc/*"},  // Engine filters before dispatch
```
{% endraw %}

**Pattern: Callback → Return**
{% raw %}
```go
// Before
s.cb(&detect.Finding{
	SigMetadata: s.GetMetadata(),
	Event:       event,
	Data:        map[string]interface{}{"file": pathname},
})
return nil

// After
return []detection.DetectorOutput{{
	Data: []*v1beta1.EventValue{
		v1beta1.NewStringValue("file", pathname),
	},
}}, nil
```
{% endraw %}

---

## Troubleshooting

### Problem: Detector Not Running

**Symptoms**: Detector code exists but never triggers

**Diagnosis**:
```bash
# Check if detector is registered
sudo ./dist/tracee list | grep detectors

# Check Tracee logs
sudo ./dist/tracee --logging debug
```

**Solutions**:
1. Verify `init() { register(&YourDetector{}) }` exists
2. Rebuild Tracee: `make clean && make tracee`
3. Check for panics in `Init()` (blocks registration)

### Problem: No Events Received

**Symptoms**: `OnEvent()` never called

**Diagnosis**:
1. Check event name: `sudo ./dist/tracee list | grep event_name`
2. Temporarily remove `DataFilters` to see if they're too restrictive
3. Add debug logging in `OnEvent()`

**Solutions**:
1. Fix event name typo in `EventRequirement.Name`
2. Adjust `DataFilters` - they might be filtering everything out
3. Check `ScopeFilters` - ensure they match your test environment

### Problem: DataStore Returns ErrNotFound

**Symptoms**: `GetProcess()`, `GetContainer()` return `ErrNotFound`

**Possible causes**:
1. Process tree not enabled: `--stores process`
2. Process exited before query
3. Container not tracked yet

**Solutions**:
{% raw %}
```go
// Always handle ErrNotFound gracefully
proc, err := d.dataStores.Processes().GetProcess(entityId)
if errors.Is(err, datastores.ErrNotFound) {
    // This is normal - process might have exited
    return nil, nil
}
if err != nil {
    return nil, fmt.Errorf("unexpected error: %w", err)
}
```
{% endraw %}

### Problem: High CPU Usage

**Symptoms**: Tracee consuming excessive CPU

**Diagnosis**:
1. Check if detector receives too many events
2. Profile detector logic

**Solutions**:
1. Add more restrictive `DataFilters`
2. Optimize detector logic - avoid expensive operations in `OnEvent()`
3. Use caching for repeated computations
4. Consider batching if doing aggregations

### Problem: Memory Leaks

**Symptoms**: Memory usage grows over time

**Causes**:
1. Unbounded maps/slices in detector state
2. Not cleaning up old entries

**Solutions**:
{% raw %}
```go
// Use LRU cache with TTL instead of unbounded maps
import "github.com/hashicorp/golang-lru/v2/expirable"

func (d *MyDetector) Init(params detection.DetectorParams) error {
    // Bounded cache with expiration
    d.cache = expirable.NewLRU[string, *data](
        1000,          // Max entries
        nil,           // No eviction callback
        5*time.Minute, // TTL
    )
    return nil
}
```
{% endraw %}

---

### Docs Detectors Datastore Api (`docs/docs/detectors/datastore-api.md`)

# DataStore API Reference

Complete API reference for Tracee's DataStore system. This provides read-only access to system state information for detectors.

## Table of Contents

1. [Overview](#overview)
2. [Core Concepts](#core-concepts)
3. [Registry Interface](#registry-interface)
4. [ProcessStore](#processstore)
5. [ContainerStore](#containerstore)
6. [SystemStore](#systemstore)
7. [SyscallStore](#syscallstore)
8. [KernelSymbolStore](#kernelsymbolstore)
9. [DNSStore](#dnsstore)
10. [Health and Metrics](#health-and-metrics)
11. [Error Handling](#error-handling)
12. [Advanced Usage](#advanced-usage)
13. [Writable DataStores](#writable-datastores)
14. [Summary](#summary)

---

## Overview

### What DataStores Provide

DataStores give detectors read-only access to system state information collected by Tracee:

- **Process Tree**: Process information, ancestry, and relationships
- **Container Registry**: Container metadata, Kubernetes pod information
- **System Information**: Immutable system details (architecture, kernel, OS)
- **Syscall Mapping**: Syscall ID ↔ name conversion
- **Kernel Symbols**: Symbol resolution for addresses
- **DNS Cache**: DNS query responses

### Registry Pattern

All datastores are accessed through a single `Registry` interface provided to detectors via `DetectorParams`:

{% raw %}
```go
func (d *MyDetector) Init(params detection.DetectorParams) error {
    d.dataStores = params.DataStores  // Registry interface

    // Access individual stores
    processStore := d.dataStores.Processes()
    containerStore := d.dataStores.Containers()

    return nil
}
```
{% endraw %}

### Read-Only Access Model

DataStores are **read-only** from the detector perspective:

- ✅ Query existing data
- ✅ Iterate over collections
- ✅ Check health and metrics
- ❌ No writes or modifications
- ❌ No data deletion

The engine manages datastore lifecycle (initialization, updates, shutdown).

### Thread Safety Guarantees

All datastores are **thread-safe** for concurrent access:

- Multiple detectors can query the same store simultaneously
- No locking required in detector code
- Internal RWMutex protects shared state

---

## Core Concepts

### DataStore Base Interface

All datastores implement this base interface:

{% raw %}
```go
type DataStore interface {
    // Name returns the name of this datastore
    Name() string

    // GetHealth returns the current health status
    GetHealth() *HealthInfo

    // GetMetrics returns operational metrics
    GetMetrics() *DataStoreMetrics
}
```
{% endraw %}

### Common Errors

Sentinel errors for standard error cases:

{% raw %}
```go
var (
    ErrNotFound        = errors.New("entity not found")
    ErrStoreUnhealthy  = errors.New("datastore is unhealthy")
    ErrNotImplemented  = errors.New("operation not implemented")
    ErrInvalidArgument = errors.New("invalid argument")
)
```
{% endraw %}

**StoreError** wraps errors with context:

{% raw %}
```go
type StoreError struct {
    StoreName string  // Which store
    Operation string  // Which operation
    Err       error   // Underlying error
}
```
{% endraw %}

### Health Status

{% raw %}
```go
type HealthStatus int

const (
    HealthUnknown   HealthStatus = iota  // 0: Unknown state
    HealthHealthy                        // 1: Operating normally
    HealthUnhealthy                      // 2: Degraded or unavailable
)
```
{% endraw %}

---

## Registry Interface

### Registry Methods

{% raw %}
```go
type Registry interface {
    // Core stores (always return non-nil, but may be unavailable)
    Processes() ProcessStore
    Containers() ContainerStore
    KernelSymbols() KernelSymbolStore
    DNS() DNSStore
    System() SystemStore
    Syscalls() SyscallStore

    // Custom stores (returns error if not found)
    GetCustom(name string) (DataStore, error)

    // Introspection
    List() []string
    IsAvailable(name string) bool
    GetMetadata(name string) (*DataStoreMetadata, error)
    GetMetrics(name string) (*DataStoreMetrics, error)
}
```
{% endraw %}

**Nil-Safety Guarantee:**

Accessor methods (`Processes()`, `Containers()`, etc.) **never return nil**. If a store is not registered or unavailable, they return a null object that:

- Implements the store interface
- Returns `ErrStoreUnhealthy` for all operations
- Has health status set to `HealthUnhealthy`

This eliminates nil checks and allows safe method chaining:

```go
// Always safe - no nil check needed
proc, err := datastores.Processes().GetProcess(entityID)
if errors.Is(err, datastores.ErrStoreUnhealthy) {
    // Store not available
}
if errors.Is(err, datastores.ErrNotFound) {
    // Process not found
}
```

### Usage Examples

**Basic access**:
{% raw %}
```go
// Direct access to core stores
processStore := d.dataStores.Processes()
containerStore := d.dataStores.Containers()

// Use immediately
proc, err := processStore.GetProcess(entityId)
if errors.Is(err, datastores.ErrNotFound) {
    // Process not found
}
```
{% endraw %}

**Check availability**:
{% raw %}
```go
// Check if optional store is available
if d.dataStores.IsAvailable("system") {
    systemStore := d.dataStores.System()
    sysInfo := systemStore.GetSystemInfo()
    // Use system info
}
```
{% endraw %}

**List all stores**:
{% raw %}
```go
stores := d.dataStores.List()
d.logger.Infow("Available datastores", "stores", stores)
// Output: ["process", "container", "symbol", "dns", "system", "syscall"]
```
{% endraw %}

**Get store metadata**:
{% raw %}
```go
metadata, err := d.dataStores.GetMetadata("process")
if err == nil {
    d.logger.Infow("Process store info",
        "name", metadata.Name,
        "description", metadata.Description)
}
```
{% endraw %}

---

## ProcessStore

### Interface

{% raw %}
```go
type ProcessStore interface {
    DataStore

    // GetProcess retrieves process information by entity ID
    // Returns ErrNotFound if not found
    GetProcess(entityId uint32) (*ProcessInfo, error)

    // GetChildProcesses returns all child processes
    // Returns empty slice if no children
    GetChildProcesses(entityId uint32) ([]*ProcessInfo, error)

    // GetAncestry retrieves the ancestry chain up to maxDepth levels
    // [0] = process itself, [1] = parent, [2] = grandparent, etc.
    // Returns empty slice if maxDepth <= 0 or process not found
    GetAncestry(entityId uint32, maxDepth int) ([]*ProcessInfo, error)
}
```
{% endraw %}

### ProcessInfo Structure

{% raw %}
```go
type ProcessInfo struct {
    EntityID  uint32    // Primary key (hash) - matches Event.Process.EntityId
    PID       uint32    // OS process ID (for display/logging)
    PPID      uint32    // OS parent PID (for display/logging)
    Name      string    // Binary name: "bash"
    Exe       string    // Full executable path: "/usr/bin/bash"
    StartTime time.Time // Process start time
    UID       uint32    // User ID
    GID       uint32    // Group ID
}
```
{% endraw %}

### Methods

#### GetProcess

Retrieve process information by entity ID.

**Signature**:
{% raw %}
```go
GetProcess(entityId uint32) (*ProcessInfo, error)
```
{% endraw %}

**Parameters**:

- `entityId`: Process entity ID (hash from ProcessTree, matches `event.Workload.Process.EntityId`)

**Returns**:

- `*ProcessInfo`: Process information if found
- `error`: `ErrNotFound` if not found, `nil` on success

**Example**:
{% raw %}
```go
entityId := event.Workload.Process.EntityId.Value
proc, err := d.dataStores.Processes().GetProcess(entityId)
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Warnw("Process not found", "entity_id", entityId)
    return nil, nil
}
if err != nil {
    return nil, fmt.Errorf("failed to get process: %w", err)
}

d.logger.Infow("Found process",
    "pid", proc.PID,           // OS PID for logging
    "name", proc.Name,         // "bash"
    "exe", proc.Exe,           // "/usr/bin/bash"
    "uid", proc.UID,
    "start_time", proc.StartTime)
```
{% endraw %}

#### GetAncestry

Retrieve process ancestry chain up to specified depth.

**Signature**:
{% raw %}
```go
GetAncestry(entityId uint32, maxDepth int) ([]*ProcessInfo, error)
```
{% endraw %}

**Parameters**:

- `entityId`: Process entity ID
- `maxDepth`: Maximum ancestry depth (typically 5)

**Returns**:

- `[]*ProcessInfo`: Ancestry chain where `[0]` = process itself, `[1]` = parent, `[2]` = grandparent, etc.
- `error`: Error if query fails

**Behavior**:

- Returns empty slice if `maxDepth <= 0`
- Returns empty slice if process not found
- Stops if parent not found in tree
- Stops if circular reference detected

**Example**:
{% raw %}
```go
ancestry, err := d.dataStores.Processes().GetAncestry(entityId, 5)
if err != nil {
    return nil, fmt.Errorf("failed to get ancestry: %w", err)
}

if len(ancestry) == 0 {
    d.logger.Debugw("No ancestry found")
    return nil, nil
}

// ancestry[0] is the process itself
// ancestry[1] is the parent
// ancestry[2] is the grandparent, etc.

for i, ancestor := range ancestry {
    d.logger.Debugw("Ancestor",
        "level", i,
        "pid", ancestor.PID,
        "name", ancestor.Name,
        "exe", ancestor.Exe)
}

// Check if parent is suspicious
if len(ancestry) > 1 {
    parent := ancestry[1]
    if strings.Contains(parent.Exe, "suspicious") {
        // Trigger detection
    }
}
```
{% endraw %}

#### GetChildProcesses

Retrieve all child processes of a given process.

**Signature**:
{% raw %}
```go
GetChildProcesses(entityId uint32) ([]*ProcessInfo, error)
```
{% endraw %}

**Parameters**:

- `entityId`: Parent process entity ID

**Returns**:

- `[]*ProcessInfo`: List of child processes (empty if no children)
- `error`: Error if query fails

**Example**:
{% raw %}
```go
children, err := d.dataStores.Processes().GetChildProcesses(entityId)
if err != nil {
    return nil, err
}

d.logger.Infow("Process has children", "count", len(children))
for _, child := range children {
    d.logger.Debugw("Child process",
        "pid", child.PID,
        "name", child.Name,
        "exe", child.Exe)
}
```
{% endraw %}

### EntityID vs PID

!!! Important "Always Use EntityID for Lookups"
    **EntityID** is the primary key for process lookups, not PID.

    **Why?**

    - PIDs can be reused after a process exits
    - EntityID is unique for the lifetime of the process
    - EntityID matches `event.Workload.Process.EntityId`

    **Usage**:

    - **EntityID**: For all datastore lookups (`GetProcess`, `GetAncestry`, etc.)
    - **PID**: Only for display, logging, and debugging

**Correct**:
{% raw %}
```go
// Get EntityID from event
entityId := event.Workload.Process.EntityId.Value

// Use EntityID for lookup
proc, err := d.dataStores.Processes().GetProcess(entityId)
if errors.Is(err, datastores.ErrNotFound) {
    return nil, nil
}

// Use PID for logging only
d.logger.Infow("Process found", "pid", proc.PID, "name", proc.Name)
```
{% endraw %}

**Incorrect**:
{% raw %}
```go
// ❌ Don't use PID for lookups
pid := event.Workload.Process.Pid.Value
proc, err := d.dataStores.Processes().GetProcess(uint32(pid))  // Wrong!
```
{% endraw %}

---

## ContainerStore

### Interface

{% raw %}
```go
type ContainerStore interface {
    DataStore

    // GetContainer retrieves container by ID
    // Returns ErrNotFound if not found
    GetContainer(id string) (*ContainerInfo, error)

    // GetContainerByName retrieves container by name
    // Returns ErrNotFound if not found
    GetContainerByName(name string) (*ContainerInfo, error)

    // ListContainers returns currently running containers, optionally filtered
    // Pass nil filter to return all containers
    // Returns empty slice if no containers match the filter
    ListContainers(filter *ContainerFilter) ([]*ContainerInfo, error)
}

// ContainerFilter allows filtering containers by various criteria
// All fields are optional - nil values mean the field is not used for filtering
type ContainerFilter struct {
    Name    *string  // Filter by container name (exact match)
    Image   *string  // Filter by image name (exact match)
    Runtime *string  // Filter by runtime (e.g., "docker", "containerd")
}
```
{% endraw %}

**Important:** Container enrichment (`--enrichment container`) is required for container metadata:

When `--enrichment container` is enabled, Tracee:
1. Queries container runtimes (Docker, containerd, CRI-O, Podman) at userspace
2. Populates the container datastore with Name, Image, Pod info, etc.
3. Attaches this metadata to Event.Workload.Container fields

**Without `--enrichment container`:**
- Only `Event.Workload.Container.Id` is available (from cgroup context)
- Container datastore queries will NOT return Name, Image, or Pod information
- Both Event fields and datastore will lack enriched metadata

**With `--enrichment container`:**
- `Event.Workload.Container.Name`, `.Image`, `.Pod` are populated in events
- Container datastore can return full metadata when queried
- Detectors can choose: read from Event fields directly OR query datastore

**Why use datastore queries if enrichment is required anyway?**
- More flexible error handling (e.g., handle not-found containers)
- Access to additional methods like `GetContainerByName()`
- Useful when container info is needed conditionally in detection logic

### ContainerInfo Structure

{% raw %}
```go
type ContainerInfo struct {
    ID          string      // Container ID
    Name        string      // Container name
    Image       string      // Container image
    ImageDigest string      // Image digest (SHA256)
    Runtime     string      // Runtime: "docker", "containerd", "crio"
    StartTime   time.Time   // Container start time
    Pod         *K8sPodInfo // Kubernetes pod info (nil for non-K8s)
}

type K8sPodInfo struct {
    Name      string // Pod name
    UID       string // Pod UID
    Namespace string // Pod namespace
    Sandbox   bool   // Whether this is a sandbox container
}
```
{% endraw %}

### Methods

#### GetContainer

Retrieve container information by container ID.

**Signature**:
{% raw %}
```go
GetContainer(id string) (*ContainerInfo, error)
```
{% endraw %}

**Parameters**:

- `id`: Container ID (full ID or short form)

**Returns**:

- `*ContainerInfo`: Container information if found
- `error`: `ErrNotFound` if not found, `nil` on success

**Example**:
{% raw %}
```go
containerID := v1beta1.GetContainerID(event)
if containerID == "" {
    // Not a container event
    return nil, nil
}

container, err := d.dataStores.Containers().GetContainer(containerID)
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Warnw("Container not found", "id", containerID)
    return nil, nil
}
if err != nil {
    return nil, fmt.Errorf("failed to get container: %w", err)
}

d.logger.Infow("Container info",
    "id", container.ID,
    "name", container.Name,
    "image", container.Image,
    "runtime", container.Runtime,
    "start_time", container.StartTime)

// Access Kubernetes pod information
if container.Pod != nil {
    d.logger.Infow("Pod info",
        "name", container.Pod.Name,
        "namespace", container.Pod.Namespace,
        "uid", container.Pod.UID,
        "sandbox", container.Pod.Sandbox)
}
```
{% endraw %}

#### GetContainerByName

Retrieve container information by container name.

**Signature**:
{% raw %}
```go
GetContainerByName(name string) (*ContainerInfo, error)
```
{% endraw %}

**Parameters**:

- `name`: Container name

**Returns**:

- `*ContainerInfo`: Container information if found
- `error`: `ErrNotFound` if not found, `nil` on success

**Example**:
{% raw %}
```go
container, err := d.dataStores.Containers().GetContainerByName("web-server")
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Debugw("Container not found", "name", "web-server")
    return nil, nil
}
if err != nil {
    return nil, fmt.Errorf("failed to get container: %w", err)
}

d.logger.Infow("Found container", "id", container.ID, "image", container.Image)
```
{% endraw %}

#### ListContainers

List all currently running containers, with optional filtering by name, image, or runtime using functional options.

**Signature**:
{% raw %}
```go
ListContainers(opts ...ContainerFilterOption) ([]*ContainerInfo, error)
```
{% endraw %}

**Filter Options**:

- `WithName(name string)`: Filter by exact container name match
- `WithImage(image string)`: Filter by exact image name match
- `WithRuntime(runtime string)`: Filter by runtime type ("docker", "containerd", "crio", "podman")

**Returns**:

- `[]*ContainerInfo`: Slice of containers matching the filter (empty if none found)
- `error`: Error if the operation fails (never returns `ErrNotFound`)

**Use Cases**:

- Container discovery at startup
- Periodic container inventory
- Finding all containers running a specific image
- Filtering containers by runtime type

**Examples**:

{% raw %}
```go
// List all containers (no filter)
containers, err := d.dataStores.Containers().ListContainers()
if err != nil {
    return fmt.Errorf("failed to list containers: %w", err)
}

// Filter by image
nginxContainers, err := d.dataStores.Containers().ListContainers(
    datastores.WithImage("nginx:latest"),
)

// Filter by multiple criteria (AND logic)
containers, err = d.dataStores.Containers().ListContainers(
    datastores.WithImage("redis:7"),
    datastores.WithRuntime("docker"),
)

// Discover containers at detector initialization
func (d *MyDetector) Init(params detection.DetectorParams) error {
    containers, err := params.DataStores.Containers().ListContainers()
    if err != nil {
        return fmt.Errorf("failed to discover containers: %w", err)
    }
    // Initialize state based on existing containers
    for _, c := range containers {
        d.trackContainer(c)
    }
    return nil
}
```
{% endraw %}

**Notes**:

- Pass `nil` filter to list all containers
- All filters use AND logic (must match all specified criteria)
- Returns only live, running containers (excludes expired/dead)
- Cleaner alternative to buffering `existing_container` events

**Container Enrichment**:

When `--enrichment container` flag is used, containers discovered at startup are automatically enriched with runtime metadata (name, image, digest, pod information). Without enrichment, only container ID and runtime are available from cgroup paths.

---

## SystemStore

### Interface

{% raw %}
```go
type SystemStore interface {
    DataStore

    // GetSystemInfo returns immutable system information
    // Collected at Tracee startup, never changes
    GetSystemInfo() *SystemInfo
}
```
{% endraw %}

### SystemInfo Structure

{% raw %}
```go
type SystemInfo struct {
    Architecture    string            // CPU architecture: "x86_64", "aarch64"
    KernelRelease   string            // Kernel version: "5.15.0-generic"
    Hostname        string            // System hostname
    BootTime        time.Time         // System boot time
    TraceeStartTime time.Time         // When Tracee started
    OSName          string            // OS name: "Ubuntu"
    OSVersion       string            // OS version: "22.04"
    OSPrettyName    string            // Full OS name: "Ubuntu 22.04 LTS"
    TraceeVersion   string            // Tracee version string
    InitNamespaces  map[string]uint32 // Init process namespaces
}
```
{% endraw %}

### Methods

#### GetSystemInfo

Retrieve immutable system information.

**Signature**:
{% raw %}
```go
GetSystemInfo() *SystemInfo
```
{% endraw %}

**Returns**:

- `*SystemInfo`: System information (never nil if store available)

**Characteristics**:

- **Immutable**: Data never changes during Tracee lifetime
- **Fast**: No lookup, just returns cached data
- **Always available**: Returns immediately with no errors

**Example**:
{% raw %}
```go
// System store always returns non-nil (per registry contract)
// Check IsAvailable() if you need to verify store is ready
systemStore := d.dataStores.System()
sysInfo := systemStore.GetSystemInfo()

d.logger.Infow("System information",
    "arch", sysInfo.Architecture,
    "kernel", sysInfo.KernelRelease,
    "os", sysInfo.OSPrettyName,
    "hostname", sysInfo.Hostname,
    "tracee_version", sysInfo.TraceeVersion,
    "boot_time", sysInfo.BootTime,
    "tracee_start", sysInfo.TraceeStartTime)

// Check architecture for platform-specific logic
if sysInfo.Architecture == "x86_64" {
    // x86-specific detection logic
} else if sysInfo.Architecture == "aarch64" {
    // ARM-specific detection logic
}

// Add to detection metadata
detection.Data = append(detection.Data,
    v1beta1.NewStringValue("system_arch", sysInfo.Architecture),
    v1beta1.NewStringValue("kernel_version", sysInfo.KernelRelease),
    v1beta1.NewStringValue("os", sysInfo.OSPrettyName))
```
{% endraw %}

---

## SyscallStore

### Interface

{% raw %}
```go
type SyscallStore interface {
    DataStore

    // GetSyscallName returns syscall name for given ID
    // Returns ErrNotFound if not found
    GetSyscallName(id int32) (string, error)

    // GetSyscallID returns syscall ID for given name
    // Returns ErrNotFound if not found
    GetSyscallID(name string) (int32, error)
}
```
{% endraw %}

### Methods

#### GetSyscallName

Get syscall name from syscall ID.

**Signature**:
{% raw %}
```go
GetSyscallName(id int32) (string, error)
```
{% endraw %}

**Parameters**:

- `id`: Syscall ID (architecture-specific)

**Returns**:

- `string`: Syscall name if found
- `error`: `ErrNotFound` if not found, `nil` on success

**Example**:
{% raw %}
```go
syscallID, found := v1beta1.GetData[int32](event, "syscall_id")
if !found {
    return nil, nil
}

name, err := d.dataStores.Syscalls().GetSyscallName(syscallID)
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Debugw("Unknown syscall", "id", syscallID)
    name = fmt.Sprintf("syscall_%d", syscallID)
}
if err != nil && !errors.Is(err, datastores.ErrNotFound) {
    return nil, fmt.Errorf("failed to get syscall name: %w", err)
}

d.logger.Infow("Syscall detected", "name", name, "id", syscallID)
```
{% endraw %}

#### GetSyscallID

Get syscall ID from syscall name.

**Signature**:
{% raw %}
```go
GetSyscallID(name string) (int32, error)
```
{% endraw %}

**Parameters**:

- `name`: Syscall name (e.g., "execve", "openat")

**Returns**:

- `int32`: Syscall ID if found
- `error`: `ErrNotFound` if not found, `nil` on success

**Example**:
{% raw %}
```go
id, err := d.dataStores.Syscalls().GetSyscallID("execve")
if err == nil {
    d.logger.Infow("Syscall", "name", "execve", "id", id)
    // ID = 59 on x86_64
}
```
{% endraw %}

!!! Warning "Architecture-Specific"
    Syscall IDs are **architecture-specific**:

    - ID 59 = `execve` on x86_64
    - ID 221 = `execve` on ARM64

    The store returns IDs for the **current architecture** only.

---

## KernelSymbolStore

### Interface

{% raw %}
```go
type KernelSymbolStore interface {
    DataStore

    // ResolveSymbolByAddress resolves address to symbols
    // Returns multiple symbols if aliases exist
    // Returns ErrNotFound if address cannot be resolved
    ResolveSymbolByAddress(addr uint64) ([]*SymbolInfo, error)

    // GetSymbolAddress returns address of named symbol
    // If multiple symbols exist, returns first one found
    // Returns ErrNotFound if symbol not found
    GetSymbolAddress(name string) (uint64, error)

    // ResolveSymbolsBatch resolves multiple addresses at once
    // Returns map of address -> symbols for found addresses
    // Missing addresses are not included in result
    ResolveSymbolsBatch(addrs []uint64) (map[uint64][]*SymbolInfo, error)
}
```
{% endraw %}

### SymbolInfo Structure

{% raw %}
```go
type SymbolInfo struct {
    Name    string // Symbol name: "do_sys_open"
    Address uint64 // Symbol address: 0xffffffff81234567
    Module  string // Module name: "system" for kernel, or module name
}
```
{% endraw %}

### Methods

#### ResolveSymbolByAddress

Resolve kernel address to symbol information.

**Signature**:
{% raw %}
```go
ResolveSymbolByAddress(addr uint64) ([]*SymbolInfo, error)
```
{% endraw %}

**Parameters**:

- `addr`: Kernel address to resolve

**Returns**:

- `[]*SymbolInfo`: List of symbols at this address (usually one, but aliases possible)
- `error`: `ErrNotFound` if address cannot be resolved

**Example**:
{% raw %}
```go
address, found := v1beta1.GetData[uint64](event, "address")
if !found {
    return nil, nil
}

symbols, err := d.dataStores.KernelSymbols().ResolveSymbolByAddress(address)
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Warnw("Unknown address", "addr", fmt.Sprintf("0x%x", address))
    return nil, nil
}
if err != nil {
    return nil, err
}

for _, symbol := range symbols {
    d.logger.Infow("Symbol",
        "name", symbol.Name,
        "address", fmt.Sprintf("0x%x", symbol.Address),
        "module", symbol.Module)

    // Check if this is a kernel module (rootkit detection)
    if symbol.Module != "system" {
        // Potential hook detected
        return []detection.DetectorOutput{{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("symbol", symbol.Name),
                v1beta1.NewStringValue("module", symbol.Module),
            },
        }}, nil
    }
}
```
{% endraw %}

#### GetSymbolAddress

Get the address of a named kernel symbol.

**Signature**:
{% raw %}
```go
GetSymbolAddress(name string) (uint64, error)
```
{% endraw %}

**Parameters**:

- `name`: Symbol name (e.g., "do_sys_open")

**Returns**:

- `uint64`: Symbol address
- `error`: `ErrNotFound` if symbol not found

**Example**:
{% raw %}
```go
addr, err := d.dataStores.KernelSymbols().GetSymbolAddress("do_sys_open")
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Warnw("Symbol not found", "name", "do_sys_open")
    return nil, nil
}
if err != nil {
    return nil, err
}

d.logger.Infow("Symbol address",
    "name", "do_sys_open",
    "address", fmt.Sprintf("0x%x", addr))
```
{% endraw %}

#### ResolveSymbolsBatch

Resolve multiple addresses in a single call (more efficient).

**Signature**:
{% raw %}
```go
ResolveSymbolsBatch(addrs []uint64) (map[uint64][]*SymbolInfo, error)
```
{% endraw %}

**Parameters**:

- `addrs`: List of addresses to resolve

**Returns**:

- `map[uint64][]*SymbolInfo`: Map of address to symbols (only includes found addresses)
- `error`: Error if batch operation fails

**Example**:
{% raw %}
```go
// Get syscall table addresses from event
addresses := []uint64{0xffffffff81234567, 0xffffffff81abcdef, ...}

// Batch resolve (more efficient than individual lookups)
symbols, err := d.dataStores.KernelSymbols().ResolveSymbolsBatch(addresses)
if err != nil {
    return nil, err
}

// Process results
for addr, symbolList := range symbols {
    for _, symbol := range symbolList {
        d.logger.Infow("Resolved",
            "address", fmt.Sprintf("0x%x", addr),
            "symbol", symbol.Name,
            "module", symbol.Module)

        // Detect hooks by checking module
        if symbol.Module != "system" {
            // Hook detected
        }
    }
}

// Check for addresses that couldn't be resolved
for _, addr := range addresses {
    if _, found := symbols[addr]; !found {
        d.logger.Warnw("Address not resolved", "addr", fmt.Sprintf("0x%x", addr))
    }
}
```
{% endraw %}

---

## DNSStore

### Interface

{% raw %}
```go
type DNSStore interface {
    DataStore

    // GetDNSResponse retrieves cached DNS response
    // Returns ErrNotFound if no cached response found
    GetDNSResponse(query string) (*DNSResponse, error)
}
```
{% endraw %}

### DNSResponse Structure

{% raw %}
```go
type DNSResponse struct {
    Query   string   // DNS query (domain name)
    IPs     []string // Resolved IP addresses
    Domains []string // Resolved domains (for reverse lookups)
}
```
{% endraw %}

### Methods

#### GetDNSResponse

Retrieve cached DNS query response.

**Signature**:
{% raw %}
```go
GetDNSResponse(query string) (*DNSResponse, error)
```
{% endraw %}

**Parameters**:

- `query`: DNS query (domain name)

**Returns**:

- `*DNSResponse`: DNS response if cached
- `error`: `ErrNotFound` if not cached, `nil` on success

**Example**:
{% raw %}
```go
domain := "malicious.example.com"
response, err := d.dataStores.DNS().GetDNSResponse(domain)
if errors.Is(err, datastores.ErrNotFound) {
    d.logger.Debugw("No DNS cache entry", "domain", domain)
    return nil, nil
}
if err != nil {
    return nil, fmt.Errorf("failed to query DNS cache: %w", err)
}

d.logger.Infow("DNS cache hit",
    "domain", domain,
    "ips", response.IPs,
    "resolved_domains", response.Domains)
```
{% endraw %}

!!! Note "Cache Scope"
    The DNS cache only contains queries observed by Tracee while running.
    It does **not** contain historical or system-wide DNS data.

---

## Health and Metrics

### Health Information

All datastores expose health status:

{% raw %}
```go
type HealthInfo struct {
    Status    HealthStatus // Current health status
    Message   string       // Empty if healthy, error details if not
    LastCheck time.Time    // When health was last checked
}
```
{% endraw %}

**Example**:
{% raw %}
```go
processStore := d.dataStores.Processes()
health := processStore.GetHealth()

if health.Status != datastores.HealthHealthy {
    d.logger.Warnw("Process store unhealthy",
        "status", health.Status.String(),
        "message", health.Message,
        "last_check", health.LastCheck)

    // Decide whether to continue or fail
    if health.Status == datastores.HealthUnhealthy {
        return nil, fmt.Errorf("process store unhealthy: %s", health.Message)
    }
}
```
{% endraw %}

### Metrics

Datastores expose operational metrics:

{% raw %}
```go
type DataStoreMetrics struct {
    ItemCount    int64     // Number of items in store
    SuccessCount uint64    // Successful requests
    ErrorCount   uint64    // Failed requests
    CacheHits    uint64    // Cache hits (if applicable)
    CacheMisses  uint64    // Cache misses (if applicable)
    LastAccess   time.Time // Last access time
}
```
{% endraw %}

**Example**:
{% raw %}
```go
processStore := d.dataStores.Processes()
metrics := processStore.GetMetrics()

d.logger.Infow("Process store metrics",
    "item_count", metrics.ItemCount,
    "success_count", metrics.SuccessCount,
    "error_count", metrics.ErrorCount,
    "cache_hits", metrics.CacheHits,
    "cache_misses", metrics.CacheMisses,
    "hit_rate", float64(metrics.CacheHits) / float64(metrics.CacheHits + metrics.CacheMisses),
    "last_access", metrics.LastAccess)

// Alert if error rate is high
if metrics.ErrorCount > 0 {
    errorRate := float64(metrics.ErrorCount) / float64(metrics.SuccessCount + metrics.ErrorCount)
    if errorRate > 0.01 {
        d.logger.Warnw("High error rate", "rate", errorRate)
    }
}
```
{% endraw %}

---

## Error Handling

### Error Classification

**Not Found (common, non-error)**:
{% raw %}
```go
proc, err := d.dataStores.Processes().GetProcess(entityId)
if errors.Is(err, datastores.ErrNotFound) {
    // Process not in tree (maybe exited) - this is normal
    d.logger.Debugw("Process not found", "entity_id", entityId)
    return nil, nil  // Skip gracefully
}
```
{% endraw %}

**Store Errors (critical)**:
{% raw %}
```go
symbols, err := d.dataStores.KernelSymbols().ResolveSymbolByAddress(addr)
if err != nil {
    // Check for specific error types
    if errors.Is(err, datastores.ErrNotFound) {
        d.logger.Warnw("Address not in symbol table", "addr", addr)
        return nil, nil  // Skip gracefully
    }

    if errors.Is(err, datastores.ErrStoreUnhealthy) {
        // Store is unhealthy - critical
        return nil, fmt.Errorf("symbol store unhealthy: %w", err)
    }

    // Other error - log and return
    return nil, fmt.Errorf("symbol resolution failed: %w", err)
}
```
{% endraw %}

### Graceful Degradation

Always handle missing/unavailable stores gracefully:

{% raw %}
```go
func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // Check if optional store is available (use IsAvailable for readiness check)
    if d.dataStores.IsAvailable("system") {
        systemStore := d.dataStores.System()
        sysInfo := systemStore.GetSystemInfo()
        // Use system info for enrichment
    } else {
        d.logger.Debugw("System store unavailable, continuing without system info")
        // Continue detection without system info
    }

    // Check for optional data
    container, err := d.dataStores.Containers().GetContainer(containerID)
    if errors.Is(err, datastores.ErrNotFound) {
        d.logger.Debugw("Container not found, continuing without container context")
        // Continue detection without container info
    } else if err != nil {
        return nil, fmt.Errorf("container store error: %w", err)
    } else {
        // Use container info for enrichment
    }

    // Core detection logic...
}
```
{% endraw %}

---

## Advanced Usage

### Combining Multiple Stores

{% raw %}
```go
func (d *ContainerEscapeDetector) analyzeEscape(event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // Get process information
    entityId := event.Workload.Process.EntityId.Value
    proc, err := d.dataStores.Processes().GetProcess(entityId)
    if errors.Is(err, datastores.ErrNotFound) {
        return nil, nil
    }
    if err != nil {
        return nil, fmt.Errorf("process store error: %w", err)
    }

    // Get process ancestry
    ancestry, err := d.dataStores.Processes().GetAncestry(entityId, 5)
    if err != nil {
        return nil, err
    }

    // Get container information
    containerID := v1beta1.GetContainerID(event)
    container, err := d.dataStores.Containers().GetContainer(containerID)
    if errors.Is(err, datastores.ErrNotFound) {
        return nil, nil
    }
    if err != nil {
        return nil, fmt.Errorf("container store error: %w", err)
    }

    // Get system information
    sysInfo := d.dataStores.System().GetSystemInfo()

    // Analyze all together
    if d.detectEscape(proc, ancestry, container, sysInfo) {
        return []detection.DetectorOutput{{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("process", proc.Exe),
                v1beta1.NewStringValue("container", container.Name),
                v1beta1.NewStringValue("kernel", sysInfo.KernelRelease),
            },
        }}, nil
    }

    return nil, nil
}
```
{% endraw %}

### Caching Store References

Cache store references in Init() for better performance:

{% raw %}
```go
type MyDetector struct {
    logger         detection.Logger
    processStore   datastores.ProcessStore
    containerStore datastores.ContainerStore
    systemStore    datastores.SystemStore
}

func (d *MyDetector) Init(params detection.DetectorParams) error {
    d.logger = params.Logger

    // Cache store references (registry always returns non-nil stores)
    d.processStore = params.DataStores.Processes()
    d.containerStore = params.DataStores.Containers()
    d.systemStore = params.DataStores.System()

    // Validate required stores are available
    if !params.DataStores.IsAvailable("process") {
        return fmt.Errorf("process store required but not available")
    }

    // Warn about optional stores
    if !params.DataStores.IsAvailable("system") {
        d.logger.Warnw("System store unavailable, some features disabled")
    }

    return nil
}

func (d *MyDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    // Use cached references (no registry lookup)
    proc, err := d.processStore.GetProcess(entityId)
    if errors.Is(err, datastores.ErrNotFound) {
        return nil, nil
    }

    container, err := d.containerStore.GetContainer(containerID)
    if errors.Is(err, datastores.ErrNotFound) {
        return nil, nil
    }

    // System store always available (cached reference)
    sysInfo := d.systemStore.GetSystemInfo()
    // Use system info

    // ...
}
```
{% endraw %}

---

## Writable DataStores

Writable datastores allow detectors to store and share custom data, enabling use cases like:
- **Threat intelligence**: Store IP/domain reputation from external feeds
- **State tracking**: Maintain correlation state across events
- **Cross-detector sharing**: Share enrichment data between detectors

### Quick Start

**Register a writable store** (owner):

{% raw %}
```go
func (d *IPReputationDetector) Init(params detection.DetectorParams) error {
    // Create and configure store
    store := ipreputation.NewIPReputationStore(
        ipreputation.MaxSeverity,   // Conflict resolution policy
        nil,                        // Source priorities (optional)
    )

    // Register - you become the owner
    err := params.DataStores.RegisterWritableStore("ip_reputation", store)
    if err != nil {
        return err
    }

    d.ipRepStore = store
    return nil
}
```
{% endraw %}

**Write to the store**:

{% raw %}
```go
func (d *IPReputationDetector) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    srcIP, _ := v1beta1.GetData[string](event, "src_ip")

    // Write reputation data
    d.ipRepStore.WriteReputation("local_detector", srcIP, &ipreputation.IPReputation{
        IP:          srcIP,
        Status:      ipreputation.ReputationBlacklisted,
        Severity:    8,
        Tags:        []string{"malware"},
        LastUpdated: time.Now(),
    })

    return nil, nil
}
```
{% endraw %}

**Read from the store** (other detectors):

{% raw %}
```go
func (d *ThreatAnalyzer) Init(params detection.DetectorParams) error {
    // Access as read-only
    store, err := params.DataStores.GetCustom("ip_reputation")
    if err != nil {
        return nil  // Optional dependency - degrade gracefully
    }

    d.ipRepStore = store.(*ipreputation.IPReputationStore)
    return nil
}

func (d *ThreatAnalyzer) OnEvent(ctx context.Context, event *v1beta1.Event) ([]detection.DetectorOutput, error) {
    srcIP, _ := v1beta1.GetData[string](event, "src_ip")

    // Query the store
    if d.ipRepStore.IsBlacklisted(srcIP) {
        return []detection.DetectorOutput{{
            Data: []*v1beta1.EventValue{
                v1beta1.NewStringValue("malicious_ip", srcIP),
            },
        }}, nil
    }

    return nil, nil
}
```
{% endraw %}

### Key Concepts

**Ownership Model:**

- The detector that calls `RegisterWritableStore()` owns the store
- Owner has read/write access (keeps concrete type reference)
- Others have read-only access (via `GetCustom()`)
- No authorization code needed - access control via typing

**Source Isolation:**

- All write operations include a `source` parameter (e.g., `"local_detector"`, `"some_feed"`)
- Enables data provenance tracking
- Supports multi-source aggregation with conflict resolution
- Cleanup by source: `Clear("external_feed")`

**Implementation-Specific:**

- Stores define their own behavior (conflict resolution, retention, etc.)
- See `pkg/datastores/ipreputation/` for reference implementation
- Custom stores implement the `WritableStore` interface

---

## Summary

### Quick Reference

| Store | Primary Use | Key Methods |
|-------|-------------|-------------|
| **ProcessStore** | Process info & ancestry | `GetProcess()`, `GetAncestry()`, `GetChildProcesses()` |
| **ContainerStore** | Container metadata | `GetContainer()`, `GetContainerByName()` |
| **SystemStore** | System information | `GetSystemInfo()` |
| **SyscallStore** | Syscall ID/name mapping | `GetSyscallName()`, `GetSyscallID()` |
| **KernelSymbolStore** | Symbol resolution | `ResolveSymbolByAddress()`, `ResolveSymbolsBatch()` |
| **DNSStore** | DNS cache | `GetDNSResponse()` |

### Best Practices

1. **Use EntityID, not PID** for process lookups
2. **Cache store references** in Init() for performance
3. **Handle missing data gracefully** (stores may be unavailable)
4. **Use batch operations** when resolving multiple items
5. **Check health status** before critical operations
6. **Monitor metrics** for performance insights
7. **Fail fast** on critical errors, degrade gracefully on optional failures

### Next Steps

- Read the [Detector API Reference](api-reference.md) for complete detector development
- Start with the [Quick Start Guide](quickstart.md) to write your first detector
- Study real detector implementations in `detectors/` directory

---

**API Version**: v1beta1
**Last Updated**: From implementation commits (60 commits analyzing detector and datastore system)

---

### Docs Detectors Yaml Detectors (`docs/docs/detectors/yaml-detectors.md`)

# YAML Detectors

YAML detectors provide a declarative way to define threat detection and derived events without writing Go code. They offer a simpler, configuration-based approach for common detection patterns.

## Overview

YAML detectors enable you to:

- **Detect threats** by filtering events and generating alerts
- **Derive events** by enriching or transforming existing events
- **Write dynamic conditions** using Common Expression Language (CEL)
- **Extract runtime data** using CEL expressions
- **Auto-populate fields** like threat metadata and process ancestry
- **Reuse policy syntax** for static event filtering
- **Chain detectors** to build complex multi-level detection logic

## Quick Start

Here's a simple YAML detector that identifies execution of networking tools:

```yaml
type: detector
id: yaml-001
produced_event:
  name: suspicious_binary_execution
  version: 1.0.0
  description: Detects execution of networking tools
  tags:
    - execution
    - defense-evasion
  fields:
    - name: binary_path
      type: string
    - name: binary_name
      type: string

requirements:
  min_tracee_version: 0.0.0
  events:
    - name: sched_process_exec
      dependency: required
      data_filters:
        - pathname=/usr/bin/nc
        - pathname=/usr/bin/ncat

threat:
  severity: medium
  description: Execution of networking tool commonly used for reverse shells
  mitre:
    technique:
      id: T1059
      name: Command and Scripting Interpreter
    tactic:
      name: Execution

auto_populate:
  threat: true
  detected_from: true

# Simple field extraction using CEL
output:
  fields:
    - name: binary_path
      expression: getEventData("pathname")
    - name: binary_name
      expression: workload.process.name
```

## Schema Reference

### Top-Level Fields

| Field | Required | Description |
|-------|----------|-------------|
| `type` | Yes | File type: `detector` (for detectors) or `string_list` (for shared lists). Case-insensitive. |
| `id` | Yes | Unique detector identifier (e.g., `yaml-001`) |
| `produced_event` | Yes | Definition of the event this detector produces |
| `requirements` | Yes | Events and conditions required for this detector |
| `threat` | No | Threat metadata (for threat detectors) |
| `auto_populate` | Yes | Fields to auto-populate by the engine |
| `output` | No | Runtime data extraction configuration |

### Produced Event

Defines the event schema that this detector will emit:

```yaml
produced_event:
  name: event_name              # Event name (required)
  version: 1.0.0                # Semantic version (required)
  description: "Description"    # Human-readable description (required)
  tags:                         # Event tags (optional)
    - tag1
    - tag2
  fields:                       # Event data fields (optional)
    - name: field_name
      type: string              # string, int32, int64, uint32, uint64, bool, bytes
```

**Supported field types:**

- `string` - Text data
- `int32` - 32-bit signed integer
- `int64` - 64-bit signed integer
- `uint32` - 32-bit unsigned integer
- `uint64` - 64-bit unsigned integer
- `bool` - Boolean value
- `bytes` - Binary data

### Requirements

Specifies what events and conditions are needed:

```yaml
requirements:
  min_tracee_version: 0.0.0     # Minimum Tracee version (optional)
  architectures:                # Supported architectures (optional)
    - x86_64
    - arm64
  events:                       # Event dependencies (required, at least one)
    - name: event_name
      dependency: required      # required or optional
      min_version: 1.0.0        # Minimum event version (optional)
      max_version: 2.0.0        # Maximum event version (optional)
      data_filters:             # Data filters (optional)
        - pathname=/etc/passwd
        - uid=0
      scope_filters:            # Scope filters (optional)
        - container=true
  enrichments:                  # Required enrichments (optional)
    - name: environment         # Enrichment name
      dependency: required      # required or optional
    - name: executable-hash
      config: digest-inode      # Enrichment-specific config
    - name: container           # Container enrichment
      dependency: required
```

**Event Filters:**

Filters use the same syntax as Tracee policies:

- **Data filters**: Filter by event data fields (e.g., `pathname=/etc/shadow`)
- **Scope filters**: Filter by event scope (e.g., `container=true`)
- **Multiple values**: Comma-separated for OR logic
- **Wildcards**: Prefix (`/tmp*`) or suffix (`*shadow`) matching

Examples:

```yaml
data_filters:
  - pathname=/etc/passwd
  - pathname=/etc/shadow      # OR with previous pathname
scope_filters:
  - container=true
```

### Threat Metadata

For threat detectors, define threat information:

```yaml
threat:
  severity: high                # low, medium, high, critical
  description: "Threat description"
  mitre:
    technique:
      id: T1003               # MITRE ATT&CK technique ID
      name: OS Credential Dumping
    tactic:
      name: Credential Access # MITRE ATT&CK tactic
```

### Auto-Populate

Specify which fields the engine should automatically populate:

```yaml
auto_populate:
  threat: true                  # Auto-populate threat metadata
  detected_from: true           # Auto-populate detection source
  process_ancestry: false       # Auto-populate process tree (expensive)
```

**Fields:**

- `threat`: Populates threat metadata from detector definition
- `detected_from`: Adds detector ID and source event information
- `process_ancestry`: Includes full process tree (performance impact)

### Conditions (CEL Expressions)

YAML detectors support dynamic runtime conditions using Common Expression Language (CEL). Conditions are evaluated after static filters and all must be true for a detection to fire.

```yaml
conditions:
  - hasData("pathname")  # Check if field exists
  - getEventData("pathname").startsWith("/tmp")  # String operations
  - workload.process.uid > 1000  # Numeric comparisons
  - workload.container.id != ""  # Check container context
```

**CEL Capabilities:**

- **Boolean logic**: `&&`, `||`, `!`, ternary `? :`
- **Comparisons**: `==`, `!=`, `<`, `<=`, `>`, `>=`
- **String methods**: `.startsWith()`, `.endsWith()`, `.contains()`, `.matches()` (regex)
- **List operations**: `in`, `.size()`, `.exists()`, `.all()`
- **Type conversions**: Automatic for compatible types

**Helper Functions:**

| Function | Description | Example |
|----------|-------------|---------|
| `getEventData("field")` | Extract data field | `getEventData("pathname")`, `getEventData("fd")` |
| `hasData("field")` | Check if data field exists | `hasData("pathname")` |

**String Utility Functions:**

| Function | Description | Example |
|----------|-------------|---------|
| `split(str, delimiter)` | Split string into list | `split("a,b,c", ",")` → `["a", "b", "c"]` |
| `join(list, delimiter)` | Join list into string | `join(["a", "b"], ",")` → `"a,b"` |
| `trim(str)` | Remove leading/trailing whitespace | `trim("  hello  ")` → `"hello"` |
| `replace(str, old, new)` | Replace all occurrences | `replace("foo bar", "bar", "baz")` → `"foo baz"` |
| `upper(str)` | Convert to uppercase | `upper("hello")` → `"HELLO"` |
| `lower(str)` | Convert to lowercase | `lower("HELLO")` → `"hello"` |
| `basename(path)` | Get filename from path | `basename("/path/to/file.txt")` → `"file.txt"` |
| `dirname(path)` | Get directory from path | `dirname("/path/to/file.txt")` → `"/path/to"` |

**Performance:**

- Conditions are evaluated with 5ms timeout by default
- Failed evaluations are logged and treated as `false`
- CEL programs are compiled once at load time

### Output Data Extraction

Extract runtime values from input events to populate output event fields using CEL expressions.

```yaml
output:
  fields:
    - name: output_field_name     # Name in output event
      expression: getEventData("pathname")  # CEL expression (simplified syntax!)
      optional: false              # Whether field is optional (default: false)
```

**CEL Expression Examples:**

| Use Case | Expression |
|----------|------------|
| Extract data field | `getEventData("pathname")` |
| Extract workload field | `workload.container.id` |
| Conditional extraction | `workload.container.id != "" ? workload.container.id : "unknown"` |
| Extract filename | `basename(getEventData("pathname"))` |
| Extract directory | `dirname(getEventData("pathname"))` |
| Split path components | `split(getEventData("pathname"), "/")` |
| Join path components | `join(["usr", "bin", "nc"], "/")` |
| Normalize case | `lower(workload.process.name)` |
| Replace substring | `replace(getEventData("pathname"), "/tmp", "/var/tmp")` |
| Combine fields | `workload.process.name + ":" + string(workload.process.pid)` |

**Field Semantics:**

- **name**: Required, the output field name
- **expression**: Required, CEL expression to compute the value
- **optional**: If `true`, missing/failed fields are skipped without error

## Working with CEL

Common Expression Language (CEL) is used throughout YAML detectors for conditions and data extraction. This section explains the key concepts and available functions.

### Data Access

#### Event Data Fields

The `getEventData()` function extracts fields from the event's `data` section:

```yaml
output:
  fields:
    - name: binary_path
      expression: getEventData("pathname")
    - name: file_descriptor
      expression: getEventData("fd")
    - name: syscall_return
      expression: getEventData("ret")
```

Use in conditions:

```yaml
conditions:
  - getEventData("pathname").startsWith("/tmp")    # String operations
  - getEventData("fd") > 2                         # Numeric comparisons
  - getEventData("ret") == 0                       # Any type supported
```

#### Workload Context

Access process, container, and Kubernetes information directly through the `workload` variable:

```yaml
conditions:
  - workload.container.id != ""
  - workload.process.pid > 1000
  - workload.process.uid == 0

output:
  fields:
    - name: container_id
      expression: workload.container.id
    - name: pod_name
      expression: workload.kubernetes.pod_name
    - name: process_name
      expression: workload.process.name
```

**Available variables:**

- `workload.process.*` - Process ID, name, uid, gid, etc.
- `workload.container.*` - Container ID, name, image
- `workload.kubernetes.*` - Pod name, namespace, labels
- `timestamp` - Event timestamp

### Complete Example

```yaml
conditions:
  - hasData("pathname")                            # Check field exists
  - getEventData("pathname").startsWith("/tmp")    # Event data
  - workload.process.uid == 0                      # Workload context
  - workload.container.id != ""                    # Container check

output:
  fields:
    - name: binary
      expression: getEventData("pathname")
    - name: uid
      expression: workload.process.uid
    - name: container
      expression: workload.container.id
```

## Shared Lists

Shared lists allow you to define reusable lists of values (e.g., shell binaries, sensitive paths) that multiple detectors can reference. This avoids duplication and makes maintenance easier.

### List Definition Format

Lists are defined in YAML files placed in the **same directory** as your detectors.

Each list file defines a named list:

```yaml
name: SHELL_BINARIES
type: string_list
values:
  - /bin/sh
  - /bin/bash
  - /bin/dash
  - /bin/zsh
  - /usr/bin/sh
  - /usr/bin/bash
```

**Naming convention:** List names must be uppercase snake_case (e.g., `SHELL_BINARIES`, `SENSITIVE_PATHS`).

**Type:** Currently, only `string_list` is supported.

### Using Lists in Detectors

Reference list variables in CEL conditions using the `in` operator:

```yaml
id: yaml-shell-exec
produced_event:
  name: shell_execution_detected
  version: 1.0.0
  description: Detects execution of shell binaries
  tags:
    - execution
  fields:
    - name: shell_path
      type: string

requirements:
  events:
    - name: sched_process_exec

conditions:
  - getEventData("pathname") in SHELL_BINARIES  # Uses shared list

output:
  fields:
    - name: shell_path
      expression: getEventData("pathname")
```

### Complex List Expressions

Lists work with standard CEL operators:

```yaml
conditions:
  # Check membership in multiple lists
  - getEventData("pathname") in SHELL_BINARIES || getEventData("pathname") in SCRIPT_INTERPRETERS

  # Combine with other conditions
  - getEventData("pathname") in SENSITIVE_PATHS && workload.container.id != ""

  # Negate membership
  - !(getEventData("pathname") in ALLOWED_BINARIES)
```

### List Loading Behavior

- Lists are loaded once at startup from the same directory as detectors
- Lists must have `type: string_list` at the top of the file
- Lists are shared across all detectors in the same directory
- Lists are optional - detectors without lists work as before
- Invalid list files prevent all detectors in that directory from loading
- Duplicate list names are not allowed

### Benefits

1. **No duplication**: Define common lists once, use in multiple detectors
2. **Easy maintenance**: Update lists in one place
3. **Zero runtime overhead**: Lists are compiled into the CEL environment at load time
4. **Type safety**: Undefined list references are caught at compile time

## Detector Composition

One of the most powerful features of YAML detectors is the ability to **compose them in chains**. A YAML detector can consume events produced by another YAML detector, enabling layered detection patterns and reusable building blocks.

### How It Works

The detector engine automatically resolves event dependencies. When a detector requires an event that's not a built-in kernel event, the engine checks if another detector produces that event. If found, it creates a subscription chain automatically.

### Example: Two-Level Chain

**Base Detector** (reusable pattern):
```yaml
id: yaml-suspicious-exec
produced_event:
  name: suspicious_binary_execution
  version: 1.0.0
  fields:
    - name: binary_path
      type: string

requirements:
  events:
    - name: sched_process_exec
      data_filters:
        - pathname=/usr/bin/nc
        - pathname=/usr/bin/ncat
        - pathname=/tmp/malware

output:
  fields:
    - name: binary_path
      expression: getEventData("pathname")
```

**Composed Detector** (adds context):
```yaml
id: yaml-container-suspicious-exec
produced_event:
  name: container_suspicious_execution
  version: 1.0.0
  fields:
    - name: binary_path
      type: string
    - name: container_id
      type: string

requirements:
  events:
    - name: suspicious_binary_execution  # ← Consumes base detector
      scope_filters:
        - container=true

threat:
  severity: high
  description: Suspicious binary executed in container

auto_populate:
  threat: true
  detected_from: true

output:
  fields:
    - name: binary_path
      expression: getEventData("binary_path")  # From base detector
    - name: container_id
      expression: workload.container.id
```

### Benefits

- **Reusability**: Share base detectors across teams and organizations
- **Maintainability**: Update base detector, all consumers benefit
- **Modularity**: Each layer has a single responsibility
- **Testing**: Test each layer independently
- **Distribution**: Package and distribute detector libraries

### Use Cases

**Threat Intelligence Integration:**
```yaml
# Community-maintained malware list (base)
id: yaml-known-malware
produced_event:
  name: known_malware_execution
requirements:
  events:
    - name: sched_process_exec
      data_filters:
        - pathname=/known/malware/path1
        - pathname=/known/malware/path2
        # ... hundreds more
```

```yaml
# Organization-specific response (composed)
id: yaml-org-malware-alert
produced_event:
  name: malware_alert
requirements:
  events:
    - name: known_malware_execution  # ← Uses community detector
threat:
  severity: critical
```

**Progressive Refinement:**

You can chain multiple levels (base pattern → container context → production scope):

```
Level 1: cryptominer_execution
    ↓
Level 2: cryptominer_in_container
    ↓
Level 3: cryptominer_production_alert
```

### Best Practices

1. **Design for Reuse**: Make base detectors generic and composable
2. **Single Responsibility**: Each detector should have one clear purpose
3. **Avoid Deep Chains**: Keep chains to 2-3 levels for maintainability
4. **Document Dependencies**: Clearly state what events are consumed
5. **Version Carefully**: Breaking changes in base detectors affect all consumers

## Advanced: Datastore Functions

For complex detections that require querying system state beyond the current event, YAML detectors can use datastore functions in CEL conditions and output expressions. This enables detectors to make decisions based on process ancestry, container metadata, system information, and more.

**Note:** Most detectors won't need these functions. Use static filters in `requirements` when possible for better performance.

### Process Functions

Query process information from Tracee's process datastore.

**`process.get(entityId)`** - Get process information by entity ID

```yaml
conditions:
  # Check if process executable is bash
  - process.get(workload.process.unique_id).exe == "/bin/bash"
  
  # Check process UID
  - process.get(workload.process.unique_id).uid == 0
```

Returns a process object with fields:

- `entity_id` (uint64) - Unique entity ID
- `pid` (uint32) - Process ID
- `ppid` (uint32) - Parent process ID
- `name` (string) - Process name
- `exe` (string) - Executable path
- `start_time` (int64) - Process start timestamp
- `uid` (uint32) - User ID
- `gid` (uint32) - Group ID

Returns `null` if process not found.

**`process.getAncestry(entityId, maxDepth)`** - Get process ancestry chain

```yaml
conditions:
  # Check if any ancestor is a shell
  - process.getAncestry(workload.process.unique_id, 5).exists(p, p.name in SHELL_BINARIES)
  
  # Check if parent process is systemd
  - process.getAncestry(workload.process.unique_id, 2)[1].name == "systemd"
  
  # Verify process depth (count of ancestors)
  - process.getAncestry(workload.process.unique_id, 10).size() < 3
```

Returns a list of process objects, where `[0]` is the process itself, `[1]` is its parent, etc.

**`process.getChildren(entityId)`** - Get child processes

```yaml
conditions:
  # Check if process has spawned children
  - process.getChildren(workload.process.unique_id).size() > 0
  
  # Check if any child is a specific binary
  - process.getChildren(workload.process.unique_id).exists(c, c.exe == "/usr/bin/nc")
```

Returns a list of child process objects.

### Container Functions

Query container information from Tracee's container datastore.

**`container.get(id)`** - Get container by ID

```yaml
conditions:
  # Check container image
  - container.get(workload.container.id).image.startsWith("malicious")
  
  # Check container runtime
  - container.get(workload.container.id).runtime == "docker"
  
  # Check if container has pod metadata
  - container.get(workload.container.id).pod != null
```

Returns a container object with fields:

- `id` (string) - Container ID
- `name` (string) - Container name
- `image` (string) - Container image
- `image_digest` (string) - Image digest (SHA256)
- `runtime` (string) - Container runtime (docker, containerd, cri-o)
- `start_time` (int64) - Container start timestamp
- `pod` (object or null) - Kubernetes pod metadata (if available)
  - `name` (string) - Pod name
  - `uid` (string) - Pod UID
  - `namespace` (string) - Pod namespace
  - `sandbox` (bool) - Whether this is a sandbox container

Returns `null` if container not found.

**`container.getByName(name)`** - Get container by name

```yaml
conditions:
  # Find container by name pattern
  - container.getByName("suspicious-app").image.contains("malware")
```

Returns the same container object as `container.get()`, or `null` if not found.

### System Functions

Access immutable system information collected at Tracee startup.

**`system.info()`** - Get system information

```yaml
conditions:
  # Check architecture
  - system.info().architecture == "x86_64"
  
  # Check kernel version
  - system.info().kernel_release.startsWith("5.")
  
  # Check OS
  - system.info().os_name == "Ubuntu" && system.info().os_version.startsWith("22.")
  
  # Check hostname
  - system.info().hostname.contains("prod")
```

Returns a system info object with fields:

- `architecture` (string) - System architecture (x86_64, arm64, etc.)
- `kernel_release` (string) - Kernel version (e.g., "5.15.0-91-generic")
- `hostname` (string) - System hostname
- `boot_time` (int64) - System boot timestamp
- `tracee_start_time` (int64) - Tracee start timestamp
- `os_name` (string) - OS name (e.g., "Ubuntu")
- `os_version` (string) - OS version (e.g., "22.04")
- `os_pretty_name` (string) - Human-readable OS name
- `tracee_version` (string) - Tracee version

Always returns a valid object (never `null`).

### Kernel Symbol Functions

Resolve kernel addresses and symbol names.

**`kernel.resolveSymbol(address)`** - Resolve address to symbol

```yaml
conditions:
  # Check if address resolves to a known function
  - kernel.resolveSymbol(getEventData("addr")).exists(s, s.name == "sys_execve")
```

Returns a list of symbol objects (multiple if aliases exist):

- `name` (string) - Symbol name
- `address` (uint64) - Symbol address
- `module` (string) - Module name (e.g., "vmlinux")

Returns empty list if address cannot be resolved.

**`kernel.getSymbolAddress(name)`** - Get symbol address

```yaml
conditions:
  # Check if hooked address differs from expected
  - getEventData("hooked_addr") != kernel.getSymbolAddress("sys_execve")
  
  # Verify symbol exists
  - kernel.getSymbolAddress("sys_read") > 0u
```

Returns the symbol address as `uint64`, or `0` if not found.

### DNS Functions

Query cached DNS responses.

**`dns.getResponse(query)`** - Get cached DNS response

```yaml
conditions:
  # Check if domain resolves to suspicious IP
  - dns.getResponse(getEventData("domain")).ips.exists(ip, ip.startsWith("192.168."))
  
  # Check number of resolved IPs
  - dns.getResponse("example.com").ips.size() > 10
```

Returns a DNS response object:

- `query` (string) - Original DNS query
- `ips` (list of strings) - Resolved IP addresses
- `domains` (list of strings) - CNAME chain

Returns `null` if no cached response found.

### Syscall Functions

Map between syscall IDs and names (architecture-specific).

**`syscall.getName(id)`** - Get syscall name from ID

```yaml
conditions:
  # Check if syscall ID is execve
  - syscall.getName(getEventData("syscall_id")) == "execve"
  
  # Check for specific syscall
  - syscall.getName(59) in ["execve", "execveat"]
```

Returns the syscall name as a string, or empty string `""` if not found.

**`syscall.getId(name)`** - Get syscall ID from name

```yaml
conditions:
  # Check if event is for specific syscalls
  - getEventData("syscall_id") == syscall.getId("execve") || 
    getEventData("syscall_id") == syscall.getId("execveat")
```

Returns the syscall ID as `int`, or `-1` if not found.

### Return Values and Error Handling

**Null handling:**

- Functions return `null` when an entity is not found (safe for conditions)
- Non-existent datastores return `null` (graceful degradation)
- Use null-safe checks: `container.get(id) != null` or `container.get(id).image`

**Error propagation:**

- Unexpected errors (not "not found") cause condition evaluation to fail
- The detector will log the error and skip the event

**Performance considerations:**

- Datastore lookups add latency (typically <1ms)
- Use judiciously in high-frequency events
- Prefer static filters (in `requirements`) over dynamic lookups when possible

### Datastore Examples

**Example 1: Detect reverse shell from containerized bash**

```yaml
id: yaml-container-reverse-shell
produced_event:
  name: container_reverse_shell
  version: 1.0.0
  description: Reverse shell spawned from container bash process

requirements:
  events:
    - name: security_socket_connect
      scope_filters:
        - container=true

conditions:
  # Check if process ancestry includes bash
  - process.getAncestry(workload.process.unique_id, 5).exists(p, 
      p.name == "bash" || p.name == "sh")
  
  # Check if container image is not trusted
  - container.get(workload.container.id).image.startsWith("suspicious/")

output:
  fields:
    - name: container_image
      expression: container.get(workload.container.id).image
    - name: shell_exe
      expression: process.getAncestry(workload.process.unique_id, 5).filter(p, 
          p.name in ["bash", "sh"])[0].exe
```

**Example 2: Detect privilege escalation**

```yaml
id: yaml-privilege-escalation
produced_event:
  name: privilege_escalation_detected
  version: 1.0.0
  description: Process escalated privileges from non-root to root

requirements:
  events:
    - name: setuid
      data_filters:
        - uid=0

conditions:
  # Check if parent was non-root
  - process.get(workload.process.unique_id).uid != 0
  
  # Check if not running as expected system process
  - !process.getAncestry(workload.process.unique_id, 3).exists(p, 
      p.name == "systemd" || p.name == "init")
```

**Example 3: Detect kernel rootkit**

```yaml
id: yaml-kernel-rootkit
produced_event:
  name: kernel_rootkit_detected
  version: 1.0.0
  description: Syscall table hooking detected

requirements:
  events:
    - name: hooked_syscalls

conditions:
  # Check if hooked address doesn't match expected symbol
  - getEventData("hooked_addr") != kernel.getSymbolAddress(getEventData("syscall_name"))
  
  # Verify it's a critical syscall
  - getEventData("syscall_name") in ["sys_read", "sys_write", "sys_open", "sys_execve"]
```

## Deployment

### Directory Structure

Detectors and lists can coexist in the same flat directory:

```
detectors/
  ├── suspicious_exec.yaml        # type: detector
  ├── hidden_file.yaml            # type: detector
  ├── shell_binaries.list.yaml    # type: string_list
  └── suspicious_ports.list.yaml  # type: string_list
```

**File Requirements:**

- All files must include a `type` field: `detector` or `string_list`
- The `type` field is case-insensitive
- Only the top-level directory is scanned - subdirectories are ignored
- List files should use `.list.yaml` suffix for clarity (optional but recommended)

### Default Search Path

Tracee automatically loads YAML detectors from `/etc/tracee/detectors/`.

### Custom Paths

Specify custom directories using:

**CLI Flag:**
```bash
tracee --detectors /custom/path
```

**Config File:**
```yaml
detectors:
  - /custom/path1
  - /custom/path2
```

## Validation

YAML detectors are validated at load time:

- **Schema validation**: Ensures all required fields are present
- **Type checking**: Validates field types match schema
- **Filter syntax**: Checks filter expressions are valid
- **Version constraints**: Validates semantic version format
- **Field extraction**: Verifies expression paths are supported

**Error Handling:**

Invalid detectors are logged as warnings and skipped. Tracee continues loading valid detectors.

## Best Practices

### 1. Use Consistent ID Convention

Choose a naming convention and stick to it:

```yaml
# Good - consistent conventions
id: yaml-001           # Simple numeric
id: TRC-YAML-001       # Prefixed with TRC
id: DRV-YAML-001       # Prefixed for derived events

# Bad - inconsistent
id: my-detector-1
id: TRC-002
id: yaml-ssh-detector  # Unnecessarily descriptive (use produced_event.name instead)
```

**Note:** IDs should be unique and stable. Use `produced_event.name` and `description` for descriptive information.

### 2. Provide Clear Descriptions

```yaml
produced_event:
  name: ssh_brute_force_attempt
  description: Multiple failed SSH authentication attempts from same source
```

### 3. Use Specific Filters

```yaml
# Good - specific paths
data_filters:
  - pathname=/etc/passwd
  - pathname=/etc/shadow

# Bad - too broad
data_filters:
  - pathname=/etc*
```

### 4. Tag Appropriately

Use consistent tags for categorization:

```yaml
tags:
  - credential-access    # MITRE tactic
  - brute-force         # Technique
  - ssh                 # Technology
```

### 5. Extract Relevant Data

Only extract fields that provide investigative value:

```yaml
output:
  fields:
    - name: source_ip      # Useful for investigation
      expression: getEventData("src_ip")
    - name: target_user    # Useful for investigation
      expression: getEventData("username")
```

### 6. Set Appropriate Severity

Match severity to actual threat level:

- `low`: Informational, may be benign
- `medium`: Suspicious, requires investigation
- `high`: Likely malicious, immediate attention
- `critical`: Active attack, urgent response

### 7. Version Your Detectors

Use semantic versioning for detector evolution:

```yaml
produced_event:
  version: 1.0.0    # Initial release
  version: 1.1.0    # Added new field
  version: 2.0.0    # Breaking change
```

## Limitations

Current limitations of YAML detectors:

- **No state management**: Cannot track state across events (use Go detectors)
- **No complex logic**: Cannot implement conditional branching or loops
- **No custom types**: Limited to basic protobuf types
- **No hot reload**: Requires Tracee restart to load new/updated detectors

For advanced use cases requiring these features, use [Go detectors](quickstart.md).

## Troubleshooting

### Detector Not Loading

Check Tracee logs for validation errors:

```bash
tracee --log debug
```

Common issues:

- Invalid YAML syntax
- Missing required fields
- Invalid filter expressions
- Unsupported field types

### No Events Generated

Verify:

1. Input event is being generated: `tracee --events event_name`
2. Filters are correct: Test with broader filters
3. Event is selected in policy: Check policy configuration

### Incorrect Data Extraction

- Verify expression path exists in input event
- Check field type matches extracted data
- Use `optional: true` for fields that may not exist

## See Also

- [Quick Start Guide](quickstart.md) - First Go detector tutorial
- [API Reference](api-reference.md) - Complete Go detector API
- [DataStore API](datastore-api.md) - Accessing system state from Go detectors
- [Policy Guide](../policies/index.md) - Event filtering syntax
- [Events Reference](../events/index.md) - Available events
- [Example Detectors](https://github.com/aquasecurity/tracee/tree/main/examples/detectors/yaml) - YAML detector examples

---

### Docs Flags Artifacts.1 (`docs/docs/flags/artifacts.1.md`)

---
title: TRACEE-ARTIFACTS
section: 1
header: Tracee Artifacts Flag Manual
date: 2026/04
...

## NAME

tracee **\-\-artifacts** - Capture artifacts that were written, executed, or found to be suspicious

## SYNOPSIS

tracee **\-\-artifacts** <artifact-option[=value]\> ...

## DESCRIPTION

The **\-\-artifacts** flag allows you to capture artifacts that were written, executed, or found to be suspicious during the execution of Tracee. The captured artifacts will appear in the 'output-path' directory.

Possible artifacts options:

- **file-write**: Capture written files. **Note**: Using `file-write.filters` automatically enables file-write, so you don't need to also specify `--artifacts file-write`.
- **file-write.filters=\<filter\>**: Filter for file writes (multiple allowed). Format: \<filter_type\>=\<filter_value\>
- **file-read**: Capture read files. **Note**: Using `file-read.filters` automatically enables file-read, so you don't need to also specify `--artifacts file-read`.
- **file-read.filters=\<filter\>**: Filter for file reads (multiple allowed). Format: \<filter_type\>=\<filter_value\>
- **executable**: Capture executed files.
- **kernel-modules**: Capture loaded kernel modules.
- **bpf-programs**: Capture loaded BPF programs bytecode.
- **memory-regions**: Capture memory regions that had write+execute (w+x) protection and then changed to execute (x) only.
- **network**: Capture network traffic. Only TCP/UDP/ICMP protocols are currently supported. **Note**: Using `network.pcap.split`, `network.pcap.options`, or `network.pcap.snaplen` automatically enables network, so you don't need to also specify `--artifacts network`.
- **network.pcap.split=\<split_mode\>**: Capture separate pcap files organized by split mode: single, process, container, command (comma-separated).
- **network.pcap.options=\<option\>**: Network capturing options: none (default) or filtered.
- **network.pcap.snaplen=\<size\>**: Sets captured payload from each packet: default, headers, max, or SIZE (e.g., 256b, 512b, 1kb, 2kb, 4kb).
- **dir.path=\<path\>**: Path where tracee will save produced artifacts. The artifact will be saved into an 'out' subdirectory (default: /var/lib/tracee).
- **dir.clear**: Clear the captured artifacts output dir before starting (default: false).

### File Capture Filters

Files captured upon read/write can be filtered to catch only specific IO operations. The different filter types have a logical 'AND' between them but a logical 'OR' between filters of the same type. The filter format is as follows: \<filter_type\>=\<filter_value\>

Filter types:

- **path**: A filter for the file path prefix (up to 50 characters). Up to 3 filters can be given. Format: path=/path/prefix\*
- **type**: A file type from the following options: 'regular', 'pipe', 'socket' and 'elf'. Format: type=regular
- **fd**: The file descriptor of the file. Can be one of the three standards: 'stdin', 'stdout' and 'stderr'. Format: fd=stdout

### Network Capture Notes

- Pcap Files:
  - If you only specify **\-\-artifacts network**, you will have a single file with all network traffic.
  - You can use **network.pcap.split=xxx,yyy** to have more than one pcap file, split by different means.

- Pcap Options:
  - If you do not specify **network.pcap.options** (or set to none), you will capture ALL network traffic into your pcap files.
  - If you specify **network.pcap.options=filtered**, events being traced will define what network traffic will be captured.

- Snap Length:
  - If you do not specify a snaplen, the default is headers only (incomplete packets in tcpdump).
  - If you specify **max** as snaplen, you will get full packets contents (pcap files will be large).
  - If you specify **headers** as snaplen, you will only get L2/L3 headers in captured packets.
  - If you specify **headers** but trace for **net_packet_dns** events, L4 DNS header will be captured.
  - If you specify **headers** but trace for **net_packet_http** events, only L2/L3 headers will be captured.

## EXAMPLES

### File capture

- To capture executed files into the default output directory, use the following flag:

  ```console
  --artifacts executable
  ```

- To capture executed files into a specific directory, clear the directory before starting, use the following flags:

  ```console
  --artifacts executable --artifacts dir.path=/my/dir --artifacts dir.clear
  ```

- To capture files that were written into anywhere under /usr/bin/ or /etc/, use the following flags:

  ```console
  --artifacts file-write.filters=path=/usr/bin/* --artifacts file-write.filters=path=/etc/*
  ```
  Note: `file-write.filters` automatically enables file-write, so `--artifacts file-write` is not needed.

- To capture file writes to socket files that are the 'stdout' of the writing process, use the following flags:

  ```console
  --artifacts file-write.filters=type=socket --artifacts file-write.filters=fd=stdout
  ```
  Note: `file-write.filters` automatically enables file-write, so `--artifacts file-write` is not needed.

### Network Capture

- To capture network traffic, use the following flag:

  ```console
  --artifacts network
  ```

- To capture network traffic and save separate pcap files for processes and commands, use the following flags:

  ```console
  --artifacts network.pcap.split=process,command
  ```
  Note: `network.pcap.split` automatically enables network, so `--artifacts network` is not needed.

- To capture network traffic and save pcap files containing only traced/filtered packets, use the following flags:

  ```console
  --artifacts network.pcap.options=filtered
  ```
  Note: `network.pcap.options` automatically enables network, so `--artifacts network` is not needed.

- To capture network traffic and set the captured payload from each packet to 1KB, use the following flags:

  ```console
  --artifacts network.pcap.snaplen=1kb
  ```
  Note: `network.pcap.snaplen` automatically enables network, so `--artifacts network` is not needed.

- To capture network traffic and save pcap files for containers and commands, use the following flags:

  ```console
  --artifacts network.pcap.split=container,command
  ```
  Note: `network.pcap.split` automatically enables network, so `--artifacts network` is not needed.

---

### Docs Flags Buffers.1 (`docs/docs/flags/buffers.1.md`)

---
title: TRACEE-BUFFERS
section: 1
header: Tracee Buffers Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-buffers** - Set kernel/user buffers size

## SYNOPSIS

tracee **\-\-buffers** [kernel.events=<size\> | kernel.artifacts=<size\> | kernel.control-plane=<size\> | pipeline=<size\>] ... [**\-\-buffers** [kernel.events=<size\> | kernel.artifacts=<size\> | kernel.control-plane=<size\> | pipeline=<size\>] ...]

## DESCRIPTION

The **\-\-buffers** flag allows you to configure the sizes of the buffers used by tracee.

Buffer sizes for perf ring buffers (kernel.events, kernel.artifacts, kernel.control-plane) are specified in pages. The pipeline buffer size is specified in event objects. The default size for perf ring buffers is 1024 pages. The default size for the pipeline buffer is 1000 event objects.

Possible buffer options:

- **kernel.events=<size\>**: Sets the size, in pages, of the internal perf ring buffer used to submit events from the kernel.

- **kernel.artifacts=<size\>**: Sets the size, in pages, of the internal perf ring buffer used to send artifacts from the kernel.

- **kernel.control-plane=<size\>**: Sets the size, in pages, of the internal perf ring buffer used to submit events from the control plane.

- **pipeline=<size\>**: Sets the size, in event objects, of each pipeline stage's output channel.

## EXAMPLES

- To set the kernel events buffer size to 2048 pages, use the following flag:

  ```console
  --buffers kernel.events=2048
  ```

- To set multiple buffer sizes at once, use multiple flags:

  ```console
  --buffers kernel.events=2048 --buffers pipeline=20000
  ```

- To set all buffer sizes in a single command, use the following flags:

  ```console
  --buffers kernel.events=2048 --buffers kernel.artifacts=1024 --buffers kernel.control-plane=512 --buffers pipeline=20000
  ```

- To only override the kernel.artifacts buffer size while keeping other buffers at their default values, use the following flag:

  ```console
  --buffers kernel.artifacts=2048
  ```

---

### Docs Flags Capabilities.1 (`docs/docs/flags/capabilities.1.md`)

---
title: TRACEE-CAPABILITIES
section: 1
header: Tracee Capabilities Flag Manual
date: 2024/11
...

## NAME

tracee **\-\-capabilities** - Opt out from dropping capabilities by default or set specific ones

## SYNOPSIS

tracee **\-\-capabilities** [<bypass=[true|false]\> | <add=cap1(,cap2...)\> | <drop=cap1(,cap2...)\>] ... [**\-\-capabilities** [<add=cap1(,cap2...)\> | <drop=cap1(,cap2...)\>] ...]

## DESCRIPTION

The **\-\-capabilities** flag allows you to control the dropping of capabilities during execution time or set specific capabilities.

Possible options:

- **bypass=[true|false]**: Keep all capabilities during execution time. Setting **bypass=true** will opt out from dropping any capabilities.
- **add=cap1(,cap2...)**: Add specific capabilities to the "required" capabilities ring. You can provide multiple capability names separated by commas.
- **drop=cap1(,cap2...)**: Drop specific capabilities from the "required" capabilities ring. You can specify multiple capability names separated by commas.

Please note that the available capabilities will depend on the running system. For the list of capabilities available on your system, see the **list-caps** command.

## EXAMPLES

- To keep all capabilities during execution time, use the following flag:

  ```console
  --capabilities bypass=true
  ```

- To add specific capabilities (e.g., cap_kill and cap_syslog) to the "required" capabilities ring, use the following flag:

  ```console
  --capabilities add=cap_kill,cap_syslog
  ```

- To drop a specific capability (e.g., cap_chown) from the "required" capabilities ring, use the following flag:

  ```console
  --capabilities drop=cap_chown
  ```

Please refer to the [documentation](../install/capabilities.md) for more information on environment capabilities.

---

### Docs Flags Config.1 (`docs/docs/flags/config.1.md`)

---
title: TRACEE-CONFIG
section: 1
header: Tracee Config Flag Manual
date: 2026/05
...

## NAME

tracee **\-\-config** - Define global configuration options for tracee

## SYNOPSIS

tracee **\-\-config** <file\>

## DESCRIPTION

The **\-\-config** flag allows you to specify global configuration options for Tracee by providing a configuration file.

The configuration file supports structured (nested) YAML under each top-level key, or a list of CLI-style flag strings under that same key. Dotted keys at the root of the file (for example `server.http-address`) are not loaded; each subsystem must appear as its own top-level key (`server`, `output`, `logging`, and so on).

## FILE FORMAT

The configuration file supports multiple formats:

- **YAML**: Recommended format with clear hierarchy and comments support
- **JSON**: Standard JSON format for programmatic generation

## USAGE

To use the **\-\-config** flag, you need to provide the path to the configuration file:

```console
tracee --config /path/to/tracee-config.yaml
```

## EXAMPLES

### Nested (Structured) Format

Example configuration file using nested YAML structure:

```yaml
output:
  destinations:
    - name: stdout_destination
      type: file
      format: json
      path: stdout

logging:
  level: info

enrichment:
  container:
    enabled: true

buffers:
  kernel:
    events: 2048
    artifacts: 1024
  pipeline: 10000

server:
  http-address: ":3366"
  metrics: true
  healthz: true
```

### CLI-Style (List) Format

The same configuration using CLI-style lists under each top-level key. Each entry is one flag string, like on the command line.

```yaml
output:
  - destinations.stdout_destination.type=file
  - destinations.stdout_destination.format=json
  - destinations.stdout_destination.path=stdout

logging:
  - level=info

enrichment:
  - container

buffers:
  - kernel.events=2048
  - kernel.artifacts=1024
  - pipeline=10000

server:
  - "http-address=:3366"
  - metrics
  - healthz
```

For a complete example configuration file with all available options, see:
https://github.com/aquasecurity/tracee/blob/main/examples/config/global_config.yaml

## NOTES

The following options are **not** supported in the configuration file and must be
provided exclusively via the CLI:

- **\-\-config**: Path to the config file itself.
- **\-\-policy**: Policy file or directory paths.
- **\-\-scope**: Scope filters.
- **\-\-events**: Event filters.

Policies can also be supplied through Kubernetes CRDs when running in a cluster.

## SEE ALSO

For more detailed information about configuration:

- **Configuration Guide**: See the [Configuration Overview](../install/config/index.md)
- **Kubernetes Configuration**: See the [Kubernetes Configuration Guide](../install/config/kubernetes.md)

---

### Docs Flags Detectors.1 (`docs/docs/flags/detectors.1.md`)

---
title: TRACEE-DETECTORS
section: 1
header: Tracee Detectors Flag Manual
date: 2026/01
...

## NAME

tracee **\-\-detectors** - Configure YAML detector search directories

## SYNOPSIS

tracee **\-\-detectors** [path...] [**\-\-detectors** path...]

## DESCRIPTION

The **\-\-detectors** flag lets you add directories or files to search for YAML detectors and shared lists.

Each path can be a directory or a YAML file. If not specified, Tracee uses the default search path `/etc/tracee/detectors`.

## EXAMPLES

1. Use the default search path:
   ```console
   tracee
   ```

2. Add a custom directory:
   ```console
   --detectors /custom/detectors
   ```

3. Add multiple directories:
   ```console
   --detectors /dir1 --detectors /dir2
   ```

4. Add a specific YAML detector file:
   ```console
   --detectors ./detectors/suspicious_exec.yaml
   ```

5. Config file format:
   ```yaml
   detectors:
     - /custom/path1
     - /custom/path2
   ```

---

### Docs Flags Enrichment.1 (`docs/docs/flags/enrichment.1.md`)

---
title: TRACEE-ENRICHMENT
section: 1
header: Tracee Enrichment Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-enrichment** - Configure enrichment for container events and other enrichment options

## SYNOPSIS

tracee **\-\-enrichment** [container|container.cgroupfs.path=*path*|container.cgroupfs.force|container.docker.socket=*socket_path*|container.containerd.socket=*socket_path*|container.crio.socket=*socket_path*|container.podman.socket=*socket_path*|fd-paths|environment|executable-hash|executable-hash.mode=*mode*|user-stack|decoded-data] [**\-\-enrichment** ...]

## DESCRIPTION

The `--enrichment` flag allows you to configure enrichment options for container events and other enrichment features.

### Flags

- **container**: Enable container enrichment with default settings. When enabled, Tracee will enrich container events with container information.

- **container.cgroupfs.path**=*path*: Enable container enrichment and configure the path to the cgroupfs where container cgroups are created. This is used as a hint for auto-detection. **Note**: Using this option automatically enables container, so you don't need to also specify `--enrichment container`.
  Example:
  ```console
  --enrichment container.cgroupfs.path=/sys/fs/cgroup
  ```

- **container.cgroupfs.force**: Force the usage of the provided mountpoint path, skipping auto-detection. **Note**: This option requires `container.cgroupfs.path` to be set. It cannot be used alone.
  Example:
  ```console
  --enrichment container.cgroupfs.path=/sys/fs/cgroup --enrichment container.cgroupfs.force
  ```

- **container.docker.socket**=*socket_path*: Enable container enrichment and configure container runtime sockets for enrichment. Configure the path to the Docker socket. **Note**: Using this option automatically enables container, so you don't need to also specify `--enrichment container`.
  Example:
  ```console
  --enrichment container.docker.socket=/var/run/docker.sock
  ```

- **container.containerd.socket**=*socket_path*: Enable container enrichment and configure container runtime sockets for enrichment. Configure the path to the Containerd socket. **Note**: Using this option automatically enables container, so you don't need to also specify `--enrichment container`.
  Example:
  ```console
  --enrichment container.containerd.socket=/var/run/containerd/containerd.sock
  ```

- **container.crio.socket**=*socket_path*: Enable container enrichment and configure container runtime sockets for enrichment. Configure the path to the CRI-O socket. **Note**: Using this option automatically enables container, so you don't need to also specify `--enrichment container`.
  Example:
  ```console
  --enrichment container.crio.socket=/var/run/crio/crio.sock
  ```

- **container.podman.socket**=*socket_path*: Enable container enrichment and configure container runtime sockets for enrichment. Configure the path to the Podman socket. **Note**: Using this option automatically enables container, so you don't need to also specify `--enrichment container`.
  Example:
  ```console
  --enrichment container.podman.socket=/var/run/podman/podman.sock
  ```

  Supported container runtimes for socket configuration:
  - CRI-O      (`crio`, `cri-o`)
  - Containerd (`containerd`)
  - Docker     (`docker`)
  - Podman     (`podman`)

- **fd-paths**: Enable fd-paths. When enabled, Tracee will resolve file descriptor arguments to show associated file paths instead of just the descriptor number. This enriches file descriptors with file path translation. May cause pipeline slowdowns.
  Example:
  ```console
  --enrichment fd-paths
  ```

- **decoded-data**: Enable decoded-data. When enabled, Tracee will decode event arguments into human-readable strings instead of raw machine-readable values. This converts numeric flags, permissions, syscall types, and other raw values into readable format (e.g., `O_RDONLY` instead of `0`, `PROT_READ` instead of `1`). Recommended for interactive use and readability, but may add processing overhead that impacts performance on high-volume event streams.
  Example:
  ```console
  --enrichment decoded-data
  ```

- **environment**: Enable environment. When enabled, Tracee will include execution environment variables in process execution events (particularly useful for `execve` events).
  Example:
  ```console
  --enrichment environment
  ```

- **executable-hash**: Enable executable-hash with default settings. When enabled, Tracee will compute hash values for executed binaries.

- **executable-hash.mode**=*mode*: Enable executable-hash and configure the mode for executable-hash. **Note**: Using this option automatically enables executable-hash, so you don't need to also specify `--enrichment executable-hash`.
  Example:
  ```console
  --enrichment executable-hash.mode=sha256
  ```

- **user-stack**
  Enable user-stack. Presence of the flag enables it, absence disables it.
  Example:
  ```console
  --enrichment user-stack
  ```

## EXAMPLES

1. Enable container enrichment:
   ```console
   --enrichment container
   ```

2. Configure Docker socket:
   ```console
   --enrichment container.docker.socket=/var/run/docker.sock
   ```
   Note: `container.docker.socket` automatically enables container, so `--enrichment container` is not needed.

3. Set the cgroupfs path:
   ```console
   --enrichment container.cgroupfs.path=/sys/fs/cgroup
   ```
   Note: `container.cgroupfs.path` automatically enables container, so `--enrichment container` is not needed.

4. Combine multiple flags:
   ```console
   --enrichment container.docker.socket=/var/run/docker.sock --enrichment container.cgroupfs.path=/sys/fs/cgroup
   ```
   Note: Since `container.docker.socket` and `container.cgroupfs.path` automatically enable container, you don't need `--enrichment container`.

5. Enable fd-paths, environment, and executable-hash:
   ```console
   --enrichment fd-paths --enrichment environment --enrichment executable-hash
   ```

6. Enable executable-hash with custom mode:
   ```console
   --enrichment executable-hash.mode=sha256
   ```
   Note: `executable-hash.mode` automatically enables executable-hash, so `--enrichment executable-hash` is not needed.

Please refer to the [documentation](../install/container-engines.md) for more information on container events enrichment.

---

### Docs Flags Events.1 (`docs/docs/flags/events.1.md`)

--
title: TRACEE-EVENTS
section: 1
header: Tracee Events Flag Manual
date: 2024/12
...

## NAME

tracee **\-\-events** - Select which events to trace

## SYNOPSIS

tracee **\-\-events** [<event-name1(,[-]event-name2...)\> | <tag=tag1(,tag2...)\> | <event1.data.data-field[=|!=]value\> | <event1.retval[=|!=|<|\>|<=|\>=]value\> | <event1.scope.field[=|!=|<|\>|<=|\>=]value\> | <event.scope.container\>] ...

## DESCRIPTION

The **\-\-events** flag allows you to select which events to trace by defining filters.

## FILTERS

- Event name: Select specific events using 'event-name1,event-name2...'. To exclude events, prepend the event name with a dash '-': '-event-name'.

- Tag selection: Select events by tag (set) using 'tag=tag1,tag2...'. Multiple tags are combined with OR logic. Common tags include: syscalls, fs, network, proc, default.

- Detector selection by threat properties: Select detectors based on their threat metadata using 'threat.property=value'. See THREAT-BASED DETECTOR SELECTION section below.

- Event data: Filter events based on their data using 'event-name.data.event_data'. The event data expression follows the syntax of a string expression.

- Event return value: Filter events based on their return value using 'event-name.retval'. The event return value expression follows the syntax of a numerical expression.

- Event scope fields: Filter events based on the non-argument fields defined in the trace.Event struct using 'event-name.scope.field'. Refer to the json tags in the trace.Event struct located in the types/trace package for the correct field names, and the event filtering section in the documentation for a full list.

## FILTER EXPRESSION

Filter expressions can be defined to operate on event options or process metadata. Only events that match all filter expressions will be traced.

Multiple flags are combined with AND logic, while multiple values within a single flag are combined with OR logic when using the equals operator '='. Multiple values can be specified using ','.

### NUMERICAL EXPRESSION OPERATORS

'=', '!=', '<', '\>', '<=', '\>='

Available for:

- return value
- scope fields

NOTE: Expressions containing '<' or '\>' tokens must be escaped!

### STRING EXPRESSION OPERATORS

'=', '!='

Available for:

- event arguments
- return value
- scope fields

Strings can be compared as a prefix if ending with '\*', or as a suffix if starting with '\*'.  If a string starts with '\*' and ends with '\*', it functions as a contains operator.

For certain event fields filtered in kernel space, the user will receive a warning if:

- String filters exceed 255 characters.
- The contains operator is used. Only exact matches, prefix, and suffix comparisons are allowed.

NOTE: Expressions containing '\*' token must be escaped!

### EXCLUSION OPERATOR (PREPENDED)

'-'

Available only for:

- event names

## THREAT-BASED DETECTOR SELECTION

Detectors can be selected based on their threat metadata properties. This allows you to enable all detectors that match specific security criteria without knowing their individual event names.

### THREAT PROPERTIES

**threat.severity** - Severity level (info, low, medium, high, critical or 0-4). Supports: =, !=, <, \>, <=, \>=

```console
--events threat.severity=critical                # Only critical threats
--events 'threat.severity>=high'                 # High and critical threats
```

**threat.mitre.technique** - MITRE ATT&CK technique ID (e.g., T1055). Supports: =, !=

```console
--events threat.mitre.technique=T1055
```

**threat.mitre.tactic** - MITRE ATT&CK tactic name (e.g., "Defense Evasion"). Supports: =, !=

```console
--events 'threat.mitre.tactic="Defense Evasion"'
```

**threat.name** - Threat name/identifier. Supports: =, !=

```console
--events threat.name=process_injection
```

Threat selection can be combined with regular events. Multiple `--events` flags are combined additively (OR logic):

```console
--events threat.severity=critical --events write    # Critical threats OR write events
--events tag=fs --events 'threat.severity>=high'    # Filesystem events OR high+ threats
```

**Note:** Detector selection based on threat properties is performed once at startup. Matching detectors are enabled; non-matching detectors are never loaded.

## TAG SELECTION

Events are categorized with tags (also known as sets). You can select all events with a specific tag:

```console
--events tag=containers       # All events tagged with "containers"
--events tag=fs               # All filesystem-related events
--events tag=detectors        # All detector events
```

Tags can be combined with other selection methods:

```console
--events tag=containers,execve            # Events with "containers" tag + execve syscall
--events tag=fs,network                   # Filesystem OR network events
--events tag=fs --events -open,-openat    # Filesystem events except open(at)
```

## EXAMPLES

- To trace only 'execve' and 'open' events, use the following flag:

  ```console
  --events execve,open
  ```

- To trace only events prefixed by "open", use the following flag:

  ```console
  --events 'open*'
  ```

- To trace all file-system related events, use the following flag:

  ```console
  --events tag=fs
  ```

- To trace all file-system related events, but not 'open' or 'openat', use the following flag:

  ```console
  --events tag=fs --events '-open,-openat'
  ```

- To trace only 'close' events that have 'fd' equal to 5, use the following flag:

  ```console
  --events close.data.fd=5
  ```

- To trace only 'openat' events that have 'pathname' prefixed by '/tmp', use the following flag:

  ```console
  --events openat.data.pathname='/tmp*'
  ```

- To trace only 'openat' events that have 'pathname' suffixed by 'shadow', use the following flag:

  ```console
  --events openat.data.pathname='*shadow'
  ```

- To trace only 'openat' events that have 'pathname' contains the substring 'pass', use the following flag:

  ```console
  --events openat.data.pathname='*pass*'
  ```

- To exclude 'openat' events that have 'pathname' equal to '/tmp/1' or '/bin/ls', use the following flag:

  ```console
  --events openat.data.pathname!=/tmp/1,/bin/ls
  ```

- To trace only 'openat' events that have 'processName' equal to 'ls', use the following flag:

  ```console
  --events openat.scope.processName=ls
  ```

- To trace only 'security_file_open' events coming from a container, use the following flag:

  ```console
  --events security_file_open.scope.container
  ```

- To trace all detectors with critical severity, use the following flag:

  ```console
  --events threat.severity=critical
  ```

- To trace all detectors with high or critical severity, use the following flag:

  ```console
  --events 'threat.severity>=high'
  ```

- To trace detectors for a specific MITRE ATT&CK technique, use the following flag:

  ```console
  --events threat.mitre.technique=T1055
  ```

- To trace detectors for a specific MITRE ATT&CK tactic, use the following flag:

  ```console
  --events 'threat.mitre.tactic="Defense Evasion"'
  ```

- To combine threat-based selection with regular events, use the following flags:

  ```console
  --events write --events threat.severity=critical
  ```

---

### Docs Flags List Detectors.1 (`docs/docs/flags/list-detectors.1.md`)

--
title: TRACEE-LIST-DETECTORS
section: 1
header: Tracee List Detectors Command Manual
date: 2026/05
...

## NAME

tracee **list detectors** - List available detectors and shared lists

## SYNOPSIS

tracee **list detectors** [paths...] [\-\-json]

## DESCRIPTION

The **list detectors** command displays all available detectors and shared lists from built-in sources and YAML files.

Detectors analyze events and produce threat detections or derived events.

Shared lists are reusable value sets (e.g., shell binaries, sensitive paths) that YAML detectors can reference in CEL expressions.

## ARGUMENTS

**[paths...]**
: Directories or files to search for YAML detectors and lists. If not specified, uses default paths (/etc/tracee/detectors).

## FLAGS

**\-\-json**, **-j**
: Output in JSON format for scripting.

## OUTPUT

The command displays two sections:

### Detectors

**ID**
: Unique detector identifier (e.g., TRC-001, DRV-001)

**Name**
: Detector event name

**Severity**
: Threat severity level (info, low, medium, high, critical)

**Required Events**
: Events the detector needs to receive

**MITRE**
: MITRE ATT&CK technique ID if applicable

### Shared Lists

**Name**
: List variable name (uppercase snake_case, e.g., SHELL_BINARIES)

**Values**
: Number of values in the list

**Source**
: Directory the shared list was loaded from (YAML scope for that list).

## JSON OUTPUT

When using `--json`, the output structure is:

```json
{
  "detectors": [
    {
      "id": "yaml-001",
      "name": "suspicious_exec",
      "severity": "HIGH",
      "required_events": ["sched_process_exec"],
      "mitre_technique": "T1059"
    }
  ],
  "lists": [
    {"name": "SHELL_BINARIES", "value_count": 6, "source_dir": "/etc/tracee/detectors"},
    {"name": "SENSITIVE_PATHS", "value_count": 12, "source_dir": "/etc/tracee/detectors"}
  ]
}
```

## EXAMPLES

- List all detectors and lists from default paths:

```console
tracee list detectors
```

- List detectors and lists from a custom directory:

```console
tracee list detectors ./my-detectors
```

- List detectors and lists from multiple directories:

```console
tracee list detectors ./dir1 ./dir2
```

- List detectors and lists in JSON format:

```console
tracee list detectors --json
```

- Filter critical detectors with jq:

```console
tracee list detectors --json | jq '.detectors[] | select(.severity == "CRITICAL")'
```

- List shared list names with jq:

```console
tracee list detectors --json | jq '.lists[].name'
```

## SEE ALSO

tracee-list(1), tracee-list-events(1), tracee-list-policies(1)

---

### Docs Flags List Events.1 (`docs/docs/flags/list-events.1.md`)

--
title: TRACEE-LIST-EVENTS
section: 1
header: Tracee List Events Command Manual
date: 2025/01
...

## NAME

tracee **list events** - List traceable events with optional filtering

## SYNOPSIS

tracee **list events** [filters...] [\-\-json]

## DESCRIPTION

The **list events** command displays all events that can be traced by Tracee, with optional filtering.

Uses the same filter syntax as **tracee \-\-events**.

## FILTER PATTERNS

**eventname**
: Exact event name match (e.g., `open`, `execve`)

**pattern\***
: Wildcard pattern match. Supports prefix (`open*`), suffix (`*write`), or contains (`*file*`)

**tag=TAG**
: Filter by tag/set (e.g., `tag=fs`, `tag=network`, `tag=syscalls`)

**tag=TAG1,TAG2**
: Filter by multiple tags with OR logic (e.g., `tag=fs,network`)

**type=TYPE**
: Filter by event type. Valid values: `syscall`, `detector`, `network`

**threat.severity=SEVERITY**
: Filter by threat severity. Valid values: `info`, `low`, `medium`, `high`, `critical`

**threat.mitre.technique=ID**
: Filter by MITRE ATT&CK technique ID (e.g., `threat.mitre.technique=T1055`)

**threat.mitre.tactic=NAME**
: Filter by MITRE ATT&CK tactic name (e.g., `threat.mitre.tactic=Execution`)

## OUTPUT FLAGS

**\-\-json**, **-j**
: Output in JSON format for scripting.

## FILTER SEMANTICS

Multiple filter arguments are combined with **AND** logic:

```console
tracee list events tag=fs threat.severity=high
# Events must have 'fs' tag AND high severity
```

Comma-separated values within a filter are combined with **OR** logic:

```console
tracee list events tag=fs,network
# Events with 'fs' OR 'network' tag
```

## EXAMPLES

- List all events:

```console
tracee list events
```

- List event by exact name:

```console
tracee list events open
```

- List events matching a pattern:

```console
tracee list events 'open*'
tracee list events '*write*'
```

- List events with a specific tag:

```console
tracee list events tag=fs
tracee list events tag=syscalls
```

- List events with multiple tags (OR):

```console
tracee list events tag=fs,network
```

- List events with multiple tags (AND):

```console
tracee list events tag=fs tag=proc
```

- List syscall events only:

```console
tracee list events type=syscall
```

- List detector events only:

```console
tracee list events type=detector
```

- List events by threat severity:

```console
tracee list events threat.severity=critical
tracee list events threat.severity=high,critical
```

- List events by MITRE technique:

```console
tracee list events threat.mitre.technique=T1055
```

- List events by MITRE tactic:

```console
tracee list events threat.mitre.tactic=Execution
```

- Combine multiple filters (AND):

```console
tracee list events tag=fs threat.severity=high
tracee list events type=detector threat.mitre.tactic=Persistence
```

- Output in JSON format:

```console
tracee list events --json
tracee list events tag=fs --json | jq '.[] | .name'
```

## SEE ALSO

tracee-list(1), tracee-list-detectors(1), tracee-list-policies(1)

---

### Docs Flags List Policies.1 (`docs/docs/flags/list-policies.1.md`)

--
title: TRACEE-LIST-POLICIES
section: 1
header: Tracee List Policies Command Manual
date: 2025/01
...

## NAME

tracee **list policies** - List policies from a directory

## SYNOPSIS

tracee **list policies** [paths...] [\-\-json]

## DESCRIPTION

The **list policies** command displays all policies from the specified directories or files.

Policies define what events to trace, how to filter them, and what actions to take.

## ARGUMENTS

**[paths...]**
: Directories or files to search for policies. If not specified, uses default path (/etc/tracee/policies).

## FLAGS

**\-\-json**, **-j**
: Output in JSON format for scripting.

## OUTPUT COLUMNS

**Name**
: Policy name

**Description**
: Policy description from annotations

**Scope**
: Summary of scope filters

**Rules**
: Number of rules in the policy

## EXAMPLES

- List policies from default path:

```console
tracee list policies
```

- List policies from a custom directory:

```console
tracee list policies ./my-policies
```

- List policies from multiple directories:

```console
tracee list policies ./dir1 ./dir2
```

- List a single policy file:

```console
tracee list policies ./my-policy.yaml
```

- List policies in JSON format:

```console
tracee list policies --json
```

## SEE ALSO

tracee-list(1), tracee-list-events(1), tracee-list-detectors(1), tracee-policy(1)

---

### Docs Flags List.1 (`docs/docs/flags/list.1.md`)

--
title: TRACEE-LIST
section: 1
header: Tracee List Command Manual
date: 2025/01
...

## NAME

tracee **list** - List traceable events, detectors, or policies

## SYNOPSIS

tracee **list** \<subcommand\> [arguments...] [options]

## DESCRIPTION

The **list** command displays available events, detectors, or policies that can be used with Tracee.

A subcommand is required.

## SUBCOMMANDS

**events** [filters...]
: List all traceable events with optional filtering. See tracee-list-events(1).

**detectors** [paths...]
: List available detectors from built-in and YAML sources. See tracee-list-detectors(1).

**policies** [paths...]
: List policies from directories or files. See tracee-list-policies(1).

## EXAMPLES

- List all events:

```console
tracee list events
```

- List events with filters:

```console
tracee list events tag=fs
tracee list events type=detector threat.severity=critical
```

- List detectors:

```console
tracee list detectors
tracee list detectors ./my-detectors
```

- List policies:

```console
tracee list policies
tracee list policies ./my-policies
```

## SEE ALSO

tracee-list-events(1), tracee-list-detectors(1), tracee-list-policies(1)

---

### Docs Flags Logging.1 (`docs/docs/flags/logging.1.md`)

---
title: TRACEE-LOGGING
section: 1
header: Tracee Logging Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-logging** - Control logger options - aggregation and level priority

## SYNOPSIS

tracee **\-\-logging** aggregate | aggregate.flush-interval=<time\> | level=<debug|info|warn|error|fatal\> | file=/path/to/file | filters.include.[msg=<value,...\>] | filters.include.[regex=<value,...\>] | filters.include.[pkg=<value,...\>] | filters.include.[file=<value,...\>] | filters.include.[level=<value,...\>] | filters.include.[libbpf] | filters.exclude.[msg=<value,...\>] | filters.exclude.[regex=<value,...\>] | filters.exclude.[pkg=<value,...\>] | filters.exclude.[file=<value,...\>] | filters.exclude.[level=<value,...\>] | filters.exclude.[libbpf]


## DESCRIPTION

The **\-\-logging** flag allows you to control logger options for the tool.

Possible log options:

- **aggregate | aggregate.flush-interval=[time]**: Turns log aggregation on, delaying output with an optional interval (default: 3s). Use `aggregate` to enable aggregation with default flush interval, or `aggregate.flush-interval=[time]` to set a custom interval. The flush-interval can be specified in seconds (s) or minutes (m).

- **level=<debug|info|warn|error|fatal\>**: Sets the log level. The default log level is 'info'.

- **file=/path/to/file**: Writes the logs to the specified file. If the file exists, it will be created or trimmed.

- **filters.include.**<option;...\>: Filters in logs that match the specified option values. Multiple filter options can be provided, separated by semicolons.

- **filters.exclude.**<option;...\>: Filters out logs that match the specified option values. Multiple filter options can be provided, separated by semicolons.

Filter options:

- **msg=<value,...\>**: Filters logs that have the message containing any of the specified values.

- **regex=<value,...\>**: Filters logs that match the specified regular expression in the message.

- **pkg=<value,...\>**: Filters logs that originate from the specified package.

- **file=<value,...\>**: Filters logs that originate from the specified file.

- **level=<value,...\>**: Filters logs that are of the specified level.

- **libbpf**: Filters logs that originate from libbpf.

## EXAMPLES

- To output debug level logs, use the following flag:

  ```console
  --logging level=debug
  ```

- To output aggregated debug level logs every 3 seconds (default), use the following flag:

  ```console
  --logging level=debug --logging aggregate
  ```

- To output aggregated logs every 5 seconds, use the following flag:

  ```console
  --logging aggregate.flush-interval=5s
  ```

- To output debug level logs to `/tmp/tracee.log`, use the following flag:

  ```console
  --logging level=debug --logging file=/tmp/tracee.log
  ```

- To filter in logs that have either 'foo' or 'bar' in the message, are from the 'core' package, and are of 'error' level, use the following flag:

  ```console
  --logging filters.include.msg=foo,bar --logging filters.include.pkg=core --logging filters.include.level=error
  ```

- To filter out logs that have either 'foo' or 'bar' in the message, are from the 'core' package, and are of 'error' level, use the following flag:

  ```console
  --logging filters.exclude.msg=foo,bar --logging filters.exclude.pkg=core --logging filters.exclude.level=error
  ```

- To filter in logs that have either 'foo' or 'bar' in the message and, based on that result, filter out logs that are from the 'core' package, use the following flag:

  ```console
  --logging filters.include.msg=foo,bar --logging filters.exclude.pkg=core
  ```

- To filter out logs that originate from the '/pkg/cmd/flags/logger.go' file, use the following flag:

  ```console
  --logging filters.exclude.file=/pkg/cmd/flags/logger.go
  ```

- To filter in logs that have messages matching the regex '^foo', use the following flag:

  ```console
  --logging filters.include.regex='^foo'
  ```

- To filter in logs that originate from libbpf, use the following flag:

  ```console
  --logging filters.include.libbpf
  ```

---

### Docs Flags Man.1 (`docs/docs/flags/man.1.md`)

--
title: TRACEE-MAN
section: 1
header: Tracee Manual Command Manual
date: 2025/01
...

## NAME

tracee **man** - Open manual pages for tracee flags and events

## SYNOPSIS

tracee **man** \<flag\> | \<event\>

## DESCRIPTION

The **man** command provides access to detailed documentation for tracee flags and events. It displays manual pages for specific flags or events directly in your terminal.

## SUBCOMMANDS

The **man** command accepts subcommands that correspond to tracee flags:

- **artifacts**, **a** - Show manual page for the --artifacts flag
- **buffers** - Show manual page for the --buffers flag
- **capabilities**, **C** - Show manual page for the --capabilities flag
- **config**, **c** - Show manual page for the --config flag
- **detectors**, **d** - Show manual page for the --detectors flag
- **enrichment**, **E** - Show manual page for the --enrichment flag
- **events**, **e** - Show manual page for the --events flag
- **list** - Show manual page for the list command
- **list-events** - Show manual page for the list events subcommand
- **list-detectors** - Show manual page for the list detectors subcommand
- **list-policies** - Show manual page for the list policies subcommand
- **logging**, **l** - Show manual page for the --logging flag
- **output**, **o** - Show manual page for the --output flag
- **policy**, **p** - Show manual page for the --policy flag
- **scope**, **s** - Show manual page for the --scope flag
- **server** - Show manual page for the --server flag
- **signatures-dir** - Show manual page for the --signatures-dir flag
- **stores** - Show manual page for the --stores flag

## EXAMPLES

- View manual page for a flag:

```console
tracee man events
tracee man output
tracee man policy
```

- View manual page for the list command:

```console
tracee man list
```

- View manual page for list subcommands:

```console
tracee man list-events
tracee man list-detectors
tracee man list-policies
```

- Using aliases:

```console
tracee m events
tracee m e
```

## SEE ALSO

tracee(1), tracee-list(1)

---

### Docs Flags Output.1 (`docs/docs/flags/output.1.md`)

---
title: TRACEE-OUTPUT
section: 1
header: Tracee Output Flag Manual
date: 2025/01
...

## NAME

tracee **\-\-output** - Control how and where output is printed

## SYNOPSIS

tracee **\-\-output** destinations.*name*.*field*=*value* | sort-events


## DESCRIPTION

The **\-\-output** flag allows you to control how and where the output is printed using destinations and output options.

### Destination Configuration

Output destinations are configured using the format: `--output destinations.<name>.<field>=<value>`

**Available Fields:**

- **type**: Type of the destination. One of `file`, `webhook`, or `forward`. Default: `file`
- **format**: Format of the event. One of `json`, `table`, or `gotemplate=/path/to/template`. Default: `table` for file, `json` for webhook and forward
- **path**: (file type only) File path to write output. Default: `stdout`
- **url**: (webhook and forward types) Destination URL

**Destination Types:**

- **file**: Output to a file or stdout in JSON, table, or custom template format
- **webhook**: Send events in JSON format to a webhook URL
- **forward**: Send events to a FluentBit receiver using the Forward protocol

### Output Options

- **sort-events**: Enable sorting events before passing them to output. May decrease overall program efficiency.

!!! Note
    The enrichment `decoded-data` option is automatically enabled when using table format output. It does not need to be specified separately via `--enrichment decoded-data`.

## EXAMPLES

- To output events as JSON to stdout using a destination named `stdout_json`:

  ```console
  --output destinations.stdout_json.format=json
  ```

- To output events as JSON to a file `/my/out.json`:

  ```console
  --output destinations.file_out.type=file --output destinations.file_out.format=json --output destinations.file_out.path=/my/out.json
  ```

- To output events using a Go template:

  ```console
  --output destinations.template_out.format=gotemplate=/path/to/my.tmpl
  ```

- To send events via the Forward protocol to a FluentBit receiver:

  ```console
  --output destinations.forward1.type=forward --output destinations.forward1.url=tcp://user:pass@127.0.0.1:24224?tag=tracee
  ```

- To send events to a webhook endpoint:

  ```console
  --output destinations.webhook1.type=webhook --output destinations.webhook1.url=http://webhook:8080
  ```

- To send events to a webhook with a timeout:

  ```console
  --output destinations.webhook1.type=webhook --output destinations.webhook1.url=http://webhook:8080?timeout=5s
  ```

## SEE ALSO

For comprehensive information about output configuration:

- **Output Formats**: See [Output Formats](../outputs/output-formats.md) for detailed format options
- **Event Structure**: See [Event Structure](../outputs/event-structure.md) for understanding event data
- **Outputs Overview**: See [Outputs Overview](../outputs/index.md) for complete output documentation

---

### Docs Flags Policy.1 (`docs/docs/flags/policy.1.md`)

---
title: TRACEE-POLICY
section: 1
header: Tracee Policy Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-policy** - Specify policy files or directories to load

## SYNOPSIS

tracee **\-\-policy** <file|dir> [**\-\-policy** <file|dir> ...]

## DESCRIPTION

The **\-\-policy** flag allows you to specify one or more policy files or directories containing policy files to load into Tracee.

Policies define which events to trace and how to filter them. When using the **\-\-policy** flag, you cannot use the **\-\-scope** or **\-\-events** flags together, as they serve different purposes for event selection.

Policy files are YAML files that follow the Tracee Policy API specification. Tracee supports two formats: **Kubernetes CRD format** and **Plain YAML format**. Both formats are fully interchangeable and Tracee automatically detects the format when loading policies. They define:

- **Scope**: Which workloads to trace (e.g., global, specific containers, processes)
- **Rules**: Which events to trace and optional filters to apply

When specifying a directory, Tracee will load all policy files found in that directory. When specifying individual files, you can use multiple **\-\-policy** flags to load multiple policies.

## EXAMPLES

- Load a single policy file:

  ```console
  --policy ./policy.yaml
  ```

- Load all policies from a directory:

  ```console
  --policy ./policies/
  ```

- Load multiple individual policy files:

  ```console
  --policy ./policy1.yaml --policy ./policy2.yaml
  ```

- Using the short form:

  ```console
  -p ./policy.yaml
  ```

- Complete example with policy:

  ```console
  tracee --policy ./security-policy.yaml --output json:events.json
  ```

## POLICY FILE FORMAT

Policy files use YAML format. Tracee supports two formats:

### Kubernetes CRD Format

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: policy-name
  annotations:
    description: Policy description
spec:
  scope:
    - global
  rules:
    - event: event_name
      filters:
        - filter_expression
```

### Plain YAML Format

```yaml
type: policy
name: policy-name
description: Policy description
scope:
  - global
rules:
  - event: event_name
    filters:
      - filter_expression
```

Tracee automatically detects the format when loading policies. Both formats are fully interchangeable and produce identical results.

For more information about policy file formats and options, refer to the [Policy Documentation](../policies/index.md).

## NOTES

- If multiple policies are loaded, they are combined and all matching events from any policy will be traced
- Policy files must be valid YAML and conform to the Tracee Policy API specification

---

### Docs Flags Scope.1 (`docs/docs/flags/scope.1.md`)

---
title: TRACEE-SCOPE
section: 1
header: Tracee Scope Flag Manual
date: 2024/09
...

## NAME

tracee **\-\-scope** - Select the scope for tracing events

## SYNOPSIS

tracee **\-\-scope** [<[uid|pid][=|!=|<|\>|<=|\>=]value1(,value2...)\> | <[mntns|pidns|tree][=|!=]value1(,value2...)\> | <[uts|comm|container|[executable|exec|binary|bin]][=|!=]value1(,value2...)\>] | <not-container\> | <container[=|!=]value\> | <[container|pid]=new\> | <follow\>]  ...

## DESCRIPTION

The **\-\-scope** flag allows you to select the scope for tracing events by defining filters.

## FILTER EXPRESSION

Filter expressions can be defined to operate on scope options or process metadata. Only events that match all filter expressions will be traced.

Multiple flags are combined with AND logic, while multiple values within a single flag are combined with OR logic when using the equals operator '='. Multiple values can be specified using ','.

### NUMERICAL EXPRESSION OPERATORS

The following numerical fields support the operators '=', '!=', '<', '>', '<=', '>=':

- uid: Select events from specific user IDs.
- pid: Select events from specific process IDs.

The following numerical fields only support the operators '=' and '!=':

- mntns: Select events from specific mount namespace IDs.
- pidns: Select events from specific process namespace IDs.
- tree: Select events that descend from specific process IDs.

NOTE: Expressions containing '<' or '\>' tokens must be escaped!

### STRING EXPRESSION OPERATORS

'=', '!='

Available for the following string fields:

- uts: Select events based on UTS (Unix Timesharing System) names.
- comm: Select events based on process command names.
- container: Select events from specific container IDs.
- executable: Select events based on the executable path.

Strings can be compared as a prefix if ending with '\*', or as a suffix if starting with '\*'.

NOTE: Expressions containing '\*' token must be escaped!

you can learn more about the wildcard in the [`event` section](./events.1.md).

### BOOLEAN OPERATOR (PREPENDED)

'!'

Available for the following boolean field:

- container: Select events based on whether they originate from a container or not.

## SPECIAL FILTERS

The following special filters can be used within the scope filter expressions:

- new: Select newly created containers or process IDs.
- started: Select events from containers that have completed initialization and started running (post-entrypoint execution). Only available for container scope (e.g., `container=started`). Note: The negation `container!=started` is not supported due to race conditions with early container event recognition.
- follow: Select events from the processes that match the criteria and their descendants.

## EXAMPLES

- To trace only events from new processes, use the following flag:

  ```console
  --scope pid=new
  ```

- To trace only events from pid 510 or pid 1709, use the following flag:

  ```console
  --scope pid=510,1709
  ```

- To trace only events from pid 510 or pid 1709 (same as above), use the following flag:

  ```console
  --scope p=510 --scope p=1709
  ```

- To trace only events from newly created containers, use the following flag:

  ```console
  --scope container=new
  ```

- To trace only events from containers that have started running (post-entrypoint execution), use the following flag:

  ```console
  --scope container=started
  ```

- To trace only events from the container with ID 'ab356bc4dd554', use the following flag:

  ```console
  --scope container=ab356bc4dd554
  ```

- To trace only events from containers, use the following flag:

  ```console
  --scope container
  ```

- To only trace events from containers (same as above), use the following flag:

  ```console
  --scope c
  ```

- To trace only events from the host, use the following flag:

  ```console
  --scope not-container
  ```

- To trace only events from uid 0, use the following flag:

  ```console
  --scope uid=0
  ```

- To trace only events from mntns id 4026531840, use the following flag:

  ```console
  --scope mntns=4026531840
  ```

- To trace only events from pidns id not equal to 4026531836, use the following flag:

  ```console
  --scope pidns!=4026531836
  ```

- To trace only events that descend from the process with pid 476165, use the following flag:

  ```console
  --scope tree=476165
  ```

- To trace only events if they do not descend from the process with pid 5023, use the following flag:

  ```console
  --scope tree!=5023
  ```

- To trace only events if they descend from 3213 or 5200, but not 3215, use the following flag:

  ```console
  --scope tree=3213,5200 --scope tree!=3215
  ```

- To trace only events from uids greater than 0, use the following flag:

  ```console
  --scope 'uid>0'
  ```

- To trace only events from pids between 0 and 1000, use the following flag:

  ```console
  --scope 'pid>0' --scope 'pid<1000'
  ```

- To trace only events from uids greater than 0 but not 1000, use the following flag:

  ```console
  --scope 'u>0' --scope u!=1000
  ```

- To exclude events from uts name 'ab356bc4dd554', use the following flag:

  ```console
  --scope uts!=ab356bc4dd554
  ```

- To trace only events from the 'ls' command, use the following flag:

  ```console
  --scope comm=ls
  ```

- To trace only events from the '/usr/bin/ls' executable, use the executable flag (or the binary alias):

  ```console
  --scope executable=/usr/bin/ls
  ```

  ```console
  --scope binary=/usr/bin/ls
  ```

- To trace only events from the '/usr/bin/ls' executable in the host mount namespace, use the following flag:

  ```console
  --scope executable=host:/usr/bin/ls
  ```

- To trace only events from the '/usr/bin/ls' executable in the 4026532448 mount namespace, use the following flag:

  ```console
  --scope executable=4026532448:/usr/bin/ls
  ```

- To trace all events that originated from 'bash' or from one of the processes spawned by 'bash', use the following flag:

  ```console
  --scope comm=bash --scope follow
  ```

---

### Docs Flags Server.1 (`docs/docs/flags/server.1.md`)

---
title: TRACEE-SERVER
section: 1
header: Tracee Server Flag Manual
date: 2026/04
...

## NAME

tracee **\-\-server** - Configure server options and endpoints

## SYNOPSIS

tracee **\-\-server** <http-address=<host:port\>> | <grpc-address=<protocol:address\>> | grpc | metrics | healthz | pprof | pyroscope

## DESCRIPTION

The **\-\-server** flag allows you to configure server options and endpoints for Tracee. This unified flag replaces the deprecated individual server flags and provides better validation and error messages.

Server address options:

- **http-address=<host:port\>**: Start an HTTP server listening on the specified address. Used for metrics, healthz, pprof, and pyroscope endpoints.

- **grpc-address=<protocol:address\>**: Start a gRPC server listening on the specified protocol and address. Supported protocols are:
  - **tcp:<port\>** or **tcp:<host\>:<port\>**: TCP connection. A bare port (e.g., tcp:4466) binds to **127.0.0.1** (loopback only). To bind to a specific interface, use host:port (e.g., tcp:0.0.0.0:4466). IPv6 addresses must be enclosed in square brackets (e.g., tcp:[::1]:4466). If no port is specified, defaults to 127.0.0.1:4466.
  - **unix:<socket_path\>**: Unix domain socket (e.g., unix:/tmp/tracee.sock). If no path is specified, defaults to /var/run/tracee.sock.

Server endpoint options (require HTTP server):

- **metrics**: Enable Prometheus metrics endpoint at /metrics. If no HTTP server is configured, defaults to 127.0.0.1:3366.

- **healthz**: Enable health check endpoint at /healthz. If no HTTP server is configured, defaults to 127.0.0.1:3366.

- **pprof**: Enable Go pprof debugging endpoints under /debug/pprof/. If no HTTP server is configured, defaults to 127.0.0.1:3366.

- **pyroscope**: Enable Pyroscope profiling agent integration. If no HTTP server is configured, defaults to 127.0.0.1:3366.

Other options:

- **grpc**: Enable gRPC server with default Unix socket at /var/run/tracee.sock. Equivalent to grpc-address=unix.

Multiple **\-\-server** flags can be specified to enable different combinations of servers and endpoints.

## EXAMPLES

- To enable gRPC server with default Unix socket:

  ```console
  --server grpc
  ```

- To enable gRPC server on a specific TCP port (loopback only):

  ```console
  --server grpc-address=tcp:4466
  ```

- To enable gRPC server on IPv6 loopback:

  ```console
  --server grpc-address=tcp:[::1]:4466
  ```

- To enable gRPC server on all interfaces (use with caution):

  ```console
  --server grpc-address=tcp:0.0.0.0:4466
  ```

- To enable gRPC server with custom Unix socket path:

  ```console
  --server grpc-address=unix:/tmp/tracee.sock
  ```

- To enable HTTP server with metrics endpoint:

  ```console
  --server http-address=:3366 --server metrics
  ```

- To enable metrics endpoint with default HTTP server:

  ```console
  --server metrics
  ```

- To enable multiple HTTP endpoints on custom address:

  ```console
  --server http-address=127.0.0.1:8080 --server metrics --server healthz --server pprof
  ```

- To enable both gRPC and HTTP servers:

  ```console
  --server grpc-address=unix:/var/run/tracee.sock --server http-address=:3366 --server metrics
  ```

- To enable all available endpoints:

  ```console
  --server http-address=:3366 --server grpc-address=tcp:4466 --server metrics --server healthz --server pprof --server pyroscope
  ```

## MIGRATION FROM DEPRECATED FLAGS

The **\-\-server** flag replaces the following deprecated flags:

| Deprecated Flag | New Server Flag Equivalent |
|---|---|
| `--metrics-endpoint` | `--server metrics` |
| `--healthz-endpoint` | `--server healthz` |
| `--pprof-endpoint` | `--server pprof` |
| `--pyroscope` | `--server pyroscope` |
| `--http-listen-addr=:3366` | `--server http-address=:3366` |
| `--grpc-listen-addr=tcp:4466` | `--server grpc-address=tcp:4466` |

Migration examples:

- Before: `--metrics-endpoint --grpc-listen-addr=tcp:4466`
- After: `--server metrics --server grpc-address=tcp:4466`

- Before: `--http-listen-addr=:8080 --metrics-endpoint --healthz-endpoint --pprof-endpoint`
- After: `--server http-address=:8080 --server metrics --server healthz --server pprof`

- Before: `--grpc-listen-addr=unix:/tmp/tracee.sock --metrics-endpoint`
- After: `--server grpc-address=unix:/tmp/tracee.sock --server metrics`

## DEFAULT BEHAVIOR

- If no **\-\-server** flags are specified, no servers are started.
- If endpoint flags (metrics, healthz, pprof, pyroscope) are specified without http-address, an HTTP server is automatically created on 127.0.0.1:3366 (loopback only).
- If **grpc-address=tcp** is specified without a port, defaults to 127.0.0.1:4466 (loopback only).
- If **grpc-address=tcp:<port\>** is specified with a bare port, binds to 127.0.0.1:<port\>.
- If **grpc-address=unix** is specified without a path, defaults to /var/run/tracee.sock.
- Existing Unix socket files are automatically cleaned up before starting the gRPC server.

## SECURITY

Both the HTTP and gRPC TCP servers default to binding on **127.0.0.1** (loopback only). Earlier versions bound the gRPC TCP server to **0.0.0.0** (all interfaces), which exposed privileged RPCs to the network without authentication. The gRPC server has no TLS or authentication; binding to a non-loopback address requires network-level access controls.

If you previously relied on the implicit wildcard bind for remote gRPC access, you must now set an explicit bind address:

- **gRPC on all interfaces (restore old behavior -- use with caution):**

  ```console
  --server grpc-address=tcp:0.0.0.0:4466
  ```

- **gRPC on specific network interface (preferred when possible):**

  ```console
  --server grpc-address=tcp:10.0.0.5:4466
  ```

If you previously relied on the implicit wildcard bind for remote HTTP access (Prometheus scraping, health checks, pprof), you must now set an explicit bind address:

- **HTTP on all interfaces:**

  ```console
  --server http-address=0.0.0.0:3366 --server metrics
  ```

- **HTTP on specific network interface (preferred when possible):**

  ```console
  --server http-address=10.0.0.5:3366 --server metrics
  ```

- **Config file equivalent:**

  ```yaml
  server:
    http-address: "0.0.0.0:3366"
    grpc-address: "tcp:0.0.0.0:4466"
    metrics: true
    healthz: true
  ```

**pprof security:** The /debug/pprof/ endpoints are served on the same HTTP address as metrics and healthz with no additional access control. pprof heap dumps can expose sensitive process memory of a privileged eBPF daemon. Keep the default loopback bind, use the NetworkPolicy (Kubernetes), or firewall rules to restrict access. For remote debugging, use an SSH tunnel or kubectl port-forward.

**Kubernetes deployments:** The Helm chart sets `0.0.0.0:3366` explicitly. Enable `networkPolicy.enabled=true` to deploy a NetworkPolicy restricting ingress to Prometheus pods (override `networkPolicy.ingressFrom` if your Prometheus uses different labels). If using TCP gRPC in Kubernetes, set an explicit bind address and ensure NetworkPolicy covers the gRPC port as well.

## SEE ALSO

**tracee**(1), **tracee-output**(1), **tracee-log**(1)

---

### Docs Flags Signatures Dir.1 (`docs/docs/flags/signatures-dir.1.md`)

---
title: TRACEE-SIGNATURES-DIR
section: 1
header: Tracee Signatures Directory Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-signatures-dir** - Specify directories to search for signature plugins

## SYNOPSIS

tracee **\-\-signatures-dir** <dir> [**\-\-signatures-dir** <dir> ...]

## DESCRIPTION

The **\-\-signatures-dir** flag allows you to specify one or more directories where Tracee should search for signature plugins in Go plugin (.so) format.

Signatures are security detection rules that can detect suspicious or malicious behavior patterns. They are compiled as Go plugins and loaded dynamically at runtime.

When Tracee starts, it searches for signature plugins in:

- The directories specified by **\-\-signatures-dir** flags, or
- Default signatures

Each signature plugin file must be a compiled Go plugin (.so file) that implements the Tracee signature interface. Loaded signatures create corresponding signature events that can be traced using policies or event flags.

## EXAMPLES

- Specify a single directory:

  ```console
  --signatures-dir /opt/tracee/signatures
  ```

- Specify multiple directories:

  ```console
  --signatures-dir /opt/tracee/signatures --signatures-dir /custom/signatures
  ```

- Use with policy to trace signature events:

  ```console
  tracee --signatures-dir /opt/tracee/signatures --policy ./security-policy.yaml
  ```

- List available signature events:

  ```console
  tracee --signatures-dir /opt/tracee/signatures list
  ```

## SIGNATURE PLUGINS

Signature plugins are compiled Go shared libraries (.so files) that implement the Tracee signature interface. Each plugin can define one or more signatures that detect specific security events or patterns.

When a signature plugin is loaded:

- The signatures it defines become available as traceable events
- These events can be referenced in policies or event flags
- Signature events are typically prefixed to distinguish them from system events

To create custom signatures, refer to the Tracee signature development documentation.

## NOTES

- Signature plugins must be compiled for the same architecture and Go version as Tracee
- Only valid Go plugin files (.so) will be loaded from the specified directories
- If no signatures are found, Tracee will continue to run but signature events will not be available
- The **\-\-signatures-dir** flag can be used with the **list** command to see available signature events before running Tracee

---

### Docs Flags Stores.1 (`docs/docs/flags/stores.1.md`)

---
title: TRACEE-STORES
section: 1
header: Tracee Stores Flag Manual
date: 2025/12
...

## NAME

tracee **\-\-stores** - Configure data stores for DNS cache and process tree

## SYNOPSIS

tracee **\-\-stores** [dns|dns.max-entries=*size*|process|process.max-processes=*size*|process.max-threads=*size*] [**\-\-stores** ...]

## DESCRIPTION

The **\-\-stores** flag allows you to configure data stores for DNS cache and process tree functionality.

### DNS Store Options

- **dns**: Enable the DNS cache store with default settings. When enabled, Tracee will cache DNS query information for enrichment of network events.

- **dns.max-entries**=*size*: Enable the DNS cache store and set the maximum number of DNS query trees to cache. Default is 5000. Further queries may be cached regardless once the limit is reached. **Note**: Using this option automatically enables DNS, so you don't need to also specify `--stores dns`.

### Process Store Options

- **process**: Enable the process tree store with default settings. When enabled, Tracee will maintain a tree of processes and threads for enrichment of events. **Note: Process tree is enabled by default.**

- **process.max-processes**=*size*: Enable the process tree store and set the maximum number of processes to cache in the process tree. Default is 10000. This is an LRU cache that will evict least recently accessed entries when full. **Note**: Using this option automatically enables process, so you don't need to also specify `--stores process`.

- **process.max-threads**=*size*: Enable the process tree store and set the maximum number of threads to cache in the process tree. Default is 0 (thread tracking disabled to save memory). This is an LRU cache that will evict least recently accessed entries when full. **Note**: Using this option automatically enables process, so you don't need to also specify `--stores process`.

**Note**: Procfs initialization happens automatically when the process tree is enabled. At startup, Tracee scans `/proc` to populate the process tree with all existing processes and threads, ensuring complete process ancestry information is available.

## EXAMPLES

1. Enable DNS cache:
   ```console
   --stores dns
   ```

2. Enable DNS cache with custom size:
   ```console
   --stores dns.max-entries=10000
   ```
   
   Note: `dns.max-entries` automatically enables DNS, so `--stores dns` is not needed.

3. Enable process tree:
   ```console
   --stores process
   ```

4. Enable process tree with custom cache sizes:
   ```console
   --stores process.max-processes=8192 --stores process.max-threads=16384
   ```
   
   Note: `process.max-processes` and `process.max-threads` automatically enable process, so `--stores process` is not needed.

5. Combine DNS and process stores:
   ```console
   --stores dns.max-entries=5000 --stores process.max-processes=8192
   ```
   
   Note: Since `dns.max-entries` automatically enables DNS and `process.max-processes` automatically enables process, you don't need `--stores dns` or `--stores process`.

6. Complete configuration example:
   ```console
   --stores dns.max-entries=5000 --stores process.max-processes=8192 --stores process.max-threads=16384
   ```
   
   Note: All process options automatically enable process, and `dns.max-entries` automatically enables DNS, so you don't need `--stores dns` or `--stores process`.

Please refer to the [DataStore API documentation](../detectors/datastore-api.md) for information about using these stores in detectors:

- [DNSStore](../detectors/datastore-api.md#dnsstore) - DNS cache access
- [ProcessStore](../detectors/datastore-api.md#processstore) - Process tree and ancestry

---

### Docs Flags Version.1 (`docs/docs/flags/version.1.md`)

--
title: TRACEE-VERSION
section: 1
header: Tracee Version Command Manual
date: 2025/01
...

## NAME

tracee **version** - Print the version number of Tracee

## SYNOPSIS

tracee **version**

## DESCRIPTION

The **version** command displays the current version number of Tracee.

## EXAMPLES

- Print the version:

```console
tracee version
```

- Using the alias:

```console
tracee v
```

## SEE ALSO

tracee(1)

---

### Docs Install Capabilities (`docs/docs/install/capabilities.md`)

# Running Tracee with Least Privileges

## Introduction to Capabilities (quoting parts of kernel docs)

For the purpose of performing permission checks, traditional UNIX
implementations distinguish two categories of processes: privileged processes
(whose effective user ID is 0, referred to as superuser or root), and
unprivileged processes (whose effective UID is nonzero).

Privileged processes bypass all kernel permission checks, while unprivileged
processes are subject to full permission checking based on the process's
credentials (usually: effective UID, effective GID, and supplementary group
list).

Linux divides the privileges traditionally associated with superuser into
distinct units, known as capabilities, which can be independently enabled and
disabled. Capabilities are a per-thread attribute.

Thread capability sets:

* Permitted: This is a limiting superset for the effective capabilities that the
thread may assume. It is also a limiting superset for the capabilities that
may be added to the inheritable set by a thread that does not have the
CAP_SETPCAP capability in its effective set.

* Inheritable: This is a set of capabilities preserved across an execve(2).
Inheritable capabilities remain inheritable when executing any program, and
inheritable capabilities are added to the permitted set when executing a
program that has the corresponding bits set in the file inheritable
set.

* Effective: This is the set of capabilities used by the kernel to perform
permission checks for the thread.

* Bounding: The capability bounding set is a mechanism that can be used to limit
the capabilities that are gained during execve(2).

## Tracee and capabilities

**tracee** tries to reduce its capabilities during its execution. The way it
does is through different "execution protection rings":

* Full:     All capabilities are effective (less secure)
* EBPF:     eBPF needed capabilities + Base capabilities
* Specific: Specific capabilities (from time to time) + Base Capabilities
* Base:     None or Some capabilities always effective (more secure)

## Listing available capabilities

You may see all available capabilities in the running environment by running:

```console
capsh --print
```

## Bypass capabilities dropping feature

!!! Attention
    This session is important if you're facing errors related to **tracee**
    dropping its capabilities OR any other permission related errors.

Some environments **won't allow capabilities dropping** because of permission
issues (for example - **AWS Lambdas**).

> It might be a result of seccomp filter for example, restricting syscalls
> access.

Failure in capabilities dropping will result tracee's exit with a matching
error, to **guarantee that tracee isn't running with excess capabilities
without the user agreement**.

To **allow tracee to run with high capabilities**, and prevent those errors, the
`--capabilities bypass=true` flag can be used. For the docker container users,
the environment variable `CAPABILITIES_BYPASS=0|1` will have the same effect.

!!! Note
    Bypassing the capabilities drop will run **tracee** with all capabilities
    set as Effective and it is only recommended if you know what you are doing.

## Capabilities Errors (Missing or Too Permissive)

During development, tracee might have bugs related to capabilities dropping
feature: one event might not have its needed capabilities set as a dependency,
for example, and you might still want to use that event.

One way to have fine grained control of "execution time" effective capabilities
is to rely on following 2 command line flags:

- `--capabilities add=cap_X,cap_Y` (docker env variable CAPABILITIES_ADD)
- `--capabilities drop=cap_Y,capZ` (docker env variable CAPABILITIES_DROP)

The first will add given capabilities to the Base ring, the ring that describe
capabilities that will always be effective while tracee is running, so events
might be able to work. The last will remove the capabilities from that same
ring.

---

### Docs Install Container Engines (`docs/docs/install/container-engines.md`)

# Tracee Events Container Enrichment

Tracee events provide context about where the collected events originated from, including information about the originating container if available. To gather this information, Tracee needs to communicate with the relevant container runtime.

## Configuration

Container enrichment is configured using the `--enrichment` flag. For complete details on all enrichment options, see the [enrichment flag reference](../flags/enrichment.1.md).

### Enabling Container Enrichment

To enable container enrichment with automatic runtime detection:

```console
tracee --enrichment container
```

Or in a configuration file:

```yaml
enrichment:
  container:
    enabled: true
```

### Configuring Runtime Sockets

You can explicitly configure container runtime socket paths using the enrichment flag. This is useful when sockets are in non-standard locations or when running Tracee in containerized environments.

**CLI:**

```console
# Docker
tracee --enrichment container.docker.socket=/var/run/docker.sock

# Containerd
tracee --enrichment container.containerd.socket=/var/run/containerd/containerd.sock

# CRI-O
tracee --enrichment container.crio.socket=/var/run/crio/crio.sock

# Podman
tracee --enrichment container.podman.socket=/var/run/podman/podman.sock
```

**Configuration file:**

```yaml
enrichment:
  container:
    enabled: true
    docker-socket: /var/run/docker.sock
    containerd-socket: /var/run/containerd/containerd.sock
    crio-socket: /var/run/crio/crio.sock
    podman-socket: /var/run/podman/podman.sock
```

!!! note
    Setting any container socket option automatically enables container enrichment, so you don't need to also specify `--enrichment container`.

## Automatic Runtime Detection

Tracee will automatically search for known supported runtimes by looking for their socket files in known locations when container enrichment is enabled.

You may track if Tracee was able to find the container runtime socket by running Tracee with `debug` log level. There will be a line for each known runtime engine socket and a message sharing its status.

## Running Tracee in Containers

When running Tracee in a container, the runtime sockets must be mounted to be available for Tracee.

For example, if running Tracee using Docker with Containerd as the container runtime:

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run/containerd:/var/run/containerd:ro \
  aquasec/tracee:latest
```

Most container runtimes have their sockets installed by default in `/var/run`, so mounting this path can also be a good option.

## Supported Container Runtimes

Tracee supports the following container runtimes and will look for their socket files in these default locations:

| Runtime    | Default Socket Path                       |
|------------|-------------------------------------------|
| Docker     | `/var/run/docker.sock`                    |
| Containerd | `/var/run/containerd/containerd.sock`     |
| CRI-O      | `/var/run/crio/crio.sock`                 |
| Podman     | `/var/run/podman/podman.sock`             |

For more details on configuring enrichment options, see the [enrichment flag reference](../flags/enrichment.1.md).

---

### Docs Install Docker (`docs/docs/install/docker.md`)

# Running Tracee with Docker

This guide will help you get started with running Tracee as a container.

## Prerequisites

- Review the [prerequisites for running Tracee](./prerequisites.md)
- If you are an Apple Mac user, please read [the Mac FAQ](./mac-faq.md)
- Ensure that you have Docker or a compatible container runtime

## Tracee container image

 Tracee container image is available in Docker Hub as [aquasec/tracee](https://hub.docker.com/r/aquasec/tracee).

- You can use the `latest` tag or a named version version e.g `aquasec/tracee:{{ git.tag }}`.
- If you are trying the most cutting edge features, there is also a `dev` tag which is built nightly from source.
- The Tracee image is a [Multi-platform](https://docs.docker.com/build/building/multi-platform/) image that includes a x86 and arm64 flavors. You can also access the platform-specific images directly with the `aarch64` and `x86_64` tags for the latest version or `aarch64-<version>` and `x86_64-<version>` for a specific version.  
- For most first time users, just use `aquasec/tracee`!

## Running Tracee container

 Here is the docker run command, we will analyze it next:

```shell
docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest
```

 1. Docker general flags:
    - `--name` - name our container so that we can interact with it easily.
    - `--rm` - remove the container once it exits, assuming this is an interactive trial of Tracee.
    - `-it` - allow the container to interact with your terminal.
 2. Since Tracee runs in a container but is instrumenting the host, it will need access to some resources from the host:
    - `--pid=host` - share the host's process namespace with Tracee's container.
    - `--cgroupns=host` - share the host's cgroup namespace with Tracee's container.
    - `--privileged` - run the Tracee container as root so it has all the [required capabilities](./prerequisites.md#process-capabilities).
    - `-v /etc/os-release:/etc/os-release-host:ro` - share the host's [OS information file](./prerequisites.md#os-information) with the Tracee container.
    - `-v /var/run:/var/run` - share the host's container runtime socket for [container enrichment](./container-engines.md)

 After running this command, you should start seeing a stream of events that Tracee is emitting.

!!! tip "Not seeing events?"
    If Tracee isn't producing events or you encounter issues, check our [Troubleshooting Guide](../troubleshooting.md) for solutions.

 For next steps, please read about Tracee [Policies](../policies/index.md)

## Installing Tracee

 If you are looking to permanently install Tracee, you would probably do the following:

 1. Remove interactive flags `-it` and replace with daemon flag `-d`
 2. Consider how to collect events from the container.

 Or you can follow the [Kubernetes guide](./kubernetes.md) which addresses these concerns.

---

### Docs Install Kernel Symbols (`docs/docs/install/kernel-symbols.md`)

# About Kernel Symbols

As explained in the [prerequisites](./prerequisites.md) doc, Tracee
needs the kernel symbol table for some operations.

A Linux kernel might lack the `/proc/kallsyms` file due to:

**Kernel Configuration**: If compiled without `CONFIG_KALLSYMS`, the kernel
won't have this file. This option enables the kernel symbol table, used mainly
for debugging.

**Security Protocols**: Some systems might hide kernel symbols to prevent
potential exploits. The `/proc/kallsyms` file could appear incomplete or even
empty to non-root users. The `CONFIG_KALLSYMS_ALL` option ensures all symbols
are visible.

The Linux kernel also offers a setting, `/proc/sys/kernel/kptr_restrict`, to
control kernel symbol visibility:

- **0**: No restrictions.
- **1**: Hide from non-privileged users.
- **2**: Hide from all users.

---

### Docs Install Kubernetes (`docs/docs/install/kubernetes.md`)

# Installing Tracee in Kubernetes 

This guide will help you get started with Tracee by installing it in a Kubernetes cluster.  


## Prerequisites

- Supported environment - please refer to the [Prerequisites](../install/prerequisites.md)
- Kubernetes - this was tested on [minikube](https://github.com/kubernetes/minikube), but should work the same with most other Kubernetes distributions
- [Helm](https://helm.sh/docs/intro/install/)

<details>
  <summary>Verify step</summary>
```console
kubectl get po -A
```

```text
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE 
kube-system   coredns-565d847f94-kd9xx           1/1     Running   0          15s 
kube-system   etcd-minikube                      1/1     Running   0          26s 
kube-system   kube-apiserver-minikube            1/1     Running   0          26s 
kube-system   kube-controller-manager-minikube   1/1     Running   0          26s 
kube-system   kube-proxy-cvqjm                   1/1     Running   0          15s 
kube-system   kube-scheduler-minikube            1/1     Running   0          26s 
kube-system   storage-provisioner                1/1     Running   0          15s 
```
</details>

## Install Tracee

The provided Helm chart will install Tracee as a DaemonSet so that it's tracing all the nodes in the cluster.

```console
helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace
```

<details>
  <summary>Verify step</summary>
```console
kubectl get pods -n tracee
```

```text
NAME           READY   STATUS    RESTARTS   AGE 
tracee-fcjmp   1/1     Running   0          4m11s
```
</details>

Once installed, Tracee immediately starts producing events. Since Tracee is deployed as a DaemonSet, a Tracee Pod is running on every node in the cluster. Every Tracee Pod is monitoring the node it is running on.

## Viewing Events

The easiest way to tap into the log stream of all Tracee Pods is with the `kubectl logs` command:

```console
kubectl logs -f daemonset/tracee -n tracee
```

!!! Note
    Tracee can produce a very high volume of events which could overwhelm kubectl's log collection command. If run in a busy cluster or with a verbose policy, this command might be slow or unresponsive.

In production scenario you would probably want to collect and ship events logs into a persistent storage that you can query.   
You can use any log collection solution of your choosing. We have a tutorial on how to do this using the open source Grafana Stack [here](../../tutorials/deploy-grafana-dashboard.md).

## Applying Policies

By default, Tracee collects a basic set of events that gives you a general overview of the cluster. If you're looking to do more with Tracee, You might want to create a new [Policy](../policies/index.md). A policy lets you capture specific set of events from a specific set of workloads. For example, if you have an application that you want to monitor more closely, or in a specialized way, you can create a policy scoped to that application, with a different set of events and filters applied. To learn more, please refer to the [Events](../events/index.md) and [Policies](../policies/index.md) sections.

When you are ready to apply a policy, it's as easy as `kubectl apply -f your-policy.yaml`. More details [here](../policies/usage/kubernetes.md).

## Configuring Tracee

In some cases you will need to configure Tracee to your preferences. For example, to change the output event format, or to set a different log level. To learn more about available configuration options please see the [configuration](../install/config/index.md) section.

Tracee's configuration is accessible as a ConfigMap in Kubernetes. Since we installed Tracee with Helm, you can also configure Tracee with it, for example: `helm upgrade tracee --set config.cache.size=1024`. More details [here](../install/config/kubernetes.md).

## Optional: Exercising a security event

To see Tracee in action, let's simulate a security event. We'll do a "file-less" execution, which is a common evasion technique used by some malware, and is flagged by Tracee as suspicious activity. To simulate this, we'll use the [tracee-tester](https://registry.hub.docker.com/r/aquasec/tracee-tester) example image it will simulate the suspicious activity without harming your environment.

```console
kubectl run tracee-tester --image=aquasec/tracee-tester -- TRC-105
```

You can see the event in the logs:

```console
kubectl logs -f ds/tracee -n tracee | grep fileless_execution 
```

## Next steps

Familiarize with the different events, filters, and configuration options in the [documentation](../overview.md).

Read other [tutorials](../../tutorials/overview.md).

!!! info "Need Help?"
    - For common issues and solutions, see our [Troubleshooting Guide](../troubleshooting.md)
    - For help and support, feel free to use [GitHub Discussions](https://github.com/aquasecurity/tracee/discussions)


## Video Content

If you prefer a video version of the Kubernetes installation guide, have a look at the following video:

Getting started with eBPF in Kubernetes - Tracee Installation Guide 

[](https://youtu.be/YQdEvf2IS9k?si=LhQM0CI8_QKvOCeK)

---

### Docs Install Lsm Support (`docs/docs/install/lsm-support.md`)

# LSM BPF Support

Linux Security Module (LSM) BPF allows eBPF programs to attach to security hooks for **enforcing security policies**. Unlike tracing, LSM programs can **allow or deny** security-sensitive operations like file access, process execution, and network connections. Most systems don't support this feature.

## Quick Check

Check if your system supports LSM BPF:

```bash
# Build the LSM check tool
make lsm-check

# Check LSM support (requires root)
sudo ./dist/lsm_check

# Quiet mode (exit code only)  
sudo ./dist/lsm_check --quiet
```

**Exit codes:**
- `0`: Supported ✅
- `1`: Not supported (normal) ⚠️  
- `2`: Error (insufficient privileges or other issues) ❌

## Requirements

### Kernel Version
- **x86_64**: Linux 5.7+ 
- **ARM64**: Linux 6.4+ (critical - earlier versions don't work)

### Configuration
Your kernel needs:
```bash
CONFIG_BPF_LSM=y               # BPF LSM support
CONFIG_LSM="...,bpf"           # Include bpf in LSM list
```

Although `CONFIG_BPF_LSM` is often enabled in many popular distributions, the `CONFIG_LSM` setting may not include "bpf" in its list, which means BPF LSM support could still be unavailable.

**ARM64 specific requirement:**
For ARM64 systems, you can check if LSM BPF support is possible by verifying:
```bash
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
```

This configuration option indicates whether the kernel has the ftrace direct call infrastructure required for ARM64 LSM BPF. If this option is not enabled or not available, LSM BPF will not work regardless of other settings (attachment of LSM programs will result `ENOTSUP` error).

### Boot Parameters

LSM modules are enabled through kernel boot parameters, which **override** the compile-time `CONFIG_LSM` setting. The `lsm=` parameter specifies which security modules to load at boot time.

**Important**: Boot parameters take precedence over `CONFIG_LSM`. If no `lsm=` parameter is specified, the kernel uses the compiled-in `CONFIG_LSM` default.

Verify current LSM configuration:
```bash
cat /sys/kernel/security/lsm
```

## Platform-Specific Notes

### ARM64 Support Issues

**Important**: ARM64 LSM BPF was broken for 3 years (Linux 5.7-6.3) due to missing kernel infrastructure.

**Fix commits:**
- **BPF Trampoline**: `efc9909fdce0` - "bpf, arm64: Add bpf trampoline for arm64" (by Xu Kuohai, Huawei)
- **Ftrace Direct Calls**: "Add ftrace direct call for arm64" patch series (by Florent Revest, Google) - merged in Linux 6.4

For detailed technical analysis of the ARM64 limitations and the underlying ftrace infrastructure issues, see: [Exploring BPF LSM support on aarch64 with ftrace](https://www.exein.io/blog/exploring-bpf-lsm-support-on-aarch64-with-ftrace)

### x86_64 Support
Kernel support works reliably on all modern distributions with kernel 5.7+. However, **BPF LSM is rarely enabled** in standard configurations.

## Why It Might Not Work

**Common reasons LSM BPF fails:**

1. **Kernel too old** (especially ARM64 < 6.4, but also < 5.7 in x86)
2. **CONFIG_BPF_LSM not enabled** in kernel
3. **The bpf LSM module is not supported** (i.e., `bpf` is missing from the `lsm=` boot parameter and/or from the `CONFIG_LSM` kernel configuration)

## Bottom Line

LSM BPF is a **security policy enforcement** feature that allows custom security rules. Tracee works fine without it. 

**Most systems support LSM infrastructure but BPF LSM is rarely enabled in standard configurations** - this is normal and expected.

---

### Docs Install Mac Faq (`docs/docs/install/mac-faq.md`)

# Tracee on Apple Mac FAQ

## Does Tracee run on MacOS?

No. Tracee runs only on Linux operating system. But you can run a Linux VM on your Mac, and then run Tracee in it. Please note though, that with this setup Tracee will only have visibility to the Linux VM it is running in, and not the host Mac machine.

## Does Tracee run on Apple Silicon?

Apple Silicon (a.k.a `M#` CPUs) utilizes the ARM64 CPU architecture. Given that [Tracee is compatible with ARM64](./prerequisites.md#processor-architecture), it should run on Apple Silicon as well.

## Does Tracee run in a Linux container on MacOS?

Yes. It is very common to install a container engine (e.g Docker Desktop) on MacOS and still be able to run Linux containers. This works by creating a Linux virtual machine inside your MacOS and running the container engine and all your containers from inside it. Note that with this setup Tracee will only have visibility to that Linux VM which is dedicated for the container engine, and not to your Mac machine. If you wanted to run something and see how Tracee reacts to it, you can run it as another container which should be visible to Tracee.

## Linuxkit

[linuxkit](https://github.com/linuxkit/linuxkit) is a popular Linux distribution used by Docker Desktop and other solutions.  
Linux kit does not have the Kernel Symbols Table feature properly configured by default, which is a [prerequisite](./prerequisites.md#kernel-symbols) for running Tracee.
It is technically possible to enable kernel symbols in linuxkit but an easier solution would be to  disable [those Tracee events that depend on kernel symbols](./prerequisites.md#kernel-symbols).

---

### Docs Install Os Requirements (`docs/docs/install/os-requirements.md`)

# Special: Overriding OS needed files

Tracee supports eBPF CO-RE (Compile Once - Run Everywhere) technology and,
because of that, it might need some information about the Linux OS it is
running at. Tracee, through libbpfgo, must have access to /etc/os-release file
AND either /boot/config-$(uname-r) OR /proc/config.gz files (KernelConfig API
at helpers/kernel_config).

> Note that, despite having this need, tracee will try to execute as CO-RE eBPF
> program in any environment it is executed.

## OS-RELEASE

Tracee will show you collected information about the running Linux OS with the
`--logging level=debug` argument:

```console
sudo ./dist/tracee --logging level=debug --scope uid=1000 --scope pid=new --events execve
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

BUT **os-release file might not exist**:

Because you're running in a distribution that does not have /etc/os-release, or
because you're running inside a container that does not support it, you may
face the following error:

```console
sudo ./dist/tracee --logging level=debug --scope uid=1000 --scope pid=new --events execve
```

```text
{"level":"debug","ts":1670976530.5685039,"msg":"osinfo", "warning: os-release file could not be found","error","open /etc/os-release: no such file or directory","pkg":"urfave","file":"urfave.go","line":33}
...
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
...

End of events stream
Stats: {EventCount:1 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
```

!!! Note
    But do note that **tracee shall continue working** (informing only the
    KERNEL_RELEASE OSInfo option).

If you need to override the underlying Linux OS information, because you're
running inside a container that already has /etc/os-release file, for example,
you may create another os-release file and inform tracee by using
LIBBPFGO's environment variable `LIBBPFGO_OSRELEASE_FILE`:

```console
sudo LIBBPFGO_OSRELEASE_FILE=/etc/os-release.orig ./dist/tracee --scope uid=1000 --scope pid=new --events execve
```

> If you're running tracee inside a docker container, you can simply bind mount
> `/etc/os-release` from the host as `/etc/os-release-host` into the guest.

## KCONFIG

Tracee needs access to kconfig file (/proc/config.gz OR /boot/config-$(uname -r)) in order to:

1. Check if the kernel of your running environment supports needed eBPF features
2. Provide kconfig variables to its eBPF counter-part (so eBPF program take decisions)

!!! Warning
    Tracee **should NOT fail** when it cannot find a kconfig file or needed options:
    
    - **missing kconfig file**

    ```console
    sudo ./dist/tracee --logging level=debug --scope uid=1000 --scope pid=new --events execve
    ```

    ```json
    {"level":"debug","ts":1670976875.7735798,"msg":"osinfo","VERSION":"\"20.04.5 LTS (Focal Fossa)\"","ID":"ubuntu","ID_LIKE":"debian","PRETTY_NAME":"\"Ubuntu 20.04.5 LTS\"","VERSION_ID":"\"20.04\"","VERSION_CODENAME":"focal","KERNEL_RELEASE":"5.4.0-91-generic","ARCH":"x86_64","pkg":"urfave","file":"urfave.go","line":53}
    ...
    {"level":"warn","ts":1670976875.7762284,"msg":"KConfig: could not check enabled kconfig features","error":"could not read /boot/config-5.4.0-91-generic: stat /boot/config-5.4.0-91-generic: no such file or directory"}
    {"level":"warn","ts":1670976875.7762842,"msg":"KConfig: assuming kconfig values, might have unexpected behavior"}
    ...
    {"level":"debug","ts":1670976876.0801573,"msg":"KConfig: warning: assuming kconfig values, might have unexpected behavior","pkg":"initialization","file":"kconfig.go","line":30}
    TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
    ...
    ```

    - **missing kconfig options**

    ```json
    {"level":"warn","ts":1698759121.4432194,"msg":"KConfig: could not detect kconfig options","options":[...]}
    ```
    
    But do have in mind it is assuming some things from the host environment and
    its behavior might have inconsistencies.
    
    If you are running tracee in an environment that does not have a kconfig file
    (nor /proc/config.gz support), it is recommended that you provide the host
    kconfig file location to tracee through the `LIBBPFGO_KCONFIG_FILE` environment
    variable:

```console
sudo LIBBPFGO_KCONFIG_FILE=/boot/config-other -E ./dist/tracee --logging level=debug --scope uid=1000 --scope pid=new --events execve
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

> If you're running tracee inside a docker container, you can simply bind mount
> /boot/config-$(uname -r) from the host as /boot/config-$(uname -r) into the
> guest and inform that through the `LIBBPFGO_KCONFIG_FILE` environment
> variable.

!!! Attention
    In case no kconfig file is found, tracee takes some decisions blindly and
    it may give you unexpected errors.

---

### Docs Install Prerequisites (`docs/docs/install/prerequisites.md`)

# Prerequisites for running Tracee

Tracee is heavily dependent on Linux and does not support any other operating system.



## Distributions and Linux Flavors

- List of supported environments:

  - With BTFHUB (check BTF session below):
    - Centos 8 (kernel < 4.18.0-193.el8)
    - Debian 10 (buster) with v5.10 kernels
    - Oracle Linux 7 and 8 with v5.4 kernels
    - Red Hat 8 (kernel <= 4.18.0-147.57.1)
    - SuSe 15.3 (kernel < 5.3.18-150300.59.90)
  - Regular Build:
    - Amazon Linux 2
    - Centos 8 and newer
    - Debian 11 (bullseye) and newer
    - Fedora 38 and newer
    - Oracle Linux 7, 8 (v5.4 kernels, some might need btfhub) and newer
    - Red Hat 8 (newer v4.18 kernels) and newer
    - SuSe 15.3 (newer kernels) and newer
    - Possibly other distributions with recent kernels.
  - Cloud Environments:
    - Amazon EKS
    - Azure AKS
    - GKE (Google Kubernetes Engine): 5.4, 5.10, 5.15 and newer
    - Minikube, Microk8s and other development environments

> This list is based on capabilities those versions provide and not necessarily
in tracee having all its features tested on each of them. Please provide
feedback if you face any issues in one of those environments.

## Kernel version

Tracee requires Linux kernel version 5.4 or newer. For RHEL-based distributions, kernel 4.18 (RHEL 8) is supported due to backported features.

You can check [kernel.org](https://kernel.org) for current supported kernels. In
addition to upstream kernels, most distributions' long-term supported kernels are
supported as well.

## BTF

Tracee needs low-level type information about the running kernel. Most modern
Linux distributions ship with the [BTF](https://www.kernel.org/doc/html/latest/bpf/btf.html)
feature that exposes this information.  

To test if this feature is enabled in your environment, check if
`/sys/kernel/btf/vmlinux` exists. If absent, you might need to upgrade to a
newer OS version, or contact your OS provider.

## Kernel symbols

Certain Tracee events require access to the Kernel Symbols Table, a feature
present in many Linux distributions.

To test if this feature is enabled in your environment, check if
`/proc/kallsyms` exists. If absent, contact your OS provider.

Alternatively, you can disable the following events which depends on kallsyms:

- `dirty_pipe_splice` (detects dirty pipe vulnerability - CVE-2022-0847)
- `hooked_syscall` (detects system call interception technique)
- `hidden_kernel_module` (detects hidden kernel modules technique)
- `hooked_proc_fops` (detects procfs file operations interception technique)
- `print_net_seq_ops` (related hooked_seq_ops event)
- `hooked_seq_ops` (detects network packets interception technique)
- `print_mem_dump` (allows memory dumping from symbols to signatures can use)

For more information and help about kernel symbols, please see [here](./kernel-symbols.md).

## OS information

Tracee will try to probe the running OS and kernel to detect available
capabilities. For this, it needs access to some standard informative files:

- For OS information please make sure the file `/etc/os-release` is available.
- For Kernel information please make sure one of the files `/boot/config-$(uname-r)` OR `/proc/config.gz` is available.

For more information and help about OS info files, please see [here](./os-requirements.md).

## Process capabilities

Tracee needs non-trivial capabilities to instrument the kernel. The easiest way
is run Tracee as "privileged" or "root".  

If you want to run Tracee with "least privileges", here are the required
capabilities and justifications:

- Manage eBPF maps limits (`CAP_SYS_RESOURCE`)
- Load and Attach eBPF programs:
    - `CAP_BPF`+`CAP_PERFMON` for recent kernels (>=5.8) where the kernel perf paranoid value in `/proc/sys/kernel/perf_event_paranoid` is equal to 2 or less
    - or `CAP_SYS_ADMIN` otherwise
- `CAP_SYS_PTRACE` (to collect information about processes)
- `CAP_NET_ADMIN` (to load cgroup_skb BPF programs for network packet capture)
- `CAP_SETPCAP` (if given - used to reduce bounding set capabilities)
- `CAP_SYSLOG` (to access kernel symbols through /proc/kallsyms)
- On some environments (e.g. Ubuntu) `CAP_IPC_LOCK` might be required as well.
- On cgroup v1 environments, `CAP_SYS_ADMIN` is recommended if running from a container in order to allow tracee to mount the cpuset cgroup controller.

For more information and help about process capabilities, please see
[here](./capabilities.md).

## LSM BPF Support

Certain Tracee features might depend on Linux Security Module (LSM) BPF support, which allows attaching BPF programs to LSM hooks for enhanced security monitoring.

To check if LSM BPF is supported in your environment, build and use the provided check tool:
```bash
# Build the LSM check tool
make lsm-check

# Check LSM support (requires root privileges)
sudo ./dist/lsm_check

# Quiet mode (just exit code)
sudo ./dist/lsm_check --quiet
```

Exit codes:

- `0`: LSM BPF is supported ✅
- `1`: LSM BPF is not supported (expected on most systems) ⚠️  
- `2`: Check failed due to insufficient privileges or other errors ❌

Note: LSM BPF requires specific kernel configuration and is not available on all systems. Tracee will function normally without LSM BPF support, but some advanced security features may be limited.

!!! tip "Having Issues?"
    If you encounter problems during installation or setup, check our [Troubleshooting Guide](../troubleshooting.md) for common solutions.

## Processor architecture

Tracee supports x86 and arm64 processors.

---

### Docs Outputs Event Structure (`docs/docs/outputs/event-structure.md`)

# Event Output Structure

Tracee outputs events using the `v1beta1.Event` protobuf structure, providing a stable, well-documented format for event data across all output formats (JSON, Go templates, webhooks, etc.).

## Event Structure Overview

The event structure is defined in the [API protobuf definitions](https://github.com/aquasecurity/tracee/blob/main/api/v1beta1/event.proto) and consists of several key components:

### Core Event Fields

```protobuf
message Event {
    google.protobuf.Timestamp timestamp = 1;  // Event timestamp
    EventId id = 2;                           // Event ID (enum)
    string name = 3;                          // Event name
    Policies policies = 4;                    // Matched policies
    Workload workload = 5;                    // Process/container context
    repeated EventValue data = 6;             // Event-specific data
    Threat threat = 7;                        // Threat information (detectors)
    Event detected_from = 8;                  // Source event (for detector events)
}
```

### Workload Context

Contains information about the process, container, and Kubernetes context:

```protobuf
message Workload {
    Process process = 1;      // Process information
    Container container = 2;  // Container information  
    Pod pod = 3;             // Kubernetes pod information
}
```

### Process Information

```protobuf
message Process {
    Thread thread = 1;                    // Thread details (name, TID, etc.)
    google.protobuf.UInt32Value pid = 2;  // Process ID
    User real_user = 3;                   // Real user ID
    Executable executable = 4;            // Executable path and metadata
    repeated Process ancestors = 5;       // Process ancestors
    // ... other fields
}
```

## JSON Output Example

```json
{
  "timestamp": {
    "seconds": "1732454400",
    "nanos": 123456789
  },
  "id": "sched_process_exec",
  "name": "sched_process_exec",
  "policies": {
    "matched": ["default-policy"]
  },
  "workload": {
    "process": {
      "thread": {
        "name": "bash",
        "start_time": {
          "seconds": "1732454350"
        },
        "unique_id": 1234567890,
        "host_tid": 12345,
        "tid": 12345,
        "syscall": "execve"
      },
      "pid": {
        "value": 12345
      },
      "real_user": {
        "id": {
          "value": 1000
        }
      },
      "executable": {
        "path": "/usr/bin/bash"
      },
      "ancestors": [
        {
          "unique_id": 9876543210,
          "host_pid": 1234,
          "pid": 1234
        }
      ]
    }
  },
  "data": [
    {
      "name": "pathname",
      "str": "/usr/bin/ls"
    },
    {
      "name": "argv",
      "str_array": {
        "value": ["ls", "-la", "/tmp"]
      }
    }
  ]
}
```

## Threat Detection Events

Events generated by detectors include additional threat information:

```json
{
  "timestamp": {...},
  "id": 6003,
  "name": "FILE_MODIFICATION",
  "policies": {
    "matched": ["file-modification-test"]
  },
  "workload": {...},
  "threat": {
    "name": "File Modification Test",
    "description": "Instrumentation events E2E Tests: File Modification",
    "mitre": {
      "tactic": {},
      "technique": {}
    },
    "properties": {
      "id": "TRC-5",
      "signatureID": "FILE_MODIFICATION"
    }
  },
  "detected_from": {
    "id": 776,
    "name": "file_modification",
    "data": [...]
  }
}
```

## Migration Guide for Template Users

If you're using Go templates, update your field references:

### Field Name Changes

| Old Field (trace.Event) | New Field (v1beta1.Event) | Notes |
|-------------------------|---------------------------|-------|
| `.EventName` | `.name` | String, event name |
| `.Timestamp` | `.timestamp.seconds` | Unix timestamp in seconds |
| - | `.timestamp.nanos` | Nanosecond fraction |
| `.ProcessName` | `.workload.process.thread.name` | Process/thread name (comm) |
| `.ProcessID` | `.workload.process.pid.value` | Wrapped uint32 |
| `.UserID` | `.workload.process.real_user.id.value` | Wrapped uint32 |
| `.Args` | `.data` | Array of EventValue |
| `.MatchedPolicies` | `.policies.matched` | Array of policy names |
| `.Container.ID` | `.workload.container.id` | Container ID |
| `.Kubernetes.PodName` | `.workload.pod.name` | Pod name |

### Template Examples

{% raw %}
**Before:**
```go
Event: {{.EventName}}
Process: {{.ProcessName}} (PID: {{.ProcessID}})
Time: {{.Timestamp}}
{{range .Args}}
  {{.Name}}: {{.Value}}
{{end}}
```

**After:**
```go
Event: {{.name}}
Process: {{if .workload}}{{if .workload.process}}{{if .workload.process.thread}}{{.workload.process.thread.name}}{{end}}{{end}}{{end}} (PID: {{if .workload}}{{if .workload.process}}{{if .workload.process.pid}}{{.workload.process.pid.value}}{{end}}{{end}}{{end}})
Time: {{if .timestamp}}{{.timestamp.seconds}}{{end}}
{{range .data}}
  {{.name}}: {{.value}}
{{end}}
```

**Simplified with better null handling:**
```go
{{- define "processName" -}}
{{- if and .workload .workload.process .workload.process.thread -}}
{{.workload.process.thread.name}}
{{- else -}}
<unknown>
{{- end -}}
{{- end -}}

{{- define "pid" -}}
{{- if and .workload .workload.process .workload.process.pid -}}
{{.workload.process.pid.value}}
{{- else -}}
0
{{- end -}}
{{- end -}}

Event: {{.name}}
Process: {{template "processName" .}} (PID: {{template "pid" .}})
Time: {{.timestamp.seconds}}.{{.timestamp.nanos}}
{{range .data}}
  {{.name}}: {{.value}}
{{end}}
```
{% endraw %}

### Accessing Event Data

Event-specific data is in the `.data` array. Each item has a `.name` and a typed `.value` field:

{% raw %}
```go
{{range .data}}
  {{if eq .name "pathname"}}
    File: {{.str}}
  {{else if eq .name "fd"}}
    FD: {{.int32}}
  {{else if eq .name "count"}}
    Count: {{.u_int64}}
  {{end}}
{{end}}
```
{% endraw %}

## Event Value Types

The `.data` array contains `EventValue` messages with typed union fields:

- `.str` - String value
- `.int32`, `.int64` - Signed integers
- `.u_int32`, `.u_int64` - Unsigned integers
- `.bool` - Boolean value
- `.bytes` - Raw bytes
- `.str_array` - String array
- `.u_int64_array` - Uint64 array
- `.sockaddr` - Socket address (IP, port)
- `.credentials` - User credentials
- `.ipv4`, `.ipv6` - IP packet metadata
- `.tcp`, `.udp`, `.icmp`, `.icmpv6` - Protocol-specific data
- `.dns_questions`, `.dns_responses` - DNS data
- `.packet_metadata` - Packet direction (ingress/egress)

## Protobuf Definition

For the complete and authoritative event structure, see:

- [event.proto](https://github.com/aquasecurity/tracee/blob/main/api/v1beta1/event.proto) - Main event structure
- [event_data.proto](https://github.com/aquasecurity/tracee/blob/main/api/v1beta1/event_data.proto) - EventValue types
- [workload.proto](https://github.com/aquasecurity/tracee/blob/main/api/v1beta1/workload.proto) - Workload/process context

## Backwards Compatibility

**Signature Development:** Signatures still work with the internal `trace.Event` structure via `protocol.Event.Payload`. No changes are needed for signature code.

**Output Only:** The v1beta1.Event structure applies only to output formats (JSON, templates, gRPC, webhooks).

## See Also

- [Output Formats](./output-formats.md) - How to format event output
- [Go Templates](./output-formats.md#go-template) - Using event fields in templates
- [Output Overview](./index.md) - Output system overview
- [Events Documentation](../events/index.md) - Available event types

---

### Docs Outputs Logging (`docs/docs/outputs/logging.md`)

# Tracee Logs

This section showcases how to configure diagnostic logs. The information provided can be used to troubleshoot Tracee.

For complete details on all logging options, see the [logging flag reference](../flags/logging.1.md).

## Configuration

### Log Level

Configure the log severity level:

**CLI:**
```console
tracee --logging level=debug
```

**Config file:**
```yaml
logging:
    level: debug
```

Available log levels: `debug`, `info` (default), `warn`, `error`, `fatal`

### Log File

Redirect logs to a file:

**CLI:**
```console
tracee --logging file=/tmp/tracee.log
```

**Config file:**
```yaml
logging:
    level: debug
    file: /tmp/tracee.log
```

### Log Aggregation

Aggregate logs for a given interval (default: 3s) to delay output:

**CLI:**
```console
tracee --logging aggregate --logging aggregate.flush-interval=10s
```

**Config file:**
```yaml
logging:
    aggregate:
        enabled: true
        flush-interval: 10s
```

The flush-interval defines how often Tracee logs will be forwarded.

## Log Filtering

### Filter by Message Content

Filter logs which message contains specified words:

**CLI:**
```console
tracee --logging filters.include.msg=foo,bar
```

**Config file:**
```yaml
logging:
    filters:
        include:
            msg: 
                - foo
                - bar
```

### Filter by Regular Expression

Filter logs using regular expressions against messages:

**CLI:**
```console
tracee --logging filters.include.regex='^pattern-one'
```

**Config file:**
```yaml
logging:
    filters:
        include:
            regex: 
                - ^pattern-one
```

### Filter by Package

Filter logs originating from a specific package:

**CLI:**
```console
tracee --logging filters.include.pkg=core
```

**Config file:**
```yaml
logging:
    filters:
        include: 
            pkg:
                - core
```

### Filter by File

Filter logs originating from a specific file:

**CLI:**
```console
tracee --logging filters.exclude.file=/pkg/cmd/flags/logger.go
```

**Config file:**
```yaml
logging:
    filters:
        include: 
            file: 
                - /pkg/cmd/flags/logger.go
```

### Filter by Log Level

Filter logs based on their severity level:

**CLI:**
```console
tracee --logging filters.include.level=error
```

**Config file:**
```yaml
logging:
    filters:
        include: 
            level:
                - error
```

### Filter libbpf Logs

Filter logs originating from libbpf:

**CLI:**
```console
tracee --logging filters.include.libbpf
```

**Config file:**
```yaml
logging:
    filters:
        include:
            libbpf: true
```

## Include vs Exclude

All `filters` options can be used with either `include` or `exclude` to achieve opposite behavior.

**Include example** - only show error logs:
```yaml
logging:
    filters:
        include:
            level:
                - error
```

**Exclude example** - hide error logs:
```yaml
logging:
    filters:
        exclude:
            level:
                - error
```

You can combine include and exclude filters to create complex filtering rules. For more examples, see the [logging flag reference](../flags/logging.1.md).

---

### Docs Outputs Output Formats (`docs/docs/outputs/output-formats.md`)

# Output Formats

Configure how Tracee events are serialized and displayed. Each destination can use a different format.

## Overview

Tracee supports three output formats:

- **JSON** - Machine-readable format for log aggregation and SIEM integration
- **Table** - Human-readable terminal output for debugging and development
- **Go Template** - Custom formatting using Go template syntax for flexible output

The format is configured per destination. See the [output flag reference](../flags/output.1.md) for complete destination configuration.

## Format Types

### JSON

JSON format outputs events as structured JSON objects, one per line. This is the recommended format for production use, log aggregation, and integration with monitoring tools.

**Example configuration:**

```yaml
output:
  destinations:
    - name: json_out
      format: json
      path: /var/log/tracee/events.json
```

**CLI:**

```bash
tracee --output destinations.json_out.format=json
```

**Use cases:**
- Integration with SIEM systems (Splunk, Elasticsearch, etc.)
- Structured logging for analysis
- Machine parsing and automation

!!! Tip
    Pipe JSON output to [jq](https://jqlang.github.io/jq/) for powerful filtering and transformation:
    ```bash
    tracee --output destinations.stdout.format=json | jq '.name'
    ```

### Table

Table format provides human-readable output formatted as a table in the terminal. This is the **default format** for file destinations when no format is specified.

**Example configuration:**

```yaml
output:
  destinations:
    - name: table_out
      format: table
```

**CLI:**

```bash
tracee --output destinations.table_out.format=table
```

**Use cases:**
- Interactive debugging and development
- Quick manual inspection of events
- Terminal-based monitoring

### Go Template

Go template format allows complete control over output formatting using Go's template syntax. Templates access Tracee's `v1beta1.Event` protobuf structure.

**Example configuration:**

```yaml
output:
  destinations:
    - name: custom_out
      format: gotemplate=/path/to/template.tmpl
      path: /var/log/tracee/custom.log
```

**CLI:**

```bash
tracee --output destinations.custom_out.format=gotemplate=/path/to/template.tmpl
```

#### Template Data Structure

Templates have access to the complete `v1beta1.Event` protobuf structure defined in the [API protobuf definitions](https://github.com/aquasecurity/tracee/blob/main/api/v1beta1/event.proto).

**Common event fields:**

| Field | Description |
|-------|-------------|
| `.timestamp` | Event timestamp (`.seconds` and `.nanos`) |
| `.id` | Event ID (protobuf enum) |
| `.name` | Event name (string) |
| `.policies.matched` | Array of matched policy names |
| `.workload.process.thread.name` | Process/thread name |
| `.workload.process.pid.value` | Process ID |
| `.workload.process.real_user.id.value` | User ID |
| `.workload.process.executable.path` | Executable path |
| `.data` | Array of event-specific data fields |
| `.threat` | Threat information (for detections) |

**For detection events:**

| Field | Description |
|-------|-------------|
| `.threat.name` | Threat/signature name |
| `.threat.description` | Threat description |
| `.threat.properties.signatureID` | Signature ID |
| `.detected_from` | Underlying event that triggered detection |

#### Template Functions

Go templates support helper functions from [Sprig](http://masterminds.github.io/sprig/) for string manipulation, formatting, and more.

**Example template:**

```go
{% raw %}
{{- range . -}}
Time: {{ .timestamp.seconds }} | Event: {{ .name }} | PID: {{ .workload.process.pid.value }}
{{- end }}
{% endraw %}
```

**Use cases:**
- Custom log formats for specific tools
- Simplified output with only needed fields
- Integration with legacy systems requiring specific formats

## Format Per Destination

Different destinations can use different formats in the same configuration:

```yaml
output:
  destinations:
    - name: json_file
      type: file
      format: json
      path: /var/log/tracee/events.json

    - name: table_console
      type: file
      format: table
      path: stdout

    - name: webhook
      type: webhook
      format: gotemplate=/etc/tracee/webhook.tmpl
      url: https://siem.example.com/events
```

## Default Format

The default format depends on the destination type:

| Destination Type | Default Format |
|------------------|----------------|
| `file` | `table` |
| `webhook` | `json` |
| `forward` | `json` |

## See Also

- [Output Flag Reference](../flags/output.1.md) - Complete destination configuration
- [Output Overview](./index.md) - Output system overview
- [Streams](./streams.md) - Filter and route events to different destinations
- [Event Structure](./event-structure.md) - Understanding event fields

---

### Docs Outputs Sorting Events (`docs/docs/outputs/sorting-events.md`)

# Sorting Events

The sorting feature sorts incoming events from the BPF programs chronologically to ensure events are output in the order they occurred.

## Enabling Sorting

Enable event sorting using the `sort-events` option:

```console
tracee --output sort-events
```

Or in a configuration file:

```yaml
output:
  sort-events: true
```

!!! Information
    There are **3 known** sources of **sorting issues**:
    
      1. In perf buffer, events are read in round robing order from CPUs buffers
         (and not according to invocation time).
    
      2. Syscall events are invoked after internal events of the syscall (though
         the syscall happened before the internal events).
    
      3. Virtual CPUs might enter sleep mode by host machine scheduler and send
         events after some delay.


## How Sorting Works

To address the perf buffer issues, events are **divided into queues according to the source CPU**. This way the events are almost sorted (except for syscalls). Syscall events are inserted to their correct chronological position manually.

This way, **all events which occurred before the last event** of the **most delaying CPU** can be sent forward with guaranteed order.

To make sure **syscall events are not missed** when sending, a **small delay** is needed. Lastly, to address the **vCPU sleep issue** (which might cause up to 2 events to be received with a delay), the events need to be sent **after a delay which is larger than the maximum possible vCPU sleep time** (which is just an increase of the syscall events delay).

## Sorting Algorithm

To summarize the algorithm's main logic, here is a textual simulation of the operation (assume that 2 scheduler ticks are larger than the maximum possible vCPU sleep time):  

Tn = Timestamp (n == TOD)  
\#m = Event's Source CPU  

1. Initial State

    ```text
           [ CPU 0 ]    [ CPU 1 ]    [ CPU 2 ]
      HEAD    T1           T2           T4
              T3           T5
              T6
      TAIL    T8
    ```

2. Scheduler Tick #1

    ```text
    Incoming events: T9#1, T11#2, T13#1, T10#2, T12#2
    
    Queues state after insert:
           [ CPU 0 ]    [ CPU 1 ]    [ CPU 2 ]
      HEAD    T1           T2           T4
              T3           T5           T10 +
              T6           T9  +        T11 +
      TAIL    T8           T13 +        T12 +
    
      - No event sent.
      - Oldest timestamp = T1.
      - T8 is oldest timestamp in most recent timestamps.
      - In 2 ticks from now: send all events up to T8.
      - Bigger timestamps than T8 (+) will be sent in future scheduling.
    ```

3. Scheduler Tick #2

    ```text
    Incoming events: T7#0, T22#1, T23#2, T20#0, T25#1, T24#2, T21#0
    
    Queues state after insert:
           [ CPU 0 ]    [ CPU 1 ]    [ CPU 2 ]
      HEAD    T1  ^        T2  ^        T4  ^
              T3  ^        T5  ^        T10
              T6  ^        T9           T11
              T7  +^       T13          T12
              T8  ^        T22 +        T23 +
              T20 +        T25 +        T24 +
      TAIL    T21 +
    
      - No event sent.
      - Oldest timestamp = T1.
      - T21 is oldest timestamp in most recent timestamps.
      - In 2 ticks from now: send all events up to T21.
      - T8 is previous oldest timestamp in most recent timestamps.
      - Next tick: send all events up to T8.
      - Bigger timestamps than T21 (+) will be sent in future scheduling.
    ```

4. Scheduler Tick #3

    ```text
    Incoming events: T30#0, T34#1, T35#2, T31#0, T36#2, T32#0, T37#2, T33#0, T38#2, T50#1, T51#1
    
    Queues state after insert:
           [ CPU 0 ]    [ CPU 1 ]    [ CPU 2 ]
      HEAD    T20 ^        T9  ^        T10 ^
              T21 ^        T13 ^        T11 ^
              T30 +        T22          T12 ^
              T31 +        T23          T24
              T32 +        T25          T35 +
              T33 +        T34 +        T36 +
                           T50 +        T37 +
       TAIL                T51 +        T38 +
    
      - Max sent timestamp = T8.
      - Oldest timestamp = T9.
      - T33 is oldest timestamp in most recent timestamps.
      - In 2 ticks from now: send all events up to T33.
      - T21 is previous oldest timestamp in most recent timestamps.
      - Next tick: send all events up to T21.
      - Bigger timestamps than T33 (+) will be sent in future scheduling.
    ```

---

### Docs Outputs Streams (`docs/docs/outputs/streams.md`)

# Streams

Streams are a Tracee mechanism that let you redirect events to different destinations with custom filters.

Using streams allows you to build complex event-routing pipelines, applying filters based on events and policies. Stream configuration is straightforward:

**yaml**

```yaml
output:
    destinations:
    - name: json_destination
      format: json

    streams:
    - name: tracing
      destinations: ["json_destination"]
      filters:
        events: []
        policies: []
      buffer:
        mode: block # or drop
        size: 1024
```

**cli**

```bash
tracee  --output destinations.json_destination.format=json \
        --output streams.tracing.destinations=json_destination \
        --output streams.tracing.buffer.mode=block \
        --output streams.tracing.buffer.size=1024
```

A stream must always reference one or more destinations.  
When a stream contains multiple destinations, every event emitted by the stream is broadcast to **all** destinations.  
Be aware that if one destination is slow, it may delay all others.

**Important:** A stream can be use multiple destinations, but a destination cannot be reused in multiple streams.

Streams are first-class citizens in Tracee. If you [declare a destination](./output-formats.md) without explicitly creating a stream, Tracee automatically creates an implicit filter-less stream to emit events to that destination.

!!! Tip
    A common pattern is to use streams to send different verbosity levels or event types to different destinations.

## Example

**yaml**

```yaml
output:
    destinations:
    - name: json_destination
      format: json
    
    - name: webhook_destination
      format: json
      url: http://localhost:8080/processes?timeout=5s
    
    - name: fluent_destination
      format: json
      url: http://localhost:2222?tag=severe

    streams:
    - name: tracing
      destinations: 
        - json_destination

    - name: processes
      destinations: 
        - webhook_destination
      filters:
        events:
          - sched_process_exec
          - sched_process_fork
          - sched_process_exit
      buffer:
        mode: drop
    
    - name: severe_events
      destinations:
        - fluent_destination
      filters:
        policies:
          - severe-events-policy
      buffer:
        mode: drop # using drop to avoid losing events
```

**cli**

```bash
tracee  --output destinations.json_destination.format=json \
        --output destinations.webhook_destination.format=json \
        --output destinations.webhook_destination.type=webhook \
        --output destinations.webhook_destination.url=http://localhost:8080/processes?timeout=5s \
        --output destinations.fluent_destination.format=json \
        --output destinations.fluent_destination.type=forward \
        --output destinations.fluent_destination.url=http://localhost:2222?tag=severe \
        --output streams.tracing.destinations=json_destination \
        --output streams.processes.destinations=webhook_destination \
        --output streams.processes.filters.events=sched_process_exec,sched_process_fork,sched_process_exit \
        --output streams.processes.buffer.mode=drop \
        --output streams.severe_events.destinations=fluent_destination \
        --output streams.severe_events.filters.policies=severe-events-policy \
        --output streams.severe_events.buffer.mode=drop
```

## CLI flags format

CLI flags have a the following structure: `--output streams.<stream_name>.<field>[.<subfield>]=<value>`. The following table describes all the available fields and subfields

### Available fields

| Field / Subfield | Usage                                                                                                               | Default |
| :--------------: | ------------------------------------------------------------------------------------------------------------------- | ------- |
| destinations     | list of destinations' names the stream refers to.                                                                   | []      |
| filters          | filters applied to the stream.                                                                                      | {}      |
| filters.events   | list of events' names to filters.                                                                                   | []      |
| filters.policies | list of policies' names to filter.                                                                                  | []      |
| buffer           | buffer settings.                                                                                                    | {}      |
| buffer.size      | number of maximum elements allowed in the stream buffer queue.                                                      | 1024    |
| buffer.mode      | `drop` of `block`. `drop` allows dropping events when the `buffer` is full. `block` does not allow events dropping. | block   |

## See Also

- [Output Flag Reference](../flags/output.1.md) - Complete output configuration
- [Output Overview](./index.md) - Output system overview
- [Output Formats](./output-formats.md) - Available event formats
- [Policies](../policies/index.md) - Policy-based filtering

---

### Docs Policies Rules (`docs/docs/policies/rules.md`)

# Rules

Rules are part of the Tracee Policy, `rules` let you define which events to trace.

`rules` have 2 sections: 

- events: let you define which events you want to trace.
- filters: enable you to refine the policy's scope.

Tracee supports many kinds of events to trace. You can find which events you can trace in the [Events section](../events/index.md).

- **NOTE:** It is possible to define multiple events within each policy.

Below is an example showcasing a policy:
```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-scope-filter
  annotations:
    description: sample scope filter
spec:
  scope:
    - global
  rules:
    - event: openat
      filters:
        - uid=1000
```
## Events

### Type of Events
The value of an `event` is the event name from the supported events.

For example: `syscall` event would be the `syscall` event name.

The [events](../events/index.md) section provides further information on the type of events that Tracee can trace.

### Threat-Based Detector Selection

Instead of specifying individual detector event names, you can select multiple detectors based on their threat metadata properties. This is especially useful for:

- Enabling all high-severity detectors without knowing their names
- Selecting detectors by MITRE ATT&CK framework categories
- Filtering by threat characteristics

#### Selecting by Severity

Select detectors based on their severity level (info, low, medium, high, critical):

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: critical-threats-only
  annotations:
    description: Enable all critical severity detectors
spec:
  scope:
    - global
  rules:
    - event: threat.severity=critical
```

You can also use comparison operators:

```yaml
rules:
  - event: threat.severity>=high  # Enable high and critical threats
```

#### Selecting by MITRE ATT&CK

Select detectors by MITRE ATT&CK technique or tactic:

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: process-injection-detectors
  annotations:
    description: Enable all process injection detectors
spec:
  scope:
    - global
  rules:
    - event: threat.mitre.technique=T1055
```

Or by tactic:

```yaml
rules:
  - event: threat.mitre.tactic=Defense Evasion
```

#### Selecting by Threat Name

Select detectors by exact threat name:

```yaml
rules:
  - event: threat.name=process_injection
```

#### Selecting by Detector Tags

Detectors can be categorized with tags. Select all detectors with a specific tag:

```yaml
rules:
  - event: containers        # All detectors with "containers" tag
  - event: malware          # All detectors with "malware" tag
```

#### Combining with Regular Events

Threat-based selection can be combined with regular event selection:

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: mixed-events
  annotations:
    description: Trace specific events and all critical threats
spec:
  scope:
    - global
  rules:
    - event: security_file_open
      filters:
        - data.pathname=/etc/*
    - event: threat.severity=critical
```

#### Available Threat Properties

| Property | Description | Example Values | Operators |
|----------|-------------|----------------|-----------|
| `threat.severity` | Severity level | info, low, medium, high, critical (or 0-4) | =, !=, <, >, <=, >= |
| `threat.mitre.technique` | MITRE technique ID | T1055, T1071 | =, != |
| `threat.mitre.tactic` | MITRE tactic name | Defense Evasion, Execution | =, != |
| `threat.name` | Threat identifier | process_injection | =, != |

**Note:** Detector selection based on threat properties is performed once when Tracee starts. Matching detectors are enabled; non-matching detectors are never loaded. Multiple rules in a policy are combined with OR logic.


## Filters

Filters enable you to refine the policy's scope by specifying conditions for particular events. This allows you to narrow down the criteria to precisely target the events you're interested in, ensuring that the policy applies only under specific circumstances.

Every `event` that is specified within the `rules` section supports three types of filters: `scope`, `data` and `return value`.

### Scope filters

Further refinement of the policy's scope is achievable through the application of scope filters:

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-scope-filter
  annotations:
    description: sample scope filter
spec:
  scope:
    - global
  rules:
    - event: sched_process_exec
      filters:
        - pid=1000
```

The scope filters supported are:

#### p, pid, processId

```yaml
event: sched_process_exec
filters:
    - pid=1000
```

#### tid, threadId

```yaml
event: sched_process_exec
filters:
    - tid=13819
```

#### ppid, parentProcessId

```yaml
event: sched_process_exec
filters:
    - ppid=1000
```

#### hostTid, hostThreadId

```yaml
event: sched_process_exec
filters:
    - hostTid=1000
```

#### hostPid

```yaml
event: sched_process_exec
filters:
    - hostPid=1000
```

#### hostParentProcessId

```yaml
event: sched_process_exec
filters:
    - hostParentProcessId=1
```

#### uid, userId

```yaml
event: sched_process_exec
filters:
    - uid=0
```

#### mntns, mountNamespace

```yaml
event: sched_process_exec
filters:
    - mntns=4026531840
```

#### pidns, pidNamespace

```yaml
event: sched_process_exec
filters:
    - pidns=4026531836
```

#### comm, processName

```yaml
event: sched_process_exec
filters:
    - comm=uname
```

#### hostName

```yaml
event: sched_process_exec
filters:
    - hostName=hostname
```

#### cgroupId

```yaml
event: sched_process_exec
filters:
    - cgroupId=5247
```

#### container

```yaml
event: sched_process_exec
filters:
    - container=66c2778945e29dfd36532d63c38c2ce4ed1
```

#### containerId

```yaml
event: sched_process_exec
filters:
    - containerId=66c2778945e29dfd36532d63c38c2ce4ed1
```

#### containerImage

```yaml
event: sched_process_exec
filters:
    - containerImage=ubuntu:latest
```

#### containerName  

```yaml
event: sched_process_exec
filters:
    - containerName=test
```

#### podName

```yaml
event: sched_process_exec
filters:
    - podName=daemonset/test
```

#### podNamespace

```yaml
event: sched_process_exec
filters:
    - podNamespace=production
```

#### podUid

```yaml
event: sched_process_exec
filters:
    - podUid=66c2778945e29dfd36532d63c38c2ce4ed16a002c44cb254b8e
```

        
### Data filter

Events contain data that can be filtered.

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-data-filter
  annotations:
    description: sample data filter
spec:
  scope:
    - global
  rules:
    - event: security_file_open
      filters:
        - data.pathname=/tmp*
```

Data fields can be found on the respective event definition, in this case [security_file_open](https://github.com/aquasecurity/tracee/blob/656eb976fbb66aba54c5f306019258e436d4814a/pkg/events/core.go#L11502-L11533) - be aware of possible changes to the definition linked above, so always check the main branch.

 Or the user can test the event output in CLI before defining a policy, e.g:

```console
tracee -e security_file_open --output json
```

```json
{"timestamp":1680182976364916505,"threadStartTime":1680179107675006774,"processorId":0,"processId":676,"cgroupId":5247,"threadId":676,"parentProcessId":1,"hostProcessId":676,"hostThreadId":676,"hostParentProcessId":1,"userId":131,"mountNamespace":4026532574,"pidNamespace":4026531836,"processName":"systemd-oomd","hostName":"josedonizetti-x","container":{},"kubernetes":{},"eventId":"730","eventName":"security_file_open","matchedPolicies":[""],"argsNum":6,"returnValue":0,"syscall":"openat","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":"/proc/meminfo"},{"name":"flags","type":"string","value":"O_RDONLY|O_LARGEFILE"},{"name":"dev","type":"dev_t","value":45},{"name":"inode","type":"unsigned long","value":4026532041},{"name":"ctime","type":"unsigned long","value":1680179108391999988},{"name":"syscall_pathname","type":"const char*","value":"/proc/meminfo"}]}
```

### Return value filter

Return values can also be filtered.

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: sample-return-value
  annotations:
    description: sample return value
spec:
  scope:
    - global
  rules:
    - event: close
      filters:
        - retval!=0
```

---

### Docs Policies Scopes (`docs/docs/policies/scopes.md`)

# Scopes

`scope` allows you to select the scope for tracing events by defining filters. 

## FILTER EXPRESSION

Filter expressions can be defined to operate on scope options or process metadata. Only events that match all filter expressions will be traced.

Multiple flags are combined with AND logic, while multiple values within a single flag are combined with OR logic when using the equals operator '='. Multiple values can be specified using ','.

### NUMERICAL EXPRESSION OPERATORS

The following numerical fields support the operators '=', '!=', '<', '>', '<=', '>=':

- uid: Select events from specific user IDs.
- pid: Select events from specific process IDs.

The following numerical fields only support the operators '=' and '!=':

- mntns: Select events from specific mount namespace IDs.
- pidns: Select events from specific process namespace IDs.
- tree: Select events that descend from specific process IDs.

NOTE: Expressions containing '<' or '\>' tokens must be escaped!

### STRING EXPRESSION OPERATORS

'=', '!='

Available for the following string fields:

- uts: Select events based on UTS (Unix Timesharing System) names.
- comm: Select events based on process command names.
- container: Select events from specific container IDs.
- executable: Select events based on the executable path.



### BOOLEAN OPERATOR (PREPENDED)

'!'

Available for the following boolean field:

- container: Select events based on whether they originate from a container or not.

## Supported Scopes

### global

Events are collected from the whole host:
```yaml
scope:
    - global
```

### uid

Events are collected from the specific user id:

```yaml
scope:
    - uid=0
```

### pid

Events are collected from the specific pid:

```yaml
scope:
    - pid=1000
```

### mntns

Events are collected from the mount namespace:

```yaml
scope:
    - mntns=4026531840
```

### pidns

Events are collected from the pid namespace:

```yaml
scope:
    - pidns=4026531836
```

### uts

Events are collected from uts namespace:

```yaml
scope:
    - uts=ab356bc4dd554
```

### comm

Events are collected from process named `uname`:

```yaml
scope:
    - comm=uname
```

### container

Events are collected only from containers:

```yaml
scope:
    - container
```

You can also filter for containers based on their state:

- **container=new**: Events are collected only from newly created containers (during container initialization):

```yaml
scope:
    - container=new
```

- **container=started**: Events are collected only from containers that have completed initialization and started running (post-entrypoint execution):

```yaml
scope:
    - container=started
```

Note: The negation `container!=started` is not supported due to race conditions with early container event recognition.

### not-container

Events are collected from everything but containers:

```yaml
scope:
    - not-container
```

### tree

Events are collected from process tree:

```yaml
scope:
    - tree=1000
```

### executable, exec

Events are collected from executable:

```yaml
scope:
    - executable=/usr/bin/dig
```

### follow

Events collected follow process children:

```yaml
scope:
    - follow
```

---

### Traceectl Commands Event (`docs/traceectl/commands/event.md`)

# Event Command Usage

The `event` command in **traceectl** is used for managing events within Tracee. It allows you to list, describe, enable, and disable various event types that Tracee can capture. Below is the usage guide for the `event` command and its subcommands.

## Usage

The `event` command is structured as follows:

```sh
traceectl event [subcommand] [flags]
```

## Subcommands

- **describe**: Retrieves detailed information about a specific **event** or **all the events**, including its fields, types, and other metadata.
  
  ```sh
  traceectl event describe EVENT
  ```

  - **`EVENT`**: The name of the event to describe (default is all events).
  - **`--format`**: Specifies the format (default is `table`).
  - **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)
  - **`--output`**: Specifies the output (default is `stdout`)

- **enable**: Enables capturing of a specific event type in Tracee.
  
  ```sh
  traceectl event enable <event_name>
  ```

  - **`EVENT`**: The name of the event to enable.
  - **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)
  - **`--output`**: Specifies the output (default is `stdout`)

- **disable**: Disables capturing of a specific event type in Tracee.
  
  ```sh
  traceectl event disable <event_name>
  ```

  - **`EVENT`**: The name of the event to disable.
  - **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)
  - **`--output`**: Specifies the output (default is `stdout`)

## Examples

- **List All Events in JSON Format**
  
  ```sh
  traceectl event describe --format json
  ```

- **Describe an Event**
  
  ```sh
  traceectl event describe execve
  ```

- **Enable an Event**
  
  ```sh
  traceectl event enable execve
  ```

- **Disable an Event**
  
  ```sh
  traceectl event disable execve
  ```

## Summary

The `event` command in traceectl is a powerful tool for managing Tracee's event capabilities. Use the, `describe`, `enable`, and `disable` subcommands to gain detailed insight and control over the events Tracee monitors.

---

### Traceectl Commands Metrics (`docs/traceectl/commands/metrics.md`)

# Metrics Command Usage

The `metrics` command in **traceectl** provides information about Tracee's performance and resource usage metrics. This command is helpful for monitoring how Tracee is functioning in real-time.

## Usage

The `metrics` command is structured as follows:

```sh
traceectl metrics 
```

- **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)

## Examples

- **Display Metrics in Table Format**
  
  ```sh
  traceectl metrics 
  ```

## Summary

The `metrics` command displays tracee's performance and resource usage metrics. Use this command to monitor Tracee's operational status.

---

### Traceectl Commands Stream (`docs/traceectl/commands/stream.md`)

# Stream Command Usage

The `stream` command in **traceectl** allows users to stream events directly from Tracee in real time. This command provides flexible output formats for better integration and readability.

## Usage

The `stream` command is structured as follows:

```sh
traceectl stream [flags] [policies]...
```

- **`[policies]`**: List of policies to stream (default is all policies)
- **`--format`**: Specifies the format (default is `table`).
- **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)
- **`--output`**: Specifies the output (default is `stdout`)

## Examples

- **Stream Events in JSON Format with a specified policies and a different unix socket**
  
  ```sh
  traceectl stream --format json --server /tmp/tracee.sock policy1 policy2
  ```

- **Stream Events to file**
  
  ```sh
  traceectl stream --output /path/to/file 
  ```

- **Stream Events in JSON Format**
  
  ```sh
  traceectl stream --format json
  ```

- **Stream Events in Table Format**
  
  ```sh
  traceectl stream --format table
  ```

## Summary

The `stream` command provides a real-time feed of Tracee events, allowing you to monitor system activity as it happens.

---

### Traceectl Commands Version (`docs/traceectl/commands/version.md`)

# Version Command

The `version` command in **traceectl** provides detailed information about the current version of the tool. This includes the version number, build date, and other relevant metadata.

## Usage

To display the version information, use the following command:

``` bash
traceectl version
```

- **`--server`**: Specifies the server unix socket path (default is `/var/run/tracee.sock`)

This command will output details such as:

- **Version Number**: The current version of traceectl.
- **Commit Hash**: The Git commit hash associated with the current build (if applicable).

### Example Output

``` bash
v0.22.0
```

### Summary

- **`traceectl version`**: Displays detailed version information.

Use this command to verify your version or to gather information for troubleshooting purposes.

---

### Traceectl Flags Format (`docs/traceectl/flags/format.md`)

# `format` Flag

The `--format` flag in **traceectl** is used to specify the output format for certain commands. Currently, this flag supports the following values for the `stream` and `event describe` commands:

- **`json`**: Outputs the data in JSON format, which is useful for automated processing or integration with other tools that consume JSON.
  
  Example:

  ```sh
  traceectl stream --format json
  ```

  In this example, the command lists all available events and outputs them in JSON format.

- **`table`**: Outputs the data in a tabular format, which is easier to read for users viewing the output directly in the terminal.
  
  Example:
  
  ```sh
  traceectl stream --format table
  ```

  In this example, the command streams events from Tracee and displays them in a table format, making it more human-readable.

The `--format` flag is helpful for customizing the output to meet different requirements, whether for readability or integration with other tools.

## Default Format

The default format for the `--format` flag is **table**. If no format is specified, the output will be displayed in a tabular format, which is more human-readable for most users.

---

### Traceectl Flags Output (`docs/traceectl/flags/output.md`)

# `output` Flag

The `--output` flag is used to specify the destination for the command's output. This flag can be set to **stdout** or a file location.

- **stdout**: This is the default output destination, which means that the command's output will be displayed on the terminal. This is convenient for users who want to see immediate results directly in their console.
  
  Example:

  ```sh
  traceectl stream --output stdout
  ```

  In this example, the command outputs the streamed events to the terminal.

- **File Output**: You can use the `--output` flag to direct the output to a specific file. This is useful if you want to save the output for later analysis or for documentation purposes.
  
  Example:

  ```sh
  traceectl stream --output file:/path/to/output.txt
  ```

  In this example, the command saves the streamed events to the file located at `/path/to/output.txt`. This is especially helpful for logging purposes or when working with large amounts of data that need to be stored for further processing.

---

### Traceectl Flags Policy (`docs/traceectl/flags/policy.md`)

# `policy` Flag

The `--policy` flag is used to specify the polices to include for the command's stream events. If this flag is set it must have a valid policy name loaded into tracee.

- **""**: This is the default policy, which means that the command's streamed events will display on the terminal all events capture by tracee. This is convenient for users who want to see immediate results directly in their console.
  
  Example:

  ```sh
  traceectl stream --policy policy1
  ```

  In this example, the command outputs the streamed events from a specific policy to the terminal.

- **Multi Policy**: You can use the `--policy` flag to specify multiple policies to include for the command's stream events. This is useful if you want to make batter analysis for different needs
  
  Example:

  ```sh
  traceectl stream --policy policy1 policy2
  ```

In this example, the command outputs the streamed events from a specific policies to the terminal. This is especially helpful for logging purposes or when working with large amounts of data that need to be sorted for further processing.

---

### Traceectl Flags Server (`docs/traceectl/flags/server.md`)

# `server` Flag

The `--server` flag in **traceectl** is used to specify the connection type that traceectl should use to communicate with the Tracee server. This connection type is **Unix socket** only.

- **Unix Socket**: This type of connection is generally used for local inter-process communication. It provides a secure and efficient means to connect to Tracee when both client and server are on the same machine.
  
  Example:

  ```sh
  traceectl --server /unix/socket/path.sock
  ```

  In this example, `/unix/socket/path.sock` is the Unix socket path where the Tracee server is listening. Using Unix sockets is beneficial for security and performance since it avoids the overhead associated with network communication.

---

### Docs Events Builtin Network Events (`docs/docs/events/builtin/network-events.md`)

Tracee offers a set of network events that makes it easy to trace network
activity in common protocols.

## Available network events

- [net_packet_ipv4](man/network/net_packet_ipv4.md)
- [net_packet_ipv6](man/network/net_packet_ipv6.md)
- [net_packet_tcp](man/network/net_packet_tcp.md)
- [net_packet_udp](man/network/net_packet_udp.md)
- [net_packet_icmp](man/network/net_packet_icmp.md)
- [net_packet_icmpv6](man/network/net_packet_icmpv6.md)
- [net_packet_dns](man/network/net_packet_dns.md)
- [net_packet_dns_request](man/network/net_packet_dns_request.md)
- [net_packet_dns_response](man/network/net_packet_dns_response.md)
- [net_packet_http](man/network/net_packet_http.md)
- [net_packet_http_request](man/network/net_packet_http_request.md)
- [net_packet_http_response](man/network/net_packet_http_response.md)

## Network Event Filtering

!!! Supported Attention
    For now it is **NOT** possible to filter the events through the
    header fields, but it **IS** possible, and recommended, to filter
    the events through **`src`**, **`dest`** fields. Not filtering
    network events might be hard to consume because of the amount of
    traced events.

Trace all TCP packets sent to port 80 anywhere, from any process:

```console
tracee --output json --events net_packet_tcp.data.dst_port=80
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

Trace all DNS packets received ONLY from Google DNS server '8.8.8.8':

```console
tracee --output json --events net_packet_dns.data.src=8.8.8.8

```

(only **systemd-resolved**, since all the other processes are resolving using local systemd-resolved server `127.0.1.1:53`):

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Network Based Detectors

It is possible to create Golang detectors for the network events. If you haven't read about how to create detectors, do it [HERE](../custom/golang.md).

!!! Examples Note
    Below is an example of how to create a detector for the `net_packet_dns` event. This same example is used by Tracee CI/CD tests and can be found at the [GitHub repository](https://github.com/aquasecurity/tracee/tree/main/detectors/e2e), together with other e2e detectors for network events.

1. **net_packet_dns** detector example

!!! Note
    For complete detector implementations, see the [detectors/e2e](https://github.com/aquasecurity/tracee/tree/main/detectors/e2e) directory and the [Detector API documentation](../../detectors/api-reference.md).

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Docs Events Builtin Security Events (`docs/docs/events/builtin/security-events.md`)

# Security Events

## Understanding Signatures in Tracee

In Tracee, a signature is a set of criteria designed to detect specific system
activities based on events such as syscalls, network interactions, and LSM hook
occurrences. When these foundational system events align with the conditions set
by a signature, Tracee generates a corresponding "security event." This process
enables Tracee to actively monitor and report potential security concerns
arising from observed system interactions.

## Security Model and Detection Boundaries

Tracee's detection capabilities vary significantly depending on the adversary:

- **Userspace Adversaries**: Tracee provides strong detection capabilities through eBPF's tamper-resistant architecture. Even root-privileged adversaries in userspace cannot bypass Tracee's monitoring.

- **Kernel-Level Adversaries**: Tracee provides best-effort detection using signature-based approaches. Rootkit detection signatures (such as syscall table hooking, hidden kernel modules, and proc filesystem hooking) may detect known techniques, but cannot guarantee detection against sophisticated kernel-level threats.

**Important**: If the kernel is compromised before Tracee starts, or if an adversary exploits kernel zero-day vulnerabilities, Tracee's detection capabilities cannot be guaranteed.

For a comprehensive understanding of Tracee's security boundaries, threat model, and limitations, please see the [Security Model](../../security-model.md) documentation.

## Functionality and Scope of Signatures

The signatures documented herein focus on key system operations. For instance,
one signature identifies attempts to manipulate the syscall tables or the
`/proc` filesystem, operations that are indicative of rootkit behaviors. Another
detects the dynamic introduction of new executables into the system, flagging
potential security issues. Upon a match, these signatures prompt Tracee to
produce a security event, capturing the specifics of the underlying event and
any associated implications.

## The Role of Security Events in Tracee

Security events play a critical role in maintaining system integrity. They
provide an analytical layer, translating raw events like syscalls into
actionable insights about potential threats or anomalies. With the power of
eBPF, Tracee efficiently monitors system activities in real-time, generating
security events that equip users with the information they need to assess and
respond to the state of their digital environments.

## Be Prepared!

For optimal utilization of Tracee and effective response to potential threats,
we strongly recommend readers to meticulously review each security event
documentation page.

A good understanding of what each signature detects will empower users to make
informed decisions and take appropriate actions when a security event arises.
Being well-versed in the nuances of each signature ensures that you're not just
alerted to risks, but also equipped to address them effectively.

## List of Default Security Events

| Name of Signature                                        | Description                                    |
|----------------------------------------------------------|------------------------------------------------|
| [Anti-Debugging Technique](man/security/anti_debugging.md) | Detects anti-debugging techniques.             |
| [ASLR Inspection](man/security/aslr_inspection.md) | Detects ASLR inspections.                      |
| [Cgroups notify_on_release File Modification](man/security/cgroup_notify_on_release_modification.md) | Monitors `notify_on_release` file changes in cgroups.|
| [Cgroups Release Agent File Modification](man/security/cgroup_release_agent_modification.md) | Detects changes to the cgroup release_agent.  |
| [Core Dumps Config File Modification](man/security/core_pattern_modification.md) | Monitors core dump configuration alterations. |
| [Default Dynamic Loader Modification](man/security/default_loader_modification.md) | Tracks changes to the default binary loader.   |
| [Container Device Mount](man/security/disk_mount.md) | Detects unauthorized container device mounts.  |
| [Docker Socket Abuse](man/security/docker_abuse.md) | Flags potential Docker socket misuse.          |
| [Dropped Executables](man/security/dropped_executable.md) | Detects runtime-dropped executables.           |
| [Dynamic Code Loading](man/security/dynamic_code_loading.md) | Monitors dynamic code loading events.          |
| [Fileless Execution](man/security/fileless_execution.md) | Flags fileless execution techniques.           |
| [Hidden Executable File Creation](man/security/hidden_file_created.md) | Detects creation of hidden executable files.   |
| [Illegitimate Shell](man/security/illegitimate_shell.md) | Flags unauthorized or unexpected shell executions.|
| [Kernel Module Loading](man/security/kernel_module_loading.md) | Monitors kernel module load events.            |
| [Kubernetes API Server Connection](man/security/kubernetes_api_connection.md) | Detects connections to the Kubernetes API server. |
| [Kubernetes TLS Certificate Theft](man/security/kubernetes_certificate_theft_attempt.md) | Flags potential theft of Kubernetes certificates.|
| [LD_PRELOAD Code Injection](man/security/ld_preload.md) | Monitors LD_PRELOAD injection attempts.        |
| [File Operations Hooking on Proc Filesystem](man/security/proc_fops_hooking.md) | Detects hooks on file operations in /proc.     |
| [Kcore Memory File Read](man/security/proc_kcore_read.md) | Monitors reads of /proc/kcore.                 |
| [Process Memory Access](man/security/proc_mem_access.md) | Flags unauthorized /proc/mem access.           |
| [Procfs Mem Code Injection](man/security/proc_mem_code_injection.md) | Detects code injections via /proc/mem.         |
| [Process VM Write Code Injection](man/security/process_vm_write_code_injection.md) | Monitors injections via process_vm_writev.     |
| [Ptrace Code Injection](man/security/ptrace_code_injection.md) | Detects ptrace-facilitated code injections.    |
| [RCD Modification](man/security/rcd_modification.md) | Monitors changes to the remote control daemon. |
| [Sched Debug Reconnaissance](man/security/sched_debug_recon.md) | Flags /proc/sched_debug reconnaissance.        |
| [Scheduled Tasks Modification](man/security/scheduled_task_modification.md) | Tracks modifications to scheduled tasks.      |
| [Process Standard Input/Output over Socket](man/security/stdio_over_socket.md) | Detects IO redirection over sockets.          |
| [Sudoers File Modification](man/security/sudoers_modification.md) | Monitors alterations to the sudoers file.      |
| [Syscall Table Hooking](man/security/syscall_table_hooking.md) | Detects syscall table hook attempts.           |
| [System Request Key Configuration Modification](man/security/system_request_key_config_modification.md) | Monitors system request key configuration changes.|

---

### Docs Events Builtin Syscall Events (`docs/docs/events/builtin/syscall-events.md`)

# Syscalls

Tracee provides two approaches for monitoring Linux system calls:

## Specific Syscall Monitoring

Each system call is available as an individual event with the same name as the syscall. For example, to trace the `open` system call, use the `open` event name. These events use dedicated kprobes for targeted, efficient monitoring of specific syscalls.

The arguments of the system call are automatically captured as event data fields, with types properly mapped from kernel types to Go types (e.g., `const char*` becomes `string`).

## Comprehensive Syscall Monitoring

For broad syscall analysis, Tracee provides comprehensive monitoring events:

- **sys_enter**: Captures all system call entries using raw tracepoints
- **sys_exit**: Captures all system call exits using raw tracepoints

These events are ideal for:

- Security auditing across all syscalls
- System-wide syscall pattern analysis
- Performance monitoring of syscall frequency
- Detecting anomalous syscall behavior

**Note**: These comprehensive events generate high volumes of data since they capture every syscall. Use filtering or sampling for production environments.

For detailed documentation about any specific system call, including its purpose, arguments, return values, and error conditions, please consult the standard Linux manual pages (man section 2). These can be accessed via:

- `man 2 syscall_name` (e.g., `man 2 open`)
- Online at [man7.org](https://man7.org/linux/man-pages/dir_section_2.html)

## Event Sets

All syscall events automatically belong to the **syscalls** set. Many syscalls also belong to additional sets based on their functionality:

- **fs**: File system operations (e.g., open, read, write)
- **net**: Network operations (e.g., socket, connect, bind)
- **proc**: Process operations (e.g., fork, execve, exit)
- **ipc**: Inter-process communication (e.g., pipe, mmap)
- **time**: Time-related operations (e.g., clock_gettime)
- **signals**: Signal handling (e.g., kill, sigaction)

## Examples

Tracing specific syscalls:
```bash
# Trace file opens
tracee -e open

# Trace process creation
tracee -e execve,fork,clone

# Trace network connections
tracee -e socket,connect,bind,accept
```

## Notes

- Event data fields use Go types (string, uint32, etc.)
- Some syscalls may have architecture-specific variants
- Some syscalls may be deprecated or not available on all systems

---

### Docs Events Custom Golang (`docs/docs/events/custom/golang.md`)

# Golang Signatures

There are 2 ways you can get your own golang signatures working with tracee.

1. **Built-In Golang signatures**

    !!! Tip
        This is the preferred way to get your own golang signatures integrated
        into Tracee, as you will find in the next part of this page, but it
        needs a better end-user experience (being worked).

    In order to get your golang signature compiled with tracee, you can create
    a file called `signatures/golang/signature_example.go` and place the
    following code in it:

    !!! Signature Example
        ```
/* Detailed source-code truncated for AI context efficiency. */
```

    Then, edit `signatures/golang/export.go` and place your new signature there:

    ```golang
    var ExportedSignatures = []detect.Signature{
        ...
        &signatureExample{},
    }
    ```

    Follow instructions on [how to build Tracee] and you will get your new signature
    available to use. You may even select only the signatures you created:

    ```console
    sudo ./dist/tracee \
        --output json \
        --events mine
    ```

    ```json
    {"timestamp":1680191445996958642,"threadStartTime":1680191445994222553,"processorId":4,"processId":329031,"cgroupId":10793,"threadId":329031,"parentProcessId":45580,"hostProcessId":329031,"hostThreadId":329031,"hostParentProcessId":45580,"userId":1000,"mountNamespace":4026531841,"pidNamespace":4026531836,"processName":"zsh","hostName":"hb","container":{},"kubernetes":{},"eventId":"6030","eventName":"mine","matchedPolicies":[""],"argsNum":0,"returnValue":11,"syscall":"","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[],"metadata":{"Version":"0.1.0","Description":"My Own Signature Detects Stuff","Tags":null,"Properties":{"signatureID":"Mine-0.1.0","signatureName":"My Own Signature"}}}
    ```

    **Be creative!** You can create signatures that would do pretty much
    anything! Examples of such signatures would: for every X event, connect to
    a cached external data-source and return a positive detection for cases A,
    B or C.

    [how to build Tracee]: ../../../contributing/building/building.md

2. Create a golang signature plugin and dynamically load it during runtime

    !!! Attention
        Eventually you will find out that Golang Plugins aren't very useful if
        you consider all the problems that emerge from using it:

        1. **Can't use different go versions** (need to compile the go plugin
           with the exact same version that was used to build Tracee).

        2. Both Tracee and your golang plugin signature must be built with the
           **exact same GOPATH** or you will get a "plugin was built with a
           different version of package XXX" error.

        3. Any **dependency** you have in your plugin should be of the **same
           version** with the dependencies of Tracee.

        4. Compiling tracee statically is sometimes useful to have a **complete
           portable eBPF tracing/detection solution**. One good example when
           statically compiling tracee is a good idea is to have a single
           binary capable of running in GLIBC (most of them) and MUSL (Alpine)
           powered Linux distros.

    At the end, creating a golang signature plugin won't have the practical
    effects as a plugin mechanism should have, so it is preferred to have
    built-in golang signatures (re)distributed with newer binaries (when you
    need to add/remove signatures from your environment) **FOR NOW**.

---

### Docs Install Config Kubernetes (`docs/docs/install/config/kubernetes.md`)

# Configuring Tracee in Kubernetes

In Kubernetes, Tracee uses a ConfigMap, called `tracee` to make Tracee configuration accessible. The ConfigMap includes a data file called `config.yaml` with the desired configuration. For example:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: tracee
    app.kubernetes.io/component: tracee
    app.kubernetes.io/part-of: tracee
  name: tracee
data:
  config.yaml: |-
    output:
      json:
        files:
          - /var/log/tracee.json
    enrichment:
      decoded-data: true
```

## Kubectl

You can use `kubectl` to interact with it:

View:

```shell
kubectl get cm tracee-config -n tracee
```

Edit:

```shell
kubectl edit cm tracee-config -n tracee
```

## Helm

You can customize specific options with the helm installation:

```
helm install tracee aqua/tracee \
        --namespace tracee --create-namespace \
        --set config.kernelBlob=1024
```

or after installation:

```
helm install tracee aqua/tracee \
        --namespace tracee --create-namespace \
        --set config.output.format=table
```

or to provide a complete config file:

```
 helm install tracee aqua/tracee \
        --namespace tracee --create-namespace \
        --set-file configFile=myconfig.yaml
```

---

### Docs Install Config Monitoring (`docs/docs/install/config/monitoring.md`)

# Monitoring

Tracee provides built-in monitoring capabilities to help you observe its performance and health status. Both features use the same HTTP server endpoint.

## Prometheus Metrics

Tracee exposes Prometheus metrics for performance monitoring of event production, consumption, and detection.

!!! Performance Attention
    Current Prometheus integration targets performance numbers for event
    production, consumption and detection. It does not target the detections
    themselves.

Prometheus scraping is **enabled by default in Kubernetes deployments** at `0.0.0.0:3366/metrics`. For CLI usage, metrics must be explicitly enabled using the `--server metrics` flag (defaults to loopback-only binding at `127.0.0.1:3366`).

### Configuration

**Enable metrics:**
```yaml
server:
  metrics: true
```

**Custom port:**
```yaml
server:
  http-address: :8080
  metrics: true
```

**CLI flags:**
```console
tracee --server metrics --server http-address=:8080
```

!!! Tip
    Check the [Grafana dashboard tutorial](../../../tutorials/deploy-grafana-dashboard.md) for a complete monitoring setup.

## Health Checks

Tracee can expose a `/healthz` endpoint that returns `OK` if everything is healthy. This follows the [common Kubernetes health check pattern](https://kubernetes.io/docs/reference/using-api/health-checks/).

Health monitoring is **disabled by default**.

### Configuration

**Enable health checks:**
```yaml
server:
  healthz: true
```

**Custom port:**
```yaml
server:
  http-address: :8080
  healthz: true
```

**CLI flags:**
```console
tracee --server healthz --server http-address=:8080
```

## Server Configuration

Both Prometheus metrics and health checks share the same HTTP server. Common configuration options:

| Option | Default | Description |
|--------|---------|-------------|
| `http-address` | `127.0.0.1:3366` | HTTP server listen address (loopback only; use `0.0.0.0:3366` for network access) |
| `metrics` | `true` | Enable Prometheus metrics endpoint |
| `healthz` | `false` | Enable health check endpoint |

### Example: Full Monitoring Setup

```yaml
server:
  http-address: :3366
  metrics: true
  healthz: true
```

This configuration makes available:
- Prometheus metrics: `http://localhost:3366/metrics`
- Health check: `http://localhost:3366/healthz`

## Migrating from Wildcard Bind Default

!!! Warning
    In previous versions, the HTTP server defaulted to binding on all interfaces
    (`:3366`). The default is now **loopback only** (`127.0.0.1:3366`). If you
    scrape metrics or run health checks from a remote host, you must set an
    explicit bind address after upgrading.

### Remote Prometheus Scraping

If a central Prometheus server scrapes Tracee from another machine, add an
explicit `http-address`:

**CLI:**
```console
tracee --server http-address=0.0.0.0:3366 --server metrics
```

**Config file:**
```yaml
server:
  http-address: "0.0.0.0:3366"
  metrics: true
  healthz: true
```

To bind to a specific network interface instead of all interfaces:

```console
tracee --server http-address=10.0.0.5:3366 --server metrics
```

### Kubernetes (Helm)

No action is required for the bind address. The Helm chart explicitly sets
`0.0.0.0:3366`. To restrict HTTP ingress to Prometheus pods, enable
`networkPolicy.enabled=true` and adjust `networkPolicy.ingressFrom` to match
your Prometheus deployment labels.

### pprof Access

The `/debug/pprof/` endpoints are served on the same HTTP address as metrics and
healthz with no additional access control. pprof heap dumps can expose sensitive
process memory, so keep the default loopback bind or restrict access via
NetworkPolicy/firewall rules. For remote debugging, use an SSH tunnel:

```console
ssh -L 3366:127.0.0.1:3366 tracee-host
curl http://localhost:3366/debug/pprof/heap
```

---

### Docs Policies Usage Cli (`docs/docs/policies/usage/cli.md`)

# CLI Policy Usage

This section details how to use policies with the Tracee CLI.

## Applying Tracee Policies

A [policy file](../index.md) can be applied using the `--policy` flag:

```console
tracee --policy ./policy.yml
```

## Using multiple policies

To specify multiple policies, users can either specify the directory, which contains all of the policies that they would like to load into Tracee, or by specifying the policies one by one.

Through a directory:

```console
tracee --policy ./policy-directory
```

By specifying individual policies:

```console
tracee --policy ./policy-one.yaml --policy ./policy-two.yaml 
```

## Event Selection Shortcuts

Instead of defining policies in YAML files, you can use the `--events` flag for quick event selection:

```console
# Trace specific events
tracee --events execve,openat

# Select detectors by threat severity
tracee --events threat.severity=critical

# Combine regular events with threat-based selection
tracee --events write --events 'threat.severity>=high'
```

See the [events flag documentation](../../flags/events.1.md) for complete details on threat-based detector selection.

## EXAMPLE

```console
tracee --config ./config.yaml --policy ./policy.yaml && cat /tmp/debug.json
```

### config.yaml (example)

```yaml
# server configuration

server:
    http-address: ":3366"
    healthz: true
    metrics: false
    pprof: false
    pyroscope: false

# feature flags

containers:
    enrich: true
    # cgroupfs:
    #     path: /host/sys/fs/cgroup
    #     force: true
    # sockets:
    #     - runtime: docker
    #       socket: /var/run/docker.sock

buffers:
    kernel:
        artifacts: 1024
        control-plane: 1024
    pipeline: 1000

# signatures

signatures-dir: ""

# features setup

capabilities:
    bypass: false
stores:
    process:
        enabled: true
        source: both
        max-processes: 8192
        max-threads: 8192
    dns:
        enabled: false

# logging

logging:
  level: debug
  file: /tmp/debug.json
  # aggregate:
  #     enabled: true
  #     flush-interval: 5s
  filters:
    exclude:
      pkg:
        - capabilities

# enrichment

enrichment:
  environment: true
  executable-hash:
    enabled: true
    mode: dev-inode
  decoded-data: true

# output

output:
  sort-events: true
  destinations:
    - name: stdout
      format: json
      path: stdout
```

### policy.yaml

```yaml
apiVersion: tracee.aquasec.com/v1beta1
kind: Policy
metadata:
  name: signatures
  annotations:
    description: traces all signatures
spec:
  scope:
    - global
  rules:
    # display security events
    - event: stdio_over_socket
    - event: k8s_api_connection
    - event: aslr_inspection
    - event: proc_mem_code_injection
    - event: docker_abuse
    - event: scheduled_task_mod
    - event: ld_preload
    - event: cgroup_notify_on_release
    - event: default_loader_mod
    - event: sudoers_modification
    - event: sched_debug_recon
    - event: system_request_key_mod
    - event: cgroup_release_agent
    - event: rcd_modification
    - event: core_pattern_modification
    - event: proc_kcore_read
    - event: proc_mem_access
    - event: hidden_file_created
    - event: anti_debugging
    - event: ptrace_code_injection
    - event: process_vm_write_inject
    - event: disk_mount
    - event: dynamic_code_loading
    - event: fileless_execution
    - event: illegitimate_shell
    - event: kernel_module_loading
    - event: k8s_cert_theft
    - event: proc_fops_hooking
    - event: syscall_hooking
    - event: dropped_executable
    # tracee open by cat in /tmp/* files
    - event: openat
      filters:
        - comm=cat
        - data.pathname=/tmp*
    # trace all container creations and removals
    - event: container_create
    - event: container_remove
```

---

--- METRICS ---
- Files Extracted: 98
- Estimated Token Budget: ~125177 tokens
- Canonical Reference: https://codewiki.google/github.com/aquasecurity/tracee