### README

# BTrace Documentation

Welcome to the BTrace documentation! BTrace is a safe, dynamic tracing tool for the Java platform that allows you to instrument running applications without stopping them.

## Quick Start

**New to BTrace?** Start here → [Getting Started Guide](GettingStarted.md)

Get up and running in 5 minutes with installation, your first script, and common usage patterns.

## Documentation Map

| Document | Description | Target Audience |
|----------|-------------|-----------------|
| **[Getting Started](GettingStarted.md)** | Installation, first script, deployment modes, common pitfalls | New users, quick start |
| **[Oneliner Guide](OnelinerGuide.md)** | DTrace-style oneliners for quick debugging without scripts | Quick debugging, ops/SRE |
| **[Quick Reference](QuickReference.md)** | Annotations, patterns, CLI commands, built-in functions | Experienced users, quick lookup |
| **[BTrace Tutorial](BTraceTutorial.md)** | Comprehensive lessons covering all features | All users, in-depth learning |
| **[Troubleshooting Guide](Troubleshooting.md)** | Common errors, debugging, performance, compatibility | Problem-solving, debugging |
| **[FAQ](FAQ.md)** | Common questions, best practices, comparisons | All users, decision-making |
| **[MCP Server](MCPServer.md)** | Using BTrace from AI clients via the Model Context Protocol | AI-assisted debugging, LLM integrations |
| **[Migration Guide 2.x → 3.0](Migration-2.x-to-3.0.md)** | Upgrading from BTrace 2.x: package rename, Java version policy, breaking changes | Existing 2.x users |
| **[Extension Development Guide](BTraceExtensionDevelopmentGuide.md)** | Writing, packaging, and publishing BTrace extensions | Extension authors, platform maintainers |
| **[3.0 Release Checklist](ReleaseChecklist.md)** | JDK, protocol, security, packaging, and RC go/no-go gates | Release maintainers |

## Learning Paths

### I'm New to BTrace
1. Read [Getting Started Guide](GettingStarted.md) (10 minutes)
2. Try the 5-minute quick start example
3. Learn [Oneliner syntax](OnelinerGuide.md) for quick debugging (5 minutes)
4. Explore [BTrace Tutorial](BTraceTutorial.md) lessons 1-3
5. Keep [Quick Reference](QuickReference.md) handy

Tip: Want latency histograms fast? See [Quick Start: Histogram Metrics Extension](GettingStarted.md#quick-start-histogram-metrics-extension) and the tutorial section [Using the Histogram Metrics Extension](BTraceTutorial.md#using-the-histogram-metrics-extension-btrace-metrics).

### I Need to Solve a Problem
1. Check [Troubleshooting Guide](Troubleshooting.md) for your error
2. Search [FAQ](FAQ.md) for similar issues
3. Review [Getting Started](GettingStarted.md) common pitfalls
4. Ask on [Slack](http://btrace.slack.com/) or [Gitter](https://gitter.im/btraceio/btrace)

### I Need a Quick Lookup
- **Quick Debug?** → [Oneliner Guide](OnelinerGuide.md) for DTrace-style one-line commands
- **Annotations?** → [Quick Reference: Core Annotations](QuickReference.md#core-annotations)
- **CLI Commands?** → [Quick Reference: CLI Commands](QuickReference.md#cli-commands)
- **Common Patterns?** → [Quick Reference: Common Patterns](QuickReference.md#common-patterns)
- **Built-in Functions?** → [Quick Reference: Built-in Functions](QuickReference.md#built-in-functions)

### I Want Advanced Features
1. **JFR Integration** → [Getting Started: JFR Integration](GettingStarted.md#advanced-jfr-integration), [Tutorial Lesson 5](BTraceTutorial.md)
2. **Sampling** → [Quick Reference: @Sampled](QuickReference.md#sampled), [FAQ: Performance](FAQ.md#performance-issues)
3. **Aggregations** → [Quick Reference: Aggregation Functions](QuickReference.md#aggregation-functions)
4. **Cloud Deployments** → [Getting Started: K8s](GettingStarted.md#btrace-in-containers-and-kubernetes), [FAQ: K8s](FAQ.md#can-i-use-btrace-with-microservices)
5. **Fat Agent JAR** → [Getting Started: Fat Agent](GettingStarted.md#fat-agent-jar-single-jar-deployment) for single-JAR deployment
6. **Level Filtering** → [Quick Reference: @Level](QuickReference.md#level)
7. **Extensions Architecture** → [Extension invokedynamic Bridge](architecture/ExtensionInvokeDynamicBridge.md)

### I'm Working on AI/LLM Applications
1. **Observe LLM API calls** → [Tutorial Lesson 12.1](BTraceTutorial.md#121-llm-inference-tracing-btrace-llm-trace) — token counts, latency, cost tracking
2. **Observe RAG pipelines** → [Tutorial Lesson 12.2](BTraceTutorial.md#122-rag-pipeline-observability-btrace-rag-quality) — vector DB queries, similarity scores
3. **Observe on-device inference** → [Tutorial Lesson 12.3](BTraceTutorial.md#123-gpu-and-inference-observability-btrace-gpu-bridge) — ONNX, DJL, TensorFlow
4. **Use BTrace from an AI agent** → [MCP Server](MCPServer.md) for Claude Desktop, Claude Code, Cursor

## Documentation by Topic

### Core Features
- **Method Tracing** → [Tutorial Lesson 1](BTraceTutorial.md), [Quick Reference: @OnMethod](QuickReference.md#onmethod)
- **Timing & Duration** → [Quick Reference: @Duration](QuickReference.md#parameter-annotations), [Pattern: Method Timing](QuickReference.md#pattern-1-method-entryexit-timing)
- **Exception Tracking** → [Quick Reference: Kind.ERROR](QuickReference.md#location-kinds), [Pattern: Exception Tracking](QuickReference.md#pattern-3-exception-tracking)
- **Field Access** → [Quick Reference: Kind.FIELD_GET/SET](QuickReference.md#location-kinds)

### Advanced Features
- **JFR Integration** → [Getting Started: JFR](GettingStarted.md#advanced-jfr-integration), [Quick Reference: @Event](QuickReference.md#event), [FAQ: JFR](FAQ.md#jfr-integration)
- **Sampling** → [Quick Reference: @Sampled](QuickReference.md#sampled), [FAQ: Performance](FAQ.md#btrace-causes-significant-slowdown)
- **Level Control** → [Quick Reference: @Level](QuickReference.md#level)
- **Aggregations** → [Quick Reference: Aggregation Functions](QuickReference.md#aggregation-functions)
- **Periodic Events** → [Quick Reference: @OnTimer](QuickReference.md#ontimer), [@PeriodicEvent](QuickReference.md#periodicevent)
- **Runtime Contracts** → [Tutorial Lesson 11](BTraceTutorial.md#lesson-11--runtime-contracts-btrace-contracts) — latency budgets, call-rate limits, assertions, tagged path profiling

### AI/LLM Observability
- **LLM Inference Tracing** → [Tutorial Lesson 12.1](BTraceTutorial.md#121-llm-inference-tracing-btrace-llm-trace) — token counts, latency, cost, streaming TTFT
- **RAG Pipeline Observability** → [Tutorial Lesson 12.2](BTraceTutorial.md#122-rag-pipeline-observability-btrace-rag-quality) — vector DB queries, similarity scores, empty retrievals
- **GPU / Inference Observability** → [Tutorial Lesson 12.3](BTraceTutorial.md#123-gpu-and-inference-observability-btrace-gpu-bridge) — ONNX Runtime, DJL, TensorFlow, Panama FFM
- **AI Agents (MCP)** → [MCP Server](MCPServer.md) — connect Claude Desktop, Claude Code, or Cursor to live JVMs

### Deployment & Operations
- **Installation** → [Getting Started: Installation](GettingStarted.md#installation)
- **Deployment Modes** → [Getting Started: Running BTrace](GettingStarted.md#four-ways-to-run-btrace)
- **Fat Agent (Single-JAR)** → [Getting Started: Fat Agent](GettingStarted.md#fat-agent-jar-single-jar-deployment), [Architecture](architecture/fat-agent-plugin.md), [Gradle Plugin](../btrace-gradle-plugin/README.md)
- **Docker & Containers** → [Getting Started: Containers](GettingStarted.md#btrace-in-containers-and-kubernetes)
- **Kubernetes** → [Getting Started: K8s](GettingStarted.md#btrace-in-containers-and-kubernetes), [FAQ: Microservices](FAQ.md#can-i-use-btrace-with-microservices), [Troubleshooting: K8s](Troubleshooting.md#kubernetes-and-cloud-deployments)
- **Performance Tuning** → [FAQ: Performance Impact](FAQ.md#whats-the-performance-impact-of-btrace), [Troubleshooting: Performance](Troubleshooting.md#performance-issues)
- **Extensions CLI (btracex)** → [Permission Policy](PermissionPolicy.md) for allow/deny and quick inspection

### Problem Solving
- **No Output** → [Troubleshooting: No Output](Troubleshooting.md#no-output-from-scripts)
- **Attachment Fails** → [Troubleshooting: JVM Attachment](Troubleshooting.md#jvm-attachment-issues)
- **Verification Errors** → [Troubleshooting: Verification](Troubleshooting.md#script-verification-errors)
- **Performance Issues** → [Troubleshooting: Performance](Troubleshooting.md#performance-issues)
- **Compatibility** → [Troubleshooting: Compatibility](Troubleshooting.md#compatibility-issues)

### Integration
- **Spring Boot** → [FAQ: Spring Boot](FAQ.md#can-i-use-btrace-with-spring-boot-applications)
- **Third-Party Libraries** → [FAQ: Third-Party](FAQ.md#how-do-i-trace-methods-from-third-party-libraries)
- **JMX Export** → [Quick Reference: @Export](QuickReference.md#export), [FAQ: Monitoring Integration](FAQ.md#can-i-integrate-btrace-with-monitoring-systems)
- **Service Mesh** → [FAQ: Service Mesh](FAQ.md#does-btrace-work-with-service-meshes-istiolinkerd)
- **AI / LLM Clients (MCP)** → [MCP Server](MCPServer.md) for Claude Desktop, Claude Code, Cursor
- **Published Extensions** → [Extension Development Guide](BTraceExtensionDevelopmentGuide.md) for writing and publishing extensions

### Architecture
- **Masked JAR** → [Masked JAR Architecture](architecture/MaskedJarArchitecture.md) — single-JAR distribution with classdata masking
- **v2 Binary Protocol** → [Version 2 Protocol Architecture](architecture/Version2ProtocolArchitecture.md) — custom binary serialization
- **Extension Framework** → [Extension invokedynamic Bridge](architecture/ExtensionInvokeDynamicBridge.md), [Extension Configuration](architecture/ExtensionConfiguration.md), [Extension Manifest](architecture/ExtensionManifestFormat.md), [Extension Storage](architecture/ExtensionStorageDesign.md)
- **Extension Catalog** → [Extension Development Guide](BTraceExtensionDevelopmentGuide.md) for packaging and distribution
- **Instrumentation Backends (ASM / ClassFile API)** → [Instrumentation Backends](architecture/InstrumentationBackends.md) — automatic backend selection for future JDK class files
- **Agent Library Attributes** → [Agent Manifest Library Attributes](architecture/agent-manifest-libs.md) — declaring boot/system JARs via the agent manifest

### Hands-On Tutorials

- **[Hands-on tutorial series](tutorials/README.md)** — scenario-driven labs with a runnable demo app; start with [Your First Trace in 2 Minutes](tutorials/01-first-trace-in-2-minutes.md)

## Sample Scripts

BTrace includes 50+ sample scripts demonstrating real-world use cases:
- Location: `btrace-dist/src/main/resources/samples/`
- Examples: File I/O tracking, JDBC monitoring, HTTP requests, memory allocation, thread analysis
- Browse: [BTrace Samples Directory](../btrace-dist/src/main/resources/samples/)

## External Resources

### Official Resources
- **GitHub Repository**: [github.com/btraceio/btrace](https://github.com/btraceio/btrace)
- **Wiki (External)**: [github.com/btraceio/btrace/wiki](https://github.com/btraceio/btrace/wiki/Home)
- **Maven Plugin (Script Compilation)**: [github.com/btraceio/btrace-maven](https://github.com/btraceio/btrace-maven)
- **Maven fat-agent status**: unpublished module removed for 3.0.0; see [Getting Started](GettingStarted.md#maven-builds)
- **Releases**: [GitHub Releases](https://github.com/btraceio/btrace/releases/latest)

### Community
- **Slack**: [btrace.slack.com](http://btrace.slack.com/)
- **Gitter Chat**: [gitter.im/btraceio/btrace](https://gitter.im/btraceio/btrace)
- **Issues**: [GitHub Issues](https://github.com/btraceio/btrace/issues)
- **Discussions**: [GitHub Discussions](https://github.com/btraceio/btrace/discussions)

### Tools & Integrations
- **VisualVM Plugin**: [visualvm.github.io](https://visualvm.github.io)
- **JDK Mission Control**: For viewing JFR events created by BTrace

## Contributing

BTrace is an open-source project welcoming contributions. To contribute:

1. Sign the [Oracle Contributor Agreement](https://oca.opensource.oracle.com/)
2. Read the [Contributing Guidelines](../README.md#contributing---important)
3. Fork the repository and create a pull request
4. See [Build Instructions](../README.md#building-btrace) for development setup

## Version Information

- **Current Version**: Check [GitHub Releases](https://github.com/btraceio/btrace/releases/latest)
- **Java Compatibility**: BTrace 3.0 runs on Java 8–25+. Running BTrace against a JVM older than Java 17 is deprecated: it continues to work throughout 3.x but emits a deprecation warning. Support for Java < 17 will be removed in the next major release (4.0). See the [migration guide](Migration-2.x-to-3.0.md).
- **License**: Apache License 2.0

## Documentation Feedback

Found an issue with the documentation? Please:
- Report it on [GitHub Issues](https://github.com/btraceio/btrace/issues)
- Tag it with `documentation` label
- Or submit a pull request with improvements

---

**Ready to get started?** → [Getting Started Guide](GettingStarted.md)

---

### Architecture/Agent Manifest Libs

# Agent Manifest Library Attributes

The BTrace agent JAR manifest can declare extra JARs to place on the bootstrap and
system class loaders, so a repackaged or fat agent can carry its dependencies without
extra command-line arguments. These attributes are read from the agent JAR's
`META-INF/MANIFEST.MF` at startup.

## Attributes

| Manifest attribute | Purpose |
|---|---|
| `BTrace-Boot-Libs` | Space-separated list of JAR paths appended to the **bootstrap** class loader search. |
| `BTrace-System-Libs` | Space-separated list of JAR paths appended to the **system** class loader search. |
| `BTrace-Libs-Root` | Optional base directory for resolving relative entries. Defaults to the agent JAR's parent directory. |
| `BTrace-Libs-Profile` | Optional named subdirectory to auto-scan for `boot/*.jar` and `system/*.jar`. Mirrors the `libs` agent argument. |

The standard `Boot-Class-Path` manifest attribute continues to be honored as well.

Entries use `Class-Path`-style space separation; each entry may be absolute or relative
to `BTrace-Libs-Root` (or the agent JAR's parent). Unusable entries are logged and skipped
rather than aborting startup.

## See also

- [Masked JAR architecture](MaskedJarArchitecture.md)
- [Migrating from libs/profiles to Extensions](migrating-from-libs-profiles.md)

---

### Architecture/ExtensionConfiguration

# BTrace Extension Configuration

## Overview

Extension configuration allows users to customize extension loading behavior, enable/disable specific extensions, and pass extension-specific settings.

## Configuration File Locations

Configuration files are searched in priority order (highest to lowest):

1. **Command-line override**: `-Dbtrace.extensions.config=/path/to/config`
2. **User config**: `~/.btrace/extensions.conf`
3. **System config**: `$BTRACE_HOME/conf/extensions.conf`

The first file found is used. Settings from multiple files are NOT merged.

## Configuration Format

Uses standard Java properties format:

```properties
# Extension Control
# Comma-separated list of extension IDs to enable (empty = all enabled)
extensions.enabled=

# Comma-separated list of extension IDs to disable
extensions.disabled=

# Auto-load extensions on demand (default: true)
extensions.autoload=true

# Extension-specific settings (passed to extension on load)
# Format: <extension-id>.<setting-name>=<value>
btrace-metrics.histogram.default-precision=3
btrace-metrics.stats.window-size=1000
```

## Configuration Properties

### Global Settings

**extensions.enabled**
- Format: comma-separated extension IDs
- Default: (empty - all enabled)
- Example: `extensions.enabled=btrace-metrics,btrace-custom`
- Description: Whitelist of extensions to load. If set, only listed extensions are loaded. If empty or not set, all discovered extensions are enabled (unless explicitly disabled).

**extensions.disabled**
- Format: comma-separated extension IDs
- Default: (empty - none disabled)
- Example: `extensions.disabled=btrace-old-metrics`
- Description: Blacklist of extensions to prevent from loading. Takes precedence over `extensions.enabled`.

**extensions.autoload**
- Format: boolean (true/false)
- Default: `true`
- Description: If `true`, extensions are loaded on demand when scripts use their services. If `false`, all enabled extensions are loaded at agent startup.

### Extension-Specific Settings

Extensions can define custom configuration properties using the pattern:
```
<extension-id>.<setting-name>=<value>
```

Examples:
```properties
# BTrace Metrics settings
btrace-metrics.histogram.default-precision=3
btrace-metrics.histogram.max-value=3600000000
btrace-metrics.stats.window-size=1000

# Custom extension settings
my-extension.feature.enabled=true
my-extension.endpoint=http://localhost:8080
```

Extension-specific settings are passed to the extension when loaded. Extensions can access these via their configuration API.

## Usage Examples

### Example 1: Disable Specific Extension

```properties
# Disable legacy metrics extension
extensions.disabled=btrace-old-metrics
```

### Example 2: Whitelist Extensions

```properties
# Only load metrics and custom extensions
extensions.enabled=btrace-metrics,my-custom-extension
```

### Example 3: Configure Extension Settings

```properties
# Configure metrics extension
btrace-metrics.histogram.default-precision=3
btrace-metrics.histogram.max-value=3600000000
```

### Example 4: Load All Extensions at Startup

```properties
# Disable lazy loading, load all extensions immediately
extensions.autoload=false
```

## API Integration

### ExtensionConfig Class

New class to manage extension configuration:

```java
public class ExtensionConfig {
  public boolean isEnabled(String extensionId);
  public boolean isAutoLoad();
  public Properties getExtensionProperties(String extensionId);
}
```

### ExtensionLoader Integration

ExtensionLoader will:
1. Load configuration on initialization
2. Filter discovered extensions based on enabled/disabled lists
3. Apply autoload setting (lazy vs eager loading)
4. Pass extension-specific properties when loading extensions

### Configuration Loading Priority

```
Command-line (-Dbtrace.extensions.config)
    ↓
User config (~/.btrace/extensions.conf)
    ↓
System config ($BTRACE_HOME/conf/extensions.conf)
    ↓
Built-in defaults (all enabled, autoload=true)
```

## Backward Compatibility

- If no configuration file exists, all extensions are enabled with autoload
- Existing deployments continue to work without any configuration
- Configuration is purely additive (opt-in customization)

## Future Enhancements

- Hot reload: Detect configuration file changes and reload extensions
- JMX integration: Expose extension configuration via JMX for runtime modification
- Validation: Warn about unknown extension IDs or malformed settings

---

### Architecture/ExtensionInvokeDynamicBridge

# Extension invokedynamic Bridge Architecture

## Overview

The BTrace extension system uses invokedynamic to bridge between script classloaders and extension classloaders, avoiding bootstrap classloader pollution while maintaining complete extension isolation.

**Key Benefits:**
- **Clean bootstrap namespace** - Only BTrace core classes in bootstrap, no extension JARs
- **Extension isolation** - Each extension in its own classloader with shaded dependencies
- **Security** - Extensions don't have bootstrap-level privileges
- **Java 8+ compatible** - Works on all supported Java versions (8+)

## Architecture Diagram

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Component Details

### 1. ExtensionIndy (btrace-runtime)

**Location:** `btrace-runtime/src/main/java/io/btrace/runtime/ExtensionIndy.java`

**Purpose:** Bootstrap methods for invokedynamic extension access

**Key Methods:**
```java
public static CallSite bootstrapFieldGet(
    MethodHandles.Lookup caller,
    String fieldName,
    MethodType type,
    String serviceClassName,
    String serviceType,
    String factoryMethod) throws Exception
```

**Responsibilities:**
- Receives invokedynamic bootstrap calls from script bytecode
- Uses ExtensionBridge to load extension service class
- Creates MethodHandle for service instantiation
- Handles both SIMPLE (no-arg) and RUNTIME (BTraceRuntime.Impl param) services
- Supports both constructors and factory methods
- Returns ConstantCallSite for zero-overhead subsequent calls

**Error Handling:**
- Gracefully degrades to null on failure
- Logs errors but doesn't throw exceptions
- Matches behavior of failed service instantiation

### 2. ExtensionBridge (btrace-core)

**Location:** `btrace-core/src/main/java/io/btrace/extension/ExtensionBridge.java`

**Purpose:** Interface for agent-to-runtime classloader bridging

**API:**
```java
public interface ExtensionBridge {
  Class<?> getExtensionClass(String serviceClassName) throws Exception;
}
```

**Why an Interface:**
- Allows btrace-runtime (bootstrap) to call btrace-agent code
- Breaks circular dependency (runtime can't depend on agent)
- Clean separation of concerns

### 3. ExtensionBridgeImpl (btrace-core)

**Location:** `btrace-core/src/main/java/io/btrace/extension/impl/ExtensionBridgeImpl.java`

**Purpose:** Agent-side implementation of ExtensionBridge

**Initialization:**
```java
static {
  try {
    Class<?> indyClz = Class.forName("io.btrace.runtime.ExtensionIndy");
    ExtensionBridge bridge = new ExtensionBridgeImpl(Main.getExtensionLoader());
    indyClz.getField("bridge").set(null, bridge);
  } catch (ClassNotFoundException e) {
    // Expected for pre-Java 8 or if ExtensionIndy unavailable
  }
}
```

**Responsibilities:**
- Sets `ExtensionIndy.bridge` field during static initialization
- Uses ExtensionLoader to find extensions providing requested services
- Loads extension classes from extension classloaders
- Ensures extensions are loaded before returning class

### 4. Preprocessor (btrace-agent)

**Location:** `btrace-agent/src/main/java/io/btrace/instr/Preprocessor.java`

**Purpose:** Transforms @Injected field references to invokedynamic

**Transformation:**
```java
// Before:
GETSTATIC ScriptClass.metrics : MetricsService

// After:
INVOKEDYNAMIC bootstrapFieldGet(
  "metrics",                           // field name (debug)
  "()Lio/btrace/metrics/MetricsService;",  // method descriptor
  ExtensionIndy.bootstrapFieldGet,     // bootstrap method
  "io.btrace.metrics.MetricsService",  // service class name
  "RUNTIME",                           // optional type hint (deprecated)
  ""                                   // factory method (empty = constructor)
) : MetricsService
```

**Modified Method:**
```java
private AbstractInsnNode updateInjectedUsage(
    ClassNode cn, FieldInsnNode fin, InsnList l, LocalVarGenerator lvg)
```

**Changes:**
- Removed local variable caching
- Removed service instantiation bytecode
- Emits InvokeDynamicInsnNode instead
- Passes service metadata as bootstrap arguments

### 5. ExtensionLoader (btrace-core)

**Location:** `btrace-core/src/main/java/io/btrace/extension/ExtensionLoader.java`

**Key Changes:**
- Avoids `instrumentation.appendToBootstrapClassLoaderSearch(jarFile)`
- Avoids an eager extension loading loop
- **Behavior:** Extensions loaded on-demand via invokedynamic

**Impact:**
- Bootstrap classloader remains clean
- Extensions only loaded when scripts use them
- Better resource utilization

## Execution Flow

### Initialization (Agent Startup)

```
1. Main.agentmain/premain()
   ↓
2. ExtensionLoader.discoverExtensions()
   - Scans extensions/ directory
   - Creates ExtensionDescriptor for each extension
   - Stores in availableExtensions map
   - Does NOT load extensions yet
   ↓
3. ExtensionBridgeImpl.<clinit>() (static initializer)
   - Creates new ExtensionBridgeImpl instance
   - Sets ExtensionIndy.bridge = instance
   - Bridge now ready for invokedynamic bootstrap calls
   ↓
4. Scripts can now be loaded and compiled
```

### Script Loading (First Access)

```
1. Script bytecode contains:
   INVOKEDYNAMIC bootstrapFieldGet(...) : MetricsService
   ↓
2. JVM calls ExtensionIndy.bootstrapFieldGet()
   (first time for this callsite)
   ↓
3. ExtensionIndy.bootstrapFieldGet():
   a. Call bridge.getExtensionClass("io.btrace.metrics.MetricsService")
   ↓
4. ExtensionBridgeImpl.getExtensionClass():
   a. Call loader.findExtensionForService(serviceClassName)
   b. If ext.isLoaded() == false:
      - loader.load(ext)  // Loads extension JAR into ExtensionClassLoader
   c. extClassLoader.loadClass(serviceClassName)
   d. Return Class<?> to bootstrap method
   ↓
5. ExtensionIndy.bootstrapFieldGet() (continued):
   a. Create MethodHandle for service instantiation:
      - Auto-detect constructor or factory
      - Initialize extensions via Extension.initialize(ExtensionContext) when first accessed
   b. Return new ConstantCallSite(methodHandle)
   ↓
6. JVM caches ConstantCallSite
   ↓
7. Subsequent calls to same INVOKEDYNAMIC:
   - Use cached CallSite (zero overhead)
   - MethodHandle directly instantiates service
   - No bootstrap method re-execution
```

### Subsequent Field Access (Same Script)

```
1. Script executes: metrics.histogram("foo")
   ↓
2. INVOKEDYNAMIC instruction
   ↓
3. JVM uses cached CallSite (no bootstrap)
   ↓
4. MethodHandle executes:
   - Instantiates MetricsService
   - Returns service instance
   ↓
5. INVOKEVIRTUAL MetricsService.histogram(String)
   - Normal virtual dispatch
   - JVM resolves from actual object class
   - Extension method executes
```

## Performance Characteristics

### First Access

**Overhead:**
- Bootstrap method execution: ~1-5ms
  - Extension class loading (if not already loaded)
  - MethodHandle creation
  - CallSite creation

**One-time cost per @Injected field per script**

### Subsequent Access

**Overhead:** Zero (compared to direct method call)
- JVM optimizes ConstantCallSite to direct call
- Identical performance to pre-invokedynamic approach
- JIT compiler inlines if possible

### Memory

**Before (Bootstrap Classpath):**
- All extension JARs in bootstrap: ~5-10MB per extension
- Permanent heap allocation
- Visible to all scripts and JVM internals

**After (invokedynamic Bridge):**
- Only loaded extensions in memory
- Extension classloaders eligible for GC when scripts unload
- Clean namespace separation

## Extension Types and Instantiation

### Extension Instances (No-arg)

Preferred pattern: extensions expose a public no-arg constructor and receive
their runtime context via `Extension.initialize(ExtensionContext)`.

**Constructor Example:**
```java
public class MyExtension extends Extension {
  public MyExtension() {
    // No-arg constructor
  }
}
```

**MethodHandle Creation:**
```java
MethodHandle constructor =
  MethodHandles.publicLookup().findConstructor(
    serviceClass,
    MethodType.methodType(void.class));
```

Factory methods are also supported but less common in the new model.

**MethodHandle Creation:**
```java
MethodHandle factory =
  MethodHandles.publicLookup().findStatic(
    serviceClass,
    "getInstance",
    MethodType.methodType(serviceClass));
```

### Runtime-Aware Initialization

If an extension needs access to runtime context, override `initialize(ExtensionContext)`
in your subclass of `Extension`. The bridge calls it after construction.

**Initialization Example:**
```java
public class MyExtension extends Extension {
  @Override
  public void initialize(ExtensionContext ctx) {
    super.initialize(ctx);
    // Use ctx as needed
  }
}
```

**MethodHandle Creation:**
```java
Class<?> runtimeImplClass =
  Class.forName("io.btrace.runtime.BTraceRuntime$Impl");

MethodHandle constructor =
  MethodHandles.publicLookup().findConstructor(
    serviceClass,
    MethodType.methodType(void.class, runtimeImplClass));

// Bind BTraceRuntime.enter() as first argument
MethodHandle getRuntimeImpl =
  MethodHandles.publicLookup().findStatic(
    BTraceRuntime.class,
    "enter",
    MethodType.methodType(runtimeImplClass));

mh = MethodHandles.filterReturnValue(getRuntimeImpl, constructor);
```

## Classloader Hierarchy

### Script Classloader

```java
// In BTraceRuntimeImpl_8.defineClass()
ClassLoader loader = new ClassLoader(null) {};  // parent = null
Class<?> cl = unsafe.defineClass(name, code, 0, code.length, loader, null);
```

**Parent:** null (delegates directly to bootstrap)
**Visibility:** Only bootstrap classes
**Impact:** Scripts can only see classes in bootstrap classpath

### Extension Classloader

```java
// In ExtensionLoader.load()
ExtensionClassLoader classLoader = new ExtensionClassLoader(
  extensionId,
  extensionVersion,
  new URL[] {jarUrl},
  parentClassLoader  // BTrace boot classloader
);
```

**Parent:** BTrace boot classloader (btrace-boot.jar)
**Visibility:** Extension JAR + BTrace core APIs
**Isolation:** Each extension in separate classloader

**Shaded Dependencies:**
- Extensions use shadow plugin to shade dependencies
- Prevents version conflicts between extensions
- Example: btrace-metrics shades HdrHistogram

### Classloader Delegation

```
Script Class
├── parent = null → Bootstrap ClassLoader
│   ├── JRE system classes
│   ├── btrace-boot.jar (BTraceRuntime, annotations)
│   └── ExtensionIndy.class
│
ExtensionClassLoader (per extension)
├── parent = BTrace Boot ClassLoader
│   └── btrace-boot.jar classes
├── Extension JAR classes
└── Shaded dependencies
```

**invokedynamic bridges the gap:**
- Script sees only bootstrap
- Bootstrap method accesses extension via bridge
- MethodHandle allows cross-classloader calls

## Java Version Compatibility

### Java 8+

**invokedynamic features used:**
- `INVOKEDYNAMIC` instruction (Java 7+)
- `MethodHandles` (Java 7+)
- `CallSite` / `ConstantCallSite` (Java 7+)
- `Handle` (Java 7+)

**NOT used:**
- Hidden classes (`Lookup.defineHiddenClass`) - Java 15+ only
- VarHandles - Java 9+ only

**Result:** Works on all BTrace-supported Java versions (8+)

### Comparison with Indy.java

**Indy.java (probe handlers):**
- Uses hidden classes (Java 15+ only)
- Creates dynamic hidden classes at link time
- Complex nested class generation

**ExtensionIndy.java (extensions):**
- Uses standard classloading (Java 8+)
- Loads classes from extension classloaders
- Simple MethodHandle creation

## Error Handling and Debugging

### Bootstrap Method Failures

**Causes:**
- Extension not found
- Extension class loading failure
- Constructor/factory method not found
- Runtime instantiation error

**Behavior:**
```java
try {
  // Load class and create MethodHandle
} catch (Throwable t) {
  // Graceful degradation: return null
  mh = MethodHandles.constant(type.returnType(), null);
}
return new ConstantCallSite(mh);
```

**Impact:**
- Script continues execution
- Service field is null
- NPE on first usage (matches current behavior)

### Debugging

**Enable logging:**
```
-Dorg.slf4j.simpleLogger.log.io.btrace.agent.extension=DEBUG
-Dorg.slf4j.simpleLogger.log.io.btrace.runtime.ExtensionIndy=DEBUG
```

**Logs show:**
- Extension discovery
- Extension loading
- Service class resolution
- MethodHandle creation

**Bytecode inspection:**
```bash
javap -v ScriptClass.class | grep -A 5 INVOKEDYNAMIC
```

**Shows:**
- Bootstrap method reference
- Bootstrap arguments
- Method descriptor

## Migration from Bootstrap Classpath Approach

### Before

**ExtensionLoader.load():**
```java
if (instrumentation != null) {
  JarFile jarFile = new JarFile(descriptor.getJarPath().toFile());
  instrumentation.appendToBootstrapClassLoaderSearch(jarFile);
}
```

**Preprocessor.updateInjectedUsage():**
```java
// Create local variable
int varIdx = lvg.newVar(implType);
// Emit instantiation bytecode
toInsert.add(new TypeInsnNode(Opcodes.NEW, implType.getInternalName()));
toInsert.add(new InsnNode(Opcodes.DUP));
toInsert.add(new MethodInsnNode(Opcodes.INVOKESPECIAL, ...));
// Store in local variable
toInsert.add(new VarInsnNode(Opcodes.ASTORE, varIdx));
```

### After

**ExtensionLoader.load():**
```java
// Extension classes accessed via invokedynamic bridge
// (no bootstrap classpath pollution)
```

**Preprocessor.updateInjectedUsage():**
```java
// Emit invokedynamic instruction
InvokeDynamicInsnNode indyInsn = new InvokeDynamicInsnNode(
  fieldName,
  methodDescriptor,
  bootstrapHandle,
  serviceClassName,
  serviceType,
  factoryMethod);
```

### Compatibility

**Existing scripts:** No changes required
- Same @Injected annotation
- Same field types
- Same usage patterns

**Existing extensions:** No changes required
- Same service classes
- Same constructors/factories
- Same MANIFEST.MF metadata

**Behavioral changes:** None
- Extension loading now on-demand
- Performance characteristics unchanged
- Error handling identical

## Security Implications

### Bootstrap Classloader Access

**Before:**
- Extensions had bootstrap-level privileges
- Could access JVM internals
- Could interfere with system classes

**After:**
- Extensions in isolated classloaders
- Limited to parent visibility (BTrace boot)
- Cannot access bootstrap internals

### MethodHandle Security

**MethodHandles.publicLookup():**
- Only creates handles for public members
- Respects Java access control
- Cannot access private/protected methods

**Lookup Context:**
- Bootstrap method has caller's lookup
- Could create privileged handles
- Current implementation uses publicLookup() for safety

## Testing

### Unit Tests

**ExtensionIndy:**
- Test bootstrap method with mock bridge
- Test SIMPLE vs RUNTIME services
- Test constructor vs factory instantiation
- Test error handling

**ExtensionBridgeImpl:**
- Test extension finding
- Test class loading
- Test extension loading on demand

### Integration Tests

**MetricsTest:**
- Uses @Injected MetricsService
- Verifies invokedynamic bootstrap execution
- Verifies service instantiation
- Verifies method calls work

**Verification:**
```bash
# Check bytecode contains INVOKEDYNAMIC
javap -v integration-tests/build/classes/java/test/MetricsTest.class | grep INVOKEDYNAMIC

# Check extension NOT in bootstrap
jcmd <pid> VM.class_hierarchy | grep -v btrace-metrics
```

## Future Enhancements

### 1. Extension API/Impl Split

**Concept:**
- `btrace-metrics-api.jar` → bootstrap (interfaces only)
- `btrace-metrics-impl.jar` → extension CL (implementations)

**Benefits:**
- Cleaner verification
- Smaller bootstrap JAR
- Better security boundaries

### 2. Service Caching

**Current:** Each invokedynamic creates new instance
**Enhancement:** Cache service instances per script

**Implementation:**
```java
// In ExtensionIndy
private static final Map<String, Object> instanceCache = new ConcurrentHashMap<>();

// In bootstrapFieldGet()
String cacheKey = caller.lookupClass().getName() + ":" + serviceClassName;
Object cached = instanceCache.get(cacheKey);
if (cached != null) {
  return new ConstantCallSite(MethodHandles.constant(type.returnType(), cached));
}
```

### 3. Hot Reload Support

**Challenge:** Extension updates require agent restart
**Enhancement:** Use MutableCallSite instead of ConstantCallSite

**Implementation:**
```java
public static CallSite bootstrapFieldGet(...) {
  MethodHandle mh = createServiceHandle(...);
  return new MutableCallSite(mh);  // Allows invalidation
}

// On extension reload:
public static void invalidateExtension(String extensionId) {
  // Find all MutableCallSites for this extension
  // Call MutableCallSite.setTarget() with new MethodHandle
}
```

## Summary

The invokedynamic extension bridge provides:

✅ **Clean bootstrap namespace** - Only BTrace core in bootstrap
✅ **Extension isolation** - Each extension in own classloader
✅ **Java 8+ compatible** - No Java 15+ features required
✅ **Zero performance overhead** - After first call
✅ **Backward compatible** - No script changes required
✅ **Better security** - Extensions lack bootstrap privileges
✅ **On-demand loading** - Extensions loaded when needed

This architecture enables scalable, secure, and performant extension loading while maintaining full compatibility with existing BTrace scripts and extensions.

---

### Architecture/ExtensionManifestFormat

# BTrace Extension MANIFEST.MF Format

## Overview

This document defines the MANIFEST.MF attribute format for BTrace extensions, replacing the previous `btrace-extension.properties` file.

## Rationale

Using MANIFEST.MF for extension metadata provides:
- **Standard Java mechanism** - Already parsed by JVM and tools
- **OSGi precedent** - Follows established patterns from OSGi bundles
- **Single source of truth** - Build tools already generate MANIFEST.MF
- **Less maintenance** - No separate properties file to manage

## Attribute Mapping

### From Properties Format

```properties
extension.id=btrace-metrics
extension.version=3.0.0
extension.name=BTrace Metrics
extension.description=High-performance metrics...
btrace.api.version=3.0+
java.version=8+
services=io.btrace.metrics.MetricsService
requires.extensions=btrace-core
shaded.packages=org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram
```

### To MANIFEST.MF Format

```
BTrace-Extension-Id: btrace-metrics
BTrace-Extension-Version: 3.0.0
BTrace-Extension-Name: BTrace Metrics
BTrace-Extension-Description: High-performance metrics with HdrHistogram
 for percentiles and lock-free statistics
BTrace-API-Version: 3.0+
BTrace-Java-Version: 8+
BTrace-Extension-Services: io.btrace.metrics.MetricsService
BTrace-Extension-Requires: btrace-core
BTrace-Shaded-Packages: org.HdrHistogram->io.btrace.metrics.sh
 aded.hdrhistogram,com.clearspring.analytics->io.btrace.metrics
 .shaded.clearspring
BTrace-Extension-Permissions: NETWORK,THREADS
```

## Attribute Definitions

### Required Attributes

**BTrace-Extension-Id**
- Format: lowercase-with-hyphens
- Example: `btrace-metrics`
- Description: Unique identifier for the extension

**BTrace-Extension-Version**
- Format: semantic version (major.minor.patch[-qualifier])
- Example: `3.0.0`, `3.0.0-SNAPSHOT`
- Description: Extension version for conflict resolution

### Optional Attributes

**BTrace-Extension-Name**
- Format: human-readable string
- Example: `BTrace Metrics`
- Description: Display name for the extension

**BTrace-Extension-Description**
- Format: multi-line text (continuation with leading space)
- Example: `High-performance metrics with HdrHistogram`
- Description: Detailed description of extension functionality

**BTrace-API-Version**
- Format: version range (major.minor+)
- Example: `3.0+` (requires BTrace API 3.0 or higher)
- Description: Required BTrace API version

**BTrace-Java-Version**
- Format: version number (8+, 11+, etc.)
- Example: `8+`
- Description: Minimum Java version required

**BTrace-Extension-Services**
- Format: comma-separated fully qualified class names
- Example: `io.btrace.metrics.MetricsService,io.btrace.metrics.StatsService`
- Description: Service classes provided by this extension

**BTrace-Extension-Requires**
- Format: comma-separated extension IDs
- Example: `btrace-core,btrace-util`
- Description: Other extensions required by this extension

**BTrace-Shaded-Packages**
- Format: comma-separated package mappings (original->shaded)
- Example: `org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram`
- Description: Package relocation mappings for shaded dependencies

## MANIFEST.MF Line Continuation

Per JAR specification, manifest attributes longer than 72 bytes must be continued on the next line with a leading space:

```
BTrace-Extension-Description: This is a very long description that excee
 ds the 72-byte limit and must be continued on the next line with a lead
 ing space character.
```

## Embedded Extension Properties (`extension.properties`)

Embedded extensions (shipped inside a fat agent JAR) are described by a
`META-INF/btrace-extensions/{id}/extension.properties` file rather than a
MANIFEST.MF. The agent reads the following keys from that file:

| Key | Required | Example | Description |
|-----|----------|---------|-------------|
| `id` | no (defaults to directory name) | `btrace-spark` | Extension identifier |
| `version` | no (defaults to `0.0.0`) | `1.2.0` | Semantic version |
| `name` | no | `BTrace Spark` | Human-readable name |
| `description` | no | `Spark job tracing` | Short description |
| `btrace.api.version` | no (defaults to `3.0+`) | `3.0.0` | Minimum BTrace API version |
| `java.version` | no (defaults to `8+`) | `11+` | Minimum Java version |
| `services` | no | `org.example.SparkService` | Comma-separated service class names |
| `probes` | no | `SparkJobTracer,SparkStageTracer` | Comma-separated bundled probe class names |
| `configurator` | no | `org.example.SparkConfigurator` | Fully qualified `ExtensionConfigurator` class for zero-config probe auto-selection (see below) |

### `configurator` — Zero-Config Probe Auto-Selection

When a `configurator` class is declared, the agent calls it during startup to
decide which bundled probes to activate based on the running JVM's environment.
This allows the extension to enable the right probes automatically (e.g. Spark
driver probes vs. executor probes) without the operator having to pass a
`probes=` agent argument.

The class must implement `io.btrace.core.extensions.ExtensionConfigurator`
and have a public no-arg constructor. It is loaded via the extension's own
classloader. See [BTraceExtensionDevelopmentGuide.md](../BTraceExtensionDevelopmentGuide.md)
for a full example.

## Backward Compatibility

The extension loader supports both formats:

1. Check for MANIFEST.MF attributes first
2. Fall back to `btrace-extension.properties` if MANIFEST attributes not found
3. Log deprecation warning if using properties file

## Gradle Configuration

Extensions should configure MANIFEST.MF generation in build.gradle:

```gradle
jar {
  manifest {
    attributes(
      'BTrace-Extension-Id': 'btrace-metrics',
      'BTrace-Extension-Version': project.version,
      'BTrace-Extension-Name': 'BTrace Metrics',
      'BTrace-Extension-Description': 'High-performance metrics...',
      'BTrace-API-Version': '3.0+',
      'BTrace-Java-Version': '8+',
      'BTrace-Extension-Services': 'io.btrace.metrics.MetricsService',
      'BTrace-Shaded-Packages': 'org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram'
    )
  }
}
```

## Migration Path

1. ✅ Design MANIFEST.MF format (this document)
2. Update ExtensionMetadata parser to read MANIFEST.MF
3. Update btrace-metrics build.gradle to generate MANIFEST attributes
4. Test with both formats for backward compatibility
5. Update documentation
6. Deprecate btrace-extension.properties (remove in future release)
**BTrace-Extension-Permissions**
- Format: comma-separated permission names
- Example: `NETWORK,THREADS,FILE_WRITE`
- Description: Permissions required by the extension implementation. Automatically inferred by the build plugin via code scanning, with optional overrides in Gradle.

---

### Architecture/ExtensionStorageDesign

# BTrace Extension Storage Design

## Overview

This document describes the design for BTrace extension storage and loading mechanism. The goal is to move extensions from the boot JAR to a dedicated extension system with built-in and user-configurable locations.

## Current State

Currently, extensions (like btrace-metrics) are packaged into the boot JAR (`btrace-boot.jar`) along with core BTrace runtime classes. This approach has several limitations:

1. **No Extension Isolation**: Extensions are mixed with core runtime code
2. **No Version Management**: Can't have multiple versions of an extension
3. **No Dynamic Loading**: All extensions loaded at startup, even if unused
4. **Deployment Friction**: Adding new extensions requires rebuilding the distribution
5. **User Extensions**: No mechanism for users to add custom extensions

## Design Goals

1. **Separation**: Extensions stored separately from core runtime
2. **Discoverability**: Automatic discovery of built-in and user extensions
3. **Configuration**: User-configurable extension locations
4. **Lazy Loading**: Load extensions only when referenced in scripts
5. **Isolation**: Extension classloading isolation to avoid conflicts
6. **Metadata**: Extension metadata for versioning and dependencies

## Directory Structure

### Distribution Layout

```
BTRACE_HOME/
├── bin/
│   ├── btrace
│   └── btracec
├── libs/
│   ├── btrace-agent.jar
│   ├── btrace-boot.jar       # Core runtime only, no extensions
│   ├── btrace-client.jar
│   └── extensions/            # Built-in extensions directory
│       ├── btrace-metrics-3.0.0.jar
│       ├── btrace-statsd-3.0.0.jar
│       └── Readme.md
└── docs/
    └── ...
```

### User Extension Locations

Extensions are discovered in the following order (later locations override earlier):

1. **Built-in**: `BTRACE_HOME/extensions/`
2. **System**: `/etc/btrace/extensions/` (Unix) or `%PROGRAMDATA%\btrace\extensions\` (Windows)
3. **User**: `~/.btrace/extensions/`
4. **Environment**: `$BTRACE_EXT_PATH` (colon-separated paths)
5. **Command-line**: `--ext-path <path>` (btrace command)
6. **Script-local**: `./.btrace/extensions/` (relative to script location)

## Extension Structure

### Extension JAR Layout

```
btrace-metrics-3.0.0.jar
├── META-INF/
│   ├── MANIFEST.MF
│   ├── btrace-extension.properties      # Extension metadata
│   └── services/
│       └── io.btrace.core.extensions.Extension
├── io/btrace/metrics/
│   ├── MetricsService.class
│   ├── histogram/
│   └── stats/
└── ... (shaded dependencies)
```

### Extension Metadata (`btrace-extension.properties`)

```properties
# Extension identity
extension.id=btrace-metrics
extension.version=3.0.0
extension.name=BTrace Metrics
extension.description=High-performance metrics with HdrHistogram

# API compatibility
btrace.api.version=3.0+
java.version=8+

# Service providers (optional, can also use META-INF/services)
services=io.btrace.metrics.MetricsService

# Dependencies on other extensions (optional)
requires.extensions=

# Shadowed packages (for conflict detection)
shaded.packages=org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram,\
                com.clearspring.analytics->io.btrace.metrics.shaded.clearspring
```

## Extension Loading Mechanism

### 1. Discovery Phase (on Agent Startup)

```java
public class ExtensionLoader {
    private final List<ExtensionRepository> repositories;

    // Scan all configured locations
    public List<ExtensionDescriptor> discoverExtensions() {
        List<ExtensionDescriptor> extensions = new ArrayList<>();

        for (ExtensionRepository repo : repositories) {
            extensions.addAll(repo.scan());
        }

        // Resolve conflicts (latest version wins)
        return resolveExtensions(extensions);
    }
}
```

### 2. Lazy Loading (on Script Compilation)

When the compiler encounters an `@Injected` service:

```java
// In Compiler.java
private void loadRequiredExtensions(List<String> serviceTypes) {
    for (String serviceType : serviceTypes) {
        ExtensionDescriptor ext = extensionLoader.findExtensionForService(serviceType);
        if (ext != null && !ext.isLoaded()) {
            extensionLoader.load(ext);
        }
    }
}
```

### 3. ClassLoader Hierarchy

```
Bootstrap ClassLoader
    |
System ClassLoader
    |
BTrace Boot ClassLoader (btrace-boot.jar)
    |
    +-- Extension ClassLoader 1 (btrace-metrics)
    |
    +-- Extension ClassLoader 2 (btrace-statsd)
    |
    +-- ...
    |
BTrace Script ClassLoader (compiled script)
```

Each extension gets its own classloader for isolation, but they can see:
- Bootstrap classes
- BTrace core API classes
- Their own classes and shaded dependencies

## Configuration

### Agent Configuration (`btrace.conf`)

```properties
# Extension directories (colon-separated on Unix, semicolon on Windows)
extension.path=${BTRACE_HOME}/extensions:${HOME}/.btrace/extensions

# Extension loading behavior
extension.lazy-load=true
extension.fail-on-missing=false
extension.conflict-resolution=latest-version

# Extension-specific settings
extension.btrace-metrics.enabled=true
extension.btrace-statsd.enabled=true
```

### Environment Variables

```bash
# Override extension path
export BTRACE_EXT_PATH="/opt/btrace-extensions:/usr/local/btrace/ext"

# Disable lazy loading (load all extensions at startup)
export BTRACE_EXT_LAZY_LOAD=false
```

### Command-line Options

```bash
# Specify additional extension directories
btrace --ext-path /custom/extensions PID script.java

# Disable built-in extensions
btrace --no-builtin-ext PID script.java

# Enable specific extensions only
btrace --ext btrace-metrics,btrace-statsd PID script.java
```

## Extension API

### Creating an Extension

1. **Implement Service Interface**:
```java
package com.example.myext;

import io.btrace.core.extensions.Extension;

public class MyExtensionService extends Extension {
    @Override
    public void onStart(BTraceRuntime runtime) {
        // Initialize extension
    }
}
```

2. **Register Service**:
No SPI file required. The Gradle plugin writes manifest attributes (BTrace-Extension-Services).
```
com.example.myext.MyExtensionService
```

3. **Add Metadata**:
Create `META-INF/btrace-extension.properties`:
```properties
extension.id=my-extension
extension.version=1.0.0
btrace.api.version=3.0+
```

4. **Build JAR**:
```gradle
shadowJar {
    // Shade dependencies to avoid conflicts
    relocate 'com.external.lib', 'com.example.myext.shaded.lib'
}
```

5. **Deploy**:
```bash
cp my-extension-1.0.0.jar $BTRACE_HOME/extensions/
# or
cp my-extension-1.0.0.jar ~/.btrace/ext/
```

## Extension Repositories

### Local File System Repository

```java
public class FileSystemExtensionRepository implements ExtensionRepository {
    private final Path extensionDir;

    @Override
    public List<ExtensionDescriptor> scan() {
        List<ExtensionDescriptor> extensions = new ArrayList<>();

        try (DirectoryStream<Path> stream = Files.newDirectoryStream(extensionDir, "*.jar")) {
            for (Path jar : stream) {
                ExtensionDescriptor desc = parseExtension(jar);
                if (desc != null) {
                    extensions.add(desc);
                }
            }
        }

        return extensions;
    }
}
```

### Future: Remote Repository (Maven-like)

```properties
# In btrace.conf
extension.repository.url=https://extensions.btrace.io/repository
extension.repository.cache=${HOME}/.btrace/cache
```

## Migration Path

### Dual mode (backward compatible)
- Keep extensions in boot JAR for backward compatibility
- Add new extension loading mechanism
- Extensions can be discovered from both locations

### Deprecation
- Move built-in extensions to `extensions/` in distribution
- Boot JAR checks if extension exists in extensions/ before loading from itself
- Log deprecation warnings

### Removal
- Remove extensions from boot JAR
- Only load from extension directories

## Implementation Plan

### Stage 1: Extension Discovery (1-2 days)
- [ ] Create `ExtensionDescriptor` class
- [ ] Create `ExtensionRepository` interface and FileSystemRepository
- [ ] Implement extension scanning and metadata parsing
- [ ] Add extension.properties parsing

### Stage 2: Extension Loading (2-3 days)
- [ ] Create `ExtensionClassLoader` with proper parent delegation
- [ ] Implement lazy loading mechanism
- [ ] Add extension lifecycle management (load/unload)
- [ ] Handle ServiceLoader integration with extensions

### Stage 3: Configuration (1-2 days)
- [ ] Add configuration file support
- [ ] Environment variable handling
- [ ] Command-line argument parsing
- [ ] Extension path resolution

### Stage 4: Integration (2-3 days)
- [ ] Integrate with compiler (detect required services)
- [ ] Integrate with agent (load extensions)
- [ ] Update verifier to work with extension classloaders
- [ ] Update build to create ext/ directory structure

### Stage 5: Migration (1-2 days)
- [ ] Move btrace-metrics to ext/ directory
- [ ] Move btrace-statsd to ext/ directory
- [ ] Update documentation
- [ ] Update integration tests

### Stage 6: Testing & Documentation (2-3 days)
- [ ] Unit tests for extension loading
- [ ] Integration tests with multiple extensions
- [ ] Test extension conflicts and resolution
- [ ] Developer guide for creating extensions
- [ ] User guide for installing extensions

**Total Estimated Effort**: 9-15 days

## Security Considerations

1. **Extension Verification**:
   - Check JAR signatures
   - Validate extension metadata
   - Sandbox extension code (SecurityManager)

2. **Trusted Locations**:
   - Built-in extensions (BTRACE_HOME) are trusted
   - User extensions require explicit trust or signature

3. **Classloading Isolation**:
   - Extensions cannot access each other's classes
   - Extensions cannot override core BTrace classes

4. **Resource Limits**:
   - Limit memory usage per extension
   - Timeout for extension initialization

## Performance Considerations

1. **Lazy Loading**: Load extensions only when needed by scripts
2. **Caching**: Cache extension metadata to avoid repeated JAR scanning
3. **Parallel Loading**: Load independent extensions in parallel
4. **Minimal Overhead**: Discovery should add <100ms to agent startup

## Backward Compatibility

1. **Boot JAR Fallback**: If extension not found in ext/, check boot JAR
2. **Service Discovery**: Support both old and new service loading
3. **Configuration**: Old configurations continue to work
4. **Scripts**: Existing scripts work without modification

## Open Questions

1. **Extension Dependencies**: How do extensions depend on each other?
2. **Version Conflicts**: What happens when script needs different versions?
3. **Extension Updates**: Hot-reload extensions without restarting agent?
4. **Distribution**: Package extensions separately or all-in-one?
5. **Remote Loading**: Allow downloading extensions from remote repositories?

## References

- [Java ServiceLoader Documentation](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html)
- [OSGi Bundle Format](https://docs.osgi.org/specification/osgi.core/7.0.0/framework.module.html)
- [Maven Extension Mechanism](https://maven.apache.org/guides/mini/guide-using-extensions.html)

---

### Architecture/Fat Agent Plugin

# Fat Agent Plugin Architecture

## Overview

The Fat Agent Plugin (`io.btrace.fat-agent`) creates self-contained agent JARs with embedded extensions for single-JAR deployment scenarios. This eliminates the need for separate extension installation in environments like Spark, Hadoop, or Kubernetes where managing multiple JARs is impractical.

## Problem Statement

Standard BTrace deployment previously required:
1. Agent JAR (`btrace-agent.jar`)
2. Boot JAR (`btrace-boot.jar`)
3. Extension JARs in `$BTRACE_HOME/extensions/`

> **Note:** The masked JAR architecture now consolidates agent and boot into a single `btrace.jar`. The multi-JAR layout above is the legacy approach.

This multi-JAR setup is problematic for:
- **Spark/Hadoop**: Driver and executors need extensions without shared filesystem
- **Kubernetes**: ConfigMaps and init containers add complexity
- **Containers**: Minimal images don't want extra layers

## Solution: Fat Agent JAR

A single JAR containing:
- All agent and boot classes
- Embedded extension API classes (as `.class` files for bootstrap)
- Embedded extension impl classes (as `.classdata` for runtime loading)
- Extension metadata in `META-INF/btrace-extensions/`

## Architecture

### Class Loading Strategy

```
┌─────────────────────────────────────────────────────────────┐
│                    Fat Agent JAR                             │
├─────────────────────────────────────────────────────────────┤
│  Bootstrap Classpath (via Boot-Class-Path manifest)         │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  io/btrace/agent/...              (agent classes)     │  │
│  │  io/btrace/core/...               (core classes)      │  │
│  │  io/btrace/instr/...              (instr classes)     │  │
│  │  org/example/ext/api/...          (extension API)     │  │
│  └───────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  Runtime-Loaded (via ClassDataLoader)                       │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  org/example/ext/impl/...classdata (extension impl)   │  │
│  └───────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  META-INF/btrace-extensions/                                │
│  ├── ext1/extension.properties                              │
│  ├── ext2/extension.properties                              │
│  └── ext3/extension.properties                              │
└─────────────────────────────────────────────────────────────┘
```

### Extension Discovery Flow

```
Agent Startup
     │
     ▼
┌─────────────────────────┐
│ Parse BTRACE_HOME       │
│ (null for embedded)     │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ Read manifest attribute │
│ BTrace-Embedded-        │
│ Extensions: ext1,ext2   │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ For each extension ID:  │
│ - Load extension.props  │
│ - Create descriptor     │
│ - Register services     │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ API classes already on  │
│ bootstrap (as .class)   │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ Impl classes loaded on  │
│ demand via ClassData-   │
│ Loader (from .classdata)│
└─────────────────────────┘
```

### Plugin Task Graph

```
fatAgentJar
├── stageExtensions
│   ├── resolveExtensions()  → ResolvedExtension[]
│   ├── stageApiClasses()    → copy as .class
│   ├── stageImplClasses()   → copy as .classdata
│   └── writeMetadata()      → extension.properties
├── stageProbes (optional)
│   ├── copyCompiledProbes() → META-INF/btrace-probes/
│   └── compileSourceProbes()
└── btraceJar (single masked JAR from btrace-dist)
```

## Implementation Details

### Extension Sources

The plugin supports three extension source types:

```groovy
embedExtensions {
    project(':my-extension')           // ProjectExtensionSource
    maven('io.btrace:ext:1.0')         // MavenExtensionSource
    file('/path/to/ext.zip')           // FileExtensionSource
}
```

Each source resolves to a `ResolvedExtension`:

```groovy
class ResolvedExtension {
    String id
    String version
    File apiJar      // contains API classes
    File implJar     // contains impl classes (shadowed)
    Properties metadata
}
```

### Staging Process

1. **API Classes**: Extracted from API JAR and copied as `.class` files
   - These end up on bootstrap classpath
   - Visible to BTrace scripts and the agent

2. **Impl Classes**: Extracted from impl JAR and renamed to `.classdata`
   - Loaded at runtime by `ClassDataLoader`
   - Isolated from target application classpath

3. **Metadata**: Written to `META-INF/btrace-extensions/{id}/extension.properties`

### Auto-Discovery

When `autoDiscover = true`, the plugin scans subprojects:

```groovy
project.gradle.projectsEvaluated {
    rootProject.subprojects.each { sp ->
        if (sp.plugins.hasPlugin('io.btrace.extension')) {
            extension.addExtensionSource(new ProjectExtensionSource(project, sp.path))
        }
    }
}
```

The selection is narrowed in two ways. `defaultExtensions` names the subset to embed when the
filter property is absent, matching either a subproject's name or its path; the filter property
overrides it and can select anything discoverable:

```bash
./gradlew fatAgentJar -PembedExtensions=btrace-metrics,btrace-statsd
```

`btrace-dist` sets `defaultExtensions` to the released extension set, so an ordinary
`fatAgentJar` build embeds exactly what a release ships rather than every extension in the
repository — the test fixture and the Spark and Hadoop examples are excluded.

### ShadowJar Integration

When ShadowJar is available on the classpath, the plugin uses it for:
- Package relocation (avoid classpath conflicts)
- Duplicate handling

```groovy
def jarTaskClass = Jar
try {
    jarTaskClass = Class.forName('com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar')
} catch (ClassNotFoundException e) {
    // Fall back to standard Jar
}
```

## Manifest Attributes

The fat agent JAR manifest includes:

| Attribute | Description |
|-----------|-------------|
| `Premain-Class` | `io.btrace.boot.Loader` |
| `Agent-Class` | `io.btrace.boot.Loader` |
| `BTrace-Agent-Main` | `io.btrace.agent.Main` |
| `Can-Redefine-Classes` | `true` |
| `Can-Retransform-Classes` | `true` |
| `Boot-Class-Path` | `btrace-agent-fat.jar` (self-reference) |
| `BTrace-Embedded-Extensions` | Comma-separated list of extension IDs |

After assembly, the plugin verifies that the JAR contains `io/btrace/boot/Loader.class`, the masked
`META-INF/btrace/agent/io/btrace/agent/Main.classdata` entry, and the mandatory loader manifest
attributes. A conventional agent JAR is not a valid base input; `agentJarTask` must reference the
task producing the masked BTrace JAR (normally `btraceJar`).

## Runtime Behavior

### Extension Loading

At agent startup:

1. `Main.initExtensions()` initializes the extension system
2. `ExtensionLoader` creates `EmbeddedExtensionRepository`
3. Repository reads `BTrace-Embedded-Extensions` from manifest
4. For each extension ID, loads `META-INF/btrace-extensions/{id}/extension.properties`
5. Creates `ExtensionDescriptorDTO` with `embedded=true`
6. API classes are already on bootstrap (no loading needed)
7. Impl classes loaded on-demand via `ClassDataLoader`

### ClassDataLoader

The `ClassDataLoader` loads `.classdata` files as classes. It registers as parallel-capable via `ClassLoader.registerAsParallelCapable()` and uses per-class-name locking via `getClassLoadingLock(name)` to allow concurrent loading of different classes while serializing attempts to load the same class:

```java
public class ClassDataLoader {
    static {
        ClassLoader.registerAsParallelCapable();
    }

    public Class<?> findClass(String className) {
        synchronized (getClassLoadingLock(className)) {
            String resourceName = className.replace('.', '/') + ".classdata";
            InputStream is = getResourceAsStream(resourceName);
            byte[] bytes = is.readAllBytes();
            return defineClass(className, bytes, 0, bytes.length);
        }
    }
}
```

## Use Cases

### 1. Spark Deployment

```groovy
btraceFatAgent {
    baseName = 'btrace-spark-agent'
    embedExtensions {
        project(':btrace-extensions:btrace-spark')
        project(':btrace-extensions:btrace-metrics')
    }
}
```

Usage:
```bash
spark-submit --conf spark.driver.extraJavaOptions=-javaagent:btrace-spark-agent.jar ...
```

### 2. Kubernetes with Pre-loaded Extensions

```dockerfile
FROM btrace/btrace:latest AS btrace
FROM openjdk:17

# Copy only the fat agent (no extension installation needed)
COPY --from=btrace /opt/btrace/libs/btrace-agent-fat.jar /opt/btrace/
```

### 3. CI/CD Pipeline

```yaml
steps:
  - name: Build Fat Agent
    run: ./gradlew fatAgentJar -PembedExtensions=btrace-metrics

  - name: Deploy
    run: kubectl cp btrace-agent-fat.jar pod:/opt/
```

## Maven Plugin

The unpublished Maven fat-agent module was removed for 3.0.0. It resolved artifacts and staged
classdata according to pre-3.0 contracts, which could produce a successful build whose embedded
services could not load. The Gradle plugin described in this guide is the supported 3.0 fat-agent
implementation.

## Limitations

1. **No Hot-Reload**: Embedded extensions cannot be updated without rebuilding the JAR
2. **Size**: Fat JAR is larger than minimal agent
3. **Classpath Conflicts**: Careful relocation needed to avoid conflicts with target app dependencies

## Related Documentation

- [Extension Development Guide](../BTraceExtensionDevelopmentGuide.md)
- [Provided-Style Extensions](provided-style-extensions.md)
- [Migrating from libs/profiles](migrating-from-libs-profiles.md)

---

### Architecture/InstrumentationBackends

# Instrumentation Backends

**Document Version:** 1.0
**Last Updated:** July 2026
**Status:** Implemented (v3.0.0+)

---

## Overview

BTrace performs bytecode instrumentation through a small internal SPI, `InstrumentationBackend`, with two implementations:

| Backend | Source set | Availability | Class file versions |
|---------|-----------|--------------|---------------------|
| `AsmInstrumentationBackend` | `src/main/java` (Java 8) | Always | ≤ 69 (up to Java 25) |
| `ClassFileApiBackend` | `src/main/java24` (Java 24) | Agent running on JDK 24+ | > 69 (Java 26+) |

All types live in the `io.btrace.instr` package of the **btrace-agent** module:

- `btrace-agent/src/main/java/io/btrace/instr/InstrumentationBackend.java`
- `btrace-agent/src/main/java/io/btrace/instr/AsmInstrumentationBackend.java`
- `btrace-agent/src/main/java/io/btrace/instr/BackendSelector.java`
- `btrace-agent/src/main/java/io/btrace/instr/ClassMeta.java`
- `btrace-agent/src/main/java24/io/btrace/instr/ClassFileApiBackend.java`

## Why: the ASM Ceiling

BTrace's instrumentation pipeline is built on ASM. ASM can only parse class files up to a major version it explicitly knows about; ASM 9.9.x tops out at class file major version **69 (Java 25)** and throws when handed anything newer:

```java
/** Highest class file major version ASM 9.9.x can parse without throwing. */
static final int MAX_ASM_MAJOR_VERSION = 69; // Java 25
```

Without an alternative backend, an application compiled for Java 26+ (class file major version 70+) could not be instrumented at all. The JDK ClassFile API (`java.lang.classfile.*`, standardized in JDK 24) always understands the class file format of the JDK it ships with, so it provides a forward-compatible path for such classes.

## The SPI

```java
interface InstrumentationBackend {

  /** Returns {@code true} when this backend can process the given class file major version. */
  boolean supports(int classFileMajorVersion);

  /**
   * Instruments {@code classfileBuffer} by applying all applicable probes.
   *
   * @param loader the classloader loading the target class (may be {@code null})
   * @param classfileBuffer raw class file bytes
   * @param probes all currently registered probes
   * @return transformed class bytes if at least one probe matched, {@code null} otherwise
   */
  byte[] instrument(ClassLoader loader, byte[] classfileBuffer, Collection<BTraceProbe> probes);
}
```

The `ClassMeta` interface complements the SPI: it exposes the minimal class metadata (Java class name, internal name, runtime-visible annotation types, classloader) needed for probe-to-class matching in `BTraceProbeSupport`, decoupling the matching logic from ASM's `ClassReader` so alternative backends can perform matching without constructing an ASM object.

## Backend Selection

`BackendSelector` chooses a backend based on the class file major version. The selection logic, verbatim:

```java
static InstrumentationBackend select(int classFileMajorVersion) {
  if (!ASM.supports(classFileMajorVersion) && CLASSFILE_API != null) {
    return CLASSFILE_API;
  }
  return ASM;
}
```

In other words:

1. Class file version ≤ 69 → ASM backend (the default, full-featured path).
2. Class file version > 69 and the ClassFile API backend is available → ClassFile API backend.
3. Class file version > 69 but the ClassFile API backend is unavailable (agent running on JDK < 24) → falls back to ASM, which will fail to parse the class; instrumentation of that class is effectively skipped.

The ClassFile API backend is loaded **reflectively** at class-initialization time so the main (Java 8-compiled) source set has no compile-time dependency on `java.lang.classfile`:

```java
Class<?> cls =
    Class.forName(
        "io.btrace.instr.ClassFileApiBackend", true, BackendSelector.class.getClassLoader());
return (InstrumentationBackend) cls.getDeclaredConstructor().newInstance();
```

On JDK < 24 the `Class.forName` fails (the compiled class targets class file version 68/Java 24 and references `java.lang.classfile`), the failure is logged at debug level, and the field stays `null`.

## ClassFile API Backend

### Requirements

- The **agent must run on JDK 24+** — the backend is compiled with `sourceCompatibility = 24` / `targetCompatibility = 24` and uses `java.lang.classfile.*`.
- It is engaged only for **class file major versions > 69** (`supports()` returns `classFileMajorVersion > AsmInstrumentationBackend.MAX_ASM_MAJOR_VERSION`).

### How It Instruments

The backend parses the class with `ClassFile.parse()`, builds a `ClassMeta` from the class model (name, runtime-visible annotations, classloader), collects applicable handlers via `BTraceProbe.getApplicableHandlers(meta)`, and injects probe calls as `invokedynamic` instructions bootstrapped by `io.btrace.runtime.IndyDispatcher.bootstrap(...)` — entry probes before the first real instruction, return probes before each `ReturnInstruction`.

### Current Limitations

Verified in `ClassFileApiBackend.java`:

- **Only `Kind.ENTRY` and `Kind.RETURN` probes are supported.** Handlers with any other probe kind (CALL, LINE, FIELD_GET/SET, ERROR, etc.) are skipped with a debug-level log; the remaining handlers are still applied.
- Method matching supports exact names and `/regex/` patterns; **type-constrained method matching** (a non-empty `type` in `@OnMethod`) is unsupported — such handlers are skipped.
- Supported handler parameters: `@ProbeClassName`, `@ProbeMethodName`, and `@Self` (on instance methods; `null` is passed for static methods and constructor entry). Handlers using other special parameters (`@Return`, `@TargetInstance`, `@Duration`, `@TargetMethodOrField`) or plain probed-method arguments are skipped.
- Classes the ClassFile API fails to parse are skipped (warning logged) rather than failing class loading.

## Packaging

The `java24` source set is declared in `btrace-agent/build.gradle` and compiled with a JDK 24 toolchain. Its output is merged into the **root** of the regular agent jar:

```gradle
jar {
    into('') {
        from sourceSets.java24.output
    }
}
```

Important consequences:

- The agent jar is **NOT a Multi-Release JAR** — there is no `META-INF/versions/24/` entry and no `Multi-Release` manifest attribute. The compiled-at-24 `ClassFileApiBackend.class` sits at the jar root next to the Java 8-compatible classes.
- The class is never referenced directly from Java 8-compiled code; it is only ever loaded **reflectively** by `BackendSelector`. On older JDKs the reflective load fails cleanly (`UnsupportedClassVersionError` caught as `Throwable`), and BTrace continues with the ASM backend only.

## Related Documents

- [ExtensionInvokeDynamicBridge](ExtensionInvokeDynamicBridge.md) — invokedynamic-based extension linkage

---

### Architecture/MaskedJarArchitecture

# Masked JAR Architecture

## Overview

BTrace uses a single-JAR distribution (`btrace.jar`) with a **classdata masking** technique to minimize bootstrap classloader pollution. This architecture packs agent, client, and shared classes into one JAR while keeping only essential API classes visible to the bootstrap classloader.

The key insight: files with a `.classdata` extension are invisible to the JVM's built-in classloading, so non-bootstrap classes can coexist in the same JAR without leaking into the bootstrap classloader.

## Problem

Traditional Java agents add their entire JAR to the bootstrap classloader via `Boot-Class-Path: .` in the manifest. For BTrace, this would expose ~1500+ classes (ASM, compiler, instrumentation engine, client) to every class in the JVM. This causes:

- Namespace pollution (classes visible where they shouldn't be)
- Potential conflicts with application dependencies (e.g., different ASM versions)
- Increased memory footprint in the bootstrap classloader

The previous multi-JAR approach (`btrace-agent.jar`, `btrace-boot.jar`, `btrace-client.jar`) solved this by splitting classes across JARs but introduced hardcoded co-location assumptions that broke alternative distribution methods like jbang and Maven repositories.

## Architecture

### JAR Structure

```
btrace.jar (~2.9 MB)
├── io/btrace/boot/*.class                      # Entry point (Loader, MaskedClassLoader, MaskedJarUtils)
├── io/btrace/core/*.class                      # Bootstrap: core API (excl. Messages, comm, extensions)
├── io/btrace/core/annotations/*.class          # Bootstrap: annotations
├── io/btrace/core/handlers/*.class             # Bootstrap: handler interfaces
├── io/btrace/core/types/*.class                # Bootstrap: type definitions
├── io/btrace/core/jfr/*.class                  # Bootstrap: JFR integration
├── io/btrace/runtime/*.class                   # Bootstrap: selected runtime support classes
├── io/btrace/extension/ExtensionBridge.class   # Bootstrap: extension bridge interface
├── io/btrace/libs/org/slf4j/**                 # Bootstrap: relocated SLF4J
├── META-INF/btrace/agent/*.classdata           # Masked: agent classes
├── META-INF/btrace/client/*.classdata          # Masked: client classes
├── META-INF/btrace/shared/*.classdata          # Masked: shared classes (ASM, protocol, etc.)
└── META-INF/MANIFEST.MF
```

**Bootstrap classes** (~112 total): Only the core API, runtime support, and SLF4J logging are stored as regular `.class` files. These are visible to the bootstrap classloader because the manifest declares `Boot-Class-Path: .`.

**Masked classes** (~1600+): Agent, client, and shared classes are stored as `.classdata` files under `META-INF/btrace/`. The JVM's class loading ignores these files entirely. They are loaded on demand by `MaskedClassLoader`.

The client section also contains the instrumentation/runtime implementation used by `btracec` to
turn compiled scripts into persisted probe packs. The `io.btrace.core.PackGenerator` service
descriptor is visible from the masked JAR root, so its `io.btrace.instr.InstrPackGenerator`
provider and non-bootstrap dependencies must be loadable by the client masked classloader as well
as by the agent classloader.

### Manifest

```
Premain-Class: io.btrace.boot.Loader
Agent-Class: io.btrace.boot.Loader
Main-Class: io.btrace.boot.Loader
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Boot-Class-Path: .
BTrace-Agent-Main: io.btrace.agent.Main
BTrace-Client-Main: io.btrace.client.Main
```

`Loader` is the single entry point for all three modes. The actual agent/client main classes are specified as manifest attributes and loaded reflectively via `MaskedClassLoader`.

## Components

### btrace-boot Module

**Location:** `btrace-boot/`

Contains three classes, all loaded by the bootstrap classloader:

#### Loader (`io.btrace.boot.Loader`)

Entry point for all three modes:

| Mode | Entry Method | Invocation |
|------|-------------|------------|
| Load-time agent | `premain(String, Instrumentation)` | `-javaagent:btrace.jar` |
| Dynamic attach | `agentmain(String, Instrumentation)` | `VirtualMachine.loadAgent()` |
| Client CLI | `main(String[])` | `java -jar btrace.jar` |

Each mode creates a `MaskedClassLoader` for the appropriate section (agent or client), loads the main class from the manifest attribute, and invokes it reflectively.

**Agent mode classloader hierarchy:**
```
Bootstrap CL (core API, runtime, Loader)
    └── MaskedClassLoader[agent] (parent=null)
            Loads from: META-INF/btrace/agent/*.classdata
            Fallback:   META-INF/btrace/shared/*.classdata
```

Using `null` as the parent ensures bootstrap-visible classes (like `BTraceRuntimeAccess`) are loaded from the bootstrap classloader rather than the agent's classloader. This is critical because probe classes—defined via the bootstrap classloader—must see the same class instances the agent uses.

**Client mode classloader hierarchy:**
```
Bootstrap CL (core API, runtime, Loader)
    └── System CL
        └── MaskedClassLoader[client] (parent=System CL)
                Loads from: META-INF/btrace/client/*.classdata
                Fallback:   META-INF/btrace/shared/*.classdata
```

#### MaskedClassLoader (`io.btrace.boot.MaskedClassLoader`)

A `URLClassLoader` subclass that loads classes from `.classdata` files. The lookup order for `findClass(name)`:

1. Check section-specific path: `META-INF/btrace/{section}/{class-path}.classdata`
2. Check shared path: `META-INF/btrace/shared/{class-path}.classdata`
3. Throw `ClassNotFoundException` (delegates to parent via standard classloader contract)

Resource loading (`findResource`, `getResourceAsStream`) follows the same pattern: section-specific first, then shared, then the JAR root.

#### MaskedJarUtils (`io.btrace.boot.MaskedJarUtils`)

Utility class for detecting masked JARs. A JAR is identified as masked by the presence of a `META-INF/btrace/shared/` entry. Used by `Client.java` and integration tests to detect whether they are running from a masked JAR or the legacy multi-JAR layout.

### Bootstrap Class Selection

**Location:** `btrace-dist/build.gradle` (the `bootIncludes` filter closure)

The `bootIncludes` filter closure defined in `btrace-dist/build.gradle` determines which classes belong in the bootstrap section. It is used by the `bootstrapJar` and `btraceJar` tasks to separate classes into `.class` (bootstrap) vs `.classdata` (masked) during JAR assembly.

**Bootstrap inclusion criteria:**
- `io/btrace/core/` — core API, including annotations and handlers (excluding `Messages`, `messages.properties`, `comm/`, and `extensions/`)
- `io/btrace/runtime/` — selected runtime support classes only (`BTraceRuntimeAccess`, `LinkingFlag`, `Indy*`, `ExtensionIndy`, `BTraceBootstrap`, `auxiliary/Auxiliary`)
- `io/btrace/BTrace.class` — flat DSL entry class (needed by javac when compiling scripts)
- `io/btrace/extension/ExtensionBridge.class` — extension bridge interface (referenced by `ExtensionIndy`)
- `io/btrace/libs/org/slf4j/` — relocated SLF4J (logging from bootstrap code)

**Explicitly excluded from bootstrap:**
- ASM classes (loaded via shared section)
- JCTools queues (loaded via shared section)
- Communication protocol classes (`io/btrace/core/comm/`)
- Extension API classes (`io/btrace/core/extensions/`)

## Build Process

The `btrace-dist/build.gradle` `btraceJar` task assembles the masked JAR:

1. **`prepareAgentClassdata`**: Extracts classes from the agent shadow JAR, renames `.class` → `.classdata`, places under `META-INF/btrace/agent/`
2. **`prepareClientClassdata`**: Same for the client shadow JAR, placed under `META-INF/btrace/client/`
3. **`btraceJar`**: Combines bootstrap `.class` files (filtered by `btrace-bootstrap`) with the `.classdata` sections and shared classes into the final JAR

Shared classes (ASM, protocol, etc.) are automatically placed in `META-INF/btrace/shared/` and are accessible to both agent and client classloaders.

## Debugging

Enable debug output with:
```bash
java -Dbtrace.boot.debug=true -jar btrace.jar ...
```

This prints classloader decisions to stderr, prefixed with `[BTrace Boot]`.

## Backward Compatibility

The masked JAR coexists with the legacy multi-JAR distribution. Detection logic in `Client.java` and `RuntimeTest` checks for `btrace.jar` first and falls back to `btrace-agent.jar` if not found. The `--agent-jar` and `--boot-jar` CLI flags allow explicit path overrides for any layout.

---

### Architecture/Migrating From Libs Profiles

# Migrating from libs/profiles to Extensions

This guide helps you move from `btrace-libs/<profile>` to extension-based integrations.

## Why Migrate

- Security & isolation: extensions use API on bootstrap and isolated impls; libs/profiles mutate global classpaths.
- Operability: discovery, enable/disable, permissions, diagnostics.
- Maintainability: versioning and conflict handling.

## Status

`libs`/profiles are **removed**. Passing `libs=<profile>` logs an error naming the profile and
loads nothing; the jars under `btrace-libs/<profile>/` are ignored. Migrate to an extension, or
use the escape hatch below as a stopgap.

If you are arriving here because custom classes stopped resolving after upgrading, that is the
expected symptom: the agent starts normally and probes fail later on the missing types.

## Migration Steps

1. Inventory usage:
   - Identify profile(s) used (e.g., `spark-driver`, `spark-executor`, `hadoop`).
   - List the APIs probes rely on; isolate to a minimal contract.

2. Create an extension:
   - One Gradle project with a single `src/main/java`, applying `io.btrace.extension`. The plugin
     partitions it into API and implementation artifacts from the declared services plus any
     `additionalExports`/`excludedExports` — there is no separate API module to create.
   - Keep application types out of the API surface: minimal interfaces and DTOs only.
   - Declare application dependencies with `implCompileOnly` so they stay off the API artifact.

3. Replace profile dependency:
   - Remove `libs=<profile>` from agent args.
   - Enable the extension in `extensions.conf` (`extensions.enabled`, `extensions.disabled`,
     `extensions.autoload`).

4. Replace classpath assumptions:
   - Update probes to pass app objects to API methods (object hand-off) instead of importing app types.
   - In impl, resolve types via TCCL/defining loader.
   - Recompile the probes against the new API artifact.

5. Configure runtime:
   - Add role/config keys to `extensions.conf` (e.g., role=driver|executor; optional classpath hint if the app doesn’t ship libs).
   - The extension plugin scans the implementation and its dependencies and writes the merged
     permission set into the API manifest, so permissions are usually not declared by hand.
     Review what it wrote: a single transitive dependency can make the whole extension privileged.
   - Permission *grants* are a separate, operator-side file — `permissions.properties`
     (`allowExtensions`, `denyExtensions`, `allowPrivileged`), not `extensions.conf`.

6. Validate:
   - Launch/attach runs; verify extension loads and links lazily/eagerly as required.
   - Injection throws by default. Marking an injection optional, or selecting shim mode, turns a
     failed link into a no-op returning defaults — convenient in production, and the quickest way
     to make an unfinished migration look complete.
   - Use `btrace -le <PID>` to see why an extension failed to link.

## Escape Hatch (Optional)

If immediate migration is not feasible and the app must see a jar on the system classpath:

```
-Dbtrace.system.appendJar=/abs/path/lib.jar -javaagent:btrace.jar=trusted=true
```

- `trusted` is an **agent argument**, not a system property; `-Dbtrace.trusted=true` has no effect
  here.
- Restricted to `BTRACE_HOME` unless `-Dbtrace.allowExternalLibs=true`. When `BTRACE_HOME` cannot
  be determined the jar is appended anyway and a warning is logged.
- One jar only; discouraged; subject to removal.

## Fat Agent Deployment

For Spark/Hadoop/Kubernetes environments where managing separate extension JARs is impractical, use fat agent builds to embed extensions directly.

### Gradle Plugin

```groovy
plugins {
    id 'io.btrace.fat-agent'
}

btraceFatAgent {
    baseName = 'my-btrace-agent'
    embedExtensions {
        project(':my-spark-extension')
        file('/path/to/btrace-metrics-3.0.0-extension.zip')
    }
}
```

Build: `./gradlew fatAgentJar`

### Maven Plugin

The unpublished Maven fat-agent module was removed for 3.0.0. Migrate fat-agent builds to the
Gradle configuration above; the external Maven plugin remains available for script compilation,
not embedded-extension packaging.

### Usage

```bash
# Spark
spark-submit --conf spark.driver.extraJavaOptions=-javaagent:my-btrace-agent.jar ...

# Kubernetes
java -javaagent:/opt/btrace/my-btrace-agent.jar MyApp
```

See [Fat Agent Plugin Architecture](fat-agent-plugin.md) for details.

## Examples & Templates

- See provided-style extension guide: `docs/architecture/provided-style-extensions.md` for Spark/Hadoop templates and `extensions.conf` snippets.

## Assisted Migration

The [BTrace agent plugins](https://github.com/btraceio/agent-plugins) marketplace provides a
[`btrace-legacy-libs-migration`](https://github.com/btraceio/agent-plugins/blob/main/plugins/btrace-observability/skills/btrace-legacy-libs-migration/SKILL.md)
skill for Claude Code, Codex, and Pi. It walks the inventory, the API/implementation split, the
service and permission metadata, and the verification steps described above.

---

### Architecture/Provided Style Extensions

# Provided-Style Extensions for App Integrations

This guide shows how to migrate profile-based integrations (e.g., Spark/Hadoop) to BTrace extensions without mutating the global classpath.

## Principles

- API on bootstrap: expose minimal, stable APIs with simple/value types.
- Impl in isolated CL: load implementation in an extension classloader; no shading of app libs.
- Runtime linking: access application types via object hand-off and TCCL instead of compile-time imports.
- No classpath injection: avoid boot/system CL changes; reserve the escape hatch for exceptional cases only.

## Helper Utilities

- `io.btrace.extension.util.ClassLoadingUtil`
  - Loaders: `tccl()`, `definingLoader(Object)`
  - Class loading: `load(String, ClassLoader)`, `load(String, Object)`, `tryLoad(String, ClassLoader)`
  - Context: `withTCCL(ClassLoader, Supplier<T>)`, `withTCCL(ClassLoader, Runnable)`, `withDefiningLoader(Object, Supplier<T>)`
  - Services: `loadService(Class<T>, ClassLoader)`, `loadServices(Class<T>, ClassLoader)`
  - Resources: `getResource(String, ClassLoader)`, `openResource(String, ClassLoader)`
  - Optional child loader: `newChildURLClassLoader(List<Path>, ClassLoader)`, `safeClose(ClassLoader)`

- `io.btrace.extension.util.MethodHandleCache`
  - Caches successful public static and virtual method lookups only. Its strong owner-class keys can retain application loaders; it has no getter, setter, or constructor helper.

## API Sketch (Spark example)

```java
// exported API (on bootstrap)
package org.example.btrace.spark.api;
public interface SparkApi {
  void onJobStart(Object jobStartEvent);
  void onStageCompleted(Object stageInfo);
}
```

## Impl Sketch

```java
// implementation (extension CL)
package org.example.btrace.spark.impl;
import org.example.btrace.spark.api.SparkApi;
import io.btrace.extension.util.ClassLoadingUtil;
import io.btrace.extension.util.MethodHandleCache;
import java.lang.invoke.MethodHandle;

public final class SparkApiImpl implements SparkApi {
  private final MethodHandleCache mh = new MethodHandleCache();

  @Override
  public void onJobStart(Object evt) {
    ClassLoadingUtil.withDefiningLoader(
      evt,
      () -> {
        try {
          Class<?> cls = ClassLoadingUtil.loadFromContext(
              "org.apache.spark.scheduler.SparkListenerJobStart", evt);
          MethodHandle getJobId = mh.findVirtual(cls, "jobId", int.class);
          int jobId = (int) getJobId.invoke(evt);
          // emit metrics/logs...
        } catch (Throwable t) {
          // log and continue
        }
        return null;
      });
  }
}
```

## External Type Adapters

`@ExternalType` supports exact public methods, opaque target-type positions, complete overload
groups, exact public field getters/setters, constructors, and `isInstance`/`cast` operations. The
canonical signatures, owner-loader, failure, and static-loader rules are in the [Extension
Development Guide](../BTraceExtensionDevelopmentGuide.md#app-type-adapters-with-externaltype).
Keep the manual pattern here for generic/array target types, bulk reflection-style work, fluent
setters, constructor selectors, runtime coercion, and non-public or JPMS-bypassing access.

For a generated static target call where the application loader is known, pass it directly:

```java
String version = VersionApi$Ext.version(appLoader);
```

Reserve `ClassLoadingUtil.withTCCL` for manual APIs that require a TCCL policy.

For fields and constructors there is no `MethodHandleCache` convenience method. Use a direct public lookup after resolving the target class:

```java
Class<?> type = ClassLoadingUtil.load("com.example.AppType", appLoader);
MethodHandle constructor = MethodHandles.publicLookup().findConstructor(type, MethodType.methodType(void.class));
```

## Role Detection & Config

- Detect driver/executor via system properties or presence of marker classes using TCCL.
- `extensions.conf` (examples):

```properties
# Spark
btrace-spark.enabled=true
btrace-spark.role=auto           # auto|driver|executor
# optional: only if the app does not ship required libs
btrace-spark.classpath=/opt/spark/jars

# Hadoop
btrace-hadoop.enabled=false
# btrace-hadoop.classpath=/opt/hadoop/share/hadoop/common
```

## Permissions

- Typical: `REFLECTION`, `THREADS`, `SYSTEM_PROPS`.
- Optional: `CLASSLOADER` if creating a child `URLClassLoader` from configured paths.

## Escape Hatch (last resort)

- If absolutely unavoidable, append a single jar to the system CL:
  - `-Dbtrace.system.appendJar=/abs/path/lib.jar -Dbtrace.trusted=true`
  - Restricted to `BTRACE_HOME` by default; override with `-Dbtrace.allowExternalLibs=true`.

## Hadoop Example (Sketch)

```java
public interface HadoopApi { void onFsOp(Object op); }
public final class HadoopApiImpl implements HadoopApi {
  // Resolve org.apache.hadoop.fs.FileSystem via TCCL and reflectively extract fields
}
```

## Migration Steps

1. Extract minimal API for probes; avoid app types.
2. Move environment-specific logic to impl; resolve app types via object hand-off/TCCL.
3. Add extension config (role, optional classpath hints).
4. Request permissions; add no-op shims when unavailable.
5. Prefer eager load if APIs must be present before probes start.

## Notes

- Keep APIs small and stable; impls can evolve independently.
- Cache MethodHandles for performance; avoid repeated reflective lookups.
- Do not rely on global classpath mutation; it’s discouraged and may be removed.

## Example Projects

- Spark example: `btrace-extensions/examples/btrace-spark`
- Hadoop example: `btrace-extensions/examples/btrace-hadoop`

See also: `docs/examples/README.md` for quick build and configuration snippets.

---

### Architecture/Version2ProtocolArchitecture

# BTrace v2 Binary Protocol Architecture

**Document Version:** 1.1
**Last Updated:** February 2026
**Status:** Implemented (first released in 3.0.0)

---

## Table of Contents

1. [Executive Summary](#executive-summary)
2. [Problem Statement](#problem-statement)
3. [High-Level Architecture](#high-level-architecture)
4. [Protocol Negotiation](#protocol-negotiation)
5. [Wire Format Specification](#wire-format-specification)
6. [Command Conversion Layer](#command-conversion-layer)
7. [Benefits and Trade-offs](#benefits-and-trade-offs)
8. [Migration Path](#migration-path)
9. [Performance Characteristics](#performance-characteristics)

---

## Executive Summary

The BTrace v2 binary protocol is a performance-optimized communication protocol that replaces Java Object Serialization with custom binary serialization. The v2 protocol delivers **3-6x faster** command transmission and **2-5x smaller** wire payloads while maintaining full backward compatibility with the existing v1 protocol through automatic protocol negotiation.

**Key Features:**
- Custom binary serialization (vs Java ObjectInputStream/ObjectOutputStream)
- Automatic protocol negotiation (once per connection)
- Compression support for large payloads (>1KB threshold)
- Thread-safe using ReentrantLock (vs synchronized blocks)
- Full backward compatibility with v1 protocol
- Zero-configuration auto-detection

---

## Problem Statement

### What Problem Does v2 Solve?

The original BTrace protocol (v1) relies on Java Object Serialization for agent-client communication. While functional, this approach has significant limitations:

#### 1. Performance Bottleneck
**Problem:** Java serialization is slow
- ObjectInputStream/ObjectOutputStream use reflection and complex state management
- Each command incurs serialization overhead (object graphs, metadata)
- High CPU usage during marshaling/unmarshaling

**Impact:**
- Limits throughput for high-frequency tracing scenarios
- Increases latency for interactive debugging
- Consumes CPU resources that could be used for actual tracing

#### 2. Large Wire Payloads
**Problem:** Java serialization produces verbose binary format
- Includes class metadata, type descriptors, stream headers
- Inefficient encoding of primitive types and strings
- No built-in compression

**Impact:**
- Increased network bandwidth usage
- Slower transmission over slow connections
- Higher memory usage for buffering

#### 3. Language Lock-in
**Problem:** Java serialization ties BTrace to JVM-only clients
- Cannot implement clients in other languages (Python, Go, JavaScript)
- Limits future extensibility (browser-based tools, IDE plugins in non-JVM languages)

**Impact:**
- Restricts ecosystem growth
- Prevents integration with non-Java monitoring tools

#### 4. Dated Concurrency Model
**Problem:** v1 uses `synchronized` blocks for thread safety
- Coarse-grained locking can become bottleneck
- Limited scalability for concurrent client connections

**Impact:**
- Performance degradation with multiple concurrent clients
- Contention under high load

### Real-World Scenario

Consider a production environment with high-frequency tracing:

**v1 Protocol:**
```
10,000 MessageCommands/second
Average size: 512 bytes serialized
Network: 5.12 MB/second
CPU overhead: ~15% for serialization
```

**v2 Protocol:**
```
10,000 MessageCommands/second
Average size: 170 bytes serialized (3x smaller with compression)
Network: 1.7 MB/second (67% reduction)
CPU overhead: ~3% for serialization (80% reduction)
```

**Result:** 67% less bandwidth, 80% less CPU overhead, same functionality

---

## High-Level Architecture

### Component Overview

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Key Components

#### 1. WireProtocol Interface
**Purpose:** Abstract wire format from business logic

**Location:** `btrace-core/src/main/java/io/btrace/core/comm/WireProtocol.java`

**Responsibilities:**
- Define contract for reading/writing Command objects
- Abstract away serialization mechanism
- Support protocol version introspection

**Interface:**
```java
public interface WireProtocol {
    Command read(InputStream in) throws IOException;
    void write(OutputStream out, Command cmd) throws IOException;
    void reset() throws IOException;  // for ObjectOutputStream.reset() in v1
    int getVersion();
}
```

#### 2. Protocol Negotiator
**Purpose:** Auto-detect and negotiate protocol version

**Location:** `btrace-core/src/main/java/io/btrace/core/comm/ProtocolNegotiator.java`

**Responsibilities:**
- Perform handshake at connection establishment
- Detect client/agent protocol capabilities
- Select optimal protocol version
- Handle negotiation timeouts and failures

#### 3. Command Adapter
**Purpose:** Convert between v1 and v2 command representations

**Location:** `btrace-core/src/main/java/io/btrace/core/comm/v2/CommandAdapter.java`

**Responsibilities:**
- Bidirectional conversion: Command ↔ BinaryCommand
- Preserve all command data during conversion
- Handle type mismatches gracefully

#### 4. Binary Protocol Layer
**Purpose:** Efficient binary serialization

**Location:** `btrace-core/src/main/java/io/btrace/core/comm/v2/`

**Components:**
- **BinaryProtocol:** Low-level primitives (readInt, writeString, etc.)
- **BinaryWireIO:** Wire format implementation (version + type + data)
- **BinaryCommand:** Base class for all binary commands
- **17 Command Implementations:** One per command type (Exit, Message, Instrument, etc.)

---

## Protocol Negotiation

### Prepared-mode authentication boundary

When the agent was loaded at JVM startup with its command server enabled, authentication occurs
before the V1/V2 choice described below. The client sends `BTA1`, a four-byte big-endian token
length, and the token bytes. The agent responds with `BTAK` on success or `BTAF` on failure. Only
after `BTAK` may the client send the V1 serialization header or the V2 `BTR2` prefix.

Both sides apply the protocol-negotiation timeout to this preamble. Invalid lengths, truncated or
incorrect tokens, and direct V1/V2 bytes are rejected before any command decoder is constructed.
V2-to-V1 fallback opens a new connection and authenticates it again; authentication failure never
causes an unauthenticated fallback. Dynamically attached agents retain the version-negotiation flow
without this prepared-mode preamble for compatibility with 2.2.x clients.

```text
prepared: TCP connect -> BTA1 + length + token -> BTAK -> V1/V2 negotiation -> commands
dynamic:  TCP connect -> V1/V2 negotiation -> commands
```

### Design Principle: Once Per Connection

**Critical:** Protocol negotiation happens **once** when a connection is established, not per command.

**Timeline:**
```
Time 0ms:     TCP socket established
Time 1ms:     Prepared mode authenticates; dynamic mode skips this step
Time 2ms:     Client sends magic bytes (BTR2) or v1 header
Time 5ms:     Agent responds with protocol acknowledgment
Time 6ms:     Protocol locked for session (v1 or v2)
Time 7ms:     First command sent (using negotiated protocol)
...           [All subsequent commands use same protocol]
Time 60000ms: Connection closed
```

**Why Once Per Connection:**
- **Performance:** Negotiating per command would add massive overhead (~5ms per command)
- **Consistency:** All commands in a session use same wire format
- **Simplicity:** WireProtocol is set once and reused
- **Statefulness:** Negotiated protocol stored in Client/RemoteClient instance

### Handshake Protocol: Magic Byte Prefix

**Approach:** Client sends 4-byte magic prefix at connection start

**v2 Magic Bytes:** `0x42 0x54 0x52 0x32` ("BTR2" in ASCII)

**Flow Diagram:**

```
Client (v2-capable)                    Agent (v2-capable)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0x42 0x54 0x52 0x32] ──────▶│  ◀── Client sends BTR2 magic
        │                                      │
        │                                      ├── Recognizes BTR2
        │                                      ├── Agent supports v2
        │                                      │
        │◀─────── [0x42 0x54 0x52 0x32] ───────┤  ◀── Agent responds with BTR2
        │                                      │
        ├── Protocol = v2 ───────────────────  ├── Protocol = v2
        │                                      │
        ├──────── SetSettingsCommand (v2) ────▶│
        ├──────── InstrumentCommand (v2) ─────▶│
        │◀─────── StatusCommand (v2) ──────────┤
        │◀─────── MessageCommand (v2) ─────────┤
        ...
```

**Fallback to v1:**

```
Client (v2-capable)                    Agent (v1-only)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0x42 0x54 0x52 0x32] ──────▶│  ◀── Client tries v2
        │                                      │
        │         [5 second timeout]           ├── Does not recognize BTR2
        │                                      ├── No response
        │                                      │
        ├── Timeout, fallback to v1 ───────────┤
        │                                      │
        ├──────── [0xAC 0xED ...] ────────────▶│  ◀── Java serialization header
        │                                      │
        │                                      ├── Recognizes Java serialization
        │                                      ├── Protocol = v1
        │                                      │
        ├── Protocol = v1 ─────────────────────┼── Protocol = v1
        │                                      │
        ├──────── SetSettingsCommand (v1) ────▶│
        ...
```

**v1-only Client:**

```
Client (v1-only)                       Agent (v2-capable)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0xAC 0xED ...] ────────────▶│  ◀── Java serialization header
        │                                      │
        │                                      ├── Detects v1 (0xAC 0xED magic)
        │                                      ├── Protocol = v1
        │                                      │
        ├── Protocol = v1 ─────────────────────┼── Protocol = v1
        │                                      │
        ├──────── SetSettingsCommand (v1) ────▶│
        ...
```

### Implementation Details

**Agent Side (RemoteClient.getClient()):**
```java
Socket sock = acceptConnection();
InputStream in = sock.getInputStream();
OutputStream out = sock.getOutputStream();

// Prepared mode only; rejects before a protocol decoder exists
ConnectionAuthenticator.authenticateAgent(in, out, expectedToken);

// Negotiate protocol (reads first bytes)
ProtocolVersion version = ProtocolNegotiator.negotiateAgent(in, out);

// Create appropriate adapter
WireProtocol wire = createWireProtocol(version, in, out);

// Store for session
remoteClient.setWireProtocol(wire);

// All subsequent commands use 'wire'
Command cmd = wire.read(in);
wire.write(out, statusResponse);
```

**Client Side (Client.submit()):**
```java
Socket sock = new Socket(host, port);
InputStream in = sock.getInputStream();
OutputStream out = sock.getOutputStream();

// Prepared mode only; repeats on every fallback connection
ConnectionAuthenticator.authenticateClient(in, out, token);

// Negotiate protocol (sends magic bytes, waits for response)
ProtocolVersion preferred = getPreferredVersion(); // from config
ProtocolVersion version = ProtocolNegotiator.negotiateClient(in, out, preferred);

// Create appropriate adapter
WireProtocol wire = createWireProtocol(version, in, out);

// Store for session
this.wire = wire;

// All subsequent commands use 'wire'
wire.write(out, setSettingsCmd);
wire.write(out, instrumentCmd);
Command status = wire.read(in);
```

### Negotiation Timeout

**Default:** 5 seconds

**Rationale:**
- Long enough for slow networks
- Short enough to fail fast
- Prevents hanging on unresponsive agents

**Configuration:**
```bash
-Dbtrace.protocol.negotiation.timeout=5000
```

### Compatibility Matrix

| Client Version | Agent Version | Negotiated Protocol | Notes |
|---------------|---------------|---------------------|-------|
| v1-only       | v1-only       | v1                  | Legacy |
| v1-only       | v2-capable    | v1                  | Agent detects v1 magic (0xAC 0xED) |
| v2-capable    | v1-only       | v1                  | Client timeout → fallback |
| v2-capable    | v2-capable    | v2                  | Optimal path |

**Key Insight:** Old clients always work with new agents, new clients always work with old agents

---

## Wire Format Specification

### v2 Protocol Format

**Overall Structure:**
```
┌──────────────┬──────────────┬────────────────────────────┐
│ Version (1B) │ Type (1B)    │ Command Data (variable)    │
└──────────────┴──────────────┴────────────────────────────┘
```

**Version Byte:** Current version is `0x03` (bumped from `0x02` after binary format changes to ErrorCommand and GridDataCommand)

**Type Byte:** Command type identifier (0-16)

| Type | Hex  | Command Name                       |
|------|------|------------------------------------|
| 0    | 0x00 | ERROR                              |
| 1    | 0x01 | EVENT                              |
| 2    | 0x02 | EXIT                               |
| 3    | 0x03 | INSTRUMENT                         |
| 4    | 0x04 | MESSAGE                            |
| 5    | 0x05 | RENAME                             |
| 6    | 0x06 | STATUS                             |
| 7    | 0x07 | NUMBER_MAP                         |
| 8    | 0x08 | STRING_MAP                         |
| 9    | 0x09 | NUMBER                             |
| 10   | 0x0A | GRID_DATA                          |
| 11   | 0x0B | RETRANSFORMATION_START             |
| 12   | 0x0C | RETRANSFORM_CLASS                  |
| 13   | 0x0D | SET_PARAMS                         |
| 14   | 0x0E | LIST_PROBES                        |
| 15   | 0x0F | DISCONNECT                         |
| 16   | 0x10 | RECONNECT                          |

### Primitive Type Encoding

**Integers (int):** 4 bytes, big-endian
```
Value: 42
Bytes: [0x00, 0x00, 0x00, 0x2A]
```

**Longs (long):** 8 bytes, big-endian
```
Value: 1234567890
Bytes: [0x00, 0x00, 0x00, 0x00, 0x49, 0x96, 0x02, 0xD2]
```

**Booleans (boolean):** 1 byte
```
true:  [0x01]
false: [0x00]
```

**Strings (String):** Length-prefixed UTF-8
```
Format: [length (4B)] [UTF-8 bytes]

Example: "Hello"
Bytes: [0x00, 0x00, 0x00, 0x05, 0x48, 0x65, 0x6C, 0x6C, 0x6F]
        └─── length=5 ────┘  └────── "Hello" UTF-8 ──────────┘
```

**Null Strings:** Length = -1
```
null: [0xFF, 0xFF, 0xFF, 0xFF]
```

**Byte Arrays (byte[]):** Length-prefixed raw bytes
```
Format: [length (4B)] [raw bytes]

Example: [0xCA, 0xFE, 0xBA, 0xBE]
Bytes: [0x00, 0x00, 0x00, 0x04, 0xCA, 0xFE, 0xBA, 0xBE]
        └─── length=4 ────┘  └──── raw bytes ────┘
```

### Example: MessageCommand

**Structure:**
```
┌─────────┬─────────┬──────────────┬─────────────────┬─────────────────┐
│ Version │ Type    │ Urgent Flag  │ Timestamp (8B)  │ Message (String)│
│ (1B)    │ (1B)    │ (1B)         │                 │                 │
└─────────┴─────────┴──────────────┴─────────────────┴─────────────────┘
  0x02      0x04      0x00/0x01      long              length + UTF-8
```

**Example Bytes:**
```
Message: "BTrace started"
Timestamp: 1638360000000
Urgent: false

Hex dump:
02 04 00 00 00 00 01 7D 28 4F 2D 00 00 00 00 0E
42 54 72 61 63 65 20 73 74 61 72 74 65 64

Breakdown:
02           - Version = 2
04           - Type = MESSAGE (4)
00           - Urgent = false
00 00 00 01 7D 28 4F 2D - Timestamp = 1638360000000
00 00 00 0E  - String length = 14
42 54 72 61 63 65 20 73 74 61 72 74 65 64 - "BTrace started" (UTF-8)
```

### Compression

**Trigger:** Message size > 1024 bytes (configurable)

**Algorithm:** Java Deflater/Inflater (BEST_SPEED)

**Format with Compression:**
```
┌─────────┬─────────┬──────────────┬──────────────────┬────────────────────┐
│ Version │ Type    │ Urgent Flag  │ Compressed Flag  │ Compressed/Raw Data│
│ (1B)    │ (1B)    │ (1B)         │ (1B)             │                    │
└─────────┴─────────┴──────────────┴──────────────────┴────────────────────┘
  0x02      0x04      0x00/0x01      0x00/0x01          byte array
```

**Compressed Data:**
```
[Original Length (4B)] [Compressed Length (4B)] [Deflated Bytes]
```

**Benefits:**
- 3-5x size reduction for large text messages
- Automatically applied for messages >1KB
- Transparent to Command layer

---

## Command Conversion Layer

### Purpose

The **CommandAdapter** provides bidirectional conversion between v1 (`Command`) and v2 (`BinaryCommand`) representations, enabling:

1. v2 wire protocol to work with v1 business logic
2. Gradual migration without rewriting all command handling
3. Testing v2 implementation against v1 baseline

### Architecture

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Conversion Examples

**v1 → v2 (Client sending command):**
```java
// Client has Command object (v1)
MessageCommand v1Cmd = new MessageCommand("Hello from BTrace");

// Convert to BinaryCommand (v2)
BinaryCommand v2Cmd = CommandAdapter.toBinaryCommand(v1Cmd);
// Result: BinaryMessageCommand with message="Hello from BTrace"

// Serialize to wire
BinaryWireIO.write(outputStream, v2Cmd);
// Wire: [0x02][0x04][urgent][timestamp][length][UTF-8 bytes]
```

**v2 → v1 (Agent receiving command):**
```java
// Read from wire
BinaryCommand v2Cmd = BinaryWireIO.read(inputStream);
// Result: BinaryMessageCommand

// Convert to Command (v1)
Command v1Cmd = CommandAdapter.toBtraceCommand(v2Cmd);
// Result: MessageCommand with same data

// Pass to v1 business logic
agent.onCommand(v1Cmd);
```

### Data Fidelity

**Guarantee:** All data is preserved during conversion

**Special Cases:**

1. **ErrorCommand:**
   - v1: Contains full `Throwable` object (type + message + stack trace)
   - v2: Contains exception class name, message, and stack trace as strings
   - **Conversion:** Adapter extracts exception class, message, and stack trace from the `Throwable`; on deserialization, wraps them in a `RemoteException` that preserves the original type and trace

2. **GridDataCommand:**
   - v1: Object[][] (mixed types), optional column names
   - v2: Typed cells (String, Integer, Long, Float, Double, Boolean, HistogramData, null), column names preserved
   - **Conversion:** Type preservation via explicit type codes; `HistogramData` has a dedicated encoding

3. **NumberMapDataCommand:**
   - v1: `Map<String, Number>` (can carry any `Number` subclass)
   - v2: Typed encoding for int/long/float/double plus dedicated codes for `BigInteger` and `BigDecimal`
   - **Conversion:** Preserves precision for all standard `Number` types

4. **StatusCommand:**
   - v1: Single int (positive=success, negative=failure)
   - v2: flag (int) + success (boolean)
   - **Conversion:** `flag = abs(v1), success = (v1 > 0)`

### WireProtocol Adapters

**WireIOV1Adapter:**
```java
public class WireIOV1Adapter implements WireProtocol {
    private ObjectInputStream ois;
    private ObjectOutputStream oos;

    public Command read(InputStream in) throws IOException {
        return WireIO.read(ois);  // Uses v1 protocol
    }

    public void write(OutputStream out, Command cmd) throws IOException {
        WireIO.write(oos, cmd);  // Uses v1 protocol
    }

    public void reset() throws IOException {
        oos.reset();  // ObjectOutputStream state management
    }
}
```

**WireIOV2Adapter:**
```java
public class WireIOV2Adapter implements WireProtocol {
    private InputStream in;
    private OutputStream out;

    public Command read(InputStream in) throws IOException {
        BinaryCommand binaryCmd = BinaryWireIO.read(in);
        return CommandAdapter.toBtraceCommand(binaryCmd);  // Convert v2→v1
    }

    public void write(OutputStream out, Command cmd) throws IOException {
        BinaryCommand binaryCmd = CommandAdapter.toBinaryCommand(cmd);  // Convert v1→v2
        BinaryWireIO.write(out, binaryCmd);
    }

    public void reset() throws IOException {
        // No-op: v2 has no state to reset
    }
}
```

---

## Benefits and Trade-offs

### Benefits

#### 1. Performance: 3-6x Faster
**Measurement:** 10,000 iterations, InstrumentCommand (100KB bytecode)

| Metric | v1 (Java Serialization) | v2 (Binary) | Improvement |
|--------|------------------------|-------------|-------------|
| Serialize | 450ms | 90ms | **5x faster** |
| Deserialize | 520ms | 110ms | **4.7x faster** |
| Round-trip | 970ms | 200ms | **4.85x faster** |

**Why:**
- No reflection overhead
- Minimal object allocation
- Direct byte manipulation
- Optimized for BTrace command patterns

#### 2. Size: 2-5x Smaller
**Measurement:** Wire size comparison

| Command Type | v1 Size | v2 Size | Reduction |
|-------------|---------|---------|-----------|
| ExitCommand | 45 bytes | 15 bytes | **3x smaller** |
| MessageCommand (small) | 180 bytes | 60 bytes | **3x smaller** |
| MessageCommand (large, 10KB) | 10,240 bytes | 2,150 bytes | **4.8x smaller** (compressed) |
| InstrumentCommand (100KB) | 102,400 bytes | 34,100 bytes | **3x smaller** (compressed) |

**Why:**
- No Java serialization metadata
- Efficient primitive encoding
- Automatic compression for large payloads
- Minimal framing overhead

#### 3. Thread Safety: ReentrantLock
**v1:** `synchronized (ObjectOutputStream)`
**v2:** `ReentrantLock` in BinaryWireIO

**Benefits:**
- Better scalability under contention
- Fairness guarantees (optional)
- Interruptible locking
- Try-lock with timeout

#### 4. Language Independence
**v1:** Requires Java client (ObjectInputStream/ObjectOutputStream)
**v2:** Simple binary format, can be implemented in any language

**Future possibilities:**
- Python BTrace client
- Go monitoring tools
- JavaScript browser-based debugger
- VS Code extension in TypeScript

#### 5. Backward Compatibility
**Zero breaking changes:** Old clients work with new agents, new clients work with old agents

**Migration path:** Automatic, no user action required

### Trade-offs

#### 1. Code Complexity
**Added:** Protocol negotiation, WireProtocol abstraction, CommandAdapter
**Mitigated by:** Clean interfaces, comprehensive tests

#### 2. Negotiation Latency
**Cost:** ~5-10ms per connection establishment
**Amortized over:** Entire session (thousands of commands)
**Net impact:** Negligible

#### 3. Compression CPU Overhead
**Cost:** Deflate/Inflate CPU usage for large messages
**Threshold:** Only for messages >1KB
**Net benefit:** Reduced network I/O usually more expensive than compression

#### 4. Testing Burden
**Requirement:** Test v1, v2, and mixed scenarios
**Mitigated by:** Automated test matrix, reusable test harness

### When to Use v2

**Recommended for:**
- High-frequency tracing (>100 commands/second)
- Large instrumentation payloads
- Remote tracing over slow networks
- Production environments with multiple agents

**v1 sufficient for:**
- Interactive debugging (low frequency)
- Local tracing (no network)
- Legacy environments (no upgrade path)

---

## Migration Path

### For End Users (Transparent)

**No action required:** Protocol negotiation is automatic

**Optional configuration** (see `ProtocolConfig` in `btrace-core/src/main/java/io/btrace/core/comm/ProtocolConfig.java`):
```bash
# Preferred protocol version (accepts 1, 2, v1, v2). Default: v2
-Dbtrace.comm.protocol=v2

# Enable/disable automatic protocol negotiation. Default: true
-Dbtrace.comm.autoNegotiate=true

# Force the configured version without negotiation. Default: false
# (cannot be combined with autoNegotiate=true)
-Dbtrace.comm.forceVersion=true
```

The default behavior (v2 preferred, auto-negotiation enabled) requires no configuration.

### For Developers

#### Completed
- All 17 command types implemented and tested
- Protocol negotiation implemented
- RemoteClient and Client refactored with WireProtocol abstraction
- Backward compatibility verified (v1 clients work with v2 agents and vice versa)
- Default to v2 with automatic fallback to v1

#### Post-Release Technical Debt
- Add v2-only end-to-end integration test suite
- Stress tests under sustained high-frequency tracing

### Rollback Plan

**If issues arise:**
1. Fall back to v1: `-Dbtrace.comm.protocol=1 -Dbtrace.comm.autoNegotiate=false -Dbtrace.comm.forceVersion=true`
2. Roll back agent/client to previous version
3. Fix issues, re-test
4. Re-enable v2

**Safety:** v1 protocol remains fully functional, no risk of data loss

---

## Performance Characteristics

### Throughput

**Scenario:** Single client, continuous command stream

| Command Type | v1 (cmds/sec) | v2 (cmds/sec) | Improvement |
|--------------|---------------|---------------|-------------|
| ExitCommand | 120,000 | 550,000 | **4.6x** |
| MessageCommand (small) | 45,000 | 180,000 | **4x** |
| MessageCommand (large) | 2,500 | 12,000 | **4.8x** |
| InstrumentCommand | 800 | 3,500 | **4.4x** |
| GridDataCommand | 8,000 | 32,000 | **4x** |

**Bottleneck (v1):** Java serialization overhead
**Bottleneck (v2):** Network I/O (achieved wire-speed)

### Latency

**Scenario:** Round-trip time (client send → agent receive → process → respond → client receive)

| Command Type | v1 p50 | v1 p99 | v2 p50 | v2 p99 | Improvement |
|--------------|--------|--------|--------|--------|-------------|
| ExitCommand | 1.2ms | 3.5ms | 0.3ms | 0.8ms | **4x faster** |
| MessageCommand | 2.8ms | 8.1ms | 0.7ms | 2.1ms | **4x faster** |
| InstrumentCommand | 45ms | 120ms | 12ms | 35ms | **3.75x faster** |

**Key insight:** v2 reduces tail latency significantly (p99)

### Memory

**Scenario:** Memory allocations per command

| Command Type | v1 Allocations | v2 Allocations | Reduction |
|--------------|----------------|----------------|-----------|
| ExitCommand | 850 bytes | 120 bytes | **7x less** |
| MessageCommand | 2.1 KB | 450 bytes | **4.7x less** |
| InstrumentCommand | 125 KB | 102 KB | **1.2x less** (bytecode dominates) |

**GC impact:** Fewer allocations → less GC pressure → smoother performance

### Network Bandwidth

**Scenario:** 10,000 MessageCommands (average 500 bytes text)

| Protocol | Wire Size | Network Usage |
|----------|-----------|---------------|
| v1 | 8.2 MB | 100% baseline |
| v2 (no compression) | 5.1 MB | 62% |
| v2 (with compression) | 1.9 MB | 23% |

**Benefit:** 77% bandwidth reduction with compression

---

## Conclusion

The BTrace v2 binary protocol delivers significant performance improvements (3-6x faster, 2-5x smaller) while maintaining full backward compatibility through automatic protocol negotiation. The architecture is clean, well-tested, and production-ready.

**Key Takeaways:**
- Protocol negotiation happens once per connection (not per command)
- Automatic fallback ensures zero breaking changes
- Performance gains are substantial and validated by benchmarks
- Migration is transparent to end users

**Implementation status:**
- Protocol version bumped to 3 after binary format changes
- All 17 command types covered by unit tests (26+ tests)
- ErrorCommand preserves exception class, message, and stack trace via RemoteException
- GridDataCommand preserves HistogramData and column names
- NumberMapDataCommand preserves BigInteger and BigDecimal

---

## References

- Implementation: `btrace-core/src/main/java/io/btrace/core/comm/v2/`
- Tests: `btrace-core/src/test/java/io/btrace/core/comm/v2/`
- README: `btrace-core/src/main/java/io/btrace/core/comm/v2/README.md`

---

### Examples/README

# Example Extensions (Provided-Style)

These examples demonstrate how to build provided-style extensions that avoid mutating the JVM classpath and instead use object hand-off + context class loaders for runtime linking.

- Spark example: `btrace-extensions/examples/btrace-spark`
- Hadoop example: `btrace-extensions/examples/btrace-hadoop`

Build (from repo root):

```
./gradlew :btrace-extensions:examples:btrace-spark:build
./gradlew :btrace-extensions:examples:btrace-hadoop:build
```

Both example modules use the default extension layout (`src/main`) while still producing the standard BTrace extension artifacts (`*-api.jar`, `*-impl.jar`, `*-extension.zip`).

Enable in `extensions.conf` (examples):

```
# Spark example
btrace-spark-example.enabled=true
# Optional role detection (example-specific key)
btrace-spark-example.role=auto           # auto|driver|executor
# Optional external classpath hint (prefer not needed)
# btrace-spark-example.classpath=/opt/spark/jars

# Hadoop example
btrace-hadoop-example.enabled=true
# Optional external classpath hint (prefer not needed)
# btrace-hadoop-example.classpath=/opt/hadoop/share/hadoop/common
```

Permissions (examples use reflection):

```
grant=REFLECTION,SYSTEM_PROPS
```

Notes
- These are templates only; they are not maintained as core modules.
- Prefer packaging application libraries with the application; avoid external classpath hints when possible.
- For rare, short-term needs, the agent supports a single-jar escape hatch (discouraged):
  - `-Dbtrace.system.appendJar=/abs/path/lib.jar -Dbtrace.trusted=true`

---

### Superpowers/Specs/2026 07 29 Issue 932 Client Close Deadlock Design

# Issue 932 — make an integration-test stall self-diagnosing

Design document.

## Symptom

`PreparedModeAuthenticationFunctionalTest` intermittently stalls on its V2/V2 parameterisations,
consumes the full 30-minute CI budget, reports no assertion failure and no stack trace, and leaves
several orphaned `java` processes behind. Last line before the silence is
`Successfully started BTrace probe`, with no `PASSED` line for the parameterisation.

## Why this design is not a root-cause fix

An earlier revision argued the stall was `Client.close()` deadlocking against its own parked reader,
fixable with `sock.shutdownInput()`. **That was refuted by measurement** and is recorded here so it
is not proposed a third time.

`NioSocketImpl.close()` does not wait for an in-flight reader. It pre-closes the descriptor and
*signals* the blocked thread; the deferred close completes on the reader's own thread. Measured on
macOS across JDK 11/17/24/26, descriptor in both blocking and non-blocking mode:

| action, with a reader parked in an unbounded `read()` | duration | reader observes |
|---|---|---|
| `socket.close()` | 0–1 ms | `SocketException: Socket closed` |
| `socket.getOutputStream().close()` | 0 ms | `SocketException: Socket closed` |
| `socket.shutdownInput()` | 0 ms | `read()` returns `-1` |

`Client.close()` is bounded and cannot be a 25-minute stall.

The supporting evidence was also misread. `log.info("Successfully started BTrace probe")` is emitted
at `Client.java:1284`, inside `submit`'s wrapping listener, *before* `listener.onCommand(cmd)` at
`Client.java:1292` reaches the test's `handleStatus`. It proves only that the worker reached line
1284 — not that `started.get(20, SECONDS)` returned. Every elimination argument rested on that
misreading.

What the evidence does support: `started.get(20, SECONDS)` may have thrown `TimeoutException`, and
that exception was then *lost*, because the enclosing `finally` runs `TestApp.stop()` →
`process.waitFor()` with no bound (`RuntimeTest.java:1006`). A pending exception whose `finally`
never returns is never delivered to JUnit. That reproduces the observed shape exactly — no assertion
failure, no `PASSED` line, orphaned JVMs — and means the 25 minutes were spent waiting on a target
JVM that never exited.

The remaining product-side defect is whatever wedged the **target** JVM. Nothing in the available
evidence identifies it; two mechanisms have now been proposed and refuted, both by reasoning
backwards from a log shape that several mechanisms explain equally well. The next attempt starts
from a thread dump, which is what this design delivers.

## What PR #934 does and does not provide

Verified against `git show 44f2c25d`: the branch bounds `TestApp.stop()`, makes the submit executors
daemon threads, and adds a speculative `shutdownInput()` in `Client.close()`. Its commit message also
describes a five-minute per-test timeout in `SEPARATE_THREAD` mode — **that annotation is not in the
diff**, and the branch carries no `junit-platform.properties`. This design therefore assumes no
per-test timeout exists, and must not rely on one to bound the capture.

## Goal

> When an integration test stalls, the CI artifacts and (where available) the job log name the
> blocked thread in the test JVM and in every live target JVM, without anyone re-running anything.

## Design

A **stall watchdog** in the integration-test harness. Evidence first, and then — only once the
evidence is on disk — it releases the stall.

The evidence-only design this document originally specified turned out not to reach the goal.
Adversarial review of the implementation established, and `javap` on `junit-jupiter-api-5.14.4`
confirmed, that `AssertTimeoutPreemptively$TimeoutThreadFactory` builds its thread with
`new Thread(Runnable, String)` and never calls `setDaemon(true)`. So when the per-test timeout below
fires, JUnit marks the test failed, interrupts the test thread — an interrupt a thread blocked in
socket I/O ignores, which is why `SEPARATE_THREAD` was chosen in the first place — and abandons it.
That surviving non-daemon thread keeps the Gradle worker JVM alive, and the job burns its whole
budget regardless. Killing the stalled test's registered targets closes the sockets the thread is
blocked on, which is what actually ends the stall; it also clears the orphaned JVMs the issue
reports. This happens only after both frames are captured, and only to targets registered by the
stalled test.

**1. `StallWatchdog`, a JUnit 5 extension.** Registered with `@ExtendWith` on the abstract
`RuntimeTest`; JUnit searches the superclass hierarchy, so all eight subclasses inherit it with no
boilerplate. `BeforeAllCallback` arms first (so a stall in `classSetup()` is covered), `beforeEach`
re-arms per invocation, `afterEach`/`afterAll` disarm. Disarm uses `ScheduledFuture.cancel(false)`
plus an `AtomicBoolean` so an in-flight capture is never attributed to the next test and `afterEach`
never blocks waiting for one.

**2. The deadline is derived, not hard-coded.** `RuntimeTest` already allows a target up to
`startupTimeoutMs = timeout * 4` (`RuntimeTest.java:784`), i.e. 240 s at the default
`btrace.test.timeoutMs = 60000`. A four-minute watchdog would fire on legitimately slow green tests.
The deadline is `btrace.test.timeoutMs * 6` (six minutes by default), overridable with
`-Pbtrace.test.stallTimeoutMs`. **`-P`, not `-D`**: `integration-tests/build.gradle:274` forwards
only Gradle project properties beginning with `btrace.` into the test JVM.

**3. Two captures, thirty seconds apart, then stop.** A single frame cannot distinguish "wedged"
from "slow" — a thread parked in `SocketInputStream.read` looks identical either way. The diff
between two frames is the evidence. Capturing twice and stopping bounds the log cost on a job that
is already lost.

**3b. The dump is rendered frame by frame,** not with `ThreadInfo.toString()`, which stops after
eight frames — and the frame that explains a stall is rarely in the top eight of a test-harness
stack. Measured: a 44-frame stack renders as 8 through `toString()`.

**4. Each capture contains:** a banner with the test's unique id and elapsed time; the test JVM's
threads via `ThreadMXBean.dumpAllThreads(true, true)` (lock-owner and monitor information, which
`Thread.getAllStackTraces()` omits and a deadlock diagnosis needs); and for every live registered
target, `jcmd <pid> Thread.print -l`.

**5. File first, stdout second.** The file is the reliable sink: `if: always()` upload steps do run
after a `timeout-minutes` cancellation, whereas a cancelled job tears down the worker→daemon→console
pipeline, and the release path redirects all test output into a file it only summarises *after* the
command exits (`scripts/run-release-gate.sh:52`). Dumps are written to
`integration-tests/build/reports/stall-dumps/`, which the existing `if: always()` glob
`integration-tests/build/reports/**/*` (`continuous.yml:171`) already uploads — no workflow change.
The stdout copy is best-effort and written second, because `System.out` is a shared synchronised
`PrintStream` that a stalled pipeline can itself block on. Every line carries a `[stall-dump] `
prefix so one `grep` recovers a capture interleaved with the `[traced app]` reader threads.
Filenames derive from the sanitised JUnit unique id, not `<class>.<method>` — the motivating test is
a `@ParameterizedTest` whose four rows share one method name.

**6. `jcmd` is treated as untrustworthy.** Its output goes to a file via `redirectOutput`, never to a
pipe: `Thread.print -l` on an instrumented target exceeds the 64 KB pipe buffer, and a blocked jcmd
would then be killed by the bound having produced nothing — losing precisely the dump we want. The
call is bounded and `destroyForcibly`-ed on overrun. If it fails or times out — the jammed
Attach Listener case, which is plausible here — the watchdog falls back to `kill -QUIT`, which the
VM's Signal Dispatcher handles independently of the attach mechanism. That dump goes to the target's
stdout, which `RuntimeTest` echoes only when `debug` is set (`RuntimeTest.java:940`), so requesting a
capture also flips a flag that makes the reader threads echo unconditionally.

**7. Target registry.** A static registry holds the `Process` plus a lazily-resolved pid, registered
at launch rather than at `ready:`, so a target that dies before reporting a pid is still known.
Entries with `!process.isAlive()` are skipped at capture time, which removes the need for
deregistration hooks and covers the leak at `BTraceFunctionalTests.java:536`, where a `TestApp` is
launched and never stopped. Four launch sites register: `TestApp`, the three raw `ProcessBuilder`
paths in `RuntimeTest` (`:348`, `:540`, `:745`), and
`Issue888RuntimeHardeningIntegrationTest.java:184`. The whole capture shares one overall time budget
rather than 10 s per target, so accumulated stale entries cannot stretch it.

**Java 8 source level.** `integration-tests` compiles tests with `sourceCompatibility = 8`, but
without `options.release`, so javac 24 happily compiles JDK 9+ APIs that then fail on a JDK 8
worker. No `var`, no `List.of`, no `ProcessHandle`/`Process.pid()`. Adding
`compileTestJava { options.release = 8 }` to enforce this is attempted; if existing test code
already violates it the change is dropped and the violation reported rather than fixed here.

## Testing

The extension's own tests drive it through an in-memory seam rather than scraping stdout, and take
their deadline from a `@StallTimeout` annotation read from the `ExtensionContext` rather than a
system property — with no `forkEvery` configured anywhere in the build, one worker JVM runs all ten
test classes, so a system property set by a test leaks into every class that follows it.

1. **Captures the test JVM.** A parked thread with a recognisable name; assert the capture contains
   the banner, the unique id, and that thread's stack.
2. **Captures a target JVM.** Launch a target, assert the capture contains a `Thread.print` section
   for its pid and a thread name only the target has.
3. **Degrades rather than hangs.** With a bogus registered pid, the capture completes within its
   budget, reports the failure inline, and still contains the test-JVM dump.
4. **Stays silent when green.** A fast test writes no dump; the assertion targets a `@TempDir`-rooted
   output directory so a file left by an earlier Gradle invocation cannot make it pass or fail
   spuriously.

## Out of scope

- Product code. Nothing in `btrace-client`, `btrace-core`, `btrace-agent` changes.
- `JBangAttachDockerTest` and `Issue884PublishedFatAgentE2ETest` do not extend `RuntimeTest` and are
  not covered. They must not be covered by registering the extension globally: their targets run
  inside containers, and a container pid handed to a host `jcmd` would attach to an unrelated host
  process.
- Bounding `TestApp.stop()`, daemon submit executors, and the missing per-test timeout — PR #934's
  territory, whatever that PR ends up containing.
- Killing or failing a stalled test.

---

### Superpowers/Plans/2026 07 12 Checklines Migration

# Migrate Remaining `checkLines` Call Sites to `Completion` Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Eliminate the remaining `int checkLines` line-count waits in `integration-tests` by migrating every call site to content-based `Completion` (`untilContains`/`untilMatches`), matching the pattern already used by `ExternalTypeAdapterIntegrationTest`.

**Architecture:** `docs/superpowers/plans/2026-07-12-test-harness-completion-conditions.md` (already merged) built the `Completion`/`OutputPump` abstraction and migrated exactly one caller. It explicitly kept the `int checkLines` overloads for backward compatibility and scoped full migration out as an optional Task 7. This plan finishes that migration: every remaining call site gets a `Completion` built from the same content markers its `ResultValidator` already asserts on, the now-dead `int` overloads and `runBTrace(String[], int, ...)` are deleted, and `testStartup`'s separate hand-rolled `CountDownLatch(checkLines)` mechanism (used by `testTraceAll`, `testThreadStart(false)`, `launchAgent_manifestLibs`) is rewired onto `OutputPump`/`Completion` the same way `attach()` was in the original plan's Task 3.

**Tech Stack:** Java, JUnit 5, `tests.harness.Completion`, `tests.harness.OutputPump` (both already implemented in `integration-tests/src/test/java/tests/harness/`).

## Global Constraints

- Every migrated call site's `Completion` must be built from markers **already asserted on** by that test's `ResultValidator` — do not invent new markers. Where the validator has no content assertion (`testOSMBean`, `testJfr`) or asserts on something the `Completion` mechanism cannot see (`testExtensionCloseCalledOnError`'s exit-code check, JFR-file checks), use the bespoke condition specified in this plan's Task 6, not a guess.
- `Completion.untilContains(...)` waits for markers to appear "in any order" across the accumulated stdout — safe for markers that are genuinely independent, but do NOT use it where a test's negative assertion depends on stopping *before* an unrelated marker could appear (see Task 5's note on `testProbeArgs`).
- Do not touch `ExternalTypeAdapterIntegrationTest.java` — already migrated, serves as the reference pattern.
- Do not touch `JBangAttachDockerTest.java` — does not use `RuntimeTest`/`Completion` at all.
- After each task, run the affected test class only (not the full suite) via `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.<ClassName>'` and confirm the class passes with **zero** `[harness] timed out` lines in the output for tests that pass. A `[harness] timed out` line followed by a PASS is itself a bug this plan exists to remove — treat it as a task failure, not a pre-existing quirk to tolerate.
- Run `./gradlew spotlessApply` and commit formatting changes before any push (standing repo rule).

---

### Task 1: Delete the dead `runBTrace(String[], int, ...)` overload

**Files:**
- Modify: `integration-tests/src/test/java/tests/RuntimeTest.java` (around line 1167, the `int checkLines` overload of `runBTrace`)

**Interfaces:**
- Consumes: nothing new.
- Produces: nothing — pure deletion. The `Completion`-based `runBTrace` overload (around line 1170) remains untouched.

- [ ] **Step 1: Confirm there are truly zero callers**

Run: `grep -rn "runBTrace(" integration-tests/src/test/java | grep -v "RuntimeTest.java"`
Expected: no output (already confirmed during inventory, re-verify before deleting).

- [ ] **Step 2: Delete the `int checkLines` overload**

Remove the `runBTrace(String[] args, int checkLines, StringBuilder stdout, StringBuilder stderr)` method body entirely (it should be a thin wrapper delegating to the `Completion`-based overload — delete the whole method, not just its body).

- [ ] **Step 3: Compile**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava`
Expected: BUILD SUCCESSFUL (proves no hidden caller existed).

- [ ] **Step 4: Commit**

```bash
git add integration-tests/src/test/java/tests/RuntimeTest.java
git commit -m "test(harness): delete dead runBTrace(String[], int, ...) overload"
```

---

### Task 2: Migrate `ClassFileApiTests.java`

**Files:**
- Modify: `integration-tests/src/test/java/tests/ClassFileApiTests.java` (4 call sites: `testEntry` ~L58, `testReturnValue` ~L75, `testDuration` ~L92, `testFeatureSmoke` ~L112)

**Interfaces:**
- Consumes: `RuntimeTest.testDynamic(String testApp, String testScript, Completion completion, ResultValidator v)` (already exists, L257-261), `Completion.untilContains(String...)`.
- Produces: nothing new for later tasks — this file has no other dependents.

- [ ] **Step 1: Add the `Completion` import**

Add `import tests.harness.Completion;` to `ClassFileApiTests.java`'s import block if not already present (check first — it may already be imported for another reason; if absent, add it).

- [ ] **Step 2: Replace the 3 single-marker call sites**

| Test | Old 3rd arg | New 3rd arg |
|---|---|---|
| `testEntry` | `5` | `Completion.untilContains("Math.abs entered: abs")` |
| `testReturnValue` | `5` | `Completion.untilContains("Math.abs returned: ")` |
| `testDuration` | `5` | `Completion.untilContains("Math.max duration: ")` |

Each is a `testDynamic(testApp, testScript, <arg>, validator)` call — replace only the third argument in place, keep everything else identical.

- [ ] **Step 3: Replace `testFeatureSmoke`'s `checkLines=100` with all 11 markers**

Read the test method body to get the exact 11 marker strings the `assertContainsAll` call already checks (they are literal substrings passed to that helper). Replace the `100` argument with:

```java
Completion.untilContains(
    "cfapi <marker-1>",
    "cfapi <marker-2>",
    // ... all 11 markers, copied verbatim from the assertContainsAll(...) call below in the
    // same method, in the same order they appear there
    "cfapi <marker-11>")
```

Do not paraphrase the markers — copy the exact strings used in the existing `assertContainsAll(...)` call so the wait condition and the assertion can never drift apart.

- [ ] **Step 4: Run the migrated class**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ClassFileApiTests'`
Expected: all 4 tests PASS (note: `testFeatureSmoke` only runs on JDK26+; if the local/CI JDK is older it will be skipped, which is fine — do not force it).

- [ ] **Step 5: Commit**

```bash
git add integration-tests/src/test/java/tests/ClassFileApiTests.java
git commit -m "test(harness): migrate ClassFileApiTests off checkLines to Completion.untilContains"
```

---

### Task 3: Migrate `ExtensionLifecycleIntegrationTest.java`

**Files:**
- Modify: `integration-tests/src/test/java/tests/ExtensionLifecycleIntegrationTest.java` (3 call sites: `testExtensionMethodCalled` ~L57, `testExtensionCloseCalledOnError` ~L78, `testMultipleExtensionsAllClosed` ~L101)

**Interfaces:**
- Consumes: same `testDynamic(..., Completion, ResultValidator)` overload as Task 2.
- Produces: nothing new for later tasks.

- [ ] **Step 1: Migrate `testExtensionMethodCalled`**

Replace its `checkLines=10` argument with:
```java
Completion.untilContains("LIFECYCLE: extension method called")
```

- [ ] **Step 2: Migrate `testMultipleExtensionsAllClosed`**

Replace its `checkLines=10` argument with:
```java
Completion.untilContains(
    "LIFECYCLE: printer extension called", "LIFECYCLE: metrics extension called")
```
(This is a textbook multi-marker case — mirrors `ExternalTypeAdapterIntegrationTest`'s already-migrated pattern exactly.)

- [ ] **Step 3: Migrate `testExtensionCloseCalledOnError` — bespoke condition, do NOT use plain `untilContains` alone**

This test asserts `retcode == 1` in addition to the two markers `"LIFECYCLE: extension method called"` and `"Triggering error exit"`. A `Completion` cannot observe the process exit code (it only sees stdout/stderr lines), so `untilContains` on the two markers is still the right wait condition — the exit-code assertion happens after `testDynamic` returns, unaffected by which `Completion` was used. Replace the `checkLines=10` argument with:
```java
Completion.untilContains("LIFECYCLE: extension method called", "Triggering error exit")
```
This is safe specifically because both markers are printed by the probe itself before the error-triggered process exit, so waiting for both before checking `retcode` is correct and no longer racy on unrelated framework output.

- [ ] **Step 4: Add the `Completion` import if missing, run the migrated class**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ExtensionLifecycleIntegrationTest'`
Expected: all 3 tests PASS, no `[harness] timed out` lines.

- [ ] **Step 5: Commit**

```bash
git add integration-tests/src/test/java/tests/ExtensionLifecycleIntegrationTest.java
git commit -m "test(harness): migrate ExtensionLifecycleIntegrationTest off checkLines to Completion.untilContains"
```

---

### Task 4: Migrate `ManifestLibsTests.java`'s `dynamicAttach_manifestLibs`

**Files:**
- Modify: `integration-tests/src/test/java/tests/ManifestLibsTests.java` (~L57)

**Interfaces:**
- Consumes: same `testDynamic(..., Completion, ResultValidator)` overload.

- [ ] **Step 1: Migrate the call site**

`dynamicAttach_manifestLibs` reuses `btrace/OnTimerArgTest.java` and asserts `contains("timer")`. Replace its `checkLines=10` argument with:
```java
Completion.untilContains("timer")
```

- [ ] **Step 2: Run and commit**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ManifestLibsTests.dynamicAttach_manifestLibs'`
Expected: PASS.

```bash
git add integration-tests/src/test/java/tests/ManifestLibsTests.java
git commit -m "test(harness): migrate ManifestLibsTests off checkLines to Completion.untilContains"
```

---

### Task 5: Migrate `BTraceFunctionalTests.java` — straightforward marker-based tests

**Files:**
- Modify: `integration-tests/src/test/java/tests/BTraceFunctionalTests.java`

**Interfaces:**
- Consumes: same `testDynamic`/`testDynamicOneliner` `Completion` overloads.

This task covers every call site in this file **except** `testOSMBean`, `testJfr`, `testOnMethodTrackRetransform`, and `testOnelinerCompilationError` — those four have no clean content-marker mapping and are handled separately in Task 6.

- [ ] **Step 1: Replace the single/multi-marker call sites per this table**

| Test | Old `checkLines` | New `Completion` expression |
|---|---|---|
| `testOnProbe` | `5` | `Completion.untilContains("[this, noargs]", "[this, args]")` |
| `testOnTimer` | `10` | `Completion.untilContains("vm version", "vm starttime", "timer")` |
| `testOnTimerArg` | `10` | `Completion.untilContains("vm version", "vm starttime", "timer")` |
| `testOnExit` | `5` | `Completion.untilContains("onexit")` |
| `testOnMethod` | `14` | `Completion.untilContains("[this, noargs]", "[this, args]", "{xxx}", "heap:init", "prop: test", "fieldSet:", "fieldGet:")` — copy the exact marker substrings from this test's existing `assertTrue(stdout.contains(...))` calls; do not paraphrase |
| `testOnelinerRuntime` | `30` | `Completion.untilContains("callB", "Hello World")` |
| `testExtensionLifecycleClose` | `10` | `Completion.untilContains("extension close: btrace-utils")` — this marker is framework/teardown output, not probe output, but it is exactly what the validator asserts on, so it is still the correct (and only) wait target |
| `testOnMethodLevel` | `5` | `Completion.untilContains("[this, noargs]", "[this, args]", "{xxx}")` |
| `testOnMethodReturn` | `5` | `Completion.untilContains("[this, anytype(void)]", "[this, void]", "[this, 2]")` |
| `testOnMethodSubclass` | `5` | `Completion.untilContains("print:class resources.Main")` — verify the exact marker text against the existing assertion in this test before typing it in; the inventory noted the marker text is approximate, so read the assertion directly rather than trusting this plan's paraphrase |
| `testProbeArgs` | `5` | `Completion.untilContains("arg#=", "arg1=", "arg2=val2")` — keep the existing negative assertion (`assertFalse(contains("matching probe"))`) unchanged; `untilContains` only affects when the harness stops *waiting*, not what it asserts afterward, so the negative check still runs against whatever accumulated by the time these 3 markers appeared |
| `testPerfCounter` | `5` | `Completion.untilContains("matching probe")` |
| `testReflection` | `5` | copy the exact 2 marker strings from this test's 2 `assertTrue(contains(...))` calls into `Completion.untilContains(marker1, marker2)` |
| `testThreadStart(dynamic=true)` | `10` | `Completion.untilContains("starting testThread")` |
| `testMetrics` | `20` | copy all distinct marker strings this test's 7 `assertTrue(contains(...))` calls check (header, report banner, Count/Mean/P50/P95/P99 labels) into a single `Completion.untilContains(...)` call |
| `testOnelinerMethodEntry` | `10` | `Completion.untilContains("callA")` |
| `testOnelinerWithArguments` | `10` | `Completion.untilContains("[1, Hello World]")` |
| `testOnelinerWithReturn` | `10` | `Completion.untilContains("callB")` |
| `testOnelinerWithRegexClassMatch` | `10` | `Completion.untilContains("callA")` |
| `testOnelinerStack` | `10` | `Completion.untilContains("resources.Main.callA")` — verify exact marker against this test's actual assertion (`contains("resources.Main.callA")` or `contains("resources.Main")`); use whichever the assertion literally checks |
| `flatDslOpsWork` | `5` | `Completion.untilContains("flat-dsl:")` |

For each row: find the call site (`testDynamic(...)` or `testDynamicOneliner(...)`), replace only the `checkLines` int argument with the `Completion` expression, and switch the call to the `Completion`-taking overload (same method name, different overload — no other change needed). Leave the `ResultValidator` bodies untouched.

- [ ] **Step 2: Add the `Completion` import if missing**

- [ ] **Step 3: Run the migrated tests**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests'`
Expected: every test migrated in this task PASSES. (`testOSMBean`, `testJfr`, `testOnMethodTrackRetransform`, `testOnelinerCompilationError` are still on `checkLines` at this point — Task 6 handles those — so don't be alarmed if they still show `[harness] timed out` noise here.)

- [ ] **Step 4: Commit**

```bash
git add integration-tests/src/test/java/tests/BTraceFunctionalTests.java
git commit -m "test(harness): migrate most of BTraceFunctionalTests off checkLines to Completion.untilContains"
```

---

### Task 6: Migrate `BTraceFunctionalTests.java` — tests with no clean content marker

**Files:**
- Modify: `integration-tests/src/test/java/tests/BTraceFunctionalTests.java` (`testOSMBean`, `testJfr`, `testOnMethodTrackRetransform`, `testOnelinerCompilationError`)

**Interfaces:**
- Consumes: `Completion.untilContains`, `Completion.lines` (kept deliberately for one case below — see Step 1), `Completion.untilMatches`.

These four don't fit the "wait for what you assert on" pattern cleanly. Handle each on its own terms rather than forcing `untilContains`:

- [ ] **Step 1: `testOSMBean` — validator has no content assertion at all**

The validator only checks `assertFalse(stdout.contains("FAILED"))` and `assertTrue(stderr.isEmpty())` — there is no positive marker to wait for, because the script (`btrace/OSMBeanTest.java`) prints exactly one line on success and two on failure, and the test doesn't know in advance which it'll get. Use:
```java
Completion.untilMatches(Pattern.compile(".+"), 1)
```
This waits for exactly one non-empty output line (success or failure path both produce at least one), replacing the old `checkLines=2` which was tuned to the failure path only. Add `import java.util.regex.Pattern;` if not already present.

- [ ] **Step 2: `testJfr` — assertion is against a JFR binary file, not stdout**

The validator's real check is `assertNotNull(jfrFile)` plus JFR event-type/value inspection via `RecordingFile` — stdout content is irrelevant to pass/fail. The probe (`btrace/JfrTest.java`) prints `"Main.callA"` exactly once. Use:
```java
Completion.untilContains("Main.callA")
```
This is the one line the probe reliably produces, giving the harness a real signal that the probe fired (and therefore likely populated the JFR recording) instead of waiting for 30 arbitrary lines of framework noise. Do not attempt to make the `Completion` aware of the JFR file itself — that is out of scope for this interface.

- [ ] **Step 3: `testOnMethodTrackRetransform` — marker is a framework/debug log line**

The validator asserts `contains("Going to retransform class")`, which the BTrace framework itself logs (not the probe). This is still valid `Completion` usage — the interface waits on any stdout content, framework or probe — so:
```java
Completion.untilContains("Going to retransform class")
```
Replace the old `checkLines=2`.

- [ ] **Step 4: `testOnelinerCompilationError` — script is expected to fail compilation, not produce output**

The oneliner is intentionally invalid (`resources.Main::callB @invalid { print }`), and the validator checks `!stderr.isEmpty() || contains("error") || contains("Error")`. Waiting for stdout content here is backwards — a compile error may produce zero stdout lines and only stderr. Use `Completion`'s stderr hook instead of a stdout-only wait:
```java
new Completion() {
  @Override
  public boolean onStdout(String line) {
    return line.toLowerCase(Locale.ROOT).contains("error");
  }

  @Override
  public boolean onStderr(String line) {
    return true;
  }

  @Override
  public String describe() {
    return "a compile error on stdout or any stderr line";
  }
}
```
Add `import java.util.Locale;` if not already present. This releases the wait the moment either signal (a stdout error message OR any stderr output) appears, instead of waiting for a fixed stdout line count that a compile-failure path may never reach.

- [ ] **Step 5: Run all four migrated tests**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests.testOSMBean' --tests 'tests.BTraceFunctionalTests.testJfr' --tests 'tests.BTraceFunctionalTests.testOnMethodTrackRetransform' --tests 'tests.BTraceFunctionalTests.testOnelinerCompilationError'`
Expected: all 4 PASS, no `[harness] timed out` lines.

- [ ] **Step 6: Commit**

```bash
git add integration-tests/src/test/java/tests/BTraceFunctionalTests.java
git commit -m "test(harness): migrate remaining BTraceFunctionalTests edge cases off checkLines"
```

---

### Task 7: Migrate `testStartup`'s hand-rolled `CountDownLatch(checkLines)` to `OutputPump`/`Completion`

**Files:**
- Modify: `integration-tests/src/test/java/tests/RuntimeTest.java` (`testStartup`, ~L658-838; its caller `test(...)`, ~L242-248)
- Modify: `integration-tests/src/test/java/tests/BTraceFunctionalTests.java` (`testTraceAll`, `testThreadStart(dynamic=false)` call sites)
- Modify: `integration-tests/src/test/java/tests/ManifestLibsTests.java` (`launchAgent_manifestLibs`)

**Interfaces:**
- Consumes: `tests.harness.OutputPump.run(...)` (same signature already used by `attach()`/`attachOneliner()` — read those two methods in `RuntimeTest.java` first to copy the exact call shape and skip-line/skip-prefix lists), `Completion.untilContains`.
- Produces: `testStartup(String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v)` — a new overload later tasks/tests could use, though none currently need to.

`testStartup` currently hand-rolls its own stdout/stderr reader threads with a raw `CountDownLatch(checkLines)` (L727) plus a bug-adjacent stderr fallback that force-releases the latch on ANY non-filtered stderr line (L788-791: `for (int i=0;i<checkLines;i++) stdoutLatch.countDown();`) — meaning today, in practice, a single qualifying stderr line already satisfies the wait regardless of `checkLines`'s value. This is the same race `OutputPump`/`Completion` were built to remove from `attach()`.

- [ ] **Step 1: Read the existing `attach()` method's `OutputPump` wiring completely**

Before writing any code, read `RuntimeTest.java`'s `attach(String pid, String trace, String[] cmdArgs, Completion completion, StringBuilder stdout, StringBuilder stderr)` method in full (the one Task 3 of the original harness plan built). It already solves the same problem (spawn process, pump stdout/stderr through `OutputPump` with a `Completion`, apply the same stderr skip-substring list `testStartup` uses). `testStartup` is going to be restructured to follow the same shape.

- [ ] **Step 2: Add a `Completion`-based `testStartup` overload**

Add a new overload:
```java
public void testStartup(
    String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v)
    throws Exception {
  // Copy testStartup's existing setup exactly (agentPath, testJavaHome, args list construction,
  // agentSetup javaagent string, ProcessBuilder pb, jfrFile handling) up to the point where it
  // currently constructs `stdoutReader`/`stderrReader`/the two hand-rolled Thread objects and the
  // CountDownLatch fields.
  //
  // Replace that reader-thread block with the same OutputPump.run(...) call attach() uses,
  // passing `completion` through instead of `Completion.lines(checkLines)`. Preserve the existing
  // `testAppLatch`/`pidStringRef` "ready:" line handling from the current stdout loop (L737-742)
  // by keeping it as a side-channel line inspection ahead of/alongside the OutputPump call --
  // read how attach() (or attachOneliner(), whichever already coexists with a "ready:" style
  // pre-scan) structures this before deciding the exact placement, since RuntimeTest.java already
  // has a working precedent for combining a "ready:" latch with an OutputPump-driven Completion
  // wait in the same method.
  //
  // Keep the rest of testStartup's body identical: the JFR dump-before-kill block (already fixed
  // in a prior commit — do not reorder it), the final v.validate(...) call, and the finally-block
  // process cleanup.
}
```

- [ ] **Step 3: Make the old `int checkLines` `testStartup` overload delegate to the new one**

```java
public void testStartup(
    String testApp, String testScript, String[] cmdArgs, int checkLines, ResultValidator v)
    throws Exception {
  testStartup(testApp, testScript, cmdArgs, Completion.lines(checkLines), v);
}
```

- [ ] **Step 4: Migrate `test(...)`'s internal call, and the 3 direct callers**

`test(...)` (L242-248) calls both `testDynamic(...)` and `testStartup(...)` with the same `checkLines`. Since `testDynamic` already takes `Completion` after Task 5/6, change `test(...)`'s signature to take a `Completion` too, and have both inner calls use it:
```java
public void test(
    String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v)
    throws Exception {
  testDynamic(testApp, testScript, cmdArgs, completion, v);
  testStartup(testApp, testScript.replace(".java", ".class"), cmdArgs, completion, v);
}
```
Keep the old `int checkLines` `test(...)` overload delegating to this one via `Completion.lines(checkLines)`, for now (Task 8 removes it once all callers are off it).

Then migrate the 3 direct callers found during inventory:
- `testTraceAll` (via `test(...)`) — reuses `traces/TraceAllTest.class`; check its `ResultValidator` for markers (if, like `testOSMBean`, it has none beyond `!contains("FAILED")`, apply the same `Completion.untilMatches(Pattern.compile(".+"), 1)` treatment as Task 6 Step 1).
- `testThreadStart(dynamic=false)` (via `testStartup` directly) — same script as the already-migrated `dynamic=true` case (Task 5's `Completion.untilContains("starting testThread")`); reuse the identical marker.
- `launchAgent_manifestLibs` in `ManifestLibsTests.java` (via `testStartup` directly, `traces/TraceAllTest.class`, `checkLines=5`) — same `TraceAllTest` script as `testTraceAll`; reuse whatever `Completion` Task 7 Step 4's first bullet settles on for `testTraceAll`, since both exercise the identical script.

- [ ] **Step 5: Run the migrated tests**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests.testTraceAll' --tests 'tests.BTraceFunctionalTests.testThreadStart' --tests 'tests.ManifestLibsTests.launchAgent_manifestLibs'`
Expected: all PASS, no `[harness] timed out` lines.

- [ ] **Step 6: Commit**

```bash
git add integration-tests/src/test/java/tests/RuntimeTest.java integration-tests/src/test/java/tests/BTraceFunctionalTests.java integration-tests/src/test/java/tests/ManifestLibsTests.java
git commit -m "test(harness): migrate testStartup's CountDownLatch(checkLines) to OutputPump/Completion"
```

---

### Task 8: Delete the now-dead `int checkLines` overloads

**Files:**
- Modify: `integration-tests/src/test/java/tests/RuntimeTest.java`

**Interfaces:**
- Consumes: nothing new.
- Produces: nothing — pure deletion, only proceed once Tasks 1-7 have removed every caller.

- [ ] **Step 1: Confirm zero remaining callers of each `int checkLines` overload**

Run: `grep -rn "checkLines" integration-tests/src/test/java/tests/*.java | grep -v "RuntimeTest.java\|harness/"`
Expected: no output. If anything remains, that call site was missed by an earlier task — go back and migrate it (do not delete the overload while a caller still needs it).

- [ ] **Step 2: Delete the `int checkLines` overloads**

Delete: `testDynamic(String, String, String[], int, ResultValidator)`, `testDynamic(String, String, int, ResultValidator)` (the 4-arg convenience wrapper — check whether it delegates to the `int` or the soon-to-be-only `Completion` 4-arg form; if it currently delegates to the `int` 5-arg form, repoint it at the `Completion` 5-arg form instead of deleting, since it's a genuinely useful convenience overload), `testDynamicOneliner(String, String, int, ResultValidator)` and its `String[] cmdArgs` sibling (same convenience-overload judgment call as `testDynamic`), `attach(..., int checkLines, ...)`, `attachOneliner(..., int checkLines, ...)`, `testStartup(..., int checkLines, ...)`, and `test(..., int checkLines, ...)`.

For each: if it's a thin `Completion.lines(checkLines)` delegator with no other logic, delete it outright. If it's a convenience overload whose only "extra" is supplying a default (e.g., `cmdArgs = null`), keep it but change its parameter from `int checkLines` to `Completion completion` and have it delegate to the `Completion`-based longer overload instead.

- [ ] **Step 3: Compile the whole module**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava`
Expected: BUILD SUCCESSFUL. Any compile error here means Step 1's grep missed a caller — find it and migrate it properly rather than papering over the compile error.

- [ ] **Step 4: Commit**

```bash
git add integration-tests/src/test/java/tests/RuntimeTest.java
git commit -m "test(harness): delete dead int-checkLines overloads now that all callers use Completion"
```

---

### Task 9: Full-suite verification

**Files:** none (verification only)

- [ ] **Step 1: Run spotlessApply**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew spotlessApply`
If it modifies any files, `git add` and commit them separately: `git commit -m "spotless"`.

- [ ] **Step 2: Run the full integration-tests suite**

Run: `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test`
Expected: all tests PASS. Capture the full output to a log file rather than relying on terminal scrollback:
`GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test > /tmp/checklines-migration-verify.log 2>&1; echo "exit: $?"`

- [ ] **Step 3: Confirm zero spurious harness timeouts**

Run: `grep -c "\[harness\] timed out" /tmp/checklines-migration-verify.log`
Expected: `0`. If any remain, find which test produced them (search the log for the preceding test name) — either that call site was missed, or its `Completion` markers don't match what the script actually prints (re-check against the actual `.java` probe source under `integration-tests/src/test/btrace/`, not this plan's transcription of it).

- [ ] **Step 4: Push and let CI confirm across all 6 JDK versions**

This is the real test — the local JDK is only one of the 6 matrix versions. Push the branch and check `gh run view --json jobs` once the run completes, the same way prior CI investigation in this session did. Do not declare this plan complete until CI shows all 6 `test (*)` jobs green, not just local success on one JDK.

---

## Self-Review

**Spec coverage:** All 27 call sites from the inventory are covered — Tasks 2-6 cover the 4 named `RuntimeTest` helpers' direct callers (`ClassFileApiTests` ×4, `ExtensionLifecycleIntegrationTest` ×3, `ManifestLibsTests.dynamicAttach_manifestLibs` ×1, `BTraceFunctionalTests` ×19 split across Tasks 5 and 6). Task 7 covers the separate `testStartup` `CountDownLatch` mechanism and its 3 callers (`testTraceAll`, `testThreadStart(dynamic=false)`, `launchAgent_manifestLibs`) that the original harness plan explicitly left out of scope. Task 1 and Task 8 clean up dead/superseded overloads. Task 9 verifies against the full suite and CI, matching this session's actual failure signal (multiple JDK jobs, different tests each time).

**Placeholder scan:** Task 7's Step 2 code block contains descriptive comments instead of full inline code for the `OutputPump` wiring — this is intentional and flagged explicitly as "read `attach()`'s existing implementation first and mirror its exact shape," because copying that ~80-line method's exact structure into this plan verbatim would drift out of sync with the real file the moment either changes; the task tells the implementer precisely which existing method to mirror and which two behaviors (the "ready:" latch, the JFR dump ordering) must be preserved. This is the one deliberate exception to "no placeholders" in this plan, scoped to a single step, with a concrete named reference implementation to copy rather than an abstract instruction.

**Type consistency:** `Completion`, `untilContains`, `untilMatches`, `lines`, `onStdout`, `onStderr`, `describe` are used identically to their existing definitions in `tests.harness.Completion` (no new methods on the interface are introduced by this plan). The new `testStartup(..., Completion, ...)` and `test(..., Completion, ...)` overloads in Task 7 match the exact parameter-ordering convention already established by `testDynamic(..., Completion, ...)`.

---

### Superpowers/Plans/2026 07 12 Extension Loader Classcircularity Investigation

# Investigation: `ClassCircularityError` in `ExtensionLoader.initialize()`

**Status:** Root cause identified. No fix implemented yet — this is a diagnosis, pending a decision on fix strategy.

## Symptom

`BTraceFunctionalTests.testJfr()` intermittently fails (observed 1-in-3 to 1-in-8 runs) on
`assertTrue(stderr.isEmpty(), "Non-empty stderr")` during the on-startup (`-javaagent`) phase.
The target JVM's `premain()` throws:

```
java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1
    at java.base/java.lang.invoke.MethodHandle.customize(MethodHandle.java:1741)
    at java.base/java.lang.invoke.MethodHandle.maybeCustomize(MethodHandle.java:1731)
    at java.base/java.lang.invoke.Invokers.maybeCustomize(Invokers.java:634)
    at java.base/java.lang.invoke.Invokers.checkCustomized(Invokers.java:628)
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
    at io.btrace.extension.ExtensionLoader.initialize(ExtensionLoader.java:95)
    at io.btrace.agent.Main.initExtensions(Main.java:432)
    at io.btrace.agent.Main.main(Main.java:287)
    at io.btrace.agent.Main.startAgent(Main.java:161)
    at io.btrace.agent.Main.premain(Main.java:151)
```

which cascades into the JVM's native instrument-agent assertion (`*** java.lang.instrument
ASSERTION FAILED ***`, `JPLISAgent.c:422`) and dumps the whole stack to the target process's
stderr.

## Reproduction

- Confirmed via 3 sequential (non-concurrent) runs of `BTraceFunctionalTests.testJfr` alone,
  with `BTRACE_TEST_DEBUG=true`: run 1 pass, run 2 pass, run 3 fail (this exact trace).
- Also observed once during the harness-redesign work on `jb/external-type-adapter` (now merged
  into `develop` via #860).
- Load-sensitive: reproduces more readily when many JVMs have recently run on the machine
  (consistent with a startup-timing race, not a deterministic bug).

## Root cause

`ExtensionLoader.initialize()` (`btrace-core/src/main/java/io/btrace/extension/ExtensionLoader.java:95`):

```java
ServiceDeclarationRegistry.setResolver(fqcn -> instance.findExtensionForService(fqcn) != null);
```

This is a **lambda expression**. javac compiles it to an `invokedynamic` call site bound to
`LambdaMetafactory`, so the *first* time this line executes, the JVM must bootstrap that call
site — which is exactly the `MethodHandleNatives.linkCallSite` → `CallSite.makeSite` →
`MethodHandle.customize` path in the stack trace above.

`ExtensionLoader.initialize()` runs from `Main.initExtensions()` → `Main.main()`, which is called
directly from `-javaagent` `premain()` — i.e., extremely early in target-JVM startup, before
`main()` (the application's own `public static void main`) has even begun. At this point the
JVM's own `java.lang.invoke` subsystem (which lazily bootstraps `MethodHandle`, `MethodHandle$1`,
`CallSite`, etc. on first use) may not have finished its own class initialization. Triggering a
*fresh* `invokedynamic` linkage this early — under load, when JIT/class-loading threads are
contending for CPU — can race with the JVM's own in-progress initialization of the same classes,
producing `ClassCircularityError`.

This is a load-sensitive startup race, not a logic bug — which matches the observed intermittency.

## Why the existing `LinkingFlag`/`MethodHandleNatives` guard does NOT protect this

The codebase already has infrastructure that looks related but solves a **different** problem:

- `Main.main()` (`btrace-agent/src/main/java/io/btrace/agent/Main.java:252-265`), right before
  `initExtensions()`, retransforms `java.lang.invoke.MethodHandleNatives` via
  `LinkerInstrumentor.addGuard(...)` (`btrace-agent/src/main/java/io/btrace/instr/LinkerInstrumentor.java`).
  This injects `io.btrace.runtime.LinkingFlag.guardLinking()` / `.reset()` calls around
  `MethodHandleNatives.linkCallSite`/`linkMethodHandleConstant`.
- `LinkingFlag` (`btrace-core/src/main/java/io/btrace/runtime/LinkingFlag.java`) is a
  **per-thread reentrancy depth counter** (`ThreadLocal<Integer>`). Its purpose, per the comment
  at `BTraceTransformer.java:135-136`, is "to be able to safely skip BTrace probes while linking
  is still in progress" — i.e., it lets BTrace's **own instrumented probe-dispatch code** detect
  "I am currently re-entering indy-linking machinery on this thread" and bail out, avoiding
  deadlock/infinite recursion when a *user's* instrumented method itself triggers indy linking.

Two reasons this doesn't help our bug:
1. **Nothing reads `LinkingFlag` on the path that crashes.** `ExtensionLoader.initialize()`'s
   lambda doesn't check it, and nothing in the `premain()` call chain up to that point does
   either. The flag is consumed by probe-dispatch code (elsewhere), not by the agent's own
   startup sequence.
2. **It's a reentrancy guard, not a readiness barrier.** Even if consulted, `LinkingFlag` only
   answers "is a link operation already in progress on my thread" — it says nothing about
   whether the JVM's `java.lang.invoke` bootstrap has *fully completed* globally. It cannot tell
   `ExtensionLoader.initialize()` "wait, the JVM isn't ready for you to trigger indy linking yet."
3. The retransform of `MethodHandleNatives` that installs the guard is itself wrapped in a
   silent `catch (Throwable t) { log.debug(...) }` (`Main.java:260-264`) — if it fails or doesn't
   land in time, there is no fallback protection at all, and no visible signal that this happened.

## Confirmed: unrelated to the just-merged test-harness redesign

The crash occurs entirely inside the **target JVM's own `premain()`**, before the harness's
`RuntimeTest` reader code (`attach`/`attachOneliner`/`runBTrace`, all migrated onto
`OutputPump`/`Completion` in #860) reads a single line of output. Traced and confirmed on
`jb/external-type-adapter` before merge: the "Dynamic attach" phase (which exercises the
migrated `attach()`) completed and printed `Detached.` cleanly in every reproduction; the failure
always occurs later, in the separate "On-Startup" (`testStartup`) phase, which is structurally
untouched by that harness work.

## Candidate fix directions (not yet decided/implemented)

1. **Avoid the lambda at the crash site.** Replace `ExtensionLoader.java:95`'s lambda with an
   anonymous inner class implementing `ServiceDeclarationRegistry.Resolver`. Anonymous classes
   are ordinary `invokespecial`/`new`-based construction — no `invokedynamic`, no
   `LambdaMetafactory` bootstrap. Minimal, surgical, but only fixes *this one* call site; any
   other lambda/method-reference/string-concat-via-indy touched this early in `premain()` remains
   equally exposed (untargeted audit of the full `premain()`→`initExtensions()` path for other
   `invokedynamic` sites has not yet been done).
2. **Warm up `java.lang.invoke` deliberately, before the agent touches it.** Add an explicit,
   early, single-threaded "touch a trivial lambda/MethodHandle and let its bootstrap fully
   resolve" step at the very start of `premain()`/`Main.main()`, before `initExtensions()` (and
   ideally before anything else in the agent that might use `invokedynamic`). This is closer to
   your instinct: force `java.lang.invoke`'s own bootstrap to complete in a controlled context
   first, rather than avoiding lambdas piecemeal. Open questions to resolve before implementing:
   - Does forcing this warm-up *from within `premain()`* still race with the JVM's own
     concurrent bootstrap, or does it need to happen even earlier (e.g., via a bootstrap-classpath
     class, or is `premain()` early enough already)?
   - Is there a reliable, catchable signal for "warm-up succeeded" vs. silently swallowing a
     `ClassCircularityError` during warm-up itself?
3. **Combine both:** do the warm-up as the primary defense (protects all future/current
   `invokedynamic` use in early agent code), and still convert the one identified lambda to an
   anonymous class as defense-in-depth for this specific known-hot call site.

## Next steps

- Decide fix direction (1, 2, or 3 above) — recommend **2+1 combined**, given the intermittency
  suggests other undiscovered `invokedynamic` sites in the early `premain()` path could trigger
  the same race even after fixing this one lambda alone.
- If proceeding, this should go through TDD/systematic-debugging per project convention:
  reproduce reliably (this doc's repro recipe), form a single hypothesis, test minimally.
- Reproducing this race reliably enough for a regression test is itself nontrivial (it's
  load-sensitive) — likely needs either a stress-loop test (spin up N target JVMs concurrently)
  or a way to artificially delay/contend `java.lang.invoke` bootstrap in a test harness.

---

### Superpowers/Plans/2026 07 29 Issue 932 Stall Watchdog Plan

# Implementation plan — stall watchdog for integration tests

Design: `docs/superpowers/specs/2026-07-29-issue-932-client-close-deadlock-design.md`.
Branch: `agent/issue-932-client-close-deadlock`, worktree `.worktrees/issue-932-client-close`.
Base: `origin/develop` @ `840564e9`. Independent of PR #934.

Revision 2. Adversarial review of revision 1 found that the SIGQUIT fallback would have written a
thread dump into the target stdout buffer that test validators assert on (`RuntimeTest.java:858` →
`ManifestLibsTests.java:81`), turning a diagnostic into a test failure. That is removed, along with
the echo flag that existed only to serve it. Other corrections are noted where they apply.

Every command below runs from this worktree, prefixed `GRADLE_USER_HOME=$(pwd)/.gradle-user`, with
output redirected to a log before filtering, per `AGENTS.md`.

## Constraints

- **Java 8 source level** for `integration-tests` tests (`build.gradle:15-18`). No `var`, no
  `List.of`, no direct `Process.pid()`. Note this is enforced by convention only, and CI actually
  runs the harness on Java 24 — the matrix JDK is the *target* JVM via `TEST_JAVA_HOME`.
- **One worker JVM for all ten test classes.** No `forkEvery`, `maxParallelForks`, or
  `junit-platform.properties` exists today. Static state is shared for the whole run, and a system
  property set by one test leaks into every class after it.
- Test knobs arrive via `-P` (`build.gradle:274` forwards project properties beginning with
  `btrace.`).
- The watchdog must never become the hang it diagnoses, and must never perturb the system under
  test. Every wait is bounded; every internal failure is recorded as text, not thrown.

## Task 1 — `TargetRegistry`

New: `integration-tests/src/test/java/tests/harness/TargetRegistry.java`

- `static Handle register(Process process, String label, String jcmdPath)` — called at launch,
  before a pid is known. **`jcmdPath` is passed in at registration**, not resolved centrally:
  `resolveTestJavaHome()` is `protected static` in package `tests` (`RuntimeTest.java:128`) and is
  unreachable from `tests.harness`, and targets do not all come from the same JDK —
  `Issue888RuntimeHardeningIntegrationTest.java:172` launches with the *build* JDK while the
  `RuntimeTest` paths use `resolveTestJavaHome()`.
- `Handle.setPid(String)` — called where a launch site parses `ready:`.
- `static List<Snapshot> liveTargets()` — entries where `process.isAlive()`. Registering at launch
  and filtering on liveness at capture removes any need for deregistration hooks, and makes the
  never-stopped `TestApp` at `BTraceFunctionalTests.java:535` harmless.
- Pid resolution when `ready:` never arrived — the startup-stall case, which is exactly when the
  registry must still be useful: fall back to `Process.pid()` **via reflection**, cached, guarded,
  returning null on failure. Reflection keeps the Java 8 source level while working on the Java 24
  worker CI actually uses.
- `Snapshot` is an immutable `(label, pid, process, jcmdPath)`.
- Backed by `CopyOnWriteArrayList`.

## Task 2 — `StallCapture`

New: `integration-tests/src/test/java/tests/harness/StallCapture.java`

`static void capture(Appendable sink, String label, long elapsedMs, int frame, List<Snapshot> targets, long budgetMs)`
— writes the report and never throws.

- Sections in order: banner (label, elapsed, frame index); test-JVM threads; then each live target.
- **The sink is flushed after every section.** Revision 1 built the whole report as a `String` and
  wrote it at the end, so a hang in any section produced zero bytes — the precise failure being
  designed against.
- The test-JVM dump is `ThreadMXBean.dumpAllThreads(true, true)` (lock owners and monitors, which
  `Thread.getAllStackTraces()` omits) computed **on a throwaway thread with a bounded join**. It
  requires a safepoint, and a JVM that cannot reach one is a live possibility here. On overrun the
  section reads `test-JVM dump timed out`.
- Per target: `jcmd <pid> Thread.print -l` using the snapshot's own `jcmdPath`, with output
  redirected to a temp file — **never a pipe**. `Thread.print -l` on an instrumented target exceeds
  the 64 KB pipe buffer, and a jcmd blocked on a full pipe would be killed by the bound having
  produced nothing.
- `budgetMs` spans all targets together, checked before each; a skipped target says so.
- **No SIGQUIT fallback.** A HotSpot SIGQUIT dump goes to the target's stdout, and for
  `testStartup` targets that stream is accumulated by `OutputPump` (`OutputPump.java:67`) into the
  buffer `v.validate(stdout.toString(), …)` asserts on (`RuntimeTest.java:858`). It would corrupt
  the system under test. A jcmd that fails or overruns is recorded as text and that is the end of it.
- Every line is prefixed `[stall-dump] ` so a capture interleaved with the `[traced app]` and
  `[btrace out]` reader threads is recoverable with one `grep`.

## Task 3 — `StallWatchdog` and `@StallTimeout`

New: `integration-tests/src/test/java/tests/harness/StallWatchdog.java`,
`integration-tests/src/test/java/tests/harness/StallTimeout.java`

`BeforeAllCallback` + `BeforeEachCallback` + `AfterEachCallback` + `AfterAllCallback`.

- Arms on `beforeAll` — JUnit fires extension `beforeAll` before the class's own `@BeforeAll`, and
  every subclass calls `classSetup()` from one, so that path is covered — and re-arms per
  `beforeEach`.
- Deadline: `@StallTimeout(millis=…)` on method or class, else `-Pbtrace.test.stallTimeoutMs`, else
  `btrace.test.timeoutMs * 6` (six minutes at the default). The annotation exists because a system
  property cannot be scoped to a single test in a single-JVM run. A shorter default is not safe:
  `RuntimeTest.java:784` already allows a target `timeout * 4` = 240 s for startup alone.
- **Targets are snapshotted once, at fire time**, and both frames capture those exact `Process`
  objects. Revision 1 re-read the live registry for frame two, so a test that unwedged during the
  gap would have had the *next* test's target captured.
- Frame two is a **separately scheduled task** at +30 s, not a sleep inside frame one, and it
  re-checks the disarm flag and skips if the test has finished. Two frames because one cannot
  distinguish "wedged" from "slow"; the diff is the evidence. Scheduler pool size 2 so one fire
  cannot defer another test's deadline.
- Disarm: `ScheduledFuture.cancel(false)` plus an `AtomicBoolean`; `afterEach` never blocks on an
  in-flight capture.
- Output dir comes from the absolute path in `btrace.test.stallDumpDir` (task 4). A relative path
  would resolve against `Test.workingDir`, which defaults to the *project* directory, yielding
  `integration-tests/integration-tests/build/...` — outside the upload glob.
- Filenames: sanitised JUnit unique id truncated to 120 chars plus a short hash of the full id.
  The full id of a parameterised invocation exceeds the 255-byte filename limit, and the motivating
  test is a `@ParameterizedTest` whose four rows share one method name.
- Sink order is **file first, stdout second**. `if: always()` upload steps do run after a
  `timeout-minutes` cancellation, whereas a cancelled job tears down the worker→daemon→console
  pipeline, and the release path redirects all test output into a file it only summarises after the
  command exits (`scripts/run-release-gate.sh:52`). Stdout is a shared synchronised `PrintStream`
  that a stalled pipeline can itself block on.
- Stale dumps from earlier runs are cleared once per JVM on first arm; `cleanTest` does not cover
  this directory.

## Task 4 — build wiring

Modify: `integration-tests/build.gradle` — in the `test` block (`:163`), add

```
systemProperty 'btrace.test.stallDumpDir',
    layout.buildDirectory.dir('reports/stall-dumps').get().asFile.absolutePath
```

Its own line, because the `-P` forwarding loop at `:274` forwards project properties only.
`build/reports/**/*` is what both `if: always()` upload steps already collect
(`continuous.yml:171`, `release.yml:377`), so no workflow change is needed. Confirm that during
task 8 rather than assuming it.

`options.release = 8` is **not** attempted: `Issue884PublishedFatAgentE2ETest` already calls
`Files.writeString` (JDK 11) at seven sites (`:173, :187, :239, :245, :279, :285, :289`), so the
module cannot compile at release 8 today. Recorded as a pre-existing violation, not fixed here.

## Task 5 — a per-test timeout, because nothing else provides one

New: `integration-tests/src/test/resources/junit-platform.properties`

```
junit.jupiter.execution.timeout.testable.method.default = 8m
junit.jupiter.execution.timeout.thread.mode.default = SEPARATE_THREAD
```

PR #934's commit message describes exactly this and **its diff contains neither** — verified with
`git show 44f2c25d`. Without it the watchdog's outcome is "30 minutes still burned, now with
evidence". With it, a stall becomes a failing test at 8 minutes.

Eight minutes, not five: a passing `test()` invocation runs `testDynamic` then `testStartup` in one
method, and `startupTimeoutMs` alone is 240 s, so five minutes risks failing green tests on slow
lanes. Eight sits above that and below `release.yml`'s 15-minute integration job (`:271`).
`SEPARATE_THREAD` is required because the default mode interrupts the test thread, and a thread
blocked in socket I/O ignores an interrupt.

Ordering: watchdog captures at 6:00 and 6:30, timeout fails the test at 8:00.

This mode change affects every integration test, so task 8's full-suite run is what qualifies it.

## Task 6 — wire the registry into the launch sites

Modify: `integration-tests/src/test/java/tests/RuntimeTest.java`

- `@ExtendWith(StallWatchdog.class)` on the class; `@ExtendWith` is `@Inherited` and all eight
  subclasses pick it up.
- Register at launch, set the pid where `ready:` is parsed. Corrected line numbers, all verified:

  | site | launch | pid parsed |
  |---|---|---|
  | `testDynamic` raw path | `:348` | `:367` |
  | `testDynamicOneliner` raw path | `:540` | `:559` |
  | `testStartup` raw path | `:745` | `:765` (in the `readyAwareCompletion` decorator, not a reader thread) |
  | `TestApp` | constructor, `:1067` caller | `:937` |

  `TestApp.pid` is a private `int` whose only accessor `getPid()` (`:1007`) blocks up to 30 s, so the
  hook goes at `:937` directly.
- No echo-flag changes anywhere: the SIGQUIT fallback that needed them is gone.

Modify: `integration-tests/src/test/java/tests/Issue888RuntimeHardeningIntegrationTest.java` —
register the target built at `:172-189`; the pid is parsed by the *caller* at `:83-85`, so
`startTarget()` must return the handle alongside the process rather than a bare `Process`.

Client-JVM spawns (`RuntimeTest.java:1109, :1255, :1380, :1493`) are registered too where a handle
can be threaded without restructuring; where it cannot, the omission is recorded in the commit
message rather than left silent. `:254` (`hasJaxbProbeDescriptorSupport`) stays unregistered — it is
bounded at 10 s and cannot stall a job.

## Task 7 — tests

New: `integration-tests/src/test/java/tests/harness/StallCaptureTest.java`,
`integration-tests/src/test/java/tests/harness/StallWatchdogTest.java`

Driven through the in-memory seam, scoped with `@StallTimeout`, output under `@TempDir`.

1. **Captures the test JVM.** Park a distinctively named thread; assert banner, label, thread name
   and stack all appear.
2. **Captures a target JVM.** Launch a JVM that blocks until told to exit (not "short-lived" — a
   target that exits mid-attach makes jcmd nondeterministic), launched from the same JDK whose
   `jcmd` is used. Assert a `Thread.print` section for its pid appears. Degrade with an assumption
   rather than a failure if attach is unavailable in the environment.
3. **Degrades rather than hangs.** A bogus pid: capture returns inside its budget, records the
   failure inline, still contains the test-JVM dump.
4. **Budget is shared.** Several unreachable targets: capture returns in roughly the budget, not
   budget × N, and names what it skipped.
5. **Silent when green.** A fast test with the watchdog armed writes nothing into the `@TempDir`
   output dir.
6. **Fires end to end, with a target.** `@StallTimeout(millis=1500)` on a test that registers a real
   blocking target and then sleeps ~3 s; assert the dump names both the sleeping thread and the
   target's `Thread.print` section. This replaces revision 1's throwaway "add a test, run it, delete
   it" verification step, which produced evidence nobody could re-verify.
7. **Two frames.** A test that stays parked past the 30 s gap produces two dump files whose thread
   sections differ in elapsed time — the frame-two mechanism is otherwise never exercised.
   Uses a shortened inter-frame gap via the same annotation so it runs in seconds.

## Task 8 — verification

1. `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew spotlessApply`, commit any formatting change.
2. `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava`.
3. `GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration :integration-tests:test --tests 'tests.harness.*'`
   — note this also runs the pre-existing `CompletionTest` and `OutputPumpTest`, and still triggers
   the full dist/probe build chain (`build.gradle:167-172`), so it is not a cheap check.
4. Full suite as CI runs it:
   `GRADLE_USER_HOME=$(pwd)/.gradle-user TEST_JAVA_HOME=<jdk> ./gradlew -Pintegration -PCI :integration-tests:test`.
   This is the gate for task 5's `SEPARATE_THREAD` change. Record the duration and confirm no
   watchdog file is produced. Docker-tagged tests are included whenever a Docker host is reachable
   (`build.gradle:176-241`), so hold Docker state constant between baseline and after.
5. Confirm `build/reports/stall-dumps/` is inside the upload glob by listing what a run leaves in
   `integration-tests/build/reports/`.

## Order

1 → 2 → 3 → 7 (tests for 1–3) → 6 → 4 → 5 → 8.


## Revision 3 — corrections found by adversarial review of the implementation

Two defects would have shipped silently, and both were confirmed by direct measurement rather than
argument:

1. **The per-test timeout was never committed.** `.gitignore`'s bare `junit*` pattern matches
   `integration-tests/src/test/resources/junit-platform.properties` at any depth, so the file existed
   locally, passed every local run, and was absent from the commit. `.gitignore` now carries a
   negation, placed *after* the pattern it overrides, since the last matching pattern wins.
2. **Test-JVM stacks were truncated to eight frames.** `ThreadInfo.toString()` caps output at
   `MAX_FRAMES = 8`; a 44-frame stack rendered as 8. The dump is now rendered frame by frame with
   locked monitors and synchronizers, and a test asserts the rendered depth exceeds eight.

Also corrected: the stdout echo was the one unbounded wait in a class that bounds everything else,
and it ran on one of only two scheduler threads — it now runs on a throwaway thread with a bounded
join. The registry is pruned on disarm, so a late stall does not spend its capture budget on
`jcmd` calls against processes from earlier classes. Dump filenames are truncated from the tail, not
the head, because a parameterised unique id is identical up to its invocation index, and carry a
sequence number so two teardown stalls in one class cannot collide on one path. The derived deadline
is clamped below the per-test timeout, since `btrace.test.timeoutMs` is an operator knob that could
otherwise push the watchdog past it. The four btrace client JVM launch sites are registered — the
client is the side the surviving issue-932 hypothesis implicates. `budgetIsSharedAcrossTargets`
passed against a per-target budget and now does not; `capturesTargetJvmThreads` would have failed a
build rather than skipped when `jcmd` cannot attach on a loaded runner.

---

### Tutorials/01 First Trace In 2 Minutes

# Your First Trace in 2 Minutes

Trace a live JVM — no code changes, no restart, no config. You'll find a latency bug and an
intermittent exception in a running application using nothing but one-line commands.

**Persona:** anyone with a JVM that's misbehaving. **Time:** ~5 minutes (first trace in 2).

## What you'll need

- JDK 11 or newer on your PATH (the demo uses single-file source launch)
- BTrace 3.0 installed — `bin/btrace` on your PATH ([installation options](../GettingStarted.md#installation))
- Two terminal windows

## Step 1 — Start the patient

The demo app ([demo/DemoApp.java](demo/DemoApp.java)) is a tiny order-processing service with two
deliberate defects hidden inside. Run it in terminal 1:

```sh
java DemoApp.java
```

**You should see** (numbers will vary):

```
[demo] order service running - stop with Ctrl+C
[demo] processed 64 orders, 8 failed
[demo] processed 134 orders, 14 failed
```

Orders are being processed, and some of them fail. Why? The app won't tell you — let's ask the JVM
directly.

## Step 2 — Find its PID

In terminal 2:

```sh
jps
```

**You should see** a line like `12345 DemoApp`. That number is the `<PID>` in every command below.

## Step 3 — Your first trace

A BTrace *oneliner* attaches to the running JVM and prints every call of a method, live:

```sh
btrace -n 'OrderService::processOrder @return { print method, time }' <PID>
```

**You should see** a stream like:

```
processOrder
execution time: 61 ms
processOrder
execution time: 74 ms
processOrder
execution time: 342 ms
```

> **What just happened?** BTrace compiled your one-liner into a tiny instrumentation program,
> injected it into the running JVM, and hooked the return of `OrderService.processOrder` — all
> while the app kept serving orders. Nothing was restarted, and when you detach, every hook is
> cleanly removed. Curious what it generated? Re-run with `-Dbtrace.oneliner.dump=true` to see the
> generated source.

Notice something? Most orders take well under 100 ms, but every now and then one takes 300+ ms.
Let's isolate the slow ones.

## Step 4 — Catch the latency bug

Stop the previous trace with `Ctrl+C`, then widen the net to *all* `OrderService` methods and keep
only the slow calls:

```sh
btrace -n 'OrderService::* @return if duration>200ms { print method, time }' <PID>
```

**You should see** two method names, always together:

```
chargeCard
execution time: 306 ms
processOrder
execution time: 331 ms
```

Case closed: `chargeCard` is the culprit (a "slow payment provider" hits ~10% of calls), and its
latency is what makes `processOrder` slow. You found it with one line, without reading a single
line of application code.

> `if duration > 200ms` filters on the method's execution time. In `print`, the raw `duration`
> value is in **nanoseconds**; the `time` action prints it converted to milliseconds for you.

## Step 5 — Catch the failures

The summary line also reported failed orders. Hook the *error path* — this probe only fires when a
method exits by throwing:

```sh
btrace -n 'OrderService::validateOrder @error { print method, stack }' <PID>
```

**You should see**, for roughly one order in twelve:

```
validateOrder
OrderService.validateOrder(DemoApp.java:93)
OrderService.processOrder(DemoApp.java:85)
...
```

The exception's origin and the exact call path — captured live, from a method that the application
catches and silently swallows.

## Step 6 — Clean up

`Ctrl+C` in the BTrace terminal detaches the client. In BTrace 3.0, detaching disables all
injected probes on the spot (they become no-ops — no restart, no lingering overhead), and you can
re-attach at any time. Stop the demo app with `Ctrl+C` in terminal 1 when you're done.

## Troubleshooting

- **`Can not attach to PID ...`** — make sure you run `btrace` as the same OS user as the target
  JVM, and check [Troubleshooting: attachment issues](../Troubleshooting.md).
- **A warning about your Java version** — if the target JVM is older than Java 17 you'll see the
  3.0 deprecation warning. Everything still works; see the
  [Java support policy](../Migration-2.x-to-3.0.md) for what it means.
- **No output appears** — the probe may not match: patterns are exact class names plus `*`/`?`
  wildcards. Try `OrderService::*` first, then narrow down.

## Go deeper

- Full oneliner syntax (filters, `args`, `count`, regex patterns): [Oneliner Guide](../OnelinerGuide.md)
- Turn a oneliner into a real script with the flat DSL: [Tutorial Lesson 7](../BTraceTutorial.md)
- Everything you can hook (`@OnMethod` locations, JFR, timers): [Quick Reference](../QuickReference.md)

---

### Tutorials/02 Oneliner To Script

# From Oneliner to Script: The Flat DSL

Turn a one-line trace into a real, version-controllable script — and give it the state a
one-liner can't have — in about 10 minutes.

**Persona:** Java devs who want a repeatable probe, not just a one-off. **Time:** ~10 minutes.

This picks up exactly where [Tutorial 1](01-first-trace-in-2-minutes.md) left off: same demo
app, same JVM, same bug hunt. If you don't have it running anymore, start it again in one
terminal (`java DemoApp.java`) and find its PID with `jps` in another — everything below uses
that `<PID>`.

## What you'll need

- JDK 11 or newer on your PATH, and the [demo app](demo/DemoApp.java) running (`java DemoApp.java`)
- BTrace 3.0 installed — `bin/btrace` on your PATH
- A text editor, and this tutorial's script,
  [demo/OrderTiming.java](demo/OrderTiming.java), open next to the demo app

## Step 1 — Peek at what your oneliner really compiled to

Tutorial 1's oneliner wasn't magic — BTrace turned it into a real Java class before compiling it.
You can watch that happen. The client reads the `btrace.oneliner.dump` system property, and
because the `btrace` launcher forwards its arguments to the client's `main()` rather than to the
JVM itself, the way to set that property is `JAVA_TOOL_OPTIONS` (the JVM will confirm it picked
the option up — that first line below is normal and harmless):

```sh
JAVA_TOOL_OPTIONS="-Dbtrace.oneliner.dump=true" btrace -n 'OrderService::processOrder @return { print method, time }' <PID>
```

**You should see**, before the familiar trace stream starts:

```
Picked up JAVA_TOOL_OPTIONS: -Dbtrace.oneliner.dump=true
=== Generated oneliner source (BTraceOneliner_1752219841233.java) ===
package io.btrace.generated;

import java.util.concurrent.atomic.AtomicInteger;
import io.btrace.core.BTraceUtils;
import io.btrace.core.annotations.*;
import io.btrace.core.types.AnyType;

@BTrace
public class BTraceOneliner_1752219841233 {

  @OnMethod(clazz="OrderService", method="processOrder",
      location=@Location(Kind.RETURN))
  public static void probe(@ProbeMethodName String method, @Duration long duration) {
    BTraceUtils.println(method);
    BTraceUtils.println("execution time: " + (duration / 1000000) + " ms");
  }
}
processOrder
execution time: 58 ms
processOrder
execution time: 296 ms
```

(The class name's numeric suffix is just `System.currentTimeMillis()` at compile time — yours
will differ. The trace stream underneath is identical to Tutorial 1's, because it's the exact
same generated class doing the work.)

> **What just happened?** The oneliner compiler (`OnelinerCodeGenerator`) turned your probe
> spec into a small, throwaway `@BTrace` class: one `@OnMethod` hook at `Kind.RETURN`, one
> `println` per action. Notice it writes classic `BTraceUtils.println(...)` calls with an
> explicit `import io.btrace.core.BTraceUtils;` — the generator predates, and doesn't need, the
> flat DSL. But *your own* scripts get the flat DSL automatically, which is what the rest of this
> tutorial is about.

`Ctrl+C` to detach before moving on.

## Step 2 — The same probe, as a real script file

Open [demo/OrderTiming.java](demo/OrderTiming.java) — for now, replace its contents with the
minimal version below (you'll build it back up over the next two steps). This is hand-written,
not generated, and needs no imports at all:

```java
@BTrace
public class OrderTiming {
    @OnMethod(clazz = "OrderService", method = "processOrder", location = @Location(Kind.RETURN))
    public static void onReturn(@ProbeMethodName String method, @Duration long duration) {
        println(method);
        println("execution time: " + (duration / 1_000_000) + " ms");
    }
}
```

Deploy it — note there's no `-n` and no quoting, just a filename after the PID:

```sh
btrace <PID> OrderTiming.java
```

**You should see** the same two-line-per-order output as Step 1:

```
processOrder
execution time: 61 ms
processOrder
execution time: 342 ms
```

> **What just happened?** `btrace <pid> <file>` is BTrace's other submission mode (`-n` is a
> shorthand for the first one). The client reads `OrderTiming.java`, and before handing it to
> `javac` it checks whether the source already imports the DSL or the annotations; since this
> file imports neither, the compiler prepends both `import static io.btrace.BTrace.*;` and
> `import io.btrace.core.annotations.*;` for you (`Compiler.injectDslImport`). That's why
> `println`, `@OnMethod`, and `@BTrace` all resolved with zero imports on your end — and it's the
> same auto-injection path whether the source came from a `.java` file like this one or was typed
> straight into a oneliner.

`Ctrl+C` to detach.

## Step 3 — Make the message richer

`println(method)` on one line and a hand-built duration string on the next is a bit clunky. The
flat DSL (`io.btrace.BTrace`) gives you small helpers for exactly this — `str()` to stringify
non-`String` values, `concat()` to join two strings, `timestamp()` for wall-clock milliseconds
(there's also `monotonic()` for nanosecond timing, which you'll use in the next step). Update the
script to build one line instead of two:

```java
@BTrace
public class OrderTiming {
    @OnMethod(clazz = "OrderService", method = "processOrder", location = @Location(Kind.RETURN))
    public static void onReturn(@ProbeMethodName String method, @Duration long duration) {
        String line = concat(method, " took ") + str(duration / 1_000_000) + "ms at " + str(timestamp());
        println(line);
    }
}
```

Redeploy the same way:

```sh
btrace <PID> OrderTiming.java
```

**You should see** one line per order instead of two:

```
processOrder took 295ms at 1783778244337
processOrder took 68ms at 1783778244419
processOrder took 48ms at 1783778244467
processOrder took 52ms at 1783778244520
```

> **What just happened?** `concat(a, b)` is a plain two-string join — it exists mainly because
> it's null-safe (returns the non-null side, or `""` if both are null), which matters once you're
> composing strings from probe data instead of literals. `str(duration / 1_000_000)` converts the
> `long` to a `String` the same way `String.valueOf` would; `+` still works fine for the rest,
> since these are ordinary Java strings. Nothing here is flat-DSL-only syntax — it's just methods
> you no longer have to import.

`Ctrl+C` to detach again — one more change to make.

## Step 4 — Track state across probes with `@TLS`

Here's something a single `@Duration` can't give you: how many orders *each* of the app's three
worker threads has handled, and how long an order actually takes end to end — from
`validateOrder`'s first line to `chargeCard`'s return — even though those are two different
methods with two different `@OnMethod` hooks. That needs state that survives between separate
probe invocations, kept separate per thread. That's what `@TLS` (thread-local storage) is for:
a field marked `@TLS` behaves like a `ThreadLocal`, transparently, with each thread seeing its own
copy.

Replace the script one more time with the full version already sitting in
[demo/OrderTiming.java](demo/OrderTiming.java):

```java
@BTrace
public class OrderTiming {

    // One copy of each field per thread - order counts and timers never mix between
    // order-worker-0, order-worker-1, and order-worker-2.
    @TLS
    private static long orderStart;

    @TLS
    private static long ordersOnThisThread;

    // First thing that happens for every order: start the clock and count the attempt
    // (even orders that fail validation a few lines later still count as "attempted").
    @OnMethod(clazz = "OrderService", method = "validateOrder")
    public static void onOrderStart() {
        orderStart = monotonic();
        ordersOnThisThread = ordersOnThisThread + 1;
    }

    // Last thing that happens for a *successful* order: chargeCard returning means
    // validateOrder and lookupInventory already completed, in order, on this same thread.
    @OnMethod(clazz = "OrderService", method = "chargeCard", location = @Location(Kind.RETURN))
    public static void onOrderDone() {
        long totalMs = (monotonic() - orderStart) / 1_000_000L;
        String who = concat("worker=", threadName(currentThread()));
        String line = who + " order #" + str(ordersOnThisThread)
                + " total=" + str(totalMs) + "ms at " + str(timestamp());
        println(line);
    }
}
```

Deploy it exactly as before:

```sh
btrace <PID> OrderTiming.java
```

**You should see** each of the three workers reporting its own running count and its own timing
(numbers, thread names, and the exact interleaving will differ):

```
worker=order-worker-1 order #1 total=43ms at 1783778034521
worker=order-worker-0 order #1 total=57ms at 1783778034534
worker=order-worker-2 order #1 total=72ms at 1783778034549
worker=order-worker-0 order #2 total=53ms at 1783778034593
worker=order-worker-1 order #2 total=80ms at 1783778034620
worker=order-worker-2 order #3 total=414ms at 1783778034964
worker=order-worker-2 order #5 total=72ms at 1783778035036
```

Notice `order-worker-2` jumps from `#1` to `#3`, and later from `#3` to `#5` — that's not a bug.
`ordersOnThisThread` is incremented the moment `validateOrder` starts, so a failed validation
(roughly 1 order in 12, per Tutorial 1) still counts as an attempt; you just never see a line for
it, because `chargeCard` — and the probe on it — never runs for that order. You also just caught
the same slow-payment defect from Tutorial 1 again, this time with a number that includes
`validateOrder` and `lookupInventory` too: `total=414ms`.

> **What just happened?** `@TLS` fields are static, but each thread reads and writes its own
> independent copy — the annotation transparently rewrites field access to go through thread-local
> storage. That's what let `onOrderStart` (hooked on `validateOrder`) hand data to `onOrderDone`
> (hooked on a completely different method, `chargeCard`) without the two probes racing on a
> shared field across `order-worker-0/1/2`. One rule to know: you can only read or write a `@TLS`
> field from inside another `@OnMethod`-annotated handler — not from `@OnTimer`, `@OnEvent`, or
> similar global callbacks.

## Step 5 — Clean up

`Ctrl+C` in the BTrace terminal detaches the client and disables the probe immediately — no
restart needed, and you can redeploy as many times as you like while you iterate on a script.
Stop the demo app with `Ctrl+C` in its terminal when you're done for good.

## Troubleshooting

- **`BTrace compilation failed`** — the client prints `javac`-style errors to stderr above this
  line; the usual cause is a typo in the `@OnMethod` clause or a public class name that doesn't
  match the file name (`OrderTiming.java` must contain `public class OrderTiming`).
- **`File not found: OrderTiming.java`** — run `btrace` from the same directory as the script, or
  pass a path: `btrace <PID> demo/OrderTiming.java`.
- **The dump never appears** — `-Dbtrace.oneliner.dump=true` has to reach the JVM running the
  `btrace` client itself, not the target JVM. If you don't see the `Picked up JAVA_TOOL_OPTIONS`
  line, the property never made it in; appending `-Dbtrace.oneliner.dump=true` directly to the
  `btrace ...` command line does **not** work, because the launcher passes its arguments straight
  through to the client's argument parser, not to the JVM's own options.
- **A `@TLS` field is always `0`** — make sure you're reading it from an `@OnMethod` handler; per
  its javadoc, `@TLS` state isn't visible from any other kind of handler.

## Go deeper

- The full flat DSL method list, `@TLS` details, and the INVOKEDYNAMIC rewrite under the hood:
  [BTrace Tutorial, Lessons 7–8](../BTraceTutorial.md)
- Every `@OnMethod` location kind, annotation, and common pattern: [Quick Reference](../QuickReference.md)
- Full oneliner syntax this tutorial started from: [Oneliner Guide](../OnelinerGuide.md)

---