README Copy Section

BTrace Documentation

Welcome to the BTrace documentation! BTrace is a safe, dynamic tracing tool for the Java platform that allows you to instrument running applications without stopping them.

Quick Start Copy Chapter

New to BTrace? Start here → Getting Started Guide

Get up and running in 5 minutes with installation, your first script, and common usage patterns.

Documentation Map Copy Chapter

| Document | Description | Target Audience |
|----------|-------------|-----------------|
| Getting Started | Installation, first script, deployment modes, common pitfalls | New users, quick start |
| Oneliner Guide | DTrace-style oneliners for quick debugging without scripts | Quick debugging, ops/SRE |
| Quick Reference | Annotations, patterns, CLI commands, built-in functions | Experienced users, quick lookup |
| BTrace Tutorial | Comprehensive lessons covering all features | All users, in-depth learning |
| Troubleshooting Guide | Common errors, debugging, performance, compatibility | Problem-solving, debugging |
| FAQ | Common questions, best practices, comparisons | All users, decision-making |
| MCP Server | Using BTrace from AI clients via the Model Context Protocol | AI-assisted debugging, LLM integrations |
| Migration Guide 2.x → 3.0 | Upgrading from BTrace 2.x: package rename, Java version policy, breaking changes | Existing 2.x users |
| Extension Development Guide | Writing, packaging, and publishing BTrace extensions | Extension authors, platform maintainers |
| 3.0 Release Checklist | JDK, protocol, security, packaging, and RC go/no-go gates | Release maintainers |

Learning Paths Copy Chapter

I'm New to BTrace Copy Section

Tip: Want latency histograms fast? See Quick Start: Histogram Metrics Extension and the tutorial section Using the Histogram Metrics Extension.

I Need to Solve a Problem Copy Section

I Need a Quick Lookup Copy Section

I Want Advanced Features Copy Section

I'm Working on AI/LLM Applications Copy Section

Documentation by Topic Copy Chapter

Core Features Copy Section

Advanced Features Copy Section

AI/LLM Observability Copy Section

Deployment & Operations Copy Section

Problem Solving Copy Section

Integration Copy Section

Architecture Copy Section

Hands-On Tutorials Copy Section

- Hands-on tutorial series — scenario-driven labs with a runnable demo app; start with Your First Trace in 2 Minutes

Sample Scripts Copy Chapter

BTrace includes 50+ sample scripts demonstrating real-world use cases:
- Location: btrace-dist/src/main/resources/samples/
- Examples: File I/O tracking, JDBC monitoring, HTTP requests, memory allocation, thread analysis
- Browse: BTrace Samples Directory

External Resources Copy Chapter

Official Resources Copy Section

Community Copy Section

Tools & Integrations Copy Section

Contributing Copy Chapter

BTrace is an open-source project welcoming contributions. To contribute:

1. Sign the Oracle Contributor Agreement
2. Read the Contributing Guidelines
3. Fork the repository and create a pull request
4. See Build Instructions for development setup

Version Information Copy Chapter

- Current Version: Check GitHub Releases
- Java Compatibility: BTrace 3.0 runs on Java 8–25+. Running BTrace against a JVM older than Java 17 is deprecated: it continues to work throughout 3.x but emits a deprecation warning. Support for Java < 17 will be removed in the next major release (4.0). See the migration guide.
- License: Apache License 2.0

Documentation Feedback Copy Chapter

Found an issue with the documentation? Please:
- Report it on GitHub Issues
- Tag it with documentation label
- Or submit a pull request with improvements

---

Ready to get started? → Getting Started Guide

---

Architecture/Agent Manifest Libs Copy Section

Agent Manifest Library Attributes

The BTrace agent JAR manifest can declare extra JARs to place on the bootstrap and
system class loaders, so a repackaged or fat agent can carry its dependencies without
extra command-line arguments. These attributes are read from the agent JAR's
META-INF/MANIFEST.MF at startup.

Attributes Copy Chapter

| Manifest attribute | Purpose |
|---|---|
| BTrace-Boot-Libs | Space-separated list of JAR paths appended to the bootstrap class loader search. |
| BTrace-System-Libs | Space-separated list of JAR paths appended to the system class loader search. |
| BTrace-Libs-Root | Optional base directory for resolving relative entries. Defaults to the agent JAR's parent directory. |
| BTrace-Libs-Profile | Optional named subdirectory to auto-scan for boot/.jar and system/.jar. Mirrors the libs agent argument. |

The standard Boot-Class-Path manifest attribute continues to be honored as well.

Entries use Class-Path-style space separation; each entry may be absolute or relative
to BTrace-Libs-Root (or the agent JAR's parent). Unusable entries are logged and skipped
rather than aborting startup.

See also Copy Chapter

- Masked JAR architecture
- Migrating from libs/profiles to Extensions

---

Architecture/ExtensionConfiguration Copy Section

BTrace Extension Configuration

Overview Copy Chapter

Extension configuration allows users to customize extension loading behavior, enable/disable specific extensions, and pass extension-specific settings.

Configuration File Locations Copy Chapter

Configuration files are searched in priority order (highest to lowest):

1. Command-line override: -Dbtrace.extensions.config=/path/to/config
2. User config: ~/.btrace/extensions.conf
3. System config: $BTRACE_HOME/conf/extensions.conf

The first file found is used. Settings from multiple files are NOT merged.

Configuration Format Copy Chapter

Uses standard Java properties format:

Extension Control

Comma-separated list of extension IDs to enable (empty = all enabled)

extensions.enabled=
Comma-separated list of extension IDs to disable

extensions.disabled=
Auto-load extensions on demand (default: true)

extensions.autoload=true
Extension-specific settings (passed to extension on load)

Format: <extension-id>.<setting-name>=<value>

btrace-metrics.histogram.default-precision=3
btrace-metrics.stats.window-size=1000

Configuration Properties Copy Chapter

Global Settings Copy Section

extensions.enabled
- Format: comma-separated extension IDs
- Default: (empty - all enabled)
- Example: extensions.enabled=btrace-metrics,btrace-custom
- Description: Whitelist of extensions to load. If set, only listed extensions are loaded. If empty or not set, all discovered extensions are enabled (unless explicitly disabled).

extensions.disabled
- Format: comma-separated extension IDs
- Default: (empty - none disabled)
- Example: extensions.disabled=btrace-old-metrics
- Description: Blacklist of extensions to prevent from loading. Takes precedence over extensions.enabled.

extensions.autoload
- Format: boolean (true/false)
- Default: true
- Description: If true, extensions are loaded on demand when scripts use their services. If false, all enabled extensions are loaded at agent startup.

Extension-Specific Settings Copy Section

Extensions can define custom configuration properties using the pattern:

<extension-id>.<setting-name>=<value>

Examples:

BTrace Metrics settings

btrace-metrics.histogram.default-precision=3
btrace-metrics.histogram.max-value=3600000000
btrace-metrics.stats.window-size=1000
Custom extension settings

my-extension.feature.enabled=true
my-extension.endpoint=http://localhost:8080

Extension-specific settings are passed to the extension when loaded. Extensions can access these via their configuration API.

Usage Examples Copy Chapter

Example 1: Disable Specific Extension Copy Section

Disable legacy metrics extension

extensions.disabled=btrace-old-metrics

Example 2: Whitelist Extensions Copy Section

Only load metrics and custom extensions

extensions.enabled=btrace-metrics,my-custom-extension

Example 3: Configure Extension Settings Copy Section

Configure metrics extension

btrace-metrics.histogram.default-precision=3
btrace-metrics.histogram.max-value=3600000000

Example 4: Load All Extensions at Startup Copy Section

Disable lazy loading, load all extensions immediately

extensions.autoload=false

API Integration Copy Chapter

ExtensionConfig Class Copy Section

New class to manage extension configuration:

public class ExtensionConfig {
  public boolean isEnabled(String extensionId);
  public boolean isAutoLoad();
  public Properties getExtensionProperties(String extensionId);
}

ExtensionLoader Integration Copy Section

ExtensionLoader will:
1. Load configuration on initialization
2. Filter discovered extensions based on enabled/disabled lists
3. Apply autoload setting (lazy vs eager loading)
4. Pass extension-specific properties when loading extensions

Configuration Loading Priority Copy Section

Command-line (-Dbtrace.extensions.config)
    ↓
User config (~/.btrace/extensions.conf)
    ↓
System config ($BTRACE_HOME/conf/extensions.conf)
    ↓
Built-in defaults (all enabled, autoload=true)

Backward Compatibility Copy Chapter

- If no configuration file exists, all extensions are enabled with autoload
- Existing deployments continue to work without any configuration
- Configuration is purely additive (opt-in customization)

Future Enhancements Copy Chapter

- Hot reload: Detect configuration file changes and reload extensions
- JMX integration: Expose extension configuration via JMX for runtime modification
- Validation: Warn about unknown extension IDs or malformed settings

---

Architecture/ExtensionInvokeDynamicBridge Copy Section

Extension invokedynamic Bridge Architecture

Overview Copy Chapter

The BTrace extension system uses invokedynamic to bridge between script classloaders and extension classloaders, avoiding bootstrap classloader pollution while maintaining complete extension isolation.

Key Benefits:
- Clean bootstrap namespace - Only BTrace core classes in bootstrap, no extension JARs
- Extension isolation - Each extension in its own classloader with shaded dependencies
- Security - Extensions don't have bootstrap-level privileges
- Java 8+ compatible - Works on all supported Java versions (8+)

Architecture Diagram Copy Chapter

/ Detailed source-code truncated for AI context efficiency. /

Component Details Copy Chapter

1. ExtensionIndy (btrace-runtime) Copy Section

Location: btrace-runtime/src/main/java/io/btrace/runtime/ExtensionIndy.java

Purpose: Bootstrap methods for invokedynamic extension access

Key Methods:

public static CallSite bootstrapFieldGet(
    MethodHandles.Lookup caller,
    String fieldName,
    MethodType type,
    String serviceClassName,
    String serviceType,
    String factoryMethod) throws Exception

Responsibilities:
- Receives invokedynamic bootstrap calls from script bytecode
- Uses ExtensionBridge to load extension service class
- Creates MethodHandle for service instantiation
- Handles both SIMPLE (no-arg) and RUNTIME (BTraceRuntime.Impl param) services
- Supports both constructors and factory methods
- Returns ConstantCallSite for zero-overhead subsequent calls

Error Handling:
- Gracefully degrades to null on failure
- Logs errors but doesn't throw exceptions
- Matches behavior of failed service instantiation

2. ExtensionBridge (btrace-core) Copy Section

Location: btrace-core/src/main/java/io/btrace/extension/ExtensionBridge.java

Purpose: Interface for agent-to-runtime classloader bridging

API:

public interface ExtensionBridge {
  Class<?> getExtensionClass(String serviceClassName) throws Exception;
}

Why an Interface:
- Allows btrace-runtime (bootstrap) to call btrace-agent code
- Breaks circular dependency (runtime can't depend on agent)
- Clean separation of concerns

3. ExtensionBridgeImpl (btrace-core) Copy Section

Location: btrace-core/src/main/java/io/btrace/extension/impl/ExtensionBridgeImpl.java

Purpose: Agent-side implementation of ExtensionBridge

Initialization:

static {
  try {
    Class<?> indyClz = Class.forName("io.btrace.runtime.ExtensionIndy");
    ExtensionBridge bridge = new ExtensionBridgeImpl(Main.getExtensionLoader());
    indyClz.getField("bridge").set(null, bridge);
  } catch (ClassNotFoundException e) {
    // Expected for pre-Java 8 or if ExtensionIndy unavailable
  }
}

Responsibilities:
- Sets ExtensionIndy.bridge field during static initialization
- Uses ExtensionLoader to find extensions providing requested services
- Loads extension classes from extension classloaders
- Ensures extensions are loaded before returning class

4. Preprocessor (btrace-agent) Copy Section

Location: btrace-agent/src/main/java/io/btrace/instr/Preprocessor.java

Purpose: Transforms @Injected field references to invokedynamic

Transformation:

// Before:
GETSTATIC ScriptClass.metrics : MetricsService
// After:
INVOKEDYNAMIC bootstrapFieldGet(
  "metrics",                           // field name (debug)
  "()Lio/btrace/metrics/MetricsService;",  // method descriptor
  ExtensionIndy.bootstrapFieldGet,     // bootstrap method
  "io.btrace.metrics.MetricsService",  // service class name
  "RUNTIME",                           // optional type hint (deprecated)
  ""                                   // factory method (empty = constructor)
) : MetricsService

Modified Method:

private AbstractInsnNode updateInjectedUsage(
    ClassNode cn, FieldInsnNode fin, InsnList l, LocalVarGenerator lvg)

Changes:
- Removed local variable caching
- Removed service instantiation bytecode
- Emits InvokeDynamicInsnNode instead
- Passes service metadata as bootstrap arguments

5. ExtensionLoader (btrace-core) Copy Section

Location: btrace-core/src/main/java/io/btrace/extension/ExtensionLoader.java

Key Changes:
- Avoids instrumentation.appendToBootstrapClassLoaderSearch(jarFile)
- Avoids an eager extension loading loop
- Behavior: Extensions loaded on-demand via invokedynamic

Impact:
- Bootstrap classloader remains clean
- Extensions only loaded when scripts use them
- Better resource utilization

Execution Flow Copy Chapter

Initialization (Agent Startup) Copy Section

1. Main.agentmain/premain()
   ↓
2. ExtensionLoader.discoverExtensions()
   - Scans extensions/ directory
   - Creates ExtensionDescriptor for each extension
   - Stores in availableExtensions map
   - Does NOT load extensions yet
   ↓
3. ExtensionBridgeImpl.<clinit>() (static initializer)
   - Creates new ExtensionBridgeImpl instance
   - Sets ExtensionIndy.bridge = instance
   - Bridge now ready for invokedynamic bootstrap calls
   ↓
4. Scripts can now be loaded and compiled

Script Loading (First Access) Copy Section

1. Script bytecode contains:
   INVOKEDYNAMIC bootstrapFieldGet(...) : MetricsService
   ↓
2. JVM calls ExtensionIndy.bootstrapFieldGet()
   (first time for this callsite)
   ↓
3. ExtensionIndy.bootstrapFieldGet():
   a. Call bridge.getExtensionClass("io.btrace.metrics.MetricsService")
   ↓
4. ExtensionBridgeImpl.getExtensionClass():
   a. Call loader.findExtensionForService(serviceClassName)
   b. If ext.isLoaded() == false:
      - loader.load(ext)  // Loads extension JAR into ExtensionClassLoader
   c. extClassLoader.loadClass(serviceClassName)
   d. Return Class<?> to bootstrap method
   ↓
5. ExtensionIndy.bootstrapFieldGet() (continued):
   a. Create MethodHandle for service instantiation:
      - Auto-detect constructor or factory
      - Initialize extensions via Extension.initialize(ExtensionContext) when first accessed
   b. Return new ConstantCallSite(methodHandle)
   ↓
6. JVM caches ConstantCallSite
   ↓
7. Subsequent calls to same INVOKEDYNAMIC:
   - Use cached CallSite (zero overhead)
   - MethodHandle directly instantiates service
   - No bootstrap method re-execution

Subsequent Field Access (Same Script) Copy Section

1. Script executes: metrics.histogram("foo")
   ↓
2. INVOKEDYNAMIC instruction
   ↓
3. JVM uses cached CallSite (no bootstrap)
   ↓
4. MethodHandle executes:
   - Instantiates MetricsService
   - Returns service instance
   ↓
5. INVOKEVIRTUAL MetricsService.histogram(String)
   - Normal virtual dispatch
   - JVM resolves from actual object class
   - Extension method executes

Performance Characteristics Copy Chapter

First Access Copy Section

Overhead:
- Bootstrap method execution: ~1-5ms
- Extension class loading (if not already loaded)
- MethodHandle creation
- CallSite creation

One-time cost per @Injected field per script

Subsequent Access Copy Section

Overhead: Zero (compared to direct method call)
- JVM optimizes ConstantCallSite to direct call
- Identical performance to pre-invokedynamic approach
- JIT compiler inlines if possible

Memory Copy Section

Before (Bootstrap Classpath):
- All extension JARs in bootstrap: ~5-10MB per extension
- Permanent heap allocation
- Visible to all scripts and JVM internals

After (invokedynamic Bridge):
- Only loaded extensions in memory
- Extension classloaders eligible for GC when scripts unload
- Clean namespace separation

Extension Types and Instantiation Copy Chapter

Extension Instances (No-arg) Copy Section

Preferred pattern: extensions expose a public no-arg constructor and receive
their runtime context via Extension.initialize(ExtensionContext).

Constructor Example:

public class MyExtension extends Extension {
  public MyExtension() {
    // No-arg constructor
  }
}

MethodHandle Creation:

MethodHandle constructor =
  MethodHandles.publicLookup().findConstructor(
    serviceClass,
    MethodType.methodType(void.class));

Factory methods are also supported but less common in the new model.

MethodHandle Creation:

MethodHandle factory =
  MethodHandles.publicLookup().findStatic(
    serviceClass,
    "getInstance",
    MethodType.methodType(serviceClass));

Runtime-Aware Initialization Copy Section

If an extension needs access to runtime context, override initialize(ExtensionContext)
in your subclass of Extension. The bridge calls it after construction.

Initialization Example:

public class MyExtension extends Extension {
  @Override
  public void initialize(ExtensionContext ctx) {
    super.initialize(ctx);
    // Use ctx as needed
  }
}

MethodHandle Creation:

Class<?> runtimeImplClass =
  Class.forName("io.btrace.runtime.BTraceRuntime$Impl");
MethodHandle constructor =
  MethodHandles.publicLookup().findConstructor(
    serviceClass,
    MethodType.methodType(void.class, runtimeImplClass));
// Bind BTraceRuntime.enter() as first argument
MethodHandle getRuntimeImpl =
  MethodHandles.publicLookup().findStatic(
    BTraceRuntime.class,
    "enter",
    MethodType.methodType(runtimeImplClass));
mh = MethodHandles.filterReturnValue(getRuntimeImpl, constructor);

Classloader Hierarchy Copy Chapter

Script Classloader Copy Section

// In BTraceRuntimeImpl_8.defineClass()
ClassLoader loader = new ClassLoader(null) {};  // parent = null
Class<?> cl = unsafe.defineClass(name, code, 0, code.length, loader, null);

Parent: null (delegates directly to bootstrap)
Visibility: Only bootstrap classes
Impact: Scripts can only see classes in bootstrap classpath

Extension Classloader Copy Section

// In ExtensionLoader.load()
ExtensionClassLoader classLoader = new ExtensionClassLoader(
  extensionId,
  extensionVersion,
  new URL[] {jarUrl},
  parentClassLoader  // BTrace boot classloader
);

Parent: BTrace boot classloader (btrace-boot.jar)
Visibility: Extension JAR + BTrace core APIs
Isolation: Each extension in separate classloader

Shaded Dependencies:
- Extensions use shadow plugin to shade dependencies
- Prevents version conflicts between extensions
- Example: btrace-metrics shades HdrHistogram

Classloader Delegation Copy Section

Script Class
├── parent = null → Bootstrap ClassLoader
│   ├── JRE system classes
│   ├── btrace-boot.jar (BTraceRuntime, annotations)
│   └── ExtensionIndy.class
│
ExtensionClassLoader (per extension)
├── parent = BTrace Boot ClassLoader
│   └── btrace-boot.jar classes
├── Extension JAR classes
└── Shaded dependencies

invokedynamic bridges the gap:
- Script sees only bootstrap
- Bootstrap method accesses extension via bridge
- MethodHandle allows cross-classloader calls

Java Version Compatibility Copy Chapter

Java 8+ Copy Section

invokedynamic features used:
- INVOKEDYNAMIC instruction (Java 7+)
- MethodHandles (Java 7+)
- CallSite / ConstantCallSite (Java 7+)
- Handle (Java 7+)

NOT used:
- Hidden classes (Lookup.defineHiddenClass) - Java 15+ only
- VarHandles - Java 9+ only

Result: Works on all BTrace-supported Java versions (8+)

Comparison with Indy.java Copy Section

Indy.java (probe handlers):
- Uses hidden classes (Java 15+ only)
- Creates dynamic hidden classes at link time
- Complex nested class generation

ExtensionIndy.java (extensions):
- Uses standard classloading (Java 8+)
- Loads classes from extension classloaders
- Simple MethodHandle creation

Error Handling and Debugging Copy Chapter

Bootstrap Method Failures Copy Section

Causes:
- Extension not found
- Extension class loading failure
- Constructor/factory method not found
- Runtime instantiation error

Behavior:

try {
  // Load class and create MethodHandle
} catch (Throwable t) {
  // Graceful degradation: return null
  mh = MethodHandles.constant(type.returnType(), null);
}
return new ConstantCallSite(mh);

Impact:
- Script continues execution
- Service field is null
- NPE on first usage (matches current behavior)

Debugging Copy Section

Enable logging:

-Dorg.slf4j.simpleLogger.log.io.btrace.agent.extension=DEBUG
-Dorg.slf4j.simpleLogger.log.io.btrace.runtime.ExtensionIndy=DEBUG

Logs show:
- Extension discovery
- Extension loading
- Service class resolution
- MethodHandle creation

Bytecode inspection:

javap -v ScriptClass.class | grep -A 5 INVOKEDYNAMIC

Shows:
- Bootstrap method reference
- Bootstrap arguments
- Method descriptor

Migration from Bootstrap Classpath Approach Copy Chapter

Before Copy Section

ExtensionLoader.load():

if (instrumentation != null) {
  JarFile jarFile = new JarFile(descriptor.getJarPath().toFile());
  instrumentation.appendToBootstrapClassLoaderSearch(jarFile);
}

Preprocessor.updateInjectedUsage():

// Create local variable
int varIdx = lvg.newVar(implType);
// Emit instantiation bytecode
toInsert.add(new TypeInsnNode(Opcodes.NEW, implType.getInternalName()));
toInsert.add(new InsnNode(Opcodes.DUP));
toInsert.add(new MethodInsnNode(Opcodes.INVOKESPECIAL, ...));
// Store in local variable
toInsert.add(new VarInsnNode(Opcodes.ASTORE, varIdx));

After Copy Section

ExtensionLoader.load():

// Extension classes accessed via invokedynamic bridge
// (no bootstrap classpath pollution)

Preprocessor.updateInjectedUsage():

// Emit invokedynamic instruction
InvokeDynamicInsnNode indyInsn = new InvokeDynamicInsnNode(
  fieldName,
  methodDescriptor,
  bootstrapHandle,
  serviceClassName,
  serviceType,
  factoryMethod);

Compatibility Copy Section

Existing scripts: No changes required
- Same @Injected annotation
- Same field types
- Same usage patterns

Existing extensions: No changes required
- Same service classes
- Same constructors/factories
- Same MANIFEST.MF metadata

Behavioral changes: None
- Extension loading now on-demand
- Performance characteristics unchanged
- Error handling identical

Security Implications Copy Chapter

Bootstrap Classloader Access Copy Section

Before:
- Extensions had bootstrap-level privileges
- Could access JVM internals
- Could interfere with system classes

After:
- Extensions in isolated classloaders
- Limited to parent visibility (BTrace boot)
- Cannot access bootstrap internals

MethodHandle Security Copy Section

MethodHandles.publicLookup():
- Only creates handles for public members
- Respects Java access control
- Cannot access private/protected methods

Lookup Context:
- Bootstrap method has caller's lookup
- Could create privileged handles
- Current implementation uses publicLookup() for safety

Testing Copy Chapter

Unit Tests Copy Section

ExtensionIndy:
- Test bootstrap method with mock bridge
- Test SIMPLE vs RUNTIME services
- Test constructor vs factory instantiation
- Test error handling

ExtensionBridgeImpl:
- Test extension finding
- Test class loading
- Test extension loading on demand

Integration Tests Copy Section

MetricsTest:
- Uses @Injected MetricsService
- Verifies invokedynamic bootstrap execution
- Verifies service instantiation
- Verifies method calls work

Verification:

Check bytecode contains INVOKEDYNAMIC

javap -v integration-tests/build/classes/java/test/MetricsTest.class | grep INVOKEDYNAMIC
Check extension NOT in bootstrap

jcmd <pid> VM.class_hierarchy | grep -v btrace-metrics

Future Enhancements Copy Chapter

1. Extension API/Impl Split Copy Section

Concept:
- btrace-metrics-api.jar → bootstrap (interfaces only)
- btrace-metrics-impl.jar → extension CL (implementations)

Benefits:
- Cleaner verification
- Smaller bootstrap JAR
- Better security boundaries

2. Service Caching Copy Section

Current: Each invokedynamic creates new instance
Enhancement: Cache service instances per script

Implementation:

// In ExtensionIndy
private static final Map<String, Object> instanceCache = new ConcurrentHashMap<>();
// In bootstrapFieldGet()
String cacheKey = caller.lookupClass().getName() + ":" + serviceClassName;
Object cached = instanceCache.get(cacheKey);
if (cached != null) {
  return new ConstantCallSite(MethodHandles.constant(type.returnType(), cached));
}

3. Hot Reload Support Copy Section

Challenge: Extension updates require agent restart
Enhancement: Use MutableCallSite instead of ConstantCallSite

Implementation:

public static CallSite bootstrapFieldGet(...) {
  MethodHandle mh = createServiceHandle(...);
  return new MutableCallSite(mh);  // Allows invalidation
}
// On extension reload:
public static void invalidateExtension(String extensionId) {
  // Find all MutableCallSites for this extension
  // Call MutableCallSite.setTarget() with new MethodHandle
}

Summary Copy Chapter

The invokedynamic extension bridge provides:

✅ Clean bootstrap namespace - Only BTrace core in bootstrap
✅ Extension isolation - Each extension in own classloader
✅ Java 8+ compatible - No Java 15+ features required
✅ Zero performance overhead - After first call
✅ Backward compatible - No script changes required
✅ Better security - Extensions lack bootstrap privileges
✅ On-demand loading - Extensions loaded when needed

This architecture enables scalable, secure, and performant extension loading while maintaining full compatibility with existing BTrace scripts and extensions.

---

Architecture/ExtensionManifestFormat Copy Section

BTrace Extension MANIFEST.MF Format

Overview Copy Chapter

This document defines the MANIFEST.MF attribute format for BTrace extensions, replacing the previous btrace-extension.properties file.

Rationale Copy Chapter

Using MANIFEST.MF for extension metadata provides:
- Standard Java mechanism - Already parsed by JVM and tools
- OSGi precedent - Follows established patterns from OSGi bundles
- Single source of truth - Build tools already generate MANIFEST.MF
- Less maintenance - No separate properties file to manage

Attribute Mapping Copy Chapter

From Properties Format Copy Section

extension.id=btrace-metrics
extension.version=3.0.0
extension.name=BTrace Metrics
extension.description=High-performance metrics...
btrace.api.version=3.0+
java.version=8+
services=io.btrace.metrics.MetricsService
requires.extensions=btrace-core
shaded.packages=org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram

To MANIFEST.MF Format Copy Section

BTrace-Extension-Id: btrace-metrics
BTrace-Extension-Version: 3.0.0
BTrace-Extension-Name: BTrace Metrics
BTrace-Extension-Description: High-performance metrics with HdrHistogram
 for percentiles and lock-free statistics
BTrace-API-Version: 3.0+
BTrace-Java-Version: 8+
BTrace-Extension-Services: io.btrace.metrics.MetricsService
BTrace-Extension-Requires: btrace-core
BTrace-Shaded-Packages: org.HdrHistogram->io.btrace.metrics.sh
 aded.hdrhistogram,com.clearspring.analytics->io.btrace.metrics
 .shaded.clearspring
BTrace-Extension-Permissions: NETWORK,THREADS

Attribute Definitions Copy Chapter

Required Attributes Copy Section

BTrace-Extension-Id
- Format: lowercase-with-hyphens
- Example: btrace-metrics
- Description: Unique identifier for the extension

BTrace-Extension-Version
- Format: semantic version (major.minor.patch[-qualifier])
- Example: 3.0.0, 3.0.0-SNAPSHOT
- Description: Extension version for conflict resolution

Optional Attributes Copy Section

BTrace-Extension-Name
- Format: human-readable string
- Example: BTrace Metrics
- Description: Display name for the extension

BTrace-Extension-Description
- Format: multi-line text (continuation with leading space)
- Example: High-performance metrics with HdrHistogram
- Description: Detailed description of extension functionality

BTrace-API-Version
- Format: version range (major.minor+)
- Example: 3.0+ (requires BTrace API 3.0 or higher)
- Description: Required BTrace API version

BTrace-Java-Version
- Format: version number (8+, 11+, etc.)
- Example: 8+
- Description: Minimum Java version required

BTrace-Extension-Services
- Format: comma-separated fully qualified class names
- Example: io.btrace.metrics.MetricsService,io.btrace.metrics.StatsService
- Description: Service classes provided by this extension

BTrace-Extension-Requires
- Format: comma-separated extension IDs
- Example: btrace-core,btrace-util
- Description: Other extensions required by this extension

BTrace-Shaded-Packages
- Format: comma-separated package mappings (original->shaded)
- Example: org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram
- Description: Package relocation mappings for shaded dependencies

MANIFEST.MF Line Continuation Copy Chapter

Per JAR specification, manifest attributes longer than 72 bytes must be continued on the next line with a leading space:

BTrace-Extension-Description: This is a very long description that excee
 ds the 72-byte limit and must be continued on the next line with a lead
 ing space character.

Embedded Extension Properties (extension.properties) extension.properties)')" title="Copy chapter prompt for LLMs"> Copy Chapter

Embedded extensions (shipped inside a fat agent JAR) are described by a
META-INF/btrace-extensions/{id}/extension.properties file rather than a
MANIFEST.MF. The agent reads the following keys from that file:

| Key | Required | Example | Description |
|-----|----------|---------|-------------|
| id | no (defaults to directory name) | btrace-spark | Extension identifier |
| version | no (defaults to 0.0.0) | 1.2.0 | Semantic version |
| name | no | BTrace Spark | Human-readable name |
| description | no | Spark job tracing | Short description |
| btrace.api.version | no (defaults to 3.0+) | 3.0.0 | Minimum BTrace API version |
| java.version | no (defaults to 8+) | 11+ | Minimum Java version |
| services | no | org.example.SparkService | Comma-separated service class names |
| probes | no | SparkJobTracer,SparkStageTracer | Comma-separated bundled probe class names |
| configurator | no | org.example.SparkConfigurator | Fully qualified ExtensionConfigurator class for zero-config probe auto-selection (see below) |

configurator — Zero-Config Probe Auto-Selection configurator — Zero-Config Probe Auto-Selection')" title="Copy section prompt for LLMs"> Copy Section

When a configurator class is declared, the agent calls it during startup to
decide which bundled probes to activate based on the running JVM's environment.
This allows the extension to enable the right probes automatically (e.g. Spark
driver probes vs. executor probes) without the operator having to pass a
probes= agent argument.

The class must implement io.btrace.core.extensions.ExtensionConfigurator
and have a public no-arg constructor. It is loaded via the extension's own
classloader. See BTraceExtensionDevelopmentGuide.md
for a full example.

Backward Compatibility Copy Chapter

The extension loader supports both formats:

1. Check for MANIFEST.MF attributes first
2. Fall back to btrace-extension.properties if MANIFEST attributes not found
3. Log deprecation warning if using properties file

Gradle Configuration Copy Chapter

Extensions should configure MANIFEST.MF generation in build.gradle:

jar {
  manifest {
    attributes(
      'BTrace-Extension-Id': 'btrace-metrics',
      'BTrace-Extension-Version': project.version,
      'BTrace-Extension-Name': 'BTrace Metrics',
      'BTrace-Extension-Description': 'High-performance metrics...',
      'BTrace-API-Version': '3.0+',
      'BTrace-Java-Version': '8+',
      'BTrace-Extension-Services': 'io.btrace.metrics.MetricsService',
      'BTrace-Shaded-Packages': 'org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram'
    )
  }
}

Migration Path Copy Chapter

1. ✅ Design MANIFEST.MF format (this document)
2. Update ExtensionMetadata parser to read MANIFEST.MF
3. Update btrace-metrics build.gradle to generate MANIFEST attributes
4. Test with both formats for backward compatibility
5. Update documentation
6. Deprecate btrace-extension.properties (remove in future release)
BTrace-Extension-Permissions
- Format: comma-separated permission names
- Example: NETWORK,THREADS,FILE_WRITE
- Description: Permissions required by the extension implementation. Automatically inferred by the build plugin via code scanning, with optional overrides in Gradle.

---

Architecture/ExtensionStorageDesign Copy Section

BTrace Extension Storage Design

Overview Copy Chapter

This document describes the design for BTrace extension storage and loading mechanism. The goal is to move extensions from the boot JAR to a dedicated extension system with built-in and user-configurable locations.

Current State Copy Chapter

Currently, extensions (like btrace-metrics) are packaged into the boot JAR (btrace-boot.jar) along with core BTrace runtime classes. This approach has several limitations:

1. No Extension Isolation: Extensions are mixed with core runtime code
2. No Version Management: Can't have multiple versions of an extension
3. No Dynamic Loading: All extensions loaded at startup, even if unused
4. Deployment Friction: Adding new extensions requires rebuilding the distribution
5. User Extensions: No mechanism for users to add custom extensions

Design Goals Copy Chapter

1. Separation: Extensions stored separately from core runtime
2. Discoverability: Automatic discovery of built-in and user extensions
3. Configuration: User-configurable extension locations
4. Lazy Loading: Load extensions only when referenced in scripts
5. Isolation: Extension classloading isolation to avoid conflicts
6. Metadata: Extension metadata for versioning and dependencies

Directory Structure Copy Chapter

Distribution Layout Copy Section

BTRACE_HOME/
├── bin/
│   ├── btrace
│   └── btracec
├── libs/
│   ├── btrace-agent.jar
│   ├── btrace-boot.jar       # Core runtime only, no extensions
│   ├── btrace-client.jar
│   └── extensions/            # Built-in extensions directory
│       ├── btrace-metrics-3.0.0.jar
│       ├── btrace-statsd-3.0.0.jar
│       └── Readme.md
└── docs/
    └── ...

User Extension Locations Copy Section

Extensions are discovered in the following order (later locations override earlier):

1. Built-in: BTRACE_HOME/extensions/
2. System: /etc/btrace/extensions/ (Unix) or %PROGRAMDATA%\btrace\extensions\ (Windows)
3. User: ~/.btrace/extensions/
4. Environment: $BTRACE_EXT_PATH (colon-separated paths)
5. Command-line: --ext-path <path> (btrace command)
6. Script-local: ./.btrace/extensions/ (relative to script location)

Extension Structure Copy Chapter

Extension JAR Layout Copy Section

btrace-metrics-3.0.0.jar
├── META-INF/
│   ├── MANIFEST.MF
│   ├── btrace-extension.properties      # Extension metadata
│   └── services/
│       └── io.btrace.core.extensions.Extension
├── io/btrace/metrics/
│   ├── MetricsService.class
│   ├── histogram/
│   └── stats/
└── ... (shaded dependencies)

Extension Metadata (btrace-extension.properties) btrace-extension.properties)')" title="Copy section prompt for LLMs"> Copy Section

Extension identity

extension.id=btrace-metrics
extension.version=3.0.0
extension.name=BTrace Metrics
extension.description=High-performance metrics with HdrHistogram
API compatibility

btrace.api.version=3.0+
java.version=8+
Service providers (optional, can also use META-INF/services)

services=io.btrace.metrics.MetricsService
Dependencies on other extensions (optional)

requires.extensions=
Shadowed packages (for conflict detection)

shaded.packages=org.HdrHistogram->io.btrace.metrics.shaded.hdrhistogram,\
                com.clearspring.analytics->io.btrace.metrics.shaded.clearspring

Extension Loading Mechanism Copy Chapter

1. Discovery Phase (on Agent Startup) Copy Section

public class ExtensionLoader {
    private final List<ExtensionRepository> repositories;
    // Scan all configured locations
    public List<ExtensionDescriptor> discoverExtensions() {
        List<ExtensionDescriptor> extensions = new ArrayList<>();
        for (ExtensionRepository repo : repositories) {
            extensions.addAll(repo.scan());
        }
        // Resolve conflicts (latest version wins)
        return resolveExtensions(extensions);
    }
}

2. Lazy Loading (on Script Compilation) Copy Section

When the compiler encounters an @Injected service:

// In Compiler.java
private void loadRequiredExtensions(List<String> serviceTypes) {
    for (String serviceType : serviceTypes) {
        ExtensionDescriptor ext = extensionLoader.findExtensionForService(serviceType);
        if (ext != null && !ext.isLoaded()) {
            extensionLoader.load(ext);
        }
    }
}

3. ClassLoader Hierarchy Copy Section

Bootstrap ClassLoader
    |
System ClassLoader
    |
BTrace Boot ClassLoader (btrace-boot.jar)
    |
    +-- Extension ClassLoader 1 (btrace-metrics)
    |
    +-- Extension ClassLoader 2 (btrace-statsd)
    |
    +-- ...
    |
BTrace Script ClassLoader (compiled script)

Each extension gets its own classloader for isolation, but they can see:
- Bootstrap classes
- BTrace core API classes
- Their own classes and shaded dependencies

Configuration Copy Chapter

Agent Configuration (btrace.conf) btrace.conf)')" title="Copy section prompt for LLMs"> Copy Section

Extension directories (colon-separated on Unix, semicolon on Windows)

extension.path=${BTRACE_HOME}/extensions:${HOME}/.btrace/extensions
Extension loading behavior

extension.lazy-load=true
extension.fail-on-missing=false
extension.conflict-resolution=latest-version
Extension-specific settings

extension.btrace-metrics.enabled=true
extension.btrace-statsd.enabled=true

Environment Variables Copy Section

Override extension path

export BTRACE_EXT_PATH="/opt/btrace-extensions:/usr/local/btrace/ext"
Disable lazy loading (load all extensions at startup)

export BTRACE_EXT_LAZY_LOAD=false

Command-line Options Copy Section

Specify additional extension directories

btrace --ext-path /custom/extensions PID script.java
Disable built-in extensions

btrace --no-builtin-ext PID script.java
Enable specific extensions only

btrace --ext btrace-metrics,btrace-statsd PID script.java

Extension API Copy Chapter

Creating an Extension Copy Section

1. Implement Service Interface:

package com.example.myext;
import io.btrace.core.extensions.Extension;
public class MyExtensionService extends Extension {
    @Override
    public void onStart(BTraceRuntime runtime) {
        // Initialize extension
    }
}

2. Register Service:
No SPI file required. The Gradle plugin writes manifest attributes (BTrace-Extension-Services).

com.example.myext.MyExtensionService

3. Add Metadata:
Create META-INF/btrace-extension.properties:

extension.id=my-extension
extension.version=1.0.0
btrace.api.version=3.0+

4. Build JAR:

shadowJar {
    // Shade dependencies to avoid conflicts
    relocate 'com.external.lib', 'com.example.myext.shaded.lib'
}

5. Deploy:

cp my-extension-1.0.0.jar $BTRACE_HOME/extensions/
or

cp my-extension-1.0.0.jar ~/.btrace/ext/

Extension Repositories Copy Chapter

Local File System Repository Copy Section

public class FileSystemExtensionRepository implements ExtensionRepository {
    private final Path extensionDir;
    @Override
    public List<ExtensionDescriptor> scan() {
        List<ExtensionDescriptor> extensions = new ArrayList<>();
        try (DirectoryStream<Path> stream = Files.newDirectoryStream(extensionDir, "*.jar")) {
            for (Path jar : stream) {
                ExtensionDescriptor desc = parseExtension(jar);
                if (desc != null) {
                    extensions.add(desc);
                }
            }
        }
        return extensions;
    }
}

Future: Remote Repository (Maven-like) Copy Section

In btrace.conf

extension.repository.url=https://extensions.btrace.io/repository
extension.repository.cache=${HOME}/.btrace/cache

Migration Path Copy Chapter

Dual mode (backward compatible) Copy Section

Deprecation Copy Section

Removal Copy Section

Implementation Plan Copy Chapter

Stage 1: Extension Discovery (1-2 days) Copy Section

Stage 2: Extension Loading (2-3 days) Copy Section

Stage 3: Configuration (1-2 days) Copy Section

Stage 4: Integration (2-3 days) Copy Section

Stage 5: Migration (1-2 days) Copy Section

Stage 6: Testing & Documentation (2-3 days) Copy Section

Total Estimated Effort: 9-15 days

Security Considerations Copy Chapter

1. Extension Verification:
- Check JAR signatures
- Validate extension metadata
- Sandbox extension code (SecurityManager)

2. Trusted Locations:
- Built-in extensions (BTRACE_HOME) are trusted
- User extensions require explicit trust or signature

3. Classloading Isolation:
- Extensions cannot access each other's classes
- Extensions cannot override core BTrace classes

4. Resource Limits:
- Limit memory usage per extension
- Timeout for extension initialization

Performance Considerations Copy Chapter

1. Lazy Loading: Load extensions only when needed by scripts
2. Caching: Cache extension metadata to avoid repeated JAR scanning
3. Parallel Loading: Load independent extensions in parallel
4. Minimal Overhead: Discovery should add <100ms to agent startup

Backward Compatibility Copy Chapter

1. Boot JAR Fallback: If extension not found in ext/, check boot JAR
2. Service Discovery: Support both old and new service loading
3. Configuration: Old configurations continue to work
4. Scripts: Existing scripts work without modification

Open Questions Copy Chapter

1. Extension Dependencies: How do extensions depend on each other?
2. Version Conflicts: What happens when script needs different versions?
3. Extension Updates: Hot-reload extensions without restarting agent?
4. Distribution: Package extensions separately or all-in-one?
5. Remote Loading: Allow downloading extensions from remote repositories?

References Copy Chapter

- Java ServiceLoader Documentation
- OSGi Bundle Format
- Maven Extension Mechanism

---

Architecture/Fat Agent Plugin Copy Section

Fat Agent Plugin Architecture

Overview Copy Chapter

The Fat Agent Plugin (io.btrace.fat-agent) creates self-contained agent JARs with embedded extensions for single-JAR deployment scenarios. This eliminates the need for separate extension installation in environments like Spark, Hadoop, or Kubernetes where managing multiple JARs is impractical.

Problem Statement Copy Chapter

Standard BTrace deployment previously required:
1. Agent JAR (btrace-agent.jar)
2. Boot JAR (btrace-boot.jar)
3. Extension JARs in $BTRACE_HOME/extensions/

Note: The masked JAR architecture now consolidates agent and boot into a single btrace.jar. The multi-JAR layout above is the legacy approach.

This multi-JAR setup is problematic for:
- Spark/Hadoop: Driver and executors need extensions without shared filesystem
- Kubernetes: ConfigMaps and init containers add complexity
- Containers: Minimal images don't want extra layers

Solution: Fat Agent JAR Copy Chapter

A single JAR containing:
- All agent and boot classes
- Embedded extension API classes (as .class files for bootstrap)
- Embedded extension impl classes (as .classdata for runtime loading)
- Extension metadata in META-INF/btrace-extensions/

Architecture Copy Chapter

Class Loading Strategy Copy Section

┌─────────────────────────────────────────────────────────────┐
│                    Fat Agent JAR                             │
├─────────────────────────────────────────────────────────────┤
│  Bootstrap Classpath (via Boot-Class-Path manifest)         │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  io/btrace/agent/...              (agent classes)     │  │
│  │  io/btrace/core/...               (core classes)      │  │
│  │  io/btrace/instr/...              (instr classes)     │  │
│  │  org/example/ext/api/...          (extension API)     │  │
│  └───────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  Runtime-Loaded (via ClassDataLoader)                       │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  org/example/ext/impl/...classdata (extension impl)   │  │
│  └───────────────────────────────────────────────────────┘  │
├─────────────────────────────────────────────────────────────┤
│  META-INF/btrace-extensions/                                │
│  ├── ext1/extension.properties                              │
│  ├── ext2/extension.properties                              │
│  └── ext3/extension.properties                              │
└─────────────────────────────────────────────────────────────┘

Extension Discovery Flow Copy Section

Agent Startup
     │
     ▼
┌─────────────────────────┐
│ Parse BTRACE_HOME       │
│ (null for embedded)     │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ Read manifest attribute │
│ BTrace-Embedded-        │
│ Extensions: ext1,ext2   │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ For each extension ID:  │
│ - Load extension.props  │
│ - Create descriptor     │
│ - Register services     │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ API classes already on  │
│ bootstrap (as .class)   │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ Impl classes loaded on  │
│ demand via ClassData-   │
│ Loader (from .classdata)│
└─────────────────────────┘

Plugin Task Graph Copy Section

fatAgentJar
├── stageExtensions
│   ├── resolveExtensions()  → ResolvedExtension[]
│   ├── stageApiClasses()    → copy as .class
│   ├── stageImplClasses()   → copy as .classdata
│   └── writeMetadata()      → extension.properties
├── stageProbes (optional)
│   ├── copyCompiledProbes() → META-INF/btrace-probes/
│   └── compileSourceProbes()
└── btraceJar (single masked JAR from btrace-dist)

Implementation Details Copy Chapter

Extension Sources Copy Section

The plugin supports three extension source types:

embedExtensions {
    project(':my-extension')           // ProjectExtensionSource
    maven('io.btrace:ext:1.0')         // MavenExtensionSource
    file('/path/to/ext.zip')           // FileExtensionSource
}

Each source resolves to a ResolvedExtension:

class ResolvedExtension {
    String id
    String version
    File apiJar      // contains API classes
    File implJar     // contains impl classes (shadowed)
    Properties metadata
}

Staging Process Copy Section

1. API Classes: Extracted from API JAR and copied as .class files
- These end up on bootstrap classpath
- Visible to BTrace scripts and the agent

2. Impl Classes: Extracted from impl JAR and renamed to .classdata
- Loaded at runtime by ClassDataLoader
- Isolated from target application classpath

3. Metadata: Written to META-INF/btrace-extensions/{id}/extension.properties

Auto-Discovery Copy Section

When autoDiscover = true, the plugin scans subprojects:

project.gradle.projectsEvaluated {
    rootProject.subprojects.each { sp ->
        if (sp.plugins.hasPlugin('io.btrace.extension')) {
            extension.addExtensionSource(new ProjectExtensionSource(project, sp.path))
        }
    }
}

The selection is narrowed in two ways. defaultExtensions names the subset to embed when the
filter property is absent, matching either a subproject's name or its path; the filter property
overrides it and can select anything discoverable:

./gradlew fatAgentJar -PembedExtensions=btrace-metrics,btrace-statsd

btrace-dist sets defaultExtensions to the released extension set, so an ordinary
fatAgentJar build embeds exactly what a release ships rather than every extension in the
repository — the test fixture and the Spark and Hadoop examples are excluded.

ShadowJar Integration Copy Section

When ShadowJar is available on the classpath, the plugin uses it for:
- Package relocation (avoid classpath conflicts)
- Duplicate handling

def jarTaskClass = Jar
try {
    jarTaskClass = Class.forName('com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar')
} catch (ClassNotFoundException e) {
    // Fall back to standard Jar
}

Manifest Attributes Copy Chapter

The fat agent JAR manifest includes:

| Attribute | Description |
|-----------|-------------|
| Premain-Class | io.btrace.boot.Loader |
| Agent-Class | io.btrace.boot.Loader |
| BTrace-Agent-Main | io.btrace.agent.Main |
| Can-Redefine-Classes | true |
| Can-Retransform-Classes | true |
| Boot-Class-Path | btrace-agent-fat.jar (self-reference) |
| BTrace-Embedded-Extensions | Comma-separated list of extension IDs |

After assembly, the plugin verifies that the JAR contains io/btrace/boot/Loader.class, the masked
META-INF/btrace/agent/io/btrace/agent/Main.classdata entry, and the mandatory loader manifest
attributes. A conventional agent JAR is not a valid base input; agentJarTask must reference the
task producing the masked BTrace JAR (normally btraceJar).

Runtime Behavior Copy Chapter

Extension Loading Copy Section

At agent startup:

1. Main.initExtensions() initializes the extension system
2. ExtensionLoader creates EmbeddedExtensionRepository
3. Repository reads BTrace-Embedded-Extensions from manifest
4. For each extension ID, loads META-INF/btrace-extensions/{id}/extension.properties
5. Creates ExtensionDescriptorDTO with embedded=true
6. API classes are already on bootstrap (no loading needed)
7. Impl classes loaded on-demand via ClassDataLoader

ClassDataLoader Copy Section

The ClassDataLoader loads .classdata files as classes. It registers as parallel-capable via ClassLoader.registerAsParallelCapable() and uses per-class-name locking via getClassLoadingLock(name) to allow concurrent loading of different classes while serializing attempts to load the same class:

public class ClassDataLoader {
    static {
        ClassLoader.registerAsParallelCapable();
    }
    public Class<?> findClass(String className) {
        synchronized (getClassLoadingLock(className)) {
            String resourceName = className.replace('.', '/') + ".classdata";
            InputStream is = getResourceAsStream(resourceName);
            byte[] bytes = is.readAllBytes();
            return defineClass(className, bytes, 0, bytes.length);
        }
    }
}

Use Cases Copy Chapter

1. Spark Deployment Copy Section

btraceFatAgent {
    baseName = 'btrace-spark-agent'
    embedExtensions {
        project(':btrace-extensions:btrace-spark')
        project(':btrace-extensions:btrace-metrics')
    }
}

Usage:

spark-submit --conf spark.driver.extraJavaOptions=-javaagent:btrace-spark-agent.jar ...

2. Kubernetes with Pre-loaded Extensions Copy Section

FROM btrace/btrace:latest AS btrace
FROM openjdk:17
Copy only the fat agent (no extension installation needed)

COPY --from=btrace /opt/btrace/libs/btrace-agent-fat.jar /opt/btrace/

3. CI/CD Pipeline Copy Section

steps:
  - name: Build Fat Agent
    run: ./gradlew fatAgentJar -PembedExtensions=btrace-metrics
  - name: Deploy
    run: kubectl cp btrace-agent-fat.jar pod:/opt/

Maven Plugin Copy Chapter

The unpublished Maven fat-agent module was removed for 3.0.0. It resolved artifacts and staged
classdata according to pre-3.0 contracts, which could produce a successful build whose embedded
services could not load. The Gradle plugin described in this guide is the supported 3.0 fat-agent
implementation.

Limitations Copy Chapter

1. No Hot-Reload: Embedded extensions cannot be updated without rebuilding the JAR
2. Size: Fat JAR is larger than minimal agent
3. Classpath Conflicts: Careful relocation needed to avoid conflicts with target app dependencies

Related Documentation Copy Chapter

- Extension Development Guide
- Provided-Style Extensions
- Migrating from libs/profiles

---

Architecture/InstrumentationBackends Copy Section

Instrumentation Backends

Document Version: 1.0
Last Updated: July 2026
Status: Implemented (v3.0.0+)

---

Overview Copy Chapter

BTrace performs bytecode instrumentation through a small internal SPI, InstrumentationBackend, with two implementations:

| Backend | Source set | Availability | Class file versions |
|---------|-----------|--------------|---------------------|
| AsmInstrumentationBackend | src/main/java (Java 8) | Always | ≤ 69 (up to Java 25) |
| ClassFileApiBackend | src/main/java24 (Java 24) | Agent running on JDK 24+ | > 69 (Java 26+) |

All types live in the io.btrace.instr package of the btrace-agent module:

- btrace-agent/src/main/java/io/btrace/instr/InstrumentationBackend.java
- btrace-agent/src/main/java/io/btrace/instr/AsmInstrumentationBackend.java
- btrace-agent/src/main/java/io/btrace/instr/BackendSelector.java
- btrace-agent/src/main/java/io/btrace/instr/ClassMeta.java
- btrace-agent/src/main/java24/io/btrace/instr/ClassFileApiBackend.java

Why: the ASM Ceiling Copy Chapter

BTrace's instrumentation pipeline is built on ASM. ASM can only parse class files up to a major version it explicitly knows about; ASM 9.9.x tops out at class file major version 69 (Java 25) and throws when handed anything newer:

/ Highest class file major version ASM 9.9.x can parse without throwing. */
static final int MAX_ASM_MAJOR_VERSION = 69; // Java 25

Without an alternative backend, an application compiled for Java 26+ (class file major version 70+) could not be instrumented at all. The JDK ClassFile API (java.lang.classfile.*, standardized in JDK 24) always understands the class file format of the JDK it ships with, so it provides a forward-compatible path for such classes.

The SPI Copy Chapter

interface InstrumentationBackend {
  / Returns {@code true} when this backend can process the given class file major version. */
  boolean supports(int classFileMajorVersion);
  /
   * Instruments {@code classfileBuffer} by applying all applicable probes.
   *
   * @param loader the classloader loading the target class (may be {@code null})
   * @param classfileBuffer raw class file bytes
   * @param probes all currently registered probes
   * @return transformed class bytes if at least one probe matched, {@code null} otherwise
   */
  byte[] instrument(ClassLoader loader, byte[] classfileBuffer, Collection<BTraceProbe> probes);
}

The ClassMeta interface complements the SPI: it exposes the minimal class metadata (Java class name, internal name, runtime-visible annotation types, classloader) needed for probe-to-class matching in BTraceProbeSupport, decoupling the matching logic from ASM's ClassReader so alternative backends can perform matching without constructing an ASM object.

Backend Selection Copy Chapter

BackendSelector chooses a backend based on the class file major version. The selection logic, verbatim:

static InstrumentationBackend select(int classFileMajorVersion) {
  if (!ASM.supports(classFileMajorVersion) && CLASSFILE_API != null) {
    return CLASSFILE_API;
  }
  return ASM;
}

In other words:

1. Class file version ≤ 69 → ASM backend (the default, full-featured path).
2. Class file version > 69 and the ClassFile API backend is available → ClassFile API backend.
3. Class file version > 69 but the ClassFile API backend is unavailable (agent running on JDK < 24) → falls back to ASM, which will fail to parse the class; instrumentation of that class is effectively skipped.

The ClassFile API backend is loaded reflectively at class-initialization time so the main (Java 8-compiled) source set has no compile-time dependency on java.lang.classfile:

Class<?> cls =
    Class.forName(
        "io.btrace.instr.ClassFileApiBackend", true, BackendSelector.class.getClassLoader());
return (InstrumentationBackend) cls.getDeclaredConstructor().newInstance();

On JDK < 24 the Class.forName fails (the compiled class targets class file version 68/Java 24 and references java.lang.classfile), the failure is logged at debug level, and the field stays null.

ClassFile API Backend Copy Chapter

Requirements Copy Section

- The agent must run on JDK 24+ — the backend is compiled with sourceCompatibility = 24 / targetCompatibility = 24 and uses java.lang.classfile.*.
- It is engaged only for class file major versions > 69 (supports() returns classFileMajorVersion > AsmInstrumentationBackend.MAX_ASM_MAJOR_VERSION).

How It Instruments Copy Section

The backend parses the class with ClassFile.parse(), builds a ClassMeta from the class model (name, runtime-visible annotations, classloader), collects applicable handlers via BTraceProbe.getApplicableHandlers(meta), and injects probe calls as invokedynamic instructions bootstrapped by io.btrace.runtime.IndyDispatcher.bootstrap(...) — entry probes before the first real instruction, return probes before each ReturnInstruction.

Current Limitations Copy Section

Verified in ClassFileApiBackend.java:

- Only Kind.ENTRY and Kind.RETURN probes are supported. Handlers with any other probe kind (CALL, LINE, FIELD_GET/SET, ERROR, etc.) are skipped with a debug-level log; the remaining handlers are still applied.
- Method matching supports exact names and /regex/ patterns; type-constrained method matching (a non-empty type in @OnMethod) is unsupported — such handlers are skipped.
- Supported handler parameters: @ProbeClassName, @ProbeMethodName, and @Self (on instance methods; null is passed for static methods and constructor entry). Handlers using other special parameters (@Return, @TargetInstance, @Duration, @TargetMethodOrField) or plain probed-method arguments are skipped.
- Classes the ClassFile API fails to parse are skipped (warning logged) rather than failing class loading.

Packaging Copy Chapter

The java24 source set is declared in btrace-agent/build.gradle and compiled with a JDK 24 toolchain. Its output is merged into the root of the regular agent jar:

jar {
    into('') {
        from sourceSets.java24.output
    }
}

Important consequences:

- The agent jar is NOT a Multi-Release JAR — there is no META-INF/versions/24/ entry and no Multi-Release manifest attribute. The compiled-at-24 ClassFileApiBackend.class sits at the jar root next to the Java 8-compatible classes.
- The class is never referenced directly from Java 8-compiled code; it is only ever loaded reflectively by BackendSelector. On older JDKs the reflective load fails cleanly (UnsupportedClassVersionError caught as Throwable), and BTrace continues with the ASM backend only.

Related Documents Copy Chapter

- ExtensionInvokeDynamicBridge — invokedynamic-based extension linkage

---

Architecture/MaskedJarArchitecture Copy Section

Masked JAR Architecture

Overview Copy Chapter

BTrace uses a single-JAR distribution (btrace.jar) with a classdata masking technique to minimize bootstrap classloader pollution. This architecture packs agent, client, and shared classes into one JAR while keeping only essential API classes visible to the bootstrap classloader.

The key insight: files with a .classdata extension are invisible to the JVM's built-in classloading, so non-bootstrap classes can coexist in the same JAR without leaking into the bootstrap classloader.

Problem Copy Chapter

Traditional Java agents add their entire JAR to the bootstrap classloader via Boot-Class-Path: . in the manifest. For BTrace, this would expose ~1500+ classes (ASM, compiler, instrumentation engine, client) to every class in the JVM. This causes:

- Namespace pollution (classes visible where they shouldn't be)
- Potential conflicts with application dependencies (e.g., different ASM versions)
- Increased memory footprint in the bootstrap classloader

The previous multi-JAR approach (btrace-agent.jar, btrace-boot.jar, btrace-client.jar) solved this by splitting classes across JARs but introduced hardcoded co-location assumptions that broke alternative distribution methods like jbang and Maven repositories.

Architecture Copy Chapter

JAR Structure Copy Section

btrace.jar (~2.9 MB)
├── io/btrace/boot/*.class                      # Entry point (Loader, MaskedClassLoader, MaskedJarUtils)
├── io/btrace/core/*.class                      # Bootstrap: core API (excl. Messages, comm, extensions)
├── io/btrace/core/annotations/*.class          # Bootstrap: annotations
├── io/btrace/core/handlers/*.class             # Bootstrap: handler interfaces
├── io/btrace/core/types/*.class                # Bootstrap: type definitions
├── io/btrace/core/jfr/*.class                  # Bootstrap: JFR integration
├── io/btrace/runtime/*.class                   # Bootstrap: selected runtime support classes
├── io/btrace/extension/ExtensionBridge.class   # Bootstrap: extension bridge interface
├── io/btrace/libs/org/slf4j/                 # Bootstrap: relocated SLF4J
├── META-INF/btrace/agent/*.classdata           # Masked: agent classes
├── META-INF/btrace/client/*.classdata          # Masked: client classes
├── META-INF/btrace/shared/*.classdata          # Masked: shared classes (ASM, protocol, etc.)
└── META-INF/MANIFEST.MF

Bootstrap classes (~112 total): Only the core API, runtime support, and SLF4J logging are stored as regular .class files. These are visible to the bootstrap classloader because the manifest declares Boot-Class-Path: ..

Masked classes (~1600+): Agent, client, and shared classes are stored as .classdata files under META-INF/btrace/. The JVM's class loading ignores these files entirely. They are loaded on demand by MaskedClassLoader.

The client section also contains the instrumentation/runtime implementation used by btracec to
turn compiled scripts into persisted probe packs. The io.btrace.core.PackGenerator service
descriptor is visible from the masked JAR root, so its io.btrace.instr.InstrPackGenerator
provider and non-bootstrap dependencies must be loadable by the client masked classloader as well
as by the agent classloader.

Manifest Copy Section

Premain-Class: io.btrace.boot.Loader
Agent-Class: io.btrace.boot.Loader
Main-Class: io.btrace.boot.Loader
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Boot-Class-Path: .
BTrace-Agent-Main: io.btrace.agent.Main
BTrace-Client-Main: io.btrace.client.Main

Loader is the single entry point for all three modes. The actual agent/client main classes are specified as manifest attributes and loaded reflectively via MaskedClassLoader.

Components Copy Chapter

btrace-boot Module Copy Section

Location: btrace-boot/

Contains three classes, all loaded by the bootstrap classloader:

#### Loader (io.btrace.boot.Loader)

Entry point for all three modes:

| Mode | Entry Method | Invocation |
|------|-------------|------------|
| Load-time agent | premain(String, Instrumentation) | -javaagent:btrace.jar |
| Dynamic attach | agentmain(String, Instrumentation) | VirtualMachine.loadAgent() |
| Client CLI | main(String[]) | java -jar btrace.jar |

Each mode creates a MaskedClassLoader for the appropriate section (agent or client), loads the main class from the manifest attribute, and invokes it reflectively.

Agent mode classloader hierarchy:

Bootstrap CL (core API, runtime, Loader)
    └── MaskedClassLoader[agent] (parent=null)
            Loads from: META-INF/btrace/agent/*.classdata
            Fallback:   META-INF/btrace/shared/*.classdata

Using null as the parent ensures bootstrap-visible classes (like BTraceRuntimeAccess) are loaded from the bootstrap classloader rather than the agent's classloader. This is critical because probe classes—defined via the bootstrap classloader—must see the same class instances the agent uses.

Client mode classloader hierarchy:

Bootstrap CL (core API, runtime, Loader)
    └── System CL
        └── MaskedClassLoader[client] (parent=System CL)
                Loads from: META-INF/btrace/client/*.classdata
                Fallback:   META-INF/btrace/shared/*.classdata

#### MaskedClassLoader (io.btrace.boot.MaskedClassLoader)

A URLClassLoader subclass that loads classes from .classdata files. The lookup order for findClass(name):

1. Check section-specific path: META-INF/btrace/{section}/{class-path}.classdata
2. Check shared path: META-INF/btrace/shared/{class-path}.classdata
3. Throw ClassNotFoundException (delegates to parent via standard classloader contract)

Resource loading (findResource, getResourceAsStream) follows the same pattern: section-specific first, then shared, then the JAR root.

#### MaskedJarUtils (io.btrace.boot.MaskedJarUtils)

Utility class for detecting masked JARs. A JAR is identified as masked by the presence of a META-INF/btrace/shared/ entry. Used by Client.java and integration tests to detect whether they are running from a masked JAR or the legacy multi-JAR layout.

Bootstrap Class Selection Copy Section

Location: btrace-dist/build.gradle (the bootIncludes filter closure)

The bootIncludes filter closure defined in btrace-dist/build.gradle determines which classes belong in the bootstrap section. It is used by the bootstrapJar and btraceJar tasks to separate classes into .class (bootstrap) vs .classdata (masked) during JAR assembly.

Bootstrap inclusion criteria:
- io/btrace/core/ — core API, including annotations and handlers (excluding Messages, messages.properties, comm/, and extensions/)
- io/btrace/runtime/ — selected runtime support classes only (BTraceRuntimeAccess, LinkingFlag, Indy*, ExtensionIndy, BTraceBootstrap, auxiliary/Auxiliary)
- io/btrace/BTrace.class — flat DSL entry class (needed by javac when compiling scripts)
- io/btrace/extension/ExtensionBridge.class — extension bridge interface (referenced by ExtensionIndy)
- io/btrace/libs/org/slf4j/ — relocated SLF4J (logging from bootstrap code)

Explicitly excluded from bootstrap:
- ASM classes (loaded via shared section)
- JCTools queues (loaded via shared section)
- Communication protocol classes (io/btrace/core/comm/)
- Extension API classes (io/btrace/core/extensions/)

Build Process Copy Chapter

The btrace-dist/build.gradle btraceJar task assembles the masked JAR:

1. prepareAgentClassdata: Extracts classes from the agent shadow JAR, renames .class → .classdata, places under META-INF/btrace/agent/
2. prepareClientClassdata: Same for the client shadow JAR, placed under META-INF/btrace/client/
3. btraceJar: Combines bootstrap .class files (filtered by btrace-bootstrap) with the .classdata sections and shared classes into the final JAR

Shared classes (ASM, protocol, etc.) are automatically placed in META-INF/btrace/shared/ and are accessible to both agent and client classloaders.

Debugging Copy Chapter

Enable debug output with:

java -Dbtrace.boot.debug=true -jar btrace.jar ...

This prints classloader decisions to stderr, prefixed with [BTrace Boot].

Backward Compatibility Copy Chapter

The masked JAR coexists with the legacy multi-JAR distribution. Detection logic in Client.java and RuntimeTest checks for btrace.jar first and falls back to btrace-agent.jar if not found. The --agent-jar and --boot-jar CLI flags allow explicit path overrides for any layout.

---

Architecture/Migrating From Libs Profiles Copy Section

Migrating from libs/profiles to Extensions

This guide helps you move from btrace-libs/<profile> to extension-based integrations.

Why Migrate Copy Chapter

- Security & isolation: extensions use API on bootstrap and isolated impls; libs/profiles mutate global classpaths.
- Operability: discovery, enable/disable, permissions, diagnostics.
- Maintainability: versioning and conflict handling.

Status Copy Chapter

libs/profiles are removed. Passing libs=<profile> logs an error naming the profile and
loads nothing; the jars under btrace-libs/<profile>/ are ignored. Migrate to an extension, or
use the escape hatch below as a stopgap.

If you are arriving here because custom classes stopped resolving after upgrading, that is the
expected symptom: the agent starts normally and probes fail later on the missing types.

Migration Steps Copy Chapter

1. Inventory usage:
- Identify profile(s) used (e.g., spark-driver, spark-executor, hadoop).
- List the APIs probes rely on; isolate to a minimal contract.

2. Create an extension:
- One Gradle project with a single src/main/java, applying io.btrace.extension. The plugin
partitions it into API and implementation artifacts from the declared services plus any
additionalExports/excludedExports — there is no separate API module to create.
- Keep application types out of the API surface: minimal interfaces and DTOs only.
- Declare application dependencies with implCompileOnly so they stay off the API artifact.

3. Replace profile dependency:
- Remove libs=<profile> from agent args.
- Enable the extension in extensions.conf (extensions.enabled, extensions.disabled,
extensions.autoload).

4. Replace classpath assumptions:
- Update probes to pass app objects to API methods (object hand-off) instead of importing app types.
- In impl, resolve types via TCCL/defining loader.
- Recompile the probes against the new API artifact.

5. Configure runtime:
- Add role/config keys to extensions.conf (e.g., role=driver|executor; optional classpath hint if the app doesn’t ship libs).
- The extension plugin scans the implementation and its dependencies and writes the merged
permission set into the API manifest, so permissions are usually not declared by hand.
Review what it wrote: a single transitive dependency can make the whole extension privileged.
- Permission grants are a separate, operator-side file — permissions.properties
(allowExtensions, denyExtensions, allowPrivileged), not extensions.conf.

6. Validate:
- Launch/attach runs; verify extension loads and links lazily/eagerly as required.
- Injection throws by default. Marking an injection optional, or selecting shim mode, turns a
failed link into a no-op returning defaults — convenient in production, and the quickest way
to make an unfinished migration look complete.
- Use btrace -le <PID> to see why an extension failed to link.

Escape Hatch (Optional) Copy Chapter

If immediate migration is not feasible and the app must see a jar on the system classpath:

-Dbtrace.system.appendJar=/abs/path/lib.jar -javaagent:btrace.jar=trusted=true

- trusted is an agent argument, not a system property; -Dbtrace.trusted=true has no effect
here.
- Restricted to BTRACE_HOME unless -Dbtrace.allowExternalLibs=true. When BTRACE_HOME cannot
be determined the jar is appended anyway and a warning is logged.
- One jar only; discouraged; subject to removal.

Fat Agent Deployment Copy Chapter

For Spark/Hadoop/Kubernetes environments where managing separate extension JARs is impractical, use fat agent builds to embed extensions directly.

Gradle Plugin Copy Section

plugins {
    id 'io.btrace.fat-agent'
}
btraceFatAgent {
    baseName = 'my-btrace-agent'
    embedExtensions {
        project(':my-spark-extension')
        file('/path/to/btrace-metrics-3.0.0-extension.zip')
    }
}

Build: ./gradlew fatAgentJar

Maven Plugin Copy Section

The unpublished Maven fat-agent module was removed for 3.0.0. Migrate fat-agent builds to the
Gradle configuration above; the external Maven plugin remains available for script compilation,
not embedded-extension packaging.

Usage Copy Section

Spark

spark-submit --conf spark.driver.extraJavaOptions=-javaagent:my-btrace-agent.jar ...
Kubernetes

java -javaagent:/opt/btrace/my-btrace-agent.jar MyApp

See Fat Agent Plugin Architecture for details.

Examples & Templates Copy Chapter

- See provided-style extension guide: docs/architecture/provided-style-extensions.md for Spark/Hadoop templates and extensions.conf snippets.

Assisted Migration Copy Chapter

The BTrace agent plugins marketplace provides a
btrace-legacy-libs-migration
skill for Claude Code, Codex, and Pi. It walks the inventory, the API/implementation split, the
service and permission metadata, and the verification steps described above.

---

Architecture/Provided Style Extensions Copy Section

Provided-Style Extensions for App Integrations

This guide shows how to migrate profile-based integrations (e.g., Spark/Hadoop) to BTrace extensions without mutating the global classpath.

Principles Copy Chapter

- API on bootstrap: expose minimal, stable APIs with simple/value types.
- Impl in isolated CL: load implementation in an extension classloader; no shading of app libs.
- Runtime linking: access application types via object hand-off and TCCL instead of compile-time imports.
- No classpath injection: avoid boot/system CL changes; reserve the escape hatch for exceptional cases only.

Helper Utilities Copy Chapter

- io.btrace.extension.util.ClassLoadingUtil
- Loaders: tccl(), definingLoader(Object)
- Class loading: load(String, ClassLoader), load(String, Object), tryLoad(String, ClassLoader)
- Context: withTCCL(ClassLoader, Supplier<T>), withTCCL(ClassLoader, Runnable), withDefiningLoader(Object, Supplier<T>)
- Services: loadService(Class<T>, ClassLoader), loadServices(Class<T>, ClassLoader)
- Resources: getResource(String, ClassLoader), openResource(String, ClassLoader)
- Optional child loader: newChildURLClassLoader(List<Path>, ClassLoader), safeClose(ClassLoader)

- io.btrace.extension.util.MethodHandleCache
- Caches successful public static and virtual method lookups only. Its strong owner-class keys can retain application loaders; it has no getter, setter, or constructor helper.

API Sketch (Spark example) Copy Chapter

// exported API (on bootstrap)
package org.example.btrace.spark.api;
public interface SparkApi {
  void onJobStart(Object jobStartEvent);
  void onStageCompleted(Object stageInfo);
}

Impl Sketch Copy Chapter

// implementation (extension CL)
package org.example.btrace.spark.impl;
import org.example.btrace.spark.api.SparkApi;
import io.btrace.extension.util.ClassLoadingUtil;
import io.btrace.extension.util.MethodHandleCache;
import java.lang.invoke.MethodHandle;
public final class SparkApiImpl implements SparkApi {
  private final MethodHandleCache mh = new MethodHandleCache();
  @Override
  public void onJobStart(Object evt) {
    ClassLoadingUtil.withDefiningLoader(
      evt,
      () -> {
        try {
          Class<?> cls = ClassLoadingUtil.loadFromContext(
              "org.apache.spark.scheduler.SparkListenerJobStart", evt);
          MethodHandle getJobId = mh.findVirtual(cls, "jobId", int.class);
          int jobId = (int) getJobId.invoke(evt);
          // emit metrics/logs...
        } catch (Throwable t) {
          // log and continue
        }
        return null;
      });
  }
}

External Type Adapters Copy Chapter

@ExternalType supports exact public methods, opaque target-type positions, complete overload
groups, exact public field getters/setters, constructors, and isInstance/cast operations. The
canonical signatures, owner-loader, failure, and static-loader rules are in the Extension
Development Guide.
Keep the manual pattern here for generic/array target types, bulk reflection-style work, fluent
setters, constructor selectors, runtime coercion, and non-public or JPMS-bypassing access.

For a generated static target call where the application loader is known, pass it directly:

String version = VersionApi$Ext.version(appLoader);

Reserve ClassLoadingUtil.withTCCL for manual APIs that require a TCCL policy.

For fields and constructors there is no MethodHandleCache convenience method. Use a direct public lookup after resolving the target class:

Class<?> type = ClassLoadingUtil.load("com.example.AppType", appLoader);
MethodHandle constructor = MethodHandles.publicLookup().findConstructor(type, MethodType.methodType(void.class));

Role Detection & Config Copy Chapter

- Detect driver/executor via system properties or presence of marker classes using TCCL.
- extensions.conf (examples):

Spark

btrace-spark.enabled=true
btrace-spark.role=auto           # auto|driver|executor
optional: only if the app does not ship required libs

btrace-spark.classpath=/opt/spark/jars
Hadoop

btrace-hadoop.enabled=false
btrace-hadoop.classpath=/opt/hadoop/share/hadoop/common

Permissions Copy Chapter

- Typical: REFLECTION, THREADS, SYSTEM_PROPS.
- Optional: CLASSLOADER if creating a child URLClassLoader from configured paths.

Escape Hatch (last resort) Copy Chapter

- If absolutely unavoidable, append a single jar to the system CL:
- -Dbtrace.system.appendJar=/abs/path/lib.jar -Dbtrace.trusted=true
- Restricted to BTRACE_HOME by default; override with -Dbtrace.allowExternalLibs=true.

Hadoop Example (Sketch) Copy Chapter

public interface HadoopApi { void onFsOp(Object op); }
public final class HadoopApiImpl implements HadoopApi {
  // Resolve org.apache.hadoop.fs.FileSystem via TCCL and reflectively extract fields
}

Migration Steps Copy Chapter

1. Extract minimal API for probes; avoid app types.
2. Move environment-specific logic to impl; resolve app types via object hand-off/TCCL.
3. Add extension config (role, optional classpath hints).
4. Request permissions; add no-op shims when unavailable.
5. Prefer eager load if APIs must be present before probes start.

Notes Copy Chapter

- Keep APIs small and stable; impls can evolve independently.
- Cache MethodHandles for performance; avoid repeated reflective lookups.
- Do not rely on global classpath mutation; it’s discouraged and may be removed.

Example Projects Copy Chapter

- Spark example: btrace-extensions/examples/btrace-spark
- Hadoop example: btrace-extensions/examples/btrace-hadoop

See also: docs/examples/README.md for quick build and configuration snippets.

---

Architecture/Version2ProtocolArchitecture Copy Section

BTrace v2 Binary Protocol Architecture

Document Version: 1.1
Last Updated: February 2026
Status: Implemented (first released in 3.0.0)

---

Table of Contents Copy Chapter

1. Executive Summary
2. Problem Statement
3. High-Level Architecture
4. Protocol Negotiation
5. Wire Format Specification
6. Command Conversion Layer
7. Benefits and Trade-offs
8. Migration Path
9. Performance Characteristics

---

Executive Summary Copy Chapter

The BTrace v2 binary protocol is a performance-optimized communication protocol that replaces Java Object Serialization with custom binary serialization. The v2 protocol delivers 3-6x faster command transmission and 2-5x smaller wire payloads while maintaining full backward compatibility with the existing v1 protocol through automatic protocol negotiation.

Key Features:
- Custom binary serialization (vs Java ObjectInputStream/ObjectOutputStream)
- Automatic protocol negotiation (once per connection)
- Compression support for large payloads (>1KB threshold)
- Thread-safe using ReentrantLock (vs synchronized blocks)
- Full backward compatibility with v1 protocol
- Zero-configuration auto-detection

---

Problem Statement Copy Chapter

What Problem Does v2 Solve? Copy Section

The original BTrace protocol (v1) relies on Java Object Serialization for agent-client communication. While functional, this approach has significant limitations:

#### 1. Performance Bottleneck
Problem: Java serialization is slow
- ObjectInputStream/ObjectOutputStream use reflection and complex state management
- Each command incurs serialization overhead (object graphs, metadata)
- High CPU usage during marshaling/unmarshaling

Impact:
- Limits throughput for high-frequency tracing scenarios
- Increases latency for interactive debugging
- Consumes CPU resources that could be used for actual tracing

#### 2. Large Wire Payloads
Problem: Java serialization produces verbose binary format
- Includes class metadata, type descriptors, stream headers
- Inefficient encoding of primitive types and strings
- No built-in compression

Impact:
- Increased network bandwidth usage
- Slower transmission over slow connections
- Higher memory usage for buffering

#### 3. Language Lock-in
Problem: Java serialization ties BTrace to JVM-only clients
- Cannot implement clients in other languages (Python, Go, JavaScript)
- Limits future extensibility (browser-based tools, IDE plugins in non-JVM languages)

Impact:
- Restricts ecosystem growth
- Prevents integration with non-Java monitoring tools

#### 4. Dated Concurrency Model
Problem: v1 uses synchronized blocks for thread safety
- Coarse-grained locking can become bottleneck
- Limited scalability for concurrent client connections

Impact:
- Performance degradation with multiple concurrent clients
- Contention under high load

Real-World Scenario Copy Section

Consider a production environment with high-frequency tracing:

v1 Protocol:

10,000 MessageCommands/second
Average size: 512 bytes serialized
Network: 5.12 MB/second
CPU overhead: ~15% for serialization

v2 Protocol:

10,000 MessageCommands/second
Average size: 170 bytes serialized (3x smaller with compression)
Network: 1.7 MB/second (67% reduction)
CPU overhead: ~3% for serialization (80% reduction)

Result: 67% less bandwidth, 80% less CPU overhead, same functionality

---

High-Level Architecture Copy Chapter

Component Overview Copy Section

/ Detailed source-code truncated for AI context efficiency. /

Key Components Copy Section

#### 1. WireProtocol Interface
Purpose: Abstract wire format from business logic

Location: btrace-core/src/main/java/io/btrace/core/comm/WireProtocol.java

Responsibilities:
- Define contract for reading/writing Command objects
- Abstract away serialization mechanism
- Support protocol version introspection

Interface:

public interface WireProtocol {
    Command read(InputStream in) throws IOException;
    void write(OutputStream out, Command cmd) throws IOException;
    void reset() throws IOException;  // for ObjectOutputStream.reset() in v1
    int getVersion();
}

#### 2. Protocol Negotiator
Purpose: Auto-detect and negotiate protocol version

Location: btrace-core/src/main/java/io/btrace/core/comm/ProtocolNegotiator.java

Responsibilities:
- Perform handshake at connection establishment
- Detect client/agent protocol capabilities
- Select optimal protocol version
- Handle negotiation timeouts and failures

#### 3. Command Adapter
Purpose: Convert between v1 and v2 command representations

Location: btrace-core/src/main/java/io/btrace/core/comm/v2/CommandAdapter.java

Responsibilities:
- Bidirectional conversion: Command ↔ BinaryCommand
- Preserve all command data during conversion
- Handle type mismatches gracefully

#### 4. Binary Protocol Layer
Purpose: Efficient binary serialization

Location: btrace-core/src/main/java/io/btrace/core/comm/v2/

Components:
- BinaryProtocol: Low-level primitives (readInt, writeString, etc.)
- BinaryWireIO: Wire format implementation (version + type + data)
- BinaryCommand: Base class for all binary commands
- 17 Command Implementations: One per command type (Exit, Message, Instrument, etc.)

---

Protocol Negotiation Copy Chapter

Prepared-mode authentication boundary Copy Section

When the agent was loaded at JVM startup with its command server enabled, authentication occurs
before the V1/V2 choice described below. The client sends BTA1, a four-byte big-endian token
length, and the token bytes. The agent responds with BTAK on success or BTAF on failure. Only
after BTAK may the client send the V1 serialization header or the V2 BTR2 prefix.

Both sides apply the protocol-negotiation timeout to this preamble. Invalid lengths, truncated or
incorrect tokens, and direct V1/V2 bytes are rejected before any command decoder is constructed.
V2-to-V1 fallback opens a new connection and authenticates it again; authentication failure never
causes an unauthenticated fallback. Dynamically attached agents retain the version-negotiation flow
without this prepared-mode preamble for compatibility with 2.2.x clients.

prepared: TCP connect -> BTA1 + length + token -> BTAK -> V1/V2 negotiation -> commands
dynamic:  TCP connect -> V1/V2 negotiation -> commands

Design Principle: Once Per Connection Copy Section

Critical: Protocol negotiation happens once when a connection is established, not per command.

Timeline:

Time 0ms:     TCP socket established
Time 1ms:     Prepared mode authenticates; dynamic mode skips this step
Time 2ms:     Client sends magic bytes (BTR2) or v1 header
Time 5ms:     Agent responds with protocol acknowledgment
Time 6ms:     Protocol locked for session (v1 or v2)
Time 7ms:     First command sent (using negotiated protocol)
...           [All subsequent commands use same protocol]
Time 60000ms: Connection closed

Why Once Per Connection:
- Performance: Negotiating per command would add massive overhead (~5ms per command)
- Consistency: All commands in a session use same wire format
- Simplicity: WireProtocol is set once and reused
- Statefulness: Negotiated protocol stored in Client/RemoteClient instance

Handshake Protocol: Magic Byte Prefix Copy Section

Approach: Client sends 4-byte magic prefix at connection start

v2 Magic Bytes: 0x42 0x54 0x52 0x32 ("BTR2" in ASCII)

Flow Diagram:

Client (v2-capable)                    Agent (v2-capable)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0x42 0x54 0x52 0x32] ──────▶│  ◀── Client sends BTR2 magic
        │                                      │
        │                                      ├── Recognizes BTR2
        │                                      ├── Agent supports v2
        │                                      │
        │◀─────── [0x42 0x54 0x52 0x32] ───────┤  ◀── Agent responds with BTR2
        │                                      │
        ├── Protocol = v2 ───────────────────  ├── Protocol = v2
        │                                      │
        ├──────── SetSettingsCommand (v2) ────▶│
        ├──────── InstrumentCommand (v2) ─────▶│
        │◀─────── StatusCommand (v2) ──────────┤
        │◀─────── MessageCommand (v2) ─────────┤
        ...

Fallback to v1:

Client (v2-capable)                    Agent (v1-only)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0x42 0x54 0x52 0x32] ──────▶│  ◀── Client tries v2
        │                                      │
        │         [5 second timeout]           ├── Does not recognize BTR2
        │                                      ├── No response
        │                                      │
        ├── Timeout, fallback to v1 ───────────┤
        │                                      │
        ├──────── [0xAC 0xED ...] ────────────▶│  ◀── Java serialization header
        │                                      │
        │                                      ├── Recognizes Java serialization
        │                                      ├── Protocol = v1
        │                                      │
        ├── Protocol = v1 ─────────────────────┼── Protocol = v1
        │                                      │
        ├──────── SetSettingsCommand (v1) ────▶│
        ...

v1-only Client:

Client (v1-only)                       Agent (v2-capable)
        │                                      │
        ├──────── TCP Connect ────────────────▶│
        │                                      │
        ├──────── [0xAC 0xED ...] ────────────▶│  ◀── Java serialization header
        │                                      │
        │                                      ├── Detects v1 (0xAC 0xED magic)
        │                                      ├── Protocol = v1
        │                                      │
        ├── Protocol = v1 ─────────────────────┼── Protocol = v1
        │                                      │
        ├──────── SetSettingsCommand (v1) ────▶│
        ...

Implementation Details Copy Section

Agent Side (RemoteClient.getClient()):

Socket sock = acceptConnection();
InputStream in = sock.getInputStream();
OutputStream out = sock.getOutputStream();
// Prepared mode only; rejects before a protocol decoder exists
ConnectionAuthenticator.authenticateAgent(in, out, expectedToken);
// Negotiate protocol (reads first bytes)
ProtocolVersion version = ProtocolNegotiator.negotiateAgent(in, out);
// Create appropriate adapter
WireProtocol wire = createWireProtocol(version, in, out);
// Store for session
remoteClient.setWireProtocol(wire);
// All subsequent commands use 'wire'
Command cmd = wire.read(in);
wire.write(out, statusResponse);

Client Side (Client.submit()):

Socket sock = new Socket(host, port);
InputStream in = sock.getInputStream();
OutputStream out = sock.getOutputStream();
// Prepared mode only; repeats on every fallback connection
ConnectionAuthenticator.authenticateClient(in, out, token);
// Negotiate protocol (sends magic bytes, waits for response)
ProtocolVersion preferred = getPreferredVersion(); // from config
ProtocolVersion version = ProtocolNegotiator.negotiateClient(in, out, preferred);
// Create appropriate adapter
WireProtocol wire = createWireProtocol(version, in, out);
// Store for session
this.wire = wire;
// All subsequent commands use 'wire'
wire.write(out, setSettingsCmd);
wire.write(out, instrumentCmd);
Command status = wire.read(in);

Negotiation Timeout Copy Section

Default: 5 seconds

Rationale:
- Long enough for slow networks
- Short enough to fail fast
- Prevents hanging on unresponsive agents

Configuration:

-Dbtrace.protocol.negotiation.timeout=5000

Compatibility Matrix Copy Section

| Client Version | Agent Version | Negotiated Protocol | Notes |
|---------------|---------------|---------------------|-------|
| v1-only | v1-only | v1 | Legacy |
| v1-only | v2-capable | v1 | Agent detects v1 magic (0xAC 0xED) |
| v2-capable | v1-only | v1 | Client timeout → fallback |
| v2-capable | v2-capable | v2 | Optimal path |

Key Insight: Old clients always work with new agents, new clients always work with old agents

---

Wire Format Specification Copy Chapter

v2 Protocol Format Copy Section

Overall Structure:

┌──────────────┬──────────────┬────────────────────────────┐
│ Version (1B) │ Type (1B)    │ Command Data (variable)    │
└──────────────┴──────────────┴────────────────────────────┘

Version Byte: Current version is 0x03 (bumped from 0x02 after binary format changes to ErrorCommand and GridDataCommand)

Type Byte: Command type identifier (0-16)

| Type | Hex | Command Name |
|------|------|------------------------------------|
| 0 | 0x00 | ERROR |
| 1 | 0x01 | EVENT |
| 2 | 0x02 | EXIT |
| 3 | 0x03 | INSTRUMENT |
| 4 | 0x04 | MESSAGE |
| 5 | 0x05 | RENAME |
| 6 | 0x06 | STATUS |
| 7 | 0x07 | NUMBER_MAP |
| 8 | 0x08 | STRING_MAP |
| 9 | 0x09 | NUMBER |
| 10 | 0x0A | GRID_DATA |
| 11 | 0x0B | RETRANSFORMATION_START |
| 12 | 0x0C | RETRANSFORM_CLASS |
| 13 | 0x0D | SET_PARAMS |
| 14 | 0x0E | LIST_PROBES |
| 15 | 0x0F | DISCONNECT |
| 16 | 0x10 | RECONNECT |

Primitive Type Encoding Copy Section

Integers (int): 4 bytes, big-endian

Value: 42
Bytes: [0x00, 0x00, 0x00, 0x2A]

Longs (long): 8 bytes, big-endian

Value: 1234567890
Bytes: [0x00, 0x00, 0x00, 0x00, 0x49, 0x96, 0x02, 0xD2]

Booleans (boolean): 1 byte

true:  [0x01]
false: [0x00]

Strings (String): Length-prefixed UTF-8

Format: [length (4B)] [UTF-8 bytes]
Example: "Hello"
Bytes: [0x00, 0x00, 0x00, 0x05, 0x48, 0x65, 0x6C, 0x6C, 0x6F]
        └─── length=5 ────┘  └────── "Hello" UTF-8 ──────────┘

Null Strings: Length = -1

null: [0xFF, 0xFF, 0xFF, 0xFF]

Byte Arrays (byte[]): Length-prefixed raw bytes

Format: [length (4B)] [raw bytes]
Example: [0xCA, 0xFE, 0xBA, 0xBE]
Bytes: [0x00, 0x00, 0x00, 0x04, 0xCA, 0xFE, 0xBA, 0xBE]
        └─── length=4 ────┘  └──── raw bytes ────┘

Example: MessageCommand Copy Section

Structure:

┌─────────┬─────────┬──────────────┬─────────────────┬─────────────────┐
│ Version │ Type    │ Urgent Flag  │ Timestamp (8B)  │ Message (String)│
│ (1B)    │ (1B)    │ (1B)         │                 │                 │
└─────────┴─────────┴──────────────┴─────────────────┴─────────────────┘
  0x02      0x04      0x00/0x01      long              length + UTF-8

Example Bytes:

Message: "BTrace started"
Timestamp: 1638360000000
Urgent: false
Hex dump:
02 04 00 00 00 00 01 7D 28 4F 2D 00 00 00 00 0E
42 54 72 61 63 65 20 73 74 61 72 74 65 64
Breakdown:
02           - Version = 2
04           - Type = MESSAGE (4)
00           - Urgent = false
00 00 00 01 7D 28 4F 2D - Timestamp = 1638360000000
00 00 00 0E  - String length = 14
42 54 72 61 63 65 20 73 74 61 72 74 65 64 - "BTrace started" (UTF-8)

Compression Copy Section

Trigger: Message size > 1024 bytes (configurable)

Algorithm: Java Deflater/Inflater (BEST_SPEED)

Format with Compression:

┌─────────┬─────────┬──────────────┬──────────────────┬────────────────────┐
│ Version │ Type    │ Urgent Flag  │ Compressed Flag  │ Compressed/Raw Data│
│ (1B)    │ (1B)    │ (1B)         │ (1B)             │                    │
└─────────┴─────────┴──────────────┴──────────────────┴────────────────────┘
  0x02      0x04      0x00/0x01      0x00/0x01          byte array

Compressed Data:

[Original Length (4B)] [Compressed Length (4B)] [Deflated Bytes]

Benefits:
- 3-5x size reduction for large text messages
- Automatically applied for messages >1KB
- Transparent to Command layer

---

Command Conversion Layer Copy Chapter

Purpose Copy Section

The CommandAdapter provides bidirectional conversion between v1 (Command) and v2 (BinaryCommand) representations, enabling:

1. v2 wire protocol to work with v1 business logic
2. Gradual migration without rewriting all command handling
3. Testing v2 implementation against v1 baseline

Architecture Copy Section

/ Detailed source-code truncated for AI context efficiency. /

Conversion Examples Copy Section

v1 → v2 (Client sending command):

// Client has Command object (v1)
MessageCommand v1Cmd = new MessageCommand("Hello from BTrace");
// Convert to BinaryCommand (v2)
BinaryCommand v2Cmd = CommandAdapter.toBinaryCommand(v1Cmd);
// Result: BinaryMessageCommand with message="Hello from BTrace"
// Serialize to wire
BinaryWireIO.write(outputStream, v2Cmd);
// Wire: [0x02][0x04][urgent][timestamp][length][UTF-8 bytes]

v2 → v1 (Agent receiving command):

// Read from wire
BinaryCommand v2Cmd = BinaryWireIO.read(inputStream);
// Result: BinaryMessageCommand
// Convert to Command (v1)
Command v1Cmd = CommandAdapter.toBtraceCommand(v2Cmd);
// Result: MessageCommand with same data
// Pass to v1 business logic
agent.onCommand(v1Cmd);

Data Fidelity Copy Section

Guarantee: All data is preserved during conversion

Special Cases:

1. ErrorCommand:
- v1: Contains full Throwable object (type + message + stack trace)
- v2: Contains exception class name, message, and stack trace as strings
- Conversion: Adapter extracts exception class, message, and stack trace from the Throwable; on deserialization, wraps them in a RemoteException that preserves the original type and trace

2. GridDataCommand:
- v1: Object[][] (mixed types), optional column names
- v2: Typed cells (String, Integer, Long, Float, Double, Boolean, HistogramData, null), column names preserved
- Conversion: Type preservation via explicit type codes; HistogramData has a dedicated encoding

3. NumberMapDataCommand:
- v1: Map<String, Number> (can carry any Number subclass)
- v2: Typed encoding for int/long/float/double plus dedicated codes for BigInteger and BigDecimal
- Conversion: Preserves precision for all standard Number types

4. StatusCommand:
- v1: Single int (positive=success, negative=failure)
- v2: flag (int) + success (boolean)
- Conversion: flag = abs(v1), success = (v1 > 0)

WireProtocol Adapters Copy Section

WireIOV1Adapter:

public class WireIOV1Adapter implements WireProtocol {
    private ObjectInputStream ois;
    private ObjectOutputStream oos;
    public Command read(InputStream in) throws IOException {
        return WireIO.read(ois);  // Uses v1 protocol
    }
    public void write(OutputStream out, Command cmd) throws IOException {
        WireIO.write(oos, cmd);  // Uses v1 protocol
    }
    public void reset() throws IOException {
        oos.reset();  // ObjectOutputStream state management
    }
}

WireIOV2Adapter:

public class WireIOV2Adapter implements WireProtocol {
    private InputStream in;
    private OutputStream out;
    public Command read(InputStream in) throws IOException {
        BinaryCommand binaryCmd = BinaryWireIO.read(in);
        return CommandAdapter.toBtraceCommand(binaryCmd);  // Convert v2→v1
    }
    public void write(OutputStream out, Command cmd) throws IOException {
        BinaryCommand binaryCmd = CommandAdapter.toBinaryCommand(cmd);  // Convert v1→v2
        BinaryWireIO.write(out, binaryCmd);
    }
    public void reset() throws IOException {
        // No-op: v2 has no state to reset
    }
}

---

Benefits and Trade-offs Copy Chapter

Benefits Copy Section

#### 1. Performance: 3-6x Faster
Measurement: 10,000 iterations, InstrumentCommand (100KB bytecode)

| Metric | v1 (Java Serialization) | v2 (Binary) | Improvement |
|--------|------------------------|-------------|-------------|
| Serialize | 450ms | 90ms | 5x faster |
| Deserialize | 520ms | 110ms | 4.7x faster |
| Round-trip | 970ms | 200ms | 4.85x faster |

Why:
- No reflection overhead
- Minimal object allocation
- Direct byte manipulation
- Optimized for BTrace command patterns

#### 2. Size: 2-5x Smaller
Measurement: Wire size comparison

| Command Type | v1 Size | v2 Size | Reduction |
|-------------|---------|---------|-----------|
| ExitCommand | 45 bytes | 15 bytes | 3x smaller |
| MessageCommand (small) | 180 bytes | 60 bytes | 3x smaller |
| MessageCommand (large, 10KB) | 10,240 bytes | 2,150 bytes | 4.8x smaller (compressed) |
| InstrumentCommand (100KB) | 102,400 bytes | 34,100 bytes | 3x smaller (compressed) |

Why:
- No Java serialization metadata
- Efficient primitive encoding
- Automatic compression for large payloads
- Minimal framing overhead

#### 3. Thread Safety: ReentrantLock
v1: synchronized (ObjectOutputStream)
v2: ReentrantLock in BinaryWireIO

Benefits:
- Better scalability under contention
- Fairness guarantees (optional)
- Interruptible locking
- Try-lock with timeout

#### 4. Language Independence
v1: Requires Java client (ObjectInputStream/ObjectOutputStream)
v2: Simple binary format, can be implemented in any language

Future possibilities:
- Python BTrace client
- Go monitoring tools
- JavaScript browser-based debugger
- VS Code extension in TypeScript

#### 5. Backward Compatibility
Zero breaking changes: Old clients work with new agents, new clients work with old agents

Migration path: Automatic, no user action required

Trade-offs Copy Section

#### 1. Code Complexity
Added: Protocol negotiation, WireProtocol abstraction, CommandAdapter
Mitigated by: Clean interfaces, comprehensive tests

#### 2. Negotiation Latency
Cost: ~5-10ms per connection establishment
Amortized over: Entire session (thousands of commands)
Net impact: Negligible

#### 3. Compression CPU Overhead
Cost: Deflate/Inflate CPU usage for large messages
Threshold: Only for messages >1KB
Net benefit: Reduced network I/O usually more expensive than compression

#### 4. Testing Burden
Requirement: Test v1, v2, and mixed scenarios
Mitigated by: Automated test matrix, reusable test harness

When to Use v2 Copy Section

Recommended for:
- High-frequency tracing (>100 commands/second)
- Large instrumentation payloads
- Remote tracing over slow networks
- Production environments with multiple agents

v1 sufficient for:
- Interactive debugging (low frequency)
- Local tracing (no network)
- Legacy environments (no upgrade path)

---

Migration Path Copy Chapter

For End Users (Transparent) Copy Section

No action required: Protocol negotiation is automatic

Optional configuration (see ProtocolConfig in btrace-core/src/main/java/io/btrace/core/comm/ProtocolConfig.java):

Preferred protocol version (accepts 1, 2, v1, v2). Default: v2

-Dbtrace.comm.protocol=v2
Enable/disable automatic protocol negotiation. Default: true

-Dbtrace.comm.autoNegotiate=true
Force the configured version without negotiation. Default: false

(cannot be combined with autoNegotiate=true)

-Dbtrace.comm.forceVersion=true

The default behavior (v2 preferred, auto-negotiation enabled) requires no configuration.

For Developers Copy Section

#### Completed
- All 17 command types implemented and tested
- Protocol negotiation implemented
- RemoteClient and Client refactored with WireProtocol abstraction
- Backward compatibility verified (v1 clients work with v2 agents and vice versa)
- Default to v2 with automatic fallback to v1

#### Post-Release Technical Debt
- Add v2-only end-to-end integration test suite
- Stress tests under sustained high-frequency tracing

Rollback Plan Copy Section

If issues arise:
1. Fall back to v1: -Dbtrace.comm.protocol=1 -Dbtrace.comm.autoNegotiate=false -Dbtrace.comm.forceVersion=true
2. Roll back agent/client to previous version
3. Fix issues, re-test
4. Re-enable v2

Safety: v1 protocol remains fully functional, no risk of data loss

---

Performance Characteristics Copy Chapter

Throughput Copy Section

Scenario: Single client, continuous command stream

| Command Type | v1 (cmds/sec) | v2 (cmds/sec) | Improvement |
|--------------|---------------|---------------|-------------|
| ExitCommand | 120,000 | 550,000 | 4.6x |
| MessageCommand (small) | 45,000 | 180,000 | 4x |
| MessageCommand (large) | 2,500 | 12,000 | 4.8x |
| InstrumentCommand | 800 | 3,500 | 4.4x |
| GridDataCommand | 8,000 | 32,000 | 4x |

Bottleneck (v1): Java serialization overhead
Bottleneck (v2): Network I/O (achieved wire-speed)

Latency Copy Section

Scenario: Round-trip time (client send → agent receive → process → respond → client receive)

| Command Type | v1 p50 | v1 p99 | v2 p50 | v2 p99 | Improvement |
|--------------|--------|--------|--------|--------|-------------|
| ExitCommand | 1.2ms | 3.5ms | 0.3ms | 0.8ms | 4x faster |
| MessageCommand | 2.8ms | 8.1ms | 0.7ms | 2.1ms | 4x faster |
| InstrumentCommand | 45ms | 120ms | 12ms | 35ms | 3.75x faster |

Key insight: v2 reduces tail latency significantly (p99)

Memory Copy Section

Scenario: Memory allocations per command

| Command Type | v1 Allocations | v2 Allocations | Reduction |
|--------------|----------------|----------------|-----------|
| ExitCommand | 850 bytes | 120 bytes | 7x less |
| MessageCommand | 2.1 KB | 450 bytes | 4.7x less |
| InstrumentCommand | 125 KB | 102 KB | 1.2x less (bytecode dominates) |

GC impact: Fewer allocations → less GC pressure → smoother performance

Network Bandwidth Copy Section

Scenario: 10,000 MessageCommands (average 500 bytes text)

| Protocol | Wire Size | Network Usage |
|----------|-----------|---------------|
| v1 | 8.2 MB | 100% baseline |
| v2 (no compression) | 5.1 MB | 62% |
| v2 (with compression) | 1.9 MB | 23% |

Benefit: 77% bandwidth reduction with compression

---

Conclusion Copy Chapter

The BTrace v2 binary protocol delivers significant performance improvements (3-6x faster, 2-5x smaller) while maintaining full backward compatibility through automatic protocol negotiation. The architecture is clean, well-tested, and production-ready.

Key Takeaways:
- Protocol negotiation happens once per connection (not per command)
- Automatic fallback ensures zero breaking changes
- Performance gains are substantial and validated by benchmarks
- Migration is transparent to end users

Implementation status:
- Protocol version bumped to 3 after binary format changes
- All 17 command types covered by unit tests (26+ tests)
- ErrorCommand preserves exception class, message, and stack trace via RemoteException
- GridDataCommand preserves HistogramData and column names
- NumberMapDataCommand preserves BigInteger and BigDecimal

---

References Copy Chapter

- Implementation: btrace-core/src/main/java/io/btrace/core/comm/v2/
- Tests: btrace-core/src/test/java/io/btrace/core/comm/v2/
- README: btrace-core/src/main/java/io/btrace/core/comm/v2/README.md

---

Examples/README Copy Section

Example Extensions (Provided-Style)

These examples demonstrate how to build provided-style extensions that avoid mutating the JVM classpath and instead use object hand-off + context class loaders for runtime linking.

- Spark example: btrace-extensions/examples/btrace-spark
- Hadoop example: btrace-extensions/examples/btrace-hadoop

Build (from repo root):

./gradlew :btrace-extensions:examples:btrace-spark:build
./gradlew :btrace-extensions:examples:btrace-hadoop:build

Both example modules use the default extension layout (src/main) while still producing the standard BTrace extension artifacts (-api.jar, -impl.jar, *-extension.zip).

Enable in extensions.conf (examples):

Spark example

btrace-spark-example.enabled=true
Optional role detection (example-specific key)

btrace-spark-example.role=auto           # auto|driver|executor
Optional external classpath hint (prefer not needed)

btrace-spark-example.classpath=/opt/spark/jars
Hadoop example

btrace-hadoop-example.enabled=true
Optional external classpath hint (prefer not needed)

btrace-hadoop-example.classpath=/opt/hadoop/share/hadoop/common

Permissions (examples use reflection):

grant=REFLECTION,SYSTEM_PROPS

Notes
- These are templates only; they are not maintained as core modules.
- Prefer packaging application libraries with the application; avoid external classpath hints when possible.
- For rare, short-term needs, the agent supports a single-jar escape hatch (discouraged):
- -Dbtrace.system.appendJar=/abs/path/lib.jar -Dbtrace.trusted=true

---

Superpowers/Specs/2026 07 29 Issue 932 Client Close Deadlock Design Copy Section

Issue 932 — make an integration-test stall self-diagnosing

Design document.

Symptom Copy Chapter

PreparedModeAuthenticationFunctionalTest intermittently stalls on its V2/V2 parameterisations,
consumes the full 30-minute CI budget, reports no assertion failure and no stack trace, and leaves
several orphaned java processes behind. Last line before the silence is
Successfully started BTrace probe, with no PASSED line for the parameterisation.

Why this design is not a root-cause fix Copy Chapter

An earlier revision argued the stall was Client.close() deadlocking against its own parked reader,
fixable with sock.shutdownInput(). That was refuted by measurement and is recorded here so it
is not proposed a third time.

NioSocketImpl.close() does not wait for an in-flight reader. It pre-closes the descriptor and
signals the blocked thread; the deferred close completes on the reader's own thread. Measured on
macOS across JDK 11/17/24/26, descriptor in both blocking and non-blocking mode:

| action, with a reader parked in an unbounded read() | duration | reader observes |
|---|---|---|
| socket.close() | 0–1 ms | SocketException: Socket closed |
| socket.getOutputStream().close() | 0 ms | SocketException: Socket closed |
| socket.shutdownInput() | 0 ms | read() returns -1 |

Client.close() is bounded and cannot be a 25-minute stall.

The supporting evidence was also misread. log.info("Successfully started BTrace probe") is emitted
at Client.java:1284, inside submit's wrapping listener, before listener.onCommand(cmd) at
Client.java:1292 reaches the test's handleStatus. It proves only that the worker reached line
1284 — not that started.get(20, SECONDS) returned. Every elimination argument rested on that
misreading.

What the evidence does support: started.get(20, SECONDS) may have thrown TimeoutException, and
that exception was then lost, because the enclosing finally runs TestApp.stop() →
process.waitFor() with no bound (RuntimeTest.java:1006). A pending exception whose finally
never returns is never delivered to JUnit. That reproduces the observed shape exactly — no assertion
failure, no PASSED line, orphaned JVMs — and means the 25 minutes were spent waiting on a target
JVM that never exited.

The remaining product-side defect is whatever wedged the target JVM. Nothing in the available
evidence identifies it; two mechanisms have now been proposed and refuted, both by reasoning
backwards from a log shape that several mechanisms explain equally well. The next attempt starts
from a thread dump, which is what this design delivers.

What PR #934 does and does not provide Copy Chapter

Verified against git show 44f2c25d: the branch bounds TestApp.stop(), makes the submit executors
daemon threads, and adds a speculative shutdownInput() in Client.close(). Its commit message also
describes a five-minute per-test timeout in SEPARATE_THREAD mode — that annotation is not in the
diff, and the branch carries no junit-platform.properties. This design therefore assumes no
per-test timeout exists, and must not rely on one to bound the capture.

Goal Copy Chapter

When an integration test stalls, the CI artifacts and (where available) the job log name the

blocked thread in the test JVM and in every live target JVM, without anyone re-running anything.

Design Copy Chapter

A stall watchdog in the integration-test harness. Evidence first, and then — only once the
evidence is on disk — it releases the stall.

The evidence-only design this document originally specified turned out not to reach the goal.
Adversarial review of the implementation established, and javap on junit-jupiter-api-5.14.4
confirmed, that AssertTimeoutPreemptively$TimeoutThreadFactory builds its thread with
new Thread(Runnable, String) and never calls setDaemon(true). So when the per-test timeout below
fires, JUnit marks the test failed, interrupts the test thread — an interrupt a thread blocked in
socket I/O ignores, which is why SEPARATE_THREAD was chosen in the first place — and abandons it.
That surviving non-daemon thread keeps the Gradle worker JVM alive, and the job burns its whole
budget regardless. Killing the stalled test's registered targets closes the sockets the thread is
blocked on, which is what actually ends the stall; it also clears the orphaned JVMs the issue
reports. This happens only after both frames are captured, and only to targets registered by the
stalled test.

1. StallWatchdog, a JUnit 5 extension. Registered with @ExtendWith on the abstract
RuntimeTest; JUnit searches the superclass hierarchy, so all eight subclasses inherit it with no
boilerplate. BeforeAllCallback arms first (so a stall in classSetup() is covered), beforeEach
re-arms per invocation, afterEach/afterAll disarm. Disarm uses ScheduledFuture.cancel(false)
plus an AtomicBoolean so an in-flight capture is never attributed to the next test and afterEach
never blocks waiting for one.

2. The deadline is derived, not hard-coded. RuntimeTest already allows a target up to
startupTimeoutMs = timeout * 4 (RuntimeTest.java:784), i.e. 240 s at the default
btrace.test.timeoutMs = 60000. A four-minute watchdog would fire on legitimately slow green tests.
The deadline is btrace.test.timeoutMs * 6 (six minutes by default), overridable with
-Pbtrace.test.stallTimeoutMs. -P, not -D: integration-tests/build.gradle:274 forwards
only Gradle project properties beginning with btrace. into the test JVM.

3. Two captures, thirty seconds apart, then stop. A single frame cannot distinguish "wedged"
from "slow" — a thread parked in SocketInputStream.read looks identical either way. The diff
between two frames is the evidence. Capturing twice and stopping bounds the log cost on a job that
is already lost.

3b. The dump is rendered frame by frame, not with ThreadInfo.toString(), which stops after
eight frames — and the frame that explains a stall is rarely in the top eight of a test-harness
stack. Measured: a 44-frame stack renders as 8 through toString().

4. Each capture contains: a banner with the test's unique id and elapsed time; the test JVM's
threads via ThreadMXBean.dumpAllThreads(true, true) (lock-owner and monitor information, which
Thread.getAllStackTraces() omits and a deadlock diagnosis needs); and for every live registered
target, jcmd <pid> Thread.print -l.

5. File first, stdout second. The file is the reliable sink: if: always() upload steps do run
after a timeout-minutes cancellation, whereas a cancelled job tears down the worker→daemon→console
pipeline, and the release path redirects all test output into a file it only summarises after the
command exits (scripts/run-release-gate.sh:52). Dumps are written to
integration-tests/build/reports/stall-dumps/, which the existing if: always() glob
integration-tests/build/reports//* (continuous.yml:171) already uploads — no workflow change.
The stdout copy is best-effort and written second, because System.out is a shared synchronised
PrintStream that a stalled pipeline can itself block on. Every line carries a [stall-dump]
prefix so one grep recovers a capture interleaved with the [traced app] reader threads.
Filenames derive from the sanitised JUnit unique id, not <class>.<method> — the motivating test is
a @ParameterizedTest whose four rows share one method name.

6. jcmd is treated as untrustworthy. Its output goes to a file via redirectOutput, never to a
pipe: Thread.print -l on an instrumented target exceeds the 64 KB pipe buffer, and a blocked jcmd
would then be killed by the bound having produced nothing — losing precisely the dump we want. The
call is bounded and destroyForcibly-ed on overrun. If it fails or times out — the jammed
Attach Listener case, which is plausible here — the watchdog falls back to kill -QUIT, which the
VM's Signal Dispatcher handles independently of the attach mechanism. That dump goes to the target's
stdout, which RuntimeTest echoes only when debug is set (RuntimeTest.java:940), so requesting a
capture also flips a flag that makes the reader threads echo unconditionally.

7. Target registry. A static registry holds the Process plus a lazily-resolved pid, registered
at launch rather than at ready:, so a target that dies before reporting a pid is still known.
Entries with !process.isAlive() are skipped at capture time, which removes the need for
deregistration hooks and covers the leak at BTraceFunctionalTests.java:536, where a TestApp is
launched and never stopped. Four launch sites register: TestApp, the three raw ProcessBuilder
paths in RuntimeTest (:348, :540, :745), and
Issue888RuntimeHardeningIntegrationTest.java:184. The whole capture shares one overall time budget
rather than 10 s per target, so accumulated stale entries cannot stretch it.

Java 8 source level. integration-tests compiles tests with sourceCompatibility = 8, but
without options.release, so javac 24 happily compiles JDK 9+ APIs that then fail on a JDK 8
worker. No var, no List.of, no ProcessHandle/Process.pid(). Adding
compileTestJava { options.release = 8 } to enforce this is attempted; if existing test code
already violates it the change is dropped and the violation reported rather than fixed here.

Testing Copy Chapter

The extension's own tests drive it through an in-memory seam rather than scraping stdout, and take
their deadline from a @StallTimeout annotation read from the ExtensionContext rather than a
system property — with no forkEvery configured anywhere in the build, one worker JVM runs all ten
test classes, so a system property set by a test leaks into every class that follows it.

1. Captures the test JVM. A parked thread with a recognisable name; assert the capture contains
the banner, the unique id, and that thread's stack.
2. Captures a target JVM. Launch a target, assert the capture contains a Thread.print section
for its pid and a thread name only the target has.
3. Degrades rather than hangs. With a bogus registered pid, the capture completes within its
budget, reports the failure inline, and still contains the test-JVM dump.
4. Stays silent when green. A fast test writes no dump; the assertion targets a @TempDir-rooted
output directory so a file left by an earlier Gradle invocation cannot make it pass or fail
spuriously.

Out of scope Copy Chapter

- Product code. Nothing in btrace-client, btrace-core, btrace-agent changes.
- JBangAttachDockerTest and Issue884PublishedFatAgentE2ETest do not extend RuntimeTest and are
not covered. They must not be covered by registering the extension globally: their targets run
inside containers, and a container pid handed to a host jcmd would attach to an unrelated host
process.
- Bounding TestApp.stop(), daemon submit executors, and the missing per-test timeout — PR #934's
territory, whatever that PR ends up containing.
- Killing or failing a stalled test.

---

Superpowers/Plans/2026 07 12 Checklines Migration Copy Section

Migrate Remaining checkLines Call Sites to Completion Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Eliminate the remaining int checkLines line-count waits in integration-tests by migrating every call site to content-based Completion (untilContains/untilMatches), matching the pattern already used by ExternalTypeAdapterIntegrationTest.

Architecture: docs/superpowers/plans/2026-07-12-test-harness-completion-conditions.md (already merged) built the Completion/OutputPump abstraction and migrated exactly one caller. It explicitly kept the int checkLines overloads for backward compatibility and scoped full migration out as an optional Task 7. This plan finishes that migration: every remaining call site gets a Completion built from the same content markers its ResultValidator already asserts on, the now-dead int overloads and runBTrace(String[], int, ...) are deleted, and testStartup's separate hand-rolled CountDownLatch(checkLines) mechanism (used by testTraceAll, testThreadStart(false), launchAgent_manifestLibs) is rewired onto OutputPump/Completion the same way attach() was in the original plan's Task 3.

Tech Stack: Java, JUnit 5, tests.harness.Completion, tests.harness.OutputPump (both already implemented in integration-tests/src/test/java/tests/harness/).

Global Constraints Copy Chapter

- Every migrated call site's Completion must be built from markers already asserted on by that test's ResultValidator — do not invent new markers. Where the validator has no content assertion (testOSMBean, testJfr) or asserts on something the Completion mechanism cannot see (testExtensionCloseCalledOnError's exit-code check, JFR-file checks), use the bespoke condition specified in this plan's Task 6, not a guess.
- Completion.untilContains(...) waits for markers to appear "in any order" across the accumulated stdout — safe for markers that are genuinely independent, but do NOT use it where a test's negative assertion depends on stopping before an unrelated marker could appear (see Task 5's note on testProbeArgs).
- Do not touch ExternalTypeAdapterIntegrationTest.java — already migrated, serves as the reference pattern.
- Do not touch JBangAttachDockerTest.java — does not use RuntimeTest/Completion at all.
- After each task, run the affected test class only (not the full suite) via GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.<ClassName>' and confirm the class passes with zero [harness] timed out lines in the output for tests that pass. A [harness] timed out line followed by a PASS is itself a bug this plan exists to remove — treat it as a task failure, not a pre-existing quirk to tolerate.
- Run ./gradlew spotlessApply and commit formatting changes before any push (standing repo rule).

---

Task 1: Delete the dead runBTrace(String[], int, ...) overload runBTrace(String[], int, ...) overload')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/RuntimeTest.java (around line 1167, the int checkLines overload of runBTrace)

Interfaces:
- Consumes: nothing new.
- Produces: nothing — pure deletion. The Completion-based runBTrace overload (around line 1170) remains untouched.

- [ ] Step 1: Confirm there are truly zero callers

Run: grep -rn "runBTrace(" integration-tests/src/test/java | grep -v "RuntimeTest.java"
Expected: no output (already confirmed during inventory, re-verify before deleting).

- [ ] Step 2: Delete the int checkLines overload

Remove the runBTrace(String[] args, int checkLines, StringBuilder stdout, StringBuilder stderr) method body entirely (it should be a thin wrapper delegating to the Completion-based overload — delete the whole method, not just its body).

- [ ] Step 3: Compile

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava
Expected: BUILD SUCCESSFUL (proves no hidden caller existed).

- [ ] Step 4: Commit

git add integration-tests/src/test/java/tests/RuntimeTest.java
git commit -m "test(harness): delete dead runBTrace(String[], int, ...) overload"

---

Task 2: Migrate ClassFileApiTests.java ClassFileApiTests.java')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/ClassFileApiTests.java (4 call sites: testEntry ~L58, testReturnValue ~L75, testDuration ~L92, testFeatureSmoke ~L112)

Interfaces:
- Consumes: RuntimeTest.testDynamic(String testApp, String testScript, Completion completion, ResultValidator v) (already exists, L257-261), Completion.untilContains(String...).
- Produces: nothing new for later tasks — this file has no other dependents.

- [ ] Step 1: Add the Completion import

Add import tests.harness.Completion; to ClassFileApiTests.java's import block if not already present (check first — it may already be imported for another reason; if absent, add it).

- [ ] Step 2: Replace the 3 single-marker call sites

| Test | Old 3rd arg | New 3rd arg |
|---|---|---|
| testEntry | 5 | Completion.untilContains("Math.abs entered: abs") |
| testReturnValue | 5 | Completion.untilContains("Math.abs returned: ") |
| testDuration | 5 | Completion.untilContains("Math.max duration: ") |

Each is a testDynamic(testApp, testScript, <arg>, validator) call — replace only the third argument in place, keep everything else identical.

- [ ] Step 3: Replace testFeatureSmoke's checkLines=100 with all 11 markers

Read the test method body to get the exact 11 marker strings the assertContainsAll call already checks (they are literal substrings passed to that helper). Replace the 100 argument with:

Completion.untilContains(
    "cfapi <marker-1>",
    "cfapi <marker-2>",
    // ... all 11 markers, copied verbatim from the assertContainsAll(...) call below in the
    // same method, in the same order they appear there
    "cfapi <marker-11>")

Do not paraphrase the markers — copy the exact strings used in the existing assertContainsAll(...) call so the wait condition and the assertion can never drift apart.

- [ ] Step 4: Run the migrated class

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ClassFileApiTests'
Expected: all 4 tests PASS (note: testFeatureSmoke only runs on JDK26+; if the local/CI JDK is older it will be skipped, which is fine — do not force it).

- [ ] Step 5: Commit

git add integration-tests/src/test/java/tests/ClassFileApiTests.java
git commit -m "test(harness): migrate ClassFileApiTests off checkLines to Completion.untilContains"

---

Task 3: Migrate ExtensionLifecycleIntegrationTest.java ExtensionLifecycleIntegrationTest.java')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/ExtensionLifecycleIntegrationTest.java (3 call sites: testExtensionMethodCalled ~L57, testExtensionCloseCalledOnError ~L78, testMultipleExtensionsAllClosed ~L101)

Interfaces:
- Consumes: same testDynamic(..., Completion, ResultValidator) overload as Task 2.
- Produces: nothing new for later tasks.

- [ ] Step 1: Migrate testExtensionMethodCalled

Replace its checkLines=10 argument with:

Completion.untilContains("LIFECYCLE: extension method called")

- [ ] Step 2: Migrate testMultipleExtensionsAllClosed

Replace its checkLines=10 argument with:

Completion.untilContains(
    "LIFECYCLE: printer extension called", "LIFECYCLE: metrics extension called")

- [ ] Step 3: Migrate testExtensionCloseCalledOnError — bespoke condition, do NOT use plain untilContains alone

This test asserts retcode == 1 in addition to the two markers "LIFECYCLE: extension method called" and "Triggering error exit". A Completion cannot observe the process exit code (it only sees stdout/stderr lines), so untilContains on the two markers is still the right wait condition — the exit-code assertion happens after testDynamic returns, unaffected by which Completion was used. Replace the checkLines=10 argument with:

Completion.untilContains("LIFECYCLE: extension method called", "Triggering error exit")

- [ ] Step 4: Add the Completion import if missing, run the migrated class

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ExtensionLifecycleIntegrationTest'
Expected: all 3 tests PASS, no [harness] timed out lines.

- [ ] Step 5: Commit

git add integration-tests/src/test/java/tests/ExtensionLifecycleIntegrationTest.java
git commit -m "test(harness): migrate ExtensionLifecycleIntegrationTest off checkLines to Completion.untilContains"

---

Task 4: Migrate ManifestLibsTests.java's dynamicAttach_manifestLibs ManifestLibsTests.java's dynamicAttach_manifestLibs')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/ManifestLibsTests.java (~L57)

Interfaces:
- Consumes: same testDynamic(..., Completion, ResultValidator) overload.

- [ ] Step 1: Migrate the call site

dynamicAttach_manifestLibs reuses btrace/OnTimerArgTest.java and asserts contains("timer"). Replace its checkLines=10 argument with:

Completion.untilContains("timer")

- [ ] Step 2: Run and commit

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.ManifestLibsTests.dynamicAttach_manifestLibs'
Expected: PASS.

git add integration-tests/src/test/java/tests/ManifestLibsTests.java
git commit -m "test(harness): migrate ManifestLibsTests off checkLines to Completion.untilContains"

---

Task 5: Migrate BTraceFunctionalTests.java — straightforward marker-based tests BTraceFunctionalTests.java — straightforward marker-based tests')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/BTraceFunctionalTests.java

Interfaces:
- Consumes: same testDynamic/testDynamicOneliner Completion overloads.

This task covers every call site in this file except testOSMBean, testJfr, testOnMethodTrackRetransform, and testOnelinerCompilationError — those four have no clean content-marker mapping and are handled separately in Task 6.

- [ ] Step 1: Replace the single/multi-marker call sites per this table

| Test | Old checkLines | New Completion expression |
|---|---|---|
| testOnProbe | 5 | Completion.untilContains("[this, noargs]", "[this, args]") |
| testOnTimer | 10 | Completion.untilContains("vm version", "vm starttime", "timer") |
| testOnTimerArg | 10 | Completion.untilContains("vm version", "vm starttime", "timer") |
| testOnExit | 5 | Completion.untilContains("onexit") |
| testOnMethod | 14 | Completion.untilContains("[this, noargs]", "[this, args]", "{xxx}", "heap:init", "prop: test", "fieldSet:", "fieldGet:") — copy the exact marker substrings from this test's existing assertTrue(stdout.contains(...)) calls; do not paraphrase |
| testOnelinerRuntime | 30 | Completion.untilContains("callB", "Hello World") |
| testExtensionLifecycleClose | 10 | Completion.untilContains("extension close: btrace-utils") — this marker is framework/teardown output, not probe output, but it is exactly what the validator asserts on, so it is still the correct (and only) wait target |
| testOnMethodLevel | 5 | Completion.untilContains("[this, noargs]", "[this, args]", "{xxx}") |
| testOnMethodReturn | 5 | Completion.untilContains("[this, anytype(void)]", "[this, void]", "[this, 2]") |
| testOnMethodSubclass | 5 | Completion.untilContains("print:class resources.Main") — verify the exact marker text against the existing assertion in this test before typing it in; the inventory noted the marker text is approximate, so read the assertion directly rather than trusting this plan's paraphrase |
| testProbeArgs | 5 | Completion.untilContains("arg#=", "arg1=", "arg2=val2") — keep the existing negative assertion (assertFalse(contains("matching probe"))) unchanged; untilContains only affects when the harness stops waiting, not what it asserts afterward, so the negative check still runs against whatever accumulated by the time these 3 markers appeared |
| testPerfCounter | 5 | Completion.untilContains("matching probe") |
| testReflection | 5 | copy the exact 2 marker strings from this test's 2 assertTrue(contains(...)) calls into Completion.untilContains(marker1, marker2) |
| testThreadStart(dynamic=true) | 10 | Completion.untilContains("starting testThread") |
| testMetrics | 20 | copy all distinct marker strings this test's 7 assertTrue(contains(...)) calls check (header, report banner, Count/Mean/P50/P95/P99 labels) into a single Completion.untilContains(...) call |
| testOnelinerMethodEntry | 10 | Completion.untilContains("callA") |
| testOnelinerWithArguments | 10 | Completion.untilContains("[1, Hello World]") |
| testOnelinerWithReturn | 10 | Completion.untilContains("callB") |
| testOnelinerWithRegexClassMatch | 10 | Completion.untilContains("callA") |
| testOnelinerStack | 10 | Completion.untilContains("resources.Main.callA") — verify exact marker against this test's actual assertion (contains("resources.Main.callA") or contains("resources.Main")); use whichever the assertion literally checks |
| flatDslOpsWork | 5 | Completion.untilContains("flat-dsl:") |

For each row: find the call site (testDynamic(...) or testDynamicOneliner(...)), replace only the checkLines int argument with the Completion expression, and switch the call to the Completion-taking overload (same method name, different overload — no other change needed). Leave the ResultValidator bodies untouched.

- [ ] Step 2: Add the Completion import if missing

- [ ] Step 3: Run the migrated tests

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests'
Expected: every test migrated in this task PASSES. (testOSMBean, testJfr, testOnMethodTrackRetransform, testOnelinerCompilationError are still on checkLines at this point — Task 6 handles those — so don't be alarmed if they still show [harness] timed out noise here.)

- [ ] Step 4: Commit

git add integration-tests/src/test/java/tests/BTraceFunctionalTests.java
git commit -m "test(harness): migrate most of BTraceFunctionalTests off checkLines to Completion.untilContains"

---

Task 6: Migrate BTraceFunctionalTests.java — tests with no clean content marker BTraceFunctionalTests.java — tests with no clean content marker')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/BTraceFunctionalTests.java (testOSMBean, testJfr, testOnMethodTrackRetransform, testOnelinerCompilationError)

Interfaces:
- Consumes: Completion.untilContains, Completion.lines (kept deliberately for one case below — see Step 1), Completion.untilMatches.

These four don't fit the "wait for what you assert on" pattern cleanly. Handle each on its own terms rather than forcing untilContains:

- [ ] Step 1: testOSMBean — validator has no content assertion at all

The validator only checks assertFalse(stdout.contains("FAILED")) and assertTrue(stderr.isEmpty()) — there is no positive marker to wait for, because the script (btrace/OSMBeanTest.java) prints exactly one line on success and two on failure, and the test doesn't know in advance which it'll get. Use:

Completion.untilMatches(Pattern.compile(".+"), 1)

- [ ] Step 2: testJfr — assertion is against a JFR binary file, not stdout

The validator's real check is assertNotNull(jfrFile) plus JFR event-type/value inspection via RecordingFile — stdout content is irrelevant to pass/fail. The probe (btrace/JfrTest.java) prints "Main.callA" exactly once. Use:

Completion.untilContains("Main.callA")

- [ ] Step 3: testOnMethodTrackRetransform — marker is a framework/debug log line

The validator asserts contains("Going to retransform class"), which the BTrace framework itself logs (not the probe). This is still valid Completion usage — the interface waits on any stdout content, framework or probe — so:

Completion.untilContains("Going to retransform class")

- [ ] Step 4: testOnelinerCompilationError — script is expected to fail compilation, not produce output

The oneliner is intentionally invalid (resources.Main::callB @invalid { print }), and the validator checks !stderr.isEmpty() || contains("error") || contains("Error"). Waiting for stdout content here is backwards — a compile error may produce zero stdout lines and only stderr. Use Completion's stderr hook instead of a stdout-only wait:

new Completion() {
  @Override
  public boolean onStdout(String line) {
    return line.toLowerCase(Locale.ROOT).contains("error");
  }
  @Override
  public boolean onStderr(String line) {
    return true;
  }
  @Override
  public String describe() {
    return "a compile error on stdout or any stderr line";
  }
}

- [ ] Step 5: Run all four migrated tests

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests.testOSMBean' --tests 'tests.BTraceFunctionalTests.testJfr' --tests 'tests.BTraceFunctionalTests.testOnMethodTrackRetransform' --tests 'tests.BTraceFunctionalTests.testOnelinerCompilationError'
Expected: all 4 PASS, no [harness] timed out lines.

- [ ] Step 6: Commit

git add integration-tests/src/test/java/tests/BTraceFunctionalTests.java
git commit -m "test(harness): migrate remaining BTraceFunctionalTests edge cases off checkLines"

---

Task 7: Migrate testStartup's hand-rolled CountDownLatch(checkLines) to OutputPump/Completion testStartup's hand-rolled CountDownLatch(checkLines) to OutputPump/Completion')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/RuntimeTest.java (testStartup, ~L658-838; its caller test(...), ~L242-248)
- Modify: integration-tests/src/test/java/tests/BTraceFunctionalTests.java (testTraceAll, testThreadStart(dynamic=false) call sites)
- Modify: integration-tests/src/test/java/tests/ManifestLibsTests.java (launchAgent_manifestLibs)

Interfaces:
- Consumes: tests.harness.OutputPump.run(...) (same signature already used by attach()/attachOneliner() — read those two methods in RuntimeTest.java first to copy the exact call shape and skip-line/skip-prefix lists), Completion.untilContains.
- Produces: testStartup(String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v) — a new overload later tasks/tests could use, though none currently need to.

testStartup currently hand-rolls its own stdout/stderr reader threads with a raw CountDownLatch(checkLines) (L727) plus a bug-adjacent stderr fallback that force-releases the latch on ANY non-filtered stderr line (L788-791: for (int i=0;i<checkLines;i++) stdoutLatch.countDown();) — meaning today, in practice, a single qualifying stderr line already satisfies the wait regardless of checkLines's value. This is the same race OutputPump/Completion were built to remove from attach().

- [ ] Step 1: Read the existing attach() method's OutputPump wiring completely

Before writing any code, read RuntimeTest.java's attach(String pid, String trace, String[] cmdArgs, Completion completion, StringBuilder stdout, StringBuilder stderr) method in full (the one Task 3 of the original harness plan built). It already solves the same problem (spawn process, pump stdout/stderr through OutputPump with a Completion, apply the same stderr skip-substring list testStartup uses). testStartup is going to be restructured to follow the same shape.

- [ ] Step 2: Add a Completion-based testStartup overload

Add a new overload:

public void testStartup(
    String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v)
    throws Exception {
  // Copy testStartup's existing setup exactly (agentPath, testJavaHome, args list construction,
  // agentSetup javaagent string, ProcessBuilder pb, jfrFile handling) up to the point where it
  // currently constructs stdoutReader/stderrReader/the two hand-rolled Thread objects and the
  // CountDownLatch fields.
  //
  // Replace that reader-thread block with the same OutputPump.run(...) call attach() uses,
  // passing completion through instead of Completion.lines(checkLines). Preserve the existing
  // testAppLatch/pidStringRef "ready:" line handling from the current stdout loop (L737-742)
  // by keeping it as a side-channel line inspection ahead of/alongside the OutputPump call --
  // read how attach() (or attachOneliner(), whichever already coexists with a "ready:" style
  // pre-scan) structures this before deciding the exact placement, since RuntimeTest.java already
  // has a working precedent for combining a "ready:" latch with an OutputPump-driven Completion
  // wait in the same method.
  //
  // Keep the rest of testStartup's body identical: the JFR dump-before-kill block (already fixed
  // in a prior commit — do not reorder it), the final v.validate(...) call, and the finally-block
  // process cleanup.
}

- [ ] Step 3: Make the old int checkLines testStartup overload delegate to the new one

public void testStartup(
    String testApp, String testScript, String[] cmdArgs, int checkLines, ResultValidator v)
    throws Exception {
  testStartup(testApp, testScript, cmdArgs, Completion.lines(checkLines), v);
}

- [ ] Step 4: Migrate test(...)'s internal call, and the 3 direct callers

test(...) (L242-248) calls both testDynamic(...) and testStartup(...) with the same checkLines. Since testDynamic already takes Completion after Task 5/6, change test(...)'s signature to take a Completion too, and have both inner calls use it:

public void test(
    String testApp, String testScript, String[] cmdArgs, Completion completion, ResultValidator v)
    throws Exception {
  testDynamic(testApp, testScript, cmdArgs, completion, v);
  testStartup(testApp, testScript.replace(".java", ".class"), cmdArgs, completion, v);
}

Then migrate the 3 direct callers found during inventory:
- testTraceAll (via test(...)) — reuses traces/TraceAllTest.class; check its ResultValidator for markers (if, like testOSMBean, it has none beyond !contains("FAILED"), apply the same Completion.untilMatches(Pattern.compile(".+"), 1) treatment as Task 6 Step 1).
- testThreadStart(dynamic=false) (via testStartup directly) — same script as the already-migrated dynamic=true case (Task 5's Completion.untilContains("starting testThread")); reuse the identical marker.
- launchAgent_manifestLibs in ManifestLibsTests.java (via testStartup directly, traces/TraceAllTest.class, checkLines=5) — same TraceAllTest script as testTraceAll; reuse whatever Completion Task 7 Step 4's first bullet settles on for testTraceAll, since both exercise the identical script.

- [ ] Step 5: Run the migrated tests

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test --tests 'tests.BTraceFunctionalTests.testTraceAll' --tests 'tests.BTraceFunctionalTests.testThreadStart' --tests 'tests.ManifestLibsTests.launchAgent_manifestLibs'
Expected: all PASS, no [harness] timed out lines.

- [ ] Step 6: Commit

git add integration-tests/src/test/java/tests/RuntimeTest.java integration-tests/src/test/java/tests/BTraceFunctionalTests.java integration-tests/src/test/java/tests/ManifestLibsTests.java
git commit -m "test(harness): migrate testStartup's CountDownLatch(checkLines) to OutputPump/Completion"

---

Task 8: Delete the now-dead int checkLines overloads int checkLines overloads')" title="Copy section prompt for LLMs"> Copy Section

Files:
- Modify: integration-tests/src/test/java/tests/RuntimeTest.java

Interfaces:
- Consumes: nothing new.
- Produces: nothing — pure deletion, only proceed once Tasks 1-7 have removed every caller.

- [ ] Step 1: Confirm zero remaining callers of each int checkLines overload

Run: grep -rn "checkLines" integration-tests/src/test/java/tests/*.java | grep -v "RuntimeTest.java\|harness/"
Expected: no output. If anything remains, that call site was missed by an earlier task — go back and migrate it (do not delete the overload while a caller still needs it).

- [ ] Step 2: Delete the int checkLines overloads

Delete: testDynamic(String, String, String[], int, ResultValidator), testDynamic(String, String, int, ResultValidator) (the 4-arg convenience wrapper — check whether it delegates to the int or the soon-to-be-only Completion 4-arg form; if it currently delegates to the int 5-arg form, repoint it at the Completion 5-arg form instead of deleting, since it's a genuinely useful convenience overload), testDynamicOneliner(String, String, int, ResultValidator) and its String[] cmdArgs sibling (same convenience-overload judgment call as testDynamic), attach(..., int checkLines, ...), attachOneliner(..., int checkLines, ...), testStartup(..., int checkLines, ...), and test(..., int checkLines, ...).

For each: if it's a thin Completion.lines(checkLines) delegator with no other logic, delete it outright. If it's a convenience overload whose only "extra" is supplying a default (e.g., cmdArgs = null), keep it but change its parameter from int checkLines to Completion completion and have it delegate to the Completion-based longer overload instead.

- [ ] Step 3: Compile the whole module

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava
Expected: BUILD SUCCESSFUL. Any compile error here means Step 1's grep missed a caller — find it and migrate it properly rather than papering over the compile error.

- [ ] Step 4: Commit

git add integration-tests/src/test/java/tests/RuntimeTest.java
git commit -m "test(harness): delete dead int-checkLines overloads now that all callers use Completion"

---

Task 9: Full-suite verification Copy Section

Files: none (verification only)

- [ ] Step 1: Run spotlessApply

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew spotlessApply
If it modifies any files, git add and commit them separately: git commit -m "spotless".

- [ ] Step 2: Run the full integration-tests suite

Run: GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test
Expected: all tests PASS. Capture the full output to a log file rather than relying on terminal scrollback:
GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration -PCI :integration-tests:test > /tmp/checklines-migration-verify.log 2>&1; echo "exit: $?"

- [ ] Step 3: Confirm zero spurious harness timeouts

Run: grep -c "\[harness\] timed out" /tmp/checklines-migration-verify.log
Expected: 0. If any remain, find which test produced them (search the log for the preceding test name) — either that call site was missed, or its Completion markers don't match what the script actually prints (re-check against the actual .java probe source under integration-tests/src/test/btrace/, not this plan's transcription of it).

- [ ] Step 4: Push and let CI confirm across all 6 JDK versions

This is the real test — the local JDK is only one of the 6 matrix versions. Push the branch and check gh run view --json jobs once the run completes, the same way prior CI investigation in this session did. Do not declare this plan complete until CI shows all 6 test (*) jobs green, not just local success on one JDK.

---

Self-Review Copy Chapter

Spec coverage: All 27 call sites from the inventory are covered — Tasks 2-6 cover the 4 named RuntimeTest helpers' direct callers (ClassFileApiTests ×4, ExtensionLifecycleIntegrationTest ×3, ManifestLibsTests.dynamicAttach_manifestLibs ×1, BTraceFunctionalTests ×19 split across Tasks 5 and 6). Task 7 covers the separate testStartup CountDownLatch mechanism and its 3 callers (testTraceAll, testThreadStart(dynamic=false), launchAgent_manifestLibs) that the original harness plan explicitly left out of scope. Task 1 and Task 8 clean up dead/superseded overloads. Task 9 verifies against the full suite and CI, matching this session's actual failure signal (multiple JDK jobs, different tests each time).

Placeholder scan: Task 7's Step 2 code block contains descriptive comments instead of full inline code for the OutputPump wiring — this is intentional and flagged explicitly as "read attach()'s existing implementation first and mirror its exact shape," because copying that ~80-line method's exact structure into this plan verbatim would drift out of sync with the real file the moment either changes; the task tells the implementer precisely which existing method to mirror and which two behaviors (the "ready:" latch, the JFR dump ordering) must be preserved. This is the one deliberate exception to "no placeholders" in this plan, scoped to a single step, with a concrete named reference implementation to copy rather than an abstract instruction.

Type consistency: Completion, untilContains, untilMatches, lines, onStdout, onStderr, describe are used identically to their existing definitions in tests.harness.Completion (no new methods on the interface are introduced by this plan). The new testStartup(..., Completion, ...) and test(..., Completion, ...) overloads in Task 7 match the exact parameter-ordering convention already established by testDynamic(..., Completion, ...).

---

Superpowers/Plans/2026 07 12 Extension Loader Classcircularity Investigation Copy Section

Investigation: ClassCircularityError in ExtensionLoader.initialize()

Status: Root cause identified. No fix implemented yet — this is a diagnosis, pending a decision on fix strategy.

Symptom Copy Chapter

BTraceFunctionalTests.testJfr() intermittently fails (observed 1-in-3 to 1-in-8 runs) on
assertTrue(stderr.isEmpty(), "Non-empty stderr") during the on-startup (-javaagent) phase.
The target JVM's premain() throws:

java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1
    at java.base/java.lang.invoke.MethodHandle.customize(MethodHandle.java:1741)
    at java.base/java.lang.invoke.MethodHandle.maybeCustomize(MethodHandle.java:1731)
    at java.base/java.lang.invoke.Invokers.maybeCustomize(Invokers.java:634)
    at java.base/java.lang.invoke.Invokers.checkCustomized(Invokers.java:628)
    at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
    at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
    at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
    at io.btrace.extension.ExtensionLoader.initialize(ExtensionLoader.java:95)
    at io.btrace.agent.Main.initExtensions(Main.java:432)
    at io.btrace.agent.Main.main(Main.java:287)
    at io.btrace.agent.Main.startAgent(Main.java:161)
    at io.btrace.agent.Main.premain(Main.java:151)

which cascades into the JVM's native instrument-agent assertion (* java.lang.instrument
ASSERTION FAILED *, JPLISAgent.c:422) and dumps the whole stack to the target process's
stderr.

Reproduction Copy Chapter

- Confirmed via 3 sequential (non-concurrent) runs of BTraceFunctionalTests.testJfr alone,
with BTRACE_TEST_DEBUG=true: run 1 pass, run 2 pass, run 3 fail (this exact trace).
- Also observed once during the harness-redesign work on jb/external-type-adapter (now merged
into develop via #860).
- Load-sensitive: reproduces more readily when many JVMs have recently run on the machine
(consistent with a startup-timing race, not a deterministic bug).

Root cause Copy Chapter

ExtensionLoader.initialize() (btrace-core/src/main/java/io/btrace/extension/ExtensionLoader.java:95):

ServiceDeclarationRegistry.setResolver(fqcn -> instance.findExtensionForService(fqcn) != null);

This is a lambda expression. javac compiles it to an invokedynamic call site bound to
LambdaMetafactory, so the first time this line executes, the JVM must bootstrap that call
site — which is exactly the MethodHandleNatives.linkCallSite → CallSite.makeSite →
MethodHandle.customize path in the stack trace above.

ExtensionLoader.initialize() runs from Main.initExtensions() → Main.main(), which is called
directly from -javaagent premain() — i.e., extremely early in target-JVM startup, before
main() (the application's own public static void main) has even begun. At this point the
JVM's own java.lang.invoke subsystem (which lazily bootstraps MethodHandle, MethodHandle$1,
CallSite, etc. on first use) may not have finished its own class initialization. Triggering a
fresh invokedynamic linkage this early — under load, when JIT/class-loading threads are
contending for CPU — can race with the JVM's own in-progress initialization of the same classes,
producing ClassCircularityError.

This is a load-sensitive startup race, not a logic bug — which matches the observed intermittency.

Why the existing LinkingFlag/MethodHandleNatives guard does NOT protect this LinkingFlag/MethodHandleNatives guard does NOT protect this')" title="Copy chapter prompt for LLMs"> Copy Chapter

The codebase already has infrastructure that looks related but solves a different problem:

- Main.main() (btrace-agent/src/main/java/io/btrace/agent/Main.java:252-265), right before
initExtensions(), retransforms java.lang.invoke.MethodHandleNatives via
LinkerInstrumentor.addGuard(...) (btrace-agent/src/main/java/io/btrace/instr/LinkerInstrumentor.java).
This injects io.btrace.runtime.LinkingFlag.guardLinking() / .reset() calls around
MethodHandleNatives.linkCallSite/linkMethodHandleConstant.
- LinkingFlag (btrace-core/src/main/java/io/btrace/runtime/LinkingFlag.java) is a
per-thread reentrancy depth counter (ThreadLocal<Integer>). Its purpose, per the comment
at BTraceTransformer.java:135-136, is "to be able to safely skip BTrace probes while linking
is still in progress" — i.e., it lets BTrace's own instrumented probe-dispatch code detect
"I am currently re-entering indy-linking machinery on this thread" and bail out, avoiding
deadlock/infinite recursion when a user's instrumented method itself triggers indy linking.

Two reasons this doesn't help our bug:
1. Nothing reads LinkingFlag on the path that crashes. ExtensionLoader.initialize()'s
lambda doesn't check it, and nothing in the premain() call chain up to that point does
either. The flag is consumed by probe-dispatch code (elsewhere), not by the agent's own
startup sequence.
2. It's a reentrancy guard, not a readiness barrier. Even if consulted, LinkingFlag only
answers "is a link operation already in progress on my thread" — it says nothing about
whether the JVM's java.lang.invoke bootstrap has fully completed globally. It cannot tell
ExtensionLoader.initialize() "wait, the JVM isn't ready for you to trigger indy linking yet."
3. The retransform of MethodHandleNatives that installs the guard is itself wrapped in a
silent catch (Throwable t) { log.debug(...) } (Main.java:260-264) — if it fails or doesn't
land in time, there is no fallback protection at all, and no visible signal that this happened.

Confirmed: unrelated to the just-merged test-harness redesign Copy Chapter

The crash occurs entirely inside the target JVM's own premain(), before the harness's
RuntimeTest reader code (attach/attachOneliner/runBTrace, all migrated onto
OutputPump/Completion in #860) reads a single line of output. Traced and confirmed on
jb/external-type-adapter before merge: the "Dynamic attach" phase (which exercises the
migrated attach()) completed and printed Detached. cleanly in every reproduction; the failure
always occurs later, in the separate "On-Startup" (testStartup) phase, which is structurally
untouched by that harness work.

Candidate fix directions (not yet decided/implemented) Copy Chapter

1. Avoid the lambda at the crash site. Replace ExtensionLoader.java:95's lambda with an
anonymous inner class implementing ServiceDeclarationRegistry.Resolver. Anonymous classes
are ordinary invokespecial/new-based construction — no invokedynamic, no
LambdaMetafactory bootstrap. Minimal, surgical, but only fixes this one call site; any
other lambda/method-reference/string-concat-via-indy touched this early in premain() remains
equally exposed (untargeted audit of the full premain()→initExtensions() path for other
invokedynamic sites has not yet been done).
2. Warm up java.lang.invoke deliberately, before the agent touches it. Add an explicit,
early, single-threaded "touch a trivial lambda/MethodHandle and let its bootstrap fully
resolve" step at the very start of premain()/Main.main(), before initExtensions() (and
ideally before anything else in the agent that might use invokedynamic). This is closer to
your instinct: force java.lang.invoke's own bootstrap to complete in a controlled context
first, rather than avoiding lambdas piecemeal. Open questions to resolve before implementing:
- Does forcing this warm-up from within premain() still race with the JVM's own
concurrent bootstrap, or does it need to happen even earlier (e.g., via a bootstrap-classpath
class, or is premain() early enough already)?
- Is there a reliable, catchable signal for "warm-up succeeded" vs. silently swallowing a
ClassCircularityError during warm-up itself?
3. Combine both: do the warm-up as the primary defense (protects all future/current
invokedynamic use in early agent code), and still convert the one identified lambda to an
anonymous class as defense-in-depth for this specific known-hot call site.

Next steps Copy Chapter

- Decide fix direction (1, 2, or 3 above) — recommend 2+1 combined, given the intermittency
suggests other undiscovered invokedynamic sites in the early premain() path could trigger
the same race even after fixing this one lambda alone.
- If proceeding, this should go through TDD/systematic-debugging per project convention:
reproduce reliably (this doc's repro recipe), form a single hypothesis, test minimally.
- Reproducing this race reliably enough for a regression test is itself nontrivial (it's
load-sensitive) — likely needs either a stress-loop test (spin up N target JVMs concurrently)
or a way to artificially delay/contend java.lang.invoke bootstrap in a test harness.

---

Superpowers/Plans/2026 07 29 Issue 932 Stall Watchdog Plan Copy Section

Implementation plan — stall watchdog for integration tests

Design: docs/superpowers/specs/2026-07-29-issue-932-client-close-deadlock-design.md.
Branch: agent/issue-932-client-close-deadlock, worktree .worktrees/issue-932-client-close.
Base: origin/develop @ 840564e9. Independent of PR #934.

Revision 2. Adversarial review of revision 1 found that the SIGQUIT fallback would have written a
thread dump into the target stdout buffer that test validators assert on (RuntimeTest.java:858 →
ManifestLibsTests.java:81), turning a diagnostic into a test failure. That is removed, along with
the echo flag that existed only to serve it. Other corrections are noted where they apply.

Every command below runs from this worktree, prefixed GRADLE_USER_HOME=$(pwd)/.gradle-user, with
output redirected to a log before filtering, per AGENTS.md.

Constraints Copy Chapter

- Java 8 source level for integration-tests tests (build.gradle:15-18). No var, no
List.of, no direct Process.pid(). Note this is enforced by convention only, and CI actually
runs the harness on Java 24 — the matrix JDK is the target JVM via TEST_JAVA_HOME.
- One worker JVM for all ten test classes. No forkEvery, maxParallelForks, or
junit-platform.properties exists today. Static state is shared for the whole run, and a system
property set by one test leaks into every class after it.
- Test knobs arrive via -P (build.gradle:274 forwards project properties beginning with
btrace.).
- The watchdog must never become the hang it diagnoses, and must never perturb the system under
test. Every wait is bounded; every internal failure is recorded as text, not thrown.

Task 1 — TargetRegistry TargetRegistry')" title="Copy chapter prompt for LLMs"> Copy Chapter

New: integration-tests/src/test/java/tests/harness/TargetRegistry.java

- static Handle register(Process process, String label, String jcmdPath) — called at launch,
before a pid is known. jcmdPath is passed in at registration, not resolved centrally:
resolveTestJavaHome() is protected static in package tests (RuntimeTest.java:128) and is
unreachable from tests.harness, and targets do not all come from the same JDK —
Issue888RuntimeHardeningIntegrationTest.java:172 launches with the build JDK while the
RuntimeTest paths use resolveTestJavaHome().
- Handle.setPid(String) — called where a launch site parses ready:.
- static List<Snapshot> liveTargets() — entries where process.isAlive(). Registering at launch
and filtering on liveness at capture removes any need for deregistration hooks, and makes the
never-stopped TestApp at BTraceFunctionalTests.java:535 harmless.
- Pid resolution when ready: never arrived — the startup-stall case, which is exactly when the
registry must still be useful: fall back to Process.pid() via reflection, cached, guarded,
returning null on failure. Reflection keeps the Java 8 source level while working on the Java 24
worker CI actually uses.
- Snapshot is an immutable (label, pid, process, jcmdPath).
- Backed by CopyOnWriteArrayList.

Task 2 — StallCapture StallCapture')" title="Copy chapter prompt for LLMs"> Copy Chapter

New: integration-tests/src/test/java/tests/harness/StallCapture.java

static void capture(Appendable sink, String label, long elapsedMs, int frame, List<Snapshot> targets, long budgetMs)
— writes the report and never throws.

- Sections in order: banner (label, elapsed, frame index); test-JVM threads; then each live target.
- The sink is flushed after every section. Revision 1 built the whole report as a String and
wrote it at the end, so a hang in any section produced zero bytes — the precise failure being
designed against.
- The test-JVM dump is ThreadMXBean.dumpAllThreads(true, true) (lock owners and monitors, which
Thread.getAllStackTraces() omits) computed on a throwaway thread with a bounded join. It
requires a safepoint, and a JVM that cannot reach one is a live possibility here. On overrun the
section reads test-JVM dump timed out.
- Per target: jcmd <pid> Thread.print -l using the snapshot's own jcmdPath, with output
redirected to a temp file — never a pipe. Thread.print -l on an instrumented target exceeds
the 64 KB pipe buffer, and a jcmd blocked on a full pipe would be killed by the bound having
produced nothing.
- budgetMs spans all targets together, checked before each; a skipped target says so.
- No SIGQUIT fallback. A HotSpot SIGQUIT dump goes to the target's stdout, and for
testStartup targets that stream is accumulated by OutputPump (OutputPump.java:67) into the
buffer v.validate(stdout.toString(), …) asserts on (RuntimeTest.java:858). It would corrupt
the system under test. A jcmd that fails or overruns is recorded as text and that is the end of it.
- Every line is prefixed [stall-dump] so a capture interleaved with the [traced app] and
[btrace out] reader threads is recoverable with one grep.

Task 3 — StallWatchdog and @StallTimeout StallWatchdog and @StallTimeout')" title="Copy chapter prompt for LLMs"> Copy Chapter

New: integration-tests/src/test/java/tests/harness/StallWatchdog.java,
integration-tests/src/test/java/tests/harness/StallTimeout.java

BeforeAllCallback + BeforeEachCallback + AfterEachCallback + AfterAllCallback.

- Arms on beforeAll — JUnit fires extension beforeAll before the class's own @BeforeAll, and
every subclass calls classSetup() from one, so that path is covered — and re-arms per
beforeEach.
- Deadline: @StallTimeout(millis=…) on method or class, else -Pbtrace.test.stallTimeoutMs, else
btrace.test.timeoutMs * 6 (six minutes at the default). The annotation exists because a system
property cannot be scoped to a single test in a single-JVM run. A shorter default is not safe:
RuntimeTest.java:784 already allows a target timeout * 4 = 240 s for startup alone.
- Targets are snapshotted once, at fire time, and both frames capture those exact Process
objects. Revision 1 re-read the live registry for frame two, so a test that unwedged during the
gap would have had the next test's target captured.
- Frame two is a separately scheduled task at +30 s, not a sleep inside frame one, and it
re-checks the disarm flag and skips if the test has finished. Two frames because one cannot
distinguish "wedged" from "slow"; the diff is the evidence. Scheduler pool size 2 so one fire
cannot defer another test's deadline.
- Disarm: ScheduledFuture.cancel(false) plus an AtomicBoolean; afterEach never blocks on an
in-flight capture.
- Output dir comes from the absolute path in btrace.test.stallDumpDir (task 4). A relative path
would resolve against Test.workingDir, which defaults to the project directory, yielding
integration-tests/integration-tests/build/... — outside the upload glob.
- Filenames: sanitised JUnit unique id truncated to 120 chars plus a short hash of the full id.
The full id of a parameterised invocation exceeds the 255-byte filename limit, and the motivating
test is a @ParameterizedTest whose four rows share one method name.
- Sink order is file first, stdout second. if: always() upload steps do run after a
timeout-minutes cancellation, whereas a cancelled job tears down the worker→daemon→console
pipeline, and the release path redirects all test output into a file it only summarises after the
command exits (scripts/run-release-gate.sh:52). Stdout is a shared synchronised PrintStream
that a stalled pipeline can itself block on.
- Stale dumps from earlier runs are cleared once per JVM on first arm; cleanTest does not cover
this directory.

Task 4 — build wiring Copy Chapter

Modify: integration-tests/build.gradle — in the test block (:163), add

systemProperty 'btrace.test.stallDumpDir',
    layout.buildDirectory.dir('reports/stall-dumps').get().asFile.absolutePath

Its own line, because the -P forwarding loop at :274 forwards project properties only.
build/reports//* is what both if: always() upload steps already collect
(continuous.yml:171, release.yml:377), so no workflow change is needed. Confirm that during
task 8 rather than assuming it.

options.release = 8 is not attempted: Issue884PublishedFatAgentE2ETest already calls
Files.writeString (JDK 11) at seven sites (:173, :187, :239, :245, :279, :285, :289), so the
module cannot compile at release 8 today. Recorded as a pre-existing violation, not fixed here.

Task 5 — a per-test timeout, because nothing else provides one Copy Chapter

New: integration-tests/src/test/resources/junit-platform.properties

junit.jupiter.execution.timeout.testable.method.default = 8m
junit.jupiter.execution.timeout.thread.mode.default = SEPARATE_THREAD

PR #934's commit message describes exactly this and its diff contains neither — verified with
git show 44f2c25d. Without it the watchdog's outcome is "30 minutes still burned, now with
evidence". With it, a stall becomes a failing test at 8 minutes.

Eight minutes, not five: a passing test() invocation runs testDynamic then testStartup in one
method, and startupTimeoutMs alone is 240 s, so five minutes risks failing green tests on slow
lanes. Eight sits above that and below release.yml's 15-minute integration job (:271).
SEPARATE_THREAD is required because the default mode interrupts the test thread, and a thread
blocked in socket I/O ignores an interrupt.

Ordering: watchdog captures at 6:00 and 6:30, timeout fails the test at 8:00.

This mode change affects every integration test, so task 8's full-suite run is what qualifies it.

Task 6 — wire the registry into the launch sites Copy Chapter

Modify: integration-tests/src/test/java/tests/RuntimeTest.java

- @ExtendWith(StallWatchdog.class) on the class; @ExtendWith is @Inherited and all eight
subclasses pick it up.
- Register at launch, set the pid where ready: is parsed. Corrected line numbers, all verified:

| site | launch | pid parsed |
|---|---|---|
| testDynamic raw path | :348 | :367 |
| testDynamicOneliner raw path | :540 | :559 |
| testStartup raw path | :745 | :765 (in the readyAwareCompletion decorator, not a reader thread) |
| TestApp | constructor, :1067 caller | :937 |

TestApp.pid is a private int whose only accessor getPid() (:1007) blocks up to 30 s, so the
hook goes at :937 directly.
- No echo-flag changes anywhere: the SIGQUIT fallback that needed them is gone.

Modify: integration-tests/src/test/java/tests/Issue888RuntimeHardeningIntegrationTest.java —
register the target built at :172-189; the pid is parsed by the caller at :83-85, so
startTarget() must return the handle alongside the process rather than a bare Process.

Client-JVM spawns (RuntimeTest.java:1109, :1255, :1380, :1493) are registered too where a handle
can be threaded without restructuring; where it cannot, the omission is recorded in the commit
message rather than left silent. :254 (hasJaxbProbeDescriptorSupport) stays unregistered — it is
bounded at 10 s and cannot stall a job.

Task 7 — tests Copy Chapter

New: integration-tests/src/test/java/tests/harness/StallCaptureTest.java,
integration-tests/src/test/java/tests/harness/StallWatchdogTest.java

Driven through the in-memory seam, scoped with @StallTimeout, output under @TempDir.

1. Captures the test JVM. Park a distinctively named thread; assert banner, label, thread name
and stack all appear.
2. Captures a target JVM. Launch a JVM that blocks until told to exit (not "short-lived" — a
target that exits mid-attach makes jcmd nondeterministic), launched from the same JDK whose
jcmd is used. Assert a Thread.print section for its pid appears. Degrade with an assumption
rather than a failure if attach is unavailable in the environment.
3. Degrades rather than hangs. A bogus pid: capture returns inside its budget, records the
failure inline, still contains the test-JVM dump.
4. Budget is shared. Several unreachable targets: capture returns in roughly the budget, not
budget × N, and names what it skipped.
5. Silent when green. A fast test with the watchdog armed writes nothing into the @TempDir
output dir.
6. Fires end to end, with a target. @StallTimeout(millis=1500) on a test that registers a real
blocking target and then sleeps ~3 s; assert the dump names both the sleeping thread and the
target's Thread.print section. This replaces revision 1's throwaway "add a test, run it, delete
it" verification step, which produced evidence nobody could re-verify.
7. Two frames. A test that stays parked past the 30 s gap produces two dump files whose thread
sections differ in elapsed time — the frame-two mechanism is otherwise never exercised.
Uses a shortened inter-frame gap via the same annotation so it runs in seconds.

Task 8 — verification Copy Chapter

1. GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew spotlessApply, commit any formatting change.
2. GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew :integration-tests:compileTestJava.
3. GRADLE_USER_HOME=$(pwd)/.gradle-user ./gradlew -Pintegration :integration-tests:test --tests 'tests.harness.*'
— note this also runs the pre-existing CompletionTest and OutputPumpTest, and still triggers
the full dist/probe build chain (build.gradle:167-172), so it is not a cheap check.
4. Full suite as CI runs it:
GRADLE_USER_HOME=$(pwd)/.gradle-user TEST_JAVA_HOME=<jdk> ./gradlew -Pintegration -PCI :integration-tests:test.
This is the gate for task 5's SEPARATE_THREAD change. Record the duration and confirm no
watchdog file is produced. Docker-tagged tests are included whenever a Docker host is reachable
(build.gradle:176-241), so hold Docker state constant between baseline and after.
5. Confirm build/reports/stall-dumps/ is inside the upload glob by listing what a run leaves in
integration-tests/build/reports/.

Order Copy Chapter

1 → 2 → 3 → 7 (tests for 1–3) → 6 → 4 → 5 → 8.

Revision 3 — corrections found by adversarial review of the implementation Copy Chapter

Two defects would have shipped silently, and both were confirmed by direct measurement rather than
argument:

1. The per-test timeout was never committed. .gitignore's bare junit* pattern matches
integration-tests/src/test/resources/junit-platform.properties at any depth, so the file existed
locally, passed every local run, and was absent from the commit. .gitignore now carries a
negation, placed after the pattern it overrides, since the last matching pattern wins.
2. Test-JVM stacks were truncated to eight frames. ThreadInfo.toString() caps output at
MAX_FRAMES = 8; a 44-frame stack rendered as 8. The dump is now rendered frame by frame with
locked monitors and synchronizers, and a test asserts the rendered depth exceeds eight.

Also corrected: the stdout echo was the one unbounded wait in a class that bounds everything else,
and it ran on one of only two scheduler threads — it now runs on a throwaway thread with a bounded
join. The registry is pruned on disarm, so a late stall does not spend its capture budget on
jcmd calls against processes from earlier classes. Dump filenames are truncated from the tail, not
the head, because a parameterised unique id is identical up to its invocation index, and carry a
sequence number so two teardown stalls in one class cannot collide on one path. The derived deadline
is clamped below the per-test timeout, since btrace.test.timeoutMs is an operator knob that could
otherwise push the watchdog past it. The four btrace client JVM launch sites are registered — the
client is the side the surviving issue-932 hypothesis implicates. budgetIsSharedAcrossTargets
passed against a per-target budget and now does not; capturesTargetJvmThreads would have failed a
build rather than skipped when jcmd cannot attach on a loaded runner.

---

Tutorials/01 First Trace In 2 Minutes Copy Section

Your First Trace in 2 Minutes

Trace a live JVM — no code changes, no restart, no config. You'll find a latency bug and an
intermittent exception in a running application using nothing but one-line commands.

Persona: anyone with a JVM that's misbehaving. Time: ~5 minutes (first trace in 2).

What you'll need Copy Chapter

- JDK 11 or newer on your PATH (the demo uses single-file source launch)
- BTrace 3.0 installed — bin/btrace on your PATH (installation options)
- Two terminal windows

Step 1 — Start the patient Copy Chapter

The demo app (demo/DemoApp.java) is a tiny order-processing service with two
deliberate defects hidden inside. Run it in terminal 1:

java DemoApp.java

You should see (numbers will vary):

[demo] order service running - stop with Ctrl+C
[demo] processed 64 orders, 8 failed
[demo] processed 134 orders, 14 failed

Orders are being processed, and some of them fail. Why? The app won't tell you — let's ask the JVM
directly.

Step 2 — Find its PID Copy Chapter

In terminal 2:

jps

You should see a line like 12345 DemoApp. That number is the <PID> in every command below.

Step 3 — Your first trace Copy Chapter

A BTrace oneliner attaches to the running JVM and prints every call of a method, live:

btrace -n 'OrderService::processOrder @return { print method, time }' <PID>

You should see a stream like:

processOrder
execution time: 61 ms
processOrder
execution time: 74 ms
processOrder
execution time: 342 ms

What just happened? BTrace compiled your one-liner into a tiny instrumentation program,

injected it into the running JVM, and hooked the return of OrderService.processOrder — all

while the app kept serving orders. Nothing was restarted, and when you detach, every hook is

cleanly removed. Curious what it generated? Re-run with -Dbtrace.oneliner.dump=true to see the

generated source.

Notice something? Most orders take well under 100 ms, but every now and then one takes 300+ ms.
Let's isolate the slow ones.

Step 4 — Catch the latency bug Copy Chapter

Stop the previous trace with Ctrl+C, then widen the net to all OrderService methods and keep
only the slow calls:

btrace -n 'OrderService::* @return if duration>200ms { print method, time }' <PID>

You should see two method names, always together:

chargeCard
execution time: 306 ms
processOrder
execution time: 331 ms

Case closed: chargeCard is the culprit (a "slow payment provider" hits ~10% of calls), and its
latency is what makes processOrder slow. You found it with one line, without reading a single
line of application code.

if duration > 200ms filters on the method's execution time. In print, the raw duration

value is in nanoseconds; the time action prints it converted to milliseconds for you.

Step 5 — Catch the failures Copy Chapter

The summary line also reported failed orders. Hook the error path — this probe only fires when a
method exits by throwing:

btrace -n 'OrderService::validateOrder @error { print method, stack }' <PID>

You should see, for roughly one order in twelve:

validateOrder
OrderService.validateOrder(DemoApp.java:93)
OrderService.processOrder(DemoApp.java:85)
...

The exception's origin and the exact call path — captured live, from a method that the application
catches and silently swallows.

Step 6 — Clean up Copy Chapter

Ctrl+C in the BTrace terminal detaches the client. In BTrace 3.0, detaching disables all
injected probes on the spot (they become no-ops — no restart, no lingering overhead), and you can
re-attach at any time. Stop the demo app with Ctrl+C in terminal 1 when you're done.

Troubleshooting Copy Chapter

- Can not attach to PID ... — make sure you run btrace as the same OS user as the target
JVM, and check Troubleshooting: attachment issues.
- A warning about your Java version — if the target JVM is older than Java 17 you'll see the
3.0 deprecation warning. Everything still works; see the
Java support policy for what it means.
- No output appears — the probe may not match: patterns are exact class names plus */?
wildcards. Try OrderService::* first, then narrow down.

Go deeper Copy Chapter

- Full oneliner syntax (filters, args, count, regex patterns): Oneliner Guide
- Turn a oneliner into a real script with the flat DSL: Tutorial Lesson 7
- Everything you can hook (@OnMethod locations, JFR, timers): Quick Reference

---

Tutorials/02 Oneliner To Script Copy Section