{"owner":"elastic","repo":"elasticsearch","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md"],"skills":{"AGENTS.md":"# Elasticsearch\n\n## Toolchain Snapshot\n- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).\n- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.\n- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.\n- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.\n- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.\n\n## Build & Run Commands\n- Refer to BUILDING.md, CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.\n\n## Verification & Lint Tasks\n- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.\n- `spotlessApply` also prunes unused imports and reorders imports automatically. Run it instead of manually hunting for unused imports after refactoring.\n\n## Project Structure\nThe repository is organized into several key directories:\n*   `server`: The core Elasticsearch server. Few third-party dependencies (Lucene plus a handful of small libraries). Key `org.elasticsearch` sub-packages: `cluster` (cluster state machine), `index` (per-index logic), `search` (query execution), `action` (transport actions), `snapshots` (snapshot/restore), plus `indices`, `repositories`, `rest`, `ingest`, etc.\n*   `modules`: Features shipped with Elasticsearch by default, but not considered \"core\" server code. Many modules provide a specific implementation of a pluggable interface defined in `server`, such as `transport-netty4` (the transport layer) or `repository-s3`/`repository-gcs`/`repository-azure` (snapshot repositories). Others integrate with external systems, such as `apm` (Application Performance Monitoring agent integration).\n*   `plugins`: Optional, not bundled by default, but officially supported. Examples: `discovery-ec2`/`discovery-gce`/`discovery-azure-classic` (cloud-aware cluster discovery).\n*   `libs`: Internal libraries used by multiple parts of the project. Examples: `logging`, `x-content` (JSON/CBOR/YAML/SMILE parsing abstraction).\n*   `client`: The official Java REST client.\n*   `test`: Test infrastructure used by the rest of the repo. `framework` holds `ESTestCase`/`ESIntegTestCase`/`ESSingleNodeTestCase`; also `test-clusters` and `yaml-rest-runner` (runner for YAML-based REST API tests).\n*   `qa`: Integration and multi-version tests. Examples: `rolling-upgrade`, `mixed-cluster`.\n*   `rest-api-spec`: JSON spec definitions for the public REST API endpoints.\n*   `docs`: Project documentation.\n*   `distribution`: Logic for building distribution packages.\n*   `x-pack`: Modules, plugins, and commercial features under the Elastic License 2.0. Example sub-plugins: `security`, `ml` (machine learning), `ccr` (cross-cluster replication), `logsdb` (optimized index mode for log data), and `stateless`.\n*   `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic. Refer to BUILDING.md for details on how these are structured and used.\n\n## Stateless Elasticsearch\n\nStateless Elasticsearch is a distribution where shard data is stored in an **object store** (e.g., S3, GCS, Azure) rather than local disk. Nodes carry no durable local state. The cluster distinguishes two node roles: **indexing nodes** (`index` role, write path + translog replication to object store) and **search nodes** (`search` role, read-only via shared blob cache). The `DiscoveryNode.STATELESS_ENABLED_SETTING` gates stateless behavior at runtime.\n\n### Plugin `deploymentTarget`\n\nPlugins can set `deploymentTarget` in `build.gradle`. That value tells the node **whether to load the plugin**: **`STATEFUL_ONLY`** (stateful clusters only), **`STATELESS_ONLY`** (stateless mode on only), or **`ALL`** (always loaded; this is the default when the property is omitted).\n\n### Plugin locations\n\n| Plugin | Gradle path | Purpose |\n|---|---|---|\n| `stateless` | `:x-pack:plugin:stateless` | Core stateless — engines, allocation, cache, object store, recovery |\n| `stateless-sigterm` | `:x-pack:plugin:stateless-sigterm` | Clean SIGTERM shutdown for Kubernetes |\n| `stateless-master-failover` | `:x-pack:plugin:stateless-master-failover` | Master failover behavior |\n| `stateless-no-wait-for-active-shards` | `:x-pack:plugin:stateless-no-wait-for-active-shards` | Suppresses wait-for-active-shards |\n| `stateless-health-shards-availability` | `:x-pack:plugin:stateless-health-shards-availability` | Shard availability health indicators |\n\n**Package**: `org.elasticsearch.xpack.stateless.*` throughout.\n\n### Key subsystems\n\n- **Object store** (`objectstore/`): `ObjectStoreService`, bucket config, GC tasks for stale indices and translogs.\n- **Commits** (`commits/`): `StatelessCommitService` manages shard commits to blob store; `HollowShardsService` manages hollow indexing shards.\n- **Cache & prewarming** (`cache/`): `StatelessSharedBlobCacheService`, online prewarming, `SearchCommitPrefetcher`.\n- **Engines** (`engine/`): `IndexEngine` (write path) and `SearchEngine` (read-only); `TranslogReplicator` replicates translog to object store.\n- **Allocation** (`allocation/`): `StatelessExistingShardsAllocator`, separate balancing weights per tier, heap-usage-aware allocation decisions.\n- **Recovery** (`recovery/`): custom primary relocation and unpromotable shard relocation protocols.\n\n## Testing Cheatsheet\n- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).\n- Single project: `./gradlew :server:test` (or other subproject path).\n- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.\n- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.\n- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.\n- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).\n- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline=\"-verbose:gc\"`, `-Dtests.output=always`, etc.\n- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.\n- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.\n- Yaml REST tests: `./gradlew \":rest-api-spec:yamlRestTest\" --tests \"org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}\"`\n- ES|QL CSV tests: `./gradlew \":x-pack:plugin:esql:internalClusterTest\" --tests \"org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*\"` (e.g. `--tests \"...CsvIT.*stats_first_last*\"`); append `*<test-name>*` to target a single test within the file.\n- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.\n- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.\n- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary.\n\n### Test Types\n- Unit Tests: Preferred. Extend `ESTestCase`.\n- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).\n- Integration: Extend `ESIntegTestCase`.\n- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.\n\n### Distribution selection for external-module tests\n- Prefer the OSS/minimal distribution over `usesDefaultDistribution` whenever possible. `usesDefaultDistribution` packages the full default distribution, which is significantly more expensive to build and run.\n- Only use `usesDefaultDistribution` when the test genuinely requires a feature that is only available in the default distribution and cannot be replicated with a custom cluster configuration that includes just the needed plugins. Always document the reason in the `usesDefaultDistribution(...)` message.\n\n## Dependency Hygiene\n- Never add a dependency without checking for existing alternatives in the repo.\n\n## Entitlement Policy\n- Never add an entitlement speculatively. Each entry in `entitlement-policy.yaml` must have a specific justification — ideally a concrete `NotEntitledException` that was observed, or at minimum a clear explanation of why the library requires that capability. Entitlements are a least-privilege mechanism; granting one \"just in case\" defeats the purpose.\n- Every use of `ESTestCase.WithoutEntitlements` must be accompanied by a comment explaining why the entitlement failure is spurious in the test context and would not occur in production.\n\n## Formatting & Imports\n- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.\n- In `switch` statements, do not use `default` as a branch for valid or expected options. Enumerate those cases explicitly and reserve `default` for throwing an exception for unexpected values, or an assertion error if this code branch is unreachable.\n\n## Types, Generics, and Suppressions\n- Prefer type-safe constructs; avoid raw types and unchecked casts.\n- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).\n- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.\n\n## Naming Conventions\n- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.\n- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.\n- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.\n\n## Logging & Error Handling\n- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.\n- Always use parameterized logging (`logger.debug(\"operation [{}]\", value)`); never build strings via concatenation.\n- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.\n- Log levels:\n  - `TRACE`: highly verbose developer diagnostics; usually read alongside code.\n  - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.\n  - `INFO`: default-enabled operational milestones; prefer factual language.\n  - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.\n  - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.\n- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.\n- Tests can assert logging via `MockLog` for complex flows.\n\n## Javadoc & Comments\n- New packages/classes/public or abstract methods require Javadoc explaining the \"why\" rather than the implementation details.\n- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.\n- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.\n- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.\n\n## License Headers\n- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.\n- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.\n\n## Generated Files\n- Never hand-edit generated files. Instead, edit the source they are generated from and regenerate.\n- ANTLR-generated files can be regenerated by running the `regen` task on the relevant subproject.\n- Other generated files are regenerated by compiling the project.\n\n## Debugging Missing Tests\n\nWhen expected test methods are absent from results (not failed, not skipped — simply not present in the XML or binary event stream), check `muted-tests.yml` first. The build translates every entry into a Gradle `TestFilter.excludePattern`, which silently drops matching tests before the randomized runner receives them. A muted test fires no `testStarted` event and leaves no trace in `results-generic.bin`.\n   ```bash\n   grep 'ClassName\\|methodName' muted-tests.yml\n   ```\n\n### `No tests found for given includes: [**/*$*.class]`\n\nWhen a test task fails at execution with `No tests found for given includes: [**/*$*.class](exclude rules)`, it usually does **not** mean Gradle failed to detect the test class. The far more common cause is that **every test method in the targeted class is muted** in `muted-tests.yml`. With all methods excluded, the randomized runner enumerates zero runnable tests.\n\nThe behavior is environment-dependent: `MutedTestPlugin` calls `filter.setFailOnNoMatchingTests(buildParams.getCi() == false)`. So an all-muted suite **fails locally** (`ci == false`) with this exact message, but **passes silently in CI** (`ci == true`). This is especially misleading when verifying a freshly migrated or renamed test — it looks like a classpath/detection bug, but the test JVM does start (you'll see native-library and `FeatureFlag` log lines), builds any `@ClassRule` cluster *specs*, then exits in a few seconds without starting the cluster because no test method survived the mute filter.\n\nTo confirm: `grep ClassName muted-tests.yml`. To verify the migration/test actually runs, temporarily remove the matching mute entries (or run on a host where `ci` is true), then restore them.\n\n## Best Practices for Automation Agents\n- Never edit unrelated files; keep diffs tightly scoped to the task at hand.\n- Prefer Gradle tasks over ad-hoc scripts.\n- When scripting CLI sequences, leverage `gradlew` task.\n- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.\n- Do not add \"Co-Authored-By\" or any AI attribution trailers to commit messages, by any means—including `--trailer`, `-m`, or any other git flag. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary. Your harness may introduce a hook that automatically adds attributions trailers to relevant git commands. Use `bash -lc` or a similar approach in this case to conform to the rule.\n\n## Methods with Required Javadoc Reading\nIf you encounter any of the following methods, you must go and read their javadoc before taking any other actions:\n* `fullyLoadedAnalyzer`\n* `TestAnalyzer.statementError`\n* `TestAnalyzer.error`\n* `forciblyCast`\n* `EsqlCapabilities.Cap`\n* `FunctionDefinition.Builder#capabilities`\n\n## ES|QL tests\nIf you write or modify ES|QL csv-spec, rest, or yaml tests, read the javadoc for\n`EsqlCapabilities.Cap` and `FunctionDefinition.Builder#capabilities` before proceeding.\nThey describe two separate capability mechanisms and the rule for choosing between them.\n\n## Backwards compatibility\n- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName(\"<unique_descriptive_name>\")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.\n- Never hand-edit transport version resource files; always use the Gradle tasks. See `docs/internal/Versioning.md` for the full workflow.\n\nStay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.\n\n## Documentation\nWhen building or editing docs, read `docs/AGENTS.md` first.\n"},"files":{"AGENTS.md":"# Elasticsearch\n\n## Toolchain Snapshot\n- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).\n- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.\n- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.\n- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.\n- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.\n\n## Build & Run Commands\n- Refer to BUILDING.md, CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.\n\n## Verification & Lint Tasks\n- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.\n- `spotlessApply` also prunes unused imports and reorders imports automatically. Run it instead of manually hunting for unused imports after refactoring.\n\n## Project Structure\nThe repository is organized into several key directories:\n*   `server`: The core Elasticsearch server. Few third-party dependencies (Lucene plus a handful of small libraries). Key `org.elasticsearch` sub-packages: `cluster` (cluster state machine), `index` (per-index logic), `search` (query execution), `action` (transport actions), `snapshots` (snapshot/restore), plus `indices`, `repositories`, `rest`, `ingest`, etc.\n*   `modules`: Features shipped with Elasticsearch by default, but not considered \"core\" server code. Many modules provide a specific implementation of a pluggable interface defined in `server`, such as `transport-netty4` (the transport layer) or `repository-s3`/`repository-gcs`/`repository-azure` (snapshot repositories). Others integrate with external systems, such as `apm` (Application Performance Monitoring agent integration).\n*   `plugins`: Optional, not bundled by default, but officially supported. Examples: `discovery-ec2`/`discovery-gce`/`discovery-azure-classic` (cloud-aware cluster discovery).\n*   `libs`: Internal libraries used by multiple parts of the project. Examples: `logging`, `x-content` (JSON/CBOR/YAML/SMILE parsing abstraction).\n*   `client`: The official Java REST client.\n*   `test`: Test infrastructure used by the rest of the repo. `framework` holds `ESTestCase`/`ESIntegTestCase`/`ESSingleNodeTestCase`; also `test-clusters` and `yaml-rest-runner` (runner for YAML-based REST API tests).\n*   `qa`: Integration and multi-version tests. Examples: `rolling-upgrade`, `mixed-cluster`.\n*   `rest-api-spec`: JSON spec definitions for the public REST API endpoints.\n*   `docs`: Project documentation.\n*   `distribution`: Logic for building distribution packages.\n*   `x-pack`: Modules, plugins, and commercial features under the Elastic License 2.0. Example sub-plugins: `security`, `ml` (machine learning), `ccr` (cross-cluster replication), `logsdb` (optimized index mode for log data), and `stateless`.\n*   `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic. Refer to BUILDING.md for details on how these are structured and used.\n\n## Stateless Elasticsearch\n\nStateless Elasticsearch is a distribution where shard data is stored in an **object store** (e.g., S3, GCS, Azure) rather than local disk. Nodes carry no durable local state. The cluster distinguishes two node roles: **indexing nodes** (`index` role, write path + translog replication to object store) and **search nodes** (`search` role, read-only via shared blob cache). The `DiscoveryNode.STATELESS_ENABLED_SETTING` gates stateless behavior at runtime.\n\n### Plugin `deploymentTarget`\n\nPlugins can set `deploymentTarget` in `build.gradle`. That value tells the node **whether to load the plugin**: **`STATEFUL_ONLY`** (stateful clusters only), **`STATELESS_ONLY`** (stateless mode on only), or **`ALL`** (always loaded; this is the default when the property is omitted).\n\n### Plugin locations\n\n| Plugin | Gradle path | Purpose |\n|---|---|---|\n| `stateless` | `:x-pack:plugin:stateless` | Core stateless — engines, allocation, cache, object store, recovery |\n| `stateless-sigterm` | `:x-pack:plugin:stateless-sigterm` | Clean SIGTERM shutdown for Kubernetes |\n| `stateless-master-failover` | `:x-pack:plugin:stateless-master-failover` | Master failover behavior |\n| `stateless-no-wait-for-active-shards` | `:x-pack:plugin:stateless-no-wait-for-active-shards` | Suppresses wait-for-active-shards |\n| `stateless-health-shards-availability` | `:x-pack:plugin:stateless-health-shards-availability` | Shard availability health indicators |\n\n**Package**: `org.elasticsearch.xpack.stateless.*` throughout.\n\n### Key subsystems\n\n- **Object store** (`objectstore/`): `ObjectStoreService`, bucket config, GC tasks for stale indices and translogs.\n- **Commits** (`commits/`): `StatelessCommitService` manages shard commits to blob store; `HollowShardsService` manages hollow indexing shards.\n- **Cache & prewarming** (`cache/`): `StatelessSharedBlobCacheService`, online prewarming, `SearchCommitPrefetcher`.\n- **Engines** (`engine/`): `IndexEngine` (write path) and `SearchEngine` (read-only); `TranslogReplicator` replicates translog to object store.\n- **Allocation** (`allocation/`): `StatelessExistingShardsAllocator`, separate balancing weights per tier, heap-usage-aware allocation decisions.\n- **Recovery** (`recovery/`): custom primary relocation and unpromotable shard relocation protocols.\n\n## Testing Cheatsheet\n- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).\n- Single project: `./gradlew :server:test` (or other subproject path).\n- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.\n- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.\n- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.\n- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).\n- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline=\"-verbose:gc\"`, `-Dtests.output=always`, etc.\n- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.\n- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.\n- Yaml REST tests: `./gradlew \":rest-api-spec:yamlRestTest\" --tests \"org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}\"`\n- ES|QL CSV tests: `./gradlew \":x-pack:plugin:esql:internalClusterTest\" --tests \"org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*\"` (e.g. `--tests \"...CsvIT.*stats_first_last*\"`); append `*<test-name>*` to target a single test within the file.\n- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.\n- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.\n- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary.\n\n### Test Types\n- Unit Tests: Preferred. Extend `ESTestCase`.\n- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).\n- Integration: Extend `ESIntegTestCase`.\n- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.\n\n### Distribution selection for external-module tests\n- Prefer the OSS/minimal distribution over `usesDefaultDistribution` whenever possible. `usesDefaultDistribution` packages the full default distribution, which is significantly more expensive to build and run.\n- Only use `usesDefaultDistribution` when the test genuinely requires a feature that is only available in the default distribution and cannot be replicated with a custom cluster configuration that includes just the needed plugins. Always document the reason in the `usesDefaultDistribution(...)` message.\n\n## Dependency Hygiene\n- Never add a dependency without checking for existing alternatives in the repo.\n\n## Entitlement Policy\n- Never add an entitlement speculatively. Each entry in `entitlement-policy.yaml` must have a specific justification — ideally a concrete `NotEntitledException` that was observed, or at minimum a clear explanation of why the library requires that capability. Entitlements are a least-privilege mechanism; granting one \"just in case\" defeats the purpose.\n- Every use of `ESTestCase.WithoutEntitlements` must be accompanied by a comment explaining why the entitlement failure is spurious in the test context and would not occur in production.\n\n## Formatting & Imports\n- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.\n- In `switch` statements, do not use `default` as a branch for valid or expected options. Enumerate those cases explicitly and reserve `default` for throwing an exception for unexpected values, or an assertion error if this code branch is unreachable.\n\n## Types, Generics, and Suppressions\n- Prefer type-safe constructs; avoid raw types and unchecked casts.\n- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).\n- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.\n\n## Naming Conventions\n- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.\n- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.\n- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.\n\n## Logging & Error Handling\n- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.\n- Always use parameterized logging (`logger.debug(\"operation [{}]\", value)`); never build strings via concatenation.\n- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.\n- Log levels:\n  - `TRACE`: highly verbose developer diagnostics; usually read alongside code.\n  - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.\n  - `INFO`: default-enabled operational milestones; prefer factual language.\n  - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.\n  - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.\n- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.\n- Tests can assert logging via `MockLog` for complex flows.\n\n## Javadoc & Comments\n- New packages/classes/public or abstract methods require Javadoc explaining the \"why\" rather than the implementation details.\n- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.\n- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.\n- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.\n\n## License Headers\n- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.\n- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.\n\n## Generated Files\n- Never hand-edit generated files. Instead, edit the source they are generated from and regenerate.\n- ANTLR-generated files can be regenerated by running the `regen` task on the relevant subproject.\n- Other generated files are regenerated by compiling the project.\n\n## Debugging Missing Tests\n\nWhen expected test methods are absent from results (not failed, not skipped — simply not present in the XML or binary event stream), check `muted-tests.yml` first. The build translates every entry into a Gradle `TestFilter.excludePattern`, which silently drops matching tests before the randomized runner receives them. A muted test fires no `testStarted` event and leaves no trace in `results-generic.bin`.\n   ```bash\n   grep 'ClassName\\|methodName' muted-tests.yml\n   ```\n\n### `No tests found for given includes: [**/*$*.class]`\n\nWhen a test task fails at execution with `No tests found for given includes: [**/*$*.class](exclude rules)`, it usually does **not** mean Gradle failed to detect the test class. The far more common cause is that **every test method in the targeted class is muted** in `muted-tests.yml`. With all methods excluded, the randomized runner enumerates zero runnable tests.\n\nThe behavior is environment-dependent: `MutedTestPlugin` calls `filter.setFailOnNoMatchingTests(buildParams.getCi() == false)`. So an all-muted suite **fails locally** (`ci == false`) with this exact message, but **passes silently in CI** (`ci == true`). This is especially misleading when verifying a freshly migrated or renamed test — it looks like a classpath/detection bug, but the test JVM does start (you'll see native-library and `FeatureFlag` log lines), builds any `@ClassRule` cluster *specs*, then exits in a few seconds without starting the cluster because no test method survived the mute filter.\n\nTo confirm: `grep ClassName muted-tests.yml`. To verify the migration/test actually runs, temporarily remove the matching mute entries (or run on a host where `ci` is true), then restore them.\n\n## Best Practices for Automation Agents\n- Never edit unrelated files; keep diffs tightly scoped to the task at hand.\n- Prefer Gradle tasks over ad-hoc scripts.\n- When scripting CLI sequences, leverage `gradlew` task.\n- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.\n- Do not add \"Co-Authored-By\" or any AI attribution trailers to commit messages, by any means—including `--trailer`, `-m`, or any other git flag. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary. Your harness may introduce a hook that automatically adds attributions trailers to relevant git commands. Use `bash -lc` or a similar approach in this case to conform to the rule.\n\n## Methods with Required Javadoc Reading\nIf you encounter any of the following methods, you must go and read their javadoc before taking any other actions:\n* `fullyLoadedAnalyzer`\n* `TestAnalyzer.statementError`\n* `TestAnalyzer.error`\n* `forciblyCast`\n* `EsqlCapabilities.Cap`\n* `FunctionDefinition.Builder#capabilities`\n\n## ES|QL tests\nIf you write or modify ES|QL csv-spec, rest, or yaml tests, read the javadoc for\n`EsqlCapabilities.Cap` and `FunctionDefinition.Builder#capabilities` before proceeding.\nThey describe two separate capability mechanisms and the rule for choosing between them.\n\n## Backwards compatibility\n- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName(\"<unique_descriptive_name>\")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.\n- Never hand-edit transport version resource files; always use the Gradle tasks. See `docs/internal/Versioning.md` for the full workflow.\n\nStay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.\n\n## Documentation\nWhen building or editing docs, read `docs/AGENTS.md` first.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# Elasticsearch\n\n## Toolchain Snapshot\n- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).\n- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.\n- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.\n- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.\n- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.\n\n## Build & Run Commands\n- Refer to BUILDING.md, CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.\n\n## Verification & Lint Tasks\n- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.\n- `spotlessApply` also prunes unused imports and reorders imports automatically. Run it instead of manually hunting for unused imports after refactoring.\n\n## Project Structure\nThe repository is organized into several key directories:\n*   `server`: The core Elasticsearch server. Few third-party dependencies (Lucene plus a handful of small libraries). Key `org.elasticsearch` sub-packages: `cluster` (cluster state machine), `index` (per-index logic), `search` (query execution), `action` (transport actions), `snapshots` (snapshot/restore), plus `indices`, `repositories`, `rest`, `ingest`, etc.\n*   `modules`: Features shipped with Elasticsearch by default, but not considered \"core\" server code. Many modules provide a specific implementation of a pluggable interface defined in `server`, such as `transport-netty4` (the transport layer) or `repository-s3`/`repository-gcs`/`repository-azure` (snapshot repositories). Others integrate with external systems, such as `apm` (Application Performance Monitoring agent integration).\n*   `plugins`: Optional, not bundled by default, but officially supported. Examples: `discovery-ec2`/`discovery-gce`/`discovery-azure-classic` (cloud-aware cluster discovery).\n*   `libs`: Internal libraries used by multiple parts of the project. Examples: `logging`, `x-content` (JSON/CBOR/YAML/SMILE parsing abstraction).\n*   `client`: The official Java REST client.\n*   `test`: Test infrastructure used by the rest of the repo. `framework` holds `ESTestCase`/`ESIntegTestCase`/`ESSingleNodeTestCase`; also `test-clusters` and `yaml-rest-runner` (runner for YAML-based REST API tests).\n*   `qa`: Integration and multi-version tests. Examples: `rolling-upgrade`, `mixed-cluster`.\n*   `rest-api-spec`: JSON spec definitions for the public REST API endpoints.\n*   `docs`: Project documentation.\n*   `distribution`: Logic for building distribution packages.\n*   `x-pack`: Modules, plugins, and commercial features under the Elastic License 2.0. Example sub-plugins: `security`, `ml` (machine learning), `ccr` (cross-cluster replication), `logsdb` (optimized index mode for log data), and `stateless`.\n*   `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic. Refer to BUILDING.md for details on how these are structured and used.\n\n## Stateless Elasticsearch\n\nStateless Elasticsearch is a distribution where shard data is stored in an **object store** (e.g., S3, GCS, Azure) rather than local disk. Nodes carry no durable local state. The cluster distinguishes two node roles: **indexing nodes** (`index` role, write path + translog replication to object store) and **search nodes** (`search` role, read-only via shared blob cache). The `DiscoveryNode.STATELESS_ENABLED_SETTING` gates stateless behavior at runtime.\n\n### Plugin `deploymentTarget`\n\nPlugins can set `deploymentTarget` in `build.gradle`. That value tells the node **whether to load the plugin**: **`STATEFUL_ONLY`** (stateful clusters only), **`STATELESS_ONLY`** (stateless mode on only), or **`ALL`** (always loaded; this is the default when the property is omitted).\n\n### Plugin locations\n\n| Plugin | Gradle path | Purpose |\n|---|---|---|\n| `stateless` | `:x-pack:plugin:stateless` | Core stateless — engines, allocation, cache, object store, recovery |\n| `stateless-sigterm` | `:x-pack:plugin:stateless-sigterm` | Clean SIGTERM shutdown for Kubernetes |\n| `stateless-master-failover` | `:x-pack:plugin:stateless-master-failover` | Master failover behavior |\n| `stateless-no-wait-for-active-shards` | `:x-pack:plugin:stateless-no-wait-for-active-shards` | Suppresses wait-for-active-shards |\n| `stateless-health-shards-availability` | `:x-pack:plugin:stateless-health-shards-availability` | Shard availability health indicators |\n\n**Package**: `org.elasticsearch.xpack.stateless.*` throughout.\n\n### Key subsystems\n\n- **Object store** (`objectstore/`): `ObjectStoreService`, bucket config, GC tasks for stale indices and translogs.\n- **Commits** (`commits/`): `StatelessCommitService` manages shard commits to blob store; `HollowShardsService` manages hollow indexing shards.\n- **Cache & prewarming** (`cache/`): `StatelessSharedBlobCacheService`, online prewarming, `SearchCommitPrefetcher`.\n- **Engines** (`engine/`): `IndexEngine` (write path) and `SearchEngine` (read-only); `TranslogReplicator` replicates translog to object store.\n- **Allocation** (`allocation/`): `StatelessExistingShardsAllocator`, separate balancing weights per tier, heap-usage-aware allocation decisions.\n- **Recovery** (`recovery/`): custom primary relocation and unpromotable shard relocation protocols.\n\n## Testing Cheatsheet\n- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).\n- Single project: `./gradlew :server:test` (or other subproject path).\n- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.\n- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.\n- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.\n- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).\n- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline=\"-verbose:gc\"`, `-Dtests.output=always`, etc.\n- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.\n- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.\n- Yaml REST tests: `./gradlew \":rest-api-spec:yamlRestTest\" --tests \"org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}\"`\n- ES|QL CSV tests: `./gradlew \":x-pack:plugin:esql:internalClusterTest\" --tests \"org.elasticsearch.xpack.esql.CsvIT.*<csv-file>*\"` (e.g. `--tests \"...CsvIT.*stats_first_last*\"`); append `*<test-name>*` to target a single test within the file.\n- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.\n- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.\n- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary.\n\n### Test Types\n- Unit Tests: Preferred. Extend `ESTestCase`.\n- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).\n- Integration: Extend `ESIntegTestCase`.\n- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.\n\n### Distribution selection for external-module tests\n- Prefer the OSS/minimal distribution over `usesDefaultDistribution` whenever possible. `usesDefaultDistribution` packages the full default distribution, which is significantly more expensive to build and run.\n- Only use `usesDefaultDistribution` when the test genuinely requires a feature that is only available in the default distribution and cannot be replicated with a custom cluster configuration that includes just the needed plugins. Always document the reason in the `usesDefaultDistribution(...)` message.\n\n## Dependency Hygiene\n- Never add a dependency without checking for existing alternatives in the repo.\n\n## Entitlement Policy\n- Never add an entitlement speculatively. Each entry in `entitlement-policy.yaml` must have a specific justification — ideally a concrete `NotEntitledException` that was observed, or at minimum a clear explanation of why the library requires that capability. Entitlements are a least-privilege mechanism; granting one \"just in case\" defeats the purpose.\n- Every use of `ESTestCase.WithoutEntitlements` must be accompanied by a comment explaining why the entitlement failure is spurious in the test context and would not occur in production.\n\n## Formatting & Imports\n- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.\n- In `switch` statements, do not use `default` as a branch for valid or expected options. Enumerate those cases explicitly and reserve `default` for throwing an exception for unexpected values, or an assertion error if this code branch is unreachable.\n\n## Types, Generics, and Suppressions\n- Prefer type-safe constructs; avoid raw types and unchecked casts.\n- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).\n- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.\n\n## Naming Conventions\n- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.\n- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.\n- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.\n\n## Logging & Error Handling\n- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.\n- Always use parameterized logging (`logger.debug(\"operation [{}]\", value)`); never build strings via concatenation.\n- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.\n- Log levels:\n  - `TRACE`: highly verbose developer diagnostics; usually read alongside code.\n  - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.\n  - `INFO`: default-enabled operational milestones; prefer factual language.\n  - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.\n  - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.\n- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.\n- Tests can assert logging via `MockLog` for complex flows.\n\n## Javadoc & Comments\n- New packages/classes/public or abstract methods require Javadoc explaining the \"why\" rather than the implementation details.\n- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.\n- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.\n- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.\n\n## License Headers\n- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.\n- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.\n\n## Generated Files\n- Never hand-edit generated files. Instead, edit the source they are generated from and regenerate.\n- ANTLR-generated files can be regenerated by running the `regen` task on the relevant subproject.\n- Other generated files are regenerated by compiling the project.\n\n## Debugging Missing Tests\n\nWhen expected test methods are absent from results (not failed, not skipped — simply not present in the XML or binary event stream), check `muted-tests.yml` first. The build translates every entry into a Gradle `TestFilter.excludePattern`, which silently drops matching tests before the randomized runner receives them. A muted test fires no `testStarted` event and leaves no trace in `results-generic.bin`.\n   ```bash\n   grep 'ClassName\\|methodName' muted-tests.yml\n   ```\n\n### `No tests found for given includes: [**/*$*.class]`\n\nWhen a test task fails at execution with `No tests found for given includes: [**/*$*.class](exclude rules)`, it usually does **not** mean Gradle failed to detect the test class. The far more common cause is that **every test method in the targeted class is muted** in `muted-tests.yml`. With all methods excluded, the randomized runner enumerates zero runnable tests.\n\nThe behavior is environment-dependent: `MutedTestPlugin` calls `filter.setFailOnNoMatchingTests(buildParams.getCi() == false)`. So an all-muted suite **fails locally** (`ci == false`) with this exact message, but **passes silently in CI** (`ci == true`). This is especially misleading when verifying a freshly migrated or renamed test — it looks like a classpath/detection bug, but the test JVM does start (you'll see native-library and `FeatureFlag` log lines), builds any `@ClassRule` cluster *specs*, then exits in a few seconds without starting the cluster because no test method survived the mute filter.\n\nTo confirm: `grep ClassName muted-tests.yml`. To verify the migration/test actually runs, temporarily remove the matching mute entries (or run on a host where `ci` is true), then restore them.\n\n## Best Practices for Automation Agents\n- Never edit unrelated files; keep diffs tightly scoped to the task at hand.\n- Prefer Gradle tasks over ad-hoc scripts.\n- When scripting CLI sequences, leverage `gradlew` task.\n- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.\n- Do not add \"Co-Authored-By\" or any AI attribution trailers to commit messages, by any means—including `--trailer`, `-m`, or any other git flag. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary. Your harness may introduce a hook that automatically adds attributions trailers to relevant git commands. Use `bash -lc` or a similar approach in this case to conform to the rule.\n\n## Methods with Required Javadoc Reading\nIf you encounter any of the following methods, you must go and read their javadoc before taking any other actions:\n* `fullyLoadedAnalyzer`\n* `TestAnalyzer.statementError`\n* `TestAnalyzer.error`\n* `forciblyCast`\n* `EsqlCapabilities.Cap`\n* `FunctionDefinition.Builder#capabilities`\n\n## ES|QL tests\nIf you write or modify ES|QL csv-spec, rest, or yaml tests, read the javadoc for\n`EsqlCapabilities.Cap` and `FunctionDefinition.Builder#capabilities` before proceeding.\nThey describe two separate capability mechanisms and the rule for choosing between them.\n\n## Backwards compatibility\n- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName(\"<unique_descriptive_name>\")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.\n- Never hand-edit transport version resource files; always use the Gradle tasks. See `docs/internal/Versioning.md` for the full workflow.\n\nStay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.\n\n## Documentation\nWhen building or editing docs, read `docs/AGENTS.md` first.\n","category":"root","tokens":4032}]}