{"owner":"apache","repo":"pinot","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md"],"skills":{"AGENTS.md":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# Apache Pinot - AGENTS Guide\n\nThis file provides quick, practical guidance for coding agents working in this\nrepo. It is intentionally short and focused on day-to-day work.\n\n## Project overview\n- Apache Pinot is a real-time distributed OLAP datastore for low-latency\n  analytics over streaming and batch data.\n- Core runtime roles: broker (query routing), server (segment storage/execution),\n  controller (cluster metadata/management), minion (async tasks).\n\n## Repository layout (high level)\n- pinot-broker: broker query planning and scatter-gather.\n- pinot-controller: controller APIs, table/segment metadata, Helix management.\n- pinot-server: server query execution, segment loading, indexing.\n- pinot-minion: background tasks (segment conversion, purge, etc).\n- pinot-common / pinot-spi: shared utils, config, and SPI interfaces.\n- pinot-segment-local / pinot-segment-spi: segment generation, indexes, storage.\n- pinot-query-planner / pinot-query-runtime: multi-stage query (MSQ) engine.\n- pinot-connectors: external tooling to connect to Pinot\n- pinot-plugins: all pinot plugins.\n- pinot-tools: CLI and quickstart scripts.\n- pinot-integration-tests: end-to-end validation suites.\n- pinot-distribution: packaging artifacts.\n\n## pinot-plugins modules\n- pinot-input-format: input format plugin family.\n  - pinot-arrow: Apache Arrow input format support.\n  - pinot-avro: Avro input format support.\n  - pinot-avro-base: shared Avro utilities and base classes.\n  - pinot-bson: MongoDB BSON input format support.\n  - pinot-clp-log: CLP log input format support.\n  - pinot-confluent-avro: Confluent Schema Registry Avro input support.\n  - pinot-confluent-json: Confluent Schema Registry JSON input support.\n  - pinot-confluent-protobuf: Confluent Schema Registry Protobuf input support.\n  - pinot-orc: ORC input format support.\n  - pinot-json: JSON input format support.\n  - pinot-parquet: Parquet input format support.\n  - pinot-csv: CSV input format support.\n  - pinot-thrift: Thrift input format support.\n  - pinot-protobuf: Protobuf input format support.\n- pinot-file-system: filesystem plugin family.\n  - pinot-adls: Azure Data Lake Storage (ADLS) filesystem support.\n  - pinot-hdfs: Hadoop HDFS filesystem support.\n  - pinot-gcs: Google Cloud Storage filesystem support.\n  - pinot-s3: Amazon S3 filesystem support.\n- pinot-batch-ingestion: batch ingestion plugin family.\n  - pinot-batch-ingestion-common: shared batch ingestion APIs and utilities.\n  - pinot-batch-ingestion-spark-base: shared Spark ingestion base classes.\n  - pinot-batch-ingestion-spark-3: Spark 3 ingestion implementation.\n  - pinot-batch-ingestion-hadoop: Hadoop MapReduce ingestion implementation.\n  - pinot-batch-ingestion-standalone: standalone batch ingestion implementation.\n- pinot-stream-ingestion: stream ingestion plugin family.\n  - pinot-kafka-base: shared Kafka ingestion base classes.\n  - pinot-kafka-3.0: Kafka 3.x ingestion implementation.\n  - pinot-kafka-4.0: Kafka 4.x ingestion implementation.\n  - pinot-kinesis: AWS Kinesis ingestion implementation.\n  - pinot-pulsar: Apache Pulsar ingestion implementation.\n- pinot-minion-tasks: minion task plugin family.\n  - pinot-minion-builtin-tasks: built-in minion task implementations.\n- pinot-metrics: metrics reporter plugin family.\n  - pinot-dropwizard: Dropwizard Metrics reporter implementation.\n  - pinot-yammer: Yammer Metrics reporter implementation.\n  - pinot-compound-metrics: compound metrics implementation.\n- pinot-segment-writer: segment writer plugin family.\n  - pinot-segment-writer-file-based: file-based segment writer implementation.\n- pinot-segment-uploader: segment uploader plugin family.\n  - pinot-segment-uploader-default: default segment uploader implementation.\n- pinot-environment: environment provider plugin family.\n  - pinot-azure: Azure environment provider implementation.\n- pinot-timeseries-lang: time series language plugin family.\n  - pinot-timeseries-m3ql: M3QL language plugin implementation.\n- assembly-descriptor: Maven assembly descriptor for plugin packaging.\n\n## Build and test\n- Build JDK: Use JDK 21+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- Runtime JRE: Broker/server/controller/minion run on Java 21+.\n- Default build: `./mvnw clean install`\n- Faster dev build: `./mvnw verify -Ppinot-fastdev`\n- Full binary/shaded build:\n  `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- Build a module with deps: `./mvnw -pl pinot-server -am test`\n- Single test example: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- Quickstart (after build): `build/bin/quick-start-batch.sh`\n\n## Integration tests\n- Single integration test example: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n\n## Coding conventions and hygiene\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments with either `/** ... */` or `///` syntax (per JEP-467); service code targets Java 21 by default.\n- Keep license headers on all new source files.\n- Use `./mvnw license:format` to add headers to new files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Checkstyle config\n- Checkstyle rules and related config files live under `config/`.\n- Use the Maven wrapper (`./mvnw` on Unix-like systems or `mvnw.cmd` on Windows) to run `spotless:apply` to format code and `checkstyle:check` to validate style.\n- Run `./mvnw license:check` to validate license headers.\n\n## Pre-commit checks\nBefore pushing a commit, always run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\n## Change guidance\n- Query changes often touch broker planning and server execution; verify both.\n- Segment/index changes usually live under `pinot-segment-local` and\n  `pinot-segment-spi`.\n- Config or API changes should update relevant configs and docs where applicable.\n\n## Reference docs\n- `README.md` for build and quickstart details.\n- `CONTRIBUTING.md` for style, licensing, and contribution guidance.\n\n## Knowledge base (tool-neutral)\nThe `kb/` directory holds AI-optimized procedures and reference material that any\ncoding agent (Claude Code, Copilot, Cursor, GPT, Qwen, Gemini, etc.) can read.\nClaude Code's `.claude/skills/<name>/SKILL.md` and `.claude/agents/<name>.md`\nfiles are thin pointers that delegate to the kb/ procedures — non-Claude agents\nshould read kb/ directly.\n\n- `kb/skills/` — operational procedures and review checklists. See\n  [`kb/skills/README.md`](kb/skills/README.md) for the index. Each file is\n  self-contained; read it and follow it when your task matches the skill name.\n  - Operations: `precommit`, `run-test`, `quickstart`, `bench-compare`,\n    `flaky-analyze`.\n  - Review (eight domains, one per file): `review-config-backcompat`,\n    `review-concurrency-state`, `review-architecture`, `review-performance`,\n    `review-correctness-nulls`, `review-testing`, `review-naming-api`,\n    `review-process-scope`.\n- `kb/agents/code-reviewer.md` — orchestrator that dispatches the eight review\n  skills in parallel, aggregates findings, and emits a consolidated severity-\n  ranked report.\n- `kb/code-review-principles.md` — Pinot-specific review principles cited by id\n  (e.g. `C2.4`, `C6.1`) from the review skills.\n- `kb/CLAUDE.md` — kb/ authoring rules (one source of truth, terse, AI-optimized).\n\n**For non-Claude agents:** when a task matches a skill name (e.g. user asks for\na pre-commit check, a benchmark comparison, a flaky-test investigation, or a\ncode review), read the corresponding `kb/skills/<name>.md` and follow its\nprocedure. For a full code review, read `kb/agents/code-reviewer.md` and run the\neight review skills as it describes.\n","CLAUDE.md":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# CLAUDE.md - Apache Pinot\n\n## What is this project?\nApache Pinot is a real-time distributed OLAP datastore for low-latency analytics over streaming and batch data. Core runtime roles: **broker** (query routing), **server** (segment storage/execution), **controller** (cluster metadata/management), **minion** (async tasks).\n\n## Repository layout\n| Directory | Purpose |\n|---|---|\n| `pinot-broker` | Broker query planning and scatter-gather |\n| `pinot-controller` | Controller APIs, table/segment metadata, Helix management |\n| `pinot-server` | Server query execution, segment loading, indexing |\n| `pinot-minion` | Background tasks (segment conversion, purge, etc.) |\n| `pinot-common` / `pinot-spi` | Shared utils, config, and SPI interfaces |\n| `pinot-segment-local` / `pinot-segment-spi` | Segment generation, indexes, storage |\n| `pinot-query-planner` / `pinot-query-runtime` | Multi-stage query engine (MSQE) |\n| `pinot-connectors` | External tooling to connect to Pinot |\n| `pinot-plugins` | All Pinot plugins (input formats, filesystems, stream/batch ingestion, metrics, etc.) |\n| `pinot-tools` | CLI and quickstart scripts |\n| `pinot-integration-tests` | End-to-end validation suites |\n| `pinot-distribution` | Packaging artifacts |\n\n## Plugin modules (`pinot-plugins`)\n- **pinot-input-format**: input format plugin family.\n  - `pinot-arrow`: Apache Arrow input format support.\n  - `pinot-avro`: Avro input format support.\n  - `pinot-avro-base`: shared Avro utilities and base classes.\n  - `pinot-bson`: MongoDB BSON input format support.\n  - `pinot-clp-log`: CLP log input format support.\n  - `pinot-confluent-avro`: Confluent Schema Registry Avro input support.\n  - `pinot-confluent-json`: Confluent Schema Registry JSON input support.\n  - `pinot-confluent-protobuf`: Confluent Schema Registry Protobuf input support.\n  - `pinot-orc`: ORC input format support.\n  - `pinot-json`: JSON input format support.\n  - `pinot-parquet`: Parquet input format support.\n  - `pinot-csv`: CSV input format support.\n  - `pinot-thrift`: Thrift input format support.\n  - `pinot-protobuf`: Protobuf input format support.\n- **pinot-file-system**: filesystem plugin family.\n  - `pinot-adls`: Azure Data Lake Storage (ADLS) filesystem support.\n  - `pinot-hdfs`: Hadoop HDFS filesystem support.\n  - `pinot-gcs`: Google Cloud Storage filesystem support.\n  - `pinot-s3`: Amazon S3 filesystem support.\n- **pinot-batch-ingestion**: batch ingestion plugin family.\n  - `pinot-batch-ingestion-common`: shared batch ingestion APIs and utilities.\n  - `pinot-batch-ingestion-spark-base`: shared Spark ingestion base classes.\n  - `pinot-batch-ingestion-spark-3`: Spark 3 ingestion implementation.\n  - `pinot-batch-ingestion-hadoop`: Hadoop MapReduce ingestion implementation.\n  - `pinot-batch-ingestion-standalone`: standalone batch ingestion implementation.\n- **pinot-stream-ingestion**: stream ingestion plugin family.\n  - `pinot-kafka-base`: shared Kafka ingestion base classes.\n  - `pinot-kafka-3.0`: Kafka 3.x ingestion implementation.\n  - `pinot-kafka-4.0`: Kafka 4.x ingestion implementation.\n  - `pinot-kinesis`: AWS Kinesis ingestion implementation.\n  - `pinot-pulsar`: Apache Pulsar ingestion implementation.\n- **pinot-minion-tasks**: minion task plugin family.\n  - `pinot-minion-builtin-tasks`: built-in minion task implementations.\n- **pinot-metrics**: metrics reporter plugin family.\n  - `pinot-dropwizard`: Dropwizard Metrics reporter implementation.\n  - `pinot-yammer`: Yammer Metrics reporter implementation.\n  - `pinot-compound-metrics`: compound metrics implementation.\n- **pinot-segment-writer**: segment writer plugin family.\n  - `pinot-segment-writer-file-based`: file-based segment writer implementation.\n- **pinot-segment-uploader**: segment uploader plugin family.\n  - `pinot-segment-uploader-default`: default segment uploader implementation.\n- **pinot-environment**: environment provider plugin family.\n  - `pinot-azure`: Azure environment provider implementation.\n- **pinot-timeseries-lang**: time series language plugin family.\n  - `pinot-timeseries-m3ql`: M3QL language plugin implementation.\n- **assembly-descriptor**: Maven assembly descriptor for plugin packaging.\n\n## Build commands\n- **JDK**: Use JDK 25+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- **Default build**: `./mvnw clean install`\n- **Fast dev build**: `./mvnw verify -Ppinot-fastdev`\n- **Full binary/shaded build**: `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- **Build a module with deps**: `./mvnw -pl pinot-server -am test`\n- **Single test**: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- **Single integration test**: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n- **Quickstart (after build)**: `build/bin/quick-start-batch.sh`\n\n## Code style and formatting\n- Run `./mvnw spotless:apply` to auto-format code.\n- Run `./mvnw checkstyle:check` to validate style. Checkstyle config is in `config/checkstyle.xml`.\n- Run `./mvnw license:format` to add license headers to new files.\n- Run `./mvnw license:check` to validate license headers.\n- Always use the Maven wrapper (`./mvnw`) rather than a system `mvn`.\n\n## Coding conventions\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments (`/** ... */` or `///` syntax); service code targets Java 25 by default.\n- Keep Apache 2.0 license headers on all new source files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n- Avoid deprecated APIs in new code. If you must reference one (e.g., for backward-compat serialization or to test the deprecated path), justify it with a comment.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Pre-commit checks\nBefore pushing a commit, run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\nAdditionally, run the compiler warning check and fix what you can:\n5. `./mvnw test-compile -pl <module> -am -Dmaven.compiler.showDeprecation=true -Dmaven.compiler.showWarnings=true` — review warnings (deprecation, unchecked, etc.) in your changed code and fix where possible.\n\nClaude Code users can invoke `/precommit` to automate all of the above.\n\n## Change guidance\n- **Query changes** often touch broker planning and server execution; verify both.\n- **Segment/index changes** usually live under `pinot-segment-local` and `pinot-segment-spi`.\n- **Config or API changes** should update relevant configs and docs where applicable.\n\n## Mandatory code review\n\nAfter completing any coding task (bug fix, feature, refactor, etc.), you MUST run the `code-reviewer` agent before presenting the work as done. This is non-negotiable.\n\n- Pass ONLY the review scope and a one-line change description. Do NOT pass your analysis, reasoning, or opinions — the reviewer must judge the code independently.\n- Example invocation: `\"Review unstaged changes in pinot-broker. Change: added timeout to scatter-gather calls.\"`\n- If the reviewer finds CRITICAL issues, fix them before proceeding. MAJOR issues should be fixed unless you have strong justification. MINOR issues are at your discretion.\n- Do not skip the review even if the change seems trivial.\n\n## Common gotchas\n- This is a large multi-module Maven project. Building the entire project takes a long time — prefer building only the modules you need with `-pl <module> -am`.\n- When running tests, use `-Dtest=ClassName` to run a specific test class rather than the full suite.\n- Mixed-version compatibility matters — do not break wire protocols or serialization formats without careful consideration.\n"},"files":{"AGENTS.md":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# Apache Pinot - AGENTS Guide\n\nThis file provides quick, practical guidance for coding agents working in this\nrepo. It is intentionally short and focused on day-to-day work.\n\n## Project overview\n- Apache Pinot is a real-time distributed OLAP datastore for low-latency\n  analytics over streaming and batch data.\n- Core runtime roles: broker (query routing), server (segment storage/execution),\n  controller (cluster metadata/management), minion (async tasks).\n\n## Repository layout (high level)\n- pinot-broker: broker query planning and scatter-gather.\n- pinot-controller: controller APIs, table/segment metadata, Helix management.\n- pinot-server: server query execution, segment loading, indexing.\n- pinot-minion: background tasks (segment conversion, purge, etc).\n- pinot-common / pinot-spi: shared utils, config, and SPI interfaces.\n- pinot-segment-local / pinot-segment-spi: segment generation, indexes, storage.\n- pinot-query-planner / pinot-query-runtime: multi-stage query (MSQ) engine.\n- pinot-connectors: external tooling to connect to Pinot\n- pinot-plugins: all pinot plugins.\n- pinot-tools: CLI and quickstart scripts.\n- pinot-integration-tests: end-to-end validation suites.\n- pinot-distribution: packaging artifacts.\n\n## pinot-plugins modules\n- pinot-input-format: input format plugin family.\n  - pinot-arrow: Apache Arrow input format support.\n  - pinot-avro: Avro input format support.\n  - pinot-avro-base: shared Avro utilities and base classes.\n  - pinot-bson: MongoDB BSON input format support.\n  - pinot-clp-log: CLP log input format support.\n  - pinot-confluent-avro: Confluent Schema Registry Avro input support.\n  - pinot-confluent-json: Confluent Schema Registry JSON input support.\n  - pinot-confluent-protobuf: Confluent Schema Registry Protobuf input support.\n  - pinot-orc: ORC input format support.\n  - pinot-json: JSON input format support.\n  - pinot-parquet: Parquet input format support.\n  - pinot-csv: CSV input format support.\n  - pinot-thrift: Thrift input format support.\n  - pinot-protobuf: Protobuf input format support.\n- pinot-file-system: filesystem plugin family.\n  - pinot-adls: Azure Data Lake Storage (ADLS) filesystem support.\n  - pinot-hdfs: Hadoop HDFS filesystem support.\n  - pinot-gcs: Google Cloud Storage filesystem support.\n  - pinot-s3: Amazon S3 filesystem support.\n- pinot-batch-ingestion: batch ingestion plugin family.\n  - pinot-batch-ingestion-common: shared batch ingestion APIs and utilities.\n  - pinot-batch-ingestion-spark-base: shared Spark ingestion base classes.\n  - pinot-batch-ingestion-spark-3: Spark 3 ingestion implementation.\n  - pinot-batch-ingestion-hadoop: Hadoop MapReduce ingestion implementation.\n  - pinot-batch-ingestion-standalone: standalone batch ingestion implementation.\n- pinot-stream-ingestion: stream ingestion plugin family.\n  - pinot-kafka-base: shared Kafka ingestion base classes.\n  - pinot-kafka-3.0: Kafka 3.x ingestion implementation.\n  - pinot-kafka-4.0: Kafka 4.x ingestion implementation.\n  - pinot-kinesis: AWS Kinesis ingestion implementation.\n  - pinot-pulsar: Apache Pulsar ingestion implementation.\n- pinot-minion-tasks: minion task plugin family.\n  - pinot-minion-builtin-tasks: built-in minion task implementations.\n- pinot-metrics: metrics reporter plugin family.\n  - pinot-dropwizard: Dropwizard Metrics reporter implementation.\n  - pinot-yammer: Yammer Metrics reporter implementation.\n  - pinot-compound-metrics: compound metrics implementation.\n- pinot-segment-writer: segment writer plugin family.\n  - pinot-segment-writer-file-based: file-based segment writer implementation.\n- pinot-segment-uploader: segment uploader plugin family.\n  - pinot-segment-uploader-default: default segment uploader implementation.\n- pinot-environment: environment provider plugin family.\n  - pinot-azure: Azure environment provider implementation.\n- pinot-timeseries-lang: time series language plugin family.\n  - pinot-timeseries-m3ql: M3QL language plugin implementation.\n- assembly-descriptor: Maven assembly descriptor for plugin packaging.\n\n## Build and test\n- Build JDK: Use JDK 21+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- Runtime JRE: Broker/server/controller/minion run on Java 21+.\n- Default build: `./mvnw clean install`\n- Faster dev build: `./mvnw verify -Ppinot-fastdev`\n- Full binary/shaded build:\n  `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- Build a module with deps: `./mvnw -pl pinot-server -am test`\n- Single test example: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- Quickstart (after build): `build/bin/quick-start-batch.sh`\n\n## Integration tests\n- Single integration test example: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n\n## Coding conventions and hygiene\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments with either `/** ... */` or `///` syntax (per JEP-467); service code targets Java 21 by default.\n- Keep license headers on all new source files.\n- Use `./mvnw license:format` to add headers to new files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Checkstyle config\n- Checkstyle rules and related config files live under `config/`.\n- Use the Maven wrapper (`./mvnw` on Unix-like systems or `mvnw.cmd` on Windows) to run `spotless:apply` to format code and `checkstyle:check` to validate style.\n- Run `./mvnw license:check` to validate license headers.\n\n## Pre-commit checks\nBefore pushing a commit, always run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\n## Change guidance\n- Query changes often touch broker planning and server execution; verify both.\n- Segment/index changes usually live under `pinot-segment-local` and\n  `pinot-segment-spi`.\n- Config or API changes should update relevant configs and docs where applicable.\n\n## Reference docs\n- `README.md` for build and quickstart details.\n- `CONTRIBUTING.md` for style, licensing, and contribution guidance.\n\n## Knowledge base (tool-neutral)\nThe `kb/` directory holds AI-optimized procedures and reference material that any\ncoding agent (Claude Code, Copilot, Cursor, GPT, Qwen, Gemini, etc.) can read.\nClaude Code's `.claude/skills/<name>/SKILL.md` and `.claude/agents/<name>.md`\nfiles are thin pointers that delegate to the kb/ procedures — non-Claude agents\nshould read kb/ directly.\n\n- `kb/skills/` — operational procedures and review checklists. See\n  [`kb/skills/README.md`](kb/skills/README.md) for the index. Each file is\n  self-contained; read it and follow it when your task matches the skill name.\n  - Operations: `precommit`, `run-test`, `quickstart`, `bench-compare`,\n    `flaky-analyze`.\n  - Review (eight domains, one per file): `review-config-backcompat`,\n    `review-concurrency-state`, `review-architecture`, `review-performance`,\n    `review-correctness-nulls`, `review-testing`, `review-naming-api`,\n    `review-process-scope`.\n- `kb/agents/code-reviewer.md` — orchestrator that dispatches the eight review\n  skills in parallel, aggregates findings, and emits a consolidated severity-\n  ranked report.\n- `kb/code-review-principles.md` — Pinot-specific review principles cited by id\n  (e.g. `C2.4`, `C6.1`) from the review skills.\n- `kb/CLAUDE.md` — kb/ authoring rules (one source of truth, terse, AI-optimized).\n\n**For non-Claude agents:** when a task matches a skill name (e.g. user asks for\na pre-commit check, a benchmark comparison, a flaky-test investigation, or a\ncode review), read the corresponding `kb/skills/<name>.md` and follow its\nprocedure. For a full code review, read `kb/agents/code-reviewer.md` and run the\neight review skills as it describes.\n","CLAUDE.md":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# CLAUDE.md - Apache Pinot\n\n## What is this project?\nApache Pinot is a real-time distributed OLAP datastore for low-latency analytics over streaming and batch data. Core runtime roles: **broker** (query routing), **server** (segment storage/execution), **controller** (cluster metadata/management), **minion** (async tasks).\n\n## Repository layout\n| Directory | Purpose |\n|---|---|\n| `pinot-broker` | Broker query planning and scatter-gather |\n| `pinot-controller` | Controller APIs, table/segment metadata, Helix management |\n| `pinot-server` | Server query execution, segment loading, indexing |\n| `pinot-minion` | Background tasks (segment conversion, purge, etc.) |\n| `pinot-common` / `pinot-spi` | Shared utils, config, and SPI interfaces |\n| `pinot-segment-local` / `pinot-segment-spi` | Segment generation, indexes, storage |\n| `pinot-query-planner` / `pinot-query-runtime` | Multi-stage query engine (MSQE) |\n| `pinot-connectors` | External tooling to connect to Pinot |\n| `pinot-plugins` | All Pinot plugins (input formats, filesystems, stream/batch ingestion, metrics, etc.) |\n| `pinot-tools` | CLI and quickstart scripts |\n| `pinot-integration-tests` | End-to-end validation suites |\n| `pinot-distribution` | Packaging artifacts |\n\n## Plugin modules (`pinot-plugins`)\n- **pinot-input-format**: input format plugin family.\n  - `pinot-arrow`: Apache Arrow input format support.\n  - `pinot-avro`: Avro input format support.\n  - `pinot-avro-base`: shared Avro utilities and base classes.\n  - `pinot-bson`: MongoDB BSON input format support.\n  - `pinot-clp-log`: CLP log input format support.\n  - `pinot-confluent-avro`: Confluent Schema Registry Avro input support.\n  - `pinot-confluent-json`: Confluent Schema Registry JSON input support.\n  - `pinot-confluent-protobuf`: Confluent Schema Registry Protobuf input support.\n  - `pinot-orc`: ORC input format support.\n  - `pinot-json`: JSON input format support.\n  - `pinot-parquet`: Parquet input format support.\n  - `pinot-csv`: CSV input format support.\n  - `pinot-thrift`: Thrift input format support.\n  - `pinot-protobuf`: Protobuf input format support.\n- **pinot-file-system**: filesystem plugin family.\n  - `pinot-adls`: Azure Data Lake Storage (ADLS) filesystem support.\n  - `pinot-hdfs`: Hadoop HDFS filesystem support.\n  - `pinot-gcs`: Google Cloud Storage filesystem support.\n  - `pinot-s3`: Amazon S3 filesystem support.\n- **pinot-batch-ingestion**: batch ingestion plugin family.\n  - `pinot-batch-ingestion-common`: shared batch ingestion APIs and utilities.\n  - `pinot-batch-ingestion-spark-base`: shared Spark ingestion base classes.\n  - `pinot-batch-ingestion-spark-3`: Spark 3 ingestion implementation.\n  - `pinot-batch-ingestion-hadoop`: Hadoop MapReduce ingestion implementation.\n  - `pinot-batch-ingestion-standalone`: standalone batch ingestion implementation.\n- **pinot-stream-ingestion**: stream ingestion plugin family.\n  - `pinot-kafka-base`: shared Kafka ingestion base classes.\n  - `pinot-kafka-3.0`: Kafka 3.x ingestion implementation.\n  - `pinot-kafka-4.0`: Kafka 4.x ingestion implementation.\n  - `pinot-kinesis`: AWS Kinesis ingestion implementation.\n  - `pinot-pulsar`: Apache Pulsar ingestion implementation.\n- **pinot-minion-tasks**: minion task plugin family.\n  - `pinot-minion-builtin-tasks`: built-in minion task implementations.\n- **pinot-metrics**: metrics reporter plugin family.\n  - `pinot-dropwizard`: Dropwizard Metrics reporter implementation.\n  - `pinot-yammer`: Yammer Metrics reporter implementation.\n  - `pinot-compound-metrics`: compound metrics implementation.\n- **pinot-segment-writer**: segment writer plugin family.\n  - `pinot-segment-writer-file-based`: file-based segment writer implementation.\n- **pinot-segment-uploader**: segment uploader plugin family.\n  - `pinot-segment-uploader-default`: default segment uploader implementation.\n- **pinot-environment**: environment provider plugin family.\n  - `pinot-azure`: Azure environment provider implementation.\n- **pinot-timeseries-lang**: time series language plugin family.\n  - `pinot-timeseries-m3ql`: M3QL language plugin implementation.\n- **assembly-descriptor**: Maven assembly descriptor for plugin packaging.\n\n## Build commands\n- **JDK**: Use JDK 25+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- **Default build**: `./mvnw clean install`\n- **Fast dev build**: `./mvnw verify -Ppinot-fastdev`\n- **Full binary/shaded build**: `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- **Build a module with deps**: `./mvnw -pl pinot-server -am test`\n- **Single test**: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- **Single integration test**: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n- **Quickstart (after build)**: `build/bin/quick-start-batch.sh`\n\n## Code style and formatting\n- Run `./mvnw spotless:apply` to auto-format code.\n- Run `./mvnw checkstyle:check` to validate style. Checkstyle config is in `config/checkstyle.xml`.\n- Run `./mvnw license:format` to add license headers to new files.\n- Run `./mvnw license:check` to validate license headers.\n- Always use the Maven wrapper (`./mvnw`) rather than a system `mvn`.\n\n## Coding conventions\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments (`/** ... */` or `///` syntax); service code targets Java 25 by default.\n- Keep Apache 2.0 license headers on all new source files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n- Avoid deprecated APIs in new code. If you must reference one (e.g., for backward-compat serialization or to test the deprecated path), justify it with a comment.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Pre-commit checks\nBefore pushing a commit, run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\nAdditionally, run the compiler warning check and fix what you can:\n5. `./mvnw test-compile -pl <module> -am -Dmaven.compiler.showDeprecation=true -Dmaven.compiler.showWarnings=true` — review warnings (deprecation, unchecked, etc.) in your changed code and fix where possible.\n\nClaude Code users can invoke `/precommit` to automate all of the above.\n\n## Change guidance\n- **Query changes** often touch broker planning and server execution; verify both.\n- **Segment/index changes** usually live under `pinot-segment-local` and `pinot-segment-spi`.\n- **Config or API changes** should update relevant configs and docs where applicable.\n\n## Mandatory code review\n\nAfter completing any coding task (bug fix, feature, refactor, etc.), you MUST run the `code-reviewer` agent before presenting the work as done. This is non-negotiable.\n\n- Pass ONLY the review scope and a one-line change description. Do NOT pass your analysis, reasoning, or opinions — the reviewer must judge the code independently.\n- Example invocation: `\"Review unstaged changes in pinot-broker. Change: added timeout to scatter-gather calls.\"`\n- If the reviewer finds CRITICAL issues, fix them before proceeding. MAJOR issues should be fixed unless you have strong justification. MINOR issues are at your discretion.\n- Do not skip the review even if the change seems trivial.\n\n## Common gotchas\n- This is a large multi-module Maven project. Building the entire project takes a long time — prefer building only the modules you need with `-pl <module> -am`.\n- When running tests, use `-Dtest=ClassName` to run a specific test class rather than the full suite.\n- Mixed-version compatibility matters — do not break wire protocols or serialization formats without careful consideration.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# Apache Pinot - AGENTS Guide\n\nThis file provides quick, practical guidance for coding agents working in this\nrepo. It is intentionally short and focused on day-to-day work.\n\n## Project overview\n- Apache Pinot is a real-time distributed OLAP datastore for low-latency\n  analytics over streaming and batch data.\n- Core runtime roles: broker (query routing), server (segment storage/execution),\n  controller (cluster metadata/management), minion (async tasks).\n\n## Repository layout (high level)\n- pinot-broker: broker query planning and scatter-gather.\n- pinot-controller: controller APIs, table/segment metadata, Helix management.\n- pinot-server: server query execution, segment loading, indexing.\n- pinot-minion: background tasks (segment conversion, purge, etc).\n- pinot-common / pinot-spi: shared utils, config, and SPI interfaces.\n- pinot-segment-local / pinot-segment-spi: segment generation, indexes, storage.\n- pinot-query-planner / pinot-query-runtime: multi-stage query (MSQ) engine.\n- pinot-connectors: external tooling to connect to Pinot\n- pinot-plugins: all pinot plugins.\n- pinot-tools: CLI and quickstart scripts.\n- pinot-integration-tests: end-to-end validation suites.\n- pinot-distribution: packaging artifacts.\n\n## pinot-plugins modules\n- pinot-input-format: input format plugin family.\n  - pinot-arrow: Apache Arrow input format support.\n  - pinot-avro: Avro input format support.\n  - pinot-avro-base: shared Avro utilities and base classes.\n  - pinot-bson: MongoDB BSON input format support.\n  - pinot-clp-log: CLP log input format support.\n  - pinot-confluent-avro: Confluent Schema Registry Avro input support.\n  - pinot-confluent-json: Confluent Schema Registry JSON input support.\n  - pinot-confluent-protobuf: Confluent Schema Registry Protobuf input support.\n  - pinot-orc: ORC input format support.\n  - pinot-json: JSON input format support.\n  - pinot-parquet: Parquet input format support.\n  - pinot-csv: CSV input format support.\n  - pinot-thrift: Thrift input format support.\n  - pinot-protobuf: Protobuf input format support.\n- pinot-file-system: filesystem plugin family.\n  - pinot-adls: Azure Data Lake Storage (ADLS) filesystem support.\n  - pinot-hdfs: Hadoop HDFS filesystem support.\n  - pinot-gcs: Google Cloud Storage filesystem support.\n  - pinot-s3: Amazon S3 filesystem support.\n- pinot-batch-ingestion: batch ingestion plugin family.\n  - pinot-batch-ingestion-common: shared batch ingestion APIs and utilities.\n  - pinot-batch-ingestion-spark-base: shared Spark ingestion base classes.\n  - pinot-batch-ingestion-spark-3: Spark 3 ingestion implementation.\n  - pinot-batch-ingestion-hadoop: Hadoop MapReduce ingestion implementation.\n  - pinot-batch-ingestion-standalone: standalone batch ingestion implementation.\n- pinot-stream-ingestion: stream ingestion plugin family.\n  - pinot-kafka-base: shared Kafka ingestion base classes.\n  - pinot-kafka-3.0: Kafka 3.x ingestion implementation.\n  - pinot-kafka-4.0: Kafka 4.x ingestion implementation.\n  - pinot-kinesis: AWS Kinesis ingestion implementation.\n  - pinot-pulsar: Apache Pulsar ingestion implementation.\n- pinot-minion-tasks: minion task plugin family.\n  - pinot-minion-builtin-tasks: built-in minion task implementations.\n- pinot-metrics: metrics reporter plugin family.\n  - pinot-dropwizard: Dropwizard Metrics reporter implementation.\n  - pinot-yammer: Yammer Metrics reporter implementation.\n  - pinot-compound-metrics: compound metrics implementation.\n- pinot-segment-writer: segment writer plugin family.\n  - pinot-segment-writer-file-based: file-based segment writer implementation.\n- pinot-segment-uploader: segment uploader plugin family.\n  - pinot-segment-uploader-default: default segment uploader implementation.\n- pinot-environment: environment provider plugin family.\n  - pinot-azure: Azure environment provider implementation.\n- pinot-timeseries-lang: time series language plugin family.\n  - pinot-timeseries-m3ql: M3QL language plugin implementation.\n- assembly-descriptor: Maven assembly descriptor for plugin packaging.\n\n## Build and test\n- Build JDK: Use JDK 21+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- Runtime JRE: Broker/server/controller/minion run on Java 21+.\n- Default build: `./mvnw clean install`\n- Faster dev build: `./mvnw verify -Ppinot-fastdev`\n- Full binary/shaded build:\n  `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- Build a module with deps: `./mvnw -pl pinot-server -am test`\n- Single test example: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- Quickstart (after build): `build/bin/quick-start-batch.sh`\n\n## Integration tests\n- Single integration test example: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n\n## Coding conventions and hygiene\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments with either `/** ... */` or `///` syntax (per JEP-467); service code targets Java 21 by default.\n- Keep license headers on all new source files.\n- Use `./mvnw license:format` to add headers to new files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Checkstyle config\n- Checkstyle rules and related config files live under `config/`.\n- Use the Maven wrapper (`./mvnw` on Unix-like systems or `mvnw.cmd` on Windows) to run `spotless:apply` to format code and `checkstyle:check` to validate style.\n- Run `./mvnw license:check` to validate license headers.\n\n## Pre-commit checks\nBefore pushing a commit, always run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\n## Change guidance\n- Query changes often touch broker planning and server execution; verify both.\n- Segment/index changes usually live under `pinot-segment-local` and\n  `pinot-segment-spi`.\n- Config or API changes should update relevant configs and docs where applicable.\n\n## Reference docs\n- `README.md` for build and quickstart details.\n- `CONTRIBUTING.md` for style, licensing, and contribution guidance.\n\n## Knowledge base (tool-neutral)\nThe `kb/` directory holds AI-optimized procedures and reference material that any\ncoding agent (Claude Code, Copilot, Cursor, GPT, Qwen, Gemini, etc.) can read.\nClaude Code's `.claude/skills/<name>/SKILL.md` and `.claude/agents/<name>.md`\nfiles are thin pointers that delegate to the kb/ procedures — non-Claude agents\nshould read kb/ directly.\n\n- `kb/skills/` — operational procedures and review checklists. See\n  [`kb/skills/README.md`](kb/skills/README.md) for the index. Each file is\n  self-contained; read it and follow it when your task matches the skill name.\n  - Operations: `precommit`, `run-test`, `quickstart`, `bench-compare`,\n    `flaky-analyze`.\n  - Review (eight domains, one per file): `review-config-backcompat`,\n    `review-concurrency-state`, `review-architecture`, `review-performance`,\n    `review-correctness-nulls`, `review-testing`, `review-naming-api`,\n    `review-process-scope`.\n- `kb/agents/code-reviewer.md` — orchestrator that dispatches the eight review\n  skills in parallel, aggregates findings, and emits a consolidated severity-\n  ranked report.\n- `kb/code-review-principles.md` — Pinot-specific review principles cited by id\n  (e.g. `C2.4`, `C6.1`) from the review skills.\n- `kb/CLAUDE.md` — kb/ authoring rules (one source of truth, terse, AI-optimized).\n\n**For non-Claude agents:** when a task matches a skill name (e.g. user asks for\na pre-commit check, a benchmark comparison, a flaky-test investigation, or a\ncode review), read the corresponding `kb/skills/<name>.md` and follow its\nprocedure. For a full code review, read `kb/agents/code-reviewer.md` and run the\neight review skills as it describes.\n","category":"root","tokens":2527},{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"<!--\n\n    Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    \"License\"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n      http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n-->\n# CLAUDE.md - Apache Pinot\n\n## What is this project?\nApache Pinot is a real-time distributed OLAP datastore for low-latency analytics over streaming and batch data. Core runtime roles: **broker** (query routing), **server** (segment storage/execution), **controller** (cluster metadata/management), **minion** (async tasks).\n\n## Repository layout\n| Directory | Purpose |\n|---|---|\n| `pinot-broker` | Broker query planning and scatter-gather |\n| `pinot-controller` | Controller APIs, table/segment metadata, Helix management |\n| `pinot-server` | Server query execution, segment loading, indexing |\n| `pinot-minion` | Background tasks (segment conversion, purge, etc.) |\n| `pinot-common` / `pinot-spi` | Shared utils, config, and SPI interfaces |\n| `pinot-segment-local` / `pinot-segment-spi` | Segment generation, indexes, storage |\n| `pinot-query-planner` / `pinot-query-runtime` | Multi-stage query engine (MSQE) |\n| `pinot-connectors` | External tooling to connect to Pinot |\n| `pinot-plugins` | All Pinot plugins (input formats, filesystems, stream/batch ingestion, metrics, etc.) |\n| `pinot-tools` | CLI and quickstart scripts |\n| `pinot-integration-tests` | End-to-end validation suites |\n| `pinot-distribution` | Packaging artifacts |\n\n## Plugin modules (`pinot-plugins`)\n- **pinot-input-format**: input format plugin family.\n  - `pinot-arrow`: Apache Arrow input format support.\n  - `pinot-avro`: Avro input format support.\n  - `pinot-avro-base`: shared Avro utilities and base classes.\n  - `pinot-bson`: MongoDB BSON input format support.\n  - `pinot-clp-log`: CLP log input format support.\n  - `pinot-confluent-avro`: Confluent Schema Registry Avro input support.\n  - `pinot-confluent-json`: Confluent Schema Registry JSON input support.\n  - `pinot-confluent-protobuf`: Confluent Schema Registry Protobuf input support.\n  - `pinot-orc`: ORC input format support.\n  - `pinot-json`: JSON input format support.\n  - `pinot-parquet`: Parquet input format support.\n  - `pinot-csv`: CSV input format support.\n  - `pinot-thrift`: Thrift input format support.\n  - `pinot-protobuf`: Protobuf input format support.\n- **pinot-file-system**: filesystem plugin family.\n  - `pinot-adls`: Azure Data Lake Storage (ADLS) filesystem support.\n  - `pinot-hdfs`: Hadoop HDFS filesystem support.\n  - `pinot-gcs`: Google Cloud Storage filesystem support.\n  - `pinot-s3`: Amazon S3 filesystem support.\n- **pinot-batch-ingestion**: batch ingestion plugin family.\n  - `pinot-batch-ingestion-common`: shared batch ingestion APIs and utilities.\n  - `pinot-batch-ingestion-spark-base`: shared Spark ingestion base classes.\n  - `pinot-batch-ingestion-spark-3`: Spark 3 ingestion implementation.\n  - `pinot-batch-ingestion-hadoop`: Hadoop MapReduce ingestion implementation.\n  - `pinot-batch-ingestion-standalone`: standalone batch ingestion implementation.\n- **pinot-stream-ingestion**: stream ingestion plugin family.\n  - `pinot-kafka-base`: shared Kafka ingestion base classes.\n  - `pinot-kafka-3.0`: Kafka 3.x ingestion implementation.\n  - `pinot-kafka-4.0`: Kafka 4.x ingestion implementation.\n  - `pinot-kinesis`: AWS Kinesis ingestion implementation.\n  - `pinot-pulsar`: Apache Pulsar ingestion implementation.\n- **pinot-minion-tasks**: minion task plugin family.\n  - `pinot-minion-builtin-tasks`: built-in minion task implementations.\n- **pinot-metrics**: metrics reporter plugin family.\n  - `pinot-dropwizard`: Dropwizard Metrics reporter implementation.\n  - `pinot-yammer`: Yammer Metrics reporter implementation.\n  - `pinot-compound-metrics`: compound metrics implementation.\n- **pinot-segment-writer**: segment writer plugin family.\n  - `pinot-segment-writer-file-based`: file-based segment writer implementation.\n- **pinot-segment-uploader**: segment uploader plugin family.\n  - `pinot-segment-uploader-default`: default segment uploader implementation.\n- **pinot-environment**: environment provider plugin family.\n  - `pinot-azure`: Azure environment provider implementation.\n- **pinot-timeseries-lang**: time series language plugin family.\n  - `pinot-timeseries-m3ql`: M3QL language plugin implementation.\n- **assembly-descriptor**: Maven assembly descriptor for plugin packaging.\n\n## Build commands\n- **JDK**: Use JDK 25+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.\n- **Default build**: `./mvnw clean install`\n- **Fast dev build**: `./mvnw verify -Ppinot-fastdev`\n- **Full binary/shaded build**: `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`\n- **Build a module with deps**: `./mvnw -pl pinot-server -am test`\n- **Single test**: `./mvnw -pl pinot-segment-local -Dtest=RangeIndexTest test`\n- **Single integration test**: `./mvnw -pl pinot-integration-tests -am -Dtest=OfflineClusterIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false test`\n- **Quickstart (after build)**: `build/bin/quick-start-batch.sh`\n\n## Code style and formatting\n- Run `./mvnw spotless:apply` to auto-format code.\n- Run `./mvnw checkstyle:check` to validate style. Checkstyle config is in `config/checkstyle.xml`.\n- Run `./mvnw license:format` to add license headers to new files.\n- Run `./mvnw license:check` to validate license headers.\n- Always use the Maven wrapper (`./mvnw`) rather than a system `mvn`.\n\n## Coding conventions\n- Add class-level Javadoc for new classes; describe behavior and thread-safety.\n- Use Javadoc comments (`/** ... */` or `///` syntax); service code targets Java 25 by default.\n- Keep Apache 2.0 license headers on all new source files.\n- Preserve backward compatibility across mixed-version broker/server/controller.\n- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).\n- Prefer `List.of()`, `Set.of()`, and `Map.of()` for non-null immutable collection literals. Checkstyle blocks\n  `Collections.emptyList()`, `Collections.emptySet()`, and `Collections.emptyMap()`; use `List.of()`, `Set.of()`, and\n  `Map.of()` instead. Do not add blanket bans for `Collections.singleton*`; use them only when an element/key/value\n  argument is intentionally null because `List.of(null)`, `Set.of(null)`, and `Map.of(...)` with null keys or values\n  throw `NullPointerException`. Before replacing empty collection factories, check whether the value flows to\n  mutating callers. See\n  `kb/code-review-principles.md` C7.12.\n- Prefer targeted unit tests; use integration tests when behavior crosses roles.\n- Avoid deprecated APIs in new code. If you must reference one (e.g., for backward-compat serialization or to test the deprecated path), justify it with a comment.\n\n## Commit messages\n- Do not include `Co-authored-by` trailers that reference AI tools (e.g., Claude, Copilot).\n  - **Why**: These trailers propagate into squash-merge commits on GitHub, making the project history appear AI-authored rather than human-authored.\n  - **Fix**: Omit the `Co-authored-by` line entirely when committing.\n\n## Pre-commit checks\nBefore pushing a commit, run the following checks on the affected modules and fix any failures:\n1. `./mvnw spotless:apply -pl <module>` — auto-format code.\n2. `./mvnw checkstyle:check -pl <module>` — validate style conformance.\n3. `./mvnw license:format -pl <module>` — add missing license headers to new files.\n4. `./mvnw license:check -pl <module>` — verify all files have correct license headers.\n\nDo not push until all four checks pass cleanly.\n\nAdditionally, run the compiler warning check and fix what you can:\n5. `./mvnw test-compile -pl <module> -am -Dmaven.compiler.showDeprecation=true -Dmaven.compiler.showWarnings=true` — review warnings (deprecation, unchecked, etc.) in your changed code and fix where possible.\n\nClaude Code users can invoke `/precommit` to automate all of the above.\n\n## Change guidance\n- **Query changes** often touch broker planning and server execution; verify both.\n- **Segment/index changes** usually live under `pinot-segment-local` and `pinot-segment-spi`.\n- **Config or API changes** should update relevant configs and docs where applicable.\n\n## Mandatory code review\n\nAfter completing any coding task (bug fix, feature, refactor, etc.), you MUST run the `code-reviewer` agent before presenting the work as done. This is non-negotiable.\n\n- Pass ONLY the review scope and a one-line change description. Do NOT pass your analysis, reasoning, or opinions — the reviewer must judge the code independently.\n- Example invocation: `\"Review unstaged changes in pinot-broker. Change: added timeout to scatter-gather calls.\"`\n- If the reviewer finds CRITICAL issues, fix them before proceeding. MAJOR issues should be fixed unless you have strong justification. MINOR issues are at your discretion.\n- Do not skip the review even if the change seems trivial.\n\n## Common gotchas\n- This is a large multi-module Maven project. Building the entire project takes a long time — prefer building only the modules you need with `-pl <module> -am`.\n- When running tests, use `-Dtest=ClassName` to run a specific test class rather than the full suite.\n- Mixed-version compatibility matters — do not break wire protocols or serialization formats without careful consideration.\n","category":"root","tokens":2492}]}