{"owner":"apache","repo":"kafka","hasSkills":true,"totalSkillsCount":5,"totalTokensCount":4002,"categories":["plugin-manifest","anthropic-skill","root-instruction","cursor-rule"],"hasMcp":false,"mcpConfig":null,"found":["committer-tools/README.md","tools/src/test/resources/log4j2.yaml","AGENTS.md","tools/tools-api/src/test/resources/log4j2.yaml",".cursorrules (scala-kafka-cursorrules-prompt-file.mdc)"],"skills":{"committer-tools/README.md":"# Committer Tools\n\nThis directory contains scripts to help Apache Kafka committers with a few chores.\nSome of the scripts require a GitHub API token with write permissions. Only\ncommitters will be able to utilize such scripts.\n\n## Requirements\n\n- Python 3.x and pip\n- The GitHub CLI\n\n## Installation\n\n### 1. Check Python installation\n\nCheck if Python and pip are installed in your system.\n\n```bash\npython --version\npip --version\n```\n\n### 2. Set up a virtual environment (optional)\n\n```bash\npython -m venv venv\n\n# For Linux/macOS\nsource venv/bin/activate\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\n### 3. Install the required dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Install the GitHub CLI\n\nSee: https://cli.github.com/\n\n```bash\nbrew install gh\n```\n\n## Find Reviewers and Update to PR body\n\nThe reviewers.py script is used to simplify the process of producing our \"Reviewers:\"\nGit trailer to PR body. It parses the Git log to gather a set of \"Authors\" and \"Reviewers\". \nSome simple string prefix matching is done to find candidates. \nAfter entering the pull request number, the script updates the \"Reviewers:\" trailer accordingly. \nIf the PR body already contains a \"Reviewers:\" trailer, the script replaces it with the updated list of reviewers.\n\nUsage:\n\n```bash\npython reviewers.py\n```\n\n## Refresh Collaborators\n\nThe Refresh Collaborators script automates the process of fetching contributor\ndata from GitHub repositories, filtering top contributors who are not part of\nthe existing committers, and updating a local configuration file (.asf.yaml) to\ninclude these new contributors.\n\n> This script requires the Python dependencies and a GitHub auth token.\n\nYou need to set up a valid GitHub token to access the repository. After you\ngenerate it (or authenticate via GitHub CLI), this can be done by setting the\nGITHUB_TOKEN environment variable.\n\n```bash\n# For Linux/macOS\nexport GITHUB_TOKEN=\"your_github_token\"\n# Or if you use GitHub CLI\nexport GITHUB_TOKEN=\"$(gh auth token)\"\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\nUsage:\n\n```bash\npython refresh_collaborators.py\n```\n\n## Find Hanging Tests\n\nThis script is used to infer hanging tests from the Gradle output. It looks for\ntests that were STARTED but do not have a corresponding FINISHED or FAILED.\n\nUsage:\n\n```bash\npython find-unfinished-test.py ~/Downloads/logs_28218821016/5_build\\ _\\ JUnit\\ tests\\ Java\\ 11.txt\n\nFound tests that were started, but not finished:\n\n2024-09-10T20:31:26.6830206Z Gradle Test Run :streams:test > Gradle Test Executor 47 > StreamThreadTest > shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean) > \"shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean).processingThreadsEnabled=true\" STARTED\n```\n","tools/src/test/resources/log4j2.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: INFO\n AppenderRef:\n - ref: STDOUT\n\n Logger:\n - name: org.apache.kafka\n level: ERROR\n\n - name: org.eclipse.jetty\n level: ERROR\n","AGENTS.md":"\n\n# Agent Guide for Apache Kafka\n\nThis file is read by automated agents (security scanners, code analyzers,\nAI assistants) operating on this repository.\n\n## Project overview\n\nApache Kafka is a distributed event streaming platform. This repo is a **Gradle**\nbuild in **Java** and **Scala 2.13**.\n\n- **Supported JDK**: 11+ for `clients`, `generator`, and `streams`; 17+ for other modules.\n- **Build/test JDK**: Java 17 and 25.\n- **Build tool**: Gradle, wrapper `./gradlew` is recommended.\n\n## Repository layout (high level)\n\n| Path | Purpose |\n|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|\n| `clients/` | Producer, consumer, admin client APIs and protocol messages |\n| `core/` | Broker runtime, log, replication, request handling (gradually being removed along with Scala code) |\n| `server/`, `server-common/` | Broker/server components and shared server code |\n| `metadata/`, `raft/` | KRaft metadata layer and Raft implementation |\n| `storage/`, `storage/api/` | Log segments, checkpoints, tiered storage APIs |\n| `group-coordinator/`, `share-coordinator/`, `transaction-coordinator/`, `coordinator-common/` | Coordinators |\n| `connect/` | Kafka Connect (api, runtime, plugins, transforms) |\n| `streams/` | Kafka Streams (+ Scala, examples, upgrade system tests) |\n| `generator/` | RPC/message code generation |\n| `tools/`, `shell/` | CLI tools and kafka metadata shell |\n| `examples/` | Kafka producer and consumer examples |\n| `jmh-benchmarks/` | Kafka benchmarks tests |\n| `trogdor/` | test framework |\n| `tests/` | System test harness (see `tests/README.md`) |\n| `config/` | Sample broker/controller configuration |\n| `checkstyle/` | Checkstyle rules and import-control XML |\n| `docs/` | Documentation sources |\n| `docker/` | Docker images, build/test scripts, and examples for JVM and native Kafka |\n\n\n## Build commands\n\n```bash\n# Compile jars\n./gradlew jar\n\n# Release tarball (output under core/build/distributions/)\n./gradlew clean releaseTarGz\n\n# Publish to local Maven (skip signing for local dev)\n./gradlew -PskipSigning=true publishToMavenLocal\n\n# Regenerate RPC/message classes after branch switches\n./gradlew processMessages processTestMessages\n\n# List tasks\n./gradlew tasks\n```\n\nModule-scoped build:\n\n```bash\n./gradlew core:jar\n./gradlew clients:jar\n./gradlew :streams:testAll\n```\n\n## Test commands\n\n```bash\n# All unit + integration tests\n./gradlew test\n\n# Split suites\n./gradlew unitTest\n./gradlew integrationTest\n\n# Include flaky tests (marked @Flaky)\n./gradlew test -Pkafka.test.run.flaky=true\n\n# Re-run without considering up-to-date\n./gradlew test --rerun-tasks\n\n# Single test class\n./gradlew clients:test --tests RequestResponseTest\n\n# Single test method\n./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic\n\n# Coverage (whole project or one module)\n./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n```\n\n**Test retries** (optional): `-PmaxTestRetries=1 -PmaxTestRetryFailures=3`\n\n**Logging**: adjust `log4j2.yaml` under `/src/test/resources/` for verbose test logs.\n\n**System tests**: see [tests/README.md](tests/README.md).\n\n## Code quality\n\nRun before submitting Java changes:\n\n```bash\n./gradlew spotlessApply # fix import order / formatting\n./gradlew checkstyleMain checkstyleTest spotlessCheck\n./gradlew spotbugsMain spotbugsTest -x test # optional static analysis\n```\n\nCheckstyle reports: `/build/reports/checkstyle/`.\n\n## Coding conventions\n\n- **Commits / PR titles**: Start with `KAFKA-XXXXX`, `MINOR`, or `HOTFIX`. Use `KAFKA-XXXXX` only when there is a valid Jira ticket for the change.\n- **PR descriptions**: The PR title and description become the squash-merged commit message. In the description, explain what the change does and why it is needed, if not obvious. Keep it short for small changes. Put any notes for reviewers, such as the tests you ran, in a separate comment.\n- **AI-generated contributions**: Follow the [AI-Generated Contributions](CONTRIBUTING.md#ai-generated-contributions) section in `CONTRIBUTING.md` — add a `Co-Authored-By` or `Generated-by` commit trailer for AI-assisted changes.\n- **Public API / KIP**: Changes to public interfaces, wire protocol, configurations, or metrics generally require a [KIP](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). See the `javadoc` `include` list in [build.gradle](build.gradle) for packages treated as public API.\n- **License header**: New source files need the standard ASF license header (see existing files in the same module).\n- **Checkstyle**: Enforced on main and test sources; rules in `checkstyle/`. Import order is checked via Spotless.\n- **Generated code**: Do not hand-edit generated RPC/message sources. Regenerate with `processMessages` / `processTestMessages`. See [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md).\n- **Scope**: Prefer targeted unit tests in the module you change; integration/system tests for cross-component behavior.\n- **Compatibility**: clients, generator, and streams modules support Java 11+; all other modules require Java 17+. Avoid APIs that break those release levels.\n\n## Security\n\nSee [SECURITY.md](./SECURITY.md) for how to report a vulnerability and for links\nto the Apache Kafka security model under [docs/security/](./docs/security/).\n\nAgents that scan this repository should consult `SECURITY.md` and the linked\nsecurity model before reporting issues. In particular, the core model defines\nwhat is in and out of scope, how reports are classified, and a list of known\nnon-findings; each component page adds its own known non-findings.\n\n## Reference docs\n\n| Topic | Location |\n|--------------------------------|----------|\n| Build, test, broker quickstart | [README.md](README.md) |\n| Contributing | [CONTRIBUTING.md](CONTRIBUTING.md), https://kafka.apache.org/contributing.html |\n| Message protocol / codegen | [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md) |\n| System tests | [tests/README.md](tests/README.md) |\n| JMH benchmarks | [jmh-benchmarks/README.md](jmh-benchmarks/README.md) |\n| Client examples | [examples/README.md](examples/README.md) |\n| Security | [SECURITY.md](SECURITY.md) |\n","tools/tools-api/src/test/resources/log4j2.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: \"OFF\"\n AppenderRef:\n - ref: STDOUT\n Logger:\n - name: kafka\n level: WARN\n\n - name: org.apache.kafka\n level: WARN\n",".cursorrules (scala-kafka-cursorrules-prompt-file.mdc)":"---\ndescription: \"Cursor rules for Scala Kafka.\"\nglobs: **/*\nalwaysApply: false\n---\n# general-scala-clean-code.mdc\n\n- Declare vals/vars as close as possible to first use.\n- Name length should be proportional to scope: 1-2 chars allowed only inside small lambdas.\n- Avoid nested for-comprehensions deeper than two levels; factor out steps into helpers.\n- Split a single source file by responsibility.\n- Use *tail-rec* optimisation (`@tailrec`) where appropriate.\n- Prefer *immutable* collections and avoid mutation during iteration.\n- When interop with Java forces mutability, wrap it in a pure facade with the use of .asScala, to retain functional API.\n- When something is nullable, wrap it into an Option, to retain functional API.\n- Keep cyclomatic complexity below 10 for any method; IDE inspections should warn.\n\n# general-scala-development-practices.mdc\n\n# ========== GENERAL PRINCIPLES ==========\n- You are an experienced Senior Scala Developer.\n- You always adhere to SOLID, DRY, KISS and YAGNI principles.\n- Prefer *pure* functions; minimise side-effects. Where effects are required, make them explicit (e.g. using scala.util.Try, Either, or cats-effect IO/Task if adopted).\n- Use *val* over *var*; collections must be immutable unless mutability is proven cheaper & safe.\n- Replace *null* with Option, Either or a domain-specific ADT.\n- Use pattern matching exhaustively and handle the *default* case only when truly open-ended.\n- Prefer for-comprehensions, map/flatMap/fold, and higher-order functions over imperative loops.\n- Prefer *case classes* and *sealed traits* for algebraic data types.\n- Extract common logic into private or package-private helpers; avoid long methods (> 30 LOC).\n- Prefer extension methods or type classes over inheritance when adding behaviour.\n- Keep public APIs small, surface only what the module owns.\n- Break every task into the smallest composable pure functions before wiring them together.\n\n# ========== NAMING & SYNTAX ==========\n- Class / object / trait names are UpperCamelCase nouns (e.g. *NotificationStreamApp*).\n- Methods & vals are lowerCamelCase verbs or nouns (e.g. *process*, *serde*, *productKey*).\n- Constants use `SCREAMING_SNAKE_CASE`.\n- Similar to Java’s static final members, if the member is final, immutable and it belongs to a package object or an object, it may be considered a constant.\n- Symbolic names (`|>`) are allowed *only* when they align with widespread FP idioms.\n- Match expressions use `match`/`case` over nested if/else chains; for simple two-branch logic prefer `if … then … else …` expressions.\n\n# ========== ERROR HANDLING & LOGGING ==========\n- Catch the most specific Exception first; convert checked Java exceptions to an ADT or `Try`.\n- No empty `catch` blocks; log at *debug* or *error* level with a meaningful message.\n- Leverage `scala.util.Using` (or cats-effect `Resource`) for auto-closing resources.\n- Avoid “defensive” logging or `println`; use SLF4J (Logback) with the *scala-logging* wrapper.\n\n# ========== TESTING ==========\n- Use ScalaTest in a **Given-When-Then** layout with the use of AnyFlatSpec.\n- Focus on critical paths and business invariants; do not over-test boilerplate.\n- Property-based tests (ScalaCheck) for pure functions with non-trivial invariants.\n- Set up integration tests as a subproject named “integration” and treat integration tests as standard tests\n\n# ========== PERFORMANCE & SAFETY ==========\n- Avoid blocking calls inside Kafka stream processing; if unavoidable, off-load to a dedicated thread-pool.\n- Convert Java collections to Scala equivalents once at the boundary; never bounce back and forth.\n- Use underscore-separated digits for large numeric literals (e.g. `val timeoutMs = 30_000`).\n\n# ========== MODERN SCALA 3 FEATURES ==========\n- Use *Enums* for finite alternatives instead of Java-style enums.\n- Embrace *opaque types* to avoid accidental misuse of primitive wrappers.\n- Use *context parameters* (`using`) for type-class evidence instead of classic implicit lists when convenient.\n- Prefer `given`/`using` syntax over `implicit` where supported.\n\n# ========== CLEAN BUILD ==========\n- The sbt build uses **scalafmt** for formatting; treat any scalafmt violation as a build error.\n\n# kafka-development-practices.mdc\n\n- All topic names config values (Typesafe Config or pure-config).\n- Use Format or Codec from the JSON or AVRO or another library that is being used in the project.\n- Streams logic must be tested with `TopologyTestDriver` (unit-test) plus an integration test against local Kafka.\n\n# linting-formatting.mdc\n\n- **scalafmt:** Enforce Google-inspired scalafmt configuration with `maxColumn = 100`.\n"},"files":{"committer-tools/README.md":"# Committer Tools\n\nThis directory contains scripts to help Apache Kafka committers with a few chores.\nSome of the scripts require a GitHub API token with write permissions. Only\ncommitters will be able to utilize such scripts.\n\n## Requirements\n\n- Python 3.x and pip\n- The GitHub CLI\n\n## Installation\n\n### 1. Check Python installation\n\nCheck if Python and pip are installed in your system.\n\n```bash\npython --version\npip --version\n```\n\n### 2. Set up a virtual environment (optional)\n\n```bash\npython -m venv venv\n\n# For Linux/macOS\nsource venv/bin/activate\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\n### 3. Install the required dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Install the GitHub CLI\n\nSee: https://cli.github.com/\n\n```bash\nbrew install gh\n```\n\n## Find Reviewers and Update to PR body\n\nThe reviewers.py script is used to simplify the process of producing our \"Reviewers:\"\nGit trailer to PR body. It parses the Git log to gather a set of \"Authors\" and \"Reviewers\". \nSome simple string prefix matching is done to find candidates. \nAfter entering the pull request number, the script updates the \"Reviewers:\" trailer accordingly. \nIf the PR body already contains a \"Reviewers:\" trailer, the script replaces it with the updated list of reviewers.\n\nUsage:\n\n```bash\npython reviewers.py\n```\n\n## Refresh Collaborators\n\nThe Refresh Collaborators script automates the process of fetching contributor\ndata from GitHub repositories, filtering top contributors who are not part of\nthe existing committers, and updating a local configuration file (.asf.yaml) to\ninclude these new contributors.\n\n> This script requires the Python dependencies and a GitHub auth token.\n\nYou need to set up a valid GitHub token to access the repository. After you\ngenerate it (or authenticate via GitHub CLI), this can be done by setting the\nGITHUB_TOKEN environment variable.\n\n```bash\n# For Linux/macOS\nexport GITHUB_TOKEN=\"your_github_token\"\n# Or if you use GitHub CLI\nexport GITHUB_TOKEN=\"$(gh auth token)\"\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\nUsage:\n\n```bash\npython refresh_collaborators.py\n```\n\n## Find Hanging Tests\n\nThis script is used to infer hanging tests from the Gradle output. It looks for\ntests that were STARTED but do not have a corresponding FINISHED or FAILED.\n\nUsage:\n\n```bash\npython find-unfinished-test.py ~/Downloads/logs_28218821016/5_build\\ _\\ JUnit\\ tests\\ Java\\ 11.txt\n\nFound tests that were started, but not finished:\n\n2024-09-10T20:31:26.6830206Z Gradle Test Run :streams:test > Gradle Test Executor 47 > StreamThreadTest > shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean) > \"shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean).processingThreadsEnabled=true\" STARTED\n```\n","tools/src/test/resources/log4j2.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: INFO\n AppenderRef:\n - ref: STDOUT\n\n Logger:\n - name: org.apache.kafka\n level: ERROR\n\n - name: org.eclipse.jetty\n level: ERROR\n","AGENTS.md":"\n\n# Agent Guide for Apache Kafka\n\nThis file is read by automated agents (security scanners, code analyzers,\nAI assistants) operating on this repository.\n\n## Project overview\n\nApache Kafka is a distributed event streaming platform. This repo is a **Gradle**\nbuild in **Java** and **Scala 2.13**.\n\n- **Supported JDK**: 11+ for `clients`, `generator`, and `streams`; 17+ for other modules.\n- **Build/test JDK**: Java 17 and 25.\n- **Build tool**: Gradle, wrapper `./gradlew` is recommended.\n\n## Repository layout (high level)\n\n| Path | Purpose |\n|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|\n| `clients/` | Producer, consumer, admin client APIs and protocol messages |\n| `core/` | Broker runtime, log, replication, request handling (gradually being removed along with Scala code) |\n| `server/`, `server-common/` | Broker/server components and shared server code |\n| `metadata/`, `raft/` | KRaft metadata layer and Raft implementation |\n| `storage/`, `storage/api/` | Log segments, checkpoints, tiered storage APIs |\n| `group-coordinator/`, `share-coordinator/`, `transaction-coordinator/`, `coordinator-common/` | Coordinators |\n| `connect/` | Kafka Connect (api, runtime, plugins, transforms) |\n| `streams/` | Kafka Streams (+ Scala, examples, upgrade system tests) |\n| `generator/` | RPC/message code generation |\n| `tools/`, `shell/` | CLI tools and kafka metadata shell |\n| `examples/` | Kafka producer and consumer examples |\n| `jmh-benchmarks/` | Kafka benchmarks tests |\n| `trogdor/` | test framework |\n| `tests/` | System test harness (see `tests/README.md`) |\n| `config/` | Sample broker/controller configuration |\n| `checkstyle/` | Checkstyle rules and import-control XML |\n| `docs/` | Documentation sources |\n| `docker/` | Docker images, build/test scripts, and examples for JVM and native Kafka |\n\n\n## Build commands\n\n```bash\n# Compile jars\n./gradlew jar\n\n# Release tarball (output under core/build/distributions/)\n./gradlew clean releaseTarGz\n\n# Publish to local Maven (skip signing for local dev)\n./gradlew -PskipSigning=true publishToMavenLocal\n\n# Regenerate RPC/message classes after branch switches\n./gradlew processMessages processTestMessages\n\n# List tasks\n./gradlew tasks\n```\n\nModule-scoped build:\n\n```bash\n./gradlew core:jar\n./gradlew clients:jar\n./gradlew :streams:testAll\n```\n\n## Test commands\n\n```bash\n# All unit + integration tests\n./gradlew test\n\n# Split suites\n./gradlew unitTest\n./gradlew integrationTest\n\n# Include flaky tests (marked @Flaky)\n./gradlew test -Pkafka.test.run.flaky=true\n\n# Re-run without considering up-to-date\n./gradlew test --rerun-tasks\n\n# Single test class\n./gradlew clients:test --tests RequestResponseTest\n\n# Single test method\n./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic\n\n# Coverage (whole project or one module)\n./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n```\n\n**Test retries** (optional): `-PmaxTestRetries=1 -PmaxTestRetryFailures=3`\n\n**Logging**: adjust `log4j2.yaml` under `/src/test/resources/` for verbose test logs.\n\n**System tests**: see [tests/README.md](tests/README.md).\n\n## Code quality\n\nRun before submitting Java changes:\n\n```bash\n./gradlew spotlessApply # fix import order / formatting\n./gradlew checkstyleMain checkstyleTest spotlessCheck\n./gradlew spotbugsMain spotbugsTest -x test # optional static analysis\n```\n\nCheckstyle reports: `/build/reports/checkstyle/`.\n\n## Coding conventions\n\n- **Commits / PR titles**: Start with `KAFKA-XXXXX`, `MINOR`, or `HOTFIX`. Use `KAFKA-XXXXX` only when there is a valid Jira ticket for the change.\n- **PR descriptions**: The PR title and description become the squash-merged commit message. In the description, explain what the change does and why it is needed, if not obvious. Keep it short for small changes. Put any notes for reviewers, such as the tests you ran, in a separate comment.\n- **AI-generated contributions**: Follow the [AI-Generated Contributions](CONTRIBUTING.md#ai-generated-contributions) section in `CONTRIBUTING.md` — add a `Co-Authored-By` or `Generated-by` commit trailer for AI-assisted changes.\n- **Public API / KIP**: Changes to public interfaces, wire protocol, configurations, or metrics generally require a [KIP](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). See the `javadoc` `include` list in [build.gradle](build.gradle) for packages treated as public API.\n- **License header**: New source files need the standard ASF license header (see existing files in the same module).\n- **Checkstyle**: Enforced on main and test sources; rules in `checkstyle/`. Import order is checked via Spotless.\n- **Generated code**: Do not hand-edit generated RPC/message sources. Regenerate with `processMessages` / `processTestMessages`. See [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md).\n- **Scope**: Prefer targeted unit tests in the module you change; integration/system tests for cross-component behavior.\n- **Compatibility**: clients, generator, and streams modules support Java 11+; all other modules require Java 17+. Avoid APIs that break those release levels.\n\n## Security\n\nSee [SECURITY.md](./SECURITY.md) for how to report a vulnerability and for links\nto the Apache Kafka security model under [docs/security/](./docs/security/).\n\nAgents that scan this repository should consult `SECURITY.md` and the linked\nsecurity model before reporting issues. In particular, the core model defines\nwhat is in and out of scope, how reports are classified, and a list of known\nnon-findings; each component page adds its own known non-findings.\n\n## Reference docs\n\n| Topic | Location |\n|--------------------------------|----------|\n| Build, test, broker quickstart | [README.md](README.md) |\n| Contributing | [CONTRIBUTING.md](CONTRIBUTING.md), https://kafka.apache.org/contributing.html |\n| Message protocol / codegen | [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md) |\n| System tests | [tests/README.md](tests/README.md) |\n| JMH benchmarks | [jmh-benchmarks/README.md](jmh-benchmarks/README.md) |\n| Client examples | [examples/README.md](examples/README.md) |\n| Security | [SECURITY.md](SECURITY.md) |\n","tools/tools-api/src/test/resources/log4j2.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: \"OFF\"\n AppenderRef:\n - ref: STDOUT\n Logger:\n - name: kafka\n level: WARN\n\n - name: org.apache.kafka\n level: WARN\n",".cursorrules (scala-kafka-cursorrules-prompt-file.mdc)":"---\ndescription: \"Cursor rules for Scala Kafka.\"\nglobs: **/*\nalwaysApply: false\n---\n# general-scala-clean-code.mdc\n\n- Declare vals/vars as close as possible to first use.\n- Name length should be proportional to scope: 1-2 chars allowed only inside small lambdas.\n- Avoid nested for-comprehensions deeper than two levels; factor out steps into helpers.\n- Split a single source file by responsibility.\n- Use *tail-rec* optimisation (`@tailrec`) where appropriate.\n- Prefer *immutable* collections and avoid mutation during iteration.\n- When interop with Java forces mutability, wrap it in a pure facade with the use of .asScala, to retain functional API.\n- When something is nullable, wrap it into an Option, to retain functional API.\n- Keep cyclomatic complexity below 10 for any method; IDE inspections should warn.\n\n# general-scala-development-practices.mdc\n\n# ========== GENERAL PRINCIPLES ==========\n- You are an experienced Senior Scala Developer.\n- You always adhere to SOLID, DRY, KISS and YAGNI principles.\n- Prefer *pure* functions; minimise side-effects. Where effects are required, make them explicit (e.g. using scala.util.Try, Either, or cats-effect IO/Task if adopted).\n- Use *val* over *var*; collections must be immutable unless mutability is proven cheaper & safe.\n- Replace *null* with Option, Either or a domain-specific ADT.\n- Use pattern matching exhaustively and handle the *default* case only when truly open-ended.\n- Prefer for-comprehensions, map/flatMap/fold, and higher-order functions over imperative loops.\n- Prefer *case classes* and *sealed traits* for algebraic data types.\n- Extract common logic into private or package-private helpers; avoid long methods (> 30 LOC).\n- Prefer extension methods or type classes over inheritance when adding behaviour.\n- Keep public APIs small, surface only what the module owns.\n- Break every task into the smallest composable pure functions before wiring them together.\n\n# ========== NAMING & SYNTAX ==========\n- Class / object / trait names are UpperCamelCase nouns (e.g. *NotificationStreamApp*).\n- Methods & vals are lowerCamelCase verbs or nouns (e.g. *process*, *serde*, *productKey*).\n- Constants use `SCREAMING_SNAKE_CASE`.\n- Similar to Java’s static final members, if the member is final, immutable and it belongs to a package object or an object, it may be considered a constant.\n- Symbolic names (`|>`) are allowed *only* when they align with widespread FP idioms.\n- Match expressions use `match`/`case` over nested if/else chains; for simple two-branch logic prefer `if … then … else …` expressions.\n\n# ========== ERROR HANDLING & LOGGING ==========\n- Catch the most specific Exception first; convert checked Java exceptions to an ADT or `Try`.\n- No empty `catch` blocks; log at *debug* or *error* level with a meaningful message.\n- Leverage `scala.util.Using` (or cats-effect `Resource`) for auto-closing resources.\n- Avoid “defensive” logging or `println`; use SLF4J (Logback) with the *scala-logging* wrapper.\n\n# ========== TESTING ==========\n- Use ScalaTest in a **Given-When-Then** layout with the use of AnyFlatSpec.\n- Focus on critical paths and business invariants; do not over-test boilerplate.\n- Property-based tests (ScalaCheck) for pure functions with non-trivial invariants.\n- Set up integration tests as a subproject named “integration” and treat integration tests as standard tests\n\n# ========== PERFORMANCE & SAFETY ==========\n- Avoid blocking calls inside Kafka stream processing; if unavoidable, off-load to a dedicated thread-pool.\n- Convert Java collections to Scala equivalents once at the boundary; never bounce back and forth.\n- Use underscore-separated digits for large numeric literals (e.g. `val timeoutMs = 30_000`).\n\n# ========== MODERN SCALA 3 FEATURES ==========\n- Use *Enums* for finite alternatives instead of Java-style enums.\n- Embrace *opaque types* to avoid accidental misuse of primitive wrappers.\n- Use *context parameters* (`using`) for type-class evidence instead of classic implicit lists when convenient.\n- Prefer `given`/`using` syntax over `implicit` where supported.\n\n# ========== CLEAN BUILD ==========\n- The sbt build uses **scalafmt** for formatting; treat any scalafmt violation as a build error.\n\n# kafka-development-practices.mdc\n\n- All topic names config values (Typesafe Config or pure-config).\n- Use Format or Codec from the JSON or AVRO or another library that is being used in the project.\n- Streams logic must be tested with `TopologyTestDriver` (unit-test) plus an integration test against local Kafka.\n\n# linting-formatting.mdc\n\n- **scalafmt:** Enforce Google-inspired scalafmt configuration with `maxColumn = 100`.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/apache/kafka/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"\n\n# Agent Guide for Apache Kafka\n\nThis file is read by automated agents (security scanners, code analyzers,\nAI assistants) operating on this repository.\n\n## Project overview\n\nApache Kafka is a distributed event streaming platform. This repo is a **Gradle**\nbuild in **Java** and **Scala 2.13**.\n\n- **Supported JDK**: 11+ for `clients`, `generator`, and `streams`; 17+ for other modules.\n- **Build/test JDK**: Java 17 and 25.\n- **Build tool**: Gradle, wrapper `./gradlew` is recommended.\n\n## Repository layout (high level)\n\n| Path | Purpose |\n|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|\n| `clients/` | Producer, consumer, admin client APIs and protocol messages |\n| `core/` | Broker runtime, log, replication, request handling (gradually being removed along with Scala code) |\n| `server/`, `server-common/` | Broker/server components and shared server code |\n| `metadata/`, `raft/` | KRaft metadata layer and Raft implementation |\n| `storage/`, `storage/api/` | Log segments, checkpoints, tiered storage APIs |\n| `group-coordinator/`, `share-coordinator/`, `transaction-coordinator/`, `coordinator-common/` | Coordinators |\n| `connect/` | Kafka Connect (api, runtime, plugins, transforms) |\n| `streams/` | Kafka Streams (+ Scala, examples, upgrade system tests) |\n| `generator/` | RPC/message code generation |\n| `tools/`, `shell/` | CLI tools and kafka metadata shell |\n| `examples/` | Kafka producer and consumer examples |\n| `jmh-benchmarks/` | Kafka benchmarks tests |\n| `trogdor/` | test framework |\n| `tests/` | System test harness (see `tests/README.md`) |\n| `config/` | Sample broker/controller configuration |\n| `checkstyle/` | Checkstyle rules and import-control XML |\n| `docs/` | Documentation sources |\n| `docker/` | Docker images, build/test scripts, and examples for JVM and native Kafka |\n\n\n## Build commands\n\n```bash\n# Compile jars\n./gradlew jar\n\n# Release tarball (output under core/build/distributions/)\n./gradlew clean releaseTarGz\n\n# Publish to local Maven (skip signing for local dev)\n./gradlew -PskipSigning=true publishToMavenLocal\n\n# Regenerate RPC/message classes after branch switches\n./gradlew processMessages processTestMessages\n\n# List tasks\n./gradlew tasks\n```\n\nModule-scoped build:\n\n```bash\n./gradlew core:jar\n./gradlew clients:jar\n./gradlew :streams:testAll\n```\n\n## Test commands\n\n```bash\n# All unit + integration tests\n./gradlew test\n\n# Split suites\n./gradlew unitTest\n./gradlew integrationTest\n\n# Include flaky tests (marked @Flaky)\n./gradlew test -Pkafka.test.run.flaky=true\n\n# Re-run without considering up-to-date\n./gradlew test --rerun-tasks\n\n# Single test class\n./gradlew clients:test --tests RequestResponseTest\n\n# Single test method\n./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic\n\n# Coverage (whole project or one module)\n./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n```\n\n**Test retries** (optional): `-PmaxTestRetries=1 -PmaxTestRetryFailures=3`\n\n**Logging**: adjust `log4j2.yaml` under `/src/test/resources/` for verbose test logs.\n\n**System tests**: see [tests/README.md](tests/README.md).\n\n## Code quality\n\nRun before submitting Java changes:\n\n```bash\n./gradlew spotlessApply # fix import order / formatting\n./gradlew checkstyleMain checkstyleTest spotlessCheck\n./gradlew spotbugsMain spotbugsTest -x test # optional static analysis\n```\n\nCheckstyle reports: `/build/reports/checkstyle/`.\n\n## Coding conventions\n\n- **Commits / PR titles**: Start with `KAFKA-XXXXX`, `MINOR`, or `HOTFIX`. Use `KAFKA-XXXXX` only when there is a valid Jira ticket for the change.\n- **PR descriptions**: The PR title and description become the squash-merged commit message. In the description, explain what the change does and why it is needed, if not obvious. Keep it short for small changes. Put any notes for reviewers, such as the tests you ran, in a separate comment.\n- **AI-generated contributions**: Follow the [AI-Generated Contributions](CONTRIBUTING.md#ai-generated-contributions) section in `CONTRIBUTING.md` — add a `Co-Authored-By` or `Generated-by` commit trailer for AI-assisted changes.\n- **Public API / KIP**: Changes to public interfaces, wire protocol, configurations, or metrics generally require a [KIP](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). See the `javadoc` `include` list in [build.gradle](build.gradle) for packages treated as public API.\n- **License header**: New source files need the standard ASF license header (see existing files in the same module).\n- **Checkstyle**: Enforced on main and test sources; rules in `checkstyle/`. Import order is checked via Spotless.\n- **Generated code**: Do not hand-edit generated RPC/message sources. Regenerate with `processMessages` / `processTestMessages`. See [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md).\n- **Scope**: Prefer targeted unit tests in the module you change; integration/system tests for cross-component behavior.\n- **Compatibility**: clients, generator, and streams modules support Java 11+; all other modules require Java 17+. Avoid APIs that break those release levels.\n\n## Security\n\nSee [SECURITY.md](./SECURITY.md) for how to report a vulnerability and for links\nto the Apache Kafka security model under [docs/security/](./docs/security/).\n\nAgents that scan this repository should consult `SECURITY.md` and the linked\nsecurity model before reporting issues. In particular, the core model defines\nwhat is in and out of scope, how reports are classified, and a list of known\nnon-findings; each component page adds its own known non-findings.\n\n## Reference docs\n\n| Topic | Location |\n|--------------------------------|----------|\n| Build, test, broker quickstart | [README.md](README.md) |\n| Contributing | [CONTRIBUTING.md](CONTRIBUTING.md), https://kafka.apache.org/contributing.html |\n| Message protocol / codegen | [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md) |\n| System tests | [tests/README.md](tests/README.md) |\n| JMH benchmarks | [jmh-benchmarks/README.md](jmh-benchmarks/README.md) |\n| Client examples | [examples/README.md](examples/README.md) |\n| Security | [SECURITY.md](SECURITY.md) |\n","isInternal":false,"tokens":1765,"sizeBytes":10000},{"name":"log4j2.yaml","path":"tools/src/test/resources/log4j2.yaml","rawUrl":"https://raw.githubusercontent.com/apache/kafka/HEAD/tools/src/test/resources/log4j2.yaml","title":"Resources Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: INFO\n AppenderRef:\n - ref: STDOUT\n\n Logger:\n - name: org.apache.kafka\n level: ERROR\n\n - name: org.eclipse.jetty\n level: ERROR\n","isInternal":false,"tokens":273,"sizeBytes":1187},{"name":"log4j2.yaml","path":"tools/tools-api/src/test/resources/log4j2.yaml","rawUrl":"https://raw.githubusercontent.com/apache/kafka/HEAD/tools/tools-api/src/test/resources/log4j2.yaml","title":"Resources Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: \"OFF\"\n AppenderRef:\n - ref: STDOUT\n Logger:\n - name: kafka\n level: WARN\n\n - name: org.apache.kafka\n level: WARN\n","isInternal":false,"tokens":271,"sizeBytes":1173},{"name":"README.md","path":"committer-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/kafka/HEAD/committer-tools/README.md","title":"committer-tools Documentation","category":"plugin-manifest","format":"markdown","content":"# Committer Tools\n\nThis directory contains scripts to help Apache Kafka committers with a few chores.\nSome of the scripts require a GitHub API token with write permissions. Only\ncommitters will be able to utilize such scripts.\n\n## Requirements\n\n- Python 3.x and pip\n- The GitHub CLI\n\n## Installation\n\n### 1. Check Python installation\n\nCheck if Python and pip are installed in your system.\n\n```bash\npython --version\npip --version\n```\n\n### 2. Set up a virtual environment (optional)\n\n```bash\npython -m venv venv\n\n# For Linux/macOS\nsource venv/bin/activate\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\n### 3. Install the required dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Install the GitHub CLI\n\nSee: https://cli.github.com/\n\n```bash\nbrew install gh\n```\n\n## Find Reviewers and Update to PR body\n\nThe reviewers.py script is used to simplify the process of producing our \"Reviewers:\"\nGit trailer to PR body. It parses the Git log to gather a set of \"Authors\" and \"Reviewers\". \nSome simple string prefix matching is done to find candidates. \nAfter entering the pull request number, the script updates the \"Reviewers:\" trailer accordingly. \nIf the PR body already contains a \"Reviewers:\" trailer, the script replaces it with the updated list of reviewers.\n\nUsage:\n\n```bash\npython reviewers.py\n```\n\n## Refresh Collaborators\n\nThe Refresh Collaborators script automates the process of fetching contributor\ndata from GitHub repositories, filtering top contributors who are not part of\nthe existing committers, and updating a local configuration file (.asf.yaml) to\ninclude these new contributors.\n\n> This script requires the Python dependencies and a GitHub auth token.\n\nYou need to set up a valid GitHub token to access the repository. After you\ngenerate it (or authenticate via GitHub CLI), this can be done by setting the\nGITHUB_TOKEN environment variable.\n\n```bash\n# For Linux/macOS\nexport GITHUB_TOKEN=\"your_github_token\"\n# Or if you use GitHub CLI\nexport GITHUB_TOKEN=\"$(gh auth token)\"\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\nUsage:\n\n```bash\npython refresh_collaborators.py\n```\n\n## Find Hanging Tests\n\nThis script is used to infer hanging tests from the Gradle output. It looks for\ntests that were STARTED but do not have a corresponding FINISHED or FAILED.\n\nUsage:\n\n```bash\npython find-unfinished-test.py ~/Downloads/logs_28218821016/5_build\\ _\\ JUnit\\ tests\\ Java\\ 11.txt\n\nFound tests that were started, but not finished:\n\n2024-09-10T20:31:26.6830206Z Gradle Test Run :streams:test > Gradle Test Executor 47 > StreamThreadTest > shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean) > \"shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean).processingThreadsEnabled=true\" STARTED\n```\n","isInternal":false,"tokens":640,"sizeBytes":2726},{"name":".cursorrules","path":".cursorrules (scala-kafka-cursorrules-prompt-file.mdc)","rawUrl":"https://raw.githubusercontent.com/PatrickJS/awesome-cursorrules/main/rules/scala-kafka-cursorrules-prompt-file.mdc","title":"Awesome Cursorrules: scala kafka","category":"cursor-rule","format":"markdown","content":"---\ndescription: \"Cursor rules for Scala Kafka.\"\nglobs: **/*\nalwaysApply: false\n---\n# general-scala-clean-code.mdc\n\n- Declare vals/vars as close as possible to first use.\n- Name length should be proportional to scope: 1-2 chars allowed only inside small lambdas.\n- Avoid nested for-comprehensions deeper than two levels; factor out steps into helpers.\n- Split a single source file by responsibility.\n- Use *tail-rec* optimisation (`@tailrec`) where appropriate.\n- Prefer *immutable* collections and avoid mutation during iteration.\n- When interop with Java forces mutability, wrap it in a pure facade with the use of .asScala, to retain functional API.\n- When something is nullable, wrap it into an Option, to retain functional API.\n- Keep cyclomatic complexity below 10 for any method; IDE inspections should warn.\n\n# general-scala-development-practices.mdc\n\n# ========== GENERAL PRINCIPLES ==========\n- You are an experienced Senior Scala Developer.\n- You always adhere to SOLID, DRY, KISS and YAGNI principles.\n- Prefer *pure* functions; minimise side-effects. Where effects are required, make them explicit (e.g. using scala.util.Try, Either, or cats-effect IO/Task if adopted).\n- Use *val* over *var*; collections must be immutable unless mutability is proven cheaper & safe.\n- Replace *null* with Option, Either or a domain-specific ADT.\n- Use pattern matching exhaustively and handle the *default* case only when truly open-ended.\n- Prefer for-comprehensions, map/flatMap/fold, and higher-order functions over imperative loops.\n- Prefer *case classes* and *sealed traits* for algebraic data types.\n- Extract common logic into private or package-private helpers; avoid long methods (> 30 LOC).\n- Prefer extension methods or type classes over inheritance when adding behaviour.\n- Keep public APIs small, surface only what the module owns.\n- Break every task into the smallest composable pure functions before wiring them together.\n\n# ========== NAMING & SYNTAX ==========\n- Class / object / trait names are UpperCamelCase nouns (e.g. *NotificationStreamApp*).\n- Methods & vals are lowerCamelCase verbs or nouns (e.g. *process*, *serde*, *productKey*).\n- Constants use `SCREAMING_SNAKE_CASE`.\n- Similar to Java’s static final members, if the member is final, immutable and it belongs to a package object or an object, it may be considered a constant.\n- Symbolic names (`|>`) are allowed *only* when they align with widespread FP idioms.\n- Match expressions use `match`/`case` over nested if/else chains; for simple two-branch logic prefer `if … then … else …` expressions.\n\n# ========== ERROR HANDLING & LOGGING ==========\n- Catch the most specific Exception first; convert checked Java exceptions to an ADT or `Try`.\n- No empty `catch` blocks; log at *debug* or *error* level with a meaningful message.\n- Leverage `scala.util.Using` (or cats-effect `Resource`) for auto-closing resources.\n- Avoid “defensive” logging or `println`; use SLF4J (Logback) with the *scala-logging* wrapper.\n\n# ========== TESTING ==========\n- Use ScalaTest in a **Given-When-Then** layout with the use of AnyFlatSpec.\n- Focus on critical paths and business invariants; do not over-test boilerplate.\n- Property-based tests (ScalaCheck) for pure functions with non-trivial invariants.\n- Set up integration tests as a subproject named “integration” and treat integration tests as standard tests\n\n# ========== PERFORMANCE & SAFETY ==========\n- Avoid blocking calls inside Kafka stream processing; if unavoidable, off-load to a dedicated thread-pool.\n- Convert Java collections to Scala equivalents once at the boundary; never bounce back and forth.\n- Use underscore-separated digits for large numeric literals (e.g. `val timeoutMs = 30_000`).\n\n# ========== MODERN SCALA 3 FEATURES ==========\n- Use *Enums* for finite alternatives instead of Java-style enums.\n- Embrace *opaque types* to avoid accidental misuse of primitive wrappers.\n- Use *context parameters* (`using`) for type-class evidence instead of classic implicit lists when convenient.\n- Prefer `given`/`using` syntax over `implicit` where supported.\n\n# ========== CLEAN BUILD ==========\n- The sbt build uses **scalafmt** for formatting; treat any scalafmt violation as a build error.\n\n# kafka-development-practices.mdc\n\n- All topic names config values (Typesafe Config or pure-config).\n- Use Format or Codec from the JSON or AVRO or another library that is being used in the project.\n- Streams logic must be tested with `TopologyTestDriver` (unit-test) plus an integration test against local Kafka.\n\n# linting-formatting.mdc\n\n- **scalafmt:** Enforce Google-inspired scalafmt configuration with `maxColumn = 100`.\n","tokens":1053,"sizeBytes":4668}],"systemPromptSnippet":"\n\n\n\n# Agent Guide for Apache Kafka\n\nThis file is read by automated agents (security scanners, code analyzers,\nAI assistants) operating on this repository.\n\n## Project overview\n\nApache Kafka is a distributed event streaming platform. This repo is a **Gradle**\nbuild in **Java** and **Scala 2.13**.\n\n- **Supported JDK**: 11+ for `clients`, `generator`, and `streams`; 17+ for other modules.\n- **Build/test JDK**: Java 17 and 25.\n- **Build tool**: Gradle, wrapper `./gradlew` is recommended.\n\n## Repository layout (high level)\n\n| Path | Purpose |\n|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|\n| `clients/` | Producer, consumer, admin client APIs and protocol messages |\n| `core/` | Broker runtime, log, replication, request handling (gradually being removed along with Scala code) |\n| `server/`, `server-common/` | Broker/server components and shared server code |\n| `metadata/`, `raft/` | KRaft metadata layer and Raft implementation |\n| `storage/`, `storage/api/` | Log segments, checkpoints, tiered storage APIs |\n| `group-coordinator/`, `share-coordinator/`, `transaction-coordinator/`, `coordinator-common/` | Coordinators |\n| `connect/` | Kafka Connect (api, runtime, plugins, transforms) |\n| `streams/` | Kafka Streams (+ Scala, examples, upgrade system tests) |\n| `generator/` | RPC/message code generation |\n| `tools/`, `shell/` | CLI tools and kafka metadata shell |\n| `examples/` | Kafka producer and consumer examples |\n| `jmh-benchmarks/` | Kafka benchmarks tests |\n| `trogdor/` | test framework |\n| `tests/` | System test harness (see `tests/README.md`) |\n| `config/` | Sample broker/controller configuration |\n| `checkstyle/` | Checkstyle rules and import-control XML |\n| `docs/` | Documentation sources |\n| `docker/` | Docker images, build/test scripts, and examples for JVM and native Kafka |\n\n\n## Build commands\n\n```bash\n# Compile jars\n./gradlew jar\n\n# Release tarball (output under core/build/distributions/)\n./gradlew clean releaseTarGz\n\n# Publish to local Maven (skip signing for local dev)\n./gradlew -PskipSigning=true publishToMavenLocal\n\n# Regenerate RPC/message classes after branch switches\n./gradlew processMessages processTestMessages\n\n# List tasks\n./gradlew tasks\n```\n\nModule-scoped build:\n\n```bash\n./gradlew core:jar\n./gradlew clients:jar\n./gradlew :streams:testAll\n```\n\n## Test commands\n\n```bash\n# All unit + integration tests\n./gradlew test\n\n# Split suites\n./gradlew unitTest\n./gradlew integrationTest\n\n# Include flaky tests (marked @Flaky)\n./gradlew test -Pkafka.test.run.flaky=true\n\n# Re-run without considering up-to-date\n./gradlew test --rerun-tasks\n\n# Single test class\n./gradlew clients:test --tests RequestResponseTest\n\n# Single test method\n./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic\n\n# Coverage (whole project or one module)\n./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false\n```\n\n**Test retries** (optional): `-PmaxTestRetries=1 -PmaxTestRetryFailures=3`\n\n**Logging**: adjust `log4j2.yaml` under `/src/test/resources/` for verbose test logs.\n\n**System tests**: see [tests/README.md](tests/README.md).\n\n## Code quality\n\nRun before submitting Java changes:\n\n```bash\n./gradlew spotlessApply # fix import order / formatting\n./gradlew checkstyleMain checkstyleTest spotlessCheck\n./gradlew spotbugsMain spotbugsTest -x test # optional static analysis\n```\n\nCheckstyle reports: `/build/reports/checkstyle/`.\n\n## Coding conventions\n\n- **Commits / PR titles**: Start with `KAFKA-XXXXX`, `MINOR`, or `HOTFIX`. Use `KAFKA-XXXXX` only when there is a valid Jira ticket for the change.\n- **PR descriptions**: The PR title and description become the squash-merged commit message. In the description, explain what the change does and why it is needed, if not obvious. Keep it short for small changes. Put any notes for reviewers, such as the tests you ran, in a separate comment.\n- **AI-generated contributions**: Follow the [AI-Generated Contributions](CONTRIBUTING.md#ai-generated-contributions) section in `CONTRIBUTING.md` — add a `Co-Authored-By` or `Generated-by` commit trailer for AI-assisted changes.\n- **Public API / KIP**: Changes to public interfaces, wire protocol, configurations, or metrics generally require a [KIP](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). See the `javadoc` `include` list in [build.gradle](build.gradle) for packages treated as public API.\n- **License header**: New source files need the standard ASF license header (see existing files in the same module).\n- **Checkstyle**: Enforced on main and test sources; rules in `checkstyle/`. Import order is checked via Spotless.\n- **Generated code**: Do not hand-edit generated RPC/message sources. Regenerate with `processMessages` / `processTestMessages`. See [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md).\n- **Scope**: Prefer targeted unit tests in the module you change; integration/system tests for cross-component behavior.\n- **Compatibility**: clients, generator, and streams modules support Java 11+; all other modules require Java 17+. Avoid APIs that break those release levels.\n\n## Security\n\nSee [SECURITY.md](./SECURITY.md) for how to report a vulnerability and for links\nto the Apache Kafka security model under [docs/security/](./docs/security/).\n\nAgents that scan this repository should consult `SECURITY.md` and the linked\nsecurity model before reporting issues. In particular, the core model defines\nwhat is in and out of scope, how reports are classified, and a list of known\nnon-findings; each component page adds its own known non-findings.\n\n## Reference docs\n\n| Topic | Location |\n|--------------------------------|----------|\n| Build, test, broker quickstart | [README.md](README.md) |\n| Contributing | [CONTRIBUTING.md](CONTRIBUTING.md), https://kafka.apache.org/contributing.html |\n| Message protocol / codegen | [clients/src/main/resources/common/message/README.md](clients/src/main/resources/common/message/README.md) |\n| System tests | [tests/README.md](tests/README.md) |\n| JMH benchmarks | [jmh-benchmarks/README.md](jmh-benchmarks/README.md) |\n| Client examples | [examples/README.md](examples/README.md) |\n| Security | [SECURITY.md](SECURITY.md) |\n\n\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: INFO\n AppenderRef:\n - ref: STDOUT\n\n Logger:\n - name: org.apache.kafka\n level: ERROR\n\n - name: org.eclipse.jetty\n level: ERROR\n\n\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nConfiguration:\n Properties:\n Property:\n - name: \"logPattern\"\n value: \"[%d] %p %m (%c:%L)%n\"\n\n Appenders:\n Console:\n name: STDOUT\n PatternLayout:\n pattern: \"${logPattern}\"\n\n Loggers:\n Root:\n level: \"OFF\"\n AppenderRef:\n - ref: STDOUT\n Logger:\n - name: kafka\n level: WARN\n\n - name: org.apache.kafka\n level: WARN\n\n\n# Committer Tools\n\nThis directory contains scripts to help Apache Kafka committers with a few chores.\nSome of the scripts require a GitHub API token with write permissions. Only\ncommitters will be able to utilize such scripts.\n\n## Requirements\n\n- Python 3.x and pip\n- The GitHub CLI\n\n## Installation\n\n### 1. Check Python installation\n\nCheck if Python and pip are installed in your system.\n\n```bash\npython --version\npip --version\n```\n\n### 2. Set up a virtual environment (optional)\n\n```bash\npython -m venv venv\n\n# For Linux/macOS\nsource venv/bin/activate\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\n### 3. Install the required dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Install the GitHub CLI\n\nSee: https://cli.github.com/\n\n```bash\nbrew install gh\n```\n\n## Find Reviewers and Update to PR body\n\nThe reviewers.py script is used to simplify the process of producing our \"Reviewers:\"\nGit trailer to PR body. It parses the Git log to gather a set of \"Authors\" and \"Reviewers\". \nSome simple string prefix matching is done to find candidates. \nAfter entering the pull request number, the script updates the \"Reviewers:\" trailer accordingly. \nIf the PR body already contains a \"Reviewers:\" trailer, the script replaces it with the updated list of reviewers.\n\nUsage:\n\n```bash\npython reviewers.py\n```\n\n## Refresh Collaborators\n\nThe Refresh Collaborators script automates the process of fetching contributor\ndata from GitHub repositories, filtering top contributors who are not part of\nthe existing committers, and updating a local configuration file (.asf.yaml) to\ninclude these new contributors.\n\n> This script requires the Python dependencies and a GitHub auth token.\n\nYou need to set up a valid GitHub token to access the repository. After you\ngenerate it (or authenticate via GitHub CLI), this can be done by setting the\nGITHUB_TOKEN environment variable.\n\n```bash\n# For Linux/macOS\nexport GITHUB_TOKEN=\"your_github_token\"\n# Or if you use GitHub CLI\nexport GITHUB_TOKEN=\"$(gh auth token)\"\n\n# On Windows:\n# .\\venv\\Scripts\\activate\n```\n\nUsage:\n\n```bash\npython refresh_collaborators.py\n```\n\n## Find Hanging Tests\n\nThis script is used to infer hanging tests from the Gradle output. It looks for\ntests that were STARTED but do not have a corresponding FINISHED or FAILED.\n\nUsage:\n\n```bash\npython find-unfinished-test.py ~/Downloads/logs_28218821016/5_build\\ _\\ JUnit\\ tests\\ Java\\ 11.txt\n\nFound tests that were started, but not finished:\n\n2024-09-10T20:31:26.6830206Z Gradle Test Run :streams:test > Gradle Test Executor 47 > StreamThreadTest > shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean) > \"shouldReturnErrorIfProducerInstanceIdNotInitialized(boolean).processingThreadsEnabled=true\" STARTED\n```\n\n\n---\ndescription: \"Cursor rules for Scala Kafka.\"\nglobs: **/*\nalwaysApply: false\n---\n# general-scala-clean-code.mdc\n\n- Declare vals/vars as close as possible to first use.\n- Name length should be proportional to scope: 1-2 chars allowed only inside small lambdas.\n- Avoid nested for-comprehensions deeper than two levels; factor out steps into helpers.\n- Split a single source file by responsibility.\n- Use *tail-rec* optimisation (`@tailrec`) where appropriate.\n- Prefer *immutable* collections and avoid mutation during iteration.\n- When interop with Java forces mutability, wrap it in a pure facade with the use of .asScala, to retain functional API.\n- When something is nullable, wrap it into an Option, to retain functional API.\n- Keep cyclomatic complexity below 10 for any method; IDE inspections should warn.\n\n# general-scala-development-practices.mdc\n\n# ========== GENERAL PRINCIPLES ==========\n- You are an experienced Senior Scala Developer.\n- You always adhere to SOLID, DRY, KISS and YAGNI principles.\n- Prefer *pure* functions; minimise side-effects. Where effects are required, make them explicit (e.g. using scala.util.Try, Either, or cats-effect IO/Task if adopted).\n- Use *val* over *var*; collections must be immutable unless mutability is proven cheaper & safe.\n- Replace *null* with Option, Either or a domain-specific ADT.\n- Use pattern matching exhaustively and handle the *default* case only when truly open-ended.\n- Prefer for-comprehensions, map/flatMap/fold, and higher-order functions over imperative loops.\n- Prefer *case classes* and *sealed traits* for algebraic data types.\n- Extract common logic into private or package-private helpers; avoid long methods (> 30 LOC).\n- Prefer extension methods or type classes over inheritance when adding behaviour.\n- Keep public APIs small, surface only what the module owns.\n- Break every task into the smallest composable pure functions before wiring them together.\n\n# ========== NAMING & SYNTAX ==========\n- Class / object / trait names are UpperCamelCase nouns (e.g. *NotificationStreamApp*).\n- Methods & vals are lowerCamelCase verbs or nouns (e.g. *process*, *serde*, *productKey*).\n- Constants use `SCREAMING_SNAKE_CASE`.\n- Similar to Java’s static final members, if the member is final, immutable and it belongs to a package object or an object, it may be considered a constant.\n- Symbolic names (`|>`) are allowed *only* when they align with widespread FP idioms.\n- Match expressions use `match`/`case` over nested if/else chains; for simple two-branch logic prefer `if … then … else …` expressions.\n\n# ========== ERROR HANDLING & LOGGING ==========\n- Catch the most specific Exception first; convert checked Java exceptions to an ADT or `Try`.\n- No empty `catch` blocks; log at *debug* or *error* level with a meaningful message.\n- Leverage `scala.util.Using` (or cats-effect `Resource`) for auto-closing resources.\n- Avoid “defensive” logging or `println`; use SLF4J (Logback) with the *scala-logging* wrapper.\n\n# ========== TESTING ==========\n- Use ScalaTest in a **Given-When-Then** layout with the use of AnyFlatSpec.\n- Focus on critical paths and business invariants; do not over-test boilerplate.\n- Property-based tests (ScalaCheck) for pure functions with non-trivial invariants.\n- Set up integration tests as a subproject named “integration” and treat integration tests as standard tests\n\n# ========== PERFORMANCE & SAFETY ==========\n- Avoid blocking calls inside Kafka stream processing; if unavoidable, off-load to a dedicated thread-pool.\n- Convert Java collections to Scala equivalents once at the boundary; never bounce back and forth.\n- Use underscore-separated digits for large numeric literals (e.g. `val timeoutMs = 30_000`).\n\n# ========== MODERN SCALA 3 FEATURES ==========\n- Use *Enums* for finite alternatives instead of Java-style enums.\n- Embrace *opaque types* to avoid accidental misuse of primitive wrappers.\n- Use *context parameters* (`using`) for type-class evidence instead of classic implicit lists when convenient.\n- Prefer `given`/`using` syntax over `implicit` where supported.\n\n# ========== CLEAN BUILD ==========\n- The sbt build uses **scalafmt** for formatting; treat any scalafmt violation as a build error.\n\n# kafka-development-practices.mdc\n\n- All topic names config values (Typesafe Config or pure-config).\n- Use Format or Codec from the JSON or AVRO or another library that is being used in the project.\n- Streams logic must be tested with `TopologyTestDriver` (unit-test) plus an integration test against local Kafka.\n\n# linting-formatting.mdc\n\n- **scalafmt:** Enforce Google-inspired scalafmt configuration with `maxColumn = 100`.\n"}