{"owner":"apache","repo":"flink-cdc","hasSkills":true,"totalSkillsCount":7,"totalTokensCount":5371,"categories":["root-instruction","anthropic-skill","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml","tools/cdcup/README.md"],"skills":{"AGENTS.md":"<!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\n# Flink CDC AI Agent Instructions\n\nThis file provides guidance for AI coding agents working with the Apache Flink CDC.\n\n## Prerequisites\n\n- Java 11 (baseline; all code must compile and run on Java 11) and Java 17\n- Maven 3.8.6 or higher\n- Git\n- Unix-like environment (Linux, macOS, WSL)\n- Docker (required for running integration tests and e2e tests)\n\n## Commands\n\n### Build\n\n- Fast dev build (skip tests and format checks): `mvn clean install -DskipTests -Dspotless.check.skip=true -Dcheckstyle.skip=true`\n- Full build against Flink 1.x (default): `mvn clean package -DskipTests`\n- Full build against Flink 2.x: `mvn clean package -DskipTests -Pflink2`\n- Single module (e.g. `flink-cdc-common`): `mvn clean package -DskipTests -pl flink-cdc-common -am`\n\n### Testing\n\n- Run all tests for a module: `mvn verify -pl flink-cdc-common`\n- Run all tests against Flink 2.x: `mvn verify -pl flink-cdc-common -Pflink2`\n- Run a single test class: `mvn -pl flink-cdc-common -Dtest=MyTest test`\n- Run a single test method: `mvn -pl flink-cdc-common -Dtest=MyTest#myMethod test`\n\n### Code Quality\n\n- Before committing changes, run `mvn spotless:apply` and `mvn spotless:check` to enforce code style rules.\n- Make sure newly added files have proper ASF license headers.\n\n## Repository Structure\n\nFlink CDC is organized around the Pipeline abstraction: a user-defined data pipeline that reads from one or more sources, optionally transforms records, and writes to one or more sinks. The core modules implement this abstraction; the connector modules provide the concrete source and sink implementations.\n\n### Core Modules\n\n- `flink-cdc-common` — Shared API and data model used across all modules: CDC event types (`DataChangeEvent`, `SchemaChangeEvent`, etc.), the schema model, data types, source/sink interfaces, `Factory` SPI, route definitions, UDF interfaces, and utility classes. Most new abstractions start here.\n- `flink-cdc-runtime` — Runtime implementation of the Pipeline: operators for reading, routing, transforming (expression evaluation via Calcite + Janino), and writing CDC events.\n- `flink-cdc-composer` — Pipeline assembly and deployment layer. Translates a `PipelineDefinition` into a runnable Flink job, wiring sources, operators, and sinks together. Supports Flink-native, Kubernetes, and YARN deployment.\n- `flink-cdc-cli` — Command-line entry point (`flink-cdc.sh`). Parses YAML pipeline definitions and delegates to `flink-cdc-composer`.\n- `flink-cdc-dist` — Distribution packaging. Produces the `flink-cdc-<version>-bin` release archive.\n\n### Flink Version Compatibility\n\nCurrently, Flink CDC supports two Flink generations simultaneously:\n\n- `flink-cdc-flink1-compat` — Flink 1.x compatibility layer (currently 1.20.3). Default profile.\n- `flink-cdc-flink2-compat` — Flink 2.x compatibility layer (currently 2.2.0). Activated via `-Pflink2`.\n\nAll modules that depend on Flink APIs must declare their Flink dependencies as `provided` and reference `${flink.version}`, which is resolved by the active profile.\nPlease verify your changes in both Flink 1.20 (LTS) and Flink 2.x (latest).\n\n### Connectors (`flink-cdc-connect/`)\n\nConnectors are split into two categories:\n\n- Source Connectors (`flink-cdc-connect/flink-cdc-source-connectors/`) are CDC sources for DataStream and Flink SQL jobs.\n- Pipeline Connectors (`flink-cdc-connect/flink-cdc-pipeline-connectors/`) are Pipeline connectors for the YAML API.\n\n### Tests\n\nWrite unit tests and integration tests in the corresponding submodules. End-to-end tests are located in these modules:\n\n- `flink-cdc-e2e-tests/` — End-to-end tests parent module.\n  - `flink-cdc-e2e-utils` — Shared test utilities (container management, assertions)\n  - `flink-cdc-source-e2e-tests` — E2E tests for source connectors\n  - `flink-cdc-pipeline-e2e-tests` — E2E tests for pipeline connectors\n\n### Documentation\n\n- `docs/` — Hugo-based documentation site. `docs/content/` for English, `docs/content.zh/` for Chinese. Update both when adding new features.\n\n## Coding Standards\n\n- **Format Java files with Spotless before every commit:** `mvn spotless:apply`.\n- **Import order** (enforced by Checkstyle): `org.apache.flink.cdc` → `org.apache.flink` → other third-party → `javax` → `java`. Static imports go last. No star imports.\n- **Forbidden imports** (enforced by Checkstyle):\n  - JUnit 4 (`org.junit.*` except `org.junit.jupiter.*`) — use JUnit 5 Jupiter instead\n  - `org.junit.jupiter.api.Assertions` and `org.hamcrest` — use AssertJ instead\n  - `com.google.common.*` — use `flink-shaded-guava` instead\n  - `com.google.common.base.Preconditions` — use Flink CDC's `Preconditions`\n  - `com.google.common.annotations.VisibleForTesting` — use `@VisibleForTesting` from `org.apache.flink.cdc.common.annotation`\n- **API stability annotations:** User-facing API types should carry stability annotations, and methods/fields/constructors only need explicit annotations when they differ from the enclosing type or when needed.\n  - `@Public` — stable across major versions\n  - `@PublicEvolving` — may change in minor versions\n  - `@Experimental` — may change at any time\n  - `@Internal` — no stability guarantee; users should not depend on it\n- **Logging:** Use SLF4J with parameterized placeholders (`LOG.info(\"foo {}\", bar)`), never string concatenation.\n- **Braces:** Always use braces for `if`/`else`/`for`/`while`/`do` blocks.\n- **Comments:** All comments should be written in English, and keep it concise. Only write comments when necessary. Avoid trivial `@param`, `@return` comments in JavaDocs.\n- Apache License 2.0 header required on all new files.\n\n## Testing Standards\n\n- Use **JUnit 5** (`org.junit.jupiter`) and **AssertJ** (`org.assertj.core.api.Assertions`) for all new tests. Do not use JUnit 4 or Hamcrest.\n- Test classes are **package-private** (no `public` modifier on the class).\n- **Unit tests** are named `*Test.java` (e.g., `SchemaUtilsTest`).\n- **Integration tests** (requiring Docker / real databases) are named `*ITCase.java` (e.g., `MySqlSourceITCase`).\n- Add tests for new behavior, covering success, failure, and edge cases. Tests should be self-explanatory; avoid verbose comments in test classes.\n- For bug fixes, verify the new test fails without the fix before confirming it passes with it.\n\n## Commits and PRs\n\n### Commit message format\n\n- `[FLINK-XXXX][component] Description` where `FLINK-XXXX` is the JIRA issue number and `component` is the affected area (e.g. `connect/mysql`, `pipeline-connector/kafka`, `docs`, `runtime`)\n- `[hotfix][component] ...` or `[docs][component] ...` for trivial fixes without a JIRA issue\n- `[ci] Description` for CI-only changes\n\n### Pull request conventions\n\n- Title format mirrors the commit message: `[FLINK-XXXX][component] Title`\n- A corresponding JIRA issue is required (except trivial changes)\n- Fill out the PR template completely: describe purpose, change log, testing approach, and documentation impact\n- Ensure CI passes before requesting review\n- Enable GitHub Actions on your fork before opening a PR\n- When AI tools were used, check the AI disclosure checkbox and uncomment the `Generated-by` line in the PR template, per [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html)\n\n## Boundaries\n\n### Ask first\n\n- Adding or changing `@Public` or `@PublicEvolving` annotations (user-facing API commitments)\n- New dependencies\n- Large cross-module refactors\n- Changes to serialization formats or checkpoint behavior\n- Changes to hot paths that could impact performance (per-record processing, state access)\n\n### Never\n\n- Commit secrets, credentials, or tokens\n- Push directly to `apache/flink-cdc`; always work from your fork\n- Mix unrelated changes into one PR\n- Use JUnit 4 or Hamcrest in new test code\n- Use `org.junit.jupiter.api.Assertions`; use AssertJ instead\n- Add `Co-Authored-By` with an AI agent in commit messages; use `Generated-by: <Tool Name and Version>` instead","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Invalid Cases\nsteps:\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      execution:\n        execution.coordinator-close-timeout: 300000\n        checkpointing:\n          interval: 600000\n          timeout: 30000\n          max-concurrent-checkpoints: 1\n          min-pause-between-checkpoints: 300000\n    error: |\n      Unexpected key `execution` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: user_course.\\.*\n          sink-table: ods.ods_user_course_<>\n          replace-symbol: <>\n          description: sync table to one destination table with given prefix ods_\n          transformers:\n            - type: cast\n              fields:\n                - name: \"*\"\n                  type: BIT\n                  target-type: TINYINT\n    error: |\n      Unexpected key `transformers` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option transformers: [{\"type\":\"cast\",\"fields\":[{\"name\":\"*\",\"type\":\"BIT\",\"target-type\":\"TINYINT\"}]}] is unexpected.\n      It was silently ignored in previous versions, and probably should be removed.","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Incomplete Structures\nsteps:\n  # Missing source\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n    error: Missing required field \"sink\" in top-level configuration\n  # Missing source\n  - type: submit\n    yaml: |\n      sink:\n        type: values\n    error: Missing required field \"source\" in top-level configuration\n  # Missing required options\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n      sink:\n        type: values\n    error: |\n      Missing required options are:\n      hostname\n      password\n      tables\n      username\n  # Missing transform components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - projection: \\*\n    error: Missing required field \"source-table\" in transform configuration\n  # Missing routing components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - sink-table: foo.bar.baz\n    error: Missing required field \"source-table\" in route configuration\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar.baz\n    error: Missing required field \"sink-table\" in route configuration\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Malformed Structures\nsteps:\n  # Transform rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        # ERROR: should be `- source-table: foo.bar`\n        source-table: foo.bar\n    error: |\n      YAML transform block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # Route rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        source-table: foo.bar\n    error: |\n      YAML route block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # UDF functions not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          name: addone\n          classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n    error: |\n      YAML UDF block is expecting an array children, but got an OBJECT ({\"name\":\"addone\",\"classpath\":\"org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\"}).\n      Perhaps you missed a dash prefix `-`?\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Vanilla Test Specs\nsteps:\n  - type: exec\n    sql: |\n      CREATE TABLE foo (idx BIGINT NOT NULL PRIMARY KEY, name VARCHAR(17));\n      INSERT INTO foo VALUES\n        (1, 'Alice'),\n        (2, 'Brett'),\n        (3, 'Cicada'),\n        (4, 'Derrida'),\n        (5, 'Eve');\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n      sink:\n        type: values\n      pipeline:\n        parallelism: 1\n  - type: check\n    jm: |\n      Successfully applied schema change event CreateTableEvent{tableId=$database$.foo, schema=columns={`idx` BIGINT NOT NULL,`name` VARCHAR(17)}, primaryKeys=idx, options=()} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[1, Alice], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[2, Brett], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[3, Cicada], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[4, Derrida], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[5, Eve], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      ALTER TABLE foo ADD COLUMN (digits DOUBLE);\n      INSERT INTO foo VALUES\n        (6, 'Faye', 3.14),\n        (7, 'Gem', 2.71828),\n        (8, 'Helen', 1.414);\n  - type: check\n    jm: |\n      Successfully applied schema change event AddColumnEvent{tableId=$database$.foo, addedColumns=[ColumnWithPosition{column=`digits` DOUBLE, position=LAST, existedColumnName=null}]} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[6, Faye, 3.14], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[7, Gem, 2.71828], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[8, Helen, 1.414], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      UPDATE foo SET digits = 42 WHERE idx < 4;\n  - type: check\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[1, Alice, null], after=[1, Alice, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[2, Brett, null], after=[2, Brett, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[3, Cicada, null], after=[3, Cicada, 42.0], op=UPDATE, meta=()}\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Unexpected Structures\nsteps:\n  # Unexpected top-level block\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      mapping:\n        - source: userUuid\n          sink: uuid\n        - source: userId\n          sink: user_id\n        - source: deviceUuid\n          sink: device_uuid\n    error: |\n      Unexpected key `mapping` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  # Unexpected transform block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - source-table: foo.bar\n          mapping:\n            - source: userUuid\n              sink: uuid\n    error: |\n      Unexpected key `mapping` in YAML transform block.\n      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, table-options.delimiter, description, converter-after-transform]\n      Note: option mapping: [{\"source\":\"userUuid\",\"sink\":\"uuid\"}] is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected route block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar\n          sink-table: fun.baz\n          regex-style: java\n    error: |\n      Unexpected key `regex-style` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option regex-style: \"java\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected UDF block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          - name: addone\n            classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n            language: clojure\n    error: |\n      Unexpected key `language` in YAML UDF block.\n      Allowed keys in this context are: [name, classpath, python-code, python-executable, python-files, options]\n      Note: option language: \"clojure\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n","tools/cdcup/README.md":"# cdcup\n\nA `docker` (`compose`) environment on Linux / macOS is required to play with this. Ruby is **not** necessary.\n\n## `./cdcup.sh init`\n\nInitialize a playground environment, and generate configuration files.\n\n## `./cdcup.sh up`\n\nStart docker containers. Note that it may take a while before database is ready.\n\n## `./cdcup.sh pipeline <yaml>`\n\nSubmit a YAML pipeline job. Before executing this, please ensure that:\n\n1. All container are running and ready for connections\n2. (For MySQL) You've created at least one database & tables to be captured\n\n## `./cdcup.sh flink`\n\nPrint Flink Web dashboard URL.\n\n## `./cdcup.sh stop`\n\nStop all running playground containers.\n\n## `./cdcup.sh down`\n\nStop and remove containers, networks, and volumes.\n"},"files":{"AGENTS.md":"<!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\n# Flink CDC AI Agent Instructions\n\nThis file provides guidance for AI coding agents working with the Apache Flink CDC.\n\n## Prerequisites\n\n- Java 11 (baseline; all code must compile and run on Java 11) and Java 17\n- Maven 3.8.6 or higher\n- Git\n- Unix-like environment (Linux, macOS, WSL)\n- Docker (required for running integration tests and e2e tests)\n\n## Commands\n\n### Build\n\n- Fast dev build (skip tests and format checks): `mvn clean install -DskipTests -Dspotless.check.skip=true -Dcheckstyle.skip=true`\n- Full build against Flink 1.x (default): `mvn clean package -DskipTests`\n- Full build against Flink 2.x: `mvn clean package -DskipTests -Pflink2`\n- Single module (e.g. `flink-cdc-common`): `mvn clean package -DskipTests -pl flink-cdc-common -am`\n\n### Testing\n\n- Run all tests for a module: `mvn verify -pl flink-cdc-common`\n- Run all tests against Flink 2.x: `mvn verify -pl flink-cdc-common -Pflink2`\n- Run a single test class: `mvn -pl flink-cdc-common -Dtest=MyTest test`\n- Run a single test method: `mvn -pl flink-cdc-common -Dtest=MyTest#myMethod test`\n\n### Code Quality\n\n- Before committing changes, run `mvn spotless:apply` and `mvn spotless:check` to enforce code style rules.\n- Make sure newly added files have proper ASF license headers.\n\n## Repository Structure\n\nFlink CDC is organized around the Pipeline abstraction: a user-defined data pipeline that reads from one or more sources, optionally transforms records, and writes to one or more sinks. The core modules implement this abstraction; the connector modules provide the concrete source and sink implementations.\n\n### Core Modules\n\n- `flink-cdc-common` — Shared API and data model used across all modules: CDC event types (`DataChangeEvent`, `SchemaChangeEvent`, etc.), the schema model, data types, source/sink interfaces, `Factory` SPI, route definitions, UDF interfaces, and utility classes. Most new abstractions start here.\n- `flink-cdc-runtime` — Runtime implementation of the Pipeline: operators for reading, routing, transforming (expression evaluation via Calcite + Janino), and writing CDC events.\n- `flink-cdc-composer` — Pipeline assembly and deployment layer. Translates a `PipelineDefinition` into a runnable Flink job, wiring sources, operators, and sinks together. Supports Flink-native, Kubernetes, and YARN deployment.\n- `flink-cdc-cli` — Command-line entry point (`flink-cdc.sh`). Parses YAML pipeline definitions and delegates to `flink-cdc-composer`.\n- `flink-cdc-dist` — Distribution packaging. Produces the `flink-cdc-<version>-bin` release archive.\n\n### Flink Version Compatibility\n\nCurrently, Flink CDC supports two Flink generations simultaneously:\n\n- `flink-cdc-flink1-compat` — Flink 1.x compatibility layer (currently 1.20.3). Default profile.\n- `flink-cdc-flink2-compat` — Flink 2.x compatibility layer (currently 2.2.0). Activated via `-Pflink2`.\n\nAll modules that depend on Flink APIs must declare their Flink dependencies as `provided` and reference `${flink.version}`, which is resolved by the active profile.\nPlease verify your changes in both Flink 1.20 (LTS) and Flink 2.x (latest).\n\n### Connectors (`flink-cdc-connect/`)\n\nConnectors are split into two categories:\n\n- Source Connectors (`flink-cdc-connect/flink-cdc-source-connectors/`) are CDC sources for DataStream and Flink SQL jobs.\n- Pipeline Connectors (`flink-cdc-connect/flink-cdc-pipeline-connectors/`) are Pipeline connectors for the YAML API.\n\n### Tests\n\nWrite unit tests and integration tests in the corresponding submodules. End-to-end tests are located in these modules:\n\n- `flink-cdc-e2e-tests/` — End-to-end tests parent module.\n  - `flink-cdc-e2e-utils` — Shared test utilities (container management, assertions)\n  - `flink-cdc-source-e2e-tests` — E2E tests for source connectors\n  - `flink-cdc-pipeline-e2e-tests` — E2E tests for pipeline connectors\n\n### Documentation\n\n- `docs/` — Hugo-based documentation site. `docs/content/` for English, `docs/content.zh/` for Chinese. Update both when adding new features.\n\n## Coding Standards\n\n- **Format Java files with Spotless before every commit:** `mvn spotless:apply`.\n- **Import order** (enforced by Checkstyle): `org.apache.flink.cdc` → `org.apache.flink` → other third-party → `javax` → `java`. Static imports go last. No star imports.\n- **Forbidden imports** (enforced by Checkstyle):\n  - JUnit 4 (`org.junit.*` except `org.junit.jupiter.*`) — use JUnit 5 Jupiter instead\n  - `org.junit.jupiter.api.Assertions` and `org.hamcrest` — use AssertJ instead\n  - `com.google.common.*` — use `flink-shaded-guava` instead\n  - `com.google.common.base.Preconditions` — use Flink CDC's `Preconditions`\n  - `com.google.common.annotations.VisibleForTesting` — use `@VisibleForTesting` from `org.apache.flink.cdc.common.annotation`\n- **API stability annotations:** User-facing API types should carry stability annotations, and methods/fields/constructors only need explicit annotations when they differ from the enclosing type or when needed.\n  - `@Public` — stable across major versions\n  - `@PublicEvolving` — may change in minor versions\n  - `@Experimental` — may change at any time\n  - `@Internal` — no stability guarantee; users should not depend on it\n- **Logging:** Use SLF4J with parameterized placeholders (`LOG.info(\"foo {}\", bar)`), never string concatenation.\n- **Braces:** Always use braces for `if`/`else`/`for`/`while`/`do` blocks.\n- **Comments:** All comments should be written in English, and keep it concise. Only write comments when necessary. Avoid trivial `@param`, `@return` comments in JavaDocs.\n- Apache License 2.0 header required on all new files.\n\n## Testing Standards\n\n- Use **JUnit 5** (`org.junit.jupiter`) and **AssertJ** (`org.assertj.core.api.Assertions`) for all new tests. Do not use JUnit 4 or Hamcrest.\n- Test classes are **package-private** (no `public` modifier on the class).\n- **Unit tests** are named `*Test.java` (e.g., `SchemaUtilsTest`).\n- **Integration tests** (requiring Docker / real databases) are named `*ITCase.java` (e.g., `MySqlSourceITCase`).\n- Add tests for new behavior, covering success, failure, and edge cases. Tests should be self-explanatory; avoid verbose comments in test classes.\n- For bug fixes, verify the new test fails without the fix before confirming it passes with it.\n\n## Commits and PRs\n\n### Commit message format\n\n- `[FLINK-XXXX][component] Description` where `FLINK-XXXX` is the JIRA issue number and `component` is the affected area (e.g. `connect/mysql`, `pipeline-connector/kafka`, `docs`, `runtime`)\n- `[hotfix][component] ...` or `[docs][component] ...` for trivial fixes without a JIRA issue\n- `[ci] Description` for CI-only changes\n\n### Pull request conventions\n\n- Title format mirrors the commit message: `[FLINK-XXXX][component] Title`\n- A corresponding JIRA issue is required (except trivial changes)\n- Fill out the PR template completely: describe purpose, change log, testing approach, and documentation impact\n- Ensure CI passes before requesting review\n- Enable GitHub Actions on your fork before opening a PR\n- When AI tools were used, check the AI disclosure checkbox and uncomment the `Generated-by` line in the PR template, per [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html)\n\n## Boundaries\n\n### Ask first\n\n- Adding or changing `@Public` or `@PublicEvolving` annotations (user-facing API commitments)\n- New dependencies\n- Large cross-module refactors\n- Changes to serialization formats or checkpoint behavior\n- Changes to hot paths that could impact performance (per-record processing, state access)\n\n### Never\n\n- Commit secrets, credentials, or tokens\n- Push directly to `apache/flink-cdc`; always work from your fork\n- Mix unrelated changes into one PR\n- Use JUnit 4 or Hamcrest in new test code\n- Use `org.junit.jupiter.api.Assertions`; use AssertJ instead\n- Add `Co-Authored-By` with an AI agent in commit messages; use `Generated-by: <Tool Name and Version>` instead","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Invalid Cases\nsteps:\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      execution:\n        execution.coordinator-close-timeout: 300000\n        checkpointing:\n          interval: 600000\n          timeout: 30000\n          max-concurrent-checkpoints: 1\n          min-pause-between-checkpoints: 300000\n    error: |\n      Unexpected key `execution` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: user_course.\\.*\n          sink-table: ods.ods_user_course_<>\n          replace-symbol: <>\n          description: sync table to one destination table with given prefix ods_\n          transformers:\n            - type: cast\n              fields:\n                - name: \"*\"\n                  type: BIT\n                  target-type: TINYINT\n    error: |\n      Unexpected key `transformers` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option transformers: [{\"type\":\"cast\",\"fields\":[{\"name\":\"*\",\"type\":\"BIT\",\"target-type\":\"TINYINT\"}]}] is unexpected.\n      It was silently ignored in previous versions, and probably should be removed.","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Incomplete Structures\nsteps:\n  # Missing source\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n    error: Missing required field \"sink\" in top-level configuration\n  # Missing source\n  - type: submit\n    yaml: |\n      sink:\n        type: values\n    error: Missing required field \"source\" in top-level configuration\n  # Missing required options\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n      sink:\n        type: values\n    error: |\n      Missing required options are:\n      hostname\n      password\n      tables\n      username\n  # Missing transform components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - projection: \\*\n    error: Missing required field \"source-table\" in transform configuration\n  # Missing routing components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - sink-table: foo.bar.baz\n    error: Missing required field \"source-table\" in route configuration\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar.baz\n    error: Missing required field \"sink-table\" in route configuration\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Malformed Structures\nsteps:\n  # Transform rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        # ERROR: should be `- source-table: foo.bar`\n        source-table: foo.bar\n    error: |\n      YAML transform block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # Route rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        source-table: foo.bar\n    error: |\n      YAML route block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # UDF functions not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          name: addone\n          classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n    error: |\n      YAML UDF block is expecting an array children, but got an OBJECT ({\"name\":\"addone\",\"classpath\":\"org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\"}).\n      Perhaps you missed a dash prefix `-`?\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Vanilla Test Specs\nsteps:\n  - type: exec\n    sql: |\n      CREATE TABLE foo (idx BIGINT NOT NULL PRIMARY KEY, name VARCHAR(17));\n      INSERT INTO foo VALUES\n        (1, 'Alice'),\n        (2, 'Brett'),\n        (3, 'Cicada'),\n        (4, 'Derrida'),\n        (5, 'Eve');\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n      sink:\n        type: values\n      pipeline:\n        parallelism: 1\n  - type: check\n    jm: |\n      Successfully applied schema change event CreateTableEvent{tableId=$database$.foo, schema=columns={`idx` BIGINT NOT NULL,`name` VARCHAR(17)}, primaryKeys=idx, options=()} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[1, Alice], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[2, Brett], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[3, Cicada], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[4, Derrida], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[5, Eve], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      ALTER TABLE foo ADD COLUMN (digits DOUBLE);\n      INSERT INTO foo VALUES\n        (6, 'Faye', 3.14),\n        (7, 'Gem', 2.71828),\n        (8, 'Helen', 1.414);\n  - type: check\n    jm: |\n      Successfully applied schema change event AddColumnEvent{tableId=$database$.foo, addedColumns=[ColumnWithPosition{column=`digits` DOUBLE, position=LAST, existedColumnName=null}]} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[6, Faye, 3.14], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[7, Gem, 2.71828], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[8, Helen, 1.414], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      UPDATE foo SET digits = 42 WHERE idx < 4;\n  - type: check\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[1, Alice, null], after=[1, Alice, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[2, Brett, null], after=[2, Brett, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[3, Cicada, null], after=[3, Cicada, 42.0], op=UPDATE, meta=()}\n","flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Unexpected Structures\nsteps:\n  # Unexpected top-level block\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      mapping:\n        - source: userUuid\n          sink: uuid\n        - source: userId\n          sink: user_id\n        - source: deviceUuid\n          sink: device_uuid\n    error: |\n      Unexpected key `mapping` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  # Unexpected transform block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - source-table: foo.bar\n          mapping:\n            - source: userUuid\n              sink: uuid\n    error: |\n      Unexpected key `mapping` in YAML transform block.\n      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, table-options.delimiter, description, converter-after-transform]\n      Note: option mapping: [{\"source\":\"userUuid\",\"sink\":\"uuid\"}] is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected route block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar\n          sink-table: fun.baz\n          regex-style: java\n    error: |\n      Unexpected key `regex-style` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option regex-style: \"java\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected UDF block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          - name: addone\n            classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n            language: clojure\n    error: |\n      Unexpected key `language` in YAML UDF block.\n      Allowed keys in this context are: [name, classpath, python-code, python-executable, python-files, options]\n      Note: option language: \"clojure\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n","tools/cdcup/README.md":"# cdcup\n\nA `docker` (`compose`) environment on Linux / macOS is required to play with this. Ruby is **not** necessary.\n\n## `./cdcup.sh init`\n\nInitialize a playground environment, and generate configuration files.\n\n## `./cdcup.sh up`\n\nStart docker containers. Note that it may take a while before database is ready.\n\n## `./cdcup.sh pipeline <yaml>`\n\nSubmit a YAML pipeline job. Before executing this, please ensure that:\n\n1. All container are running and ready for connections\n2. (For MySQL) You've created at least one database & tables to be captured\n\n## `./cdcup.sh flink`\n\nPrint Flink Web dashboard URL.\n\n## `./cdcup.sh stop`\n\nStop all running playground containers.\n\n## `./cdcup.sh down`\n\nStop and remove containers, networks, and volumes.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"<!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\n# Flink CDC AI Agent Instructions\n\nThis file provides guidance for AI coding agents working with the Apache Flink CDC.\n\n## Prerequisites\n\n- Java 11 (baseline; all code must compile and run on Java 11) and Java 17\n- Maven 3.8.6 or higher\n- Git\n- Unix-like environment (Linux, macOS, WSL)\n- Docker (required for running integration tests and e2e tests)\n\n## Commands\n\n### Build\n\n- Fast dev build (skip tests and format checks): `mvn clean install -DskipTests -Dspotless.check.skip=true -Dcheckstyle.skip=true`\n- Full build against Flink 1.x (default): `mvn clean package -DskipTests`\n- Full build against Flink 2.x: `mvn clean package -DskipTests -Pflink2`\n- Single module (e.g. `flink-cdc-common`): `mvn clean package -DskipTests -pl flink-cdc-common -am`\n\n### Testing\n\n- Run all tests for a module: `mvn verify -pl flink-cdc-common`\n- Run all tests against Flink 2.x: `mvn verify -pl flink-cdc-common -Pflink2`\n- Run a single test class: `mvn -pl flink-cdc-common -Dtest=MyTest test`\n- Run a single test method: `mvn -pl flink-cdc-common -Dtest=MyTest#myMethod test`\n\n### Code Quality\n\n- Before committing changes, run `mvn spotless:apply` and `mvn spotless:check` to enforce code style rules.\n- Make sure newly added files have proper ASF license headers.\n\n## Repository Structure\n\nFlink CDC is organized around the Pipeline abstraction: a user-defined data pipeline that reads from one or more sources, optionally transforms records, and writes to one or more sinks. The core modules implement this abstraction; the connector modules provide the concrete source and sink implementations.\n\n### Core Modules\n\n- `flink-cdc-common` — Shared API and data model used across all modules: CDC event types (`DataChangeEvent`, `SchemaChangeEvent`, etc.), the schema model, data types, source/sink interfaces, `Factory` SPI, route definitions, UDF interfaces, and utility classes. Most new abstractions start here.\n- `flink-cdc-runtime` — Runtime implementation of the Pipeline: operators for reading, routing, transforming (expression evaluation via Calcite + Janino), and writing CDC events.\n- `flink-cdc-composer` — Pipeline assembly and deployment layer. Translates a `PipelineDefinition` into a runnable Flink job, wiring sources, operators, and sinks together. Supports Flink-native, Kubernetes, and YARN deployment.\n- `flink-cdc-cli` — Command-line entry point (`flink-cdc.sh`). Parses YAML pipeline definitions and delegates to `flink-cdc-composer`.\n- `flink-cdc-dist` — Distribution packaging. Produces the `flink-cdc-<version>-bin` release archive.\n\n### Flink Version Compatibility\n\nCurrently, Flink CDC supports two Flink generations simultaneously:\n\n- `flink-cdc-flink1-compat` — Flink 1.x compatibility layer (currently 1.20.3). Default profile.\n- `flink-cdc-flink2-compat` — Flink 2.x compatibility layer (currently 2.2.0). Activated via `-Pflink2`.\n\nAll modules that depend on Flink APIs must declare their Flink dependencies as `provided` and reference `${flink.version}`, which is resolved by the active profile.\nPlease verify your changes in both Flink 1.20 (LTS) and Flink 2.x (latest).\n\n### Connectors (`flink-cdc-connect/`)\n\nConnectors are split into two categories:\n\n- Source Connectors (`flink-cdc-connect/flink-cdc-source-connectors/`) are CDC sources for DataStream and Flink SQL jobs.\n- Pipeline Connectors (`flink-cdc-connect/flink-cdc-pipeline-connectors/`) are Pipeline connectors for the YAML API.\n\n### Tests\n\nWrite unit tests and integration tests in the corresponding submodules. End-to-end tests are located in these modules:\n\n- `flink-cdc-e2e-tests/` — End-to-end tests parent module.\n  - `flink-cdc-e2e-utils` — Shared test utilities (container management, assertions)\n  - `flink-cdc-source-e2e-tests` — E2E tests for source connectors\n  - `flink-cdc-pipeline-e2e-tests` — E2E tests for pipeline connectors\n\n### Documentation\n\n- `docs/` — Hugo-based documentation site. `docs/content/` for English, `docs/content.zh/` for Chinese. Update both when adding new features.\n\n## Coding Standards\n\n- **Format Java files with Spotless before every commit:** `mvn spotless:apply`.\n- **Import order** (enforced by Checkstyle): `org.apache.flink.cdc` → `org.apache.flink` → other third-party → `javax` → `java`. Static imports go last. No star imports.\n- **Forbidden imports** (enforced by Checkstyle):\n  - JUnit 4 (`org.junit.*` except `org.junit.jupiter.*`) — use JUnit 5 Jupiter instead\n  - `org.junit.jupiter.api.Assertions` and `org.hamcrest` — use AssertJ instead\n  - `com.google.common.*` — use `flink-shaded-guava` instead\n  - `com.google.common.base.Preconditions` — use Flink CDC's `Preconditions`\n  - `com.google.common.annotations.VisibleForTesting` — use `@VisibleForTesting` from `org.apache.flink.cdc.common.annotation`\n- **API stability annotations:** User-facing API types should carry stability annotations, and methods/fields/constructors only need explicit annotations when they differ from the enclosing type or when needed.\n  - `@Public` — stable across major versions\n  - `@PublicEvolving` — may change in minor versions\n  - `@Experimental` — may change at any time\n  - `@Internal` — no stability guarantee; users should not depend on it\n- **Logging:** Use SLF4J with parameterized placeholders (`LOG.info(\"foo {}\", bar)`), never string concatenation.\n- **Braces:** Always use braces for `if`/`else`/`for`/`while`/`do` blocks.\n- **Comments:** All comments should be written in English, and keep it concise. Only write comments when necessary. Avoid trivial `@param`, `@return` comments in JavaDocs.\n- Apache License 2.0 header required on all new files.\n\n## Testing Standards\n\n- Use **JUnit 5** (`org.junit.jupiter`) and **AssertJ** (`org.assertj.core.api.Assertions`) for all new tests. Do not use JUnit 4 or Hamcrest.\n- Test classes are **package-private** (no `public` modifier on the class).\n- **Unit tests** are named `*Test.java` (e.g., `SchemaUtilsTest`).\n- **Integration tests** (requiring Docker / real databases) are named `*ITCase.java` (e.g., `MySqlSourceITCase`).\n- Add tests for new behavior, covering success, failure, and edge cases. Tests should be self-explanatory; avoid verbose comments in test classes.\n- For bug fixes, verify the new test fails without the fix before confirming it passes with it.\n\n## Commits and PRs\n\n### Commit message format\n\n- `[FLINK-XXXX][component] Description` where `FLINK-XXXX` is the JIRA issue number and `component` is the affected area (e.g. `connect/mysql`, `pipeline-connector/kafka`, `docs`, `runtime`)\n- `[hotfix][component] ...` or `[docs][component] ...` for trivial fixes without a JIRA issue\n- `[ci] Description` for CI-only changes\n\n### Pull request conventions\n\n- Title format mirrors the commit message: `[FLINK-XXXX][component] Title`\n- A corresponding JIRA issue is required (except trivial changes)\n- Fill out the PR template completely: describe purpose, change log, testing approach, and documentation impact\n- Ensure CI passes before requesting review\n- Enable GitHub Actions on your fork before opening a PR\n- When AI tools were used, check the AI disclosure checkbox and uncomment the `Generated-by` line in the PR template, per [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html)\n\n## Boundaries\n\n### Ask first\n\n- Adding or changing `@Public` or `@PublicEvolving` annotations (user-facing API commitments)\n- New dependencies\n- Large cross-module refactors\n- Changes to serialization formats or checkpoint behavior\n- Changes to hot paths that could impact performance (per-record processing, state access)\n\n### Never\n\n- Commit secrets, credentials, or tokens\n- Push directly to `apache/flink-cdc`; always work from your fork\n- Mix unrelated changes into one PR\n- Use JUnit 4 or Hamcrest in new test code\n- Use `org.junit.jupiter.api.Assertions`; use AssertJ instead\n- Add `Co-Authored-By` with an AI agent in commit messages; use `Generated-by: <Tool Name and Version>` instead","isInternal":false,"tokens":2138,"sizeBytes":8736},{"name":"incomplete.yaml","path":"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Incomplete Structures\nsteps:\n  # Missing source\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n    error: Missing required field \"sink\" in top-level configuration\n  # Missing source\n  - type: submit\n    yaml: |\n      sink:\n        type: values\n    error: Missing required field \"source\" in top-level configuration\n  # Missing required options\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n      sink:\n        type: values\n    error: |\n      Missing required options are:\n      hostname\n      password\n      tables\n      username\n  # Missing transform components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - projection: \\*\n    error: Missing required field \"source-table\" in transform configuration\n  # Missing routing components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - sink-table: foo.bar.baz\n    error: Missing required field \"source-table\" in route configuration\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar.baz\n    error: Missing required field \"sink-table\" in route configuration\n","isInternal":false,"tokens":499,"sizeBytes":2194},{"name":"invalid.yaml","path":"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Invalid Cases\nsteps:\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      execution:\n        execution.coordinator-close-timeout: 300000\n        checkpointing:\n          interval: 600000\n          timeout: 30000\n          max-concurrent-checkpoints: 1\n          min-pause-between-checkpoints: 300000\n    error: |\n      Unexpected key `execution` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: user_course.\\.*\n          sink-table: ods.ods_user_course_<>\n          replace-symbol: <>\n          description: sync table to one destination table with given prefix ods_\n          transformers:\n            - type: cast\n              fields:\n                - name: \"*\"\n                  type: BIT\n                  target-type: TINYINT\n    error: |\n      Unexpected key `transformers` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option transformers: [{\"type\":\"cast\",\"fields\":[{\"name\":\"*\",\"type\":\"BIT\",\"target-type\":\"TINYINT\"}]}] is unexpected.\n      It was silently ignored in previous versions, and probably should be removed.","isInternal":false,"tokens":498,"sizeBytes":2259},{"name":"malformed.yaml","path":"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Malformed Structures\nsteps:\n  # Transform rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        # ERROR: should be `- source-table: foo.bar`\n        source-table: foo.bar\n    error: |\n      YAML transform block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # Route rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        source-table: foo.bar\n    error: |\n      YAML route block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # UDF functions not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          name: addone\n          classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n    error: |\n      YAML UDF block is expecting an array children, but got an OBJECT ({\"name\":\"addone\",\"classpath\":\"org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\"}).\n      Perhaps you missed a dash prefix `-`?\n","isInternal":false,"tokens":475,"sizeBytes":2050},{"name":"unexpected.yaml","path":"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Unexpected Structures\nsteps:\n  # Unexpected top-level block\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      mapping:\n        - source: userUuid\n          sink: uuid\n        - source: userId\n          sink: user_id\n        - source: deviceUuid\n          sink: device_uuid\n    error: |\n      Unexpected key `mapping` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  # Unexpected transform block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - source-table: foo.bar\n          mapping:\n            - source: userUuid\n              sink: uuid\n    error: |\n      Unexpected key `mapping` in YAML transform block.\n      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, table-options.delimiter, description, converter-after-transform]\n      Note: option mapping: [{\"source\":\"userUuid\",\"sink\":\"uuid\"}] is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected route block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar\n          sink-table: fun.baz\n          regex-style: java\n    error: |\n      Unexpected key `regex-style` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option regex-style: \"java\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected UDF block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          - name: addone\n            classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n            language: clojure\n    error: |\n      Unexpected key `language` in YAML UDF block.\n      Allowed keys in this context are: [name, classpath, python-code, python-executable, python-files, options]\n      Note: option language: \"clojure\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n","isInternal":false,"tokens":713,"sizeBytes":3202},{"name":"vanilla.yaml","path":"flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Vanilla Test Specs\nsteps:\n  - type: exec\n    sql: |\n      CREATE TABLE foo (idx BIGINT NOT NULL PRIMARY KEY, name VARCHAR(17));\n      INSERT INTO foo VALUES\n        (1, 'Alice'),\n        (2, 'Brett'),\n        (3, 'Cicada'),\n        (4, 'Derrida'),\n        (5, 'Eve');\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n      sink:\n        type: values\n      pipeline:\n        parallelism: 1\n  - type: check\n    jm: |\n      Successfully applied schema change event CreateTableEvent{tableId=$database$.foo, schema=columns={`idx` BIGINT NOT NULL,`name` VARCHAR(17)}, primaryKeys=idx, options=()} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[1, Alice], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[2, Brett], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[3, Cicada], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[4, Derrida], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[5, Eve], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      ALTER TABLE foo ADD COLUMN (digits DOUBLE);\n      INSERT INTO foo VALUES\n        (6, 'Faye', 3.14),\n        (7, 'Gem', 2.71828),\n        (8, 'Helen', 1.414);\n  - type: check\n    jm: |\n      Successfully applied schema change event AddColumnEvent{tableId=$database$.foo, addedColumns=[ColumnWithPosition{column=`digits` DOUBLE, position=LAST, existedColumnName=null}]} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[6, Faye, 3.14], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[7, Gem, 2.71828], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[8, Helen, 1.414], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      UPDATE foo SET digits = 42 WHERE idx < 4;\n  - type: check\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[1, Alice, null], after=[1, Alice, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[2, Brett, null], after=[2, Brett, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[3, Cicada, null], after=[3, Cicada, 42.0], op=UPDATE, meta=()}\n","isInternal":false,"tokens":872,"sizeBytes":3213},{"name":"README.md","path":"tools/cdcup/README.md","rawUrl":"https://raw.githubusercontent.com/apache/flink-cdc/HEAD/tools/cdcup/README.md","title":"cdcup Documentation","category":"plugin-manifest","format":"markdown","content":"# cdcup\n\nA `docker` (`compose`) environment on Linux / macOS is required to play with this. Ruby is **not** necessary.\n\n## `./cdcup.sh init`\n\nInitialize a playground environment, and generate configuration files.\n\n## `./cdcup.sh up`\n\nStart docker containers. Note that it may take a while before database is ready.\n\n## `./cdcup.sh pipeline <yaml>`\n\nSubmit a YAML pipeline job. Before executing this, please ensure that:\n\n1. All container are running and ready for connections\n2. (For MySQL) You've created at least one database & tables to be captured\n\n## `./cdcup.sh flink`\n\nPrint Flink Web dashboard URL.\n\n## `./cdcup.sh stop`\n\nStop all running playground containers.\n\n## `./cdcup.sh down`\n\nStop and remove containers, networks, and volumes.\n","isInternal":false,"tokens":176,"sizeBytes":744}],"systemPromptSnippet":"<yakaai_skills repo=\"apache/flink-cdc\">\n<!-- File: AGENTS.md (Tokens: ~2138 | Category: root-instruction) -->\n<!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\n# Flink CDC AI Agent Instructions\n\nThis file provides guidance for AI coding agents working with the Apache Flink CDC.\n\n## Prerequisites\n\n- Java 11 (baseline; all code must compile and run on Java 11) and Java 17\n- Maven 3.8.6 or higher\n- Git\n- Unix-like environment (Linux, macOS, WSL)\n- Docker (required for running integration tests and e2e tests)\n\n## Commands\n\n### Build\n\n- Fast dev build (skip tests and format checks): `mvn clean install -DskipTests -Dspotless.check.skip=true -Dcheckstyle.skip=true`\n- Full build against Flink 1.x (default): `mvn clean package -DskipTests`\n- Full build against Flink 2.x: `mvn clean package -DskipTests -Pflink2`\n- Single module (e.g. `flink-cdc-common`): `mvn clean package -DskipTests -pl flink-cdc-common -am`\n\n### Testing\n\n- Run all tests for a module: `mvn verify -pl flink-cdc-common`\n- Run all tests against Flink 2.x: `mvn verify -pl flink-cdc-common -Pflink2`\n- Run a single test class: `mvn -pl flink-cdc-common -Dtest=MyTest test`\n- Run a single test method: `mvn -pl flink-cdc-common -Dtest=MyTest#myMethod test`\n\n### Code Quality\n\n- Before committing changes, run `mvn spotless:apply` and `mvn spotless:check` to enforce code style rules.\n- Make sure newly added files have proper ASF license headers.\n\n## Repository Structure\n\nFlink CDC is organized around the Pipeline abstraction: a user-defined data pipeline that reads from one or more sources, optionally transforms records, and writes to one or more sinks. The core modules implement this abstraction; the connector modules provide the concrete source and sink implementations.\n\n### Core Modules\n\n- `flink-cdc-common` — Shared API and data model used across all modules: CDC event types (`DataChangeEvent`, `SchemaChangeEvent`, etc.), the schema model, data types, source/sink interfaces, `Factory` SPI, route definitions, UDF interfaces, and utility classes. Most new abstractions start here.\n- `flink-cdc-runtime` — Runtime implementation of the Pipeline: operators for reading, routing, transforming (expression evaluation via Calcite + Janino), and writing CDC events.\n- `flink-cdc-composer` — Pipeline assembly and deployment layer. Translates a `PipelineDefinition` into a runnable Flink job, wiring sources, operators, and sinks together. Supports Flink-native, Kubernetes, and YARN deployment.\n- `flink-cdc-cli` — Command-line entry point (`flink-cdc.sh`). Parses YAML pipeline definitions and delegates to `flink-cdc-composer`.\n- `flink-cdc-dist` — Distribution packaging. Produces the `flink-cdc-<version>-bin` release archive.\n\n### Flink Version Compatibility\n\nCurrently, Flink CDC supports two Flink generations simultaneously:\n\n- `flink-cdc-flink1-compat` — Flink 1.x compatibility layer (currently 1.20.3). Default profile.\n- `flink-cdc-flink2-compat` — Flink 2.x compatibility layer (currently 2.2.0). Activated via `-Pflink2`.\n\nAll modules that depend on Flink APIs must declare their Flink dependencies as `provided` and reference `${flink.version}`, which is resolved by the active profile.\nPlease verify your changes in both Flink 1.20 (LTS) and Flink 2.x (latest).\n\n### Connectors (`flink-cdc-connect/`)\n\nConnectors are split into two categories:\n\n- Source Connectors (`flink-cdc-connect/flink-cdc-source-connectors/`) are CDC sources for DataStream and Flink SQL jobs.\n- Pipeline Connectors (`flink-cdc-connect/flink-cdc-pipeline-connectors/`) are Pipeline connectors for the YAML API.\n\n### Tests\n\nWrite unit tests and integration tests in the corresponding submodules. End-to-end tests are located in these modules:\n\n- `flink-cdc-e2e-tests/` — End-to-end tests parent module.\n  - `flink-cdc-e2e-utils` — Shared test utilities (container management, assertions)\n  - `flink-cdc-source-e2e-tests` — E2E tests for source connectors\n  - `flink-cdc-pipeline-e2e-tests` — E2E tests for pipeline connectors\n\n### Documentation\n\n- `docs/` — Hugo-based documentation site. `docs/content/` for English, `docs/content.zh/` for Chinese. Update both when adding new features.\n\n## Coding Standards\n\n- **Format Java files with Spotless before every commit:** `mvn spotless:apply`.\n- **Import order** (enforced by Checkstyle): `org.apache.flink.cdc` → `org.apache.flink` → other third-party → `javax` → `java`. Static imports go last. No star imports.\n- **Forbidden imports** (enforced by Checkstyle):\n  - JUnit 4 (`org.junit.*` except `org.junit.jupiter.*`) — use JUnit 5 Jupiter instead\n  - `org.junit.jupiter.api.Assertions` and `org.hamcrest` — use AssertJ instead\n  - `com.google.common.*` — use `flink-shaded-guava` instead\n  - `com.google.common.base.Preconditions` — use Flink CDC's `Preconditions`\n  - `com.google.common.annotations.VisibleForTesting` — use `@VisibleForTesting` from `org.apache.flink.cdc.common.annotation`\n- **API stability annotations:** User-facing API types should carry stability annotations, and methods/fields/constructors only need explicit annotations when they differ from the enclosing type or when needed.\n  - `@Public` — stable across major versions\n  - `@PublicEvolving` — may change in minor versions\n  - `@Experimental` — may change at any time\n  - `@Internal` — no stability guarantee; users should not depend on it\n- **Logging:** Use SLF4J with parameterized placeholders (`LOG.info(\"foo {}\", bar)`), never string concatenation.\n- **Braces:** Always use braces for `if`/`else`/`for`/`while`/`do` blocks.\n- **Comments:** All comments should be written in English, and keep it concise. Only write comments when necessary. Avoid trivial `@param`, `@return` comments in JavaDocs.\n- Apache License 2.0 header required on all new files.\n\n## Testing Standards\n\n- Use **JUnit 5** (`org.junit.jupiter`) and **AssertJ** (`org.assertj.core.api.Assertions`) for all new tests. Do not use JUnit 4 or Hamcrest.\n- Test classes are **package-private** (no `public` modifier on the class).\n- **Unit tests** are named `*Test.java` (e.g., `SchemaUtilsTest`).\n- **Integration tests** (requiring Docker / real databases) are named `*ITCase.java` (e.g., `MySqlSourceITCase`).\n- Add tests for new behavior, covering success, failure, and edge cases. Tests should be self-explanatory; avoid verbose comments in test classes.\n- For bug fixes, verify the new test fails without the fix before confirming it passes with it.\n\n## Commits and PRs\n\n### Commit message format\n\n- `[FLINK-XXXX][component] Description` where `FLINK-XXXX` is the JIRA issue number and `component` is the affected area (e.g. `connect/mysql`, `pipeline-connector/kafka`, `docs`, `runtime`)\n- `[hotfix][component] ...` or `[docs][component] ...` for trivial fixes without a JIRA issue\n- `[ci] Description` for CI-only changes\n\n### Pull request conventions\n\n- Title format mirrors the commit message: `[FLINK-XXXX][component] Title`\n- A corresponding JIRA issue is required (except trivial changes)\n- Fill out the PR template completely: describe purpose, change log, testing approach, and documentation impact\n- Ensure CI passes before requesting review\n- Enable GitHub Actions on your fork before opening a PR\n- When AI tools were used, check the AI disclosure checkbox and uncomment the `Generated-by` line in the PR template, per [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html)\n\n## Boundaries\n\n### Ask first\n\n- Adding or changing `@Public` or `@PublicEvolving` annotations (user-facing API commitments)\n- New dependencies\n- Large cross-module refactors\n- Changes to serialization formats or checkpoint behavior\n- Changes to hot paths that could impact performance (per-record processing, state access)\n\n### Never\n\n- Commit secrets, credentials, or tokens\n- Push directly to `apache/flink-cdc`; always work from your fork\n- Mix unrelated changes into one PR\n- Use JUnit 4 or Hamcrest in new test code\n- Use `org.junit.jupiter.api.Assertions`; use AssertJ instead\n- Add `Co-Authored-By` with an AI agent in commit messages; use `Generated-by: <Tool Name and Version>` instead\n\n<!-- File: flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/incomplete.yaml (Tokens: ~499 | Category: anthropic-skill) -->\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Incomplete Structures\nsteps:\n  # Missing source\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n    error: Missing required field \"sink\" in top-level configuration\n  # Missing source\n  - type: submit\n    yaml: |\n      sink:\n        type: values\n    error: Missing required field \"source\" in top-level configuration\n  # Missing required options\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n      sink:\n        type: values\n    error: |\n      Missing required options are:\n      hostname\n      password\n      tables\n      username\n  # Missing transform components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - projection: \\*\n    error: Missing required field \"source-table\" in transform configuration\n  # Missing routing components\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - sink-table: foo.bar.baz\n    error: Missing required field \"source-table\" in route configuration\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar.baz\n    error: Missing required field \"sink-table\" in route configuration\n\n<!-- File: flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/invalid.yaml (Tokens: ~498 | Category: anthropic-skill) -->\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Invalid Cases\nsteps:\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      execution:\n        execution.coordinator-close-timeout: 300000\n        checkpointing:\n          interval: 600000\n          timeout: 30000\n          max-concurrent-checkpoints: 1\n          min-pause-between-checkpoints: 300000\n    error: |\n      Unexpected key `execution` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: user_course.\\.*\n          sink-table: ods.ods_user_course_<>\n          replace-symbol: <>\n          description: sync table to one destination table with given prefix ods_\n          transformers:\n            - type: cast\n              fields:\n                - name: \"*\"\n                  type: BIT\n                  target-type: TINYINT\n    error: |\n      Unexpected key `transformers` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option transformers: [{\"type\":\"cast\",\"fields\":[{\"name\":\"*\",\"type\":\"BIT\",\"target-type\":\"TINYINT\"}]}] is unexpected.\n      It was silently ignored in previous versions, and probably should be removed.\n\n<!-- File: flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/malformed.yaml (Tokens: ~475 | Category: anthropic-skill) -->\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Malformed Structures\nsteps:\n  # Transform rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        # ERROR: should be `- source-table: foo.bar`\n        source-table: foo.bar\n    error: |\n      YAML transform block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # Route rules not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        source-table: foo.bar\n    error: |\n      YAML route block is expecting an array children, but got an OBJECT ({\"source-table\":\"foo.bar\"}).\n      Perhaps you missed a dash prefix `-`?\n  # UDF functions not an array\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          name: addone\n          classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n    error: |\n      YAML UDF block is expecting an array children, but got an OBJECT ({\"name\":\"addone\",\"classpath\":\"org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\"}).\n      Perhaps you missed a dash prefix `-`?\n\n<!-- File: flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/unexpected.yaml (Tokens: ~713 | Category: anthropic-skill) -->\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Unexpected Structures\nsteps:\n  # Unexpected top-level block\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      mapping:\n        - source: userUuid\n          sink: uuid\n        - source: userId\n          sink: user_id\n        - source: deviceUuid\n          sink: device_uuid\n    error: |\n      Unexpected key `mapping` in YAML top-level block.\n      Allowed keys in this context are: [source, sink, route, transform, pipeline]\n      Note: Flink configurations should be defined in \"Runtime Configurations\" instead of YAML scripts.\n  # Unexpected transform block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      transform:\n        - source-table: foo.bar\n          mapping:\n            - source: userUuid\n              sink: uuid\n    error: |\n      Unexpected key `mapping` in YAML transform block.\n      Allowed keys in this context are: [source-table, projection, filter, primary-keys, partition-keys, table-options, table-options.delimiter, description, converter-after-transform]\n      Note: option mapping: [{\"source\":\"userUuid\",\"sink\":\"uuid\"}] is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected route block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      route:\n        - source-table: foo.bar\n          sink-table: fun.baz\n          regex-style: java\n    error: |\n      Unexpected key `regex-style` in YAML route block.\n      Allowed keys in this context are: [source-table, sink-table, replace-symbol, description]\n      Note: option regex-style: \"java\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n  # Unexpected UDF block keys\n  - type: submit\n    yaml: |\n      source:\n        type: values\n      sink:\n        type: values\n      pipeline:\n        user-defined-function:\n          - name: addone\n            classpath: org.apache.flink.cdc.udf.examples.%s.AddOneFunctionClass\n            language: clojure\n    error: |\n      Unexpected key `language` in YAML UDF block.\n      Allowed keys in this context are: [name, classpath, python-code, python-executable, python-files, options]\n      Note: option language: \"clojure\" is unexpected. It was silently ignored in previous versions, and probably should be removed.\n\n<!-- File: flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/rules/vanilla.yaml (Tokens: ~872 | Category: anthropic-skill) -->\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements.  See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License.  You may obtain a copy of the License at\n#\n#      http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\ngroup: Common\nspec: Vanilla Test Specs\nsteps:\n  - type: exec\n    sql: |\n      CREATE TABLE foo (idx BIGINT NOT NULL PRIMARY KEY, name VARCHAR(17));\n      INSERT INTO foo VALUES\n        (1, 'Alice'),\n        (2, 'Brett'),\n        (3, 'Cicada'),\n        (4, 'Derrida'),\n        (5, 'Eve');\n  - type: submit\n    yaml: |\n      source:\n        type: mysql\n        hostname: $hostname$\n        tables: $database$.\\.*\n        username: $username$\n        password: $password$\n      sink:\n        type: values\n      pipeline:\n        parallelism: 1\n  - type: check\n    jm: |\n      Successfully applied schema change event CreateTableEvent{tableId=$database$.foo, schema=columns={`idx` BIGINT NOT NULL,`name` VARCHAR(17)}, primaryKeys=idx, options=()} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[1, Alice], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[2, Brett], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[3, Cicada], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[4, Derrida], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[5, Eve], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      ALTER TABLE foo ADD COLUMN (digits DOUBLE);\n      INSERT INTO foo VALUES\n        (6, 'Faye', 3.14),\n        (7, 'Gem', 2.71828),\n        (8, 'Helen', 1.414);\n  - type: check\n    jm: |\n      Successfully applied schema change event AddColumnEvent{tableId=$database$.foo, addedColumns=[ColumnWithPosition{column=`digits` DOUBLE, position=LAST, existedColumnName=null}]} to external system.\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[6, Faye, 3.14], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[7, Gem, 2.71828], op=INSERT, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[], after=[8, Helen, 1.414], op=INSERT, meta=()}\n  - type: exec\n    sql: |\n      UPDATE foo SET digits = 42 WHERE idx < 4;\n  - type: check\n    tm: |\n      DataChangeEvent{tableId=$database$.foo, before=[1, Alice, null], after=[1, Alice, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[2, Brett, null], after=[2, Brett, 42.0], op=UPDATE, meta=()}\n      DataChangeEvent{tableId=$database$.foo, before=[3, Cicada, null], after=[3, Cicada, 42.0], op=UPDATE, meta=()}\n\n<!-- File: tools/cdcup/README.md (Tokens: ~176 | Category: plugin-manifest) -->\n# cdcup\n\nA `docker` (`compose`) environment on Linux / macOS is required to play with this. Ruby is **not** necessary.\n\n## `./cdcup.sh init`\n\nInitialize a playground environment, and generate configuration files.\n\n## `./cdcup.sh up`\n\nStart docker containers. Note that it may take a while before database is ready.\n\n## `./cdcup.sh pipeline <yaml>`\n\nSubmit a YAML pipeline job. Before executing this, please ensure that:\n\n1. All container are running and ready for connections\n2. (For MySQL) You've created at least one database & tables to be captured\n\n## `./cdcup.sh flink`\n\nPrint Flink Web dashboard URL.\n\n## `./cdcup.sh stop`\n\nStop all running playground containers.\n\n## `./cdcup.sh down`\n\nStop and remove containers, networks, and volumes.\n</yakaai_skills>"}