{"owner":"citusdata","repo":"citus","hasSkills":true,"totalSkillsCount":7,"totalTokensCount":17324,"categories":["copilot-instructions","anthropic-skill","command-prompt"],"hasMcp":false,"mcpConfig":null,"found":[".github/copilot-instructions.md",".github/skills/citus-backport/references/manual-upgrade-testing.md",".github/skills/citus-backport/references/ci-triage.md",".github/skills/README.md",".github/skills/citus-backport/references/sql-schema-backport.md",".github/skills/citus-backport/SKILL.md","src/backend/distributed/commands/README.md"],"skills":{".github/copilot-instructions.md":"# Copilot instructions for citus\n\nFollow the repository's normal contribution conventions: see `CONTRIBUTING.md` for\nbuild/style/PR rules and `src/test/regress/README.md` for the regression-test workflow.\n\n## Agent skills\n\nThis repo ships task-specific **agent skills** under `.github/skills/`. Before starting\na task that matches one, load that skill's `SKILL.md` and follow it so you don't re-derive\nit. Each skill's `SKILL.md` frontmatter `description` states exactly when to use it; the\nindex below is a quick map.\n\n| Task | Skill |\n|------|-------|\n| Backport a merged `main` commit/PR to a release branch (`release-13.2` / `release-14.0` / newest two majors) — including SQL-schema migrations, the upgrade/downgrade ladder, N-1 / Major-Version-Upgrade safety, and triaging release-branch CI | [`citus-backport`](skills/citus-backport/SKILL.md) |\n| Merge a batch of Citus PRs one at a time: sync each with its base branch, fix a red `check-style` job, retry other red checks with give-up rules, then squash-merge with the PR description as the commit message | [`citus-merge-loop`](skills/citus-merge-loop/SKILL.md) |\n| Fix a failing `check-style` job on whatever branch is already checked out, by rebuilding the exact `uncrustify`/`citus_indent` versions that branch itself pins (in its own `STYLEGUIDE.md` and `.github/workflows/build_and_test.yml`), then running `make reindent` and committing the style-only diff locally (no push, and no switching to or inspecting any other Citus branch) | [`citus-check-style-reindent`](skills/citus-check-style-reindent/SKILL.md) |\n\nSee `.github/skills/README.md` for the skills layout and how to add a new one.\n",".github/skills/citus-backport/references/manual-upgrade-testing.md":"# Manual MVU (major-version-upgrade) testing for backport branches\n\nBackporting a SQL-schema change and propagating the upgrade ladder (see\n`sql-schema-backport.md` §Step 5) only *claims* MVU safety. This doc is how you **prove** it\nend-to-end on a real multi-node cluster: stand up N nodes at an OLD Citus major, then walk the\nextension forward across both backport branches — and, one hop further onto the\n`main`-targeting PR branch — with `ALTER EXTENSION citus UPDATE`, capturing `citus_version()` AND\n`pg_extension.extversion` on every node after every hop.\n\nThis is the \"manual testing for backport branches\" step. Do it after CI is green, before\ndeclaring a SQL-schema backport family done. A C-only backport does NOT need it.\n\n## What this proves\n- A cluster shipped at the 13.x tail (`default_version` of `release-13.2`, e.g. `13.4-1`) can\n  still cross to 14.x — i.e. a future managed-service 13→14 MVU won't strand the customer.\n- **Final hop:** a 14.x-tail cluster (e.g. `14.2-1`) can cross to the NEXT major on\n  `main` (`15.0-1`) via the PR branch you opened against `main` — proving the SAME object also\n  rides a future 14.x→15.0 MVU. `main` jumps `14.0-1 → 15.0-1` directly (it has NO 14.1/14.2\n  *upgrade-in-line*), but it MUST carry the `14.2-1--14.1-1` and `14.1-1--14.0-1` **downgrades**\n  so PG can route `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. If that route is missing you get \"no\n  update path from 14.2-1 to 15.0-1\" — a real MVU gap in the ladder propagation to `main`.\n- The forward/downgrade **ladder** you propagated actually routes end to end.\n- The backported object (UDF/GUC/column) is present and the cluster is functional after each jump.\n\n## The single hard requirement: ONE PG, three Citus builds\n`ALTER EXTENSION citus UPDATE` walks the version graph made of the `citus--A--B.sql` scripts\n**installed in the current PG's `SHAREDIR/extension/`**. So all three Citus versions in the walk\n(old baseline + both backport branches) must be installed, one at a time, into the SAME PG\ninstall / SAME cluster data dirs. You swap the installed Citus between hops and restart the nodes;\nthe catalog stays put and `ALTER EXTENSION` migrates it.\n\nUse ONE dedicated PG install as that single PG (its Citus gets overwritten repeatedly — that's the\npoint — then you restore it at the end, §Cleanup). Do the builds in **detached worktrees** so your\nreviewed branch checkouts are never disturbed.\n\n### `make install-all`, NOT `make install` — MANDATORY here\n`make install` lays down only **forward** scripts. The cross-major route from the 13.x tail is\n**not** a straight climb — there is no `13.4-1--14.x` edge. It goes:\n\n```\n13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1\n  (down)   (down)   (BRIDGE up)  (up)    (up)\n```\n\nso it needs the **downgrade** scripts `13.4-1--13.3-1`, `13.3-1--13.2-1` and the cross-major\n**bridge** `13.2-1--14.0-1` (all shipped under `sql/downgrades/` on `release-14.0`). `make install`\nomits every downgrade → `ALTER EXTENSION citus UPDATE` fails with \"no update path from 13.4-1 to\n14.2-1\". Always build the test PG with **`make install-all`** (it runs `install` +\n`install-downgrades` for both the distributed and columnar extensions).\n\n### PG version must be common to all builds\nBaseline Citus must support the chosen PG. 13.0/13.2 support PG15/16/17; 14.0 supports 16/17/18;\n`main` supports 16/17/18 → **PG 17** is the common denominator for all four. Pick a PG 17 install\nso it works for the whole 13.0→13.4→14.2→15.0 walk.\n\n## Recipe (verified with a 3-node cluster: citus 13.0.5 → 13.4.0 → 14.2.0 → 15.0devel)\n\nPick a shared PG (`SHARED=<path to your PG17 install>`), a spare coordinator port (e.g. `10600`;\nworkers take `+1/+2`), and the three source trees. Use detached worktrees so the reviewed\nbranch worktrees are never disturbed:\n\n```bash\ncd <main clone>            # holds all the branches/tags\ngit worktree add --detach /tmp/mvu/wt130 v13.0.5            # OLD baseline (a real tag)\ngit worktree add --detach /tmp/mvu/wt132 <bp-release-13.2-head-sha>\ngit worktree add --detach /tmp/mvu/wt140 <bp-release-14.0-head-sha>\ngit worktree add --detach /tmp/mvu/wtmain <bp-main-PR-head-sha>\n```\n\nBuild helper — reconfigure each tree against the SHARED PG and `install-all`:\n\n```bash\nbuild() {  # build <worktree> ; installs into $SHARED\n  cd \"$1\"\n  ./configure PG_CONFIG=\"$SHARED/bin/pg_config\" >/tmp/cfg.log 2>&1\n  local JOBS; JOBS=$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)   # portable: Linux || macOS\n  make -sj\"$JOBS\"             >/tmp/mk.log  2>&1\n  make -sj\"$JOBS\" install-all >/tmp/ins.log 2>&1\n  ls \"$SHARED/share/extension/\" | grep -E '^citus--1[3-5]' | sort   # sanity: see the ladder\n}\n```\nRe-use the same cluster data dirs across hops:\n\n1. **Baseline 13.0.** `build /tmp/mvu/wt130`. Stand up a cluster on `$SHARED` with `citus_dev`\n   (it `CREATE EXTENSION citus` at `default_version` = the currently-installed `13.0-1`). Verify\n   `SELECT extversion FROM pg_extension WHERE extname='citus'` and `SELECT citus_version()` = 13.0\n   on coordinator + both workers. (`citus_dev`'s default DB is `$USER`, NOT `postgres`.)\n2. **Up to the 13.2 branch.** `build /tmp/mvu/wt132` (overwrites Citus in `$SHARED`, now\n   `default_version = 13.4-1`, installs the 13.x up+down ladder). **Restart every node** so the new\n   `.so` loads. Then, on EACH node: `ALTER EXTENSION citus UPDATE;` → catalog goes `13.0-1 → 13.4-1`\n   in one call (walks the whole chain). Capture `citus_version()` per node — expect\n   `13.4.0 ... (sha: <13.2-head>)`.\n3. **Cross-major to the 14.0 branch.** `build /tmp/mvu/wt140` (`default_version = 14.2-1`, installs\n   the full up+down+bridge ladder). **Restart every node.** On EACH node: `ALTER EXTENSION citus\n   UPDATE;` → routes `13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1`. Capture\n   `citus_version()` per node — expect `14.2.0 ... (sha: <14.0-head>)`.\n4. **Functional check** on the upgraded cluster (coordinator): active `pg_dist_node`,\n   `create_distributed_table` + INSERT + distributed `count(*)`, `pg_dist_shard` count, and that\n   the backported object is present (e.g. `SELECT proname FROM pg_proc WHERE proname =\n   'distribute_object' AND pronamespace = 'citus_internal'::regnamespace;`).\n5. **One more hop onto the `main` PR branch.** Detached worktree at the `bp-<pr>-main`\n   head, `build` it into the SAME `$SHARED` (overwrites Citus to `default_version = 15.0-1`,\n   installs main's up+down ladder incl. the 14.x downgrades). **Restart every node.** On EACH node:\n   `ALTER EXTENSION citus UPDATE;` → routes `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. Re-run the\n   functional check. Expect `Citus 15.0devel ... (sha: <main-PR-head>)` and the backported object\n   still present at `15.0-1`.\n\n**KEY NUANCE — `citus_version()` reports the BINARY, `extversion` reports the CATALOG.** After you\ninstall a newer Citus + restart (but BEFORE `ALTER EXTENSION citus UPDATE`), `citus_version()`\nalready prints the new version (it is compiled into the `.so`), while `pg_extension.extversion` is\nstill the old catalog version. The migration is only proven by **`extversion` moving to the new\n`default_version`**. On the first hops the jump is a whole major so `citus_version()` alone is\ntelling; on the `14.2 → 15.0devel` hop both 14.2 and 15.0 builds may print `15.0devel`-ish strings,\nso capture `extversion` before+after (`14.2-1 → 15.0-1`) as the real evidence.\n\nRun the same `ALTER EXTENSION citus UPDATE` on each node by connecting to each port directly\n(coordinator does NOT auto-propagate the extension version to workers) — that's the \"update all\nnodes\" the ask means.\n\n## Environment notes (neutral, but they save time)\n- **Put SQL in `.sql` files and run `psql -f`; put shell logic in `.sh` files and run them.**\n  Piping SQL / loops inline through several shell layers (esp. cross-OS terminals) mangles quoting\n  and `$USER`. If a script is authored on a different-newline OS, strip CRLF first (`sed -i\n  's/\\r$//' file`) or `set -o pipefail` may die with \"invalid option name\".\n- **`citus_dev` records RELATIVE data-dir paths** (`cluster/<role>`). `pg_ctl -D <abs>` then fails\n  with \"could not access directory …/cluster/<role>\". Fix: `cd` into the clusters parent and use\n  `pg_ctl -D cluster/<role> …`.\n- **If `pg_ctl -w start` blocks**, the `-w` readiness probe can hang against an `ssl=on` citus_dev\n  config even though the server logged \"ready to accept connections\". Start WITHOUT `-w` and poll\n  the TCP port yourself (`ss -ltn | grep :<port>`). `-w` on **stop** is fine.\n- On slow disks the first post-restart checkpoint can take ~100s; judge node readiness by\n  \"database system is ready to accept connections\" in the node logfile + the port listening, not by\n  a `pg_ctl -w` wrapper returning.\n\n## Cleanup / restore\n- The shared PG's Citus was left at whatever you installed last (after the final main hop that\n  is main's `15.0-1`). **Restore the env's own build:** rebuild the PG's normal Citus branch with\n  `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install-all` (verify `default_version` + the ladder edges are back).\n- Stop the test cluster (`pg_ctl -D cluster/<role> -w -m fast stop` per node, or your cluster\n  helper's stop command) and `git worktree remove` the throwaway build trees (+ `git worktree\n  prune`) when done.\n\n## Expected evidence shape (what a PASS looks like)\n```\nBASELINE (citus 13.0):    all nodes → extversion 13.0-1  | Citus 13.0.5 ... gitref: v13.0.5\nafter 13.2-branch UPDATE: all nodes → extversion 13.4-1  | Citus 13.4.0 ... (sha: <13.2 head>)\nafter 14.0-branch UPDATE: all nodes → extversion 14.2-1  | Citus 14.2.0 ... (sha: <14.0 head>)\nafter main-PR   UPDATE:   all nodes → extversion 14.2-1→15.0-1 | Citus 15.0devel ... (sha: <main head>)\nfunctional (each hop): 3 active nodes, create_distributed_table OK, count matches, backported obj present\n```\nReport the per-node `extversion` (before+after) AND `citus_version()` for every hop — not a\nsummary — since the whole point is proving every node crossed each boundary.\n",".github/skills/citus-backport/references/ci-triage.md":"# CI triage for backports — PG-version compat + the baseline-red release branches\n\nTwo independent things make backport CI confusing: (1) `main` might support different PG majors\nthan a release branch, so a \"trivial C-only\" cherry-pick can fail to COMPILE on an old PG; and\n(2) the release branches are BASELINE-RED — several jobs fail on the pristine base branch with\nno backport. Handle both before declaring a red \"your fault\".\n\n## PG-version compat (the #1 non-obvious C-backport failure)\n`main` may support different PG majors than the release branches, so code `main` wrote unconditionally\nagainst a newer PG API breaks the older-PG build on a release branch. Check each branch's supported\nPG set in `.github/workflows/build_and_test.yml` (the `pgNN_version` params block). Today, these are the\nsupported PG majors by main and two most recent release branches:\n\n| Branch        | PG majors built      |\n|---------------|----------------------|\n| release-13.2  | **15**, 16, 17       |\n| release-14.0  | 16, 17, 18 (no 15)   |\n| main          | 16, 17, 18 (dropped 15) |\n\n**Concrete example (confirmed):** `Query.rteperminfos` was added in PG16. `main` sets\n`q->rteperminfos = NIL;` bare. On release-13.2 that is `error: 'Query' has no member named\n'rteperminfos'` on the **PG15** build. Fix = the citus idiom the branch ALREADY uses everywhere\nelse for that field:\n\n```c\n#if PG_VERSION_NUM >= PG_VERSION_16\n    q->rteperminfos = NIL;\n#endif\n```\n\nGrep the target branch for the same symbol to copy its exact guard macro. **General rule:** after a\nC cherry-pick, scan the added `+` lines for struct fields / API calls introduced in a PG major that\nthe branch's OLDEST PG predates, and guard them. **Backport to the oldest-PG branch first** to\nsurface these early. A single build environment builds ONE PG so it can't catch this locally — rely\non CI's per-PG `Build for PGNN` jobs.\n\n### Reading a Build-for-PGNN failure mid-run\n`gh run view --log` / `--log-failed` say \"still in progress\" until the WHOLE run finishes, but a\nsingle finished job's log is fetchable immediately:\n\n```\ngh api repos/<your-fork>/citus/actions/jobs/<jobId>/logs > build.log\n```\n\nThen grep for `error:` / `make.*Error`. A `Build for PGxx` failure with a real `error:` line = a\ngenuine bug (fix it). An \"exit code N\" annotation with no compiler `error:` in the log = usually\ninfra/flaky (rerun). After fixing, **amend the single cherry-pick commit** (keep 1 commit/PR),\n`git push --force-with-lease`, and re-dispatch:\n`gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`.\n\n## The release branches are BASELINE-RED\n`release-13.2` and `release-14.0` have KNOWN-FAILING jobs on their own pristine upstream CI. Do NOT\npanic when your backport branch is red — **subtract the baseline first.** As of this writing the\nbase branches fail these SAME jobs with no backport applied:\n\n- **N-1 mixed-version jobs where the OLD lib runs on the COORDINATOR:**\n  - Test Citus **Lib N-1** — check-multi, check-multi-1, check-add-backup-node\n  - Test Citus **Coordinator N-1** — check-multi, check-multi-1, check-add-backup-node\n  - (The SQL N-1 and Worker N-1 variants of the same tests PASS — old lib is only on workers there.)\n- Occasional **Test flakyness (N)** shards and a stray check-enterprise / check-operations.\n- **Test Citus Failure / check-query-generator** — a randomized differential fuzzer (compares\n  distributed vs local results of random nested-CTE/join queries). Flaky by nature; a random query\n  can trip `citus.max_intermediate_result_size` etc. **Tell:** one PG major fails while another PG\n  major of the SAME commit passes. Not a backport regression.\n\n**Root cause of the N-1 reds (not yours):** these base branches already carry in-flight\nminor-targeted features (e.g. clone-promotion sequence *re-ranging*;\n`citus_internal.delete_placement_metadata` behavior) whose NEW behavior lives only in the NEW lib.\nWhen the OLD lib (the previous minor's binary) is loaded on the coordinator, the expected `.out`\n(written for the new lib) mismatches → diff. Inherent to N-1 testing of in-flight features;\nunrelated to a feature backport that doesn't touch those areas.\n\n## How to PROVE a red is pre-existing (do this — gather the evidence)\n1. **Upstream base-branch run is itself failing:**\n   ```\n   gh run list --repo citusdata/citus --branch release-13.2 --workflow \"Build & Test\" --limit 6\n   ```\n   Note the base-branch runs are themselves `failure`.\n2. **Your failing set ⊆ upstream's failing set:**\n   ```\n   gh run view <upstreamRunId> --repo citusdata/citus --json jobs \\\n     --jq '.jobs[]|select(.conclusion==\"failure\")|.name'\n   ```\n   If your branch's failing job names are a subset, they're baseline, not yours.\n3. **The diff cites a pre-existing feature, not your object:** download your failed job's artifact\n   and read `regression.diffs`:\n   ```\n   gh run download <runId> --repo <your-fork>/citus -n \"<artifactName>\" -D <dir>\n   ```\n   Artifact naming: `<pgmajor>_<test>_[<citus_version>_]<libLabel>_<mode>`\n   (e.g. `18_check-enterprise_14.1-1_all` = SQL N-1; `..._v14.1.0_coordinatoronly` = Coordinator\n   N-1). Confirm the diff mentions the pre-existing feature (`re-ranged N sequence(s)`,\n   `delete_placement_metadata`) and has **ZERO** mentions of your backported symbol.\n4. **Best control:** run CI on your C-ONLY sibling backport (touches none of the SQL/dependency\n   code). If it shows the IDENTICAL N-1 reds, that confirms the reds track the base branch, not the\n   change.\n\n## What WOULD be your fault (fix these)\n- A red job whose `regression.diffs` names YOUR backported object **in a NORMAL (same-version)\n  job** — check-multi/-mx/-1 on a supported PG. (See the N-1 handling immediately below.)\n- A `Build for PGxx` with a real `error:` in its log.\n- A NEW job red that is GREEN on the upstream base branch. If it's a NEW-GUC/UDF N-1 red, the fix is\n  the schedule move below (not a test/output change); everything else, fix at the source.\n\nEverything else on these branches is baseline noise.\n\n## The NEW N-1 red a feature backport ADDS — move it to the N-1-excluded schedule\nBackporting a feature that adds a **new GUC or UDF** whose regression test runs under an N-1\nschedule (`multi_schedule` / `multi_1_schedule` → `check-multi`/`check-multi-1`) makes the **N-1\n(mixed-version)** jobs show that test as a NEW red. The red itself is EXPECTED graceful-degradation\n(not a defect), but the CLEAN fix is to relocate the test so N-1 never runs it — do NOT mangle the\nreviewed test, fabricate output, or just \"leave + document\". Proven on both the 13.2 and 14.0\nbackports of the `citus.allow_unsafe_insert_select_pushdown` GUC:\n- Test `allow_unsafe_insert_select_pushdown` (backported GUC `citus.allow_unsafe_insert_select_pushdown`).\n- Under N-1 the OLD lib runs on the coordinator (`lib-v13.3.0` / `lib-v14.1.0`), which lacks the GUC.\n  `SET citus.allow_unsafe_insert_select_pushdown` → `ERROR: invalid configuration parameter name ...\n  \"citus\" is a reserved prefix`, then the EXPLAIN falls back to the DEFAULT (pull-to-coordinator) plan.\n- **That fallback IS the N-1 contract** (old binary → GUC absent → safe pre-feature behavior); the diff\n  *demonstrates* compatibility. But a red CI job is still noise, so move the test off the N-1 matrix.\n\n**THE FIX: move the test line from its schedule into the placeholder section of\n`multi_1_create_citus_schedule`.** The N-1 jobs' `make_targets` (build_and_test.yml Coordinator/Lib\nN-1 blocks) list `check-multi`, `check-multi-1`, `check-add-backup-node`, ... but **NOT\n`check-multi-1-create-citus`**, so any test in `multi_1_create_citus_schedule` is N-1-invisible while\nSTILL running under `check-multi-1-create-citus` on every normal PG job (coverage preserved). Steps:\n1. Remove `test: <name>` from `multi_schedule` (mind the surrounding blank lines / parallel-group\n   comments — don't strand a comment that referenced it).\n2. Append a block under the placeholder header in `multi_1_create_citus_schedule`\n   (`# Have this placeholder section at the end for the tests that need to be moved back ... to\n   prevent N-1 test failures for the features that are not present in the minor version.`), mirroring\n   the EXISTING `citus_cluster_changes_block` block right there — same `# ----------` fencing and\n   phrasing so new additions look like the existing file.\n3. `run_test.py` auto-discovers the new schedule (it prints `SCHEDULE: multi_1_create_citus_schedule`);\n   the `.out` file is schedule-independent, so NO `.out` regen is needed — it's a pure schedule move.\n\n**N-1 version label = the actual N-1 CI version = the PREVIOUS MINOR (it advances every release).**\nRead the authoritative number from `build_and_test.yml`'s N-1 jobs — the `citus_version:` (SQL /\nWorker / Coordinator N-1) and `citus_libdir:` (Lib / Coordinator N-1) pins, not from\nmaybe-stale comments at `multi_1_create_citus_schedule`. As of this writing: release-13.2 pins\n`citus_version: \"13.3-1\"` / `citus_libdir: v13.3.0` (its `default_version` is 13.4-1, so N-1 = 13.3-1);\nrelease-14.0 pins `14.1-1` / `v14.1.0` (default 14.2-1, N-1 = 14.1-1). The rule is simply\n`default_version` minus one minor, and it is bumped whenever the branch ships a new minor.\n\n- **Write the label with today's N-1 version.** For a GUC introduced in the branch's current\n  `default_version` (e.g. 13.4 on release-13.2), the N-1 binary is the previous minor (13.3-1) and the\n  GUC is absent there, so: `GUC introduced in 13.4; not present in the N-1 (13.3-1) Citus binary ...\n  Move back to multi_schedule at Citus <next major, 14>.` On release-14.0 the corresponding numbers\n  are 14.2 / N-1 14.1-1 / next major 15.\n- **Sanity gate:** the move is only warranted when the object is genuinely ABSENT in that N-1 version\n  (introduced in a minor strictly newer than N-1). If your object was introduced at or below N-1, the\n  N-1 binary already has it — no red, no move.\n\n- **Why a new UDF may NOT need this move** (asymmetry): a test already placed in\n  `multi_1_create_citus_schedule` (e.g. `citus_internal_distribute_object`) is ALREADY N-1-invisible,\n  so it never shows an N-1 red and needs no move. Grep which schedule your test is in first.\n- Distinguish this N-1 red from a genuine bug: a same-version job (no `lib-vX` / `_coordinatoronly` /\n  `_workeronly` / `_all` suffix on the artifact) naming your object IS yours — see the section above.\n- Shipping: fold the move into a not-yet-reviewed fix commit (amend + `--force-with-lease`), or add a\n  NEW appended commit if the tip has already been reviewed (plain push, no force).\n\n## Genuinely-new failures a backport WILL introduce on a release branch (and the fix)\nThe release branch supports an OLDER PG than `main` (e.g. release-13.2 still builds **PG15**), and it\nships **version-pinned `expected/*.out`** files. A backport that is green on `main` can therefore go\nred on the release branch for two reasons that ARE yours to fix (they are not baseline noise):\n\n### (a) A shared helper's SIGNATURE changed → version-pinned .out that prints the signature mismatches\nIf your backport chain brings a new/edited SQL helper whose *signature* changed (e.g.\n`explain_filter(text)` → `explain_filter(text, keep_numbers boolean DEFAULT false)`), any expected\n`.out` that prints that signature in a `CONTEXT:` / `PL/pgSQL function ... line N` / error line will\nmismatch. **`normalize.sed` normalizes only the line NUMBER (`line [0-9]+` → `line XX`), NOT the\nsignature text**, so the signature diff is real. Fix = regenerate/`sed`-update those signature\nstrings in the pinned `.out`. Find them all with `grep -rl \"helper_name(oldargs)\" expected/`.\n- **Do NOT mistake a required helper for \"over-reach\" and revert it.** If the backported *test*\n  calls the new form (`explain_filter($$...$$, true)`), the helper is REQUIRED; reverting it yields\n  `ERROR: function ...(unknown, boolean) does not exist`. Before deleting an \"extra\" hunk a\n  cherry-pick carried, check whether the backported test depends on it.\n\n### (b) PG-version planner divergence on the older PG the release branch still supports\nThe same EXPLAIN can differ by PG major. Real example: PG15 omits a functionally-redundant trailing\nsort column that PG16+ keeps — `Sort Key: q.batch` (PG15) vs `Sort Key: q.batch, q.text_id` (PG16+).\nrelease-13.2 hits it on the PG15 `check-multi`; main/release-14.0 never see it (no PG15).\nFix, in the order the repo prefers:\n1. **An explain helper from `main`'s `multi_test_helpers.sql`** — check whether one already\n   normalizes the divergent bit. As of 13.x/14.x none normalizes Sort Key (there are\n   `coordinator_plan`, `plan_normalize_memory`, `explain_with_pg16_subplan_format`,\n   `explain_with_pg17_initplan_format`, and boolean `explain_has_distributed_subplan` /\n   `explain_has_single_task` — the booleans would need an invasive test rewrite).\n2. **MOVE the diverging blocks into a PG-version-guarded test file (PREFERRED on release-13.2 when\n   the divergence is PG15-only).** release-13.2 ships `sql/pg16.sql`, which `\\q`-skips below PG16\n   (guard at top: `SELECT substring(:'server_version','\\d+')::int >= 16 ... \\if ... \\else \\q \\endif`)\n   AND already lives in `multi_1_create_citus_schedule` (the N-1-excluded schedule). So relocating the\n   PG15-diverging EXPLAIN cases out of the normal test (e.g. `allow_unsafe_insert_select_pushdown.sql`)\n   into a self-contained section appended to `pg16.sql` means: PG15 never runs them (no diff),\n   PG16/17/18 still cover them, and — bonus — they leave the N-1 matrix too. Recipe:\n   - Cut the exact diverging blocks from the source `.sql`; fix any dangling comment they leave behind.\n   - Append AFTER `pg16.sql`'s final `DROP SCHEMA pg16 CASCADE;` a self-contained section: its OWN\n     schema (e.g. `allow_unsafe_insert_select_pushdown_pg16`), a distinct `next_shard_id`\n     (e.g. 14100000) to avoid shard-id clashes, recreate the tables/functions the blocks need, the\n     moved EXPLAIN blocks, then `DROP SCHEMA ... CASCADE;`. It must not depend on the parent test's\n     schema (schedules run independently).\n   - Regenerate BOTH `.out`s (source shrinks, `pg16.out` grows) via the runner; never hand-edit.\n3. Add a new **`normalize.sed` rule** or update an existing one to generalize the coverage, or add an\n   alternative expected file as a last resort.\n\n### (c) Flaky detector goes all-red because your changed-file set is widely depended-on\n\"Test flakyness (1..32)\" computes a changed-test set by DEPENDENCY EXPANSION of the PR's changed\nfiles. Editing a widely-depended helper/migration (`multi_test_helpers.sql`, a migration `.sql`,\n`multi_extension.sql`) balloons the set across every shard and drags in inherently-flaky base tests\n(`start_stop_metadata_sync`, `multi_metadata_sync`, ...). The shards only settle once your\n*actually-changed* tests pass; the base-flaky tests pulled in are not yours to fix. Sanity check with\n`gh pr view <n> --json files` — a helper appearing in the flaky `tests=` list but NOT in the PR's\nfile list is pure dependency-expansion, not a change you made.\n\n**Do NOT hand-wave \"it's flaky\".** Two pieces of hard evidence, both cheap:\n1. **Read the actual diff** from the shard's uploaded artifact (they DO upload one, named\n   `test_flakyness_parallel_<N>`; see fetch recipe below). Confirm the failing test is a base test\n   you did not touch AND the diff contains ZERO of your backported symbols. The signature of\n   `--use-whole-schedule-line` running the whole schedule line in parallel so other tests consume\n   colocation IDs first. Scan: `grep -c <your_symbol> regression.diffs` (want 0),\n   `grep -oE \"expected/[a-z_0-9]+[.]out\"` (only the base test).\n2. **Reproduce on the PRISTINE base** to prove pre-existing, not introduced. From `src/test/regress`:\n   `python citus_tests/run_test.py <flaky_test> --repeat 8 --use-whole-schedule-line` on your branch\n   head, then `git checkout <merge-base> && make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install` and run the SAME command. If the\n   base fails the identical N/N under the identical flags, it's 100% pre-existing (checkout your\n   branch + rebuild to restore afterward). Example proven this way: `start_stop_metadata_sync` failed\n   8/8 on both `bp-release-13.2` tip and its pristine base, same colocationid/FK diff — not the\n   backport's fault. NOTE the flaky detector's PG major varies — check the shard log's `PG_MAJOR:`\n   line and reproduce on the SAME PG for an exact comparison.\n\n### Shipping the fix\nOne NEW fix commit on top of the reviewed tip (do not rewrite reviewed SHAs). If a prior *bad* fix\ncommit is already on the remote, REPLACE it with `git push --force-with-lease=<branch>:<badsha>`\n(the reviewed commits below it stay). Force-push freely only while the commits are still unreviewed.\n\n## Fetching CI diffs reliably (tooling gotchas)\n- `gh run view --job <id> --log-failed` streams often die with `stream error: stream ID 1; CANCEL` —\n  retry or avoid.\n- Jobs that upload artifacts (`NN_check-multi`, `NN_check-operations`, AND the flaky shards):\n  list with a SINGLE page, not `--paginate` (`--paginate` concatenates JSON pages so a single `--jq`\n  chokes with \"Additional text after finished reading JSON content\"):\n  `gh api \"repos/citusdata/citus/actions/runs/<run>/artifacts?per_page=100\" --jq '.artifacts[]|select(.name==\"15_check-multi\").id'`\n  then `gh api .../actions/artifacts/<id>/zip > x.zip` (this redirect is fine for the small per-artifact\n  zip; read `regression.diffs` from it).\n- Flaky-detector jobs DO upload a named artifact `test_flakyness_parallel_<N>` (contains the shard's\n  `regression.diffs` under `__w/citus/citus/src/test/regress/`) — fetch it like the per-artifact zip\n  above; that's the reliable way to read the flaky diff body. The per-job logs endpoint only NAMES the\n  failing test (`SCHEDULE_LINE:` / `not ok`) and points at regression.diffs in the artifact.\n- Do NOT re-encode a downloaded zip through a shell pipe that changes encoding (corrupts it); write it\n  to disk directly. The whole-run `/logs` zip often arrives partial (\"End of Central Directory record\n  could not be found\") — prefer the per-job / per-artifact endpoints above.\n\n## Mechanics / gotchas\n- Overall run status stays `queued`/`in_progress` until EVERY matrix job ends; per-job logs and\n  finished-job artifacts are still fetchable mid-run (above).\n- Dispatching several full matrix runs (~190 jobs each) saturates the fork runner pool and starves\n  the job you actually care about; `gh run cancel <id>` the fully-triaged runs to free capacity for\n  a critical re-verification.\n",".github/skills/README.md":"# Agent skills for citus\n\nThis directory holds **agent skills** — folders of instructions (and optionally\nscripts/resources) that an AI coding agent can load on demand when it works on a\nparticular kind of task in this repository. Skills let us capture hard-won,\nrepo-specific know-how once and have every agent (and human) reuse it, instead of\nre-deriving it from scratch each time.\n\n## Layout\n\nEach skill lives in its own subdirectory containing a `SKILL.md`:\n\n```\n.github/skills/\n├── README.md                     ← this file\n└── citus-backport/\n    ├── SKILL.md                  ← entry point (loaded first)\n    └── references/               ← deep-dive docs loaded on demand\n        ├── sql-schema-backport.md\n        ├── ci-triage.md\n        └── manual-upgrade-testing.md\n```\n\n- Subdirectory names are lowercase with hyphens.\n- The entry file **must** be named `SKILL.md`.\n- Keep `SKILL.md` a concise hub; push long procedures into `references/` files it\n  links to, so the agent only pulls in what it needs.\n\n## `SKILL.md` format\n\nA `SKILL.md` is Markdown with YAML frontmatter:\n\n```markdown\n---\nname: my-skill                 # required; lowercase-hyphenated, matches the folder\ndescription: >-                # required; WHAT it does + WHEN to use it (drives auto-selection)\n  One or two sentences describing the task this skill helps with and the\n  trigger conditions that should make an agent load it.\nlicense: See the repository LICENSE file.   # optional\n---\n\n# Human-readable title\n\nInstructions, examples, and guidelines for the agent to follow…\n```\n\nThe `description` is the most important field for discovery: agents match a user's\nrequest against it, so state both the capability and the trigger phrases plainly.\n\n## Where these are picked up\n\nProject skills under `.github/skills/` are discovered automatically by GitHub\nCopilot CLI. (The Agent Skills format is intentionally portable; other agent tools\nmay read `.claude/skills/` or `.agents/skills/` instead — see each tool's docs.)\n\n`.github/copilot-instructions.md` (auto-loaded repo-wide) also carries a short\n`task → skill` index so agents are pointed here even before they browse this\ndirectory. When you add a skill, add a row there too.\n\n## Adding a new skill\n\n1. Create `./<skill-name>/SKILL.md` with valid frontmatter.\n2. Put anything long or optional in `./<skill-name>/references/` and link to it\n   from `SKILL.md`.\n3. Keep it repo-specific and evidence-based; avoid machine- or user-specific paths\n   so it works for anyone who clones citus.\n4. Add a `task → skill` row to `.github/copilot-instructions.md` so agents discover it.\n",".github/skills/citus-backport/references/sql-schema-backport.md":"# SQL-schema backport recipe (the hard case)\n\nA C-only cherry-pick usually just applies. The moment a commit touches the SQL schema (new UDF,\nnew column, new migration step) you must **remap it to the target branch's real version**, because\n`main`'s migration files are named for the NEXT major (e.g. `15.0-1`) and the release branch is on\na different `default_version`. Do NOT accept `main`'s files as-is.\n\n## Step 0 — read the target version (drives everything)\nOn each target branch read `src/backend/distributed/citus.control`:\n\n```\ndefault_version = '<X.Y-Z>'\n```\n\nThen find that branch's current dev migration + downgrade under\n`src/backend/distributed/sql/`:\n- upgrade:   `citus--<prev>--<default_version>.sql`\n- downgrade: `citus--<default_version>--<prev>.sql`\n\nWorked example (versions as of this writing):\n\n| Branch        | default_version | dev upgrade migration        | dev downgrade                | udf versioned file |\n|---------------|-----------------|------------------------------|------------------------------|--------------------|\n| release-13.2  | `13.4-1`        | `citus--13.3-1--13.4-1.sql`  | `citus--13.4-1--13.3-1.sql`  | `13.4-1.sql`       |\n| release-14.0  | `14.2-1`        | `citus--14.1-1--14.2-1.sql`  | `citus--14.2-1--14.1-1.sql`  | `14.2-1.sql`       |\n| main          | `15.0-1`        | `citus--14.0-1--15.0-1.sql`  | `citus--15.0-1--14.0-1.sql`  | `15.0-1.sql`       |\n\n## Step 1 — remap the migration files\nThe main commit added its schema hunk to `citus--14.0-1--15.0-1.sql` (and the DROP to the matching\n`15.0-1--14.0-1` downgrade), plus a versioned udf file `sql/udfs/<name>/15.0-1.sql`. On the target\nbranch:\n\n1. `git rm` the main-only `15.0-1` upgrade + downgrade migration files if the cherry-pick created\n   them (they don't belong on the release branch).\n2. `git mv src/.../sql/udfs/<name>/15.0-1.sql  src/.../sql/udfs/<name>/<default_version>.sql`\n   (rename the versioned udf file to the branch's version).\n3. Add the `#include \"udfs/<name>/<default_version>.sql\"` line to the branch's **dev upgrade\n   migration** (`citus--<prev>--<default_version>.sql`) — same include the main commit added to\n   `14.0-1--15.0-1.sql`, just in the branch's file.\n4. Add the reverse `DROP FUNCTION IF EXISTS <fully-qualified-signature>;` (or DROP for whatever\n   object) to the branch's **dev downgrade** (`citus--<default_version>--<prev>.sql`).\n5. Keep `sql/udfs/<name>/latest.sql` — refresh it to identical content as the versioned file.\n\n`ci/check_migration_files.sh` (\"Check for missing downgrade scripts\") gates step 4 — every upgrade\nstep needs a matching downgrade. The CI `check-sql-snapshots` job validates these manual edits, so\ngreen there is strong confirmation the remap is correct.\n\n## Step 2 — fix `multi_extension.out`\n`multi_extension` walks the whole upgrade+downgrade ladder and prints a \"Snapshot of state at\n`<version>`\" section per version. The new object's row must land under the **target branch's**\nversion section (e.g. `13.4-1`), NOT main's `15.0-1`. The snapshot table's column widths are\ndynamic — **regenerate the .out with the runner, don't hand-align** it. From `src/test/regress`:\n\n```\npython citus_tests/run_test.py multi_extension\n```\n\nThe new line typically shows the new function/object with `(1 row)` (or the appropriate count) in\nthe target-version section.\n\n## Step 3 — fix `upgrade_list_citus_objects.out` (edit BY CONSTRUCTION)\nThis test **cannot be run standalone** via `run_test.py` — it depends on a `citus_schema` created\nby an earlier test in `after_pg_upgrade_with_columnar_schedule`; run alone it dies with\n`ERROR: no schema has been selected to create in`. So validate its edit by construction:\n- the new function line auto-merges alphabetically in the cherry-pick (leave it);\n- fix only the trailing **row COUNT** = base_count + (number of new citus objects). Read the base\n  count from the branch's own expected file. Observed: release-13.2 `378 → 379`, release-14.0\n  `384 → 385` for one new UDF.\n\n## Step 4 — build WITH downgrades, then test\n`make install` does NOT install downgrade scripts, and `multi_extension` exercises the downgrade\npath, so after every rebuild run both:\n\n```\nmake -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install\nmake -C src/backend/distributed install-downgrades\nmake -C src/backend/columnar    install-downgrades\n```\n\n(`columnar` is a SEPARATE extension with its own downgrades; omitting it surfaces as a\n`citus_columnar` \"no update path\" error. `make install-all` runs both.)\n\nThen run the feature's own test + `multi_extension`.\n\n## Step 5 — propagate the upgrade ladder ACROSS branches (MVU safety)\n\nBackporting the object DOWN to a release branch is not enough. An MVU replays the migration\n**ladder**, so a new SQL object must be reachable on EVERY higher branch's ladder too — otherwise\na cluster upgraded 13.x→14.y would silently lose it. Net effect: **the same object is defined\nMULTIPLE times per branch, once per major's introduction step.** This is intentional, not\nduplication to \"fix\".\n\n**Topology** (worked example: `citus_internal.distribute_object`, introduced on main at\n`14.0-1--15.0-1`, backported to 13.2 at `13.3-1--13.4-1` and to 14.0 at `14.1-1--14.2-1`):\n\n| Ladder step (+ its downgrade + versioned udf) | release-13.2 | release-14.0 | main |\n|---|---|---|---|\n| `13.3-1--13.4-1` (udf `13.4-1`) — 13.x introduction | ✅ (the backport itself) | ✅ copy here | ✅ copy here |\n| `14.1-1--14.2-1` (udf `14.2-1`) — 14.x introduction | n/a | ✅ (the backport itself) | ✅ copy here |\n| `14.0-1--15.0-1` (udf `15.0-1`) — main's own top    | n/a | n/a | ✅ already there (orig land) — LEAVE UNTOUCHED |\n\nRule of thumb: **a step introduced on major N must be copied onto every branch for major ≥ N.**\n13.x paths → release-14.0 + main; 14.x paths → main; main's own top path stays as merged.\n\n**How to do it (per higher branch, e.g. release-14.0 then main):**\n1. This is a **SEPARATE commit** on top of the (unchanged) cherry-pick — never amend the\n   cherry-pick. On main it's its own PR.\n2. Copy the step + downgrade + versioned-udf files **byte-for-byte** from the branch that owns\n   the introduction. `git show <branch>:<path>` reads any branch's blob without checking it out\n   (worktrees of one clone share an object store, so this is trivial):\n   ```\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/citus--13.3-1--13.4-1.sql          > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/downgrades/citus--13.4-1--13.3-1.sql > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/udfs/<name>/13.4-1.sql             > <same path>\n   ```\n   (and the analogous `14.1-1--14.2-1` trio from `bp-<pr>-release-14.0` when copying onto main.)\n3. **Keep the header verbatim** — even the `-- bump version to 13.4-1` line stays on downstream\n   branches where 13.4 is not the default. Confirmed identical across branches in the #8621 UDF family.\n4. **Purely additive.** Do NOT touch `citus.control` / `default_version`. Do NOT touch the\n   `13.2-1--14.0-1` major-bridge migration. main's `latest.sql` still points at its own top\n   version (15.0-1); leave it. `check_sql_snapshots.sh` verifies latest.sql == highest udf.\n5. New step files contain ONLY the object being propagated (header + single `#include`); the\n   downgrade is header + `DROP FUNCTION IF EXISTS <signature>;`. Don't drag along unrelated objects\n   that happen to share the source branch's step file.\n6. Build + `install-downgrades` + `multi_extension` on the higher branch (expect PASS, usually no\n   `.out` change needed since the new sections are additive). Run the three migration CI checks\n   locally before pushing: `ci/check_migration_files.sh`, `ci/check_sql_snapshots.sh`,\n   `ci/disallow_c_comments_in_migrations.sh` (all must exit 0).\n7. **Add `multi_extension.sql` ladder-test coverage for the new step** (or prove it's already\n   covered) — see the dedicated section below.\n\n**Template PR family** (study these to see the end-state, don't reverse-engineer from one PR):\nfind the original feature PR, then every PR that mentions it in title/desc. Worked example — the\n`citus_internal.distribute_object` repair UDF, introduced on main in **#8621**: backported\none-branch-per-target via **#8684**→release-13.2 and **#8685**→release-14.0 (each branch also carries\nthe sibling GUC PR **#8625** `citus.allow_unsafe_insert_select_pushdown`, unsquashed as its own\ncommit), with the 13.x/14.x upgrade paths propagated to `main` in **#8686**. The main ladder PR\n(#8686, \"Add 13.x/14.x upgrade paths for citus_internal.distribute_object to main\") is the model for\nthe main ladder commit's title/body; it's internal plumbing so usually **no `DESCRIPTION:` line**.\n\n## multi_extension.sql ladder tests for new steps (off-walk detour pattern)\n\nAdding a ladder step (Step 5 backport OR Step 5 propagation) is not fully covered until\n`src/test/regress/sql/multi_extension.sql` exercises it — or you can prove the existing walk already\ndoes. Model the additions on the template PR family, adapted per branch.\n\n- **The test walks ONE path only** — the main upgrade path (`13.2 → 13.3 → 14.0 → 14.1 → 15.0`).\n  A step you add for MVU may sit **off** that walk:\n  - the 13.x maintenance-tail step `13.3-1--13.4-1` (off-walk on release-14.0 and main), and\n  - the 14.x maintenance-tail step `14.1-1--14.2-1` (off-walk on **main**, whose walk jumps 14.1 → 15.0).\n  Off-walk steps' upgrade AND downgrade scripts are otherwise **never executed** by the test.\n- **On-walk / top-of-branch steps need NOTHING extra.** A branch's own `default_version` (13.4-1 on\n  release-13.2, 14.2-1 on release-14.0, 15.0-1 on main) is already the target of the existing\n  `-- Snapshot of state at X` block; the propagated object simply appears in that snapshot's `.out`.\n  ⇒ release-13.2 needed **no** `multi_extension.sql` change; release-14.0 needed **one** detour\n  (13.4-1); main needed **two** (13.4-1 and 14.2-1).\n- **PATTERN = round-trip no-op → snapshot at the detour → rejoin the walk FROM the detour.** Match\n  the shape of the existing on-walk blocks — visit the off-walk version the same way every other\n  version is visited, so the additions look like they belong. Do NOT add editorial parentheticals\n  (no \"(13.x maintenance line, off the ... path)\"). At the point the walk sits at the detour's\n  PARENT, insert two blocks AND retarget the *existing* next round-trip so it bounces off the\n  detour instead of the parent:\n  ```sql\n  -- Test downgrade to <parent> from <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';   -- e.g. 13.4-1\n  ALTER EXTENSION citus UPDATE TO '<parent>';   -- e.g. 13.3-1\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Snapshot of state at <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Test downgrade to <detour> from <next>   <-- was \"<parent> from <next>\"; retarget it\n  ALTER EXTENSION citus UPDATE TO '<next>';    -- the next ON-WALK version (e.g. 14.0-1 / 15.0-1)\n  ALTER EXTENSION citus UPDATE TO '<detour>';  -- bounce back to the detour, NOT the parent\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n  ```\n  Block 1 (round-trip) proves the up+down pair is a perfect no-op. Block 2 (snapshot) shows the new\n  object actually appearing at the detour version (`+ function citus_internal.distribute_object...`,\n  `(1 row)`). The snapshot leaves the walk sitting at `<detour>` with the `print_extension_changes()`\n  baseline `= {detour}`. Instead of adding a separate \"return to parent\" block, you **retarget the\n  existing next round-trip** (which used to bounce `<parent> ↔ <next>`) so it bounces `<detour> ↔\n  <next>` — this keeps the walk moving forward from the detour and lands it back at `<detour>` with\n  baseline reset to `{detour}`.\n- **The one intended downstream change:** because the walk now enters the following `-- Snapshot of\n  state at <next>` block with baseline `{detour}` (not `{parent}`), that ONE snapshot shifts by\n  exactly the detour's object, and nothing else changes:\n  - If `<next>` does NOT contain the object (e.g. `14.0-1`, a 13.x-line object): the snapshot GAINS a\n    `previous_object` row (the object shown as dropped when leaving the maintenance line).\n  - If `<next>` DOES contain the object (e.g. `15.0-1` when the object is already at `14.2-1`): the\n    snapshot LOSES the `current_object` row it used to show (it's no longer newly-added — it was\n    already displayed at the detour snapshot).\n  Both are self-consistent and desirable: the object is introduced exactly once, at its detour\n  snapshot. Every block AFTER that `<next>` snapshot stays byte-identical (print resets baseline to\n  `{next}` regardless). Regenerate `.out` and eyeball that only that single snapshot moved.\n- **`.out` regen:** edit `.sql` → runner \"fails\" (writes `results/multi_extension.out`) →\n  `cp results/multi_extension.out expected/multi_extension.out` → re-run → \"All 1 tests passed\".\n  **Never hand-edit `.out`.**\n- **`diff -w` header gotcha:** pg_regress compares with `diff -dU10 -w`, so snapshot-table header\n  column widths / leading spaces are IGNORED for pass/fail. A regen may normalize a pre-existing\n  HAND-EDITED narrow header (left over from an earlier `.out` hand-edit) to the runner's canonical\n  wider spacing. Whitespace-only and harmless — **keep the canonical runner output**, note it in the\n  commit message. Prove it's pre-existing: run the test on pristine HEAD (no change) — same wide\n  header, still PASSES.\n- **APPEND-ONLY once reviewed:** if the backport branch/PR has already been reviewed, do NOT\n  force-push — add the `multi_extension` test work as a new commit on top.\n\n## Flaky / gotcha notes for SQL backports\n- **`multi_extension` flaky maintenance-daemon crash:** the issue #3409 `DROP EXTENSION` under a\n  non-superuser role section can report \"server process was terminated\" once, then PASS on a clean\n  re-run. If the PG log shows only a clean fast-shutdown (no PANIC / segfault / Assert), it's\n  timing/flaky — re-run before investigating. Sanity-check by running the same test on the PRISTINE\n  base branch.\n- **`configure` gets regenerated by builds** — never part of a backport. `git checkout -- configure`\n  before committing.\n- **`git stash -u` mid-cherry-pick** drops `CHERRY_PICK_HEAD` but keeps the file changes; on restore\n  you must `git commit` manually and re-add the original message + `(cherry picked from commit <sha>)`.\n- **Regenerate .out with the runner, never hand-edit** — snapshot column widths and object counts\n  are computed.\n",".github/skills/citus-backport/SKILL.md":"---\nname: citus-backport\ndescription: >-\n  Backport one or more merged citusdata/citus `main` commits/PRs onto the active release\n  branches. Covers the release model, cherry-picking, remapping SQL schema changes to each\n  branch's default_version, the SQL upgrade/downgrade ladder for Major-Version-Upgrade safety,\n  running regression tests, one clean commit per PR per branch, and triaging release-branch CI\n  (pre-existing baseline reds vs reds the backport introduced). USE WHEN asked to backport / port\n  / cherry-pick a change to release-13.2 / release-14.0 (or the newest two majors), when a\n  backport hits a SQL migration / udf / multi_extension conflict, when a \"trivial C-only\" backport\n  fails to compile on an older PG the release branch still supports (e.g. PG15 `rteperminfos`),\n  when release-branch CI is red and you must prove which reds are pre-existing, or when you must\n  prove a SQL-schema backport is Major-Version-Upgrade-safe via `ALTER EXTENSION citus UPDATE`\n  across a real multi-node cluster. DO NOT USE for non-Citus repos, new features, or PR review.\nlicense: See the repository LICENSE file.\n---\n\n# Backport a Citus change to the release branches\n\nThis skill encodes how to backport merged `main` commit(s) to Citus release branches: the release\nmodel, the SQL-schema remap recipe, the PG-version compatibility traps, and how to triage the\nrelease branches' baseline-red CI. Follow the repository's normal build/test/style conventions\n(see `CONTRIBUTING.md` and `src/test/regress/README.md`) on top of the guidance here.\n\n## Reference deep-dives (load on demand)\nAll live next to this file under `references/`:\n- **`references/sql-schema-backport.md`** — the hard case: a commit that changes the SQL schema\n  (new UDF / column / migration). Exact file-by-file remap recipe, the `multi_extension.out` and\n  `upgrade_list_citus_objects.out` edits, the `install-downgrades` trap, and the cross-branch\n  **ladder-propagation** step (copy the introduction paths UP to every higher major for MVU safety).\n- **`references/ci-triage.md`** — reading backport CI: the PG-version compile traps (main drops\n  old PGs the release branch keeps), the N-1 mixed-version jobs, and the step-by-step proof that a\n  red job is pre-existing baseline noise vs a real regression you introduced.\n- **`references/manual-upgrade-testing.md`** — the manual MVU proof: stand up a multi-node cluster\n  at an OLD Citus major, then `ALTER EXTENSION citus UPDATE` across the backport branches (13.x\n  tail → cross-major → 14.x tail), capturing `citus_version()` + `pg_extension.extversion` per node\n  per hop. The `make install-all` / single-shared-PG requirement.\n\nA C-only backport usually needs only this hub + `ci-triage.md`. A SQL-schema backport needs all\nfour (add `manual-upgrade-testing.md` as the final proof once CI is green).\n\n## Release model (internalize this first)\n- **ONE branch per MAJOR.** `release-13.2` serves the WHOLE 13.x line; `release-14.0` serves ALL of\n  14.x; `main` is the next major. Citus no longer cuts a branch per minor (this changed ~2026-05).\n- **Backport FEATURES, not just bugfixes**, to the **newest two majors** (13 & 14 as of this\n  writing).\n- **Branch names LIE about the version.** `release-13.2` currently ships 13.4; `release-14.0` ships\n  14.2. **NEVER infer the target minor from the branch name** — read `default_version` from\n  `src/backend/distributed/citus.control` on each target branch. That value drives every SQL edit.\n- **N-1 compatibility is required and CI-gated.** A backport is N-1-safe when it is purely\n  ADDITIVE (new UDF / GUC / column) and does not change an existing UDF signature or wire format\n  used cross-version. If it's additive, it's safe. See `references/ci-triage.md` for the jobs.\n\n## Workflow (per PR, per target branch)\n1. **Use an isolated build per target major.** A dedicated checkout/worktree + its own PostgreSQL\n   install per release branch keeps builds/tests from clobbering each other and your main checkout.\n   Two worktrees of the same clone SHARE one git object store — all branches are visible/pushable\n   from either, and a `+`-prefixed branch in `git branch` is checked out in a sibling worktree.\n2. **Make the SHAs reachable**, then `git checkout -b bp-<pr>-<release-branch> origin/<release-branch>`\n   and `git cherry-pick -x <sha>` (the `-x` records `(cherry picked from commit <sha>)`).\n3. **Resolve:**\n   - **C-only:** usually applies clean or with trivial context conflicts. THEN check PG-version\n     compat (`references/ci-triage.md` §PG-version) — the #1 non-obvious C-backport failure.\n   - **SQL-schema:** migration / udf / `multi_extension.out` conflict → remap by hand per\n     `references/sql-schema-backport.md`. Do NOT just accept `main`'s next-major (e.g. `15.0-1`) files.\n4. **Build in the isolated env:** `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install`. For any SQL change ALSO install\n   downgrades (see `references/sql-schema-backport.md` — plain `make install` skips them and\n   `multi_extension` will fail with cascading \"no update path\" errors otherwise; use\n   `make install-all`).\n5. **Test only what's relevant** (CI runs the whole suite): the feature's own test + `multi_extension`\n   for SQL changes. From `src/test/regress`: `python citus_tests/run_test.py <test>` (it manages its\n   own cluster). Regenerate any `expected/*.out` with the runner — **never hand-edit .out** (column\n   widths / row counts are dynamic).\n6. **Collapse to ONE commit per PR per branch.** Keep the original message; keep a `DESCRIPTION:`\n   first line iff the change is user-facing (this feeds the changelog). `git checkout -- configure`\n   if a build regenerated it (configure is never part of a backport). **Multiple PRs to the SAME\n   target → ONE branch per target, not per PR** (unsquashed: one commit per PR, in main's\n   chronological merge order; SQL-ladder commits sit right after their PR's cherry-pick). Name it\n   `bp-release-<major>` (e.g. `bp-release-13.2`). Build cleanly from the SQL-heavy PR's branch, then\n   cherry-pick the C-only PRs on top. **Cherry-pick an already-adapted backport commit WITHOUT\n   `-x`** — its message already carries the single `(cherry picked from <orig>)` trailer; `-x` would\n   double it.\n7. **Push to your fork.** `bp-*` branches don't match CI's push trigger (`main`/`release-*` only),\n   so kick CI with `gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`. Open\n   PRs only when the change owner asks.\n8. **Wait for CI and triage** per `references/ci-triage.md`. Separate baseline-red noise (N-1 /\n   flaky / coverage) from **genuinely-new failures you introduced**. The two you WILL hit on a\n   release branch: (a) a version-pinned `.out` that prints a shared helper's changed *signature*, and\n   (b) a PG-version planner divergence on the OLDER PG the branch keeps but `main` dropped (PREFER a\n   PG-guarded file like `pg16.sql` over a `<test>_N.out` alt file). A third you must NOT chase green\n   by mangling the test: a new GUC/UDF whose test runs under N-1 shows a NEW red in the (non-blocking)\n   **N-1 mixed-version** jobs — the old lib lacks it and falls back, which IS the N-1 contract. The\n   clean fix is to move that test line into the `multi_1_create_citus_schedule` placeholder section\n   (schedule mechanics + N-1 label rule in the hard-rules below and `ci-triage.md §\"NEW N-1 red\"`).\n   **Fix-commit rule (all fixes):** pre-review → amend + `git push --force-with-lease`; post-review →\n   add ONE new fix commit, never rewrite reviewed SHAs (replace a bad remote fix commit with\n   `--force-with-lease=<branch>:<badsha>`). Report evidence; don't declare done on red.\n9. **Propagate the ladder UP (SQL-schema backports only).** A new SQL object must also be reachable\n   on every HIGHER major's upgrade ladder, or a future 13.x→14.y Major Version Upgrade loses it.\n   As SEPARATE commits (its own PR on main), copy the introduction step/downgrade/udf files\n   byte-for-byte onto release-14.0 (for a 13.x path) and main (for 13.x and 14.x paths, given today's\n   two most recent major versions); main's own top path stays untouched. Full topology + recipe in\n   `references/sql-schema-backport.md` §Step 5.\n10. **Add `multi_extension.sql` ladder-test coverage** for every new step. The test only walks the\n    main upgrade path, so a step OFF that path (a 13.x/14.x maintenance-tail detour) is never\n    exercised unless you add a detour block matching the on-walk shape (and the file's existing\n    comment style — no editorial parentheticals): a round-trip no-op + a snapshot at the detour,\n    then retarget the *existing* next round-trip to bounce `<detour> ↔ <next>` rather than adding a\n    separate \"return\" block. That shifts exactly one downstream snapshot; everything after stays\n    byte-identical. A branch's own top/default_version is already covered by its snapshot block.\n    Never hand-edit `.out`; prefer append-only on reviewed branches. Full pattern + the `diff -w`\n    header gotcha in `references/sql-schema-backport.md`.\n11. **Prove MVU safety on a real cluster (SQL-schema backports, after CI is green).** The ladder\n    propagation only *claims* a future 13→14 Major Version Upgrade keeps the object; prove it. Stand\n    up a multi-node cluster at an OLD major, then `ALTER EXTENSION citus UPDATE` on EVERY node across\n    the backport branches (13.x tail → cross-major → 14.x tail → cross-major → main tail), capturing\n    `citus_version()` and `pg_extension.extversion` per node per hop, plus a functional check that the\n    backported object survived. Build all Citus versions into ONE shared PG with **`make install-all`**\n    (plain `install` omits the downgrade + bridge scripts the cross-major route needs → \"no update path\").\n    Full recipe and the single-PG requirement in `references/manual-upgrade-testing.md`.\n\n## Hard rules / traps (full detail in the references)\n- **Do NOT run `make reindent` in a build environment whose `citus_indent` version differs from the\n  branch's** — a version-skewed formatter reformats 100+ unrelated files and errors out. Trust\n  cherry-picked formatting; CI `check-style` verifies it. If you ran it by accident:\n  `git reset --hard HEAD`.\n- **`make install` does NOT install downgrade scripts.** For any SQL change run BOTH\n  `make -C src/backend/distributed install-downgrades` and `make -C src/backend/columnar\n  install-downgrades` (columnar is a separate extension). `make install-all` does both.\n- **A single build environment builds ONE PG only** — it cannot catch a compile break on another PG\n  the release branch supports. Rely on CI's per-PG `Build for PGNN` jobs; scan cherry-picked C for\n  struct fields / APIs newer than the branch's OLDEST PG and add `#if PG_VERSION_NUM >= PG_VERSION_NN`.\n- **The release branches are BASELINE-RED.** Several N-1 / flaky jobs fail on the pristine base\n  branch with no backport. Subtract the baseline before blaming your change — prove it with the\n  upstream `citusdata/citus` base-branch run. Details + exact commands in `references/ci-triage.md`.\n- **A new SQL object is defined MULTIPLE times on purpose.** After ladder propagation the same UDF\n  appears in the 13.x, 14.x AND main introduction steps (triple-defined on main). That is\n  intentional MVU/N-1 safety, NOT duplication to collapse. Never \"dedupe\" it. Details:\n  `references/sql-schema-backport.md` §Step 5.\n- **Show evidence.** Present the proof (build-job outcomes, `regression.diffs` excerpts, the\n  upstream-baseline comparison), not a bare \"CI is green/red\".\n- **A backport onto an older-PG release branch surfaces failures `main` never saw** (release-13.2\n  still builds PG15; the branch ships version-pinned `expected/*.out`). Two patterns: (1) a shared\n  helper whose SIGNATURE you changed prints in a pinned `.out` (`normalize.sed` masks the line\n  NUMBER, not the signature) — do NOT \"fix\" it by reverting the helper if the backported TEST needs\n  it (it's required, not over-reach); (2) a PG15-vs-PG16+ EXPLAIN/planner diff → PREFER a PG-guarded\n  file (`pg16.sql` `\\q`-skips <PG16 and already lives in the N-1-excluded schedule) over a\n  `<test>_0.out` alt. Full recipe: `references/ci-triage.md` §Genuinely-new failures.\n- **A NEW GUC/UDF whose test runs under N-1 has a CLEAN fix, not \"leave it\".** Move the test line\n  from its schedule into the `multi_1_create_citus_schedule` placeholder section (N-1 make_targets\n  omit `check-multi-1-create-citus`). N-1 version label = the branch's CURRENT N-1 = the previous\n  minor (13.3-1 on release-13.2, 14.1-1 on release-14.0), read from the live `citus_version:` /\n  `citus_libdir:` pin in build_and_test.yml — not the (possibly-stale) number in the neighbor comment.\n  Details: `references/ci-triage.md` §\"NEW N-1 red\".\n","src/backend/distributed/commands/README.md":"# Commands\n\nThe commands module is modeled after `backend/commands` from the postgres repository and\ncontains the logic for Citus on how to run these commands on distributed objects. Even\nthough the structure of the directory has some resemblence to its postgres relative, files\nhere are somewhat more fine-grained. This is due to the nature of citus commands that are\nheavily focused on distributed tables. Instead of having all commands in `tablecmds.c`\nthey are often moved to files that are named after the command.\n\n| File                         | Description |\n|------------------------------|-------------|\n| `create_distributed_table.c` | Implementation of UDF's for creating distributed tables |\n| `drop_distributed_table.c`   | Implementation for dropping metadata for partitions of distributed tables |\n| `extension.c`                | Implementation of `CREATE EXTENSION` commands for citus specific checks |\n| `foreign_constraint.c`       | Implementation of and helper functions for foreign key constraints |\n| `grant.c`                    | Implementation of `GRANT` commands for roles/users on relations |\n| `index.c`                    | Implementation of commands specific to indices on distributed tables |\n| `multi_copy.c`               | Implementation of `COPY` command. There are multiple different copy modes which are described in detail below |\n| `policy.c`                   | Implementation of `CREATE\\ALTER POLICY` commands. |\n| `rename.c`                   | Implementation of `ALTER ... RENAME ...` commands. It implements the renaming of applicable objects, otherwise provides the user with a warning |\n| `schema.c`                   | |\n| `sequence.c`                 | Implementation of `CREATE/ALTER SEQUENCE` commands. Primarily checks correctness of sequence statements as they are not propagated to the worker nodes |\n| `table.c`                    | |\n| `transmit.c`                 | Implementation of `COPY` commands with `format transmit` set in the options. This format is used to transfer files from one node to another node |\n| `truncate.c`                 | Implementation of `TRUNCATE` commands on distributed tables |\n| `utility_hook.c`             | This is the entry point from postgres into the commands module of citus. It contains the implementation that gets registered in postgres' `ProcessUtility_hook` callback to extends the functionality of the original ProcessUtility. This code is used to route the incoming commands to their respective implementation in Citus |\n| `vacuum.c`                   | Implementation of `VACUUM` commands on distributed tables |\n\n# COPY\n\nThe copy command is overloaded for a couple of use-cases specific to citus. The syntax of\nthe command stays the same, however the implementation might slightly differ from the\nstock implementation. The overloading is mostly done via extra options that Citus uses to\nindicate how to operate the copy. The options used are described below.\n\n## FORMAT transmit\n\nImplemented in `transmit.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n\n## FORMAT result\n\nImplemented in `multi_copy.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n"},"files":{".github/copilot-instructions.md":"# Copilot instructions for citus\n\nFollow the repository's normal contribution conventions: see `CONTRIBUTING.md` for\nbuild/style/PR rules and `src/test/regress/README.md` for the regression-test workflow.\n\n## Agent skills\n\nThis repo ships task-specific **agent skills** under `.github/skills/`. Before starting\na task that matches one, load that skill's `SKILL.md` and follow it so you don't re-derive\nit. Each skill's `SKILL.md` frontmatter `description` states exactly when to use it; the\nindex below is a quick map.\n\n| Task | Skill |\n|------|-------|\n| Backport a merged `main` commit/PR to a release branch (`release-13.2` / `release-14.0` / newest two majors) — including SQL-schema migrations, the upgrade/downgrade ladder, N-1 / Major-Version-Upgrade safety, and triaging release-branch CI | [`citus-backport`](skills/citus-backport/SKILL.md) |\n| Merge a batch of Citus PRs one at a time: sync each with its base branch, fix a red `check-style` job, retry other red checks with give-up rules, then squash-merge with the PR description as the commit message | [`citus-merge-loop`](skills/citus-merge-loop/SKILL.md) |\n| Fix a failing `check-style` job on whatever branch is already checked out, by rebuilding the exact `uncrustify`/`citus_indent` versions that branch itself pins (in its own `STYLEGUIDE.md` and `.github/workflows/build_and_test.yml`), then running `make reindent` and committing the style-only diff locally (no push, and no switching to or inspecting any other Citus branch) | [`citus-check-style-reindent`](skills/citus-check-style-reindent/SKILL.md) |\n\nSee `.github/skills/README.md` for the skills layout and how to add a new one.\n",".github/skills/citus-backport/references/manual-upgrade-testing.md":"# Manual MVU (major-version-upgrade) testing for backport branches\n\nBackporting a SQL-schema change and propagating the upgrade ladder (see\n`sql-schema-backport.md` §Step 5) only *claims* MVU safety. This doc is how you **prove** it\nend-to-end on a real multi-node cluster: stand up N nodes at an OLD Citus major, then walk the\nextension forward across both backport branches — and, one hop further onto the\n`main`-targeting PR branch — with `ALTER EXTENSION citus UPDATE`, capturing `citus_version()` AND\n`pg_extension.extversion` on every node after every hop.\n\nThis is the \"manual testing for backport branches\" step. Do it after CI is green, before\ndeclaring a SQL-schema backport family done. A C-only backport does NOT need it.\n\n## What this proves\n- A cluster shipped at the 13.x tail (`default_version` of `release-13.2`, e.g. `13.4-1`) can\n  still cross to 14.x — i.e. a future managed-service 13→14 MVU won't strand the customer.\n- **Final hop:** a 14.x-tail cluster (e.g. `14.2-1`) can cross to the NEXT major on\n  `main` (`15.0-1`) via the PR branch you opened against `main` — proving the SAME object also\n  rides a future 14.x→15.0 MVU. `main` jumps `14.0-1 → 15.0-1` directly (it has NO 14.1/14.2\n  *upgrade-in-line*), but it MUST carry the `14.2-1--14.1-1` and `14.1-1--14.0-1` **downgrades**\n  so PG can route `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. If that route is missing you get \"no\n  update path from 14.2-1 to 15.0-1\" — a real MVU gap in the ladder propagation to `main`.\n- The forward/downgrade **ladder** you propagated actually routes end to end.\n- The backported object (UDF/GUC/column) is present and the cluster is functional after each jump.\n\n## The single hard requirement: ONE PG, three Citus builds\n`ALTER EXTENSION citus UPDATE` walks the version graph made of the `citus--A--B.sql` scripts\n**installed in the current PG's `SHAREDIR/extension/`**. So all three Citus versions in the walk\n(old baseline + both backport branches) must be installed, one at a time, into the SAME PG\ninstall / SAME cluster data dirs. You swap the installed Citus between hops and restart the nodes;\nthe catalog stays put and `ALTER EXTENSION` migrates it.\n\nUse ONE dedicated PG install as that single PG (its Citus gets overwritten repeatedly — that's the\npoint — then you restore it at the end, §Cleanup). Do the builds in **detached worktrees** so your\nreviewed branch checkouts are never disturbed.\n\n### `make install-all`, NOT `make install` — MANDATORY here\n`make install` lays down only **forward** scripts. The cross-major route from the 13.x tail is\n**not** a straight climb — there is no `13.4-1--14.x` edge. It goes:\n\n```\n13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1\n  (down)   (down)   (BRIDGE up)  (up)    (up)\n```\n\nso it needs the **downgrade** scripts `13.4-1--13.3-1`, `13.3-1--13.2-1` and the cross-major\n**bridge** `13.2-1--14.0-1` (all shipped under `sql/downgrades/` on `release-14.0`). `make install`\nomits every downgrade → `ALTER EXTENSION citus UPDATE` fails with \"no update path from 13.4-1 to\n14.2-1\". Always build the test PG with **`make install-all`** (it runs `install` +\n`install-downgrades` for both the distributed and columnar extensions).\n\n### PG version must be common to all builds\nBaseline Citus must support the chosen PG. 13.0/13.2 support PG15/16/17; 14.0 supports 16/17/18;\n`main` supports 16/17/18 → **PG 17** is the common denominator for all four. Pick a PG 17 install\nso it works for the whole 13.0→13.4→14.2→15.0 walk.\n\n## Recipe (verified with a 3-node cluster: citus 13.0.5 → 13.4.0 → 14.2.0 → 15.0devel)\n\nPick a shared PG (`SHARED=<path to your PG17 install>`), a spare coordinator port (e.g. `10600`;\nworkers take `+1/+2`), and the three source trees. Use detached worktrees so the reviewed\nbranch worktrees are never disturbed:\n\n```bash\ncd <main clone>            # holds all the branches/tags\ngit worktree add --detach /tmp/mvu/wt130 v13.0.5            # OLD baseline (a real tag)\ngit worktree add --detach /tmp/mvu/wt132 <bp-release-13.2-head-sha>\ngit worktree add --detach /tmp/mvu/wt140 <bp-release-14.0-head-sha>\ngit worktree add --detach /tmp/mvu/wtmain <bp-main-PR-head-sha>\n```\n\nBuild helper — reconfigure each tree against the SHARED PG and `install-all`:\n\n```bash\nbuild() {  # build <worktree> ; installs into $SHARED\n  cd \"$1\"\n  ./configure PG_CONFIG=\"$SHARED/bin/pg_config\" >/tmp/cfg.log 2>&1\n  local JOBS; JOBS=$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)   # portable: Linux || macOS\n  make -sj\"$JOBS\"             >/tmp/mk.log  2>&1\n  make -sj\"$JOBS\" install-all >/tmp/ins.log 2>&1\n  ls \"$SHARED/share/extension/\" | grep -E '^citus--1[3-5]' | sort   # sanity: see the ladder\n}\n```\nRe-use the same cluster data dirs across hops:\n\n1. **Baseline 13.0.** `build /tmp/mvu/wt130`. Stand up a cluster on `$SHARED` with `citus_dev`\n   (it `CREATE EXTENSION citus` at `default_version` = the currently-installed `13.0-1`). Verify\n   `SELECT extversion FROM pg_extension WHERE extname='citus'` and `SELECT citus_version()` = 13.0\n   on coordinator + both workers. (`citus_dev`'s default DB is `$USER`, NOT `postgres`.)\n2. **Up to the 13.2 branch.** `build /tmp/mvu/wt132` (overwrites Citus in `$SHARED`, now\n   `default_version = 13.4-1`, installs the 13.x up+down ladder). **Restart every node** so the new\n   `.so` loads. Then, on EACH node: `ALTER EXTENSION citus UPDATE;` → catalog goes `13.0-1 → 13.4-1`\n   in one call (walks the whole chain). Capture `citus_version()` per node — expect\n   `13.4.0 ... (sha: <13.2-head>)`.\n3. **Cross-major to the 14.0 branch.** `build /tmp/mvu/wt140` (`default_version = 14.2-1`, installs\n   the full up+down+bridge ladder). **Restart every node.** On EACH node: `ALTER EXTENSION citus\n   UPDATE;` → routes `13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1`. Capture\n   `citus_version()` per node — expect `14.2.0 ... (sha: <14.0-head>)`.\n4. **Functional check** on the upgraded cluster (coordinator): active `pg_dist_node`,\n   `create_distributed_table` + INSERT + distributed `count(*)`, `pg_dist_shard` count, and that\n   the backported object is present (e.g. `SELECT proname FROM pg_proc WHERE proname =\n   'distribute_object' AND pronamespace = 'citus_internal'::regnamespace;`).\n5. **One more hop onto the `main` PR branch.** Detached worktree at the `bp-<pr>-main`\n   head, `build` it into the SAME `$SHARED` (overwrites Citus to `default_version = 15.0-1`,\n   installs main's up+down ladder incl. the 14.x downgrades). **Restart every node.** On EACH node:\n   `ALTER EXTENSION citus UPDATE;` → routes `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. Re-run the\n   functional check. Expect `Citus 15.0devel ... (sha: <main-PR-head>)` and the backported object\n   still present at `15.0-1`.\n\n**KEY NUANCE — `citus_version()` reports the BINARY, `extversion` reports the CATALOG.** After you\ninstall a newer Citus + restart (but BEFORE `ALTER EXTENSION citus UPDATE`), `citus_version()`\nalready prints the new version (it is compiled into the `.so`), while `pg_extension.extversion` is\nstill the old catalog version. The migration is only proven by **`extversion` moving to the new\n`default_version`**. On the first hops the jump is a whole major so `citus_version()` alone is\ntelling; on the `14.2 → 15.0devel` hop both 14.2 and 15.0 builds may print `15.0devel`-ish strings,\nso capture `extversion` before+after (`14.2-1 → 15.0-1`) as the real evidence.\n\nRun the same `ALTER EXTENSION citus UPDATE` on each node by connecting to each port directly\n(coordinator does NOT auto-propagate the extension version to workers) — that's the \"update all\nnodes\" the ask means.\n\n## Environment notes (neutral, but they save time)\n- **Put SQL in `.sql` files and run `psql -f`; put shell logic in `.sh` files and run them.**\n  Piping SQL / loops inline through several shell layers (esp. cross-OS terminals) mangles quoting\n  and `$USER`. If a script is authored on a different-newline OS, strip CRLF first (`sed -i\n  's/\\r$//' file`) or `set -o pipefail` may die with \"invalid option name\".\n- **`citus_dev` records RELATIVE data-dir paths** (`cluster/<role>`). `pg_ctl -D <abs>` then fails\n  with \"could not access directory …/cluster/<role>\". Fix: `cd` into the clusters parent and use\n  `pg_ctl -D cluster/<role> …`.\n- **If `pg_ctl -w start` blocks**, the `-w` readiness probe can hang against an `ssl=on` citus_dev\n  config even though the server logged \"ready to accept connections\". Start WITHOUT `-w` and poll\n  the TCP port yourself (`ss -ltn | grep :<port>`). `-w` on **stop** is fine.\n- On slow disks the first post-restart checkpoint can take ~100s; judge node readiness by\n  \"database system is ready to accept connections\" in the node logfile + the port listening, not by\n  a `pg_ctl -w` wrapper returning.\n\n## Cleanup / restore\n- The shared PG's Citus was left at whatever you installed last (after the final main hop that\n  is main's `15.0-1`). **Restore the env's own build:** rebuild the PG's normal Citus branch with\n  `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install-all` (verify `default_version` + the ladder edges are back).\n- Stop the test cluster (`pg_ctl -D cluster/<role> -w -m fast stop` per node, or your cluster\n  helper's stop command) and `git worktree remove` the throwaway build trees (+ `git worktree\n  prune`) when done.\n\n## Expected evidence shape (what a PASS looks like)\n```\nBASELINE (citus 13.0):    all nodes → extversion 13.0-1  | Citus 13.0.5 ... gitref: v13.0.5\nafter 13.2-branch UPDATE: all nodes → extversion 13.4-1  | Citus 13.4.0 ... (sha: <13.2 head>)\nafter 14.0-branch UPDATE: all nodes → extversion 14.2-1  | Citus 14.2.0 ... (sha: <14.0 head>)\nafter main-PR   UPDATE:   all nodes → extversion 14.2-1→15.0-1 | Citus 15.0devel ... (sha: <main head>)\nfunctional (each hop): 3 active nodes, create_distributed_table OK, count matches, backported obj present\n```\nReport the per-node `extversion` (before+after) AND `citus_version()` for every hop — not a\nsummary — since the whole point is proving every node crossed each boundary.\n",".github/skills/citus-backport/references/ci-triage.md":"# CI triage for backports — PG-version compat + the baseline-red release branches\n\nTwo independent things make backport CI confusing: (1) `main` might support different PG majors\nthan a release branch, so a \"trivial C-only\" cherry-pick can fail to COMPILE on an old PG; and\n(2) the release branches are BASELINE-RED — several jobs fail on the pristine base branch with\nno backport. Handle both before declaring a red \"your fault\".\n\n## PG-version compat (the #1 non-obvious C-backport failure)\n`main` may support different PG majors than the release branches, so code `main` wrote unconditionally\nagainst a newer PG API breaks the older-PG build on a release branch. Check each branch's supported\nPG set in `.github/workflows/build_and_test.yml` (the `pgNN_version` params block). Today, these are the\nsupported PG majors by main and two most recent release branches:\n\n| Branch        | PG majors built      |\n|---------------|----------------------|\n| release-13.2  | **15**, 16, 17       |\n| release-14.0  | 16, 17, 18 (no 15)   |\n| main          | 16, 17, 18 (dropped 15) |\n\n**Concrete example (confirmed):** `Query.rteperminfos` was added in PG16. `main` sets\n`q->rteperminfos = NIL;` bare. On release-13.2 that is `error: 'Query' has no member named\n'rteperminfos'` on the **PG15** build. Fix = the citus idiom the branch ALREADY uses everywhere\nelse for that field:\n\n```c\n#if PG_VERSION_NUM >= PG_VERSION_16\n    q->rteperminfos = NIL;\n#endif\n```\n\nGrep the target branch for the same symbol to copy its exact guard macro. **General rule:** after a\nC cherry-pick, scan the added `+` lines for struct fields / API calls introduced in a PG major that\nthe branch's OLDEST PG predates, and guard them. **Backport to the oldest-PG branch first** to\nsurface these early. A single build environment builds ONE PG so it can't catch this locally — rely\non CI's per-PG `Build for PGNN` jobs.\n\n### Reading a Build-for-PGNN failure mid-run\n`gh run view --log` / `--log-failed` say \"still in progress\" until the WHOLE run finishes, but a\nsingle finished job's log is fetchable immediately:\n\n```\ngh api repos/<your-fork>/citus/actions/jobs/<jobId>/logs > build.log\n```\n\nThen grep for `error:` / `make.*Error`. A `Build for PGxx` failure with a real `error:` line = a\ngenuine bug (fix it). An \"exit code N\" annotation with no compiler `error:` in the log = usually\ninfra/flaky (rerun). After fixing, **amend the single cherry-pick commit** (keep 1 commit/PR),\n`git push --force-with-lease`, and re-dispatch:\n`gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`.\n\n## The release branches are BASELINE-RED\n`release-13.2` and `release-14.0` have KNOWN-FAILING jobs on their own pristine upstream CI. Do NOT\npanic when your backport branch is red — **subtract the baseline first.** As of this writing the\nbase branches fail these SAME jobs with no backport applied:\n\n- **N-1 mixed-version jobs where the OLD lib runs on the COORDINATOR:**\n  - Test Citus **Lib N-1** — check-multi, check-multi-1, check-add-backup-node\n  - Test Citus **Coordinator N-1** — check-multi, check-multi-1, check-add-backup-node\n  - (The SQL N-1 and Worker N-1 variants of the same tests PASS — old lib is only on workers there.)\n- Occasional **Test flakyness (N)** shards and a stray check-enterprise / check-operations.\n- **Test Citus Failure / check-query-generator** — a randomized differential fuzzer (compares\n  distributed vs local results of random nested-CTE/join queries). Flaky by nature; a random query\n  can trip `citus.max_intermediate_result_size` etc. **Tell:** one PG major fails while another PG\n  major of the SAME commit passes. Not a backport regression.\n\n**Root cause of the N-1 reds (not yours):** these base branches already carry in-flight\nminor-targeted features (e.g. clone-promotion sequence *re-ranging*;\n`citus_internal.delete_placement_metadata` behavior) whose NEW behavior lives only in the NEW lib.\nWhen the OLD lib (the previous minor's binary) is loaded on the coordinator, the expected `.out`\n(written for the new lib) mismatches → diff. Inherent to N-1 testing of in-flight features;\nunrelated to a feature backport that doesn't touch those areas.\n\n## How to PROVE a red is pre-existing (do this — gather the evidence)\n1. **Upstream base-branch run is itself failing:**\n   ```\n   gh run list --repo citusdata/citus --branch release-13.2 --workflow \"Build & Test\" --limit 6\n   ```\n   Note the base-branch runs are themselves `failure`.\n2. **Your failing set ⊆ upstream's failing set:**\n   ```\n   gh run view <upstreamRunId> --repo citusdata/citus --json jobs \\\n     --jq '.jobs[]|select(.conclusion==\"failure\")|.name'\n   ```\n   If your branch's failing job names are a subset, they're baseline, not yours.\n3. **The diff cites a pre-existing feature, not your object:** download your failed job's artifact\n   and read `regression.diffs`:\n   ```\n   gh run download <runId> --repo <your-fork>/citus -n \"<artifactName>\" -D <dir>\n   ```\n   Artifact naming: `<pgmajor>_<test>_[<citus_version>_]<libLabel>_<mode>`\n   (e.g. `18_check-enterprise_14.1-1_all` = SQL N-1; `..._v14.1.0_coordinatoronly` = Coordinator\n   N-1). Confirm the diff mentions the pre-existing feature (`re-ranged N sequence(s)`,\n   `delete_placement_metadata`) and has **ZERO** mentions of your backported symbol.\n4. **Best control:** run CI on your C-ONLY sibling backport (touches none of the SQL/dependency\n   code). If it shows the IDENTICAL N-1 reds, that confirms the reds track the base branch, not the\n   change.\n\n## What WOULD be your fault (fix these)\n- A red job whose `regression.diffs` names YOUR backported object **in a NORMAL (same-version)\n  job** — check-multi/-mx/-1 on a supported PG. (See the N-1 handling immediately below.)\n- A `Build for PGxx` with a real `error:` in its log.\n- A NEW job red that is GREEN on the upstream base branch. If it's a NEW-GUC/UDF N-1 red, the fix is\n  the schedule move below (not a test/output change); everything else, fix at the source.\n\nEverything else on these branches is baseline noise.\n\n## The NEW N-1 red a feature backport ADDS — move it to the N-1-excluded schedule\nBackporting a feature that adds a **new GUC or UDF** whose regression test runs under an N-1\nschedule (`multi_schedule` / `multi_1_schedule` → `check-multi`/`check-multi-1`) makes the **N-1\n(mixed-version)** jobs show that test as a NEW red. The red itself is EXPECTED graceful-degradation\n(not a defect), but the CLEAN fix is to relocate the test so N-1 never runs it — do NOT mangle the\nreviewed test, fabricate output, or just \"leave + document\". Proven on both the 13.2 and 14.0\nbackports of the `citus.allow_unsafe_insert_select_pushdown` GUC:\n- Test `allow_unsafe_insert_select_pushdown` (backported GUC `citus.allow_unsafe_insert_select_pushdown`).\n- Under N-1 the OLD lib runs on the coordinator (`lib-v13.3.0` / `lib-v14.1.0`), which lacks the GUC.\n  `SET citus.allow_unsafe_insert_select_pushdown` → `ERROR: invalid configuration parameter name ...\n  \"citus\" is a reserved prefix`, then the EXPLAIN falls back to the DEFAULT (pull-to-coordinator) plan.\n- **That fallback IS the N-1 contract** (old binary → GUC absent → safe pre-feature behavior); the diff\n  *demonstrates* compatibility. But a red CI job is still noise, so move the test off the N-1 matrix.\n\n**THE FIX: move the test line from its schedule into the placeholder section of\n`multi_1_create_citus_schedule`.** The N-1 jobs' `make_targets` (build_and_test.yml Coordinator/Lib\nN-1 blocks) list `check-multi`, `check-multi-1`, `check-add-backup-node`, ... but **NOT\n`check-multi-1-create-citus`**, so any test in `multi_1_create_citus_schedule` is N-1-invisible while\nSTILL running under `check-multi-1-create-citus` on every normal PG job (coverage preserved). Steps:\n1. Remove `test: <name>` from `multi_schedule` (mind the surrounding blank lines / parallel-group\n   comments — don't strand a comment that referenced it).\n2. Append a block under the placeholder header in `multi_1_create_citus_schedule`\n   (`# Have this placeholder section at the end for the tests that need to be moved back ... to\n   prevent N-1 test failures for the features that are not present in the minor version.`), mirroring\n   the EXISTING `citus_cluster_changes_block` block right there — same `# ----------` fencing and\n   phrasing so new additions look like the existing file.\n3. `run_test.py` auto-discovers the new schedule (it prints `SCHEDULE: multi_1_create_citus_schedule`);\n   the `.out` file is schedule-independent, so NO `.out` regen is needed — it's a pure schedule move.\n\n**N-1 version label = the actual N-1 CI version = the PREVIOUS MINOR (it advances every release).**\nRead the authoritative number from `build_and_test.yml`'s N-1 jobs — the `citus_version:` (SQL /\nWorker / Coordinator N-1) and `citus_libdir:` (Lib / Coordinator N-1) pins, not from\nmaybe-stale comments at `multi_1_create_citus_schedule`. As of this writing: release-13.2 pins\n`citus_version: \"13.3-1\"` / `citus_libdir: v13.3.0` (its `default_version` is 13.4-1, so N-1 = 13.3-1);\nrelease-14.0 pins `14.1-1` / `v14.1.0` (default 14.2-1, N-1 = 14.1-1). The rule is simply\n`default_version` minus one minor, and it is bumped whenever the branch ships a new minor.\n\n- **Write the label with today's N-1 version.** For a GUC introduced in the branch's current\n  `default_version` (e.g. 13.4 on release-13.2), the N-1 binary is the previous minor (13.3-1) and the\n  GUC is absent there, so: `GUC introduced in 13.4; not present in the N-1 (13.3-1) Citus binary ...\n  Move back to multi_schedule at Citus <next major, 14>.` On release-14.0 the corresponding numbers\n  are 14.2 / N-1 14.1-1 / next major 15.\n- **Sanity gate:** the move is only warranted when the object is genuinely ABSENT in that N-1 version\n  (introduced in a minor strictly newer than N-1). If your object was introduced at or below N-1, the\n  N-1 binary already has it — no red, no move.\n\n- **Why a new UDF may NOT need this move** (asymmetry): a test already placed in\n  `multi_1_create_citus_schedule` (e.g. `citus_internal_distribute_object`) is ALREADY N-1-invisible,\n  so it never shows an N-1 red and needs no move. Grep which schedule your test is in first.\n- Distinguish this N-1 red from a genuine bug: a same-version job (no `lib-vX` / `_coordinatoronly` /\n  `_workeronly` / `_all` suffix on the artifact) naming your object IS yours — see the section above.\n- Shipping: fold the move into a not-yet-reviewed fix commit (amend + `--force-with-lease`), or add a\n  NEW appended commit if the tip has already been reviewed (plain push, no force).\n\n## Genuinely-new failures a backport WILL introduce on a release branch (and the fix)\nThe release branch supports an OLDER PG than `main` (e.g. release-13.2 still builds **PG15**), and it\nships **version-pinned `expected/*.out`** files. A backport that is green on `main` can therefore go\nred on the release branch for two reasons that ARE yours to fix (they are not baseline noise):\n\n### (a) A shared helper's SIGNATURE changed → version-pinned .out that prints the signature mismatches\nIf your backport chain brings a new/edited SQL helper whose *signature* changed (e.g.\n`explain_filter(text)` → `explain_filter(text, keep_numbers boolean DEFAULT false)`), any expected\n`.out` that prints that signature in a `CONTEXT:` / `PL/pgSQL function ... line N` / error line will\nmismatch. **`normalize.sed` normalizes only the line NUMBER (`line [0-9]+` → `line XX`), NOT the\nsignature text**, so the signature diff is real. Fix = regenerate/`sed`-update those signature\nstrings in the pinned `.out`. Find them all with `grep -rl \"helper_name(oldargs)\" expected/`.\n- **Do NOT mistake a required helper for \"over-reach\" and revert it.** If the backported *test*\n  calls the new form (`explain_filter($$...$$, true)`), the helper is REQUIRED; reverting it yields\n  `ERROR: function ...(unknown, boolean) does not exist`. Before deleting an \"extra\" hunk a\n  cherry-pick carried, check whether the backported test depends on it.\n\n### (b) PG-version planner divergence on the older PG the release branch still supports\nThe same EXPLAIN can differ by PG major. Real example: PG15 omits a functionally-redundant trailing\nsort column that PG16+ keeps — `Sort Key: q.batch` (PG15) vs `Sort Key: q.batch, q.text_id` (PG16+).\nrelease-13.2 hits it on the PG15 `check-multi`; main/release-14.0 never see it (no PG15).\nFix, in the order the repo prefers:\n1. **An explain helper from `main`'s `multi_test_helpers.sql`** — check whether one already\n   normalizes the divergent bit. As of 13.x/14.x none normalizes Sort Key (there are\n   `coordinator_plan`, `plan_normalize_memory`, `explain_with_pg16_subplan_format`,\n   `explain_with_pg17_initplan_format`, and boolean `explain_has_distributed_subplan` /\n   `explain_has_single_task` — the booleans would need an invasive test rewrite).\n2. **MOVE the diverging blocks into a PG-version-guarded test file (PREFERRED on release-13.2 when\n   the divergence is PG15-only).** release-13.2 ships `sql/pg16.sql`, which `\\q`-skips below PG16\n   (guard at top: `SELECT substring(:'server_version','\\d+')::int >= 16 ... \\if ... \\else \\q \\endif`)\n   AND already lives in `multi_1_create_citus_schedule` (the N-1-excluded schedule). So relocating the\n   PG15-diverging EXPLAIN cases out of the normal test (e.g. `allow_unsafe_insert_select_pushdown.sql`)\n   into a self-contained section appended to `pg16.sql` means: PG15 never runs them (no diff),\n   PG16/17/18 still cover them, and — bonus — they leave the N-1 matrix too. Recipe:\n   - Cut the exact diverging blocks from the source `.sql`; fix any dangling comment they leave behind.\n   - Append AFTER `pg16.sql`'s final `DROP SCHEMA pg16 CASCADE;` a self-contained section: its OWN\n     schema (e.g. `allow_unsafe_insert_select_pushdown_pg16`), a distinct `next_shard_id`\n     (e.g. 14100000) to avoid shard-id clashes, recreate the tables/functions the blocks need, the\n     moved EXPLAIN blocks, then `DROP SCHEMA ... CASCADE;`. It must not depend on the parent test's\n     schema (schedules run independently).\n   - Regenerate BOTH `.out`s (source shrinks, `pg16.out` grows) via the runner; never hand-edit.\n3. Add a new **`normalize.sed` rule** or update an existing one to generalize the coverage, or add an\n   alternative expected file as a last resort.\n\n### (c) Flaky detector goes all-red because your changed-file set is widely depended-on\n\"Test flakyness (1..32)\" computes a changed-test set by DEPENDENCY EXPANSION of the PR's changed\nfiles. Editing a widely-depended helper/migration (`multi_test_helpers.sql`, a migration `.sql`,\n`multi_extension.sql`) balloons the set across every shard and drags in inherently-flaky base tests\n(`start_stop_metadata_sync`, `multi_metadata_sync`, ...). The shards only settle once your\n*actually-changed* tests pass; the base-flaky tests pulled in are not yours to fix. Sanity check with\n`gh pr view <n> --json files` — a helper appearing in the flaky `tests=` list but NOT in the PR's\nfile list is pure dependency-expansion, not a change you made.\n\n**Do NOT hand-wave \"it's flaky\".** Two pieces of hard evidence, both cheap:\n1. **Read the actual diff** from the shard's uploaded artifact (they DO upload one, named\n   `test_flakyness_parallel_<N>`; see fetch recipe below). Confirm the failing test is a base test\n   you did not touch AND the diff contains ZERO of your backported symbols. The signature of\n   `--use-whole-schedule-line` running the whole schedule line in parallel so other tests consume\n   colocation IDs first. Scan: `grep -c <your_symbol> regression.diffs` (want 0),\n   `grep -oE \"expected/[a-z_0-9]+[.]out\"` (only the base test).\n2. **Reproduce on the PRISTINE base** to prove pre-existing, not introduced. From `src/test/regress`:\n   `python citus_tests/run_test.py <flaky_test> --repeat 8 --use-whole-schedule-line` on your branch\n   head, then `git checkout <merge-base> && make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install` and run the SAME command. If the\n   base fails the identical N/N under the identical flags, it's 100% pre-existing (checkout your\n   branch + rebuild to restore afterward). Example proven this way: `start_stop_metadata_sync` failed\n   8/8 on both `bp-release-13.2` tip and its pristine base, same colocationid/FK diff — not the\n   backport's fault. NOTE the flaky detector's PG major varies — check the shard log's `PG_MAJOR:`\n   line and reproduce on the SAME PG for an exact comparison.\n\n### Shipping the fix\nOne NEW fix commit on top of the reviewed tip (do not rewrite reviewed SHAs). If a prior *bad* fix\ncommit is already on the remote, REPLACE it with `git push --force-with-lease=<branch>:<badsha>`\n(the reviewed commits below it stay). Force-push freely only while the commits are still unreviewed.\n\n## Fetching CI diffs reliably (tooling gotchas)\n- `gh run view --job <id> --log-failed` streams often die with `stream error: stream ID 1; CANCEL` —\n  retry or avoid.\n- Jobs that upload artifacts (`NN_check-multi`, `NN_check-operations`, AND the flaky shards):\n  list with a SINGLE page, not `--paginate` (`--paginate` concatenates JSON pages so a single `--jq`\n  chokes with \"Additional text after finished reading JSON content\"):\n  `gh api \"repos/citusdata/citus/actions/runs/<run>/artifacts?per_page=100\" --jq '.artifacts[]|select(.name==\"15_check-multi\").id'`\n  then `gh api .../actions/artifacts/<id>/zip > x.zip` (this redirect is fine for the small per-artifact\n  zip; read `regression.diffs` from it).\n- Flaky-detector jobs DO upload a named artifact `test_flakyness_parallel_<N>` (contains the shard's\n  `regression.diffs` under `__w/citus/citus/src/test/regress/`) — fetch it like the per-artifact zip\n  above; that's the reliable way to read the flaky diff body. The per-job logs endpoint only NAMES the\n  failing test (`SCHEDULE_LINE:` / `not ok`) and points at regression.diffs in the artifact.\n- Do NOT re-encode a downloaded zip through a shell pipe that changes encoding (corrupts it); write it\n  to disk directly. The whole-run `/logs` zip often arrives partial (\"End of Central Directory record\n  could not be found\") — prefer the per-job / per-artifact endpoints above.\n\n## Mechanics / gotchas\n- Overall run status stays `queued`/`in_progress` until EVERY matrix job ends; per-job logs and\n  finished-job artifacts are still fetchable mid-run (above).\n- Dispatching several full matrix runs (~190 jobs each) saturates the fork runner pool and starves\n  the job you actually care about; `gh run cancel <id>` the fully-triaged runs to free capacity for\n  a critical re-verification.\n",".github/skills/README.md":"# Agent skills for citus\n\nThis directory holds **agent skills** — folders of instructions (and optionally\nscripts/resources) that an AI coding agent can load on demand when it works on a\nparticular kind of task in this repository. Skills let us capture hard-won,\nrepo-specific know-how once and have every agent (and human) reuse it, instead of\nre-deriving it from scratch each time.\n\n## Layout\n\nEach skill lives in its own subdirectory containing a `SKILL.md`:\n\n```\n.github/skills/\n├── README.md                     ← this file\n└── citus-backport/\n    ├── SKILL.md                  ← entry point (loaded first)\n    └── references/               ← deep-dive docs loaded on demand\n        ├── sql-schema-backport.md\n        ├── ci-triage.md\n        └── manual-upgrade-testing.md\n```\n\n- Subdirectory names are lowercase with hyphens.\n- The entry file **must** be named `SKILL.md`.\n- Keep `SKILL.md` a concise hub; push long procedures into `references/` files it\n  links to, so the agent only pulls in what it needs.\n\n## `SKILL.md` format\n\nA `SKILL.md` is Markdown with YAML frontmatter:\n\n```markdown\n---\nname: my-skill                 # required; lowercase-hyphenated, matches the folder\ndescription: >-                # required; WHAT it does + WHEN to use it (drives auto-selection)\n  One or two sentences describing the task this skill helps with and the\n  trigger conditions that should make an agent load it.\nlicense: See the repository LICENSE file.   # optional\n---\n\n# Human-readable title\n\nInstructions, examples, and guidelines for the agent to follow…\n```\n\nThe `description` is the most important field for discovery: agents match a user's\nrequest against it, so state both the capability and the trigger phrases plainly.\n\n## Where these are picked up\n\nProject skills under `.github/skills/` are discovered automatically by GitHub\nCopilot CLI. (The Agent Skills format is intentionally portable; other agent tools\nmay read `.claude/skills/` or `.agents/skills/` instead — see each tool's docs.)\n\n`.github/copilot-instructions.md` (auto-loaded repo-wide) also carries a short\n`task → skill` index so agents are pointed here even before they browse this\ndirectory. When you add a skill, add a row there too.\n\n## Adding a new skill\n\n1. Create `./<skill-name>/SKILL.md` with valid frontmatter.\n2. Put anything long or optional in `./<skill-name>/references/` and link to it\n   from `SKILL.md`.\n3. Keep it repo-specific and evidence-based; avoid machine- or user-specific paths\n   so it works for anyone who clones citus.\n4. Add a `task → skill` row to `.github/copilot-instructions.md` so agents discover it.\n",".github/skills/citus-backport/references/sql-schema-backport.md":"# SQL-schema backport recipe (the hard case)\n\nA C-only cherry-pick usually just applies. The moment a commit touches the SQL schema (new UDF,\nnew column, new migration step) you must **remap it to the target branch's real version**, because\n`main`'s migration files are named for the NEXT major (e.g. `15.0-1`) and the release branch is on\na different `default_version`. Do NOT accept `main`'s files as-is.\n\n## Step 0 — read the target version (drives everything)\nOn each target branch read `src/backend/distributed/citus.control`:\n\n```\ndefault_version = '<X.Y-Z>'\n```\n\nThen find that branch's current dev migration + downgrade under\n`src/backend/distributed/sql/`:\n- upgrade:   `citus--<prev>--<default_version>.sql`\n- downgrade: `citus--<default_version>--<prev>.sql`\n\nWorked example (versions as of this writing):\n\n| Branch        | default_version | dev upgrade migration        | dev downgrade                | udf versioned file |\n|---------------|-----------------|------------------------------|------------------------------|--------------------|\n| release-13.2  | `13.4-1`        | `citus--13.3-1--13.4-1.sql`  | `citus--13.4-1--13.3-1.sql`  | `13.4-1.sql`       |\n| release-14.0  | `14.2-1`        | `citus--14.1-1--14.2-1.sql`  | `citus--14.2-1--14.1-1.sql`  | `14.2-1.sql`       |\n| main          | `15.0-1`        | `citus--14.0-1--15.0-1.sql`  | `citus--15.0-1--14.0-1.sql`  | `15.0-1.sql`       |\n\n## Step 1 — remap the migration files\nThe main commit added its schema hunk to `citus--14.0-1--15.0-1.sql` (and the DROP to the matching\n`15.0-1--14.0-1` downgrade), plus a versioned udf file `sql/udfs/<name>/15.0-1.sql`. On the target\nbranch:\n\n1. `git rm` the main-only `15.0-1` upgrade + downgrade migration files if the cherry-pick created\n   them (they don't belong on the release branch).\n2. `git mv src/.../sql/udfs/<name>/15.0-1.sql  src/.../sql/udfs/<name>/<default_version>.sql`\n   (rename the versioned udf file to the branch's version).\n3. Add the `#include \"udfs/<name>/<default_version>.sql\"` line to the branch's **dev upgrade\n   migration** (`citus--<prev>--<default_version>.sql`) — same include the main commit added to\n   `14.0-1--15.0-1.sql`, just in the branch's file.\n4. Add the reverse `DROP FUNCTION IF EXISTS <fully-qualified-signature>;` (or DROP for whatever\n   object) to the branch's **dev downgrade** (`citus--<default_version>--<prev>.sql`).\n5. Keep `sql/udfs/<name>/latest.sql` — refresh it to identical content as the versioned file.\n\n`ci/check_migration_files.sh` (\"Check for missing downgrade scripts\") gates step 4 — every upgrade\nstep needs a matching downgrade. The CI `check-sql-snapshots` job validates these manual edits, so\ngreen there is strong confirmation the remap is correct.\n\n## Step 2 — fix `multi_extension.out`\n`multi_extension` walks the whole upgrade+downgrade ladder and prints a \"Snapshot of state at\n`<version>`\" section per version. The new object's row must land under the **target branch's**\nversion section (e.g. `13.4-1`), NOT main's `15.0-1`. The snapshot table's column widths are\ndynamic — **regenerate the .out with the runner, don't hand-align** it. From `src/test/regress`:\n\n```\npython citus_tests/run_test.py multi_extension\n```\n\nThe new line typically shows the new function/object with `(1 row)` (or the appropriate count) in\nthe target-version section.\n\n## Step 3 — fix `upgrade_list_citus_objects.out` (edit BY CONSTRUCTION)\nThis test **cannot be run standalone** via `run_test.py` — it depends on a `citus_schema` created\nby an earlier test in `after_pg_upgrade_with_columnar_schedule`; run alone it dies with\n`ERROR: no schema has been selected to create in`. So validate its edit by construction:\n- the new function line auto-merges alphabetically in the cherry-pick (leave it);\n- fix only the trailing **row COUNT** = base_count + (number of new citus objects). Read the base\n  count from the branch's own expected file. Observed: release-13.2 `378 → 379`, release-14.0\n  `384 → 385` for one new UDF.\n\n## Step 4 — build WITH downgrades, then test\n`make install` does NOT install downgrade scripts, and `multi_extension` exercises the downgrade\npath, so after every rebuild run both:\n\n```\nmake -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install\nmake -C src/backend/distributed install-downgrades\nmake -C src/backend/columnar    install-downgrades\n```\n\n(`columnar` is a SEPARATE extension with its own downgrades; omitting it surfaces as a\n`citus_columnar` \"no update path\" error. `make install-all` runs both.)\n\nThen run the feature's own test + `multi_extension`.\n\n## Step 5 — propagate the upgrade ladder ACROSS branches (MVU safety)\n\nBackporting the object DOWN to a release branch is not enough. An MVU replays the migration\n**ladder**, so a new SQL object must be reachable on EVERY higher branch's ladder too — otherwise\na cluster upgraded 13.x→14.y would silently lose it. Net effect: **the same object is defined\nMULTIPLE times per branch, once per major's introduction step.** This is intentional, not\nduplication to \"fix\".\n\n**Topology** (worked example: `citus_internal.distribute_object`, introduced on main at\n`14.0-1--15.0-1`, backported to 13.2 at `13.3-1--13.4-1` and to 14.0 at `14.1-1--14.2-1`):\n\n| Ladder step (+ its downgrade + versioned udf) | release-13.2 | release-14.0 | main |\n|---|---|---|---|\n| `13.3-1--13.4-1` (udf `13.4-1`) — 13.x introduction | ✅ (the backport itself) | ✅ copy here | ✅ copy here |\n| `14.1-1--14.2-1` (udf `14.2-1`) — 14.x introduction | n/a | ✅ (the backport itself) | ✅ copy here |\n| `14.0-1--15.0-1` (udf `15.0-1`) — main's own top    | n/a | n/a | ✅ already there (orig land) — LEAVE UNTOUCHED |\n\nRule of thumb: **a step introduced on major N must be copied onto every branch for major ≥ N.**\n13.x paths → release-14.0 + main; 14.x paths → main; main's own top path stays as merged.\n\n**How to do it (per higher branch, e.g. release-14.0 then main):**\n1. This is a **SEPARATE commit** on top of the (unchanged) cherry-pick — never amend the\n   cherry-pick. On main it's its own PR.\n2. Copy the step + downgrade + versioned-udf files **byte-for-byte** from the branch that owns\n   the introduction. `git show <branch>:<path>` reads any branch's blob without checking it out\n   (worktrees of one clone share an object store, so this is trivial):\n   ```\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/citus--13.3-1--13.4-1.sql          > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/downgrades/citus--13.4-1--13.3-1.sql > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/udfs/<name>/13.4-1.sql             > <same path>\n   ```\n   (and the analogous `14.1-1--14.2-1` trio from `bp-<pr>-release-14.0` when copying onto main.)\n3. **Keep the header verbatim** — even the `-- bump version to 13.4-1` line stays on downstream\n   branches where 13.4 is not the default. Confirmed identical across branches in the #8621 UDF family.\n4. **Purely additive.** Do NOT touch `citus.control` / `default_version`. Do NOT touch the\n   `13.2-1--14.0-1` major-bridge migration. main's `latest.sql` still points at its own top\n   version (15.0-1); leave it. `check_sql_snapshots.sh` verifies latest.sql == highest udf.\n5. New step files contain ONLY the object being propagated (header + single `#include`); the\n   downgrade is header + `DROP FUNCTION IF EXISTS <signature>;`. Don't drag along unrelated objects\n   that happen to share the source branch's step file.\n6. Build + `install-downgrades` + `multi_extension` on the higher branch (expect PASS, usually no\n   `.out` change needed since the new sections are additive). Run the three migration CI checks\n   locally before pushing: `ci/check_migration_files.sh`, `ci/check_sql_snapshots.sh`,\n   `ci/disallow_c_comments_in_migrations.sh` (all must exit 0).\n7. **Add `multi_extension.sql` ladder-test coverage for the new step** (or prove it's already\n   covered) — see the dedicated section below.\n\n**Template PR family** (study these to see the end-state, don't reverse-engineer from one PR):\nfind the original feature PR, then every PR that mentions it in title/desc. Worked example — the\n`citus_internal.distribute_object` repair UDF, introduced on main in **#8621**: backported\none-branch-per-target via **#8684**→release-13.2 and **#8685**→release-14.0 (each branch also carries\nthe sibling GUC PR **#8625** `citus.allow_unsafe_insert_select_pushdown`, unsquashed as its own\ncommit), with the 13.x/14.x upgrade paths propagated to `main` in **#8686**. The main ladder PR\n(#8686, \"Add 13.x/14.x upgrade paths for citus_internal.distribute_object to main\") is the model for\nthe main ladder commit's title/body; it's internal plumbing so usually **no `DESCRIPTION:` line**.\n\n## multi_extension.sql ladder tests for new steps (off-walk detour pattern)\n\nAdding a ladder step (Step 5 backport OR Step 5 propagation) is not fully covered until\n`src/test/regress/sql/multi_extension.sql` exercises it — or you can prove the existing walk already\ndoes. Model the additions on the template PR family, adapted per branch.\n\n- **The test walks ONE path only** — the main upgrade path (`13.2 → 13.3 → 14.0 → 14.1 → 15.0`).\n  A step you add for MVU may sit **off** that walk:\n  - the 13.x maintenance-tail step `13.3-1--13.4-1` (off-walk on release-14.0 and main), and\n  - the 14.x maintenance-tail step `14.1-1--14.2-1` (off-walk on **main**, whose walk jumps 14.1 → 15.0).\n  Off-walk steps' upgrade AND downgrade scripts are otherwise **never executed** by the test.\n- **On-walk / top-of-branch steps need NOTHING extra.** A branch's own `default_version` (13.4-1 on\n  release-13.2, 14.2-1 on release-14.0, 15.0-1 on main) is already the target of the existing\n  `-- Snapshot of state at X` block; the propagated object simply appears in that snapshot's `.out`.\n  ⇒ release-13.2 needed **no** `multi_extension.sql` change; release-14.0 needed **one** detour\n  (13.4-1); main needed **two** (13.4-1 and 14.2-1).\n- **PATTERN = round-trip no-op → snapshot at the detour → rejoin the walk FROM the detour.** Match\n  the shape of the existing on-walk blocks — visit the off-walk version the same way every other\n  version is visited, so the additions look like they belong. Do NOT add editorial parentheticals\n  (no \"(13.x maintenance line, off the ... path)\"). At the point the walk sits at the detour's\n  PARENT, insert two blocks AND retarget the *existing* next round-trip so it bounces off the\n  detour instead of the parent:\n  ```sql\n  -- Test downgrade to <parent> from <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';   -- e.g. 13.4-1\n  ALTER EXTENSION citus UPDATE TO '<parent>';   -- e.g. 13.3-1\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Snapshot of state at <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Test downgrade to <detour> from <next>   <-- was \"<parent> from <next>\"; retarget it\n  ALTER EXTENSION citus UPDATE TO '<next>';    -- the next ON-WALK version (e.g. 14.0-1 / 15.0-1)\n  ALTER EXTENSION citus UPDATE TO '<detour>';  -- bounce back to the detour, NOT the parent\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n  ```\n  Block 1 (round-trip) proves the up+down pair is a perfect no-op. Block 2 (snapshot) shows the new\n  object actually appearing at the detour version (`+ function citus_internal.distribute_object...`,\n  `(1 row)`). The snapshot leaves the walk sitting at `<detour>` with the `print_extension_changes()`\n  baseline `= {detour}`. Instead of adding a separate \"return to parent\" block, you **retarget the\n  existing next round-trip** (which used to bounce `<parent> ↔ <next>`) so it bounces `<detour> ↔\n  <next>` — this keeps the walk moving forward from the detour and lands it back at `<detour>` with\n  baseline reset to `{detour}`.\n- **The one intended downstream change:** because the walk now enters the following `-- Snapshot of\n  state at <next>` block with baseline `{detour}` (not `{parent}`), that ONE snapshot shifts by\n  exactly the detour's object, and nothing else changes:\n  - If `<next>` does NOT contain the object (e.g. `14.0-1`, a 13.x-line object): the snapshot GAINS a\n    `previous_object` row (the object shown as dropped when leaving the maintenance line).\n  - If `<next>` DOES contain the object (e.g. `15.0-1` when the object is already at `14.2-1`): the\n    snapshot LOSES the `current_object` row it used to show (it's no longer newly-added — it was\n    already displayed at the detour snapshot).\n  Both are self-consistent and desirable: the object is introduced exactly once, at its detour\n  snapshot. Every block AFTER that `<next>` snapshot stays byte-identical (print resets baseline to\n  `{next}` regardless). Regenerate `.out` and eyeball that only that single snapshot moved.\n- **`.out` regen:** edit `.sql` → runner \"fails\" (writes `results/multi_extension.out`) →\n  `cp results/multi_extension.out expected/multi_extension.out` → re-run → \"All 1 tests passed\".\n  **Never hand-edit `.out`.**\n- **`diff -w` header gotcha:** pg_regress compares with `diff -dU10 -w`, so snapshot-table header\n  column widths / leading spaces are IGNORED for pass/fail. A regen may normalize a pre-existing\n  HAND-EDITED narrow header (left over from an earlier `.out` hand-edit) to the runner's canonical\n  wider spacing. Whitespace-only and harmless — **keep the canonical runner output**, note it in the\n  commit message. Prove it's pre-existing: run the test on pristine HEAD (no change) — same wide\n  header, still PASSES.\n- **APPEND-ONLY once reviewed:** if the backport branch/PR has already been reviewed, do NOT\n  force-push — add the `multi_extension` test work as a new commit on top.\n\n## Flaky / gotcha notes for SQL backports\n- **`multi_extension` flaky maintenance-daemon crash:** the issue #3409 `DROP EXTENSION` under a\n  non-superuser role section can report \"server process was terminated\" once, then PASS on a clean\n  re-run. If the PG log shows only a clean fast-shutdown (no PANIC / segfault / Assert), it's\n  timing/flaky — re-run before investigating. Sanity-check by running the same test on the PRISTINE\n  base branch.\n- **`configure` gets regenerated by builds** — never part of a backport. `git checkout -- configure`\n  before committing.\n- **`git stash -u` mid-cherry-pick** drops `CHERRY_PICK_HEAD` but keeps the file changes; on restore\n  you must `git commit` manually and re-add the original message + `(cherry picked from commit <sha>)`.\n- **Regenerate .out with the runner, never hand-edit** — snapshot column widths and object counts\n  are computed.\n",".github/skills/citus-backport/SKILL.md":"---\nname: citus-backport\ndescription: >-\n  Backport one or more merged citusdata/citus `main` commits/PRs onto the active release\n  branches. Covers the release model, cherry-picking, remapping SQL schema changes to each\n  branch's default_version, the SQL upgrade/downgrade ladder for Major-Version-Upgrade safety,\n  running regression tests, one clean commit per PR per branch, and triaging release-branch CI\n  (pre-existing baseline reds vs reds the backport introduced). USE WHEN asked to backport / port\n  / cherry-pick a change to release-13.2 / release-14.0 (or the newest two majors), when a\n  backport hits a SQL migration / udf / multi_extension conflict, when a \"trivial C-only\" backport\n  fails to compile on an older PG the release branch still supports (e.g. PG15 `rteperminfos`),\n  when release-branch CI is red and you must prove which reds are pre-existing, or when you must\n  prove a SQL-schema backport is Major-Version-Upgrade-safe via `ALTER EXTENSION citus UPDATE`\n  across a real multi-node cluster. DO NOT USE for non-Citus repos, new features, or PR review.\nlicense: See the repository LICENSE file.\n---\n\n# Backport a Citus change to the release branches\n\nThis skill encodes how to backport merged `main` commit(s) to Citus release branches: the release\nmodel, the SQL-schema remap recipe, the PG-version compatibility traps, and how to triage the\nrelease branches' baseline-red CI. Follow the repository's normal build/test/style conventions\n(see `CONTRIBUTING.md` and `src/test/regress/README.md`) on top of the guidance here.\n\n## Reference deep-dives (load on demand)\nAll live next to this file under `references/`:\n- **`references/sql-schema-backport.md`** — the hard case: a commit that changes the SQL schema\n  (new UDF / column / migration). Exact file-by-file remap recipe, the `multi_extension.out` and\n  `upgrade_list_citus_objects.out` edits, the `install-downgrades` trap, and the cross-branch\n  **ladder-propagation** step (copy the introduction paths UP to every higher major for MVU safety).\n- **`references/ci-triage.md`** — reading backport CI: the PG-version compile traps (main drops\n  old PGs the release branch keeps), the N-1 mixed-version jobs, and the step-by-step proof that a\n  red job is pre-existing baseline noise vs a real regression you introduced.\n- **`references/manual-upgrade-testing.md`** — the manual MVU proof: stand up a multi-node cluster\n  at an OLD Citus major, then `ALTER EXTENSION citus UPDATE` across the backport branches (13.x\n  tail → cross-major → 14.x tail), capturing `citus_version()` + `pg_extension.extversion` per node\n  per hop. The `make install-all` / single-shared-PG requirement.\n\nA C-only backport usually needs only this hub + `ci-triage.md`. A SQL-schema backport needs all\nfour (add `manual-upgrade-testing.md` as the final proof once CI is green).\n\n## Release model (internalize this first)\n- **ONE branch per MAJOR.** `release-13.2` serves the WHOLE 13.x line; `release-14.0` serves ALL of\n  14.x; `main` is the next major. Citus no longer cuts a branch per minor (this changed ~2026-05).\n- **Backport FEATURES, not just bugfixes**, to the **newest two majors** (13 & 14 as of this\n  writing).\n- **Branch names LIE about the version.** `release-13.2` currently ships 13.4; `release-14.0` ships\n  14.2. **NEVER infer the target minor from the branch name** — read `default_version` from\n  `src/backend/distributed/citus.control` on each target branch. That value drives every SQL edit.\n- **N-1 compatibility is required and CI-gated.** A backport is N-1-safe when it is purely\n  ADDITIVE (new UDF / GUC / column) and does not change an existing UDF signature or wire format\n  used cross-version. If it's additive, it's safe. See `references/ci-triage.md` for the jobs.\n\n## Workflow (per PR, per target branch)\n1. **Use an isolated build per target major.** A dedicated checkout/worktree + its own PostgreSQL\n   install per release branch keeps builds/tests from clobbering each other and your main checkout.\n   Two worktrees of the same clone SHARE one git object store — all branches are visible/pushable\n   from either, and a `+`-prefixed branch in `git branch` is checked out in a sibling worktree.\n2. **Make the SHAs reachable**, then `git checkout -b bp-<pr>-<release-branch> origin/<release-branch>`\n   and `git cherry-pick -x <sha>` (the `-x` records `(cherry picked from commit <sha>)`).\n3. **Resolve:**\n   - **C-only:** usually applies clean or with trivial context conflicts. THEN check PG-version\n     compat (`references/ci-triage.md` §PG-version) — the #1 non-obvious C-backport failure.\n   - **SQL-schema:** migration / udf / `multi_extension.out` conflict → remap by hand per\n     `references/sql-schema-backport.md`. Do NOT just accept `main`'s next-major (e.g. `15.0-1`) files.\n4. **Build in the isolated env:** `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install`. For any SQL change ALSO install\n   downgrades (see `references/sql-schema-backport.md` — plain `make install` skips them and\n   `multi_extension` will fail with cascading \"no update path\" errors otherwise; use\n   `make install-all`).\n5. **Test only what's relevant** (CI runs the whole suite): the feature's own test + `multi_extension`\n   for SQL changes. From `src/test/regress`: `python citus_tests/run_test.py <test>` (it manages its\n   own cluster). Regenerate any `expected/*.out` with the runner — **never hand-edit .out** (column\n   widths / row counts are dynamic).\n6. **Collapse to ONE commit per PR per branch.** Keep the original message; keep a `DESCRIPTION:`\n   first line iff the change is user-facing (this feeds the changelog). `git checkout -- configure`\n   if a build regenerated it (configure is never part of a backport). **Multiple PRs to the SAME\n   target → ONE branch per target, not per PR** (unsquashed: one commit per PR, in main's\n   chronological merge order; SQL-ladder commits sit right after their PR's cherry-pick). Name it\n   `bp-release-<major>` (e.g. `bp-release-13.2`). Build cleanly from the SQL-heavy PR's branch, then\n   cherry-pick the C-only PRs on top. **Cherry-pick an already-adapted backport commit WITHOUT\n   `-x`** — its message already carries the single `(cherry picked from <orig>)` trailer; `-x` would\n   double it.\n7. **Push to your fork.** `bp-*` branches don't match CI's push trigger (`main`/`release-*` only),\n   so kick CI with `gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`. Open\n   PRs only when the change owner asks.\n8. **Wait for CI and triage** per `references/ci-triage.md`. Separate baseline-red noise (N-1 /\n   flaky / coverage) from **genuinely-new failures you introduced**. The two you WILL hit on a\n   release branch: (a) a version-pinned `.out` that prints a shared helper's changed *signature*, and\n   (b) a PG-version planner divergence on the OLDER PG the branch keeps but `main` dropped (PREFER a\n   PG-guarded file like `pg16.sql` over a `<test>_N.out` alt file). A third you must NOT chase green\n   by mangling the test: a new GUC/UDF whose test runs under N-1 shows a NEW red in the (non-blocking)\n   **N-1 mixed-version** jobs — the old lib lacks it and falls back, which IS the N-1 contract. The\n   clean fix is to move that test line into the `multi_1_create_citus_schedule` placeholder section\n   (schedule mechanics + N-1 label rule in the hard-rules below and `ci-triage.md §\"NEW N-1 red\"`).\n   **Fix-commit rule (all fixes):** pre-review → amend + `git push --force-with-lease`; post-review →\n   add ONE new fix commit, never rewrite reviewed SHAs (replace a bad remote fix commit with\n   `--force-with-lease=<branch>:<badsha>`). Report evidence; don't declare done on red.\n9. **Propagate the ladder UP (SQL-schema backports only).** A new SQL object must also be reachable\n   on every HIGHER major's upgrade ladder, or a future 13.x→14.y Major Version Upgrade loses it.\n   As SEPARATE commits (its own PR on main), copy the introduction step/downgrade/udf files\n   byte-for-byte onto release-14.0 (for a 13.x path) and main (for 13.x and 14.x paths, given today's\n   two most recent major versions); main's own top path stays untouched. Full topology + recipe in\n   `references/sql-schema-backport.md` §Step 5.\n10. **Add `multi_extension.sql` ladder-test coverage** for every new step. The test only walks the\n    main upgrade path, so a step OFF that path (a 13.x/14.x maintenance-tail detour) is never\n    exercised unless you add a detour block matching the on-walk shape (and the file's existing\n    comment style — no editorial parentheticals): a round-trip no-op + a snapshot at the detour,\n    then retarget the *existing* next round-trip to bounce `<detour> ↔ <next>` rather than adding a\n    separate \"return\" block. That shifts exactly one downstream snapshot; everything after stays\n    byte-identical. A branch's own top/default_version is already covered by its snapshot block.\n    Never hand-edit `.out`; prefer append-only on reviewed branches. Full pattern + the `diff -w`\n    header gotcha in `references/sql-schema-backport.md`.\n11. **Prove MVU safety on a real cluster (SQL-schema backports, after CI is green).** The ladder\n    propagation only *claims* a future 13→14 Major Version Upgrade keeps the object; prove it. Stand\n    up a multi-node cluster at an OLD major, then `ALTER EXTENSION citus UPDATE` on EVERY node across\n    the backport branches (13.x tail → cross-major → 14.x tail → cross-major → main tail), capturing\n    `citus_version()` and `pg_extension.extversion` per node per hop, plus a functional check that the\n    backported object survived. Build all Citus versions into ONE shared PG with **`make install-all`**\n    (plain `install` omits the downgrade + bridge scripts the cross-major route needs → \"no update path\").\n    Full recipe and the single-PG requirement in `references/manual-upgrade-testing.md`.\n\n## Hard rules / traps (full detail in the references)\n- **Do NOT run `make reindent` in a build environment whose `citus_indent` version differs from the\n  branch's** — a version-skewed formatter reformats 100+ unrelated files and errors out. Trust\n  cherry-picked formatting; CI `check-style` verifies it. If you ran it by accident:\n  `git reset --hard HEAD`.\n- **`make install` does NOT install downgrade scripts.** For any SQL change run BOTH\n  `make -C src/backend/distributed install-downgrades` and `make -C src/backend/columnar\n  install-downgrades` (columnar is a separate extension). `make install-all` does both.\n- **A single build environment builds ONE PG only** — it cannot catch a compile break on another PG\n  the release branch supports. Rely on CI's per-PG `Build for PGNN` jobs; scan cherry-picked C for\n  struct fields / APIs newer than the branch's OLDEST PG and add `#if PG_VERSION_NUM >= PG_VERSION_NN`.\n- **The release branches are BASELINE-RED.** Several N-1 / flaky jobs fail on the pristine base\n  branch with no backport. Subtract the baseline before blaming your change — prove it with the\n  upstream `citusdata/citus` base-branch run. Details + exact commands in `references/ci-triage.md`.\n- **A new SQL object is defined MULTIPLE times on purpose.** After ladder propagation the same UDF\n  appears in the 13.x, 14.x AND main introduction steps (triple-defined on main). That is\n  intentional MVU/N-1 safety, NOT duplication to collapse. Never \"dedupe\" it. Details:\n  `references/sql-schema-backport.md` §Step 5.\n- **Show evidence.** Present the proof (build-job outcomes, `regression.diffs` excerpts, the\n  upstream-baseline comparison), not a bare \"CI is green/red\".\n- **A backport onto an older-PG release branch surfaces failures `main` never saw** (release-13.2\n  still builds PG15; the branch ships version-pinned `expected/*.out`). Two patterns: (1) a shared\n  helper whose SIGNATURE you changed prints in a pinned `.out` (`normalize.sed` masks the line\n  NUMBER, not the signature) — do NOT \"fix\" it by reverting the helper if the backported TEST needs\n  it (it's required, not over-reach); (2) a PG15-vs-PG16+ EXPLAIN/planner diff → PREFER a PG-guarded\n  file (`pg16.sql` `\\q`-skips <PG16 and already lives in the N-1-excluded schedule) over a\n  `<test>_0.out` alt. Full recipe: `references/ci-triage.md` §Genuinely-new failures.\n- **A NEW GUC/UDF whose test runs under N-1 has a CLEAN fix, not \"leave it\".** Move the test line\n  from its schedule into the `multi_1_create_citus_schedule` placeholder section (N-1 make_targets\n  omit `check-multi-1-create-citus`). N-1 version label = the branch's CURRENT N-1 = the previous\n  minor (13.3-1 on release-13.2, 14.1-1 on release-14.0), read from the live `citus_version:` /\n  `citus_libdir:` pin in build_and_test.yml — not the (possibly-stale) number in the neighbor comment.\n  Details: `references/ci-triage.md` §\"NEW N-1 red\".\n","src/backend/distributed/commands/README.md":"# Commands\n\nThe commands module is modeled after `backend/commands` from the postgres repository and\ncontains the logic for Citus on how to run these commands on distributed objects. Even\nthough the structure of the directory has some resemblence to its postgres relative, files\nhere are somewhat more fine-grained. This is due to the nature of citus commands that are\nheavily focused on distributed tables. Instead of having all commands in `tablecmds.c`\nthey are often moved to files that are named after the command.\n\n| File                         | Description |\n|------------------------------|-------------|\n| `create_distributed_table.c` | Implementation of UDF's for creating distributed tables |\n| `drop_distributed_table.c`   | Implementation for dropping metadata for partitions of distributed tables |\n| `extension.c`                | Implementation of `CREATE EXTENSION` commands for citus specific checks |\n| `foreign_constraint.c`       | Implementation of and helper functions for foreign key constraints |\n| `grant.c`                    | Implementation of `GRANT` commands for roles/users on relations |\n| `index.c`                    | Implementation of commands specific to indices on distributed tables |\n| `multi_copy.c`               | Implementation of `COPY` command. There are multiple different copy modes which are described in detail below |\n| `policy.c`                   | Implementation of `CREATE\\ALTER POLICY` commands. |\n| `rename.c`                   | Implementation of `ALTER ... RENAME ...` commands. It implements the renaming of applicable objects, otherwise provides the user with a warning |\n| `schema.c`                   | |\n| `sequence.c`                 | Implementation of `CREATE/ALTER SEQUENCE` commands. Primarily checks correctness of sequence statements as they are not propagated to the worker nodes |\n| `table.c`                    | |\n| `transmit.c`                 | Implementation of `COPY` commands with `format transmit` set in the options. This format is used to transfer files from one node to another node |\n| `truncate.c`                 | Implementation of `TRUNCATE` commands on distributed tables |\n| `utility_hook.c`             | This is the entry point from postgres into the commands module of citus. It contains the implementation that gets registered in postgres' `ProcessUtility_hook` callback to extends the functionality of the original ProcessUtility. This code is used to route the incoming commands to their respective implementation in Citus |\n| `vacuum.c`                   | Implementation of `VACUUM` commands on distributed tables |\n\n# COPY\n\nThe copy command is overloaded for a couple of use-cases specific to citus. The syntax of\nthe command stays the same, however the implementation might slightly differ from the\nstock implementation. The overloading is mostly done via extra options that Citus uses to\nindicate how to operate the copy. The options used are described below.\n\n## FORMAT transmit\n\nImplemented in `transmit.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n\n## FORMAT result\n\nImplemented in `multi_copy.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n"},"items":[{"name":"ci-triage.md","path":".github/skills/citus-backport/references/ci-triage.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/skills/citus-backport/references/ci-triage.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# CI triage for backports — PG-version compat + the baseline-red release branches\n\nTwo independent things make backport CI confusing: (1) `main` might support different PG majors\nthan a release branch, so a \"trivial C-only\" cherry-pick can fail to COMPILE on an old PG; and\n(2) the release branches are BASELINE-RED — several jobs fail on the pristine base branch with\nno backport. Handle both before declaring a red \"your fault\".\n\n## PG-version compat (the #1 non-obvious C-backport failure)\n`main` may support different PG majors than the release branches, so code `main` wrote unconditionally\nagainst a newer PG API breaks the older-PG build on a release branch. Check each branch's supported\nPG set in `.github/workflows/build_and_test.yml` (the `pgNN_version` params block). Today, these are the\nsupported PG majors by main and two most recent release branches:\n\n| Branch        | PG majors built      |\n|---------------|----------------------|\n| release-13.2  | **15**, 16, 17       |\n| release-14.0  | 16, 17, 18 (no 15)   |\n| main          | 16, 17, 18 (dropped 15) |\n\n**Concrete example (confirmed):** `Query.rteperminfos` was added in PG16. `main` sets\n`q->rteperminfos = NIL;` bare. On release-13.2 that is `error: 'Query' has no member named\n'rteperminfos'` on the **PG15** build. Fix = the citus idiom the branch ALREADY uses everywhere\nelse for that field:\n\n```c\n#if PG_VERSION_NUM >= PG_VERSION_16\n    q->rteperminfos = NIL;\n#endif\n```\n\nGrep the target branch for the same symbol to copy its exact guard macro. **General rule:** after a\nC cherry-pick, scan the added `+` lines for struct fields / API calls introduced in a PG major that\nthe branch's OLDEST PG predates, and guard them. **Backport to the oldest-PG branch first** to\nsurface these early. A single build environment builds ONE PG so it can't catch this locally — rely\non CI's per-PG `Build for PGNN` jobs.\n\n### Reading a Build-for-PGNN failure mid-run\n`gh run view --log` / `--log-failed` say \"still in progress\" until the WHOLE run finishes, but a\nsingle finished job's log is fetchable immediately:\n\n```\ngh api repos/<your-fork>/citus/actions/jobs/<jobId>/logs > build.log\n```\n\nThen grep for `error:` / `make.*Error`. A `Build for PGxx` failure with a real `error:` line = a\ngenuine bug (fix it). An \"exit code N\" annotation with no compiler `error:` in the log = usually\ninfra/flaky (rerun). After fixing, **amend the single cherry-pick commit** (keep 1 commit/PR),\n`git push --force-with-lease`, and re-dispatch:\n`gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`.\n\n## The release branches are BASELINE-RED\n`release-13.2` and `release-14.0` have KNOWN-FAILING jobs on their own pristine upstream CI. Do NOT\npanic when your backport branch is red — **subtract the baseline first.** As of this writing the\nbase branches fail these SAME jobs with no backport applied:\n\n- **N-1 mixed-version jobs where the OLD lib runs on the COORDINATOR:**\n  - Test Citus **Lib N-1** — check-multi, check-multi-1, check-add-backup-node\n  - Test Citus **Coordinator N-1** — check-multi, check-multi-1, check-add-backup-node\n  - (The SQL N-1 and Worker N-1 variants of the same tests PASS — old lib is only on workers there.)\n- Occasional **Test flakyness (N)** shards and a stray check-enterprise / check-operations.\n- **Test Citus Failure / check-query-generator** — a randomized differential fuzzer (compares\n  distributed vs local results of random nested-CTE/join queries). Flaky by nature; a random query\n  can trip `citus.max_intermediate_result_size` etc. **Tell:** one PG major fails while another PG\n  major of the SAME commit passes. Not a backport regression.\n\n**Root cause of the N-1 reds (not yours):** these base branches already carry in-flight\nminor-targeted features (e.g. clone-promotion sequence *re-ranging*;\n`citus_internal.delete_placement_metadata` behavior) whose NEW behavior lives only in the NEW lib.\nWhen the OLD lib (the previous minor's binary) is loaded on the coordinator, the expected `.out`\n(written for the new lib) mismatches → diff. Inherent to N-1 testing of in-flight features;\nunrelated to a feature backport that doesn't touch those areas.\n\n## How to PROVE a red is pre-existing (do this — gather the evidence)\n1. **Upstream base-branch run is itself failing:**\n   ```\n   gh run list --repo citusdata/citus --branch release-13.2 --workflow \"Build & Test\" --limit 6\n   ```\n   Note the base-branch runs are themselves `failure`.\n2. **Your failing set ⊆ upstream's failing set:**\n   ```\n   gh run view <upstreamRunId> --repo citusdata/citus --json jobs \\\n     --jq '.jobs[]|select(.conclusion==\"failure\")|.name'\n   ```\n   If your branch's failing job names are a subset, they're baseline, not yours.\n3. **The diff cites a pre-existing feature, not your object:** download your failed job's artifact\n   and read `regression.diffs`:\n   ```\n   gh run download <runId> --repo <your-fork>/citus -n \"<artifactName>\" -D <dir>\n   ```\n   Artifact naming: `<pgmajor>_<test>_[<citus_version>_]<libLabel>_<mode>`\n   (e.g. `18_check-enterprise_14.1-1_all` = SQL N-1; `..._v14.1.0_coordinatoronly` = Coordinator\n   N-1). Confirm the diff mentions the pre-existing feature (`re-ranged N sequence(s)`,\n   `delete_placement_metadata`) and has **ZERO** mentions of your backported symbol.\n4. **Best control:** run CI on your C-ONLY sibling backport (touches none of the SQL/dependency\n   code). If it shows the IDENTICAL N-1 reds, that confirms the reds track the base branch, not the\n   change.\n\n## What WOULD be your fault (fix these)\n- A red job whose `regression.diffs` names YOUR backported object **in a NORMAL (same-version)\n  job** — check-multi/-mx/-1 on a supported PG. (See the N-1 handling immediately below.)\n- A `Build for PGxx` with a real `error:` in its log.\n- A NEW job red that is GREEN on the upstream base branch. If it's a NEW-GUC/UDF N-1 red, the fix is\n  the schedule move below (not a test/output change); everything else, fix at the source.\n\nEverything else on these branches is baseline noise.\n\n## The NEW N-1 red a feature backport ADDS — move it to the N-1-excluded schedule\nBackporting a feature that adds a **new GUC or UDF** whose regression test runs under an N-1\nschedule (`multi_schedule` / `multi_1_schedule` → `check-multi`/`check-multi-1`) makes the **N-1\n(mixed-version)** jobs show that test as a NEW red. The red itself is EXPECTED graceful-degradation\n(not a defect), but the CLEAN fix is to relocate the test so N-1 never runs it — do NOT mangle the\nreviewed test, fabricate output, or just \"leave + document\". Proven on both the 13.2 and 14.0\nbackports of the `citus.allow_unsafe_insert_select_pushdown` GUC:\n- Test `allow_unsafe_insert_select_pushdown` (backported GUC `citus.allow_unsafe_insert_select_pushdown`).\n- Under N-1 the OLD lib runs on the coordinator (`lib-v13.3.0` / `lib-v14.1.0`), which lacks the GUC.\n  `SET citus.allow_unsafe_insert_select_pushdown` → `ERROR: invalid configuration parameter name ...\n  \"citus\" is a reserved prefix`, then the EXPLAIN falls back to the DEFAULT (pull-to-coordinator) plan.\n- **That fallback IS the N-1 contract** (old binary → GUC absent → safe pre-feature behavior); the diff\n  *demonstrates* compatibility. But a red CI job is still noise, so move the test off the N-1 matrix.\n\n**THE FIX: move the test line from its schedule into the placeholder section of\n`multi_1_create_citus_schedule`.** The N-1 jobs' `make_targets` (build_and_test.yml Coordinator/Lib\nN-1 blocks) list `check-multi`, `check-multi-1`, `check-add-backup-node`, ... but **NOT\n`check-multi-1-create-citus`**, so any test in `multi_1_create_citus_schedule` is N-1-invisible while\nSTILL running under `check-multi-1-create-citus` on every normal PG job (coverage preserved). Steps:\n1. Remove `test: <name>` from `multi_schedule` (mind the surrounding blank lines / parallel-group\n   comments — don't strand a comment that referenced it).\n2. Append a block under the placeholder header in `multi_1_create_citus_schedule`\n   (`# Have this placeholder section at the end for the tests that need to be moved back ... to\n   prevent N-1 test failures for the features that are not present in the minor version.`), mirroring\n   the EXISTING `citus_cluster_changes_block` block right there — same `# ----------` fencing and\n   phrasing so new additions look like the existing file.\n3. `run_test.py` auto-discovers the new schedule (it prints `SCHEDULE: multi_1_create_citus_schedule`);\n   the `.out` file is schedule-independent, so NO `.out` regen is needed — it's a pure schedule move.\n\n**N-1 version label = the actual N-1 CI version = the PREVIOUS MINOR (it advances every release).**\nRead the authoritative number from `build_and_test.yml`'s N-1 jobs — the `citus_version:` (SQL /\nWorker / Coordinator N-1) and `citus_libdir:` (Lib / Coordinator N-1) pins, not from\nmaybe-stale comments at `multi_1_create_citus_schedule`. As of this writing: release-13.2 pins\n`citus_version: \"13.3-1\"` / `citus_libdir: v13.3.0` (its `default_version` is 13.4-1, so N-1 = 13.3-1);\nrelease-14.0 pins `14.1-1` / `v14.1.0` (default 14.2-1, N-1 = 14.1-1). The rule is simply\n`default_version` minus one minor, and it is bumped whenever the branch ships a new minor.\n\n- **Write the label with today's N-1 version.** For a GUC introduced in the branch's current\n  `default_version` (e.g. 13.4 on release-13.2), the N-1 binary is the previous minor (13.3-1) and the\n  GUC is absent there, so: `GUC introduced in 13.4; not present in the N-1 (13.3-1) Citus binary ...\n  Move back to multi_schedule at Citus <next major, 14>.` On release-14.0 the corresponding numbers\n  are 14.2 / N-1 14.1-1 / next major 15.\n- **Sanity gate:** the move is only warranted when the object is genuinely ABSENT in that N-1 version\n  (introduced in a minor strictly newer than N-1). If your object was introduced at or below N-1, the\n  N-1 binary already has it — no red, no move.\n\n- **Why a new UDF may NOT need this move** (asymmetry): a test already placed in\n  `multi_1_create_citus_schedule` (e.g. `citus_internal_distribute_object`) is ALREADY N-1-invisible,\n  so it never shows an N-1 red and needs no move. Grep which schedule your test is in first.\n- Distinguish this N-1 red from a genuine bug: a same-version job (no `lib-vX` / `_coordinatoronly` /\n  `_workeronly` / `_all` suffix on the artifact) naming your object IS yours — see the section above.\n- Shipping: fold the move into a not-yet-reviewed fix commit (amend + `--force-with-lease`), or add a\n  NEW appended commit if the tip has already been reviewed (plain push, no force).\n\n## Genuinely-new failures a backport WILL introduce on a release branch (and the fix)\nThe release branch supports an OLDER PG than `main` (e.g. release-13.2 still builds **PG15**), and it\nships **version-pinned `expected/*.out`** files. A backport that is green on `main` can therefore go\nred on the release branch for two reasons that ARE yours to fix (they are not baseline noise):\n\n### (a) A shared helper's SIGNATURE changed → version-pinned .out that prints the signature mismatches\nIf your backport chain brings a new/edited SQL helper whose *signature* changed (e.g.\n`explain_filter(text)` → `explain_filter(text, keep_numbers boolean DEFAULT false)`), any expected\n`.out` that prints that signature in a `CONTEXT:` / `PL/pgSQL function ... line N` / error line will\nmismatch. **`normalize.sed` normalizes only the line NUMBER (`line [0-9]+` → `line XX`), NOT the\nsignature text**, so the signature diff is real. Fix = regenerate/`sed`-update those signature\nstrings in the pinned `.out`. Find them all with `grep -rl \"helper_name(oldargs)\" expected/`.\n- **Do NOT mistake a required helper for \"over-reach\" and revert it.** If the backported *test*\n  calls the new form (`explain_filter($$...$$, true)`), the helper is REQUIRED; reverting it yields\n  `ERROR: function ...(unknown, boolean) does not exist`. Before deleting an \"extra\" hunk a\n  cherry-pick carried, check whether the backported test depends on it.\n\n### (b) PG-version planner divergence on the older PG the release branch still supports\nThe same EXPLAIN can differ by PG major. Real example: PG15 omits a functionally-redundant trailing\nsort column that PG16+ keeps — `Sort Key: q.batch` (PG15) vs `Sort Key: q.batch, q.text_id` (PG16+).\nrelease-13.2 hits it on the PG15 `check-multi`; main/release-14.0 never see it (no PG15).\nFix, in the order the repo prefers:\n1. **An explain helper from `main`'s `multi_test_helpers.sql`** — check whether one already\n   normalizes the divergent bit. As of 13.x/14.x none normalizes Sort Key (there are\n   `coordinator_plan`, `plan_normalize_memory`, `explain_with_pg16_subplan_format`,\n   `explain_with_pg17_initplan_format`, and boolean `explain_has_distributed_subplan` /\n   `explain_has_single_task` — the booleans would need an invasive test rewrite).\n2. **MOVE the diverging blocks into a PG-version-guarded test file (PREFERRED on release-13.2 when\n   the divergence is PG15-only).** release-13.2 ships `sql/pg16.sql`, which `\\q`-skips below PG16\n   (guard at top: `SELECT substring(:'server_version','\\d+')::int >= 16 ... \\if ... \\else \\q \\endif`)\n   AND already lives in `multi_1_create_citus_schedule` (the N-1-excluded schedule). So relocating the\n   PG15-diverging EXPLAIN cases out of the normal test (e.g. `allow_unsafe_insert_select_pushdown.sql`)\n   into a self-contained section appended to `pg16.sql` means: PG15 never runs them (no diff),\n   PG16/17/18 still cover them, and — bonus — they leave the N-1 matrix too. Recipe:\n   - Cut the exact diverging blocks from the source `.sql`; fix any dangling comment they leave behind.\n   - Append AFTER `pg16.sql`'s final `DROP SCHEMA pg16 CASCADE;` a self-contained section: its OWN\n     schema (e.g. `allow_unsafe_insert_select_pushdown_pg16`), a distinct `next_shard_id`\n     (e.g. 14100000) to avoid shard-id clashes, recreate the tables/functions the blocks need, the\n     moved EXPLAIN blocks, then `DROP SCHEMA ... CASCADE;`. It must not depend on the parent test's\n     schema (schedules run independently).\n   - Regenerate BOTH `.out`s (source shrinks, `pg16.out` grows) via the runner; never hand-edit.\n3. Add a new **`normalize.sed` rule** or update an existing one to generalize the coverage, or add an\n   alternative expected file as a last resort.\n\n### (c) Flaky detector goes all-red because your changed-file set is widely depended-on\n\"Test flakyness (1..32)\" computes a changed-test set by DEPENDENCY EXPANSION of the PR's changed\nfiles. Editing a widely-depended helper/migration (`multi_test_helpers.sql`, a migration `.sql`,\n`multi_extension.sql`) balloons the set across every shard and drags in inherently-flaky base tests\n(`start_stop_metadata_sync`, `multi_metadata_sync`, ...). The shards only settle once your\n*actually-changed* tests pass; the base-flaky tests pulled in are not yours to fix. Sanity check with\n`gh pr view <n> --json files` — a helper appearing in the flaky `tests=` list but NOT in the PR's\nfile list is pure dependency-expansion, not a change you made.\n\n**Do NOT hand-wave \"it's flaky\".** Two pieces of hard evidence, both cheap:\n1. **Read the actual diff** from the shard's uploaded artifact (they DO upload one, named\n   `test_flakyness_parallel_<N>`; see fetch recipe below). Confirm the failing test is a base test\n   you did not touch AND the diff contains ZERO of your backported symbols. The signature of\n   `--use-whole-schedule-line` running the whole schedule line in parallel so other tests consume\n   colocation IDs first. Scan: `grep -c <your_symbol> regression.diffs` (want 0),\n   `grep -oE \"expected/[a-z_0-9]+[.]out\"` (only the base test).\n2. **Reproduce on the PRISTINE base** to prove pre-existing, not introduced. From `src/test/regress`:\n   `python citus_tests/run_test.py <flaky_test> --repeat 8 --use-whole-schedule-line` on your branch\n   head, then `git checkout <merge-base> && make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install` and run the SAME command. If the\n   base fails the identical N/N under the identical flags, it's 100% pre-existing (checkout your\n   branch + rebuild to restore afterward). Example proven this way: `start_stop_metadata_sync` failed\n   8/8 on both `bp-release-13.2` tip and its pristine base, same colocationid/FK diff — not the\n   backport's fault. NOTE the flaky detector's PG major varies — check the shard log's `PG_MAJOR:`\n   line and reproduce on the SAME PG for an exact comparison.\n\n### Shipping the fix\nOne NEW fix commit on top of the reviewed tip (do not rewrite reviewed SHAs). If a prior *bad* fix\ncommit is already on the remote, REPLACE it with `git push --force-with-lease=<branch>:<badsha>`\n(the reviewed commits below it stay). Force-push freely only while the commits are still unreviewed.\n\n## Fetching CI diffs reliably (tooling gotchas)\n- `gh run view --job <id> --log-failed` streams often die with `stream error: stream ID 1; CANCEL` —\n  retry or avoid.\n- Jobs that upload artifacts (`NN_check-multi`, `NN_check-operations`, AND the flaky shards):\n  list with a SINGLE page, not `--paginate` (`--paginate` concatenates JSON pages so a single `--jq`\n  chokes with \"Additional text after finished reading JSON content\"):\n  `gh api \"repos/citusdata/citus/actions/runs/<run>/artifacts?per_page=100\" --jq '.artifacts[]|select(.name==\"15_check-multi\").id'`\n  then `gh api .../actions/artifacts/<id>/zip > x.zip` (this redirect is fine for the small per-artifact\n  zip; read `regression.diffs` from it).\n- Flaky-detector jobs DO upload a named artifact `test_flakyness_parallel_<N>` (contains the shard's\n  `regression.diffs` under `__w/citus/citus/src/test/regress/`) — fetch it like the per-artifact zip\n  above; that's the reliable way to read the flaky diff body. The per-job logs endpoint only NAMES the\n  failing test (`SCHEDULE_LINE:` / `not ok`) and points at regression.diffs in the artifact.\n- Do NOT re-encode a downloaded zip through a shell pipe that changes encoding (corrupts it); write it\n  to disk directly. The whole-run `/logs` zip often arrives partial (\"End of Central Directory record\n  could not be found\") — prefer the per-job / per-artifact endpoints above.\n\n## Mechanics / gotchas\n- Overall run status stays `queued`/`in_progress` until EVERY matrix job ends; per-job logs and\n  finished-job artifacts are still fetchable mid-run (above).\n- Dispatching several full matrix runs (~190 jobs each) saturates the fork runner pool and starves\n  the job you actually care about; `gh run cancel <id>` the fully-triaged runs to free capacity for\n  a critical re-verification.\n","isInternal":false,"tokens":5018,"sizeBytes":18641},{"name":"manual-upgrade-testing.md","path":".github/skills/citus-backport/references/manual-upgrade-testing.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/skills/citus-backport/references/manual-upgrade-testing.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Manual MVU (major-version-upgrade) testing for backport branches\n\nBackporting a SQL-schema change and propagating the upgrade ladder (see\n`sql-schema-backport.md` §Step 5) only *claims* MVU safety. This doc is how you **prove** it\nend-to-end on a real multi-node cluster: stand up N nodes at an OLD Citus major, then walk the\nextension forward across both backport branches — and, one hop further onto the\n`main`-targeting PR branch — with `ALTER EXTENSION citus UPDATE`, capturing `citus_version()` AND\n`pg_extension.extversion` on every node after every hop.\n\nThis is the \"manual testing for backport branches\" step. Do it after CI is green, before\ndeclaring a SQL-schema backport family done. A C-only backport does NOT need it.\n\n## What this proves\n- A cluster shipped at the 13.x tail (`default_version` of `release-13.2`, e.g. `13.4-1`) can\n  still cross to 14.x — i.e. a future managed-service 13→14 MVU won't strand the customer.\n- **Final hop:** a 14.x-tail cluster (e.g. `14.2-1`) can cross to the NEXT major on\n  `main` (`15.0-1`) via the PR branch you opened against `main` — proving the SAME object also\n  rides a future 14.x→15.0 MVU. `main` jumps `14.0-1 → 15.0-1` directly (it has NO 14.1/14.2\n  *upgrade-in-line*), but it MUST carry the `14.2-1--14.1-1` and `14.1-1--14.0-1` **downgrades**\n  so PG can route `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. If that route is missing you get \"no\n  update path from 14.2-1 to 15.0-1\" — a real MVU gap in the ladder propagation to `main`.\n- The forward/downgrade **ladder** you propagated actually routes end to end.\n- The backported object (UDF/GUC/column) is present and the cluster is functional after each jump.\n\n## The single hard requirement: ONE PG, three Citus builds\n`ALTER EXTENSION citus UPDATE` walks the version graph made of the `citus--A--B.sql` scripts\n**installed in the current PG's `SHAREDIR/extension/`**. So all three Citus versions in the walk\n(old baseline + both backport branches) must be installed, one at a time, into the SAME PG\ninstall / SAME cluster data dirs. You swap the installed Citus between hops and restart the nodes;\nthe catalog stays put and `ALTER EXTENSION` migrates it.\n\nUse ONE dedicated PG install as that single PG (its Citus gets overwritten repeatedly — that's the\npoint — then you restore it at the end, §Cleanup). Do the builds in **detached worktrees** so your\nreviewed branch checkouts are never disturbed.\n\n### `make install-all`, NOT `make install` — MANDATORY here\n`make install` lays down only **forward** scripts. The cross-major route from the 13.x tail is\n**not** a straight climb — there is no `13.4-1--14.x` edge. It goes:\n\n```\n13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1\n  (down)   (down)   (BRIDGE up)  (up)    (up)\n```\n\nso it needs the **downgrade** scripts `13.4-1--13.3-1`, `13.3-1--13.2-1` and the cross-major\n**bridge** `13.2-1--14.0-1` (all shipped under `sql/downgrades/` on `release-14.0`). `make install`\nomits every downgrade → `ALTER EXTENSION citus UPDATE` fails with \"no update path from 13.4-1 to\n14.2-1\". Always build the test PG with **`make install-all`** (it runs `install` +\n`install-downgrades` for both the distributed and columnar extensions).\n\n### PG version must be common to all builds\nBaseline Citus must support the chosen PG. 13.0/13.2 support PG15/16/17; 14.0 supports 16/17/18;\n`main` supports 16/17/18 → **PG 17** is the common denominator for all four. Pick a PG 17 install\nso it works for the whole 13.0→13.4→14.2→15.0 walk.\n\n## Recipe (verified with a 3-node cluster: citus 13.0.5 → 13.4.0 → 14.2.0 → 15.0devel)\n\nPick a shared PG (`SHARED=<path to your PG17 install>`), a spare coordinator port (e.g. `10600`;\nworkers take `+1/+2`), and the three source trees. Use detached worktrees so the reviewed\nbranch worktrees are never disturbed:\n\n```bash\ncd <main clone>            # holds all the branches/tags\ngit worktree add --detach /tmp/mvu/wt130 v13.0.5            # OLD baseline (a real tag)\ngit worktree add --detach /tmp/mvu/wt132 <bp-release-13.2-head-sha>\ngit worktree add --detach /tmp/mvu/wt140 <bp-release-14.0-head-sha>\ngit worktree add --detach /tmp/mvu/wtmain <bp-main-PR-head-sha>\n```\n\nBuild helper — reconfigure each tree against the SHARED PG and `install-all`:\n\n```bash\nbuild() {  # build <worktree> ; installs into $SHARED\n  cd \"$1\"\n  ./configure PG_CONFIG=\"$SHARED/bin/pg_config\" >/tmp/cfg.log 2>&1\n  local JOBS; JOBS=$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)   # portable: Linux || macOS\n  make -sj\"$JOBS\"             >/tmp/mk.log  2>&1\n  make -sj\"$JOBS\" install-all >/tmp/ins.log 2>&1\n  ls \"$SHARED/share/extension/\" | grep -E '^citus--1[3-5]' | sort   # sanity: see the ladder\n}\n```\nRe-use the same cluster data dirs across hops:\n\n1. **Baseline 13.0.** `build /tmp/mvu/wt130`. Stand up a cluster on `$SHARED` with `citus_dev`\n   (it `CREATE EXTENSION citus` at `default_version` = the currently-installed `13.0-1`). Verify\n   `SELECT extversion FROM pg_extension WHERE extname='citus'` and `SELECT citus_version()` = 13.0\n   on coordinator + both workers. (`citus_dev`'s default DB is `$USER`, NOT `postgres`.)\n2. **Up to the 13.2 branch.** `build /tmp/mvu/wt132` (overwrites Citus in `$SHARED`, now\n   `default_version = 13.4-1`, installs the 13.x up+down ladder). **Restart every node** so the new\n   `.so` loads. Then, on EACH node: `ALTER EXTENSION citus UPDATE;` → catalog goes `13.0-1 → 13.4-1`\n   in one call (walks the whole chain). Capture `citus_version()` per node — expect\n   `13.4.0 ... (sha: <13.2-head>)`.\n3. **Cross-major to the 14.0 branch.** `build /tmp/mvu/wt140` (`default_version = 14.2-1`, installs\n   the full up+down+bridge ladder). **Restart every node.** On EACH node: `ALTER EXTENSION citus\n   UPDATE;` → routes `13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1`. Capture\n   `citus_version()` per node — expect `14.2.0 ... (sha: <14.0-head>)`.\n4. **Functional check** on the upgraded cluster (coordinator): active `pg_dist_node`,\n   `create_distributed_table` + INSERT + distributed `count(*)`, `pg_dist_shard` count, and that\n   the backported object is present (e.g. `SELECT proname FROM pg_proc WHERE proname =\n   'distribute_object' AND pronamespace = 'citus_internal'::regnamespace;`).\n5. **One more hop onto the `main` PR branch.** Detached worktree at the `bp-<pr>-main`\n   head, `build` it into the SAME `$SHARED` (overwrites Citus to `default_version = 15.0-1`,\n   installs main's up+down ladder incl. the 14.x downgrades). **Restart every node.** On EACH node:\n   `ALTER EXTENSION citus UPDATE;` → routes `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. Re-run the\n   functional check. Expect `Citus 15.0devel ... (sha: <main-PR-head>)` and the backported object\n   still present at `15.0-1`.\n\n**KEY NUANCE — `citus_version()` reports the BINARY, `extversion` reports the CATALOG.** After you\ninstall a newer Citus + restart (but BEFORE `ALTER EXTENSION citus UPDATE`), `citus_version()`\nalready prints the new version (it is compiled into the `.so`), while `pg_extension.extversion` is\nstill the old catalog version. The migration is only proven by **`extversion` moving to the new\n`default_version`**. On the first hops the jump is a whole major so `citus_version()` alone is\ntelling; on the `14.2 → 15.0devel` hop both 14.2 and 15.0 builds may print `15.0devel`-ish strings,\nso capture `extversion` before+after (`14.2-1 → 15.0-1`) as the real evidence.\n\nRun the same `ALTER EXTENSION citus UPDATE` on each node by connecting to each port directly\n(coordinator does NOT auto-propagate the extension version to workers) — that's the \"update all\nnodes\" the ask means.\n\n## Environment notes (neutral, but they save time)\n- **Put SQL in `.sql` files and run `psql -f`; put shell logic in `.sh` files and run them.**\n  Piping SQL / loops inline through several shell layers (esp. cross-OS terminals) mangles quoting\n  and `$USER`. If a script is authored on a different-newline OS, strip CRLF first (`sed -i\n  's/\\r$//' file`) or `set -o pipefail` may die with \"invalid option name\".\n- **`citus_dev` records RELATIVE data-dir paths** (`cluster/<role>`). `pg_ctl -D <abs>` then fails\n  with \"could not access directory …/cluster/<role>\". Fix: `cd` into the clusters parent and use\n  `pg_ctl -D cluster/<role> …`.\n- **If `pg_ctl -w start` blocks**, the `-w` readiness probe can hang against an `ssl=on` citus_dev\n  config even though the server logged \"ready to accept connections\". Start WITHOUT `-w` and poll\n  the TCP port yourself (`ss -ltn | grep :<port>`). `-w` on **stop** is fine.\n- On slow disks the first post-restart checkpoint can take ~100s; judge node readiness by\n  \"database system is ready to accept connections\" in the node logfile + the port listening, not by\n  a `pg_ctl -w` wrapper returning.\n\n## Cleanup / restore\n- The shared PG's Citus was left at whatever you installed last (after the final main hop that\n  is main's `15.0-1`). **Restore the env's own build:** rebuild the PG's normal Citus branch with\n  `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install-all` (verify `default_version` + the ladder edges are back).\n- Stop the test cluster (`pg_ctl -D cluster/<role> -w -m fast stop` per node, or your cluster\n  helper's stop command) and `git worktree remove` the throwaway build trees (+ `git worktree\n  prune`) when done.\n\n## Expected evidence shape (what a PASS looks like)\n```\nBASELINE (citus 13.0):    all nodes → extversion 13.0-1  | Citus 13.0.5 ... gitref: v13.0.5\nafter 13.2-branch UPDATE: all nodes → extversion 13.4-1  | Citus 13.4.0 ... (sha: <13.2 head>)\nafter 14.0-branch UPDATE: all nodes → extversion 14.2-1  | Citus 14.2.0 ... (sha: <14.0 head>)\nafter main-PR   UPDATE:   all nodes → extversion 14.2-1→15.0-1 | Citus 15.0devel ... (sha: <main head>)\nfunctional (each hop): 3 active nodes, create_distributed_table OK, count matches, backported obj present\n```\nReport the per-node `extversion` (before+after) AND `citus_version()` for every hop — not a\nsummary — since the whole point is proving every node crossed each boundary.\n","isInternal":false,"tokens":3089,"sizeBytes":10111},{"name":"sql-schema-backport.md","path":".github/skills/citus-backport/references/sql-schema-backport.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/skills/citus-backport/references/sql-schema-backport.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# SQL-schema backport recipe (the hard case)\n\nA C-only cherry-pick usually just applies. The moment a commit touches the SQL schema (new UDF,\nnew column, new migration step) you must **remap it to the target branch's real version**, because\n`main`'s migration files are named for the NEXT major (e.g. `15.0-1`) and the release branch is on\na different `default_version`. Do NOT accept `main`'s files as-is.\n\n## Step 0 — read the target version (drives everything)\nOn each target branch read `src/backend/distributed/citus.control`:\n\n```\ndefault_version = '<X.Y-Z>'\n```\n\nThen find that branch's current dev migration + downgrade under\n`src/backend/distributed/sql/`:\n- upgrade:   `citus--<prev>--<default_version>.sql`\n- downgrade: `citus--<default_version>--<prev>.sql`\n\nWorked example (versions as of this writing):\n\n| Branch        | default_version | dev upgrade migration        | dev downgrade                | udf versioned file |\n|---------------|-----------------|------------------------------|------------------------------|--------------------|\n| release-13.2  | `13.4-1`        | `citus--13.3-1--13.4-1.sql`  | `citus--13.4-1--13.3-1.sql`  | `13.4-1.sql`       |\n| release-14.0  | `14.2-1`        | `citus--14.1-1--14.2-1.sql`  | `citus--14.2-1--14.1-1.sql`  | `14.2-1.sql`       |\n| main          | `15.0-1`        | `citus--14.0-1--15.0-1.sql`  | `citus--15.0-1--14.0-1.sql`  | `15.0-1.sql`       |\n\n## Step 1 — remap the migration files\nThe main commit added its schema hunk to `citus--14.0-1--15.0-1.sql` (and the DROP to the matching\n`15.0-1--14.0-1` downgrade), plus a versioned udf file `sql/udfs/<name>/15.0-1.sql`. On the target\nbranch:\n\n1. `git rm` the main-only `15.0-1` upgrade + downgrade migration files if the cherry-pick created\n   them (they don't belong on the release branch).\n2. `git mv src/.../sql/udfs/<name>/15.0-1.sql  src/.../sql/udfs/<name>/<default_version>.sql`\n   (rename the versioned udf file to the branch's version).\n3. Add the `#include \"udfs/<name>/<default_version>.sql\"` line to the branch's **dev upgrade\n   migration** (`citus--<prev>--<default_version>.sql`) — same include the main commit added to\n   `14.0-1--15.0-1.sql`, just in the branch's file.\n4. Add the reverse `DROP FUNCTION IF EXISTS <fully-qualified-signature>;` (or DROP for whatever\n   object) to the branch's **dev downgrade** (`citus--<default_version>--<prev>.sql`).\n5. Keep `sql/udfs/<name>/latest.sql` — refresh it to identical content as the versioned file.\n\n`ci/check_migration_files.sh` (\"Check for missing downgrade scripts\") gates step 4 — every upgrade\nstep needs a matching downgrade. The CI `check-sql-snapshots` job validates these manual edits, so\ngreen there is strong confirmation the remap is correct.\n\n## Step 2 — fix `multi_extension.out`\n`multi_extension` walks the whole upgrade+downgrade ladder and prints a \"Snapshot of state at\n`<version>`\" section per version. The new object's row must land under the **target branch's**\nversion section (e.g. `13.4-1`), NOT main's `15.0-1`. The snapshot table's column widths are\ndynamic — **regenerate the .out with the runner, don't hand-align** it. From `src/test/regress`:\n\n```\npython citus_tests/run_test.py multi_extension\n```\n\nThe new line typically shows the new function/object with `(1 row)` (or the appropriate count) in\nthe target-version section.\n\n## Step 3 — fix `upgrade_list_citus_objects.out` (edit BY CONSTRUCTION)\nThis test **cannot be run standalone** via `run_test.py` — it depends on a `citus_schema` created\nby an earlier test in `after_pg_upgrade_with_columnar_schedule`; run alone it dies with\n`ERROR: no schema has been selected to create in`. So validate its edit by construction:\n- the new function line auto-merges alphabetically in the cherry-pick (leave it);\n- fix only the trailing **row COUNT** = base_count + (number of new citus objects). Read the base\n  count from the branch's own expected file. Observed: release-13.2 `378 → 379`, release-14.0\n  `384 → 385` for one new UDF.\n\n## Step 4 — build WITH downgrades, then test\n`make install` does NOT install downgrade scripts, and `multi_extension` exercises the downgrade\npath, so after every rebuild run both:\n\n```\nmake -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install\nmake -C src/backend/distributed install-downgrades\nmake -C src/backend/columnar    install-downgrades\n```\n\n(`columnar` is a SEPARATE extension with its own downgrades; omitting it surfaces as a\n`citus_columnar` \"no update path\" error. `make install-all` runs both.)\n\nThen run the feature's own test + `multi_extension`.\n\n## Step 5 — propagate the upgrade ladder ACROSS branches (MVU safety)\n\nBackporting the object DOWN to a release branch is not enough. An MVU replays the migration\n**ladder**, so a new SQL object must be reachable on EVERY higher branch's ladder too — otherwise\na cluster upgraded 13.x→14.y would silently lose it. Net effect: **the same object is defined\nMULTIPLE times per branch, once per major's introduction step.** This is intentional, not\nduplication to \"fix\".\n\n**Topology** (worked example: `citus_internal.distribute_object`, introduced on main at\n`14.0-1--15.0-1`, backported to 13.2 at `13.3-1--13.4-1` and to 14.0 at `14.1-1--14.2-1`):\n\n| Ladder step (+ its downgrade + versioned udf) | release-13.2 | release-14.0 | main |\n|---|---|---|---|\n| `13.3-1--13.4-1` (udf `13.4-1`) — 13.x introduction | ✅ (the backport itself) | ✅ copy here | ✅ copy here |\n| `14.1-1--14.2-1` (udf `14.2-1`) — 14.x introduction | n/a | ✅ (the backport itself) | ✅ copy here |\n| `14.0-1--15.0-1` (udf `15.0-1`) — main's own top    | n/a | n/a | ✅ already there (orig land) — LEAVE UNTOUCHED |\n\nRule of thumb: **a step introduced on major N must be copied onto every branch for major ≥ N.**\n13.x paths → release-14.0 + main; 14.x paths → main; main's own top path stays as merged.\n\n**How to do it (per higher branch, e.g. release-14.0 then main):**\n1. This is a **SEPARATE commit** on top of the (unchanged) cherry-pick — never amend the\n   cherry-pick. On main it's its own PR.\n2. Copy the step + downgrade + versioned-udf files **byte-for-byte** from the branch that owns\n   the introduction. `git show <branch>:<path>` reads any branch's blob without checking it out\n   (worktrees of one clone share an object store, so this is trivial):\n   ```\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/citus--13.3-1--13.4-1.sql          > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/downgrades/citus--13.4-1--13.3-1.sql > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/udfs/<name>/13.4-1.sql             > <same path>\n   ```\n   (and the analogous `14.1-1--14.2-1` trio from `bp-<pr>-release-14.0` when copying onto main.)\n3. **Keep the header verbatim** — even the `-- bump version to 13.4-1` line stays on downstream\n   branches where 13.4 is not the default. Confirmed identical across branches in the #8621 UDF family.\n4. **Purely additive.** Do NOT touch `citus.control` / `default_version`. Do NOT touch the\n   `13.2-1--14.0-1` major-bridge migration. main's `latest.sql` still points at its own top\n   version (15.0-1); leave it. `check_sql_snapshots.sh` verifies latest.sql == highest udf.\n5. New step files contain ONLY the object being propagated (header + single `#include`); the\n   downgrade is header + `DROP FUNCTION IF EXISTS <signature>;`. Don't drag along unrelated objects\n   that happen to share the source branch's step file.\n6. Build + `install-downgrades` + `multi_extension` on the higher branch (expect PASS, usually no\n   `.out` change needed since the new sections are additive). Run the three migration CI checks\n   locally before pushing: `ci/check_migration_files.sh`, `ci/check_sql_snapshots.sh`,\n   `ci/disallow_c_comments_in_migrations.sh` (all must exit 0).\n7. **Add `multi_extension.sql` ladder-test coverage for the new step** (or prove it's already\n   covered) — see the dedicated section below.\n\n**Template PR family** (study these to see the end-state, don't reverse-engineer from one PR):\nfind the original feature PR, then every PR that mentions it in title/desc. Worked example — the\n`citus_internal.distribute_object` repair UDF, introduced on main in **#8621**: backported\none-branch-per-target via **#8684**→release-13.2 and **#8685**→release-14.0 (each branch also carries\nthe sibling GUC PR **#8625** `citus.allow_unsafe_insert_select_pushdown`, unsquashed as its own\ncommit), with the 13.x/14.x upgrade paths propagated to `main` in **#8686**. The main ladder PR\n(#8686, \"Add 13.x/14.x upgrade paths for citus_internal.distribute_object to main\") is the model for\nthe main ladder commit's title/body; it's internal plumbing so usually **no `DESCRIPTION:` line**.\n\n## multi_extension.sql ladder tests for new steps (off-walk detour pattern)\n\nAdding a ladder step (Step 5 backport OR Step 5 propagation) is not fully covered until\n`src/test/regress/sql/multi_extension.sql` exercises it — or you can prove the existing walk already\ndoes. Model the additions on the template PR family, adapted per branch.\n\n- **The test walks ONE path only** — the main upgrade path (`13.2 → 13.3 → 14.0 → 14.1 → 15.0`).\n  A step you add for MVU may sit **off** that walk:\n  - the 13.x maintenance-tail step `13.3-1--13.4-1` (off-walk on release-14.0 and main), and\n  - the 14.x maintenance-tail step `14.1-1--14.2-1` (off-walk on **main**, whose walk jumps 14.1 → 15.0).\n  Off-walk steps' upgrade AND downgrade scripts are otherwise **never executed** by the test.\n- **On-walk / top-of-branch steps need NOTHING extra.** A branch's own `default_version` (13.4-1 on\n  release-13.2, 14.2-1 on release-14.0, 15.0-1 on main) is already the target of the existing\n  `-- Snapshot of state at X` block; the propagated object simply appears in that snapshot's `.out`.\n  ⇒ release-13.2 needed **no** `multi_extension.sql` change; release-14.0 needed **one** detour\n  (13.4-1); main needed **two** (13.4-1 and 14.2-1).\n- **PATTERN = round-trip no-op → snapshot at the detour → rejoin the walk FROM the detour.** Match\n  the shape of the existing on-walk blocks — visit the off-walk version the same way every other\n  version is visited, so the additions look like they belong. Do NOT add editorial parentheticals\n  (no \"(13.x maintenance line, off the ... path)\"). At the point the walk sits at the detour's\n  PARENT, insert two blocks AND retarget the *existing* next round-trip so it bounces off the\n  detour instead of the parent:\n  ```sql\n  -- Test downgrade to <parent> from <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';   -- e.g. 13.4-1\n  ALTER EXTENSION citus UPDATE TO '<parent>';   -- e.g. 13.3-1\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Snapshot of state at <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Test downgrade to <detour> from <next>   <-- was \"<parent> from <next>\"; retarget it\n  ALTER EXTENSION citus UPDATE TO '<next>';    -- the next ON-WALK version (e.g. 14.0-1 / 15.0-1)\n  ALTER EXTENSION citus UPDATE TO '<detour>';  -- bounce back to the detour, NOT the parent\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n  ```\n  Block 1 (round-trip) proves the up+down pair is a perfect no-op. Block 2 (snapshot) shows the new\n  object actually appearing at the detour version (`+ function citus_internal.distribute_object...`,\n  `(1 row)`). The snapshot leaves the walk sitting at `<detour>` with the `print_extension_changes()`\n  baseline `= {detour}`. Instead of adding a separate \"return to parent\" block, you **retarget the\n  existing next round-trip** (which used to bounce `<parent> ↔ <next>`) so it bounces `<detour> ↔\n  <next>` — this keeps the walk moving forward from the detour and lands it back at `<detour>` with\n  baseline reset to `{detour}`.\n- **The one intended downstream change:** because the walk now enters the following `-- Snapshot of\n  state at <next>` block with baseline `{detour}` (not `{parent}`), that ONE snapshot shifts by\n  exactly the detour's object, and nothing else changes:\n  - If `<next>` does NOT contain the object (e.g. `14.0-1`, a 13.x-line object): the snapshot GAINS a\n    `previous_object` row (the object shown as dropped when leaving the maintenance line).\n  - If `<next>` DOES contain the object (e.g. `15.0-1` when the object is already at `14.2-1`): the\n    snapshot LOSES the `current_object` row it used to show (it's no longer newly-added — it was\n    already displayed at the detour snapshot).\n  Both are self-consistent and desirable: the object is introduced exactly once, at its detour\n  snapshot. Every block AFTER that `<next>` snapshot stays byte-identical (print resets baseline to\n  `{next}` regardless). Regenerate `.out` and eyeball that only that single snapshot moved.\n- **`.out` regen:** edit `.sql` → runner \"fails\" (writes `results/multi_extension.out`) →\n  `cp results/multi_extension.out expected/multi_extension.out` → re-run → \"All 1 tests passed\".\n  **Never hand-edit `.out`.**\n- **`diff -w` header gotcha:** pg_regress compares with `diff -dU10 -w`, so snapshot-table header\n  column widths / leading spaces are IGNORED for pass/fail. A regen may normalize a pre-existing\n  HAND-EDITED narrow header (left over from an earlier `.out` hand-edit) to the runner's canonical\n  wider spacing. Whitespace-only and harmless — **keep the canonical runner output**, note it in the\n  commit message. Prove it's pre-existing: run the test on pristine HEAD (no change) — same wide\n  header, still PASSES.\n- **APPEND-ONLY once reviewed:** if the backport branch/PR has already been reviewed, do NOT\n  force-push — add the `multi_extension` test work as a new commit on top.\n\n## Flaky / gotcha notes for SQL backports\n- **`multi_extension` flaky maintenance-daemon crash:** the issue #3409 `DROP EXTENSION` under a\n  non-superuser role section can report \"server process was terminated\" once, then PASS on a clean\n  re-run. If the PG log shows only a clean fast-shutdown (no PANIC / segfault / Assert), it's\n  timing/flaky — re-run before investigating. Sanity-check by running the same test on the PRISTINE\n  base branch.\n- **`configure` gets regenerated by builds** — never part of a backport. `git checkout -- configure`\n  before committing.\n- **`git stash -u` mid-cherry-pick** drops `CHERRY_PICK_HEAD` but keeps the file changes; on restore\n  you must `git commit` manually and re-add the original message + `(cherry picked from commit <sha>)`.\n- **Regenerate .out with the runner, never hand-edit** — snapshot column widths and object counts\n  are computed.\n","isInternal":false,"tokens":4168,"sizeBytes":14848},{"name":"SKILL.md","path":".github/skills/citus-backport/SKILL.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/skills/citus-backport/SKILL.md","title":"citus-backport","category":"anthropic-skill","format":"markdown","content":"---\nname: citus-backport\ndescription: >-\n  Backport one or more merged citusdata/citus `main` commits/PRs onto the active release\n  branches. Covers the release model, cherry-picking, remapping SQL schema changes to each\n  branch's default_version, the SQL upgrade/downgrade ladder for Major-Version-Upgrade safety,\n  running regression tests, one clean commit per PR per branch, and triaging release-branch CI\n  (pre-existing baseline reds vs reds the backport introduced). USE WHEN asked to backport / port\n  / cherry-pick a change to release-13.2 / release-14.0 (or the newest two majors), when a\n  backport hits a SQL migration / udf / multi_extension conflict, when a \"trivial C-only\" backport\n  fails to compile on an older PG the release branch still supports (e.g. PG15 `rteperminfos`),\n  when release-branch CI is red and you must prove which reds are pre-existing, or when you must\n  prove a SQL-schema backport is Major-Version-Upgrade-safe via `ALTER EXTENSION citus UPDATE`\n  across a real multi-node cluster. DO NOT USE for non-Citus repos, new features, or PR review.\nlicense: See the repository LICENSE file.\n---\n\n# Backport a Citus change to the release branches\n\nThis skill encodes how to backport merged `main` commit(s) to Citus release branches: the release\nmodel, the SQL-schema remap recipe, the PG-version compatibility traps, and how to triage the\nrelease branches' baseline-red CI. Follow the repository's normal build/test/style conventions\n(see `CONTRIBUTING.md` and `src/test/regress/README.md`) on top of the guidance here.\n\n## Reference deep-dives (load on demand)\nAll live next to this file under `references/`:\n- **`references/sql-schema-backport.md`** — the hard case: a commit that changes the SQL schema\n  (new UDF / column / migration). Exact file-by-file remap recipe, the `multi_extension.out` and\n  `upgrade_list_citus_objects.out` edits, the `install-downgrades` trap, and the cross-branch\n  **ladder-propagation** step (copy the introduction paths UP to every higher major for MVU safety).\n- **`references/ci-triage.md`** — reading backport CI: the PG-version compile traps (main drops\n  old PGs the release branch keeps), the N-1 mixed-version jobs, and the step-by-step proof that a\n  red job is pre-existing baseline noise vs a real regression you introduced.\n- **`references/manual-upgrade-testing.md`** — the manual MVU proof: stand up a multi-node cluster\n  at an OLD Citus major, then `ALTER EXTENSION citus UPDATE` across the backport branches (13.x\n  tail → cross-major → 14.x tail), capturing `citus_version()` + `pg_extension.extversion` per node\n  per hop. The `make install-all` / single-shared-PG requirement.\n\nA C-only backport usually needs only this hub + `ci-triage.md`. A SQL-schema backport needs all\nfour (add `manual-upgrade-testing.md` as the final proof once CI is green).\n\n## Release model (internalize this first)\n- **ONE branch per MAJOR.** `release-13.2` serves the WHOLE 13.x line; `release-14.0` serves ALL of\n  14.x; `main` is the next major. Citus no longer cuts a branch per minor (this changed ~2026-05).\n- **Backport FEATURES, not just bugfixes**, to the **newest two majors** (13 & 14 as of this\n  writing).\n- **Branch names LIE about the version.** `release-13.2` currently ships 13.4; `release-14.0` ships\n  14.2. **NEVER infer the target minor from the branch name** — read `default_version` from\n  `src/backend/distributed/citus.control` on each target branch. That value drives every SQL edit.\n- **N-1 compatibility is required and CI-gated.** A backport is N-1-safe when it is purely\n  ADDITIVE (new UDF / GUC / column) and does not change an existing UDF signature or wire format\n  used cross-version. If it's additive, it's safe. See `references/ci-triage.md` for the jobs.\n\n## Workflow (per PR, per target branch)\n1. **Use an isolated build per target major.** A dedicated checkout/worktree + its own PostgreSQL\n   install per release branch keeps builds/tests from clobbering each other and your main checkout.\n   Two worktrees of the same clone SHARE one git object store — all branches are visible/pushable\n   from either, and a `+`-prefixed branch in `git branch` is checked out in a sibling worktree.\n2. **Make the SHAs reachable**, then `git checkout -b bp-<pr>-<release-branch> origin/<release-branch>`\n   and `git cherry-pick -x <sha>` (the `-x` records `(cherry picked from commit <sha>)`).\n3. **Resolve:**\n   - **C-only:** usually applies clean or with trivial context conflicts. THEN check PG-version\n     compat (`references/ci-triage.md` §PG-version) — the #1 non-obvious C-backport failure.\n   - **SQL-schema:** migration / udf / `multi_extension.out` conflict → remap by hand per\n     `references/sql-schema-backport.md`. Do NOT just accept `main`'s next-major (e.g. `15.0-1`) files.\n4. **Build in the isolated env:** `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install`. For any SQL change ALSO install\n   downgrades (see `references/sql-schema-backport.md` — plain `make install` skips them and\n   `multi_extension` will fail with cascading \"no update path\" errors otherwise; use\n   `make install-all`).\n5. **Test only what's relevant** (CI runs the whole suite): the feature's own test + `multi_extension`\n   for SQL changes. From `src/test/regress`: `python citus_tests/run_test.py <test>` (it manages its\n   own cluster). Regenerate any `expected/*.out` with the runner — **never hand-edit .out** (column\n   widths / row counts are dynamic).\n6. **Collapse to ONE commit per PR per branch.** Keep the original message; keep a `DESCRIPTION:`\n   first line iff the change is user-facing (this feeds the changelog). `git checkout -- configure`\n   if a build regenerated it (configure is never part of a backport). **Multiple PRs to the SAME\n   target → ONE branch per target, not per PR** (unsquashed: one commit per PR, in main's\n   chronological merge order; SQL-ladder commits sit right after their PR's cherry-pick). Name it\n   `bp-release-<major>` (e.g. `bp-release-13.2`). Build cleanly from the SQL-heavy PR's branch, then\n   cherry-pick the C-only PRs on top. **Cherry-pick an already-adapted backport commit WITHOUT\n   `-x`** — its message already carries the single `(cherry picked from <orig>)` trailer; `-x` would\n   double it.\n7. **Push to your fork.** `bp-*` branches don't match CI's push trigger (`main`/`release-*` only),\n   so kick CI with `gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`. Open\n   PRs only when the change owner asks.\n8. **Wait for CI and triage** per `references/ci-triage.md`. Separate baseline-red noise (N-1 /\n   flaky / coverage) from **genuinely-new failures you introduced**. The two you WILL hit on a\n   release branch: (a) a version-pinned `.out` that prints a shared helper's changed *signature*, and\n   (b) a PG-version planner divergence on the OLDER PG the branch keeps but `main` dropped (PREFER a\n   PG-guarded file like `pg16.sql` over a `<test>_N.out` alt file). A third you must NOT chase green\n   by mangling the test: a new GUC/UDF whose test runs under N-1 shows a NEW red in the (non-blocking)\n   **N-1 mixed-version** jobs — the old lib lacks it and falls back, which IS the N-1 contract. The\n   clean fix is to move that test line into the `multi_1_create_citus_schedule` placeholder section\n   (schedule mechanics + N-1 label rule in the hard-rules below and `ci-triage.md §\"NEW N-1 red\"`).\n   **Fix-commit rule (all fixes):** pre-review → amend + `git push --force-with-lease`; post-review →\n   add ONE new fix commit, never rewrite reviewed SHAs (replace a bad remote fix commit with\n   `--force-with-lease=<branch>:<badsha>`). Report evidence; don't declare done on red.\n9. **Propagate the ladder UP (SQL-schema backports only).** A new SQL object must also be reachable\n   on every HIGHER major's upgrade ladder, or a future 13.x→14.y Major Version Upgrade loses it.\n   As SEPARATE commits (its own PR on main), copy the introduction step/downgrade/udf files\n   byte-for-byte onto release-14.0 (for a 13.x path) and main (for 13.x and 14.x paths, given today's\n   two most recent major versions); main's own top path stays untouched. Full topology + recipe in\n   `references/sql-schema-backport.md` §Step 5.\n10. **Add `multi_extension.sql` ladder-test coverage** for every new step. The test only walks the\n    main upgrade path, so a step OFF that path (a 13.x/14.x maintenance-tail detour) is never\n    exercised unless you add a detour block matching the on-walk shape (and the file's existing\n    comment style — no editorial parentheticals): a round-trip no-op + a snapshot at the detour,\n    then retarget the *existing* next round-trip to bounce `<detour> ↔ <next>` rather than adding a\n    separate \"return\" block. That shifts exactly one downstream snapshot; everything after stays\n    byte-identical. A branch's own top/default_version is already covered by its snapshot block.\n    Never hand-edit `.out`; prefer append-only on reviewed branches. Full pattern + the `diff -w`\n    header gotcha in `references/sql-schema-backport.md`.\n11. **Prove MVU safety on a real cluster (SQL-schema backports, after CI is green).** The ladder\n    propagation only *claims* a future 13→14 Major Version Upgrade keeps the object; prove it. Stand\n    up a multi-node cluster at an OLD major, then `ALTER EXTENSION citus UPDATE` on EVERY node across\n    the backport branches (13.x tail → cross-major → 14.x tail → cross-major → main tail), capturing\n    `citus_version()` and `pg_extension.extversion` per node per hop, plus a functional check that the\n    backported object survived. Build all Citus versions into ONE shared PG with **`make install-all`**\n    (plain `install` omits the downgrade + bridge scripts the cross-major route needs → \"no update path\").\n    Full recipe and the single-PG requirement in `references/manual-upgrade-testing.md`.\n\n## Hard rules / traps (full detail in the references)\n- **Do NOT run `make reindent` in a build environment whose `citus_indent` version differs from the\n  branch's** — a version-skewed formatter reformats 100+ unrelated files and errors out. Trust\n  cherry-picked formatting; CI `check-style` verifies it. If you ran it by accident:\n  `git reset --hard HEAD`.\n- **`make install` does NOT install downgrade scripts.** For any SQL change run BOTH\n  `make -C src/backend/distributed install-downgrades` and `make -C src/backend/columnar\n  install-downgrades` (columnar is a separate extension). `make install-all` does both.\n- **A single build environment builds ONE PG only** — it cannot catch a compile break on another PG\n  the release branch supports. Rely on CI's per-PG `Build for PGNN` jobs; scan cherry-picked C for\n  struct fields / APIs newer than the branch's OLDEST PG and add `#if PG_VERSION_NUM >= PG_VERSION_NN`.\n- **The release branches are BASELINE-RED.** Several N-1 / flaky jobs fail on the pristine base\n  branch with no backport. Subtract the baseline before blaming your change — prove it with the\n  upstream `citusdata/citus` base-branch run. Details + exact commands in `references/ci-triage.md`.\n- **A new SQL object is defined MULTIPLE times on purpose.** After ladder propagation the same UDF\n  appears in the 13.x, 14.x AND main introduction steps (triple-defined on main). That is\n  intentional MVU/N-1 safety, NOT duplication to collapse. Never \"dedupe\" it. Details:\n  `references/sql-schema-backport.md` §Step 5.\n- **Show evidence.** Present the proof (build-job outcomes, `regression.diffs` excerpts, the\n  upstream-baseline comparison), not a bare \"CI is green/red\".\n- **A backport onto an older-PG release branch surfaces failures `main` never saw** (release-13.2\n  still builds PG15; the branch ships version-pinned `expected/*.out`). Two patterns: (1) a shared\n  helper whose SIGNATURE you changed prints in a pinned `.out` (`normalize.sed` masks the line\n  NUMBER, not the signature) — do NOT \"fix\" it by reverting the helper if the backported TEST needs\n  it (it's required, not over-reach); (2) a PG15-vs-PG16+ EXPLAIN/planner diff → PREFER a PG-guarded\n  file (`pg16.sql` `\\q`-skips <PG16 and already lives in the N-1-excluded schedule) over a\n  `<test>_0.out` alt. Full recipe: `references/ci-triage.md` §Genuinely-new failures.\n- **A NEW GUC/UDF whose test runs under N-1 has a CLEAN fix, not \"leave it\".** Move the test line\n  from its schedule into the `multi_1_create_citus_schedule` placeholder section (N-1 make_targets\n  omit `check-multi-1-create-citus`). N-1 version label = the branch's CURRENT N-1 = the previous\n  minor (13.3-1 on release-13.2, 14.1-1 on release-14.0), read from the live `citus_version:` /\n  `citus_libdir:` pin in build_and_test.yml — not the (possibly-stale) number in the neighbor comment.\n  Details: `references/ci-triage.md` §\"NEW N-1 red\".\n","frontmatter":{"name":"citus-backport","description":"Backport one or more merged citusdata/citus `main` commits/PRs onto the active release branches. Covers the release model, cherry-picking, remapping SQL schema changes to each branch's default_version, the SQL upgrade/downgrade ladder for Major-Version-Upgrade safety, running regression tests, one clean commit per PR per branch, and triaging release-branch CI (pre-existing baseline reds vs reds the backport introduced). USE WHEN asked to backport / port / cherry-pick a change to release-13.2 / release-14.0 (or the newest two majors), when a backport hits a SQL migration / udf / multi_extension conflict, when a \"trivial C-only\" backport fails to compile on an older PG the release branch still supports (e.g. PG15 `rteperminfos`), when release-branch CI is red and you must prove which reds are pre-existing, or when you must prove a SQL-schema backport is Major-Version-Upgrade-safe via `ALTER EXTENSION citus UPDATE` across a real multi-node cluster. DO NOT USE for non-Citus repos, new features, or PR review.","license":"See the repository LICENSE file."},"isInternal":false,"tokens":3359,"sizeBytes":12838},{"name":"README.md","path":".github/skills/README.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/skills/README.md","title":"Skills Skill","category":"anthropic-skill","format":"markdown","content":"# Agent skills for citus\n\nThis directory holds **agent skills** — folders of instructions (and optionally\nscripts/resources) that an AI coding agent can load on demand when it works on a\nparticular kind of task in this repository. Skills let us capture hard-won,\nrepo-specific know-how once and have every agent (and human) reuse it, instead of\nre-deriving it from scratch each time.\n\n## Layout\n\nEach skill lives in its own subdirectory containing a `SKILL.md`:\n\n```\n.github/skills/\n├── README.md                     ← this file\n└── citus-backport/\n    ├── SKILL.md                  ← entry point (loaded first)\n    └── references/               ← deep-dive docs loaded on demand\n        ├── sql-schema-backport.md\n        ├── ci-triage.md\n        └── manual-upgrade-testing.md\n```\n\n- Subdirectory names are lowercase with hyphens.\n- The entry file **must** be named `SKILL.md`.\n- Keep `SKILL.md` a concise hub; push long procedures into `references/` files it\n  links to, so the agent only pulls in what it needs.\n\n## `SKILL.md` format\n\nA `SKILL.md` is Markdown with YAML frontmatter:\n\n```markdown\n---\nname: my-skill                 # required; lowercase-hyphenated, matches the folder\ndescription: >-                # required; WHAT it does + WHEN to use it (drives auto-selection)\n  One or two sentences describing the task this skill helps with and the\n  trigger conditions that should make an agent load it.\nlicense: See the repository LICENSE file.   # optional\n---\n\n# Human-readable title\n\nInstructions, examples, and guidelines for the agent to follow…\n```\n\nThe `description` is the most important field for discovery: agents match a user's\nrequest against it, so state both the capability and the trigger phrases plainly.\n\n## Where these are picked up\n\nProject skills under `.github/skills/` are discovered automatically by GitHub\nCopilot CLI. (The Agent Skills format is intentionally portable; other agent tools\nmay read `.claude/skills/` or `.agents/skills/` instead — see each tool's docs.)\n\n`.github/copilot-instructions.md` (auto-loaded repo-wide) also carries a short\n`task → skill` index so agents are pointed here even before they browse this\ndirectory. When you add a skill, add a row there too.\n\n## Adding a new skill\n\n1. Create `./<skill-name>/SKILL.md` with valid frontmatter.\n2. Put anything long or optional in `./<skill-name>/references/` and link to it\n   from `SKILL.md`.\n3. Keep it repo-specific and evidence-based; avoid machine- or user-specific paths\n   so it works for anyone who clones citus.\n4. Add a `task → skill` row to `.github/copilot-instructions.md` so agents discover it.\n","isInternal":false,"tokens":622,"sizeBytes":2672},{"name":"copilot-instructions.md","path":".github/copilot-instructions.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/.github/copilot-instructions.md","title":"GitHub Copilot Instructions","category":"copilot-instructions","format":"markdown","content":"# Copilot instructions for citus\n\nFollow the repository's normal contribution conventions: see `CONTRIBUTING.md` for\nbuild/style/PR rules and `src/test/regress/README.md` for the regression-test workflow.\n\n## Agent skills\n\nThis repo ships task-specific **agent skills** under `.github/skills/`. Before starting\na task that matches one, load that skill's `SKILL.md` and follow it so you don't re-derive\nit. Each skill's `SKILL.md` frontmatter `description` states exactly when to use it; the\nindex below is a quick map.\n\n| Task | Skill |\n|------|-------|\n| Backport a merged `main` commit/PR to a release branch (`release-13.2` / `release-14.0` / newest two majors) — including SQL-schema migrations, the upgrade/downgrade ladder, N-1 / Major-Version-Upgrade safety, and triaging release-branch CI | [`citus-backport`](skills/citus-backport/SKILL.md) |\n| Merge a batch of Citus PRs one at a time: sync each with its base branch, fix a red `check-style` job, retry other red checks with give-up rules, then squash-merge with the PR description as the commit message | [`citus-merge-loop`](skills/citus-merge-loop/SKILL.md) |\n| Fix a failing `check-style` job on whatever branch is already checked out, by rebuilding the exact `uncrustify`/`citus_indent` versions that branch itself pins (in its own `STYLEGUIDE.md` and `.github/workflows/build_and_test.yml`), then running `make reindent` and committing the style-only diff locally (no push, and no switching to or inspecting any other Citus branch) | [`citus-check-style-reindent`](skills/citus-check-style-reindent/SKILL.md) |\n\nSee `.github/skills/README.md` for the skills layout and how to add a new one.\n","isInternal":false,"tokens":422,"sizeBytes":1659},{"name":"README.md","path":"src/backend/distributed/commands/README.md","rawUrl":"https://raw.githubusercontent.com/citusdata/citus/HEAD/src/backend/distributed/commands/README.md","title":"Command: /readme","category":"command-prompt","format":"markdown","content":"# Commands\n\nThe commands module is modeled after `backend/commands` from the postgres repository and\ncontains the logic for Citus on how to run these commands on distributed objects. Even\nthough the structure of the directory has some resemblence to its postgres relative, files\nhere are somewhat more fine-grained. This is due to the nature of citus commands that are\nheavily focused on distributed tables. Instead of having all commands in `tablecmds.c`\nthey are often moved to files that are named after the command.\n\n| File                         | Description |\n|------------------------------|-------------|\n| `create_distributed_table.c` | Implementation of UDF's for creating distributed tables |\n| `drop_distributed_table.c`   | Implementation for dropping metadata for partitions of distributed tables |\n| `extension.c`                | Implementation of `CREATE EXTENSION` commands for citus specific checks |\n| `foreign_constraint.c`       | Implementation of and helper functions for foreign key constraints |\n| `grant.c`                    | Implementation of `GRANT` commands for roles/users on relations |\n| `index.c`                    | Implementation of commands specific to indices on distributed tables |\n| `multi_copy.c`               | Implementation of `COPY` command. There are multiple different copy modes which are described in detail below |\n| `policy.c`                   | Implementation of `CREATE\\ALTER POLICY` commands. |\n| `rename.c`                   | Implementation of `ALTER ... RENAME ...` commands. It implements the renaming of applicable objects, otherwise provides the user with a warning |\n| `schema.c`                   | |\n| `sequence.c`                 | Implementation of `CREATE/ALTER SEQUENCE` commands. Primarily checks correctness of sequence statements as they are not propagated to the worker nodes |\n| `table.c`                    | |\n| `transmit.c`                 | Implementation of `COPY` commands with `format transmit` set in the options. This format is used to transfer files from one node to another node |\n| `truncate.c`                 | Implementation of `TRUNCATE` commands on distributed tables |\n| `utility_hook.c`             | This is the entry point from postgres into the commands module of citus. It contains the implementation that gets registered in postgres' `ProcessUtility_hook` callback to extends the functionality of the original ProcessUtility. This code is used to route the incoming commands to their respective implementation in Citus |\n| `vacuum.c`                   | Implementation of `VACUUM` commands on distributed tables |\n\n# COPY\n\nThe copy command is overloaded for a couple of use-cases specific to citus. The syntax of\nthe command stays the same, however the implementation might slightly differ from the\nstock implementation. The overloading is mostly done via extra options that Citus uses to\nindicate how to operate the copy. The options used are described below.\n\n## FORMAT transmit\n\nImplemented in `transmit.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n\n## FORMAT result\n\nImplemented in `multi_copy.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n","isInternal":false,"tokens":646,"sizeBytes":3249}],"systemPromptSnippet":"<yakaai_skills repo=\"citusdata/citus\">\n<!-- File: .github/skills/citus-backport/references/ci-triage.md (Tokens: ~5018 | Category: anthropic-skill) -->\n# CI triage for backports — PG-version compat + the baseline-red release branches\n\nTwo independent things make backport CI confusing: (1) `main` might support different PG majors\nthan a release branch, so a \"trivial C-only\" cherry-pick can fail to COMPILE on an old PG; and\n(2) the release branches are BASELINE-RED — several jobs fail on the pristine base branch with\nno backport. Handle both before declaring a red \"your fault\".\n\n## PG-version compat (the #1 non-obvious C-backport failure)\n`main` may support different PG majors than the release branches, so code `main` wrote unconditionally\nagainst a newer PG API breaks the older-PG build on a release branch. Check each branch's supported\nPG set in `.github/workflows/build_and_test.yml` (the `pgNN_version` params block). Today, these are the\nsupported PG majors by main and two most recent release branches:\n\n| Branch        | PG majors built      |\n|---------------|----------------------|\n| release-13.2  | **15**, 16, 17       |\n| release-14.0  | 16, 17, 18 (no 15)   |\n| main          | 16, 17, 18 (dropped 15) |\n\n**Concrete example (confirmed):** `Query.rteperminfos` was added in PG16. `main` sets\n`q->rteperminfos = NIL;` bare. On release-13.2 that is `error: 'Query' has no member named\n'rteperminfos'` on the **PG15** build. Fix = the citus idiom the branch ALREADY uses everywhere\nelse for that field:\n\n```c\n#if PG_VERSION_NUM >= PG_VERSION_16\n    q->rteperminfos = NIL;\n#endif\n```\n\nGrep the target branch for the same symbol to copy its exact guard macro. **General rule:** after a\nC cherry-pick, scan the added `+` lines for struct fields / API calls introduced in a PG major that\nthe branch's OLDEST PG predates, and guard them. **Backport to the oldest-PG branch first** to\nsurface these early. A single build environment builds ONE PG so it can't catch this locally — rely\non CI's per-PG `Build for PGNN` jobs.\n\n### Reading a Build-for-PGNN failure mid-run\n`gh run view --log` / `--log-failed` say \"still in progress\" until the WHOLE run finishes, but a\nsingle finished job's log is fetchable immediately:\n\n```\ngh api repos/<your-fork>/citus/actions/jobs/<jobId>/logs > build.log\n```\n\nThen grep for `error:` / `make.*Error`. A `Build for PGxx` failure with a real `error:` line = a\ngenuine bug (fix it). An \"exit code N\" annotation with no compiler `error:` in the log = usually\ninfra/flaky (rerun). After fixing, **amend the single cherry-pick commit** (keep 1 commit/PR),\n`git push --force-with-lease`, and re-dispatch:\n`gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`.\n\n## The release branches are BASELINE-RED\n`release-13.2` and `release-14.0` have KNOWN-FAILING jobs on their own pristine upstream CI. Do NOT\npanic when your backport branch is red — **subtract the baseline first.** As of this writing the\nbase branches fail these SAME jobs with no backport applied:\n\n- **N-1 mixed-version jobs where the OLD lib runs on the COORDINATOR:**\n  - Test Citus **Lib N-1** — check-multi, check-multi-1, check-add-backup-node\n  - Test Citus **Coordinator N-1** — check-multi, check-multi-1, check-add-backup-node\n  - (The SQL N-1 and Worker N-1 variants of the same tests PASS — old lib is only on workers there.)\n- Occasional **Test flakyness (N)** shards and a stray check-enterprise / check-operations.\n- **Test Citus Failure / check-query-generator** — a randomized differential fuzzer (compares\n  distributed vs local results of random nested-CTE/join queries). Flaky by nature; a random query\n  can trip `citus.max_intermediate_result_size` etc. **Tell:** one PG major fails while another PG\n  major of the SAME commit passes. Not a backport regression.\n\n**Root cause of the N-1 reds (not yours):** these base branches already carry in-flight\nminor-targeted features (e.g. clone-promotion sequence *re-ranging*;\n`citus_internal.delete_placement_metadata` behavior) whose NEW behavior lives only in the NEW lib.\nWhen the OLD lib (the previous minor's binary) is loaded on the coordinator, the expected `.out`\n(written for the new lib) mismatches → diff. Inherent to N-1 testing of in-flight features;\nunrelated to a feature backport that doesn't touch those areas.\n\n## How to PROVE a red is pre-existing (do this — gather the evidence)\n1. **Upstream base-branch run is itself failing:**\n   ```\n   gh run list --repo citusdata/citus --branch release-13.2 --workflow \"Build & Test\" --limit 6\n   ```\n   Note the base-branch runs are themselves `failure`.\n2. **Your failing set ⊆ upstream's failing set:**\n   ```\n   gh run view <upstreamRunId> --repo citusdata/citus --json jobs \\\n     --jq '.jobs[]|select(.conclusion==\"failure\")|.name'\n   ```\n   If your branch's failing job names are a subset, they're baseline, not yours.\n3. **The diff cites a pre-existing feature, not your object:** download your failed job's artifact\n   and read `regression.diffs`:\n   ```\n   gh run download <runId> --repo <your-fork>/citus -n \"<artifactName>\" -D <dir>\n   ```\n   Artifact naming: `<pgmajor>_<test>_[<citus_version>_]<libLabel>_<mode>`\n   (e.g. `18_check-enterprise_14.1-1_all` = SQL N-1; `..._v14.1.0_coordinatoronly` = Coordinator\n   N-1). Confirm the diff mentions the pre-existing feature (`re-ranged N sequence(s)`,\n   `delete_placement_metadata`) and has **ZERO** mentions of your backported symbol.\n4. **Best control:** run CI on your C-ONLY sibling backport (touches none of the SQL/dependency\n   code). If it shows the IDENTICAL N-1 reds, that confirms the reds track the base branch, not the\n   change.\n\n## What WOULD be your fault (fix these)\n- A red job whose `regression.diffs` names YOUR backported object **in a NORMAL (same-version)\n  job** — check-multi/-mx/-1 on a supported PG. (See the N-1 handling immediately below.)\n- A `Build for PGxx` with a real `error:` in its log.\n- A NEW job red that is GREEN on the upstream base branch. If it's a NEW-GUC/UDF N-1 red, the fix is\n  the schedule move below (not a test/output change); everything else, fix at the source.\n\nEverything else on these branches is baseline noise.\n\n## The NEW N-1 red a feature backport ADDS — move it to the N-1-excluded schedule\nBackporting a feature that adds a **new GUC or UDF** whose regression test runs under an N-1\nschedule (`multi_schedule` / `multi_1_schedule` → `check-multi`/`check-multi-1`) makes the **N-1\n(mixed-version)** jobs show that test as a NEW red. The red itself is EXPECTED graceful-degradation\n(not a defect), but the CLEAN fix is to relocate the test so N-1 never runs it — do NOT mangle the\nreviewed test, fabricate output, or just \"leave + document\". Proven on both the 13.2 and 14.0\nbackports of the `citus.allow_unsafe_insert_select_pushdown` GUC:\n- Test `allow_unsafe_insert_select_pushdown` (backported GUC `citus.allow_unsafe_insert_select_pushdown`).\n- Under N-1 the OLD lib runs on the coordinator (`lib-v13.3.0` / `lib-v14.1.0`), which lacks the GUC.\n  `SET citus.allow_unsafe_insert_select_pushdown` → `ERROR: invalid configuration parameter name ...\n  \"citus\" is a reserved prefix`, then the EXPLAIN falls back to the DEFAULT (pull-to-coordinator) plan.\n- **That fallback IS the N-1 contract** (old binary → GUC absent → safe pre-feature behavior); the diff\n  *demonstrates* compatibility. But a red CI job is still noise, so move the test off the N-1 matrix.\n\n**THE FIX: move the test line from its schedule into the placeholder section of\n`multi_1_create_citus_schedule`.** The N-1 jobs' `make_targets` (build_and_test.yml Coordinator/Lib\nN-1 blocks) list `check-multi`, `check-multi-1`, `check-add-backup-node`, ... but **NOT\n`check-multi-1-create-citus`**, so any test in `multi_1_create_citus_schedule` is N-1-invisible while\nSTILL running under `check-multi-1-create-citus` on every normal PG job (coverage preserved). Steps:\n1. Remove `test: <name>` from `multi_schedule` (mind the surrounding blank lines / parallel-group\n   comments — don't strand a comment that referenced it).\n2. Append a block under the placeholder header in `multi_1_create_citus_schedule`\n   (`# Have this placeholder section at the end for the tests that need to be moved back ... to\n   prevent N-1 test failures for the features that are not present in the minor version.`), mirroring\n   the EXISTING `citus_cluster_changes_block` block right there — same `# ----------` fencing and\n   phrasing so new additions look like the existing file.\n3. `run_test.py` auto-discovers the new schedule (it prints `SCHEDULE: multi_1_create_citus_schedule`);\n   the `.out` file is schedule-independent, so NO `.out` regen is needed — it's a pure schedule move.\n\n**N-1 version label = the actual N-1 CI version = the PREVIOUS MINOR (it advances every release).**\nRead the authoritative number from `build_and_test.yml`'s N-1 jobs — the `citus_version:` (SQL /\nWorker / Coordinator N-1) and `citus_libdir:` (Lib / Coordinator N-1) pins, not from\nmaybe-stale comments at `multi_1_create_citus_schedule`. As of this writing: release-13.2 pins\n`citus_version: \"13.3-1\"` / `citus_libdir: v13.3.0` (its `default_version` is 13.4-1, so N-1 = 13.3-1);\nrelease-14.0 pins `14.1-1` / `v14.1.0` (default 14.2-1, N-1 = 14.1-1). The rule is simply\n`default_version` minus one minor, and it is bumped whenever the branch ships a new minor.\n\n- **Write the label with today's N-1 version.** For a GUC introduced in the branch's current\n  `default_version` (e.g. 13.4 on release-13.2), the N-1 binary is the previous minor (13.3-1) and the\n  GUC is absent there, so: `GUC introduced in 13.4; not present in the N-1 (13.3-1) Citus binary ...\n  Move back to multi_schedule at Citus <next major, 14>.` On release-14.0 the corresponding numbers\n  are 14.2 / N-1 14.1-1 / next major 15.\n- **Sanity gate:** the move is only warranted when the object is genuinely ABSENT in that N-1 version\n  (introduced in a minor strictly newer than N-1). If your object was introduced at or below N-1, the\n  N-1 binary already has it — no red, no move.\n\n- **Why a new UDF may NOT need this move** (asymmetry): a test already placed in\n  `multi_1_create_citus_schedule` (e.g. `citus_internal_distribute_object`) is ALREADY N-1-invisible,\n  so it never shows an N-1 red and needs no move. Grep which schedule your test is in first.\n- Distinguish this N-1 red from a genuine bug: a same-version job (no `lib-vX` / `_coordinatoronly` /\n  `_workeronly` / `_all` suffix on the artifact) naming your object IS yours — see the section above.\n- Shipping: fold the move into a not-yet-reviewed fix commit (amend + `--force-with-lease`), or add a\n  NEW appended commit if the tip has already been reviewed (plain push, no force).\n\n## Genuinely-new failures a backport WILL introduce on a release branch (and the fix)\nThe release branch supports an OLDER PG than `main` (e.g. release-13.2 still builds **PG15**), and it\nships **version-pinned `expected/*.out`** files. A backport that is green on `main` can therefore go\nred on the release branch for two reasons that ARE yours to fix (they are not baseline noise):\n\n### (a) A shared helper's SIGNATURE changed → version-pinned .out that prints the signature mismatches\nIf your backport chain brings a new/edited SQL helper whose *signature* changed (e.g.\n`explain_filter(text)` → `explain_filter(text, keep_numbers boolean DEFAULT false)`), any expected\n`.out` that prints that signature in a `CONTEXT:` / `PL/pgSQL function ... line N` / error line will\nmismatch. **`normalize.sed` normalizes only the line NUMBER (`line [0-9]+` → `line XX`), NOT the\nsignature text**, so the signature diff is real. Fix = regenerate/`sed`-update those signature\nstrings in the pinned `.out`. Find them all with `grep -rl \"helper_name(oldargs)\" expected/`.\n- **Do NOT mistake a required helper for \"over-reach\" and revert it.** If the backported *test*\n  calls the new form (`explain_filter($$...$$, true)`), the helper is REQUIRED; reverting it yields\n  `ERROR: function ...(unknown, boolean) does not exist`. Before deleting an \"extra\" hunk a\n  cherry-pick carried, check whether the backported test depends on it.\n\n### (b) PG-version planner divergence on the older PG the release branch still supports\nThe same EXPLAIN can differ by PG major. Real example: PG15 omits a functionally-redundant trailing\nsort column that PG16+ keeps — `Sort Key: q.batch` (PG15) vs `Sort Key: q.batch, q.text_id` (PG16+).\nrelease-13.2 hits it on the PG15 `check-multi`; main/release-14.0 never see it (no PG15).\nFix, in the order the repo prefers:\n1. **An explain helper from `main`'s `multi_test_helpers.sql`** — check whether one already\n   normalizes the divergent bit. As of 13.x/14.x none normalizes Sort Key (there are\n   `coordinator_plan`, `plan_normalize_memory`, `explain_with_pg16_subplan_format`,\n   `explain_with_pg17_initplan_format`, and boolean `explain_has_distributed_subplan` /\n   `explain_has_single_task` — the booleans would need an invasive test rewrite).\n2. **MOVE the diverging blocks into a PG-version-guarded test file (PREFERRED on release-13.2 when\n   the divergence is PG15-only).** release-13.2 ships `sql/pg16.sql`, which `\\q`-skips below PG16\n   (guard at top: `SELECT substring(:'server_version','\\d+')::int >= 16 ... \\if ... \\else \\q \\endif`)\n   AND already lives in `multi_1_create_citus_schedule` (the N-1-excluded schedule). So relocating the\n   PG15-diverging EXPLAIN cases out of the normal test (e.g. `allow_unsafe_insert_select_pushdown.sql`)\n   into a self-contained section appended to `pg16.sql` means: PG15 never runs them (no diff),\n   PG16/17/18 still cover them, and — bonus — they leave the N-1 matrix too. Recipe:\n   - Cut the exact diverging blocks from the source `.sql`; fix any dangling comment they leave behind.\n   - Append AFTER `pg16.sql`'s final `DROP SCHEMA pg16 CASCADE;` a self-contained section: its OWN\n     schema (e.g. `allow_unsafe_insert_select_pushdown_pg16`), a distinct `next_shard_id`\n     (e.g. 14100000) to avoid shard-id clashes, recreate the tables/functions the blocks need, the\n     moved EXPLAIN blocks, then `DROP SCHEMA ... CASCADE;`. It must not depend on the parent test's\n     schema (schedules run independently).\n   - Regenerate BOTH `.out`s (source shrinks, `pg16.out` grows) via the runner; never hand-edit.\n3. Add a new **`normalize.sed` rule** or update an existing one to generalize the coverage, or add an\n   alternative expected file as a last resort.\n\n### (c) Flaky detector goes all-red because your changed-file set is widely depended-on\n\"Test flakyness (1..32)\" computes a changed-test set by DEPENDENCY EXPANSION of the PR's changed\nfiles. Editing a widely-depended helper/migration (`multi_test_helpers.sql`, a migration `.sql`,\n`multi_extension.sql`) balloons the set across every shard and drags in inherently-flaky base tests\n(`start_stop_metadata_sync`, `multi_metadata_sync`, ...). The shards only settle once your\n*actually-changed* tests pass; the base-flaky tests pulled in are not yours to fix. Sanity check with\n`gh pr view <n> --json files` — a helper appearing in the flaky `tests=` list but NOT in the PR's\nfile list is pure dependency-expansion, not a change you made.\n\n**Do NOT hand-wave \"it's flaky\".** Two pieces of hard evidence, both cheap:\n1. **Read the actual diff** from the shard's uploaded artifact (they DO upload one, named\n   `test_flakyness_parallel_<N>`; see fetch recipe below). Confirm the failing test is a base test\n   you did not touch AND the diff contains ZERO of your backported symbols. The signature of\n   `--use-whole-schedule-line` running the whole schedule line in parallel so other tests consume\n   colocation IDs first. Scan: `grep -c <your_symbol> regression.diffs` (want 0),\n   `grep -oE \"expected/[a-z_0-9]+[.]out\"` (only the base test).\n2. **Reproduce on the PRISTINE base** to prove pre-existing, not introduced. From `src/test/regress`:\n   `python citus_tests/run_test.py <flaky_test> --repeat 8 --use-whole-schedule-line` on your branch\n   head, then `git checkout <merge-base> && make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install` and run the SAME command. If the\n   base fails the identical N/N under the identical flags, it's 100% pre-existing (checkout your\n   branch + rebuild to restore afterward). Example proven this way: `start_stop_metadata_sync` failed\n   8/8 on both `bp-release-13.2` tip and its pristine base, same colocationid/FK diff — not the\n   backport's fault. NOTE the flaky detector's PG major varies — check the shard log's `PG_MAJOR:`\n   line and reproduce on the SAME PG for an exact comparison.\n\n### Shipping the fix\nOne NEW fix commit on top of the reviewed tip (do not rewrite reviewed SHAs). If a prior *bad* fix\ncommit is already on the remote, REPLACE it with `git push --force-with-lease=<branch>:<badsha>`\n(the reviewed commits below it stay). Force-push freely only while the commits are still unreviewed.\n\n## Fetching CI diffs reliably (tooling gotchas)\n- `gh run view --job <id> --log-failed` streams often die with `stream error: stream ID 1; CANCEL` —\n  retry or avoid.\n- Jobs that upload artifacts (`NN_check-multi`, `NN_check-operations`, AND the flaky shards):\n  list with a SINGLE page, not `--paginate` (`--paginate` concatenates JSON pages so a single `--jq`\n  chokes with \"Additional text after finished reading JSON content\"):\n  `gh api \"repos/citusdata/citus/actions/runs/<run>/artifacts?per_page=100\" --jq '.artifacts[]|select(.name==\"15_check-multi\").id'`\n  then `gh api .../actions/artifacts/<id>/zip > x.zip` (this redirect is fine for the small per-artifact\n  zip; read `regression.diffs` from it).\n- Flaky-detector jobs DO upload a named artifact `test_flakyness_parallel_<N>` (contains the shard's\n  `regression.diffs` under `__w/citus/citus/src/test/regress/`) — fetch it like the per-artifact zip\n  above; that's the reliable way to read the flaky diff body. The per-job logs endpoint only NAMES the\n  failing test (`SCHEDULE_LINE:` / `not ok`) and points at regression.diffs in the artifact.\n- Do NOT re-encode a downloaded zip through a shell pipe that changes encoding (corrupts it); write it\n  to disk directly. The whole-run `/logs` zip often arrives partial (\"End of Central Directory record\n  could not be found\") — prefer the per-job / per-artifact endpoints above.\n\n## Mechanics / gotchas\n- Overall run status stays `queued`/`in_progress` until EVERY matrix job ends; per-job logs and\n  finished-job artifacts are still fetchable mid-run (above).\n- Dispatching several full matrix runs (~190 jobs each) saturates the fork runner pool and starves\n  the job you actually care about; `gh run cancel <id>` the fully-triaged runs to free capacity for\n  a critical re-verification.\n\n<!-- File: .github/skills/citus-backport/references/manual-upgrade-testing.md (Tokens: ~3089 | Category: anthropic-skill) -->\n# Manual MVU (major-version-upgrade) testing for backport branches\n\nBackporting a SQL-schema change and propagating the upgrade ladder (see\n`sql-schema-backport.md` §Step 5) only *claims* MVU safety. This doc is how you **prove** it\nend-to-end on a real multi-node cluster: stand up N nodes at an OLD Citus major, then walk the\nextension forward across both backport branches — and, one hop further onto the\n`main`-targeting PR branch — with `ALTER EXTENSION citus UPDATE`, capturing `citus_version()` AND\n`pg_extension.extversion` on every node after every hop.\n\nThis is the \"manual testing for backport branches\" step. Do it after CI is green, before\ndeclaring a SQL-schema backport family done. A C-only backport does NOT need it.\n\n## What this proves\n- A cluster shipped at the 13.x tail (`default_version` of `release-13.2`, e.g. `13.4-1`) can\n  still cross to 14.x — i.e. a future managed-service 13→14 MVU won't strand the customer.\n- **Final hop:** a 14.x-tail cluster (e.g. `14.2-1`) can cross to the NEXT major on\n  `main` (`15.0-1`) via the PR branch you opened against `main` — proving the SAME object also\n  rides a future 14.x→15.0 MVU. `main` jumps `14.0-1 → 15.0-1` directly (it has NO 14.1/14.2\n  *upgrade-in-line*), but it MUST carry the `14.2-1--14.1-1` and `14.1-1--14.0-1` **downgrades**\n  so PG can route `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. If that route is missing you get \"no\n  update path from 14.2-1 to 15.0-1\" — a real MVU gap in the ladder propagation to `main`.\n- The forward/downgrade **ladder** you propagated actually routes end to end.\n- The backported object (UDF/GUC/column) is present and the cluster is functional after each jump.\n\n## The single hard requirement: ONE PG, three Citus builds\n`ALTER EXTENSION citus UPDATE` walks the version graph made of the `citus--A--B.sql` scripts\n**installed in the current PG's `SHAREDIR/extension/`**. So all three Citus versions in the walk\n(old baseline + both backport branches) must be installed, one at a time, into the SAME PG\ninstall / SAME cluster data dirs. You swap the installed Citus between hops and restart the nodes;\nthe catalog stays put and `ALTER EXTENSION` migrates it.\n\nUse ONE dedicated PG install as that single PG (its Citus gets overwritten repeatedly — that's the\npoint — then you restore it at the end, §Cleanup). Do the builds in **detached worktrees** so your\nreviewed branch checkouts are never disturbed.\n\n### `make install-all`, NOT `make install` — MANDATORY here\n`make install` lays down only **forward** scripts. The cross-major route from the 13.x tail is\n**not** a straight climb — there is no `13.4-1--14.x` edge. It goes:\n\n```\n13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1\n  (down)   (down)   (BRIDGE up)  (up)    (up)\n```\n\nso it needs the **downgrade** scripts `13.4-1--13.3-1`, `13.3-1--13.2-1` and the cross-major\n**bridge** `13.2-1--14.0-1` (all shipped under `sql/downgrades/` on `release-14.0`). `make install`\nomits every downgrade → `ALTER EXTENSION citus UPDATE` fails with \"no update path from 13.4-1 to\n14.2-1\". Always build the test PG with **`make install-all`** (it runs `install` +\n`install-downgrades` for both the distributed and columnar extensions).\n\n### PG version must be common to all builds\nBaseline Citus must support the chosen PG. 13.0/13.2 support PG15/16/17; 14.0 supports 16/17/18;\n`main` supports 16/17/18 → **PG 17** is the common denominator for all four. Pick a PG 17 install\nso it works for the whole 13.0→13.4→14.2→15.0 walk.\n\n## Recipe (verified with a 3-node cluster: citus 13.0.5 → 13.4.0 → 14.2.0 → 15.0devel)\n\nPick a shared PG (`SHARED=<path to your PG17 install>`), a spare coordinator port (e.g. `10600`;\nworkers take `+1/+2`), and the three source trees. Use detached worktrees so the reviewed\nbranch worktrees are never disturbed:\n\n```bash\ncd <main clone>            # holds all the branches/tags\ngit worktree add --detach /tmp/mvu/wt130 v13.0.5            # OLD baseline (a real tag)\ngit worktree add --detach /tmp/mvu/wt132 <bp-release-13.2-head-sha>\ngit worktree add --detach /tmp/mvu/wt140 <bp-release-14.0-head-sha>\ngit worktree add --detach /tmp/mvu/wtmain <bp-main-PR-head-sha>\n```\n\nBuild helper — reconfigure each tree against the SHARED PG and `install-all`:\n\n```bash\nbuild() {  # build <worktree> ; installs into $SHARED\n  cd \"$1\"\n  ./configure PG_CONFIG=\"$SHARED/bin/pg_config\" >/tmp/cfg.log 2>&1\n  local JOBS; JOBS=$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)   # portable: Linux || macOS\n  make -sj\"$JOBS\"             >/tmp/mk.log  2>&1\n  make -sj\"$JOBS\" install-all >/tmp/ins.log 2>&1\n  ls \"$SHARED/share/extension/\" | grep -E '^citus--1[3-5]' | sort   # sanity: see the ladder\n}\n```\nRe-use the same cluster data dirs across hops:\n\n1. **Baseline 13.0.** `build /tmp/mvu/wt130`. Stand up a cluster on `$SHARED` with `citus_dev`\n   (it `CREATE EXTENSION citus` at `default_version` = the currently-installed `13.0-1`). Verify\n   `SELECT extversion FROM pg_extension WHERE extname='citus'` and `SELECT citus_version()` = 13.0\n   on coordinator + both workers. (`citus_dev`'s default DB is `$USER`, NOT `postgres`.)\n2. **Up to the 13.2 branch.** `build /tmp/mvu/wt132` (overwrites Citus in `$SHARED`, now\n   `default_version = 13.4-1`, installs the 13.x up+down ladder). **Restart every node** so the new\n   `.so` loads. Then, on EACH node: `ALTER EXTENSION citus UPDATE;` → catalog goes `13.0-1 → 13.4-1`\n   in one call (walks the whole chain). Capture `citus_version()` per node — expect\n   `13.4.0 ... (sha: <13.2-head>)`.\n3. **Cross-major to the 14.0 branch.** `build /tmp/mvu/wt140` (`default_version = 14.2-1`, installs\n   the full up+down+bridge ladder). **Restart every node.** On EACH node: `ALTER EXTENSION citus\n   UPDATE;` → routes `13.4-1 → 13.3-1 → 13.2-1 → 14.0-1 → 14.1-1 → 14.2-1`. Capture\n   `citus_version()` per node — expect `14.2.0 ... (sha: <14.0-head>)`.\n4. **Functional check** on the upgraded cluster (coordinator): active `pg_dist_node`,\n   `create_distributed_table` + INSERT + distributed `count(*)`, `pg_dist_shard` count, and that\n   the backported object is present (e.g. `SELECT proname FROM pg_proc WHERE proname =\n   'distribute_object' AND pronamespace = 'citus_internal'::regnamespace;`).\n5. **One more hop onto the `main` PR branch.** Detached worktree at the `bp-<pr>-main`\n   head, `build` it into the SAME `$SHARED` (overwrites Citus to `default_version = 15.0-1`,\n   installs main's up+down ladder incl. the 14.x downgrades). **Restart every node.** On EACH node:\n   `ALTER EXTENSION citus UPDATE;` → routes `14.2-1 → 14.1-1 → 14.0-1 → 15.0-1`. Re-run the\n   functional check. Expect `Citus 15.0devel ... (sha: <main-PR-head>)` and the backported object\n   still present at `15.0-1`.\n\n**KEY NUANCE — `citus_version()` reports the BINARY, `extversion` reports the CATALOG.** After you\ninstall a newer Citus + restart (but BEFORE `ALTER EXTENSION citus UPDATE`), `citus_version()`\nalready prints the new version (it is compiled into the `.so`), while `pg_extension.extversion` is\nstill the old catalog version. The migration is only proven by **`extversion` moving to the new\n`default_version`**. On the first hops the jump is a whole major so `citus_version()` alone is\ntelling; on the `14.2 → 15.0devel` hop both 14.2 and 15.0 builds may print `15.0devel`-ish strings,\nso capture `extversion` before+after (`14.2-1 → 15.0-1`) as the real evidence.\n\nRun the same `ALTER EXTENSION citus UPDATE` on each node by connecting to each port directly\n(coordinator does NOT auto-propagate the extension version to workers) — that's the \"update all\nnodes\" the ask means.\n\n## Environment notes (neutral, but they save time)\n- **Put SQL in `.sql` files and run `psql -f`; put shell logic in `.sh` files and run them.**\n  Piping SQL / loops inline through several shell layers (esp. cross-OS terminals) mangles quoting\n  and `$USER`. If a script is authored on a different-newline OS, strip CRLF first (`sed -i\n  's/\\r$//' file`) or `set -o pipefail` may die with \"invalid option name\".\n- **`citus_dev` records RELATIVE data-dir paths** (`cluster/<role>`). `pg_ctl -D <abs>` then fails\n  with \"could not access directory …/cluster/<role>\". Fix: `cd` into the clusters parent and use\n  `pg_ctl -D cluster/<role> …`.\n- **If `pg_ctl -w start` blocks**, the `-w` readiness probe can hang against an `ssl=on` citus_dev\n  config even though the server logged \"ready to accept connections\". Start WITHOUT `-w` and poll\n  the TCP port yourself (`ss -ltn | grep :<port>`). `-w` on **stop** is fine.\n- On slow disks the first post-restart checkpoint can take ~100s; judge node readiness by\n  \"database system is ready to accept connections\" in the node logfile + the port listening, not by\n  a `pg_ctl -w` wrapper returning.\n\n## Cleanup / restore\n- The shared PG's Citus was left at whatever you installed last (after the final main hop that\n  is main's `15.0-1`). **Restore the env's own build:** rebuild the PG's normal Citus branch with\n  `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install-all` (verify `default_version` + the ladder edges are back).\n- Stop the test cluster (`pg_ctl -D cluster/<role> -w -m fast stop` per node, or your cluster\n  helper's stop command) and `git worktree remove` the throwaway build trees (+ `git worktree\n  prune`) when done.\n\n## Expected evidence shape (what a PASS looks like)\n```\nBASELINE (citus 13.0):    all nodes → extversion 13.0-1  | Citus 13.0.5 ... gitref: v13.0.5\nafter 13.2-branch UPDATE: all nodes → extversion 13.4-1  | Citus 13.4.0 ... (sha: <13.2 head>)\nafter 14.0-branch UPDATE: all nodes → extversion 14.2-1  | Citus 14.2.0 ... (sha: <14.0 head>)\nafter main-PR   UPDATE:   all nodes → extversion 14.2-1→15.0-1 | Citus 15.0devel ... (sha: <main head>)\nfunctional (each hop): 3 active nodes, create_distributed_table OK, count matches, backported obj present\n```\nReport the per-node `extversion` (before+after) AND `citus_version()` for every hop — not a\nsummary — since the whole point is proving every node crossed each boundary.\n\n<!-- File: .github/skills/citus-backport/references/sql-schema-backport.md (Tokens: ~4168 | Category: anthropic-skill) -->\n# SQL-schema backport recipe (the hard case)\n\nA C-only cherry-pick usually just applies. The moment a commit touches the SQL schema (new UDF,\nnew column, new migration step) you must **remap it to the target branch's real version**, because\n`main`'s migration files are named for the NEXT major (e.g. `15.0-1`) and the release branch is on\na different `default_version`. Do NOT accept `main`'s files as-is.\n\n## Step 0 — read the target version (drives everything)\nOn each target branch read `src/backend/distributed/citus.control`:\n\n```\ndefault_version = '<X.Y-Z>'\n```\n\nThen find that branch's current dev migration + downgrade under\n`src/backend/distributed/sql/`:\n- upgrade:   `citus--<prev>--<default_version>.sql`\n- downgrade: `citus--<default_version>--<prev>.sql`\n\nWorked example (versions as of this writing):\n\n| Branch        | default_version | dev upgrade migration        | dev downgrade                | udf versioned file |\n|---------------|-----------------|------------------------------|------------------------------|--------------------|\n| release-13.2  | `13.4-1`        | `citus--13.3-1--13.4-1.sql`  | `citus--13.4-1--13.3-1.sql`  | `13.4-1.sql`       |\n| release-14.0  | `14.2-1`        | `citus--14.1-1--14.2-1.sql`  | `citus--14.2-1--14.1-1.sql`  | `14.2-1.sql`       |\n| main          | `15.0-1`        | `citus--14.0-1--15.0-1.sql`  | `citus--15.0-1--14.0-1.sql`  | `15.0-1.sql`       |\n\n## Step 1 — remap the migration files\nThe main commit added its schema hunk to `citus--14.0-1--15.0-1.sql` (and the DROP to the matching\n`15.0-1--14.0-1` downgrade), plus a versioned udf file `sql/udfs/<name>/15.0-1.sql`. On the target\nbranch:\n\n1. `git rm` the main-only `15.0-1` upgrade + downgrade migration files if the cherry-pick created\n   them (they don't belong on the release branch).\n2. `git mv src/.../sql/udfs/<name>/15.0-1.sql  src/.../sql/udfs/<name>/<default_version>.sql`\n   (rename the versioned udf file to the branch's version).\n3. Add the `#include \"udfs/<name>/<default_version>.sql\"` line to the branch's **dev upgrade\n   migration** (`citus--<prev>--<default_version>.sql`) — same include the main commit added to\n   `14.0-1--15.0-1.sql`, just in the branch's file.\n4. Add the reverse `DROP FUNCTION IF EXISTS <fully-qualified-signature>;` (or DROP for whatever\n   object) to the branch's **dev downgrade** (`citus--<default_version>--<prev>.sql`).\n5. Keep `sql/udfs/<name>/latest.sql` — refresh it to identical content as the versioned file.\n\n`ci/check_migration_files.sh` (\"Check for missing downgrade scripts\") gates step 4 — every upgrade\nstep needs a matching downgrade. The CI `check-sql-snapshots` job validates these manual edits, so\ngreen there is strong confirmation the remap is correct.\n\n## Step 2 — fix `multi_extension.out`\n`multi_extension` walks the whole upgrade+downgrade ladder and prints a \"Snapshot of state at\n`<version>`\" section per version. The new object's row must land under the **target branch's**\nversion section (e.g. `13.4-1`), NOT main's `15.0-1`. The snapshot table's column widths are\ndynamic — **regenerate the .out with the runner, don't hand-align** it. From `src/test/regress`:\n\n```\npython citus_tests/run_test.py multi_extension\n```\n\nThe new line typically shows the new function/object with `(1 row)` (or the appropriate count) in\nthe target-version section.\n\n## Step 3 — fix `upgrade_list_citus_objects.out` (edit BY CONSTRUCTION)\nThis test **cannot be run standalone** via `run_test.py` — it depends on a `citus_schema` created\nby an earlier test in `after_pg_upgrade_with_columnar_schedule`; run alone it dies with\n`ERROR: no schema has been selected to create in`. So validate its edit by construction:\n- the new function line auto-merges alphabetically in the cherry-pick (leave it);\n- fix only the trailing **row COUNT** = base_count + (number of new citus objects). Read the base\n  count from the branch's own expected file. Observed: release-13.2 `378 → 379`, release-14.0\n  `384 → 385` for one new UDF.\n\n## Step 4 — build WITH downgrades, then test\n`make install` does NOT install downgrade scripts, and `multi_extension` exercises the downgrade\npath, so after every rebuild run both:\n\n```\nmake -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install\nmake -C src/backend/distributed install-downgrades\nmake -C src/backend/columnar    install-downgrades\n```\n\n(`columnar` is a SEPARATE extension with its own downgrades; omitting it surfaces as a\n`citus_columnar` \"no update path\" error. `make install-all` runs both.)\n\nThen run the feature's own test + `multi_extension`.\n\n## Step 5 — propagate the upgrade ladder ACROSS branches (MVU safety)\n\nBackporting the object DOWN to a release branch is not enough. An MVU replays the migration\n**ladder**, so a new SQL object must be reachable on EVERY higher branch's ladder too — otherwise\na cluster upgraded 13.x→14.y would silently lose it. Net effect: **the same object is defined\nMULTIPLE times per branch, once per major's introduction step.** This is intentional, not\nduplication to \"fix\".\n\n**Topology** (worked example: `citus_internal.distribute_object`, introduced on main at\n`14.0-1--15.0-1`, backported to 13.2 at `13.3-1--13.4-1` and to 14.0 at `14.1-1--14.2-1`):\n\n| Ladder step (+ its downgrade + versioned udf) | release-13.2 | release-14.0 | main |\n|---|---|---|---|\n| `13.3-1--13.4-1` (udf `13.4-1`) — 13.x introduction | ✅ (the backport itself) | ✅ copy here | ✅ copy here |\n| `14.1-1--14.2-1` (udf `14.2-1`) — 14.x introduction | n/a | ✅ (the backport itself) | ✅ copy here |\n| `14.0-1--15.0-1` (udf `15.0-1`) — main's own top    | n/a | n/a | ✅ already there (orig land) — LEAVE UNTOUCHED |\n\nRule of thumb: **a step introduced on major N must be copied onto every branch for major ≥ N.**\n13.x paths → release-14.0 + main; 14.x paths → main; main's own top path stays as merged.\n\n**How to do it (per higher branch, e.g. release-14.0 then main):**\n1. This is a **SEPARATE commit** on top of the (unchanged) cherry-pick — never amend the\n   cherry-pick. On main it's its own PR.\n2. Copy the step + downgrade + versioned-udf files **byte-for-byte** from the branch that owns\n   the introduction. `git show <branch>:<path>` reads any branch's blob without checking it out\n   (worktrees of one clone share an object store, so this is trivial):\n   ```\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/citus--13.3-1--13.4-1.sql          > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/downgrades/citus--13.4-1--13.3-1.sql > <same path>\n   git show bp-<pr>-release-13.2:src/backend/distributed/sql/udfs/<name>/13.4-1.sql             > <same path>\n   ```\n   (and the analogous `14.1-1--14.2-1` trio from `bp-<pr>-release-14.0` when copying onto main.)\n3. **Keep the header verbatim** — even the `-- bump version to 13.4-1` line stays on downstream\n   branches where 13.4 is not the default. Confirmed identical across branches in the #8621 UDF family.\n4. **Purely additive.** Do NOT touch `citus.control` / `default_version`. Do NOT touch the\n   `13.2-1--14.0-1` major-bridge migration. main's `latest.sql` still points at its own top\n   version (15.0-1); leave it. `check_sql_snapshots.sh` verifies latest.sql == highest udf.\n5. New step files contain ONLY the object being propagated (header + single `#include`); the\n   downgrade is header + `DROP FUNCTION IF EXISTS <signature>;`. Don't drag along unrelated objects\n   that happen to share the source branch's step file.\n6. Build + `install-downgrades` + `multi_extension` on the higher branch (expect PASS, usually no\n   `.out` change needed since the new sections are additive). Run the three migration CI checks\n   locally before pushing: `ci/check_migration_files.sh`, `ci/check_sql_snapshots.sh`,\n   `ci/disallow_c_comments_in_migrations.sh` (all must exit 0).\n7. **Add `multi_extension.sql` ladder-test coverage for the new step** (or prove it's already\n   covered) — see the dedicated section below.\n\n**Template PR family** (study these to see the end-state, don't reverse-engineer from one PR):\nfind the original feature PR, then every PR that mentions it in title/desc. Worked example — the\n`citus_internal.distribute_object` repair UDF, introduced on main in **#8621**: backported\none-branch-per-target via **#8684**→release-13.2 and **#8685**→release-14.0 (each branch also carries\nthe sibling GUC PR **#8625** `citus.allow_unsafe_insert_select_pushdown`, unsquashed as its own\ncommit), with the 13.x/14.x upgrade paths propagated to `main` in **#8686**. The main ladder PR\n(#8686, \"Add 13.x/14.x upgrade paths for citus_internal.distribute_object to main\") is the model for\nthe main ladder commit's title/body; it's internal plumbing so usually **no `DESCRIPTION:` line**.\n\n## multi_extension.sql ladder tests for new steps (off-walk detour pattern)\n\nAdding a ladder step (Step 5 backport OR Step 5 propagation) is not fully covered until\n`src/test/regress/sql/multi_extension.sql` exercises it — or you can prove the existing walk already\ndoes. Model the additions on the template PR family, adapted per branch.\n\n- **The test walks ONE path only** — the main upgrade path (`13.2 → 13.3 → 14.0 → 14.1 → 15.0`).\n  A step you add for MVU may sit **off** that walk:\n  - the 13.x maintenance-tail step `13.3-1--13.4-1` (off-walk on release-14.0 and main), and\n  - the 14.x maintenance-tail step `14.1-1--14.2-1` (off-walk on **main**, whose walk jumps 14.1 → 15.0).\n  Off-walk steps' upgrade AND downgrade scripts are otherwise **never executed** by the test.\n- **On-walk / top-of-branch steps need NOTHING extra.** A branch's own `default_version` (13.4-1 on\n  release-13.2, 14.2-1 on release-14.0, 15.0-1 on main) is already the target of the existing\n  `-- Snapshot of state at X` block; the propagated object simply appears in that snapshot's `.out`.\n  ⇒ release-13.2 needed **no** `multi_extension.sql` change; release-14.0 needed **one** detour\n  (13.4-1); main needed **two** (13.4-1 and 14.2-1).\n- **PATTERN = round-trip no-op → snapshot at the detour → rejoin the walk FROM the detour.** Match\n  the shape of the existing on-walk blocks — visit the off-walk version the same way every other\n  version is visited, so the additions look like they belong. Do NOT add editorial parentheticals\n  (no \"(13.x maintenance line, off the ... path)\"). At the point the walk sits at the detour's\n  PARENT, insert two blocks AND retarget the *existing* next round-trip so it bounces off the\n  detour instead of the parent:\n  ```sql\n  -- Test downgrade to <parent> from <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';   -- e.g. 13.4-1\n  ALTER EXTENSION citus UPDATE TO '<parent>';   -- e.g. 13.3-1\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Snapshot of state at <detour>\n  ALTER EXTENSION citus UPDATE TO '<detour>';\n  SELECT * FROM multi_extension.print_extension_changes();\n\n  -- Test downgrade to <detour> from <next>   <-- was \"<parent> from <next>\"; retarget it\n  ALTER EXTENSION citus UPDATE TO '<next>';    -- the next ON-WALK version (e.g. 14.0-1 / 15.0-1)\n  ALTER EXTENSION citus UPDATE TO '<detour>';  -- bounce back to the detour, NOT the parent\n  -- Should be empty result since upgrade+downgrade should be a no-op\n  SELECT * FROM multi_extension.print_extension_changes();\n  ```\n  Block 1 (round-trip) proves the up+down pair is a perfect no-op. Block 2 (snapshot) shows the new\n  object actually appearing at the detour version (`+ function citus_internal.distribute_object...`,\n  `(1 row)`). The snapshot leaves the walk sitting at `<detour>` with the `print_extension_changes()`\n  baseline `= {detour}`. Instead of adding a separate \"return to parent\" block, you **retarget the\n  existing next round-trip** (which used to bounce `<parent> ↔ <next>`) so it bounces `<detour> ↔\n  <next>` — this keeps the walk moving forward from the detour and lands it back at `<detour>` with\n  baseline reset to `{detour}`.\n- **The one intended downstream change:** because the walk now enters the following `-- Snapshot of\n  state at <next>` block with baseline `{detour}` (not `{parent}`), that ONE snapshot shifts by\n  exactly the detour's object, and nothing else changes:\n  - If `<next>` does NOT contain the object (e.g. `14.0-1`, a 13.x-line object): the snapshot GAINS a\n    `previous_object` row (the object shown as dropped when leaving the maintenance line).\n  - If `<next>` DOES contain the object (e.g. `15.0-1` when the object is already at `14.2-1`): the\n    snapshot LOSES the `current_object` row it used to show (it's no longer newly-added — it was\n    already displayed at the detour snapshot).\n  Both are self-consistent and desirable: the object is introduced exactly once, at its detour\n  snapshot. Every block AFTER that `<next>` snapshot stays byte-identical (print resets baseline to\n  `{next}` regardless). Regenerate `.out` and eyeball that only that single snapshot moved.\n- **`.out` regen:** edit `.sql` → runner \"fails\" (writes `results/multi_extension.out`) →\n  `cp results/multi_extension.out expected/multi_extension.out` → re-run → \"All 1 tests passed\".\n  **Never hand-edit `.out`.**\n- **`diff -w` header gotcha:** pg_regress compares with `diff -dU10 -w`, so snapshot-table header\n  column widths / leading spaces are IGNORED for pass/fail. A regen may normalize a pre-existing\n  HAND-EDITED narrow header (left over from an earlier `.out` hand-edit) to the runner's canonical\n  wider spacing. Whitespace-only and harmless — **keep the canonical runner output**, note it in the\n  commit message. Prove it's pre-existing: run the test on pristine HEAD (no change) — same wide\n  header, still PASSES.\n- **APPEND-ONLY once reviewed:** if the backport branch/PR has already been reviewed, do NOT\n  force-push — add the `multi_extension` test work as a new commit on top.\n\n## Flaky / gotcha notes for SQL backports\n- **`multi_extension` flaky maintenance-daemon crash:** the issue #3409 `DROP EXTENSION` under a\n  non-superuser role section can report \"server process was terminated\" once, then PASS on a clean\n  re-run. If the PG log shows only a clean fast-shutdown (no PANIC / segfault / Assert), it's\n  timing/flaky — re-run before investigating. Sanity-check by running the same test on the PRISTINE\n  base branch.\n- **`configure` gets regenerated by builds** — never part of a backport. `git checkout -- configure`\n  before committing.\n- **`git stash -u` mid-cherry-pick** drops `CHERRY_PICK_HEAD` but keeps the file changes; on restore\n  you must `git commit` manually and re-add the original message + `(cherry picked from commit <sha>)`.\n- **Regenerate .out with the runner, never hand-edit** — snapshot column widths and object counts\n  are computed.\n\n<!-- File: .github/skills/citus-backport/SKILL.md (Tokens: ~3359 | Category: anthropic-skill) -->\n---\nname: citus-backport\ndescription: >-\n  Backport one or more merged citusdata/citus `main` commits/PRs onto the active release\n  branches. Covers the release model, cherry-picking, remapping SQL schema changes to each\n  branch's default_version, the SQL upgrade/downgrade ladder for Major-Version-Upgrade safety,\n  running regression tests, one clean commit per PR per branch, and triaging release-branch CI\n  (pre-existing baseline reds vs reds the backport introduced). USE WHEN asked to backport / port\n  / cherry-pick a change to release-13.2 / release-14.0 (or the newest two majors), when a\n  backport hits a SQL migration / udf / multi_extension conflict, when a \"trivial C-only\" backport\n  fails to compile on an older PG the release branch still supports (e.g. PG15 `rteperminfos`),\n  when release-branch CI is red and you must prove which reds are pre-existing, or when you must\n  prove a SQL-schema backport is Major-Version-Upgrade-safe via `ALTER EXTENSION citus UPDATE`\n  across a real multi-node cluster. DO NOT USE for non-Citus repos, new features, or PR review.\nlicense: See the repository LICENSE file.\n---\n\n# Backport a Citus change to the release branches\n\nThis skill encodes how to backport merged `main` commit(s) to Citus release branches: the release\nmodel, the SQL-schema remap recipe, the PG-version compatibility traps, and how to triage the\nrelease branches' baseline-red CI. Follow the repository's normal build/test/style conventions\n(see `CONTRIBUTING.md` and `src/test/regress/README.md`) on top of the guidance here.\n\n## Reference deep-dives (load on demand)\nAll live next to this file under `references/`:\n- **`references/sql-schema-backport.md`** — the hard case: a commit that changes the SQL schema\n  (new UDF / column / migration). Exact file-by-file remap recipe, the `multi_extension.out` and\n  `upgrade_list_citus_objects.out` edits, the `install-downgrades` trap, and the cross-branch\n  **ladder-propagation** step (copy the introduction paths UP to every higher major for MVU safety).\n- **`references/ci-triage.md`** — reading backport CI: the PG-version compile traps (main drops\n  old PGs the release branch keeps), the N-1 mixed-version jobs, and the step-by-step proof that a\n  red job is pre-existing baseline noise vs a real regression you introduced.\n- **`references/manual-upgrade-testing.md`** — the manual MVU proof: stand up a multi-node cluster\n  at an OLD Citus major, then `ALTER EXTENSION citus UPDATE` across the backport branches (13.x\n  tail → cross-major → 14.x tail), capturing `citus_version()` + `pg_extension.extversion` per node\n  per hop. The `make install-all` / single-shared-PG requirement.\n\nA C-only backport usually needs only this hub + `ci-triage.md`. A SQL-schema backport needs all\nfour (add `manual-upgrade-testing.md` as the final proof once CI is green).\n\n## Release model (internalize this first)\n- **ONE branch per MAJOR.** `release-13.2` serves the WHOLE 13.x line; `release-14.0` serves ALL of\n  14.x; `main` is the next major. Citus no longer cuts a branch per minor (this changed ~2026-05).\n- **Backport FEATURES, not just bugfixes**, to the **newest two majors** (13 & 14 as of this\n  writing).\n- **Branch names LIE about the version.** `release-13.2` currently ships 13.4; `release-14.0` ships\n  14.2. **NEVER infer the target minor from the branch name** — read `default_version` from\n  `src/backend/distributed/citus.control` on each target branch. That value drives every SQL edit.\n- **N-1 compatibility is required and CI-gated.** A backport is N-1-safe when it is purely\n  ADDITIVE (new UDF / GUC / column) and does not change an existing UDF signature or wire format\n  used cross-version. If it's additive, it's safe. See `references/ci-triage.md` for the jobs.\n\n## Workflow (per PR, per target branch)\n1. **Use an isolated build per target major.** A dedicated checkout/worktree + its own PostgreSQL\n   install per release branch keeps builds/tests from clobbering each other and your main checkout.\n   Two worktrees of the same clone SHARE one git object store — all branches are visible/pushable\n   from either, and a `+`-prefixed branch in `git branch` is checked out in a sibling worktree.\n2. **Make the SHAs reachable**, then `git checkout -b bp-<pr>-<release-branch> origin/<release-branch>`\n   and `git cherry-pick -x <sha>` (the `-x` records `(cherry picked from commit <sha>)`).\n3. **Resolve:**\n   - **C-only:** usually applies clean or with trivial context conflicts. THEN check PG-version\n     compat (`references/ci-triage.md` §PG-version) — the #1 non-obvious C-backport failure.\n   - **SQL-schema:** migration / udf / `multi_extension.out` conflict → remap by hand per\n     `references/sql-schema-backport.md`. Do NOT just accept `main`'s next-major (e.g. `15.0-1`) files.\n4. **Build in the isolated env:** `make -sj\"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)\" install`. For any SQL change ALSO install\n   downgrades (see `references/sql-schema-backport.md` — plain `make install` skips them and\n   `multi_extension` will fail with cascading \"no update path\" errors otherwise; use\n   `make install-all`).\n5. **Test only what's relevant** (CI runs the whole suite): the feature's own test + `multi_extension`\n   for SQL changes. From `src/test/regress`: `python citus_tests/run_test.py <test>` (it manages its\n   own cluster). Regenerate any `expected/*.out` with the runner — **never hand-edit .out** (column\n   widths / row counts are dynamic).\n6. **Collapse to ONE commit per PR per branch.** Keep the original message; keep a `DESCRIPTION:`\n   first line iff the change is user-facing (this feeds the changelog). `git checkout -- configure`\n   if a build regenerated it (configure is never part of a backport). **Multiple PRs to the SAME\n   target → ONE branch per target, not per PR** (unsquashed: one commit per PR, in main's\n   chronological merge order; SQL-ladder commits sit right after their PR's cherry-pick). Name it\n   `bp-release-<major>` (e.g. `bp-release-13.2`). Build cleanly from the SQL-heavy PR's branch, then\n   cherry-pick the C-only PRs on top. **Cherry-pick an already-adapted backport commit WITHOUT\n   `-x`** — its message already carries the single `(cherry picked from <orig>)` trailer; `-x` would\n   double it.\n7. **Push to your fork.** `bp-*` branches don't match CI's push trigger (`main`/`release-*` only),\n   so kick CI with `gh workflow run \"Build & Test\" --repo <your-fork>/citus --ref <branch>`. Open\n   PRs only when the change owner asks.\n8. **Wait for CI and triage** per `references/ci-triage.md`. Separate baseline-red noise (N-1 /\n   flaky / coverage) from **genuinely-new failures you introduced**. The two you WILL hit on a\n   release branch: (a) a version-pinned `.out` that prints a shared helper's changed *signature*, and\n   (b) a PG-version planner divergence on the OLDER PG the branch keeps but `main` dropped (PREFER a\n   PG-guarded file like `pg16.sql` over a `<test>_N.out` alt file). A third you must NOT chase green\n   by mangling the test: a new GUC/UDF whose test runs under N-1 shows a NEW red in the (non-blocking)\n   **N-1 mixed-version** jobs — the old lib lacks it and falls back, which IS the N-1 contract. The\n   clean fix is to move that test line into the `multi_1_create_citus_schedule` placeholder section\n   (schedule mechanics + N-1 label rule in the hard-rules below and `ci-triage.md §\"NEW N-1 red\"`).\n   **Fix-commit rule (all fixes):** pre-review → amend + `git push --force-with-lease`; post-review →\n   add ONE new fix commit, never rewrite reviewed SHAs (replace a bad remote fix commit with\n   `--force-with-lease=<branch>:<badsha>`). Report evidence; don't declare done on red.\n9. **Propagate the ladder UP (SQL-schema backports only).** A new SQL object must also be reachable\n   on every HIGHER major's upgrade ladder, or a future 13.x→14.y Major Version Upgrade loses it.\n   As SEPARATE commits (its own PR on main), copy the introduction step/downgrade/udf files\n   byte-for-byte onto release-14.0 (for a 13.x path) and main (for 13.x and 14.x paths, given today's\n   two most recent major versions); main's own top path stays untouched. Full topology + recipe in\n   `references/sql-schema-backport.md` §Step 5.\n10. **Add `multi_extension.sql` ladder-test coverage** for every new step. The test only walks the\n    main upgrade path, so a step OFF that path (a 13.x/14.x maintenance-tail detour) is never\n    exercised unless you add a detour block matching the on-walk shape (and the file's existing\n    comment style — no editorial parentheticals): a round-trip no-op + a snapshot at the detour,\n    then retarget the *existing* next round-trip to bounce `<detour> ↔ <next>` rather than adding a\n    separate \"return\" block. That shifts exactly one downstream snapshot; everything after stays\n    byte-identical. A branch's own top/default_version is already covered by its snapshot block.\n    Never hand-edit `.out`; prefer append-only on reviewed branches. Full pattern + the `diff -w`\n    header gotcha in `references/sql-schema-backport.md`.\n11. **Prove MVU safety on a real cluster (SQL-schema backports, after CI is green).** The ladder\n    propagation only *claims* a future 13→14 Major Version Upgrade keeps the object; prove it. Stand\n    up a multi-node cluster at an OLD major, then `ALTER EXTENSION citus UPDATE` on EVERY node across\n    the backport branches (13.x tail → cross-major → 14.x tail → cross-major → main tail), capturing\n    `citus_version()` and `pg_extension.extversion` per node per hop, plus a functional check that the\n    backported object survived. Build all Citus versions into ONE shared PG with **`make install-all`**\n    (plain `install` omits the downgrade + bridge scripts the cross-major route needs → \"no update path\").\n    Full recipe and the single-PG requirement in `references/manual-upgrade-testing.md`.\n\n## Hard rules / traps (full detail in the references)\n- **Do NOT run `make reindent` in a build environment whose `citus_indent` version differs from the\n  branch's** — a version-skewed formatter reformats 100+ unrelated files and errors out. Trust\n  cherry-picked formatting; CI `check-style` verifies it. If you ran it by accident:\n  `git reset --hard HEAD`.\n- **`make install` does NOT install downgrade scripts.** For any SQL change run BOTH\n  `make -C src/backend/distributed install-downgrades` and `make -C src/backend/columnar\n  install-downgrades` (columnar is a separate extension). `make install-all` does both.\n- **A single build environment builds ONE PG only** — it cannot catch a compile break on another PG\n  the release branch supports. Rely on CI's per-PG `Build for PGNN` jobs; scan cherry-picked C for\n  struct fields / APIs newer than the branch's OLDEST PG and add `#if PG_VERSION_NUM >= PG_VERSION_NN`.\n- **The release branches are BASELINE-RED.** Several N-1 / flaky jobs fail on the pristine base\n  branch with no backport. Subtract the baseline before blaming your change — prove it with the\n  upstream `citusdata/citus` base-branch run. Details + exact commands in `references/ci-triage.md`.\n- **A new SQL object is defined MULTIPLE times on purpose.** After ladder propagation the same UDF\n  appears in the 13.x, 14.x AND main introduction steps (triple-defined on main). That is\n  intentional MVU/N-1 safety, NOT duplication to collapse. Never \"dedupe\" it. Details:\n  `references/sql-schema-backport.md` §Step 5.\n- **Show evidence.** Present the proof (build-job outcomes, `regression.diffs` excerpts, the\n  upstream-baseline comparison), not a bare \"CI is green/red\".\n- **A backport onto an older-PG release branch surfaces failures `main` never saw** (release-13.2\n  still builds PG15; the branch ships version-pinned `expected/*.out`). Two patterns: (1) a shared\n  helper whose SIGNATURE you changed prints in a pinned `.out` (`normalize.sed` masks the line\n  NUMBER, not the signature) — do NOT \"fix\" it by reverting the helper if the backported TEST needs\n  it (it's required, not over-reach); (2) a PG15-vs-PG16+ EXPLAIN/planner diff → PREFER a PG-guarded\n  file (`pg16.sql` `\\q`-skips <PG16 and already lives in the N-1-excluded schedule) over a\n  `<test>_0.out` alt. Full recipe: `references/ci-triage.md` §Genuinely-new failures.\n- **A NEW GUC/UDF whose test runs under N-1 has a CLEAN fix, not \"leave it\".** Move the test line\n  from its schedule into the `multi_1_create_citus_schedule` placeholder section (N-1 make_targets\n  omit `check-multi-1-create-citus`). N-1 version label = the branch's CURRENT N-1 = the previous\n  minor (13.3-1 on release-13.2, 14.1-1 on release-14.0), read from the live `citus_version:` /\n  `citus_libdir:` pin in build_and_test.yml — not the (possibly-stale) number in the neighbor comment.\n  Details: `references/ci-triage.md` §\"NEW N-1 red\".\n\n<!-- File: .github/skills/README.md (Tokens: ~622 | Category: anthropic-skill) -->\n# Agent skills for citus\n\nThis directory holds **agent skills** — folders of instructions (and optionally\nscripts/resources) that an AI coding agent can load on demand when it works on a\nparticular kind of task in this repository. Skills let us capture hard-won,\nrepo-specific know-how once and have every agent (and human) reuse it, instead of\nre-deriving it from scratch each time.\n\n## Layout\n\nEach skill lives in its own subdirectory containing a `SKILL.md`:\n\n```\n.github/skills/\n├── README.md                     ← this file\n└── citus-backport/\n    ├── SKILL.md                  ← entry point (loaded first)\n    └── references/               ← deep-dive docs loaded on demand\n        ├── sql-schema-backport.md\n        ├── ci-triage.md\n        └── manual-upgrade-testing.md\n```\n\n- Subdirectory names are lowercase with hyphens.\n- The entry file **must** be named `SKILL.md`.\n- Keep `SKILL.md` a concise hub; push long procedures into `references/` files it\n  links to, so the agent only pulls in what it needs.\n\n## `SKILL.md` format\n\nA `SKILL.md` is Markdown with YAML frontmatter:\n\n```markdown\n---\nname: my-skill                 # required; lowercase-hyphenated, matches the folder\ndescription: >-                # required; WHAT it does + WHEN to use it (drives auto-selection)\n  One or two sentences describing the task this skill helps with and the\n  trigger conditions that should make an agent load it.\nlicense: See the repository LICENSE file.   # optional\n---\n\n# Human-readable title\n\nInstructions, examples, and guidelines for the agent to follow…\n```\n\nThe `description` is the most important field for discovery: agents match a user's\nrequest against it, so state both the capability and the trigger phrases plainly.\n\n## Where these are picked up\n\nProject skills under `.github/skills/` are discovered automatically by GitHub\nCopilot CLI. (The Agent Skills format is intentionally portable; other agent tools\nmay read `.claude/skills/` or `.agents/skills/` instead — see each tool's docs.)\n\n`.github/copilot-instructions.md` (auto-loaded repo-wide) also carries a short\n`task → skill` index so agents are pointed here even before they browse this\ndirectory. When you add a skill, add a row there too.\n\n## Adding a new skill\n\n1. Create `./<skill-name>/SKILL.md` with valid frontmatter.\n2. Put anything long or optional in `./<skill-name>/references/` and link to it\n   from `SKILL.md`.\n3. Keep it repo-specific and evidence-based; avoid machine- or user-specific paths\n   so it works for anyone who clones citus.\n4. Add a `task → skill` row to `.github/copilot-instructions.md` so agents discover it.\n\n<!-- File: .github/copilot-instructions.md (Tokens: ~422 | Category: copilot-instructions) -->\n# Copilot instructions for citus\n\nFollow the repository's normal contribution conventions: see `CONTRIBUTING.md` for\nbuild/style/PR rules and `src/test/regress/README.md` for the regression-test workflow.\n\n## Agent skills\n\nThis repo ships task-specific **agent skills** under `.github/skills/`. Before starting\na task that matches one, load that skill's `SKILL.md` and follow it so you don't re-derive\nit. Each skill's `SKILL.md` frontmatter `description` states exactly when to use it; the\nindex below is a quick map.\n\n| Task | Skill |\n|------|-------|\n| Backport a merged `main` commit/PR to a release branch (`release-13.2` / `release-14.0` / newest two majors) — including SQL-schema migrations, the upgrade/downgrade ladder, N-1 / Major-Version-Upgrade safety, and triaging release-branch CI | [`citus-backport`](skills/citus-backport/SKILL.md) |\n| Merge a batch of Citus PRs one at a time: sync each with its base branch, fix a red `check-style` job, retry other red checks with give-up rules, then squash-merge with the PR description as the commit message | [`citus-merge-loop`](skills/citus-merge-loop/SKILL.md) |\n| Fix a failing `check-style` job on whatever branch is already checked out, by rebuilding the exact `uncrustify`/`citus_indent` versions that branch itself pins (in its own `STYLEGUIDE.md` and `.github/workflows/build_and_test.yml`), then running `make reindent` and committing the style-only diff locally (no push, and no switching to or inspecting any other Citus branch) | [`citus-check-style-reindent`](skills/citus-check-style-reindent/SKILL.md) |\n\nSee `.github/skills/README.md` for the skills layout and how to add a new one.\n\n<!-- File: src/backend/distributed/commands/README.md (Tokens: ~646 | Category: command-prompt) -->\n# Commands\n\nThe commands module is modeled after `backend/commands` from the postgres repository and\ncontains the logic for Citus on how to run these commands on distributed objects. Even\nthough the structure of the directory has some resemblence to its postgres relative, files\nhere are somewhat more fine-grained. This is due to the nature of citus commands that are\nheavily focused on distributed tables. Instead of having all commands in `tablecmds.c`\nthey are often moved to files that are named after the command.\n\n| File                         | Description |\n|------------------------------|-------------|\n| `create_distributed_table.c` | Implementation of UDF's for creating distributed tables |\n| `drop_distributed_table.c`   | Implementation for dropping metadata for partitions of distributed tables |\n| `extension.c`                | Implementation of `CREATE EXTENSION` commands for citus specific checks |\n| `foreign_constraint.c`       | Implementation of and helper functions for foreign key constraints |\n| `grant.c`                    | Implementation of `GRANT` commands for roles/users on relations |\n| `index.c`                    | Implementation of commands specific to indices on distributed tables |\n| `multi_copy.c`               | Implementation of `COPY` command. There are multiple different copy modes which are described in detail below |\n| `policy.c`                   | Implementation of `CREATE\\ALTER POLICY` commands. |\n| `rename.c`                   | Implementation of `ALTER ... RENAME ...` commands. It implements the renaming of applicable objects, otherwise provides the user with a warning |\n| `schema.c`                   | |\n| `sequence.c`                 | Implementation of `CREATE/ALTER SEQUENCE` commands. Primarily checks correctness of sequence statements as they are not propagated to the worker nodes |\n| `table.c`                    | |\n| `transmit.c`                 | Implementation of `COPY` commands with `format transmit` set in the options. This format is used to transfer files from one node to another node |\n| `truncate.c`                 | Implementation of `TRUNCATE` commands on distributed tables |\n| `utility_hook.c`             | This is the entry point from postgres into the commands module of citus. It contains the implementation that gets registered in postgres' `ProcessUtility_hook` callback to extends the functionality of the original ProcessUtility. This code is used to route the incoming commands to their respective implementation in Citus |\n| `vacuum.c`                   | Implementation of `VACUUM` commands on distributed tables |\n\n# COPY\n\nThe copy command is overloaded for a couple of use-cases specific to citus. The syntax of\nthe command stays the same, however the implementation might slightly differ from the\nstock implementation. The overloading is mostly done via extra options that Citus uses to\nindicate how to operate the copy. The options used are described below.\n\n## FORMAT transmit\n\nImplemented in `transmit.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n\n## FORMAT result\n\nImplemented in `multi_copy.c`\n\nTODO: to be written by someone with enough knowledge to write how, when and why it is used.\n</yakaai_skills>"}