{"owner":"mozilla-services","repo":"syncstorage-rs","hasSkills":true,"totalSkillsCount":34,"totalTokensCount":59069,"categories":["command-prompt","anthropic-skill","claude-rule","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":[".claude/commands/env-check.md",".claude/commands/git-history-check.md",".claude/commands/migration-review.md",".claude/commands/pr-description.md",".claude/commands/py-checks.md",".claude/commands/rust-backend-check.md",".claude/skills/jira-status/SKILL.md",".claude/skills/multi-backend-consistency/SKILL.md",".claude/skills/pytest-fixture-review/SKILL.md",".claude/skills/rust-code-smell/SKILL.md",".claude/skills/rust-error-review/SKILL.md",".claude/skills/sentry-health/SKILL.md",".claude/skills/spanner-local-tests/SKILL.md",".claude/skills/syncserver-explain-code/SKILL.md",".claude/skills/syncserver-jira-description/SKILL.md","CLAUDE.md","docs/src/tools/load-testing.md","docs/src/tools/make_hawk_token.md","docs/src/tools/payload-offload-infrastructure.md","docs/src/tools/payload-offload-load-testing.md","docs/src/tools/payload_link_reconciler.md","docs/src/tools/postgres_purge_ttl.md","docs/src/tools/purge_old_records_tokenserver.md","tools/README.md","tools/examples/README.md","tools/hawk/README.md","tools/payload-link-dataflow/README.md","tools/payload-link-dataflow/payload-link-publisher-py/README.md","tools/payload-reconciler/README.md","tools/postgres/README.md","tools/spanner/README.md","tools/syncstorage-loadtest/README.md","tools/tokenserver/README.md","tools/tokenserver/loadtests/README.md"],"skills":{".claude/commands/env-check.md":"Verify all required environment variables are set before running integration tests.\n\nYou are a pre-flight assistant for syncstorage-rs integration tests. Your job is to check that the shell environment is correctly configured so tests will actually run, and surface any missing or suspicious values before the user wastes time on a failing test run.\n\nDo NOT print or log the values of any secrets. Use placeholders like `[set]` or `[not set]` when reporting on secret variables.\n\n## Variables to check\n\n### Required for all integration test runs\n\n| Variable | Expected | Notes |\n|---|---|---|\n| `SYNC_MASTER_SECRET` | any non-empty string | secret — report [set] or [not set] only |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | starts with `mysql://`, `postgresql://`, or `spanner://` | |\n| `SYNC_TOKENSERVER__DATABASE_URL` | starts with `mysql://` or `postgresql://` | |\n| `TOKENSERVER_HOST` | starts with `http://` or `https://` | default: `http://localhost:8000` |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | starts with `http://` or `https://` | default: `http://localhost:6000` for local |\n| `PYTHONPATH` | contains `tools/` path | required for PyO3 and test imports |\n\n### Required only for real FxA e2e tests (`test_e2e.py`)\n\n| Variable | Notes |\n|---|---|\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E` | secret — report [set] or [not set] only |\n\n## Step 1 — Check each variable\n\n```bash\necho \"SYNC_MASTER_SECRET: $([ -n \"$SYNC_MASTER_SECRET\" ] && echo '[set]' || echo '[NOT SET]')\"\necho \"SYNC_SYNCSTORAGE__DATABASE_URL: ${SYNC_SYNCSTORAGE__DATABASE_URL:-[NOT SET]}\"\necho \"SYNC_TOKENSERVER__DATABASE_URL: ${SYNC_TOKENSERVER__DATABASE_URL:-[NOT SET]}\"\necho \"TOKENSERVER_HOST: ${TOKENSERVER_HOST:-[NOT SET - default: http://localhost:8000]}\"\necho \"SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: ${SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL:-[NOT SET - default: http://localhost:6000]}\"\necho \"PYTHONPATH: ${PYTHONPATH:-[NOT SET]}\"\n```\n\n## Step 2 — Validate database URL format\n\nFor `SYNC_SYNCSTORAGE__DATABASE_URL` and `SYNC_TOKENSERVER__DATABASE_URL`:\n- Check the scheme matches a supported backend (`mysql`, `postgresql`, `spanner`)\n- For `postgres://` prefix: note that SQLAlchemy 1.4+ requires `postgresql://` — flag this if present\n- For Spanner: confirm `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` is set if the URL contains `test-project` or `emulator`\n\n## Step 3 — Check PYTHONPATH contains tools/\n\n```bash\npython3 -c \"import tools.integration_tests\" 2>&1\n```\n\nIf this fails, the `PYTHONPATH` is not set correctly for test imports.\n\n## Step 4 — Check server reachability (if TOKENSERVER_HOST is set)\n\n```bash\ncurl -sf \"${TOKENSERVER_HOST:-http://localhost:8000}/__heartbeat__\" && echo \"server: reachable\" || echo \"server: NOT REACHABLE\"\n```\n\n## Output format\n\nReport each variable as **OK**, **WARNING**, or **MISSING**:\n\n| Variable | Status | Detail |\n|---|---|---|\n\nThen list any blockers (MISSING required vars, unreachable server) followed by warnings (format issues, defaults in use).\n\nEnd with **Ready to run tests** or **N issues need to be resolved**.\n",".claude/commands/git-history-check.md":"Examine git history of files changed in the current branch to identify regressions, re-introduced bugs, or changes that conflict with past fixes.\n\nYou are a senior engineer doing a history-aware code review. Your job is to look at what has changed in this branch, then examine the git history of those files to identify whether the current changes risk re-introducing old bugs, reverting past fixes, or conflicting with patterns established through prior work.\n\n## Step 1 — Get the current diff\n\n```bash\ngit diff main...HEAD\n```\n\n## Step 2 — Get the list of changed files\n\n```bash\ngit diff main...HEAD --name-only\n```\n\n## Step 3 — For each changed file, examine recent commit history\n\n```bash\ngit log --oneline -20 -- \n```\n\n## Step 4 — Inspect relevant commits in full\n\nFor each commit that looks relevant (bug fixes, reverts, security patches, refactors on the same lines), inspect the full diff:\n\n```bash\ngit show \n```\n\n## Step 5 — Look for reverted or fix commits\n\n```bash\ngit log --oneline --all --grep=\"revert\" -- \ngit log --oneline --all --grep=\"fix\" -- \n```\n\nUse your judgment about which historical commits are worth digging into. Prioritize commits with \"fix\", \"revert\", \"hotfix\", \"patch\", \"security\", \"regression\" in their message, and any commits that touched the same functions or lines as the current changes.\n\n## Analysis checklist\n\nFor each changed file, work through the following. Report findings with:\n- **Severity:** High / Medium / Low\n- **Location:** file:line\n- **Issue:** what the historical context reveals\n- **Relevant commit(s):** hash + message\n- **Recommendation:** what to verify or change\n\n### 1. Re-introduced bugs\n- Does the current diff restore code that was previously removed by a bug fix?\n- Are there commits in the history that explicitly fixed logic that the current change modifies or removes?\n\n### 2. Reverted or rolled-back patterns\n- Has this code been reverted before? If so, why — and does the current change repeat the same pattern?\n- Are there \"Revert X\" commits in the history that suggest a prior attempt at this change failed?\n\n### 3. Previously fixed security issues\n- Does the history show security-related fixes on these lines?\n- Does the current change touch or weaken those protections?\n- Pay particular attention to: HAWK/JWT token handling, FxA OAuth validation, gRPC error suppression logic, DB migration constraints.\n\n### 4. Repeated churn\n- Has this file or function been modified many times in a short period? High churn is a signal of instability.\n- Does the current change look like it continues a pattern of patching symptoms rather than fixing the root cause?\n\n### 5. Conflicting intent\n- Do commit messages indicate a deliberate design decision that the current change reverses without explanation?\n- Are there TODOs or FIXMEs introduced by prior commits that the current change should have addressed but didn't?\n\n### 6. Migration or deprecation conflicts\n- Does the history show a migration away from a pattern that the current change re-introduces?\n- Examples: callback → async, legacy auth → FxA OAuth, MySQL-only → multi-backend.\n\n### 7. Test regressions\n- Were tests previously added to cover a bug that the current change might invalidate?\n- Do any historical fix commits include test additions that are now at risk of being bypassed?\n\n## Output format\n\nLead with a summary table:\n\n| Severity | File | Issue | Relevant commit |\n|---|---|---|---|\n\nFollow with per-file detail. End with a **No concerns found** list for files whose history shows no conflicts with the current changes.\n\nIf the git history is shallow or sparse for a file, note that and flag it as low confidence.\n",".claude/commands/migration-review.md":"Review database migration changes against the repo's published-migration policy.\n\nYou are a database migration reviewer for syncstorage-rs. The repo has a hard rule: **never edit a published migration file**. Only new forward migrations with a corresponding rollback are allowed. Your job is to enforce this and catch violations before merge.\n\n## Migration locations\n\n```\nsyncstorage-mysql/migrations/ # up.sql / down.sql per timestamped dir\nsyncstorage-postgres/migrations/\ntokenserver-mysql/migrations/\ntokenserver-postgres/migrations/\nsyncstorage-spanner/src/schema.ddl # Spanner schema — treated as published\nsyncstorage-spanner/src/*.sql # inline Spanner queries\n```\n\nEach migration directory is named `YYYY-MM-DD-hhmmss_description/` and contains `up.sql` and `down.sql`.\n\n## Step 1 — Find changed migration files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"migrations/|schema\\.ddl|\\.sql\"\n```\n\n## Step 2 — Classify each changed file\n\nFor each changed SQL file:\n\n- **New migration directory** (path did not exist on main) → allowed, proceed to Step 3\n- **Existing migration file modified** (path existed on main and content changed) → **VIOLATION** — flag as a blocker\n- **schema.ddl modified** → **VIOLATION** — flag as a blocker\n- **Inline .sql query file modified** (non-migration) → review for intent, flag if it changes schema\n\nCheck whether a file existed on main:\n```bash\ngit show main:\n```\n\n## Step 3 — Validate new migrations\n\nFor each new migration directory:\n\n1. Confirm both `up.sql` and `down.sql` exist\n2. Confirm `down.sql` is a genuine rollback of `up.sql` (not empty, not a no-op)\n3. Check that the timestamp prefix is newer than all existing migrations in that crate\n4. Check for destructive operations in `up.sql` (`DROP TABLE`, `DROP COLUMN`, `TRUNCATE`) — flag these for explicit review\n5. Check that column types and constraints are consistent with existing schema in the same crate\n\n## Step 4 — Cross-backend consistency\n\nIf migrations exist for both mysql and postgres variants of the same crate, verify the schema changes are semantically equivalent (same columns, types mapped appropriately, same constraints).\n\n## Output format\n\n**Violations (blockers):**\nList any edits to published files with file path and the nature of the change.\n\n**New migrations:**\nFor each new migration: name, what it does, whether down.sql is valid, any destructive ops.\n\n**Warnings:**\nNon-blocking concerns (e.g. missing index, risky destructive op, cross-backend inconsistency).\n\n**Verdict:** Pass / Violations found\n",".claude/commands/pr-description.md":"Generate a PR title and summary from the current branch diff, following the repo's commit and PR conventions.\n\nYou are a PR description writer for syncstorage-rs. Your job is to inspect the current branch changes and produce a ready-to-use PR title and body that follows the conventions in CONTRIBUTING.md.\n\n## Step 1 — Gather branch context\n\n```bash\ngit diff main...HEAD --name-only\ngit log main...HEAD --oneline\ngit diff main...HEAD --stat\n```\n\n## Step 2 — Read the full diff for context\n\n```bash\ngit diff main...HEAD\n```\n\nFocus on understanding *what changed and why*, not just listing files.\n\n## Step 3 — Classify the change type\n\nBased on changed files, determine the primary `type`:\n\n- `feat` — new functionality added\n- `fix` — bug fix\n- `refactor` — code restructured, no behavior change\n- `test` — test additions or changes only\n- `chore` — build, CI, tooling, dependencies\n- `docs` — documentation only\n- `perf` — performance improvement\n- `style` — formatting, lint fixes only\n\nIf multiple types apply, use the most significant one.\n\n## Step 4 — Determine scope (optional)\n\nIf the change is clearly scoped to one subsystem, add it in parentheses:\n- `tokenserver`, `syncstorage`, `spanner`, `mysql`, `postgres`, `auth`, `migrations`, `ci`, `tools`\n\nExample: `refactor(tokenserver): ...`\n\n## Step 5 — Draft the PR title\n\nRules from CONTRIBUTING.md:\n- Format: `type(scope): subject` or `type: subject`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Keep it under 70 characters\n\n## Step 6 — Draft the PR body\n\n```markdown\n## Summary\n\n<2–4 bullet points covering what changed and why. Lead with the motivating\nproblem or context, then the solution. Skip obvious restatements of the title.>\n\n## Test plan\n\n\n```\n\nPre-fill the test plan based on file types changed:\n\n- Rust source changed → `- [ ] \\`make clippy_\\` passes`, `- [ ] Unit tests pass (\\`make test\\`)`\n- Python source changed → `- [ ] \\`make ruff-lint\\` and \\`make mypy\\` pass`, `- [ ] Integration tests pass (\\`make run_local_e2e_tests\\`)`\n- SQL migrations added → `- [ ] \\`/migration-review\\` passes`, `- [ ] Migration applies cleanly on a fresh DB`\n- `.github/` changed → `- [ ] CI workflow runs successfully on this branch`\n- `CLAUDE.md` or docs changed → `- [ ] Renders correctly on GitHub`\n\n## Step 7 — Output\n\nPrint the final PR title and body, ready to paste into GitHub. Also include the suggested branch name if the current branch name doesn't follow `type/description-STOR-####` or `type/description-####` format.\n\nNote any breaking changes that should be called out explicitly.\n",".claude/commands/py-checks.md":"Run all Python lint, format, type, and security checks for the tools/ directory.\n\nYou are a Python code quality assistant for the syncstorage-rs tools and integration tests. Your job is to run the full suite of Python checks in the correct order, report findings clearly, and tell the user exactly what needs to be fixed before the PR can merge.\n\nAll checks operate on the `tools/` directory. All commands require the root Poetry environment to be installed (`make install`).\n\n## Step 1 — Check Poetry environment\n\n```bash\npoetry env info\n```\n\nIf no environment is active, tell the user to run `make install` first and stop.\n\n## Step 2 — Format check\n\n```bash\nmake ruff-fmt-chk\n```\n\nIf there are formatting issues, tell the user to run `make ruff-format` to auto-fix, then re-run. Do not proceed past format failures.\n\n## Step 3 — Lint\n\n```bash\nmake ruff-lint\n```\n\nReport each warning/error with file:line and rule code.\n\n## Step 4 — Type checking\n\n```bash\nmake mypy\n```\n\nReport type errors with file:line. Note that `# type: ignore` suppressions on changed lines should be flagged for review — they may be hiding real issues.\n\n## Step 5 — Docstring validation\n\n```bash\nmake pydocstyle\n```\n\nReport missing or malformed docstrings. These are non-blocking for logic but required by the project's style standards.\n\n## Step 6 — Security scan\n\n```bash\nmake bandit\n```\n\nFlag any **HIGH** severity findings as blockers. List **MEDIUM** as warnings.\n\n## Output format\n\nReport each check as **PASS**, **WARNINGS**, or **FAIL**:\n\n| Check | Status | Issues |\n|---|---|---|\n| Format (ruff) | | |\n| Lint (ruff) | | |\n| Types (mypy) | | |\n| Docstrings (pydocstyle) | | |\n| Security (bandit) | | |\n\nList all actionable findings below the table with file:line references.\n\nEnd with **All checks passed** or **N checks need attention** verdict.\n",".claude/commands/rust-backend-check.md":"Run Clippy, fmt check, and cargo audit for a specific Rust backend.\n\nYou are a Rust build and lint assistant for the syncstorage-rs workspace. Your job is to run the correct set of checks for a given backend, report findings clearly, and flag anything that needs attention before the code is merged.\n\n## Step 1 — Determine the backend\n\nAsk the user which backend to check if not already specified: `mysql`, `postgres`, or `spanner`. If the current branch name or open files suggest a backend, infer it and confirm.\n\n## Step 2 — Run format check\n\n```bash\ncargo fmt -- --check\n```\n\nReport any files that are not formatted. If there are formatting issues, stop and tell the user to run `cargo fmt` before proceeding.\n\n## Step 3 — Run Clippy for the chosen backend\n\nUse the exact make target:\n\n```bash\nmake clippy_mysql # for mysql\nmake clippy_postgres # for postgres\nmake clippy_spanner # for spanner\n```\n\n## Step 4 — Run release-mode Clippy\n\n```bash\nmake clippy_release_mysql # for mysql\nmake clippy_release_postgres # for postgres\nmake clippy_release_spanner # for spanner\n```\n\nRelease mode catches dead code and issues only visible with optimizations enabled.\n\n## Step 5 — Run cargo audit\n\n```bash\ncargo audit\n```\n\nFlag any advisories with severity **high** or **critical** as blockers. List medium/low advisories as warnings.\n\n## Output format\n\nSummarize findings in this order:\n\n1. **Format** — pass or list of unformatted files\n2. **Clippy (debug)** — pass or list of warnings/errors with file:line\n3. **Clippy (release)** — pass or list of warnings/errors with file:line\n4. **Audit** — pass or list of advisories (severity, crate, advisory ID)\n\nEnd with a clear **Ready to merge** / **Needs attention** verdict.\n",".claude/skills/jira-status/SKILL.md":"---\nname: jira-status\ndescription: Queries, summarizes, creates, and edits Jira issues for syncstorage-rs (project STOR). Supports user-centric work views, epic breakdowns, structured summaries with impact/context/highlights, and ticket creation or modification from natural language.\nuser-invocable: true\nargument-hint: \"me | epic | epics | sprint | summary | create | update | status (default: status)\"\n---\n\nStart by reading `.claude/mcp-context.md` for project IDs, JQL patterns, issue types, and field conventions.\n\nThen call `mcp__atlassian__getAccessibleAtlassianResources` to confirm the Jira cloud ID and base URL for `mozilla-hub.atlassian.net`. Store the base URL as `JIRA_BASE` — all issue links are formatted as `JIRA_BASE/browse/STOR-####`.\n\nParse `$ARGUMENTS` (trimmed). Match against the modes below. Default is `status`.\n\n---\n\n## Mode: `status` (default)\n\nOverall project task landscape. Use when no argument is given.\n\nRun these JQL queries in parallel:\n\n```jql\n-- Open sprint issues\nproject = STOR AND sprint in openSprints() AND statusCategory != Done ORDER BY priority DESC, updated DESC\n\n-- Unassigned open issues\nproject = STOR AND statusCategory != Done AND assignee is EMPTY ORDER BY priority DESC\n\n-- Recently updated (last 3 days)\nproject = STOR AND updated >= -3d AND statusCategory != Done ORDER BY updated DESC\n```\n\n**Output:**\n\n### Project Status\nOne-line overall signal: `Active` / `Needs attention` / `Stalled`.\n\n### Current Sprint\nTable: Key (linked) | Summary | Status | Assignee | Priority\nGroup by status category (In Progress first, then To Do, then Done).\n\n### Unassigned Open Issues\nTable: Key (linked) | Summary | Priority | Issue Type\nFlag Blockers and Criticals explicitly.\n\n### Recently Updated\nTable: Key (linked) | Summary | Updated | By\n\n---\n\n## Mode: `me`\n\nWork assigned to the person running this skill. Ask: \"What is your Jira email or display name?\" — use their response to filter.\n\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory != Done ORDER BY updated DESC\n```\n\nAlso fetch recently completed by the user (for summary context):\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory = Done AND resolutiondate >= -30d ORDER BY resolutiondate DESC\n```\n\n**Output:**\n\n### Your Active Work\nTable: Key (linked) | Summary | Status | Priority | Last Updated\nHighlight anything In Progress that hasn't been updated in > 3 days.\n\n### Your Recent Completions (Last 30 Days)\nTable: Key (linked) | Summary | Resolved On | Issue Type\n\n### Work Summary\nA plain-English paragraph for each active item: what it is, where it stands, and whether anything is blocking it. Lead with the highest-priority item.\n\n---\n\n## Mode: `epic `\n\nRetrieve a specific epic and all its child issues.\n\nSteps:\n1. Call `mcp__atlassian__getJiraIssue` with the provided KEY to get the epic details.\n2. Fetch all children:\n ```jql\n project = STOR AND parent = ORDER BY priority DESC, status ASC\n ```\n3. Also fetch linked issues (blocks/is blocked by) on the epic itself.\n\n**Output:**\n\n### Epic: [STOR-####](JIRA_BASE/browse/STOR-####) — {Epic Summary}\n**Status:** {status} | **Assignee:** {assignee or Unassigned} | **Priority:** {priority}\n\n**Description:**\n{epic description, condensed to key intent — 2–4 sentences max}\n\n### Child Issues\nTable: Key (linked) | Summary | Status | Assignee | Priority | Issue Type\nGroup: In Progress → To Do → Done (collapsed count for Done if > 5)\n\n### Epic Progress\n- Total issues: N\n- Done: N (N%)\n- In Progress: N\n- Blocked: N (flag if any)\n\n### Blockers & Dependencies\nList any linked \"blocks\" or \"is blocked by\" relationships. If none, omit this section.\n\n---\n\n## Mode: `epics`\n\nList all open epics in the STOR project with their child counts and progress.\n\n```jql\nproject = STOR AND issuetype = Epic AND statusCategory != Done ORDER BY priority DESC, updated DESC\n```\n\nFor each epic, fetch child counts via:\n```jql\nproject = STOR AND parent = \n```\n(batch these — fetch all children in one query with `parent in (KEY1, KEY2, ...)` if the tool supports it, otherwise query each epic individually)\n\n**Output:**\n\n### Open Epics\n\nFor each epic:\n**[STOR-####](link) — {Summary}**\nStatus: {status} | Assignee: {assignee} | Children: {done}/{total}\nOne-line description of the epic's goal (from its description field, condensed).\n\n---\n\n## Mode: `sprint`\n\nCurrent sprint state focused on progress and blockers — not health framing, just what is happening.\n\n```jql\nproject = STOR AND sprint in openSprints() ORDER BY statusCategory ASC, priority DESC\n```\n\n**Output:**\n\n### Sprint: {Sprint Name}\nDates: {start} → {end} | Days remaining: N\n\n**In Progress** (N issues)\nTable: Key (linked) | Summary | Assignee | Days In Progress\n\n**To Do** (N issues)\nTable: Key (linked) | Summary | Assignee | Priority\n\n**Done** (N issues)\nCount only unless ≤ 5 — then list.\n\n**Blockers in sprint:**\nList any issue linked as \"blocks\" another sprint issue, or with priority Blocker.\n\n---\n\n## Mode: `summary `\n\nGenerate a structured summary suitable for a status update, PR description, or stakeholder report. The argument can be:\n- A single issue key: `summary STOR-1234`\n- An epic key: `summary STOR-100` (auto-detects if it's an epic)\n- A freeform scope description: `summary tokenserver auth work` (runs JQL to find matching issues)\n- `summary me` — summarize the current user's recent work (ask for their name if not known)\n- `summary sprint` — summarize the current sprint's work\n\n**Steps:**\n1. Resolve the scope to a set of issues. Fetch full details for each.\n2. For epics: fetch children too.\n3. Identify: what was done, what is in progress, what is blocked.\n4. Synthesize the summary.\n\n**Output format — always structured as:**\n\n### Impact\nWhat this work changes for users or the system. Lead with the most significant outcome. Use concrete language: \"reduces\", \"enables\", \"removes\", \"fixes\". If there are multiple issues, consolidate into the top 2–3 outcomes. Link relevant issues inline: e.g. \"Adds PostgreSQL support for tokenserver ([STOR-42](link)).\"\n\n### Context\nWhy this work was needed. Background the reader should have to understand the impact. Reference the driving requirement, bug, or design decision. Keep to 3–5 sentences. Link to the primary issue or epic.\n\n### Highlights\nNotable implementation decisions, technical wins, or interesting constraints resolved. Bullet points. Each item links to the relevant issue. Focus on what would surprise or interest a fellow engineer — not just a list of what was done.\n\n---\n\n## Mode: `create `\n\nCreate a new Jira issue in STOR from natural language.\n\n**Steps:**\n\n1. Parse the description to infer:\n - **Summary** — one clear sentence (≤ 100 chars)\n - **Issue type** — Bug / Task / Story / Spike (infer from language; ask if ambiguous)\n - **Priority** — infer from language (\"critical\", \"blocking\" → Critical; \"nice to have\" → Minor; default → Major). Valid STOR priorities: `Blocker`, `Critical`, `Major`, `Minor`. `Trivial` is rejected by the project — do not use it.\n - **Work category** — `Feature Engineering` / `Engineering Excellence` / `Operational Excellence`. See [Work categories](#work-categories) below for definitions and inference rules. Always classify; if unsure between EE and OpEx, the planned/reactive distinction decides it.\n - **Assignee** — if this `create` invocation is part of the same workflow that opens a GitHub PR for the work, set the assignee to the current user. Look up their accountId via `mcp__atlassian__lookupJiraAccountId` using the email from the runtime `userEmail` context. Otherwise leave unassigned unless the user named a specific person.\n - **Parent** — if the description references an epic (e.g. \"under STOR-506\", \"in STOR-506\", \"linked to \"), set `parent` to that key when calling `createJiraIssue`.\n - **Description body** — expand the user's language into a structured description, leading with the category:\n - **Category:** {one of the three categories} — *one-sentence rationale*\n - **Background:** why this is needed\n - **Acceptance Criteria:** observable, testable outcomes\n - **Notes:** any constraints or references mentioned\n\n2. Show the draft to the user before creating:\n ```\n Ready to create:\n Type: {type}\n Project: STOR\n Parent: {parent key or none}\n Priority: {priority}\n Category: {category}\n Assignee: {display name or \"unassigned\"}\n Summary: {summary}\n Description:\n {description}\n\n Create this issue? (yes / edit first / cancel)\n ```\n\n3. On confirmation, call `mcp__atlassian__createJiraIssue` with:\n - `projectKey: STOR`\n - `issueTypeName: {type}`\n - `summary: {summary}`\n - `description: {description}` with `contentFormat: markdown`\n - `additional_fields: { \"priority\": { \"name\": \"{priority}\" } }`\n - `assignee_account_id: {accountId}` — only when auto-assignment applies or the user named an assignee\n - `parent: {epic_key}` — only when a parent epic was identified\n\n4. Return the created issue key and direct link: `[STOR-####](JIRA_BASE/browse/STOR-####)`\n\n---\n\n## Mode: `update `\n\nModify an existing issue. Changes can be natural language.\n\n**Steps:**\n\n1. Fetch the current issue with `mcp__atlassian__getJiraIssue`.\n\n2. Parse the `` to determine what to modify:\n - Field updates (summary, description, priority, assignee) → use `mcp__atlassian__editJiraIssue`\n - Status transition (\"mark as done\", \"move to in progress\") → use `mcp__atlassian__getTransitionsForJiraIssue` then `mcp__atlassian__transitionJiraIssue`\n - Comment (\"add a comment: ...\") → use `mcp__atlassian__addCommentToJiraIssue`\n\n3. Show the proposed changes before applying:\n ```\n Updating STOR-####: {current summary}\n Changes:\n - {field}: \"{old value}\" → \"{new value}\"\n - {transition}: {new status}\n - Comment: \"{comment text}\"\n\n Apply? (yes / edit / cancel)\n ```\n\n4. Apply on confirmation. Return confirmation with the issue link.\n\n---\n\n## General guidelines\n\n- **All issue references must be linked**: format as `[STOR-####](JIRA_BASE/browse/STOR-####)` — never bare key text\n- **Summaries lead with impact**: what changes for users or the system, not what was done mechanically\n- **Do not fabricate data**: if a query returns nothing, say so plainly\n- **Ask once for missing identity**: if the user's Jira name is needed and not known, ask once; do not repeat the question across modes in the same session\n- **Confirm before write operations**: always show a draft before calling `createJiraIssue`, `editJiraIssue`, `transitionJiraIssue`, or `addCommentToJiraIssue`\n- **Permissions**: create/edit/transition operations require the user's Jira account to have project contributor access to STOR; if a call fails with 403, report it clearly and do not retry\n- **Issue type inference**: \"bug\" → Bug; \"task\"/\"chore\" → Task; \"feature\"/\"story\"/\"as a user\" → Story; \"investigate\"/\"explore\"/\"spike\" → Spike\n- **Classify every ticket**: when creating or summarizing, identify the [work category](#work-categories) (Feature Engineering / Engineering Excellence / Operational Excellence). Include it as the leading line of the description body for created tickets.\n- **Auto-assign when PR-bundled**: if a `create` invocation is part of the same workflow that opens a GitHub PR for the same work, set the assignee to the current user (look up their accountId with `mcp__atlassian__lookupJiraAccountId`). Setting an assignee may transition the issue to In Progress via Jira workflow rules — that's expected.\n\n---\n\n## Work categories\n\nPer [Feature Engineering, Engineering Excellence, and Operational Excellence](https://mozilla-hub.atlassian.net/wiki/spaces/IP/pages/1397817345). Classify every STOR ticket into one of three categories. Target distribution per quarter: ≤ 50% Feature Engineering, ≤ 30% Engineering Excellence, ≤ 20% Operational Excellence.\n\n### Feature Engineering — *build & deliver new things*\n\nPlanned work that changes what a customer (internal or external) sees beyond speed, reliability, or bugfixes.\n\nExamples: new API endpoints, new auth methods (passkeys), launching a region or platform, UI/UX redesigns, A/B testing, expanded user-configuration options, changes to core business logic, bug fixes against unlaunched feature work.\n\n### Engineering Excellence — *sustain & improve velocity*\n\nPlanned work that improves code maintainability, reduces tech debt, or boosts engineering velocity. Most non-hotfix bug fixes land here. \"Keeping the lights on\" proactively.\n\nExamples: refactors, test-coverage improvements, observability and monitoring, CI/CD tooling, dependency upgrades, removing dead code, fixing flaky tests, reducing on-call toil, automating manual deployments, fixes for issues in already-released features.\n\n### Operational Excellence — *react & keep production healthy*\n\nUnplanned work. Reliability, stability, security, and operational efficiency in the production environment. \"Keeping the lights on\" reactively.\n\nExamples: security patches and CVE remediations, manual scaling under load, incident response and postmortems, hotfix patches for production incidents, resolving performance issues impacting SLAs.\n\n### Inference rules\n\n- **Planned vs. unplanned** decides EE vs. OpEx: if it was on a roadmap/sprint plan, it's EE; if it interrupted planned work to keep prod healthy, it's OpEx.\n- **Customer-visible change** decides FE vs. EE: if the customer sees something new (beyond speed/reliability/bugfix), it's FE.\n- **Dependency upgrades, CI/CD tooling, refactors, test improvements** → Engineering Excellence (unless the upgrade is a CVE remediation, then OpEx).\n- **Security patches and CVEs** → Operational Excellence.\n- **Bug fixes**: pre-launch → FE; against released features, planned → EE; via hotfix → OpEx.\n",".claude/skills/multi-backend-consistency/SKILL.md":"---\nname: multi-backend-consistency\ndescription: Checks that trait method changes in syncstorage-db or tokenserver-db are consistently implemented across all backend crates (mysql, postgres, spanner). Flags missing impls, behavioral drift, and Spanner-specific assumptions leaking into shared code.\nuser-invocable: true\n---\n\n# Multi-Backend Consistency Check\n\nYou are a Rust trait/implementation consistency reviewer for syncstorage-rs. Changes to shared traits must be reflected in every backend. Your job is to find gaps before they reach CI.\n\n## Backend map\n\n```\nsyncstorage-db/ ← trait definitions (DbPool, Db, etc.)\n → syncstorage-mysql/\n → syncstorage-postgres/\n → syncstorage-spanner/\n\ntokenserver-db/ ← trait definitions\n → tokenserver-mysql/\n → tokenserver-postgres/\n```\n\n## Step 1 — Find changed trait files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-db/|tokenserver-db/\"\n```\n\nIf no trait files changed, check whether any backend implementation changed without a corresponding trait change — that may indicate the trait was bypassed or duplicated.\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-(mysql|postgres|spanner)/|tokenserver-(mysql|postgres)/\"\n```\n\n## Step 2 — Diff the traits\n\nFor each changed trait file:\n\n```bash\ngit diff main...HEAD -- \n```\n\nList every added, removed, or modified method signature.\n\n## Step 3 — Check each backend for the same change\n\nFor each modified method, grep for its name in every backend crate:\n\n```bash\ngrep -rn \"fn \" syncstorage-mysql/src/ syncstorage-postgres/src/ syncstorage-spanner/src/\ngrep -rn \"fn \" tokenserver-mysql/src/ tokenserver-postgres/src/\n```\n\nFor each backend, determine:\n- **Present and updated** — matches the new signature ✓\n- **Present but stale** — exists but hasn't been updated to match ✗\n- **Missing** — not implemented at all ✗\n\n## Step 4 — Check for cross-backend behavioral assumptions\n\nReview the diff for patterns that are safe on one backend but wrong on another:\n\n- **Auto-increment IDs** — MySQL/Postgres use serial/auto-increment; Spanner uses explicit INT64. Code that relies on `last_insert_id()` or assumes monotonic IDs will break on Spanner.\n- **Transaction semantics** — Spanner transactions are bounded; MySQL/Postgres allow longer-lived transactions. Check for assumptions about transaction scope.\n- **NULL handling** — Spanner is stricter about NULLs in some column types than MySQL.\n- **RETURNING clause** — Postgres supports `RETURNING id`; MySQL uses `last_insert_id()`; Spanner uses a separate read. Check that insert helpers follow the per-backend pattern established in `helpers.py` / existing db impls.\n- **Timestamp precision** — Spanner uses microseconds; MySQL DATETIME has second precision unless explicitly specified.\n- **Feature flags** — confirm the change compiles under `--features=syncstorage-db/mysql`, `--features=syncstorage-db/postgres`, and `--features=syncstorage-db/spanner` independently.\n\n## Step 5 — Check shared db-common crates\n\nChanges to `syncstorage-db-common/` or `tokenserver-db-common/` affect all backends. Verify nothing in those changes makes backend-specific assumptions.\n\n```bash\ngit diff main...HEAD --name-only | grep \"db-common\"\n```\n\n## Output format\n\n**Trait changes detected:**\nList each modified method with its new signature.\n\n**Backend coverage:**\n\n| Method | mysql | postgres | spanner |\n|---|---|---|---|\n| `method_name` | ✓ updated / ✗ stale / ✗ missing | ... | ... |\n\n**Behavioral drift warnings:**\nAny cross-backend assumptions found, with file:line.\n\n**Verdict:** All backends consistent / N gaps found\n",".claude/skills/pytest-fixture-review/SKILL.md":"---\nname: pytest-fixture-review\ndescription: Reviews conftest.py and helpers.py files in tools/integration_tests/ against the repo's fixture/helper separation standard. Flags helpers in conftest, missing teardown, wrong scopes, and raw DB operations outside helper functions.\nuser-invocable: true\n---\n\n# Pytest Fixture Review\n\nYou are a pytest code reviewer for syncstorage-rs integration tests. This repo has a deliberate structure: `conftest.py` contains only pytest fixtures, and all utility functions live in `helpers.py`. Your job is to enforce that standard and catch common fixture mistakes before they cause flaky or confusing tests.\n\n## The standard\n\n```\ntools/integration_tests/\n conftest.py ← fixtures ONLY (st_ctx)\n helpers.py ← retry helpers, auth state, switch_user, constants\n\ntools/integration_tests/tokenserver/\n conftest.py ← fixtures ONLY (ts_db_conn, ts_app, ts_service_id, ts_ctx, fxa_auth)\n helpers.py ← DB helpers, auth helpers, node/user/service helpers\n```\n\nFixtures yield context dicts. Helpers are plain functions imported directly by test files.\n\n## Step 1 — Find changed test files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"tools/integration_tests\"\n```\n\nRead each changed file in full.\n\n## Step 2 — conftest.py checks\n\nFor each `conftest.py` in the diff:\n\n**Non-fixture code in conftest:**\n- Are there any module-level functions that are not decorated with `@pytest.fixture`?\n- Are there utility functions, helper classes, constants, or context managers defined in conftest that belong in `helpers.py`?\n- Exception: module-level side effects that must run at import time (e.g. the `SYNC_TEST_LOG_HTTP` logging patch) may stay in conftest with a comment explaining why.\n\n**Fixture scopes:**\n- `scope=\"function\"` — default, correct for most fixtures; state is reset per test\n- `scope=\"session\"` — only justified for expensive external calls (e.g. `fxa_auth` creates a real FxA account). Any new session-scoped fixture must have a comment explaining why.\n- `scope=\"module\"` or `scope=\"class\"` — flag for review; not currently used in this repo\n\n**Teardown:**\n- Does every fixture that creates state also clean it up?\n- `ts_ctx` pattern: `clear_db()` before yield AND after yield. Verify both sides are present.\n- `fxa_auth` pattern: `acct.clear()` + `client.destroy_account()` in teardown. Verify error handling wraps the teardown.\n- `st_ctx` pattern: `config.end()` + `del os.environ[\"MOZSVC_UUID\"]`. Verify both.\n\n**Database connections:**\n- `ts_db_conn` uses `AUTOCOMMIT` isolation. If a new fixture creates a connection, verify isolation level is set explicitly.\n- Connections must be closed in teardown (`conn.close()`).\n\n## Step 3 — helpers.py checks\n\n**No fixtures in helpers:**\n- Are there any functions decorated with `@pytest.fixture` in `helpers.py`? These must move to `conftest.py`.\n\n**Raw SQL outside helper functions:**\n- Is `sqltext()` / `execute_sql()` called directly in a test file rather than through a helper?\n- Direct SQL in tests bypasses the helper layer and makes tests brittle. Flag with file:line.\n\n**Imports:**\n- Are helpers importing from `conftest`? This creates a circular dependency risk. Helpers should have no pytest imports and no conftest imports.\n\n**Constants:**\n- Are test constants (NODE_ID, NODE_URL, FXA_EMAIL_DOMAIN, etc.) defined in helpers, not scattered in test files or conftest?\n\n## Step 4 — Test file import checks\n\nFor each changed test file:\n\n```bash\ngrep -n \"from.*conftest import\" \n```\n\nTest files should import helpers from `helpers.py`, not from `conftest.py`. Any `from tools.integration_tests.conftest import` or `from integration_tests.tokenserver.conftest import` is wrong — flag it with the correct import path.\n\n## Step 5 — New fixture pattern check\n\nIf a new fixture was added, verify:\n- It follows the `yield`-based teardown pattern (not `return`)\n- It is typed or documented enough that its contents are clear\n- If it seeds database state, it cleans up after itself\n- Its scope is the narrowest possible for its purpose\n\n## Output format\n\n**conftest.py issues:**\nList each violation with file:line and what needs to move or change.\n\n**helpers.py issues:**\nList each violation with file:line.\n\n**Import violations:**\nList any test files importing from conftest instead of helpers.\n\n**New fixtures:**\nFor each new fixture: name, scope, teardown present (yes/no), any concerns.\n\n**Verdict:** Follows conventions / N issues found\n",".claude/skills/rust-code-smell/SKILL.md":"---\nname: rust-code-smell\ndescription: Runs clippy, cargo audit, and all relevant multi-backend Makefile targets for files changed in the current branch. Detects code smells, anti-patterns specific to this codebase, and surfaces issues across every affected backend before CI catches them. Also handles dependency-update verification (Cargo.toml / Cargo.lock changes from dependabot or manual bumps) via the `deps` mode.\nuser-invocable: true\n---\n\n# Rust Code Smell Detector\n\nYou are a Rust code quality reviewer for syncstorage-rs. Your job is to run the full set of mechanical checks relevant to the current branch's changes, then layer on a manual smell pass for patterns that tooling misses. Run checks for every backend touched by the diff — not just the default.\n\n## Standing authorization\n\nWhen this skill is invoked, you have standing authorization to run the following without per-command confirmation:\n\n- `cargo fmt -- --check`\n- `cargo clippy ...` (any feature combination)\n- `make clippy_mysql`, `make clippy_postgres`, `make clippy_spanner`\n- `make clippy_release_mysql`, `make clippy_release_postgres`, `make clippy_release_spanner`\n- `cargo audit`\n- `cargo check --workspace ...`\n- `cargo test --no-run` (compile-check only — do NOT run actual tests without explicit permission, since they require DB setup)\n- `cargo update --precise -p ` (lockfile-only adjustments during bisection)\n- `git stash` / `git stash pop` (for baseline comparison)\n\nDo not run anything destructive (commit, push, rebase, branch deletion, `cargo clean` of the whole workspace, `make run_*`, docker-compose up/down) without asking.\n\n## Mode selection\n\n- **Default mode** — branch under review contains Rust source changes (not just `Cargo.toml`/`Cargo.lock`). Run Steps 1–5 below.\n- **`deps` mode** — branch only changes `Cargo.toml` / `Cargo.lock`, or the user invokes the skill in the context of a dependabot PR / dependency bump. Jump to the \"Dependency update verification\" section near the end and follow that flow instead.\n\n## Step 1 — Identify which backends are affected\n\n```bash\ngit diff main...HEAD --name-only\n```\n\nDetermine which backends the changes touch:\n\n- Files in `syncstorage-mysql/`, `tokenserver-mysql/`, or `syncstorage-db/` with MySQL defaults → **mysql**\n- Files in `syncstorage-postgres/`, `tokenserver-postgres/` → **postgres**\n- Files in `syncstorage-spanner/` → **spanner**\n- Files in `syncserver/`, `syncserver-common/`, `syncserver-settings/`, `syncstorage-db-common/`, `tokenserver-auth/`, `tokenserver-common/` → **all three backends**\n\n## Step 2 — Format check\n\n```bash\ncargo fmt -- --check\n```\n\nIf formatting issues exist, stop and report them. Do not proceed until format is clean — other tools will produce misleading output on unformatted code.\n\n## Step 3 — Run Clippy for each affected backend\n\nRun only the backends identified in Step 1. Use the exact Makefile targets:\n\n```bash\nmake clippy_mysql # if mysql affected\nmake clippy_postgres # if postgres affected\nmake clippy_spanner # if spanner affected\n```\n\nThen run release-mode Clippy for each affected backend (catches dead code, unused imports with optimizations, and issues only visible at `--release`):\n\n```bash\nmake clippy_release_mysql # if mysql affected\nmake clippy_release_postgres # if postgres affected\nmake clippy_release_spanner # if spanner affected\n```\n\nCollect all warnings and errors. Do not deduplicate across backends — the same logical issue may manifest differently per backend due to feature flags.\n\n## Step 4 — Security audit\n\n```bash\ncargo audit\n```\n\nFlag any **critical** or **high** severity advisories as blockers. List **medium** as warnings. Note the affected crate and advisory ID for each.\n\n## Step 5 — Manual smell pass\n\nRead the diff:\n\n```bash\ngit diff main...HEAD\n```\n\nCheck for the following patterns that Clippy does not catch:\n\n### Async/blocking\n- `std::thread::sleep` inside an async function — use `tokio::time::sleep`\n- Blocking DB calls inside async handlers without `spawn_blocking`\n- `.unwrap()` or `.expect()` on `Result`/`Option` in production paths (test code is fine)\n\n### gRPC / Spanner-specific\n- `grpcio::RpcStatus` errors caught and silently dropped without either Sentry classification or metric emission — see `is_sentry_event()` / `is_ignored_internal()` pattern\n- Retry logic that doesn't cap attempts or uses `std::thread::sleep` instead of exponential backoff\n- Spanner mutations not batched where they could be\n\n### Error handling\n- `map_err(|_| ...)` that discards the original error without logging — information loss\n- `unwrap()` on lock acquisition (implies the lock is poisonable)\n- New `From` impls that convert specific errors into generic `InternalError` without preserving context\n\n### Trait object vs generic\n- `Box` or `Arc` allocations in hot paths where a generic would avoid the vtable overhead\n- Inconsistency with how the rest of the codebase handles the same trait (check existing usage)\n\n### Diesel / query patterns\n- N+1 query patterns: loops that issue individual DB calls instead of a single query\n- Missing index hints or queries on unindexed columns in high-traffic paths\n- Raw string SQL (`sqltext(...)`) where a typed Diesel query builder already exists\n\n### Configuration / secrets\n- Secrets or credentials hardcoded or defaulting to non-placeholder values outside of test modules\n- Environment variable reads outside of the settings structs (should go through `syncserver-settings`)\n\n### Feature flag hygiene\n- `#[cfg(feature = \"...\")]` on items that should be available to all backends\n- Code that compiles under one feature but silently no-ops under another without a clear comment\n\n## Output format\n\n**Format:** pass / fail (list files)\n\n**Clippy results per backend:**\n\n| Backend | Debug | Release | Issues |\n|---|---|---|---|\n| mysql | pass/fail | pass/fail | N warnings/errors |\n| postgres | pass/fail | pass/fail | |\n| spanner | pass/fail | pass/fail | |\n\n**Audit:** pass / N advisories (list crate, severity, advisory ID)\n\n**Manual smells found:**\nList each with file:line, smell type, and a one-line explanation.\n\n**Verdict:** Clean / N issues need attention — list blockers vs warnings separately.\n\n---\n\n# Dependency update verification (`deps` mode)\n\nUse this flow when the branch only changes `Cargo.toml` / `Cargo.lock`, the user is evaluating a dependabot PR, or the user is splitting a grouped dependency PR into smaller pieces.\n\n## When to enter this mode\n\n- User invokes the skill on a dependabot PR (e.g. mozilla-services/syncstorage-rs#NNNN).\n- `git diff main...HEAD --name-only` returns only `Cargo.toml` and/or `Cargo.lock`.\n- User explicitly says \"check this dep bump\" / \"verify the cargo updates\" / \"is this PR safe\".\n\n## Step D1 — Establish the baseline\n\nThe single most valuable signal is whether the gates already pass on `main` before the bump. Stash, test, restore:\n\n```bash\ngit stash\nmake clippy_mysql 2>&1 | tail -10\ngit stash pop\n```\n\nIf `main` already fails a gate, stop and surface that to the user — the bump did not introduce the failure and pretending otherwise wastes time.\n\n## Step D2 — Classify the bumps\n\nCategorize every changed crate before running anything else. Read the manifest diff plus dependabot's release notes (in the PR body) and assign each crate to one of:\n\n- **Routine** — patch/minor bumps to leaf-ish deps (`env_logger`, `reqwest`, `utoipa`, `uuid`, `cadence`, `jsonwebtoken`, `config`, `pyo3` patches). Group these.\n- **Ecosystem-coupled** — crates whose versions must move together. In this repo:\n - `diesel-async` and `deadpool` are paired — Cargo.toml comments already say so.\n - `diesel_migrations` patch bumps can interact with `diesel-async`'s major version (a 2.3.2 bump broke `IntoUpdateTarget` trait selection against diesel-async 0.7; the same bump composes fine with diesel-async 0.9). Treat `diesel_migrations` as ecosystem-coupled even though it looks like a patch.\n - `protobuf` for the Spanner backend must match what `google-cloud-rust-raw` resolves. Do not bump `syncstorage-spanner`'s pinned `protobuf` ahead of `google-cloud-rust-raw`.\n- **Major / breaking** — any `x.0 → y.0` or `0.x → 0.y` bump on a direct dep. Flag for human review even if compilation passes.\n\nRecommend splitting a grouped dependabot PR along these lines: one PR per category, with the riskier categories held until their constraints clear.\n\n## Step D3 — Apply bumps surgically\n\nWhen applying a chosen group, use `--precise` per crate so transitive deps don't drift:\n\n```bash\ncargo update --precise -p \n```\n\nDo not run a bare `cargo update -p ` — that will pull in unrelated transitive bumps (e.g. `hashlink 0.10 → 0.11`, `anstream 0.6 → 1.0`) and you'll then have to debug whether those broke something. `--precise` keeps the diff to exactly what was requested.\n\nCargo only accepts one `--precise` per invocation. To apply N crates, run N sequential `cargo update --precise` commands.\n\nIf two crates share a version qualifier (`reqwest@0.13.2`), cargo will demand the disambiguator — use the form it suggests in the error.\n\nAfter every cargo-update batch, run:\n\n```bash\ngit diff origin/master -- Cargo.lock | grep -E '^[-+]version = ' | sort | uniq -c | sort -rn\n```\n\nVerify the version-change set matches the intended group. Anything you did not request showing up is a red flag.\n\n## Step D4 — Run the gates\n\nRun, in order, stopping at the first failure:\n\n```bash\ncargo fmt -- --check\ncargo audit\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\ncargo test --workspace --no-run # compile-check only\n```\n\n`make test` (running the suite) requires a real MySQL on `localhost:3306` with `sample_user:sample_password` and the schemas pre-created. If the user's environment doesn't have it, report that explicitly and recommend `make docker_run_mysql_e2e_tests` / `make docker_run_postgres_e2e_tests` for full verification. Do not attempt to set up the DB.\n\n`cargo test --no-run` also requires `libpython3.9` linkable on disk because `pyo3` is a default-feature dep. If that fails at the link stage with `library 'python3.9' not found`, report it as a local env issue, not a bump regression. Confirm by running the same `--no-run` against the baseline (Step D1) — if it fails there too, it's not your problem.\n\n## Step D5 — Bisect on failure\n\nIf a gate fails, do not start guessing. Bisect:\n\n1. Revert the *most likely* single crate to its pre-bump version with `cargo update --precise -p `.\n2. Re-run only the failing gate. If it now passes, that crate is the culprit. If not, restore it and try the next candidate.\n3. Likely-culprit ordering for diesel/diesel-async-related failures: `diesel_migrations` first, then `uuid` (it's a diesel feature type), then anything else that touches DB.\n4. For trait-selection / type-inference errors in `syncstorage-postgres` or `syncstorage-mysql` under a *different* backend's clippy target (e.g. clippy_mysql failing in syncstorage-postgres source), the failure is almost always an ecosystem-coupled crate moving alone. Re-classify and move it to the appropriate group.\n\nWhen a crate is identified as incompatible with the current group, remove it from the group, return it to the manifest's previous version, and update the Jira ticket (or PR description) to note it's been moved to whichever group it now belongs with.\n\n## Step D6 — Report\n\nOutput one table:\n\n| Crate | Old | New | Status |\n|---|---|---|---|\n| ... | ... | ... | applied / held / failed / moved to Group X |\n\nPlus the verification matrix:\n\n| Gate | Result | Notes |\n|---|---|---|\n| `cargo fmt --check` | pass/fail | |\n| `cargo audit` | pass/fail | N advisories listed if any |\n| `make clippy_mysql` | pass/fail | |\n| `make clippy_postgres` | pass/fail | |\n| `make clippy_spanner` | pass/fail | |\n| `cargo test --no-run` | pass/fail/env-skip | note libpython if env-skipped |\n\nEnd with one paragraph: what landed, what was excluded and why, and what should ride with the next group.\n\n## Known coupling notes for this repo\n\nKeep these in mind when classifying bumps — they cost real time to rediscover:\n\n- **`diesel-async` + `deadpool`** are paired. Cargo.toml comments say so. Do not move one without the other.\n- **`diesel_migrations` 2.3.2** breaks `IntoUpdateTarget` trait selection in `syncstorage-postgres` source when paired with `diesel-async 0.7`. It needs to ride with the `diesel-async 0.9` upgrade.\n- **`protobuf`** is pinned with `=` in `syncstorage-spanner/Cargo.toml` deliberately. The pin must match what `google-cloud-rust-raw` resolves to. Currently both should be `2.28.0`. Bumping the spanner pin ahead of `google-cloud-rust-raw` will cause two protobuf majors in the lockfile and break compilation (RepeatedField doesn't exist in protobuf 3.x).\n- **`make clippy_mysql` / `clippy_postgres` / `clippy_spanner`** all use `--workspace --all-targets` — every backend crate compiles under every feature set. A failure in `syncstorage-postgres` source under `clippy_mysql` is not a misconfiguration; it means a workspace-shared dep changed something diesel-postgres can't tolerate.\n- **`pyo3` + `libpython3.9` link** is required for default-feature builds on this repo. Local `cargo test --no-run` will fail at link without it. Docker e2e is the fallback for full test verification.\n",".claude/skills/rust-error-review/SKILL.md":"---\nname: rust-error-review\ndescription: Reviews new or modified error variants in syncstorage-rs for correct Sentry routing, HTTP status mapping, and response safety. Enforces the is_sentry_event() / maybe_emit_metrics() classification pattern.\nuser-invocable: true\n---\n\n# Rust Error Review\n\nYou are a Rust error handling reviewer for syncstorage-rs. This repo has a specific, non-obvious error routing pattern: every error variant must be explicitly classified as either a Sentry event or a suppressed metric. Getting this wrong causes either noisy Sentry alerts (on transient infrastructure errors) or silent failures (on real bugs). Your job is to catch misclassification before merge.\n\n## The pattern\n\n```\nSyncstorageError / DbError\n → is_sentry_event() → true → sent to Sentry\n → is_sentry_event() → false → maybe_emit_metrics() → StatsD counter\n```\n\nKey files:\n- `syncstorage-spanner/src/error.rs` — `DbError`, `DbErrorKind`, `is_sentry_event()`, `is_ignored_internal()`\n- `syncstorage-db-common/src/error.rs` — shared error types\n- `syncserver/src/error.rs` — top-level `ApiError`, HTTP response mapping\n- `tokenserver-common/src/error.rs` — tokenserver error types\n\n## Step 1 — Find changed error files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"error\\.rs\"\n```\n\nAlso check for new `From<>` impls that convert external errors into repo error types:\n\n```bash\ngit diff main...HEAD | grep -E \"impl From|Into<.*Error\"\n```\n\n## Step 2 — Diff each error file\n\n```bash\ngit diff main...HEAD -- \n```\n\n## Step 3 — Review checklist\n\nFor each new or modified error variant:\n\n### Sentry classification\n- Is there a new arm in `is_sentry_event()` for this variant?\n- Is the classification correct?\n - **Should be Sentry:** logic bugs, unexpected states, data integrity issues, auth failures that indicate an attack, panics\n - **Should be metric only:** transient infrastructure errors (gRPC RST_STREAM, connection timeouts, 503s from upstream), expected client errors (404, 409 conflict, quota exceeded)\n- For `INTERNAL` gRPC errors: is `is_ignored_internal()` doing substring matching (`.iter().any(|s| msg.contains(s))`)? Exact equality will miss real RST_STREAM messages.\n\n### HTTP status mapping\n- Does the new variant map to an appropriate HTTP status in the `From for ApiError` or `status_code()` impl?\n- Are client errors (4xx) distinguished from server errors (5xx)?\n- Is a 500 being returned for something that should be a 409 or 404?\n\n### Response body safety\n- Does the error's `Display` or serialization leak internal state (SQL query text, file paths, internal IDs, stack traces)?\n- Check `ApiError`'s response serialization — internal error messages must not reach the client.\n\n### Backtrace\n- Is `capture_backtrace()` called for new unexpected/internal error variants?\n- Is it *not* called for expected operational errors (quota, not found, conflict)?\n\n### Metric naming\n- If the variant routes to `maybe_emit_metrics()`, is the metric name consistent with existing conventions (`storage.spanner.grpc.internal`, etc.)?\n\n## Step 4 — Check `From` impl chains\n\nNew error types often get wrapped through multiple `From` impls before reaching the HTTP layer. Trace the chain to verify classification survives the conversion:\n\n```bash\ngrep -rn \"From<\" syncstorage-spanner/src/ syncstorage-db/src/ syncserver/src/ | grep -i error\n```\n\n## Output format\n\nFor each changed error variant:\n- **Variant:** name\n- **Classification:** Sentry / Metric / Unclear\n- **HTTP status:** correct / wrong (should be X) / missing\n- **Response safety:** safe / leaks internal detail at file:line\n- **Issues:** list\n\nEnd with **All error variants correctly classified** or **N classification issues found**.\n",".claude/skills/sentry-health/SKILL.md":"---\nname: sentry-health\ndescription: Queries Sentry for syncstorage-rs (project syncstorage-prod) to surface error trends, new issues, regressions, and production health signals. Reports as engineer/manager view of what requires attention.\nuser-invocable: true\nargument-hint: \"new | regressions | volume | full (default: full)\"\n---\n\nYou are acting as **both engineer and engineering manager** for syncstorage-rs. You are reviewing Sentry to understand production error health, spot regressions, and flag issues that need triage or fixes.\n\nStart by reading `.claude/mcp-context.md` for project slugs, environments, what to flag, and what is expected noise.\n\nParse `$ARGUMENTS` (trimmed, lowercased). Default is `full`. Valid modes:\n- `new` — issues first seen in the last 7 days\n- `regressions` — previously resolved issues that have recurred\n- `volume` — high-event-count issues\n- `full` — all of the above\n\n---\n\n## Step 1: Confirm the project\n\nCall `mcp__sentry__find_projects` and locate the `syncstorage-prod` project under the `mozilla` organization. Confirm the project slug and organization slug before querying. If the project is not found, report the available projects and stop.\n\n---\n\n## Step 2: Gather issue data\n\nUse available Sentry MCP tools to fetch issues for `syncstorage-prod`. Apply environment filter `production` unless the user has specified otherwise.\n\nFetch based on the selected mode:\n\n### New issues (`new`, `full`)\nIssues with `firstSeen` in the last 7 days that have not been resolved. These are the riskiest — they represent new failure modes introduced recently.\n\nSort by: event count descending.\n\nFor each, capture:\n- Issue ID and title\n- First seen timestamp\n- Event count in the period\n- Affected users (if available)\n- Platform / culprit (the Rust file/function if available)\n\n### Regressions (`regressions`, `full`)\nIssues that were previously marked resolved but have reoccurred. These indicate incomplete fixes or flaky patches.\n\nFor each, capture:\n- Issue ID and title\n- Original resolution date\n- Recurrence date\n- Event count since recurrence\n\n### High-volume issues (`volume`, `full`)\nIssues with the most events in the last 24 hours, regardless of age. These are the most actively impacting users.\n\nFor each, capture:\n- Issue ID and title\n- Event count (24h)\n- Is this a known/triaged issue or unacknowledged?\n\n---\n\n## Step 3: Classify each issue\n\nFor every issue surfaced, classify it against the known patterns in `.claude/mcp-context.md`:\n\n**Tokenserver auth path** — involves `hawk_identifier`, FxA OAuth, `tokenserver-auth/`, `client_state`, `keys_changed_at`. These are user-facing auth failures — high priority.\n\n**Database layer** — involves `DbError`, `diesel`, `deadpool`, connection pool, migration, query timeout. May indicate infrastructure problems.\n\n**Syncstorage storage path** — involves BSO read/write, collection operations, quota. Usually lower priority unless volume is very high.\n\n**Expected noise** — 409 Conflict, 503 during deployment, staging-environment-only errors. Mark as \"noise\" and exclude from the action list.\n\n**Spanner-specific** — RST_STREAM, gRPC errors in production (not emulator). These should NOT appear in production — flag immediately.\n\n---\n\n## Step 4: Cross-reference with Jira (if context available)\n\nIf recent Jira data is available in this conversation, check whether any Sentry issues correspond to open `STOR-` bugs. Note the link if found.\n\nIf a high-volume or regression Sentry issue has no corresponding Jira ticket, that is a gap — flag it as \"needs ticket\".\n\n---\n\n## Step 5: Output format\n\n### Production Health: [OK / Watch / Degraded]\nOne-line overall assessment.\n\n### New Issues (Last 7 Days)\nTable: Issue ID | Title | First Seen | Events | Classification\nIf none: \"No new issues in the last 7 days.\"\n\n### Regressions\nTable: Issue ID | Title | Resolved On | Recurred On | Events Since\nIf none: \"No regressions detected.\"\n\n### High-Volume Issues (Last 24h)\nTable: Issue ID | Title | Events (24h) | Triaged? | Classification\nExclude known noise (mark separately if many).\n\n### Error Pattern Analysis\nBulleted findings only if genuine patterns are present:\n- Auth path errors (tokenserver risk)\n- DB layer errors (infrastructure risk)\n- Sudden volume spikes (availability risk)\n- Patterns that correlate with a recent deploy or migration\n\n### Recommended Actions\nUp to 5 concrete items. Each: issue ID, what to do, urgency. Skip if no action is warranted.\n\n### Noise / Expected Errors (Acknowledged)\nBrief list of issues seen but classified as expected noise. One line each. This confirms you looked at them.\n\n---\n\n## Guidelines\n\n- Do not fabricate event counts or timestamps — use exactly what Sentry returns\n- If a query fails or a tool is unavailable, note it and work with available data\n- `is_sentry_event()` in `syncserver/src/error.rs` is the canonical routing decision — errors that are HTTP 4xx should generally not be in Sentry at high volume; flag if they are\n- Focus on production environment; staging errors are informational only\n- An issue being \"old\" does not mean it's not important — check event count trends, not just age\n- Sentry issue IDs should be referenced in full (e.g. `SYNCSTORAGE-PROD-####`) if available\n",".claude/skills/spanner-local-tests/SKILL.md":"---\nname: spanner-local-tests\ndescription: Brings up the Spanner emulator, provisions the test database with the project schema, and runs the syncstorage-rs Spanner unit-test suite locally. Repeatable single-command flow plus full from-scratch documentation. Also covers the e2e (docker-compose) path.\nuser-invocable: true\n---\n\n# Run Spanner Tests Locally\n\nSet up a local Spanner emulator + schema, then run the spanner-backed unit tests. The whole loop is < 90 seconds once the emulator image is cached. This is the **fast inner loop** for any change touching `syncstorage-spanner/`.\n\nThere are two test surfaces in this repo:\n\n| Surface | What it covers | Time | Needs |\n|---|---|---|---|\n| **Unit tests** (this skill's default) | Direct calls into `SpannerDb` — exercises every DML path through `update_user_collection_quotas`, batch ops, BSO CRUD, tombstones, quota. | ~30s after warm-up | emulator only |\n| **Docker e2e** | Full syncserver + emulator + tokenserver + mock FxA, hit over HTTP by the Python integration suite. | 5-15 min | freshly-built `app:build` image |\n\nDefault to unit tests. Reach for e2e only when changes touch HTTP routing, token handling, or cross-backend behavior CI would catch first.\n\n## Quick re-run (after first-time setup)\n\nIf the emulator is already up and provisioned:\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nIf the emulator is **not** running, use the helper:\n\n```bash\n./scripts/spanner-local-test-setup.sh\n```\n\nThat script brings up the emulator, provisions the database/schema, and runs the suite. It's idempotent — re-running just executes the tests if the stack is already up.\n\n## From scratch — the full flow\n\n### 1. Prerequisites\n\n- **Docker Desktop running.** The emulator runs as a container.\n- **Rust toolchain** matching `rust-toolchain.toml` (or close).\n- For e2e only: enough Docker memory (≥ 8 GB recommended) — the `app:build` image compiles grpcio-sys from source and OOMs on small allocations.\n\n### 2. Start the emulator\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml up -d sync-db\n```\n\nThis starts the [`gcr.io/cloud-spanner-emulator/emulator`](https://cloud.google.com/spanner/docs/emulator) image. Two ports:\n\n- **9010** — gRPC. The Rust client (and production) talk this.\n- **9020** — REST. Used for admin (create instance / create database / DDL).\n\nWait ~2 seconds for the emulator to be ready. Confirm:\n\n```bash\ncurl -fsS -o /dev/null http://localhost:9020/v1/projects/test-project/instances && echo ok\n```\n\n### 3. Create the instance and database\n\n```bash\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"instance\":{\"config\":\"emulator-test-config\",\"nodeCount\":1,\"displayName\":\"Test\"},\"instanceId\":\"test-instance\"}'\n\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"createStatement\":\"CREATE DATABASE `test-database`\"}'\n```\n\n### 4. Apply the schema\n\n⚠️ **Do not run `scripts/prepare-spanner.sh` directly on macOS.** Its `sed` invocation uses GNU `\\+` syntax that BSD `sed` doesn't accept. The script is designed to run inside the Linux container (where GNU sed lives). When you run it from a macOS host, the DDL parsing silently returns an empty array, the database gets created with no tables, and tests fail with cryptic *\"Table not found: collections\"* errors.\n\nUse this macOS-safe inline equivalent:\n\n```bash\nDDL_STATEMENTS=$(grep -v '^--' syncstorage-spanner/src/schema.ddl \\\n | tr '\\n' ' ' \\\n | tr -s ' ' \\\n | sed 's/;[[:space:]]*$//' \\\n | jq -R -s -c 'split(\";\") | map(select(length > 0)) | map(gsub(\"^\\\\s+|\\\\s+$\";\"\"))')\n\nPAYLOAD=$(jq -n --argjson stmts \"$DDL_STATEMENTS\" '{statements:$stmts}')\n\ncurl -fsS -X PATCH \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n -H 'Content-Type: application/json' \\\n -d \"$PAYLOAD\"\n```\n\nThen verify the tables landed:\n\n```bash\ncurl -fsS \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n | jq '.statements | length'\n# expected: 8 (5 tables + 3 indexes + 2 row-deletion policies, the last 2 are folded into CREATE TABLE statements by the emulator)\n```\n\n### 5. Run the unit tests\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nTwo non-obvious env vars:\n\n- `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` — the gRPC endpoint. **This is what tells the client to skip OAuth.** If you leave it unset, every test fails with *\"Error occurred when fetching oauth2 token\"* because the client tries to authenticate against the real Google API.\n- `RUST_TEST_THREADS=1` — required. Many tests share the same database and assert on counts/timestamps; parallel runs race.\n\nExpected output: `test result: ok. 39 passed; 0 failed; …`.\n\n### 6. Run a single test\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db \\\n tests::db::delete_collection_tombstone -- --nocapture\n```\n\n`--nocapture` is essential when tests fail — without it, errors get swallowed.\n\n### 7. Teardown\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml down -v\n```\n\nThe `-v` wipes named volumes for the whole compose project — including `tokenserver_db_data`. If you have the full e2e stack (or anything else from this compose file) running in parallel, prefer the helper script's narrower teardown instead:\n\n```bash\n./scripts/spanner-local-test-setup.sh down\n```\n\nThat only stops + removes the `sync-db` container. The emulator keeps no data in named volumes, so next run starts from a fresh database either way.\n\n## Common failures\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `Error occurred when fetching oauth2 token` | Forgot `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST`. Client is trying real Google OAuth. | Export the var. |\n| `Table not found: collections [at 2:25]` | DDL didn't apply. Most often because `prepare-spanner.sh` was run on macOS and produced an empty statement list. | Use the inline DDL block in step 4. |\n| `INVALID_ARGUMENT: …` on `INSERT OR UPDATE` | Emulator version too old (< 1.5.x). | Update `docker/docker-compose.spanner.yaml`'s image tag. |\n| 39 tests \"pass\" in 0.2s | Tests didn't actually connect — they're failing the OAuth check and treating it as a fresh start. Check the first test's error. | Same as the OAuth row above. |\n| Tests intermittently fail with timestamp/count mismatches | `RUST_TEST_THREADS=1` not set. | Set it. |\n| `cargo test` error: *the package 'syncstorage-db' does not contain this feature: py_verifier* | `py_verifier` is on the `syncserver` crate. | Drop `--features=py_verifier` when running tests on `syncstorage-db` alone. |\n\n## Docker e2e (optional, slower path)\n\nThe Makefile target `docker_run_spanner_e2e_tests` spins up the full stack and runs the Python integration suite twice (with and without JWK cache). It requires the `app:build` image to be **current** — built against the working tree's code.\n\n```bash\ndocker build -t app:build .\nmkdir -p workflow/test-results\nSYNCSTORAGE_RS_IMAGE=app:build make docker_run_spanner_e2e_tests\n```\n\nTwo local-run-only gotchas the make target assumes CI handles:\n\n- `SYNCSTORAGE_RS_IMAGE=app:build` must be exported. `docker-compose.spanner.yaml:52` references `${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}`; without the override compose tries to pull a non-existent `syncstorage-rs:latest` from Docker Hub. CI sets it in `.github/workflows/main-workflow.yml`. Local doesn't get it for free.\n- `workflow/test-results/` must exist before the make target runs. The target uses `docker cp ... ${INT_SPANNER_NO_JWK_JUNIT_XML}` which writes into that directory; if missing, the docker cp fails with `invalid output path` and aborts the run.\n\nTwo tests will always fail locally and can be safely ignored:\n\n- `tools/integration_tests/tokenserver/test_e2e.py::test_unauthorized_oauth_error_status`\n- `tools/integration_tests/tokenserver/test_e2e.py::test_valid_oauth_request`\n\nBoth hit **real FxA stage** (`api-accounts.stage.mozaws.net`, `oauth.stage.mozaws.net`) and need live network + production credentials the docker stack doesn't have. They surface as `fxa.errors.OutOfProtocolError: API responded with non-json content-type` — FxA returning an HTML error page because the request can't authenticate. CLAUDE.md excludes these from `make run_local_e2e_tests` via `--ignore=tokenserver/test_e2e.py`; the docker e2e target doesn't apply that exclusion, so expect `149 passed, 2 errors` not `151 passed`.\n\nNotes:\n\n- The image build compiles grpcio-sys from C++ source. On Docker Desktop for Mac, allocate **≥ 8 GB** memory.\n- The build takes 10-20 minutes from cold. Subsequent rebuilds reuse cache layers and are much faster *unless* `Cargo.lock` or the Rust toolchain changed.\n- Results land in `workspace/test-results/` as JUnit XML.\n- The make target tears the stack down on completion.\n\n### `ar: unable to copy file 'libgrpc.a'; reason: Success`\n\nMisleading message — the linker wasn't killed, the BuildKit cache mount got into a corrupt state. Seen on macOS with stale buildkit caches (mine had ~19 GB accumulated).\n\nFix:\n\n> ⚠️ `docker builder prune -f` clears BuildKit cache mounts for **every** Docker project on the host, not just this repo. Unrelated projects' next builds will go from cached to cold. Run it only when the link error reproduces and you've ruled out a real OOM.\n\n```bash\ndocker builder prune -f\ndocker build -t app:build .\n```\n\nThe prune drops the cache mounts grpcio-sys writes its 100+ MB `libgrpc.a` into. The next build re-creates them clean and the link step succeeds. If it fails a second time after prune, *then* suspect a real OOM and bump Docker Desktop memory.\n\nIf the local build won't cooperate, CI on push runs the same e2e suite — relying on it is reasonable for refactors that don't touch HTTP/auth/routing.\n\n## When to invoke this skill\n\nReach for it when:\n\n- You're editing anything under `syncstorage-spanner/` and want to validate before pushing.\n- A spanner test is failing in CI and you need to reproduce locally.\n- You're investigating an emulator-vs-prod behavior question.\n- You forgot the env-var dance for the third time this month.\n\nDon't invoke it for changes that don't touch Spanner — the MySQL or Postgres backends have their own test paths through the Makefile's `test_with_coverage` and `postgres_test_with_coverage` targets.\n\n## Output format\n\nAfter running, report:\n\n1. **Emulator state:** running / fresh / pre-existing\n2. **Schema applied:** N statements\n3. **Test results:** `X passed; Y failed; Z ignored` — list the failed test names if any\n4. **Total wall time**\n5. **Teardown:** done / left running for follow-up\n\nIf anything failed, surface the actual error message (not just the test name) — Spanner emulator errors are verbose but informative, and the first line is usually enough to diagnose.\n",".claude/skills/syncserver-explain-code/SKILL.md":"---\ndescription: Explains code for experienced engineers working on syncstorage-rs. Covers what changed, why it works, non-obvious decisions, gotchas, and data/control flow. Defaults to git diff vs main; accepts an optional file or path argument.\nargument-hint: file-or-path\n---\n\nYou are a senior engineer explaining code to another experienced engineer working on **syncstorage-rs** — Mozilla's Rust implementation of the Firefox Sync storage server and token server.\n\nSkip basics and language fundamentals. Focus on what this code does, why it was written this way, non-obvious decisions, and things that could surprise or bite someone.\n\n## Service boundary — critical context\n\nThis repo owns two services:\n- **Syncstorage** — the storage node that holds encrypted user sync data (bookmarks, passwords, tabs, etc.). Database backends: MySQL, PostgreSQL, Google Cloud Spanner.\n- **Tokenserver** — the allocation service that authenticates Firefox clients via FxA OAuth and assigns them to a storage node. Database backends: MySQL, PostgreSQL.\n\n**Firefox Accounts (FxA) is an external upstream service.** This repo does not own FxA. When explaining code:\n- References to `fxa_uid`, `fxa_kid`, `client_state`, FxA OAuth tokens, JWK keys, the FxA OAuth server URL — these are **inputs from FxA that this service validates and consumes**, not code this repo owns.\n- The `mock-fxa-server` in `scripts/` is a test stub only — it is not a real FxA implementation.\n- `tokenserver-auth/src/oauth/` and `tokenserver-auth/src/crypto.rs` are **this repo's FxA token verification logic**, not FxA's own code.\n\nMake this boundary explicit whenever the code touches the FxA/tokenserver interface.\n\n## How to gather the code\n\nIf `$ARGUMENTS` is provided, read that file or directory.\n\nOtherwise, run:\n\n```bash\ngit diff main...HEAD\n```\n\nand read the full diff. Follow imports or read related files as needed to give accurate explanations — do not explain in isolation if context from a related file matters.\n\n## Explanation structure\n\nWork through the code and produce an explanation covering all sections below. Omit a section only if it genuinely doesn't apply.\n\n### 1. One-paragraph summary\n\nPlain English. What does this code do, and what problem does it solve? Write for someone who hasn't read the ticket.\n\n### 2. Architecture & data flow\n\nWhere this fits in the broader system. Include an ASCII diagram if it clarifies the flow.\n\nUseful reference shapes for this codebase:\n\n```\nFirefox client\n → Tokenserver (validates FxA OAuth token, assigns node)\n → FxA OAuth server [external — validates token, not our code]\n → tokenserver-db (MySQL/Postgres — users, nodes, services tables)\n → Syncstorage node (stores/retrieves BSOs)\n → syncstorage-db (MySQL / Postgres / Spanner)\n```\n\n```\nRequest → Actix-web (syncserver/)\n → Extractors (hawk_identifier, fxa_uid, token validation)\n → Handler\n → db trait (syncstorage-db or tokenserver-db)\n → backend impl (mysql / postgres / spanner)\n```\n\n### 3. Annotated walkthrough\n\nStep through the key functions, types, or request paths. For each:\n- What it does\n- Why it's structured this way (if non-obvious)\n- How it connects to the next step\n\nFocus on the critical path. Don't exhaustively document trivial helpers.\n\nNote Rust-specific patterns where relevant: trait objects vs generics, `Arc`, `actix_web::web::Data<>`, `grpcio` vs `tonic`, `diesel` query builder patterns, `serde` field renaming.\n\n### 4. Gotchas & non-obvious bits\n\nThe most important section. Flag:\n\n- Implicit assumptions or preconditions the caller must satisfy\n- Surprising behavior or edge cases (off-by-one, async ordering, race conditions)\n- Why an obvious alternative wasn't taken (if inferrable)\n- Error handling that silently swallows failures or has unexpected fallback behavior\n- State mutated in non-obvious places\n- Performance characteristics worth knowing (N+1 queries, large allocations, blocking calls in async context)\n- Security-sensitive paths — especially: HAWK token validation, FxA OAuth scope checking, `client_state` / `keys_changed_at` handling, anything touching node assignment\n- gRPC-specific issues (Spanner backend): RST_STREAM handling, retry logic, emulator vs production behavioral differences\n- Migration safety: anything that changes DB schema or touches published migration files\n\n### 5. Dependencies & integrations\n\nExternal systems or crates this code depends on that aren't obvious from the code alone. Distinguish:\n- **Owned by this repo:** `syncstorage-*`, `tokenserver-*`, `syncserver-*` crates\n- **External upstream:** FxA OAuth server, Google Cloud Spanner, `tokenlib` (Python), `hawkauthlib`\n- **Test infrastructure only:** mock FxA server, Spanner emulator\n\nNote any version constraints or behavioral quirks.\n\n### 6. Testing notes\n\nHow is this code tested? Are there gaps? Note:\n- Unit tests (Rust `#[cfg(test)]` modules)\n- Integration tests (`tools/integration_tests/` — pytest, requires a running server)\n- E2E tests (`tokenserver/test_e2e.py` — hits real FxA staging, session-scoped, slow)\n- Any known flaky behavior (Spanner emulator quirks, timing-sensitive tests, 409/503 retry paths)\n\n## Style guidelines\n\n- Be direct and dense. Skip preamble.\n- Use code formatting for identifiers, file paths, and values.\n- Use ASCII diagrams when they save more words than they cost.\n- If something is genuinely straightforward, say so in one sentence and move on.\n- When the FxA boundary is crossed, say so explicitly rather than implying this repo owns that behavior.\n",".claude/skills/syncserver-jira-description/SKILL.md":"---\nname: syncserver-jira-description\ndescription: Drafts a concise Jira description for a syncstorage-rs task. Gathers context via targeted interview, researches relevant patterns in the repo, then outputs a clean description ready for an engineer to hand to Claude for implementation.\nuser-invocable: true\n---\n\n# Syncserver Jira Description\n\nDraft a Jira description for a syncstorage-rs task. Output the description only — do not create, edit, or suggest changes to any source files.\n\n## Step 1: Gather context\n\nIf a planning doc, epic description, or tech spec was provided, read it first and infer what, why, crates, and constraints before asking anything.\n\nRequired information:\n- **What:** What is being built or changed, in one sentence\n- **Why:** Motivation — user need, requirement, bug, or tech debt\n- **Crates:** Which specific crate(s) will be modified (e.g. `syncstorage-spanner`, `tokenserver-auth`, `syncserver`, `tokenserver-postgres`)\n- **Constraints:** DB migration required, multi-backend impact (MySQL/Postgres/Spanner), breaking API change, feature flag, FxA contract change — or none\n\nIf all four are clear from provided context, proceed directly to Step 2. Otherwise ask for only what is missing in a single message. Also invite related PRs, tickets, existing approach notes, or architecture notes that would add useful context.\n\n## Step 2: Research\n\nSearch only the crates identified in Step 1. Find the most relevant existing patterns: similar feature, nearby handler, equivalent DB query, related error type. Expand to the broader workspace only if nothing relevant is found there.\n\nIdentify:\n- Key files an implementer will need to touch\n- The closest existing reference implementation to follow\n- Which database backends are affected and whether the change must be replicated across MySQL, PostgreSQL, and/or Spanner\n- Whether new migrations are required (and which crates: `syncstorage-mysql`, `syncstorage-postgres`, `tokenserver-mysql`, `tokenserver-postgres`)\n- Whether tests, metrics (Glean/StatsD), or Sentry error handling apply\n\nIncorporate findings directly into the draft — do not list them separately or ask for confirmation. Surface genuine blockers as Open Questions.\n\n## Step 3: Output\n\n**Background:**\nWhy this is needed and what it enables. 2–4 sentences. Note which service is affected — Syncstorage (storage node), Tokenserver (node allocation + FxA auth), or both.\n\n**Acceptance Criteria:**\nObservable, testable outcomes. Each item verifiable without reading the code. Include criteria for tests, metrics emission, and Sentry error classification where applicable.\n\n**Implementation Steps:**\nNumbered steps with crate paths, file paths, trait/function names, and structural guidance. Reference the nearest existing pattern for each step. For multi-backend changes, call out which backends need matching implementations. No code snippets — file locations, types, and patterns only.\n\n**Database Migrations:** *(omit if no schema changes)*\nWhich migration crates need new files (`syncstorage-mysql/migrations/`, `tokenserver-postgres/migrations/`, etc.). Reminder: never edit published migrations — add new `up.sql`/`down.sql` pairs only.\n\n**Tests:**\nWhat needs to be tested. Unit (`#[cfg(test)]` in the relevant crate), integration (`tools/integration_tests/` pytest — requires running server), and E2E (`tokenserver/test_e2e.py` — hits FxA staging, session-scoped) coverage expectations. Reference the nearest existing test file as a pattern.\n\n**Metrics & Error Handling:** *(omit if not applicable)*\nAny Glean metrics, StatsD counters, or Sentry event classifications that should be emitted or updated. Note whether new error variants should be routed to Sentry or suppressed as metrics (see `is_sentry_event()` pattern in `syncstorage-spanner/src/error.rs`).\n\n**Key Reference Files:**\nSpecific files the implementer should read before starting. One line each.\n\n**Out of Scope:** *(omit if not needed)*\n\n**Open Questions:** *(omit if none)*\n\n## Guidelines\n\n- Output the description only — no source file changes\n- Implementation Steps should give enough detail to start work without follow-up questions — crate paths, file paths, and patterns, not prose\n- Multi-backend tasks: always explicitly name which backends (MySQL / PostgreSQL / Spanner) need matching changes; do not say \"all backends\" without listing them\n- Migration tasks: always reference the no-edit-published-migrations rule\n- Omit redundant or obvious acceptance criteria\n- Include Database Migrations, Metrics & Error Handling only when relevant\n- If motivation or scope remain unclear after asking, flag as an Open Question rather than assuming\n- Branch naming reminder for the assignee: Mozilla engineers use `type/description-STOR-####`; community contributors use `type/description-####`\n","CLAUDE.md":"# syncstorage-rs\n\nFirefox Sync storage and token server, written in Rust. The workspace contains a\nRust multi-crate server plus Python utilities and integration/end-to-end tests.\n\n## Agent rules\n\nSecurity takes absolute precedence. This repository handles synchronization of\nuser data and token management to access storage nodes.\n\n### 1) Scope & writes\n\n- Operate strictly within this repo root; normalize paths; do not follow\n symlinks outside the repo.\n- Writes are allowed to the working tree. Always present a diff for review\n before any staging or commit.\n- Do not modify files adjacent to a requested change. Mention issues found\n but do not fix them unless asked.\n- Ask before running any command (build, test, install, DB ops, git,\n service management). Do not run `git add`, `git commit`, `git push`, or\n `git rebase` unless explicitly instructed.\n\n### 2) Non-negotiables\n\n- **Secrets:** never read, print, summarize, or transmit secret files or\n values. Use placeholders (`YOUR_API_KEY_HERE`) in examples.\n- **External network:** only with explicit approval to a trusted, documented\n endpoint.\n- **Pipelines & contracts:** flag breaking API or contract changes. Do not\n alter GitHub Actions defined in `.github/`, any configuration for CI/CD workflows or git hooks without explicit, reviewed\n justification.\n- **Published DB migrations:** never edit existing published migration files or schema definitions (ex. `syncstorage-spanner/src/schema.ddl`), whether for MySQL, Postgres, or Spanner.\n Always add a new forward migration and a separate rollback migration.\n- **Workspace recommendations:** the repo deliberately ships no\n `.vscode/extensions.json`. Do not re-add one. Useful extensions are listed\n in `.vscode/README.md` for contributors to install at their own discretion.\n\n### 3) Do-not-touch paths\n\nNever read or write these paths. This list mirrors `.gitignore` — if there is\na discrepancy, `.gitignore` is the source of truth.\n\n```\nservice-account.json # GCP credentials\n.sentryclirc # Sentry auth token\n.envrc # direnv secrets\nconfig/local.toml # local server config (may contain secrets)\ntools/tokenserver/loadtests/*.pem\ntools/tokenserver/loadtests/*.pub\ntarget/ # Rust build artifacts\n*.xml # test result files\nopenapi.json # generated artifact\nbook/ # generated docs\ndocs/output/ # generated docs\nvenv/ # Python virtual environments\n.install.stamp # Poetry install marker\nworkspace/ # CI artifact directory\n```\n\n### 4) Git commit messages\n\nFollow the commit message format defined in [CONTRIBUTING.md](CONTRIBUTING.md) Git Commit Guidelines.\n\n- Format: `type(scope): subject` — `type` is one of `feat`, `fix`, `docs`,\n `style`, `refactor`, `perf`, `test`, `chore`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Body: explain motivation and contrast with previous behavior\n- Footer: `Closes STOR-1234` (Mozilla engineers — matches the Jira ticket) or\n `Closes #N` / `Issue #N` (community contributors — matches the GitHub issue number);\n `BREAKING CHANGE:` for breaking changes\n- Branch naming (Mozilla engineers): `type/description-STOR-1234` where `1234` must match the numeric suffix of the associated Jira ticket\n- Branch naming (community): `type/description-1112`\n- All commits must be GPG/SSH signed (enforced by CI)\n\nExample:\n\n```\nfeat: add node reassignment on capacity overflow\n\nNodes at 100% capacity previously returned 503 rather than\nassigning the user to an available node.\n\nCloses #42.\n```\n\n## Project layout\n\n```\nsyncserver/ Main Actix-web server (entry point)\nsyncstorage-db/ Syncstorage database abstraction layer\nsyncstorage-db-common/ Syncstorage shared database types and traits\nsyncstorage-mysql/ MySQL backend for syncstorage\nsyncstorage-postgres/ PostgreSQL backend for syncstorage\nsyncstorage-spanner/ Google Cloud Spanner backend for syncstorage\nsyncstorage-settings/ Syncstorage configuration types\ntokenserver-db/ Tokenserver database abstraction layer\ntokenserver-db-common/ Tokenserver shared database types\ntokenserver-mysql/ MySQL backend for tokenserver\ntokenserver-postgres/ PostgreSQL backend for tokenserver\ntokenserver-auth/ HAWK token generation and verification\ntokenserver-common/ Shared tokenserver utilities\ntokenserver-settings/ Tokenserver configuration types\nsyncserver-common/ Shared server utilities\nsyncserver-db-common/ Shared database utilities\nsyncserver-settings/ Syncserver configuration types\nglean/ Glean telemetry crate\ntools/\n integration_tests/ Pytest-based integration + e2e tests\n tokenserver/ Tokenserver utilities and load tests\n spanner/ Spanner utilities\n hawk/ HAWK token generation utility\n syncstorage-loadtest/ Molotov load tests\ndocker/ Docker Compose files for each backend\nconfig/ Local TOML configuration files\ndocs/ mdBook documentation source\n.github/ GitHub CI Actions and configuration\n```\n\n## Rust builds\n\nThe default feature set targets MySQL, however production uses the Spanner database with MySQL Tokenserver. Backend features are mutually exclusive.\n\n```bash\n# MySQL (default)\ncargo build\n\n# PostgreSQL\ncargo build --no-default-features --features=syncstorage-db/postgres --features=tokenserver-db/postgres --features=py_verifier\n\n# Spanner\ncargo build --no-default-features --features=syncstorage-db/spanner --features=py_verifier\n```\n\n## Rust tests\n\nTests require `RUST_TEST_THREADS=1` (many tests share a real database).\n\n```bash\n# MySQL unit tests (default DATABASE_URL envs set in Makefile)\nmake test\n\n# Specific package\nmake test ARGS=\"--package syncstorage-db\"\n\n# With coverage\nmake test_with_coverage # MySQL\nmake postgres_test_with_coverage\nmake spanner_test_with_coverage\n```\n\n## Clippy / formatting / audit\n\nFormatting checks are necessary before applying any code changes, as they\ncan capture problematic patterns or compiler issues in advance.\n\n```bash\ncargo fmt -- --check # format check\ncargo fmt # auto-format\n\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\n\nmake clippy_release_mysql # release-mode (catches dead code etc.)\nmake clippy_release_postgres\nmake clippy_release_spanner\n\ncargo audit # check dependencies for known CVEs (run by CI on every push)\n```\n\nAfter editing Rust code, run clippy and the unit tests for the affected backend before finishing the task; the `rust-code-smell` skill covers changed files across backends.\n\n## Python tooling\n\nAll Python tooling uses [Poetry](https://python-poetry.org/). The root\n`pyproject.toml` covers the `tools/` tree. Each subdirectory under `tools/` has\nits own `pyproject.toml` for its specific dependencies.\n\nAll of these Python checks must be run with any related change to a Python\nutility or test.\n\n```bash\nmake install # install root dependencies\nmake integration-test # install integration test dependencies\nmake tokenserver # install tokenserver utility dependencies\n```\n\nLint / format / type-check:\n\n```bash\nmake ruff-lint # ruff check tools/\nmake ruff-format # ruff format tools/\nmake ruff-fmt-chk # format diff check\nmake mypy # mypy type checking\nmake pydocstyle # docstring validation\nmake bandit # security linting\n```\n\n## Integration / E2E tests\n\n### Local (against a running server)\n\n```bash\n# Run all integration tests except real FxA e2e\nPYTHONPATH=$(pwd)/tools \\\nSYNC_MASTER_SECRET=secret0 \\\nSYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=http://localhost:6000 \\\nTOKENSERVER_HOST=http://localhost:8000 \\\npoetry -C tools/integration_tests run pytest . --ignore=tokenserver/test_e2e.py\n```\n\nOr via the Makefile shortcut (sets the env vars above):\n\n```bash\nmake run_local_e2e_tests\n```\n\n### Docker-based E2E (full stack)\n\n```bash\nmake docker_run_mysql_e2e_tests\nmake docker_run_postgres_e2e_tests\nmake docker_run_spanner_e2e_tests\n```\n\nEach command spins up two compose stacks (with and without JWK cache), copies\nJUnit XML results out of the container, then tears everything down.\n\n## Running the server locally\n\n```bash\n# MySQL\nmake run_mysql\n\n# Spanner (requires GCP service account key at ./service-account.json)\nmake run_spanner\n```\n\nServer listens on `http://localhost:8000`. Health check: `GET /__heartbeat__`.\n\n## Key environment variables\n\n| Variable | Purpose |\n|---|---|\n| `SYNC_MASTER_SECRET` | HAWK token signing secret |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | Syncstorage DB connection string |\n| `SYNC_TOKENSERVER__DATABASE_URL` | Tokenserver DB connection string |\n| `SYNC_TOKENSERVER__NODE_TYPE` | `mysql` / `postgres` / `spanner` |\n| `SYNC_TOKENSERVER__RUN_MIGRATIONS` | Auto-migrate on startup (`true`/`false`) |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | FxA OAuth endpoint |\n| `RUST_TEST_THREADS` | Set to `1` for all test runs |\n| `RUST_LOG` | Log level (`debug`, `info`, etc.) |\n| `SYNC_TEST_LOG_HTTP` | Set any value to log all HTTP req/resp in integration tests |\n| `SYNC_SERVER_URL` | Integration test base URL (default: `http://localhost:8000`) |\n| `TOKENSERVER_HOST` | Tokenserver host for integration tests |\n| `PYTHONPATH` | Must point to `$(pwd)/tools` when running integration tests |\n\n## Docker Compose files\n\n| File | Purpose |\n|---|---|\n| `docker/docker-compose.mysql.yaml` | MySQL syncstorage + MySQL tokenserver |\n| `docker/docker-compose.postgres.yaml` | PostgreSQL syncstorage + PostgreSQL tokenserver |\n| `docker/docker-compose.spanner.yaml` | Spanner emulator + MySQL tokenserver |\n| `docker/docker-compose.e2e.{mysql,postgres,spanner}.yaml` | E2E test runner overlay |\n| `docker/docker-compose.e2e.jwk-cache.yaml` | Enable JWK caching overlay |\n| `docker/docker-compose.e2e.no-jwk-cache.yaml` | Disable JWK caching overlay |\n\n## Documentation\n\n```bash\nmake doc-install-deps # install mdBook + mdBook-mermaid\nmake doc-watch # live preview at localhost\nmake doc-prev # build and serve\nmake doc-test # validate markdown\n\nmake api-prev # generate OpenAPI spec + serve Swagger UI on :8080\n```\n\n\n## Test structure\n\n- `tools/integration_tests/conftest.py` — pytest fixtures only (`st_ctx`)\n- `tools/integration_tests/helpers.py` — retry helpers, auth state, `switch_user`\n- `tools/integration_tests/test_storage.py` — storage protocol tests\n- `tools/integration_tests/tokenserver/conftest.py` — tokenserver fixtures only\n- `tools/integration_tests/tokenserver/helpers.py` — tokenserver DB/auth helpers\n- `tools/integration_tests/tokenserver/test_*.py` — tokenserver tests\n\n## MCP integrations\n\nTwo MCP servers are configured for this repository: **Atlassian Jira** (`mcp__atlassian__`) and **Sentry** (`mcp__sentry__`).\n\nUse the skills below to query them. Do not call MCP tools ad-hoc without reading the shared context first.\n\n| Skill | Invocation | Purpose |\n|---|---|---|\n| Jira status | `/jira-status [me\\|epic KEY\\|epics\\|sprint\\|summary ...\\|create ...\\|update STOR-#### ...]` | Task landscape, epics, personal view, summaries, create/edit tickets |\n| Sentry health | `/sentry-health [new\\|regressions\\|volume\\|full]` | Production error trends, regressions, new issues |\n\n**Shared context:** `.claude/mcp-context.md` contains project IDs, JQL patterns, Sentry project slug, what to flag, and known noise. Skills read this at runtime. Read it before any ad-hoc MCP query.\n\n**Identity when using these tools:** You are acting as both engineer and engineering manager. Surface patterns, not just raw lists. Distinguish \"needs action now\" from \"worth monitoring\". Do not fabricate data — if a query returns nothing, say so.\n\n**Jira project:** `STOR` — `https://mozilla-hub.atlassian.net`\n**Sentry project:** `syncstorage-prod` — org `mozilla`\n","docs/src/tools/load-testing.md":"# Load Testing\n\n## Quickstart: load test the change stream system (STOR-628)\n\nFour commands. Needs Docker, plus `curl` and `jq`. No GCP account, no\ncredentials, no VM.\n\n```console\nmake loadtest-offload-image # once: builds the server image\nmake loadtest-offload-changestream # run the load test\nmake loadtest-offload-report # did the pipeline keep up?\nmake loadtest-offload-down # tear it all down\n```\n\nThe report is the answer. Three numbers matter:\n\n| Look for | Meaning |\n| --- | --- |\n| `filtered` non-zero | The pipeline correctly dropped change records that have no payload file attached. This is the thing STOR-628 is asking about. |\n| `handler errors` zero | Nothing failed while finalizing or deleting objects. |\n| `noop_skips` zero | Nothing inert slipped past the filter. |\n\nIt prints `OK: pipeline drained clean` at the end, or a list of `FAIL` lines\nexplaining what went wrong, and exits non-zero on a real problem so it can gate\na script.\n\nOne thing to expect: a `NOTE` saying `batch_commit_skips` is 0. That is a known\nemulator limitation rather than your run going wrong. See\n[The emulator drops transaction tags](#the-emulator-drops-transaction-tags).\n\nFor a real run rather than a smoke test, pass molotov your own arguments:\n\n```console\nmake loadtest-offload-changestream \\\n MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 3600 -v\"\n```\n\nTo load test the handler read/write path instead (STOR-629), swap in\n`make loadtest-offload-handler`. Same report, same teardown.\n\nEverything below is detail you only need when a run looks wrong, or when you\nneed real latency numbers and have to go to GCP.\n\n---\n\n## Summary\n\nThis is the runbook for load testing syncstorage, with the offloaded-payload\npipeline as the main case. It replaces the Confluence page \"Syncstorage &\nTokenserver Runbooks\", whose molotov and locust instructions had become\ninterleaved and unreadable.\n\nThere are two load tests in the repo and they are unrelated:\n\n| | Target | Tool | Location |\n| --- | --- | --- | --- |\n| Syncstorage | storage read/write API | molotov | `tools/syncstorage-loadtest/` |\n| Tokenserver | `/1.0/sync/1.5` token issuance | locust | `tools/tokenserver/loadtests/` |\n\nEverything below is about the syncstorage/molotov one unless it says otherwise.\nFor tokenserver, see [Tokenserver load tests](#tokenserver-load-tests-locust).\n\nTwo tickets drive the offload work, and they want different traffic shapes:\n\n| Ticket | Goal | Traffic shape |\n| --- | --- | --- |\n| STOR-629 | Performance of the handler's GCS read/write path | 100% offloaded, every payload expanded |\n| STOR-628 | The change stream to reconciler pipeline, including the filtering of NULL `payload_link` records | Mixed offloaded and inline |\n\nThe distinction matters. STOR-628 is only a real test if some writes are\n*not* offloaded, because the record it is checking for is the one the\npublisher's filter is supposed to drop.\n\n---\n\n## Which rig to use\n\n```mermaid\nflowchart TB\n q1{\"What are you
measuring?\"}\n q1 -->|\"pipeline correctness:
does every object get
finalized, is the filter
dropping NULL records\"| local[\"Local docker rig
(emulators)\"]\n q1 -->|\"real latency and
throughput numbers
for a capacity decision\"| gcp[\"GCP dev environment\"]\n local --> both[\"Run local first either way:
it catches wiring bugs in
minutes with no GCP spend\"]\n```\n\nThe local rig runs the whole pipeline on emulators. It is the right default:\nit needs no GCP project, no credentials and no VM, and it answers every\ncorrectness question. What it cannot give you is a trustworthy latency\nnumber, because fake-gcs-server on a loopback interface is nothing like GCS\nover the network, and the Spanner emulator is single-node and in-process.\n\nUse GCP dev when the question is \"how fast\" rather than \"does it work\".\n\n---\n\n## Local docker rig\n\n### What it stands up\n\nThe rig is the existing reconciliation e2e stack plus a molotov container and\na metrics sink. Compose overlay: `docker/docker-compose.loadtest.yaml`.\n\n```mermaid\nflowchart LR\n lt[\"loadtest
molotov\"]\n\n subgraph fg[\"Foreground: the request path (STOR-629)\"]\n ss[\"syncserver
offload enabled\"]\n end\n\n gcs[(\"fake-gcs
test-payloads
filesystem backend\")]\n db[(\"Spanner emulator
bsos, batch_bsos
payload_link_changes\")]\n\n subgraph bg[\"Background: the cleanup arm (STOR-628)\"]\n pub[\"payload-link-publisher
stands in for Dataflow\"]\n ps[\"Pub/Sub emulator\"]\n rec[\"payload-reconciler
long-running mode\"]\n end\n\n drop([\"dropped: inert records
payload_link NULL both sides\"])\n sd[\"statsd-exporter
:9102\"]\n rep[\"loadtest-report.sh
the verdict\"]\n\n lt -->|\"Hawk, direct access
no tokenserver or FxA\"| ss\n ss -->|\"upload payload\"| gcs\n ss -->|\"store gs:// in payload_link\"| db\n gcs -->|\"download on read\"| ss\n db -->|\"change stream\"| pub\n pub -.->|filtered| drop\n pub -->|published| ps\n ps --> rec\n rec -->|\"finalize: committed=true
delete: orphans\"| gcs\n\n ss -.->|dogstatsd| sd\n pub -.->|dogstatsd| sd\n rec -.->|dogstatsd| sd\n sd -->|\"scrape counters\"| rep\n gcs -->|\"list objects,
check committed + customTime\"| rep\n```\n\nThe two halves map onto the two tickets. Everything left of the change stream\nis STOR-629's territory, and it costs the user latency. Everything right of it\nis STOR-628's, and no user waits on it, but a wrong decision there deletes a\nfile a live row still needs.\n\nNote where the report gets its evidence: counters from the statsd sink, and\nobject state read straight out of the bucket. It never trusts molotov's exit\nstatus, because the cleanup arm runs after the response is sent and can fail\nlong after the client saw a 200.\n\n| Service | Role |\n| --- | --- |\n| `sync-db` | Spanner emulator, carrying the `payload_link_changes` change stream from `schema.ddl` |\n| `pubsub-emulator` | Pub/Sub |\n| `fake-gcs` | GCS, on its **filesystem** backend for this rig |\n| `reconciliation-setup` | One-shot: creates the topic, subscription and bucket |\n| `payload-link-publisher` | Python change-stream reader, stands in for the Dataflow job |\n| `payload-reconciler` | Drains Pub/Sub, finalizes and deletes GCS objects |\n| `syncserver` | Offload enabled, limits raised |\n| `statsd` | `prom/statsd-exporter`, so the publisher and reconciler counters are readable |\n| `loadtest` | molotov |\n\nThe publisher is the Python one, not the Java Dataflow template. It publishes\nthe identical JSON wire format, so everything downstream of it is\nproduction code. If you need to reproduce something Java-specific, layer\n`docker-compose.e2e.reconciliation.java.yaml` on top; see\n[Reconciliation Pipeline](payload_link_reconciler.md).\n\nOn the NULL-filtering half of STOR-628, be precise about what the local rig\nproves. The filter exists twice: `isPayloadLinkActionable` in the Java flex\ntemplate, and `is_payload_link_actionable` in\n`payload-link-publisher-py/utils.py`, which is written to mirror it. A local\nrun exercises the **Python** one. That is a real test of the filter's logic\nand of everything downstream, but it is not a test of the Java code that runs\nin production. To exercise the shipping filter, either use the Java overlay\nlocally or run against GCP dev, where the Dataflow job is the publisher.\n\n### Prerequisites\n\n- Docker running, with roughly 8 GB available to the VM. Expanded payloads\n are held in memory in several places at once.\n- `curl` and `jq` on the host, for the report script.\n- Nothing else. No GCP credentials, no `service-account.json`.\n\n### The stale publisher image trap\n\nRead this before trusting any local result.\n\n`payload-link-publisher` is the only service in the stack built from a compose\n`build:` context rather than pulled or shared with `app:build`. Plain\n`docker compose up` reuses whatever `docker-payload-link-publisher` image is\nalready in your local cache and **will not rebuild it when its source\nchanges**. So a publisher image built before a change to\n`payload-link-publisher-py/` keeps running against new server code, silently.\n\nThis is not hypothetical. It is how the rig behaved on first use here: a\nfour-week-old publisher image had no `transaction_tag` support at all, so no\nrecord ever carried the `batch_commit` tag, so the reconciler treated every\nbatch-commit handoff as a genuine `batch_bsos` removal and deleted GCS objects\nthat committed `bsos` rows still pointed at. The symptom was HTTP 500s on\nreads, `payload_reconciler.batch_commit_skips` flat at zero, and\n`gcs_404{op=finalize}` running nearly 1:1 with `finalizes`. It looked exactly\nlike the STOR-657 production bug, and it was entirely a stale image.\n\nThe `loadtest-offload-*` targets therefore pass `--build` on every `up`. The\nfirst build is slow from a cold cache; afterwards it is cached and cheap.\n\nTwo things follow. If you drive compose by hand instead of through the\nMakefile, pass `--build` yourself. And note that\n`make docker_run_reconciliation_e2e_tests` does **not** pass it, so a local run\nof the reconciliation e2e suite can exercise a stale publisher. CI is unaffected\nbecause its cache starts empty.\n\nThe tell, if you suspect it:\n\n```console\ndocker compose ... exec payload-link-publisher grep -c transaction_tag /app/publisher.py\n```\n\nZero means the image predates transaction-tag support.\n\n### Run it\n\n```console\nmake loadtest-offload-image # build app:build (spanner backend), once\n\nmake loadtest-offload-handler # STOR-629\nmake loadtest-offload-report\n\nmake loadtest-offload-changestream # STOR-628\nmake loadtest-offload-report\n\nmake loadtest-offload-down # tear down, including the fake-gcs volume\n```\n\n`loadtest-offload-report` waits for the reconciler to stop making progress\nbefore reporting, so run it straight after the load test rather than trying\nto guess a sleep. It exits non-zero if it finds a problem, which makes it\nusable as a gate.\n\nBoth scenario targets take `MOLOTOV_ARGS`:\n\n```console\nmake loadtest-offload-handler MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 600 -v\"\n```\n\nStart small. Every worker can be holding a multi-MiB payload, on both the\nclient and the server, so worker count multiplies memory rather than just CPU.\nFive workers writing 10 MiB records is already a meaningful amount of traffic\nthrough this stack.\n\n### Reading the report\n\n```text\nRECONCILER (payload_reconciler.*)\n payload_reconciler_finalizes 1843\n payload_reconciler_orphan_deletes 291\n payload_reconciler_batch_commit_skips 412\n...\nGCS OBJECTS (test-payloads)\n objects 1552\n committed=true 1552\n committed!=true 0\n customTime pinned 1552\n...\nVERDICT\n OK: pipeline drained clean.\n```\n\nWhat each number should look like:\n\n| Metric | Healthy | If it is wrong |\n| --- | --- | --- |\n| `published` | Tracks actionable change records | Zero means the publisher is not reading the change stream at all |\n| `filtered` | **Non-zero on a mixed run.** This is the positive evidence for STOR-628: the publisher dropped records with `payload_link` NULL on both sides | Zero on a mixed run means either the filter is passing inert records through, or every write happened to be offloaded and the path went untested |\n| `finalizes` | Tracks offloaded write volume | Zero with objects in the bucket means the change stream, publisher or Pub/Sub leg is broken |\n| `orphan_deletes` | Non-zero, tracks overwrites and deletes | Zero means the load test never deleted anything; check `DISABLE_DELETES` |\n| `batch_commit_skips` | **Zero on the local rig** -- see the emulator limitation below. Non-zero is what you require on GCP dev | Zero on *GCP dev* means the transaction tag is not reaching the change stream, which is the STOR-657/668 bug class |\n| `noop_skips` | **Zero** | Non-zero is the STOR-628 regression: the filter let through a record with `payload_link` NULL on both sides |\n| `errors kind:handler` | **Zero** | Non-zero means messages went unacked. In prod five failures on one message routes it to the DLQ |\n| `committed!=true` | Zero once drained | A small tail is normal if you cut the run off mid-flight. A large or growing count means the pipeline is not keeping up with the write rate |\n\n`committed=true` count and `customTime pinned` count must agree. Finalize sets\nboth in a single `blob.patch()`, so a divergence means something other than\nthe reconciler is writing object metadata.\n\nOne thing that surprises people: even the \"100% offloaded\" STOR-629 run\nproduces a non-zero `filtered` count. `OFFLOAD_COLLECTIONS` only governs the\nbatch-write arm, while the load test also PUTs `meta/global` and POSTs\n`clients` on every scenario pass. Those two are never in the offload list, so\nthey generate exactly the inert NULL-on-both-sides records the filter is\nthere to drop. So STOR-629 covers the filter incidentally, and STOR-628 is\nstill the run that covers it deliberately, at volume, alongside offloaded\ntraffic in the same stream.\n\n### Reading the 404 counters, and why they are the whole signal\n\nThe reconciler treats `404 NotFound` as success on both of its operations.\nFrom `reconcile_payload_links.py`, a finalize whose target is gone logs at\n`debug`, bumps `gcs_404{op=finalize}`, and **acks the message**. Same shape for\ndelete. [Reconciliation Pipeline](payload_link_reconciler.md) states the\nrationale plainly: \"Both reconciler actions are idempotent, so a redelivered\nrecord is harmless.\"\n\nThat is correct and deliberate, and it has a consequence worth internalising\nbefore you read a load test result. Because a missing object is success, a run\nthat is silently destroying data produces:\n\n- `errors` `kind:handler` at **zero**\n- nothing in the DLQ\n- no `WARN` or `ERROR` in the reconciler log, only `debug`\n- an elevated `gcs_404` counter, and nothing else\n\nSo on the offload path `gcs_404{op=finalize}` is not a nice-to-have gauge. It\nis the *only* signal that separates a healthy pipeline from one that is\ndeleting live payloads. Everything else looks green.\n\nThat existing doc's failure-modes table gives two causes for a sustained\n`gcs_404{op=finalize}`:\n\n| Documented cause | Applies to a local run? |\n| --- | --- |\n| The lifecycle rule reclaimed the object before it was finalized | **No.** The 30 day `daysSinceCustomTime` rule is a bucket policy in `bucket.tf`. fake-gcs has no lifecycle policy at all, so nothing can reap an object locally. |\n| The same message was redelivered after a successful prior run (the at-least-once tax) | **No**, not at volume. Redelivery is a background trickle, not a rate approaching `finalizes`. |\n\nMeasured on these runs, neither explains what was observed:\n\n| Run | `finalizes` | `gcs_404{op=finalize}` | Ratio |\n| --- | --- | --- | --- |\n| STOR-629 handler | 930 | 716 | 77% |\n| STOR-628 changestream | 1853 | 1589 | 86% |\n\nA ratio near 1:1 is a third cause that the table does not list: **the\nreconciler deleted the object itself**, as an orphan, and the finalize for that\nsame object then 404s. Locally that is the emulator dropping the\n`batch_commit` transaction tag (see below), so every batch-commit handoff is\nmisread as a genuine `batch_bsos` removal.\n\nThe signature to recognise, wherever you see it:\n\n```text\nbatch_commit_skips 0 <- tag never arrived\ngcs_404{op=finalize} ~= finalizes <- finalizing objects already deleted\norphan_deletes high <- the deletes that did the damage\nerrors kind:handler 0 <- and nothing reports a problem\n```\n\n`gcs_404{op=delete}` is the benign one, and the existing doc's read of it holds\nwithout qualification: the object was already gone, the operation is idempotent\nby design, and a redelivery or a concurrent cleanup produces it. It carries no\ndata-loss implication on its own.\n\nTwo things follow for the alerting the reconciler doc already recommends. Its\nsuggestion to watch \"`gcs_404` `op:finalize` rising as a share of `finalizes`\"\nis the right alert, and this gives it a concrete threshold to reason about: a\nlow single-digit share is the redelivery tax, while anything approaching\n`finalizes` means objects are being deleted out from under committed rows. And\npairing it with `batch_commit_skips == 0` distinguishes the batch-commit cause\nfrom a lifecycle-window problem.\n\n### Knobs\n\nServer side, set on the `syncserver` container:\n\n| Variable | Default in the rig | Meaning |\n| --- | --- | --- |\n| `LOADTEST_OFFLOAD_COLLECTIONS` | `bookmarks,history` | Which collections syncserver offloads. **This is what decides mixed vs fully offloaded.** |\n| `LOADTEST_COLLECTION_LIMITS` | 10 MiB record / 20 MiB post / 21 MiB request, for `bookmarks` and `history` only | Per-collection raised limits, as a JSON map. Keys must be a subset of the offload list -- see below |\n| `LOADTEST_GCS_MAX_CONCURRENCY` | 4 | Concurrent GCS ops within one batch request. The main server-side throughput knob, worth sweeping |\n| `LOADTEST_DB_POOL_MAX_SIZE` | 40 | The stock 10 saturates first once workers climb |\n\n### Raise limits per collection, never globally\n\nThis is the one thing that will bite you, so it is worth stating plainly.\n\nSpanner caps a single `STRING(MAX)` value at **2621440 bytes (2.5 MiB)**. That\nhard cap is the reason this whole project exists. Raising the *global*\n`max_record_payload_bytes` above it does not fail at the API, it fails in the\ndatabase, on every collection that is not offloaded:\n\n```text\nFAILED_PRECONDITION New value exceeds the maximum size limit for\nthis column: bsos.payload, size: 10485760, limit: 2621440\n```\n\nAn offloaded collection is exempt, because its payload goes to GCS and the row\nstores only a `gs://` URL in `payload_link`, leaving `payload` NULL. So a limit\nabove 2.5 MiB is only ever correct for a collection that is in\n`gcs_payload_offload_collections`. That is exactly what the per-collection\noverride exists for, and how production raises it:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":10485760,\"max_post_bytes\":20971520,\"max_request_bytes\":22020096}}'\n```\n\nLeave the globals alone. Any collection not named in the map keeps stock\n2.5 MiB behaviour and writes inline safely. `max_post_bytes` and\n`max_request_bytes` are overridable per collection too; actix's payload config\nis sized from the largest `max_request_bytes` across all overrides\n(`ServerLimits::effective_max_request_bytes`), so a big body is not rejected at\nthe resource level before per-collection enforcement runs.\n\nThere is no per-collection `max_total_bytes`. The global default (clamped to\n`MAX_SPANNER_LOAD_SIZE`, 100 MB, by `Settings::normalize` on spanner) is\nalready generous, so the rig leaves it untouched.\n\nOn a mixed run this arrangement gives you exactly the right traffic for free:\noffloaded collections take expanded payloads, and inline collections get\n`LARGE_PAYLOAD_PROB` applied but capped at 2.5 MiB by\n`payload_target_length`, which is realistic \"large but not offloaded\" traffic.\n\nClient side, set on the `loadtest` container:\n\n| Variable | Meaning |\n| --- | --- |\n| `LARGE_PAYLOAD_PROB` | Fraction of BSOs given an expanded payload. `1.0` for STOR-629 |\n| `LARGE_PAYLOAD_SIZE` | Explicit target size in bytes. Unset means \"use the server's `max_record_payload_bytes`\" |\n| `OFFLOAD_COLLECTIONS` | Restricts batch writes to these collections. **Leave unset for mixed runs** |\n| `DISABLE_DELETES` | Set truthy to suppress the DELETE arm. Leave off, or the reconciler's `orphan_deletes` branch is never exercised |\n\nThe two collection lists are easy to confuse:\n\n- `LOADTEST_OFFLOAD_COLLECTIONS` (server) decides which collections get\n offloaded.\n- `OFFLOAD_COLLECTIONS` (client) decides which collections get *written to*.\n\nSetting the client one to a single collection sends every batch write there,\nwhich is what you want for STOR-629 and exactly what you must not do for\nSTOR-628. For a mixed run, leave the client variable unset so writes spread\nacross molotov's default five (`bookmarks`, `forms`, `passwords`, `history`,\n`prefs`) and set the server list to a subset of those.\n\n### Per-collection limit overrides\n\nProduction raises limits for one collection rather than globally, via a JSON\nmap. The load test resolves limits per target collection out of\n`/info/configuration`, preferring a collection's override, so this path is\ncovered too:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":20971520,\"max_post_bytes\":26214400,\"max_request_bytes\":26218496}}'\n```\n\n### Known limits of the local rig\n\n- **The emulator does not carry transaction tags, so the batch-commit skip\n cannot be validated locally.** This one is load bearing; details below.\n- **No DLQ.** `docker/reconciliation-setup.sh` creates the topic and\n subscription but no dead-letter topic, so dead-letter routing after five\n failed deliveries cannot be exercised locally. Verify that on GCP dev.\n- **No Dataflow.** The Python publisher replaces the Java job. Autoscaling,\n partition-split handling under real load and the connector's Spanner\n metadata writes are all Dataflow behaviours that only appear on GCP.\n- **Latency is not meaningful.** fake-gcs over loopback and a single-node\n Spanner emulator. Use it for correctness and relative comparisons only.\n- **No lifecycle policy.** The 30 day `daysSinceCustomTime` rule that reaps\n unfinalized objects is a bucket policy, so the local rig can only check\n that `customTime` was pinned, not that the reaping behaves.\n\n### The emulator drops transaction tags\n\nThe Spanner emulator emits the `transaction_tag` field on change stream\nDataChangeRecords but never populates it. Measured on emulator v1.5.52: the\nDataChangeRecord struct has arity 13, so index 11 (`transaction_tag`) exists\nand the publisher reads it correctly, but every record carries the empty\nstring, including the `batch_bsos` DELETE that *is* the batch-commit handoff.\n\nThat single gap has a large consequence. The reconciler decides whether to keep\nor delete a `batch_bsos` removal's object by checking\n`transactionTag == \"batch_commit\"` (see\n[Reconciliation Pipeline](payload_link_reconciler.md)). With the tag always\nempty, every batch-commit handoff looks like a genuine removal, so the\nreconciler deletes the GCS object that the just-committed `bsos` row now points\nat. The BSO becomes unreadable and syncserver returns a 500 backed by a GCS 404.\n\nSo on the local rig, expect all of this, and do not read it as a regression:\n\n- `payload_reconciler.batch_commit_skips` flat at zero.\n- `gcs_404{op=finalize}` running close to 1:1 with `finalizes`.\n- Objects missing for BSOs written in a batch create/append request.\n- A residual molotov failure rate, because roughly 50% of its POST scenarios\n are transactional batches. Reads of those BSOs 500.\n\n`docker/batch-commit-probe.py` reproduces it in isolation: one two-request\ntransactional batch, then a read of both BSOs, printing the bucket contents at\neach stage.\n\n```console\nmake loadtest-offload-up\nSYNCSTORAGE_RS_IMAGE=app:build docker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n -f docker/docker-compose.loadtest.yaml \\\n run --rm --entrypoint python3 \\\n -v \"$PWD/docker/batch-commit-probe.py:/probe.py:ro\" loadtest /probe.py\n```\n\nAdding `--no-deps` to that command keeps the reconciler stopped, and the probe\nthen passes with both payloads intact. That is the control which proves the\nreconciler is the thing deleting the object, rather than the batch commit\nlosing the `payload_link`.\n\nWhat this does **not** tell you is whether production is affected. The app side\nis correct: `handlers.rs` calls\n`with_transaction_tag(BATCH_COMMIT_TRANSACTION_TAG)` on the commit path, and\nreal Spanner does record transaction tags on change streams. The evidence\npoints at an emulator gap rather than a code defect, but the local rig cannot\nclose that question. **Confirm the batch-commit skip on GCP dev**, where the\nDataflow job reads a real change stream: run the probe there and require\n`payload_reconciler.batch_commit_skips` to be non-zero.\n\nTwo follow-ups worth filing rather than leaving implicit:\n\n1. There is no e2e test for the batch-commit handoff.\n `tools/integration_tests/test_payload_link_reconciliation.py` covers upload,\n update and delete, but not a transactional batch. That is presumably because\n it cannot pass against the emulator, which is exactly the point: the gap is\n invisible in CI.\n2. If the Java overlay behaves the same way, the emulator is confirmed as the\n cause. That is a cheap check and worth doing before spending GCP time.\n\n---\n\n## GCP dev environment\n\nUse this for real numbers. The dev pipeline is fully provisioned; see\n[GCP Infrastructure](payload-offload-infrastructure.md) for what lives where\nand which repo owns it. That page is the authority on the resources\nthemselves. The diagram below is only the load-testing view: where the\ngenerator sits, what it traverses, and where you go to observe each leg.\n\n```mermaid\nflowchart TB\n subgraph gen[\"Load generator (you create this)\"]\n vm[\"C4D VM, Ubuntu
molotov + load_test.pem
in moz-fx-sync-nonprod\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (shared GKE)\"]\n ts[\"tokenserver
trusts your public JWK\"]\n ss[\"syncserver
needs GCS_PAYLOAD_BUCKET +
OFFLOAD_COLLECTIONS + raised limits\"]\n rec[\"payload-reconciler
cronjob, ~5 min cadence\"]\n end\n\n subgraph v1[\"moz-fx-sync-nonprod-904c (GCPv1, cloudops-infra)\"]\n sp[(\"Spanner syncdb-dev
change stream:
payload_link_changes\")]\n end\n\n subgraph tenant[\"moz-fx-sync-nonprod (tenant, webservices-infra)\"]\n df[\"Dataflow flex template
the REAL Java filter\"]\n ps[\"Pub/Sub
payload-link-changes\"]\n dlq[\"Pub/Sub DLQ
after 5 failed deliveries\"]\n b[(\"GCS
sync-nonprod-dev-
syncstorage-payloads
30d lifecycle rule\")]\n end\n\n vm -->|\"1. self-signed JWT\"| ts\n ts -->|\"2. Hawk token\"| vm\n vm -->|\"3. storage reads + writes\"| ss\n ss -->|\"upload / download\"| b\n ss -->|\"payload_link\"| sp\n sp -->|\"change stream read\"| df\n df -->|publish| ps\n ps -->|\"pull subscription\"| rec\n ps -.-> dlq\n rec -->|\"finalize / delete\"| b\n```\n\nThree things this buys you that the local rig cannot:\n\n| Leg | Why it only exists here |\n| --- | --- |\n| Dataflow | The local rig substitutes a Python publisher. This is the actual Java `isPayloadLinkActionable` filter that ships, plus autoscaling and partition-split behaviour under real load. |\n| Real Spanner | Populates `transaction_tag`, so the batch-commit skip can finally be verified. Also the only place to measure what the change stream costs in Spanner storage, which has never been quantified. |\n| DLQ and lifecycle rule | Neither exists locally. `reconciliation-setup.sh` creates no dead-letter topic, and fake-gcs has no lifecycle policy. |\n\n### Load generator VM\n\nGenerate load from inside GCP, not from a laptop. A laptop's uplink is the\nbottleneck long before the server is, and multi-MiB payloads make that worse.\n\n1. Console -> project `moz-fx-sync-nonprod` -> Compute Engine -> VM instances\n -> Create instance.\n2. Machine type: `C4D` series. OS: Ubuntu, not the Debian default.\n3. The project already has IAP TCP forwarding and the firewall rules. If SSH\n is refused, check the project inherited them; see the SRE space's IAP page.\n\nSet the VM up:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\ngit clone https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\ncurl -sSL https://install.python-poetry.org | python3 -\npoetry install\n```\n\nDocker is an alternative to poetry on the VM, and avoids the Python version\ndance:\n\n```console\ndocker build . --tag syncstorage-loadtest:local\ndocker run --rm -e SERVER_URL -e OAUTH_PRIVATE_KEY_FILE \\\n -e LARGE_PAYLOAD_PROB -e OFFLOAD_COLLECTIONS \\\n -v /path/to/load_test.pem:/keys/load_test.pem:ro \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\"\n```\n\n### Authentication\n\nThree modes, in `storage/auth.py`. For a dev run use self-signed JWTs: no FxA\naccounts to create or clean up, and the token lifetime is yours to set.\n\n```console\npoetry run ./generate-keys.sh\n```\n\nThat writes `load_test.pem` (private, stays on the VM) and `jwk.json`\n(public). The dev tokenserver has to trust that public key, which means\nlanding the JWK in the tokenserver FxA JWK configuration in\n`webservices-infra` under `sync/k8s/`. That is a PR plus an ArgoCD sync, so do\nit before you book time for the run.\n\nThen:\n\n```console\nSERVER_URL=\"https://\" \\\n OAUTH_PRIVATE_KEY_FILE=./load_test.pem \\\n LARGE_PAYLOAD_PROB=1.0 \\\n OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\n```\n\nDirect-access mode (`SERVER_URL` with a `#secret` fragment) bypasses\ntokenserver entirely and is what the local rig uses. It needs the deployed\nmaster secret, so do not use it against a shared environment.\n\n### Server side prerequisites on dev\n\nThe pipeline being provisioned is not the same as offload being on. Offload\ndoes nothing until both of these are set on syncserver, and expanded payloads\ndo nothing until the limits are raised:\n\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET`\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS`\n- `SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES` and the\n `MAX_POST_BYTES` / `MAX_REQUEST_BYTES` / `MAX_TOTAL_BYTES` that go with it\n\nAll four are Helm values in `webservices-infra` under `sync/k8s/sync/`. Raise\nany front-end proxy body-size limit to match, or requests get a 413 before\nthey reach the server.\n\nConfirm what actually took effect by reading it back from the server rather\nthan from the values file:\n\n```console\ncurl -s https:///1.5//info/configuration | jq .\n```\n\nThe load test sizes its batches from exactly this response, so if the limits\nare not in there, the run is not testing expanded payloads no matter what the\nenvironment variables say.\n\n### Verifying a dev run\n\nEverything the local report script checks, plus the parts that only exist on\nGCP:\n\n| What | Where |\n| --- | --- |\n| Reconciler counters (`payload_reconciler.*`) | The metrics pipeline, same as any other service metric |\n| Oldest unacked message age on `payload-link-reconciler-sub` | Pub/Sub console. The most direct measure of finalize latency; should sit in minutes |\n| Anything at all in `payload-link-changes-dlq` | Pub/Sub console. A message only lands there after five failed deliveries, so it always needs a human |\n| Dataflow job lag, autoscaling, worker errors | Dataflow console |\n| Spanner CPU and change-stream storage cost | Spanner console on `moz-fx-sync-nonprod-904c` |\n| Object count and finalize state in the bucket | GCS console, or `gcloud storage ls -L` |\n\nChange-stream storage cost is worth capturing deliberately. It has not been\nquantified, and the plan for prod is to measure it before enabling offload\ntraffic, so a dev run is the cheapest place to get a number.\n\n---\n\n## What to record\n\nStandard long run is 60 to 65 minutes. Log results to the shared documents\nrather than only in a ticket:\n\n- Syncstorage molotov runs: \"Sync Load Test Run Document\"\n- Tokenserver locust runs: the tokenserver load test results document\n\nCapture at minimum: the git SHA under test, molotov arguments, the effective\n`/info/configuration`, the server-side offload collection list, molotov's\nown summary (request counts, failures, RPS), and the reconciler counters plus\nGCS finalize state from the report.\n\n---\n\n## Tokenserver load tests (locust)\n\nSeparate tool, separate target, unrelated to payload offload. Locust rather\nthan molotov, and it has a web UI.\n\n```console\ncd tools/tokenserver/loadtests\npoetry install\npoetry run ./generate-keys.sh\nOAUTH_PRIVATE_KEY_FILE=./load_test.pem poetry run locust # UI on :8089\n```\n\nRunning on a VM, tunnel the UI out over IAP:\n\n```console\ngcloud compute ssh --zone \"\" \"\" --project \"\" \\\n --tunnel-through-iap -- -L 8090:localhost:8089\n```\n\nThen open `localhost:8090`. See `tools/tokenserver/loadtests/README.md` for\nthe rest.\n\n---\n\n## Gaps worth knowing about\n\n- **GCS time can be separated from request time for the collection routes.**\n `storage.gcs.payload.batch` is the wall time one `get_collection` or\n `post_collection` request spent on GCS across all of its objects, tagged with\n `handler`, and the request middleware emits `request.duration` tagged by\n route and method. Dividing `rate(storage.gcs.payload.batch_sum{handler=\"post_collection\"})`\n by the `request.duration` `rate(_count)` for `/1.5/{uid}/storage/{collection}`\n with `method=\"POST\"` (and `get_collection` against `GET` likewise) gives the\n GCS share of an average collection write or read. It is still rough, because\n the denominator counts requests whose payloads stayed inline, under the\n offload threshold. The single-BSO routes (`/{bso}` GET and PUT) emit only\n the per-object `.upload` / `.download` timings, which carry no route tag, so\n they cannot be attributed the same way; measure those with a separate load\n shape. Either way a STOR-629 run can say \"GCS accounted for about N ms of a\n collection request\" rather than only \"the handler got slower\".\n- **No per-collection metric dimension**, so a mixed run cannot separate\n offloaded from inline request latency from the server side either. Working\n around it means either running the two shapes separately and comparing, or\n adding a collection tag.\n","docs/src/tools/make_hawk_token.md":"# Documentation for `make_hawk_token.py`\n\n## Summary\n\n`make_hawk_token.py` is a utility script for generating a [Hawk](https://github.com/mozilla/hawk) authentication token and header for use when testing the syncstorage-rs API. It derives a token and signing key from a shared secret and produces either a raw token tuple or a ready-to-use `Authorization: Hawk ...` header.\n\nThe script lives in [`tools/hawk/`](https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/hawk).\n\n---\n\n## Setup\n\n### Prerequisites\n\n- Python ≥ 3.12\n- [Poetry](https://python-poetry.org/docs/#installation) installed\n\n### Install dependencies\n\n```shell\ncd tools/hawk\npoetry install\n```\n\n---\n\n## Usage\n\n```shell\npoetry run python make_hawk_token.py [options]\n```\n\nBy default (no arguments), a token is generated for `http://localhost:8000/1.5/1/storage/col2/` using a built-in test secret.\n\n### Options\n\n| Option | Description | Default |\n|---|---|---|\n| `--uri` | Request URI path | `/1.5/1/storage/col2/` |\n| `--secret` | Shared `SYNC_MASTER_SECRET` value | *(built-in test value)* |\n| `--method` | HTTP method for the request | `GET` |\n| `--as_header` | Print a ready-to-use `Authorization: Hawk ...` header | *(off)* |\n\nUse `-h` for the full option list.\n\n---\n\n## Examples\n\n### Generate a token for a GET request\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET\n```\n\n### Generate a ready-to-use Authorization header\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nOutput:\n```\nAuthorization: Hawk id=\"...\", ts=\"...\", nonce=\"...\", mac=\"...\"\n```\n\n### Non-GET methods\n\nFor methods other than GET (e.g. PUT, POST, DELETE), pass `--method`:\n\n```shell\npoetry run python make_hawk_token.py \\\n --method PUT \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nSee [`tools/examples/put.bash`](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for a full curl example using this output.\n\n---\n\n## Notes\n\n- The `--secret` value must match the `SYNC_MASTER_SECRET` configured in the running syncstorage-rs instance.\n- The generated token is scoped to the exact URI and method — a token generated for `GET /foo` is not valid for `PUT /foo`.\n","docs/src/tools/payload-offload-infrastructure.md":"# Payload Offload Infrastructure\n\n## Summary\n\nThis document is a map of the GCP infrastructure behind offloaded BSO\npayloads: which project each resource lives in, which repo defines it,\nand what applies it. It covers where things run rather than how the\npipeline behaves. For the behaviour of the change stream, Dataflow job,\nPub/Sub topics and reconciler, see\n[Payload Link Reconciler](payload_link_reconciler.md).\n\nTracking epic: STOR-372, \"Expand spanner storage beyond 2.5MB\".\n\n## The two-project split\n\nThe single most confusing thing about this setup is that Spanner and\neverything else live in different GCP projects, under different\ngenerations of Mozilla's cloud tooling.\n\n| | Spanner | Everything else |\n| --- | --- | --- |\n| Project (dev) | `moz-fx-sync-nonprod-904c` | `moz-fx-sync-nonprod` |\n| Generation | GCPv1 (legacy) | GCPv2 (MozCloud tenant) |\n| Managed from | `cloudops-infra` | `webservices-infra` + `global-platform-admin` |\n\nThe Spanner instances stayed in GCPv1 on purpose. Migrating the large\nprod instance is not cheap, so they were left in place while the rest of\nthe service moved to MozCloud. The practical consequences:\n\n1. Terraform in `webservices-infra` cannot configure the Spanner\n instances or databases. Those changes need `cloudops-infra`, which\n means an extra PR and a MozCloud engineer to apply it.\n2. The webservices Terraform admin service account has no\n `setIamPolicy` rights on `-904c`, so IAM grants that target the\n Spanner database have to be applied out of band. See\n [Out-of-band steps](#out-of-band-steps) below.\n\nThe Dataflow job bridges the two projects: it runs in the tenant project\nand reads a change stream in the legacy project.\n\n## Diagram\n\n```mermaid\nflowchart TB\n gha[\"GitHub Actions\n syncstorage-rs\"]\n\n subgraph v1[\"GCPv1: moz-fx-sync-nonprod-904c (cloudops-infra)\"]\n syncdb[\"Spanner syncdb\n instance: sync\n database: syncdb-dev\n change stream: payload_link_changes\n access role: payload_link_reader\"]\n metadb[\"Spanner Dataflow metadata\n instance: sync\n database: syncdb-pldf-meta-dev\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (global-platform-admin)\"]\n gke[\"Shared GKE cluster\n syncserver, tokenserver,\n payload-reconciler cronjob\"]\n end\n\n subgraph tenant[\"GCPv2 tenant: moz-fx-sync-nonprod (webservices-infra)\"]\n df[\"Dataflow flex template job\n sync-nonprod-dev-payload-link-dataflow\n dedicated VPC and firewall\"]\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n payloads[(\"GCS\n sync-nonprod-dev-syncstorage-payloads\")]\n jobbucket[(\"GCS\n sync-nonprod-dev-payload-link-dataflow\n template spec, staging, tmp\")]\n end\n\n gha -.->|\"publish template spec, via WIF\"| jobbucket\n jobbucket -.->|template spec| df\n syncdb -->|change stream read| df\n df -->|partition state| metadb\n df -->|publish| topic\n topic -->|pull subscription| gke\n topic -.->|\"after 5 failed deliveries\"| dlq\n gke -->|finalize and delete objects| payloads\n gke -->|upload and download payloads| payloads\n gke -->|read and write BSO rows| syncdb\n\n```\n\nNote that the workloads do not run in the tenant project. Sync is\n`risk_level: high`, so its pods run on the shared high risk GKE cluster\nin `moz-fx-webservices-high-nonpro` (nonprod) and\n`moz-fx-webservices-high-prod` (prod). The tenant project holds the\ndata plane resources: buckets, Pub/Sub, secrets, service accounts, DNS,\nand the Dataflow job. Pods reach tenant resources through workload\nidentity on the tenant GKE service account.\n\n## Where each piece is defined\n\n| Resource | Defined in | Repo |\n| --- | --- | --- |\n| Tenant definition (envs, charts, risk rating, project ids) | `tenants/sync.yaml` | global-platform-admin |\n| Tenant GCP projects, enabled APIs, folders, DNS, GAR | `projects/tf/webservices/locals.tf` | global-platform-admin |\n| Shared GKE, GCLB, logging, monitoring | `webservices-high/tf/{nonprod,prod}` | global-platform-admin |\n| GCS payload bucket and lifecycle rule | `sync/tf/dev/bucket.tf` | webservices-infra |\n| Dataflow job, network, firewall, job bucket | `sync/tf/dev/dataflow.tf` | webservices-infra |\n| Dataflow service account and IAM | `sync/tf/dev/dataflow_iam.tf` | webservices-infra |\n| Pub/Sub topics, subscriptions, DLQ routing | `sync/tf/dev/pubsub.tf` | webservices-infra |\n| Spanner instance, database, DDL, database IAM | `projects/sync` | cloudops-infra |\n| Change stream and access role DDL source of truth | `syncstorage-spanner/src/schema.ddl` | syncstorage-rs |\n| Dataflow flex template image and metadata | `tools/payload-link-dataflow/` | syncstorage-rs |\n| Image build and template spec publish | `.github/workflows/mozcloud-publish.yaml` | syncstorage-rs |\n| Helm charts for sync, tokenserver, sync-jobs, sync-test | `sync/k8s/` | webservices-infra |\n\n## Repos and how each is applied\n\n| Repo | Scope | Automation |\n| --- | --- | --- |\n| [global-platform-admin](https://github.com/mozilla/global-platform-admin) | Cross tenant platform: projects, APIs, shared clusters, tenant definitions | Spacelift. The `admin-webservices-projects` stack tracks `main` with autodeploy, so a merge plans and applies without a manual step. |\n| [webservices-infra](https://github.com/mozilla/webservices-infra) | Per tenant resources for the webservices function | Atlantis, autodiscovered per directory. Plan runs on the PR, then `atlantis apply` in a comment. |\n| [cloudops-infra](https://github.com/mozilla-services/cloudops-infra) | Legacy GCPv1 resources, including sync Spanner | Atlantis, applied by a MozCloud engineer |\n| [syncstorage-rs](https://github.com/mozilla-services/syncstorage-rs) | Application code, Spanner DDL, Dataflow template, Helm chart values | GitHub Actions builds and publishes images; ArgoCD deploys |\n\nTerraform state for both global-platform-admin's `projects/tf/webservices`\nand webservices-infra's `sync/tf/*` lives in the same bucket,\n`moz-fx-webservices-terraform-state-global`, accessed by impersonating\n`tf-webservices@moz-fx-websvc-terraform-admin.iam.gserviceaccount.com`.\nThe per environment state is at prefix `projects/sync/`. The project\nlevel state is at `projects/projects/global`, and\n`sync/tf/dev/data.tf` reads it as a remote state data source to resolve\nproject ids and numbers. That prefix is load bearing, do not rename it\nwithout migrating the state object.\n\n## Concrete dev resources\n\nNames are derived from `${application}-${realm}-${environment}` in\n`sync/tf/dev/locals.tf`.\n\n| Thing | Value |\n| --- | --- |\n| Tenant project | `moz-fx-sync-nonprod`, number `960020799362` |\n| Spanner project | `moz-fx-sync-nonprod-904c` |\n| Spanner instance and database | `sync` / `syncdb-dev` |\n| Dataflow metadata database | `sync` / `syncdb-pldf-meta-dev` |\n| Dataflow metadata table | `Metadata_payload_link`, pinned via `spannerMetadataTableName` |\n| Change stream | `payload_link_changes` |\n| Spanner database role | `payload_link_reader` |\n| Payload bucket | `sync-nonprod-dev-syncstorage-payloads`, us-west1 |\n| Dataflow job bucket | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow job | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow service account | `sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Template publisher service account | `sync-nonprod-dev-tmpl-pub@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Dataflow VPC | `sync-nonprod-dev-dataflow`, internal ingress on tcp 12345-12346 |\n| Pub/Sub topic | `payload-link-changes` |\n| Pub/Sub DLQ | `payload-link-changes-dlq`, 5 delivery attempts |\n| Reconciler subscription | `payload-link-reconciler-sub`, 60s ack, 7d retention |\n| DLQ inspection subscription | `payload-link-changes-dlq-sub`, 7d retention |\n| Flex template image | `us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow` |\n| Template spec | `gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json` |\n\nOnly dev is built out today.\n\n## Standing up stage or prod\n\nEverything below is per environment. Names derive from\n`${application}-${realm}-${environment}`, so copying the dev Terraform\ninto the target environment directory produces the right names without\nediting them.\n\n1. **Terraform.** Copy `bucket.tf`, `pubsub.tf`, `dataflow.tf` and\n `dataflow_iam.tf` from `sync/tf/dev/` into the target environment\n directory in webservices-infra.\n\n2. **A dedicated Spanner metadata database.** The change stream\n connector keeps its partition state in a Spanner database of its own.\n Each environment should get a separate database, for example\n `syncdb-pldf-meta-stage`, so the connector holds no write access\n to the syncstorage database. This is a cloudops-infra change.\n\n Pin the table inside it as well, with\n `spannerMetadataTableName = \"Metadata_payload_link\"` on the job.\n Dev uses that name; reusing it per environment is fine, since the\n databases are already separate. The parameter is optional and the\n consequence of skipping it is silent, so it is easy to miss: see\n step 6.\n\n3. **DDL.** Apply the change stream and the `payload_link_reader` role to\n the target database. See [Out-of-band steps](#out-of-band-steps).\n\n4. **Cross-project IAM.** A `-904c` admin grants the new environment's\n Dataflow service account `roles/spanner.databaseUser` on the\n syncstorage database and on the metadata database.\n\n5. **Template spec.** Add a `write-payload-link-dataflow-spec-` job\n to `.github/workflows/mozcloud-publish.yaml`, and make sure the\n environment's `-tmpl-pub` service account exists first. The spec has\n to be in the bucket before the Terraform job resource is applied.\n\n6. **Job lifecycle.** Keep `on_delete = \"cancel\"`. This is not a\n dev-only shortcut: the SpannerIO change stream connector does not\n support draining at all, so `\"drain\"` is never the right setting for\n this pipeline. See\n [Draining a change streams pipeline](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining).\n The change stream's 7 day retention is what covers the gap while a\n replacement job comes up.\n\n Cancel is only safe to rely on because the metadata table name is\n pinned. The connector generates that name from a random UUID at\n graph-construction time, so an unpinned job that is cancelled and\n relaunched, or updated in place, comes up against an empty table and\n resumes from `Timestamp.now()` instead of the previous watermarks.\n Nothing errors; the records committed in between are simply never\n published, and the reconciler never learns those objects need\n finalizing or deleting. Set the name once and do not change it.\n\n7. **Reconciler cronjob.** Enable `payloadReconciler` in the\n environment's values file in `sync/k8s/sync/`. The chart requires\n `runBudgetSeconds` to be less than `activeDeadlineSeconds`.\n\n8. **Measure before enabling on prod.** The change stream costs Spanner\n storage that has not been quantified. Turning the stream on ahead of\n any offload traffic is the cheap way to find out.\n\nEnabled APIs need no change. `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so `moz-fx-sync-prod` already has them.\n\nNote that provisioning the pipeline is safe on its own. Offload does\nnothing until `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET` and\n`SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` are both set on\nsyncserver, so an environment can carry the whole pipeline with zero\ntraffic through it while it is verified.\n\n## Enabled GCP APIs\n\nThe tenant project's enabled APIs are declared in\n`projects/tf/webservices/locals.tf` under the `sync` entry. Enabling an\nAPI there is a PR to global-platform-admin, since tenant teams do not\nhold the IAM to enable services directly.\n\nCurrent list for the sync tenant:\n\n- `spanner.googleapis.com`\n- `secretmanager.googleapis.com`\n- `dataflow.googleapis.com`\n- `datapipelines.googleapis.com`\n- `cloudscheduler.googleapis.com`\n\nThe last three were added in\n[global-platform-admin PR\n#7031](https://github.com/mozilla/global-platform-admin/pull/7031) for\nthis work, tracked by STOR-656 and SREIN-1633.\n`compute.googleapis.com` is enabled by default for every tenant project\nby the shared `google_project` module, which is what Dataflow workers\nneed.\n\nTwo caveats on that list. First, `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so those three APIs are enabled in `moz-fx-sync-prod`\nas well as `moz-fx-sync-nonprod`. There is no per realm override.\nSecond, the current implementation launches\n`google_dataflow_flex_template_job` directly, so nothing in\n`webservices-infra/sync` references Data Pipelines or Cloud Scheduler\nyet. Those two were requested up front in case the job moves to a\nscheduled Data Pipelines definition. They are enabled but unused.\n\n### Security assessment\n\nSync as a whole stays at `risk_level: high` with\n`risk_rating: {confidentiality: 2, integrity: 1, availability: 4}` in\n`tenants/sync.yaml`. The offload itself is low risk.\n\nBSO payloads are encrypted client side with the user's Sync key before\nthey reach syncserver, so GCS holds ciphertext the server cannot\ndecrypt. The offload moves the same class of opaque blob that Spanner\nalready holds, rather than creating a new plaintext exposure. Object\nkeys and metadata do carry identifiers such as fxa_uid, collection and\nbso_id, which is metadata rather than content.\n\nEnabling a service grants no access on its own and creates no\nidentities, so IAM still gates every call. The one item worth care is\nDataflow: a principal with `roles/dataflow.developer` plus `actAs` on\nthe worker service account can run arbitrary pipeline code on GCE VMs\ninside the tenant VPC with that account's permissions. That is why the\njob runs as a dedicated per environment service account holding only\n`roles/dataflow.worker`, `roles/pubsub.publisher` on one topic,\n`roles/storage.objectAdmin` on the job bucket, and\n`roles/spanner.databaseUser` on one database, rather than the Compute\nEngine default account which carries `roles/editor`.\n\nSpanner access is narrowed a second time inside the database. The job\nruns under the `payload_link_reader` database role, which is granted\n`SELECT` on the one change stream and `EXECUTE` on its read function and\nnothing else, so the IAM grant alone does not let the job read BSO rows.\n\n## Out-of-band steps\n\nThree things are not managed by Terraform and have to be done by hand in\nthe right order. Getting this wrong is the usual reason a fresh\nenvironment fails to come up.\n\n1. Apply the change stream DDL to the Spanner database. The DDL in\n `syncstorage-spanner/src/schema.ddl` is not auto-applied. Run\n `gcloud spanner databases ddl update` against the target database\n after merging.\n\n That same DDL block also creates the fine-grained access role the\n Dataflow job reads through:\n\n ```sql\n CREATE ROLE payload_link_reader;\n GRANT SELECT ON CHANGE STREAM payload_link_changes TO ROLE payload_link_reader;\n GRANT EXECUTE ON TABLE FUNCTION READ_payload_link_changes TO ROLE payload_link_reader;\n ```\n\n The job is launched with `spannerDatabaseRole = \"payload_link_reader\"`\n (see `sync/tf/dev/dataflow.tf`), so if the role does not exist the job\n fails at startup rather than falling back to broader access. Apply the\n role and its grants in the same DDL update as the change stream.\n\n2. Grant the Dataflow service account `roles/spanner.databaseUser` on\n the Spanner database. This targets `-904c`, where the webservices\n Terraform admin account has no `setIamPolicy` rights, so a `-904c`\n admin runs it. Tracked by STOR-655.\n\n ```console\n gcloud spanner databases add-iam-policy-binding syncdb-dev \\\n --instance=sync --project=moz-fx-sync-nonprod-904c \\\n --role=roles/spanner.databaseUser \\\n --member=serviceAccount:sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com\n ```\n\n3. Publish the flex template spec into the job bucket before the\n Terraform job resource is applied, or the job has nothing to launch.\n The `write-payload-link-dataflow-spec-dev` job in\n `.github/workflows/mozcloud-publish.yaml` does this on publish.\n\n CI does not use the pipeline service account for this. A separate\n `-tmpl-pub` account exists purely to upload the spec, and the\n syncstorage-rs GitHub Actions workflow impersonates it through\n workload identity federation. Its bucket grant carries an IAM\n condition restricting writes to the `templates/` prefix, so CI cannot\n touch the runtime `staging/` or `tmp/` paths, and it holds nothing on\n Spanner or Pub/Sub. Both accounts are defined in\n `sync/tf/dev/dataflow_iam.tf`.\n\n To publish manually:\n\n ```console\n gcloud dataflow flex-template build \\\n gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json \\\n --image us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n ```\n\n## Consoles and dashboards\n\nSubstitute the project for other environments.\n\n- [Enabled APIs](https://console.cloud.google.com/apis/dashboard?project=moz-fx-sync-nonprod)\n- [Dataflow jobs](https://console.cloud.google.com/dataflow/jobs?project=moz-fx-sync-nonprod)\n- [Pub/Sub topics](https://console.cloud.google.com/cloudpubsub/topic/list?project=moz-fx-sync-nonprod)\n- [GCS buckets](https://console.cloud.google.com/storage/browser?project=moz-fx-sync-nonprod)\n- [Service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts?project=moz-fx-sync-nonprod)\n- [Spanner databases](https://console.cloud.google.com/spanner/instances/sync/databases?project=moz-fx-sync-nonprod-904c)\n\nThe tenant project Terraform is applied by the `admin-webservices-projects`\nstack in Spacelift.\n\n## Reference\n\nMozilla internal documents, not linked here:\n\n- Design document: \"Sync Spanner 10MB Payloads\"\n- \"Syncstorage Payload Expansion Project Plan\"\n- Confluence: \"Bootstrapping a tenant\", under the SRE space\n- Confluence: \"MozCloud Customer Documentation\", under the SRE space\n","docs/src/tools/payload-offload-load-testing.md":"# Load Testing Payload Offload\n\nThis provides a guide on running load tests (`tools/syncstorage-loadtest/`) on the\nGCS payload offload path with Molotov in GCP. This focuses on using gcloud cli\ncommands and short scripts as opposed to the UI.\n\nTwo methods:\n\n- **Method A, against dev env.** Real Spanner, real GCS, real change stream and\n reconciler. Payloads capped at whatever dev's limits allow. Use this to test\n the whole pipeline end to end. Note: it requires webservices-infra updates\n should you need to change application limits.\n- **Method B, local syncserver on the VM.** Spanner emulator, real GCS, your own\n limits. No change stream or reconciler. Use this to test expanded payloads\n larger than the server's default ceiling. This allows for easier ad-hoc\n changing of application limits.\n\nBoth use the load tester's **direct access** mode, which creates its own Hawk\ntoken from the master secret and bypasses Tokenserver.\n\n---\n\n## Prerequisites\n\n### Check your gcloud configuration first\n\nIf you have a configuration for the Spanner emulator and it is enabled,\nit has the bad habit of breaking everything in confusing ways. Check:\n\n```console\ngcloud config list\n```\n\nA configuration containing `auth/disable_credentials = true` or\n`api_endpoint_overrides/spanner` sends every request out unauthenticated and\nredirects Spanner calls to localhost. Switch to a normal configuration:\n\n```console\ngcloud config configurations activate default\ngcloud auth login\n```\n\nSwitch back with `gcloud config configurations activate ` when\nyou return to local emulator work.\n\n### Access you need\n\n| For | You need |\n| --- | --- |\n| Creating the VM | `roles/compute.instanceAdmin.v1` on `moz-fx-sync-nonprod`, which `sync/developers` holds. There is no equivalent grant on prod. |\n| Method A | `roles/secretmanager.secretAccessor` on the tenant project, to read the master secret. |\n| Method B | Write access to a GCS bucket you control. |\n\nSync is onboarded to PAM, so anything beyond the standing grants above can be\nrequested just-in-time. List what is available with `gcloud pam entitlements\nsearch`.\n\n---\n\n## Part 1: Create the load generator VM\n\nUse your own project under the org's `developers` folder (get the folder_no in GCP)\nrather than a tenant project.\n\n```console\nPROJ=\n\ngcloud projects create $PROJ --folder=\ngcloud billing accounts list\ngcloud billing projects link $PROJ --billing-account=\ngcloud services enable compute.googleapis.com --project=$PROJ\n```\n\nNew projects have no default VPC, because the org enforces\n`compute.skipDefaultNetworkCreation`. Create one:\n\n```console\ngcloud compute networks create default --subnet-mode=auto --project=$PROJ\n```\n\nFirewall for IAP SSH required. Without this, the SSH in the next step fails\nwhen no firewall rules are set. Org rules forbid opening tcp/22 to `0.0.0.0/0`:\n\n```console\ngcloud compute firewall-rules create allow-iap-ssh \\\n --project=$PROJ --network=default \\\n --direction=INGRESS --action=allow --rules=tcp:22 \\\n --source-ranges=35.235.240.0/20\n```\n\nCreate the instance in **us-west1**, matching where dev runs.\nAlways check region and match to limit latency.\n\n```console\ngcloud compute instances create sync-loadtest \\\n --project=$PROJ \\\n --zone=us-west1-b \\\n --machine-type=n2-standard-8 \\\n --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \\\n --boot-disk-size=200GB\n```\n\n> **Note on Machine family.** New projects get zero quota for newer families, so C4D\n> fails with `Quota 'CPUS_PER_VM_FAMILY' exceeded. Limit: 0.0`. Most families have\n> their own regional quota (`N2_CPUS`, `E2_CPUS`, `C2_CPUS`); only N1 draws on the\n> general `CPUS` pool. Check whichever matches the family you pick, with\n> `gcloud compute regions describe us-west1 --project=$PROJ --flatten=\"quotas[]\"\n> --format=\"table(quotas.metric,quotas.limit,quotas.usage)\"`.\n\nTo Connect via terminal to instance (you'll likely use this command a lot):\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap\n```\n\nInstall sync & load tests on the VM:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\nmkdir -p ~/src && cd ~/src\ngit clone --depth=1 https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\npython3 -m venv venv && source venv/bin/activate\npip install poetry && poetry install\n```\n\n---\n\n## Part 2: Method A - load test against dev\n\n### Fetch the master secret programmatically\n\nDo not copy and paste the secret. It can contain spaces and a paste\nmay result in confusing 401 errors that are just a mismatched token.\n\n```console\ngcloud auth login --no-launch-browser # VM's own service account cannot read it\n\nSECRET=$(gcloud secrets versions access latest --secret=dev-gke-app-secrets \\\n --project=moz-fx-sync-nonprod \\\n | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"SYNC_MASTER_SECRET\"])')\n\nexport SERVER_URL=\"https://dev.sync.nonprod.webservices.mozgcp.net#${SECRET}\"\necho \"${SERVER_URL%%#*}\"\n```\n\nThat last line prints the origin without revealing the secret. If empty,\nstop: an unset `SERVER_URL` results in a fall back to\n`https://token.stage.mozaws.net` and take the OAuth path instead.\n\n### Verify authentication before running\n\n```console\npoetry run python -c \"\nimport asyncio, aiohttp\nfrom storage.client import StorageClient\nasync def m():\n async with aiohttp.ClientSession() as s:\n c = StorageClient(s)\n r, _ = await c.get('/info/collections', (200, 404))\n print(r.status)\nasyncio.run(m())\n\"\n```\n\nUse `/info/collections`, not `/info/configuration`. The latter takes no auth\nextractor at all and returns 200 to anyone, so it cannot tell you whether your\nsecret is right.\n\n### Run\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n`OFFLOAD_COLLECTIONS` must name a collection the server actually offloads.\nCheck `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` in the deployment's\nvalues file.\n\nThen scale up, redirecting output to a file:\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n nohup poetry run molotov --processes 1 --workers 5 --duration 600 -v loadtest.py \\\n > /tmp/lt.log 2>&1 &\n```\n\nRamp gradually. Dev is small, and a sudden jump to high concurrency will hit\nits ceiling rather quickly.\n\n### Verify offload actually occurred\n\n```console\ngcloud storage ls -r \"gs://sync-nonprod-dev-syncstorage-payloads/**\" \\\n --project=moz-fx-sync-nonprod | grep -v '[/:]$' | tail -5\n```\n\nThen check at least one object was finalized by the reconciler:\n\n```console\ngcloud storage objects describe --project=moz-fx-sync-nonprod\n```\n\nLook for `custom_fields.committed: 'true'` and `custom_time` pinned to the far\nfuture. The gap between `creation_time` and `update_time` is your\nend-to-end reconciliation latency, which can be worth noting.\n\n> Note: the metadata key is `custom_fields` in `gcloud storage` output, not\n> `metadata`.\n\n---\n\n## Part 3: Method B, local syncserver with raised limits\n\nUse this when you need payloads larger than the deployed server allows. There\nis no runtime or client-side override: limits are read from the server's\nenvironment at startup, which you can easily change on the fly.\nIt runs on whatever `/info/configuration`\nis set to. Exceeding said server-side limits returns 413.\n\n### Your own bucket\n\nDo not use a shared environment's bucket. Make one you own:\n\n```console\ngcloud storage buckets create gs:// --project=$PROJ --location=us-west1\n```\n\nAdd a lifecycle rule. These objects are never finalized in this setup, so\nnothing else will clean them up. A 20 minute run can write hundreds of GB, so we\ndon't necessarily want these sticking around:\n\n```console\necho '{\"rule\":[{\"action\":{\"type\":\"Delete\"},\"condition\":{\"age\":1}}]}' > /tmp/lc.json\ngcloud storage buckets update gs:// --lifecycle-file=/tmp/lc.json --project=$PROJ\n```\n\n### Give the VM write access\n\nThe default compute service account is scoped to `devstorage.read_only`, so it\ncannot write, regardless of IAM settings. Note that scopes can only change while the instance\nis stopped:\n\n```console\nSA=$(gcloud compute instances describe sync-loadtest --project=$PROJ \\\n --zone=us-west1-b --format=\"value(serviceAccounts[0].email)\")\n\ngcloud storage buckets add-iam-policy-binding gs:// \\\n --member=serviceAccount:$SA --role=roles/storage.objectAdmin --project=$PROJ\n\ngcloud compute instances stop sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud compute instances set-service-account sync-loadtest \\\n --project=$PROJ --zone=us-west1-b --service-account=$SA \\\n --scopes=https://www.googleapis.com/auth/cloud-platform\ngcloud compute instances start sync-loadtest --project=$PROJ --zone=us-west1-b\n```\n\nDisk contents survive the stop.\n\nConfirm the new scope took effect by writing **from the VM**. Run locally this\nwould use your own credentials and prove nothing about the instance service\naccount, and `$PROJ` is a local shell variable that does not exist on the VM:\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap --command \\\n 'echo hello | gcloud storage cp - gs:///write-check.txt && \\\n gcloud storage rm gs:///write-check.txt'\n```\n\nUsing the instance service account this way means no expiring credentials and\nnothing to mount into the container.\n\n### Build the Docker image\n\n```console\nsudo apt install -y docker.io docker-compose-v2 docker-buildx\nsudo usermod -aG docker $USER && newgrp docker\n\ncd ~/src/syncstorage-rs\nDOCKER_BUILDKIT=1 nohup docker build -t app:build . > /tmp/build.log 2>&1 &\n```\n\n> `DOCKER_BUILDKIT=1` is required. The Dockerfile uses BuildKit cache mounts and\n> the legacy builder fails with \"the --mount option requires BuildKit\".\n\nPulling the published image instead is much faster if you have Artifact\nRegistry read, after `gcloud auth configure-docker us-docker.pkg.dev`. Tag it as\n`app:build` either way, because the compose file hardcodes that tag for its\nsetup container.\n\n### Compose override\n\n```console\ncd ~/src/syncstorage-rs\ncat > docker/docker-compose.offload-local.yaml <<'YAML'\nservices:\n syncserver:\n environment:\n SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET: \n SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS: bookmarks\n SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES: \"20971520\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_POST_BYTES: \"26214400\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_REQUEST_BYTES: \"26218496\"\nYAML\n```\n\nNo nginx runs in this stack, therefore the chart's `nginxClientMaxBodySize` (default\n`2568k`) does not apply. It does apply to any deployed environment, though.\nNote that raising syncserver limits without it gets you a 413 from nginx,\nbefore syncserver sees a request.\n\n### Docker compose\n\n```console\ndocker tag app:build syncstorage-rs:latest\n\ndocker compose -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.offload-local.yaml up -d\n\ndocker compose -f docker/docker-compose.spanner.yaml ps\n```\n\nThe compose default image is `syncstorage-rs:latest`, hence the tag. Otherwise\nset `SYNCSTORAGE_RS_IMAGE=app:build` on every compose.\n\nConfirm the limits are valid:\n\n```console\ncurl -s http://localhost:8000/1.5/1/info/configuration | python3 -m json.tool\n```\n\n`max_record_payload_bytes` must show your raised value.\n\n### Run\n\nThe master secret in this compose config is `secret0`:\n\n```console\ncd ~/src/syncstorage-rs/tools/syncstorage-loadtest\nsource venv/bin/activate\nexport SERVER_URL=\"http://localhost:8000#secret0\"\n\nLARGE_PAYLOAD_PROB=1.0 LARGE_PAYLOAD_SIZE=20971520 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\nConfirm the object sizes:\n\n```console\ngcloud storage ls -r --long \"gs:///**\" --project=$PROJ | tail -6\n```\n\n### Sampling memory during a run\n\nMemory can be easily measured here, and is a valuable performance metric.\nUse the below command to sample it to a file (rather than watching a streaming display):\n\n```console\nnohup bash -c 'while true; do date -u +%H:%M:%S | tr \"\\n\" \" \"; docker stats --no-stream --format \"{{.MemUsage}} {{.CPUPerc}}\" docker-syncserver-1; sleep 5; done' > /tmp/mem.log 2>&1 &\n```\n\nThen run the load tests. Then `pkill -f \"docker stats\"` and read `/tmp/mem.log`.\n\n---\n\n## Caveats\n\n**Molotov hides its errors.** A failing `setup_session` is counted and the\ntraceback is discarded, so any auth or setup problem shows as\n`SUCCESSES: 0 | FAILURES: 1` with nothing else.\n\nAlso note `_retry` only checks the HTTP status when the caller passes an\nexpected-statuses tuple. The scenario's `/info/configuration` call passes none,\nso a 503 appears as a JSON decode error rather than \"Response 503\".\n\n**Molotov's display can mangle terminals.** To save yourself the possible headache, always\nredirect to a file for anything but a short test, and if your terminal\ngoes blank or stops echoing, run `reset`.\n\n**Always redirect, never pipe.** Piping molotov's output through `head` or\n`grep` produces basically unhelpful output.\n\n**Co-locating the server and the load tester skews results.** Note that to some degree, Method B\nresults in both compete for the same CPU and memory. Use low concurrency, or alternatively a second VM.\n\n**The emulator is not Spanner.** Method B's write throughput and latency numbers\naren't super helpful and indicative of live spanner performance.\n\n**Fastly NGWAF is the edge, in blocking mode.** Dev sets\n`ngwaf_agent_level = \"block\"`, so sustained 403s partway through a run against a\ndeployed environment are worth checking there before blaming the load tester.\nThe `sync-policy` Cloud Armor resource still exists, but neither backend config\nattaches it, so it is not the cause.\n\n**Check where the Spanner metrics live.** Spanner for the sync tenant is in a\nseparate project, `moz-fx-sync-nonprod-904c` (now named `sync-legacy-nonprod`),\nand `sync/developers` has no access to it, so the console is unavailable. Since\nthe move to `sync-legacy` the metrics come from Grafana's `gcp-v2-nonprod`\ndatasource, not `gcp-v1-nonprod`, which no longer returns them.\n\n---\n\n## Cleanup\n\n```console\ngcloud compute instances delete sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud storage rm -r gs:///** --project=$PROJ\n```\n\nDelete the instance when you are done. An idle load generator bills\nindefinitely!\n","docs/src/tools/payload_link_reconciler.md":"# GCS Payload Reconciliation Pipeline\n\n## Summary\n\nWhen syncserver offloads a large BSO payload to Google Cloud Storage\n(`syncserver/src/web/payload_offload.rs`), the GCS object is created\nwith custom metadata `committed=false` and `customTime=now`, and the\nBSO row's `payload_link` column points at it. A separate pipeline must:\n\n- **Finalize** newly-committed objects — flip metadata to\n `committed=true` and pin `customTime` to its maximum value\n (`2200-12-31T23:59:59Z`) so the bucket's lifecycle policy (see\n below) cannot reclaim them.\n- **Garbage-collect orphans** — delete GCS objects whose row's\n `payload_link` was replaced (UPDATE) or removed (DELETE, including\n Spanner row-deletion-policy TTL deletes).\n\nAn object whose write never reached a Spanner commit never receives a\nfinalize, and there are two shapes of that. If the database transaction\nfails or rolls back, syncserver issues an inline best-effort **delete**\nof the objects it uploaded for that request and ignores the result. If an\nupload itself fails, the request is abandoned fail-fast and objects\nalready uploaded for it are not cleaned up inline at all. Either way an\nobject can be left stranded at `committed=false` with its upload-time\n`customTime`.\n\nAnything that slips through is reaped by a GCS **lifecycle policy**\n(`bucket.tf` in `webservices-infra/sync`) that deletes objects whose\n`customTime` is more than **30 days** old. Flipping `customTime` to the\nmax sentinel is what protects committed objects from that policy:\n`daysSinceCustomTime` goes permanently negative once finalized, so the\npolicy cannot touch them regardless of object age.\n\nThe 30 day window is the safety margin for the whole asynchronous arm. It\nhas to comfortably exceed the worst case time from upload to finalize,\nwhich is set by the cronjob cadence plus any Pub/Sub or Dataflow backlog.\nAt a five minute cadence the margin is four orders of magnitude, so the\nnumber only becomes interesting if the pipeline is stopped for weeks.\n\nThis document covers that pipeline. It consumes the\n`payload_link_changes` Spanner change stream defined in\n`syncstorage-spanner/src/schema.ddl`.\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n stream[(\"Spanner change stream\n payload_link_changes\n OLD_AND_NEW_VALUES, 7d retention\")]\n\n subgraph dataflow[\"Custom Dataflow flex template\"]\n filter{\"payload_link NULL\n on both sides?\"}\n end\n\n drop([\"dropped\"])\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n\n subgraph reconciler[\"Reconciler cronjob\"]\n route{\"which side\n carries a link?\"}\n finalize[\"finalize the object\n committed=true\n customTime=MAX\"]\n delete[\"delete the object\"]\n end\n\n stream -->|DataChangeRecord| filter\n filter -->|yes| drop\n filter -->|\"no, publish as JSON\"| topic\n topic -->|pull subscription| route\n topic -.->|\"after 5 failed deliveries\"| dlq\n route -->|new link| finalize\n route -->|\"old link, replaced or removed\"| delete\n```\n\nBoth reconciler actions are idempotent, so a redelivered record is\nharmless.\n\n## Components\n\n### 1. Spanner change stream — `payload_link_changes`\n\nDefined in `syncstorage-spanner/src/schema.ddl`:\n\n```sql\nCREATE CHANGE STREAM payload_link_changes\n FOR bsos(payload_link), batch_bsos(payload_link)\n OPTIONS (\n retention_period = '7d',\n value_capture_type = 'OLD_AND_NEW_VALUES'\n );\n```\n\nColumn-scoped: an UPDATE that does not touch `payload_link` produces no\nrecord. INSERTs and DELETEs always produce a record, even when\n`payload_link` is NULL — those are dropped at the next stage.\n\nThe Spanner DDL is not auto-applied; run `gcloud spanner databases ddl\nupdate` against the target database after merging.\n\n### 2. Custom Dataflow flex template — `tools/payload-link-dataflow/`\n\nA standalone Apache Beam pipeline (Java, Beam 2.60.0) that:\n\n1. Reads `payload_link_changes` via `SpannerIO.readChangeStream()`.\n2. Applies a `Filter.by(isPayloadLinkActionable)` step that drops\n records whose every mod has `payload_link` NULL on both sides.\n Malformed records pass through so the reconciler / DLQ surfaces\n them — not the filter.\n3. Serializes each surviving `DataChangeRecord` to JSON and publishes\n to a Pub/Sub topic.\n\nThe pipeline is **not** vendored from\n[`GoogleCloudPlatform/DataflowTemplates`](https://github.com/GoogleCloudPlatform/DataflowTemplates).\nWe own a small standalone source tree under `src/`; the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` template is referenced for\nintent comparison via `upstream-customization.patch` (documentation\nonly — not a build input).\n\n**Build / publish** (operator runs from `webservices-infra`):\n\n```bash\ndocker build -t /syncserver-payload-link-dataflow: \\\n tools/payload-link-dataflow\ndocker push /syncserver-payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/syncserver-payload-link-dataflow.json \\\n --image /syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n```\n\n**Launch parameters** (full list in\n`tools/payload-link-dataflow/metadata.json`):\n\n- `spannerProjectId`, `spannerInstanceId`, `spannerDatabase` — the\n syncstorage Spanner database.\n- `spannerMetadataInstanceId`, `spannerMetadataDatabase` — where the\n change-stream connector keeps its partition-state table. Dev uses a\n dedicated database, `syncdb-pldf-meta-dev`, so the connector holds no\n write access to the syncstorage database. Every environment should do\n the same.\n- `spannerMetadataTableName=Metadata_payload_link` names the\n partition-state table itself. Optional in the template, but pin it on\n any streaming launch. Left unset, the connector generates a random\n table name inside `expand()` and bakes it into the pipeline graph, so a\n `--update` or a cancel-then-relaunch comes up against a fresh empty\n table, reads from `Timestamp.now()`, and silently drops every record\n committed between the two jobs. The name only has to be a legal Spanner\n identifier and stable for the life of the environment; the connector\n creates the table on first use.\n See \"Job update and restart continuity\" in\n `tools/payload-link-dataflow/README.md`.\n- `changeStreamName=payload_link_changes`.\n- `spannerDatabaseRole=payload_link_reader` — the fine-grained access\n role the job reads the stream through, created by the DDL in\n `syncstorage-spanner/src/schema.ddl`. If the role is absent the job\n fails at startup rather than falling back to broader access.\n- `pubsubTopic=projects//topics/payload-link-changes`.\n\n**Service account requires:**\n\n- `roles/spanner.databaseUser` on the syncstorage database and on the\n metadata database. The one role covers both reading the change stream\n and maintaining the connector's partition metadata, but since the two\n live in separate databases the grant is needed on each. Note the IAM\n grant targets the `-904c` project, so it is applied out of band; see\n [GCP Infrastructure](payload-offload-infrastructure.md#out-of-band-steps).\n- Membership of the `payload_link_reader` database role, which is what\n actually narrows the job to the change stream. The IAM grant alone\n does not let it read BSO rows.\n- `roles/pubsub.publisher` on the destination topic.\n- `roles/storage.objectAdmin` on the Dataflow job bucket, for `staging/`\n and `tmp/`.\n- `roles/dataflow.worker` at the project level.\n\n### 2b. Dev/E2E Python publisher — `tools/payload-link-dataflow/payload-link-publisher-py/`\n\nPlain Python script (not Beam) that polls the\n`READ_payload_link_changes` TVF via `google.cloud.spanner.Client` and\npublishes the same JSON wire format the Java job produces. Sub-second\nstartup, no JVM. **Dev/E2E only** — used by the compose stack\ndescribed below; **Java remains the prod publisher.**\n\nFollows partition splits: reads `_root`, picks up child-partition\ntokens from `child_partitions_record`s, reads each child from its\nadvertised `start_timestamp`, retires parents once their children are\nannounced, and drops any partition that responds `OUT_OF_RANGE`.\nRequired in practice — the emulator routes DataChangeRecords to child\npartitions immediately, so a `_root`-only reader sees zero DCRs. Full\ndetails in the tool's own README.\n\n### 3. Pub/Sub topic + DLQ\n\nProvisioned from `webservices-infra/sync`:\n\n- Topic: `payload-link-changes`\n- Dead-letter topic: `payload-link-changes-dlq`\n- Pull subscription: `payload-link-reconciler-sub`\n - 60s ack deadline\n - 7d message retention\n - DLQ routing after 5 delivery attempts\n\n### 4. Reconciler — `tools/payload-reconciler/`\n\nPython script with one job: pull messages, perform GCS operations, ack.\nSync-pull drain loop with two deployment modes selected by whether\n`RUN_BUDGET_SECONDS` is set:\n\n- **Cronjob mode** (default deployment): `RUN_BUDGET_SECONDS` set (e.g.\n `240`). The script drains the subscription up to that many seconds\n or until the queue idles, then exits 0. K8s cronjob at ~5 min\n cadence.\n- **Long-running mode**: `RUN_BUDGET_SECONDS` unset. The script polls\n forever, never exiting on idle. Deploy as a K8s Deployment when\n finalize-flip latency below the cronjob cadence matters.\n\n**Per-message handling** (`reconcile_payload_links.py:handle_message_body`):\n\nFor each mod in the change record:\n\n- New `payload_link` present: `blob.patch()` sets\n `metadata.committed = \"true\"` and `customTime = \"2200-12-31T23:59:59Z\"`.\n- Old `payload_link` present and not equal to the new value: `blob.delete()`,\n with one exception. A `batch_bsos` row removed under the batch commit\n transaction tag (`transactionTag` equals `batch_commit`, set by syncstorage in\n the Spanner `commit_batch` path) is skipped\n (`payload_reconciler.batch_commit_skips`). On commit the link moves into the\n permanent `bsos` row in the same transaction, so its object must be kept\n (deleting it was the STOR-657 bug). Any other `batch_bsos` removal is a\n genuine delete: `batch_bsos` has no deletion policy of its own, so TTL expiry\n and `user_collections` deletes reach it as cascade deletes with no such tag,\n and their objects are removed. `bsos` changes and `batch_bsos` overwrites are\n unaffected. See STOR-668.\n\nBoth operations tolerate `404 NotFound` as success, see *Failure\nmodes* below.\n\n**Environment**\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | — | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | — | `payload-link-reconciler-sub` in prod. |\n| `GCS_PAYLOAD_BUCKET` | yes | — | Cross-bucket links abort the message. |\n| `RUN_BUDGET_SECONDS` | no | — | Set (e.g. `240`) → cronjob mode; drain up to N seconds then exit 0. Unset → long-running mode; poll forever, never exit on idle. |\n| `SYNC_STATSD_HOST` | no | localhost | statsd metrics server host. |\n| `SYNC_STATSD_PORT` | no | 8125 | statsd metrics server port. |\n\n**Service account requires:**\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`.\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and the `delete` operation).\n\n**Deployment.** Default is a K8s cronjob (~5 min cadence) with\n`RUN_BUDGET_SECONDS` set. When lower finalize-flip latency matters,\ndeploy as a K8s Deployment without `RUN_BUDGET_SECONDS` to run\nlong-running. The cronjob template is\n`sync/k8s/sync/templates/payload-reconciler-cronjob.yaml` in\nwebservices-infra, gated on `payloadReconciler.enabled`. Note it lives in\nthe `sync` chart rather than `sync-jobs`.\n\nFour settings in that manifest are load bearing, and changing any of them\nbreaks an assumption the script relies on:\n\n- `concurrencyPolicy: Forbid`. A run that overruns its window is never\n joined by a second one. The subscription is the queue, so the next tick\n simply picks up where the last left off. Two concurrent drains would\n race on the same messages, which is survivable given idempotency but\n wastes the ack deadline.\n- `backoffLimit: 0`. No in-window retry. Every operation is idempotent\n and unacked messages come back on their own, so the next scheduled run\n *is* the retry. A backoff would just re-drain the same queue sooner.\n- `runBudgetSeconds` strictly less than `activeDeadlineSeconds`. The\n chart fails the render if this is violated. The budget is what makes\n the drain loop exit cleanly on its own; if the deadline fired first the\n pod would be killed mid-message and the run would always look failed.\n- The command override. The image entrypoint is the syncserver binary, so\n the cronjob invokes\n `python3 /app/tools/payload-reconciler/reconcile_payload_links.py`\n explicitly. Invoking by path is also what puts the script's directory\n on `sys.path`, which is how its `import utils` resolves.\n\nCredentials come from workload identity: the pod runs as the tenant GKE\nservice account, which holds the two roles listed above. There is no key\nfile and no secret mounted.\n\n---\n\n## Wire format\n\nEach surviving change record reaches the reconciler as a JSON Pub/Sub\nmessage:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-30T00:00:00.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"transactionTag\": \"\",\n \"isSystemTransaction\": false,\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nMod fields (`keys`, `oldValues`, `newValues`) carry **JSON strings**\nthat the reconciler parses with a second `json.loads`, matching\nSpanner's change-streams wire convention.\n\n`transactionTag` carries the transaction tag Spanner records for the change,\nwhich the reconciler uses to skip batch commit handoffs (see above).\n`isSystemTransaction` is carried too but not currently consumed.\n\n---\n\n## Local e2e compose stack\n\n`make docker_run_reconciliation_e2e_tests` brings up a full-stack\ncompose environment that exercises the entire pipeline against\nemulators:\n\n| Service | Image | Role |\n|---|---|---|\n| `sync-db` | Spanner emulator (existing) | Spanner + change stream |\n| `pubsub-emulator` | `google-cloud-cli:emulators` | Pub/Sub |\n| `fake-gcs` | `fsouza/fake-gcs-server` | GCS |\n| `reconciliation-setup` | one-shot | creates Pub/Sub topic + subscription + bucket |\n| `payload-link-publisher` | Python publisher (default) | polls change stream → publishes to Pub/Sub |\n| `payload-reconciler` | primary image, entrypoint override | drains Pub/Sub → patches/deletes GCS objects |\n| `syncserver` | primary image | offload enabled for all `test_storage.py` collections |\n| `e2e-tests` | primary image | runs `pytest tools/integration_tests/ tools/tokenserver/` |\n\nTwo publisher variants:\n\n- **Python (default)** — `docker-compose.e2e.reconciliation.yaml`.\n Sub-second startup; the day-to-day iteration path.\n- **Java (swap-in)** — layer\n `docker-compose.e2e.reconciliation.java.yaml` on top. Runs the same\n Java flex-template image under Beam's DirectRunner. See\n `tools/payload-link-dataflow/README.md` for the exact invocation.\n Use when you need to reproduce a Java-specific issue.\n\nThe compose stack doubles as regression coverage for `test_storage.py`:\nby opting every collection the storage tests use into GCS offload, each\nBSO write flows through the offload path (upload + `payload_link`\nstorage), and each read flows through `download_payload`. New\nreconciler-specific tests live in\n`tools/integration_tests/test_payload_link_reconciliation.py`, gated\nby a module-level `pytest.mark.skipif` on `GCS_PAYLOAD_BUCKET` so they\nauto-skip in the existing spanner e2e stack (where the env var is\nunset) and un-skip here.\n\n**Emulator fallback.** If a future Spanner emulator upgrade breaks\n`SELECT * FROM READ_payload_link_changes(...)`, the fall-back is the\nJava swap-in overlay above (or running the compose stack against a\nreal dev Spanner instance).\n\n---\n\n## Metrics\n\nBoth halves report. The reconciler emits under the\n`payload_reconciler` statsd namespace, syncserver under\n`storage.gcs.payload`. Between them every GCS operation reports both a\ncount and a latency.\n\n### Reconciler\n\nCounters:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalizes` | Objects flipped to `committed=true` | Tracks offloaded write volume |\n| `orphan_deletes` | Objects deleted because a row stopped pointing at them | Tracks overwrite and delete volume |\n| `gcs_404` `op:finalize` | Finalize target was gone | Low and flat |\n| `gcs_404` `op:delete` | Delete target was already gone | Low and flat, redeliveries are normal |\n| `batch_commit_skips` | A `batch_bsos` removal carrying the `batch_commit` transaction tag was kept rather than deleted | Tracks batch commit volume on offloaded collections |\n| `noop_skips` | A record arrived with nothing to do | Near zero; the Dataflow filter should have dropped it |\n| `errors` `kind:handler` | Handler raised, message left unacked | Zero |\n| `runs` | One per drain, emitted unconditionally | Tracks the cronjob cadence |\n| `messages_processed` | Change records pulled and handled, counted per batch | Tracks pipeline volume |\n| `budget_exhausted` | A run hit `RUN_BUDGET_SECONDS` before the queue idled | Zero |\n\nTimings:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalize_age` | Spanner commit to finalize, in ms | Seconds to minutes. This is the window the 30 day lifecycle policy is sized against |\n| `gcs_op` | One GCS round trip. Tagged `op` (`finalize` or `delete`) and `result` (`success`, `not_found`, `error`) | Low. Separates our own GCS latency from Dataflow and Pub/Sub delay |\n| `drain_duration` | How long one run took | Comfortably under `RUN_BUDGET_SECONDS` |\n\n`runs` is the only unconditional series. Every other metric here fires\nonly on the event it names, so a quiet pipeline emits almost nothing and\n\"idle\" would otherwise be indistinguishable from \"not running\".\n\n### Syncserver offload path\n\nEmitted from `syncserver/src/web/payload_offload.rs`. All three GCS\noperations report latency, each tagged with a `result`.\n\n| Metric | Type | Tags | Meaning |\n|---|---|---|---|\n| `storage.gcs.payload.upload` | timing | `result` | One object uploaded |\n| `storage.gcs.payload.download` | timing | `result` | One object read back, streaming included |\n| `storage.gcs.payload.cleanup.duration` | timing | `result` | The compensating delete after a failed write transaction |\n| `storage.gcs.payload.batch` | timing | `op`, `handler`, `result` | Total GCS time for one request, across all of its BSOs |\n| `storage.gcs.payload.size` | histogram | `op` | Distribution of individual payload sizes, in bytes |\n| `storage.gcs.payload.upload.bytes` | counter | none | Bytes written, for throughput |\n| `storage.gcs.payload.download.bytes` | counter | none | Bytes read back |\n| `storage.gcs.payload.cleanup` | counter | `handler`, `result`, `reason` | Outcome of the compensating delete |\n\nStatsd timers carry their own count, so there is no separate counter for\nthe number of uploads or downloads.\n\n`storage.gcs.payload.batch` is not the sum of the per-object timings. A\nrequest works on up to `gcs_payload_max_concurrency` objects at once, so\nits total lands somewhere between the slowest single object and the sum\nof all of them. It is the delay the client waits through, and it is\nrecorded even when the batch fails part way, hence the `result` tag: an\nabandoned batch stopped early and is not comparable to a complete one.\n\n`storage.gcs.payload.size` is a histogram rather than a tag on the\ntimings, so the aggregator derives percentiles from it. The byte\ncounters give a total and therefore a mean; the tail is the part that\nmatters on a project about raising a size ceiling.\n\nTags on `storage.gcs.payload.cleanup`:\n\n| Tag | Values | Meaning |\n|---|---|---|\n| `handler` | `put_bso`, `post_collection` | Which write handler issued the cleanup |\n| `result` | `success`, `failure` | Whether the object was deleted |\n| `reason` | `invalid_url`, `gcs_error` | Failure only. `invalid_url` means the `gs://` URL did not parse and no delete was attempted; `gcs_error` means GCS rejected it |\n\nThe cleanup counter firing at all means writes are failing after their\npayload reached GCS, so read it against the write error rate rather than\non its own. `result:failure` is the more interesting cut: those objects\nare now leaked until the 30 day lifecycle policy reaps them. A steady\n`reason:invalid_url` is a code bug rather than an operational problem,\nsince the URL is one syncserver just generated. Note that an unparseable\nURL issues no GCS call and so has no `cleanup.duration` alongside it.\n\nWorth alerting on:\n\n- Anything at all in `payload-link-changes-dlq`. A message only lands\n there after five failed deliveries, so it means a record cannot be\n handled and needs a human. Inspect it through\n `payload-link-changes-dlq-sub`. Note the DLQ is terminal: its\n subscription carries no dead-letter policy of its own, so a message\n sits there until the 7 day retention drops it and no further queue\n catches it. That is the same 7 days as the change stream retention,\n so once a dead-lettered record ages out the underlying change is\n unrecoverable from both. Seven days is a response deadline, not a\n margin.\n- `errors` `kind:handler` sustained above zero.\n- `noop_skips` climbing, which means the Dataflow filter regressed and\n the pipeline is paying Pub/Sub for records it discards.\n- `gcs_404` `op:finalize` rising as a share of `finalizes`, which is the\n signal that objects are being reaped before they get finalized.\n- Oldest unacked message age on `payload-link-reconciler-sub`. This is a\n Pub/Sub metric rather than one of ours, and it is the most direct\n measure of finalize latency. It should sit in minutes. The 30 day\n lifecycle window is the deadline it must never approach.\n- `storage.gcs.payload.cleanup` with `result:failure`, which counts\n objects that leaked because the compensating delete did not land.\n- `finalize_age` climbing. It is the direct measure of how long objects\n sit unprotected, and unlike the Pub/Sub metric above it covers the\n whole path including the GCS patch.\n- `budget_exhausted` sustained above zero, meaning runs keep ending on\n the clock rather than on an idle queue, so the drain is not keeping up.\n- `runs` going flat. Every other reconciler metric is conditional, so\n this is the one that separates a quiet pipeline from a stopped one.\n\n---\n\n## Failure modes\n\n| Symptom | Cause | Behaviour |\n|---|---|---|\n| Spike in `payload_reconciler.noop_skips` | Dataflow filter is letting inert records through | Investigate; should be ~0 if the filter works. Records still ack — no harm but extra Pub/Sub cost. |\n| Sustained `payload_reconciler.gcs_404` with `op:finalize` | Lifecycle rule reclaimed the object before the reconciler finalized it, OR the same message redelivered after a successful prior run (at-least-once tax) | Acceptable up to a low background level. Sharp rise = lifecycle window too aggressive vs. cronjob cadence. |\n| Sustained `payload_reconciler.gcs_404` with `op:delete` | Object was already deleted (redelivery or concurrent cleanup) | Acceptable; idempotent by design. |\n| Messages in `payload-link-changes-dlq` | Repeated handler exceptions on the same message after 5 retries (malformed JSON, cross-bucket link, GCS auth failure) | Inspect the DLQ payload; fix and re-publish or discard. The main subscription continues to drain. |\n| `payload_reconciler.errors` with `kind:handler` non-zero | Same as above before reaching DLQ. | Same. |\n| `storage.gcs.payload.cleanup` with `result:failure` | The compensating delete after a failed write transaction did not land | The object is stranded at `committed=false` and the 30 day lifecycle policy is the only thing left to reap it. Occasional failures are tolerable; sustained means the write path is failing and cleanup is failing with it. |\n| Finalize latency jumps after a Dataflow deploy, with a gap in `finalizes` | The job relaunched against a fresh partition-metadata table and resumed from `Timestamp.now()` | Records committed during the gap are lost for good; the change stream's 7 day retention only helps if you notice in time. Check that `spannerMetadataTableName` is pinned and unchanged. |\n\nA `payload_link` pointing at a bucket other than `GCS_PAYLOAD_BUCKET`\nraises `ValueError` and the message is left unacked — it retries up to\nthe DLQ rather than mutating an unrelated bucket. This is a hard guard.\n\n---\n\n## Keeping the reference patch accurate\n\n`tools/payload-link-dataflow/upstream-customization.patch` is\ndocumentation only — not a build input. If the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` pipeline evolves in ways that\nchange the conceptual diff, refresh it (see the README in that\ndirectory). This is a doc refresh, not a code change.\n","docs/src/tools/postgres_purge_ttl.md":"# Documentation for `purge_ttl.py` (PostgreSQL)\n\n## Summary\n\nThe `purge_ttl.py` script is a utility for purging expired Time-To-Live (TTL) records from a Sync storage PostgreSQL database. It deletes expired entries from the `batches` and/or `bsos` tables, with optional filtering by collection ID and a dry-run mode for verifying query logic without modifying data.\n\n> Spanner no longer uses this script — it expires records natively via its built-in [row deletion (TTL) policies](https://cloud.google.com/spanner/docs/ttl). This utility applies to PostgreSQL deployments only.\n\n---\n\n## Status\n- Runs as a regularly scheduled job (e.g. a cron job) in PostgreSQL (\"enterprise\") deployments.\n- The script is bundled in the main `syncserver-postgres` image and invoked from there; there is no separate utility image.\n\n## Specifics\n\n- **Database**: PostgreSQL (accepts a `postgresql://` or `postgres://` DSN).\n- **Tables**:\n - `batches`: Contains batch entries (child `batch_bsos` rows are removed via the schema's foreign-key cascade).\n - `bsos`: Stores Sync Basic Storage Objects (BSO).\n- **Supported Modes**:\n - `batches`: Purges expired entries in the `batches` table.\n - `bsos`: Purges expired entries in the `bsos` table.\n - `both`: Performs purges on both tables.\n- **Expiry Modes**:\n - `now`: Purges entries with `expiry < CURRENT_TIMESTAMP`.\n - `midnight`: Purges entries with `expiry < DATE_TRUNC('day', CURRENT_TIMESTAMP AT TIME ZONE 'UTC')`.\n\nThe script tracks execution duration and rows affected using StatsD metrics for performance monitoring.\n\n---\n\n## Notes\n\n- Ensure the database role used by the DSN has `DELETE` privileges on the target tables.\n- Use the `--dryrun` option to verify query logic before actual purging.\n- Consider setting up automated monitoring for long-running operations or performance issues.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3 is installed.\n2. **Poetry**: Install [Poetry](https://python-poetry.org/docs/#installation).\n3. **Dependencies**: From `tools/postgres`, run `poetry install`.\n4. **Environment Variables:**\n `SYNC_SYNCSTORAGE__DATABASE_URL`: Database connection URL (e.g., `postgresql://user:pass@host:5432/syncstorage`). Used when `--database_url` is not supplied.\n\n### Usage\n\nRun the script using the following command:\n ```bash\n poetry run python purge_ttl.py [options]\n ```\n\n#### Options\n\n| Option | Description | Default |\n|---------------------------------|----------------------------------------------------------------------------------------|----------------------------------|\n| `-u`, `--database_url` | PostgreSQL DSN (`postgresql://...` or `postgres://...`). Required if the env var is unset. | `SYNC_SYNCSTORAGE__DATABASE_URL` |\n| `--collection_ids`, `--ids` | Comma-separated list of collection IDs to purge. | `[]` |\n| `--mode` | Purge mode: `batches`, `bsos`, or `both`. | `both` |\n| `--expiry_mode` | Expiry mode: `now` (current timestamp) or `midnight` (start of current day, UTC). | `midnight` |\n| `--dryrun` | Perform a dry run without making changes to the database. | `False` |\n\n#### Examples\n\n##### Example 1: Basic Purge\nPurge expired entries from both `batches` and `bsos` tables using the DSN from the environment:\n```bash\n poetry run python purge_ttl.py\n```\n\n##### Example 2: Specify the Database URL\n```bash\n poetry run python purge_ttl.py -u postgresql://user:pass@host:5432/syncstorage\n```\n\n##### Example 3: Filter by Collection IDs\nPurge only for specific collection IDs:\n```bash\n poetry run python purge_ttl.py --collection_ids [123,456,789]\n```\n\n##### Example 4: Purge a Single Table\n```bash\n poetry run python purge_ttl.py --mode bsos\n```\n\n##### Example 5: Perform a Dry Run\nTest the script without making actual changes:\n```bash\n poetry run python purge_ttl.py --dryrun\n```\n\n### Detailed Usage\n\n1. **Connecting to PostgreSQL**:\n - The script builds a SQLAlchemy engine from the supplied DSN. A `postgres://` scheme is normalized to `postgresql://` for newer SQLAlchemy versions.\n\n2. **Purge Modes**:\n - `batches`: Deletes expired entries from the `batches` table.\n - `bsos`: Deletes expired BSOs.\n - `both`: Executes purges on both `batches` and `bsos`.\n\n3. **Expiry Conditions**:\n - `now`: Purge entries that have already expired at the current timestamp.\n - `midnight`: Purge entries that expired at or before the start of the current UTC day.\n\n4. **Query Customization**:\n - An optional collection-ID filter is appended via the `add_conditions` helper using bound parameters.\n\n5. **Performance Monitoring**:\n - Metrics for execution duration and rows affected are logged and sent to StatsD for monitoring.\n\n6. **Dry Run**:\n - Enabling the `--dryrun` flag ensures that the queries are constructed and logged without executing them on the database.\n","docs/src/tools/purge_old_records_tokenserver.md":"# Documentation for `purge_old_records.py`\n\n## Summary\n\nThe `purge_old_records.py` script is an administrative utility for managing obsolete user records in Tokenserver. It removes outdated user records from the database and deletes associated data from storage nodes. This process helps reduce storage overhead, improve database performance, and maintain the health of the Tokenserver system.\n\nObsolete records are those replaced by newer records for the same user or marked for deletion if the user has deleted their account. The script can run in batch mode for periodic cleanup and includes options for dry-run testing and forced purging when nodes are down.\n\n---\n\n## Status\n - Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-purge-old-records-0`\n - See YAML configuration when editing each job.\n - See Kubernetes Engine Workload Panel in [sync-prod](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information. \n\n---\n\n## Specifics\n\n- **Primary Functionality**:\n - Deletes obsolete user records.\n - Issues delete requests to user storage nodes to purge related data.\n- **Optional Administrative Task**:\n - The script complements internal record replacement handled by the backend but is not mandatory for system operation.\n- **Batch Processing**:\n - Operates in loops, processing records in batches of a configurable size.\n- **Grace Period**:\n - Provides a grace period to avoid prematurely deleting recently replaced records.\n- **Dry Run**:\n - Offers a non-destructive mode for testing.\n\n---\n\n## Notes\n- **Regular Use**:\n - Running this script regularly can help maintain system performance and reduce storage overhead.\n- **Concurrency**:\n - When running multiple instances of this script, use the `--max-offset` option to reduce collisions.\n- **Forced Deletion**:\n - Use the `--force` option cautiously, especially if storage nodes are down.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3.7+ is installed.\n2. **Dependencies**:\n - Install required Python packages:\n `pip install requests hawkauthlib backoff`\n\n3. **Configuration**:\n - Set up access to the Tokenserver database.\n - Provide necessary metrics and logging configurations.\n\n### Usage\n\nRun the script using the following command:\n```bash\npython purge_old_records.py [options] secret\n```\n\n\n### Options\n\n| Option | Description | Default |\n|-------------------------|-------------------------------------------------------------------------------------------------|--------------|\n| `--purge-interval` | Interval in seconds to sleep between purging runs. | `3600` |\n| `--grace-period` | Grace period in seconds before deleting records. | `86700` (24h5m) |\n| `--max-per-loop` | Maximum number of items to process in each batch. | `10` |\n| `--max-offset` | Random offset to select records for purging, reducing collisions in concurrent tasks. | `0` |\n| `--max-records` | Maximum number of records to purge before exiting. | `0` (no limit) |\n| `--request-timeout` | Timeout in seconds for delete requests to storage nodes. | `60` |\n| `--oneshot` | Perform a single purge run and exit. | Disabled |\n| `--dryrun` | Test the script without making destructive changes. | Disabled |\n| `--force` | Force purging even if the user's storage node is marked as down. | Disabled |\n| `--override_node` | Specify a node to override for deletions if data is copied. | None |\n| `--range_start` | Start of UID range to process. | None |\n| `--range_end` | End of UID range to process. | None |\n| `--human_logs` | Enable human-readable logs. | Disabled |\n| `--last-seen-staging-table` | BigQuery last-seen staging table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-table`. | None |\n| `--last-seen-table` | BigQuery last-seen final table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-staging-table`. | None |\n\n### Examples\n\n#### Example 1: Basic Purge\nPerform a basic purge of obsolete user records:\n```bash\npython purge_old_records.py secret_key\n```\n#### Example 2: Grace Period and Dry Run\nPurge records with a 48-hour grace period in dry-run mode:\n```bash\npython purge_old_records.py --grace-period 172800 --dryrun secret_key\n```\n\n#### Example 3: Specify Range and Offset\nPurge records within a UID range with a random offset:\n\n```bash\npython purge_old_records.py --range_start uid_start --range_end uid_end --max-offset 50 secret_key\n```\n\n#### Example 4: Force Purge on Downed Nodes\nForce the deletion of data on downed nodes:\n```bash\npython purge_old_records.py --force secret_key\n```\n\n---\n\n## Detailed Usage\n\n1. **Batch Processing**:\n - The script processes records in batches defined by the `--max-per-loop` option.\n - Each batch is fetched from the database using random offsets to avoid overlapping with concurrent runs.\n\n2. **Grace Period**:\n - The grace period ensures that recently replaced records are not prematurely deleted.\n\n3. **Storage Node Cleanup**:\n - For each user, the script sends a delete request to their storage node to remove associated data.\n\n4. **Metrics Tracking**:\n - Tracks operations like user record deletions, service data deletions, and errors using metrics integration.\n\n5. **Error Handling**:\n - Uses exponential backoff to retry failed HTTP requests.\n - Detects loops in batch processing and raises exceptions.\n\n6. **Dry Run Mode**:\n - Simulates deletions without modifying the database or storage nodes, useful for testing.\n\n7. **Account Last-Seen Deletion**:\n - With both `--last-seen-*` options, each record retired by an FxA account deletion first has its account's rows deleted from the BigQuery last-seen tables, just before the record itself.\n - If that job fails or times out, the retired record stays, `replaced_at` is bumped, and it is retried after another grace period. Other records continue normally.\n - The default grace period (24 hours 5 minutes) outlasts the one-day OAuth token lifetime plus the upload delay, so an account's last activity normally reaches BigQuery before purge deletes it.\n\n---\n\n","tools/README.md":"# A collection of Sync Tools and utilities\n\nSee each directory for details:\n\n* [hawk](hawk) - a tool for generating test HAWK authorization headers\n* [spanner](spanner) - Google Cloud Platform Spanner tools for maintenance and testing\n* [payload-link-dataflow](payload-link-dataflow) - custom Dataflow flex template feeding the payload-link reconciliation pipeline\n* [payload-reconciler](payload-reconciler) - Pub/Sub-driven reconciler that finalizes and garbage-collects offloaded GCS payload objects\n## Installation\n\nThese tools are mostly written in python. It is recommended that you create a commonly shared virtual environment using something like:\n\n`python3 -m venv venv`\n\nto create a `/venv` directory. To activate this, call `sh /venv/bin/activate`.\n\nScript dependencies can be installed via poetry:\n\n```shell\npip install poetry\npoetry install\n```\n\nfor each tool.\n","tools/examples/README.md":"# Example scripts for working with syncstorage-rs\n\n* `put.bash` - First `chmod +x put.bash` then `./put.bash` to simulate a PUT.\n","tools/hawk/README.md":"# Make a Hawk compatible Auth header\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs `make_hawk_token.py` script, you'll need a Python =>3.10 development environment with `Poetry` installed. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `hawk/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/hawk\n # pyenv version install\n $ pyenv install 3.14\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.14 hawk # or whatever project name you like.\n $ pyenv local hawk # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\n`poetry run python make_hawk_token.py`\n\n## Create a Token Header:\n\nYou'll need to pass along your `SYNC_MASTER_SECRET` and the uri you'll be testing in order to generate a valid Hawk Id:\n\n`poetry run python make_hawk_token.py --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`\n\n** For testing against uri's using methods other than GET, you'll need to pass along the `--method` flag to generate your token. Ie, `poetry run python make_hawk_token.py --method PUT --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`. See [examples/put.bash](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for an example of this.\n\nUse `-h` for help.\n\nBy default, with no passed arguments, a default Hawk Id token will be generated mapping to your localhost:8000 with\nthe path `http://localhost:8000/1.5/1/storage/col2/`.","tools/payload-link-dataflow/README.md":"# payload-link-dataflow\n\nCustom Dataflow flex template that reads the `payload_link_changes`\nSpanner change stream, drops records whose every mod has `payload_link`\nNULL on both sides, and publishes the remainder to a Pub/Sub topic.\n\nThe downstream consumer is the `payload-reconciler` cronjob\n(`tools/payload-reconciler/`), which finalizes newly-committed GCS\npayload objects (`committed=true`, `customTime=MAX`) and deletes\norphaned ones.\n\n## Directory layout\n\nThe Java source tree under `src/` is a standalone Apache Beam pipeline\nwe own and build directly. It is intentionally minimal: it uses Beam's\nstock `SpannerIO.readChangeStream` and `PubsubIO.writeStrings`, plus the\nfilter and a small JSON serializer. We do **not** vendor or depend on\nthe upstream GoogleCloudPlatform/DataflowTemplates Java tree.\n\n| Path | Purpose |\n|---|---|\n| `pom.xml` | Maven project (Apache Beam + Jackson + SLF4J). |\n| `src/main/java/.../PayloadLinkChangesToPubSub.java` | The pipeline: read change stream → filter → serialize → publish. |\n| `src/main/java/.../PayloadLinkOptions.java` | Flex template parameters. |\n| `src/test/java/.../PayloadLinkChangesToPubSubTest.java` | Unit tests for the filter predicate. |\n| `metadata.json` | Flex template parameter schema (consumed by `gcloud dataflow flex-template build`). |\n| `Dockerfile` | Multi-stage build: Maven → flex template launcher base image. |\n| `upstream-customization.patch` | **Reference documentation only.** Unified diff showing the **filter delta only** against the upstream template; see \"Beyond the filter\" below for the rest. Not a build input. |\n| `upstream.txt` | The upstream commit SHA the reference patch was last reviewed against. Update when re-reviewing intent against upstream. |\n| `generate-full-delta.sh` | On-demand: produces a unified diff of the **complete** delta between upstream (at the pinned SHA) and our impl. Not maintained as a checked-in artifact -- runs against the source tree at execution time. |\n| `payload-link-publisher-py/` | Dev/E2E-only Python variant of this publisher (not Beam, not shipped to prod). See its own README. |\n| `README.md` | This file. |\n\n## Build\n\n```bash\ndocker build -t payload-link-dataflow:local tools/payload-link-dataflow\n```\n\n`mvn package` runs the unit tests, so a change that breaks the filter\ncontract fails the image build.\n\nFor a local non-Docker build:\n\n```bash\ncd tools/payload-link-dataflow\nmvn -B package\n```\n\nThe bundled jar lands at\n`target/payload-link-dataflow-bundled-1.0-SNAPSHOT.jar`.\n\n## Publish + launch (operator workflow; lives in webservices-infra)\n\n```bash\ndocker tag payload-link-dataflow:local /payload-link-dataflow:\ndocker push /payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/payload-link-dataflow.json \\\n --image /payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n\ngcloud dataflow flex-template run payload-link-dataflow- \\\n --template-file-gcs-location gs:///templates/payload-link-dataflow.json \\\n --parameters spannerProjectId= \\\n --parameters spannerInstanceId= \\\n --parameters spannerDatabase= \\\n --parameters spannerMetadataInstanceId= \\\n --parameters spannerMetadataDatabase= \\\n --parameters spannerMetadataTableName=Metadata_payload_link \\\n --parameters changeStreamName=payload_link_changes \\\n --parameters pubsubTopic=projects//topics/payload-link-changes\n```\n\n`spannerMetadataTableName` is optional but strongly recommended for\nstreaming launches -- see \"Job update and restart continuity\" below.\n\nThe Dataflow job's service account needs:\n- `roles/spanner.databaseReader` on the syncstorage database\n- `roles/spanner.databaseUser` on the metadata database (the change\n stream connector writes its own partition state there)\n- `roles/pubsub.publisher` on the destination topic\n- `roles/dataflow.worker`\n\n### Fine-grained access control (optional)\n\nPass `spannerDatabaseRole` to read the change stream under a Spanner\ndatabase role instead of a database-wide IAM grant. `payload_link_reader`\nin `syncstorage-spanner/src/schema.ddl` is that role: it holds `SELECT`\non `payload_link_changes` and `EXECUTE` on the generated\n`READ_payload_link_changes` table function, and nothing else.\n\nThe role applies to the change stream read only. Beam's\n`MetadataSpannerConfigFactory` deliberately does not copy `databaseRole`\nonto the connector's metadata database config, so that connection still\nuses the service account's own IAM grants. Two consequences, both from\n[Spanner's FGAC guidance](https://docs.cloud.google.com/spanner/docs/fgac-change-streams):\n\n- `spannerMetadataDatabase` **must** be a different database than\n `spannerDatabase`. The connector needs database-level write access for\n its partition state, and that grant would override the role's\n restrictions on the database being read.\n- the service account must **not** hold `roles/spanner.databaseUser` on\n `spannerDatabase` -- it bypasses fine-grained access control outright.\n Grant `roles/spanner.fineGrainedAccessUser`, plus\n `roles/spanner.databaseRoleUser` conditioned on the role resource,\n instead.\n\nIf the metadata tables share the syncstorage database, setting\n`spannerDatabaseRole` buys nothing security-wise: the role is assumed,\nbut the database-level grant the connector needs supersedes it.\nProvision a dedicated metadata database first.\n\n### Job update and restart continuity\n\nPass `spannerMetadataTableName` on any streaming (unbounded) launch so\n`--update` and cancel-then-restart can resume from the previous job's\nwatermarks. The connector generates the partition metadata table name\ninside `expand()` from a `UUID.randomUUID()` call, and that string\ngets baked into the pipeline graph; if the new job's binary produces\na different UUID, it reinitializes against a fresh empty table and\nreads from `startTimestamp` (defaulting to `Timestamp.now()`),\nsilently dropping records committed between the two jobs.\n\nPick any Spanner-legal identifier -- e.g. `Metadata_payload_link` -- and\nkeep it stable across launches. The connector creates the table on\nfirst initialization if it does not already exist, then reuses it.\nBeam's own recommendation: *\"you have to explicitly set the metadata\ntable name parameter of the new job when updating it. Use the value\nof the `metadataTable` pipeline option from the job you are\nupdating.\"*\n\nNotes on stop modes:\n\n- `--update`: preserves Dataflow's checkpointed state but does **not**\n transfer the metadata table binding -- pin the name or lose state.\n- `--cancel`: leaves the metadata table intact; the next launch with\n the same name resumes cleanly.\n- `--drain`: the change stream connector does not support drain\n (per [Google's guidance](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining)).\n Use `--cancel` or `--update` instead.\n\n## Filter behaviour\n\n`PayloadLinkChangesToPubSub.isPayloadLinkActionable` keeps a\n`DataChangeRecord` when any of its mods has a non-null `payload_link`\non either side. It drops only the case where **every** mod has both\nold and new `payload_link` NULL -- the inert INSERT/DELETE noise from\ncolumn-scoped change streams. Malformed records pass through so the\nreconciler / DLQ -- not this filter -- surfaces them.\n\n## Beyond the filter: other differences from upstream\n\n`upstream-customization.patch` expresses only the filter step. Our\npipeline additionally diverges from upstream's\n`Cloud_Spanner_Change_Streams_to_PubSub` template in ways that diff\ndoes not attempt to capture. Run `./generate-full-delta.sh` to see\nthe complete diff against the pinned upstream SHA; the substantive\ndifferences are:\n\n- **Sink swap.** `PubsubIO.writeStrings` + a custom `serializeRecord`,\n not upstream's `FileFormatFactorySpannerChangeStreamsToPubSub`.\n Emits raw JSON strings; no PubsubMessage attributes.\n- **JSON wire format.** Emits only `{commitTimestamp, modType,\n tableName, mods[]}`. Omits fields upstream would emit:\n `serverTransactionId`, `rowType[]`,\n `numberOfRecordsInTransaction`,\n `numberOfPartitionsInTransaction`, `partitionToken`,\n `recordSequence`, `isLastRecordInTransactionInPartition`,\n `valueCaptureType`.\n- **No `UncaughtExceptionLogger.register()`.** Uncaught exceptions\n flow to the runner's default handler.\n- **No `@Template` / `TemplateCategory` annotations.** Upstream uses\n the `com.google.cloud.teleport.metadata` annotation-driven\n registrar for flex-template metadata. We register via\n `metadata.json` + `gcloud dataflow flex-template build` instead --\n simpler, no annotation-processor dependency.\n- **No support for these upstream options:**\n `useSpannerEmulatorHost`, `spannerHost`,\n `spannerChangeStreamTvfNameList`, `outputMessageMetadata`,\n `outputDataFormat` (JSON/Avro switch), `pubsubAPI`. Our\n `PayloadLinkOptions` deliberately has a smaller surface (~12\n options vs. upstream's ~25).\n- **No ValueProvider indirection on options.** Upstream wraps\n several config fields in `ValueProvider` for template\n parameterisation; we take plain strings.\n- **No explicit `streaming=true` mutation.** Beam infers streaming\n from the source (`SpannerIO.readChangeStream` is\n `@UnboundedPerElement`) and `DataflowRunner` flips the flag at\n translation time.\n\nThese deltas were deliberate scope choices, not oversights. The\nfilter-only patch is kept as the primary review artifact because\nit's small enough to eyeball, applies cleanly against the pinned\nSHA, and is insensitive to upstream churn on features we don't use.\n\n`main()` matches upstream on two pipeline-shape invariants that are\nnot deployment choices for this source: it appends `use_runner_v2`\nto `options.experiments` and calls `setEnableStreamingEngine(true)`.\n`BundleFinalizer` (used by `SpannerIO.readChangeStream`) requires the\nDataflow Portable Runner (\"Runner V2\"), and V2 streaming jobs\nrequire Streaming Engine -- so both settings are pinned in code\nrather than left to launch-time flags.\n\n## Output wire format\n\nEach surviving `DataChangeRecord` is serialized to a JSON Pub/Sub\nmessage of the form:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-29T12:34:56.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nEach mod field carries a **JSON string** (escaped) -- the downstream\nPython reconciler reads them with a second `json.loads` per mod. This\nmatches the Spanner change-streams wire convention.\n\n## Running against the compose e2e stack\n\nThe `docker_run_reconciliation_e2e_tests` Makefile target brings up\nSpanner + Pub/Sub + fake-GCS emulators and a publisher container\nlocally. There are two publisher variants; both produce the exact same\nPub/Sub wire format so the reconciler consumes either\nindistinguishably.\n\n### Default: Python publisher\n\n`docker/docker-compose.e2e.reconciliation.yaml` builds and runs the\nPython variant at `payload-link-publisher-py/`. Sub-second startup,\nno JVM. Use this for day-to-day iteration.\n\n```bash\nmake docker_run_reconciliation_e2e_tests\n```\n\n### Swap-in: this Java publisher\n\n`docker/docker-compose.e2e.reconciliation.java.yaml` overrides the\n`payload-link-publisher` service to build **this** directory's Docker\nimage and run the shaded JAR under Beam's DirectRunner (bypassing the\nflex-template launcher's Dataflow-submission entrypoint). Use when you\nneed to reproduce a prod issue against the actual Java pipeline or\nverify wire-format parity between the two variants.\n\n```bash\ndocker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.reconciliation.java.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n up --exit-code-from e2e-tests --abort-on-container-exit\n```\n\nExpect a ~15-30s JVM+Beam warmup and a heavier memory footprint than\nthe Python variant. If Docker's default resource limits are tight on\nyour dev machine, bump them.\n\n## Keeping the reference patch accurate\n\n`upstream-customization.patch` is reference documentation, not a build\ninput. If upstream's pipeline shape evolves or our pipeline drifts in\nways that change the conceptual diff, regenerate it:\n\n```bash\ngit clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git upstream\ncd upstream\ngit checkout \n# Manually re-author the filter customization against the current\n# upstream SpannerChangeStreamsToPubSub.java (in our src/ we own the\n# whole pipeline; here we only need to express the *delta*).\ngit diff > ../tools/payload-link-dataflow/upstream-customization.patch\necho \"\" > ../tools/payload-link-dataflow/upstream.txt\n```\n\nAfter updating the SHA, run `./generate-full-delta.sh` and skim the\noutput to check that our impl's other divergences (sink swap, JSON\nshape, dropped options -- see \"Beyond the filter\" above) still make\nsense against upstream's new state. If upstream restructured\nsomething we deliberately dropped, update the \"Beyond the filter\"\nbullets accordingly.\n\nThis is purely a documentation refresh -- it does not affect the build.\n","tools/payload-link-dataflow/payload-link-publisher-py/README.md":"# payload-link-publisher-py\n\nDev/E2E-only Python variant of the payload-link Dataflow publisher.\n\n**Prod uses the Java flex template one directory up.** This variant\nexists so the docker-compose e2e stack can run without a JVM, without\nDataflow, and without minute-scale iteration times.\n\n## Design summary\n\n- **Not Apache Beam.** Python's Beam SDK has no Spanner change-stream\n connector; a Beam pipeline in Python would either require a\n cross-language Java expansion (JVM back in) or a custom Source\n (more code than the whole non-Beam approach). Instead we poll the\n `READ_payload_link_changes` TVF directly via\n `google.cloud.spanner.Client`.\n- **Wire-format identical to the Java job.** `serialize_record`\n produces the exact JSON shape `PayloadLinkChangesToPubSub.serializeRecord`\n produces, so the reconciler consumes either interchangeably.\n- **Partition tracking.** Follows partition splits: reads `_root`,\n picks up child-partition tokens from any `child_partitions_record`,\n reads each child from its advertised `start_timestamp`, and retires\n a parent when a child announces it. Handles `OUT_OF_RANGE` by\n dropping ended partitions. Required in practice — the Spanner\n emulator routes DataChangeRecords to child partitions almost\n immediately, so a `_root`-only reader sees zero DCRs.\n- **In-memory cursor.** On restart, resumes from `now - POLL_OVERLAP_SECONDS`\n on `_root`. Fine for tests (compose brings the pod up alongside the\n stream); would lose events across restarts in prod.\n- **Hardcoded row-shape offsets.** Rather than introspect the\n `READ_` TVF's ChangeRecord / DataChangeRecord /\n ChildPartitionsRecord STRUCT layout at runtime, `publisher.py`\n documents the field positions and accesses them positionally. If a\n future Spanner rev reorders the STRUCT, tests fail loudly and one\n offset map gets updated. Acceptable trade-off for a dev/E2E tool.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-spanner`, `google-cloud-pubsub`, `statsd`. |\n| `publisher.py` | Entry point: poll TVF → filter → serialize → publish. |\n| `utils.py` | `extract_payload_link`, `is_payload_link_actionable` — mirrors the Java filter. |\n| `test_publisher.py` | Unit tests for the pure functions. |\n| `Dockerfile` | `python:3.13-slim` + poetry install. |\n\n## Environment\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `SPANNER_PROJECT_ID` | yes | — | |\n| `SPANNER_INSTANCE_ID` | yes | — | |\n| `SPANNER_DATABASE_ID` | yes | — | |\n| `PUBSUB_PROJECT_ID` | yes | — | |\n| `PUBSUB_TOPIC` | yes | — | e.g. `payload-link-changes` |\n| `SPANNER_CHANGE_STREAM_NAME` | no | `payload_link_changes` | |\n| `HEARTBEAT_MS` | no | `1000` | TVF heartbeat interval. |\n| `POLL_OVERLAP_SECONDS` | no | `5` | Restart re-reads this window. |\n| `SPANNER_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-spanner`. |\n| `PUBSUB_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-pubsub`. |\n| `STATSD_HOST` / `STATSD_PORT` | no | — | Standard `statsd.defaults.env` pair. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-link-dataflow/payload-link-publisher-py\npoetry install\npoetry run pytest\npoetry run mypy .\npoetry run ruff check .\n```\n\n`poetry.lock` is generated on first install (not yet checked in).\n\nUnit tests cover `extract_payload_link`, `is_payload_link_actionable`,\n`serialize_record`, and `_to_json_string`. Not covered by unit tests:\n`poll_partition` (needs a live Spanner emulator) and\n`publish_if_actionable` (needs a Pub/Sub emulator). Both are\nexercised end-to-end by the docker-compose reconciliation stack\n(`make docker_run_reconciliation_e2e_tests`).\n\n## Metrics\n\n- `payload_link_publisher.published` — records published to Pub/Sub\n- `payload_link_publisher.filtered` — inert records dropped by\n `is_payload_link_actionable`\n- `payload_link_publisher.errors.publish` — publish failures\n\n## Related\n\n- Java flex template (prod): `tools/payload-link-dataflow/` (parent dir)\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (`payload_link_changes`)\n- Downstream reconciler: `tools/payload-reconciler/`\n- Runbook: `docs/src/tools/payload_link_reconciler.md`\n","tools/payload-reconciler/README.md":"# payload-reconciler\n\nPub/Sub-driven reconciler that closes the loop on GCS payload offload.\n\nReads the `payload-link-changes` Pub/Sub topic (populated by the\n`tools/payload-link-dataflow/` flex template). For each mod inside each\nchange record:\n\n- **New `payload_link` non-null** → patch the referenced GCS object's\n metadata to `committed=true` and `customTime=MAX`. Completes the\n 2-phase commit started in `syncserver/src/web/payload_offload.rs`.\n- **Old `payload_link` non-null and ≠ new** → delete the referenced GCS\n object. It is unreferenced now (the row's `payload_link` moved, was\n cleared, or the row was deleted — including TTL row-deletion-policy\n deletes).\n\nBoth operations are idempotent (re-setting metadata is a no-op; 404 on\ndelete is treated as success), so Pub/Sub at-least-once redelivery is\nsafe and no ordering key is needed.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-pubsub`, `google-cloud-storage`, `datadog`. |\n| `reconcile_payload_links.py` | Entry point: sync-pull drain loop + per-message handler. |\n| `utils.py` | `parse_gs_url()` — mirrors the Rust uploader's URL grammar. |\n| `test_utils.py` | Tests for `parse_gs_url`. |\n| `test_reconcile_payload_links.py` | Handler tests with a mocked GCS client. |\n\n## Environment\n\n| Variable | Required | Notes |\n|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | e.g. `payload-link-reconciler-sub`. |\n| `GCS_PAYLOAD_BUCKET` | yes | The bucket the syncserver writes payloads into. Cross-bucket links abort the message and route it to the DLQ. |\n| `RUN_BUDGET_SECONDS` | no | Selects deployment mode. **Set** (e.g. `240`): cronjob mode -- drain up to that many seconds then exit 0. **Unset**: long-running mode -- poll forever, never exit on idle; the deployment supervises restarts. Cronjob mode is the default deployment. |\n| `SYNC_STATSD_HOST` | no | statsd server host. |\n| `SYNC_STATSD_PORT` | no | statsd server port. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-reconciler\npoetry install\npoetry run pytest\n```\n\nUnit tests cover `parse_gs_url` and the per-message handler. The\n`drain` loop (sync-pull, budget vs. long-running exit conditions) has\nno unit coverage — exercise it against the Pub/Sub emulator when\nchanging it.\n\n## Metrics\n\nStatsd counters (all under the `payload_reconciler.` prefix):\n\n- `finalizes` — successful `committed=true` + `customTime=MAX` patch\n- `orphan_deletes` — successful delete of an unreferenced object\n- `noop_skips` — message processed with zero ops. Should be ~0 if the\n Dataflow filter is working; non-zero is the alarm bell that the\n filter regressed.\n- `gcs_404` + `op:finalize` / `op:delete` — 404 on the respective op.\n Expected non-zero under at-least-once redelivery (idempotency tax);\n a sharp rise suggests upstream state is gone unexpectedly.\n- `errors` + `kind:handler` — Sustained non-zero values feed the DLQ.\n\n## Deployment\n\n**The reconciler ships in the primary syncserver image**, built from\nthe root `Dockerfile`. Its script and Python deps are baked in via\nthat Dockerfile's per-tool poetry-export / pip-wheel / pip-install\nchain; K8s picks between the syncserver binary and the reconciler\nscript via the pod's `command:`.\n\nDefault: K8s cronjob (definition lives in webservices-infra), cadence\n~5 min, exports `RUN_BUDGET_SECONDS` (e.g. `240`).\n\nAlternative: long-running K8s Deployment. Omit `RUN_BUDGET_SECONDS`;\nthe pod polls forever. Choose this when finalize-flip latency below\nthe cronjob cadence matters (e.g. an operator wants offloaded\npayloads to become `committed=true` within seconds of upload rather\nthan up to one cronjob cycle later).\n\nService account needs the same roles in either mode:\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and `delete` operations)\n\n## Related\n\n- Producer: `syncserver/src/web/payload_offload.rs`\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (look for `payload_link_changes`)\n- Upstream pipeline: `tools/payload-link-dataflow/`\n","tools/postgres/README.md":"# PostgreSQL Utility Scripts\n\nCurrently there is one script: `purge_ttl.py`.\n\n## purge_ttl.py\n\nPurge expired items (BSOs and batches) from a Sync storage PostgreSQL instance.\nThe script is intended to be invoked regularly, e.g. as a cronjob.\n\n### Prerequisites\n\n- Python 3\n- [Poetry](https://python-poetry.org/docs/#installation)\n\n### Run Script Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run python purge_ttl.py\"\n```\n\nPass `-h` to see the list of options.\n\n### Run Tests Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run pytest test_purge_ttl.py -v\"\n```\n\n### Run with Docker\n\nA Docker image with the PostgreSQL build is not published anywhere at the\nmoment. Once that's available, this readme will be updated to include that\nstep.\n\n```bash\ndocker run --rm \\\n --entrypoint python3 \\\n -e SYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@host:5432/syncstorage\" \\\n syncstorage-postgres-image:tag \\\n /app/tools/postgres/purge_ttl.py\n```\n\n","tools/spanner/README.md":"# Spanner Tools and Scripts\n\nThese tools are supplemental scripts for working with the Google Cloud Platform. Follow [the general installation instructions](https://cloud.google.com/spanner/docs/getting-started/python/), as well as fetch the proper service account credentials file.\n\nRemember, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable should point to the absolute path location of your service account credential file.\n\ne.g.\n```bash\nGOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/project-id-service-cred.json venv/bin/python count_expired_rows.py\n```\nSee each script for details about function and use.\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs spanner untilities, you'll need a Python =>3.12 development environment with `Poetry` installed. These scripts typically run in a deployed GCP workflow, so their Dockerfile config will generally prepare all of this for you. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `spanner/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/spanner\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 spanner # or whatever project name you like.\n $ pyenv local spanner # activates virtual env whenever you enter this directory.\n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python count_expired_rows.py`\n","tools/syncstorage-loadtest/README.md":"# Storage load test\n\n**Note:** This requires Python version 3.10+\n\nThe tests can execute in three modes:\n\n1. **Direct access**: use a known master secret\n2. **Firefox accounts OAuth**: create test accounts and use OAuth JWTs\n3. **Self-signed JWT**: generate and sign JWTs locally with a given private key\n\n## Installation\n\n### Environment Setup\n\nTo run the syncstorage load tests, you'll need a Python >=3.10 development environment with `Poetry` installed. You can also directly use `poetry run` to execute commands as described in the usage examples below.\n\nThe easiest solution is to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments as a way to isolate the dependencies from other projects.\n\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\n See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Example:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n\n4. Install Python version, create virtualenv, activate and install dependencies from inside the project directory.\n\n **Note:** You can skip creating a virtual environment and invoke commands directly using `poetry run`.\n\n ```shell\n $ cd syncstorage-loadtest\n\n # Install Python 3.10+\n $ pyenv install 3.10\n\n # Create named, associated virtualenv\n $ pyenv virtualenv 3.10 syncstorage-loadtest # or whatever name you prefer\n $ pyenv local syncstorage-loadtest # activates virtual env whenever you enter this directory\n\n # Install Poetry and dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. Once you're in your virtual environment, run the load tests using:\n ```bash\n poetry run molotov [options] loadtest.py\n ```\n\n### Quick Install\n\nIf you already have Poetry installed:\n\n```bash\npoetry install\n```\n### Generate Key\n Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```bash\npoetry run ./generate-keys.sh\n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```bash\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n```\n## Mode 1: Direct Access\n\nWith a known syncstorage master secret:\n\n```bash\nSERVER_URL=\"http://localhost:8000#secretValue\" poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n## Mode 2: Firefox Accounts OAuth\n\nWith FxA stage accounts:\n\n```bash\nSERVER_URL=\"https://token.stage.mozaws.net\" \\\n FXA_API_HOST=\"https://api-accounts.stage.mozaws.net\" \\\n FXA_OAUTH_HOST=\"https://oauth.stage.mozaws.net\" \\\n poetry run molotov --workers 3 --duration 60 -v loadtest.py\n```\n\n## Mode 3: Self-Signed JWTs\n\nAssuming an RSA keypair was generated, the private key pem was saved and accessible, and the token server was started with the public key JWK configured. The presence of `OAUTH_PRIVATE_KEY_FILE` triggers this mode when using OAuth.\n\nRun with self-signed JWTs:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n## Expanded payloads (GCS offload)\n\nThe batch write path can generate larger-than-default payloads so you can load\ntest collections whose payloads are offloaded to Google Cloud Storage. These\nknobs are opt-in via environment variables; unset, the load test behaves\nexactly as before.\n\n| Variable | Default | Effect |\n|---|---|---|\n| `LARGE_PAYLOAD_PROB` | `0.0` | Fraction (`0.0`–`1.0`) of BSOs given an expanded payload. `1.0` makes **every** payload large; `0.0` disables the feature. |\n| `LARGE_PAYLOAD_SIZE` | *(unset)* | Explicit target size in bytes for large payloads. When unset, large payloads target the server's `max_record_payload_bytes`. |\n| `OFFLOAD_COLLECTIONS` | *(unset)* | Comma-separated collection names to target for batch writes. Must match the server's `gcs_payload_offload_collections`. When unset, the default collections are used. |\n\n`LARGE_PAYLOAD_PROB`/`LARGE_PAYLOAD_SIZE` control payload **size**; whether a\nwrite is offloaded to GCS is a separate, per-collection server setting. To drive\na fully offloaded collection at maximum size, set `OFFLOAD_COLLECTIONS` to that\ncollection and `LARGE_PAYLOAD_PROB=1.0`.\n\nIndividual payloads are capped at the target collection's\n`max_record_payload_bytes`, and each batch is kept under its `max_post_bytes`,\nso a large enough payload results in fewer records per request.\n\nExample — every write is a 5 MiB payload sent to the `tabs` collection:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n LARGE_PAYLOAD_PROB=1.0 \\\n LARGE_PAYLOAD_SIZE=5242880 \\\n OFFLOAD_COLLECTIONS=tabs \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n### Server-side prerequisites\n\nTo actually exercise expanded/offloaded payloads end to end, the target server\nmust be configured for it:\n\n- `gcs_payload_bucket` set and the target collection listed in\n `gcs_payload_offload_collections` (syncstorage settings).\n- Raised limits to allow payloads beyond the 2.5 MiB default:\n `max_record_payload_bytes`, `max_post_bytes`, `max_request_bytes`, and\n `max_total_bytes`. These are read from `/info/configuration`. A collection\n may also raise its own limits via the `collections` section of that\n response; the load test resolves limits per target collection, preferring a\n collection's override and falling back to the global value.\n- Any front-end proxy (e.g. nginx) request body-size limit raised to match.\n\nNote: many workers each holding multi-MiB payloads is a real memory footprint\non the load-generating host — size the harness accordingly.\n\n## Docker\n\nBuild and run this directory's image:\n\n```bash\ndocker build . --tag syncstorage-loadtest:local\n\ndocker run --rm --net=host \\\n -e SERVER_URL=\"http://localhost:8000#secretValue\" \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 1 --workers 5 --duration 60 -v loadtest.py\"\n```\n\n## Full offload pipeline on docker\n\nFor the offload and change-stream load tests (STOR-628, STOR-629) there is a\ncompose rig that stands up syncserver with offload enabled, a Spanner emulator\ncarrying the change stream, Pub/Sub, GCS, the publisher, the reconciler and\nthis load test, then reports whether the pipeline drained clean:\n\n```bash\nmake loadtest-offload-image\nmake loadtest-offload-handler # 100% offloaded\nmake loadtest-offload-report\nmake loadtest-offload-down\n```\n\nSee `docs/src/tools/load-testing.md` for the runbook, the knobs, and the GCP\ndev path.\n\nHappy Breaking!\n","tools/tokenserver/README.md":"# Summary of Files in tools/Tokenserver\n\n| File Name | Description |\n|--------------------------------|-----------------------------------------------------------------------------|\n| `add_node.py` | Adds new node to the tokenserver's database, registering it for user allocation. |\n| `allocate_user.py` | Script to allocate a specific user to a node. |\n| `conftest.py` | Pytest configuration and fixtures for shared test setup. |\n| `count_users.py` | Script to emit total-user-count metrics for exec dashboard. |\n| `database.py` | Shared database utility queries and functions used by multiple scripts. |\n| `purge_old_records.py` | Script to purge user records that have been replaced. |\n| `pytest.ini` | Configuration file for pytest, specifying options like test output format. |\n| `remove_node.py` | Script to remove a node from the system. |\n| `test_database.py` | Unit tests for `database.py`. |\n| `test_purge_old_records.py` | Tests for `purge_old_records.py`, validating cleanup operations. |\n| `test_scripts.py` | Testing module to test the various scripts in this directory. |\n| `unassign_node.py` | Removes a node from the system and clears any assignments to the named node.| \n| `update_node.py` | Script to update node status in the db. |","tools/tokenserver/loadtests/README.md":"# Tokenserver Load Tests\n\nThis directory contains everything needed to run the suite of load tests for Tokenserver.\n\n## Prerequisite Dependency and Environment Setup:\nTo use the syncstorage-rs tokenserver load tests, you'll need a Python =>3.12 development environment with `Poetry` installed. This testing script can be run within a spun up GCP workflow where you can execute the locust commands from the UI. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `loadtests/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/tokenserver/loadtests\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 loadtests # or whatever project name you like.\n $ pyenv local loadtests # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python locustfile.py`\n\n\n## Building and Running\n\n1. Install the load testing dependencies as described above:\n\n ```sh\n poetry install\n ```\n\n2. Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```sh\npoetry run ./generate-keys.sh \n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```sh\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n ```\n\n3. Set the following environment variables/settings on Tokenserver:\n\n ```sh\n # Should be set to the \"n\" component of the JWK\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N\n # Should be set to the \"e\" component of the JWK (this value should almost always be \"AQAB\")\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY=RSA\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__USE=sig\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__ALG=RS256\n\n # These two environment variables don't affect the load tests, but they need to be set:\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID=\"\"\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__FXA_CREATED_AT=0\n ```\n\n Note that, because these settings cache the JWK used to verify OAuth tokens, no verification requests will be made to FxA, so the value of `SYNC_TOKENSERVER__FXA_OAUTH_VERIFIER_URL` does not matter; however, Tokenserver expects it to be set, so setting it to something like `http://localhost` will suffice.\n\n4. Set the following environment variables on the machine that will be running the load tests:\n\n - `OAUTH_PEM_FILE` should be set to the location of the private RSA key generated in a previous step\n\n5. Tokenserver uses [locust](https://locust.io/) for load testing. To run the load tests, simply run the following command in this directory:\n\n ```sh\n locust\n ```\n\n6. Navigate your browser to , where you'll find the locust GUI. Enter the following information:\n\n - Number of users: The peak number of Tokenserver users to be used during the load tests\n - Spawn rate: The rate at which new users are spawned\n - Host: The URL of the server to be load tested. Note that this URL must include the protocol (e.g. \"http://\")\n\n7. Click the \"Start swarming\" button to begin the load tests.\n\n## Populating the Database\n\nThis directory includes an optional `populate_db.py` script that can be used to add test users to the database en masse. The script can be run like so:\n\n```sh\npoetry run python populate_db.py \n```\n\nOr if in built virtualenv:\n```sh\npython3 populate_db.py \n```\n\nwhere `sqluri` is the URL of the Tokenserver database, `nodes` is a comma-separated list of nodes **that are already present in the database** to which the users will be randomly assigned, and `number of users` is the number of users to be created.\n"},"files":{".claude/commands/env-check.md":"Verify all required environment variables are set before running integration tests.\n\nYou are a pre-flight assistant for syncstorage-rs integration tests. Your job is to check that the shell environment is correctly configured so tests will actually run, and surface any missing or suspicious values before the user wastes time on a failing test run.\n\nDo NOT print or log the values of any secrets. Use placeholders like `[set]` or `[not set]` when reporting on secret variables.\n\n## Variables to check\n\n### Required for all integration test runs\n\n| Variable | Expected | Notes |\n|---|---|---|\n| `SYNC_MASTER_SECRET` | any non-empty string | secret — report [set] or [not set] only |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | starts with `mysql://`, `postgresql://`, or `spanner://` | |\n| `SYNC_TOKENSERVER__DATABASE_URL` | starts with `mysql://` or `postgresql://` | |\n| `TOKENSERVER_HOST` | starts with `http://` or `https://` | default: `http://localhost:8000` |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | starts with `http://` or `https://` | default: `http://localhost:6000` for local |\n| `PYTHONPATH` | contains `tools/` path | required for PyO3 and test imports |\n\n### Required only for real FxA e2e tests (`test_e2e.py`)\n\n| Variable | Notes |\n|---|---|\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E` | secret — report [set] or [not set] only |\n\n## Step 1 — Check each variable\n\n```bash\necho \"SYNC_MASTER_SECRET: $([ -n \"$SYNC_MASTER_SECRET\" ] && echo '[set]' || echo '[NOT SET]')\"\necho \"SYNC_SYNCSTORAGE__DATABASE_URL: ${SYNC_SYNCSTORAGE__DATABASE_URL:-[NOT SET]}\"\necho \"SYNC_TOKENSERVER__DATABASE_URL: ${SYNC_TOKENSERVER__DATABASE_URL:-[NOT SET]}\"\necho \"TOKENSERVER_HOST: ${TOKENSERVER_HOST:-[NOT SET - default: http://localhost:8000]}\"\necho \"SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: ${SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL:-[NOT SET - default: http://localhost:6000]}\"\necho \"PYTHONPATH: ${PYTHONPATH:-[NOT SET]}\"\n```\n\n## Step 2 — Validate database URL format\n\nFor `SYNC_SYNCSTORAGE__DATABASE_URL` and `SYNC_TOKENSERVER__DATABASE_URL`:\n- Check the scheme matches a supported backend (`mysql`, `postgresql`, `spanner`)\n- For `postgres://` prefix: note that SQLAlchemy 1.4+ requires `postgresql://` — flag this if present\n- For Spanner: confirm `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` is set if the URL contains `test-project` or `emulator`\n\n## Step 3 — Check PYTHONPATH contains tools/\n\n```bash\npython3 -c \"import tools.integration_tests\" 2>&1\n```\n\nIf this fails, the `PYTHONPATH` is not set correctly for test imports.\n\n## Step 4 — Check server reachability (if TOKENSERVER_HOST is set)\n\n```bash\ncurl -sf \"${TOKENSERVER_HOST:-http://localhost:8000}/__heartbeat__\" && echo \"server: reachable\" || echo \"server: NOT REACHABLE\"\n```\n\n## Output format\n\nReport each variable as **OK**, **WARNING**, or **MISSING**:\n\n| Variable | Status | Detail |\n|---|---|---|\n\nThen list any blockers (MISSING required vars, unreachable server) followed by warnings (format issues, defaults in use).\n\nEnd with **Ready to run tests** or **N issues need to be resolved**.\n",".claude/commands/git-history-check.md":"Examine git history of files changed in the current branch to identify regressions, re-introduced bugs, or changes that conflict with past fixes.\n\nYou are a senior engineer doing a history-aware code review. Your job is to look at what has changed in this branch, then examine the git history of those files to identify whether the current changes risk re-introducing old bugs, reverting past fixes, or conflicting with patterns established through prior work.\n\n## Step 1 — Get the current diff\n\n```bash\ngit diff main...HEAD\n```\n\n## Step 2 — Get the list of changed files\n\n```bash\ngit diff main...HEAD --name-only\n```\n\n## Step 3 — For each changed file, examine recent commit history\n\n```bash\ngit log --oneline -20 -- \n```\n\n## Step 4 — Inspect relevant commits in full\n\nFor each commit that looks relevant (bug fixes, reverts, security patches, refactors on the same lines), inspect the full diff:\n\n```bash\ngit show \n```\n\n## Step 5 — Look for reverted or fix commits\n\n```bash\ngit log --oneline --all --grep=\"revert\" -- \ngit log --oneline --all --grep=\"fix\" -- \n```\n\nUse your judgment about which historical commits are worth digging into. Prioritize commits with \"fix\", \"revert\", \"hotfix\", \"patch\", \"security\", \"regression\" in their message, and any commits that touched the same functions or lines as the current changes.\n\n## Analysis checklist\n\nFor each changed file, work through the following. Report findings with:\n- **Severity:** High / Medium / Low\n- **Location:** file:line\n- **Issue:** what the historical context reveals\n- **Relevant commit(s):** hash + message\n- **Recommendation:** what to verify or change\n\n### 1. Re-introduced bugs\n- Does the current diff restore code that was previously removed by a bug fix?\n- Are there commits in the history that explicitly fixed logic that the current change modifies or removes?\n\n### 2. Reverted or rolled-back patterns\n- Has this code been reverted before? If so, why — and does the current change repeat the same pattern?\n- Are there \"Revert X\" commits in the history that suggest a prior attempt at this change failed?\n\n### 3. Previously fixed security issues\n- Does the history show security-related fixes on these lines?\n- Does the current change touch or weaken those protections?\n- Pay particular attention to: HAWK/JWT token handling, FxA OAuth validation, gRPC error suppression logic, DB migration constraints.\n\n### 4. Repeated churn\n- Has this file or function been modified many times in a short period? High churn is a signal of instability.\n- Does the current change look like it continues a pattern of patching symptoms rather than fixing the root cause?\n\n### 5. Conflicting intent\n- Do commit messages indicate a deliberate design decision that the current change reverses without explanation?\n- Are there TODOs or FIXMEs introduced by prior commits that the current change should have addressed but didn't?\n\n### 6. Migration or deprecation conflicts\n- Does the history show a migration away from a pattern that the current change re-introduces?\n- Examples: callback → async, legacy auth → FxA OAuth, MySQL-only → multi-backend.\n\n### 7. Test regressions\n- Were tests previously added to cover a bug that the current change might invalidate?\n- Do any historical fix commits include test additions that are now at risk of being bypassed?\n\n## Output format\n\nLead with a summary table:\n\n| Severity | File | Issue | Relevant commit |\n|---|---|---|---|\n\nFollow with per-file detail. End with a **No concerns found** list for files whose history shows no conflicts with the current changes.\n\nIf the git history is shallow or sparse for a file, note that and flag it as low confidence.\n",".claude/commands/migration-review.md":"Review database migration changes against the repo's published-migration policy.\n\nYou are a database migration reviewer for syncstorage-rs. The repo has a hard rule: **never edit a published migration file**. Only new forward migrations with a corresponding rollback are allowed. Your job is to enforce this and catch violations before merge.\n\n## Migration locations\n\n```\nsyncstorage-mysql/migrations/ # up.sql / down.sql per timestamped dir\nsyncstorage-postgres/migrations/\ntokenserver-mysql/migrations/\ntokenserver-postgres/migrations/\nsyncstorage-spanner/src/schema.ddl # Spanner schema — treated as published\nsyncstorage-spanner/src/*.sql # inline Spanner queries\n```\n\nEach migration directory is named `YYYY-MM-DD-hhmmss_description/` and contains `up.sql` and `down.sql`.\n\n## Step 1 — Find changed migration files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"migrations/|schema\\.ddl|\\.sql\"\n```\n\n## Step 2 — Classify each changed file\n\nFor each changed SQL file:\n\n- **New migration directory** (path did not exist on main) → allowed, proceed to Step 3\n- **Existing migration file modified** (path existed on main and content changed) → **VIOLATION** — flag as a blocker\n- **schema.ddl modified** → **VIOLATION** — flag as a blocker\n- **Inline .sql query file modified** (non-migration) → review for intent, flag if it changes schema\n\nCheck whether a file existed on main:\n```bash\ngit show main:\n```\n\n## Step 3 — Validate new migrations\n\nFor each new migration directory:\n\n1. Confirm both `up.sql` and `down.sql` exist\n2. Confirm `down.sql` is a genuine rollback of `up.sql` (not empty, not a no-op)\n3. Check that the timestamp prefix is newer than all existing migrations in that crate\n4. Check for destructive operations in `up.sql` (`DROP TABLE`, `DROP COLUMN`, `TRUNCATE`) — flag these for explicit review\n5. Check that column types and constraints are consistent with existing schema in the same crate\n\n## Step 4 — Cross-backend consistency\n\nIf migrations exist for both mysql and postgres variants of the same crate, verify the schema changes are semantically equivalent (same columns, types mapped appropriately, same constraints).\n\n## Output format\n\n**Violations (blockers):**\nList any edits to published files with file path and the nature of the change.\n\n**New migrations:**\nFor each new migration: name, what it does, whether down.sql is valid, any destructive ops.\n\n**Warnings:**\nNon-blocking concerns (e.g. missing index, risky destructive op, cross-backend inconsistency).\n\n**Verdict:** Pass / Violations found\n",".claude/commands/pr-description.md":"Generate a PR title and summary from the current branch diff, following the repo's commit and PR conventions.\n\nYou are a PR description writer for syncstorage-rs. Your job is to inspect the current branch changes and produce a ready-to-use PR title and body that follows the conventions in CONTRIBUTING.md.\n\n## Step 1 — Gather branch context\n\n```bash\ngit diff main...HEAD --name-only\ngit log main...HEAD --oneline\ngit diff main...HEAD --stat\n```\n\n## Step 2 — Read the full diff for context\n\n```bash\ngit diff main...HEAD\n```\n\nFocus on understanding *what changed and why*, not just listing files.\n\n## Step 3 — Classify the change type\n\nBased on changed files, determine the primary `type`:\n\n- `feat` — new functionality added\n- `fix` — bug fix\n- `refactor` — code restructured, no behavior change\n- `test` — test additions or changes only\n- `chore` — build, CI, tooling, dependencies\n- `docs` — documentation only\n- `perf` — performance improvement\n- `style` — formatting, lint fixes only\n\nIf multiple types apply, use the most significant one.\n\n## Step 4 — Determine scope (optional)\n\nIf the change is clearly scoped to one subsystem, add it in parentheses:\n- `tokenserver`, `syncstorage`, `spanner`, `mysql`, `postgres`, `auth`, `migrations`, `ci`, `tools`\n\nExample: `refactor(tokenserver): ...`\n\n## Step 5 — Draft the PR title\n\nRules from CONTRIBUTING.md:\n- Format: `type(scope): subject` or `type: subject`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Keep it under 70 characters\n\n## Step 6 — Draft the PR body\n\n```markdown\n## Summary\n\n<2–4 bullet points covering what changed and why. Lead with the motivating\nproblem or context, then the solution. Skip obvious restatements of the title.>\n\n## Test plan\n\n\n```\n\nPre-fill the test plan based on file types changed:\n\n- Rust source changed → `- [ ] \\`make clippy_\\` passes`, `- [ ] Unit tests pass (\\`make test\\`)`\n- Python source changed → `- [ ] \\`make ruff-lint\\` and \\`make mypy\\` pass`, `- [ ] Integration tests pass (\\`make run_local_e2e_tests\\`)`\n- SQL migrations added → `- [ ] \\`/migration-review\\` passes`, `- [ ] Migration applies cleanly on a fresh DB`\n- `.github/` changed → `- [ ] CI workflow runs successfully on this branch`\n- `CLAUDE.md` or docs changed → `- [ ] Renders correctly on GitHub`\n\n## Step 7 — Output\n\nPrint the final PR title and body, ready to paste into GitHub. Also include the suggested branch name if the current branch name doesn't follow `type/description-STOR-####` or `type/description-####` format.\n\nNote any breaking changes that should be called out explicitly.\n",".claude/commands/py-checks.md":"Run all Python lint, format, type, and security checks for the tools/ directory.\n\nYou are a Python code quality assistant for the syncstorage-rs tools and integration tests. Your job is to run the full suite of Python checks in the correct order, report findings clearly, and tell the user exactly what needs to be fixed before the PR can merge.\n\nAll checks operate on the `tools/` directory. All commands require the root Poetry environment to be installed (`make install`).\n\n## Step 1 — Check Poetry environment\n\n```bash\npoetry env info\n```\n\nIf no environment is active, tell the user to run `make install` first and stop.\n\n## Step 2 — Format check\n\n```bash\nmake ruff-fmt-chk\n```\n\nIf there are formatting issues, tell the user to run `make ruff-format` to auto-fix, then re-run. Do not proceed past format failures.\n\n## Step 3 — Lint\n\n```bash\nmake ruff-lint\n```\n\nReport each warning/error with file:line and rule code.\n\n## Step 4 — Type checking\n\n```bash\nmake mypy\n```\n\nReport type errors with file:line. Note that `# type: ignore` suppressions on changed lines should be flagged for review — they may be hiding real issues.\n\n## Step 5 — Docstring validation\n\n```bash\nmake pydocstyle\n```\n\nReport missing or malformed docstrings. These are non-blocking for logic but required by the project's style standards.\n\n## Step 6 — Security scan\n\n```bash\nmake bandit\n```\n\nFlag any **HIGH** severity findings as blockers. List **MEDIUM** as warnings.\n\n## Output format\n\nReport each check as **PASS**, **WARNINGS**, or **FAIL**:\n\n| Check | Status | Issues |\n|---|---|---|\n| Format (ruff) | | |\n| Lint (ruff) | | |\n| Types (mypy) | | |\n| Docstrings (pydocstyle) | | |\n| Security (bandit) | | |\n\nList all actionable findings below the table with file:line references.\n\nEnd with **All checks passed** or **N checks need attention** verdict.\n",".claude/commands/rust-backend-check.md":"Run Clippy, fmt check, and cargo audit for a specific Rust backend.\n\nYou are a Rust build and lint assistant for the syncstorage-rs workspace. Your job is to run the correct set of checks for a given backend, report findings clearly, and flag anything that needs attention before the code is merged.\n\n## Step 1 — Determine the backend\n\nAsk the user which backend to check if not already specified: `mysql`, `postgres`, or `spanner`. If the current branch name or open files suggest a backend, infer it and confirm.\n\n## Step 2 — Run format check\n\n```bash\ncargo fmt -- --check\n```\n\nReport any files that are not formatted. If there are formatting issues, stop and tell the user to run `cargo fmt` before proceeding.\n\n## Step 3 — Run Clippy for the chosen backend\n\nUse the exact make target:\n\n```bash\nmake clippy_mysql # for mysql\nmake clippy_postgres # for postgres\nmake clippy_spanner # for spanner\n```\n\n## Step 4 — Run release-mode Clippy\n\n```bash\nmake clippy_release_mysql # for mysql\nmake clippy_release_postgres # for postgres\nmake clippy_release_spanner # for spanner\n```\n\nRelease mode catches dead code and issues only visible with optimizations enabled.\n\n## Step 5 — Run cargo audit\n\n```bash\ncargo audit\n```\n\nFlag any advisories with severity **high** or **critical** as blockers. List medium/low advisories as warnings.\n\n## Output format\n\nSummarize findings in this order:\n\n1. **Format** — pass or list of unformatted files\n2. **Clippy (debug)** — pass or list of warnings/errors with file:line\n3. **Clippy (release)** — pass or list of warnings/errors with file:line\n4. **Audit** — pass or list of advisories (severity, crate, advisory ID)\n\nEnd with a clear **Ready to merge** / **Needs attention** verdict.\n",".claude/skills/jira-status/SKILL.md":"---\nname: jira-status\ndescription: Queries, summarizes, creates, and edits Jira issues for syncstorage-rs (project STOR). Supports user-centric work views, epic breakdowns, structured summaries with impact/context/highlights, and ticket creation or modification from natural language.\nuser-invocable: true\nargument-hint: \"me | epic | epics | sprint | summary | create | update | status (default: status)\"\n---\n\nStart by reading `.claude/mcp-context.md` for project IDs, JQL patterns, issue types, and field conventions.\n\nThen call `mcp__atlassian__getAccessibleAtlassianResources` to confirm the Jira cloud ID and base URL for `mozilla-hub.atlassian.net`. Store the base URL as `JIRA_BASE` — all issue links are formatted as `JIRA_BASE/browse/STOR-####`.\n\nParse `$ARGUMENTS` (trimmed). Match against the modes below. Default is `status`.\n\n---\n\n## Mode: `status` (default)\n\nOverall project task landscape. Use when no argument is given.\n\nRun these JQL queries in parallel:\n\n```jql\n-- Open sprint issues\nproject = STOR AND sprint in openSprints() AND statusCategory != Done ORDER BY priority DESC, updated DESC\n\n-- Unassigned open issues\nproject = STOR AND statusCategory != Done AND assignee is EMPTY ORDER BY priority DESC\n\n-- Recently updated (last 3 days)\nproject = STOR AND updated >= -3d AND statusCategory != Done ORDER BY updated DESC\n```\n\n**Output:**\n\n### Project Status\nOne-line overall signal: `Active` / `Needs attention` / `Stalled`.\n\n### Current Sprint\nTable: Key (linked) | Summary | Status | Assignee | Priority\nGroup by status category (In Progress first, then To Do, then Done).\n\n### Unassigned Open Issues\nTable: Key (linked) | Summary | Priority | Issue Type\nFlag Blockers and Criticals explicitly.\n\n### Recently Updated\nTable: Key (linked) | Summary | Updated | By\n\n---\n\n## Mode: `me`\n\nWork assigned to the person running this skill. Ask: \"What is your Jira email or display name?\" — use their response to filter.\n\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory != Done ORDER BY updated DESC\n```\n\nAlso fetch recently completed by the user (for summary context):\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory = Done AND resolutiondate >= -30d ORDER BY resolutiondate DESC\n```\n\n**Output:**\n\n### Your Active Work\nTable: Key (linked) | Summary | Status | Priority | Last Updated\nHighlight anything In Progress that hasn't been updated in > 3 days.\n\n### Your Recent Completions (Last 30 Days)\nTable: Key (linked) | Summary | Resolved On | Issue Type\n\n### Work Summary\nA plain-English paragraph for each active item: what it is, where it stands, and whether anything is blocking it. Lead with the highest-priority item.\n\n---\n\n## Mode: `epic `\n\nRetrieve a specific epic and all its child issues.\n\nSteps:\n1. Call `mcp__atlassian__getJiraIssue` with the provided KEY to get the epic details.\n2. Fetch all children:\n ```jql\n project = STOR AND parent = ORDER BY priority DESC, status ASC\n ```\n3. Also fetch linked issues (blocks/is blocked by) on the epic itself.\n\n**Output:**\n\n### Epic: [STOR-####](JIRA_BASE/browse/STOR-####) — {Epic Summary}\n**Status:** {status} | **Assignee:** {assignee or Unassigned} | **Priority:** {priority}\n\n**Description:**\n{epic description, condensed to key intent — 2–4 sentences max}\n\n### Child Issues\nTable: Key (linked) | Summary | Status | Assignee | Priority | Issue Type\nGroup: In Progress → To Do → Done (collapsed count for Done if > 5)\n\n### Epic Progress\n- Total issues: N\n- Done: N (N%)\n- In Progress: N\n- Blocked: N (flag if any)\n\n### Blockers & Dependencies\nList any linked \"blocks\" or \"is blocked by\" relationships. If none, omit this section.\n\n---\n\n## Mode: `epics`\n\nList all open epics in the STOR project with their child counts and progress.\n\n```jql\nproject = STOR AND issuetype = Epic AND statusCategory != Done ORDER BY priority DESC, updated DESC\n```\n\nFor each epic, fetch child counts via:\n```jql\nproject = STOR AND parent = \n```\n(batch these — fetch all children in one query with `parent in (KEY1, KEY2, ...)` if the tool supports it, otherwise query each epic individually)\n\n**Output:**\n\n### Open Epics\n\nFor each epic:\n**[STOR-####](link) — {Summary}**\nStatus: {status} | Assignee: {assignee} | Children: {done}/{total}\nOne-line description of the epic's goal (from its description field, condensed).\n\n---\n\n## Mode: `sprint`\n\nCurrent sprint state focused on progress and blockers — not health framing, just what is happening.\n\n```jql\nproject = STOR AND sprint in openSprints() ORDER BY statusCategory ASC, priority DESC\n```\n\n**Output:**\n\n### Sprint: {Sprint Name}\nDates: {start} → {end} | Days remaining: N\n\n**In Progress** (N issues)\nTable: Key (linked) | Summary | Assignee | Days In Progress\n\n**To Do** (N issues)\nTable: Key (linked) | Summary | Assignee | Priority\n\n**Done** (N issues)\nCount only unless ≤ 5 — then list.\n\n**Blockers in sprint:**\nList any issue linked as \"blocks\" another sprint issue, or with priority Blocker.\n\n---\n\n## Mode: `summary `\n\nGenerate a structured summary suitable for a status update, PR description, or stakeholder report. The argument can be:\n- A single issue key: `summary STOR-1234`\n- An epic key: `summary STOR-100` (auto-detects if it's an epic)\n- A freeform scope description: `summary tokenserver auth work` (runs JQL to find matching issues)\n- `summary me` — summarize the current user's recent work (ask for their name if not known)\n- `summary sprint` — summarize the current sprint's work\n\n**Steps:**\n1. Resolve the scope to a set of issues. Fetch full details for each.\n2. For epics: fetch children too.\n3. Identify: what was done, what is in progress, what is blocked.\n4. Synthesize the summary.\n\n**Output format — always structured as:**\n\n### Impact\nWhat this work changes for users or the system. Lead with the most significant outcome. Use concrete language: \"reduces\", \"enables\", \"removes\", \"fixes\". If there are multiple issues, consolidate into the top 2–3 outcomes. Link relevant issues inline: e.g. \"Adds PostgreSQL support for tokenserver ([STOR-42](link)).\"\n\n### Context\nWhy this work was needed. Background the reader should have to understand the impact. Reference the driving requirement, bug, or design decision. Keep to 3–5 sentences. Link to the primary issue or epic.\n\n### Highlights\nNotable implementation decisions, technical wins, or interesting constraints resolved. Bullet points. Each item links to the relevant issue. Focus on what would surprise or interest a fellow engineer — not just a list of what was done.\n\n---\n\n## Mode: `create `\n\nCreate a new Jira issue in STOR from natural language.\n\n**Steps:**\n\n1. Parse the description to infer:\n - **Summary** — one clear sentence (≤ 100 chars)\n - **Issue type** — Bug / Task / Story / Spike (infer from language; ask if ambiguous)\n - **Priority** — infer from language (\"critical\", \"blocking\" → Critical; \"nice to have\" → Minor; default → Major). Valid STOR priorities: `Blocker`, `Critical`, `Major`, `Minor`. `Trivial` is rejected by the project — do not use it.\n - **Work category** — `Feature Engineering` / `Engineering Excellence` / `Operational Excellence`. See [Work categories](#work-categories) below for definitions and inference rules. Always classify; if unsure between EE and OpEx, the planned/reactive distinction decides it.\n - **Assignee** — if this `create` invocation is part of the same workflow that opens a GitHub PR for the work, set the assignee to the current user. Look up their accountId via `mcp__atlassian__lookupJiraAccountId` using the email from the runtime `userEmail` context. Otherwise leave unassigned unless the user named a specific person.\n - **Parent** — if the description references an epic (e.g. \"under STOR-506\", \"in STOR-506\", \"linked to \"), set `parent` to that key when calling `createJiraIssue`.\n - **Description body** — expand the user's language into a structured description, leading with the category:\n - **Category:** {one of the three categories} — *one-sentence rationale*\n - **Background:** why this is needed\n - **Acceptance Criteria:** observable, testable outcomes\n - **Notes:** any constraints or references mentioned\n\n2. Show the draft to the user before creating:\n ```\n Ready to create:\n Type: {type}\n Project: STOR\n Parent: {parent key or none}\n Priority: {priority}\n Category: {category}\n Assignee: {display name or \"unassigned\"}\n Summary: {summary}\n Description:\n {description}\n\n Create this issue? (yes / edit first / cancel)\n ```\n\n3. On confirmation, call `mcp__atlassian__createJiraIssue` with:\n - `projectKey: STOR`\n - `issueTypeName: {type}`\n - `summary: {summary}`\n - `description: {description}` with `contentFormat: markdown`\n - `additional_fields: { \"priority\": { \"name\": \"{priority}\" } }`\n - `assignee_account_id: {accountId}` — only when auto-assignment applies or the user named an assignee\n - `parent: {epic_key}` — only when a parent epic was identified\n\n4. Return the created issue key and direct link: `[STOR-####](JIRA_BASE/browse/STOR-####)`\n\n---\n\n## Mode: `update `\n\nModify an existing issue. Changes can be natural language.\n\n**Steps:**\n\n1. Fetch the current issue with `mcp__atlassian__getJiraIssue`.\n\n2. Parse the `` to determine what to modify:\n - Field updates (summary, description, priority, assignee) → use `mcp__atlassian__editJiraIssue`\n - Status transition (\"mark as done\", \"move to in progress\") → use `mcp__atlassian__getTransitionsForJiraIssue` then `mcp__atlassian__transitionJiraIssue`\n - Comment (\"add a comment: ...\") → use `mcp__atlassian__addCommentToJiraIssue`\n\n3. Show the proposed changes before applying:\n ```\n Updating STOR-####: {current summary}\n Changes:\n - {field}: \"{old value}\" → \"{new value}\"\n - {transition}: {new status}\n - Comment: \"{comment text}\"\n\n Apply? (yes / edit / cancel)\n ```\n\n4. Apply on confirmation. Return confirmation with the issue link.\n\n---\n\n## General guidelines\n\n- **All issue references must be linked**: format as `[STOR-####](JIRA_BASE/browse/STOR-####)` — never bare key text\n- **Summaries lead with impact**: what changes for users or the system, not what was done mechanically\n- **Do not fabricate data**: if a query returns nothing, say so plainly\n- **Ask once for missing identity**: if the user's Jira name is needed and not known, ask once; do not repeat the question across modes in the same session\n- **Confirm before write operations**: always show a draft before calling `createJiraIssue`, `editJiraIssue`, `transitionJiraIssue`, or `addCommentToJiraIssue`\n- **Permissions**: create/edit/transition operations require the user's Jira account to have project contributor access to STOR; if a call fails with 403, report it clearly and do not retry\n- **Issue type inference**: \"bug\" → Bug; \"task\"/\"chore\" → Task; \"feature\"/\"story\"/\"as a user\" → Story; \"investigate\"/\"explore\"/\"spike\" → Spike\n- **Classify every ticket**: when creating or summarizing, identify the [work category](#work-categories) (Feature Engineering / Engineering Excellence / Operational Excellence). Include it as the leading line of the description body for created tickets.\n- **Auto-assign when PR-bundled**: if a `create` invocation is part of the same workflow that opens a GitHub PR for the same work, set the assignee to the current user (look up their accountId with `mcp__atlassian__lookupJiraAccountId`). Setting an assignee may transition the issue to In Progress via Jira workflow rules — that's expected.\n\n---\n\n## Work categories\n\nPer [Feature Engineering, Engineering Excellence, and Operational Excellence](https://mozilla-hub.atlassian.net/wiki/spaces/IP/pages/1397817345). Classify every STOR ticket into one of three categories. Target distribution per quarter: ≤ 50% Feature Engineering, ≤ 30% Engineering Excellence, ≤ 20% Operational Excellence.\n\n### Feature Engineering — *build & deliver new things*\n\nPlanned work that changes what a customer (internal or external) sees beyond speed, reliability, or bugfixes.\n\nExamples: new API endpoints, new auth methods (passkeys), launching a region or platform, UI/UX redesigns, A/B testing, expanded user-configuration options, changes to core business logic, bug fixes against unlaunched feature work.\n\n### Engineering Excellence — *sustain & improve velocity*\n\nPlanned work that improves code maintainability, reduces tech debt, or boosts engineering velocity. Most non-hotfix bug fixes land here. \"Keeping the lights on\" proactively.\n\nExamples: refactors, test-coverage improvements, observability and monitoring, CI/CD tooling, dependency upgrades, removing dead code, fixing flaky tests, reducing on-call toil, automating manual deployments, fixes for issues in already-released features.\n\n### Operational Excellence — *react & keep production healthy*\n\nUnplanned work. Reliability, stability, security, and operational efficiency in the production environment. \"Keeping the lights on\" reactively.\n\nExamples: security patches and CVE remediations, manual scaling under load, incident response and postmortems, hotfix patches for production incidents, resolving performance issues impacting SLAs.\n\n### Inference rules\n\n- **Planned vs. unplanned** decides EE vs. OpEx: if it was on a roadmap/sprint plan, it's EE; if it interrupted planned work to keep prod healthy, it's OpEx.\n- **Customer-visible change** decides FE vs. EE: if the customer sees something new (beyond speed/reliability/bugfix), it's FE.\n- **Dependency upgrades, CI/CD tooling, refactors, test improvements** → Engineering Excellence (unless the upgrade is a CVE remediation, then OpEx).\n- **Security patches and CVEs** → Operational Excellence.\n- **Bug fixes**: pre-launch → FE; against released features, planned → EE; via hotfix → OpEx.\n",".claude/skills/multi-backend-consistency/SKILL.md":"---\nname: multi-backend-consistency\ndescription: Checks that trait method changes in syncstorage-db or tokenserver-db are consistently implemented across all backend crates (mysql, postgres, spanner). Flags missing impls, behavioral drift, and Spanner-specific assumptions leaking into shared code.\nuser-invocable: true\n---\n\n# Multi-Backend Consistency Check\n\nYou are a Rust trait/implementation consistency reviewer for syncstorage-rs. Changes to shared traits must be reflected in every backend. Your job is to find gaps before they reach CI.\n\n## Backend map\n\n```\nsyncstorage-db/ ← trait definitions (DbPool, Db, etc.)\n → syncstorage-mysql/\n → syncstorage-postgres/\n → syncstorage-spanner/\n\ntokenserver-db/ ← trait definitions\n → tokenserver-mysql/\n → tokenserver-postgres/\n```\n\n## Step 1 — Find changed trait files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-db/|tokenserver-db/\"\n```\n\nIf no trait files changed, check whether any backend implementation changed without a corresponding trait change — that may indicate the trait was bypassed or duplicated.\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-(mysql|postgres|spanner)/|tokenserver-(mysql|postgres)/\"\n```\n\n## Step 2 — Diff the traits\n\nFor each changed trait file:\n\n```bash\ngit diff main...HEAD -- \n```\n\nList every added, removed, or modified method signature.\n\n## Step 3 — Check each backend for the same change\n\nFor each modified method, grep for its name in every backend crate:\n\n```bash\ngrep -rn \"fn \" syncstorage-mysql/src/ syncstorage-postgres/src/ syncstorage-spanner/src/\ngrep -rn \"fn \" tokenserver-mysql/src/ tokenserver-postgres/src/\n```\n\nFor each backend, determine:\n- **Present and updated** — matches the new signature ✓\n- **Present but stale** — exists but hasn't been updated to match ✗\n- **Missing** — not implemented at all ✗\n\n## Step 4 — Check for cross-backend behavioral assumptions\n\nReview the diff for patterns that are safe on one backend but wrong on another:\n\n- **Auto-increment IDs** — MySQL/Postgres use serial/auto-increment; Spanner uses explicit INT64. Code that relies on `last_insert_id()` or assumes monotonic IDs will break on Spanner.\n- **Transaction semantics** — Spanner transactions are bounded; MySQL/Postgres allow longer-lived transactions. Check for assumptions about transaction scope.\n- **NULL handling** — Spanner is stricter about NULLs in some column types than MySQL.\n- **RETURNING clause** — Postgres supports `RETURNING id`; MySQL uses `last_insert_id()`; Spanner uses a separate read. Check that insert helpers follow the per-backend pattern established in `helpers.py` / existing db impls.\n- **Timestamp precision** — Spanner uses microseconds; MySQL DATETIME has second precision unless explicitly specified.\n- **Feature flags** — confirm the change compiles under `--features=syncstorage-db/mysql`, `--features=syncstorage-db/postgres`, and `--features=syncstorage-db/spanner` independently.\n\n## Step 5 — Check shared db-common crates\n\nChanges to `syncstorage-db-common/` or `tokenserver-db-common/` affect all backends. Verify nothing in those changes makes backend-specific assumptions.\n\n```bash\ngit diff main...HEAD --name-only | grep \"db-common\"\n```\n\n## Output format\n\n**Trait changes detected:**\nList each modified method with its new signature.\n\n**Backend coverage:**\n\n| Method | mysql | postgres | spanner |\n|---|---|---|---|\n| `method_name` | ✓ updated / ✗ stale / ✗ missing | ... | ... |\n\n**Behavioral drift warnings:**\nAny cross-backend assumptions found, with file:line.\n\n**Verdict:** All backends consistent / N gaps found\n",".claude/skills/pytest-fixture-review/SKILL.md":"---\nname: pytest-fixture-review\ndescription: Reviews conftest.py and helpers.py files in tools/integration_tests/ against the repo's fixture/helper separation standard. Flags helpers in conftest, missing teardown, wrong scopes, and raw DB operations outside helper functions.\nuser-invocable: true\n---\n\n# Pytest Fixture Review\n\nYou are a pytest code reviewer for syncstorage-rs integration tests. This repo has a deliberate structure: `conftest.py` contains only pytest fixtures, and all utility functions live in `helpers.py`. Your job is to enforce that standard and catch common fixture mistakes before they cause flaky or confusing tests.\n\n## The standard\n\n```\ntools/integration_tests/\n conftest.py ← fixtures ONLY (st_ctx)\n helpers.py ← retry helpers, auth state, switch_user, constants\n\ntools/integration_tests/tokenserver/\n conftest.py ← fixtures ONLY (ts_db_conn, ts_app, ts_service_id, ts_ctx, fxa_auth)\n helpers.py ← DB helpers, auth helpers, node/user/service helpers\n```\n\nFixtures yield context dicts. Helpers are plain functions imported directly by test files.\n\n## Step 1 — Find changed test files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"tools/integration_tests\"\n```\n\nRead each changed file in full.\n\n## Step 2 — conftest.py checks\n\nFor each `conftest.py` in the diff:\n\n**Non-fixture code in conftest:**\n- Are there any module-level functions that are not decorated with `@pytest.fixture`?\n- Are there utility functions, helper classes, constants, or context managers defined in conftest that belong in `helpers.py`?\n- Exception: module-level side effects that must run at import time (e.g. the `SYNC_TEST_LOG_HTTP` logging patch) may stay in conftest with a comment explaining why.\n\n**Fixture scopes:**\n- `scope=\"function\"` — default, correct for most fixtures; state is reset per test\n- `scope=\"session\"` — only justified for expensive external calls (e.g. `fxa_auth` creates a real FxA account). Any new session-scoped fixture must have a comment explaining why.\n- `scope=\"module\"` or `scope=\"class\"` — flag for review; not currently used in this repo\n\n**Teardown:**\n- Does every fixture that creates state also clean it up?\n- `ts_ctx` pattern: `clear_db()` before yield AND after yield. Verify both sides are present.\n- `fxa_auth` pattern: `acct.clear()` + `client.destroy_account()` in teardown. Verify error handling wraps the teardown.\n- `st_ctx` pattern: `config.end()` + `del os.environ[\"MOZSVC_UUID\"]`. Verify both.\n\n**Database connections:**\n- `ts_db_conn` uses `AUTOCOMMIT` isolation. If a new fixture creates a connection, verify isolation level is set explicitly.\n- Connections must be closed in teardown (`conn.close()`).\n\n## Step 3 — helpers.py checks\n\n**No fixtures in helpers:**\n- Are there any functions decorated with `@pytest.fixture` in `helpers.py`? These must move to `conftest.py`.\n\n**Raw SQL outside helper functions:**\n- Is `sqltext()` / `execute_sql()` called directly in a test file rather than through a helper?\n- Direct SQL in tests bypasses the helper layer and makes tests brittle. Flag with file:line.\n\n**Imports:**\n- Are helpers importing from `conftest`? This creates a circular dependency risk. Helpers should have no pytest imports and no conftest imports.\n\n**Constants:**\n- Are test constants (NODE_ID, NODE_URL, FXA_EMAIL_DOMAIN, etc.) defined in helpers, not scattered in test files or conftest?\n\n## Step 4 — Test file import checks\n\nFor each changed test file:\n\n```bash\ngrep -n \"from.*conftest import\" \n```\n\nTest files should import helpers from `helpers.py`, not from `conftest.py`. Any `from tools.integration_tests.conftest import` or `from integration_tests.tokenserver.conftest import` is wrong — flag it with the correct import path.\n\n## Step 5 — New fixture pattern check\n\nIf a new fixture was added, verify:\n- It follows the `yield`-based teardown pattern (not `return`)\n- It is typed or documented enough that its contents are clear\n- If it seeds database state, it cleans up after itself\n- Its scope is the narrowest possible for its purpose\n\n## Output format\n\n**conftest.py issues:**\nList each violation with file:line and what needs to move or change.\n\n**helpers.py issues:**\nList each violation with file:line.\n\n**Import violations:**\nList any test files importing from conftest instead of helpers.\n\n**New fixtures:**\nFor each new fixture: name, scope, teardown present (yes/no), any concerns.\n\n**Verdict:** Follows conventions / N issues found\n",".claude/skills/rust-code-smell/SKILL.md":"---\nname: rust-code-smell\ndescription: Runs clippy, cargo audit, and all relevant multi-backend Makefile targets for files changed in the current branch. Detects code smells, anti-patterns specific to this codebase, and surfaces issues across every affected backend before CI catches them. Also handles dependency-update verification (Cargo.toml / Cargo.lock changes from dependabot or manual bumps) via the `deps` mode.\nuser-invocable: true\n---\n\n# Rust Code Smell Detector\n\nYou are a Rust code quality reviewer for syncstorage-rs. Your job is to run the full set of mechanical checks relevant to the current branch's changes, then layer on a manual smell pass for patterns that tooling misses. Run checks for every backend touched by the diff — not just the default.\n\n## Standing authorization\n\nWhen this skill is invoked, you have standing authorization to run the following without per-command confirmation:\n\n- `cargo fmt -- --check`\n- `cargo clippy ...` (any feature combination)\n- `make clippy_mysql`, `make clippy_postgres`, `make clippy_spanner`\n- `make clippy_release_mysql`, `make clippy_release_postgres`, `make clippy_release_spanner`\n- `cargo audit`\n- `cargo check --workspace ...`\n- `cargo test --no-run` (compile-check only — do NOT run actual tests without explicit permission, since they require DB setup)\n- `cargo update --precise -p ` (lockfile-only adjustments during bisection)\n- `git stash` / `git stash pop` (for baseline comparison)\n\nDo not run anything destructive (commit, push, rebase, branch deletion, `cargo clean` of the whole workspace, `make run_*`, docker-compose up/down) without asking.\n\n## Mode selection\n\n- **Default mode** — branch under review contains Rust source changes (not just `Cargo.toml`/`Cargo.lock`). Run Steps 1–5 below.\n- **`deps` mode** — branch only changes `Cargo.toml` / `Cargo.lock`, or the user invokes the skill in the context of a dependabot PR / dependency bump. Jump to the \"Dependency update verification\" section near the end and follow that flow instead.\n\n## Step 1 — Identify which backends are affected\n\n```bash\ngit diff main...HEAD --name-only\n```\n\nDetermine which backends the changes touch:\n\n- Files in `syncstorage-mysql/`, `tokenserver-mysql/`, or `syncstorage-db/` with MySQL defaults → **mysql**\n- Files in `syncstorage-postgres/`, `tokenserver-postgres/` → **postgres**\n- Files in `syncstorage-spanner/` → **spanner**\n- Files in `syncserver/`, `syncserver-common/`, `syncserver-settings/`, `syncstorage-db-common/`, `tokenserver-auth/`, `tokenserver-common/` → **all three backends**\n\n## Step 2 — Format check\n\n```bash\ncargo fmt -- --check\n```\n\nIf formatting issues exist, stop and report them. Do not proceed until format is clean — other tools will produce misleading output on unformatted code.\n\n## Step 3 — Run Clippy for each affected backend\n\nRun only the backends identified in Step 1. Use the exact Makefile targets:\n\n```bash\nmake clippy_mysql # if mysql affected\nmake clippy_postgres # if postgres affected\nmake clippy_spanner # if spanner affected\n```\n\nThen run release-mode Clippy for each affected backend (catches dead code, unused imports with optimizations, and issues only visible at `--release`):\n\n```bash\nmake clippy_release_mysql # if mysql affected\nmake clippy_release_postgres # if postgres affected\nmake clippy_release_spanner # if spanner affected\n```\n\nCollect all warnings and errors. Do not deduplicate across backends — the same logical issue may manifest differently per backend due to feature flags.\n\n## Step 4 — Security audit\n\n```bash\ncargo audit\n```\n\nFlag any **critical** or **high** severity advisories as blockers. List **medium** as warnings. Note the affected crate and advisory ID for each.\n\n## Step 5 — Manual smell pass\n\nRead the diff:\n\n```bash\ngit diff main...HEAD\n```\n\nCheck for the following patterns that Clippy does not catch:\n\n### Async/blocking\n- `std::thread::sleep` inside an async function — use `tokio::time::sleep`\n- Blocking DB calls inside async handlers without `spawn_blocking`\n- `.unwrap()` or `.expect()` on `Result`/`Option` in production paths (test code is fine)\n\n### gRPC / Spanner-specific\n- `grpcio::RpcStatus` errors caught and silently dropped without either Sentry classification or metric emission — see `is_sentry_event()` / `is_ignored_internal()` pattern\n- Retry logic that doesn't cap attempts or uses `std::thread::sleep` instead of exponential backoff\n- Spanner mutations not batched where they could be\n\n### Error handling\n- `map_err(|_| ...)` that discards the original error without logging — information loss\n- `unwrap()` on lock acquisition (implies the lock is poisonable)\n- New `From` impls that convert specific errors into generic `InternalError` without preserving context\n\n### Trait object vs generic\n- `Box` or `Arc` allocations in hot paths where a generic would avoid the vtable overhead\n- Inconsistency with how the rest of the codebase handles the same trait (check existing usage)\n\n### Diesel / query patterns\n- N+1 query patterns: loops that issue individual DB calls instead of a single query\n- Missing index hints or queries on unindexed columns in high-traffic paths\n- Raw string SQL (`sqltext(...)`) where a typed Diesel query builder already exists\n\n### Configuration / secrets\n- Secrets or credentials hardcoded or defaulting to non-placeholder values outside of test modules\n- Environment variable reads outside of the settings structs (should go through `syncserver-settings`)\n\n### Feature flag hygiene\n- `#[cfg(feature = \"...\")]` on items that should be available to all backends\n- Code that compiles under one feature but silently no-ops under another without a clear comment\n\n## Output format\n\n**Format:** pass / fail (list files)\n\n**Clippy results per backend:**\n\n| Backend | Debug | Release | Issues |\n|---|---|---|---|\n| mysql | pass/fail | pass/fail | N warnings/errors |\n| postgres | pass/fail | pass/fail | |\n| spanner | pass/fail | pass/fail | |\n\n**Audit:** pass / N advisories (list crate, severity, advisory ID)\n\n**Manual smells found:**\nList each with file:line, smell type, and a one-line explanation.\n\n**Verdict:** Clean / N issues need attention — list blockers vs warnings separately.\n\n---\n\n# Dependency update verification (`deps` mode)\n\nUse this flow when the branch only changes `Cargo.toml` / `Cargo.lock`, the user is evaluating a dependabot PR, or the user is splitting a grouped dependency PR into smaller pieces.\n\n## When to enter this mode\n\n- User invokes the skill on a dependabot PR (e.g. mozilla-services/syncstorage-rs#NNNN).\n- `git diff main...HEAD --name-only` returns only `Cargo.toml` and/or `Cargo.lock`.\n- User explicitly says \"check this dep bump\" / \"verify the cargo updates\" / \"is this PR safe\".\n\n## Step D1 — Establish the baseline\n\nThe single most valuable signal is whether the gates already pass on `main` before the bump. Stash, test, restore:\n\n```bash\ngit stash\nmake clippy_mysql 2>&1 | tail -10\ngit stash pop\n```\n\nIf `main` already fails a gate, stop and surface that to the user — the bump did not introduce the failure and pretending otherwise wastes time.\n\n## Step D2 — Classify the bumps\n\nCategorize every changed crate before running anything else. Read the manifest diff plus dependabot's release notes (in the PR body) and assign each crate to one of:\n\n- **Routine** — patch/minor bumps to leaf-ish deps (`env_logger`, `reqwest`, `utoipa`, `uuid`, `cadence`, `jsonwebtoken`, `config`, `pyo3` patches). Group these.\n- **Ecosystem-coupled** — crates whose versions must move together. In this repo:\n - `diesel-async` and `deadpool` are paired — Cargo.toml comments already say so.\n - `diesel_migrations` patch bumps can interact with `diesel-async`'s major version (a 2.3.2 bump broke `IntoUpdateTarget` trait selection against diesel-async 0.7; the same bump composes fine with diesel-async 0.9). Treat `diesel_migrations` as ecosystem-coupled even though it looks like a patch.\n - `protobuf` for the Spanner backend must match what `google-cloud-rust-raw` resolves. Do not bump `syncstorage-spanner`'s pinned `protobuf` ahead of `google-cloud-rust-raw`.\n- **Major / breaking** — any `x.0 → y.0` or `0.x → 0.y` bump on a direct dep. Flag for human review even if compilation passes.\n\nRecommend splitting a grouped dependabot PR along these lines: one PR per category, with the riskier categories held until their constraints clear.\n\n## Step D3 — Apply bumps surgically\n\nWhen applying a chosen group, use `--precise` per crate so transitive deps don't drift:\n\n```bash\ncargo update --precise -p \n```\n\nDo not run a bare `cargo update -p ` — that will pull in unrelated transitive bumps (e.g. `hashlink 0.10 → 0.11`, `anstream 0.6 → 1.0`) and you'll then have to debug whether those broke something. `--precise` keeps the diff to exactly what was requested.\n\nCargo only accepts one `--precise` per invocation. To apply N crates, run N sequential `cargo update --precise` commands.\n\nIf two crates share a version qualifier (`reqwest@0.13.2`), cargo will demand the disambiguator — use the form it suggests in the error.\n\nAfter every cargo-update batch, run:\n\n```bash\ngit diff origin/master -- Cargo.lock | grep -E '^[-+]version = ' | sort | uniq -c | sort -rn\n```\n\nVerify the version-change set matches the intended group. Anything you did not request showing up is a red flag.\n\n## Step D4 — Run the gates\n\nRun, in order, stopping at the first failure:\n\n```bash\ncargo fmt -- --check\ncargo audit\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\ncargo test --workspace --no-run # compile-check only\n```\n\n`make test` (running the suite) requires a real MySQL on `localhost:3306` with `sample_user:sample_password` and the schemas pre-created. If the user's environment doesn't have it, report that explicitly and recommend `make docker_run_mysql_e2e_tests` / `make docker_run_postgres_e2e_tests` for full verification. Do not attempt to set up the DB.\n\n`cargo test --no-run` also requires `libpython3.9` linkable on disk because `pyo3` is a default-feature dep. If that fails at the link stage with `library 'python3.9' not found`, report it as a local env issue, not a bump regression. Confirm by running the same `--no-run` against the baseline (Step D1) — if it fails there too, it's not your problem.\n\n## Step D5 — Bisect on failure\n\nIf a gate fails, do not start guessing. Bisect:\n\n1. Revert the *most likely* single crate to its pre-bump version with `cargo update --precise -p `.\n2. Re-run only the failing gate. If it now passes, that crate is the culprit. If not, restore it and try the next candidate.\n3. Likely-culprit ordering for diesel/diesel-async-related failures: `diesel_migrations` first, then `uuid` (it's a diesel feature type), then anything else that touches DB.\n4. For trait-selection / type-inference errors in `syncstorage-postgres` or `syncstorage-mysql` under a *different* backend's clippy target (e.g. clippy_mysql failing in syncstorage-postgres source), the failure is almost always an ecosystem-coupled crate moving alone. Re-classify and move it to the appropriate group.\n\nWhen a crate is identified as incompatible with the current group, remove it from the group, return it to the manifest's previous version, and update the Jira ticket (or PR description) to note it's been moved to whichever group it now belongs with.\n\n## Step D6 — Report\n\nOutput one table:\n\n| Crate | Old | New | Status |\n|---|---|---|---|\n| ... | ... | ... | applied / held / failed / moved to Group X |\n\nPlus the verification matrix:\n\n| Gate | Result | Notes |\n|---|---|---|\n| `cargo fmt --check` | pass/fail | |\n| `cargo audit` | pass/fail | N advisories listed if any |\n| `make clippy_mysql` | pass/fail | |\n| `make clippy_postgres` | pass/fail | |\n| `make clippy_spanner` | pass/fail | |\n| `cargo test --no-run` | pass/fail/env-skip | note libpython if env-skipped |\n\nEnd with one paragraph: what landed, what was excluded and why, and what should ride with the next group.\n\n## Known coupling notes for this repo\n\nKeep these in mind when classifying bumps — they cost real time to rediscover:\n\n- **`diesel-async` + `deadpool`** are paired. Cargo.toml comments say so. Do not move one without the other.\n- **`diesel_migrations` 2.3.2** breaks `IntoUpdateTarget` trait selection in `syncstorage-postgres` source when paired with `diesel-async 0.7`. It needs to ride with the `diesel-async 0.9` upgrade.\n- **`protobuf`** is pinned with `=` in `syncstorage-spanner/Cargo.toml` deliberately. The pin must match what `google-cloud-rust-raw` resolves to. Currently both should be `2.28.0`. Bumping the spanner pin ahead of `google-cloud-rust-raw` will cause two protobuf majors in the lockfile and break compilation (RepeatedField doesn't exist in protobuf 3.x).\n- **`make clippy_mysql` / `clippy_postgres` / `clippy_spanner`** all use `--workspace --all-targets` — every backend crate compiles under every feature set. A failure in `syncstorage-postgres` source under `clippy_mysql` is not a misconfiguration; it means a workspace-shared dep changed something diesel-postgres can't tolerate.\n- **`pyo3` + `libpython3.9` link** is required for default-feature builds on this repo. Local `cargo test --no-run` will fail at link without it. Docker e2e is the fallback for full test verification.\n",".claude/skills/rust-error-review/SKILL.md":"---\nname: rust-error-review\ndescription: Reviews new or modified error variants in syncstorage-rs for correct Sentry routing, HTTP status mapping, and response safety. Enforces the is_sentry_event() / maybe_emit_metrics() classification pattern.\nuser-invocable: true\n---\n\n# Rust Error Review\n\nYou are a Rust error handling reviewer for syncstorage-rs. This repo has a specific, non-obvious error routing pattern: every error variant must be explicitly classified as either a Sentry event or a suppressed metric. Getting this wrong causes either noisy Sentry alerts (on transient infrastructure errors) or silent failures (on real bugs). Your job is to catch misclassification before merge.\n\n## The pattern\n\n```\nSyncstorageError / DbError\n → is_sentry_event() → true → sent to Sentry\n → is_sentry_event() → false → maybe_emit_metrics() → StatsD counter\n```\n\nKey files:\n- `syncstorage-spanner/src/error.rs` — `DbError`, `DbErrorKind`, `is_sentry_event()`, `is_ignored_internal()`\n- `syncstorage-db-common/src/error.rs` — shared error types\n- `syncserver/src/error.rs` — top-level `ApiError`, HTTP response mapping\n- `tokenserver-common/src/error.rs` — tokenserver error types\n\n## Step 1 — Find changed error files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"error\\.rs\"\n```\n\nAlso check for new `From<>` impls that convert external errors into repo error types:\n\n```bash\ngit diff main...HEAD | grep -E \"impl From|Into<.*Error\"\n```\n\n## Step 2 — Diff each error file\n\n```bash\ngit diff main...HEAD -- \n```\n\n## Step 3 — Review checklist\n\nFor each new or modified error variant:\n\n### Sentry classification\n- Is there a new arm in `is_sentry_event()` for this variant?\n- Is the classification correct?\n - **Should be Sentry:** logic bugs, unexpected states, data integrity issues, auth failures that indicate an attack, panics\n - **Should be metric only:** transient infrastructure errors (gRPC RST_STREAM, connection timeouts, 503s from upstream), expected client errors (404, 409 conflict, quota exceeded)\n- For `INTERNAL` gRPC errors: is `is_ignored_internal()` doing substring matching (`.iter().any(|s| msg.contains(s))`)? Exact equality will miss real RST_STREAM messages.\n\n### HTTP status mapping\n- Does the new variant map to an appropriate HTTP status in the `From for ApiError` or `status_code()` impl?\n- Are client errors (4xx) distinguished from server errors (5xx)?\n- Is a 500 being returned for something that should be a 409 or 404?\n\n### Response body safety\n- Does the error's `Display` or serialization leak internal state (SQL query text, file paths, internal IDs, stack traces)?\n- Check `ApiError`'s response serialization — internal error messages must not reach the client.\n\n### Backtrace\n- Is `capture_backtrace()` called for new unexpected/internal error variants?\n- Is it *not* called for expected operational errors (quota, not found, conflict)?\n\n### Metric naming\n- If the variant routes to `maybe_emit_metrics()`, is the metric name consistent with existing conventions (`storage.spanner.grpc.internal`, etc.)?\n\n## Step 4 — Check `From` impl chains\n\nNew error types often get wrapped through multiple `From` impls before reaching the HTTP layer. Trace the chain to verify classification survives the conversion:\n\n```bash\ngrep -rn \"From<\" syncstorage-spanner/src/ syncstorage-db/src/ syncserver/src/ | grep -i error\n```\n\n## Output format\n\nFor each changed error variant:\n- **Variant:** name\n- **Classification:** Sentry / Metric / Unclear\n- **HTTP status:** correct / wrong (should be X) / missing\n- **Response safety:** safe / leaks internal detail at file:line\n- **Issues:** list\n\nEnd with **All error variants correctly classified** or **N classification issues found**.\n",".claude/skills/sentry-health/SKILL.md":"---\nname: sentry-health\ndescription: Queries Sentry for syncstorage-rs (project syncstorage-prod) to surface error trends, new issues, regressions, and production health signals. Reports as engineer/manager view of what requires attention.\nuser-invocable: true\nargument-hint: \"new | regressions | volume | full (default: full)\"\n---\n\nYou are acting as **both engineer and engineering manager** for syncstorage-rs. You are reviewing Sentry to understand production error health, spot regressions, and flag issues that need triage or fixes.\n\nStart by reading `.claude/mcp-context.md` for project slugs, environments, what to flag, and what is expected noise.\n\nParse `$ARGUMENTS` (trimmed, lowercased). Default is `full`. Valid modes:\n- `new` — issues first seen in the last 7 days\n- `regressions` — previously resolved issues that have recurred\n- `volume` — high-event-count issues\n- `full` — all of the above\n\n---\n\n## Step 1: Confirm the project\n\nCall `mcp__sentry__find_projects` and locate the `syncstorage-prod` project under the `mozilla` organization. Confirm the project slug and organization slug before querying. If the project is not found, report the available projects and stop.\n\n---\n\n## Step 2: Gather issue data\n\nUse available Sentry MCP tools to fetch issues for `syncstorage-prod`. Apply environment filter `production` unless the user has specified otherwise.\n\nFetch based on the selected mode:\n\n### New issues (`new`, `full`)\nIssues with `firstSeen` in the last 7 days that have not been resolved. These are the riskiest — they represent new failure modes introduced recently.\n\nSort by: event count descending.\n\nFor each, capture:\n- Issue ID and title\n- First seen timestamp\n- Event count in the period\n- Affected users (if available)\n- Platform / culprit (the Rust file/function if available)\n\n### Regressions (`regressions`, `full`)\nIssues that were previously marked resolved but have reoccurred. These indicate incomplete fixes or flaky patches.\n\nFor each, capture:\n- Issue ID and title\n- Original resolution date\n- Recurrence date\n- Event count since recurrence\n\n### High-volume issues (`volume`, `full`)\nIssues with the most events in the last 24 hours, regardless of age. These are the most actively impacting users.\n\nFor each, capture:\n- Issue ID and title\n- Event count (24h)\n- Is this a known/triaged issue or unacknowledged?\n\n---\n\n## Step 3: Classify each issue\n\nFor every issue surfaced, classify it against the known patterns in `.claude/mcp-context.md`:\n\n**Tokenserver auth path** — involves `hawk_identifier`, FxA OAuth, `tokenserver-auth/`, `client_state`, `keys_changed_at`. These are user-facing auth failures — high priority.\n\n**Database layer** — involves `DbError`, `diesel`, `deadpool`, connection pool, migration, query timeout. May indicate infrastructure problems.\n\n**Syncstorage storage path** — involves BSO read/write, collection operations, quota. Usually lower priority unless volume is very high.\n\n**Expected noise** — 409 Conflict, 503 during deployment, staging-environment-only errors. Mark as \"noise\" and exclude from the action list.\n\n**Spanner-specific** — RST_STREAM, gRPC errors in production (not emulator). These should NOT appear in production — flag immediately.\n\n---\n\n## Step 4: Cross-reference with Jira (if context available)\n\nIf recent Jira data is available in this conversation, check whether any Sentry issues correspond to open `STOR-` bugs. Note the link if found.\n\nIf a high-volume or regression Sentry issue has no corresponding Jira ticket, that is a gap — flag it as \"needs ticket\".\n\n---\n\n## Step 5: Output format\n\n### Production Health: [OK / Watch / Degraded]\nOne-line overall assessment.\n\n### New Issues (Last 7 Days)\nTable: Issue ID | Title | First Seen | Events | Classification\nIf none: \"No new issues in the last 7 days.\"\n\n### Regressions\nTable: Issue ID | Title | Resolved On | Recurred On | Events Since\nIf none: \"No regressions detected.\"\n\n### High-Volume Issues (Last 24h)\nTable: Issue ID | Title | Events (24h) | Triaged? | Classification\nExclude known noise (mark separately if many).\n\n### Error Pattern Analysis\nBulleted findings only if genuine patterns are present:\n- Auth path errors (tokenserver risk)\n- DB layer errors (infrastructure risk)\n- Sudden volume spikes (availability risk)\n- Patterns that correlate with a recent deploy or migration\n\n### Recommended Actions\nUp to 5 concrete items. Each: issue ID, what to do, urgency. Skip if no action is warranted.\n\n### Noise / Expected Errors (Acknowledged)\nBrief list of issues seen but classified as expected noise. One line each. This confirms you looked at them.\n\n---\n\n## Guidelines\n\n- Do not fabricate event counts or timestamps — use exactly what Sentry returns\n- If a query fails or a tool is unavailable, note it and work with available data\n- `is_sentry_event()` in `syncserver/src/error.rs` is the canonical routing decision — errors that are HTTP 4xx should generally not be in Sentry at high volume; flag if they are\n- Focus on production environment; staging errors are informational only\n- An issue being \"old\" does not mean it's not important — check event count trends, not just age\n- Sentry issue IDs should be referenced in full (e.g. `SYNCSTORAGE-PROD-####`) if available\n",".claude/skills/spanner-local-tests/SKILL.md":"---\nname: spanner-local-tests\ndescription: Brings up the Spanner emulator, provisions the test database with the project schema, and runs the syncstorage-rs Spanner unit-test suite locally. Repeatable single-command flow plus full from-scratch documentation. Also covers the e2e (docker-compose) path.\nuser-invocable: true\n---\n\n# Run Spanner Tests Locally\n\nSet up a local Spanner emulator + schema, then run the spanner-backed unit tests. The whole loop is < 90 seconds once the emulator image is cached. This is the **fast inner loop** for any change touching `syncstorage-spanner/`.\n\nThere are two test surfaces in this repo:\n\n| Surface | What it covers | Time | Needs |\n|---|---|---|---|\n| **Unit tests** (this skill's default) | Direct calls into `SpannerDb` — exercises every DML path through `update_user_collection_quotas`, batch ops, BSO CRUD, tombstones, quota. | ~30s after warm-up | emulator only |\n| **Docker e2e** | Full syncserver + emulator + tokenserver + mock FxA, hit over HTTP by the Python integration suite. | 5-15 min | freshly-built `app:build` image |\n\nDefault to unit tests. Reach for e2e only when changes touch HTTP routing, token handling, or cross-backend behavior CI would catch first.\n\n## Quick re-run (after first-time setup)\n\nIf the emulator is already up and provisioned:\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nIf the emulator is **not** running, use the helper:\n\n```bash\n./scripts/spanner-local-test-setup.sh\n```\n\nThat script brings up the emulator, provisions the database/schema, and runs the suite. It's idempotent — re-running just executes the tests if the stack is already up.\n\n## From scratch — the full flow\n\n### 1. Prerequisites\n\n- **Docker Desktop running.** The emulator runs as a container.\n- **Rust toolchain** matching `rust-toolchain.toml` (or close).\n- For e2e only: enough Docker memory (≥ 8 GB recommended) — the `app:build` image compiles grpcio-sys from source and OOMs on small allocations.\n\n### 2. Start the emulator\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml up -d sync-db\n```\n\nThis starts the [`gcr.io/cloud-spanner-emulator/emulator`](https://cloud.google.com/spanner/docs/emulator) image. Two ports:\n\n- **9010** — gRPC. The Rust client (and production) talk this.\n- **9020** — REST. Used for admin (create instance / create database / DDL).\n\nWait ~2 seconds for the emulator to be ready. Confirm:\n\n```bash\ncurl -fsS -o /dev/null http://localhost:9020/v1/projects/test-project/instances && echo ok\n```\n\n### 3. Create the instance and database\n\n```bash\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"instance\":{\"config\":\"emulator-test-config\",\"nodeCount\":1,\"displayName\":\"Test\"},\"instanceId\":\"test-instance\"}'\n\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"createStatement\":\"CREATE DATABASE `test-database`\"}'\n```\n\n### 4. Apply the schema\n\n⚠️ **Do not run `scripts/prepare-spanner.sh` directly on macOS.** Its `sed` invocation uses GNU `\\+` syntax that BSD `sed` doesn't accept. The script is designed to run inside the Linux container (where GNU sed lives). When you run it from a macOS host, the DDL parsing silently returns an empty array, the database gets created with no tables, and tests fail with cryptic *\"Table not found: collections\"* errors.\n\nUse this macOS-safe inline equivalent:\n\n```bash\nDDL_STATEMENTS=$(grep -v '^--' syncstorage-spanner/src/schema.ddl \\\n | tr '\\n' ' ' \\\n | tr -s ' ' \\\n | sed 's/;[[:space:]]*$//' \\\n | jq -R -s -c 'split(\";\") | map(select(length > 0)) | map(gsub(\"^\\\\s+|\\\\s+$\";\"\"))')\n\nPAYLOAD=$(jq -n --argjson stmts \"$DDL_STATEMENTS\" '{statements:$stmts}')\n\ncurl -fsS -X PATCH \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n -H 'Content-Type: application/json' \\\n -d \"$PAYLOAD\"\n```\n\nThen verify the tables landed:\n\n```bash\ncurl -fsS \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n | jq '.statements | length'\n# expected: 8 (5 tables + 3 indexes + 2 row-deletion policies, the last 2 are folded into CREATE TABLE statements by the emulator)\n```\n\n### 5. Run the unit tests\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nTwo non-obvious env vars:\n\n- `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` — the gRPC endpoint. **This is what tells the client to skip OAuth.** If you leave it unset, every test fails with *\"Error occurred when fetching oauth2 token\"* because the client tries to authenticate against the real Google API.\n- `RUST_TEST_THREADS=1` — required. Many tests share the same database and assert on counts/timestamps; parallel runs race.\n\nExpected output: `test result: ok. 39 passed; 0 failed; …`.\n\n### 6. Run a single test\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db \\\n tests::db::delete_collection_tombstone -- --nocapture\n```\n\n`--nocapture` is essential when tests fail — without it, errors get swallowed.\n\n### 7. Teardown\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml down -v\n```\n\nThe `-v` wipes named volumes for the whole compose project — including `tokenserver_db_data`. If you have the full e2e stack (or anything else from this compose file) running in parallel, prefer the helper script's narrower teardown instead:\n\n```bash\n./scripts/spanner-local-test-setup.sh down\n```\n\nThat only stops + removes the `sync-db` container. The emulator keeps no data in named volumes, so next run starts from a fresh database either way.\n\n## Common failures\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `Error occurred when fetching oauth2 token` | Forgot `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST`. Client is trying real Google OAuth. | Export the var. |\n| `Table not found: collections [at 2:25]` | DDL didn't apply. Most often because `prepare-spanner.sh` was run on macOS and produced an empty statement list. | Use the inline DDL block in step 4. |\n| `INVALID_ARGUMENT: …` on `INSERT OR UPDATE` | Emulator version too old (< 1.5.x). | Update `docker/docker-compose.spanner.yaml`'s image tag. |\n| 39 tests \"pass\" in 0.2s | Tests didn't actually connect — they're failing the OAuth check and treating it as a fresh start. Check the first test's error. | Same as the OAuth row above. |\n| Tests intermittently fail with timestamp/count mismatches | `RUST_TEST_THREADS=1` not set. | Set it. |\n| `cargo test` error: *the package 'syncstorage-db' does not contain this feature: py_verifier* | `py_verifier` is on the `syncserver` crate. | Drop `--features=py_verifier` when running tests on `syncstorage-db` alone. |\n\n## Docker e2e (optional, slower path)\n\nThe Makefile target `docker_run_spanner_e2e_tests` spins up the full stack and runs the Python integration suite twice (with and without JWK cache). It requires the `app:build` image to be **current** — built against the working tree's code.\n\n```bash\ndocker build -t app:build .\nmkdir -p workflow/test-results\nSYNCSTORAGE_RS_IMAGE=app:build make docker_run_spanner_e2e_tests\n```\n\nTwo local-run-only gotchas the make target assumes CI handles:\n\n- `SYNCSTORAGE_RS_IMAGE=app:build` must be exported. `docker-compose.spanner.yaml:52` references `${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}`; without the override compose tries to pull a non-existent `syncstorage-rs:latest` from Docker Hub. CI sets it in `.github/workflows/main-workflow.yml`. Local doesn't get it for free.\n- `workflow/test-results/` must exist before the make target runs. The target uses `docker cp ... ${INT_SPANNER_NO_JWK_JUNIT_XML}` which writes into that directory; if missing, the docker cp fails with `invalid output path` and aborts the run.\n\nTwo tests will always fail locally and can be safely ignored:\n\n- `tools/integration_tests/tokenserver/test_e2e.py::test_unauthorized_oauth_error_status`\n- `tools/integration_tests/tokenserver/test_e2e.py::test_valid_oauth_request`\n\nBoth hit **real FxA stage** (`api-accounts.stage.mozaws.net`, `oauth.stage.mozaws.net`) and need live network + production credentials the docker stack doesn't have. They surface as `fxa.errors.OutOfProtocolError: API responded with non-json content-type` — FxA returning an HTML error page because the request can't authenticate. CLAUDE.md excludes these from `make run_local_e2e_tests` via `--ignore=tokenserver/test_e2e.py`; the docker e2e target doesn't apply that exclusion, so expect `149 passed, 2 errors` not `151 passed`.\n\nNotes:\n\n- The image build compiles grpcio-sys from C++ source. On Docker Desktop for Mac, allocate **≥ 8 GB** memory.\n- The build takes 10-20 minutes from cold. Subsequent rebuilds reuse cache layers and are much faster *unless* `Cargo.lock` or the Rust toolchain changed.\n- Results land in `workspace/test-results/` as JUnit XML.\n- The make target tears the stack down on completion.\n\n### `ar: unable to copy file 'libgrpc.a'; reason: Success`\n\nMisleading message — the linker wasn't killed, the BuildKit cache mount got into a corrupt state. Seen on macOS with stale buildkit caches (mine had ~19 GB accumulated).\n\nFix:\n\n> ⚠️ `docker builder prune -f` clears BuildKit cache mounts for **every** Docker project on the host, not just this repo. Unrelated projects' next builds will go from cached to cold. Run it only when the link error reproduces and you've ruled out a real OOM.\n\n```bash\ndocker builder prune -f\ndocker build -t app:build .\n```\n\nThe prune drops the cache mounts grpcio-sys writes its 100+ MB `libgrpc.a` into. The next build re-creates them clean and the link step succeeds. If it fails a second time after prune, *then* suspect a real OOM and bump Docker Desktop memory.\n\nIf the local build won't cooperate, CI on push runs the same e2e suite — relying on it is reasonable for refactors that don't touch HTTP/auth/routing.\n\n## When to invoke this skill\n\nReach for it when:\n\n- You're editing anything under `syncstorage-spanner/` and want to validate before pushing.\n- A spanner test is failing in CI and you need to reproduce locally.\n- You're investigating an emulator-vs-prod behavior question.\n- You forgot the env-var dance for the third time this month.\n\nDon't invoke it for changes that don't touch Spanner — the MySQL or Postgres backends have their own test paths through the Makefile's `test_with_coverage` and `postgres_test_with_coverage` targets.\n\n## Output format\n\nAfter running, report:\n\n1. **Emulator state:** running / fresh / pre-existing\n2. **Schema applied:** N statements\n3. **Test results:** `X passed; Y failed; Z ignored` — list the failed test names if any\n4. **Total wall time**\n5. **Teardown:** done / left running for follow-up\n\nIf anything failed, surface the actual error message (not just the test name) — Spanner emulator errors are verbose but informative, and the first line is usually enough to diagnose.\n",".claude/skills/syncserver-explain-code/SKILL.md":"---\ndescription: Explains code for experienced engineers working on syncstorage-rs. Covers what changed, why it works, non-obvious decisions, gotchas, and data/control flow. Defaults to git diff vs main; accepts an optional file or path argument.\nargument-hint: file-or-path\n---\n\nYou are a senior engineer explaining code to another experienced engineer working on **syncstorage-rs** — Mozilla's Rust implementation of the Firefox Sync storage server and token server.\n\nSkip basics and language fundamentals. Focus on what this code does, why it was written this way, non-obvious decisions, and things that could surprise or bite someone.\n\n## Service boundary — critical context\n\nThis repo owns two services:\n- **Syncstorage** — the storage node that holds encrypted user sync data (bookmarks, passwords, tabs, etc.). Database backends: MySQL, PostgreSQL, Google Cloud Spanner.\n- **Tokenserver** — the allocation service that authenticates Firefox clients via FxA OAuth and assigns them to a storage node. Database backends: MySQL, PostgreSQL.\n\n**Firefox Accounts (FxA) is an external upstream service.** This repo does not own FxA. When explaining code:\n- References to `fxa_uid`, `fxa_kid`, `client_state`, FxA OAuth tokens, JWK keys, the FxA OAuth server URL — these are **inputs from FxA that this service validates and consumes**, not code this repo owns.\n- The `mock-fxa-server` in `scripts/` is a test stub only — it is not a real FxA implementation.\n- `tokenserver-auth/src/oauth/` and `tokenserver-auth/src/crypto.rs` are **this repo's FxA token verification logic**, not FxA's own code.\n\nMake this boundary explicit whenever the code touches the FxA/tokenserver interface.\n\n## How to gather the code\n\nIf `$ARGUMENTS` is provided, read that file or directory.\n\nOtherwise, run:\n\n```bash\ngit diff main...HEAD\n```\n\nand read the full diff. Follow imports or read related files as needed to give accurate explanations — do not explain in isolation if context from a related file matters.\n\n## Explanation structure\n\nWork through the code and produce an explanation covering all sections below. Omit a section only if it genuinely doesn't apply.\n\n### 1. One-paragraph summary\n\nPlain English. What does this code do, and what problem does it solve? Write for someone who hasn't read the ticket.\n\n### 2. Architecture & data flow\n\nWhere this fits in the broader system. Include an ASCII diagram if it clarifies the flow.\n\nUseful reference shapes for this codebase:\n\n```\nFirefox client\n → Tokenserver (validates FxA OAuth token, assigns node)\n → FxA OAuth server [external — validates token, not our code]\n → tokenserver-db (MySQL/Postgres — users, nodes, services tables)\n → Syncstorage node (stores/retrieves BSOs)\n → syncstorage-db (MySQL / Postgres / Spanner)\n```\n\n```\nRequest → Actix-web (syncserver/)\n → Extractors (hawk_identifier, fxa_uid, token validation)\n → Handler\n → db trait (syncstorage-db or tokenserver-db)\n → backend impl (mysql / postgres / spanner)\n```\n\n### 3. Annotated walkthrough\n\nStep through the key functions, types, or request paths. For each:\n- What it does\n- Why it's structured this way (if non-obvious)\n- How it connects to the next step\n\nFocus on the critical path. Don't exhaustively document trivial helpers.\n\nNote Rust-specific patterns where relevant: trait objects vs generics, `Arc`, `actix_web::web::Data<>`, `grpcio` vs `tonic`, `diesel` query builder patterns, `serde` field renaming.\n\n### 4. Gotchas & non-obvious bits\n\nThe most important section. Flag:\n\n- Implicit assumptions or preconditions the caller must satisfy\n- Surprising behavior or edge cases (off-by-one, async ordering, race conditions)\n- Why an obvious alternative wasn't taken (if inferrable)\n- Error handling that silently swallows failures or has unexpected fallback behavior\n- State mutated in non-obvious places\n- Performance characteristics worth knowing (N+1 queries, large allocations, blocking calls in async context)\n- Security-sensitive paths — especially: HAWK token validation, FxA OAuth scope checking, `client_state` / `keys_changed_at` handling, anything touching node assignment\n- gRPC-specific issues (Spanner backend): RST_STREAM handling, retry logic, emulator vs production behavioral differences\n- Migration safety: anything that changes DB schema or touches published migration files\n\n### 5. Dependencies & integrations\n\nExternal systems or crates this code depends on that aren't obvious from the code alone. Distinguish:\n- **Owned by this repo:** `syncstorage-*`, `tokenserver-*`, `syncserver-*` crates\n- **External upstream:** FxA OAuth server, Google Cloud Spanner, `tokenlib` (Python), `hawkauthlib`\n- **Test infrastructure only:** mock FxA server, Spanner emulator\n\nNote any version constraints or behavioral quirks.\n\n### 6. Testing notes\n\nHow is this code tested? Are there gaps? Note:\n- Unit tests (Rust `#[cfg(test)]` modules)\n- Integration tests (`tools/integration_tests/` — pytest, requires a running server)\n- E2E tests (`tokenserver/test_e2e.py` — hits real FxA staging, session-scoped, slow)\n- Any known flaky behavior (Spanner emulator quirks, timing-sensitive tests, 409/503 retry paths)\n\n## Style guidelines\n\n- Be direct and dense. Skip preamble.\n- Use code formatting for identifiers, file paths, and values.\n- Use ASCII diagrams when they save more words than they cost.\n- If something is genuinely straightforward, say so in one sentence and move on.\n- When the FxA boundary is crossed, say so explicitly rather than implying this repo owns that behavior.\n",".claude/skills/syncserver-jira-description/SKILL.md":"---\nname: syncserver-jira-description\ndescription: Drafts a concise Jira description for a syncstorage-rs task. Gathers context via targeted interview, researches relevant patterns in the repo, then outputs a clean description ready for an engineer to hand to Claude for implementation.\nuser-invocable: true\n---\n\n# Syncserver Jira Description\n\nDraft a Jira description for a syncstorage-rs task. Output the description only — do not create, edit, or suggest changes to any source files.\n\n## Step 1: Gather context\n\nIf a planning doc, epic description, or tech spec was provided, read it first and infer what, why, crates, and constraints before asking anything.\n\nRequired information:\n- **What:** What is being built or changed, in one sentence\n- **Why:** Motivation — user need, requirement, bug, or tech debt\n- **Crates:** Which specific crate(s) will be modified (e.g. `syncstorage-spanner`, `tokenserver-auth`, `syncserver`, `tokenserver-postgres`)\n- **Constraints:** DB migration required, multi-backend impact (MySQL/Postgres/Spanner), breaking API change, feature flag, FxA contract change — or none\n\nIf all four are clear from provided context, proceed directly to Step 2. Otherwise ask for only what is missing in a single message. Also invite related PRs, tickets, existing approach notes, or architecture notes that would add useful context.\n\n## Step 2: Research\n\nSearch only the crates identified in Step 1. Find the most relevant existing patterns: similar feature, nearby handler, equivalent DB query, related error type. Expand to the broader workspace only if nothing relevant is found there.\n\nIdentify:\n- Key files an implementer will need to touch\n- The closest existing reference implementation to follow\n- Which database backends are affected and whether the change must be replicated across MySQL, PostgreSQL, and/or Spanner\n- Whether new migrations are required (and which crates: `syncstorage-mysql`, `syncstorage-postgres`, `tokenserver-mysql`, `tokenserver-postgres`)\n- Whether tests, metrics (Glean/StatsD), or Sentry error handling apply\n\nIncorporate findings directly into the draft — do not list them separately or ask for confirmation. Surface genuine blockers as Open Questions.\n\n## Step 3: Output\n\n**Background:**\nWhy this is needed and what it enables. 2–4 sentences. Note which service is affected — Syncstorage (storage node), Tokenserver (node allocation + FxA auth), or both.\n\n**Acceptance Criteria:**\nObservable, testable outcomes. Each item verifiable without reading the code. Include criteria for tests, metrics emission, and Sentry error classification where applicable.\n\n**Implementation Steps:**\nNumbered steps with crate paths, file paths, trait/function names, and structural guidance. Reference the nearest existing pattern for each step. For multi-backend changes, call out which backends need matching implementations. No code snippets — file locations, types, and patterns only.\n\n**Database Migrations:** *(omit if no schema changes)*\nWhich migration crates need new files (`syncstorage-mysql/migrations/`, `tokenserver-postgres/migrations/`, etc.). Reminder: never edit published migrations — add new `up.sql`/`down.sql` pairs only.\n\n**Tests:**\nWhat needs to be tested. Unit (`#[cfg(test)]` in the relevant crate), integration (`tools/integration_tests/` pytest — requires running server), and E2E (`tokenserver/test_e2e.py` — hits FxA staging, session-scoped) coverage expectations. Reference the nearest existing test file as a pattern.\n\n**Metrics & Error Handling:** *(omit if not applicable)*\nAny Glean metrics, StatsD counters, or Sentry event classifications that should be emitted or updated. Note whether new error variants should be routed to Sentry or suppressed as metrics (see `is_sentry_event()` pattern in `syncstorage-spanner/src/error.rs`).\n\n**Key Reference Files:**\nSpecific files the implementer should read before starting. One line each.\n\n**Out of Scope:** *(omit if not needed)*\n\n**Open Questions:** *(omit if none)*\n\n## Guidelines\n\n- Output the description only — no source file changes\n- Implementation Steps should give enough detail to start work without follow-up questions — crate paths, file paths, and patterns, not prose\n- Multi-backend tasks: always explicitly name which backends (MySQL / PostgreSQL / Spanner) need matching changes; do not say \"all backends\" without listing them\n- Migration tasks: always reference the no-edit-published-migrations rule\n- Omit redundant or obvious acceptance criteria\n- Include Database Migrations, Metrics & Error Handling only when relevant\n- If motivation or scope remain unclear after asking, flag as an Open Question rather than assuming\n- Branch naming reminder for the assignee: Mozilla engineers use `type/description-STOR-####`; community contributors use `type/description-####`\n","CLAUDE.md":"# syncstorage-rs\n\nFirefox Sync storage and token server, written in Rust. The workspace contains a\nRust multi-crate server plus Python utilities and integration/end-to-end tests.\n\n## Agent rules\n\nSecurity takes absolute precedence. This repository handles synchronization of\nuser data and token management to access storage nodes.\n\n### 1) Scope & writes\n\n- Operate strictly within this repo root; normalize paths; do not follow\n symlinks outside the repo.\n- Writes are allowed to the working tree. Always present a diff for review\n before any staging or commit.\n- Do not modify files adjacent to a requested change. Mention issues found\n but do not fix them unless asked.\n- Ask before running any command (build, test, install, DB ops, git,\n service management). Do not run `git add`, `git commit`, `git push`, or\n `git rebase` unless explicitly instructed.\n\n### 2) Non-negotiables\n\n- **Secrets:** never read, print, summarize, or transmit secret files or\n values. Use placeholders (`YOUR_API_KEY_HERE`) in examples.\n- **External network:** only with explicit approval to a trusted, documented\n endpoint.\n- **Pipelines & contracts:** flag breaking API or contract changes. Do not\n alter GitHub Actions defined in `.github/`, any configuration for CI/CD workflows or git hooks without explicit, reviewed\n justification.\n- **Published DB migrations:** never edit existing published migration files or schema definitions (ex. `syncstorage-spanner/src/schema.ddl`), whether for MySQL, Postgres, or Spanner.\n Always add a new forward migration and a separate rollback migration.\n- **Workspace recommendations:** the repo deliberately ships no\n `.vscode/extensions.json`. Do not re-add one. Useful extensions are listed\n in `.vscode/README.md` for contributors to install at their own discretion.\n\n### 3) Do-not-touch paths\n\nNever read or write these paths. This list mirrors `.gitignore` — if there is\na discrepancy, `.gitignore` is the source of truth.\n\n```\nservice-account.json # GCP credentials\n.sentryclirc # Sentry auth token\n.envrc # direnv secrets\nconfig/local.toml # local server config (may contain secrets)\ntools/tokenserver/loadtests/*.pem\ntools/tokenserver/loadtests/*.pub\ntarget/ # Rust build artifacts\n*.xml # test result files\nopenapi.json # generated artifact\nbook/ # generated docs\ndocs/output/ # generated docs\nvenv/ # Python virtual environments\n.install.stamp # Poetry install marker\nworkspace/ # CI artifact directory\n```\n\n### 4) Git commit messages\n\nFollow the commit message format defined in [CONTRIBUTING.md](CONTRIBUTING.md) Git Commit Guidelines.\n\n- Format: `type(scope): subject` — `type` is one of `feat`, `fix`, `docs`,\n `style`, `refactor`, `perf`, `test`, `chore`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Body: explain motivation and contrast with previous behavior\n- Footer: `Closes STOR-1234` (Mozilla engineers — matches the Jira ticket) or\n `Closes #N` / `Issue #N` (community contributors — matches the GitHub issue number);\n `BREAKING CHANGE:` for breaking changes\n- Branch naming (Mozilla engineers): `type/description-STOR-1234` where `1234` must match the numeric suffix of the associated Jira ticket\n- Branch naming (community): `type/description-1112`\n- All commits must be GPG/SSH signed (enforced by CI)\n\nExample:\n\n```\nfeat: add node reassignment on capacity overflow\n\nNodes at 100% capacity previously returned 503 rather than\nassigning the user to an available node.\n\nCloses #42.\n```\n\n## Project layout\n\n```\nsyncserver/ Main Actix-web server (entry point)\nsyncstorage-db/ Syncstorage database abstraction layer\nsyncstorage-db-common/ Syncstorage shared database types and traits\nsyncstorage-mysql/ MySQL backend for syncstorage\nsyncstorage-postgres/ PostgreSQL backend for syncstorage\nsyncstorage-spanner/ Google Cloud Spanner backend for syncstorage\nsyncstorage-settings/ Syncstorage configuration types\ntokenserver-db/ Tokenserver database abstraction layer\ntokenserver-db-common/ Tokenserver shared database types\ntokenserver-mysql/ MySQL backend for tokenserver\ntokenserver-postgres/ PostgreSQL backend for tokenserver\ntokenserver-auth/ HAWK token generation and verification\ntokenserver-common/ Shared tokenserver utilities\ntokenserver-settings/ Tokenserver configuration types\nsyncserver-common/ Shared server utilities\nsyncserver-db-common/ Shared database utilities\nsyncserver-settings/ Syncserver configuration types\nglean/ Glean telemetry crate\ntools/\n integration_tests/ Pytest-based integration + e2e tests\n tokenserver/ Tokenserver utilities and load tests\n spanner/ Spanner utilities\n hawk/ HAWK token generation utility\n syncstorage-loadtest/ Molotov load tests\ndocker/ Docker Compose files for each backend\nconfig/ Local TOML configuration files\ndocs/ mdBook documentation source\n.github/ GitHub CI Actions and configuration\n```\n\n## Rust builds\n\nThe default feature set targets MySQL, however production uses the Spanner database with MySQL Tokenserver. Backend features are mutually exclusive.\n\n```bash\n# MySQL (default)\ncargo build\n\n# PostgreSQL\ncargo build --no-default-features --features=syncstorage-db/postgres --features=tokenserver-db/postgres --features=py_verifier\n\n# Spanner\ncargo build --no-default-features --features=syncstorage-db/spanner --features=py_verifier\n```\n\n## Rust tests\n\nTests require `RUST_TEST_THREADS=1` (many tests share a real database).\n\n```bash\n# MySQL unit tests (default DATABASE_URL envs set in Makefile)\nmake test\n\n# Specific package\nmake test ARGS=\"--package syncstorage-db\"\n\n# With coverage\nmake test_with_coverage # MySQL\nmake postgres_test_with_coverage\nmake spanner_test_with_coverage\n```\n\n## Clippy / formatting / audit\n\nFormatting checks are necessary before applying any code changes, as they\ncan capture problematic patterns or compiler issues in advance.\n\n```bash\ncargo fmt -- --check # format check\ncargo fmt # auto-format\n\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\n\nmake clippy_release_mysql # release-mode (catches dead code etc.)\nmake clippy_release_postgres\nmake clippy_release_spanner\n\ncargo audit # check dependencies for known CVEs (run by CI on every push)\n```\n\nAfter editing Rust code, run clippy and the unit tests for the affected backend before finishing the task; the `rust-code-smell` skill covers changed files across backends.\n\n## Python tooling\n\nAll Python tooling uses [Poetry](https://python-poetry.org/). The root\n`pyproject.toml` covers the `tools/` tree. Each subdirectory under `tools/` has\nits own `pyproject.toml` for its specific dependencies.\n\nAll of these Python checks must be run with any related change to a Python\nutility or test.\n\n```bash\nmake install # install root dependencies\nmake integration-test # install integration test dependencies\nmake tokenserver # install tokenserver utility dependencies\n```\n\nLint / format / type-check:\n\n```bash\nmake ruff-lint # ruff check tools/\nmake ruff-format # ruff format tools/\nmake ruff-fmt-chk # format diff check\nmake mypy # mypy type checking\nmake pydocstyle # docstring validation\nmake bandit # security linting\n```\n\n## Integration / E2E tests\n\n### Local (against a running server)\n\n```bash\n# Run all integration tests except real FxA e2e\nPYTHONPATH=$(pwd)/tools \\\nSYNC_MASTER_SECRET=secret0 \\\nSYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=http://localhost:6000 \\\nTOKENSERVER_HOST=http://localhost:8000 \\\npoetry -C tools/integration_tests run pytest . --ignore=tokenserver/test_e2e.py\n```\n\nOr via the Makefile shortcut (sets the env vars above):\n\n```bash\nmake run_local_e2e_tests\n```\n\n### Docker-based E2E (full stack)\n\n```bash\nmake docker_run_mysql_e2e_tests\nmake docker_run_postgres_e2e_tests\nmake docker_run_spanner_e2e_tests\n```\n\nEach command spins up two compose stacks (with and without JWK cache), copies\nJUnit XML results out of the container, then tears everything down.\n\n## Running the server locally\n\n```bash\n# MySQL\nmake run_mysql\n\n# Spanner (requires GCP service account key at ./service-account.json)\nmake run_spanner\n```\n\nServer listens on `http://localhost:8000`. Health check: `GET /__heartbeat__`.\n\n## Key environment variables\n\n| Variable | Purpose |\n|---|---|\n| `SYNC_MASTER_SECRET` | HAWK token signing secret |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | Syncstorage DB connection string |\n| `SYNC_TOKENSERVER__DATABASE_URL` | Tokenserver DB connection string |\n| `SYNC_TOKENSERVER__NODE_TYPE` | `mysql` / `postgres` / `spanner` |\n| `SYNC_TOKENSERVER__RUN_MIGRATIONS` | Auto-migrate on startup (`true`/`false`) |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | FxA OAuth endpoint |\n| `RUST_TEST_THREADS` | Set to `1` for all test runs |\n| `RUST_LOG` | Log level (`debug`, `info`, etc.) |\n| `SYNC_TEST_LOG_HTTP` | Set any value to log all HTTP req/resp in integration tests |\n| `SYNC_SERVER_URL` | Integration test base URL (default: `http://localhost:8000`) |\n| `TOKENSERVER_HOST` | Tokenserver host for integration tests |\n| `PYTHONPATH` | Must point to `$(pwd)/tools` when running integration tests |\n\n## Docker Compose files\n\n| File | Purpose |\n|---|---|\n| `docker/docker-compose.mysql.yaml` | MySQL syncstorage + MySQL tokenserver |\n| `docker/docker-compose.postgres.yaml` | PostgreSQL syncstorage + PostgreSQL tokenserver |\n| `docker/docker-compose.spanner.yaml` | Spanner emulator + MySQL tokenserver |\n| `docker/docker-compose.e2e.{mysql,postgres,spanner}.yaml` | E2E test runner overlay |\n| `docker/docker-compose.e2e.jwk-cache.yaml` | Enable JWK caching overlay |\n| `docker/docker-compose.e2e.no-jwk-cache.yaml` | Disable JWK caching overlay |\n\n## Documentation\n\n```bash\nmake doc-install-deps # install mdBook + mdBook-mermaid\nmake doc-watch # live preview at localhost\nmake doc-prev # build and serve\nmake doc-test # validate markdown\n\nmake api-prev # generate OpenAPI spec + serve Swagger UI on :8080\n```\n\n\n## Test structure\n\n- `tools/integration_tests/conftest.py` — pytest fixtures only (`st_ctx`)\n- `tools/integration_tests/helpers.py` — retry helpers, auth state, `switch_user`\n- `tools/integration_tests/test_storage.py` — storage protocol tests\n- `tools/integration_tests/tokenserver/conftest.py` — tokenserver fixtures only\n- `tools/integration_tests/tokenserver/helpers.py` — tokenserver DB/auth helpers\n- `tools/integration_tests/tokenserver/test_*.py` — tokenserver tests\n\n## MCP integrations\n\nTwo MCP servers are configured for this repository: **Atlassian Jira** (`mcp__atlassian__`) and **Sentry** (`mcp__sentry__`).\n\nUse the skills below to query them. Do not call MCP tools ad-hoc without reading the shared context first.\n\n| Skill | Invocation | Purpose |\n|---|---|---|\n| Jira status | `/jira-status [me\\|epic KEY\\|epics\\|sprint\\|summary ...\\|create ...\\|update STOR-#### ...]` | Task landscape, epics, personal view, summaries, create/edit tickets |\n| Sentry health | `/sentry-health [new\\|regressions\\|volume\\|full]` | Production error trends, regressions, new issues |\n\n**Shared context:** `.claude/mcp-context.md` contains project IDs, JQL patterns, Sentry project slug, what to flag, and known noise. Skills read this at runtime. Read it before any ad-hoc MCP query.\n\n**Identity when using these tools:** You are acting as both engineer and engineering manager. Surface patterns, not just raw lists. Distinguish \"needs action now\" from \"worth monitoring\". Do not fabricate data — if a query returns nothing, say so.\n\n**Jira project:** `STOR` — `https://mozilla-hub.atlassian.net`\n**Sentry project:** `syncstorage-prod` — org `mozilla`\n","docs/src/tools/load-testing.md":"# Load Testing\n\n## Quickstart: load test the change stream system (STOR-628)\n\nFour commands. Needs Docker, plus `curl` and `jq`. No GCP account, no\ncredentials, no VM.\n\n```console\nmake loadtest-offload-image # once: builds the server image\nmake loadtest-offload-changestream # run the load test\nmake loadtest-offload-report # did the pipeline keep up?\nmake loadtest-offload-down # tear it all down\n```\n\nThe report is the answer. Three numbers matter:\n\n| Look for | Meaning |\n| --- | --- |\n| `filtered` non-zero | The pipeline correctly dropped change records that have no payload file attached. This is the thing STOR-628 is asking about. |\n| `handler errors` zero | Nothing failed while finalizing or deleting objects. |\n| `noop_skips` zero | Nothing inert slipped past the filter. |\n\nIt prints `OK: pipeline drained clean` at the end, or a list of `FAIL` lines\nexplaining what went wrong, and exits non-zero on a real problem so it can gate\na script.\n\nOne thing to expect: a `NOTE` saying `batch_commit_skips` is 0. That is a known\nemulator limitation rather than your run going wrong. See\n[The emulator drops transaction tags](#the-emulator-drops-transaction-tags).\n\nFor a real run rather than a smoke test, pass molotov your own arguments:\n\n```console\nmake loadtest-offload-changestream \\\n MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 3600 -v\"\n```\n\nTo load test the handler read/write path instead (STOR-629), swap in\n`make loadtest-offload-handler`. Same report, same teardown.\n\nEverything below is detail you only need when a run looks wrong, or when you\nneed real latency numbers and have to go to GCP.\n\n---\n\n## Summary\n\nThis is the runbook for load testing syncstorage, with the offloaded-payload\npipeline as the main case. It replaces the Confluence page \"Syncstorage &\nTokenserver Runbooks\", whose molotov and locust instructions had become\ninterleaved and unreadable.\n\nThere are two load tests in the repo and they are unrelated:\n\n| | Target | Tool | Location |\n| --- | --- | --- | --- |\n| Syncstorage | storage read/write API | molotov | `tools/syncstorage-loadtest/` |\n| Tokenserver | `/1.0/sync/1.5` token issuance | locust | `tools/tokenserver/loadtests/` |\n\nEverything below is about the syncstorage/molotov one unless it says otherwise.\nFor tokenserver, see [Tokenserver load tests](#tokenserver-load-tests-locust).\n\nTwo tickets drive the offload work, and they want different traffic shapes:\n\n| Ticket | Goal | Traffic shape |\n| --- | --- | --- |\n| STOR-629 | Performance of the handler's GCS read/write path | 100% offloaded, every payload expanded |\n| STOR-628 | The change stream to reconciler pipeline, including the filtering of NULL `payload_link` records | Mixed offloaded and inline |\n\nThe distinction matters. STOR-628 is only a real test if some writes are\n*not* offloaded, because the record it is checking for is the one the\npublisher's filter is supposed to drop.\n\n---\n\n## Which rig to use\n\n```mermaid\nflowchart TB\n q1{\"What are you
measuring?\"}\n q1 -->|\"pipeline correctness:
does every object get
finalized, is the filter
dropping NULL records\"| local[\"Local docker rig
(emulators)\"]\n q1 -->|\"real latency and
throughput numbers
for a capacity decision\"| gcp[\"GCP dev environment\"]\n local --> both[\"Run local first either way:
it catches wiring bugs in
minutes with no GCP spend\"]\n```\n\nThe local rig runs the whole pipeline on emulators. It is the right default:\nit needs no GCP project, no credentials and no VM, and it answers every\ncorrectness question. What it cannot give you is a trustworthy latency\nnumber, because fake-gcs-server on a loopback interface is nothing like GCS\nover the network, and the Spanner emulator is single-node and in-process.\n\nUse GCP dev when the question is \"how fast\" rather than \"does it work\".\n\n---\n\n## Local docker rig\n\n### What it stands up\n\nThe rig is the existing reconciliation e2e stack plus a molotov container and\na metrics sink. Compose overlay: `docker/docker-compose.loadtest.yaml`.\n\n```mermaid\nflowchart LR\n lt[\"loadtest
molotov\"]\n\n subgraph fg[\"Foreground: the request path (STOR-629)\"]\n ss[\"syncserver
offload enabled\"]\n end\n\n gcs[(\"fake-gcs
test-payloads
filesystem backend\")]\n db[(\"Spanner emulator
bsos, batch_bsos
payload_link_changes\")]\n\n subgraph bg[\"Background: the cleanup arm (STOR-628)\"]\n pub[\"payload-link-publisher
stands in for Dataflow\"]\n ps[\"Pub/Sub emulator\"]\n rec[\"payload-reconciler
long-running mode\"]\n end\n\n drop([\"dropped: inert records
payload_link NULL both sides\"])\n sd[\"statsd-exporter
:9102\"]\n rep[\"loadtest-report.sh
the verdict\"]\n\n lt -->|\"Hawk, direct access
no tokenserver or FxA\"| ss\n ss -->|\"upload payload\"| gcs\n ss -->|\"store gs:// in payload_link\"| db\n gcs -->|\"download on read\"| ss\n db -->|\"change stream\"| pub\n pub -.->|filtered| drop\n pub -->|published| ps\n ps --> rec\n rec -->|\"finalize: committed=true
delete: orphans\"| gcs\n\n ss -.->|dogstatsd| sd\n pub -.->|dogstatsd| sd\n rec -.->|dogstatsd| sd\n sd -->|\"scrape counters\"| rep\n gcs -->|\"list objects,
check committed + customTime\"| rep\n```\n\nThe two halves map onto the two tickets. Everything left of the change stream\nis STOR-629's territory, and it costs the user latency. Everything right of it\nis STOR-628's, and no user waits on it, but a wrong decision there deletes a\nfile a live row still needs.\n\nNote where the report gets its evidence: counters from the statsd sink, and\nobject state read straight out of the bucket. It never trusts molotov's exit\nstatus, because the cleanup arm runs after the response is sent and can fail\nlong after the client saw a 200.\n\n| Service | Role |\n| --- | --- |\n| `sync-db` | Spanner emulator, carrying the `payload_link_changes` change stream from `schema.ddl` |\n| `pubsub-emulator` | Pub/Sub |\n| `fake-gcs` | GCS, on its **filesystem** backend for this rig |\n| `reconciliation-setup` | One-shot: creates the topic, subscription and bucket |\n| `payload-link-publisher` | Python change-stream reader, stands in for the Dataflow job |\n| `payload-reconciler` | Drains Pub/Sub, finalizes and deletes GCS objects |\n| `syncserver` | Offload enabled, limits raised |\n| `statsd` | `prom/statsd-exporter`, so the publisher and reconciler counters are readable |\n| `loadtest` | molotov |\n\nThe publisher is the Python one, not the Java Dataflow template. It publishes\nthe identical JSON wire format, so everything downstream of it is\nproduction code. If you need to reproduce something Java-specific, layer\n`docker-compose.e2e.reconciliation.java.yaml` on top; see\n[Reconciliation Pipeline](payload_link_reconciler.md).\n\nOn the NULL-filtering half of STOR-628, be precise about what the local rig\nproves. The filter exists twice: `isPayloadLinkActionable` in the Java flex\ntemplate, and `is_payload_link_actionable` in\n`payload-link-publisher-py/utils.py`, which is written to mirror it. A local\nrun exercises the **Python** one. That is a real test of the filter's logic\nand of everything downstream, but it is not a test of the Java code that runs\nin production. To exercise the shipping filter, either use the Java overlay\nlocally or run against GCP dev, where the Dataflow job is the publisher.\n\n### Prerequisites\n\n- Docker running, with roughly 8 GB available to the VM. Expanded payloads\n are held in memory in several places at once.\n- `curl` and `jq` on the host, for the report script.\n- Nothing else. No GCP credentials, no `service-account.json`.\n\n### The stale publisher image trap\n\nRead this before trusting any local result.\n\n`payload-link-publisher` is the only service in the stack built from a compose\n`build:` context rather than pulled or shared with `app:build`. Plain\n`docker compose up` reuses whatever `docker-payload-link-publisher` image is\nalready in your local cache and **will not rebuild it when its source\nchanges**. So a publisher image built before a change to\n`payload-link-publisher-py/` keeps running against new server code, silently.\n\nThis is not hypothetical. It is how the rig behaved on first use here: a\nfour-week-old publisher image had no `transaction_tag` support at all, so no\nrecord ever carried the `batch_commit` tag, so the reconciler treated every\nbatch-commit handoff as a genuine `batch_bsos` removal and deleted GCS objects\nthat committed `bsos` rows still pointed at. The symptom was HTTP 500s on\nreads, `payload_reconciler.batch_commit_skips` flat at zero, and\n`gcs_404{op=finalize}` running nearly 1:1 with `finalizes`. It looked exactly\nlike the STOR-657 production bug, and it was entirely a stale image.\n\nThe `loadtest-offload-*` targets therefore pass `--build` on every `up`. The\nfirst build is slow from a cold cache; afterwards it is cached and cheap.\n\nTwo things follow. If you drive compose by hand instead of through the\nMakefile, pass `--build` yourself. And note that\n`make docker_run_reconciliation_e2e_tests` does **not** pass it, so a local run\nof the reconciliation e2e suite can exercise a stale publisher. CI is unaffected\nbecause its cache starts empty.\n\nThe tell, if you suspect it:\n\n```console\ndocker compose ... exec payload-link-publisher grep -c transaction_tag /app/publisher.py\n```\n\nZero means the image predates transaction-tag support.\n\n### Run it\n\n```console\nmake loadtest-offload-image # build app:build (spanner backend), once\n\nmake loadtest-offload-handler # STOR-629\nmake loadtest-offload-report\n\nmake loadtest-offload-changestream # STOR-628\nmake loadtest-offload-report\n\nmake loadtest-offload-down # tear down, including the fake-gcs volume\n```\n\n`loadtest-offload-report` waits for the reconciler to stop making progress\nbefore reporting, so run it straight after the load test rather than trying\nto guess a sleep. It exits non-zero if it finds a problem, which makes it\nusable as a gate.\n\nBoth scenario targets take `MOLOTOV_ARGS`:\n\n```console\nmake loadtest-offload-handler MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 600 -v\"\n```\n\nStart small. Every worker can be holding a multi-MiB payload, on both the\nclient and the server, so worker count multiplies memory rather than just CPU.\nFive workers writing 10 MiB records is already a meaningful amount of traffic\nthrough this stack.\n\n### Reading the report\n\n```text\nRECONCILER (payload_reconciler.*)\n payload_reconciler_finalizes 1843\n payload_reconciler_orphan_deletes 291\n payload_reconciler_batch_commit_skips 412\n...\nGCS OBJECTS (test-payloads)\n objects 1552\n committed=true 1552\n committed!=true 0\n customTime pinned 1552\n...\nVERDICT\n OK: pipeline drained clean.\n```\n\nWhat each number should look like:\n\n| Metric | Healthy | If it is wrong |\n| --- | --- | --- |\n| `published` | Tracks actionable change records | Zero means the publisher is not reading the change stream at all |\n| `filtered` | **Non-zero on a mixed run.** This is the positive evidence for STOR-628: the publisher dropped records with `payload_link` NULL on both sides | Zero on a mixed run means either the filter is passing inert records through, or every write happened to be offloaded and the path went untested |\n| `finalizes` | Tracks offloaded write volume | Zero with objects in the bucket means the change stream, publisher or Pub/Sub leg is broken |\n| `orphan_deletes` | Non-zero, tracks overwrites and deletes | Zero means the load test never deleted anything; check `DISABLE_DELETES` |\n| `batch_commit_skips` | **Zero on the local rig** -- see the emulator limitation below. Non-zero is what you require on GCP dev | Zero on *GCP dev* means the transaction tag is not reaching the change stream, which is the STOR-657/668 bug class |\n| `noop_skips` | **Zero** | Non-zero is the STOR-628 regression: the filter let through a record with `payload_link` NULL on both sides |\n| `errors kind:handler` | **Zero** | Non-zero means messages went unacked. In prod five failures on one message routes it to the DLQ |\n| `committed!=true` | Zero once drained | A small tail is normal if you cut the run off mid-flight. A large or growing count means the pipeline is not keeping up with the write rate |\n\n`committed=true` count and `customTime pinned` count must agree. Finalize sets\nboth in a single `blob.patch()`, so a divergence means something other than\nthe reconciler is writing object metadata.\n\nOne thing that surprises people: even the \"100% offloaded\" STOR-629 run\nproduces a non-zero `filtered` count. `OFFLOAD_COLLECTIONS` only governs the\nbatch-write arm, while the load test also PUTs `meta/global` and POSTs\n`clients` on every scenario pass. Those two are never in the offload list, so\nthey generate exactly the inert NULL-on-both-sides records the filter is\nthere to drop. So STOR-629 covers the filter incidentally, and STOR-628 is\nstill the run that covers it deliberately, at volume, alongside offloaded\ntraffic in the same stream.\n\n### Reading the 404 counters, and why they are the whole signal\n\nThe reconciler treats `404 NotFound` as success on both of its operations.\nFrom `reconcile_payload_links.py`, a finalize whose target is gone logs at\n`debug`, bumps `gcs_404{op=finalize}`, and **acks the message**. Same shape for\ndelete. [Reconciliation Pipeline](payload_link_reconciler.md) states the\nrationale plainly: \"Both reconciler actions are idempotent, so a redelivered\nrecord is harmless.\"\n\nThat is correct and deliberate, and it has a consequence worth internalising\nbefore you read a load test result. Because a missing object is success, a run\nthat is silently destroying data produces:\n\n- `errors` `kind:handler` at **zero**\n- nothing in the DLQ\n- no `WARN` or `ERROR` in the reconciler log, only `debug`\n- an elevated `gcs_404` counter, and nothing else\n\nSo on the offload path `gcs_404{op=finalize}` is not a nice-to-have gauge. It\nis the *only* signal that separates a healthy pipeline from one that is\ndeleting live payloads. Everything else looks green.\n\nThat existing doc's failure-modes table gives two causes for a sustained\n`gcs_404{op=finalize}`:\n\n| Documented cause | Applies to a local run? |\n| --- | --- |\n| The lifecycle rule reclaimed the object before it was finalized | **No.** The 30 day `daysSinceCustomTime` rule is a bucket policy in `bucket.tf`. fake-gcs has no lifecycle policy at all, so nothing can reap an object locally. |\n| The same message was redelivered after a successful prior run (the at-least-once tax) | **No**, not at volume. Redelivery is a background trickle, not a rate approaching `finalizes`. |\n\nMeasured on these runs, neither explains what was observed:\n\n| Run | `finalizes` | `gcs_404{op=finalize}` | Ratio |\n| --- | --- | --- | --- |\n| STOR-629 handler | 930 | 716 | 77% |\n| STOR-628 changestream | 1853 | 1589 | 86% |\n\nA ratio near 1:1 is a third cause that the table does not list: **the\nreconciler deleted the object itself**, as an orphan, and the finalize for that\nsame object then 404s. Locally that is the emulator dropping the\n`batch_commit` transaction tag (see below), so every batch-commit handoff is\nmisread as a genuine `batch_bsos` removal.\n\nThe signature to recognise, wherever you see it:\n\n```text\nbatch_commit_skips 0 <- tag never arrived\ngcs_404{op=finalize} ~= finalizes <- finalizing objects already deleted\norphan_deletes high <- the deletes that did the damage\nerrors kind:handler 0 <- and nothing reports a problem\n```\n\n`gcs_404{op=delete}` is the benign one, and the existing doc's read of it holds\nwithout qualification: the object was already gone, the operation is idempotent\nby design, and a redelivery or a concurrent cleanup produces it. It carries no\ndata-loss implication on its own.\n\nTwo things follow for the alerting the reconciler doc already recommends. Its\nsuggestion to watch \"`gcs_404` `op:finalize` rising as a share of `finalizes`\"\nis the right alert, and this gives it a concrete threshold to reason about: a\nlow single-digit share is the redelivery tax, while anything approaching\n`finalizes` means objects are being deleted out from under committed rows. And\npairing it with `batch_commit_skips == 0` distinguishes the batch-commit cause\nfrom a lifecycle-window problem.\n\n### Knobs\n\nServer side, set on the `syncserver` container:\n\n| Variable | Default in the rig | Meaning |\n| --- | --- | --- |\n| `LOADTEST_OFFLOAD_COLLECTIONS` | `bookmarks,history` | Which collections syncserver offloads. **This is what decides mixed vs fully offloaded.** |\n| `LOADTEST_COLLECTION_LIMITS` | 10 MiB record / 20 MiB post / 21 MiB request, for `bookmarks` and `history` only | Per-collection raised limits, as a JSON map. Keys must be a subset of the offload list -- see below |\n| `LOADTEST_GCS_MAX_CONCURRENCY` | 4 | Concurrent GCS ops within one batch request. The main server-side throughput knob, worth sweeping |\n| `LOADTEST_DB_POOL_MAX_SIZE` | 40 | The stock 10 saturates first once workers climb |\n\n### Raise limits per collection, never globally\n\nThis is the one thing that will bite you, so it is worth stating plainly.\n\nSpanner caps a single `STRING(MAX)` value at **2621440 bytes (2.5 MiB)**. That\nhard cap is the reason this whole project exists. Raising the *global*\n`max_record_payload_bytes` above it does not fail at the API, it fails in the\ndatabase, on every collection that is not offloaded:\n\n```text\nFAILED_PRECONDITION New value exceeds the maximum size limit for\nthis column: bsos.payload, size: 10485760, limit: 2621440\n```\n\nAn offloaded collection is exempt, because its payload goes to GCS and the row\nstores only a `gs://` URL in `payload_link`, leaving `payload` NULL. So a limit\nabove 2.5 MiB is only ever correct for a collection that is in\n`gcs_payload_offload_collections`. That is exactly what the per-collection\noverride exists for, and how production raises it:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":10485760,\"max_post_bytes\":20971520,\"max_request_bytes\":22020096}}'\n```\n\nLeave the globals alone. Any collection not named in the map keeps stock\n2.5 MiB behaviour and writes inline safely. `max_post_bytes` and\n`max_request_bytes` are overridable per collection too; actix's payload config\nis sized from the largest `max_request_bytes` across all overrides\n(`ServerLimits::effective_max_request_bytes`), so a big body is not rejected at\nthe resource level before per-collection enforcement runs.\n\nThere is no per-collection `max_total_bytes`. The global default (clamped to\n`MAX_SPANNER_LOAD_SIZE`, 100 MB, by `Settings::normalize` on spanner) is\nalready generous, so the rig leaves it untouched.\n\nOn a mixed run this arrangement gives you exactly the right traffic for free:\noffloaded collections take expanded payloads, and inline collections get\n`LARGE_PAYLOAD_PROB` applied but capped at 2.5 MiB by\n`payload_target_length`, which is realistic \"large but not offloaded\" traffic.\n\nClient side, set on the `loadtest` container:\n\n| Variable | Meaning |\n| --- | --- |\n| `LARGE_PAYLOAD_PROB` | Fraction of BSOs given an expanded payload. `1.0` for STOR-629 |\n| `LARGE_PAYLOAD_SIZE` | Explicit target size in bytes. Unset means \"use the server's `max_record_payload_bytes`\" |\n| `OFFLOAD_COLLECTIONS` | Restricts batch writes to these collections. **Leave unset for mixed runs** |\n| `DISABLE_DELETES` | Set truthy to suppress the DELETE arm. Leave off, or the reconciler's `orphan_deletes` branch is never exercised |\n\nThe two collection lists are easy to confuse:\n\n- `LOADTEST_OFFLOAD_COLLECTIONS` (server) decides which collections get\n offloaded.\n- `OFFLOAD_COLLECTIONS` (client) decides which collections get *written to*.\n\nSetting the client one to a single collection sends every batch write there,\nwhich is what you want for STOR-629 and exactly what you must not do for\nSTOR-628. For a mixed run, leave the client variable unset so writes spread\nacross molotov's default five (`bookmarks`, `forms`, `passwords`, `history`,\n`prefs`) and set the server list to a subset of those.\n\n### Per-collection limit overrides\n\nProduction raises limits for one collection rather than globally, via a JSON\nmap. The load test resolves limits per target collection out of\n`/info/configuration`, preferring a collection's override, so this path is\ncovered too:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":20971520,\"max_post_bytes\":26214400,\"max_request_bytes\":26218496}}'\n```\n\n### Known limits of the local rig\n\n- **The emulator does not carry transaction tags, so the batch-commit skip\n cannot be validated locally.** This one is load bearing; details below.\n- **No DLQ.** `docker/reconciliation-setup.sh` creates the topic and\n subscription but no dead-letter topic, so dead-letter routing after five\n failed deliveries cannot be exercised locally. Verify that on GCP dev.\n- **No Dataflow.** The Python publisher replaces the Java job. Autoscaling,\n partition-split handling under real load and the connector's Spanner\n metadata writes are all Dataflow behaviours that only appear on GCP.\n- **Latency is not meaningful.** fake-gcs over loopback and a single-node\n Spanner emulator. Use it for correctness and relative comparisons only.\n- **No lifecycle policy.** The 30 day `daysSinceCustomTime` rule that reaps\n unfinalized objects is a bucket policy, so the local rig can only check\n that `customTime` was pinned, not that the reaping behaves.\n\n### The emulator drops transaction tags\n\nThe Spanner emulator emits the `transaction_tag` field on change stream\nDataChangeRecords but never populates it. Measured on emulator v1.5.52: the\nDataChangeRecord struct has arity 13, so index 11 (`transaction_tag`) exists\nand the publisher reads it correctly, but every record carries the empty\nstring, including the `batch_bsos` DELETE that *is* the batch-commit handoff.\n\nThat single gap has a large consequence. The reconciler decides whether to keep\nor delete a `batch_bsos` removal's object by checking\n`transactionTag == \"batch_commit\"` (see\n[Reconciliation Pipeline](payload_link_reconciler.md)). With the tag always\nempty, every batch-commit handoff looks like a genuine removal, so the\nreconciler deletes the GCS object that the just-committed `bsos` row now points\nat. The BSO becomes unreadable and syncserver returns a 500 backed by a GCS 404.\n\nSo on the local rig, expect all of this, and do not read it as a regression:\n\n- `payload_reconciler.batch_commit_skips` flat at zero.\n- `gcs_404{op=finalize}` running close to 1:1 with `finalizes`.\n- Objects missing for BSOs written in a batch create/append request.\n- A residual molotov failure rate, because roughly 50% of its POST scenarios\n are transactional batches. Reads of those BSOs 500.\n\n`docker/batch-commit-probe.py` reproduces it in isolation: one two-request\ntransactional batch, then a read of both BSOs, printing the bucket contents at\neach stage.\n\n```console\nmake loadtest-offload-up\nSYNCSTORAGE_RS_IMAGE=app:build docker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n -f docker/docker-compose.loadtest.yaml \\\n run --rm --entrypoint python3 \\\n -v \"$PWD/docker/batch-commit-probe.py:/probe.py:ro\" loadtest /probe.py\n```\n\nAdding `--no-deps` to that command keeps the reconciler stopped, and the probe\nthen passes with both payloads intact. That is the control which proves the\nreconciler is the thing deleting the object, rather than the batch commit\nlosing the `payload_link`.\n\nWhat this does **not** tell you is whether production is affected. The app side\nis correct: `handlers.rs` calls\n`with_transaction_tag(BATCH_COMMIT_TRANSACTION_TAG)` on the commit path, and\nreal Spanner does record transaction tags on change streams. The evidence\npoints at an emulator gap rather than a code defect, but the local rig cannot\nclose that question. **Confirm the batch-commit skip on GCP dev**, where the\nDataflow job reads a real change stream: run the probe there and require\n`payload_reconciler.batch_commit_skips` to be non-zero.\n\nTwo follow-ups worth filing rather than leaving implicit:\n\n1. There is no e2e test for the batch-commit handoff.\n `tools/integration_tests/test_payload_link_reconciliation.py` covers upload,\n update and delete, but not a transactional batch. That is presumably because\n it cannot pass against the emulator, which is exactly the point: the gap is\n invisible in CI.\n2. If the Java overlay behaves the same way, the emulator is confirmed as the\n cause. That is a cheap check and worth doing before spending GCP time.\n\n---\n\n## GCP dev environment\n\nUse this for real numbers. The dev pipeline is fully provisioned; see\n[GCP Infrastructure](payload-offload-infrastructure.md) for what lives where\nand which repo owns it. That page is the authority on the resources\nthemselves. The diagram below is only the load-testing view: where the\ngenerator sits, what it traverses, and where you go to observe each leg.\n\n```mermaid\nflowchart TB\n subgraph gen[\"Load generator (you create this)\"]\n vm[\"C4D VM, Ubuntu
molotov + load_test.pem
in moz-fx-sync-nonprod\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (shared GKE)\"]\n ts[\"tokenserver
trusts your public JWK\"]\n ss[\"syncserver
needs GCS_PAYLOAD_BUCKET +
OFFLOAD_COLLECTIONS + raised limits\"]\n rec[\"payload-reconciler
cronjob, ~5 min cadence\"]\n end\n\n subgraph v1[\"moz-fx-sync-nonprod-904c (GCPv1, cloudops-infra)\"]\n sp[(\"Spanner syncdb-dev
change stream:
payload_link_changes\")]\n end\n\n subgraph tenant[\"moz-fx-sync-nonprod (tenant, webservices-infra)\"]\n df[\"Dataflow flex template
the REAL Java filter\"]\n ps[\"Pub/Sub
payload-link-changes\"]\n dlq[\"Pub/Sub DLQ
after 5 failed deliveries\"]\n b[(\"GCS
sync-nonprod-dev-
syncstorage-payloads
30d lifecycle rule\")]\n end\n\n vm -->|\"1. self-signed JWT\"| ts\n ts -->|\"2. Hawk token\"| vm\n vm -->|\"3. storage reads + writes\"| ss\n ss -->|\"upload / download\"| b\n ss -->|\"payload_link\"| sp\n sp -->|\"change stream read\"| df\n df -->|publish| ps\n ps -->|\"pull subscription\"| rec\n ps -.-> dlq\n rec -->|\"finalize / delete\"| b\n```\n\nThree things this buys you that the local rig cannot:\n\n| Leg | Why it only exists here |\n| --- | --- |\n| Dataflow | The local rig substitutes a Python publisher. This is the actual Java `isPayloadLinkActionable` filter that ships, plus autoscaling and partition-split behaviour under real load. |\n| Real Spanner | Populates `transaction_tag`, so the batch-commit skip can finally be verified. Also the only place to measure what the change stream costs in Spanner storage, which has never been quantified. |\n| DLQ and lifecycle rule | Neither exists locally. `reconciliation-setup.sh` creates no dead-letter topic, and fake-gcs has no lifecycle policy. |\n\n### Load generator VM\n\nGenerate load from inside GCP, not from a laptop. A laptop's uplink is the\nbottleneck long before the server is, and multi-MiB payloads make that worse.\n\n1. Console -> project `moz-fx-sync-nonprod` -> Compute Engine -> VM instances\n -> Create instance.\n2. Machine type: `C4D` series. OS: Ubuntu, not the Debian default.\n3. The project already has IAP TCP forwarding and the firewall rules. If SSH\n is refused, check the project inherited them; see the SRE space's IAP page.\n\nSet the VM up:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\ngit clone https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\ncurl -sSL https://install.python-poetry.org | python3 -\npoetry install\n```\n\nDocker is an alternative to poetry on the VM, and avoids the Python version\ndance:\n\n```console\ndocker build . --tag syncstorage-loadtest:local\ndocker run --rm -e SERVER_URL -e OAUTH_PRIVATE_KEY_FILE \\\n -e LARGE_PAYLOAD_PROB -e OFFLOAD_COLLECTIONS \\\n -v /path/to/load_test.pem:/keys/load_test.pem:ro \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\"\n```\n\n### Authentication\n\nThree modes, in `storage/auth.py`. For a dev run use self-signed JWTs: no FxA\naccounts to create or clean up, and the token lifetime is yours to set.\n\n```console\npoetry run ./generate-keys.sh\n```\n\nThat writes `load_test.pem` (private, stays on the VM) and `jwk.json`\n(public). The dev tokenserver has to trust that public key, which means\nlanding the JWK in the tokenserver FxA JWK configuration in\n`webservices-infra` under `sync/k8s/`. That is a PR plus an ArgoCD sync, so do\nit before you book time for the run.\n\nThen:\n\n```console\nSERVER_URL=\"https://\" \\\n OAUTH_PRIVATE_KEY_FILE=./load_test.pem \\\n LARGE_PAYLOAD_PROB=1.0 \\\n OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\n```\n\nDirect-access mode (`SERVER_URL` with a `#secret` fragment) bypasses\ntokenserver entirely and is what the local rig uses. It needs the deployed\nmaster secret, so do not use it against a shared environment.\n\n### Server side prerequisites on dev\n\nThe pipeline being provisioned is not the same as offload being on. Offload\ndoes nothing until both of these are set on syncserver, and expanded payloads\ndo nothing until the limits are raised:\n\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET`\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS`\n- `SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES` and the\n `MAX_POST_BYTES` / `MAX_REQUEST_BYTES` / `MAX_TOTAL_BYTES` that go with it\n\nAll four are Helm values in `webservices-infra` under `sync/k8s/sync/`. Raise\nany front-end proxy body-size limit to match, or requests get a 413 before\nthey reach the server.\n\nConfirm what actually took effect by reading it back from the server rather\nthan from the values file:\n\n```console\ncurl -s https:///1.5//info/configuration | jq .\n```\n\nThe load test sizes its batches from exactly this response, so if the limits\nare not in there, the run is not testing expanded payloads no matter what the\nenvironment variables say.\n\n### Verifying a dev run\n\nEverything the local report script checks, plus the parts that only exist on\nGCP:\n\n| What | Where |\n| --- | --- |\n| Reconciler counters (`payload_reconciler.*`) | The metrics pipeline, same as any other service metric |\n| Oldest unacked message age on `payload-link-reconciler-sub` | Pub/Sub console. The most direct measure of finalize latency; should sit in minutes |\n| Anything at all in `payload-link-changes-dlq` | Pub/Sub console. A message only lands there after five failed deliveries, so it always needs a human |\n| Dataflow job lag, autoscaling, worker errors | Dataflow console |\n| Spanner CPU and change-stream storage cost | Spanner console on `moz-fx-sync-nonprod-904c` |\n| Object count and finalize state in the bucket | GCS console, or `gcloud storage ls -L` |\n\nChange-stream storage cost is worth capturing deliberately. It has not been\nquantified, and the plan for prod is to measure it before enabling offload\ntraffic, so a dev run is the cheapest place to get a number.\n\n---\n\n## What to record\n\nStandard long run is 60 to 65 minutes. Log results to the shared documents\nrather than only in a ticket:\n\n- Syncstorage molotov runs: \"Sync Load Test Run Document\"\n- Tokenserver locust runs: the tokenserver load test results document\n\nCapture at minimum: the git SHA under test, molotov arguments, the effective\n`/info/configuration`, the server-side offload collection list, molotov's\nown summary (request counts, failures, RPS), and the reconciler counters plus\nGCS finalize state from the report.\n\n---\n\n## Tokenserver load tests (locust)\n\nSeparate tool, separate target, unrelated to payload offload. Locust rather\nthan molotov, and it has a web UI.\n\n```console\ncd tools/tokenserver/loadtests\npoetry install\npoetry run ./generate-keys.sh\nOAUTH_PRIVATE_KEY_FILE=./load_test.pem poetry run locust # UI on :8089\n```\n\nRunning on a VM, tunnel the UI out over IAP:\n\n```console\ngcloud compute ssh --zone \"\" \"\" --project \"\" \\\n --tunnel-through-iap -- -L 8090:localhost:8089\n```\n\nThen open `localhost:8090`. See `tools/tokenserver/loadtests/README.md` for\nthe rest.\n\n---\n\n## Gaps worth knowing about\n\n- **GCS time can be separated from request time for the collection routes.**\n `storage.gcs.payload.batch` is the wall time one `get_collection` or\n `post_collection` request spent on GCS across all of its objects, tagged with\n `handler`, and the request middleware emits `request.duration` tagged by\n route and method. Dividing `rate(storage.gcs.payload.batch_sum{handler=\"post_collection\"})`\n by the `request.duration` `rate(_count)` for `/1.5/{uid}/storage/{collection}`\n with `method=\"POST\"` (and `get_collection` against `GET` likewise) gives the\n GCS share of an average collection write or read. It is still rough, because\n the denominator counts requests whose payloads stayed inline, under the\n offload threshold. The single-BSO routes (`/{bso}` GET and PUT) emit only\n the per-object `.upload` / `.download` timings, which carry no route tag, so\n they cannot be attributed the same way; measure those with a separate load\n shape. Either way a STOR-629 run can say \"GCS accounted for about N ms of a\n collection request\" rather than only \"the handler got slower\".\n- **No per-collection metric dimension**, so a mixed run cannot separate\n offloaded from inline request latency from the server side either. Working\n around it means either running the two shapes separately and comparing, or\n adding a collection tag.\n","docs/src/tools/make_hawk_token.md":"# Documentation for `make_hawk_token.py`\n\n## Summary\n\n`make_hawk_token.py` is a utility script for generating a [Hawk](https://github.com/mozilla/hawk) authentication token and header for use when testing the syncstorage-rs API. It derives a token and signing key from a shared secret and produces either a raw token tuple or a ready-to-use `Authorization: Hawk ...` header.\n\nThe script lives in [`tools/hawk/`](https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/hawk).\n\n---\n\n## Setup\n\n### Prerequisites\n\n- Python ≥ 3.12\n- [Poetry](https://python-poetry.org/docs/#installation) installed\n\n### Install dependencies\n\n```shell\ncd tools/hawk\npoetry install\n```\n\n---\n\n## Usage\n\n```shell\npoetry run python make_hawk_token.py [options]\n```\n\nBy default (no arguments), a token is generated for `http://localhost:8000/1.5/1/storage/col2/` using a built-in test secret.\n\n### Options\n\n| Option | Description | Default |\n|---|---|---|\n| `--uri` | Request URI path | `/1.5/1/storage/col2/` |\n| `--secret` | Shared `SYNC_MASTER_SECRET` value | *(built-in test value)* |\n| `--method` | HTTP method for the request | `GET` |\n| `--as_header` | Print a ready-to-use `Authorization: Hawk ...` header | *(off)* |\n\nUse `-h` for the full option list.\n\n---\n\n## Examples\n\n### Generate a token for a GET request\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET\n```\n\n### Generate a ready-to-use Authorization header\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nOutput:\n```\nAuthorization: Hawk id=\"...\", ts=\"...\", nonce=\"...\", mac=\"...\"\n```\n\n### Non-GET methods\n\nFor methods other than GET (e.g. PUT, POST, DELETE), pass `--method`:\n\n```shell\npoetry run python make_hawk_token.py \\\n --method PUT \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nSee [`tools/examples/put.bash`](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for a full curl example using this output.\n\n---\n\n## Notes\n\n- The `--secret` value must match the `SYNC_MASTER_SECRET` configured in the running syncstorage-rs instance.\n- The generated token is scoped to the exact URI and method — a token generated for `GET /foo` is not valid for `PUT /foo`.\n","docs/src/tools/payload-offload-infrastructure.md":"# Payload Offload Infrastructure\n\n## Summary\n\nThis document is a map of the GCP infrastructure behind offloaded BSO\npayloads: which project each resource lives in, which repo defines it,\nand what applies it. It covers where things run rather than how the\npipeline behaves. For the behaviour of the change stream, Dataflow job,\nPub/Sub topics and reconciler, see\n[Payload Link Reconciler](payload_link_reconciler.md).\n\nTracking epic: STOR-372, \"Expand spanner storage beyond 2.5MB\".\n\n## The two-project split\n\nThe single most confusing thing about this setup is that Spanner and\neverything else live in different GCP projects, under different\ngenerations of Mozilla's cloud tooling.\n\n| | Spanner | Everything else |\n| --- | --- | --- |\n| Project (dev) | `moz-fx-sync-nonprod-904c` | `moz-fx-sync-nonprod` |\n| Generation | GCPv1 (legacy) | GCPv2 (MozCloud tenant) |\n| Managed from | `cloudops-infra` | `webservices-infra` + `global-platform-admin` |\n\nThe Spanner instances stayed in GCPv1 on purpose. Migrating the large\nprod instance is not cheap, so they were left in place while the rest of\nthe service moved to MozCloud. The practical consequences:\n\n1. Terraform in `webservices-infra` cannot configure the Spanner\n instances or databases. Those changes need `cloudops-infra`, which\n means an extra PR and a MozCloud engineer to apply it.\n2. The webservices Terraform admin service account has no\n `setIamPolicy` rights on `-904c`, so IAM grants that target the\n Spanner database have to be applied out of band. See\n [Out-of-band steps](#out-of-band-steps) below.\n\nThe Dataflow job bridges the two projects: it runs in the tenant project\nand reads a change stream in the legacy project.\n\n## Diagram\n\n```mermaid\nflowchart TB\n gha[\"GitHub Actions\n syncstorage-rs\"]\n\n subgraph v1[\"GCPv1: moz-fx-sync-nonprod-904c (cloudops-infra)\"]\n syncdb[\"Spanner syncdb\n instance: sync\n database: syncdb-dev\n change stream: payload_link_changes\n access role: payload_link_reader\"]\n metadb[\"Spanner Dataflow metadata\n instance: sync\n database: syncdb-pldf-meta-dev\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (global-platform-admin)\"]\n gke[\"Shared GKE cluster\n syncserver, tokenserver,\n payload-reconciler cronjob\"]\n end\n\n subgraph tenant[\"GCPv2 tenant: moz-fx-sync-nonprod (webservices-infra)\"]\n df[\"Dataflow flex template job\n sync-nonprod-dev-payload-link-dataflow\n dedicated VPC and firewall\"]\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n payloads[(\"GCS\n sync-nonprod-dev-syncstorage-payloads\")]\n jobbucket[(\"GCS\n sync-nonprod-dev-payload-link-dataflow\n template spec, staging, tmp\")]\n end\n\n gha -.->|\"publish template spec, via WIF\"| jobbucket\n jobbucket -.->|template spec| df\n syncdb -->|change stream read| df\n df -->|partition state| metadb\n df -->|publish| topic\n topic -->|pull subscription| gke\n topic -.->|\"after 5 failed deliveries\"| dlq\n gke -->|finalize and delete objects| payloads\n gke -->|upload and download payloads| payloads\n gke -->|read and write BSO rows| syncdb\n\n```\n\nNote that the workloads do not run in the tenant project. Sync is\n`risk_level: high`, so its pods run on the shared high risk GKE cluster\nin `moz-fx-webservices-high-nonpro` (nonprod) and\n`moz-fx-webservices-high-prod` (prod). The tenant project holds the\ndata plane resources: buckets, Pub/Sub, secrets, service accounts, DNS,\nand the Dataflow job. Pods reach tenant resources through workload\nidentity on the tenant GKE service account.\n\n## Where each piece is defined\n\n| Resource | Defined in | Repo |\n| --- | --- | --- |\n| Tenant definition (envs, charts, risk rating, project ids) | `tenants/sync.yaml` | global-platform-admin |\n| Tenant GCP projects, enabled APIs, folders, DNS, GAR | `projects/tf/webservices/locals.tf` | global-platform-admin |\n| Shared GKE, GCLB, logging, monitoring | `webservices-high/tf/{nonprod,prod}` | global-platform-admin |\n| GCS payload bucket and lifecycle rule | `sync/tf/dev/bucket.tf` | webservices-infra |\n| Dataflow job, network, firewall, job bucket | `sync/tf/dev/dataflow.tf` | webservices-infra |\n| Dataflow service account and IAM | `sync/tf/dev/dataflow_iam.tf` | webservices-infra |\n| Pub/Sub topics, subscriptions, DLQ routing | `sync/tf/dev/pubsub.tf` | webservices-infra |\n| Spanner instance, database, DDL, database IAM | `projects/sync` | cloudops-infra |\n| Change stream and access role DDL source of truth | `syncstorage-spanner/src/schema.ddl` | syncstorage-rs |\n| Dataflow flex template image and metadata | `tools/payload-link-dataflow/` | syncstorage-rs |\n| Image build and template spec publish | `.github/workflows/mozcloud-publish.yaml` | syncstorage-rs |\n| Helm charts for sync, tokenserver, sync-jobs, sync-test | `sync/k8s/` | webservices-infra |\n\n## Repos and how each is applied\n\n| Repo | Scope | Automation |\n| --- | --- | --- |\n| [global-platform-admin](https://github.com/mozilla/global-platform-admin) | Cross tenant platform: projects, APIs, shared clusters, tenant definitions | Spacelift. The `admin-webservices-projects` stack tracks `main` with autodeploy, so a merge plans and applies without a manual step. |\n| [webservices-infra](https://github.com/mozilla/webservices-infra) | Per tenant resources for the webservices function | Atlantis, autodiscovered per directory. Plan runs on the PR, then `atlantis apply` in a comment. |\n| [cloudops-infra](https://github.com/mozilla-services/cloudops-infra) | Legacy GCPv1 resources, including sync Spanner | Atlantis, applied by a MozCloud engineer |\n| [syncstorage-rs](https://github.com/mozilla-services/syncstorage-rs) | Application code, Spanner DDL, Dataflow template, Helm chart values | GitHub Actions builds and publishes images; ArgoCD deploys |\n\nTerraform state for both global-platform-admin's `projects/tf/webservices`\nand webservices-infra's `sync/tf/*` lives in the same bucket,\n`moz-fx-webservices-terraform-state-global`, accessed by impersonating\n`tf-webservices@moz-fx-websvc-terraform-admin.iam.gserviceaccount.com`.\nThe per environment state is at prefix `projects/sync/`. The project\nlevel state is at `projects/projects/global`, and\n`sync/tf/dev/data.tf` reads it as a remote state data source to resolve\nproject ids and numbers. That prefix is load bearing, do not rename it\nwithout migrating the state object.\n\n## Concrete dev resources\n\nNames are derived from `${application}-${realm}-${environment}` in\n`sync/tf/dev/locals.tf`.\n\n| Thing | Value |\n| --- | --- |\n| Tenant project | `moz-fx-sync-nonprod`, number `960020799362` |\n| Spanner project | `moz-fx-sync-nonprod-904c` |\n| Spanner instance and database | `sync` / `syncdb-dev` |\n| Dataflow metadata database | `sync` / `syncdb-pldf-meta-dev` |\n| Dataflow metadata table | `Metadata_payload_link`, pinned via `spannerMetadataTableName` |\n| Change stream | `payload_link_changes` |\n| Spanner database role | `payload_link_reader` |\n| Payload bucket | `sync-nonprod-dev-syncstorage-payloads`, us-west1 |\n| Dataflow job bucket | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow job | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow service account | `sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Template publisher service account | `sync-nonprod-dev-tmpl-pub@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Dataflow VPC | `sync-nonprod-dev-dataflow`, internal ingress on tcp 12345-12346 |\n| Pub/Sub topic | `payload-link-changes` |\n| Pub/Sub DLQ | `payload-link-changes-dlq`, 5 delivery attempts |\n| Reconciler subscription | `payload-link-reconciler-sub`, 60s ack, 7d retention |\n| DLQ inspection subscription | `payload-link-changes-dlq-sub`, 7d retention |\n| Flex template image | `us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow` |\n| Template spec | `gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json` |\n\nOnly dev is built out today.\n\n## Standing up stage or prod\n\nEverything below is per environment. Names derive from\n`${application}-${realm}-${environment}`, so copying the dev Terraform\ninto the target environment directory produces the right names without\nediting them.\n\n1. **Terraform.** Copy `bucket.tf`, `pubsub.tf`, `dataflow.tf` and\n `dataflow_iam.tf` from `sync/tf/dev/` into the target environment\n directory in webservices-infra.\n\n2. **A dedicated Spanner metadata database.** The change stream\n connector keeps its partition state in a Spanner database of its own.\n Each environment should get a separate database, for example\n `syncdb-pldf-meta-stage`, so the connector holds no write access\n to the syncstorage database. This is a cloudops-infra change.\n\n Pin the table inside it as well, with\n `spannerMetadataTableName = \"Metadata_payload_link\"` on the job.\n Dev uses that name; reusing it per environment is fine, since the\n databases are already separate. The parameter is optional and the\n consequence of skipping it is silent, so it is easy to miss: see\n step 6.\n\n3. **DDL.** Apply the change stream and the `payload_link_reader` role to\n the target database. See [Out-of-band steps](#out-of-band-steps).\n\n4. **Cross-project IAM.** A `-904c` admin grants the new environment's\n Dataflow service account `roles/spanner.databaseUser` on the\n syncstorage database and on the metadata database.\n\n5. **Template spec.** Add a `write-payload-link-dataflow-spec-` job\n to `.github/workflows/mozcloud-publish.yaml`, and make sure the\n environment's `-tmpl-pub` service account exists first. The spec has\n to be in the bucket before the Terraform job resource is applied.\n\n6. **Job lifecycle.** Keep `on_delete = \"cancel\"`. This is not a\n dev-only shortcut: the SpannerIO change stream connector does not\n support draining at all, so `\"drain\"` is never the right setting for\n this pipeline. See\n [Draining a change streams pipeline](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining).\n The change stream's 7 day retention is what covers the gap while a\n replacement job comes up.\n\n Cancel is only safe to rely on because the metadata table name is\n pinned. The connector generates that name from a random UUID at\n graph-construction time, so an unpinned job that is cancelled and\n relaunched, or updated in place, comes up against an empty table and\n resumes from `Timestamp.now()` instead of the previous watermarks.\n Nothing errors; the records committed in between are simply never\n published, and the reconciler never learns those objects need\n finalizing or deleting. Set the name once and do not change it.\n\n7. **Reconciler cronjob.** Enable `payloadReconciler` in the\n environment's values file in `sync/k8s/sync/`. The chart requires\n `runBudgetSeconds` to be less than `activeDeadlineSeconds`.\n\n8. **Measure before enabling on prod.** The change stream costs Spanner\n storage that has not been quantified. Turning the stream on ahead of\n any offload traffic is the cheap way to find out.\n\nEnabled APIs need no change. `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so `moz-fx-sync-prod` already has them.\n\nNote that provisioning the pipeline is safe on its own. Offload does\nnothing until `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET` and\n`SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` are both set on\nsyncserver, so an environment can carry the whole pipeline with zero\ntraffic through it while it is verified.\n\n## Enabled GCP APIs\n\nThe tenant project's enabled APIs are declared in\n`projects/tf/webservices/locals.tf` under the `sync` entry. Enabling an\nAPI there is a PR to global-platform-admin, since tenant teams do not\nhold the IAM to enable services directly.\n\nCurrent list for the sync tenant:\n\n- `spanner.googleapis.com`\n- `secretmanager.googleapis.com`\n- `dataflow.googleapis.com`\n- `datapipelines.googleapis.com`\n- `cloudscheduler.googleapis.com`\n\nThe last three were added in\n[global-platform-admin PR\n#7031](https://github.com/mozilla/global-platform-admin/pull/7031) for\nthis work, tracked by STOR-656 and SREIN-1633.\n`compute.googleapis.com` is enabled by default for every tenant project\nby the shared `google_project` module, which is what Dataflow workers\nneed.\n\nTwo caveats on that list. First, `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so those three APIs are enabled in `moz-fx-sync-prod`\nas well as `moz-fx-sync-nonprod`. There is no per realm override.\nSecond, the current implementation launches\n`google_dataflow_flex_template_job` directly, so nothing in\n`webservices-infra/sync` references Data Pipelines or Cloud Scheduler\nyet. Those two were requested up front in case the job moves to a\nscheduled Data Pipelines definition. They are enabled but unused.\n\n### Security assessment\n\nSync as a whole stays at `risk_level: high` with\n`risk_rating: {confidentiality: 2, integrity: 1, availability: 4}` in\n`tenants/sync.yaml`. The offload itself is low risk.\n\nBSO payloads are encrypted client side with the user's Sync key before\nthey reach syncserver, so GCS holds ciphertext the server cannot\ndecrypt. The offload moves the same class of opaque blob that Spanner\nalready holds, rather than creating a new plaintext exposure. Object\nkeys and metadata do carry identifiers such as fxa_uid, collection and\nbso_id, which is metadata rather than content.\n\nEnabling a service grants no access on its own and creates no\nidentities, so IAM still gates every call. The one item worth care is\nDataflow: a principal with `roles/dataflow.developer` plus `actAs` on\nthe worker service account can run arbitrary pipeline code on GCE VMs\ninside the tenant VPC with that account's permissions. That is why the\njob runs as a dedicated per environment service account holding only\n`roles/dataflow.worker`, `roles/pubsub.publisher` on one topic,\n`roles/storage.objectAdmin` on the job bucket, and\n`roles/spanner.databaseUser` on one database, rather than the Compute\nEngine default account which carries `roles/editor`.\n\nSpanner access is narrowed a second time inside the database. The job\nruns under the `payload_link_reader` database role, which is granted\n`SELECT` on the one change stream and `EXECUTE` on its read function and\nnothing else, so the IAM grant alone does not let the job read BSO rows.\n\n## Out-of-band steps\n\nThree things are not managed by Terraform and have to be done by hand in\nthe right order. Getting this wrong is the usual reason a fresh\nenvironment fails to come up.\n\n1. Apply the change stream DDL to the Spanner database. The DDL in\n `syncstorage-spanner/src/schema.ddl` is not auto-applied. Run\n `gcloud spanner databases ddl update` against the target database\n after merging.\n\n That same DDL block also creates the fine-grained access role the\n Dataflow job reads through:\n\n ```sql\n CREATE ROLE payload_link_reader;\n GRANT SELECT ON CHANGE STREAM payload_link_changes TO ROLE payload_link_reader;\n GRANT EXECUTE ON TABLE FUNCTION READ_payload_link_changes TO ROLE payload_link_reader;\n ```\n\n The job is launched with `spannerDatabaseRole = \"payload_link_reader\"`\n (see `sync/tf/dev/dataflow.tf`), so if the role does not exist the job\n fails at startup rather than falling back to broader access. Apply the\n role and its grants in the same DDL update as the change stream.\n\n2. Grant the Dataflow service account `roles/spanner.databaseUser` on\n the Spanner database. This targets `-904c`, where the webservices\n Terraform admin account has no `setIamPolicy` rights, so a `-904c`\n admin runs it. Tracked by STOR-655.\n\n ```console\n gcloud spanner databases add-iam-policy-binding syncdb-dev \\\n --instance=sync --project=moz-fx-sync-nonprod-904c \\\n --role=roles/spanner.databaseUser \\\n --member=serviceAccount:sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com\n ```\n\n3. Publish the flex template spec into the job bucket before the\n Terraform job resource is applied, or the job has nothing to launch.\n The `write-payload-link-dataflow-spec-dev` job in\n `.github/workflows/mozcloud-publish.yaml` does this on publish.\n\n CI does not use the pipeline service account for this. A separate\n `-tmpl-pub` account exists purely to upload the spec, and the\n syncstorage-rs GitHub Actions workflow impersonates it through\n workload identity federation. Its bucket grant carries an IAM\n condition restricting writes to the `templates/` prefix, so CI cannot\n touch the runtime `staging/` or `tmp/` paths, and it holds nothing on\n Spanner or Pub/Sub. Both accounts are defined in\n `sync/tf/dev/dataflow_iam.tf`.\n\n To publish manually:\n\n ```console\n gcloud dataflow flex-template build \\\n gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json \\\n --image us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n ```\n\n## Consoles and dashboards\n\nSubstitute the project for other environments.\n\n- [Enabled APIs](https://console.cloud.google.com/apis/dashboard?project=moz-fx-sync-nonprod)\n- [Dataflow jobs](https://console.cloud.google.com/dataflow/jobs?project=moz-fx-sync-nonprod)\n- [Pub/Sub topics](https://console.cloud.google.com/cloudpubsub/topic/list?project=moz-fx-sync-nonprod)\n- [GCS buckets](https://console.cloud.google.com/storage/browser?project=moz-fx-sync-nonprod)\n- [Service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts?project=moz-fx-sync-nonprod)\n- [Spanner databases](https://console.cloud.google.com/spanner/instances/sync/databases?project=moz-fx-sync-nonprod-904c)\n\nThe tenant project Terraform is applied by the `admin-webservices-projects`\nstack in Spacelift.\n\n## Reference\n\nMozilla internal documents, not linked here:\n\n- Design document: \"Sync Spanner 10MB Payloads\"\n- \"Syncstorage Payload Expansion Project Plan\"\n- Confluence: \"Bootstrapping a tenant\", under the SRE space\n- Confluence: \"MozCloud Customer Documentation\", under the SRE space\n","docs/src/tools/payload-offload-load-testing.md":"# Load Testing Payload Offload\n\nThis provides a guide on running load tests (`tools/syncstorage-loadtest/`) on the\nGCS payload offload path with Molotov in GCP. This focuses on using gcloud cli\ncommands and short scripts as opposed to the UI.\n\nTwo methods:\n\n- **Method A, against dev env.** Real Spanner, real GCS, real change stream and\n reconciler. Payloads capped at whatever dev's limits allow. Use this to test\n the whole pipeline end to end. Note: it requires webservices-infra updates\n should you need to change application limits.\n- **Method B, local syncserver on the VM.** Spanner emulator, real GCS, your own\n limits. No change stream or reconciler. Use this to test expanded payloads\n larger than the server's default ceiling. This allows for easier ad-hoc\n changing of application limits.\n\nBoth use the load tester's **direct access** mode, which creates its own Hawk\ntoken from the master secret and bypasses Tokenserver.\n\n---\n\n## Prerequisites\n\n### Check your gcloud configuration first\n\nIf you have a configuration for the Spanner emulator and it is enabled,\nit has the bad habit of breaking everything in confusing ways. Check:\n\n```console\ngcloud config list\n```\n\nA configuration containing `auth/disable_credentials = true` or\n`api_endpoint_overrides/spanner` sends every request out unauthenticated and\nredirects Spanner calls to localhost. Switch to a normal configuration:\n\n```console\ngcloud config configurations activate default\ngcloud auth login\n```\n\nSwitch back with `gcloud config configurations activate ` when\nyou return to local emulator work.\n\n### Access you need\n\n| For | You need |\n| --- | --- |\n| Creating the VM | `roles/compute.instanceAdmin.v1` on `moz-fx-sync-nonprod`, which `sync/developers` holds. There is no equivalent grant on prod. |\n| Method A | `roles/secretmanager.secretAccessor` on the tenant project, to read the master secret. |\n| Method B | Write access to a GCS bucket you control. |\n\nSync is onboarded to PAM, so anything beyond the standing grants above can be\nrequested just-in-time. List what is available with `gcloud pam entitlements\nsearch`.\n\n---\n\n## Part 1: Create the load generator VM\n\nUse your own project under the org's `developers` folder (get the folder_no in GCP)\nrather than a tenant project.\n\n```console\nPROJ=\n\ngcloud projects create $PROJ --folder=\ngcloud billing accounts list\ngcloud billing projects link $PROJ --billing-account=\ngcloud services enable compute.googleapis.com --project=$PROJ\n```\n\nNew projects have no default VPC, because the org enforces\n`compute.skipDefaultNetworkCreation`. Create one:\n\n```console\ngcloud compute networks create default --subnet-mode=auto --project=$PROJ\n```\n\nFirewall for IAP SSH required. Without this, the SSH in the next step fails\nwhen no firewall rules are set. Org rules forbid opening tcp/22 to `0.0.0.0/0`:\n\n```console\ngcloud compute firewall-rules create allow-iap-ssh \\\n --project=$PROJ --network=default \\\n --direction=INGRESS --action=allow --rules=tcp:22 \\\n --source-ranges=35.235.240.0/20\n```\n\nCreate the instance in **us-west1**, matching where dev runs.\nAlways check region and match to limit latency.\n\n```console\ngcloud compute instances create sync-loadtest \\\n --project=$PROJ \\\n --zone=us-west1-b \\\n --machine-type=n2-standard-8 \\\n --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \\\n --boot-disk-size=200GB\n```\n\n> **Note on Machine family.** New projects get zero quota for newer families, so C4D\n> fails with `Quota 'CPUS_PER_VM_FAMILY' exceeded. Limit: 0.0`. Most families have\n> their own regional quota (`N2_CPUS`, `E2_CPUS`, `C2_CPUS`); only N1 draws on the\n> general `CPUS` pool. Check whichever matches the family you pick, with\n> `gcloud compute regions describe us-west1 --project=$PROJ --flatten=\"quotas[]\"\n> --format=\"table(quotas.metric,quotas.limit,quotas.usage)\"`.\n\nTo Connect via terminal to instance (you'll likely use this command a lot):\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap\n```\n\nInstall sync & load tests on the VM:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\nmkdir -p ~/src && cd ~/src\ngit clone --depth=1 https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\npython3 -m venv venv && source venv/bin/activate\npip install poetry && poetry install\n```\n\n---\n\n## Part 2: Method A - load test against dev\n\n### Fetch the master secret programmatically\n\nDo not copy and paste the secret. It can contain spaces and a paste\nmay result in confusing 401 errors that are just a mismatched token.\n\n```console\ngcloud auth login --no-launch-browser # VM's own service account cannot read it\n\nSECRET=$(gcloud secrets versions access latest --secret=dev-gke-app-secrets \\\n --project=moz-fx-sync-nonprod \\\n | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"SYNC_MASTER_SECRET\"])')\n\nexport SERVER_URL=\"https://dev.sync.nonprod.webservices.mozgcp.net#${SECRET}\"\necho \"${SERVER_URL%%#*}\"\n```\n\nThat last line prints the origin without revealing the secret. If empty,\nstop: an unset `SERVER_URL` results in a fall back to\n`https://token.stage.mozaws.net` and take the OAuth path instead.\n\n### Verify authentication before running\n\n```console\npoetry run python -c \"\nimport asyncio, aiohttp\nfrom storage.client import StorageClient\nasync def m():\n async with aiohttp.ClientSession() as s:\n c = StorageClient(s)\n r, _ = await c.get('/info/collections', (200, 404))\n print(r.status)\nasyncio.run(m())\n\"\n```\n\nUse `/info/collections`, not `/info/configuration`. The latter takes no auth\nextractor at all and returns 200 to anyone, so it cannot tell you whether your\nsecret is right.\n\n### Run\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n`OFFLOAD_COLLECTIONS` must name a collection the server actually offloads.\nCheck `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` in the deployment's\nvalues file.\n\nThen scale up, redirecting output to a file:\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n nohup poetry run molotov --processes 1 --workers 5 --duration 600 -v loadtest.py \\\n > /tmp/lt.log 2>&1 &\n```\n\nRamp gradually. Dev is small, and a sudden jump to high concurrency will hit\nits ceiling rather quickly.\n\n### Verify offload actually occurred\n\n```console\ngcloud storage ls -r \"gs://sync-nonprod-dev-syncstorage-payloads/**\" \\\n --project=moz-fx-sync-nonprod | grep -v '[/:]$' | tail -5\n```\n\nThen check at least one object was finalized by the reconciler:\n\n```console\ngcloud storage objects describe --project=moz-fx-sync-nonprod\n```\n\nLook for `custom_fields.committed: 'true'` and `custom_time` pinned to the far\nfuture. The gap between `creation_time` and `update_time` is your\nend-to-end reconciliation latency, which can be worth noting.\n\n> Note: the metadata key is `custom_fields` in `gcloud storage` output, not\n> `metadata`.\n\n---\n\n## Part 3: Method B, local syncserver with raised limits\n\nUse this when you need payloads larger than the deployed server allows. There\nis no runtime or client-side override: limits are read from the server's\nenvironment at startup, which you can easily change on the fly.\nIt runs on whatever `/info/configuration`\nis set to. Exceeding said server-side limits returns 413.\n\n### Your own bucket\n\nDo not use a shared environment's bucket. Make one you own:\n\n```console\ngcloud storage buckets create gs:// --project=$PROJ --location=us-west1\n```\n\nAdd a lifecycle rule. These objects are never finalized in this setup, so\nnothing else will clean them up. A 20 minute run can write hundreds of GB, so we\ndon't necessarily want these sticking around:\n\n```console\necho '{\"rule\":[{\"action\":{\"type\":\"Delete\"},\"condition\":{\"age\":1}}]}' > /tmp/lc.json\ngcloud storage buckets update gs:// --lifecycle-file=/tmp/lc.json --project=$PROJ\n```\n\n### Give the VM write access\n\nThe default compute service account is scoped to `devstorage.read_only`, so it\ncannot write, regardless of IAM settings. Note that scopes can only change while the instance\nis stopped:\n\n```console\nSA=$(gcloud compute instances describe sync-loadtest --project=$PROJ \\\n --zone=us-west1-b --format=\"value(serviceAccounts[0].email)\")\n\ngcloud storage buckets add-iam-policy-binding gs:// \\\n --member=serviceAccount:$SA --role=roles/storage.objectAdmin --project=$PROJ\n\ngcloud compute instances stop sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud compute instances set-service-account sync-loadtest \\\n --project=$PROJ --zone=us-west1-b --service-account=$SA \\\n --scopes=https://www.googleapis.com/auth/cloud-platform\ngcloud compute instances start sync-loadtest --project=$PROJ --zone=us-west1-b\n```\n\nDisk contents survive the stop.\n\nConfirm the new scope took effect by writing **from the VM**. Run locally this\nwould use your own credentials and prove nothing about the instance service\naccount, and `$PROJ` is a local shell variable that does not exist on the VM:\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap --command \\\n 'echo hello | gcloud storage cp - gs:///write-check.txt && \\\n gcloud storage rm gs:///write-check.txt'\n```\n\nUsing the instance service account this way means no expiring credentials and\nnothing to mount into the container.\n\n### Build the Docker image\n\n```console\nsudo apt install -y docker.io docker-compose-v2 docker-buildx\nsudo usermod -aG docker $USER && newgrp docker\n\ncd ~/src/syncstorage-rs\nDOCKER_BUILDKIT=1 nohup docker build -t app:build . > /tmp/build.log 2>&1 &\n```\n\n> `DOCKER_BUILDKIT=1` is required. The Dockerfile uses BuildKit cache mounts and\n> the legacy builder fails with \"the --mount option requires BuildKit\".\n\nPulling the published image instead is much faster if you have Artifact\nRegistry read, after `gcloud auth configure-docker us-docker.pkg.dev`. Tag it as\n`app:build` either way, because the compose file hardcodes that tag for its\nsetup container.\n\n### Compose override\n\n```console\ncd ~/src/syncstorage-rs\ncat > docker/docker-compose.offload-local.yaml <<'YAML'\nservices:\n syncserver:\n environment:\n SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET: \n SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS: bookmarks\n SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES: \"20971520\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_POST_BYTES: \"26214400\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_REQUEST_BYTES: \"26218496\"\nYAML\n```\n\nNo nginx runs in this stack, therefore the chart's `nginxClientMaxBodySize` (default\n`2568k`) does not apply. It does apply to any deployed environment, though.\nNote that raising syncserver limits without it gets you a 413 from nginx,\nbefore syncserver sees a request.\n\n### Docker compose\n\n```console\ndocker tag app:build syncstorage-rs:latest\n\ndocker compose -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.offload-local.yaml up -d\n\ndocker compose -f docker/docker-compose.spanner.yaml ps\n```\n\nThe compose default image is `syncstorage-rs:latest`, hence the tag. Otherwise\nset `SYNCSTORAGE_RS_IMAGE=app:build` on every compose.\n\nConfirm the limits are valid:\n\n```console\ncurl -s http://localhost:8000/1.5/1/info/configuration | python3 -m json.tool\n```\n\n`max_record_payload_bytes` must show your raised value.\n\n### Run\n\nThe master secret in this compose config is `secret0`:\n\n```console\ncd ~/src/syncstorage-rs/tools/syncstorage-loadtest\nsource venv/bin/activate\nexport SERVER_URL=\"http://localhost:8000#secret0\"\n\nLARGE_PAYLOAD_PROB=1.0 LARGE_PAYLOAD_SIZE=20971520 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\nConfirm the object sizes:\n\n```console\ngcloud storage ls -r --long \"gs:///**\" --project=$PROJ | tail -6\n```\n\n### Sampling memory during a run\n\nMemory can be easily measured here, and is a valuable performance metric.\nUse the below command to sample it to a file (rather than watching a streaming display):\n\n```console\nnohup bash -c 'while true; do date -u +%H:%M:%S | tr \"\\n\" \" \"; docker stats --no-stream --format \"{{.MemUsage}} {{.CPUPerc}}\" docker-syncserver-1; sleep 5; done' > /tmp/mem.log 2>&1 &\n```\n\nThen run the load tests. Then `pkill -f \"docker stats\"` and read `/tmp/mem.log`.\n\n---\n\n## Caveats\n\n**Molotov hides its errors.** A failing `setup_session` is counted and the\ntraceback is discarded, so any auth or setup problem shows as\n`SUCCESSES: 0 | FAILURES: 1` with nothing else.\n\nAlso note `_retry` only checks the HTTP status when the caller passes an\nexpected-statuses tuple. The scenario's `/info/configuration` call passes none,\nso a 503 appears as a JSON decode error rather than \"Response 503\".\n\n**Molotov's display can mangle terminals.** To save yourself the possible headache, always\nredirect to a file for anything but a short test, and if your terminal\ngoes blank or stops echoing, run `reset`.\n\n**Always redirect, never pipe.** Piping molotov's output through `head` or\n`grep` produces basically unhelpful output.\n\n**Co-locating the server and the load tester skews results.** Note that to some degree, Method B\nresults in both compete for the same CPU and memory. Use low concurrency, or alternatively a second VM.\n\n**The emulator is not Spanner.** Method B's write throughput and latency numbers\naren't super helpful and indicative of live spanner performance.\n\n**Fastly NGWAF is the edge, in blocking mode.** Dev sets\n`ngwaf_agent_level = \"block\"`, so sustained 403s partway through a run against a\ndeployed environment are worth checking there before blaming the load tester.\nThe `sync-policy` Cloud Armor resource still exists, but neither backend config\nattaches it, so it is not the cause.\n\n**Check where the Spanner metrics live.** Spanner for the sync tenant is in a\nseparate project, `moz-fx-sync-nonprod-904c` (now named `sync-legacy-nonprod`),\nand `sync/developers` has no access to it, so the console is unavailable. Since\nthe move to `sync-legacy` the metrics come from Grafana's `gcp-v2-nonprod`\ndatasource, not `gcp-v1-nonprod`, which no longer returns them.\n\n---\n\n## Cleanup\n\n```console\ngcloud compute instances delete sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud storage rm -r gs:///** --project=$PROJ\n```\n\nDelete the instance when you are done. An idle load generator bills\nindefinitely!\n","docs/src/tools/payload_link_reconciler.md":"# GCS Payload Reconciliation Pipeline\n\n## Summary\n\nWhen syncserver offloads a large BSO payload to Google Cloud Storage\n(`syncserver/src/web/payload_offload.rs`), the GCS object is created\nwith custom metadata `committed=false` and `customTime=now`, and the\nBSO row's `payload_link` column points at it. A separate pipeline must:\n\n- **Finalize** newly-committed objects — flip metadata to\n `committed=true` and pin `customTime` to its maximum value\n (`2200-12-31T23:59:59Z`) so the bucket's lifecycle policy (see\n below) cannot reclaim them.\n- **Garbage-collect orphans** — delete GCS objects whose row's\n `payload_link` was replaced (UPDATE) or removed (DELETE, including\n Spanner row-deletion-policy TTL deletes).\n\nAn object whose write never reached a Spanner commit never receives a\nfinalize, and there are two shapes of that. If the database transaction\nfails or rolls back, syncserver issues an inline best-effort **delete**\nof the objects it uploaded for that request and ignores the result. If an\nupload itself fails, the request is abandoned fail-fast and objects\nalready uploaded for it are not cleaned up inline at all. Either way an\nobject can be left stranded at `committed=false` with its upload-time\n`customTime`.\n\nAnything that slips through is reaped by a GCS **lifecycle policy**\n(`bucket.tf` in `webservices-infra/sync`) that deletes objects whose\n`customTime` is more than **30 days** old. Flipping `customTime` to the\nmax sentinel is what protects committed objects from that policy:\n`daysSinceCustomTime` goes permanently negative once finalized, so the\npolicy cannot touch them regardless of object age.\n\nThe 30 day window is the safety margin for the whole asynchronous arm. It\nhas to comfortably exceed the worst case time from upload to finalize,\nwhich is set by the cronjob cadence plus any Pub/Sub or Dataflow backlog.\nAt a five minute cadence the margin is four orders of magnitude, so the\nnumber only becomes interesting if the pipeline is stopped for weeks.\n\nThis document covers that pipeline. It consumes the\n`payload_link_changes` Spanner change stream defined in\n`syncstorage-spanner/src/schema.ddl`.\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n stream[(\"Spanner change stream\n payload_link_changes\n OLD_AND_NEW_VALUES, 7d retention\")]\n\n subgraph dataflow[\"Custom Dataflow flex template\"]\n filter{\"payload_link NULL\n on both sides?\"}\n end\n\n drop([\"dropped\"])\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n\n subgraph reconciler[\"Reconciler cronjob\"]\n route{\"which side\n carries a link?\"}\n finalize[\"finalize the object\n committed=true\n customTime=MAX\"]\n delete[\"delete the object\"]\n end\n\n stream -->|DataChangeRecord| filter\n filter -->|yes| drop\n filter -->|\"no, publish as JSON\"| topic\n topic -->|pull subscription| route\n topic -.->|\"after 5 failed deliveries\"| dlq\n route -->|new link| finalize\n route -->|\"old link, replaced or removed\"| delete\n```\n\nBoth reconciler actions are idempotent, so a redelivered record is\nharmless.\n\n## Components\n\n### 1. Spanner change stream — `payload_link_changes`\n\nDefined in `syncstorage-spanner/src/schema.ddl`:\n\n```sql\nCREATE CHANGE STREAM payload_link_changes\n FOR bsos(payload_link), batch_bsos(payload_link)\n OPTIONS (\n retention_period = '7d',\n value_capture_type = 'OLD_AND_NEW_VALUES'\n );\n```\n\nColumn-scoped: an UPDATE that does not touch `payload_link` produces no\nrecord. INSERTs and DELETEs always produce a record, even when\n`payload_link` is NULL — those are dropped at the next stage.\n\nThe Spanner DDL is not auto-applied; run `gcloud spanner databases ddl\nupdate` against the target database after merging.\n\n### 2. Custom Dataflow flex template — `tools/payload-link-dataflow/`\n\nA standalone Apache Beam pipeline (Java, Beam 2.60.0) that:\n\n1. Reads `payload_link_changes` via `SpannerIO.readChangeStream()`.\n2. Applies a `Filter.by(isPayloadLinkActionable)` step that drops\n records whose every mod has `payload_link` NULL on both sides.\n Malformed records pass through so the reconciler / DLQ surfaces\n them — not the filter.\n3. Serializes each surviving `DataChangeRecord` to JSON and publishes\n to a Pub/Sub topic.\n\nThe pipeline is **not** vendored from\n[`GoogleCloudPlatform/DataflowTemplates`](https://github.com/GoogleCloudPlatform/DataflowTemplates).\nWe own a small standalone source tree under `src/`; the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` template is referenced for\nintent comparison via `upstream-customization.patch` (documentation\nonly — not a build input).\n\n**Build / publish** (operator runs from `webservices-infra`):\n\n```bash\ndocker build -t /syncserver-payload-link-dataflow: \\\n tools/payload-link-dataflow\ndocker push /syncserver-payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/syncserver-payload-link-dataflow.json \\\n --image /syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n```\n\n**Launch parameters** (full list in\n`tools/payload-link-dataflow/metadata.json`):\n\n- `spannerProjectId`, `spannerInstanceId`, `spannerDatabase` — the\n syncstorage Spanner database.\n- `spannerMetadataInstanceId`, `spannerMetadataDatabase` — where the\n change-stream connector keeps its partition-state table. Dev uses a\n dedicated database, `syncdb-pldf-meta-dev`, so the connector holds no\n write access to the syncstorage database. Every environment should do\n the same.\n- `spannerMetadataTableName=Metadata_payload_link` names the\n partition-state table itself. Optional in the template, but pin it on\n any streaming launch. Left unset, the connector generates a random\n table name inside `expand()` and bakes it into the pipeline graph, so a\n `--update` or a cancel-then-relaunch comes up against a fresh empty\n table, reads from `Timestamp.now()`, and silently drops every record\n committed between the two jobs. The name only has to be a legal Spanner\n identifier and stable for the life of the environment; the connector\n creates the table on first use.\n See \"Job update and restart continuity\" in\n `tools/payload-link-dataflow/README.md`.\n- `changeStreamName=payload_link_changes`.\n- `spannerDatabaseRole=payload_link_reader` — the fine-grained access\n role the job reads the stream through, created by the DDL in\n `syncstorage-spanner/src/schema.ddl`. If the role is absent the job\n fails at startup rather than falling back to broader access.\n- `pubsubTopic=projects//topics/payload-link-changes`.\n\n**Service account requires:**\n\n- `roles/spanner.databaseUser` on the syncstorage database and on the\n metadata database. The one role covers both reading the change stream\n and maintaining the connector's partition metadata, but since the two\n live in separate databases the grant is needed on each. Note the IAM\n grant targets the `-904c` project, so it is applied out of band; see\n [GCP Infrastructure](payload-offload-infrastructure.md#out-of-band-steps).\n- Membership of the `payload_link_reader` database role, which is what\n actually narrows the job to the change stream. The IAM grant alone\n does not let it read BSO rows.\n- `roles/pubsub.publisher` on the destination topic.\n- `roles/storage.objectAdmin` on the Dataflow job bucket, for `staging/`\n and `tmp/`.\n- `roles/dataflow.worker` at the project level.\n\n### 2b. Dev/E2E Python publisher — `tools/payload-link-dataflow/payload-link-publisher-py/`\n\nPlain Python script (not Beam) that polls the\n`READ_payload_link_changes` TVF via `google.cloud.spanner.Client` and\npublishes the same JSON wire format the Java job produces. Sub-second\nstartup, no JVM. **Dev/E2E only** — used by the compose stack\ndescribed below; **Java remains the prod publisher.**\n\nFollows partition splits: reads `_root`, picks up child-partition\ntokens from `child_partitions_record`s, reads each child from its\nadvertised `start_timestamp`, retires parents once their children are\nannounced, and drops any partition that responds `OUT_OF_RANGE`.\nRequired in practice — the emulator routes DataChangeRecords to child\npartitions immediately, so a `_root`-only reader sees zero DCRs. Full\ndetails in the tool's own README.\n\n### 3. Pub/Sub topic + DLQ\n\nProvisioned from `webservices-infra/sync`:\n\n- Topic: `payload-link-changes`\n- Dead-letter topic: `payload-link-changes-dlq`\n- Pull subscription: `payload-link-reconciler-sub`\n - 60s ack deadline\n - 7d message retention\n - DLQ routing after 5 delivery attempts\n\n### 4. Reconciler — `tools/payload-reconciler/`\n\nPython script with one job: pull messages, perform GCS operations, ack.\nSync-pull drain loop with two deployment modes selected by whether\n`RUN_BUDGET_SECONDS` is set:\n\n- **Cronjob mode** (default deployment): `RUN_BUDGET_SECONDS` set (e.g.\n `240`). The script drains the subscription up to that many seconds\n or until the queue idles, then exits 0. K8s cronjob at ~5 min\n cadence.\n- **Long-running mode**: `RUN_BUDGET_SECONDS` unset. The script polls\n forever, never exiting on idle. Deploy as a K8s Deployment when\n finalize-flip latency below the cronjob cadence matters.\n\n**Per-message handling** (`reconcile_payload_links.py:handle_message_body`):\n\nFor each mod in the change record:\n\n- New `payload_link` present: `blob.patch()` sets\n `metadata.committed = \"true\"` and `customTime = \"2200-12-31T23:59:59Z\"`.\n- Old `payload_link` present and not equal to the new value: `blob.delete()`,\n with one exception. A `batch_bsos` row removed under the batch commit\n transaction tag (`transactionTag` equals `batch_commit`, set by syncstorage in\n the Spanner `commit_batch` path) is skipped\n (`payload_reconciler.batch_commit_skips`). On commit the link moves into the\n permanent `bsos` row in the same transaction, so its object must be kept\n (deleting it was the STOR-657 bug). Any other `batch_bsos` removal is a\n genuine delete: `batch_bsos` has no deletion policy of its own, so TTL expiry\n and `user_collections` deletes reach it as cascade deletes with no such tag,\n and their objects are removed. `bsos` changes and `batch_bsos` overwrites are\n unaffected. See STOR-668.\n\nBoth operations tolerate `404 NotFound` as success, see *Failure\nmodes* below.\n\n**Environment**\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | — | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | — | `payload-link-reconciler-sub` in prod. |\n| `GCS_PAYLOAD_BUCKET` | yes | — | Cross-bucket links abort the message. |\n| `RUN_BUDGET_SECONDS` | no | — | Set (e.g. `240`) → cronjob mode; drain up to N seconds then exit 0. Unset → long-running mode; poll forever, never exit on idle. |\n| `SYNC_STATSD_HOST` | no | localhost | statsd metrics server host. |\n| `SYNC_STATSD_PORT` | no | 8125 | statsd metrics server port. |\n\n**Service account requires:**\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`.\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and the `delete` operation).\n\n**Deployment.** Default is a K8s cronjob (~5 min cadence) with\n`RUN_BUDGET_SECONDS` set. When lower finalize-flip latency matters,\ndeploy as a K8s Deployment without `RUN_BUDGET_SECONDS` to run\nlong-running. The cronjob template is\n`sync/k8s/sync/templates/payload-reconciler-cronjob.yaml` in\nwebservices-infra, gated on `payloadReconciler.enabled`. Note it lives in\nthe `sync` chart rather than `sync-jobs`.\n\nFour settings in that manifest are load bearing, and changing any of them\nbreaks an assumption the script relies on:\n\n- `concurrencyPolicy: Forbid`. A run that overruns its window is never\n joined by a second one. The subscription is the queue, so the next tick\n simply picks up where the last left off. Two concurrent drains would\n race on the same messages, which is survivable given idempotency but\n wastes the ack deadline.\n- `backoffLimit: 0`. No in-window retry. Every operation is idempotent\n and unacked messages come back on their own, so the next scheduled run\n *is* the retry. A backoff would just re-drain the same queue sooner.\n- `runBudgetSeconds` strictly less than `activeDeadlineSeconds`. The\n chart fails the render if this is violated. The budget is what makes\n the drain loop exit cleanly on its own; if the deadline fired first the\n pod would be killed mid-message and the run would always look failed.\n- The command override. The image entrypoint is the syncserver binary, so\n the cronjob invokes\n `python3 /app/tools/payload-reconciler/reconcile_payload_links.py`\n explicitly. Invoking by path is also what puts the script's directory\n on `sys.path`, which is how its `import utils` resolves.\n\nCredentials come from workload identity: the pod runs as the tenant GKE\nservice account, which holds the two roles listed above. There is no key\nfile and no secret mounted.\n\n---\n\n## Wire format\n\nEach surviving change record reaches the reconciler as a JSON Pub/Sub\nmessage:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-30T00:00:00.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"transactionTag\": \"\",\n \"isSystemTransaction\": false,\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nMod fields (`keys`, `oldValues`, `newValues`) carry **JSON strings**\nthat the reconciler parses with a second `json.loads`, matching\nSpanner's change-streams wire convention.\n\n`transactionTag` carries the transaction tag Spanner records for the change,\nwhich the reconciler uses to skip batch commit handoffs (see above).\n`isSystemTransaction` is carried too but not currently consumed.\n\n---\n\n## Local e2e compose stack\n\n`make docker_run_reconciliation_e2e_tests` brings up a full-stack\ncompose environment that exercises the entire pipeline against\nemulators:\n\n| Service | Image | Role |\n|---|---|---|\n| `sync-db` | Spanner emulator (existing) | Spanner + change stream |\n| `pubsub-emulator` | `google-cloud-cli:emulators` | Pub/Sub |\n| `fake-gcs` | `fsouza/fake-gcs-server` | GCS |\n| `reconciliation-setup` | one-shot | creates Pub/Sub topic + subscription + bucket |\n| `payload-link-publisher` | Python publisher (default) | polls change stream → publishes to Pub/Sub |\n| `payload-reconciler` | primary image, entrypoint override | drains Pub/Sub → patches/deletes GCS objects |\n| `syncserver` | primary image | offload enabled for all `test_storage.py` collections |\n| `e2e-tests` | primary image | runs `pytest tools/integration_tests/ tools/tokenserver/` |\n\nTwo publisher variants:\n\n- **Python (default)** — `docker-compose.e2e.reconciliation.yaml`.\n Sub-second startup; the day-to-day iteration path.\n- **Java (swap-in)** — layer\n `docker-compose.e2e.reconciliation.java.yaml` on top. Runs the same\n Java flex-template image under Beam's DirectRunner. See\n `tools/payload-link-dataflow/README.md` for the exact invocation.\n Use when you need to reproduce a Java-specific issue.\n\nThe compose stack doubles as regression coverage for `test_storage.py`:\nby opting every collection the storage tests use into GCS offload, each\nBSO write flows through the offload path (upload + `payload_link`\nstorage), and each read flows through `download_payload`. New\nreconciler-specific tests live in\n`tools/integration_tests/test_payload_link_reconciliation.py`, gated\nby a module-level `pytest.mark.skipif` on `GCS_PAYLOAD_BUCKET` so they\nauto-skip in the existing spanner e2e stack (where the env var is\nunset) and un-skip here.\n\n**Emulator fallback.** If a future Spanner emulator upgrade breaks\n`SELECT * FROM READ_payload_link_changes(...)`, the fall-back is the\nJava swap-in overlay above (or running the compose stack against a\nreal dev Spanner instance).\n\n---\n\n## Metrics\n\nBoth halves report. The reconciler emits under the\n`payload_reconciler` statsd namespace, syncserver under\n`storage.gcs.payload`. Between them every GCS operation reports both a\ncount and a latency.\n\n### Reconciler\n\nCounters:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalizes` | Objects flipped to `committed=true` | Tracks offloaded write volume |\n| `orphan_deletes` | Objects deleted because a row stopped pointing at them | Tracks overwrite and delete volume |\n| `gcs_404` `op:finalize` | Finalize target was gone | Low and flat |\n| `gcs_404` `op:delete` | Delete target was already gone | Low and flat, redeliveries are normal |\n| `batch_commit_skips` | A `batch_bsos` removal carrying the `batch_commit` transaction tag was kept rather than deleted | Tracks batch commit volume on offloaded collections |\n| `noop_skips` | A record arrived with nothing to do | Near zero; the Dataflow filter should have dropped it |\n| `errors` `kind:handler` | Handler raised, message left unacked | Zero |\n| `runs` | One per drain, emitted unconditionally | Tracks the cronjob cadence |\n| `messages_processed` | Change records pulled and handled, counted per batch | Tracks pipeline volume |\n| `budget_exhausted` | A run hit `RUN_BUDGET_SECONDS` before the queue idled | Zero |\n\nTimings:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalize_age` | Spanner commit to finalize, in ms | Seconds to minutes. This is the window the 30 day lifecycle policy is sized against |\n| `gcs_op` | One GCS round trip. Tagged `op` (`finalize` or `delete`) and `result` (`success`, `not_found`, `error`) | Low. Separates our own GCS latency from Dataflow and Pub/Sub delay |\n| `drain_duration` | How long one run took | Comfortably under `RUN_BUDGET_SECONDS` |\n\n`runs` is the only unconditional series. Every other metric here fires\nonly on the event it names, so a quiet pipeline emits almost nothing and\n\"idle\" would otherwise be indistinguishable from \"not running\".\n\n### Syncserver offload path\n\nEmitted from `syncserver/src/web/payload_offload.rs`. All three GCS\noperations report latency, each tagged with a `result`.\n\n| Metric | Type | Tags | Meaning |\n|---|---|---|---|\n| `storage.gcs.payload.upload` | timing | `result` | One object uploaded |\n| `storage.gcs.payload.download` | timing | `result` | One object read back, streaming included |\n| `storage.gcs.payload.cleanup.duration` | timing | `result` | The compensating delete after a failed write transaction |\n| `storage.gcs.payload.batch` | timing | `op`, `handler`, `result` | Total GCS time for one request, across all of its BSOs |\n| `storage.gcs.payload.size` | histogram | `op` | Distribution of individual payload sizes, in bytes |\n| `storage.gcs.payload.upload.bytes` | counter | none | Bytes written, for throughput |\n| `storage.gcs.payload.download.bytes` | counter | none | Bytes read back |\n| `storage.gcs.payload.cleanup` | counter | `handler`, `result`, `reason` | Outcome of the compensating delete |\n\nStatsd timers carry their own count, so there is no separate counter for\nthe number of uploads or downloads.\n\n`storage.gcs.payload.batch` is not the sum of the per-object timings. A\nrequest works on up to `gcs_payload_max_concurrency` objects at once, so\nits total lands somewhere between the slowest single object and the sum\nof all of them. It is the delay the client waits through, and it is\nrecorded even when the batch fails part way, hence the `result` tag: an\nabandoned batch stopped early and is not comparable to a complete one.\n\n`storage.gcs.payload.size` is a histogram rather than a tag on the\ntimings, so the aggregator derives percentiles from it. The byte\ncounters give a total and therefore a mean; the tail is the part that\nmatters on a project about raising a size ceiling.\n\nTags on `storage.gcs.payload.cleanup`:\n\n| Tag | Values | Meaning |\n|---|---|---|\n| `handler` | `put_bso`, `post_collection` | Which write handler issued the cleanup |\n| `result` | `success`, `failure` | Whether the object was deleted |\n| `reason` | `invalid_url`, `gcs_error` | Failure only. `invalid_url` means the `gs://` URL did not parse and no delete was attempted; `gcs_error` means GCS rejected it |\n\nThe cleanup counter firing at all means writes are failing after their\npayload reached GCS, so read it against the write error rate rather than\non its own. `result:failure` is the more interesting cut: those objects\nare now leaked until the 30 day lifecycle policy reaps them. A steady\n`reason:invalid_url` is a code bug rather than an operational problem,\nsince the URL is one syncserver just generated. Note that an unparseable\nURL issues no GCS call and so has no `cleanup.duration` alongside it.\n\nWorth alerting on:\n\n- Anything at all in `payload-link-changes-dlq`. A message only lands\n there after five failed deliveries, so it means a record cannot be\n handled and needs a human. Inspect it through\n `payload-link-changes-dlq-sub`. Note the DLQ is terminal: its\n subscription carries no dead-letter policy of its own, so a message\n sits there until the 7 day retention drops it and no further queue\n catches it. That is the same 7 days as the change stream retention,\n so once a dead-lettered record ages out the underlying change is\n unrecoverable from both. Seven days is a response deadline, not a\n margin.\n- `errors` `kind:handler` sustained above zero.\n- `noop_skips` climbing, which means the Dataflow filter regressed and\n the pipeline is paying Pub/Sub for records it discards.\n- `gcs_404` `op:finalize` rising as a share of `finalizes`, which is the\n signal that objects are being reaped before they get finalized.\n- Oldest unacked message age on `payload-link-reconciler-sub`. This is a\n Pub/Sub metric rather than one of ours, and it is the most direct\n measure of finalize latency. It should sit in minutes. The 30 day\n lifecycle window is the deadline it must never approach.\n- `storage.gcs.payload.cleanup` with `result:failure`, which counts\n objects that leaked because the compensating delete did not land.\n- `finalize_age` climbing. It is the direct measure of how long objects\n sit unprotected, and unlike the Pub/Sub metric above it covers the\n whole path including the GCS patch.\n- `budget_exhausted` sustained above zero, meaning runs keep ending on\n the clock rather than on an idle queue, so the drain is not keeping up.\n- `runs` going flat. Every other reconciler metric is conditional, so\n this is the one that separates a quiet pipeline from a stopped one.\n\n---\n\n## Failure modes\n\n| Symptom | Cause | Behaviour |\n|---|---|---|\n| Spike in `payload_reconciler.noop_skips` | Dataflow filter is letting inert records through | Investigate; should be ~0 if the filter works. Records still ack — no harm but extra Pub/Sub cost. |\n| Sustained `payload_reconciler.gcs_404` with `op:finalize` | Lifecycle rule reclaimed the object before the reconciler finalized it, OR the same message redelivered after a successful prior run (at-least-once tax) | Acceptable up to a low background level. Sharp rise = lifecycle window too aggressive vs. cronjob cadence. |\n| Sustained `payload_reconciler.gcs_404` with `op:delete` | Object was already deleted (redelivery or concurrent cleanup) | Acceptable; idempotent by design. |\n| Messages in `payload-link-changes-dlq` | Repeated handler exceptions on the same message after 5 retries (malformed JSON, cross-bucket link, GCS auth failure) | Inspect the DLQ payload; fix and re-publish or discard. The main subscription continues to drain. |\n| `payload_reconciler.errors` with `kind:handler` non-zero | Same as above before reaching DLQ. | Same. |\n| `storage.gcs.payload.cleanup` with `result:failure` | The compensating delete after a failed write transaction did not land | The object is stranded at `committed=false` and the 30 day lifecycle policy is the only thing left to reap it. Occasional failures are tolerable; sustained means the write path is failing and cleanup is failing with it. |\n| Finalize latency jumps after a Dataflow deploy, with a gap in `finalizes` | The job relaunched against a fresh partition-metadata table and resumed from `Timestamp.now()` | Records committed during the gap are lost for good; the change stream's 7 day retention only helps if you notice in time. Check that `spannerMetadataTableName` is pinned and unchanged. |\n\nA `payload_link` pointing at a bucket other than `GCS_PAYLOAD_BUCKET`\nraises `ValueError` and the message is left unacked — it retries up to\nthe DLQ rather than mutating an unrelated bucket. This is a hard guard.\n\n---\n\n## Keeping the reference patch accurate\n\n`tools/payload-link-dataflow/upstream-customization.patch` is\ndocumentation only — not a build input. If the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` pipeline evolves in ways that\nchange the conceptual diff, refresh it (see the README in that\ndirectory). This is a doc refresh, not a code change.\n","docs/src/tools/postgres_purge_ttl.md":"# Documentation for `purge_ttl.py` (PostgreSQL)\n\n## Summary\n\nThe `purge_ttl.py` script is a utility for purging expired Time-To-Live (TTL) records from a Sync storage PostgreSQL database. It deletes expired entries from the `batches` and/or `bsos` tables, with optional filtering by collection ID and a dry-run mode for verifying query logic without modifying data.\n\n> Spanner no longer uses this script — it expires records natively via its built-in [row deletion (TTL) policies](https://cloud.google.com/spanner/docs/ttl). This utility applies to PostgreSQL deployments only.\n\n---\n\n## Status\n- Runs as a regularly scheduled job (e.g. a cron job) in PostgreSQL (\"enterprise\") deployments.\n- The script is bundled in the main `syncserver-postgres` image and invoked from there; there is no separate utility image.\n\n## Specifics\n\n- **Database**: PostgreSQL (accepts a `postgresql://` or `postgres://` DSN).\n- **Tables**:\n - `batches`: Contains batch entries (child `batch_bsos` rows are removed via the schema's foreign-key cascade).\n - `bsos`: Stores Sync Basic Storage Objects (BSO).\n- **Supported Modes**:\n - `batches`: Purges expired entries in the `batches` table.\n - `bsos`: Purges expired entries in the `bsos` table.\n - `both`: Performs purges on both tables.\n- **Expiry Modes**:\n - `now`: Purges entries with `expiry < CURRENT_TIMESTAMP`.\n - `midnight`: Purges entries with `expiry < DATE_TRUNC('day', CURRENT_TIMESTAMP AT TIME ZONE 'UTC')`.\n\nThe script tracks execution duration and rows affected using StatsD metrics for performance monitoring.\n\n---\n\n## Notes\n\n- Ensure the database role used by the DSN has `DELETE` privileges on the target tables.\n- Use the `--dryrun` option to verify query logic before actual purging.\n- Consider setting up automated monitoring for long-running operations or performance issues.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3 is installed.\n2. **Poetry**: Install [Poetry](https://python-poetry.org/docs/#installation).\n3. **Dependencies**: From `tools/postgres`, run `poetry install`.\n4. **Environment Variables:**\n `SYNC_SYNCSTORAGE__DATABASE_URL`: Database connection URL (e.g., `postgresql://user:pass@host:5432/syncstorage`). Used when `--database_url` is not supplied.\n\n### Usage\n\nRun the script using the following command:\n ```bash\n poetry run python purge_ttl.py [options]\n ```\n\n#### Options\n\n| Option | Description | Default |\n|---------------------------------|----------------------------------------------------------------------------------------|----------------------------------|\n| `-u`, `--database_url` | PostgreSQL DSN (`postgresql://...` or `postgres://...`). Required if the env var is unset. | `SYNC_SYNCSTORAGE__DATABASE_URL` |\n| `--collection_ids`, `--ids` | Comma-separated list of collection IDs to purge. | `[]` |\n| `--mode` | Purge mode: `batches`, `bsos`, or `both`. | `both` |\n| `--expiry_mode` | Expiry mode: `now` (current timestamp) or `midnight` (start of current day, UTC). | `midnight` |\n| `--dryrun` | Perform a dry run without making changes to the database. | `False` |\n\n#### Examples\n\n##### Example 1: Basic Purge\nPurge expired entries from both `batches` and `bsos` tables using the DSN from the environment:\n```bash\n poetry run python purge_ttl.py\n```\n\n##### Example 2: Specify the Database URL\n```bash\n poetry run python purge_ttl.py -u postgresql://user:pass@host:5432/syncstorage\n```\n\n##### Example 3: Filter by Collection IDs\nPurge only for specific collection IDs:\n```bash\n poetry run python purge_ttl.py --collection_ids [123,456,789]\n```\n\n##### Example 4: Purge a Single Table\n```bash\n poetry run python purge_ttl.py --mode bsos\n```\n\n##### Example 5: Perform a Dry Run\nTest the script without making actual changes:\n```bash\n poetry run python purge_ttl.py --dryrun\n```\n\n### Detailed Usage\n\n1. **Connecting to PostgreSQL**:\n - The script builds a SQLAlchemy engine from the supplied DSN. A `postgres://` scheme is normalized to `postgresql://` for newer SQLAlchemy versions.\n\n2. **Purge Modes**:\n - `batches`: Deletes expired entries from the `batches` table.\n - `bsos`: Deletes expired BSOs.\n - `both`: Executes purges on both `batches` and `bsos`.\n\n3. **Expiry Conditions**:\n - `now`: Purge entries that have already expired at the current timestamp.\n - `midnight`: Purge entries that expired at or before the start of the current UTC day.\n\n4. **Query Customization**:\n - An optional collection-ID filter is appended via the `add_conditions` helper using bound parameters.\n\n5. **Performance Monitoring**:\n - Metrics for execution duration and rows affected are logged and sent to StatsD for monitoring.\n\n6. **Dry Run**:\n - Enabling the `--dryrun` flag ensures that the queries are constructed and logged without executing them on the database.\n","docs/src/tools/purge_old_records_tokenserver.md":"# Documentation for `purge_old_records.py`\n\n## Summary\n\nThe `purge_old_records.py` script is an administrative utility for managing obsolete user records in Tokenserver. It removes outdated user records from the database and deletes associated data from storage nodes. This process helps reduce storage overhead, improve database performance, and maintain the health of the Tokenserver system.\n\nObsolete records are those replaced by newer records for the same user or marked for deletion if the user has deleted their account. The script can run in batch mode for periodic cleanup and includes options for dry-run testing and forced purging when nodes are down.\n\n---\n\n## Status\n - Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-purge-old-records-0`\n - See YAML configuration when editing each job.\n - See Kubernetes Engine Workload Panel in [sync-prod](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information. \n\n---\n\n## Specifics\n\n- **Primary Functionality**:\n - Deletes obsolete user records.\n - Issues delete requests to user storage nodes to purge related data.\n- **Optional Administrative Task**:\n - The script complements internal record replacement handled by the backend but is not mandatory for system operation.\n- **Batch Processing**:\n - Operates in loops, processing records in batches of a configurable size.\n- **Grace Period**:\n - Provides a grace period to avoid prematurely deleting recently replaced records.\n- **Dry Run**:\n - Offers a non-destructive mode for testing.\n\n---\n\n## Notes\n- **Regular Use**:\n - Running this script regularly can help maintain system performance and reduce storage overhead.\n- **Concurrency**:\n - When running multiple instances of this script, use the `--max-offset` option to reduce collisions.\n- **Forced Deletion**:\n - Use the `--force` option cautiously, especially if storage nodes are down.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3.7+ is installed.\n2. **Dependencies**:\n - Install required Python packages:\n `pip install requests hawkauthlib backoff`\n\n3. **Configuration**:\n - Set up access to the Tokenserver database.\n - Provide necessary metrics and logging configurations.\n\n### Usage\n\nRun the script using the following command:\n```bash\npython purge_old_records.py [options] secret\n```\n\n\n### Options\n\n| Option | Description | Default |\n|-------------------------|-------------------------------------------------------------------------------------------------|--------------|\n| `--purge-interval` | Interval in seconds to sleep between purging runs. | `3600` |\n| `--grace-period` | Grace period in seconds before deleting records. | `86700` (24h5m) |\n| `--max-per-loop` | Maximum number of items to process in each batch. | `10` |\n| `--max-offset` | Random offset to select records for purging, reducing collisions in concurrent tasks. | `0` |\n| `--max-records` | Maximum number of records to purge before exiting. | `0` (no limit) |\n| `--request-timeout` | Timeout in seconds for delete requests to storage nodes. | `60` |\n| `--oneshot` | Perform a single purge run and exit. | Disabled |\n| `--dryrun` | Test the script without making destructive changes. | Disabled |\n| `--force` | Force purging even if the user's storage node is marked as down. | Disabled |\n| `--override_node` | Specify a node to override for deletions if data is copied. | None |\n| `--range_start` | Start of UID range to process. | None |\n| `--range_end` | End of UID range to process. | None |\n| `--human_logs` | Enable human-readable logs. | Disabled |\n| `--last-seen-staging-table` | BigQuery last-seen staging table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-table`. | None |\n| `--last-seen-table` | BigQuery last-seen final table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-staging-table`. | None |\n\n### Examples\n\n#### Example 1: Basic Purge\nPerform a basic purge of obsolete user records:\n```bash\npython purge_old_records.py secret_key\n```\n#### Example 2: Grace Period and Dry Run\nPurge records with a 48-hour grace period in dry-run mode:\n```bash\npython purge_old_records.py --grace-period 172800 --dryrun secret_key\n```\n\n#### Example 3: Specify Range and Offset\nPurge records within a UID range with a random offset:\n\n```bash\npython purge_old_records.py --range_start uid_start --range_end uid_end --max-offset 50 secret_key\n```\n\n#### Example 4: Force Purge on Downed Nodes\nForce the deletion of data on downed nodes:\n```bash\npython purge_old_records.py --force secret_key\n```\n\n---\n\n## Detailed Usage\n\n1. **Batch Processing**:\n - The script processes records in batches defined by the `--max-per-loop` option.\n - Each batch is fetched from the database using random offsets to avoid overlapping with concurrent runs.\n\n2. **Grace Period**:\n - The grace period ensures that recently replaced records are not prematurely deleted.\n\n3. **Storage Node Cleanup**:\n - For each user, the script sends a delete request to their storage node to remove associated data.\n\n4. **Metrics Tracking**:\n - Tracks operations like user record deletions, service data deletions, and errors using metrics integration.\n\n5. **Error Handling**:\n - Uses exponential backoff to retry failed HTTP requests.\n - Detects loops in batch processing and raises exceptions.\n\n6. **Dry Run Mode**:\n - Simulates deletions without modifying the database or storage nodes, useful for testing.\n\n7. **Account Last-Seen Deletion**:\n - With both `--last-seen-*` options, each record retired by an FxA account deletion first has its account's rows deleted from the BigQuery last-seen tables, just before the record itself.\n - If that job fails or times out, the retired record stays, `replaced_at` is bumped, and it is retried after another grace period. Other records continue normally.\n - The default grace period (24 hours 5 minutes) outlasts the one-day OAuth token lifetime plus the upload delay, so an account's last activity normally reaches BigQuery before purge deletes it.\n\n---\n\n","tools/README.md":"# A collection of Sync Tools and utilities\n\nSee each directory for details:\n\n* [hawk](hawk) - a tool for generating test HAWK authorization headers\n* [spanner](spanner) - Google Cloud Platform Spanner tools for maintenance and testing\n* [payload-link-dataflow](payload-link-dataflow) - custom Dataflow flex template feeding the payload-link reconciliation pipeline\n* [payload-reconciler](payload-reconciler) - Pub/Sub-driven reconciler that finalizes and garbage-collects offloaded GCS payload objects\n## Installation\n\nThese tools are mostly written in python. It is recommended that you create a commonly shared virtual environment using something like:\n\n`python3 -m venv venv`\n\nto create a `/venv` directory. To activate this, call `sh /venv/bin/activate`.\n\nScript dependencies can be installed via poetry:\n\n```shell\npip install poetry\npoetry install\n```\n\nfor each tool.\n","tools/examples/README.md":"# Example scripts for working with syncstorage-rs\n\n* `put.bash` - First `chmod +x put.bash` then `./put.bash` to simulate a PUT.\n","tools/hawk/README.md":"# Make a Hawk compatible Auth header\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs `make_hawk_token.py` script, you'll need a Python =>3.10 development environment with `Poetry` installed. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `hawk/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/hawk\n # pyenv version install\n $ pyenv install 3.14\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.14 hawk # or whatever project name you like.\n $ pyenv local hawk # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\n`poetry run python make_hawk_token.py`\n\n## Create a Token Header:\n\nYou'll need to pass along your `SYNC_MASTER_SECRET` and the uri you'll be testing in order to generate a valid Hawk Id:\n\n`poetry run python make_hawk_token.py --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`\n\n** For testing against uri's using methods other than GET, you'll need to pass along the `--method` flag to generate your token. Ie, `poetry run python make_hawk_token.py --method PUT --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`. See [examples/put.bash](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for an example of this.\n\nUse `-h` for help.\n\nBy default, with no passed arguments, a default Hawk Id token will be generated mapping to your localhost:8000 with\nthe path `http://localhost:8000/1.5/1/storage/col2/`.","tools/payload-link-dataflow/README.md":"# payload-link-dataflow\n\nCustom Dataflow flex template that reads the `payload_link_changes`\nSpanner change stream, drops records whose every mod has `payload_link`\nNULL on both sides, and publishes the remainder to a Pub/Sub topic.\n\nThe downstream consumer is the `payload-reconciler` cronjob\n(`tools/payload-reconciler/`), which finalizes newly-committed GCS\npayload objects (`committed=true`, `customTime=MAX`) and deletes\norphaned ones.\n\n## Directory layout\n\nThe Java source tree under `src/` is a standalone Apache Beam pipeline\nwe own and build directly. It is intentionally minimal: it uses Beam's\nstock `SpannerIO.readChangeStream` and `PubsubIO.writeStrings`, plus the\nfilter and a small JSON serializer. We do **not** vendor or depend on\nthe upstream GoogleCloudPlatform/DataflowTemplates Java tree.\n\n| Path | Purpose |\n|---|---|\n| `pom.xml` | Maven project (Apache Beam + Jackson + SLF4J). |\n| `src/main/java/.../PayloadLinkChangesToPubSub.java` | The pipeline: read change stream → filter → serialize → publish. |\n| `src/main/java/.../PayloadLinkOptions.java` | Flex template parameters. |\n| `src/test/java/.../PayloadLinkChangesToPubSubTest.java` | Unit tests for the filter predicate. |\n| `metadata.json` | Flex template parameter schema (consumed by `gcloud dataflow flex-template build`). |\n| `Dockerfile` | Multi-stage build: Maven → flex template launcher base image. |\n| `upstream-customization.patch` | **Reference documentation only.** Unified diff showing the **filter delta only** against the upstream template; see \"Beyond the filter\" below for the rest. Not a build input. |\n| `upstream.txt` | The upstream commit SHA the reference patch was last reviewed against. Update when re-reviewing intent against upstream. |\n| `generate-full-delta.sh` | On-demand: produces a unified diff of the **complete** delta between upstream (at the pinned SHA) and our impl. Not maintained as a checked-in artifact -- runs against the source tree at execution time. |\n| `payload-link-publisher-py/` | Dev/E2E-only Python variant of this publisher (not Beam, not shipped to prod). See its own README. |\n| `README.md` | This file. |\n\n## Build\n\n```bash\ndocker build -t payload-link-dataflow:local tools/payload-link-dataflow\n```\n\n`mvn package` runs the unit tests, so a change that breaks the filter\ncontract fails the image build.\n\nFor a local non-Docker build:\n\n```bash\ncd tools/payload-link-dataflow\nmvn -B package\n```\n\nThe bundled jar lands at\n`target/payload-link-dataflow-bundled-1.0-SNAPSHOT.jar`.\n\n## Publish + launch (operator workflow; lives in webservices-infra)\n\n```bash\ndocker tag payload-link-dataflow:local /payload-link-dataflow:\ndocker push /payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/payload-link-dataflow.json \\\n --image /payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n\ngcloud dataflow flex-template run payload-link-dataflow- \\\n --template-file-gcs-location gs:///templates/payload-link-dataflow.json \\\n --parameters spannerProjectId= \\\n --parameters spannerInstanceId= \\\n --parameters spannerDatabase= \\\n --parameters spannerMetadataInstanceId= \\\n --parameters spannerMetadataDatabase= \\\n --parameters spannerMetadataTableName=Metadata_payload_link \\\n --parameters changeStreamName=payload_link_changes \\\n --parameters pubsubTopic=projects//topics/payload-link-changes\n```\n\n`spannerMetadataTableName` is optional but strongly recommended for\nstreaming launches -- see \"Job update and restart continuity\" below.\n\nThe Dataflow job's service account needs:\n- `roles/spanner.databaseReader` on the syncstorage database\n- `roles/spanner.databaseUser` on the metadata database (the change\n stream connector writes its own partition state there)\n- `roles/pubsub.publisher` on the destination topic\n- `roles/dataflow.worker`\n\n### Fine-grained access control (optional)\n\nPass `spannerDatabaseRole` to read the change stream under a Spanner\ndatabase role instead of a database-wide IAM grant. `payload_link_reader`\nin `syncstorage-spanner/src/schema.ddl` is that role: it holds `SELECT`\non `payload_link_changes` and `EXECUTE` on the generated\n`READ_payload_link_changes` table function, and nothing else.\n\nThe role applies to the change stream read only. Beam's\n`MetadataSpannerConfigFactory` deliberately does not copy `databaseRole`\nonto the connector's metadata database config, so that connection still\nuses the service account's own IAM grants. Two consequences, both from\n[Spanner's FGAC guidance](https://docs.cloud.google.com/spanner/docs/fgac-change-streams):\n\n- `spannerMetadataDatabase` **must** be a different database than\n `spannerDatabase`. The connector needs database-level write access for\n its partition state, and that grant would override the role's\n restrictions on the database being read.\n- the service account must **not** hold `roles/spanner.databaseUser` on\n `spannerDatabase` -- it bypasses fine-grained access control outright.\n Grant `roles/spanner.fineGrainedAccessUser`, plus\n `roles/spanner.databaseRoleUser` conditioned on the role resource,\n instead.\n\nIf the metadata tables share the syncstorage database, setting\n`spannerDatabaseRole` buys nothing security-wise: the role is assumed,\nbut the database-level grant the connector needs supersedes it.\nProvision a dedicated metadata database first.\n\n### Job update and restart continuity\n\nPass `spannerMetadataTableName` on any streaming (unbounded) launch so\n`--update` and cancel-then-restart can resume from the previous job's\nwatermarks. The connector generates the partition metadata table name\ninside `expand()` from a `UUID.randomUUID()` call, and that string\ngets baked into the pipeline graph; if the new job's binary produces\na different UUID, it reinitializes against a fresh empty table and\nreads from `startTimestamp` (defaulting to `Timestamp.now()`),\nsilently dropping records committed between the two jobs.\n\nPick any Spanner-legal identifier -- e.g. `Metadata_payload_link` -- and\nkeep it stable across launches. The connector creates the table on\nfirst initialization if it does not already exist, then reuses it.\nBeam's own recommendation: *\"you have to explicitly set the metadata\ntable name parameter of the new job when updating it. Use the value\nof the `metadataTable` pipeline option from the job you are\nupdating.\"*\n\nNotes on stop modes:\n\n- `--update`: preserves Dataflow's checkpointed state but does **not**\n transfer the metadata table binding -- pin the name or lose state.\n- `--cancel`: leaves the metadata table intact; the next launch with\n the same name resumes cleanly.\n- `--drain`: the change stream connector does not support drain\n (per [Google's guidance](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining)).\n Use `--cancel` or `--update` instead.\n\n## Filter behaviour\n\n`PayloadLinkChangesToPubSub.isPayloadLinkActionable` keeps a\n`DataChangeRecord` when any of its mods has a non-null `payload_link`\non either side. It drops only the case where **every** mod has both\nold and new `payload_link` NULL -- the inert INSERT/DELETE noise from\ncolumn-scoped change streams. Malformed records pass through so the\nreconciler / DLQ -- not this filter -- surfaces them.\n\n## Beyond the filter: other differences from upstream\n\n`upstream-customization.patch` expresses only the filter step. Our\npipeline additionally diverges from upstream's\n`Cloud_Spanner_Change_Streams_to_PubSub` template in ways that diff\ndoes not attempt to capture. Run `./generate-full-delta.sh` to see\nthe complete diff against the pinned upstream SHA; the substantive\ndifferences are:\n\n- **Sink swap.** `PubsubIO.writeStrings` + a custom `serializeRecord`,\n not upstream's `FileFormatFactorySpannerChangeStreamsToPubSub`.\n Emits raw JSON strings; no PubsubMessage attributes.\n- **JSON wire format.** Emits only `{commitTimestamp, modType,\n tableName, mods[]}`. Omits fields upstream would emit:\n `serverTransactionId`, `rowType[]`,\n `numberOfRecordsInTransaction`,\n `numberOfPartitionsInTransaction`, `partitionToken`,\n `recordSequence`, `isLastRecordInTransactionInPartition`,\n `valueCaptureType`.\n- **No `UncaughtExceptionLogger.register()`.** Uncaught exceptions\n flow to the runner's default handler.\n- **No `@Template` / `TemplateCategory` annotations.** Upstream uses\n the `com.google.cloud.teleport.metadata` annotation-driven\n registrar for flex-template metadata. We register via\n `metadata.json` + `gcloud dataflow flex-template build` instead --\n simpler, no annotation-processor dependency.\n- **No support for these upstream options:**\n `useSpannerEmulatorHost`, `spannerHost`,\n `spannerChangeStreamTvfNameList`, `outputMessageMetadata`,\n `outputDataFormat` (JSON/Avro switch), `pubsubAPI`. Our\n `PayloadLinkOptions` deliberately has a smaller surface (~12\n options vs. upstream's ~25).\n- **No ValueProvider indirection on options.** Upstream wraps\n several config fields in `ValueProvider` for template\n parameterisation; we take plain strings.\n- **No explicit `streaming=true` mutation.** Beam infers streaming\n from the source (`SpannerIO.readChangeStream` is\n `@UnboundedPerElement`) and `DataflowRunner` flips the flag at\n translation time.\n\nThese deltas were deliberate scope choices, not oversights. The\nfilter-only patch is kept as the primary review artifact because\nit's small enough to eyeball, applies cleanly against the pinned\nSHA, and is insensitive to upstream churn on features we don't use.\n\n`main()` matches upstream on two pipeline-shape invariants that are\nnot deployment choices for this source: it appends `use_runner_v2`\nto `options.experiments` and calls `setEnableStreamingEngine(true)`.\n`BundleFinalizer` (used by `SpannerIO.readChangeStream`) requires the\nDataflow Portable Runner (\"Runner V2\"), and V2 streaming jobs\nrequire Streaming Engine -- so both settings are pinned in code\nrather than left to launch-time flags.\n\n## Output wire format\n\nEach surviving `DataChangeRecord` is serialized to a JSON Pub/Sub\nmessage of the form:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-29T12:34:56.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nEach mod field carries a **JSON string** (escaped) -- the downstream\nPython reconciler reads them with a second `json.loads` per mod. This\nmatches the Spanner change-streams wire convention.\n\n## Running against the compose e2e stack\n\nThe `docker_run_reconciliation_e2e_tests` Makefile target brings up\nSpanner + Pub/Sub + fake-GCS emulators and a publisher container\nlocally. There are two publisher variants; both produce the exact same\nPub/Sub wire format so the reconciler consumes either\nindistinguishably.\n\n### Default: Python publisher\n\n`docker/docker-compose.e2e.reconciliation.yaml` builds and runs the\nPython variant at `payload-link-publisher-py/`. Sub-second startup,\nno JVM. Use this for day-to-day iteration.\n\n```bash\nmake docker_run_reconciliation_e2e_tests\n```\n\n### Swap-in: this Java publisher\n\n`docker/docker-compose.e2e.reconciliation.java.yaml` overrides the\n`payload-link-publisher` service to build **this** directory's Docker\nimage and run the shaded JAR under Beam's DirectRunner (bypassing the\nflex-template launcher's Dataflow-submission entrypoint). Use when you\nneed to reproduce a prod issue against the actual Java pipeline or\nverify wire-format parity between the two variants.\n\n```bash\ndocker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.reconciliation.java.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n up --exit-code-from e2e-tests --abort-on-container-exit\n```\n\nExpect a ~15-30s JVM+Beam warmup and a heavier memory footprint than\nthe Python variant. If Docker's default resource limits are tight on\nyour dev machine, bump them.\n\n## Keeping the reference patch accurate\n\n`upstream-customization.patch` is reference documentation, not a build\ninput. If upstream's pipeline shape evolves or our pipeline drifts in\nways that change the conceptual diff, regenerate it:\n\n```bash\ngit clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git upstream\ncd upstream\ngit checkout \n# Manually re-author the filter customization against the current\n# upstream SpannerChangeStreamsToPubSub.java (in our src/ we own the\n# whole pipeline; here we only need to express the *delta*).\ngit diff > ../tools/payload-link-dataflow/upstream-customization.patch\necho \"\" > ../tools/payload-link-dataflow/upstream.txt\n```\n\nAfter updating the SHA, run `./generate-full-delta.sh` and skim the\noutput to check that our impl's other divergences (sink swap, JSON\nshape, dropped options -- see \"Beyond the filter\" above) still make\nsense against upstream's new state. If upstream restructured\nsomething we deliberately dropped, update the \"Beyond the filter\"\nbullets accordingly.\n\nThis is purely a documentation refresh -- it does not affect the build.\n","tools/payload-link-dataflow/payload-link-publisher-py/README.md":"# payload-link-publisher-py\n\nDev/E2E-only Python variant of the payload-link Dataflow publisher.\n\n**Prod uses the Java flex template one directory up.** This variant\nexists so the docker-compose e2e stack can run without a JVM, without\nDataflow, and without minute-scale iteration times.\n\n## Design summary\n\n- **Not Apache Beam.** Python's Beam SDK has no Spanner change-stream\n connector; a Beam pipeline in Python would either require a\n cross-language Java expansion (JVM back in) or a custom Source\n (more code than the whole non-Beam approach). Instead we poll the\n `READ_payload_link_changes` TVF directly via\n `google.cloud.spanner.Client`.\n- **Wire-format identical to the Java job.** `serialize_record`\n produces the exact JSON shape `PayloadLinkChangesToPubSub.serializeRecord`\n produces, so the reconciler consumes either interchangeably.\n- **Partition tracking.** Follows partition splits: reads `_root`,\n picks up child-partition tokens from any `child_partitions_record`,\n reads each child from its advertised `start_timestamp`, and retires\n a parent when a child announces it. Handles `OUT_OF_RANGE` by\n dropping ended partitions. Required in practice — the Spanner\n emulator routes DataChangeRecords to child partitions almost\n immediately, so a `_root`-only reader sees zero DCRs.\n- **In-memory cursor.** On restart, resumes from `now - POLL_OVERLAP_SECONDS`\n on `_root`. Fine for tests (compose brings the pod up alongside the\n stream); would lose events across restarts in prod.\n- **Hardcoded row-shape offsets.** Rather than introspect the\n `READ_` TVF's ChangeRecord / DataChangeRecord /\n ChildPartitionsRecord STRUCT layout at runtime, `publisher.py`\n documents the field positions and accesses them positionally. If a\n future Spanner rev reorders the STRUCT, tests fail loudly and one\n offset map gets updated. Acceptable trade-off for a dev/E2E tool.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-spanner`, `google-cloud-pubsub`, `statsd`. |\n| `publisher.py` | Entry point: poll TVF → filter → serialize → publish. |\n| `utils.py` | `extract_payload_link`, `is_payload_link_actionable` — mirrors the Java filter. |\n| `test_publisher.py` | Unit tests for the pure functions. |\n| `Dockerfile` | `python:3.13-slim` + poetry install. |\n\n## Environment\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `SPANNER_PROJECT_ID` | yes | — | |\n| `SPANNER_INSTANCE_ID` | yes | — | |\n| `SPANNER_DATABASE_ID` | yes | — | |\n| `PUBSUB_PROJECT_ID` | yes | — | |\n| `PUBSUB_TOPIC` | yes | — | e.g. `payload-link-changes` |\n| `SPANNER_CHANGE_STREAM_NAME` | no | `payload_link_changes` | |\n| `HEARTBEAT_MS` | no | `1000` | TVF heartbeat interval. |\n| `POLL_OVERLAP_SECONDS` | no | `5` | Restart re-reads this window. |\n| `SPANNER_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-spanner`. |\n| `PUBSUB_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-pubsub`. |\n| `STATSD_HOST` / `STATSD_PORT` | no | — | Standard `statsd.defaults.env` pair. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-link-dataflow/payload-link-publisher-py\npoetry install\npoetry run pytest\npoetry run mypy .\npoetry run ruff check .\n```\n\n`poetry.lock` is generated on first install (not yet checked in).\n\nUnit tests cover `extract_payload_link`, `is_payload_link_actionable`,\n`serialize_record`, and `_to_json_string`. Not covered by unit tests:\n`poll_partition` (needs a live Spanner emulator) and\n`publish_if_actionable` (needs a Pub/Sub emulator). Both are\nexercised end-to-end by the docker-compose reconciliation stack\n(`make docker_run_reconciliation_e2e_tests`).\n\n## Metrics\n\n- `payload_link_publisher.published` — records published to Pub/Sub\n- `payload_link_publisher.filtered` — inert records dropped by\n `is_payload_link_actionable`\n- `payload_link_publisher.errors.publish` — publish failures\n\n## Related\n\n- Java flex template (prod): `tools/payload-link-dataflow/` (parent dir)\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (`payload_link_changes`)\n- Downstream reconciler: `tools/payload-reconciler/`\n- Runbook: `docs/src/tools/payload_link_reconciler.md`\n","tools/payload-reconciler/README.md":"# payload-reconciler\n\nPub/Sub-driven reconciler that closes the loop on GCS payload offload.\n\nReads the `payload-link-changes` Pub/Sub topic (populated by the\n`tools/payload-link-dataflow/` flex template). For each mod inside each\nchange record:\n\n- **New `payload_link` non-null** → patch the referenced GCS object's\n metadata to `committed=true` and `customTime=MAX`. Completes the\n 2-phase commit started in `syncserver/src/web/payload_offload.rs`.\n- **Old `payload_link` non-null and ≠ new** → delete the referenced GCS\n object. It is unreferenced now (the row's `payload_link` moved, was\n cleared, or the row was deleted — including TTL row-deletion-policy\n deletes).\n\nBoth operations are idempotent (re-setting metadata is a no-op; 404 on\ndelete is treated as success), so Pub/Sub at-least-once redelivery is\nsafe and no ordering key is needed.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-pubsub`, `google-cloud-storage`, `datadog`. |\n| `reconcile_payload_links.py` | Entry point: sync-pull drain loop + per-message handler. |\n| `utils.py` | `parse_gs_url()` — mirrors the Rust uploader's URL grammar. |\n| `test_utils.py` | Tests for `parse_gs_url`. |\n| `test_reconcile_payload_links.py` | Handler tests with a mocked GCS client. |\n\n## Environment\n\n| Variable | Required | Notes |\n|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | e.g. `payload-link-reconciler-sub`. |\n| `GCS_PAYLOAD_BUCKET` | yes | The bucket the syncserver writes payloads into. Cross-bucket links abort the message and route it to the DLQ. |\n| `RUN_BUDGET_SECONDS` | no | Selects deployment mode. **Set** (e.g. `240`): cronjob mode -- drain up to that many seconds then exit 0. **Unset**: long-running mode -- poll forever, never exit on idle; the deployment supervises restarts. Cronjob mode is the default deployment. |\n| `SYNC_STATSD_HOST` | no | statsd server host. |\n| `SYNC_STATSD_PORT` | no | statsd server port. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-reconciler\npoetry install\npoetry run pytest\n```\n\nUnit tests cover `parse_gs_url` and the per-message handler. The\n`drain` loop (sync-pull, budget vs. long-running exit conditions) has\nno unit coverage — exercise it against the Pub/Sub emulator when\nchanging it.\n\n## Metrics\n\nStatsd counters (all under the `payload_reconciler.` prefix):\n\n- `finalizes` — successful `committed=true` + `customTime=MAX` patch\n- `orphan_deletes` — successful delete of an unreferenced object\n- `noop_skips` — message processed with zero ops. Should be ~0 if the\n Dataflow filter is working; non-zero is the alarm bell that the\n filter regressed.\n- `gcs_404` + `op:finalize` / `op:delete` — 404 on the respective op.\n Expected non-zero under at-least-once redelivery (idempotency tax);\n a sharp rise suggests upstream state is gone unexpectedly.\n- `errors` + `kind:handler` — Sustained non-zero values feed the DLQ.\n\n## Deployment\n\n**The reconciler ships in the primary syncserver image**, built from\nthe root `Dockerfile`. Its script and Python deps are baked in via\nthat Dockerfile's per-tool poetry-export / pip-wheel / pip-install\nchain; K8s picks between the syncserver binary and the reconciler\nscript via the pod's `command:`.\n\nDefault: K8s cronjob (definition lives in webservices-infra), cadence\n~5 min, exports `RUN_BUDGET_SECONDS` (e.g. `240`).\n\nAlternative: long-running K8s Deployment. Omit `RUN_BUDGET_SECONDS`;\nthe pod polls forever. Choose this when finalize-flip latency below\nthe cronjob cadence matters (e.g. an operator wants offloaded\npayloads to become `committed=true` within seconds of upload rather\nthan up to one cronjob cycle later).\n\nService account needs the same roles in either mode:\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and `delete` operations)\n\n## Related\n\n- Producer: `syncserver/src/web/payload_offload.rs`\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (look for `payload_link_changes`)\n- Upstream pipeline: `tools/payload-link-dataflow/`\n","tools/postgres/README.md":"# PostgreSQL Utility Scripts\n\nCurrently there is one script: `purge_ttl.py`.\n\n## purge_ttl.py\n\nPurge expired items (BSOs and batches) from a Sync storage PostgreSQL instance.\nThe script is intended to be invoked regularly, e.g. as a cronjob.\n\n### Prerequisites\n\n- Python 3\n- [Poetry](https://python-poetry.org/docs/#installation)\n\n### Run Script Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run python purge_ttl.py\"\n```\n\nPass `-h` to see the list of options.\n\n### Run Tests Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run pytest test_purge_ttl.py -v\"\n```\n\n### Run with Docker\n\nA Docker image with the PostgreSQL build is not published anywhere at the\nmoment. Once that's available, this readme will be updated to include that\nstep.\n\n```bash\ndocker run --rm \\\n --entrypoint python3 \\\n -e SYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@host:5432/syncstorage\" \\\n syncstorage-postgres-image:tag \\\n /app/tools/postgres/purge_ttl.py\n```\n\n","tools/spanner/README.md":"# Spanner Tools and Scripts\n\nThese tools are supplemental scripts for working with the Google Cloud Platform. Follow [the general installation instructions](https://cloud.google.com/spanner/docs/getting-started/python/), as well as fetch the proper service account credentials file.\n\nRemember, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable should point to the absolute path location of your service account credential file.\n\ne.g.\n```bash\nGOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/project-id-service-cred.json venv/bin/python count_expired_rows.py\n```\nSee each script for details about function and use.\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs spanner untilities, you'll need a Python =>3.12 development environment with `Poetry` installed. These scripts typically run in a deployed GCP workflow, so their Dockerfile config will generally prepare all of this for you. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `spanner/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/spanner\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 spanner # or whatever project name you like.\n $ pyenv local spanner # activates virtual env whenever you enter this directory.\n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python count_expired_rows.py`\n","tools/syncstorage-loadtest/README.md":"# Storage load test\n\n**Note:** This requires Python version 3.10+\n\nThe tests can execute in three modes:\n\n1. **Direct access**: use a known master secret\n2. **Firefox accounts OAuth**: create test accounts and use OAuth JWTs\n3. **Self-signed JWT**: generate and sign JWTs locally with a given private key\n\n## Installation\n\n### Environment Setup\n\nTo run the syncstorage load tests, you'll need a Python >=3.10 development environment with `Poetry` installed. You can also directly use `poetry run` to execute commands as described in the usage examples below.\n\nThe easiest solution is to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments as a way to isolate the dependencies from other projects.\n\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\n See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Example:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n\n4. Install Python version, create virtualenv, activate and install dependencies from inside the project directory.\n\n **Note:** You can skip creating a virtual environment and invoke commands directly using `poetry run`.\n\n ```shell\n $ cd syncstorage-loadtest\n\n # Install Python 3.10+\n $ pyenv install 3.10\n\n # Create named, associated virtualenv\n $ pyenv virtualenv 3.10 syncstorage-loadtest # or whatever name you prefer\n $ pyenv local syncstorage-loadtest # activates virtual env whenever you enter this directory\n\n # Install Poetry and dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. Once you're in your virtual environment, run the load tests using:\n ```bash\n poetry run molotov [options] loadtest.py\n ```\n\n### Quick Install\n\nIf you already have Poetry installed:\n\n```bash\npoetry install\n```\n### Generate Key\n Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```bash\npoetry run ./generate-keys.sh\n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```bash\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n```\n## Mode 1: Direct Access\n\nWith a known syncstorage master secret:\n\n```bash\nSERVER_URL=\"http://localhost:8000#secretValue\" poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n## Mode 2: Firefox Accounts OAuth\n\nWith FxA stage accounts:\n\n```bash\nSERVER_URL=\"https://token.stage.mozaws.net\" \\\n FXA_API_HOST=\"https://api-accounts.stage.mozaws.net\" \\\n FXA_OAUTH_HOST=\"https://oauth.stage.mozaws.net\" \\\n poetry run molotov --workers 3 --duration 60 -v loadtest.py\n```\n\n## Mode 3: Self-Signed JWTs\n\nAssuming an RSA keypair was generated, the private key pem was saved and accessible, and the token server was started with the public key JWK configured. The presence of `OAUTH_PRIVATE_KEY_FILE` triggers this mode when using OAuth.\n\nRun with self-signed JWTs:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n## Expanded payloads (GCS offload)\n\nThe batch write path can generate larger-than-default payloads so you can load\ntest collections whose payloads are offloaded to Google Cloud Storage. These\nknobs are opt-in via environment variables; unset, the load test behaves\nexactly as before.\n\n| Variable | Default | Effect |\n|---|---|---|\n| `LARGE_PAYLOAD_PROB` | `0.0` | Fraction (`0.0`–`1.0`) of BSOs given an expanded payload. `1.0` makes **every** payload large; `0.0` disables the feature. |\n| `LARGE_PAYLOAD_SIZE` | *(unset)* | Explicit target size in bytes for large payloads. When unset, large payloads target the server's `max_record_payload_bytes`. |\n| `OFFLOAD_COLLECTIONS` | *(unset)* | Comma-separated collection names to target for batch writes. Must match the server's `gcs_payload_offload_collections`. When unset, the default collections are used. |\n\n`LARGE_PAYLOAD_PROB`/`LARGE_PAYLOAD_SIZE` control payload **size**; whether a\nwrite is offloaded to GCS is a separate, per-collection server setting. To drive\na fully offloaded collection at maximum size, set `OFFLOAD_COLLECTIONS` to that\ncollection and `LARGE_PAYLOAD_PROB=1.0`.\n\nIndividual payloads are capped at the target collection's\n`max_record_payload_bytes`, and each batch is kept under its `max_post_bytes`,\nso a large enough payload results in fewer records per request.\n\nExample — every write is a 5 MiB payload sent to the `tabs` collection:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n LARGE_PAYLOAD_PROB=1.0 \\\n LARGE_PAYLOAD_SIZE=5242880 \\\n OFFLOAD_COLLECTIONS=tabs \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n### Server-side prerequisites\n\nTo actually exercise expanded/offloaded payloads end to end, the target server\nmust be configured for it:\n\n- `gcs_payload_bucket` set and the target collection listed in\n `gcs_payload_offload_collections` (syncstorage settings).\n- Raised limits to allow payloads beyond the 2.5 MiB default:\n `max_record_payload_bytes`, `max_post_bytes`, `max_request_bytes`, and\n `max_total_bytes`. These are read from `/info/configuration`. A collection\n may also raise its own limits via the `collections` section of that\n response; the load test resolves limits per target collection, preferring a\n collection's override and falling back to the global value.\n- Any front-end proxy (e.g. nginx) request body-size limit raised to match.\n\nNote: many workers each holding multi-MiB payloads is a real memory footprint\non the load-generating host — size the harness accordingly.\n\n## Docker\n\nBuild and run this directory's image:\n\n```bash\ndocker build . --tag syncstorage-loadtest:local\n\ndocker run --rm --net=host \\\n -e SERVER_URL=\"http://localhost:8000#secretValue\" \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 1 --workers 5 --duration 60 -v loadtest.py\"\n```\n\n## Full offload pipeline on docker\n\nFor the offload and change-stream load tests (STOR-628, STOR-629) there is a\ncompose rig that stands up syncserver with offload enabled, a Spanner emulator\ncarrying the change stream, Pub/Sub, GCS, the publisher, the reconciler and\nthis load test, then reports whether the pipeline drained clean:\n\n```bash\nmake loadtest-offload-image\nmake loadtest-offload-handler # 100% offloaded\nmake loadtest-offload-report\nmake loadtest-offload-down\n```\n\nSee `docs/src/tools/load-testing.md` for the runbook, the knobs, and the GCP\ndev path.\n\nHappy Breaking!\n","tools/tokenserver/README.md":"# Summary of Files in tools/Tokenserver\n\n| File Name | Description |\n|--------------------------------|-----------------------------------------------------------------------------|\n| `add_node.py` | Adds new node to the tokenserver's database, registering it for user allocation. |\n| `allocate_user.py` | Script to allocate a specific user to a node. |\n| `conftest.py` | Pytest configuration and fixtures for shared test setup. |\n| `count_users.py` | Script to emit total-user-count metrics for exec dashboard. |\n| `database.py` | Shared database utility queries and functions used by multiple scripts. |\n| `purge_old_records.py` | Script to purge user records that have been replaced. |\n| `pytest.ini` | Configuration file for pytest, specifying options like test output format. |\n| `remove_node.py` | Script to remove a node from the system. |\n| `test_database.py` | Unit tests for `database.py`. |\n| `test_purge_old_records.py` | Tests for `purge_old_records.py`, validating cleanup operations. |\n| `test_scripts.py` | Testing module to test the various scripts in this directory. |\n| `unassign_node.py` | Removes a node from the system and clears any assignments to the named node.| \n| `update_node.py` | Script to update node status in the db. |","tools/tokenserver/loadtests/README.md":"# Tokenserver Load Tests\n\nThis directory contains everything needed to run the suite of load tests for Tokenserver.\n\n## Prerequisite Dependency and Environment Setup:\nTo use the syncstorage-rs tokenserver load tests, you'll need a Python =>3.12 development environment with `Poetry` installed. This testing script can be run within a spun up GCP workflow where you can execute the locust commands from the UI. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `loadtests/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/tokenserver/loadtests\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 loadtests # or whatever project name you like.\n $ pyenv local loadtests # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python locustfile.py`\n\n\n## Building and Running\n\n1. Install the load testing dependencies as described above:\n\n ```sh\n poetry install\n ```\n\n2. Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```sh\npoetry run ./generate-keys.sh \n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```sh\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n ```\n\n3. Set the following environment variables/settings on Tokenserver:\n\n ```sh\n # Should be set to the \"n\" component of the JWK\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N\n # Should be set to the \"e\" component of the JWK (this value should almost always be \"AQAB\")\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY=RSA\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__USE=sig\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__ALG=RS256\n\n # These two environment variables don't affect the load tests, but they need to be set:\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID=\"\"\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__FXA_CREATED_AT=0\n ```\n\n Note that, because these settings cache the JWK used to verify OAuth tokens, no verification requests will be made to FxA, so the value of `SYNC_TOKENSERVER__FXA_OAUTH_VERIFIER_URL` does not matter; however, Tokenserver expects it to be set, so setting it to something like `http://localhost` will suffice.\n\n4. Set the following environment variables on the machine that will be running the load tests:\n\n - `OAUTH_PEM_FILE` should be set to the location of the private RSA key generated in a previous step\n\n5. Tokenserver uses [locust](https://locust.io/) for load testing. To run the load tests, simply run the following command in this directory:\n\n ```sh\n locust\n ```\n\n6. Navigate your browser to , where you'll find the locust GUI. Enter the following information:\n\n - Number of users: The peak number of Tokenserver users to be used during the load tests\n - Spawn rate: The rate at which new users are spawned\n - Host: The URL of the server to be load tested. Note that this URL must include the protocol (e.g. \"http://\")\n\n7. Click the \"Start swarming\" button to begin the load tests.\n\n## Populating the Database\n\nThis directory includes an optional `populate_db.py` script that can be used to add test users to the database en masse. The script can be run like so:\n\n```sh\npoetry run python populate_db.py \n```\n\nOr if in built virtualenv:\n```sh\npython3 populate_db.py \n```\n\nwhere `sqluri` is the URL of the Tokenserver database, `nodes` is a comma-separated list of nodes **that are already present in the database** to which the users will be randomly assigned, and `number of users` is the number of users to be created.\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# syncstorage-rs\n\nFirefox Sync storage and token server, written in Rust. The workspace contains a\nRust multi-crate server plus Python utilities and integration/end-to-end tests.\n\n## Agent rules\n\nSecurity takes absolute precedence. This repository handles synchronization of\nuser data and token management to access storage nodes.\n\n### 1) Scope & writes\n\n- Operate strictly within this repo root; normalize paths; do not follow\n symlinks outside the repo.\n- Writes are allowed to the working tree. Always present a diff for review\n before any staging or commit.\n- Do not modify files adjacent to a requested change. Mention issues found\n but do not fix them unless asked.\n- Ask before running any command (build, test, install, DB ops, git,\n service management). Do not run `git add`, `git commit`, `git push`, or\n `git rebase` unless explicitly instructed.\n\n### 2) Non-negotiables\n\n- **Secrets:** never read, print, summarize, or transmit secret files or\n values. Use placeholders (`YOUR_API_KEY_HERE`) in examples.\n- **External network:** only with explicit approval to a trusted, documented\n endpoint.\n- **Pipelines & contracts:** flag breaking API or contract changes. Do not\n alter GitHub Actions defined in `.github/`, any configuration for CI/CD workflows or git hooks without explicit, reviewed\n justification.\n- **Published DB migrations:** never edit existing published migration files or schema definitions (ex. `syncstorage-spanner/src/schema.ddl`), whether for MySQL, Postgres, or Spanner.\n Always add a new forward migration and a separate rollback migration.\n- **Workspace recommendations:** the repo deliberately ships no\n `.vscode/extensions.json`. Do not re-add one. Useful extensions are listed\n in `.vscode/README.md` for contributors to install at their own discretion.\n\n### 3) Do-not-touch paths\n\nNever read or write these paths. This list mirrors `.gitignore` — if there is\na discrepancy, `.gitignore` is the source of truth.\n\n```\nservice-account.json # GCP credentials\n.sentryclirc # Sentry auth token\n.envrc # direnv secrets\nconfig/local.toml # local server config (may contain secrets)\ntools/tokenserver/loadtests/*.pem\ntools/tokenserver/loadtests/*.pub\ntarget/ # Rust build artifacts\n*.xml # test result files\nopenapi.json # generated artifact\nbook/ # generated docs\ndocs/output/ # generated docs\nvenv/ # Python virtual environments\n.install.stamp # Poetry install marker\nworkspace/ # CI artifact directory\n```\n\n### 4) Git commit messages\n\nFollow the commit message format defined in [CONTRIBUTING.md](CONTRIBUTING.md) Git Commit Guidelines.\n\n- Format: `type(scope): subject` — `type` is one of `feat`, `fix`, `docs`,\n `style`, `refactor`, `perf`, `test`, `chore`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Body: explain motivation and contrast with previous behavior\n- Footer: `Closes STOR-1234` (Mozilla engineers — matches the Jira ticket) or\n `Closes #N` / `Issue #N` (community contributors — matches the GitHub issue number);\n `BREAKING CHANGE:` for breaking changes\n- Branch naming (Mozilla engineers): `type/description-STOR-1234` where `1234` must match the numeric suffix of the associated Jira ticket\n- Branch naming (community): `type/description-1112`\n- All commits must be GPG/SSH signed (enforced by CI)\n\nExample:\n\n```\nfeat: add node reassignment on capacity overflow\n\nNodes at 100% capacity previously returned 503 rather than\nassigning the user to an available node.\n\nCloses #42.\n```\n\n## Project layout\n\n```\nsyncserver/ Main Actix-web server (entry point)\nsyncstorage-db/ Syncstorage database abstraction layer\nsyncstorage-db-common/ Syncstorage shared database types and traits\nsyncstorage-mysql/ MySQL backend for syncstorage\nsyncstorage-postgres/ PostgreSQL backend for syncstorage\nsyncstorage-spanner/ Google Cloud Spanner backend for syncstorage\nsyncstorage-settings/ Syncstorage configuration types\ntokenserver-db/ Tokenserver database abstraction layer\ntokenserver-db-common/ Tokenserver shared database types\ntokenserver-mysql/ MySQL backend for tokenserver\ntokenserver-postgres/ PostgreSQL backend for tokenserver\ntokenserver-auth/ HAWK token generation and verification\ntokenserver-common/ Shared tokenserver utilities\ntokenserver-settings/ Tokenserver configuration types\nsyncserver-common/ Shared server utilities\nsyncserver-db-common/ Shared database utilities\nsyncserver-settings/ Syncserver configuration types\nglean/ Glean telemetry crate\ntools/\n integration_tests/ Pytest-based integration + e2e tests\n tokenserver/ Tokenserver utilities and load tests\n spanner/ Spanner utilities\n hawk/ HAWK token generation utility\n syncstorage-loadtest/ Molotov load tests\ndocker/ Docker Compose files for each backend\nconfig/ Local TOML configuration files\ndocs/ mdBook documentation source\n.github/ GitHub CI Actions and configuration\n```\n\n## Rust builds\n\nThe default feature set targets MySQL, however production uses the Spanner database with MySQL Tokenserver. Backend features are mutually exclusive.\n\n```bash\n# MySQL (default)\ncargo build\n\n# PostgreSQL\ncargo build --no-default-features --features=syncstorage-db/postgres --features=tokenserver-db/postgres --features=py_verifier\n\n# Spanner\ncargo build --no-default-features --features=syncstorage-db/spanner --features=py_verifier\n```\n\n## Rust tests\n\nTests require `RUST_TEST_THREADS=1` (many tests share a real database).\n\n```bash\n# MySQL unit tests (default DATABASE_URL envs set in Makefile)\nmake test\n\n# Specific package\nmake test ARGS=\"--package syncstorage-db\"\n\n# With coverage\nmake test_with_coverage # MySQL\nmake postgres_test_with_coverage\nmake spanner_test_with_coverage\n```\n\n## Clippy / formatting / audit\n\nFormatting checks are necessary before applying any code changes, as they\ncan capture problematic patterns or compiler issues in advance.\n\n```bash\ncargo fmt -- --check # format check\ncargo fmt # auto-format\n\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\n\nmake clippy_release_mysql # release-mode (catches dead code etc.)\nmake clippy_release_postgres\nmake clippy_release_spanner\n\ncargo audit # check dependencies for known CVEs (run by CI on every push)\n```\n\nAfter editing Rust code, run clippy and the unit tests for the affected backend before finishing the task; the `rust-code-smell` skill covers changed files across backends.\n\n## Python tooling\n\nAll Python tooling uses [Poetry](https://python-poetry.org/). The root\n`pyproject.toml` covers the `tools/` tree. Each subdirectory under `tools/` has\nits own `pyproject.toml` for its specific dependencies.\n\nAll of these Python checks must be run with any related change to a Python\nutility or test.\n\n```bash\nmake install # install root dependencies\nmake integration-test # install integration test dependencies\nmake tokenserver # install tokenserver utility dependencies\n```\n\nLint / format / type-check:\n\n```bash\nmake ruff-lint # ruff check tools/\nmake ruff-format # ruff format tools/\nmake ruff-fmt-chk # format diff check\nmake mypy # mypy type checking\nmake pydocstyle # docstring validation\nmake bandit # security linting\n```\n\n## Integration / E2E tests\n\n### Local (against a running server)\n\n```bash\n# Run all integration tests except real FxA e2e\nPYTHONPATH=$(pwd)/tools \\\nSYNC_MASTER_SECRET=secret0 \\\nSYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=http://localhost:6000 \\\nTOKENSERVER_HOST=http://localhost:8000 \\\npoetry -C tools/integration_tests run pytest . --ignore=tokenserver/test_e2e.py\n```\n\nOr via the Makefile shortcut (sets the env vars above):\n\n```bash\nmake run_local_e2e_tests\n```\n\n### Docker-based E2E (full stack)\n\n```bash\nmake docker_run_mysql_e2e_tests\nmake docker_run_postgres_e2e_tests\nmake docker_run_spanner_e2e_tests\n```\n\nEach command spins up two compose stacks (with and without JWK cache), copies\nJUnit XML results out of the container, then tears everything down.\n\n## Running the server locally\n\n```bash\n# MySQL\nmake run_mysql\n\n# Spanner (requires GCP service account key at ./service-account.json)\nmake run_spanner\n```\n\nServer listens on `http://localhost:8000`. Health check: `GET /__heartbeat__`.\n\n## Key environment variables\n\n| Variable | Purpose |\n|---|---|\n| `SYNC_MASTER_SECRET` | HAWK token signing secret |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | Syncstorage DB connection string |\n| `SYNC_TOKENSERVER__DATABASE_URL` | Tokenserver DB connection string |\n| `SYNC_TOKENSERVER__NODE_TYPE` | `mysql` / `postgres` / `spanner` |\n| `SYNC_TOKENSERVER__RUN_MIGRATIONS` | Auto-migrate on startup (`true`/`false`) |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | FxA OAuth endpoint |\n| `RUST_TEST_THREADS` | Set to `1` for all test runs |\n| `RUST_LOG` | Log level (`debug`, `info`, etc.) |\n| `SYNC_TEST_LOG_HTTP` | Set any value to log all HTTP req/resp in integration tests |\n| `SYNC_SERVER_URL` | Integration test base URL (default: `http://localhost:8000`) |\n| `TOKENSERVER_HOST` | Tokenserver host for integration tests |\n| `PYTHONPATH` | Must point to `$(pwd)/tools` when running integration tests |\n\n## Docker Compose files\n\n| File | Purpose |\n|---|---|\n| `docker/docker-compose.mysql.yaml` | MySQL syncstorage + MySQL tokenserver |\n| `docker/docker-compose.postgres.yaml` | PostgreSQL syncstorage + PostgreSQL tokenserver |\n| `docker/docker-compose.spanner.yaml` | Spanner emulator + MySQL tokenserver |\n| `docker/docker-compose.e2e.{mysql,postgres,spanner}.yaml` | E2E test runner overlay |\n| `docker/docker-compose.e2e.jwk-cache.yaml` | Enable JWK caching overlay |\n| `docker/docker-compose.e2e.no-jwk-cache.yaml` | Disable JWK caching overlay |\n\n## Documentation\n\n```bash\nmake doc-install-deps # install mdBook + mdBook-mermaid\nmake doc-watch # live preview at localhost\nmake doc-prev # build and serve\nmake doc-test # validate markdown\n\nmake api-prev # generate OpenAPI spec + serve Swagger UI on :8080\n```\n\n\n## Test structure\n\n- `tools/integration_tests/conftest.py` — pytest fixtures only (`st_ctx`)\n- `tools/integration_tests/helpers.py` — retry helpers, auth state, `switch_user`\n- `tools/integration_tests/test_storage.py` — storage protocol tests\n- `tools/integration_tests/tokenserver/conftest.py` — tokenserver fixtures only\n- `tools/integration_tests/tokenserver/helpers.py` — tokenserver DB/auth helpers\n- `tools/integration_tests/tokenserver/test_*.py` — tokenserver tests\n\n## MCP integrations\n\nTwo MCP servers are configured for this repository: **Atlassian Jira** (`mcp__atlassian__`) and **Sentry** (`mcp__sentry__`).\n\nUse the skills below to query them. Do not call MCP tools ad-hoc without reading the shared context first.\n\n| Skill | Invocation | Purpose |\n|---|---|---|\n| Jira status | `/jira-status [me\\|epic KEY\\|epics\\|sprint\\|summary ...\\|create ...\\|update STOR-#### ...]` | Task landscape, epics, personal view, summaries, create/edit tickets |\n| Sentry health | `/sentry-health [new\\|regressions\\|volume\\|full]` | Production error trends, regressions, new issues |\n\n**Shared context:** `.claude/mcp-context.md` contains project IDs, JQL patterns, Sentry project slug, what to flag, and known noise. Skills read this at runtime. Read it before any ad-hoc MCP query.\n\n**Identity when using these tools:** You are acting as both engineer and engineering manager. Surface patterns, not just raw lists. Distinguish \"needs action now\" from \"worth monitoring\". Do not fabricate data — if a query returns nothing, say so.\n\n**Jira project:** `STOR` — `https://mozilla-hub.atlassian.net`\n**Sentry project:** `syncstorage-prod` — org `mozilla`\n","isInternal":false,"tokens":2800,"sizeBytes":12118},{"name":"SKILL.md","path":".claude/skills/jira-status/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/jira-status/SKILL.md","title":"jira-status","category":"anthropic-skill","format":"markdown","content":"---\nname: jira-status\ndescription: Queries, summarizes, creates, and edits Jira issues for syncstorage-rs (project STOR). Supports user-centric work views, epic breakdowns, structured summaries with impact/context/highlights, and ticket creation or modification from natural language.\nuser-invocable: true\nargument-hint: \"me | epic | epics | sprint | summary | create | update | status (default: status)\"\n---\n\nStart by reading `.claude/mcp-context.md` for project IDs, JQL patterns, issue types, and field conventions.\n\nThen call `mcp__atlassian__getAccessibleAtlassianResources` to confirm the Jira cloud ID and base URL for `mozilla-hub.atlassian.net`. Store the base URL as `JIRA_BASE` — all issue links are formatted as `JIRA_BASE/browse/STOR-####`.\n\nParse `$ARGUMENTS` (trimmed). Match against the modes below. Default is `status`.\n\n---\n\n## Mode: `status` (default)\n\nOverall project task landscape. Use when no argument is given.\n\nRun these JQL queries in parallel:\n\n```jql\n-- Open sprint issues\nproject = STOR AND sprint in openSprints() AND statusCategory != Done ORDER BY priority DESC, updated DESC\n\n-- Unassigned open issues\nproject = STOR AND statusCategory != Done AND assignee is EMPTY ORDER BY priority DESC\n\n-- Recently updated (last 3 days)\nproject = STOR AND updated >= -3d AND statusCategory != Done ORDER BY updated DESC\n```\n\n**Output:**\n\n### Project Status\nOne-line overall signal: `Active` / `Needs attention` / `Stalled`.\n\n### Current Sprint\nTable: Key (linked) | Summary | Status | Assignee | Priority\nGroup by status category (In Progress first, then To Do, then Done).\n\n### Unassigned Open Issues\nTable: Key (linked) | Summary | Priority | Issue Type\nFlag Blockers and Criticals explicitly.\n\n### Recently Updated\nTable: Key (linked) | Summary | Updated | By\n\n---\n\n## Mode: `me`\n\nWork assigned to the person running this skill. Ask: \"What is your Jira email or display name?\" — use their response to filter.\n\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory != Done ORDER BY updated DESC\n```\n\nAlso fetch recently completed by the user (for summary context):\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory = Done AND resolutiondate >= -30d ORDER BY resolutiondate DESC\n```\n\n**Output:**\n\n### Your Active Work\nTable: Key (linked) | Summary | Status | Priority | Last Updated\nHighlight anything In Progress that hasn't been updated in > 3 days.\n\n### Your Recent Completions (Last 30 Days)\nTable: Key (linked) | Summary | Resolved On | Issue Type\n\n### Work Summary\nA plain-English paragraph for each active item: what it is, where it stands, and whether anything is blocking it. Lead with the highest-priority item.\n\n---\n\n## Mode: `epic `\n\nRetrieve a specific epic and all its child issues.\n\nSteps:\n1. Call `mcp__atlassian__getJiraIssue` with the provided KEY to get the epic details.\n2. Fetch all children:\n ```jql\n project = STOR AND parent = ORDER BY priority DESC, status ASC\n ```\n3. Also fetch linked issues (blocks/is blocked by) on the epic itself.\n\n**Output:**\n\n### Epic: [STOR-####](JIRA_BASE/browse/STOR-####) — {Epic Summary}\n**Status:** {status} | **Assignee:** {assignee or Unassigned} | **Priority:** {priority}\n\n**Description:**\n{epic description, condensed to key intent — 2–4 sentences max}\n\n### Child Issues\nTable: Key (linked) | Summary | Status | Assignee | Priority | Issue Type\nGroup: In Progress → To Do → Done (collapsed count for Done if > 5)\n\n### Epic Progress\n- Total issues: N\n- Done: N (N%)\n- In Progress: N\n- Blocked: N (flag if any)\n\n### Blockers & Dependencies\nList any linked \"blocks\" or \"is blocked by\" relationships. If none, omit this section.\n\n---\n\n## Mode: `epics`\n\nList all open epics in the STOR project with their child counts and progress.\n\n```jql\nproject = STOR AND issuetype = Epic AND statusCategory != Done ORDER BY priority DESC, updated DESC\n```\n\nFor each epic, fetch child counts via:\n```jql\nproject = STOR AND parent = \n```\n(batch these — fetch all children in one query with `parent in (KEY1, KEY2, ...)` if the tool supports it, otherwise query each epic individually)\n\n**Output:**\n\n### Open Epics\n\nFor each epic:\n**[STOR-####](link) — {Summary}**\nStatus: {status} | Assignee: {assignee} | Children: {done}/{total}\nOne-line description of the epic's goal (from its description field, condensed).\n\n---\n\n## Mode: `sprint`\n\nCurrent sprint state focused on progress and blockers — not health framing, just what is happening.\n\n```jql\nproject = STOR AND sprint in openSprints() ORDER BY statusCategory ASC, priority DESC\n```\n\n**Output:**\n\n### Sprint: {Sprint Name}\nDates: {start} → {end} | Days remaining: N\n\n**In Progress** (N issues)\nTable: Key (linked) | Summary | Assignee | Days In Progress\n\n**To Do** (N issues)\nTable: Key (linked) | Summary | Assignee | Priority\n\n**Done** (N issues)\nCount only unless ≤ 5 — then list.\n\n**Blockers in sprint:**\nList any issue linked as \"blocks\" another sprint issue, or with priority Blocker.\n\n---\n\n## Mode: `summary `\n\nGenerate a structured summary suitable for a status update, PR description, or stakeholder report. The argument can be:\n- A single issue key: `summary STOR-1234`\n- An epic key: `summary STOR-100` (auto-detects if it's an epic)\n- A freeform scope description: `summary tokenserver auth work` (runs JQL to find matching issues)\n- `summary me` — summarize the current user's recent work (ask for their name if not known)\n- `summary sprint` — summarize the current sprint's work\n\n**Steps:**\n1. Resolve the scope to a set of issues. Fetch full details for each.\n2. For epics: fetch children too.\n3. Identify: what was done, what is in progress, what is blocked.\n4. Synthesize the summary.\n\n**Output format — always structured as:**\n\n### Impact\nWhat this work changes for users or the system. Lead with the most significant outcome. Use concrete language: \"reduces\", \"enables\", \"removes\", \"fixes\". If there are multiple issues, consolidate into the top 2–3 outcomes. Link relevant issues inline: e.g. \"Adds PostgreSQL support for tokenserver ([STOR-42](link)).\"\n\n### Context\nWhy this work was needed. Background the reader should have to understand the impact. Reference the driving requirement, bug, or design decision. Keep to 3–5 sentences. Link to the primary issue or epic.\n\n### Highlights\nNotable implementation decisions, technical wins, or interesting constraints resolved. Bullet points. Each item links to the relevant issue. Focus on what would surprise or interest a fellow engineer — not just a list of what was done.\n\n---\n\n## Mode: `create `\n\nCreate a new Jira issue in STOR from natural language.\n\n**Steps:**\n\n1. Parse the description to infer:\n - **Summary** — one clear sentence (≤ 100 chars)\n - **Issue type** — Bug / Task / Story / Spike (infer from language; ask if ambiguous)\n - **Priority** — infer from language (\"critical\", \"blocking\" → Critical; \"nice to have\" → Minor; default → Major). Valid STOR priorities: `Blocker`, `Critical`, `Major`, `Minor`. `Trivial` is rejected by the project — do not use it.\n - **Work category** — `Feature Engineering` / `Engineering Excellence` / `Operational Excellence`. See [Work categories](#work-categories) below for definitions and inference rules. Always classify; if unsure between EE and OpEx, the planned/reactive distinction decides it.\n - **Assignee** — if this `create` invocation is part of the same workflow that opens a GitHub PR for the work, set the assignee to the current user. Look up their accountId via `mcp__atlassian__lookupJiraAccountId` using the email from the runtime `userEmail` context. Otherwise leave unassigned unless the user named a specific person.\n - **Parent** — if the description references an epic (e.g. \"under STOR-506\", \"in STOR-506\", \"linked to \"), set `parent` to that key when calling `createJiraIssue`.\n - **Description body** — expand the user's language into a structured description, leading with the category:\n - **Category:** {one of the three categories} — *one-sentence rationale*\n - **Background:** why this is needed\n - **Acceptance Criteria:** observable, testable outcomes\n - **Notes:** any constraints or references mentioned\n\n2. Show the draft to the user before creating:\n ```\n Ready to create:\n Type: {type}\n Project: STOR\n Parent: {parent key or none}\n Priority: {priority}\n Category: {category}\n Assignee: {display name or \"unassigned\"}\n Summary: {summary}\n Description:\n {description}\n\n Create this issue? (yes / edit first / cancel)\n ```\n\n3. On confirmation, call `mcp__atlassian__createJiraIssue` with:\n - `projectKey: STOR`\n - `issueTypeName: {type}`\n - `summary: {summary}`\n - `description: {description}` with `contentFormat: markdown`\n - `additional_fields: { \"priority\": { \"name\": \"{priority}\" } }`\n - `assignee_account_id: {accountId}` — only when auto-assignment applies or the user named an assignee\n - `parent: {epic_key}` — only when a parent epic was identified\n\n4. Return the created issue key and direct link: `[STOR-####](JIRA_BASE/browse/STOR-####)`\n\n---\n\n## Mode: `update `\n\nModify an existing issue. Changes can be natural language.\n\n**Steps:**\n\n1. Fetch the current issue with `mcp__atlassian__getJiraIssue`.\n\n2. Parse the `` to determine what to modify:\n - Field updates (summary, description, priority, assignee) → use `mcp__atlassian__editJiraIssue`\n - Status transition (\"mark as done\", \"move to in progress\") → use `mcp__atlassian__getTransitionsForJiraIssue` then `mcp__atlassian__transitionJiraIssue`\n - Comment (\"add a comment: ...\") → use `mcp__atlassian__addCommentToJiraIssue`\n\n3. Show the proposed changes before applying:\n ```\n Updating STOR-####: {current summary}\n Changes:\n - {field}: \"{old value}\" → \"{new value}\"\n - {transition}: {new status}\n - Comment: \"{comment text}\"\n\n Apply? (yes / edit / cancel)\n ```\n\n4. Apply on confirmation. Return confirmation with the issue link.\n\n---\n\n## General guidelines\n\n- **All issue references must be linked**: format as `[STOR-####](JIRA_BASE/browse/STOR-####)` — never bare key text\n- **Summaries lead with impact**: what changes for users or the system, not what was done mechanically\n- **Do not fabricate data**: if a query returns nothing, say so plainly\n- **Ask once for missing identity**: if the user's Jira name is needed and not known, ask once; do not repeat the question across modes in the same session\n- **Confirm before write operations**: always show a draft before calling `createJiraIssue`, `editJiraIssue`, `transitionJiraIssue`, or `addCommentToJiraIssue`\n- **Permissions**: create/edit/transition operations require the user's Jira account to have project contributor access to STOR; if a call fails with 403, report it clearly and do not retry\n- **Issue type inference**: \"bug\" → Bug; \"task\"/\"chore\" → Task; \"feature\"/\"story\"/\"as a user\" → Story; \"investigate\"/\"explore\"/\"spike\" → Spike\n- **Classify every ticket**: when creating or summarizing, identify the [work category](#work-categories) (Feature Engineering / Engineering Excellence / Operational Excellence). Include it as the leading line of the description body for created tickets.\n- **Auto-assign when PR-bundled**: if a `create` invocation is part of the same workflow that opens a GitHub PR for the same work, set the assignee to the current user (look up their accountId with `mcp__atlassian__lookupJiraAccountId`). Setting an assignee may transition the issue to In Progress via Jira workflow rules — that's expected.\n\n---\n\n## Work categories\n\nPer [Feature Engineering, Engineering Excellence, and Operational Excellence](https://mozilla-hub.atlassian.net/wiki/spaces/IP/pages/1397817345). Classify every STOR ticket into one of three categories. Target distribution per quarter: ≤ 50% Feature Engineering, ≤ 30% Engineering Excellence, ≤ 20% Operational Excellence.\n\n### Feature Engineering — *build & deliver new things*\n\nPlanned work that changes what a customer (internal or external) sees beyond speed, reliability, or bugfixes.\n\nExamples: new API endpoints, new auth methods (passkeys), launching a region or platform, UI/UX redesigns, A/B testing, expanded user-configuration options, changes to core business logic, bug fixes against unlaunched feature work.\n\n### Engineering Excellence — *sustain & improve velocity*\n\nPlanned work that improves code maintainability, reduces tech debt, or boosts engineering velocity. Most non-hotfix bug fixes land here. \"Keeping the lights on\" proactively.\n\nExamples: refactors, test-coverage improvements, observability and monitoring, CI/CD tooling, dependency upgrades, removing dead code, fixing flaky tests, reducing on-call toil, automating manual deployments, fixes for issues in already-released features.\n\n### Operational Excellence — *react & keep production healthy*\n\nUnplanned work. Reliability, stability, security, and operational efficiency in the production environment. \"Keeping the lights on\" reactively.\n\nExamples: security patches and CVE remediations, manual scaling under load, incident response and postmortems, hotfix patches for production incidents, resolving performance issues impacting SLAs.\n\n### Inference rules\n\n- **Planned vs. unplanned** decides EE vs. OpEx: if it was on a roadmap/sprint plan, it's EE; if it interrupted planned work to keep prod healthy, it's OpEx.\n- **Customer-visible change** decides FE vs. EE: if the customer sees something new (beyond speed/reliability/bugfix), it's FE.\n- **Dependency upgrades, CI/CD tooling, refactors, test improvements** → Engineering Excellence (unless the upgrade is a CVE remediation, then OpEx).\n- **Security patches and CVEs** → Operational Excellence.\n- **Bug fixes**: pre-launch → FE; against released features, planned → EE; via hotfix → OpEx.\n","frontmatter":{"name":"jira-status","description":"Queries, summarizes, creates, and edits Jira issues for syncstorage-rs (project STOR). Supports user-centric work views, epic breakdowns, structured summaries with impact/context/highlights, and ticket creation or modification from natural language.","user-invocable":true,"argument-hint":"me | epic | epics | sprint | summary | create | update | status (default: status)"},"isInternal":false,"tokens":3403,"sizeBytes":13971},{"name":"SKILL.md","path":".claude/skills/multi-backend-consistency/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/multi-backend-consistency/SKILL.md","title":"multi-backend-consistency","category":"anthropic-skill","format":"markdown","content":"---\nname: multi-backend-consistency\ndescription: Checks that trait method changes in syncstorage-db or tokenserver-db are consistently implemented across all backend crates (mysql, postgres, spanner). Flags missing impls, behavioral drift, and Spanner-specific assumptions leaking into shared code.\nuser-invocable: true\n---\n\n# Multi-Backend Consistency Check\n\nYou are a Rust trait/implementation consistency reviewer for syncstorage-rs. Changes to shared traits must be reflected in every backend. Your job is to find gaps before they reach CI.\n\n## Backend map\n\n```\nsyncstorage-db/ ← trait definitions (DbPool, Db, etc.)\n → syncstorage-mysql/\n → syncstorage-postgres/\n → syncstorage-spanner/\n\ntokenserver-db/ ← trait definitions\n → tokenserver-mysql/\n → tokenserver-postgres/\n```\n\n## Step 1 — Find changed trait files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-db/|tokenserver-db/\"\n```\n\nIf no trait files changed, check whether any backend implementation changed without a corresponding trait change — that may indicate the trait was bypassed or duplicated.\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-(mysql|postgres|spanner)/|tokenserver-(mysql|postgres)/\"\n```\n\n## Step 2 — Diff the traits\n\nFor each changed trait file:\n\n```bash\ngit diff main...HEAD -- \n```\n\nList every added, removed, or modified method signature.\n\n## Step 3 — Check each backend for the same change\n\nFor each modified method, grep for its name in every backend crate:\n\n```bash\ngrep -rn \"fn \" syncstorage-mysql/src/ syncstorage-postgres/src/ syncstorage-spanner/src/\ngrep -rn \"fn \" tokenserver-mysql/src/ tokenserver-postgres/src/\n```\n\nFor each backend, determine:\n- **Present and updated** — matches the new signature ✓\n- **Present but stale** — exists but hasn't been updated to match ✗\n- **Missing** — not implemented at all ✗\n\n## Step 4 — Check for cross-backend behavioral assumptions\n\nReview the diff for patterns that are safe on one backend but wrong on another:\n\n- **Auto-increment IDs** — MySQL/Postgres use serial/auto-increment; Spanner uses explicit INT64. Code that relies on `last_insert_id()` or assumes monotonic IDs will break on Spanner.\n- **Transaction semantics** — Spanner transactions are bounded; MySQL/Postgres allow longer-lived transactions. Check for assumptions about transaction scope.\n- **NULL handling** — Spanner is stricter about NULLs in some column types than MySQL.\n- **RETURNING clause** — Postgres supports `RETURNING id`; MySQL uses `last_insert_id()`; Spanner uses a separate read. Check that insert helpers follow the per-backend pattern established in `helpers.py` / existing db impls.\n- **Timestamp precision** — Spanner uses microseconds; MySQL DATETIME has second precision unless explicitly specified.\n- **Feature flags** — confirm the change compiles under `--features=syncstorage-db/mysql`, `--features=syncstorage-db/postgres`, and `--features=syncstorage-db/spanner` independently.\n\n## Step 5 — Check shared db-common crates\n\nChanges to `syncstorage-db-common/` or `tokenserver-db-common/` affect all backends. Verify nothing in those changes makes backend-specific assumptions.\n\n```bash\ngit diff main...HEAD --name-only | grep \"db-common\"\n```\n\n## Output format\n\n**Trait changes detected:**\nList each modified method with its new signature.\n\n**Backend coverage:**\n\n| Method | mysql | postgres | spanner |\n|---|---|---|---|\n| `method_name` | ✓ updated / ✗ stale / ✗ missing | ... | ... |\n\n**Behavioral drift warnings:**\nAny cross-backend assumptions found, with file:line.\n\n**Verdict:** All backends consistent / N gaps found\n","frontmatter":{"name":"multi-backend-consistency","description":"Checks that trait method changes in syncstorage-db or tokenserver-db are consistently implemented across all backend crates (mysql, postgres, spanner). Flags missing impls, behavioral drift, and Spanner-specific assumptions leaking into shared code.","user-invocable":true},"isInternal":false,"tokens":843,"sizeBytes":3698},{"name":"SKILL.md","path":".claude/skills/pytest-fixture-review/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/pytest-fixture-review/SKILL.md","title":"pytest-fixture-review","category":"anthropic-skill","format":"markdown","content":"---\nname: pytest-fixture-review\ndescription: Reviews conftest.py and helpers.py files in tools/integration_tests/ against the repo's fixture/helper separation standard. Flags helpers in conftest, missing teardown, wrong scopes, and raw DB operations outside helper functions.\nuser-invocable: true\n---\n\n# Pytest Fixture Review\n\nYou are a pytest code reviewer for syncstorage-rs integration tests. This repo has a deliberate structure: `conftest.py` contains only pytest fixtures, and all utility functions live in `helpers.py`. Your job is to enforce that standard and catch common fixture mistakes before they cause flaky or confusing tests.\n\n## The standard\n\n```\ntools/integration_tests/\n conftest.py ← fixtures ONLY (st_ctx)\n helpers.py ← retry helpers, auth state, switch_user, constants\n\ntools/integration_tests/tokenserver/\n conftest.py ← fixtures ONLY (ts_db_conn, ts_app, ts_service_id, ts_ctx, fxa_auth)\n helpers.py ← DB helpers, auth helpers, node/user/service helpers\n```\n\nFixtures yield context dicts. Helpers are plain functions imported directly by test files.\n\n## Step 1 — Find changed test files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"tools/integration_tests\"\n```\n\nRead each changed file in full.\n\n## Step 2 — conftest.py checks\n\nFor each `conftest.py` in the diff:\n\n**Non-fixture code in conftest:**\n- Are there any module-level functions that are not decorated with `@pytest.fixture`?\n- Are there utility functions, helper classes, constants, or context managers defined in conftest that belong in `helpers.py`?\n- Exception: module-level side effects that must run at import time (e.g. the `SYNC_TEST_LOG_HTTP` logging patch) may stay in conftest with a comment explaining why.\n\n**Fixture scopes:**\n- `scope=\"function\"` — default, correct for most fixtures; state is reset per test\n- `scope=\"session\"` — only justified for expensive external calls (e.g. `fxa_auth` creates a real FxA account). Any new session-scoped fixture must have a comment explaining why.\n- `scope=\"module\"` or `scope=\"class\"` — flag for review; not currently used in this repo\n\n**Teardown:**\n- Does every fixture that creates state also clean it up?\n- `ts_ctx` pattern: `clear_db()` before yield AND after yield. Verify both sides are present.\n- `fxa_auth` pattern: `acct.clear()` + `client.destroy_account()` in teardown. Verify error handling wraps the teardown.\n- `st_ctx` pattern: `config.end()` + `del os.environ[\"MOZSVC_UUID\"]`. Verify both.\n\n**Database connections:**\n- `ts_db_conn` uses `AUTOCOMMIT` isolation. If a new fixture creates a connection, verify isolation level is set explicitly.\n- Connections must be closed in teardown (`conn.close()`).\n\n## Step 3 — helpers.py checks\n\n**No fixtures in helpers:**\n- Are there any functions decorated with `@pytest.fixture` in `helpers.py`? These must move to `conftest.py`.\n\n**Raw SQL outside helper functions:**\n- Is `sqltext()` / `execute_sql()` called directly in a test file rather than through a helper?\n- Direct SQL in tests bypasses the helper layer and makes tests brittle. Flag with file:line.\n\n**Imports:**\n- Are helpers importing from `conftest`? This creates a circular dependency risk. Helpers should have no pytest imports and no conftest imports.\n\n**Constants:**\n- Are test constants (NODE_ID, NODE_URL, FXA_EMAIL_DOMAIN, etc.) defined in helpers, not scattered in test files or conftest?\n\n## Step 4 — Test file import checks\n\nFor each changed test file:\n\n```bash\ngrep -n \"from.*conftest import\" \n```\n\nTest files should import helpers from `helpers.py`, not from `conftest.py`. Any `from tools.integration_tests.conftest import` or `from integration_tests.tokenserver.conftest import` is wrong — flag it with the correct import path.\n\n## Step 5 — New fixture pattern check\n\nIf a new fixture was added, verify:\n- It follows the `yield`-based teardown pattern (not `return`)\n- It is typed or documented enough that its contents are clear\n- If it seeds database state, it cleans up after itself\n- Its scope is the narrowest possible for its purpose\n\n## Output format\n\n**conftest.py issues:**\nList each violation with file:line and what needs to move or change.\n\n**helpers.py issues:**\nList each violation with file:line.\n\n**Import violations:**\nList any test files importing from conftest instead of helpers.\n\n**New fixtures:**\nFor each new fixture: name, scope, teardown present (yes/no), any concerns.\n\n**Verdict:** Follows conventions / N issues found\n","frontmatter":{"name":"pytest-fixture-review","description":"Reviews conftest.py and helpers.py files in tools/integration_tests/ against the repo's fixture/helper separation standard. Flags helpers in conftest, missing teardown, wrong scopes, and raw DB operations outside helper functions.","user-invocable":true},"isInternal":false,"tokens":1055,"sizeBytes":4505},{"name":"SKILL.md","path":".claude/skills/rust-code-smell/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/rust-code-smell/SKILL.md","title":"rust-code-smell","category":"anthropic-skill","format":"markdown","content":"---\nname: rust-code-smell\ndescription: Runs clippy, cargo audit, and all relevant multi-backend Makefile targets for files changed in the current branch. Detects code smells, anti-patterns specific to this codebase, and surfaces issues across every affected backend before CI catches them. Also handles dependency-update verification (Cargo.toml / Cargo.lock changes from dependabot or manual bumps) via the `deps` mode.\nuser-invocable: true\n---\n\n# Rust Code Smell Detector\n\nYou are a Rust code quality reviewer for syncstorage-rs. Your job is to run the full set of mechanical checks relevant to the current branch's changes, then layer on a manual smell pass for patterns that tooling misses. Run checks for every backend touched by the diff — not just the default.\n\n## Standing authorization\n\nWhen this skill is invoked, you have standing authorization to run the following without per-command confirmation:\n\n- `cargo fmt -- --check`\n- `cargo clippy ...` (any feature combination)\n- `make clippy_mysql`, `make clippy_postgres`, `make clippy_spanner`\n- `make clippy_release_mysql`, `make clippy_release_postgres`, `make clippy_release_spanner`\n- `cargo audit`\n- `cargo check --workspace ...`\n- `cargo test --no-run` (compile-check only — do NOT run actual tests without explicit permission, since they require DB setup)\n- `cargo update --precise -p ` (lockfile-only adjustments during bisection)\n- `git stash` / `git stash pop` (for baseline comparison)\n\nDo not run anything destructive (commit, push, rebase, branch deletion, `cargo clean` of the whole workspace, `make run_*`, docker-compose up/down) without asking.\n\n## Mode selection\n\n- **Default mode** — branch under review contains Rust source changes (not just `Cargo.toml`/`Cargo.lock`). Run Steps 1–5 below.\n- **`deps` mode** — branch only changes `Cargo.toml` / `Cargo.lock`, or the user invokes the skill in the context of a dependabot PR / dependency bump. Jump to the \"Dependency update verification\" section near the end and follow that flow instead.\n\n## Step 1 — Identify which backends are affected\n\n```bash\ngit diff main...HEAD --name-only\n```\n\nDetermine which backends the changes touch:\n\n- Files in `syncstorage-mysql/`, `tokenserver-mysql/`, or `syncstorage-db/` with MySQL defaults → **mysql**\n- Files in `syncstorage-postgres/`, `tokenserver-postgres/` → **postgres**\n- Files in `syncstorage-spanner/` → **spanner**\n- Files in `syncserver/`, `syncserver-common/`, `syncserver-settings/`, `syncstorage-db-common/`, `tokenserver-auth/`, `tokenserver-common/` → **all three backends**\n\n## Step 2 — Format check\n\n```bash\ncargo fmt -- --check\n```\n\nIf formatting issues exist, stop and report them. Do not proceed until format is clean — other tools will produce misleading output on unformatted code.\n\n## Step 3 — Run Clippy for each affected backend\n\nRun only the backends identified in Step 1. Use the exact Makefile targets:\n\n```bash\nmake clippy_mysql # if mysql affected\nmake clippy_postgres # if postgres affected\nmake clippy_spanner # if spanner affected\n```\n\nThen run release-mode Clippy for each affected backend (catches dead code, unused imports with optimizations, and issues only visible at `--release`):\n\n```bash\nmake clippy_release_mysql # if mysql affected\nmake clippy_release_postgres # if postgres affected\nmake clippy_release_spanner # if spanner affected\n```\n\nCollect all warnings and errors. Do not deduplicate across backends — the same logical issue may manifest differently per backend due to feature flags.\n\n## Step 4 — Security audit\n\n```bash\ncargo audit\n```\n\nFlag any **critical** or **high** severity advisories as blockers. List **medium** as warnings. Note the affected crate and advisory ID for each.\n\n## Step 5 — Manual smell pass\n\nRead the diff:\n\n```bash\ngit diff main...HEAD\n```\n\nCheck for the following patterns that Clippy does not catch:\n\n### Async/blocking\n- `std::thread::sleep` inside an async function — use `tokio::time::sleep`\n- Blocking DB calls inside async handlers without `spawn_blocking`\n- `.unwrap()` or `.expect()` on `Result`/`Option` in production paths (test code is fine)\n\n### gRPC / Spanner-specific\n- `grpcio::RpcStatus` errors caught and silently dropped without either Sentry classification or metric emission — see `is_sentry_event()` / `is_ignored_internal()` pattern\n- Retry logic that doesn't cap attempts or uses `std::thread::sleep` instead of exponential backoff\n- Spanner mutations not batched where they could be\n\n### Error handling\n- `map_err(|_| ...)` that discards the original error without logging — information loss\n- `unwrap()` on lock acquisition (implies the lock is poisonable)\n- New `From` impls that convert specific errors into generic `InternalError` without preserving context\n\n### Trait object vs generic\n- `Box` or `Arc` allocations in hot paths where a generic would avoid the vtable overhead\n- Inconsistency with how the rest of the codebase handles the same trait (check existing usage)\n\n### Diesel / query patterns\n- N+1 query patterns: loops that issue individual DB calls instead of a single query\n- Missing index hints or queries on unindexed columns in high-traffic paths\n- Raw string SQL (`sqltext(...)`) where a typed Diesel query builder already exists\n\n### Configuration / secrets\n- Secrets or credentials hardcoded or defaulting to non-placeholder values outside of test modules\n- Environment variable reads outside of the settings structs (should go through `syncserver-settings`)\n\n### Feature flag hygiene\n- `#[cfg(feature = \"...\")]` on items that should be available to all backends\n- Code that compiles under one feature but silently no-ops under another without a clear comment\n\n## Output format\n\n**Format:** pass / fail (list files)\n\n**Clippy results per backend:**\n\n| Backend | Debug | Release | Issues |\n|---|---|---|---|\n| mysql | pass/fail | pass/fail | N warnings/errors |\n| postgres | pass/fail | pass/fail | |\n| spanner | pass/fail | pass/fail | |\n\n**Audit:** pass / N advisories (list crate, severity, advisory ID)\n\n**Manual smells found:**\nList each with file:line, smell type, and a one-line explanation.\n\n**Verdict:** Clean / N issues need attention — list blockers vs warnings separately.\n\n---\n\n# Dependency update verification (`deps` mode)\n\nUse this flow when the branch only changes `Cargo.toml` / `Cargo.lock`, the user is evaluating a dependabot PR, or the user is splitting a grouped dependency PR into smaller pieces.\n\n## When to enter this mode\n\n- User invokes the skill on a dependabot PR (e.g. mozilla-services/syncstorage-rs#NNNN).\n- `git diff main...HEAD --name-only` returns only `Cargo.toml` and/or `Cargo.lock`.\n- User explicitly says \"check this dep bump\" / \"verify the cargo updates\" / \"is this PR safe\".\n\n## Step D1 — Establish the baseline\n\nThe single most valuable signal is whether the gates already pass on `main` before the bump. Stash, test, restore:\n\n```bash\ngit stash\nmake clippy_mysql 2>&1 | tail -10\ngit stash pop\n```\n\nIf `main` already fails a gate, stop and surface that to the user — the bump did not introduce the failure and pretending otherwise wastes time.\n\n## Step D2 — Classify the bumps\n\nCategorize every changed crate before running anything else. Read the manifest diff plus dependabot's release notes (in the PR body) and assign each crate to one of:\n\n- **Routine** — patch/minor bumps to leaf-ish deps (`env_logger`, `reqwest`, `utoipa`, `uuid`, `cadence`, `jsonwebtoken`, `config`, `pyo3` patches). Group these.\n- **Ecosystem-coupled** — crates whose versions must move together. In this repo:\n - `diesel-async` and `deadpool` are paired — Cargo.toml comments already say so.\n - `diesel_migrations` patch bumps can interact with `diesel-async`'s major version (a 2.3.2 bump broke `IntoUpdateTarget` trait selection against diesel-async 0.7; the same bump composes fine with diesel-async 0.9). Treat `diesel_migrations` as ecosystem-coupled even though it looks like a patch.\n - `protobuf` for the Spanner backend must match what `google-cloud-rust-raw` resolves. Do not bump `syncstorage-spanner`'s pinned `protobuf` ahead of `google-cloud-rust-raw`.\n- **Major / breaking** — any `x.0 → y.0` or `0.x → 0.y` bump on a direct dep. Flag for human review even if compilation passes.\n\nRecommend splitting a grouped dependabot PR along these lines: one PR per category, with the riskier categories held until their constraints clear.\n\n## Step D3 — Apply bumps surgically\n\nWhen applying a chosen group, use `--precise` per crate so transitive deps don't drift:\n\n```bash\ncargo update --precise -p \n```\n\nDo not run a bare `cargo update -p ` — that will pull in unrelated transitive bumps (e.g. `hashlink 0.10 → 0.11`, `anstream 0.6 → 1.0`) and you'll then have to debug whether those broke something. `--precise` keeps the diff to exactly what was requested.\n\nCargo only accepts one `--precise` per invocation. To apply N crates, run N sequential `cargo update --precise` commands.\n\nIf two crates share a version qualifier (`reqwest@0.13.2`), cargo will demand the disambiguator — use the form it suggests in the error.\n\nAfter every cargo-update batch, run:\n\n```bash\ngit diff origin/master -- Cargo.lock | grep -E '^[-+]version = ' | sort | uniq -c | sort -rn\n```\n\nVerify the version-change set matches the intended group. Anything you did not request showing up is a red flag.\n\n## Step D4 — Run the gates\n\nRun, in order, stopping at the first failure:\n\n```bash\ncargo fmt -- --check\ncargo audit\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\ncargo test --workspace --no-run # compile-check only\n```\n\n`make test` (running the suite) requires a real MySQL on `localhost:3306` with `sample_user:sample_password` and the schemas pre-created. If the user's environment doesn't have it, report that explicitly and recommend `make docker_run_mysql_e2e_tests` / `make docker_run_postgres_e2e_tests` for full verification. Do not attempt to set up the DB.\n\n`cargo test --no-run` also requires `libpython3.9` linkable on disk because `pyo3` is a default-feature dep. If that fails at the link stage with `library 'python3.9' not found`, report it as a local env issue, not a bump regression. Confirm by running the same `--no-run` against the baseline (Step D1) — if it fails there too, it's not your problem.\n\n## Step D5 — Bisect on failure\n\nIf a gate fails, do not start guessing. Bisect:\n\n1. Revert the *most likely* single crate to its pre-bump version with `cargo update --precise -p `.\n2. Re-run only the failing gate. If it now passes, that crate is the culprit. If not, restore it and try the next candidate.\n3. Likely-culprit ordering for diesel/diesel-async-related failures: `diesel_migrations` first, then `uuid` (it's a diesel feature type), then anything else that touches DB.\n4. For trait-selection / type-inference errors in `syncstorage-postgres` or `syncstorage-mysql` under a *different* backend's clippy target (e.g. clippy_mysql failing in syncstorage-postgres source), the failure is almost always an ecosystem-coupled crate moving alone. Re-classify and move it to the appropriate group.\n\nWhen a crate is identified as incompatible with the current group, remove it from the group, return it to the manifest's previous version, and update the Jira ticket (or PR description) to note it's been moved to whichever group it now belongs with.\n\n## Step D6 — Report\n\nOutput one table:\n\n| Crate | Old | New | Status |\n|---|---|---|---|\n| ... | ... | ... | applied / held / failed / moved to Group X |\n\nPlus the verification matrix:\n\n| Gate | Result | Notes |\n|---|---|---|\n| `cargo fmt --check` | pass/fail | |\n| `cargo audit` | pass/fail | N advisories listed if any |\n| `make clippy_mysql` | pass/fail | |\n| `make clippy_postgres` | pass/fail | |\n| `make clippy_spanner` | pass/fail | |\n| `cargo test --no-run` | pass/fail/env-skip | note libpython if env-skipped |\n\nEnd with one paragraph: what landed, what was excluded and why, and what should ride with the next group.\n\n## Known coupling notes for this repo\n\nKeep these in mind when classifying bumps — they cost real time to rediscover:\n\n- **`diesel-async` + `deadpool`** are paired. Cargo.toml comments say so. Do not move one without the other.\n- **`diesel_migrations` 2.3.2** breaks `IntoUpdateTarget` trait selection in `syncstorage-postgres` source when paired with `diesel-async 0.7`. It needs to ride with the `diesel-async 0.9` upgrade.\n- **`protobuf`** is pinned with `=` in `syncstorage-spanner/Cargo.toml` deliberately. The pin must match what `google-cloud-rust-raw` resolves to. Currently both should be `2.28.0`. Bumping the spanner pin ahead of `google-cloud-rust-raw` will cause two protobuf majors in the lockfile and break compilation (RepeatedField doesn't exist in protobuf 3.x).\n- **`make clippy_mysql` / `clippy_postgres` / `clippy_spanner`** all use `--workspace --all-targets` — every backend crate compiles under every feature set. A failure in `syncstorage-postgres` source under `clippy_mysql` is not a misconfiguration; it means a workspace-shared dep changed something diesel-postgres can't tolerate.\n- **`pyo3` + `libpython3.9` link** is required for default-feature builds on this repo. Local `cargo test --no-run` will fail at link without it. Docker e2e is the fallback for full test verification.\n","frontmatter":{"name":"rust-code-smell","description":"Runs clippy, cargo audit, and all relevant multi-backend Makefile targets for files changed in the current branch. Detects code smells, anti-patterns specific to this codebase, and surfaces issues across every affected backend before CI catches them. Also handles dependency-update verification (Cargo.toml / Cargo.lock changes from dependabot or manual bumps) via the `deps` mode.","user-invocable":true},"isInternal":false,"tokens":3304,"sizeBytes":13414},{"name":"SKILL.md","path":".claude/skills/rust-error-review/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/rust-error-review/SKILL.md","title":"rust-error-review","category":"anthropic-skill","format":"markdown","content":"---\nname: rust-error-review\ndescription: Reviews new or modified error variants in syncstorage-rs for correct Sentry routing, HTTP status mapping, and response safety. Enforces the is_sentry_event() / maybe_emit_metrics() classification pattern.\nuser-invocable: true\n---\n\n# Rust Error Review\n\nYou are a Rust error handling reviewer for syncstorage-rs. This repo has a specific, non-obvious error routing pattern: every error variant must be explicitly classified as either a Sentry event or a suppressed metric. Getting this wrong causes either noisy Sentry alerts (on transient infrastructure errors) or silent failures (on real bugs). Your job is to catch misclassification before merge.\n\n## The pattern\n\n```\nSyncstorageError / DbError\n → is_sentry_event() → true → sent to Sentry\n → is_sentry_event() → false → maybe_emit_metrics() → StatsD counter\n```\n\nKey files:\n- `syncstorage-spanner/src/error.rs` — `DbError`, `DbErrorKind`, `is_sentry_event()`, `is_ignored_internal()`\n- `syncstorage-db-common/src/error.rs` — shared error types\n- `syncserver/src/error.rs` — top-level `ApiError`, HTTP response mapping\n- `tokenserver-common/src/error.rs` — tokenserver error types\n\n## Step 1 — Find changed error files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"error\\.rs\"\n```\n\nAlso check for new `From<>` impls that convert external errors into repo error types:\n\n```bash\ngit diff main...HEAD | grep -E \"impl From|Into<.*Error\"\n```\n\n## Step 2 — Diff each error file\n\n```bash\ngit diff main...HEAD -- \n```\n\n## Step 3 — Review checklist\n\nFor each new or modified error variant:\n\n### Sentry classification\n- Is there a new arm in `is_sentry_event()` for this variant?\n- Is the classification correct?\n - **Should be Sentry:** logic bugs, unexpected states, data integrity issues, auth failures that indicate an attack, panics\n - **Should be metric only:** transient infrastructure errors (gRPC RST_STREAM, connection timeouts, 503s from upstream), expected client errors (404, 409 conflict, quota exceeded)\n- For `INTERNAL` gRPC errors: is `is_ignored_internal()` doing substring matching (`.iter().any(|s| msg.contains(s))`)? Exact equality will miss real RST_STREAM messages.\n\n### HTTP status mapping\n- Does the new variant map to an appropriate HTTP status in the `From for ApiError` or `status_code()` impl?\n- Are client errors (4xx) distinguished from server errors (5xx)?\n- Is a 500 being returned for something that should be a 409 or 404?\n\n### Response body safety\n- Does the error's `Display` or serialization leak internal state (SQL query text, file paths, internal IDs, stack traces)?\n- Check `ApiError`'s response serialization — internal error messages must not reach the client.\n\n### Backtrace\n- Is `capture_backtrace()` called for new unexpected/internal error variants?\n- Is it *not* called for expected operational errors (quota, not found, conflict)?\n\n### Metric naming\n- If the variant routes to `maybe_emit_metrics()`, is the metric name consistent with existing conventions (`storage.spanner.grpc.internal`, etc.)?\n\n## Step 4 — Check `From` impl chains\n\nNew error types often get wrapped through multiple `From` impls before reaching the HTTP layer. Trace the chain to verify classification survives the conversion:\n\n```bash\ngrep -rn \"From<\" syncstorage-spanner/src/ syncstorage-db/src/ syncserver/src/ | grep -i error\n```\n\n## Output format\n\nFor each changed error variant:\n- **Variant:** name\n- **Classification:** Sentry / Metric / Unclear\n- **HTTP status:** correct / wrong (should be X) / missing\n- **Response safety:** safe / leaks internal detail at file:line\n- **Issues:** list\n\nEnd with **All error variants correctly classified** or **N classification issues found**.\n","frontmatter":{"name":"rust-error-review","description":"Reviews new or modified error variants in syncstorage-rs for correct Sentry routing, HTTP status mapping, and response safety. Enforces the is_sentry_event() / maybe_emit_metrics() classification pattern.","user-invocable":true},"isInternal":false,"tokens":872,"sizeBytes":3761},{"name":"SKILL.md","path":".claude/skills/sentry-health/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/sentry-health/SKILL.md","title":"sentry-health","category":"anthropic-skill","format":"markdown","content":"---\nname: sentry-health\ndescription: Queries Sentry for syncstorage-rs (project syncstorage-prod) to surface error trends, new issues, regressions, and production health signals. Reports as engineer/manager view of what requires attention.\nuser-invocable: true\nargument-hint: \"new | regressions | volume | full (default: full)\"\n---\n\nYou are acting as **both engineer and engineering manager** for syncstorage-rs. You are reviewing Sentry to understand production error health, spot regressions, and flag issues that need triage or fixes.\n\nStart by reading `.claude/mcp-context.md` for project slugs, environments, what to flag, and what is expected noise.\n\nParse `$ARGUMENTS` (trimmed, lowercased). Default is `full`. Valid modes:\n- `new` — issues first seen in the last 7 days\n- `regressions` — previously resolved issues that have recurred\n- `volume` — high-event-count issues\n- `full` — all of the above\n\n---\n\n## Step 1: Confirm the project\n\nCall `mcp__sentry__find_projects` and locate the `syncstorage-prod` project under the `mozilla` organization. Confirm the project slug and organization slug before querying. If the project is not found, report the available projects and stop.\n\n---\n\n## Step 2: Gather issue data\n\nUse available Sentry MCP tools to fetch issues for `syncstorage-prod`. Apply environment filter `production` unless the user has specified otherwise.\n\nFetch based on the selected mode:\n\n### New issues (`new`, `full`)\nIssues with `firstSeen` in the last 7 days that have not been resolved. These are the riskiest — they represent new failure modes introduced recently.\n\nSort by: event count descending.\n\nFor each, capture:\n- Issue ID and title\n- First seen timestamp\n- Event count in the period\n- Affected users (if available)\n- Platform / culprit (the Rust file/function if available)\n\n### Regressions (`regressions`, `full`)\nIssues that were previously marked resolved but have reoccurred. These indicate incomplete fixes or flaky patches.\n\nFor each, capture:\n- Issue ID and title\n- Original resolution date\n- Recurrence date\n- Event count since recurrence\n\n### High-volume issues (`volume`, `full`)\nIssues with the most events in the last 24 hours, regardless of age. These are the most actively impacting users.\n\nFor each, capture:\n- Issue ID and title\n- Event count (24h)\n- Is this a known/triaged issue or unacknowledged?\n\n---\n\n## Step 3: Classify each issue\n\nFor every issue surfaced, classify it against the known patterns in `.claude/mcp-context.md`:\n\n**Tokenserver auth path** — involves `hawk_identifier`, FxA OAuth, `tokenserver-auth/`, `client_state`, `keys_changed_at`. These are user-facing auth failures — high priority.\n\n**Database layer** — involves `DbError`, `diesel`, `deadpool`, connection pool, migration, query timeout. May indicate infrastructure problems.\n\n**Syncstorage storage path** — involves BSO read/write, collection operations, quota. Usually lower priority unless volume is very high.\n\n**Expected noise** — 409 Conflict, 503 during deployment, staging-environment-only errors. Mark as \"noise\" and exclude from the action list.\n\n**Spanner-specific** — RST_STREAM, gRPC errors in production (not emulator). These should NOT appear in production — flag immediately.\n\n---\n\n## Step 4: Cross-reference with Jira (if context available)\n\nIf recent Jira data is available in this conversation, check whether any Sentry issues correspond to open `STOR-` bugs. Note the link if found.\n\nIf a high-volume or regression Sentry issue has no corresponding Jira ticket, that is a gap — flag it as \"needs ticket\".\n\n---\n\n## Step 5: Output format\n\n### Production Health: [OK / Watch / Degraded]\nOne-line overall assessment.\n\n### New Issues (Last 7 Days)\nTable: Issue ID | Title | First Seen | Events | Classification\nIf none: \"No new issues in the last 7 days.\"\n\n### Regressions\nTable: Issue ID | Title | Resolved On | Recurred On | Events Since\nIf none: \"No regressions detected.\"\n\n### High-Volume Issues (Last 24h)\nTable: Issue ID | Title | Events (24h) | Triaged? | Classification\nExclude known noise (mark separately if many).\n\n### Error Pattern Analysis\nBulleted findings only if genuine patterns are present:\n- Auth path errors (tokenserver risk)\n- DB layer errors (infrastructure risk)\n- Sudden volume spikes (availability risk)\n- Patterns that correlate with a recent deploy or migration\n\n### Recommended Actions\nUp to 5 concrete items. Each: issue ID, what to do, urgency. Skip if no action is warranted.\n\n### Noise / Expected Errors (Acknowledged)\nBrief list of issues seen but classified as expected noise. One line each. This confirms you looked at them.\n\n---\n\n## Guidelines\n\n- Do not fabricate event counts or timestamps — use exactly what Sentry returns\n- If a query fails or a tool is unavailable, note it and work with available data\n- `is_sentry_event()` in `syncserver/src/error.rs` is the canonical routing decision — errors that are HTTP 4xx should generally not be in Sentry at high volume; flag if they are\n- Focus on production environment; staging errors are informational only\n- An issue being \"old\" does not mean it's not important — check event count trends, not just age\n- Sentry issue IDs should be referenced in full (e.g. `SYNCSTORAGE-PROD-####`) if available\n","frontmatter":{"name":"sentry-health","description":"Queries Sentry for syncstorage-rs (project syncstorage-prod) to surface error trends, new issues, regressions, and production health signals. Reports as engineer/manager view of what requires attention.","user-invocable":true,"argument-hint":"new | regressions | volume | full (default: full)"},"isInternal":false,"tokens":1200,"sizeBytes":5261},{"name":"SKILL.md","path":".claude/skills/spanner-local-tests/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/spanner-local-tests/SKILL.md","title":"spanner-local-tests","category":"anthropic-skill","format":"markdown","content":"---\nname: spanner-local-tests\ndescription: Brings up the Spanner emulator, provisions the test database with the project schema, and runs the syncstorage-rs Spanner unit-test suite locally. Repeatable single-command flow plus full from-scratch documentation. Also covers the e2e (docker-compose) path.\nuser-invocable: true\n---\n\n# Run Spanner Tests Locally\n\nSet up a local Spanner emulator + schema, then run the spanner-backed unit tests. The whole loop is < 90 seconds once the emulator image is cached. This is the **fast inner loop** for any change touching `syncstorage-spanner/`.\n\nThere are two test surfaces in this repo:\n\n| Surface | What it covers | Time | Needs |\n|---|---|---|---|\n| **Unit tests** (this skill's default) | Direct calls into `SpannerDb` — exercises every DML path through `update_user_collection_quotas`, batch ops, BSO CRUD, tombstones, quota. | ~30s after warm-up | emulator only |\n| **Docker e2e** | Full syncserver + emulator + tokenserver + mock FxA, hit over HTTP by the Python integration suite. | 5-15 min | freshly-built `app:build` image |\n\nDefault to unit tests. Reach for e2e only when changes touch HTTP routing, token handling, or cross-backend behavior CI would catch first.\n\n## Quick re-run (after first-time setup)\n\nIf the emulator is already up and provisioned:\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nIf the emulator is **not** running, use the helper:\n\n```bash\n./scripts/spanner-local-test-setup.sh\n```\n\nThat script brings up the emulator, provisions the database/schema, and runs the suite. It's idempotent — re-running just executes the tests if the stack is already up.\n\n## From scratch — the full flow\n\n### 1. Prerequisites\n\n- **Docker Desktop running.** The emulator runs as a container.\n- **Rust toolchain** matching `rust-toolchain.toml` (or close).\n- For e2e only: enough Docker memory (≥ 8 GB recommended) — the `app:build` image compiles grpcio-sys from source and OOMs on small allocations.\n\n### 2. Start the emulator\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml up -d sync-db\n```\n\nThis starts the [`gcr.io/cloud-spanner-emulator/emulator`](https://cloud.google.com/spanner/docs/emulator) image. Two ports:\n\n- **9010** — gRPC. The Rust client (and production) talk this.\n- **9020** — REST. Used for admin (create instance / create database / DDL).\n\nWait ~2 seconds for the emulator to be ready. Confirm:\n\n```bash\ncurl -fsS -o /dev/null http://localhost:9020/v1/projects/test-project/instances && echo ok\n```\n\n### 3. Create the instance and database\n\n```bash\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"instance\":{\"config\":\"emulator-test-config\",\"nodeCount\":1,\"displayName\":\"Test\"},\"instanceId\":\"test-instance\"}'\n\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"createStatement\":\"CREATE DATABASE `test-database`\"}'\n```\n\n### 4. Apply the schema\n\n⚠️ **Do not run `scripts/prepare-spanner.sh` directly on macOS.** Its `sed` invocation uses GNU `\\+` syntax that BSD `sed` doesn't accept. The script is designed to run inside the Linux container (where GNU sed lives). When you run it from a macOS host, the DDL parsing silently returns an empty array, the database gets created with no tables, and tests fail with cryptic *\"Table not found: collections\"* errors.\n\nUse this macOS-safe inline equivalent:\n\n```bash\nDDL_STATEMENTS=$(grep -v '^--' syncstorage-spanner/src/schema.ddl \\\n | tr '\\n' ' ' \\\n | tr -s ' ' \\\n | sed 's/;[[:space:]]*$//' \\\n | jq -R -s -c 'split(\";\") | map(select(length > 0)) | map(gsub(\"^\\\\s+|\\\\s+$\";\"\"))')\n\nPAYLOAD=$(jq -n --argjson stmts \"$DDL_STATEMENTS\" '{statements:$stmts}')\n\ncurl -fsS -X PATCH \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n -H 'Content-Type: application/json' \\\n -d \"$PAYLOAD\"\n```\n\nThen verify the tables landed:\n\n```bash\ncurl -fsS \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n | jq '.statements | length'\n# expected: 8 (5 tables + 3 indexes + 2 row-deletion policies, the last 2 are folded into CREATE TABLE statements by the emulator)\n```\n\n### 5. Run the unit tests\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nTwo non-obvious env vars:\n\n- `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` — the gRPC endpoint. **This is what tells the client to skip OAuth.** If you leave it unset, every test fails with *\"Error occurred when fetching oauth2 token\"* because the client tries to authenticate against the real Google API.\n- `RUST_TEST_THREADS=1` — required. Many tests share the same database and assert on counts/timestamps; parallel runs race.\n\nExpected output: `test result: ok. 39 passed; 0 failed; …`.\n\n### 6. Run a single test\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db \\\n tests::db::delete_collection_tombstone -- --nocapture\n```\n\n`--nocapture` is essential when tests fail — without it, errors get swallowed.\n\n### 7. Teardown\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml down -v\n```\n\nThe `-v` wipes named volumes for the whole compose project — including `tokenserver_db_data`. If you have the full e2e stack (or anything else from this compose file) running in parallel, prefer the helper script's narrower teardown instead:\n\n```bash\n./scripts/spanner-local-test-setup.sh down\n```\n\nThat only stops + removes the `sync-db` container. The emulator keeps no data in named volumes, so next run starts from a fresh database either way.\n\n## Common failures\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `Error occurred when fetching oauth2 token` | Forgot `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST`. Client is trying real Google OAuth. | Export the var. |\n| `Table not found: collections [at 2:25]` | DDL didn't apply. Most often because `prepare-spanner.sh` was run on macOS and produced an empty statement list. | Use the inline DDL block in step 4. |\n| `INVALID_ARGUMENT: …` on `INSERT OR UPDATE` | Emulator version too old (< 1.5.x). | Update `docker/docker-compose.spanner.yaml`'s image tag. |\n| 39 tests \"pass\" in 0.2s | Tests didn't actually connect — they're failing the OAuth check and treating it as a fresh start. Check the first test's error. | Same as the OAuth row above. |\n| Tests intermittently fail with timestamp/count mismatches | `RUST_TEST_THREADS=1` not set. | Set it. |\n| `cargo test` error: *the package 'syncstorage-db' does not contain this feature: py_verifier* | `py_verifier` is on the `syncserver` crate. | Drop `--features=py_verifier` when running tests on `syncstorage-db` alone. |\n\n## Docker e2e (optional, slower path)\n\nThe Makefile target `docker_run_spanner_e2e_tests` spins up the full stack and runs the Python integration suite twice (with and without JWK cache). It requires the `app:build` image to be **current** — built against the working tree's code.\n\n```bash\ndocker build -t app:build .\nmkdir -p workflow/test-results\nSYNCSTORAGE_RS_IMAGE=app:build make docker_run_spanner_e2e_tests\n```\n\nTwo local-run-only gotchas the make target assumes CI handles:\n\n- `SYNCSTORAGE_RS_IMAGE=app:build` must be exported. `docker-compose.spanner.yaml:52` references `${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}`; without the override compose tries to pull a non-existent `syncstorage-rs:latest` from Docker Hub. CI sets it in `.github/workflows/main-workflow.yml`. Local doesn't get it for free.\n- `workflow/test-results/` must exist before the make target runs. The target uses `docker cp ... ${INT_SPANNER_NO_JWK_JUNIT_XML}` which writes into that directory; if missing, the docker cp fails with `invalid output path` and aborts the run.\n\nTwo tests will always fail locally and can be safely ignored:\n\n- `tools/integration_tests/tokenserver/test_e2e.py::test_unauthorized_oauth_error_status`\n- `tools/integration_tests/tokenserver/test_e2e.py::test_valid_oauth_request`\n\nBoth hit **real FxA stage** (`api-accounts.stage.mozaws.net`, `oauth.stage.mozaws.net`) and need live network + production credentials the docker stack doesn't have. They surface as `fxa.errors.OutOfProtocolError: API responded with non-json content-type` — FxA returning an HTML error page because the request can't authenticate. CLAUDE.md excludes these from `make run_local_e2e_tests` via `--ignore=tokenserver/test_e2e.py`; the docker e2e target doesn't apply that exclusion, so expect `149 passed, 2 errors` not `151 passed`.\n\nNotes:\n\n- The image build compiles grpcio-sys from C++ source. On Docker Desktop for Mac, allocate **≥ 8 GB** memory.\n- The build takes 10-20 minutes from cold. Subsequent rebuilds reuse cache layers and are much faster *unless* `Cargo.lock` or the Rust toolchain changed.\n- Results land in `workspace/test-results/` as JUnit XML.\n- The make target tears the stack down on completion.\n\n### `ar: unable to copy file 'libgrpc.a'; reason: Success`\n\nMisleading message — the linker wasn't killed, the BuildKit cache mount got into a corrupt state. Seen on macOS with stale buildkit caches (mine had ~19 GB accumulated).\n\nFix:\n\n> ⚠️ `docker builder prune -f` clears BuildKit cache mounts for **every** Docker project on the host, not just this repo. Unrelated projects' next builds will go from cached to cold. Run it only when the link error reproduces and you've ruled out a real OOM.\n\n```bash\ndocker builder prune -f\ndocker build -t app:build .\n```\n\nThe prune drops the cache mounts grpcio-sys writes its 100+ MB `libgrpc.a` into. The next build re-creates them clean and the link step succeeds. If it fails a second time after prune, *then* suspect a real OOM and bump Docker Desktop memory.\n\nIf the local build won't cooperate, CI on push runs the same e2e suite — relying on it is reasonable for refactors that don't touch HTTP/auth/routing.\n\n## When to invoke this skill\n\nReach for it when:\n\n- You're editing anything under `syncstorage-spanner/` and want to validate before pushing.\n- A spanner test is failing in CI and you need to reproduce locally.\n- You're investigating an emulator-vs-prod behavior question.\n- You forgot the env-var dance for the third time this month.\n\nDon't invoke it for changes that don't touch Spanner — the MySQL or Postgres backends have their own test paths through the Makefile's `test_with_coverage` and `postgres_test_with_coverage` targets.\n\n## Output format\n\nAfter running, report:\n\n1. **Emulator state:** running / fresh / pre-existing\n2. **Schema applied:** N statements\n3. **Test results:** `X passed; Y failed; Z ignored` — list the failed test names if any\n4. **Total wall time**\n5. **Teardown:** done / left running for follow-up\n\nIf anything failed, surface the actual error message (not just the test name) — Spanner emulator errors are verbose but informative, and the first line is usually enough to diagnose.\n","frontmatter":{"name":"spanner-local-tests","description":"Brings up the Spanner emulator, provisions the test database with the project schema, and runs the syncstorage-rs Spanner unit-test suite locally. Repeatable single-command flow plus full from-scratch documentation. Also covers the e2e (docker-compose) path.","user-invocable":true},"isInternal":false,"tokens":2918,"sizeBytes":11641},{"name":"SKILL.md","path":".claude/skills/syncserver-explain-code/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/syncserver-explain-code/SKILL.md","title":"syncserver-explain-code","category":"anthropic-skill","format":"markdown","content":"---\ndescription: Explains code for experienced engineers working on syncstorage-rs. Covers what changed, why it works, non-obvious decisions, gotchas, and data/control flow. Defaults to git diff vs main; accepts an optional file or path argument.\nargument-hint: file-or-path\n---\n\nYou are a senior engineer explaining code to another experienced engineer working on **syncstorage-rs** — Mozilla's Rust implementation of the Firefox Sync storage server and token server.\n\nSkip basics and language fundamentals. Focus on what this code does, why it was written this way, non-obvious decisions, and things that could surprise or bite someone.\n\n## Service boundary — critical context\n\nThis repo owns two services:\n- **Syncstorage** — the storage node that holds encrypted user sync data (bookmarks, passwords, tabs, etc.). Database backends: MySQL, PostgreSQL, Google Cloud Spanner.\n- **Tokenserver** — the allocation service that authenticates Firefox clients via FxA OAuth and assigns them to a storage node. Database backends: MySQL, PostgreSQL.\n\n**Firefox Accounts (FxA) is an external upstream service.** This repo does not own FxA. When explaining code:\n- References to `fxa_uid`, `fxa_kid`, `client_state`, FxA OAuth tokens, JWK keys, the FxA OAuth server URL — these are **inputs from FxA that this service validates and consumes**, not code this repo owns.\n- The `mock-fxa-server` in `scripts/` is a test stub only — it is not a real FxA implementation.\n- `tokenserver-auth/src/oauth/` and `tokenserver-auth/src/crypto.rs` are **this repo's FxA token verification logic**, not FxA's own code.\n\nMake this boundary explicit whenever the code touches the FxA/tokenserver interface.\n\n## How to gather the code\n\nIf `$ARGUMENTS` is provided, read that file or directory.\n\nOtherwise, run:\n\n```bash\ngit diff main...HEAD\n```\n\nand read the full diff. Follow imports or read related files as needed to give accurate explanations — do not explain in isolation if context from a related file matters.\n\n## Explanation structure\n\nWork through the code and produce an explanation covering all sections below. Omit a section only if it genuinely doesn't apply.\n\n### 1. One-paragraph summary\n\nPlain English. What does this code do, and what problem does it solve? Write for someone who hasn't read the ticket.\n\n### 2. Architecture & data flow\n\nWhere this fits in the broader system. Include an ASCII diagram if it clarifies the flow.\n\nUseful reference shapes for this codebase:\n\n```\nFirefox client\n → Tokenserver (validates FxA OAuth token, assigns node)\n → FxA OAuth server [external — validates token, not our code]\n → tokenserver-db (MySQL/Postgres — users, nodes, services tables)\n → Syncstorage node (stores/retrieves BSOs)\n → syncstorage-db (MySQL / Postgres / Spanner)\n```\n\n```\nRequest → Actix-web (syncserver/)\n → Extractors (hawk_identifier, fxa_uid, token validation)\n → Handler\n → db trait (syncstorage-db or tokenserver-db)\n → backend impl (mysql / postgres / spanner)\n```\n\n### 3. Annotated walkthrough\n\nStep through the key functions, types, or request paths. For each:\n- What it does\n- Why it's structured this way (if non-obvious)\n- How it connects to the next step\n\nFocus on the critical path. Don't exhaustively document trivial helpers.\n\nNote Rust-specific patterns where relevant: trait objects vs generics, `Arc`, `actix_web::web::Data<>`, `grpcio` vs `tonic`, `diesel` query builder patterns, `serde` field renaming.\n\n### 4. Gotchas & non-obvious bits\n\nThe most important section. Flag:\n\n- Implicit assumptions or preconditions the caller must satisfy\n- Surprising behavior or edge cases (off-by-one, async ordering, race conditions)\n- Why an obvious alternative wasn't taken (if inferrable)\n- Error handling that silently swallows failures or has unexpected fallback behavior\n- State mutated in non-obvious places\n- Performance characteristics worth knowing (N+1 queries, large allocations, blocking calls in async context)\n- Security-sensitive paths — especially: HAWK token validation, FxA OAuth scope checking, `client_state` / `keys_changed_at` handling, anything touching node assignment\n- gRPC-specific issues (Spanner backend): RST_STREAM handling, retry logic, emulator vs production behavioral differences\n- Migration safety: anything that changes DB schema or touches published migration files\n\n### 5. Dependencies & integrations\n\nExternal systems or crates this code depends on that aren't obvious from the code alone. Distinguish:\n- **Owned by this repo:** `syncstorage-*`, `tokenserver-*`, `syncserver-*` crates\n- **External upstream:** FxA OAuth server, Google Cloud Spanner, `tokenlib` (Python), `hawkauthlib`\n- **Test infrastructure only:** mock FxA server, Spanner emulator\n\nNote any version constraints or behavioral quirks.\n\n### 6. Testing notes\n\nHow is this code tested? Are there gaps? Note:\n- Unit tests (Rust `#[cfg(test)]` modules)\n- Integration tests (`tools/integration_tests/` — pytest, requires a running server)\n- E2E tests (`tokenserver/test_e2e.py` — hits real FxA staging, session-scoped, slow)\n- Any known flaky behavior (Spanner emulator quirks, timing-sensitive tests, 409/503 retry paths)\n\n## Style guidelines\n\n- Be direct and dense. Skip preamble.\n- Use code formatting for identifiers, file paths, and values.\n- Use ASCII diagrams when they save more words than they cost.\n- If something is genuinely straightforward, say so in one sentence and move on.\n- When the FxA boundary is crossed, say so explicitly rather than implying this repo owns that behavior.\n","frontmatter":{"description":"Explains code for experienced engineers working on syncstorage-rs. Covers what changed, why it works, non-obvious decisions, gotchas, and data/control flow. Defaults to git diff vs main; accepts an optional file or path argument.","argument-hint":"file-or-path"},"isInternal":false,"tokens":1263,"sizeBytes":5604},{"name":"SKILL.md","path":".claude/skills/syncserver-jira-description/SKILL.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/skills/syncserver-jira-description/SKILL.md","title":"syncserver-jira-description","category":"anthropic-skill","format":"markdown","content":"---\nname: syncserver-jira-description\ndescription: Drafts a concise Jira description for a syncstorage-rs task. Gathers context via targeted interview, researches relevant patterns in the repo, then outputs a clean description ready for an engineer to hand to Claude for implementation.\nuser-invocable: true\n---\n\n# Syncserver Jira Description\n\nDraft a Jira description for a syncstorage-rs task. Output the description only — do not create, edit, or suggest changes to any source files.\n\n## Step 1: Gather context\n\nIf a planning doc, epic description, or tech spec was provided, read it first and infer what, why, crates, and constraints before asking anything.\n\nRequired information:\n- **What:** What is being built or changed, in one sentence\n- **Why:** Motivation — user need, requirement, bug, or tech debt\n- **Crates:** Which specific crate(s) will be modified (e.g. `syncstorage-spanner`, `tokenserver-auth`, `syncserver`, `tokenserver-postgres`)\n- **Constraints:** DB migration required, multi-backend impact (MySQL/Postgres/Spanner), breaking API change, feature flag, FxA contract change — or none\n\nIf all four are clear from provided context, proceed directly to Step 2. Otherwise ask for only what is missing in a single message. Also invite related PRs, tickets, existing approach notes, or architecture notes that would add useful context.\n\n## Step 2: Research\n\nSearch only the crates identified in Step 1. Find the most relevant existing patterns: similar feature, nearby handler, equivalent DB query, related error type. Expand to the broader workspace only if nothing relevant is found there.\n\nIdentify:\n- Key files an implementer will need to touch\n- The closest existing reference implementation to follow\n- Which database backends are affected and whether the change must be replicated across MySQL, PostgreSQL, and/or Spanner\n- Whether new migrations are required (and which crates: `syncstorage-mysql`, `syncstorage-postgres`, `tokenserver-mysql`, `tokenserver-postgres`)\n- Whether tests, metrics (Glean/StatsD), or Sentry error handling apply\n\nIncorporate findings directly into the draft — do not list them separately or ask for confirmation. Surface genuine blockers as Open Questions.\n\n## Step 3: Output\n\n**Background:**\nWhy this is needed and what it enables. 2–4 sentences. Note which service is affected — Syncstorage (storage node), Tokenserver (node allocation + FxA auth), or both.\n\n**Acceptance Criteria:**\nObservable, testable outcomes. Each item verifiable without reading the code. Include criteria for tests, metrics emission, and Sentry error classification where applicable.\n\n**Implementation Steps:**\nNumbered steps with crate paths, file paths, trait/function names, and structural guidance. Reference the nearest existing pattern for each step. For multi-backend changes, call out which backends need matching implementations. No code snippets — file locations, types, and patterns only.\n\n**Database Migrations:** *(omit if no schema changes)*\nWhich migration crates need new files (`syncstorage-mysql/migrations/`, `tokenserver-postgres/migrations/`, etc.). Reminder: never edit published migrations — add new `up.sql`/`down.sql` pairs only.\n\n**Tests:**\nWhat needs to be tested. Unit (`#[cfg(test)]` in the relevant crate), integration (`tools/integration_tests/` pytest — requires running server), and E2E (`tokenserver/test_e2e.py` — hits FxA staging, session-scoped) coverage expectations. Reference the nearest existing test file as a pattern.\n\n**Metrics & Error Handling:** *(omit if not applicable)*\nAny Glean metrics, StatsD counters, or Sentry event classifications that should be emitted or updated. Note whether new error variants should be routed to Sentry or suppressed as metrics (see `is_sentry_event()` pattern in `syncstorage-spanner/src/error.rs`).\n\n**Key Reference Files:**\nSpecific files the implementer should read before starting. One line each.\n\n**Out of Scope:** *(omit if not needed)*\n\n**Open Questions:** *(omit if none)*\n\n## Guidelines\n\n- Output the description only — no source file changes\n- Implementation Steps should give enough detail to start work without follow-up questions — crate paths, file paths, and patterns, not prose\n- Multi-backend tasks: always explicitly name which backends (MySQL / PostgreSQL / Spanner) need matching changes; do not say \"all backends\" without listing them\n- Migration tasks: always reference the no-edit-published-migrations rule\n- Omit redundant or obvious acceptance criteria\n- Include Database Migrations, Metrics & Error Handling only when relevant\n- If motivation or scope remain unclear after asking, flag as an Open Question rather than assuming\n- Branch naming reminder for the assignee: Mozilla engineers use `type/description-STOR-####`; community contributors use `type/description-####`\n","frontmatter":{"name":"syncserver-jira-description","description":"Drafts a concise Jira description for a syncstorage-rs task. Gathers context via targeted interview, researches relevant patterns in the repo, then outputs a clean description ready for an engineer to hand to Claude for implementation.","user-invocable":true},"isInternal":false,"tokens":1029,"sizeBytes":4831},{"name":"load-testing.md","path":"docs/src/tools/load-testing.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/load-testing.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Load Testing\n\n## Quickstart: load test the change stream system (STOR-628)\n\nFour commands. Needs Docker, plus `curl` and `jq`. No GCP account, no\ncredentials, no VM.\n\n```console\nmake loadtest-offload-image # once: builds the server image\nmake loadtest-offload-changestream # run the load test\nmake loadtest-offload-report # did the pipeline keep up?\nmake loadtest-offload-down # tear it all down\n```\n\nThe report is the answer. Three numbers matter:\n\n| Look for | Meaning |\n| --- | --- |\n| `filtered` non-zero | The pipeline correctly dropped change records that have no payload file attached. This is the thing STOR-628 is asking about. |\n| `handler errors` zero | Nothing failed while finalizing or deleting objects. |\n| `noop_skips` zero | Nothing inert slipped past the filter. |\n\nIt prints `OK: pipeline drained clean` at the end, or a list of `FAIL` lines\nexplaining what went wrong, and exits non-zero on a real problem so it can gate\na script.\n\nOne thing to expect: a `NOTE` saying `batch_commit_skips` is 0. That is a known\nemulator limitation rather than your run going wrong. See\n[The emulator drops transaction tags](#the-emulator-drops-transaction-tags).\n\nFor a real run rather than a smoke test, pass molotov your own arguments:\n\n```console\nmake loadtest-offload-changestream \\\n MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 3600 -v\"\n```\n\nTo load test the handler read/write path instead (STOR-629), swap in\n`make loadtest-offload-handler`. Same report, same teardown.\n\nEverything below is detail you only need when a run looks wrong, or when you\nneed real latency numbers and have to go to GCP.\n\n---\n\n## Summary\n\nThis is the runbook for load testing syncstorage, with the offloaded-payload\npipeline as the main case. It replaces the Confluence page \"Syncstorage &\nTokenserver Runbooks\", whose molotov and locust instructions had become\ninterleaved and unreadable.\n\nThere are two load tests in the repo and they are unrelated:\n\n| | Target | Tool | Location |\n| --- | --- | --- | --- |\n| Syncstorage | storage read/write API | molotov | `tools/syncstorage-loadtest/` |\n| Tokenserver | `/1.0/sync/1.5` token issuance | locust | `tools/tokenserver/loadtests/` |\n\nEverything below is about the syncstorage/molotov one unless it says otherwise.\nFor tokenserver, see [Tokenserver load tests](#tokenserver-load-tests-locust).\n\nTwo tickets drive the offload work, and they want different traffic shapes:\n\n| Ticket | Goal | Traffic shape |\n| --- | --- | --- |\n| STOR-629 | Performance of the handler's GCS read/write path | 100% offloaded, every payload expanded |\n| STOR-628 | The change stream to reconciler pipeline, including the filtering of NULL `payload_link` records | Mixed offloaded and inline |\n\nThe distinction matters. STOR-628 is only a real test if some writes are\n*not* offloaded, because the record it is checking for is the one the\npublisher's filter is supposed to drop.\n\n---\n\n## Which rig to use\n\n```mermaid\nflowchart TB\n q1{\"What are you
measuring?\"}\n q1 -->|\"pipeline correctness:
does every object get
finalized, is the filter
dropping NULL records\"| local[\"Local docker rig
(emulators)\"]\n q1 -->|\"real latency and
throughput numbers
for a capacity decision\"| gcp[\"GCP dev environment\"]\n local --> both[\"Run local first either way:
it catches wiring bugs in
minutes with no GCP spend\"]\n```\n\nThe local rig runs the whole pipeline on emulators. It is the right default:\nit needs no GCP project, no credentials and no VM, and it answers every\ncorrectness question. What it cannot give you is a trustworthy latency\nnumber, because fake-gcs-server on a loopback interface is nothing like GCS\nover the network, and the Spanner emulator is single-node and in-process.\n\nUse GCP dev when the question is \"how fast\" rather than \"does it work\".\n\n---\n\n## Local docker rig\n\n### What it stands up\n\nThe rig is the existing reconciliation e2e stack plus a molotov container and\na metrics sink. Compose overlay: `docker/docker-compose.loadtest.yaml`.\n\n```mermaid\nflowchart LR\n lt[\"loadtest
molotov\"]\n\n subgraph fg[\"Foreground: the request path (STOR-629)\"]\n ss[\"syncserver
offload enabled\"]\n end\n\n gcs[(\"fake-gcs
test-payloads
filesystem backend\")]\n db[(\"Spanner emulator
bsos, batch_bsos
payload_link_changes\")]\n\n subgraph bg[\"Background: the cleanup arm (STOR-628)\"]\n pub[\"payload-link-publisher
stands in for Dataflow\"]\n ps[\"Pub/Sub emulator\"]\n rec[\"payload-reconciler
long-running mode\"]\n end\n\n drop([\"dropped: inert records
payload_link NULL both sides\"])\n sd[\"statsd-exporter
:9102\"]\n rep[\"loadtest-report.sh
the verdict\"]\n\n lt -->|\"Hawk, direct access
no tokenserver or FxA\"| ss\n ss -->|\"upload payload\"| gcs\n ss -->|\"store gs:// in payload_link\"| db\n gcs -->|\"download on read\"| ss\n db -->|\"change stream\"| pub\n pub -.->|filtered| drop\n pub -->|published| ps\n ps --> rec\n rec -->|\"finalize: committed=true
delete: orphans\"| gcs\n\n ss -.->|dogstatsd| sd\n pub -.->|dogstatsd| sd\n rec -.->|dogstatsd| sd\n sd -->|\"scrape counters\"| rep\n gcs -->|\"list objects,
check committed + customTime\"| rep\n```\n\nThe two halves map onto the two tickets. Everything left of the change stream\nis STOR-629's territory, and it costs the user latency. Everything right of it\nis STOR-628's, and no user waits on it, but a wrong decision there deletes a\nfile a live row still needs.\n\nNote where the report gets its evidence: counters from the statsd sink, and\nobject state read straight out of the bucket. It never trusts molotov's exit\nstatus, because the cleanup arm runs after the response is sent and can fail\nlong after the client saw a 200.\n\n| Service | Role |\n| --- | --- |\n| `sync-db` | Spanner emulator, carrying the `payload_link_changes` change stream from `schema.ddl` |\n| `pubsub-emulator` | Pub/Sub |\n| `fake-gcs` | GCS, on its **filesystem** backend for this rig |\n| `reconciliation-setup` | One-shot: creates the topic, subscription and bucket |\n| `payload-link-publisher` | Python change-stream reader, stands in for the Dataflow job |\n| `payload-reconciler` | Drains Pub/Sub, finalizes and deletes GCS objects |\n| `syncserver` | Offload enabled, limits raised |\n| `statsd` | `prom/statsd-exporter`, so the publisher and reconciler counters are readable |\n| `loadtest` | molotov |\n\nThe publisher is the Python one, not the Java Dataflow template. It publishes\nthe identical JSON wire format, so everything downstream of it is\nproduction code. If you need to reproduce something Java-specific, layer\n`docker-compose.e2e.reconciliation.java.yaml` on top; see\n[Reconciliation Pipeline](payload_link_reconciler.md).\n\nOn the NULL-filtering half of STOR-628, be precise about what the local rig\nproves. The filter exists twice: `isPayloadLinkActionable` in the Java flex\ntemplate, and `is_payload_link_actionable` in\n`payload-link-publisher-py/utils.py`, which is written to mirror it. A local\nrun exercises the **Python** one. That is a real test of the filter's logic\nand of everything downstream, but it is not a test of the Java code that runs\nin production. To exercise the shipping filter, either use the Java overlay\nlocally or run against GCP dev, where the Dataflow job is the publisher.\n\n### Prerequisites\n\n- Docker running, with roughly 8 GB available to the VM. Expanded payloads\n are held in memory in several places at once.\n- `curl` and `jq` on the host, for the report script.\n- Nothing else. No GCP credentials, no `service-account.json`.\n\n### The stale publisher image trap\n\nRead this before trusting any local result.\n\n`payload-link-publisher` is the only service in the stack built from a compose\n`build:` context rather than pulled or shared with `app:build`. Plain\n`docker compose up` reuses whatever `docker-payload-link-publisher` image is\nalready in your local cache and **will not rebuild it when its source\nchanges**. So a publisher image built before a change to\n`payload-link-publisher-py/` keeps running against new server code, silently.\n\nThis is not hypothetical. It is how the rig behaved on first use here: a\nfour-week-old publisher image had no `transaction_tag` support at all, so no\nrecord ever carried the `batch_commit` tag, so the reconciler treated every\nbatch-commit handoff as a genuine `batch_bsos` removal and deleted GCS objects\nthat committed `bsos` rows still pointed at. The symptom was HTTP 500s on\nreads, `payload_reconciler.batch_commit_skips` flat at zero, and\n`gcs_404{op=finalize}` running nearly 1:1 with `finalizes`. It looked exactly\nlike the STOR-657 production bug, and it was entirely a stale image.\n\nThe `loadtest-offload-*` targets therefore pass `--build` on every `up`. The\nfirst build is slow from a cold cache; afterwards it is cached and cheap.\n\nTwo things follow. If you drive compose by hand instead of through the\nMakefile, pass `--build` yourself. And note that\n`make docker_run_reconciliation_e2e_tests` does **not** pass it, so a local run\nof the reconciliation e2e suite can exercise a stale publisher. CI is unaffected\nbecause its cache starts empty.\n\nThe tell, if you suspect it:\n\n```console\ndocker compose ... exec payload-link-publisher grep -c transaction_tag /app/publisher.py\n```\n\nZero means the image predates transaction-tag support.\n\n### Run it\n\n```console\nmake loadtest-offload-image # build app:build (spanner backend), once\n\nmake loadtest-offload-handler # STOR-629\nmake loadtest-offload-report\n\nmake loadtest-offload-changestream # STOR-628\nmake loadtest-offload-report\n\nmake loadtest-offload-down # tear down, including the fake-gcs volume\n```\n\n`loadtest-offload-report` waits for the reconciler to stop making progress\nbefore reporting, so run it straight after the load test rather than trying\nto guess a sleep. It exits non-zero if it finds a problem, which makes it\nusable as a gate.\n\nBoth scenario targets take `MOLOTOV_ARGS`:\n\n```console\nmake loadtest-offload-handler MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 600 -v\"\n```\n\nStart small. Every worker can be holding a multi-MiB payload, on both the\nclient and the server, so worker count multiplies memory rather than just CPU.\nFive workers writing 10 MiB records is already a meaningful amount of traffic\nthrough this stack.\n\n### Reading the report\n\n```text\nRECONCILER (payload_reconciler.*)\n payload_reconciler_finalizes 1843\n payload_reconciler_orphan_deletes 291\n payload_reconciler_batch_commit_skips 412\n...\nGCS OBJECTS (test-payloads)\n objects 1552\n committed=true 1552\n committed!=true 0\n customTime pinned 1552\n...\nVERDICT\n OK: pipeline drained clean.\n```\n\nWhat each number should look like:\n\n| Metric | Healthy | If it is wrong |\n| --- | --- | --- |\n| `published` | Tracks actionable change records | Zero means the publisher is not reading the change stream at all |\n| `filtered` | **Non-zero on a mixed run.** This is the positive evidence for STOR-628: the publisher dropped records with `payload_link` NULL on both sides | Zero on a mixed run means either the filter is passing inert records through, or every write happened to be offloaded and the path went untested |\n| `finalizes` | Tracks offloaded write volume | Zero with objects in the bucket means the change stream, publisher or Pub/Sub leg is broken |\n| `orphan_deletes` | Non-zero, tracks overwrites and deletes | Zero means the load test never deleted anything; check `DISABLE_DELETES` |\n| `batch_commit_skips` | **Zero on the local rig** -- see the emulator limitation below. Non-zero is what you require on GCP dev | Zero on *GCP dev* means the transaction tag is not reaching the change stream, which is the STOR-657/668 bug class |\n| `noop_skips` | **Zero** | Non-zero is the STOR-628 regression: the filter let through a record with `payload_link` NULL on both sides |\n| `errors kind:handler` | **Zero** | Non-zero means messages went unacked. In prod five failures on one message routes it to the DLQ |\n| `committed!=true` | Zero once drained | A small tail is normal if you cut the run off mid-flight. A large or growing count means the pipeline is not keeping up with the write rate |\n\n`committed=true` count and `customTime pinned` count must agree. Finalize sets\nboth in a single `blob.patch()`, so a divergence means something other than\nthe reconciler is writing object metadata.\n\nOne thing that surprises people: even the \"100% offloaded\" STOR-629 run\nproduces a non-zero `filtered` count. `OFFLOAD_COLLECTIONS` only governs the\nbatch-write arm, while the load test also PUTs `meta/global` and POSTs\n`clients` on every scenario pass. Those two are never in the offload list, so\nthey generate exactly the inert NULL-on-both-sides records the filter is\nthere to drop. So STOR-629 covers the filter incidentally, and STOR-628 is\nstill the run that covers it deliberately, at volume, alongside offloaded\ntraffic in the same stream.\n\n### Reading the 404 counters, and why they are the whole signal\n\nThe reconciler treats `404 NotFound` as success on both of its operations.\nFrom `reconcile_payload_links.py`, a finalize whose target is gone logs at\n`debug`, bumps `gcs_404{op=finalize}`, and **acks the message**. Same shape for\ndelete. [Reconciliation Pipeline](payload_link_reconciler.md) states the\nrationale plainly: \"Both reconciler actions are idempotent, so a redelivered\nrecord is harmless.\"\n\nThat is correct and deliberate, and it has a consequence worth internalising\nbefore you read a load test result. Because a missing object is success, a run\nthat is silently destroying data produces:\n\n- `errors` `kind:handler` at **zero**\n- nothing in the DLQ\n- no `WARN` or `ERROR` in the reconciler log, only `debug`\n- an elevated `gcs_404` counter, and nothing else\n\nSo on the offload path `gcs_404{op=finalize}` is not a nice-to-have gauge. It\nis the *only* signal that separates a healthy pipeline from one that is\ndeleting live payloads. Everything else looks green.\n\nThat existing doc's failure-modes table gives two causes for a sustained\n`gcs_404{op=finalize}`:\n\n| Documented cause | Applies to a local run? |\n| --- | --- |\n| The lifecycle rule reclaimed the object before it was finalized | **No.** The 30 day `daysSinceCustomTime` rule is a bucket policy in `bucket.tf`. fake-gcs has no lifecycle policy at all, so nothing can reap an object locally. |\n| The same message was redelivered after a successful prior run (the at-least-once tax) | **No**, not at volume. Redelivery is a background trickle, not a rate approaching `finalizes`. |\n\nMeasured on these runs, neither explains what was observed:\n\n| Run | `finalizes` | `gcs_404{op=finalize}` | Ratio |\n| --- | --- | --- | --- |\n| STOR-629 handler | 930 | 716 | 77% |\n| STOR-628 changestream | 1853 | 1589 | 86% |\n\nA ratio near 1:1 is a third cause that the table does not list: **the\nreconciler deleted the object itself**, as an orphan, and the finalize for that\nsame object then 404s. Locally that is the emulator dropping the\n`batch_commit` transaction tag (see below), so every batch-commit handoff is\nmisread as a genuine `batch_bsos` removal.\n\nThe signature to recognise, wherever you see it:\n\n```text\nbatch_commit_skips 0 <- tag never arrived\ngcs_404{op=finalize} ~= finalizes <- finalizing objects already deleted\norphan_deletes high <- the deletes that did the damage\nerrors kind:handler 0 <- and nothing reports a problem\n```\n\n`gcs_404{op=delete}` is the benign one, and the existing doc's read of it holds\nwithout qualification: the object was already gone, the operation is idempotent\nby design, and a redelivery or a concurrent cleanup produces it. It carries no\ndata-loss implication on its own.\n\nTwo things follow for the alerting the reconciler doc already recommends. Its\nsuggestion to watch \"`gcs_404` `op:finalize` rising as a share of `finalizes`\"\nis the right alert, and this gives it a concrete threshold to reason about: a\nlow single-digit share is the redelivery tax, while anything approaching\n`finalizes` means objects are being deleted out from under committed rows. And\npairing it with `batch_commit_skips == 0` distinguishes the batch-commit cause\nfrom a lifecycle-window problem.\n\n### Knobs\n\nServer side, set on the `syncserver` container:\n\n| Variable | Default in the rig | Meaning |\n| --- | --- | --- |\n| `LOADTEST_OFFLOAD_COLLECTIONS` | `bookmarks,history` | Which collections syncserver offloads. **This is what decides mixed vs fully offloaded.** |\n| `LOADTEST_COLLECTION_LIMITS` | 10 MiB record / 20 MiB post / 21 MiB request, for `bookmarks` and `history` only | Per-collection raised limits, as a JSON map. Keys must be a subset of the offload list -- see below |\n| `LOADTEST_GCS_MAX_CONCURRENCY` | 4 | Concurrent GCS ops within one batch request. The main server-side throughput knob, worth sweeping |\n| `LOADTEST_DB_POOL_MAX_SIZE` | 40 | The stock 10 saturates first once workers climb |\n\n### Raise limits per collection, never globally\n\nThis is the one thing that will bite you, so it is worth stating plainly.\n\nSpanner caps a single `STRING(MAX)` value at **2621440 bytes (2.5 MiB)**. That\nhard cap is the reason this whole project exists. Raising the *global*\n`max_record_payload_bytes` above it does not fail at the API, it fails in the\ndatabase, on every collection that is not offloaded:\n\n```text\nFAILED_PRECONDITION New value exceeds the maximum size limit for\nthis column: bsos.payload, size: 10485760, limit: 2621440\n```\n\nAn offloaded collection is exempt, because its payload goes to GCS and the row\nstores only a `gs://` URL in `payload_link`, leaving `payload` NULL. So a limit\nabove 2.5 MiB is only ever correct for a collection that is in\n`gcs_payload_offload_collections`. That is exactly what the per-collection\noverride exists for, and how production raises it:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":10485760,\"max_post_bytes\":20971520,\"max_request_bytes\":22020096}}'\n```\n\nLeave the globals alone. Any collection not named in the map keeps stock\n2.5 MiB behaviour and writes inline safely. `max_post_bytes` and\n`max_request_bytes` are overridable per collection too; actix's payload config\nis sized from the largest `max_request_bytes` across all overrides\n(`ServerLimits::effective_max_request_bytes`), so a big body is not rejected at\nthe resource level before per-collection enforcement runs.\n\nThere is no per-collection `max_total_bytes`. The global default (clamped to\n`MAX_SPANNER_LOAD_SIZE`, 100 MB, by `Settings::normalize` on spanner) is\nalready generous, so the rig leaves it untouched.\n\nOn a mixed run this arrangement gives you exactly the right traffic for free:\noffloaded collections take expanded payloads, and inline collections get\n`LARGE_PAYLOAD_PROB` applied but capped at 2.5 MiB by\n`payload_target_length`, which is realistic \"large but not offloaded\" traffic.\n\nClient side, set on the `loadtest` container:\n\n| Variable | Meaning |\n| --- | --- |\n| `LARGE_PAYLOAD_PROB` | Fraction of BSOs given an expanded payload. `1.0` for STOR-629 |\n| `LARGE_PAYLOAD_SIZE` | Explicit target size in bytes. Unset means \"use the server's `max_record_payload_bytes`\" |\n| `OFFLOAD_COLLECTIONS` | Restricts batch writes to these collections. **Leave unset for mixed runs** |\n| `DISABLE_DELETES` | Set truthy to suppress the DELETE arm. Leave off, or the reconciler's `orphan_deletes` branch is never exercised |\n\nThe two collection lists are easy to confuse:\n\n- `LOADTEST_OFFLOAD_COLLECTIONS` (server) decides which collections get\n offloaded.\n- `OFFLOAD_COLLECTIONS` (client) decides which collections get *written to*.\n\nSetting the client one to a single collection sends every batch write there,\nwhich is what you want for STOR-629 and exactly what you must not do for\nSTOR-628. For a mixed run, leave the client variable unset so writes spread\nacross molotov's default five (`bookmarks`, `forms`, `passwords`, `history`,\n`prefs`) and set the server list to a subset of those.\n\n### Per-collection limit overrides\n\nProduction raises limits for one collection rather than globally, via a JSON\nmap. The load test resolves limits per target collection out of\n`/info/configuration`, preferring a collection's override, so this path is\ncovered too:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":20971520,\"max_post_bytes\":26214400,\"max_request_bytes\":26218496}}'\n```\n\n### Known limits of the local rig\n\n- **The emulator does not carry transaction tags, so the batch-commit skip\n cannot be validated locally.** This one is load bearing; details below.\n- **No DLQ.** `docker/reconciliation-setup.sh` creates the topic and\n subscription but no dead-letter topic, so dead-letter routing after five\n failed deliveries cannot be exercised locally. Verify that on GCP dev.\n- **No Dataflow.** The Python publisher replaces the Java job. Autoscaling,\n partition-split handling under real load and the connector's Spanner\n metadata writes are all Dataflow behaviours that only appear on GCP.\n- **Latency is not meaningful.** fake-gcs over loopback and a single-node\n Spanner emulator. Use it for correctness and relative comparisons only.\n- **No lifecycle policy.** The 30 day `daysSinceCustomTime` rule that reaps\n unfinalized objects is a bucket policy, so the local rig can only check\n that `customTime` was pinned, not that the reaping behaves.\n\n### The emulator drops transaction tags\n\nThe Spanner emulator emits the `transaction_tag` field on change stream\nDataChangeRecords but never populates it. Measured on emulator v1.5.52: the\nDataChangeRecord struct has arity 13, so index 11 (`transaction_tag`) exists\nand the publisher reads it correctly, but every record carries the empty\nstring, including the `batch_bsos` DELETE that *is* the batch-commit handoff.\n\nThat single gap has a large consequence. The reconciler decides whether to keep\nor delete a `batch_bsos` removal's object by checking\n`transactionTag == \"batch_commit\"` (see\n[Reconciliation Pipeline](payload_link_reconciler.md)). With the tag always\nempty, every batch-commit handoff looks like a genuine removal, so the\nreconciler deletes the GCS object that the just-committed `bsos` row now points\nat. The BSO becomes unreadable and syncserver returns a 500 backed by a GCS 404.\n\nSo on the local rig, expect all of this, and do not read it as a regression:\n\n- `payload_reconciler.batch_commit_skips` flat at zero.\n- `gcs_404{op=finalize}` running close to 1:1 with `finalizes`.\n- Objects missing for BSOs written in a batch create/append request.\n- A residual molotov failure rate, because roughly 50% of its POST scenarios\n are transactional batches. Reads of those BSOs 500.\n\n`docker/batch-commit-probe.py` reproduces it in isolation: one two-request\ntransactional batch, then a read of both BSOs, printing the bucket contents at\neach stage.\n\n```console\nmake loadtest-offload-up\nSYNCSTORAGE_RS_IMAGE=app:build docker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n -f docker/docker-compose.loadtest.yaml \\\n run --rm --entrypoint python3 \\\n -v \"$PWD/docker/batch-commit-probe.py:/probe.py:ro\" loadtest /probe.py\n```\n\nAdding `--no-deps` to that command keeps the reconciler stopped, and the probe\nthen passes with both payloads intact. That is the control which proves the\nreconciler is the thing deleting the object, rather than the batch commit\nlosing the `payload_link`.\n\nWhat this does **not** tell you is whether production is affected. The app side\nis correct: `handlers.rs` calls\n`with_transaction_tag(BATCH_COMMIT_TRANSACTION_TAG)` on the commit path, and\nreal Spanner does record transaction tags on change streams. The evidence\npoints at an emulator gap rather than a code defect, but the local rig cannot\nclose that question. **Confirm the batch-commit skip on GCP dev**, where the\nDataflow job reads a real change stream: run the probe there and require\n`payload_reconciler.batch_commit_skips` to be non-zero.\n\nTwo follow-ups worth filing rather than leaving implicit:\n\n1. There is no e2e test for the batch-commit handoff.\n `tools/integration_tests/test_payload_link_reconciliation.py` covers upload,\n update and delete, but not a transactional batch. That is presumably because\n it cannot pass against the emulator, which is exactly the point: the gap is\n invisible in CI.\n2. If the Java overlay behaves the same way, the emulator is confirmed as the\n cause. That is a cheap check and worth doing before spending GCP time.\n\n---\n\n## GCP dev environment\n\nUse this for real numbers. The dev pipeline is fully provisioned; see\n[GCP Infrastructure](payload-offload-infrastructure.md) for what lives where\nand which repo owns it. That page is the authority on the resources\nthemselves. The diagram below is only the load-testing view: where the\ngenerator sits, what it traverses, and where you go to observe each leg.\n\n```mermaid\nflowchart TB\n subgraph gen[\"Load generator (you create this)\"]\n vm[\"C4D VM, Ubuntu
molotov + load_test.pem
in moz-fx-sync-nonprod\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (shared GKE)\"]\n ts[\"tokenserver
trusts your public JWK\"]\n ss[\"syncserver
needs GCS_PAYLOAD_BUCKET +
OFFLOAD_COLLECTIONS + raised limits\"]\n rec[\"payload-reconciler
cronjob, ~5 min cadence\"]\n end\n\n subgraph v1[\"moz-fx-sync-nonprod-904c (GCPv1, cloudops-infra)\"]\n sp[(\"Spanner syncdb-dev
change stream:
payload_link_changes\")]\n end\n\n subgraph tenant[\"moz-fx-sync-nonprod (tenant, webservices-infra)\"]\n df[\"Dataflow flex template
the REAL Java filter\"]\n ps[\"Pub/Sub
payload-link-changes\"]\n dlq[\"Pub/Sub DLQ
after 5 failed deliveries\"]\n b[(\"GCS
sync-nonprod-dev-
syncstorage-payloads
30d lifecycle rule\")]\n end\n\n vm -->|\"1. self-signed JWT\"| ts\n ts -->|\"2. Hawk token\"| vm\n vm -->|\"3. storage reads + writes\"| ss\n ss -->|\"upload / download\"| b\n ss -->|\"payload_link\"| sp\n sp -->|\"change stream read\"| df\n df -->|publish| ps\n ps -->|\"pull subscription\"| rec\n ps -.-> dlq\n rec -->|\"finalize / delete\"| b\n```\n\nThree things this buys you that the local rig cannot:\n\n| Leg | Why it only exists here |\n| --- | --- |\n| Dataflow | The local rig substitutes a Python publisher. This is the actual Java `isPayloadLinkActionable` filter that ships, plus autoscaling and partition-split behaviour under real load. |\n| Real Spanner | Populates `transaction_tag`, so the batch-commit skip can finally be verified. Also the only place to measure what the change stream costs in Spanner storage, which has never been quantified. |\n| DLQ and lifecycle rule | Neither exists locally. `reconciliation-setup.sh` creates no dead-letter topic, and fake-gcs has no lifecycle policy. |\n\n### Load generator VM\n\nGenerate load from inside GCP, not from a laptop. A laptop's uplink is the\nbottleneck long before the server is, and multi-MiB payloads make that worse.\n\n1. Console -> project `moz-fx-sync-nonprod` -> Compute Engine -> VM instances\n -> Create instance.\n2. Machine type: `C4D` series. OS: Ubuntu, not the Debian default.\n3. The project already has IAP TCP forwarding and the firewall rules. If SSH\n is refused, check the project inherited them; see the SRE space's IAP page.\n\nSet the VM up:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\ngit clone https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\ncurl -sSL https://install.python-poetry.org | python3 -\npoetry install\n```\n\nDocker is an alternative to poetry on the VM, and avoids the Python version\ndance:\n\n```console\ndocker build . --tag syncstorage-loadtest:local\ndocker run --rm -e SERVER_URL -e OAUTH_PRIVATE_KEY_FILE \\\n -e LARGE_PAYLOAD_PROB -e OFFLOAD_COLLECTIONS \\\n -v /path/to/load_test.pem:/keys/load_test.pem:ro \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\"\n```\n\n### Authentication\n\nThree modes, in `storage/auth.py`. For a dev run use self-signed JWTs: no FxA\naccounts to create or clean up, and the token lifetime is yours to set.\n\n```console\npoetry run ./generate-keys.sh\n```\n\nThat writes `load_test.pem` (private, stays on the VM) and `jwk.json`\n(public). The dev tokenserver has to trust that public key, which means\nlanding the JWK in the tokenserver FxA JWK configuration in\n`webservices-infra` under `sync/k8s/`. That is a PR plus an ArgoCD sync, so do\nit before you book time for the run.\n\nThen:\n\n```console\nSERVER_URL=\"https://\" \\\n OAUTH_PRIVATE_KEY_FILE=./load_test.pem \\\n LARGE_PAYLOAD_PROB=1.0 \\\n OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\n```\n\nDirect-access mode (`SERVER_URL` with a `#secret` fragment) bypasses\ntokenserver entirely and is what the local rig uses. It needs the deployed\nmaster secret, so do not use it against a shared environment.\n\n### Server side prerequisites on dev\n\nThe pipeline being provisioned is not the same as offload being on. Offload\ndoes nothing until both of these are set on syncserver, and expanded payloads\ndo nothing until the limits are raised:\n\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET`\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS`\n- `SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES` and the\n `MAX_POST_BYTES` / `MAX_REQUEST_BYTES` / `MAX_TOTAL_BYTES` that go with it\n\nAll four are Helm values in `webservices-infra` under `sync/k8s/sync/`. Raise\nany front-end proxy body-size limit to match, or requests get a 413 before\nthey reach the server.\n\nConfirm what actually took effect by reading it back from the server rather\nthan from the values file:\n\n```console\ncurl -s https:///1.5//info/configuration | jq .\n```\n\nThe load test sizes its batches from exactly this response, so if the limits\nare not in there, the run is not testing expanded payloads no matter what the\nenvironment variables say.\n\n### Verifying a dev run\n\nEverything the local report script checks, plus the parts that only exist on\nGCP:\n\n| What | Where |\n| --- | --- |\n| Reconciler counters (`payload_reconciler.*`) | The metrics pipeline, same as any other service metric |\n| Oldest unacked message age on `payload-link-reconciler-sub` | Pub/Sub console. The most direct measure of finalize latency; should sit in minutes |\n| Anything at all in `payload-link-changes-dlq` | Pub/Sub console. A message only lands there after five failed deliveries, so it always needs a human |\n| Dataflow job lag, autoscaling, worker errors | Dataflow console |\n| Spanner CPU and change-stream storage cost | Spanner console on `moz-fx-sync-nonprod-904c` |\n| Object count and finalize state in the bucket | GCS console, or `gcloud storage ls -L` |\n\nChange-stream storage cost is worth capturing deliberately. It has not been\nquantified, and the plan for prod is to measure it before enabling offload\ntraffic, so a dev run is the cheapest place to get a number.\n\n---\n\n## What to record\n\nStandard long run is 60 to 65 minutes. Log results to the shared documents\nrather than only in a ticket:\n\n- Syncstorage molotov runs: \"Sync Load Test Run Document\"\n- Tokenserver locust runs: the tokenserver load test results document\n\nCapture at minimum: the git SHA under test, molotov arguments, the effective\n`/info/configuration`, the server-side offload collection list, molotov's\nown summary (request counts, failures, RPS), and the reconciler counters plus\nGCS finalize state from the report.\n\n---\n\n## Tokenserver load tests (locust)\n\nSeparate tool, separate target, unrelated to payload offload. Locust rather\nthan molotov, and it has a web UI.\n\n```console\ncd tools/tokenserver/loadtests\npoetry install\npoetry run ./generate-keys.sh\nOAUTH_PRIVATE_KEY_FILE=./load_test.pem poetry run locust # UI on :8089\n```\n\nRunning on a VM, tunnel the UI out over IAP:\n\n```console\ngcloud compute ssh --zone \"\" \"\" --project \"\" \\\n --tunnel-through-iap -- -L 8090:localhost:8089\n```\n\nThen open `localhost:8090`. See `tools/tokenserver/loadtests/README.md` for\nthe rest.\n\n---\n\n## Gaps worth knowing about\n\n- **GCS time can be separated from request time for the collection routes.**\n `storage.gcs.payload.batch` is the wall time one `get_collection` or\n `post_collection` request spent on GCS across all of its objects, tagged with\n `handler`, and the request middleware emits `request.duration` tagged by\n route and method. Dividing `rate(storage.gcs.payload.batch_sum{handler=\"post_collection\"})`\n by the `request.duration` `rate(_count)` for `/1.5/{uid}/storage/{collection}`\n with `method=\"POST\"` (and `get_collection` against `GET` likewise) gives the\n GCS share of an average collection write or read. It is still rough, because\n the denominator counts requests whose payloads stayed inline, under the\n offload threshold. The single-BSO routes (`/{bso}` GET and PUT) emit only\n the per-object `.upload` / `.download` timings, which carry no route tag, so\n they cannot be attributed the same way; measure those with a separate load\n shape. Either way a STOR-629 run can say \"GCS accounted for about N ms of a\n collection request\" rather than only \"the handler got slower\".\n- **No per-collection metric dimension**, so a mixed run cannot separate\n offloaded from inline request latency from the server side either. Working\n around it means either running the two shapes separately and comparing, or\n adding a collection tag.\n","isInternal":false,"tokens":8417,"sizeBytes":33691},{"name":"make_hawk_token.md","path":"docs/src/tools/make_hawk_token.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/make_hawk_token.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Documentation for `make_hawk_token.py`\n\n## Summary\n\n`make_hawk_token.py` is a utility script for generating a [Hawk](https://github.com/mozilla/hawk) authentication token and header for use when testing the syncstorage-rs API. It derives a token and signing key from a shared secret and produces either a raw token tuple or a ready-to-use `Authorization: Hawk ...` header.\n\nThe script lives in [`tools/hawk/`](https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/hawk).\n\n---\n\n## Setup\n\n### Prerequisites\n\n- Python ≥ 3.12\n- [Poetry](https://python-poetry.org/docs/#installation) installed\n\n### Install dependencies\n\n```shell\ncd tools/hawk\npoetry install\n```\n\n---\n\n## Usage\n\n```shell\npoetry run python make_hawk_token.py [options]\n```\n\nBy default (no arguments), a token is generated for `http://localhost:8000/1.5/1/storage/col2/` using a built-in test secret.\n\n### Options\n\n| Option | Description | Default |\n|---|---|---|\n| `--uri` | Request URI path | `/1.5/1/storage/col2/` |\n| `--secret` | Shared `SYNC_MASTER_SECRET` value | *(built-in test value)* |\n| `--method` | HTTP method for the request | `GET` |\n| `--as_header` | Print a ready-to-use `Authorization: Hawk ...` header | *(off)* |\n\nUse `-h` for the full option list.\n\n---\n\n## Examples\n\n### Generate a token for a GET request\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET\n```\n\n### Generate a ready-to-use Authorization header\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nOutput:\n```\nAuthorization: Hawk id=\"...\", ts=\"...\", nonce=\"...\", mac=\"...\"\n```\n\n### Non-GET methods\n\nFor methods other than GET (e.g. PUT, POST, DELETE), pass `--method`:\n\n```shell\npoetry run python make_hawk_token.py \\\n --method PUT \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nSee [`tools/examples/put.bash`](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for a full curl example using this output.\n\n---\n\n## Notes\n\n- The `--secret` value must match the `SYNC_MASTER_SECRET` configured in the running syncstorage-rs instance.\n- The generated token is scoped to the exact URI and method — a token generated for `GET /foo` is not valid for `PUT /foo`.\n","isInternal":false,"tokens":626,"sizeBytes":2357},{"name":"payload_link_reconciler.md","path":"docs/src/tools/payload_link_reconciler.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/payload_link_reconciler.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# GCS Payload Reconciliation Pipeline\n\n## Summary\n\nWhen syncserver offloads a large BSO payload to Google Cloud Storage\n(`syncserver/src/web/payload_offload.rs`), the GCS object is created\nwith custom metadata `committed=false` and `customTime=now`, and the\nBSO row's `payload_link` column points at it. A separate pipeline must:\n\n- **Finalize** newly-committed objects — flip metadata to\n `committed=true` and pin `customTime` to its maximum value\n (`2200-12-31T23:59:59Z`) so the bucket's lifecycle policy (see\n below) cannot reclaim them.\n- **Garbage-collect orphans** — delete GCS objects whose row's\n `payload_link` was replaced (UPDATE) or removed (DELETE, including\n Spanner row-deletion-policy TTL deletes).\n\nAn object whose write never reached a Spanner commit never receives a\nfinalize, and there are two shapes of that. If the database transaction\nfails or rolls back, syncserver issues an inline best-effort **delete**\nof the objects it uploaded for that request and ignores the result. If an\nupload itself fails, the request is abandoned fail-fast and objects\nalready uploaded for it are not cleaned up inline at all. Either way an\nobject can be left stranded at `committed=false` with its upload-time\n`customTime`.\n\nAnything that slips through is reaped by a GCS **lifecycle policy**\n(`bucket.tf` in `webservices-infra/sync`) that deletes objects whose\n`customTime` is more than **30 days** old. Flipping `customTime` to the\nmax sentinel is what protects committed objects from that policy:\n`daysSinceCustomTime` goes permanently negative once finalized, so the\npolicy cannot touch them regardless of object age.\n\nThe 30 day window is the safety margin for the whole asynchronous arm. It\nhas to comfortably exceed the worst case time from upload to finalize,\nwhich is set by the cronjob cadence plus any Pub/Sub or Dataflow backlog.\nAt a five minute cadence the margin is four orders of magnitude, so the\nnumber only becomes interesting if the pipeline is stopped for weeks.\n\nThis document covers that pipeline. It consumes the\n`payload_link_changes` Spanner change stream defined in\n`syncstorage-spanner/src/schema.ddl`.\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n stream[(\"Spanner change stream\n payload_link_changes\n OLD_AND_NEW_VALUES, 7d retention\")]\n\n subgraph dataflow[\"Custom Dataflow flex template\"]\n filter{\"payload_link NULL\n on both sides?\"}\n end\n\n drop([\"dropped\"])\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n\n subgraph reconciler[\"Reconciler cronjob\"]\n route{\"which side\n carries a link?\"}\n finalize[\"finalize the object\n committed=true\n customTime=MAX\"]\n delete[\"delete the object\"]\n end\n\n stream -->|DataChangeRecord| filter\n filter -->|yes| drop\n filter -->|\"no, publish as JSON\"| topic\n topic -->|pull subscription| route\n topic -.->|\"after 5 failed deliveries\"| dlq\n route -->|new link| finalize\n route -->|\"old link, replaced or removed\"| delete\n```\n\nBoth reconciler actions are idempotent, so a redelivered record is\nharmless.\n\n## Components\n\n### 1. Spanner change stream — `payload_link_changes`\n\nDefined in `syncstorage-spanner/src/schema.ddl`:\n\n```sql\nCREATE CHANGE STREAM payload_link_changes\n FOR bsos(payload_link), batch_bsos(payload_link)\n OPTIONS (\n retention_period = '7d',\n value_capture_type = 'OLD_AND_NEW_VALUES'\n );\n```\n\nColumn-scoped: an UPDATE that does not touch `payload_link` produces no\nrecord. INSERTs and DELETEs always produce a record, even when\n`payload_link` is NULL — those are dropped at the next stage.\n\nThe Spanner DDL is not auto-applied; run `gcloud spanner databases ddl\nupdate` against the target database after merging.\n\n### 2. Custom Dataflow flex template — `tools/payload-link-dataflow/`\n\nA standalone Apache Beam pipeline (Java, Beam 2.60.0) that:\n\n1. Reads `payload_link_changes` via `SpannerIO.readChangeStream()`.\n2. Applies a `Filter.by(isPayloadLinkActionable)` step that drops\n records whose every mod has `payload_link` NULL on both sides.\n Malformed records pass through so the reconciler / DLQ surfaces\n them — not the filter.\n3. Serializes each surviving `DataChangeRecord` to JSON and publishes\n to a Pub/Sub topic.\n\nThe pipeline is **not** vendored from\n[`GoogleCloudPlatform/DataflowTemplates`](https://github.com/GoogleCloudPlatform/DataflowTemplates).\nWe own a small standalone source tree under `src/`; the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` template is referenced for\nintent comparison via `upstream-customization.patch` (documentation\nonly — not a build input).\n\n**Build / publish** (operator runs from `webservices-infra`):\n\n```bash\ndocker build -t /syncserver-payload-link-dataflow: \\\n tools/payload-link-dataflow\ndocker push /syncserver-payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/syncserver-payload-link-dataflow.json \\\n --image /syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n```\n\n**Launch parameters** (full list in\n`tools/payload-link-dataflow/metadata.json`):\n\n- `spannerProjectId`, `spannerInstanceId`, `spannerDatabase` — the\n syncstorage Spanner database.\n- `spannerMetadataInstanceId`, `spannerMetadataDatabase` — where the\n change-stream connector keeps its partition-state table. Dev uses a\n dedicated database, `syncdb-pldf-meta-dev`, so the connector holds no\n write access to the syncstorage database. Every environment should do\n the same.\n- `spannerMetadataTableName=Metadata_payload_link` names the\n partition-state table itself. Optional in the template, but pin it on\n any streaming launch. Left unset, the connector generates a random\n table name inside `expand()` and bakes it into the pipeline graph, so a\n `--update` or a cancel-then-relaunch comes up against a fresh empty\n table, reads from `Timestamp.now()`, and silently drops every record\n committed between the two jobs. The name only has to be a legal Spanner\n identifier and stable for the life of the environment; the connector\n creates the table on first use.\n See \"Job update and restart continuity\" in\n `tools/payload-link-dataflow/README.md`.\n- `changeStreamName=payload_link_changes`.\n- `spannerDatabaseRole=payload_link_reader` — the fine-grained access\n role the job reads the stream through, created by the DDL in\n `syncstorage-spanner/src/schema.ddl`. If the role is absent the job\n fails at startup rather than falling back to broader access.\n- `pubsubTopic=projects//topics/payload-link-changes`.\n\n**Service account requires:**\n\n- `roles/spanner.databaseUser` on the syncstorage database and on the\n metadata database. The one role covers both reading the change stream\n and maintaining the connector's partition metadata, but since the two\n live in separate databases the grant is needed on each. Note the IAM\n grant targets the `-904c` project, so it is applied out of band; see\n [GCP Infrastructure](payload-offload-infrastructure.md#out-of-band-steps).\n- Membership of the `payload_link_reader` database role, which is what\n actually narrows the job to the change stream. The IAM grant alone\n does not let it read BSO rows.\n- `roles/pubsub.publisher` on the destination topic.\n- `roles/storage.objectAdmin` on the Dataflow job bucket, for `staging/`\n and `tmp/`.\n- `roles/dataflow.worker` at the project level.\n\n### 2b. Dev/E2E Python publisher — `tools/payload-link-dataflow/payload-link-publisher-py/`\n\nPlain Python script (not Beam) that polls the\n`READ_payload_link_changes` TVF via `google.cloud.spanner.Client` and\npublishes the same JSON wire format the Java job produces. Sub-second\nstartup, no JVM. **Dev/E2E only** — used by the compose stack\ndescribed below; **Java remains the prod publisher.**\n\nFollows partition splits: reads `_root`, picks up child-partition\ntokens from `child_partitions_record`s, reads each child from its\nadvertised `start_timestamp`, retires parents once their children are\nannounced, and drops any partition that responds `OUT_OF_RANGE`.\nRequired in practice — the emulator routes DataChangeRecords to child\npartitions immediately, so a `_root`-only reader sees zero DCRs. Full\ndetails in the tool's own README.\n\n### 3. Pub/Sub topic + DLQ\n\nProvisioned from `webservices-infra/sync`:\n\n- Topic: `payload-link-changes`\n- Dead-letter topic: `payload-link-changes-dlq`\n- Pull subscription: `payload-link-reconciler-sub`\n - 60s ack deadline\n - 7d message retention\n - DLQ routing after 5 delivery attempts\n\n### 4. Reconciler — `tools/payload-reconciler/`\n\nPython script with one job: pull messages, perform GCS operations, ack.\nSync-pull drain loop with two deployment modes selected by whether\n`RUN_BUDGET_SECONDS` is set:\n\n- **Cronjob mode** (default deployment): `RUN_BUDGET_SECONDS` set (e.g.\n `240`). The script drains the subscription up to that many seconds\n or until the queue idles, then exits 0. K8s cronjob at ~5 min\n cadence.\n- **Long-running mode**: `RUN_BUDGET_SECONDS` unset. The script polls\n forever, never exiting on idle. Deploy as a K8s Deployment when\n finalize-flip latency below the cronjob cadence matters.\n\n**Per-message handling** (`reconcile_payload_links.py:handle_message_body`):\n\nFor each mod in the change record:\n\n- New `payload_link` present: `blob.patch()` sets\n `metadata.committed = \"true\"` and `customTime = \"2200-12-31T23:59:59Z\"`.\n- Old `payload_link` present and not equal to the new value: `blob.delete()`,\n with one exception. A `batch_bsos` row removed under the batch commit\n transaction tag (`transactionTag` equals `batch_commit`, set by syncstorage in\n the Spanner `commit_batch` path) is skipped\n (`payload_reconciler.batch_commit_skips`). On commit the link moves into the\n permanent `bsos` row in the same transaction, so its object must be kept\n (deleting it was the STOR-657 bug). Any other `batch_bsos` removal is a\n genuine delete: `batch_bsos` has no deletion policy of its own, so TTL expiry\n and `user_collections` deletes reach it as cascade deletes with no such tag,\n and their objects are removed. `bsos` changes and `batch_bsos` overwrites are\n unaffected. See STOR-668.\n\nBoth operations tolerate `404 NotFound` as success, see *Failure\nmodes* below.\n\n**Environment**\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | — | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | — | `payload-link-reconciler-sub` in prod. |\n| `GCS_PAYLOAD_BUCKET` | yes | — | Cross-bucket links abort the message. |\n| `RUN_BUDGET_SECONDS` | no | — | Set (e.g. `240`) → cronjob mode; drain up to N seconds then exit 0. Unset → long-running mode; poll forever, never exit on idle. |\n| `SYNC_STATSD_HOST` | no | localhost | statsd metrics server host. |\n| `SYNC_STATSD_PORT` | no | 8125 | statsd metrics server port. |\n\n**Service account requires:**\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`.\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and the `delete` operation).\n\n**Deployment.** Default is a K8s cronjob (~5 min cadence) with\n`RUN_BUDGET_SECONDS` set. When lower finalize-flip latency matters,\ndeploy as a K8s Deployment without `RUN_BUDGET_SECONDS` to run\nlong-running. The cronjob template is\n`sync/k8s/sync/templates/payload-reconciler-cronjob.yaml` in\nwebservices-infra, gated on `payloadReconciler.enabled`. Note it lives in\nthe `sync` chart rather than `sync-jobs`.\n\nFour settings in that manifest are load bearing, and changing any of them\nbreaks an assumption the script relies on:\n\n- `concurrencyPolicy: Forbid`. A run that overruns its window is never\n joined by a second one. The subscription is the queue, so the next tick\n simply picks up where the last left off. Two concurrent drains would\n race on the same messages, which is survivable given idempotency but\n wastes the ack deadline.\n- `backoffLimit: 0`. No in-window retry. Every operation is idempotent\n and unacked messages come back on their own, so the next scheduled run\n *is* the retry. A backoff would just re-drain the same queue sooner.\n- `runBudgetSeconds` strictly less than `activeDeadlineSeconds`. The\n chart fails the render if this is violated. The budget is what makes\n the drain loop exit cleanly on its own; if the deadline fired first the\n pod would be killed mid-message and the run would always look failed.\n- The command override. The image entrypoint is the syncserver binary, so\n the cronjob invokes\n `python3 /app/tools/payload-reconciler/reconcile_payload_links.py`\n explicitly. Invoking by path is also what puts the script's directory\n on `sys.path`, which is how its `import utils` resolves.\n\nCredentials come from workload identity: the pod runs as the tenant GKE\nservice account, which holds the two roles listed above. There is no key\nfile and no secret mounted.\n\n---\n\n## Wire format\n\nEach surviving change record reaches the reconciler as a JSON Pub/Sub\nmessage:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-30T00:00:00.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"transactionTag\": \"\",\n \"isSystemTransaction\": false,\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nMod fields (`keys`, `oldValues`, `newValues`) carry **JSON strings**\nthat the reconciler parses with a second `json.loads`, matching\nSpanner's change-streams wire convention.\n\n`transactionTag` carries the transaction tag Spanner records for the change,\nwhich the reconciler uses to skip batch commit handoffs (see above).\n`isSystemTransaction` is carried too but not currently consumed.\n\n---\n\n## Local e2e compose stack\n\n`make docker_run_reconciliation_e2e_tests` brings up a full-stack\ncompose environment that exercises the entire pipeline against\nemulators:\n\n| Service | Image | Role |\n|---|---|---|\n| `sync-db` | Spanner emulator (existing) | Spanner + change stream |\n| `pubsub-emulator` | `google-cloud-cli:emulators` | Pub/Sub |\n| `fake-gcs` | `fsouza/fake-gcs-server` | GCS |\n| `reconciliation-setup` | one-shot | creates Pub/Sub topic + subscription + bucket |\n| `payload-link-publisher` | Python publisher (default) | polls change stream → publishes to Pub/Sub |\n| `payload-reconciler` | primary image, entrypoint override | drains Pub/Sub → patches/deletes GCS objects |\n| `syncserver` | primary image | offload enabled for all `test_storage.py` collections |\n| `e2e-tests` | primary image | runs `pytest tools/integration_tests/ tools/tokenserver/` |\n\nTwo publisher variants:\n\n- **Python (default)** — `docker-compose.e2e.reconciliation.yaml`.\n Sub-second startup; the day-to-day iteration path.\n- **Java (swap-in)** — layer\n `docker-compose.e2e.reconciliation.java.yaml` on top. Runs the same\n Java flex-template image under Beam's DirectRunner. See\n `tools/payload-link-dataflow/README.md` for the exact invocation.\n Use when you need to reproduce a Java-specific issue.\n\nThe compose stack doubles as regression coverage for `test_storage.py`:\nby opting every collection the storage tests use into GCS offload, each\nBSO write flows through the offload path (upload + `payload_link`\nstorage), and each read flows through `download_payload`. New\nreconciler-specific tests live in\n`tools/integration_tests/test_payload_link_reconciliation.py`, gated\nby a module-level `pytest.mark.skipif` on `GCS_PAYLOAD_BUCKET` so they\nauto-skip in the existing spanner e2e stack (where the env var is\nunset) and un-skip here.\n\n**Emulator fallback.** If a future Spanner emulator upgrade breaks\n`SELECT * FROM READ_payload_link_changes(...)`, the fall-back is the\nJava swap-in overlay above (or running the compose stack against a\nreal dev Spanner instance).\n\n---\n\n## Metrics\n\nBoth halves report. The reconciler emits under the\n`payload_reconciler` statsd namespace, syncserver under\n`storage.gcs.payload`. Between them every GCS operation reports both a\ncount and a latency.\n\n### Reconciler\n\nCounters:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalizes` | Objects flipped to `committed=true` | Tracks offloaded write volume |\n| `orphan_deletes` | Objects deleted because a row stopped pointing at them | Tracks overwrite and delete volume |\n| `gcs_404` `op:finalize` | Finalize target was gone | Low and flat |\n| `gcs_404` `op:delete` | Delete target was already gone | Low and flat, redeliveries are normal |\n| `batch_commit_skips` | A `batch_bsos` removal carrying the `batch_commit` transaction tag was kept rather than deleted | Tracks batch commit volume on offloaded collections |\n| `noop_skips` | A record arrived with nothing to do | Near zero; the Dataflow filter should have dropped it |\n| `errors` `kind:handler` | Handler raised, message left unacked | Zero |\n| `runs` | One per drain, emitted unconditionally | Tracks the cronjob cadence |\n| `messages_processed` | Change records pulled and handled, counted per batch | Tracks pipeline volume |\n| `budget_exhausted` | A run hit `RUN_BUDGET_SECONDS` before the queue idled | Zero |\n\nTimings:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalize_age` | Spanner commit to finalize, in ms | Seconds to minutes. This is the window the 30 day lifecycle policy is sized against |\n| `gcs_op` | One GCS round trip. Tagged `op` (`finalize` or `delete`) and `result` (`success`, `not_found`, `error`) | Low. Separates our own GCS latency from Dataflow and Pub/Sub delay |\n| `drain_duration` | How long one run took | Comfortably under `RUN_BUDGET_SECONDS` |\n\n`runs` is the only unconditional series. Every other metric here fires\nonly on the event it names, so a quiet pipeline emits almost nothing and\n\"idle\" would otherwise be indistinguishable from \"not running\".\n\n### Syncserver offload path\n\nEmitted from `syncserver/src/web/payload_offload.rs`. All three GCS\noperations report latency, each tagged with a `result`.\n\n| Metric | Type | Tags | Meaning |\n|---|---|---|---|\n| `storage.gcs.payload.upload` | timing | `result` | One object uploaded |\n| `storage.gcs.payload.download` | timing | `result` | One object read back, streaming included |\n| `storage.gcs.payload.cleanup.duration` | timing | `result` | The compensating delete after a failed write transaction |\n| `storage.gcs.payload.batch` | timing | `op`, `handler`, `result` | Total GCS time for one request, across all of its BSOs |\n| `storage.gcs.payload.size` | histogram | `op` | Distribution of individual payload sizes, in bytes |\n| `storage.gcs.payload.upload.bytes` | counter | none | Bytes written, for throughput |\n| `storage.gcs.payload.download.bytes` | counter | none | Bytes read back |\n| `storage.gcs.payload.cleanup` | counter | `handler`, `result`, `reason` | Outcome of the compensating delete |\n\nStatsd timers carry their own count, so there is no separate counter for\nthe number of uploads or downloads.\n\n`storage.gcs.payload.batch` is not the sum of the per-object timings. A\nrequest works on up to `gcs_payload_max_concurrency` objects at once, so\nits total lands somewhere between the slowest single object and the sum\nof all of them. It is the delay the client waits through, and it is\nrecorded even when the batch fails part way, hence the `result` tag: an\nabandoned batch stopped early and is not comparable to a complete one.\n\n`storage.gcs.payload.size` is a histogram rather than a tag on the\ntimings, so the aggregator derives percentiles from it. The byte\ncounters give a total and therefore a mean; the tail is the part that\nmatters on a project about raising a size ceiling.\n\nTags on `storage.gcs.payload.cleanup`:\n\n| Tag | Values | Meaning |\n|---|---|---|\n| `handler` | `put_bso`, `post_collection` | Which write handler issued the cleanup |\n| `result` | `success`, `failure` | Whether the object was deleted |\n| `reason` | `invalid_url`, `gcs_error` | Failure only. `invalid_url` means the `gs://` URL did not parse and no delete was attempted; `gcs_error` means GCS rejected it |\n\nThe cleanup counter firing at all means writes are failing after their\npayload reached GCS, so read it against the write error rate rather than\non its own. `result:failure` is the more interesting cut: those objects\nare now leaked until the 30 day lifecycle policy reaps them. A steady\n`reason:invalid_url` is a code bug rather than an operational problem,\nsince the URL is one syncserver just generated. Note that an unparseable\nURL issues no GCS call and so has no `cleanup.duration` alongside it.\n\nWorth alerting on:\n\n- Anything at all in `payload-link-changes-dlq`. A message only lands\n there after five failed deliveries, so it means a record cannot be\n handled and needs a human. Inspect it through\n `payload-link-changes-dlq-sub`. Note the DLQ is terminal: its\n subscription carries no dead-letter policy of its own, so a message\n sits there until the 7 day retention drops it and no further queue\n catches it. That is the same 7 days as the change stream retention,\n so once a dead-lettered record ages out the underlying change is\n unrecoverable from both. Seven days is a response deadline, not a\n margin.\n- `errors` `kind:handler` sustained above zero.\n- `noop_skips` climbing, which means the Dataflow filter regressed and\n the pipeline is paying Pub/Sub for records it discards.\n- `gcs_404` `op:finalize` rising as a share of `finalizes`, which is the\n signal that objects are being reaped before they get finalized.\n- Oldest unacked message age on `payload-link-reconciler-sub`. This is a\n Pub/Sub metric rather than one of ours, and it is the most direct\n measure of finalize latency. It should sit in minutes. The 30 day\n lifecycle window is the deadline it must never approach.\n- `storage.gcs.payload.cleanup` with `result:failure`, which counts\n objects that leaked because the compensating delete did not land.\n- `finalize_age` climbing. It is the direct measure of how long objects\n sit unprotected, and unlike the Pub/Sub metric above it covers the\n whole path including the GCS patch.\n- `budget_exhausted` sustained above zero, meaning runs keep ending on\n the clock rather than on an idle queue, so the drain is not keeping up.\n- `runs` going flat. Every other reconciler metric is conditional, so\n this is the one that separates a quiet pipeline from a stopped one.\n\n---\n\n## Failure modes\n\n| Symptom | Cause | Behaviour |\n|---|---|---|\n| Spike in `payload_reconciler.noop_skips` | Dataflow filter is letting inert records through | Investigate; should be ~0 if the filter works. Records still ack — no harm but extra Pub/Sub cost. |\n| Sustained `payload_reconciler.gcs_404` with `op:finalize` | Lifecycle rule reclaimed the object before the reconciler finalized it, OR the same message redelivered after a successful prior run (at-least-once tax) | Acceptable up to a low background level. Sharp rise = lifecycle window too aggressive vs. cronjob cadence. |\n| Sustained `payload_reconciler.gcs_404` with `op:delete` | Object was already deleted (redelivery or concurrent cleanup) | Acceptable; idempotent by design. |\n| Messages in `payload-link-changes-dlq` | Repeated handler exceptions on the same message after 5 retries (malformed JSON, cross-bucket link, GCS auth failure) | Inspect the DLQ payload; fix and re-publish or discard. The main subscription continues to drain. |\n| `payload_reconciler.errors` with `kind:handler` non-zero | Same as above before reaching DLQ. | Same. |\n| `storage.gcs.payload.cleanup` with `result:failure` | The compensating delete after a failed write transaction did not land | The object is stranded at `committed=false` and the 30 day lifecycle policy is the only thing left to reap it. Occasional failures are tolerable; sustained means the write path is failing and cleanup is failing with it. |\n| Finalize latency jumps after a Dataflow deploy, with a gap in `finalizes` | The job relaunched against a fresh partition-metadata table and resumed from `Timestamp.now()` | Records committed during the gap are lost for good; the change stream's 7 day retention only helps if you notice in time. Check that `spannerMetadataTableName` is pinned and unchanged. |\n\nA `payload_link` pointing at a bucket other than `GCS_PAYLOAD_BUCKET`\nraises `ValueError` and the message is left unacked — it retries up to\nthe DLQ rather than mutating an unrelated bucket. This is a hard guard.\n\n---\n\n## Keeping the reference patch accurate\n\n`tools/payload-link-dataflow/upstream-customization.patch` is\ndocumentation only — not a build input. If the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` pipeline evolves in ways that\nchange the conceptual diff, refresh it (see the README in that\ndirectory). This is a doc refresh, not a code change.\n","isInternal":false,"tokens":6160,"sizeBytes":24946},{"name":"payload-offload-infrastructure.md","path":"docs/src/tools/payload-offload-infrastructure.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/payload-offload-infrastructure.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Payload Offload Infrastructure\n\n## Summary\n\nThis document is a map of the GCP infrastructure behind offloaded BSO\npayloads: which project each resource lives in, which repo defines it,\nand what applies it. It covers where things run rather than how the\npipeline behaves. For the behaviour of the change stream, Dataflow job,\nPub/Sub topics and reconciler, see\n[Payload Link Reconciler](payload_link_reconciler.md).\n\nTracking epic: STOR-372, \"Expand spanner storage beyond 2.5MB\".\n\n## The two-project split\n\nThe single most confusing thing about this setup is that Spanner and\neverything else live in different GCP projects, under different\ngenerations of Mozilla's cloud tooling.\n\n| | Spanner | Everything else |\n| --- | --- | --- |\n| Project (dev) | `moz-fx-sync-nonprod-904c` | `moz-fx-sync-nonprod` |\n| Generation | GCPv1 (legacy) | GCPv2 (MozCloud tenant) |\n| Managed from | `cloudops-infra` | `webservices-infra` + `global-platform-admin` |\n\nThe Spanner instances stayed in GCPv1 on purpose. Migrating the large\nprod instance is not cheap, so they were left in place while the rest of\nthe service moved to MozCloud. The practical consequences:\n\n1. Terraform in `webservices-infra` cannot configure the Spanner\n instances or databases. Those changes need `cloudops-infra`, which\n means an extra PR and a MozCloud engineer to apply it.\n2. The webservices Terraform admin service account has no\n `setIamPolicy` rights on `-904c`, so IAM grants that target the\n Spanner database have to be applied out of band. See\n [Out-of-band steps](#out-of-band-steps) below.\n\nThe Dataflow job bridges the two projects: it runs in the tenant project\nand reads a change stream in the legacy project.\n\n## Diagram\n\n```mermaid\nflowchart TB\n gha[\"GitHub Actions\n syncstorage-rs\"]\n\n subgraph v1[\"GCPv1: moz-fx-sync-nonprod-904c (cloudops-infra)\"]\n syncdb[\"Spanner syncdb\n instance: sync\n database: syncdb-dev\n change stream: payload_link_changes\n access role: payload_link_reader\"]\n metadb[\"Spanner Dataflow metadata\n instance: sync\n database: syncdb-pldf-meta-dev\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (global-platform-admin)\"]\n gke[\"Shared GKE cluster\n syncserver, tokenserver,\n payload-reconciler cronjob\"]\n end\n\n subgraph tenant[\"GCPv2 tenant: moz-fx-sync-nonprod (webservices-infra)\"]\n df[\"Dataflow flex template job\n sync-nonprod-dev-payload-link-dataflow\n dedicated VPC and firewall\"]\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n payloads[(\"GCS\n sync-nonprod-dev-syncstorage-payloads\")]\n jobbucket[(\"GCS\n sync-nonprod-dev-payload-link-dataflow\n template spec, staging, tmp\")]\n end\n\n gha -.->|\"publish template spec, via WIF\"| jobbucket\n jobbucket -.->|template spec| df\n syncdb -->|change stream read| df\n df -->|partition state| metadb\n df -->|publish| topic\n topic -->|pull subscription| gke\n topic -.->|\"after 5 failed deliveries\"| dlq\n gke -->|finalize and delete objects| payloads\n gke -->|upload and download payloads| payloads\n gke -->|read and write BSO rows| syncdb\n\n```\n\nNote that the workloads do not run in the tenant project. Sync is\n`risk_level: high`, so its pods run on the shared high risk GKE cluster\nin `moz-fx-webservices-high-nonpro` (nonprod) and\n`moz-fx-webservices-high-prod` (prod). The tenant project holds the\ndata plane resources: buckets, Pub/Sub, secrets, service accounts, DNS,\nand the Dataflow job. Pods reach tenant resources through workload\nidentity on the tenant GKE service account.\n\n## Where each piece is defined\n\n| Resource | Defined in | Repo |\n| --- | --- | --- |\n| Tenant definition (envs, charts, risk rating, project ids) | `tenants/sync.yaml` | global-platform-admin |\n| Tenant GCP projects, enabled APIs, folders, DNS, GAR | `projects/tf/webservices/locals.tf` | global-platform-admin |\n| Shared GKE, GCLB, logging, monitoring | `webservices-high/tf/{nonprod,prod}` | global-platform-admin |\n| GCS payload bucket and lifecycle rule | `sync/tf/dev/bucket.tf` | webservices-infra |\n| Dataflow job, network, firewall, job bucket | `sync/tf/dev/dataflow.tf` | webservices-infra |\n| Dataflow service account and IAM | `sync/tf/dev/dataflow_iam.tf` | webservices-infra |\n| Pub/Sub topics, subscriptions, DLQ routing | `sync/tf/dev/pubsub.tf` | webservices-infra |\n| Spanner instance, database, DDL, database IAM | `projects/sync` | cloudops-infra |\n| Change stream and access role DDL source of truth | `syncstorage-spanner/src/schema.ddl` | syncstorage-rs |\n| Dataflow flex template image and metadata | `tools/payload-link-dataflow/` | syncstorage-rs |\n| Image build and template spec publish | `.github/workflows/mozcloud-publish.yaml` | syncstorage-rs |\n| Helm charts for sync, tokenserver, sync-jobs, sync-test | `sync/k8s/` | webservices-infra |\n\n## Repos and how each is applied\n\n| Repo | Scope | Automation |\n| --- | --- | --- |\n| [global-platform-admin](https://github.com/mozilla/global-platform-admin) | Cross tenant platform: projects, APIs, shared clusters, tenant definitions | Spacelift. The `admin-webservices-projects` stack tracks `main` with autodeploy, so a merge plans and applies without a manual step. |\n| [webservices-infra](https://github.com/mozilla/webservices-infra) | Per tenant resources for the webservices function | Atlantis, autodiscovered per directory. Plan runs on the PR, then `atlantis apply` in a comment. |\n| [cloudops-infra](https://github.com/mozilla-services/cloudops-infra) | Legacy GCPv1 resources, including sync Spanner | Atlantis, applied by a MozCloud engineer |\n| [syncstorage-rs](https://github.com/mozilla-services/syncstorage-rs) | Application code, Spanner DDL, Dataflow template, Helm chart values | GitHub Actions builds and publishes images; ArgoCD deploys |\n\nTerraform state for both global-platform-admin's `projects/tf/webservices`\nand webservices-infra's `sync/tf/*` lives in the same bucket,\n`moz-fx-webservices-terraform-state-global`, accessed by impersonating\n`tf-webservices@moz-fx-websvc-terraform-admin.iam.gserviceaccount.com`.\nThe per environment state is at prefix `projects/sync/`. The project\nlevel state is at `projects/projects/global`, and\n`sync/tf/dev/data.tf` reads it as a remote state data source to resolve\nproject ids and numbers. That prefix is load bearing, do not rename it\nwithout migrating the state object.\n\n## Concrete dev resources\n\nNames are derived from `${application}-${realm}-${environment}` in\n`sync/tf/dev/locals.tf`.\n\n| Thing | Value |\n| --- | --- |\n| Tenant project | `moz-fx-sync-nonprod`, number `960020799362` |\n| Spanner project | `moz-fx-sync-nonprod-904c` |\n| Spanner instance and database | `sync` / `syncdb-dev` |\n| Dataflow metadata database | `sync` / `syncdb-pldf-meta-dev` |\n| Dataflow metadata table | `Metadata_payload_link`, pinned via `spannerMetadataTableName` |\n| Change stream | `payload_link_changes` |\n| Spanner database role | `payload_link_reader` |\n| Payload bucket | `sync-nonprod-dev-syncstorage-payloads`, us-west1 |\n| Dataflow job bucket | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow job | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow service account | `sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Template publisher service account | `sync-nonprod-dev-tmpl-pub@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Dataflow VPC | `sync-nonprod-dev-dataflow`, internal ingress on tcp 12345-12346 |\n| Pub/Sub topic | `payload-link-changes` |\n| Pub/Sub DLQ | `payload-link-changes-dlq`, 5 delivery attempts |\n| Reconciler subscription | `payload-link-reconciler-sub`, 60s ack, 7d retention |\n| DLQ inspection subscription | `payload-link-changes-dlq-sub`, 7d retention |\n| Flex template image | `us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow` |\n| Template spec | `gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json` |\n\nOnly dev is built out today.\n\n## Standing up stage or prod\n\nEverything below is per environment. Names derive from\n`${application}-${realm}-${environment}`, so copying the dev Terraform\ninto the target environment directory produces the right names without\nediting them.\n\n1. **Terraform.** Copy `bucket.tf`, `pubsub.tf`, `dataflow.tf` and\n `dataflow_iam.tf` from `sync/tf/dev/` into the target environment\n directory in webservices-infra.\n\n2. **A dedicated Spanner metadata database.** The change stream\n connector keeps its partition state in a Spanner database of its own.\n Each environment should get a separate database, for example\n `syncdb-pldf-meta-stage`, so the connector holds no write access\n to the syncstorage database. This is a cloudops-infra change.\n\n Pin the table inside it as well, with\n `spannerMetadataTableName = \"Metadata_payload_link\"` on the job.\n Dev uses that name; reusing it per environment is fine, since the\n databases are already separate. The parameter is optional and the\n consequence of skipping it is silent, so it is easy to miss: see\n step 6.\n\n3. **DDL.** Apply the change stream and the `payload_link_reader` role to\n the target database. See [Out-of-band steps](#out-of-band-steps).\n\n4. **Cross-project IAM.** A `-904c` admin grants the new environment's\n Dataflow service account `roles/spanner.databaseUser` on the\n syncstorage database and on the metadata database.\n\n5. **Template spec.** Add a `write-payload-link-dataflow-spec-` job\n to `.github/workflows/mozcloud-publish.yaml`, and make sure the\n environment's `-tmpl-pub` service account exists first. The spec has\n to be in the bucket before the Terraform job resource is applied.\n\n6. **Job lifecycle.** Keep `on_delete = \"cancel\"`. This is not a\n dev-only shortcut: the SpannerIO change stream connector does not\n support draining at all, so `\"drain\"` is never the right setting for\n this pipeline. See\n [Draining a change streams pipeline](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining).\n The change stream's 7 day retention is what covers the gap while a\n replacement job comes up.\n\n Cancel is only safe to rely on because the metadata table name is\n pinned. The connector generates that name from a random UUID at\n graph-construction time, so an unpinned job that is cancelled and\n relaunched, or updated in place, comes up against an empty table and\n resumes from `Timestamp.now()` instead of the previous watermarks.\n Nothing errors; the records committed in between are simply never\n published, and the reconciler never learns those objects need\n finalizing or deleting. Set the name once and do not change it.\n\n7. **Reconciler cronjob.** Enable `payloadReconciler` in the\n environment's values file in `sync/k8s/sync/`. The chart requires\n `runBudgetSeconds` to be less than `activeDeadlineSeconds`.\n\n8. **Measure before enabling on prod.** The change stream costs Spanner\n storage that has not been quantified. Turning the stream on ahead of\n any offload traffic is the cheap way to find out.\n\nEnabled APIs need no change. `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so `moz-fx-sync-prod` already has them.\n\nNote that provisioning the pipeline is safe on its own. Offload does\nnothing until `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET` and\n`SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` are both set on\nsyncserver, so an environment can carry the whole pipeline with zero\ntraffic through it while it is verified.\n\n## Enabled GCP APIs\n\nThe tenant project's enabled APIs are declared in\n`projects/tf/webservices/locals.tf` under the `sync` entry. Enabling an\nAPI there is a PR to global-platform-admin, since tenant teams do not\nhold the IAM to enable services directly.\n\nCurrent list for the sync tenant:\n\n- `spanner.googleapis.com`\n- `secretmanager.googleapis.com`\n- `dataflow.googleapis.com`\n- `datapipelines.googleapis.com`\n- `cloudscheduler.googleapis.com`\n\nThe last three were added in\n[global-platform-admin PR\n#7031](https://github.com/mozilla/global-platform-admin/pull/7031) for\nthis work, tracked by STOR-656 and SREIN-1633.\n`compute.googleapis.com` is enabled by default for every tenant project\nby the shared `google_project` module, which is what Dataflow workers\nneed.\n\nTwo caveats on that list. First, `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so those three APIs are enabled in `moz-fx-sync-prod`\nas well as `moz-fx-sync-nonprod`. There is no per realm override.\nSecond, the current implementation launches\n`google_dataflow_flex_template_job` directly, so nothing in\n`webservices-infra/sync` references Data Pipelines or Cloud Scheduler\nyet. Those two were requested up front in case the job moves to a\nscheduled Data Pipelines definition. They are enabled but unused.\n\n### Security assessment\n\nSync as a whole stays at `risk_level: high` with\n`risk_rating: {confidentiality: 2, integrity: 1, availability: 4}` in\n`tenants/sync.yaml`. The offload itself is low risk.\n\nBSO payloads are encrypted client side with the user's Sync key before\nthey reach syncserver, so GCS holds ciphertext the server cannot\ndecrypt. The offload moves the same class of opaque blob that Spanner\nalready holds, rather than creating a new plaintext exposure. Object\nkeys and metadata do carry identifiers such as fxa_uid, collection and\nbso_id, which is metadata rather than content.\n\nEnabling a service grants no access on its own and creates no\nidentities, so IAM still gates every call. The one item worth care is\nDataflow: a principal with `roles/dataflow.developer` plus `actAs` on\nthe worker service account can run arbitrary pipeline code on GCE VMs\ninside the tenant VPC with that account's permissions. That is why the\njob runs as a dedicated per environment service account holding only\n`roles/dataflow.worker`, `roles/pubsub.publisher` on one topic,\n`roles/storage.objectAdmin` on the job bucket, and\n`roles/spanner.databaseUser` on one database, rather than the Compute\nEngine default account which carries `roles/editor`.\n\nSpanner access is narrowed a second time inside the database. The job\nruns under the `payload_link_reader` database role, which is granted\n`SELECT` on the one change stream and `EXECUTE` on its read function and\nnothing else, so the IAM grant alone does not let the job read BSO rows.\n\n## Out-of-band steps\n\nThree things are not managed by Terraform and have to be done by hand in\nthe right order. Getting this wrong is the usual reason a fresh\nenvironment fails to come up.\n\n1. Apply the change stream DDL to the Spanner database. The DDL in\n `syncstorage-spanner/src/schema.ddl` is not auto-applied. Run\n `gcloud spanner databases ddl update` against the target database\n after merging.\n\n That same DDL block also creates the fine-grained access role the\n Dataflow job reads through:\n\n ```sql\n CREATE ROLE payload_link_reader;\n GRANT SELECT ON CHANGE STREAM payload_link_changes TO ROLE payload_link_reader;\n GRANT EXECUTE ON TABLE FUNCTION READ_payload_link_changes TO ROLE payload_link_reader;\n ```\n\n The job is launched with `spannerDatabaseRole = \"payload_link_reader\"`\n (see `sync/tf/dev/dataflow.tf`), so if the role does not exist the job\n fails at startup rather than falling back to broader access. Apply the\n role and its grants in the same DDL update as the change stream.\n\n2. Grant the Dataflow service account `roles/spanner.databaseUser` on\n the Spanner database. This targets `-904c`, where the webservices\n Terraform admin account has no `setIamPolicy` rights, so a `-904c`\n admin runs it. Tracked by STOR-655.\n\n ```console\n gcloud spanner databases add-iam-policy-binding syncdb-dev \\\n --instance=sync --project=moz-fx-sync-nonprod-904c \\\n --role=roles/spanner.databaseUser \\\n --member=serviceAccount:sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com\n ```\n\n3. Publish the flex template spec into the job bucket before the\n Terraform job resource is applied, or the job has nothing to launch.\n The `write-payload-link-dataflow-spec-dev` job in\n `.github/workflows/mozcloud-publish.yaml` does this on publish.\n\n CI does not use the pipeline service account for this. A separate\n `-tmpl-pub` account exists purely to upload the spec, and the\n syncstorage-rs GitHub Actions workflow impersonates it through\n workload identity federation. Its bucket grant carries an IAM\n condition restricting writes to the `templates/` prefix, so CI cannot\n touch the runtime `staging/` or `tmp/` paths, and it holds nothing on\n Spanner or Pub/Sub. Both accounts are defined in\n `sync/tf/dev/dataflow_iam.tf`.\n\n To publish manually:\n\n ```console\n gcloud dataflow flex-template build \\\n gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json \\\n --image us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n ```\n\n## Consoles and dashboards\n\nSubstitute the project for other environments.\n\n- [Enabled APIs](https://console.cloud.google.com/apis/dashboard?project=moz-fx-sync-nonprod)\n- [Dataflow jobs](https://console.cloud.google.com/dataflow/jobs?project=moz-fx-sync-nonprod)\n- [Pub/Sub topics](https://console.cloud.google.com/cloudpubsub/topic/list?project=moz-fx-sync-nonprod)\n- [GCS buckets](https://console.cloud.google.com/storage/browser?project=moz-fx-sync-nonprod)\n- [Service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts?project=moz-fx-sync-nonprod)\n- [Spanner databases](https://console.cloud.google.com/spanner/instances/sync/databases?project=moz-fx-sync-nonprod-904c)\n\nThe tenant project Terraform is applied by the `admin-webservices-projects`\nstack in Spacelift.\n\n## Reference\n\nMozilla internal documents, not linked here:\n\n- Design document: \"Sync Spanner 10MB Payloads\"\n- \"Syncstorage Payload Expansion Project Plan\"\n- Confluence: \"Bootstrapping a tenant\", under the SRE space\n- Confluence: \"MozCloud Customer Documentation\", under the SRE space\n","isInternal":false,"tokens":4576,"sizeBytes":18337},{"name":"payload-offload-load-testing.md","path":"docs/src/tools/payload-offload-load-testing.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/payload-offload-load-testing.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Load Testing Payload Offload\n\nThis provides a guide on running load tests (`tools/syncstorage-loadtest/`) on the\nGCS payload offload path with Molotov in GCP. This focuses on using gcloud cli\ncommands and short scripts as opposed to the UI.\n\nTwo methods:\n\n- **Method A, against dev env.** Real Spanner, real GCS, real change stream and\n reconciler. Payloads capped at whatever dev's limits allow. Use this to test\n the whole pipeline end to end. Note: it requires webservices-infra updates\n should you need to change application limits.\n- **Method B, local syncserver on the VM.** Spanner emulator, real GCS, your own\n limits. No change stream or reconciler. Use this to test expanded payloads\n larger than the server's default ceiling. This allows for easier ad-hoc\n changing of application limits.\n\nBoth use the load tester's **direct access** mode, which creates its own Hawk\ntoken from the master secret and bypasses Tokenserver.\n\n---\n\n## Prerequisites\n\n### Check your gcloud configuration first\n\nIf you have a configuration for the Spanner emulator and it is enabled,\nit has the bad habit of breaking everything in confusing ways. Check:\n\n```console\ngcloud config list\n```\n\nA configuration containing `auth/disable_credentials = true` or\n`api_endpoint_overrides/spanner` sends every request out unauthenticated and\nredirects Spanner calls to localhost. Switch to a normal configuration:\n\n```console\ngcloud config configurations activate default\ngcloud auth login\n```\n\nSwitch back with `gcloud config configurations activate ` when\nyou return to local emulator work.\n\n### Access you need\n\n| For | You need |\n| --- | --- |\n| Creating the VM | `roles/compute.instanceAdmin.v1` on `moz-fx-sync-nonprod`, which `sync/developers` holds. There is no equivalent grant on prod. |\n| Method A | `roles/secretmanager.secretAccessor` on the tenant project, to read the master secret. |\n| Method B | Write access to a GCS bucket you control. |\n\nSync is onboarded to PAM, so anything beyond the standing grants above can be\nrequested just-in-time. List what is available with `gcloud pam entitlements\nsearch`.\n\n---\n\n## Part 1: Create the load generator VM\n\nUse your own project under the org's `developers` folder (get the folder_no in GCP)\nrather than a tenant project.\n\n```console\nPROJ=\n\ngcloud projects create $PROJ --folder=\ngcloud billing accounts list\ngcloud billing projects link $PROJ --billing-account=\ngcloud services enable compute.googleapis.com --project=$PROJ\n```\n\nNew projects have no default VPC, because the org enforces\n`compute.skipDefaultNetworkCreation`. Create one:\n\n```console\ngcloud compute networks create default --subnet-mode=auto --project=$PROJ\n```\n\nFirewall for IAP SSH required. Without this, the SSH in the next step fails\nwhen no firewall rules are set. Org rules forbid opening tcp/22 to `0.0.0.0/0`:\n\n```console\ngcloud compute firewall-rules create allow-iap-ssh \\\n --project=$PROJ --network=default \\\n --direction=INGRESS --action=allow --rules=tcp:22 \\\n --source-ranges=35.235.240.0/20\n```\n\nCreate the instance in **us-west1**, matching where dev runs.\nAlways check region and match to limit latency.\n\n```console\ngcloud compute instances create sync-loadtest \\\n --project=$PROJ \\\n --zone=us-west1-b \\\n --machine-type=n2-standard-8 \\\n --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \\\n --boot-disk-size=200GB\n```\n\n> **Note on Machine family.** New projects get zero quota for newer families, so C4D\n> fails with `Quota 'CPUS_PER_VM_FAMILY' exceeded. Limit: 0.0`. Most families have\n> their own regional quota (`N2_CPUS`, `E2_CPUS`, `C2_CPUS`); only N1 draws on the\n> general `CPUS` pool. Check whichever matches the family you pick, with\n> `gcloud compute regions describe us-west1 --project=$PROJ --flatten=\"quotas[]\"\n> --format=\"table(quotas.metric,quotas.limit,quotas.usage)\"`.\n\nTo Connect via terminal to instance (you'll likely use this command a lot):\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap\n```\n\nInstall sync & load tests on the VM:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\nmkdir -p ~/src && cd ~/src\ngit clone --depth=1 https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\npython3 -m venv venv && source venv/bin/activate\npip install poetry && poetry install\n```\n\n---\n\n## Part 2: Method A - load test against dev\n\n### Fetch the master secret programmatically\n\nDo not copy and paste the secret. It can contain spaces and a paste\nmay result in confusing 401 errors that are just a mismatched token.\n\n```console\ngcloud auth login --no-launch-browser # VM's own service account cannot read it\n\nSECRET=$(gcloud secrets versions access latest --secret=dev-gke-app-secrets \\\n --project=moz-fx-sync-nonprod \\\n | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"SYNC_MASTER_SECRET\"])')\n\nexport SERVER_URL=\"https://dev.sync.nonprod.webservices.mozgcp.net#${SECRET}\"\necho \"${SERVER_URL%%#*}\"\n```\n\nThat last line prints the origin without revealing the secret. If empty,\nstop: an unset `SERVER_URL` results in a fall back to\n`https://token.stage.mozaws.net` and take the OAuth path instead.\n\n### Verify authentication before running\n\n```console\npoetry run python -c \"\nimport asyncio, aiohttp\nfrom storage.client import StorageClient\nasync def m():\n async with aiohttp.ClientSession() as s:\n c = StorageClient(s)\n r, _ = await c.get('/info/collections', (200, 404))\n print(r.status)\nasyncio.run(m())\n\"\n```\n\nUse `/info/collections`, not `/info/configuration`. The latter takes no auth\nextractor at all and returns 200 to anyone, so it cannot tell you whether your\nsecret is right.\n\n### Run\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n`OFFLOAD_COLLECTIONS` must name a collection the server actually offloads.\nCheck `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` in the deployment's\nvalues file.\n\nThen scale up, redirecting output to a file:\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n nohup poetry run molotov --processes 1 --workers 5 --duration 600 -v loadtest.py \\\n > /tmp/lt.log 2>&1 &\n```\n\nRamp gradually. Dev is small, and a sudden jump to high concurrency will hit\nits ceiling rather quickly.\n\n### Verify offload actually occurred\n\n```console\ngcloud storage ls -r \"gs://sync-nonprod-dev-syncstorage-payloads/**\" \\\n --project=moz-fx-sync-nonprod | grep -v '[/:]$' | tail -5\n```\n\nThen check at least one object was finalized by the reconciler:\n\n```console\ngcloud storage objects describe --project=moz-fx-sync-nonprod\n```\n\nLook for `custom_fields.committed: 'true'` and `custom_time` pinned to the far\nfuture. The gap between `creation_time` and `update_time` is your\nend-to-end reconciliation latency, which can be worth noting.\n\n> Note: the metadata key is `custom_fields` in `gcloud storage` output, not\n> `metadata`.\n\n---\n\n## Part 3: Method B, local syncserver with raised limits\n\nUse this when you need payloads larger than the deployed server allows. There\nis no runtime or client-side override: limits are read from the server's\nenvironment at startup, which you can easily change on the fly.\nIt runs on whatever `/info/configuration`\nis set to. Exceeding said server-side limits returns 413.\n\n### Your own bucket\n\nDo not use a shared environment's bucket. Make one you own:\n\n```console\ngcloud storage buckets create gs:// --project=$PROJ --location=us-west1\n```\n\nAdd a lifecycle rule. These objects are never finalized in this setup, so\nnothing else will clean them up. A 20 minute run can write hundreds of GB, so we\ndon't necessarily want these sticking around:\n\n```console\necho '{\"rule\":[{\"action\":{\"type\":\"Delete\"},\"condition\":{\"age\":1}}]}' > /tmp/lc.json\ngcloud storage buckets update gs:// --lifecycle-file=/tmp/lc.json --project=$PROJ\n```\n\n### Give the VM write access\n\nThe default compute service account is scoped to `devstorage.read_only`, so it\ncannot write, regardless of IAM settings. Note that scopes can only change while the instance\nis stopped:\n\n```console\nSA=$(gcloud compute instances describe sync-loadtest --project=$PROJ \\\n --zone=us-west1-b --format=\"value(serviceAccounts[0].email)\")\n\ngcloud storage buckets add-iam-policy-binding gs:// \\\n --member=serviceAccount:$SA --role=roles/storage.objectAdmin --project=$PROJ\n\ngcloud compute instances stop sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud compute instances set-service-account sync-loadtest \\\n --project=$PROJ --zone=us-west1-b --service-account=$SA \\\n --scopes=https://www.googleapis.com/auth/cloud-platform\ngcloud compute instances start sync-loadtest --project=$PROJ --zone=us-west1-b\n```\n\nDisk contents survive the stop.\n\nConfirm the new scope took effect by writing **from the VM**. Run locally this\nwould use your own credentials and prove nothing about the instance service\naccount, and `$PROJ` is a local shell variable that does not exist on the VM:\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap --command \\\n 'echo hello | gcloud storage cp - gs:///write-check.txt && \\\n gcloud storage rm gs:///write-check.txt'\n```\n\nUsing the instance service account this way means no expiring credentials and\nnothing to mount into the container.\n\n### Build the Docker image\n\n```console\nsudo apt install -y docker.io docker-compose-v2 docker-buildx\nsudo usermod -aG docker $USER && newgrp docker\n\ncd ~/src/syncstorage-rs\nDOCKER_BUILDKIT=1 nohup docker build -t app:build . > /tmp/build.log 2>&1 &\n```\n\n> `DOCKER_BUILDKIT=1` is required. The Dockerfile uses BuildKit cache mounts and\n> the legacy builder fails with \"the --mount option requires BuildKit\".\n\nPulling the published image instead is much faster if you have Artifact\nRegistry read, after `gcloud auth configure-docker us-docker.pkg.dev`. Tag it as\n`app:build` either way, because the compose file hardcodes that tag for its\nsetup container.\n\n### Compose override\n\n```console\ncd ~/src/syncstorage-rs\ncat > docker/docker-compose.offload-local.yaml <<'YAML'\nservices:\n syncserver:\n environment:\n SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET: \n SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS: bookmarks\n SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES: \"20971520\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_POST_BYTES: \"26214400\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_REQUEST_BYTES: \"26218496\"\nYAML\n```\n\nNo nginx runs in this stack, therefore the chart's `nginxClientMaxBodySize` (default\n`2568k`) does not apply. It does apply to any deployed environment, though.\nNote that raising syncserver limits without it gets you a 413 from nginx,\nbefore syncserver sees a request.\n\n### Docker compose\n\n```console\ndocker tag app:build syncstorage-rs:latest\n\ndocker compose -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.offload-local.yaml up -d\n\ndocker compose -f docker/docker-compose.spanner.yaml ps\n```\n\nThe compose default image is `syncstorage-rs:latest`, hence the tag. Otherwise\nset `SYNCSTORAGE_RS_IMAGE=app:build` on every compose.\n\nConfirm the limits are valid:\n\n```console\ncurl -s http://localhost:8000/1.5/1/info/configuration | python3 -m json.tool\n```\n\n`max_record_payload_bytes` must show your raised value.\n\n### Run\n\nThe master secret in this compose config is `secret0`:\n\n```console\ncd ~/src/syncstorage-rs/tools/syncstorage-loadtest\nsource venv/bin/activate\nexport SERVER_URL=\"http://localhost:8000#secret0\"\n\nLARGE_PAYLOAD_PROB=1.0 LARGE_PAYLOAD_SIZE=20971520 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\nConfirm the object sizes:\n\n```console\ngcloud storage ls -r --long \"gs:///**\" --project=$PROJ | tail -6\n```\n\n### Sampling memory during a run\n\nMemory can be easily measured here, and is a valuable performance metric.\nUse the below command to sample it to a file (rather than watching a streaming display):\n\n```console\nnohup bash -c 'while true; do date -u +%H:%M:%S | tr \"\\n\" \" \"; docker stats --no-stream --format \"{{.MemUsage}} {{.CPUPerc}}\" docker-syncserver-1; sleep 5; done' > /tmp/mem.log 2>&1 &\n```\n\nThen run the load tests. Then `pkill -f \"docker stats\"` and read `/tmp/mem.log`.\n\n---\n\n## Caveats\n\n**Molotov hides its errors.** A failing `setup_session` is counted and the\ntraceback is discarded, so any auth or setup problem shows as\n`SUCCESSES: 0 | FAILURES: 1` with nothing else.\n\nAlso note `_retry` only checks the HTTP status when the caller passes an\nexpected-statuses tuple. The scenario's `/info/configuration` call passes none,\nso a 503 appears as a JSON decode error rather than \"Response 503\".\n\n**Molotov's display can mangle terminals.** To save yourself the possible headache, always\nredirect to a file for anything but a short test, and if your terminal\ngoes blank or stops echoing, run `reset`.\n\n**Always redirect, never pipe.** Piping molotov's output through `head` or\n`grep` produces basically unhelpful output.\n\n**Co-locating the server and the load tester skews results.** Note that to some degree, Method B\nresults in both compete for the same CPU and memory. Use low concurrency, or alternatively a second VM.\n\n**The emulator is not Spanner.** Method B's write throughput and latency numbers\naren't super helpful and indicative of live spanner performance.\n\n**Fastly NGWAF is the edge, in blocking mode.** Dev sets\n`ngwaf_agent_level = \"block\"`, so sustained 403s partway through a run against a\ndeployed environment are worth checking there before blaming the load tester.\nThe `sync-policy` Cloud Armor resource still exists, but neither backend config\nattaches it, so it is not the cause.\n\n**Check where the Spanner metrics live.** Spanner for the sync tenant is in a\nseparate project, `moz-fx-sync-nonprod-904c` (now named `sync-legacy-nonprod`),\nand `sync/developers` has no access to it, so the console is unavailable. Since\nthe move to `sync-legacy` the metrics come from Grafana's `gcp-v2-nonprod`\ndatasource, not `gcp-v1-nonprod`, which no longer returns them.\n\n---\n\n## Cleanup\n\n```console\ngcloud compute instances delete sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud storage rm -r gs:///** --project=$PROJ\n```\n\nDelete the instance when you are done. An idle load generator bills\nindefinitely!\n","isInternal":false,"tokens":3684,"sizeBytes":14480},{"name":"postgres_purge_ttl.md","path":"docs/src/tools/postgres_purge_ttl.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/postgres_purge_ttl.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Documentation for `purge_ttl.py` (PostgreSQL)\n\n## Summary\n\nThe `purge_ttl.py` script is a utility for purging expired Time-To-Live (TTL) records from a Sync storage PostgreSQL database. It deletes expired entries from the `batches` and/or `bsos` tables, with optional filtering by collection ID and a dry-run mode for verifying query logic without modifying data.\n\n> Spanner no longer uses this script — it expires records natively via its built-in [row deletion (TTL) policies](https://cloud.google.com/spanner/docs/ttl). This utility applies to PostgreSQL deployments only.\n\n---\n\n## Status\n- Runs as a regularly scheduled job (e.g. a cron job) in PostgreSQL (\"enterprise\") deployments.\n- The script is bundled in the main `syncserver-postgres` image and invoked from there; there is no separate utility image.\n\n## Specifics\n\n- **Database**: PostgreSQL (accepts a `postgresql://` or `postgres://` DSN).\n- **Tables**:\n - `batches`: Contains batch entries (child `batch_bsos` rows are removed via the schema's foreign-key cascade).\n - `bsos`: Stores Sync Basic Storage Objects (BSO).\n- **Supported Modes**:\n - `batches`: Purges expired entries in the `batches` table.\n - `bsos`: Purges expired entries in the `bsos` table.\n - `both`: Performs purges on both tables.\n- **Expiry Modes**:\n - `now`: Purges entries with `expiry < CURRENT_TIMESTAMP`.\n - `midnight`: Purges entries with `expiry < DATE_TRUNC('day', CURRENT_TIMESTAMP AT TIME ZONE 'UTC')`.\n\nThe script tracks execution duration and rows affected using StatsD metrics for performance monitoring.\n\n---\n\n## Notes\n\n- Ensure the database role used by the DSN has `DELETE` privileges on the target tables.\n- Use the `--dryrun` option to verify query logic before actual purging.\n- Consider setting up automated monitoring for long-running operations or performance issues.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3 is installed.\n2. **Poetry**: Install [Poetry](https://python-poetry.org/docs/#installation).\n3. **Dependencies**: From `tools/postgres`, run `poetry install`.\n4. **Environment Variables:**\n `SYNC_SYNCSTORAGE__DATABASE_URL`: Database connection URL (e.g., `postgresql://user:pass@host:5432/syncstorage`). Used when `--database_url` is not supplied.\n\n### Usage\n\nRun the script using the following command:\n ```bash\n poetry run python purge_ttl.py [options]\n ```\n\n#### Options\n\n| Option | Description | Default |\n|---------------------------------|----------------------------------------------------------------------------------------|----------------------------------|\n| `-u`, `--database_url` | PostgreSQL DSN (`postgresql://...` or `postgres://...`). Required if the env var is unset. | `SYNC_SYNCSTORAGE__DATABASE_URL` |\n| `--collection_ids`, `--ids` | Comma-separated list of collection IDs to purge. | `[]` |\n| `--mode` | Purge mode: `batches`, `bsos`, or `both`. | `both` |\n| `--expiry_mode` | Expiry mode: `now` (current timestamp) or `midnight` (start of current day, UTC). | `midnight` |\n| `--dryrun` | Perform a dry run without making changes to the database. | `False` |\n\n#### Examples\n\n##### Example 1: Basic Purge\nPurge expired entries from both `batches` and `bsos` tables using the DSN from the environment:\n```bash\n poetry run python purge_ttl.py\n```\n\n##### Example 2: Specify the Database URL\n```bash\n poetry run python purge_ttl.py -u postgresql://user:pass@host:5432/syncstorage\n```\n\n##### Example 3: Filter by Collection IDs\nPurge only for specific collection IDs:\n```bash\n poetry run python purge_ttl.py --collection_ids [123,456,789]\n```\n\n##### Example 4: Purge a Single Table\n```bash\n poetry run python purge_ttl.py --mode bsos\n```\n\n##### Example 5: Perform a Dry Run\nTest the script without making actual changes:\n```bash\n poetry run python purge_ttl.py --dryrun\n```\n\n### Detailed Usage\n\n1. **Connecting to PostgreSQL**:\n - The script builds a SQLAlchemy engine from the supplied DSN. A `postgres://` scheme is normalized to `postgresql://` for newer SQLAlchemy versions.\n\n2. **Purge Modes**:\n - `batches`: Deletes expired entries from the `batches` table.\n - `bsos`: Deletes expired BSOs.\n - `both`: Executes purges on both `batches` and `bsos`.\n\n3. **Expiry Conditions**:\n - `now`: Purge entries that have already expired at the current timestamp.\n - `midnight`: Purge entries that expired at or before the start of the current UTC day.\n\n4. **Query Customization**:\n - An optional collection-ID filter is appended via the `add_conditions` helper using bound parameters.\n\n5. **Performance Monitoring**:\n - Metrics for execution duration and rows affected are logged and sent to StatsD for monitoring.\n\n6. **Dry Run**:\n - Enabling the `--dryrun` flag ensures that the queries are constructed and logged without executing them on the database.\n","isInternal":false,"tokens":1188,"sizeBytes":5270},{"name":"purge_old_records_tokenserver.md","path":"docs/src/tools/purge_old_records_tokenserver.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/docs/src/tools/purge_old_records_tokenserver.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Documentation for `purge_old_records.py`\n\n## Summary\n\nThe `purge_old_records.py` script is an administrative utility for managing obsolete user records in Tokenserver. It removes outdated user records from the database and deletes associated data from storage nodes. This process helps reduce storage overhead, improve database performance, and maintain the health of the Tokenserver system.\n\nObsolete records are those replaced by newer records for the same user or marked for deletion if the user has deleted their account. The script can run in batch mode for periodic cleanup and includes options for dry-run testing and forced purging when nodes are down.\n\n---\n\n## Status\n - Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-purge-old-records-0`\n - See YAML configuration when editing each job.\n - See Kubernetes Engine Workload Panel in [sync-prod](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information. \n\n---\n\n## Specifics\n\n- **Primary Functionality**:\n - Deletes obsolete user records.\n - Issues delete requests to user storage nodes to purge related data.\n- **Optional Administrative Task**:\n - The script complements internal record replacement handled by the backend but is not mandatory for system operation.\n- **Batch Processing**:\n - Operates in loops, processing records in batches of a configurable size.\n- **Grace Period**:\n - Provides a grace period to avoid prematurely deleting recently replaced records.\n- **Dry Run**:\n - Offers a non-destructive mode for testing.\n\n---\n\n## Notes\n- **Regular Use**:\n - Running this script regularly can help maintain system performance and reduce storage overhead.\n- **Concurrency**:\n - When running multiple instances of this script, use the `--max-offset` option to reduce collisions.\n- **Forced Deletion**:\n - Use the `--force` option cautiously, especially if storage nodes are down.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3.7+ is installed.\n2. **Dependencies**:\n - Install required Python packages:\n `pip install requests hawkauthlib backoff`\n\n3. **Configuration**:\n - Set up access to the Tokenserver database.\n - Provide necessary metrics and logging configurations.\n\n### Usage\n\nRun the script using the following command:\n```bash\npython purge_old_records.py [options] secret\n```\n\n\n### Options\n\n| Option | Description | Default |\n|-------------------------|-------------------------------------------------------------------------------------------------|--------------|\n| `--purge-interval` | Interval in seconds to sleep between purging runs. | `3600` |\n| `--grace-period` | Grace period in seconds before deleting records. | `86700` (24h5m) |\n| `--max-per-loop` | Maximum number of items to process in each batch. | `10` |\n| `--max-offset` | Random offset to select records for purging, reducing collisions in concurrent tasks. | `0` |\n| `--max-records` | Maximum number of records to purge before exiting. | `0` (no limit) |\n| `--request-timeout` | Timeout in seconds for delete requests to storage nodes. | `60` |\n| `--oneshot` | Perform a single purge run and exit. | Disabled |\n| `--dryrun` | Test the script without making destructive changes. | Disabled |\n| `--force` | Force purging even if the user's storage node is marked as down. | Disabled |\n| `--override_node` | Specify a node to override for deletions if data is copied. | None |\n| `--range_start` | Start of UID range to process. | None |\n| `--range_end` | End of UID range to process. | None |\n| `--human_logs` | Enable human-readable logs. | Disabled |\n| `--last-seen-staging-table` | BigQuery last-seen staging table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-table`. | None |\n| `--last-seen-table` | BigQuery last-seen final table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-staging-table`. | None |\n\n### Examples\n\n#### Example 1: Basic Purge\nPerform a basic purge of obsolete user records:\n```bash\npython purge_old_records.py secret_key\n```\n#### Example 2: Grace Period and Dry Run\nPurge records with a 48-hour grace period in dry-run mode:\n```bash\npython purge_old_records.py --grace-period 172800 --dryrun secret_key\n```\n\n#### Example 3: Specify Range and Offset\nPurge records within a UID range with a random offset:\n\n```bash\npython purge_old_records.py --range_start uid_start --range_end uid_end --max-offset 50 secret_key\n```\n\n#### Example 4: Force Purge on Downed Nodes\nForce the deletion of data on downed nodes:\n```bash\npython purge_old_records.py --force secret_key\n```\n\n---\n\n## Detailed Usage\n\n1. **Batch Processing**:\n - The script processes records in batches defined by the `--max-per-loop` option.\n - Each batch is fetched from the database using random offsets to avoid overlapping with concurrent runs.\n\n2. **Grace Period**:\n - The grace period ensures that recently replaced records are not prematurely deleted.\n\n3. **Storage Node Cleanup**:\n - For each user, the script sends a delete request to their storage node to remove associated data.\n\n4. **Metrics Tracking**:\n - Tracks operations like user record deletions, service data deletions, and errors using metrics integration.\n\n5. **Error Handling**:\n - Uses exponential backoff to retry failed HTTP requests.\n - Detects loops in batch processing and raises exceptions.\n\n6. **Dry Run Mode**:\n - Simulates deletions without modifying the database or storage nodes, useful for testing.\n\n7. **Account Last-Seen Deletion**:\n - With both `--last-seen-*` options, each record retired by an FxA account deletion first has its account's rows deleted from the BigQuery last-seen tables, just before the record itself.\n - If that job fails or times out, the retired record stays, `replaced_at` is bumped, and it is retried after another grace period. Other records continue normally.\n - The default grace period (24 hours 5 minutes) outlasts the one-day OAuth token lifetime plus the upload delay, so an account's last activity normally reaches BigQuery before purge deletes it.\n\n---\n\n","isInternal":false,"tokens":1445,"sizeBytes":7031},{"name":"env-check.md","path":".claude/commands/env-check.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/env-check.md","title":"Command: /env-check","category":"command-prompt","format":"markdown","content":"Verify all required environment variables are set before running integration tests.\n\nYou are a pre-flight assistant for syncstorage-rs integration tests. Your job is to check that the shell environment is correctly configured so tests will actually run, and surface any missing or suspicious values before the user wastes time on a failing test run.\n\nDo NOT print or log the values of any secrets. Use placeholders like `[set]` or `[not set]` when reporting on secret variables.\n\n## Variables to check\n\n### Required for all integration test runs\n\n| Variable | Expected | Notes |\n|---|---|---|\n| `SYNC_MASTER_SECRET` | any non-empty string | secret — report [set] or [not set] only |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | starts with `mysql://`, `postgresql://`, or `spanner://` | |\n| `SYNC_TOKENSERVER__DATABASE_URL` | starts with `mysql://` or `postgresql://` | |\n| `TOKENSERVER_HOST` | starts with `http://` or `https://` | default: `http://localhost:8000` |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | starts with `http://` or `https://` | default: `http://localhost:6000` for local |\n| `PYTHONPATH` | contains `tools/` path | required for PyO3 and test imports |\n\n### Required only for real FxA e2e tests (`test_e2e.py`)\n\n| Variable | Notes |\n|---|---|\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E` | secret — report [set] or [not set] only |\n\n## Step 1 — Check each variable\n\n```bash\necho \"SYNC_MASTER_SECRET: $([ -n \"$SYNC_MASTER_SECRET\" ] && echo '[set]' || echo '[NOT SET]')\"\necho \"SYNC_SYNCSTORAGE__DATABASE_URL: ${SYNC_SYNCSTORAGE__DATABASE_URL:-[NOT SET]}\"\necho \"SYNC_TOKENSERVER__DATABASE_URL: ${SYNC_TOKENSERVER__DATABASE_URL:-[NOT SET]}\"\necho \"TOKENSERVER_HOST: ${TOKENSERVER_HOST:-[NOT SET - default: http://localhost:8000]}\"\necho \"SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: ${SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL:-[NOT SET - default: http://localhost:6000]}\"\necho \"PYTHONPATH: ${PYTHONPATH:-[NOT SET]}\"\n```\n\n## Step 2 — Validate database URL format\n\nFor `SYNC_SYNCSTORAGE__DATABASE_URL` and `SYNC_TOKENSERVER__DATABASE_URL`:\n- Check the scheme matches a supported backend (`mysql`, `postgresql`, `spanner`)\n- For `postgres://` prefix: note that SQLAlchemy 1.4+ requires `postgresql://` — flag this if present\n- For Spanner: confirm `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` is set if the URL contains `test-project` or `emulator`\n\n## Step 3 — Check PYTHONPATH contains tools/\n\n```bash\npython3 -c \"import tools.integration_tests\" 2>&1\n```\n\nIf this fails, the `PYTHONPATH` is not set correctly for test imports.\n\n## Step 4 — Check server reachability (if TOKENSERVER_HOST is set)\n\n```bash\ncurl -sf \"${TOKENSERVER_HOST:-http://localhost:8000}/__heartbeat__\" && echo \"server: reachable\" || echo \"server: NOT REACHABLE\"\n```\n\n## Output format\n\nReport each variable as **OK**, **WARNING**, or **MISSING**:\n\n| Variable | Status | Detail |\n|---|---|---|\n\nThen list any blockers (MISSING required vars, unreachable server) followed by warnings (format issues, defaults in use).\n\nEnd with **Ready to run tests** or **N issues need to be resolved**.\n","isInternal":false,"tokens":889,"sizeBytes":3351},{"name":"git-history-check.md","path":".claude/commands/git-history-check.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/git-history-check.md","title":"Command: /git-history-check","category":"command-prompt","format":"markdown","content":"Examine git history of files changed in the current branch to identify regressions, re-introduced bugs, or changes that conflict with past fixes.\n\nYou are a senior engineer doing a history-aware code review. Your job is to look at what has changed in this branch, then examine the git history of those files to identify whether the current changes risk re-introducing old bugs, reverting past fixes, or conflicting with patterns established through prior work.\n\n## Step 1 — Get the current diff\n\n```bash\ngit diff main...HEAD\n```\n\n## Step 2 — Get the list of changed files\n\n```bash\ngit diff main...HEAD --name-only\n```\n\n## Step 3 — For each changed file, examine recent commit history\n\n```bash\ngit log --oneline -20 -- \n```\n\n## Step 4 — Inspect relevant commits in full\n\nFor each commit that looks relevant (bug fixes, reverts, security patches, refactors on the same lines), inspect the full diff:\n\n```bash\ngit show \n```\n\n## Step 5 — Look for reverted or fix commits\n\n```bash\ngit log --oneline --all --grep=\"revert\" -- \ngit log --oneline --all --grep=\"fix\" -- \n```\n\nUse your judgment about which historical commits are worth digging into. Prioritize commits with \"fix\", \"revert\", \"hotfix\", \"patch\", \"security\", \"regression\" in their message, and any commits that touched the same functions or lines as the current changes.\n\n## Analysis checklist\n\nFor each changed file, work through the following. Report findings with:\n- **Severity:** High / Medium / Low\n- **Location:** file:line\n- **Issue:** what the historical context reveals\n- **Relevant commit(s):** hash + message\n- **Recommendation:** what to verify or change\n\n### 1. Re-introduced bugs\n- Does the current diff restore code that was previously removed by a bug fix?\n- Are there commits in the history that explicitly fixed logic that the current change modifies or removes?\n\n### 2. Reverted or rolled-back patterns\n- Has this code been reverted before? If so, why — and does the current change repeat the same pattern?\n- Are there \"Revert X\" commits in the history that suggest a prior attempt at this change failed?\n\n### 3. Previously fixed security issues\n- Does the history show security-related fixes on these lines?\n- Does the current change touch or weaken those protections?\n- Pay particular attention to: HAWK/JWT token handling, FxA OAuth validation, gRPC error suppression logic, DB migration constraints.\n\n### 4. Repeated churn\n- Has this file or function been modified many times in a short period? High churn is a signal of instability.\n- Does the current change look like it continues a pattern of patching symptoms rather than fixing the root cause?\n\n### 5. Conflicting intent\n- Do commit messages indicate a deliberate design decision that the current change reverses without explanation?\n- Are there TODOs or FIXMEs introduced by prior commits that the current change should have addressed but didn't?\n\n### 6. Migration or deprecation conflicts\n- Does the history show a migration away from a pattern that the current change re-introduces?\n- Examples: callback → async, legacy auth → FxA OAuth, MySQL-only → multi-backend.\n\n### 7. Test regressions\n- Were tests previously added to cover a bug that the current change might invalidate?\n- Do any historical fix commits include test additions that are now at risk of being bypassed?\n\n## Output format\n\nLead with a summary table:\n\n| Severity | File | Issue | Relevant commit |\n|---|---|---|---|\n\nFollow with per-file detail. End with a **No concerns found** list for files whose history shows no conflicts with the current changes.\n\nIf the git history is shallow or sparse for a file, note that and flag it as low confidence.\n","isInternal":false,"tokens":826,"sizeBytes":3697},{"name":"migration-review.md","path":".claude/commands/migration-review.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/migration-review.md","title":"Command: /migration-review","category":"command-prompt","format":"markdown","content":"Review database migration changes against the repo's published-migration policy.\n\nYou are a database migration reviewer for syncstorage-rs. The repo has a hard rule: **never edit a published migration file**. Only new forward migrations with a corresponding rollback are allowed. Your job is to enforce this and catch violations before merge.\n\n## Migration locations\n\n```\nsyncstorage-mysql/migrations/ # up.sql / down.sql per timestamped dir\nsyncstorage-postgres/migrations/\ntokenserver-mysql/migrations/\ntokenserver-postgres/migrations/\nsyncstorage-spanner/src/schema.ddl # Spanner schema — treated as published\nsyncstorage-spanner/src/*.sql # inline Spanner queries\n```\n\nEach migration directory is named `YYYY-MM-DD-hhmmss_description/` and contains `up.sql` and `down.sql`.\n\n## Step 1 — Find changed migration files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"migrations/|schema\\.ddl|\\.sql\"\n```\n\n## Step 2 — Classify each changed file\n\nFor each changed SQL file:\n\n- **New migration directory** (path did not exist on main) → allowed, proceed to Step 3\n- **Existing migration file modified** (path existed on main and content changed) → **VIOLATION** — flag as a blocker\n- **schema.ddl modified** → **VIOLATION** — flag as a blocker\n- **Inline .sql query file modified** (non-migration) → review for intent, flag if it changes schema\n\nCheck whether a file existed on main:\n```bash\ngit show main:\n```\n\n## Step 3 — Validate new migrations\n\nFor each new migration directory:\n\n1. Confirm both `up.sql` and `down.sql` exist\n2. Confirm `down.sql` is a genuine rollback of `up.sql` (not empty, not a no-op)\n3. Check that the timestamp prefix is newer than all existing migrations in that crate\n4. Check for destructive operations in `up.sql` (`DROP TABLE`, `DROP COLUMN`, `TRUNCATE`) — flag these for explicit review\n5. Check that column types and constraints are consistent with existing schema in the same crate\n\n## Step 4 — Cross-backend consistency\n\nIf migrations exist for both mysql and postgres variants of the same crate, verify the schema changes are semantically equivalent (same columns, types mapped appropriately, same constraints).\n\n## Output format\n\n**Violations (blockers):**\nList any edits to published files with file path and the nature of the change.\n\n**New migrations:**\nFor each new migration: name, what it does, whether down.sql is valid, any destructive ops.\n\n**Warnings:**\nNon-blocking concerns (e.g. missing index, risky destructive op, cross-backend inconsistency).\n\n**Verdict:** Pass / Violations found\n","isInternal":false,"tokens":590,"sizeBytes":2580},{"name":"pr-description.md","path":".claude/commands/pr-description.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/pr-description.md","title":"Command: /pr-description","category":"command-prompt","format":"markdown","content":"Generate a PR title and summary from the current branch diff, following the repo's commit and PR conventions.\n\nYou are a PR description writer for syncstorage-rs. Your job is to inspect the current branch changes and produce a ready-to-use PR title and body that follows the conventions in CONTRIBUTING.md.\n\n## Step 1 — Gather branch context\n\n```bash\ngit diff main...HEAD --name-only\ngit log main...HEAD --oneline\ngit diff main...HEAD --stat\n```\n\n## Step 2 — Read the full diff for context\n\n```bash\ngit diff main...HEAD\n```\n\nFocus on understanding *what changed and why*, not just listing files.\n\n## Step 3 — Classify the change type\n\nBased on changed files, determine the primary `type`:\n\n- `feat` — new functionality added\n- `fix` — bug fix\n- `refactor` — code restructured, no behavior change\n- `test` — test additions or changes only\n- `chore` — build, CI, tooling, dependencies\n- `docs` — documentation only\n- `perf` — performance improvement\n- `style` — formatting, lint fixes only\n\nIf multiple types apply, use the most significant one.\n\n## Step 4 — Determine scope (optional)\n\nIf the change is clearly scoped to one subsystem, add it in parentheses:\n- `tokenserver`, `syncstorage`, `spanner`, `mysql`, `postgres`, `auth`, `migrations`, `ci`, `tools`\n\nExample: `refactor(tokenserver): ...`\n\n## Step 5 — Draft the PR title\n\nRules from CONTRIBUTING.md:\n- Format: `type(scope): subject` or `type: subject`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Keep it under 70 characters\n\n## Step 6 — Draft the PR body\n\n```markdown\n## Summary\n\n<2–4 bullet points covering what changed and why. Lead with the motivating\nproblem or context, then the solution. Skip obvious restatements of the title.>\n\n## Test plan\n\n\n```\n\nPre-fill the test plan based on file types changed:\n\n- Rust source changed → `- [ ] \\`make clippy_\\` passes`, `- [ ] Unit tests pass (\\`make test\\`)`\n- Python source changed → `- [ ] \\`make ruff-lint\\` and \\`make mypy\\` pass`, `- [ ] Integration tests pass (\\`make run_local_e2e_tests\\`)`\n- SQL migrations added → `- [ ] \\`/migration-review\\` passes`, `- [ ] Migration applies cleanly on a fresh DB`\n- `.github/` changed → `- [ ] CI workflow runs successfully on this branch`\n- `CLAUDE.md` or docs changed → `- [ ] Renders correctly on GitHub`\n\n## Step 7 — Output\n\nPrint the final PR title and body, ready to paste into GitHub. Also include the suggested branch name if the current branch name doesn't follow `type/description-STOR-####` or `type/description-####` format.\n\nNote any breaking changes that should be called out explicitly.\n","isInternal":false,"tokens":661,"sizeBytes":2696},{"name":"py-checks.md","path":".claude/commands/py-checks.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/py-checks.md","title":"Command: /py-checks","category":"command-prompt","format":"markdown","content":"Run all Python lint, format, type, and security checks for the tools/ directory.\n\nYou are a Python code quality assistant for the syncstorage-rs tools and integration tests. Your job is to run the full suite of Python checks in the correct order, report findings clearly, and tell the user exactly what needs to be fixed before the PR can merge.\n\nAll checks operate on the `tools/` directory. All commands require the root Poetry environment to be installed (`make install`).\n\n## Step 1 — Check Poetry environment\n\n```bash\npoetry env info\n```\n\nIf no environment is active, tell the user to run `make install` first and stop.\n\n## Step 2 — Format check\n\n```bash\nmake ruff-fmt-chk\n```\n\nIf there are formatting issues, tell the user to run `make ruff-format` to auto-fix, then re-run. Do not proceed past format failures.\n\n## Step 3 — Lint\n\n```bash\nmake ruff-lint\n```\n\nReport each warning/error with file:line and rule code.\n\n## Step 4 — Type checking\n\n```bash\nmake mypy\n```\n\nReport type errors with file:line. Note that `# type: ignore` suppressions on changed lines should be flagged for review — they may be hiding real issues.\n\n## Step 5 — Docstring validation\n\n```bash\nmake pydocstyle\n```\n\nReport missing or malformed docstrings. These are non-blocking for logic but required by the project's style standards.\n\n## Step 6 — Security scan\n\n```bash\nmake bandit\n```\n\nFlag any **HIGH** severity findings as blockers. List **MEDIUM** as warnings.\n\n## Output format\n\nReport each check as **PASS**, **WARNINGS**, or **FAIL**:\n\n| Check | Status | Issues |\n|---|---|---|\n| Format (ruff) | | |\n| Lint (ruff) | | |\n| Types (mypy) | | |\n| Docstrings (pydocstyle) | | |\n| Security (bandit) | | |\n\nList all actionable findings below the table with file:line references.\n\nEnd with **All checks passed** or **N checks need attention** verdict.\n","isInternal":false,"tokens":458,"sizeBytes":1842},{"name":"rust-backend-check.md","path":".claude/commands/rust-backend-check.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/.claude/commands/rust-backend-check.md","title":"Command: /rust-backend-check","category":"command-prompt","format":"markdown","content":"Run Clippy, fmt check, and cargo audit for a specific Rust backend.\n\nYou are a Rust build and lint assistant for the syncstorage-rs workspace. Your job is to run the correct set of checks for a given backend, report findings clearly, and flag anything that needs attention before the code is merged.\n\n## Step 1 — Determine the backend\n\nAsk the user which backend to check if not already specified: `mysql`, `postgres`, or `spanner`. If the current branch name or open files suggest a backend, infer it and confirm.\n\n## Step 2 — Run format check\n\n```bash\ncargo fmt -- --check\n```\n\nReport any files that are not formatted. If there are formatting issues, stop and tell the user to run `cargo fmt` before proceeding.\n\n## Step 3 — Run Clippy for the chosen backend\n\nUse the exact make target:\n\n```bash\nmake clippy_mysql # for mysql\nmake clippy_postgres # for postgres\nmake clippy_spanner # for spanner\n```\n\n## Step 4 — Run release-mode Clippy\n\n```bash\nmake clippy_release_mysql # for mysql\nmake clippy_release_postgres # for postgres\nmake clippy_release_spanner # for spanner\n```\n\nRelease mode catches dead code and issues only visible with optimizations enabled.\n\n## Step 5 — Run cargo audit\n\n```bash\ncargo audit\n```\n\nFlag any advisories with severity **high** or **critical** as blockers. List medium/low advisories as warnings.\n\n## Output format\n\nSummarize findings in this order:\n\n1. **Format** — pass or list of unformatted files\n2. **Clippy (debug)** — pass or list of warnings/errors with file:line\n3. **Clippy (release)** — pass or list of warnings/errors with file:line\n4. **Audit** — pass or list of advisories (severity, crate, advisory ID)\n\nEnd with a clear **Ready to merge** / **Needs attention** verdict.\n","isInternal":false,"tokens":420,"sizeBytes":1760},{"name":"README.md","path":"tools/examples/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/examples/README.md","title":"examples Documentation","category":"plugin-manifest","format":"markdown","content":"# Example scripts for working with syncstorage-rs\n\n* `put.bash` - First `chmod +x put.bash` then `./put.bash` to simulate a PUT.\n","isInternal":false,"tokens":39,"sizeBytes":129},{"name":"README.md","path":"tools/hawk/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/hawk/README.md","title":"hawk Documentation","category":"plugin-manifest","format":"markdown","content":"# Make a Hawk compatible Auth header\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs `make_hawk_token.py` script, you'll need a Python =>3.10 development environment with `Poetry` installed. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `hawk/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/hawk\n # pyenv version install\n $ pyenv install 3.14\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.14 hawk # or whatever project name you like.\n $ pyenv local hawk # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\n`poetry run python make_hawk_token.py`\n\n## Create a Token Header:\n\nYou'll need to pass along your `SYNC_MASTER_SECRET` and the uri you'll be testing in order to generate a valid Hawk Id:\n\n`poetry run python make_hawk_token.py --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`\n\n** For testing against uri's using methods other than GET, you'll need to pass along the `--method` flag to generate your token. Ie, `poetry run python make_hawk_token.py --method PUT --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`. See [examples/put.bash](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for an example of this.\n\nUse `-h` for help.\n\nBy default, with no passed arguments, a default Hawk Id token will be generated mapping to your localhost:8000 with\nthe path `http://localhost:8000/1.5/1/storage/col2/`.","isInternal":false,"tokens":638,"sizeBytes":2587},{"name":"README.md","path":"tools/payload-link-dataflow/payload-link-publisher-py/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/payload-link-dataflow/payload-link-publisher-py/README.md","title":"payload-link-publisher-py Documentation","category":"plugin-manifest","format":"markdown","content":"# payload-link-publisher-py\n\nDev/E2E-only Python variant of the payload-link Dataflow publisher.\n\n**Prod uses the Java flex template one directory up.** This variant\nexists so the docker-compose e2e stack can run without a JVM, without\nDataflow, and without minute-scale iteration times.\n\n## Design summary\n\n- **Not Apache Beam.** Python's Beam SDK has no Spanner change-stream\n connector; a Beam pipeline in Python would either require a\n cross-language Java expansion (JVM back in) or a custom Source\n (more code than the whole non-Beam approach). Instead we poll the\n `READ_payload_link_changes` TVF directly via\n `google.cloud.spanner.Client`.\n- **Wire-format identical to the Java job.** `serialize_record`\n produces the exact JSON shape `PayloadLinkChangesToPubSub.serializeRecord`\n produces, so the reconciler consumes either interchangeably.\n- **Partition tracking.** Follows partition splits: reads `_root`,\n picks up child-partition tokens from any `child_partitions_record`,\n reads each child from its advertised `start_timestamp`, and retires\n a parent when a child announces it. Handles `OUT_OF_RANGE` by\n dropping ended partitions. Required in practice — the Spanner\n emulator routes DataChangeRecords to child partitions almost\n immediately, so a `_root`-only reader sees zero DCRs.\n- **In-memory cursor.** On restart, resumes from `now - POLL_OVERLAP_SECONDS`\n on `_root`. Fine for tests (compose brings the pod up alongside the\n stream); would lose events across restarts in prod.\n- **Hardcoded row-shape offsets.** Rather than introspect the\n `READ_` TVF's ChangeRecord / DataChangeRecord /\n ChildPartitionsRecord STRUCT layout at runtime, `publisher.py`\n documents the field positions and accesses them positionally. If a\n future Spanner rev reorders the STRUCT, tests fail loudly and one\n offset map gets updated. Acceptable trade-off for a dev/E2E tool.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-spanner`, `google-cloud-pubsub`, `statsd`. |\n| `publisher.py` | Entry point: poll TVF → filter → serialize → publish. |\n| `utils.py` | `extract_payload_link`, `is_payload_link_actionable` — mirrors the Java filter. |\n| `test_publisher.py` | Unit tests for the pure functions. |\n| `Dockerfile` | `python:3.13-slim` + poetry install. |\n\n## Environment\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `SPANNER_PROJECT_ID` | yes | — | |\n| `SPANNER_INSTANCE_ID` | yes | — | |\n| `SPANNER_DATABASE_ID` | yes | — | |\n| `PUBSUB_PROJECT_ID` | yes | — | |\n| `PUBSUB_TOPIC` | yes | — | e.g. `payload-link-changes` |\n| `SPANNER_CHANGE_STREAM_NAME` | no | `payload_link_changes` | |\n| `HEARTBEAT_MS` | no | `1000` | TVF heartbeat interval. |\n| `POLL_OVERLAP_SECONDS` | no | `5` | Restart re-reads this window. |\n| `SPANNER_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-spanner`. |\n| `PUBSUB_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-pubsub`. |\n| `STATSD_HOST` / `STATSD_PORT` | no | — | Standard `statsd.defaults.env` pair. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-link-dataflow/payload-link-publisher-py\npoetry install\npoetry run pytest\npoetry run mypy .\npoetry run ruff check .\n```\n\n`poetry.lock` is generated on first install (not yet checked in).\n\nUnit tests cover `extract_payload_link`, `is_payload_link_actionable`,\n`serialize_record`, and `_to_json_string`. Not covered by unit tests:\n`poll_partition` (needs a live Spanner emulator) and\n`publish_if_actionable` (needs a Pub/Sub emulator). Both are\nexercised end-to-end by the docker-compose reconciliation stack\n(`make docker_run_reconciliation_e2e_tests`).\n\n## Metrics\n\n- `payload_link_publisher.published` — records published to Pub/Sub\n- `payload_link_publisher.filtered` — inert records dropped by\n `is_payload_link_actionable`\n- `payload_link_publisher.errors.publish` — publish failures\n\n## Related\n\n- Java flex template (prod): `tools/payload-link-dataflow/` (parent dir)\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (`payload_link_changes`)\n- Downstream reconciler: `tools/payload-reconciler/`\n- Runbook: `docs/src/tools/payload_link_reconciler.md`\n","isInternal":false,"tokens":1080,"sizeBytes":4210},{"name":"README.md","path":"tools/payload-link-dataflow/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/payload-link-dataflow/README.md","title":"payload-link-dataflow Documentation","category":"plugin-manifest","format":"markdown","content":"# payload-link-dataflow\n\nCustom Dataflow flex template that reads the `payload_link_changes`\nSpanner change stream, drops records whose every mod has `payload_link`\nNULL on both sides, and publishes the remainder to a Pub/Sub topic.\n\nThe downstream consumer is the `payload-reconciler` cronjob\n(`tools/payload-reconciler/`), which finalizes newly-committed GCS\npayload objects (`committed=true`, `customTime=MAX`) and deletes\norphaned ones.\n\n## Directory layout\n\nThe Java source tree under `src/` is a standalone Apache Beam pipeline\nwe own and build directly. It is intentionally minimal: it uses Beam's\nstock `SpannerIO.readChangeStream` and `PubsubIO.writeStrings`, plus the\nfilter and a small JSON serializer. We do **not** vendor or depend on\nthe upstream GoogleCloudPlatform/DataflowTemplates Java tree.\n\n| Path | Purpose |\n|---|---|\n| `pom.xml` | Maven project (Apache Beam + Jackson + SLF4J). |\n| `src/main/java/.../PayloadLinkChangesToPubSub.java` | The pipeline: read change stream → filter → serialize → publish. |\n| `src/main/java/.../PayloadLinkOptions.java` | Flex template parameters. |\n| `src/test/java/.../PayloadLinkChangesToPubSubTest.java` | Unit tests for the filter predicate. |\n| `metadata.json` | Flex template parameter schema (consumed by `gcloud dataflow flex-template build`). |\n| `Dockerfile` | Multi-stage build: Maven → flex template launcher base image. |\n| `upstream-customization.patch` | **Reference documentation only.** Unified diff showing the **filter delta only** against the upstream template; see \"Beyond the filter\" below for the rest. Not a build input. |\n| `upstream.txt` | The upstream commit SHA the reference patch was last reviewed against. Update when re-reviewing intent against upstream. |\n| `generate-full-delta.sh` | On-demand: produces a unified diff of the **complete** delta between upstream (at the pinned SHA) and our impl. Not maintained as a checked-in artifact -- runs against the source tree at execution time. |\n| `payload-link-publisher-py/` | Dev/E2E-only Python variant of this publisher (not Beam, not shipped to prod). See its own README. |\n| `README.md` | This file. |\n\n## Build\n\n```bash\ndocker build -t payload-link-dataflow:local tools/payload-link-dataflow\n```\n\n`mvn package` runs the unit tests, so a change that breaks the filter\ncontract fails the image build.\n\nFor a local non-Docker build:\n\n```bash\ncd tools/payload-link-dataflow\nmvn -B package\n```\n\nThe bundled jar lands at\n`target/payload-link-dataflow-bundled-1.0-SNAPSHOT.jar`.\n\n## Publish + launch (operator workflow; lives in webservices-infra)\n\n```bash\ndocker tag payload-link-dataflow:local /payload-link-dataflow:\ndocker push /payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/payload-link-dataflow.json \\\n --image /payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n\ngcloud dataflow flex-template run payload-link-dataflow- \\\n --template-file-gcs-location gs:///templates/payload-link-dataflow.json \\\n --parameters spannerProjectId= \\\n --parameters spannerInstanceId= \\\n --parameters spannerDatabase= \\\n --parameters spannerMetadataInstanceId= \\\n --parameters spannerMetadataDatabase= \\\n --parameters spannerMetadataTableName=Metadata_payload_link \\\n --parameters changeStreamName=payload_link_changes \\\n --parameters pubsubTopic=projects//topics/payload-link-changes\n```\n\n`spannerMetadataTableName` is optional but strongly recommended for\nstreaming launches -- see \"Job update and restart continuity\" below.\n\nThe Dataflow job's service account needs:\n- `roles/spanner.databaseReader` on the syncstorage database\n- `roles/spanner.databaseUser` on the metadata database (the change\n stream connector writes its own partition state there)\n- `roles/pubsub.publisher` on the destination topic\n- `roles/dataflow.worker`\n\n### Fine-grained access control (optional)\n\nPass `spannerDatabaseRole` to read the change stream under a Spanner\ndatabase role instead of a database-wide IAM grant. `payload_link_reader`\nin `syncstorage-spanner/src/schema.ddl` is that role: it holds `SELECT`\non `payload_link_changes` and `EXECUTE` on the generated\n`READ_payload_link_changes` table function, and nothing else.\n\nThe role applies to the change stream read only. Beam's\n`MetadataSpannerConfigFactory` deliberately does not copy `databaseRole`\nonto the connector's metadata database config, so that connection still\nuses the service account's own IAM grants. Two consequences, both from\n[Spanner's FGAC guidance](https://docs.cloud.google.com/spanner/docs/fgac-change-streams):\n\n- `spannerMetadataDatabase` **must** be a different database than\n `spannerDatabase`. The connector needs database-level write access for\n its partition state, and that grant would override the role's\n restrictions on the database being read.\n- the service account must **not** hold `roles/spanner.databaseUser` on\n `spannerDatabase` -- it bypasses fine-grained access control outright.\n Grant `roles/spanner.fineGrainedAccessUser`, plus\n `roles/spanner.databaseRoleUser` conditioned on the role resource,\n instead.\n\nIf the metadata tables share the syncstorage database, setting\n`spannerDatabaseRole` buys nothing security-wise: the role is assumed,\nbut the database-level grant the connector needs supersedes it.\nProvision a dedicated metadata database first.\n\n### Job update and restart continuity\n\nPass `spannerMetadataTableName` on any streaming (unbounded) launch so\n`--update` and cancel-then-restart can resume from the previous job's\nwatermarks. The connector generates the partition metadata table name\ninside `expand()` from a `UUID.randomUUID()` call, and that string\ngets baked into the pipeline graph; if the new job's binary produces\na different UUID, it reinitializes against a fresh empty table and\nreads from `startTimestamp` (defaulting to `Timestamp.now()`),\nsilently dropping records committed between the two jobs.\n\nPick any Spanner-legal identifier -- e.g. `Metadata_payload_link` -- and\nkeep it stable across launches. The connector creates the table on\nfirst initialization if it does not already exist, then reuses it.\nBeam's own recommendation: *\"you have to explicitly set the metadata\ntable name parameter of the new job when updating it. Use the value\nof the `metadataTable` pipeline option from the job you are\nupdating.\"*\n\nNotes on stop modes:\n\n- `--update`: preserves Dataflow's checkpointed state but does **not**\n transfer the metadata table binding -- pin the name or lose state.\n- `--cancel`: leaves the metadata table intact; the next launch with\n the same name resumes cleanly.\n- `--drain`: the change stream connector does not support drain\n (per [Google's guidance](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining)).\n Use `--cancel` or `--update` instead.\n\n## Filter behaviour\n\n`PayloadLinkChangesToPubSub.isPayloadLinkActionable` keeps a\n`DataChangeRecord` when any of its mods has a non-null `payload_link`\non either side. It drops only the case where **every** mod has both\nold and new `payload_link` NULL -- the inert INSERT/DELETE noise from\ncolumn-scoped change streams. Malformed records pass through so the\nreconciler / DLQ -- not this filter -- surfaces them.\n\n## Beyond the filter: other differences from upstream\n\n`upstream-customization.patch` expresses only the filter step. Our\npipeline additionally diverges from upstream's\n`Cloud_Spanner_Change_Streams_to_PubSub` template in ways that diff\ndoes not attempt to capture. Run `./generate-full-delta.sh` to see\nthe complete diff against the pinned upstream SHA; the substantive\ndifferences are:\n\n- **Sink swap.** `PubsubIO.writeStrings` + a custom `serializeRecord`,\n not upstream's `FileFormatFactorySpannerChangeStreamsToPubSub`.\n Emits raw JSON strings; no PubsubMessage attributes.\n- **JSON wire format.** Emits only `{commitTimestamp, modType,\n tableName, mods[]}`. Omits fields upstream would emit:\n `serverTransactionId`, `rowType[]`,\n `numberOfRecordsInTransaction`,\n `numberOfPartitionsInTransaction`, `partitionToken`,\n `recordSequence`, `isLastRecordInTransactionInPartition`,\n `valueCaptureType`.\n- **No `UncaughtExceptionLogger.register()`.** Uncaught exceptions\n flow to the runner's default handler.\n- **No `@Template` / `TemplateCategory` annotations.** Upstream uses\n the `com.google.cloud.teleport.metadata` annotation-driven\n registrar for flex-template metadata. We register via\n `metadata.json` + `gcloud dataflow flex-template build` instead --\n simpler, no annotation-processor dependency.\n- **No support for these upstream options:**\n `useSpannerEmulatorHost`, `spannerHost`,\n `spannerChangeStreamTvfNameList`, `outputMessageMetadata`,\n `outputDataFormat` (JSON/Avro switch), `pubsubAPI`. Our\n `PayloadLinkOptions` deliberately has a smaller surface (~12\n options vs. upstream's ~25).\n- **No ValueProvider indirection on options.** Upstream wraps\n several config fields in `ValueProvider` for template\n parameterisation; we take plain strings.\n- **No explicit `streaming=true` mutation.** Beam infers streaming\n from the source (`SpannerIO.readChangeStream` is\n `@UnboundedPerElement`) and `DataflowRunner` flips the flag at\n translation time.\n\nThese deltas were deliberate scope choices, not oversights. The\nfilter-only patch is kept as the primary review artifact because\nit's small enough to eyeball, applies cleanly against the pinned\nSHA, and is insensitive to upstream churn on features we don't use.\n\n`main()` matches upstream on two pipeline-shape invariants that are\nnot deployment choices for this source: it appends `use_runner_v2`\nto `options.experiments` and calls `setEnableStreamingEngine(true)`.\n`BundleFinalizer` (used by `SpannerIO.readChangeStream`) requires the\nDataflow Portable Runner (\"Runner V2\"), and V2 streaming jobs\nrequire Streaming Engine -- so both settings are pinned in code\nrather than left to launch-time flags.\n\n## Output wire format\n\nEach surviving `DataChangeRecord` is serialized to a JSON Pub/Sub\nmessage of the form:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-29T12:34:56.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nEach mod field carries a **JSON string** (escaped) -- the downstream\nPython reconciler reads them with a second `json.loads` per mod. This\nmatches the Spanner change-streams wire convention.\n\n## Running against the compose e2e stack\n\nThe `docker_run_reconciliation_e2e_tests` Makefile target brings up\nSpanner + Pub/Sub + fake-GCS emulators and a publisher container\nlocally. There are two publisher variants; both produce the exact same\nPub/Sub wire format so the reconciler consumes either\nindistinguishably.\n\n### Default: Python publisher\n\n`docker/docker-compose.e2e.reconciliation.yaml` builds and runs the\nPython variant at `payload-link-publisher-py/`. Sub-second startup,\nno JVM. Use this for day-to-day iteration.\n\n```bash\nmake docker_run_reconciliation_e2e_tests\n```\n\n### Swap-in: this Java publisher\n\n`docker/docker-compose.e2e.reconciliation.java.yaml` overrides the\n`payload-link-publisher` service to build **this** directory's Docker\nimage and run the shaded JAR under Beam's DirectRunner (bypassing the\nflex-template launcher's Dataflow-submission entrypoint). Use when you\nneed to reproduce a prod issue against the actual Java pipeline or\nverify wire-format parity between the two variants.\n\n```bash\ndocker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.reconciliation.java.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n up --exit-code-from e2e-tests --abort-on-container-exit\n```\n\nExpect a ~15-30s JVM+Beam warmup and a heavier memory footprint than\nthe Python variant. If Docker's default resource limits are tight on\nyour dev machine, bump them.\n\n## Keeping the reference patch accurate\n\n`upstream-customization.patch` is reference documentation, not a build\ninput. If upstream's pipeline shape evolves or our pipeline drifts in\nways that change the conceptual diff, regenerate it:\n\n```bash\ngit clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git upstream\ncd upstream\ngit checkout \n# Manually re-author the filter customization against the current\n# upstream SpannerChangeStreamsToPubSub.java (in our src/ we own the\n# whole pipeline; here we only need to express the *delta*).\ngit diff > ../tools/payload-link-dataflow/upstream-customization.patch\necho \"\" > ../tools/payload-link-dataflow/upstream.txt\n```\n\nAfter updating the SHA, run `./generate-full-delta.sh` and skim the\noutput to check that our impl's other divergences (sink swap, JSON\nshape, dropped options -- see \"Beyond the filter\" above) still make\nsense against upstream's new state. If upstream restructured\nsomething we deliberately dropped, update the \"Beyond the filter\"\nbullets accordingly.\n\nThis is purely a documentation refresh -- it does not affect the build.\n","isInternal":false,"tokens":3169,"sizeBytes":13414},{"name":"README.md","path":"tools/payload-reconciler/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/payload-reconciler/README.md","title":"payload-reconciler Documentation","category":"plugin-manifest","format":"markdown","content":"# payload-reconciler\n\nPub/Sub-driven reconciler that closes the loop on GCS payload offload.\n\nReads the `payload-link-changes` Pub/Sub topic (populated by the\n`tools/payload-link-dataflow/` flex template). For each mod inside each\nchange record:\n\n- **New `payload_link` non-null** → patch the referenced GCS object's\n metadata to `committed=true` and `customTime=MAX`. Completes the\n 2-phase commit started in `syncserver/src/web/payload_offload.rs`.\n- **Old `payload_link` non-null and ≠ new** → delete the referenced GCS\n object. It is unreferenced now (the row's `payload_link` moved, was\n cleared, or the row was deleted — including TTL row-deletion-policy\n deletes).\n\nBoth operations are idempotent (re-setting metadata is a no-op; 404 on\ndelete is treated as success), so Pub/Sub at-least-once redelivery is\nsafe and no ordering key is needed.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-pubsub`, `google-cloud-storage`, `datadog`. |\n| `reconcile_payload_links.py` | Entry point: sync-pull drain loop + per-message handler. |\n| `utils.py` | `parse_gs_url()` — mirrors the Rust uploader's URL grammar. |\n| `test_utils.py` | Tests for `parse_gs_url`. |\n| `test_reconcile_payload_links.py` | Handler tests with a mocked GCS client. |\n\n## Environment\n\n| Variable | Required | Notes |\n|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | e.g. `payload-link-reconciler-sub`. |\n| `GCS_PAYLOAD_BUCKET` | yes | The bucket the syncserver writes payloads into. Cross-bucket links abort the message and route it to the DLQ. |\n| `RUN_BUDGET_SECONDS` | no | Selects deployment mode. **Set** (e.g. `240`): cronjob mode -- drain up to that many seconds then exit 0. **Unset**: long-running mode -- poll forever, never exit on idle; the deployment supervises restarts. Cronjob mode is the default deployment. |\n| `SYNC_STATSD_HOST` | no | statsd server host. |\n| `SYNC_STATSD_PORT` | no | statsd server port. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-reconciler\npoetry install\npoetry run pytest\n```\n\nUnit tests cover `parse_gs_url` and the per-message handler. The\n`drain` loop (sync-pull, budget vs. long-running exit conditions) has\nno unit coverage — exercise it against the Pub/Sub emulator when\nchanging it.\n\n## Metrics\n\nStatsd counters (all under the `payload_reconciler.` prefix):\n\n- `finalizes` — successful `committed=true` + `customTime=MAX` patch\n- `orphan_deletes` — successful delete of an unreferenced object\n- `noop_skips` — message processed with zero ops. Should be ~0 if the\n Dataflow filter is working; non-zero is the alarm bell that the\n filter regressed.\n- `gcs_404` + `op:finalize` / `op:delete` — 404 on the respective op.\n Expected non-zero under at-least-once redelivery (idempotency tax);\n a sharp rise suggests upstream state is gone unexpectedly.\n- `errors` + `kind:handler` — Sustained non-zero values feed the DLQ.\n\n## Deployment\n\n**The reconciler ships in the primary syncserver image**, built from\nthe root `Dockerfile`. Its script and Python deps are baked in via\nthat Dockerfile's per-tool poetry-export / pip-wheel / pip-install\nchain; K8s picks between the syncserver binary and the reconciler\nscript via the pod's `command:`.\n\nDefault: K8s cronjob (definition lives in webservices-infra), cadence\n~5 min, exports `RUN_BUDGET_SECONDS` (e.g. `240`).\n\nAlternative: long-running K8s Deployment. Omit `RUN_BUDGET_SECONDS`;\nthe pod polls forever. Choose this when finalize-flip latency below\nthe cronjob cadence matters (e.g. an operator wants offloaded\npayloads to become `committed=true` within seconds of upload rather\nthan up to one cronjob cycle later).\n\nService account needs the same roles in either mode:\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and `delete` operations)\n\n## Related\n\n- Producer: `syncserver/src/web/payload_offload.rs`\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (look for `payload_link_changes`)\n- Upstream pipeline: `tools/payload-link-dataflow/`\n","isInternal":false,"tokens":1091,"sizeBytes":4176},{"name":"README.md","path":"tools/postgres/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/postgres/README.md","title":"postgres Documentation","category":"plugin-manifest","format":"markdown","content":"# PostgreSQL Utility Scripts\n\nCurrently there is one script: `purge_ttl.py`.\n\n## purge_ttl.py\n\nPurge expired items (BSOs and batches) from a Sync storage PostgreSQL instance.\nThe script is intended to be invoked regularly, e.g. as a cronjob.\n\n### Prerequisites\n\n- Python 3\n- [Poetry](https://python-poetry.org/docs/#installation)\n\n### Run Script Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run python purge_ttl.py\"\n```\n\nPass `-h` to see the list of options.\n\n### Run Tests Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run pytest test_purge_ttl.py -v\"\n```\n\n### Run with Docker\n\nA Docker image with the PostgreSQL build is not published anywhere at the\nmoment. Once that's available, this readme will be updated to include that\nstep.\n\n```bash\ndocker run --rm \\\n --entrypoint python3 \\\n -e SYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@host:5432/syncstorage\" \\\n syncstorage-postgres-image:tag \\\n /app/tools/postgres/purge_ttl.py\n```\n\n","isInternal":false,"tokens":286,"sizeBytes":1124},{"name":"README.md","path":"tools/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/README.md","title":"tools Documentation","category":"plugin-manifest","format":"markdown","content":"# A collection of Sync Tools and utilities\n\nSee each directory for details:\n\n* [hawk](hawk) - a tool for generating test HAWK authorization headers\n* [spanner](spanner) - Google Cloud Platform Spanner tools for maintenance and testing\n* [payload-link-dataflow](payload-link-dataflow) - custom Dataflow flex template feeding the payload-link reconciliation pipeline\n* [payload-reconciler](payload-reconciler) - Pub/Sub-driven reconciler that finalizes and garbage-collects offloaded GCS payload objects\n## Installation\n\nThese tools are mostly written in python. It is recommended that you create a commonly shared virtual environment using something like:\n\n`python3 -m venv venv`\n\nto create a `/venv` directory. To activate this, call `sh /venv/bin/activate`.\n\nScript dependencies can be installed via poetry:\n\n```shell\npip install poetry\npoetry install\n```\n\nfor each tool.\n","isInternal":false,"tokens":195,"sizeBytes":873},{"name":"README.md","path":"tools/spanner/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/spanner/README.md","title":"spanner Documentation","category":"plugin-manifest","format":"markdown","content":"# Spanner Tools and Scripts\n\nThese tools are supplemental scripts for working with the Google Cloud Platform. Follow [the general installation instructions](https://cloud.google.com/spanner/docs/getting-started/python/), as well as fetch the proper service account credentials file.\n\nRemember, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable should point to the absolute path location of your service account credential file.\n\ne.g.\n```bash\nGOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/project-id-service-cred.json venv/bin/python count_expired_rows.py\n```\nSee each script for details about function and use.\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs spanner untilities, you'll need a Python =>3.12 development environment with `Poetry` installed. These scripts typically run in a deployed GCP workflow, so their Dockerfile config will generally prepare all of this for you. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `spanner/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/spanner\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 spanner # or whatever project name you like.\n $ pyenv local spanner # activates virtual env whenever you enter this directory.\n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python count_expired_rows.py`\n","isInternal":false,"tokens":576,"sizeBytes":2525},{"name":"README.md","path":"tools/syncstorage-loadtest/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/syncstorage-loadtest/README.md","title":"syncstorage-loadtest Documentation","category":"plugin-manifest","format":"markdown","content":"# Storage load test\n\n**Note:** This requires Python version 3.10+\n\nThe tests can execute in three modes:\n\n1. **Direct access**: use a known master secret\n2. **Firefox accounts OAuth**: create test accounts and use OAuth JWTs\n3. **Self-signed JWT**: generate and sign JWTs locally with a given private key\n\n## Installation\n\n### Environment Setup\n\nTo run the syncstorage load tests, you'll need a Python >=3.10 development environment with `Poetry` installed. You can also directly use `poetry run` to execute commands as described in the usage examples below.\n\nThe easiest solution is to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments as a way to isolate the dependencies from other projects.\n\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\n See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Example:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n\n4. Install Python version, create virtualenv, activate and install dependencies from inside the project directory.\n\n **Note:** You can skip creating a virtual environment and invoke commands directly using `poetry run`.\n\n ```shell\n $ cd syncstorage-loadtest\n\n # Install Python 3.10+\n $ pyenv install 3.10\n\n # Create named, associated virtualenv\n $ pyenv virtualenv 3.10 syncstorage-loadtest # or whatever name you prefer\n $ pyenv local syncstorage-loadtest # activates virtual env whenever you enter this directory\n\n # Install Poetry and dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. Once you're in your virtual environment, run the load tests using:\n ```bash\n poetry run molotov [options] loadtest.py\n ```\n\n### Quick Install\n\nIf you already have Poetry installed:\n\n```bash\npoetry install\n```\n### Generate Key\n Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```bash\npoetry run ./generate-keys.sh\n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```bash\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n```\n## Mode 1: Direct Access\n\nWith a known syncstorage master secret:\n\n```bash\nSERVER_URL=\"http://localhost:8000#secretValue\" poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n## Mode 2: Firefox Accounts OAuth\n\nWith FxA stage accounts:\n\n```bash\nSERVER_URL=\"https://token.stage.mozaws.net\" \\\n FXA_API_HOST=\"https://api-accounts.stage.mozaws.net\" \\\n FXA_OAUTH_HOST=\"https://oauth.stage.mozaws.net\" \\\n poetry run molotov --workers 3 --duration 60 -v loadtest.py\n```\n\n## Mode 3: Self-Signed JWTs\n\nAssuming an RSA keypair was generated, the private key pem was saved and accessible, and the token server was started with the public key JWK configured. The presence of `OAUTH_PRIVATE_KEY_FILE` triggers this mode when using OAuth.\n\nRun with self-signed JWTs:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n## Expanded payloads (GCS offload)\n\nThe batch write path can generate larger-than-default payloads so you can load\ntest collections whose payloads are offloaded to Google Cloud Storage. These\nknobs are opt-in via environment variables; unset, the load test behaves\nexactly as before.\n\n| Variable | Default | Effect |\n|---|---|---|\n| `LARGE_PAYLOAD_PROB` | `0.0` | Fraction (`0.0`–`1.0`) of BSOs given an expanded payload. `1.0` makes **every** payload large; `0.0` disables the feature. |\n| `LARGE_PAYLOAD_SIZE` | *(unset)* | Explicit target size in bytes for large payloads. When unset, large payloads target the server's `max_record_payload_bytes`. |\n| `OFFLOAD_COLLECTIONS` | *(unset)* | Comma-separated collection names to target for batch writes. Must match the server's `gcs_payload_offload_collections`. When unset, the default collections are used. |\n\n`LARGE_PAYLOAD_PROB`/`LARGE_PAYLOAD_SIZE` control payload **size**; whether a\nwrite is offloaded to GCS is a separate, per-collection server setting. To drive\na fully offloaded collection at maximum size, set `OFFLOAD_COLLECTIONS` to that\ncollection and `LARGE_PAYLOAD_PROB=1.0`.\n\nIndividual payloads are capped at the target collection's\n`max_record_payload_bytes`, and each batch is kept under its `max_post_bytes`,\nso a large enough payload results in fewer records per request.\n\nExample — every write is a 5 MiB payload sent to the `tabs` collection:\n\n```bash\nSERVER_URL=\"http://localhost:8000\" \\\n OAUTH_PRIVATE_KEY_FILE=\"/path/to/load_test.pem\" \\\n LARGE_PAYLOAD_PROB=1.0 \\\n LARGE_PAYLOAD_SIZE=5242880 \\\n OFFLOAD_COLLECTIONS=tabs \\\n poetry run molotov --workers 100 --duration 300 -v loadtest.py\n```\n\n### Server-side prerequisites\n\nTo actually exercise expanded/offloaded payloads end to end, the target server\nmust be configured for it:\n\n- `gcs_payload_bucket` set and the target collection listed in\n `gcs_payload_offload_collections` (syncstorage settings).\n- Raised limits to allow payloads beyond the 2.5 MiB default:\n `max_record_payload_bytes`, `max_post_bytes`, `max_request_bytes`, and\n `max_total_bytes`. These are read from `/info/configuration`. A collection\n may also raise its own limits via the `collections` section of that\n response; the load test resolves limits per target collection, preferring a\n collection's override and falling back to the global value.\n- Any front-end proxy (e.g. nginx) request body-size limit raised to match.\n\nNote: many workers each holding multi-MiB payloads is a real memory footprint\non the load-generating host — size the harness accordingly.\n\n## Docker\n\nBuild and run this directory's image:\n\n```bash\ndocker build . --tag syncstorage-loadtest:local\n\ndocker run --rm --net=host \\\n -e SERVER_URL=\"http://localhost:8000#secretValue\" \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 1 --workers 5 --duration 60 -v loadtest.py\"\n```\n\n## Full offload pipeline on docker\n\nFor the offload and change-stream load tests (STOR-628, STOR-629) there is a\ncompose rig that stands up syncserver with offload enabled, a Spanner emulator\ncarrying the change stream, Pub/Sub, GCS, the publisher, the reconciler and\nthis load test, then reports whether the pipeline drained clean:\n\n```bash\nmake loadtest-offload-image\nmake loadtest-offload-handler # 100% offloaded\nmake loadtest-offload-report\nmake loadtest-offload-down\n```\n\nSee `docs/src/tools/load-testing.md` for the runbook, the knobs, and the GCP\ndev path.\n\nHappy Breaking!\n","isInternal":false,"tokens":1774,"sizeBytes":7070},{"name":"README.md","path":"tools/tokenserver/loadtests/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/tokenserver/loadtests/README.md","title":"loadtests Documentation","category":"plugin-manifest","format":"markdown","content":"# Tokenserver Load Tests\n\nThis directory contains everything needed to run the suite of load tests for Tokenserver.\n\n## Prerequisite Dependency and Environment Setup:\nTo use the syncstorage-rs tokenserver load tests, you'll need a Python =>3.12 development environment with `Poetry` installed. This testing script can be run within a spun up GCP workflow where you can execute the locust commands from the UI. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `loadtests/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/tokenserver/loadtests\n # pyenv version install\n $ pyenv install 3.10\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.10 loadtests # or whatever project name you like.\n $ pyenv local loadtests # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\nEx. `poetry run python locustfile.py`\n\n\n## Building and Running\n\n1. Install the load testing dependencies as described above:\n\n ```sh\n poetry install\n ```\n\n2. Run the `generate-keys.sh` script to generate an RSA keypair and derive the public JWK.\n\nSince this script calls `get_jwk.py` and it has a dependency on `autlib`, call the shell script using Poetry:\n\n```sh\npoetry run ./generate-keys.sh \n```\n\nOtherwise, if in built virtual environment with installed Poetry dependencies:\n\n```sh\n./generate-keys.sh\n```\n\nThis script will output two files:\n\n- `load_test.pem`: The private key to be used by the load tests to create OAuth tokens\n- `jwk.json`: The public JWK associated with the private key. This is a key of the form:\n\n```json\n{\n \"n\": ...,\n \"e\": ...,\n \"kty\": \"RSA\"\n}\n ```\n\n3. Set the following environment variables/settings on Tokenserver:\n\n ```sh\n # Should be set to the \"n\" component of the JWK\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N\n # Should be set to the \"e\" component of the JWK (this value should almost always be \"AQAB\")\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY=RSA\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__USE=sig\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__ALG=RS256\n\n # These two environment variables don't affect the load tests, but they need to be set:\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID=\"\"\n SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__FXA_CREATED_AT=0\n ```\n\n Note that, because these settings cache the JWK used to verify OAuth tokens, no verification requests will be made to FxA, so the value of `SYNC_TOKENSERVER__FXA_OAUTH_VERIFIER_URL` does not matter; however, Tokenserver expects it to be set, so setting it to something like `http://localhost` will suffice.\n\n4. Set the following environment variables on the machine that will be running the load tests:\n\n - `OAUTH_PEM_FILE` should be set to the location of the private RSA key generated in a previous step\n\n5. Tokenserver uses [locust](https://locust.io/) for load testing. To run the load tests, simply run the following command in this directory:\n\n ```sh\n locust\n ```\n\n6. Navigate your browser to , where you'll find the locust GUI. Enter the following information:\n\n - Number of users: The peak number of Tokenserver users to be used during the load tests\n - Spawn rate: The rate at which new users are spawned\n - Host: The URL of the server to be load tested. Note that this URL must include the protocol (e.g. \"http://\")\n\n7. Click the \"Start swarming\" button to begin the load tests.\n\n## Populating the Database\n\nThis directory includes an optional `populate_db.py` script that can be used to add test users to the database en masse. The script can be run like so:\n\n```sh\npoetry run python populate_db.py \n```\n\nOr if in built virtualenv:\n```sh\npython3 populate_db.py \n```\n\nwhere `sqluri` is the URL of the Tokenserver database, `nodes` is a comma-separated list of nodes **that are already present in the database** to which the users will be randomly assigned, and `number of users` is the number of users to be created.\n","isInternal":false,"tokens":1293,"sizeBytes":5218},{"name":"README.md","path":"tools/tokenserver/README.md","rawUrl":"https://raw.githubusercontent.com/mozilla-services/syncstorage-rs/HEAD/tools/tokenserver/README.md","title":"tokenserver Documentation","category":"plugin-manifest","format":"markdown","content":"# Summary of Files in tools/Tokenserver\n\n| File Name | Description |\n|--------------------------------|-----------------------------------------------------------------------------|\n| `add_node.py` | Adds new node to the tokenserver's database, registering it for user allocation. |\n| `allocate_user.py` | Script to allocate a specific user to a node. |\n| `conftest.py` | Pytest configuration and fixtures for shared test setup. |\n| `count_users.py` | Script to emit total-user-count metrics for exec dashboard. |\n| `database.py` | Shared database utility queries and functions used by multiple scripts. |\n| `purge_old_records.py` | Script to purge user records that have been replaced. |\n| `pytest.ini` | Configuration file for pytest, specifying options like test output format. |\n| `remove_node.py` | Script to remove a node from the system. |\n| `test_database.py` | Unit tests for `database.py`. |\n| `test_purge_old_records.py` | Tests for `purge_old_records.py`, validating cleanup operations. |\n| `test_scripts.py` | Testing module to test the various scripts in this directory. |\n| `unassign_node.py` | Removes a node from the system and clears any assignments to the named node.| \n| `update_node.py` | Script to update node status in the db. |","isInternal":false,"tokens":301,"sizeBytes":1715}],"systemPromptSnippet":"\n\n# syncstorage-rs\n\nFirefox Sync storage and token server, written in Rust. The workspace contains a\nRust multi-crate server plus Python utilities and integration/end-to-end tests.\n\n## Agent rules\n\nSecurity takes absolute precedence. This repository handles synchronization of\nuser data and token management to access storage nodes.\n\n### 1) Scope & writes\n\n- Operate strictly within this repo root; normalize paths; do not follow\n symlinks outside the repo.\n- Writes are allowed to the working tree. Always present a diff for review\n before any staging or commit.\n- Do not modify files adjacent to a requested change. Mention issues found\n but do not fix them unless asked.\n- Ask before running any command (build, test, install, DB ops, git,\n service management). Do not run `git add`, `git commit`, `git push`, or\n `git rebase` unless explicitly instructed.\n\n### 2) Non-negotiables\n\n- **Secrets:** never read, print, summarize, or transmit secret files or\n values. Use placeholders (`YOUR_API_KEY_HERE`) in examples.\n- **External network:** only with explicit approval to a trusted, documented\n endpoint.\n- **Pipelines & contracts:** flag breaking API or contract changes. Do not\n alter GitHub Actions defined in `.github/`, any configuration for CI/CD workflows or git hooks without explicit, reviewed\n justification.\n- **Published DB migrations:** never edit existing published migration files or schema definitions (ex. `syncstorage-spanner/src/schema.ddl`), whether for MySQL, Postgres, or Spanner.\n Always add a new forward migration and a separate rollback migration.\n- **Workspace recommendations:** the repo deliberately ships no\n `.vscode/extensions.json`. Do not re-add one. Useful extensions are listed\n in `.vscode/README.md` for contributors to install at their own discretion.\n\n### 3) Do-not-touch paths\n\nNever read or write these paths. This list mirrors `.gitignore` — if there is\na discrepancy, `.gitignore` is the source of truth.\n\n```\nservice-account.json # GCP credentials\n.sentryclirc # Sentry auth token\n.envrc # direnv secrets\nconfig/local.toml # local server config (may contain secrets)\ntools/tokenserver/loadtests/*.pem\ntools/tokenserver/loadtests/*.pub\ntarget/ # Rust build artifacts\n*.xml # test result files\nopenapi.json # generated artifact\nbook/ # generated docs\ndocs/output/ # generated docs\nvenv/ # Python virtual environments\n.install.stamp # Poetry install marker\nworkspace/ # CI artifact directory\n```\n\n### 4) Git commit messages\n\nFollow the commit message format defined in [CONTRIBUTING.md](CONTRIBUTING.md) Git Commit Guidelines.\n\n- Format: `type(scope): subject` — `type` is one of `feat`, `fix`, `docs`,\n `style`, `refactor`, `perf`, `test`, `chore`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Body: explain motivation and contrast with previous behavior\n- Footer: `Closes STOR-1234` (Mozilla engineers — matches the Jira ticket) or\n `Closes #N` / `Issue #N` (community contributors — matches the GitHub issue number);\n `BREAKING CHANGE:` for breaking changes\n- Branch naming (Mozilla engineers): `type/description-STOR-1234` where `1234` must match the numeric suffix of the associated Jira ticket\n- Branch naming (community): `type/description-1112`\n- All commits must be GPG/SSH signed (enforced by CI)\n\nExample:\n\n```\nfeat: add node reassignment on capacity overflow\n\nNodes at 100% capacity previously returned 503 rather than\nassigning the user to an available node.\n\nCloses #42.\n```\n\n## Project layout\n\n```\nsyncserver/ Main Actix-web server (entry point)\nsyncstorage-db/ Syncstorage database abstraction layer\nsyncstorage-db-common/ Syncstorage shared database types and traits\nsyncstorage-mysql/ MySQL backend for syncstorage\nsyncstorage-postgres/ PostgreSQL backend for syncstorage\nsyncstorage-spanner/ Google Cloud Spanner backend for syncstorage\nsyncstorage-settings/ Syncstorage configuration types\ntokenserver-db/ Tokenserver database abstraction layer\ntokenserver-db-common/ Tokenserver shared database types\ntokenserver-mysql/ MySQL backend for tokenserver\ntokenserver-postgres/ PostgreSQL backend for tokenserver\ntokenserver-auth/ HAWK token generation and verification\ntokenserver-common/ Shared tokenserver utilities\ntokenserver-settings/ Tokenserver configuration types\nsyncserver-common/ Shared server utilities\nsyncserver-db-common/ Shared database utilities\nsyncserver-settings/ Syncserver configuration types\nglean/ Glean telemetry crate\ntools/\n integration_tests/ Pytest-based integration + e2e tests\n tokenserver/ Tokenserver utilities and load tests\n spanner/ Spanner utilities\n hawk/ HAWK token generation utility\n syncstorage-loadtest/ Molotov load tests\ndocker/ Docker Compose files for each backend\nconfig/ Local TOML configuration files\ndocs/ mdBook documentation source\n.github/ GitHub CI Actions and configuration\n```\n\n## Rust builds\n\nThe default feature set targets MySQL, however production uses the Spanner database with MySQL Tokenserver. Backend features are mutually exclusive.\n\n```bash\n# MySQL (default)\ncargo build\n\n# PostgreSQL\ncargo build --no-default-features --features=syncstorage-db/postgres --features=tokenserver-db/postgres --features=py_verifier\n\n# Spanner\ncargo build --no-default-features --features=syncstorage-db/spanner --features=py_verifier\n```\n\n## Rust tests\n\nTests require `RUST_TEST_THREADS=1` (many tests share a real database).\n\n```bash\n# MySQL unit tests (default DATABASE_URL envs set in Makefile)\nmake test\n\n# Specific package\nmake test ARGS=\"--package syncstorage-db\"\n\n# With coverage\nmake test_with_coverage # MySQL\nmake postgres_test_with_coverage\nmake spanner_test_with_coverage\n```\n\n## Clippy / formatting / audit\n\nFormatting checks are necessary before applying any code changes, as they\ncan capture problematic patterns or compiler issues in advance.\n\n```bash\ncargo fmt -- --check # format check\ncargo fmt # auto-format\n\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\n\nmake clippy_release_mysql # release-mode (catches dead code etc.)\nmake clippy_release_postgres\nmake clippy_release_spanner\n\ncargo audit # check dependencies for known CVEs (run by CI on every push)\n```\n\nAfter editing Rust code, run clippy and the unit tests for the affected backend before finishing the task; the `rust-code-smell` skill covers changed files across backends.\n\n## Python tooling\n\nAll Python tooling uses [Poetry](https://python-poetry.org/). The root\n`pyproject.toml` covers the `tools/` tree. Each subdirectory under `tools/` has\nits own `pyproject.toml` for its specific dependencies.\n\nAll of these Python checks must be run with any related change to a Python\nutility or test.\n\n```bash\nmake install # install root dependencies\nmake integration-test # install integration test dependencies\nmake tokenserver # install tokenserver utility dependencies\n```\n\nLint / format / type-check:\n\n```bash\nmake ruff-lint # ruff check tools/\nmake ruff-format # ruff format tools/\nmake ruff-fmt-chk # format diff check\nmake mypy # mypy type checking\nmake pydocstyle # docstring validation\nmake bandit # security linting\n```\n\n## Integration / E2E tests\n\n### Local (against a running server)\n\n```bash\n# Run all integration tests except real FxA e2e\nPYTHONPATH=$(pwd)/tools \\\nSYNC_MASTER_SECRET=secret0 \\\nSYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL=http://localhost:6000 \\\nTOKENSERVER_HOST=http://localhost:8000 \\\npoetry -C tools/integration_tests run pytest . --ignore=tokenserver/test_e2e.py\n```\n\nOr via the Makefile shortcut (sets the env vars above):\n\n```bash\nmake run_local_e2e_tests\n```\n\n### Docker-based E2E (full stack)\n\n```bash\nmake docker_run_mysql_e2e_tests\nmake docker_run_postgres_e2e_tests\nmake docker_run_spanner_e2e_tests\n```\n\nEach command spins up two compose stacks (with and without JWK cache), copies\nJUnit XML results out of the container, then tears everything down.\n\n## Running the server locally\n\n```bash\n# MySQL\nmake run_mysql\n\n# Spanner (requires GCP service account key at ./service-account.json)\nmake run_spanner\n```\n\nServer listens on `http://localhost:8000`. Health check: `GET /__heartbeat__`.\n\n## Key environment variables\n\n| Variable | Purpose |\n|---|---|\n| `SYNC_MASTER_SECRET` | HAWK token signing secret |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | Syncstorage DB connection string |\n| `SYNC_TOKENSERVER__DATABASE_URL` | Tokenserver DB connection string |\n| `SYNC_TOKENSERVER__NODE_TYPE` | `mysql` / `postgres` / `spanner` |\n| `SYNC_TOKENSERVER__RUN_MIGRATIONS` | Auto-migrate on startup (`true`/`false`) |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | FxA OAuth endpoint |\n| `RUST_TEST_THREADS` | Set to `1` for all test runs |\n| `RUST_LOG` | Log level (`debug`, `info`, etc.) |\n| `SYNC_TEST_LOG_HTTP` | Set any value to log all HTTP req/resp in integration tests |\n| `SYNC_SERVER_URL` | Integration test base URL (default: `http://localhost:8000`) |\n| `TOKENSERVER_HOST` | Tokenserver host for integration tests |\n| `PYTHONPATH` | Must point to `$(pwd)/tools` when running integration tests |\n\n## Docker Compose files\n\n| File | Purpose |\n|---|---|\n| `docker/docker-compose.mysql.yaml` | MySQL syncstorage + MySQL tokenserver |\n| `docker/docker-compose.postgres.yaml` | PostgreSQL syncstorage + PostgreSQL tokenserver |\n| `docker/docker-compose.spanner.yaml` | Spanner emulator + MySQL tokenserver |\n| `docker/docker-compose.e2e.{mysql,postgres,spanner}.yaml` | E2E test runner overlay |\n| `docker/docker-compose.e2e.jwk-cache.yaml` | Enable JWK caching overlay |\n| `docker/docker-compose.e2e.no-jwk-cache.yaml` | Disable JWK caching overlay |\n\n## Documentation\n\n```bash\nmake doc-install-deps # install mdBook + mdBook-mermaid\nmake doc-watch # live preview at localhost\nmake doc-prev # build and serve\nmake doc-test # validate markdown\n\nmake api-prev # generate OpenAPI spec + serve Swagger UI on :8080\n```\n\n\n## Test structure\n\n- `tools/integration_tests/conftest.py` — pytest fixtures only (`st_ctx`)\n- `tools/integration_tests/helpers.py` — retry helpers, auth state, `switch_user`\n- `tools/integration_tests/test_storage.py` — storage protocol tests\n- `tools/integration_tests/tokenserver/conftest.py` — tokenserver fixtures only\n- `tools/integration_tests/tokenserver/helpers.py` — tokenserver DB/auth helpers\n- `tools/integration_tests/tokenserver/test_*.py` — tokenserver tests\n\n## MCP integrations\n\nTwo MCP servers are configured for this repository: **Atlassian Jira** (`mcp__atlassian__`) and **Sentry** (`mcp__sentry__`).\n\nUse the skills below to query them. Do not call MCP tools ad-hoc without reading the shared context first.\n\n| Skill | Invocation | Purpose |\n|---|---|---|\n| Jira status | `/jira-status [me\\|epic KEY\\|epics\\|sprint\\|summary ...\\|create ...\\|update STOR-#### ...]` | Task landscape, epics, personal view, summaries, create/edit tickets |\n| Sentry health | `/sentry-health [new\\|regressions\\|volume\\|full]` | Production error trends, regressions, new issues |\n\n**Shared context:** `.claude/mcp-context.md` contains project IDs, JQL patterns, Sentry project slug, what to flag, and known noise. Skills read this at runtime. Read it before any ad-hoc MCP query.\n\n**Identity when using these tools:** You are acting as both engineer and engineering manager. Surface patterns, not just raw lists. Distinguish \"needs action now\" from \"worth monitoring\". Do not fabricate data — if a query returns nothing, say so.\n\n**Jira project:** `STOR` — `https://mozilla-hub.atlassian.net`\n**Sentry project:** `syncstorage-prod` — org `mozilla`\n\n\n---\nname: jira-status\ndescription: Queries, summarizes, creates, and edits Jira issues for syncstorage-rs (project STOR). Supports user-centric work views, epic breakdowns, structured summaries with impact/context/highlights, and ticket creation or modification from natural language.\nuser-invocable: true\nargument-hint: \"me | epic | epics | sprint | summary | create | update | status (default: status)\"\n---\n\nStart by reading `.claude/mcp-context.md` for project IDs, JQL patterns, issue types, and field conventions.\n\nThen call `mcp__atlassian__getAccessibleAtlassianResources` to confirm the Jira cloud ID and base URL for `mozilla-hub.atlassian.net`. Store the base URL as `JIRA_BASE` — all issue links are formatted as `JIRA_BASE/browse/STOR-####`.\n\nParse `$ARGUMENTS` (trimmed). Match against the modes below. Default is `status`.\n\n---\n\n## Mode: `status` (default)\n\nOverall project task landscape. Use when no argument is given.\n\nRun these JQL queries in parallel:\n\n```jql\n-- Open sprint issues\nproject = STOR AND sprint in openSprints() AND statusCategory != Done ORDER BY priority DESC, updated DESC\n\n-- Unassigned open issues\nproject = STOR AND statusCategory != Done AND assignee is EMPTY ORDER BY priority DESC\n\n-- Recently updated (last 3 days)\nproject = STOR AND updated >= -3d AND statusCategory != Done ORDER BY updated DESC\n```\n\n**Output:**\n\n### Project Status\nOne-line overall signal: `Active` / `Needs attention` / `Stalled`.\n\n### Current Sprint\nTable: Key (linked) | Summary | Status | Assignee | Priority\nGroup by status category (In Progress first, then To Do, then Done).\n\n### Unassigned Open Issues\nTable: Key (linked) | Summary | Priority | Issue Type\nFlag Blockers and Criticals explicitly.\n\n### Recently Updated\nTable: Key (linked) | Summary | Updated | By\n\n---\n\n## Mode: `me`\n\nWork assigned to the person running this skill. Ask: \"What is your Jira email or display name?\" — use their response to filter.\n\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory != Done ORDER BY updated DESC\n```\n\nAlso fetch recently completed by the user (for summary context):\n```jql\nproject = STOR AND assignee = \"\" AND statusCategory = Done AND resolutiondate >= -30d ORDER BY resolutiondate DESC\n```\n\n**Output:**\n\n### Your Active Work\nTable: Key (linked) | Summary | Status | Priority | Last Updated\nHighlight anything In Progress that hasn't been updated in > 3 days.\n\n### Your Recent Completions (Last 30 Days)\nTable: Key (linked) | Summary | Resolved On | Issue Type\n\n### Work Summary\nA plain-English paragraph for each active item: what it is, where it stands, and whether anything is blocking it. Lead with the highest-priority item.\n\n---\n\n## Mode: `epic `\n\nRetrieve a specific epic and all its child issues.\n\nSteps:\n1. Call `mcp__atlassian__getJiraIssue` with the provided KEY to get the epic details.\n2. Fetch all children:\n ```jql\n project = STOR AND parent = ORDER BY priority DESC, status ASC\n ```\n3. Also fetch linked issues (blocks/is blocked by) on the epic itself.\n\n**Output:**\n\n### Epic: [STOR-####](JIRA_BASE/browse/STOR-####) — {Epic Summary}\n**Status:** {status} | **Assignee:** {assignee or Unassigned} | **Priority:** {priority}\n\n**Description:**\n{epic description, condensed to key intent — 2–4 sentences max}\n\n### Child Issues\nTable: Key (linked) | Summary | Status | Assignee | Priority | Issue Type\nGroup: In Progress → To Do → Done (collapsed count for Done if > 5)\n\n### Epic Progress\n- Total issues: N\n- Done: N (N%)\n- In Progress: N\n- Blocked: N (flag if any)\n\n### Blockers & Dependencies\nList any linked \"blocks\" or \"is blocked by\" relationships. If none, omit this section.\n\n---\n\n## Mode: `epics`\n\nList all open epics in the STOR project with their child counts and progress.\n\n```jql\nproject = STOR AND issuetype = Epic AND statusCategory != Done ORDER BY priority DESC, updated DESC\n```\n\nFor each epic, fetch child counts via:\n```jql\nproject = STOR AND parent = \n```\n(batch these — fetch all children in one query with `parent in (KEY1, KEY2, ...)` if the tool supports it, otherwise query each epic individually)\n\n**Output:**\n\n### Open Epics\n\nFor each epic:\n**[STOR-####](link) — {Summary}**\nStatus: {status} | Assignee: {assignee} | Children: {done}/{total}\nOne-line description of the epic's goal (from its description field, condensed).\n\n---\n\n## Mode: `sprint`\n\nCurrent sprint state focused on progress and blockers — not health framing, just what is happening.\n\n```jql\nproject = STOR AND sprint in openSprints() ORDER BY statusCategory ASC, priority DESC\n```\n\n**Output:**\n\n### Sprint: {Sprint Name}\nDates: {start} → {end} | Days remaining: N\n\n**In Progress** (N issues)\nTable: Key (linked) | Summary | Assignee | Days In Progress\n\n**To Do** (N issues)\nTable: Key (linked) | Summary | Assignee | Priority\n\n**Done** (N issues)\nCount only unless ≤ 5 — then list.\n\n**Blockers in sprint:**\nList any issue linked as \"blocks\" another sprint issue, or with priority Blocker.\n\n---\n\n## Mode: `summary `\n\nGenerate a structured summary suitable for a status update, PR description, or stakeholder report. The argument can be:\n- A single issue key: `summary STOR-1234`\n- An epic key: `summary STOR-100` (auto-detects if it's an epic)\n- A freeform scope description: `summary tokenserver auth work` (runs JQL to find matching issues)\n- `summary me` — summarize the current user's recent work (ask for their name if not known)\n- `summary sprint` — summarize the current sprint's work\n\n**Steps:**\n1. Resolve the scope to a set of issues. Fetch full details for each.\n2. For epics: fetch children too.\n3. Identify: what was done, what is in progress, what is blocked.\n4. Synthesize the summary.\n\n**Output format — always structured as:**\n\n### Impact\nWhat this work changes for users or the system. Lead with the most significant outcome. Use concrete language: \"reduces\", \"enables\", \"removes\", \"fixes\". If there are multiple issues, consolidate into the top 2–3 outcomes. Link relevant issues inline: e.g. \"Adds PostgreSQL support for tokenserver ([STOR-42](link)).\"\n\n### Context\nWhy this work was needed. Background the reader should have to understand the impact. Reference the driving requirement, bug, or design decision. Keep to 3–5 sentences. Link to the primary issue or epic.\n\n### Highlights\nNotable implementation decisions, technical wins, or interesting constraints resolved. Bullet points. Each item links to the relevant issue. Focus on what would surprise or interest a fellow engineer — not just a list of what was done.\n\n---\n\n## Mode: `create `\n\nCreate a new Jira issue in STOR from natural language.\n\n**Steps:**\n\n1. Parse the description to infer:\n - **Summary** — one clear sentence (≤ 100 chars)\n - **Issue type** — Bug / Task / Story / Spike (infer from language; ask if ambiguous)\n - **Priority** — infer from language (\"critical\", \"blocking\" → Critical; \"nice to have\" → Minor; default → Major). Valid STOR priorities: `Blocker`, `Critical`, `Major`, `Minor`. `Trivial` is rejected by the project — do not use it.\n - **Work category** — `Feature Engineering` / `Engineering Excellence` / `Operational Excellence`. See [Work categories](#work-categories) below for definitions and inference rules. Always classify; if unsure between EE and OpEx, the planned/reactive distinction decides it.\n - **Assignee** — if this `create` invocation is part of the same workflow that opens a GitHub PR for the work, set the assignee to the current user. Look up their accountId via `mcp__atlassian__lookupJiraAccountId` using the email from the runtime `userEmail` context. Otherwise leave unassigned unless the user named a specific person.\n - **Parent** — if the description references an epic (e.g. \"under STOR-506\", \"in STOR-506\", \"linked to \"), set `parent` to that key when calling `createJiraIssue`.\n - **Description body** — expand the user's language into a structured description, leading with the category:\n - **Category:** {one of the three categories} — *one-sentence rationale*\n - **Background:** why this is needed\n - **Acceptance Criteria:** observable, testable outcomes\n - **Notes:** any constraints or references mentioned\n\n2. Show the draft to the user before creating:\n ```\n Ready to create:\n Type: {type}\n Project: STOR\n Parent: {parent key or none}\n Priority: {priority}\n Category: {category}\n Assignee: {display name or \"unassigned\"}\n Summary: {summary}\n Description:\n {description}\n\n Create this issue? (yes / edit first / cancel)\n ```\n\n3. On confirmation, call `mcp__atlassian__createJiraIssue` with:\n - `projectKey: STOR`\n - `issueTypeName: {type}`\n - `summary: {summary}`\n - `description: {description}` with `contentFormat: markdown`\n - `additional_fields: { \"priority\": { \"name\": \"{priority}\" } }`\n - `assignee_account_id: {accountId}` — only when auto-assignment applies or the user named an assignee\n - `parent: {epic_key}` — only when a parent epic was identified\n\n4. Return the created issue key and direct link: `[STOR-####](JIRA_BASE/browse/STOR-####)`\n\n---\n\n## Mode: `update `\n\nModify an existing issue. Changes can be natural language.\n\n**Steps:**\n\n1. Fetch the current issue with `mcp__atlassian__getJiraIssue`.\n\n2. Parse the `` to determine what to modify:\n - Field updates (summary, description, priority, assignee) → use `mcp__atlassian__editJiraIssue`\n - Status transition (\"mark as done\", \"move to in progress\") → use `mcp__atlassian__getTransitionsForJiraIssue` then `mcp__atlassian__transitionJiraIssue`\n - Comment (\"add a comment: ...\") → use `mcp__atlassian__addCommentToJiraIssue`\n\n3. Show the proposed changes before applying:\n ```\n Updating STOR-####: {current summary}\n Changes:\n - {field}: \"{old value}\" → \"{new value}\"\n - {transition}: {new status}\n - Comment: \"{comment text}\"\n\n Apply? (yes / edit / cancel)\n ```\n\n4. Apply on confirmation. Return confirmation with the issue link.\n\n---\n\n## General guidelines\n\n- **All issue references must be linked**: format as `[STOR-####](JIRA_BASE/browse/STOR-####)` — never bare key text\n- **Summaries lead with impact**: what changes for users or the system, not what was done mechanically\n- **Do not fabricate data**: if a query returns nothing, say so plainly\n- **Ask once for missing identity**: if the user's Jira name is needed and not known, ask once; do not repeat the question across modes in the same session\n- **Confirm before write operations**: always show a draft before calling `createJiraIssue`, `editJiraIssue`, `transitionJiraIssue`, or `addCommentToJiraIssue`\n- **Permissions**: create/edit/transition operations require the user's Jira account to have project contributor access to STOR; if a call fails with 403, report it clearly and do not retry\n- **Issue type inference**: \"bug\" → Bug; \"task\"/\"chore\" → Task; \"feature\"/\"story\"/\"as a user\" → Story; \"investigate\"/\"explore\"/\"spike\" → Spike\n- **Classify every ticket**: when creating or summarizing, identify the [work category](#work-categories) (Feature Engineering / Engineering Excellence / Operational Excellence). Include it as the leading line of the description body for created tickets.\n- **Auto-assign when PR-bundled**: if a `create` invocation is part of the same workflow that opens a GitHub PR for the same work, set the assignee to the current user (look up their accountId with `mcp__atlassian__lookupJiraAccountId`). Setting an assignee may transition the issue to In Progress via Jira workflow rules — that's expected.\n\n---\n\n## Work categories\n\nPer [Feature Engineering, Engineering Excellence, and Operational Excellence](https://mozilla-hub.atlassian.net/wiki/spaces/IP/pages/1397817345). Classify every STOR ticket into one of three categories. Target distribution per quarter: ≤ 50% Feature Engineering, ≤ 30% Engineering Excellence, ≤ 20% Operational Excellence.\n\n### Feature Engineering — *build & deliver new things*\n\nPlanned work that changes what a customer (internal or external) sees beyond speed, reliability, or bugfixes.\n\nExamples: new API endpoints, new auth methods (passkeys), launching a region or platform, UI/UX redesigns, A/B testing, expanded user-configuration options, changes to core business logic, bug fixes against unlaunched feature work.\n\n### Engineering Excellence — *sustain & improve velocity*\n\nPlanned work that improves code maintainability, reduces tech debt, or boosts engineering velocity. Most non-hotfix bug fixes land here. \"Keeping the lights on\" proactively.\n\nExamples: refactors, test-coverage improvements, observability and monitoring, CI/CD tooling, dependency upgrades, removing dead code, fixing flaky tests, reducing on-call toil, automating manual deployments, fixes for issues in already-released features.\n\n### Operational Excellence — *react & keep production healthy*\n\nUnplanned work. Reliability, stability, security, and operational efficiency in the production environment. \"Keeping the lights on\" reactively.\n\nExamples: security patches and CVE remediations, manual scaling under load, incident response and postmortems, hotfix patches for production incidents, resolving performance issues impacting SLAs.\n\n### Inference rules\n\n- **Planned vs. unplanned** decides EE vs. OpEx: if it was on a roadmap/sprint plan, it's EE; if it interrupted planned work to keep prod healthy, it's OpEx.\n- **Customer-visible change** decides FE vs. EE: if the customer sees something new (beyond speed/reliability/bugfix), it's FE.\n- **Dependency upgrades, CI/CD tooling, refactors, test improvements** → Engineering Excellence (unless the upgrade is a CVE remediation, then OpEx).\n- **Security patches and CVEs** → Operational Excellence.\n- **Bug fixes**: pre-launch → FE; against released features, planned → EE; via hotfix → OpEx.\n\n\n---\nname: multi-backend-consistency\ndescription: Checks that trait method changes in syncstorage-db or tokenserver-db are consistently implemented across all backend crates (mysql, postgres, spanner). Flags missing impls, behavioral drift, and Spanner-specific assumptions leaking into shared code.\nuser-invocable: true\n---\n\n# Multi-Backend Consistency Check\n\nYou are a Rust trait/implementation consistency reviewer for syncstorage-rs. Changes to shared traits must be reflected in every backend. Your job is to find gaps before they reach CI.\n\n## Backend map\n\n```\nsyncstorage-db/ ← trait definitions (DbPool, Db, etc.)\n → syncstorage-mysql/\n → syncstorage-postgres/\n → syncstorage-spanner/\n\ntokenserver-db/ ← trait definitions\n → tokenserver-mysql/\n → tokenserver-postgres/\n```\n\n## Step 1 — Find changed trait files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-db/|tokenserver-db/\"\n```\n\nIf no trait files changed, check whether any backend implementation changed without a corresponding trait change — that may indicate the trait was bypassed or duplicated.\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"syncstorage-(mysql|postgres|spanner)/|tokenserver-(mysql|postgres)/\"\n```\n\n## Step 2 — Diff the traits\n\nFor each changed trait file:\n\n```bash\ngit diff main...HEAD -- \n```\n\nList every added, removed, or modified method signature.\n\n## Step 3 — Check each backend for the same change\n\nFor each modified method, grep for its name in every backend crate:\n\n```bash\ngrep -rn \"fn \" syncstorage-mysql/src/ syncstorage-postgres/src/ syncstorage-spanner/src/\ngrep -rn \"fn \" tokenserver-mysql/src/ tokenserver-postgres/src/\n```\n\nFor each backend, determine:\n- **Present and updated** — matches the new signature ✓\n- **Present but stale** — exists but hasn't been updated to match ✗\n- **Missing** — not implemented at all ✗\n\n## Step 4 — Check for cross-backend behavioral assumptions\n\nReview the diff for patterns that are safe on one backend but wrong on another:\n\n- **Auto-increment IDs** — MySQL/Postgres use serial/auto-increment; Spanner uses explicit INT64. Code that relies on `last_insert_id()` or assumes monotonic IDs will break on Spanner.\n- **Transaction semantics** — Spanner transactions are bounded; MySQL/Postgres allow longer-lived transactions. Check for assumptions about transaction scope.\n- **NULL handling** — Spanner is stricter about NULLs in some column types than MySQL.\n- **RETURNING clause** — Postgres supports `RETURNING id`; MySQL uses `last_insert_id()`; Spanner uses a separate read. Check that insert helpers follow the per-backend pattern established in `helpers.py` / existing db impls.\n- **Timestamp precision** — Spanner uses microseconds; MySQL DATETIME has second precision unless explicitly specified.\n- **Feature flags** — confirm the change compiles under `--features=syncstorage-db/mysql`, `--features=syncstorage-db/postgres`, and `--features=syncstorage-db/spanner` independently.\n\n## Step 5 — Check shared db-common crates\n\nChanges to `syncstorage-db-common/` or `tokenserver-db-common/` affect all backends. Verify nothing in those changes makes backend-specific assumptions.\n\n```bash\ngit diff main...HEAD --name-only | grep \"db-common\"\n```\n\n## Output format\n\n**Trait changes detected:**\nList each modified method with its new signature.\n\n**Backend coverage:**\n\n| Method | mysql | postgres | spanner |\n|---|---|---|---|\n| `method_name` | ✓ updated / ✗ stale / ✗ missing | ... | ... |\n\n**Behavioral drift warnings:**\nAny cross-backend assumptions found, with file:line.\n\n**Verdict:** All backends consistent / N gaps found\n\n\n---\nname: pytest-fixture-review\ndescription: Reviews conftest.py and helpers.py files in tools/integration_tests/ against the repo's fixture/helper separation standard. Flags helpers in conftest, missing teardown, wrong scopes, and raw DB operations outside helper functions.\nuser-invocable: true\n---\n\n# Pytest Fixture Review\n\nYou are a pytest code reviewer for syncstorage-rs integration tests. This repo has a deliberate structure: `conftest.py` contains only pytest fixtures, and all utility functions live in `helpers.py`. Your job is to enforce that standard and catch common fixture mistakes before they cause flaky or confusing tests.\n\n## The standard\n\n```\ntools/integration_tests/\n conftest.py ← fixtures ONLY (st_ctx)\n helpers.py ← retry helpers, auth state, switch_user, constants\n\ntools/integration_tests/tokenserver/\n conftest.py ← fixtures ONLY (ts_db_conn, ts_app, ts_service_id, ts_ctx, fxa_auth)\n helpers.py ← DB helpers, auth helpers, node/user/service helpers\n```\n\nFixtures yield context dicts. Helpers are plain functions imported directly by test files.\n\n## Step 1 — Find changed test files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"tools/integration_tests\"\n```\n\nRead each changed file in full.\n\n## Step 2 — conftest.py checks\n\nFor each `conftest.py` in the diff:\n\n**Non-fixture code in conftest:**\n- Are there any module-level functions that are not decorated with `@pytest.fixture`?\n- Are there utility functions, helper classes, constants, or context managers defined in conftest that belong in `helpers.py`?\n- Exception: module-level side effects that must run at import time (e.g. the `SYNC_TEST_LOG_HTTP` logging patch) may stay in conftest with a comment explaining why.\n\n**Fixture scopes:**\n- `scope=\"function\"` — default, correct for most fixtures; state is reset per test\n- `scope=\"session\"` — only justified for expensive external calls (e.g. `fxa_auth` creates a real FxA account). Any new session-scoped fixture must have a comment explaining why.\n- `scope=\"module\"` or `scope=\"class\"` — flag for review; not currently used in this repo\n\n**Teardown:**\n- Does every fixture that creates state also clean it up?\n- `ts_ctx` pattern: `clear_db()` before yield AND after yield. Verify both sides are present.\n- `fxa_auth` pattern: `acct.clear()` + `client.destroy_account()` in teardown. Verify error handling wraps the teardown.\n- `st_ctx` pattern: `config.end()` + `del os.environ[\"MOZSVC_UUID\"]`. Verify both.\n\n**Database connections:**\n- `ts_db_conn` uses `AUTOCOMMIT` isolation. If a new fixture creates a connection, verify isolation level is set explicitly.\n- Connections must be closed in teardown (`conn.close()`).\n\n## Step 3 — helpers.py checks\n\n**No fixtures in helpers:**\n- Are there any functions decorated with `@pytest.fixture` in `helpers.py`? These must move to `conftest.py`.\n\n**Raw SQL outside helper functions:**\n- Is `sqltext()` / `execute_sql()` called directly in a test file rather than through a helper?\n- Direct SQL in tests bypasses the helper layer and makes tests brittle. Flag with file:line.\n\n**Imports:**\n- Are helpers importing from `conftest`? This creates a circular dependency risk. Helpers should have no pytest imports and no conftest imports.\n\n**Constants:**\n- Are test constants (NODE_ID, NODE_URL, FXA_EMAIL_DOMAIN, etc.) defined in helpers, not scattered in test files or conftest?\n\n## Step 4 — Test file import checks\n\nFor each changed test file:\n\n```bash\ngrep -n \"from.*conftest import\" \n```\n\nTest files should import helpers from `helpers.py`, not from `conftest.py`. Any `from tools.integration_tests.conftest import` or `from integration_tests.tokenserver.conftest import` is wrong — flag it with the correct import path.\n\n## Step 5 — New fixture pattern check\n\nIf a new fixture was added, verify:\n- It follows the `yield`-based teardown pattern (not `return`)\n- It is typed or documented enough that its contents are clear\n- If it seeds database state, it cleans up after itself\n- Its scope is the narrowest possible for its purpose\n\n## Output format\n\n**conftest.py issues:**\nList each violation with file:line and what needs to move or change.\n\n**helpers.py issues:**\nList each violation with file:line.\n\n**Import violations:**\nList any test files importing from conftest instead of helpers.\n\n**New fixtures:**\nFor each new fixture: name, scope, teardown present (yes/no), any concerns.\n\n**Verdict:** Follows conventions / N issues found\n\n\n---\nname: rust-code-smell\ndescription: Runs clippy, cargo audit, and all relevant multi-backend Makefile targets for files changed in the current branch. Detects code smells, anti-patterns specific to this codebase, and surfaces issues across every affected backend before CI catches them. Also handles dependency-update verification (Cargo.toml / Cargo.lock changes from dependabot or manual bumps) via the `deps` mode.\nuser-invocable: true\n---\n\n# Rust Code Smell Detector\n\nYou are a Rust code quality reviewer for syncstorage-rs. Your job is to run the full set of mechanical checks relevant to the current branch's changes, then layer on a manual smell pass for patterns that tooling misses. Run checks for every backend touched by the diff — not just the default.\n\n## Standing authorization\n\nWhen this skill is invoked, you have standing authorization to run the following without per-command confirmation:\n\n- `cargo fmt -- --check`\n- `cargo clippy ...` (any feature combination)\n- `make clippy_mysql`, `make clippy_postgres`, `make clippy_spanner`\n- `make clippy_release_mysql`, `make clippy_release_postgres`, `make clippy_release_spanner`\n- `cargo audit`\n- `cargo check --workspace ...`\n- `cargo test --no-run` (compile-check only — do NOT run actual tests without explicit permission, since they require DB setup)\n- `cargo update --precise -p ` (lockfile-only adjustments during bisection)\n- `git stash` / `git stash pop` (for baseline comparison)\n\nDo not run anything destructive (commit, push, rebase, branch deletion, `cargo clean` of the whole workspace, `make run_*`, docker-compose up/down) without asking.\n\n## Mode selection\n\n- **Default mode** — branch under review contains Rust source changes (not just `Cargo.toml`/`Cargo.lock`). Run Steps 1–5 below.\n- **`deps` mode** — branch only changes `Cargo.toml` / `Cargo.lock`, or the user invokes the skill in the context of a dependabot PR / dependency bump. Jump to the \"Dependency update verification\" section near the end and follow that flow instead.\n\n## Step 1 — Identify which backends are affected\n\n```bash\ngit diff main...HEAD --name-only\n```\n\nDetermine which backends the changes touch:\n\n- Files in `syncstorage-mysql/`, `tokenserver-mysql/`, or `syncstorage-db/` with MySQL defaults → **mysql**\n- Files in `syncstorage-postgres/`, `tokenserver-postgres/` → **postgres**\n- Files in `syncstorage-spanner/` → **spanner**\n- Files in `syncserver/`, `syncserver-common/`, `syncserver-settings/`, `syncstorage-db-common/`, `tokenserver-auth/`, `tokenserver-common/` → **all three backends**\n\n## Step 2 — Format check\n\n```bash\ncargo fmt -- --check\n```\n\nIf formatting issues exist, stop and report them. Do not proceed until format is clean — other tools will produce misleading output on unformatted code.\n\n## Step 3 — Run Clippy for each affected backend\n\nRun only the backends identified in Step 1. Use the exact Makefile targets:\n\n```bash\nmake clippy_mysql # if mysql affected\nmake clippy_postgres # if postgres affected\nmake clippy_spanner # if spanner affected\n```\n\nThen run release-mode Clippy for each affected backend (catches dead code, unused imports with optimizations, and issues only visible at `--release`):\n\n```bash\nmake clippy_release_mysql # if mysql affected\nmake clippy_release_postgres # if postgres affected\nmake clippy_release_spanner # if spanner affected\n```\n\nCollect all warnings and errors. Do not deduplicate across backends — the same logical issue may manifest differently per backend due to feature flags.\n\n## Step 4 — Security audit\n\n```bash\ncargo audit\n```\n\nFlag any **critical** or **high** severity advisories as blockers. List **medium** as warnings. Note the affected crate and advisory ID for each.\n\n## Step 5 — Manual smell pass\n\nRead the diff:\n\n```bash\ngit diff main...HEAD\n```\n\nCheck for the following patterns that Clippy does not catch:\n\n### Async/blocking\n- `std::thread::sleep` inside an async function — use `tokio::time::sleep`\n- Blocking DB calls inside async handlers without `spawn_blocking`\n- `.unwrap()` or `.expect()` on `Result`/`Option` in production paths (test code is fine)\n\n### gRPC / Spanner-specific\n- `grpcio::RpcStatus` errors caught and silently dropped without either Sentry classification or metric emission — see `is_sentry_event()` / `is_ignored_internal()` pattern\n- Retry logic that doesn't cap attempts or uses `std::thread::sleep` instead of exponential backoff\n- Spanner mutations not batched where they could be\n\n### Error handling\n- `map_err(|_| ...)` that discards the original error without logging — information loss\n- `unwrap()` on lock acquisition (implies the lock is poisonable)\n- New `From` impls that convert specific errors into generic `InternalError` without preserving context\n\n### Trait object vs generic\n- `Box` or `Arc` allocations in hot paths where a generic would avoid the vtable overhead\n- Inconsistency with how the rest of the codebase handles the same trait (check existing usage)\n\n### Diesel / query patterns\n- N+1 query patterns: loops that issue individual DB calls instead of a single query\n- Missing index hints or queries on unindexed columns in high-traffic paths\n- Raw string SQL (`sqltext(...)`) where a typed Diesel query builder already exists\n\n### Configuration / secrets\n- Secrets or credentials hardcoded or defaulting to non-placeholder values outside of test modules\n- Environment variable reads outside of the settings structs (should go through `syncserver-settings`)\n\n### Feature flag hygiene\n- `#[cfg(feature = \"...\")]` on items that should be available to all backends\n- Code that compiles under one feature but silently no-ops under another without a clear comment\n\n## Output format\n\n**Format:** pass / fail (list files)\n\n**Clippy results per backend:**\n\n| Backend | Debug | Release | Issues |\n|---|---|---|---|\n| mysql | pass/fail | pass/fail | N warnings/errors |\n| postgres | pass/fail | pass/fail | |\n| spanner | pass/fail | pass/fail | |\n\n**Audit:** pass / N advisories (list crate, severity, advisory ID)\n\n**Manual smells found:**\nList each with file:line, smell type, and a one-line explanation.\n\n**Verdict:** Clean / N issues need attention — list blockers vs warnings separately.\n\n---\n\n# Dependency update verification (`deps` mode)\n\nUse this flow when the branch only changes `Cargo.toml` / `Cargo.lock`, the user is evaluating a dependabot PR, or the user is splitting a grouped dependency PR into smaller pieces.\n\n## When to enter this mode\n\n- User invokes the skill on a dependabot PR (e.g. mozilla-services/syncstorage-rs#NNNN).\n- `git diff main...HEAD --name-only` returns only `Cargo.toml` and/or `Cargo.lock`.\n- User explicitly says \"check this dep bump\" / \"verify the cargo updates\" / \"is this PR safe\".\n\n## Step D1 — Establish the baseline\n\nThe single most valuable signal is whether the gates already pass on `main` before the bump. Stash, test, restore:\n\n```bash\ngit stash\nmake clippy_mysql 2>&1 | tail -10\ngit stash pop\n```\n\nIf `main` already fails a gate, stop and surface that to the user — the bump did not introduce the failure and pretending otherwise wastes time.\n\n## Step D2 — Classify the bumps\n\nCategorize every changed crate before running anything else. Read the manifest diff plus dependabot's release notes (in the PR body) and assign each crate to one of:\n\n- **Routine** — patch/minor bumps to leaf-ish deps (`env_logger`, `reqwest`, `utoipa`, `uuid`, `cadence`, `jsonwebtoken`, `config`, `pyo3` patches). Group these.\n- **Ecosystem-coupled** — crates whose versions must move together. In this repo:\n - `diesel-async` and `deadpool` are paired — Cargo.toml comments already say so.\n - `diesel_migrations` patch bumps can interact with `diesel-async`'s major version (a 2.3.2 bump broke `IntoUpdateTarget` trait selection against diesel-async 0.7; the same bump composes fine with diesel-async 0.9). Treat `diesel_migrations` as ecosystem-coupled even though it looks like a patch.\n - `protobuf` for the Spanner backend must match what `google-cloud-rust-raw` resolves. Do not bump `syncstorage-spanner`'s pinned `protobuf` ahead of `google-cloud-rust-raw`.\n- **Major / breaking** — any `x.0 → y.0` or `0.x → 0.y` bump on a direct dep. Flag for human review even if compilation passes.\n\nRecommend splitting a grouped dependabot PR along these lines: one PR per category, with the riskier categories held until their constraints clear.\n\n## Step D3 — Apply bumps surgically\n\nWhen applying a chosen group, use `--precise` per crate so transitive deps don't drift:\n\n```bash\ncargo update --precise -p \n```\n\nDo not run a bare `cargo update -p ` — that will pull in unrelated transitive bumps (e.g. `hashlink 0.10 → 0.11`, `anstream 0.6 → 1.0`) and you'll then have to debug whether those broke something. `--precise` keeps the diff to exactly what was requested.\n\nCargo only accepts one `--precise` per invocation. To apply N crates, run N sequential `cargo update --precise` commands.\n\nIf two crates share a version qualifier (`reqwest@0.13.2`), cargo will demand the disambiguator — use the form it suggests in the error.\n\nAfter every cargo-update batch, run:\n\n```bash\ngit diff origin/master -- Cargo.lock | grep -E '^[-+]version = ' | sort | uniq -c | sort -rn\n```\n\nVerify the version-change set matches the intended group. Anything you did not request showing up is a red flag.\n\n## Step D4 — Run the gates\n\nRun, in order, stopping at the first failure:\n\n```bash\ncargo fmt -- --check\ncargo audit\nmake clippy_mysql\nmake clippy_postgres\nmake clippy_spanner\ncargo test --workspace --no-run # compile-check only\n```\n\n`make test` (running the suite) requires a real MySQL on `localhost:3306` with `sample_user:sample_password` and the schemas pre-created. If the user's environment doesn't have it, report that explicitly and recommend `make docker_run_mysql_e2e_tests` / `make docker_run_postgres_e2e_tests` for full verification. Do not attempt to set up the DB.\n\n`cargo test --no-run` also requires `libpython3.9` linkable on disk because `pyo3` is a default-feature dep. If that fails at the link stage with `library 'python3.9' not found`, report it as a local env issue, not a bump regression. Confirm by running the same `--no-run` against the baseline (Step D1) — if it fails there too, it's not your problem.\n\n## Step D5 — Bisect on failure\n\nIf a gate fails, do not start guessing. Bisect:\n\n1. Revert the *most likely* single crate to its pre-bump version with `cargo update --precise -p `.\n2. Re-run only the failing gate. If it now passes, that crate is the culprit. If not, restore it and try the next candidate.\n3. Likely-culprit ordering for diesel/diesel-async-related failures: `diesel_migrations` first, then `uuid` (it's a diesel feature type), then anything else that touches DB.\n4. For trait-selection / type-inference errors in `syncstorage-postgres` or `syncstorage-mysql` under a *different* backend's clippy target (e.g. clippy_mysql failing in syncstorage-postgres source), the failure is almost always an ecosystem-coupled crate moving alone. Re-classify and move it to the appropriate group.\n\nWhen a crate is identified as incompatible with the current group, remove it from the group, return it to the manifest's previous version, and update the Jira ticket (or PR description) to note it's been moved to whichever group it now belongs with.\n\n## Step D6 — Report\n\nOutput one table:\n\n| Crate | Old | New | Status |\n|---|---|---|---|\n| ... | ... | ... | applied / held / failed / moved to Group X |\n\nPlus the verification matrix:\n\n| Gate | Result | Notes |\n|---|---|---|\n| `cargo fmt --check` | pass/fail | |\n| `cargo audit` | pass/fail | N advisories listed if any |\n| `make clippy_mysql` | pass/fail | |\n| `make clippy_postgres` | pass/fail | |\n| `make clippy_spanner` | pass/fail | |\n| `cargo test --no-run` | pass/fail/env-skip | note libpython if env-skipped |\n\nEnd with one paragraph: what landed, what was excluded and why, and what should ride with the next group.\n\n## Known coupling notes for this repo\n\nKeep these in mind when classifying bumps — they cost real time to rediscover:\n\n- **`diesel-async` + `deadpool`** are paired. Cargo.toml comments say so. Do not move one without the other.\n- **`diesel_migrations` 2.3.2** breaks `IntoUpdateTarget` trait selection in `syncstorage-postgres` source when paired with `diesel-async 0.7`. It needs to ride with the `diesel-async 0.9` upgrade.\n- **`protobuf`** is pinned with `=` in `syncstorage-spanner/Cargo.toml` deliberately. The pin must match what `google-cloud-rust-raw` resolves to. Currently both should be `2.28.0`. Bumping the spanner pin ahead of `google-cloud-rust-raw` will cause two protobuf majors in the lockfile and break compilation (RepeatedField doesn't exist in protobuf 3.x).\n- **`make clippy_mysql` / `clippy_postgres` / `clippy_spanner`** all use `--workspace --all-targets` — every backend crate compiles under every feature set. A failure in `syncstorage-postgres` source under `clippy_mysql` is not a misconfiguration; it means a workspace-shared dep changed something diesel-postgres can't tolerate.\n- **`pyo3` + `libpython3.9` link** is required for default-feature builds on this repo. Local `cargo test --no-run` will fail at link without it. Docker e2e is the fallback for full test verification.\n\n\n---\nname: rust-error-review\ndescription: Reviews new or modified error variants in syncstorage-rs for correct Sentry routing, HTTP status mapping, and response safety. Enforces the is_sentry_event() / maybe_emit_metrics() classification pattern.\nuser-invocable: true\n---\n\n# Rust Error Review\n\nYou are a Rust error handling reviewer for syncstorage-rs. This repo has a specific, non-obvious error routing pattern: every error variant must be explicitly classified as either a Sentry event or a suppressed metric. Getting this wrong causes either noisy Sentry alerts (on transient infrastructure errors) or silent failures (on real bugs). Your job is to catch misclassification before merge.\n\n## The pattern\n\n```\nSyncstorageError / DbError\n → is_sentry_event() → true → sent to Sentry\n → is_sentry_event() → false → maybe_emit_metrics() → StatsD counter\n```\n\nKey files:\n- `syncstorage-spanner/src/error.rs` — `DbError`, `DbErrorKind`, `is_sentry_event()`, `is_ignored_internal()`\n- `syncstorage-db-common/src/error.rs` — shared error types\n- `syncserver/src/error.rs` — top-level `ApiError`, HTTP response mapping\n- `tokenserver-common/src/error.rs` — tokenserver error types\n\n## Step 1 — Find changed error files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"error\\.rs\"\n```\n\nAlso check for new `From<>` impls that convert external errors into repo error types:\n\n```bash\ngit diff main...HEAD | grep -E \"impl From|Into<.*Error\"\n```\n\n## Step 2 — Diff each error file\n\n```bash\ngit diff main...HEAD -- \n```\n\n## Step 3 — Review checklist\n\nFor each new or modified error variant:\n\n### Sentry classification\n- Is there a new arm in `is_sentry_event()` for this variant?\n- Is the classification correct?\n - **Should be Sentry:** logic bugs, unexpected states, data integrity issues, auth failures that indicate an attack, panics\n - **Should be metric only:** transient infrastructure errors (gRPC RST_STREAM, connection timeouts, 503s from upstream), expected client errors (404, 409 conflict, quota exceeded)\n- For `INTERNAL` gRPC errors: is `is_ignored_internal()` doing substring matching (`.iter().any(|s| msg.contains(s))`)? Exact equality will miss real RST_STREAM messages.\n\n### HTTP status mapping\n- Does the new variant map to an appropriate HTTP status in the `From for ApiError` or `status_code()` impl?\n- Are client errors (4xx) distinguished from server errors (5xx)?\n- Is a 500 being returned for something that should be a 409 or 404?\n\n### Response body safety\n- Does the error's `Display` or serialization leak internal state (SQL query text, file paths, internal IDs, stack traces)?\n- Check `ApiError`'s response serialization — internal error messages must not reach the client.\n\n### Backtrace\n- Is `capture_backtrace()` called for new unexpected/internal error variants?\n- Is it *not* called for expected operational errors (quota, not found, conflict)?\n\n### Metric naming\n- If the variant routes to `maybe_emit_metrics()`, is the metric name consistent with existing conventions (`storage.spanner.grpc.internal`, etc.)?\n\n## Step 4 — Check `From` impl chains\n\nNew error types often get wrapped through multiple `From` impls before reaching the HTTP layer. Trace the chain to verify classification survives the conversion:\n\n```bash\ngrep -rn \"From<\" syncstorage-spanner/src/ syncstorage-db/src/ syncserver/src/ | grep -i error\n```\n\n## Output format\n\nFor each changed error variant:\n- **Variant:** name\n- **Classification:** Sentry / Metric / Unclear\n- **HTTP status:** correct / wrong (should be X) / missing\n- **Response safety:** safe / leaks internal detail at file:line\n- **Issues:** list\n\nEnd with **All error variants correctly classified** or **N classification issues found**.\n\n\n---\nname: sentry-health\ndescription: Queries Sentry for syncstorage-rs (project syncstorage-prod) to surface error trends, new issues, regressions, and production health signals. Reports as engineer/manager view of what requires attention.\nuser-invocable: true\nargument-hint: \"new | regressions | volume | full (default: full)\"\n---\n\nYou are acting as **both engineer and engineering manager** for syncstorage-rs. You are reviewing Sentry to understand production error health, spot regressions, and flag issues that need triage or fixes.\n\nStart by reading `.claude/mcp-context.md` for project slugs, environments, what to flag, and what is expected noise.\n\nParse `$ARGUMENTS` (trimmed, lowercased). Default is `full`. Valid modes:\n- `new` — issues first seen in the last 7 days\n- `regressions` — previously resolved issues that have recurred\n- `volume` — high-event-count issues\n- `full` — all of the above\n\n---\n\n## Step 1: Confirm the project\n\nCall `mcp__sentry__find_projects` and locate the `syncstorage-prod` project under the `mozilla` organization. Confirm the project slug and organization slug before querying. If the project is not found, report the available projects and stop.\n\n---\n\n## Step 2: Gather issue data\n\nUse available Sentry MCP tools to fetch issues for `syncstorage-prod`. Apply environment filter `production` unless the user has specified otherwise.\n\nFetch based on the selected mode:\n\n### New issues (`new`, `full`)\nIssues with `firstSeen` in the last 7 days that have not been resolved. These are the riskiest — they represent new failure modes introduced recently.\n\nSort by: event count descending.\n\nFor each, capture:\n- Issue ID and title\n- First seen timestamp\n- Event count in the period\n- Affected users (if available)\n- Platform / culprit (the Rust file/function if available)\n\n### Regressions (`regressions`, `full`)\nIssues that were previously marked resolved but have reoccurred. These indicate incomplete fixes or flaky patches.\n\nFor each, capture:\n- Issue ID and title\n- Original resolution date\n- Recurrence date\n- Event count since recurrence\n\n### High-volume issues (`volume`, `full`)\nIssues with the most events in the last 24 hours, regardless of age. These are the most actively impacting users.\n\nFor each, capture:\n- Issue ID and title\n- Event count (24h)\n- Is this a known/triaged issue or unacknowledged?\n\n---\n\n## Step 3: Classify each issue\n\nFor every issue surfaced, classify it against the known patterns in `.claude/mcp-context.md`:\n\n**Tokenserver auth path** — involves `hawk_identifier`, FxA OAuth, `tokenserver-auth/`, `client_state`, `keys_changed_at`. These are user-facing auth failures — high priority.\n\n**Database layer** — involves `DbError`, `diesel`, `deadpool`, connection pool, migration, query timeout. May indicate infrastructure problems.\n\n**Syncstorage storage path** — involves BSO read/write, collection operations, quota. Usually lower priority unless volume is very high.\n\n**Expected noise** — 409 Conflict, 503 during deployment, staging-environment-only errors. Mark as \"noise\" and exclude from the action list.\n\n**Spanner-specific** — RST_STREAM, gRPC errors in production (not emulator). These should NOT appear in production — flag immediately.\n\n---\n\n## Step 4: Cross-reference with Jira (if context available)\n\nIf recent Jira data is available in this conversation, check whether any Sentry issues correspond to open `STOR-` bugs. Note the link if found.\n\nIf a high-volume or regression Sentry issue has no corresponding Jira ticket, that is a gap — flag it as \"needs ticket\".\n\n---\n\n## Step 5: Output format\n\n### Production Health: [OK / Watch / Degraded]\nOne-line overall assessment.\n\n### New Issues (Last 7 Days)\nTable: Issue ID | Title | First Seen | Events | Classification\nIf none: \"No new issues in the last 7 days.\"\n\n### Regressions\nTable: Issue ID | Title | Resolved On | Recurred On | Events Since\nIf none: \"No regressions detected.\"\n\n### High-Volume Issues (Last 24h)\nTable: Issue ID | Title | Events (24h) | Triaged? | Classification\nExclude known noise (mark separately if many).\n\n### Error Pattern Analysis\nBulleted findings only if genuine patterns are present:\n- Auth path errors (tokenserver risk)\n- DB layer errors (infrastructure risk)\n- Sudden volume spikes (availability risk)\n- Patterns that correlate with a recent deploy or migration\n\n### Recommended Actions\nUp to 5 concrete items. Each: issue ID, what to do, urgency. Skip if no action is warranted.\n\n### Noise / Expected Errors (Acknowledged)\nBrief list of issues seen but classified as expected noise. One line each. This confirms you looked at them.\n\n---\n\n## Guidelines\n\n- Do not fabricate event counts or timestamps — use exactly what Sentry returns\n- If a query fails or a tool is unavailable, note it and work with available data\n- `is_sentry_event()` in `syncserver/src/error.rs` is the canonical routing decision — errors that are HTTP 4xx should generally not be in Sentry at high volume; flag if they are\n- Focus on production environment; staging errors are informational only\n- An issue being \"old\" does not mean it's not important — check event count trends, not just age\n- Sentry issue IDs should be referenced in full (e.g. `SYNCSTORAGE-PROD-####`) if available\n\n\n---\nname: spanner-local-tests\ndescription: Brings up the Spanner emulator, provisions the test database with the project schema, and runs the syncstorage-rs Spanner unit-test suite locally. Repeatable single-command flow plus full from-scratch documentation. Also covers the e2e (docker-compose) path.\nuser-invocable: true\n---\n\n# Run Spanner Tests Locally\n\nSet up a local Spanner emulator + schema, then run the spanner-backed unit tests. The whole loop is < 90 seconds once the emulator image is cached. This is the **fast inner loop** for any change touching `syncstorage-spanner/`.\n\nThere are two test surfaces in this repo:\n\n| Surface | What it covers | Time | Needs |\n|---|---|---|---|\n| **Unit tests** (this skill's default) | Direct calls into `SpannerDb` — exercises every DML path through `update_user_collection_quotas`, batch ops, BSO CRUD, tombstones, quota. | ~30s after warm-up | emulator only |\n| **Docker e2e** | Full syncserver + emulator + tokenserver + mock FxA, hit over HTTP by the Python integration suite. | 5-15 min | freshly-built `app:build` image |\n\nDefault to unit tests. Reach for e2e only when changes touch HTTP routing, token handling, or cross-backend behavior CI would catch first.\n\n## Quick re-run (after first-time setup)\n\nIf the emulator is already up and provisioned:\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nIf the emulator is **not** running, use the helper:\n\n```bash\n./scripts/spanner-local-test-setup.sh\n```\n\nThat script brings up the emulator, provisions the database/schema, and runs the suite. It's idempotent — re-running just executes the tests if the stack is already up.\n\n## From scratch — the full flow\n\n### 1. Prerequisites\n\n- **Docker Desktop running.** The emulator runs as a container.\n- **Rust toolchain** matching `rust-toolchain.toml` (or close).\n- For e2e only: enough Docker memory (≥ 8 GB recommended) — the `app:build` image compiles grpcio-sys from source and OOMs on small allocations.\n\n### 2. Start the emulator\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml up -d sync-db\n```\n\nThis starts the [`gcr.io/cloud-spanner-emulator/emulator`](https://cloud.google.com/spanner/docs/emulator) image. Two ports:\n\n- **9010** — gRPC. The Rust client (and production) talk this.\n- **9020** — REST. Used for admin (create instance / create database / DDL).\n\nWait ~2 seconds for the emulator to be ready. Confirm:\n\n```bash\ncurl -fsS -o /dev/null http://localhost:9020/v1/projects/test-project/instances && echo ok\n```\n\n### 3. Create the instance and database\n\n```bash\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"instance\":{\"config\":\"emulator-test-config\",\"nodeCount\":1,\"displayName\":\"Test\"},\"instanceId\":\"test-instance\"}'\n\ncurl -fsS -X POST \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"createStatement\":\"CREATE DATABASE `test-database`\"}'\n```\n\n### 4. Apply the schema\n\n⚠️ **Do not run `scripts/prepare-spanner.sh` directly on macOS.** Its `sed` invocation uses GNU `\\+` syntax that BSD `sed` doesn't accept. The script is designed to run inside the Linux container (where GNU sed lives). When you run it from a macOS host, the DDL parsing silently returns an empty array, the database gets created with no tables, and tests fail with cryptic *\"Table not found: collections\"* errors.\n\nUse this macOS-safe inline equivalent:\n\n```bash\nDDL_STATEMENTS=$(grep -v '^--' syncstorage-spanner/src/schema.ddl \\\n | tr '\\n' ' ' \\\n | tr -s ' ' \\\n | sed 's/;[[:space:]]*$//' \\\n | jq -R -s -c 'split(\";\") | map(select(length > 0)) | map(gsub(\"^\\\\s+|\\\\s+$\";\"\"))')\n\nPAYLOAD=$(jq -n --argjson stmts \"$DDL_STATEMENTS\" '{statements:$stmts}')\n\ncurl -fsS -X PATCH \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n -H 'Content-Type: application/json' \\\n -d \"$PAYLOAD\"\n```\n\nThen verify the tables landed:\n\n```bash\ncurl -fsS \"http://localhost:9020/v1/projects/test-project/instances/test-instance/databases/test-database/ddl\" \\\n | jq '.statements | length'\n# expected: 8 (5 tables + 3 indexes + 2 row-deletion policies, the last 2 are folded into CREATE TABLE statements by the emulator)\n```\n\n### 5. Run the unit tests\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db\n```\n\nTwo non-obvious env vars:\n\n- `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` — the gRPC endpoint. **This is what tells the client to skip OAuth.** If you leave it unset, every test fails with *\"Error occurred when fetching oauth2 token\"* because the client tries to authenticate against the real Google API.\n- `RUST_TEST_THREADS=1` — required. Many tests share the same database and assert on counts/timestamps; parallel runs race.\n\nExpected output: `test result: ok. 39 passed; 0 failed; …`.\n\n### 6. Run a single test\n\n```bash\nSYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 \\\nSYNC_SYNCSTORAGE__DATABASE_URL=spanner://projects/test-project/instances/test-instance/databases/test-database \\\nRUST_TEST_THREADS=1 \\\ncargo test --no-default-features --features=syncstorage-db/spanner --package syncstorage-db \\\n tests::db::delete_collection_tombstone -- --nocapture\n```\n\n`--nocapture` is essential when tests fail — without it, errors get swallowed.\n\n### 7. Teardown\n\n```bash\ndocker compose -f docker/docker-compose.spanner.yaml down -v\n```\n\nThe `-v` wipes named volumes for the whole compose project — including `tokenserver_db_data`. If you have the full e2e stack (or anything else from this compose file) running in parallel, prefer the helper script's narrower teardown instead:\n\n```bash\n./scripts/spanner-local-test-setup.sh down\n```\n\nThat only stops + removes the `sync-db` container. The emulator keeps no data in named volumes, so next run starts from a fresh database either way.\n\n## Common failures\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `Error occurred when fetching oauth2 token` | Forgot `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST`. Client is trying real Google OAuth. | Export the var. |\n| `Table not found: collections [at 2:25]` | DDL didn't apply. Most often because `prepare-spanner.sh` was run on macOS and produced an empty statement list. | Use the inline DDL block in step 4. |\n| `INVALID_ARGUMENT: …` on `INSERT OR UPDATE` | Emulator version too old (< 1.5.x). | Update `docker/docker-compose.spanner.yaml`'s image tag. |\n| 39 tests \"pass\" in 0.2s | Tests didn't actually connect — they're failing the OAuth check and treating it as a fresh start. Check the first test's error. | Same as the OAuth row above. |\n| Tests intermittently fail with timestamp/count mismatches | `RUST_TEST_THREADS=1` not set. | Set it. |\n| `cargo test` error: *the package 'syncstorage-db' does not contain this feature: py_verifier* | `py_verifier` is on the `syncserver` crate. | Drop `--features=py_verifier` when running tests on `syncstorage-db` alone. |\n\n## Docker e2e (optional, slower path)\n\nThe Makefile target `docker_run_spanner_e2e_tests` spins up the full stack and runs the Python integration suite twice (with and without JWK cache). It requires the `app:build` image to be **current** — built against the working tree's code.\n\n```bash\ndocker build -t app:build .\nmkdir -p workflow/test-results\nSYNCSTORAGE_RS_IMAGE=app:build make docker_run_spanner_e2e_tests\n```\n\nTwo local-run-only gotchas the make target assumes CI handles:\n\n- `SYNCSTORAGE_RS_IMAGE=app:build` must be exported. `docker-compose.spanner.yaml:52` references `${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}`; without the override compose tries to pull a non-existent `syncstorage-rs:latest` from Docker Hub. CI sets it in `.github/workflows/main-workflow.yml`. Local doesn't get it for free.\n- `workflow/test-results/` must exist before the make target runs. The target uses `docker cp ... ${INT_SPANNER_NO_JWK_JUNIT_XML}` which writes into that directory; if missing, the docker cp fails with `invalid output path` and aborts the run.\n\nTwo tests will always fail locally and can be safely ignored:\n\n- `tools/integration_tests/tokenserver/test_e2e.py::test_unauthorized_oauth_error_status`\n- `tools/integration_tests/tokenserver/test_e2e.py::test_valid_oauth_request`\n\nBoth hit **real FxA stage** (`api-accounts.stage.mozaws.net`, `oauth.stage.mozaws.net`) and need live network + production credentials the docker stack doesn't have. They surface as `fxa.errors.OutOfProtocolError: API responded with non-json content-type` — FxA returning an HTML error page because the request can't authenticate. CLAUDE.md excludes these from `make run_local_e2e_tests` via `--ignore=tokenserver/test_e2e.py`; the docker e2e target doesn't apply that exclusion, so expect `149 passed, 2 errors` not `151 passed`.\n\nNotes:\n\n- The image build compiles grpcio-sys from C++ source. On Docker Desktop for Mac, allocate **≥ 8 GB** memory.\n- The build takes 10-20 minutes from cold. Subsequent rebuilds reuse cache layers and are much faster *unless* `Cargo.lock` or the Rust toolchain changed.\n- Results land in `workspace/test-results/` as JUnit XML.\n- The make target tears the stack down on completion.\n\n### `ar: unable to copy file 'libgrpc.a'; reason: Success`\n\nMisleading message — the linker wasn't killed, the BuildKit cache mount got into a corrupt state. Seen on macOS with stale buildkit caches (mine had ~19 GB accumulated).\n\nFix:\n\n> ⚠️ `docker builder prune -f` clears BuildKit cache mounts for **every** Docker project on the host, not just this repo. Unrelated projects' next builds will go from cached to cold. Run it only when the link error reproduces and you've ruled out a real OOM.\n\n```bash\ndocker builder prune -f\ndocker build -t app:build .\n```\n\nThe prune drops the cache mounts grpcio-sys writes its 100+ MB `libgrpc.a` into. The next build re-creates them clean and the link step succeeds. If it fails a second time after prune, *then* suspect a real OOM and bump Docker Desktop memory.\n\nIf the local build won't cooperate, CI on push runs the same e2e suite — relying on it is reasonable for refactors that don't touch HTTP/auth/routing.\n\n## When to invoke this skill\n\nReach for it when:\n\n- You're editing anything under `syncstorage-spanner/` and want to validate before pushing.\n- A spanner test is failing in CI and you need to reproduce locally.\n- You're investigating an emulator-vs-prod behavior question.\n- You forgot the env-var dance for the third time this month.\n\nDon't invoke it for changes that don't touch Spanner — the MySQL or Postgres backends have their own test paths through the Makefile's `test_with_coverage` and `postgres_test_with_coverage` targets.\n\n## Output format\n\nAfter running, report:\n\n1. **Emulator state:** running / fresh / pre-existing\n2. **Schema applied:** N statements\n3. **Test results:** `X passed; Y failed; Z ignored` — list the failed test names if any\n4. **Total wall time**\n5. **Teardown:** done / left running for follow-up\n\nIf anything failed, surface the actual error message (not just the test name) — Spanner emulator errors are verbose but informative, and the first line is usually enough to diagnose.\n\n\n---\ndescription: Explains code for experienced engineers working on syncstorage-rs. Covers what changed, why it works, non-obvious decisions, gotchas, and data/control flow. Defaults to git diff vs main; accepts an optional file or path argument.\nargument-hint: file-or-path\n---\n\nYou are a senior engineer explaining code to another experienced engineer working on **syncstorage-rs** — Mozilla's Rust implementation of the Firefox Sync storage server and token server.\n\nSkip basics and language fundamentals. Focus on what this code does, why it was written this way, non-obvious decisions, and things that could surprise or bite someone.\n\n## Service boundary — critical context\n\nThis repo owns two services:\n- **Syncstorage** — the storage node that holds encrypted user sync data (bookmarks, passwords, tabs, etc.). Database backends: MySQL, PostgreSQL, Google Cloud Spanner.\n- **Tokenserver** — the allocation service that authenticates Firefox clients via FxA OAuth and assigns them to a storage node. Database backends: MySQL, PostgreSQL.\n\n**Firefox Accounts (FxA) is an external upstream service.** This repo does not own FxA. When explaining code:\n- References to `fxa_uid`, `fxa_kid`, `client_state`, FxA OAuth tokens, JWK keys, the FxA OAuth server URL — these are **inputs from FxA that this service validates and consumes**, not code this repo owns.\n- The `mock-fxa-server` in `scripts/` is a test stub only — it is not a real FxA implementation.\n- `tokenserver-auth/src/oauth/` and `tokenserver-auth/src/crypto.rs` are **this repo's FxA token verification logic**, not FxA's own code.\n\nMake this boundary explicit whenever the code touches the FxA/tokenserver interface.\n\n## How to gather the code\n\nIf `$ARGUMENTS` is provided, read that file or directory.\n\nOtherwise, run:\n\n```bash\ngit diff main...HEAD\n```\n\nand read the full diff. Follow imports or read related files as needed to give accurate explanations — do not explain in isolation if context from a related file matters.\n\n## Explanation structure\n\nWork through the code and produce an explanation covering all sections below. Omit a section only if it genuinely doesn't apply.\n\n### 1. One-paragraph summary\n\nPlain English. What does this code do, and what problem does it solve? Write for someone who hasn't read the ticket.\n\n### 2. Architecture & data flow\n\nWhere this fits in the broader system. Include an ASCII diagram if it clarifies the flow.\n\nUseful reference shapes for this codebase:\n\n```\nFirefox client\n → Tokenserver (validates FxA OAuth token, assigns node)\n → FxA OAuth server [external — validates token, not our code]\n → tokenserver-db (MySQL/Postgres — users, nodes, services tables)\n → Syncstorage node (stores/retrieves BSOs)\n → syncstorage-db (MySQL / Postgres / Spanner)\n```\n\n```\nRequest → Actix-web (syncserver/)\n → Extractors (hawk_identifier, fxa_uid, token validation)\n → Handler\n → db trait (syncstorage-db or tokenserver-db)\n → backend impl (mysql / postgres / spanner)\n```\n\n### 3. Annotated walkthrough\n\nStep through the key functions, types, or request paths. For each:\n- What it does\n- Why it's structured this way (if non-obvious)\n- How it connects to the next step\n\nFocus on the critical path. Don't exhaustively document trivial helpers.\n\nNote Rust-specific patterns where relevant: trait objects vs generics, `Arc`, `actix_web::web::Data<>`, `grpcio` vs `tonic`, `diesel` query builder patterns, `serde` field renaming.\n\n### 4. Gotchas & non-obvious bits\n\nThe most important section. Flag:\n\n- Implicit assumptions or preconditions the caller must satisfy\n- Surprising behavior or edge cases (off-by-one, async ordering, race conditions)\n- Why an obvious alternative wasn't taken (if inferrable)\n- Error handling that silently swallows failures or has unexpected fallback behavior\n- State mutated in non-obvious places\n- Performance characteristics worth knowing (N+1 queries, large allocations, blocking calls in async context)\n- Security-sensitive paths — especially: HAWK token validation, FxA OAuth scope checking, `client_state` / `keys_changed_at` handling, anything touching node assignment\n- gRPC-specific issues (Spanner backend): RST_STREAM handling, retry logic, emulator vs production behavioral differences\n- Migration safety: anything that changes DB schema or touches published migration files\n\n### 5. Dependencies & integrations\n\nExternal systems or crates this code depends on that aren't obvious from the code alone. Distinguish:\n- **Owned by this repo:** `syncstorage-*`, `tokenserver-*`, `syncserver-*` crates\n- **External upstream:** FxA OAuth server, Google Cloud Spanner, `tokenlib` (Python), `hawkauthlib`\n- **Test infrastructure only:** mock FxA server, Spanner emulator\n\nNote any version constraints or behavioral quirks.\n\n### 6. Testing notes\n\nHow is this code tested? Are there gaps? Note:\n- Unit tests (Rust `#[cfg(test)]` modules)\n- Integration tests (`tools/integration_tests/` — pytest, requires a running server)\n- E2E tests (`tokenserver/test_e2e.py` — hits real FxA staging, session-scoped, slow)\n- Any known flaky behavior (Spanner emulator quirks, timing-sensitive tests, 409/503 retry paths)\n\n## Style guidelines\n\n- Be direct and dense. Skip preamble.\n- Use code formatting for identifiers, file paths, and values.\n- Use ASCII diagrams when they save more words than they cost.\n- If something is genuinely straightforward, say so in one sentence and move on.\n- When the FxA boundary is crossed, say so explicitly rather than implying this repo owns that behavior.\n\n\n---\nname: syncserver-jira-description\ndescription: Drafts a concise Jira description for a syncstorage-rs task. Gathers context via targeted interview, researches relevant patterns in the repo, then outputs a clean description ready for an engineer to hand to Claude for implementation.\nuser-invocable: true\n---\n\n# Syncserver Jira Description\n\nDraft a Jira description for a syncstorage-rs task. Output the description only — do not create, edit, or suggest changes to any source files.\n\n## Step 1: Gather context\n\nIf a planning doc, epic description, or tech spec was provided, read it first and infer what, why, crates, and constraints before asking anything.\n\nRequired information:\n- **What:** What is being built or changed, in one sentence\n- **Why:** Motivation — user need, requirement, bug, or tech debt\n- **Crates:** Which specific crate(s) will be modified (e.g. `syncstorage-spanner`, `tokenserver-auth`, `syncserver`, `tokenserver-postgres`)\n- **Constraints:** DB migration required, multi-backend impact (MySQL/Postgres/Spanner), breaking API change, feature flag, FxA contract change — or none\n\nIf all four are clear from provided context, proceed directly to Step 2. Otherwise ask for only what is missing in a single message. Also invite related PRs, tickets, existing approach notes, or architecture notes that would add useful context.\n\n## Step 2: Research\n\nSearch only the crates identified in Step 1. Find the most relevant existing patterns: similar feature, nearby handler, equivalent DB query, related error type. Expand to the broader workspace only if nothing relevant is found there.\n\nIdentify:\n- Key files an implementer will need to touch\n- The closest existing reference implementation to follow\n- Which database backends are affected and whether the change must be replicated across MySQL, PostgreSQL, and/or Spanner\n- Whether new migrations are required (and which crates: `syncstorage-mysql`, `syncstorage-postgres`, `tokenserver-mysql`, `tokenserver-postgres`)\n- Whether tests, metrics (Glean/StatsD), or Sentry error handling apply\n\nIncorporate findings directly into the draft — do not list them separately or ask for confirmation. Surface genuine blockers as Open Questions.\n\n## Step 3: Output\n\n**Background:**\nWhy this is needed and what it enables. 2–4 sentences. Note which service is affected — Syncstorage (storage node), Tokenserver (node allocation + FxA auth), or both.\n\n**Acceptance Criteria:**\nObservable, testable outcomes. Each item verifiable without reading the code. Include criteria for tests, metrics emission, and Sentry error classification where applicable.\n\n**Implementation Steps:**\nNumbered steps with crate paths, file paths, trait/function names, and structural guidance. Reference the nearest existing pattern for each step. For multi-backend changes, call out which backends need matching implementations. No code snippets — file locations, types, and patterns only.\n\n**Database Migrations:** *(omit if no schema changes)*\nWhich migration crates need new files (`syncstorage-mysql/migrations/`, `tokenserver-postgres/migrations/`, etc.). Reminder: never edit published migrations — add new `up.sql`/`down.sql` pairs only.\n\n**Tests:**\nWhat needs to be tested. Unit (`#[cfg(test)]` in the relevant crate), integration (`tools/integration_tests/` pytest — requires running server), and E2E (`tokenserver/test_e2e.py` — hits FxA staging, session-scoped) coverage expectations. Reference the nearest existing test file as a pattern.\n\n**Metrics & Error Handling:** *(omit if not applicable)*\nAny Glean metrics, StatsD counters, or Sentry event classifications that should be emitted or updated. Note whether new error variants should be routed to Sentry or suppressed as metrics (see `is_sentry_event()` pattern in `syncstorage-spanner/src/error.rs`).\n\n**Key Reference Files:**\nSpecific files the implementer should read before starting. One line each.\n\n**Out of Scope:** *(omit if not needed)*\n\n**Open Questions:** *(omit if none)*\n\n## Guidelines\n\n- Output the description only — no source file changes\n- Implementation Steps should give enough detail to start work without follow-up questions — crate paths, file paths, and patterns, not prose\n- Multi-backend tasks: always explicitly name which backends (MySQL / PostgreSQL / Spanner) need matching changes; do not say \"all backends\" without listing them\n- Migration tasks: always reference the no-edit-published-migrations rule\n- Omit redundant or obvious acceptance criteria\n- Include Database Migrations, Metrics & Error Handling only when relevant\n- If motivation or scope remain unclear after asking, flag as an Open Question rather than assuming\n- Branch naming reminder for the assignee: Mozilla engineers use `type/description-STOR-####`; community contributors use `type/description-####`\n\n\n# Load Testing\n\n## Quickstart: load test the change stream system (STOR-628)\n\nFour commands. Needs Docker, plus `curl` and `jq`. No GCP account, no\ncredentials, no VM.\n\n```console\nmake loadtest-offload-image # once: builds the server image\nmake loadtest-offload-changestream # run the load test\nmake loadtest-offload-report # did the pipeline keep up?\nmake loadtest-offload-down # tear it all down\n```\n\nThe report is the answer. Three numbers matter:\n\n| Look for | Meaning |\n| --- | --- |\n| `filtered` non-zero | The pipeline correctly dropped change records that have no payload file attached. This is the thing STOR-628 is asking about. |\n| `handler errors` zero | Nothing failed while finalizing or deleting objects. |\n| `noop_skips` zero | Nothing inert slipped past the filter. |\n\nIt prints `OK: pipeline drained clean` at the end, or a list of `FAIL` lines\nexplaining what went wrong, and exits non-zero on a real problem so it can gate\na script.\n\nOne thing to expect: a `NOTE` saying `batch_commit_skips` is 0. That is a known\nemulator limitation rather than your run going wrong. See\n[The emulator drops transaction tags](#the-emulator-drops-transaction-tags).\n\nFor a real run rather than a smoke test, pass molotov your own arguments:\n\n```console\nmake loadtest-offload-changestream \\\n MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 3600 -v\"\n```\n\nTo load test the handler read/write path instead (STOR-629), swap in\n`make loadtest-offload-handler`. Same report, same teardown.\n\nEverything below is detail you only need when a run looks wrong, or when you\nneed real latency numbers and have to go to GCP.\n\n---\n\n## Summary\n\nThis is the runbook for load testing syncstorage, with the offloaded-payload\npipeline as the main case. It replaces the Confluence page \"Syncstorage &\nTokenserver Runbooks\", whose molotov and locust instructions had become\ninterleaved and unreadable.\n\nThere are two load tests in the repo and they are unrelated:\n\n| | Target | Tool | Location |\n| --- | --- | --- | --- |\n| Syncstorage | storage read/write API | molotov | `tools/syncstorage-loadtest/` |\n| Tokenserver | `/1.0/sync/1.5` token issuance | locust | `tools/tokenserver/loadtests/` |\n\nEverything below is about the syncstorage/molotov one unless it says otherwise.\nFor tokenserver, see [Tokenserver load tests](#tokenserver-load-tests-locust).\n\nTwo tickets drive the offload work, and they want different traffic shapes:\n\n| Ticket | Goal | Traffic shape |\n| --- | --- | --- |\n| STOR-629 | Performance of the handler's GCS read/write path | 100% offloaded, every payload expanded |\n| STOR-628 | The change stream to reconciler pipeline, including the filtering of NULL `payload_link` records | Mixed offloaded and inline |\n\nThe distinction matters. STOR-628 is only a real test if some writes are\n*not* offloaded, because the record it is checking for is the one the\npublisher's filter is supposed to drop.\n\n---\n\n## Which rig to use\n\n```mermaid\nflowchart TB\n q1{\"What are you
measuring?\"}\n q1 -->|\"pipeline correctness:
does every object get
finalized, is the filter
dropping NULL records\"| local[\"Local docker rig
(emulators)\"]\n q1 -->|\"real latency and
throughput numbers
for a capacity decision\"| gcp[\"GCP dev environment\"]\n local --> both[\"Run local first either way:
it catches wiring bugs in
minutes with no GCP spend\"]\n```\n\nThe local rig runs the whole pipeline on emulators. It is the right default:\nit needs no GCP project, no credentials and no VM, and it answers every\ncorrectness question. What it cannot give you is a trustworthy latency\nnumber, because fake-gcs-server on a loopback interface is nothing like GCS\nover the network, and the Spanner emulator is single-node and in-process.\n\nUse GCP dev when the question is \"how fast\" rather than \"does it work\".\n\n---\n\n## Local docker rig\n\n### What it stands up\n\nThe rig is the existing reconciliation e2e stack plus a molotov container and\na metrics sink. Compose overlay: `docker/docker-compose.loadtest.yaml`.\n\n```mermaid\nflowchart LR\n lt[\"loadtest
molotov\"]\n\n subgraph fg[\"Foreground: the request path (STOR-629)\"]\n ss[\"syncserver
offload enabled\"]\n end\n\n gcs[(\"fake-gcs
test-payloads
filesystem backend\")]\n db[(\"Spanner emulator
bsos, batch_bsos
payload_link_changes\")]\n\n subgraph bg[\"Background: the cleanup arm (STOR-628)\"]\n pub[\"payload-link-publisher
stands in for Dataflow\"]\n ps[\"Pub/Sub emulator\"]\n rec[\"payload-reconciler
long-running mode\"]\n end\n\n drop([\"dropped: inert records
payload_link NULL both sides\"])\n sd[\"statsd-exporter
:9102\"]\n rep[\"loadtest-report.sh
the verdict\"]\n\n lt -->|\"Hawk, direct access
no tokenserver or FxA\"| ss\n ss -->|\"upload payload\"| gcs\n ss -->|\"store gs:// in payload_link\"| db\n gcs -->|\"download on read\"| ss\n db -->|\"change stream\"| pub\n pub -.->|filtered| drop\n pub -->|published| ps\n ps --> rec\n rec -->|\"finalize: committed=true
delete: orphans\"| gcs\n\n ss -.->|dogstatsd| sd\n pub -.->|dogstatsd| sd\n rec -.->|dogstatsd| sd\n sd -->|\"scrape counters\"| rep\n gcs -->|\"list objects,
check committed + customTime\"| rep\n```\n\nThe two halves map onto the two tickets. Everything left of the change stream\nis STOR-629's territory, and it costs the user latency. Everything right of it\nis STOR-628's, and no user waits on it, but a wrong decision there deletes a\nfile a live row still needs.\n\nNote where the report gets its evidence: counters from the statsd sink, and\nobject state read straight out of the bucket. It never trusts molotov's exit\nstatus, because the cleanup arm runs after the response is sent and can fail\nlong after the client saw a 200.\n\n| Service | Role |\n| --- | --- |\n| `sync-db` | Spanner emulator, carrying the `payload_link_changes` change stream from `schema.ddl` |\n| `pubsub-emulator` | Pub/Sub |\n| `fake-gcs` | GCS, on its **filesystem** backend for this rig |\n| `reconciliation-setup` | One-shot: creates the topic, subscription and bucket |\n| `payload-link-publisher` | Python change-stream reader, stands in for the Dataflow job |\n| `payload-reconciler` | Drains Pub/Sub, finalizes and deletes GCS objects |\n| `syncserver` | Offload enabled, limits raised |\n| `statsd` | `prom/statsd-exporter`, so the publisher and reconciler counters are readable |\n| `loadtest` | molotov |\n\nThe publisher is the Python one, not the Java Dataflow template. It publishes\nthe identical JSON wire format, so everything downstream of it is\nproduction code. If you need to reproduce something Java-specific, layer\n`docker-compose.e2e.reconciliation.java.yaml` on top; see\n[Reconciliation Pipeline](payload_link_reconciler.md).\n\nOn the NULL-filtering half of STOR-628, be precise about what the local rig\nproves. The filter exists twice: `isPayloadLinkActionable` in the Java flex\ntemplate, and `is_payload_link_actionable` in\n`payload-link-publisher-py/utils.py`, which is written to mirror it. A local\nrun exercises the **Python** one. That is a real test of the filter's logic\nand of everything downstream, but it is not a test of the Java code that runs\nin production. To exercise the shipping filter, either use the Java overlay\nlocally or run against GCP dev, where the Dataflow job is the publisher.\n\n### Prerequisites\n\n- Docker running, with roughly 8 GB available to the VM. Expanded payloads\n are held in memory in several places at once.\n- `curl` and `jq` on the host, for the report script.\n- Nothing else. No GCP credentials, no `service-account.json`.\n\n### The stale publisher image trap\n\nRead this before trusting any local result.\n\n`payload-link-publisher` is the only service in the stack built from a compose\n`build:` context rather than pulled or shared with `app:build`. Plain\n`docker compose up` reuses whatever `docker-payload-link-publisher` image is\nalready in your local cache and **will not rebuild it when its source\nchanges**. So a publisher image built before a change to\n`payload-link-publisher-py/` keeps running against new server code, silently.\n\nThis is not hypothetical. It is how the rig behaved on first use here: a\nfour-week-old publisher image had no `transaction_tag` support at all, so no\nrecord ever carried the `batch_commit` tag, so the reconciler treated every\nbatch-commit handoff as a genuine `batch_bsos` removal and deleted GCS objects\nthat committed `bsos` rows still pointed at. The symptom was HTTP 500s on\nreads, `payload_reconciler.batch_commit_skips` flat at zero, and\n`gcs_404{op=finalize}` running nearly 1:1 with `finalizes`. It looked exactly\nlike the STOR-657 production bug, and it was entirely a stale image.\n\nThe `loadtest-offload-*` targets therefore pass `--build` on every `up`. The\nfirst build is slow from a cold cache; afterwards it is cached and cheap.\n\nTwo things follow. If you drive compose by hand instead of through the\nMakefile, pass `--build` yourself. And note that\n`make docker_run_reconciliation_e2e_tests` does **not** pass it, so a local run\nof the reconciliation e2e suite can exercise a stale publisher. CI is unaffected\nbecause its cache starts empty.\n\nThe tell, if you suspect it:\n\n```console\ndocker compose ... exec payload-link-publisher grep -c transaction_tag /app/publisher.py\n```\n\nZero means the image predates transaction-tag support.\n\n### Run it\n\n```console\nmake loadtest-offload-image # build app:build (spanner backend), once\n\nmake loadtest-offload-handler # STOR-629\nmake loadtest-offload-report\n\nmake loadtest-offload-changestream # STOR-628\nmake loadtest-offload-report\n\nmake loadtest-offload-down # tear down, including the fake-gcs volume\n```\n\n`loadtest-offload-report` waits for the reconciler to stop making progress\nbefore reporting, so run it straight after the load test rather than trying\nto guess a sleep. It exits non-zero if it finds a problem, which makes it\nusable as a gate.\n\nBoth scenario targets take `MOLOTOV_ARGS`:\n\n```console\nmake loadtest-offload-handler MOLOTOV_ARGS=\"--processes 2 --workers 20 --duration 600 -v\"\n```\n\nStart small. Every worker can be holding a multi-MiB payload, on both the\nclient and the server, so worker count multiplies memory rather than just CPU.\nFive workers writing 10 MiB records is already a meaningful amount of traffic\nthrough this stack.\n\n### Reading the report\n\n```text\nRECONCILER (payload_reconciler.*)\n payload_reconciler_finalizes 1843\n payload_reconciler_orphan_deletes 291\n payload_reconciler_batch_commit_skips 412\n...\nGCS OBJECTS (test-payloads)\n objects 1552\n committed=true 1552\n committed!=true 0\n customTime pinned 1552\n...\nVERDICT\n OK: pipeline drained clean.\n```\n\nWhat each number should look like:\n\n| Metric | Healthy | If it is wrong |\n| --- | --- | --- |\n| `published` | Tracks actionable change records | Zero means the publisher is not reading the change stream at all |\n| `filtered` | **Non-zero on a mixed run.** This is the positive evidence for STOR-628: the publisher dropped records with `payload_link` NULL on both sides | Zero on a mixed run means either the filter is passing inert records through, or every write happened to be offloaded and the path went untested |\n| `finalizes` | Tracks offloaded write volume | Zero with objects in the bucket means the change stream, publisher or Pub/Sub leg is broken |\n| `orphan_deletes` | Non-zero, tracks overwrites and deletes | Zero means the load test never deleted anything; check `DISABLE_DELETES` |\n| `batch_commit_skips` | **Zero on the local rig** -- see the emulator limitation below. Non-zero is what you require on GCP dev | Zero on *GCP dev* means the transaction tag is not reaching the change stream, which is the STOR-657/668 bug class |\n| `noop_skips` | **Zero** | Non-zero is the STOR-628 regression: the filter let through a record with `payload_link` NULL on both sides |\n| `errors kind:handler` | **Zero** | Non-zero means messages went unacked. In prod five failures on one message routes it to the DLQ |\n| `committed!=true` | Zero once drained | A small tail is normal if you cut the run off mid-flight. A large or growing count means the pipeline is not keeping up with the write rate |\n\n`committed=true` count and `customTime pinned` count must agree. Finalize sets\nboth in a single `blob.patch()`, so a divergence means something other than\nthe reconciler is writing object metadata.\n\nOne thing that surprises people: even the \"100% offloaded\" STOR-629 run\nproduces a non-zero `filtered` count. `OFFLOAD_COLLECTIONS` only governs the\nbatch-write arm, while the load test also PUTs `meta/global` and POSTs\n`clients` on every scenario pass. Those two are never in the offload list, so\nthey generate exactly the inert NULL-on-both-sides records the filter is\nthere to drop. So STOR-629 covers the filter incidentally, and STOR-628 is\nstill the run that covers it deliberately, at volume, alongside offloaded\ntraffic in the same stream.\n\n### Reading the 404 counters, and why they are the whole signal\n\nThe reconciler treats `404 NotFound` as success on both of its operations.\nFrom `reconcile_payload_links.py`, a finalize whose target is gone logs at\n`debug`, bumps `gcs_404{op=finalize}`, and **acks the message**. Same shape for\ndelete. [Reconciliation Pipeline](payload_link_reconciler.md) states the\nrationale plainly: \"Both reconciler actions are idempotent, so a redelivered\nrecord is harmless.\"\n\nThat is correct and deliberate, and it has a consequence worth internalising\nbefore you read a load test result. Because a missing object is success, a run\nthat is silently destroying data produces:\n\n- `errors` `kind:handler` at **zero**\n- nothing in the DLQ\n- no `WARN` or `ERROR` in the reconciler log, only `debug`\n- an elevated `gcs_404` counter, and nothing else\n\nSo on the offload path `gcs_404{op=finalize}` is not a nice-to-have gauge. It\nis the *only* signal that separates a healthy pipeline from one that is\ndeleting live payloads. Everything else looks green.\n\nThat existing doc's failure-modes table gives two causes for a sustained\n`gcs_404{op=finalize}`:\n\n| Documented cause | Applies to a local run? |\n| --- | --- |\n| The lifecycle rule reclaimed the object before it was finalized | **No.** The 30 day `daysSinceCustomTime` rule is a bucket policy in `bucket.tf`. fake-gcs has no lifecycle policy at all, so nothing can reap an object locally. |\n| The same message was redelivered after a successful prior run (the at-least-once tax) | **No**, not at volume. Redelivery is a background trickle, not a rate approaching `finalizes`. |\n\nMeasured on these runs, neither explains what was observed:\n\n| Run | `finalizes` | `gcs_404{op=finalize}` | Ratio |\n| --- | --- | --- | --- |\n| STOR-629 handler | 930 | 716 | 77% |\n| STOR-628 changestream | 1853 | 1589 | 86% |\n\nA ratio near 1:1 is a third cause that the table does not list: **the\nreconciler deleted the object itself**, as an orphan, and the finalize for that\nsame object then 404s. Locally that is the emulator dropping the\n`batch_commit` transaction tag (see below), so every batch-commit handoff is\nmisread as a genuine `batch_bsos` removal.\n\nThe signature to recognise, wherever you see it:\n\n```text\nbatch_commit_skips 0 <- tag never arrived\ngcs_404{op=finalize} ~= finalizes <- finalizing objects already deleted\norphan_deletes high <- the deletes that did the damage\nerrors kind:handler 0 <- and nothing reports a problem\n```\n\n`gcs_404{op=delete}` is the benign one, and the existing doc's read of it holds\nwithout qualification: the object was already gone, the operation is idempotent\nby design, and a redelivery or a concurrent cleanup produces it. It carries no\ndata-loss implication on its own.\n\nTwo things follow for the alerting the reconciler doc already recommends. Its\nsuggestion to watch \"`gcs_404` `op:finalize` rising as a share of `finalizes`\"\nis the right alert, and this gives it a concrete threshold to reason about: a\nlow single-digit share is the redelivery tax, while anything approaching\n`finalizes` means objects are being deleted out from under committed rows. And\npairing it with `batch_commit_skips == 0` distinguishes the batch-commit cause\nfrom a lifecycle-window problem.\n\n### Knobs\n\nServer side, set on the `syncserver` container:\n\n| Variable | Default in the rig | Meaning |\n| --- | --- | --- |\n| `LOADTEST_OFFLOAD_COLLECTIONS` | `bookmarks,history` | Which collections syncserver offloads. **This is what decides mixed vs fully offloaded.** |\n| `LOADTEST_COLLECTION_LIMITS` | 10 MiB record / 20 MiB post / 21 MiB request, for `bookmarks` and `history` only | Per-collection raised limits, as a JSON map. Keys must be a subset of the offload list -- see below |\n| `LOADTEST_GCS_MAX_CONCURRENCY` | 4 | Concurrent GCS ops within one batch request. The main server-side throughput knob, worth sweeping |\n| `LOADTEST_DB_POOL_MAX_SIZE` | 40 | The stock 10 saturates first once workers climb |\n\n### Raise limits per collection, never globally\n\nThis is the one thing that will bite you, so it is worth stating plainly.\n\nSpanner caps a single `STRING(MAX)` value at **2621440 bytes (2.5 MiB)**. That\nhard cap is the reason this whole project exists. Raising the *global*\n`max_record_payload_bytes` above it does not fail at the API, it fails in the\ndatabase, on every collection that is not offloaded:\n\n```text\nFAILED_PRECONDITION New value exceeds the maximum size limit for\nthis column: bsos.payload, size: 10485760, limit: 2621440\n```\n\nAn offloaded collection is exempt, because its payload goes to GCS and the row\nstores only a `gs://` URL in `payload_link`, leaving `payload` NULL. So a limit\nabove 2.5 MiB is only ever correct for a collection that is in\n`gcs_payload_offload_collections`. That is exactly what the per-collection\noverride exists for, and how production raises it:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":10485760,\"max_post_bytes\":20971520,\"max_request_bytes\":22020096}}'\n```\n\nLeave the globals alone. Any collection not named in the map keeps stock\n2.5 MiB behaviour and writes inline safely. `max_post_bytes` and\n`max_request_bytes` are overridable per collection too; actix's payload config\nis sized from the largest `max_request_bytes` across all overrides\n(`ServerLimits::effective_max_request_bytes`), so a big body is not rejected at\nthe resource level before per-collection enforcement runs.\n\nThere is no per-collection `max_total_bytes`. The global default (clamped to\n`MAX_SPANNER_LOAD_SIZE`, 100 MB, by `Settings::normalize` on spanner) is\nalready generous, so the rig leaves it untouched.\n\nOn a mixed run this arrangement gives you exactly the right traffic for free:\noffloaded collections take expanded payloads, and inline collections get\n`LARGE_PAYLOAD_PROB` applied but capped at 2.5 MiB by\n`payload_target_length`, which is realistic \"large but not offloaded\" traffic.\n\nClient side, set on the `loadtest` container:\n\n| Variable | Meaning |\n| --- | --- |\n| `LARGE_PAYLOAD_PROB` | Fraction of BSOs given an expanded payload. `1.0` for STOR-629 |\n| `LARGE_PAYLOAD_SIZE` | Explicit target size in bytes. Unset means \"use the server's `max_record_payload_bytes`\" |\n| `OFFLOAD_COLLECTIONS` | Restricts batch writes to these collections. **Leave unset for mixed runs** |\n| `DISABLE_DELETES` | Set truthy to suppress the DELETE arm. Leave off, or the reconciler's `orphan_deletes` branch is never exercised |\n\nThe two collection lists are easy to confuse:\n\n- `LOADTEST_OFFLOAD_COLLECTIONS` (server) decides which collections get\n offloaded.\n- `OFFLOAD_COLLECTIONS` (client) decides which collections get *written to*.\n\nSetting the client one to a single collection sends every batch write there,\nwhich is what you want for STOR-629 and exactly what you must not do for\nSTOR-628. For a mixed run, leave the client variable unset so writes spread\nacross molotov's default five (`bookmarks`, `forms`, `passwords`, `history`,\n`prefs`) and set the server list to a subset of those.\n\n### Per-collection limit overrides\n\nProduction raises limits for one collection rather than globally, via a JSON\nmap. The load test resolves limits per target collection out of\n`/info/configuration`, preferring a collection's override, so this path is\ncovered too:\n\n```console\nSYNC_SYNCSTORAGE__LIMITS__COLLECTIONS='{\"bookmarks\":{\"max_record_payload_bytes\":20971520,\"max_post_bytes\":26214400,\"max_request_bytes\":26218496}}'\n```\n\n### Known limits of the local rig\n\n- **The emulator does not carry transaction tags, so the batch-commit skip\n cannot be validated locally.** This one is load bearing; details below.\n- **No DLQ.** `docker/reconciliation-setup.sh` creates the topic and\n subscription but no dead-letter topic, so dead-letter routing after five\n failed deliveries cannot be exercised locally. Verify that on GCP dev.\n- **No Dataflow.** The Python publisher replaces the Java job. Autoscaling,\n partition-split handling under real load and the connector's Spanner\n metadata writes are all Dataflow behaviours that only appear on GCP.\n- **Latency is not meaningful.** fake-gcs over loopback and a single-node\n Spanner emulator. Use it for correctness and relative comparisons only.\n- **No lifecycle policy.** The 30 day `daysSinceCustomTime` rule that reaps\n unfinalized objects is a bucket policy, so the local rig can only check\n that `customTime` was pinned, not that the reaping behaves.\n\n### The emulator drops transaction tags\n\nThe Spanner emulator emits the `transaction_tag` field on change stream\nDataChangeRecords but never populates it. Measured on emulator v1.5.52: the\nDataChangeRecord struct has arity 13, so index 11 (`transaction_tag`) exists\nand the publisher reads it correctly, but every record carries the empty\nstring, including the `batch_bsos` DELETE that *is* the batch-commit handoff.\n\nThat single gap has a large consequence. The reconciler decides whether to keep\nor delete a `batch_bsos` removal's object by checking\n`transactionTag == \"batch_commit\"` (see\n[Reconciliation Pipeline](payload_link_reconciler.md)). With the tag always\nempty, every batch-commit handoff looks like a genuine removal, so the\nreconciler deletes the GCS object that the just-committed `bsos` row now points\nat. The BSO becomes unreadable and syncserver returns a 500 backed by a GCS 404.\n\nSo on the local rig, expect all of this, and do not read it as a regression:\n\n- `payload_reconciler.batch_commit_skips` flat at zero.\n- `gcs_404{op=finalize}` running close to 1:1 with `finalizes`.\n- Objects missing for BSOs written in a batch create/append request.\n- A residual molotov failure rate, because roughly 50% of its POST scenarios\n are transactional batches. Reads of those BSOs 500.\n\n`docker/batch-commit-probe.py` reproduces it in isolation: one two-request\ntransactional batch, then a read of both BSOs, printing the bucket contents at\neach stage.\n\n```console\nmake loadtest-offload-up\nSYNCSTORAGE_RS_IMAGE=app:build docker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n -f docker/docker-compose.loadtest.yaml \\\n run --rm --entrypoint python3 \\\n -v \"$PWD/docker/batch-commit-probe.py:/probe.py:ro\" loadtest /probe.py\n```\n\nAdding `--no-deps` to that command keeps the reconciler stopped, and the probe\nthen passes with both payloads intact. That is the control which proves the\nreconciler is the thing deleting the object, rather than the batch commit\nlosing the `payload_link`.\n\nWhat this does **not** tell you is whether production is affected. The app side\nis correct: `handlers.rs` calls\n`with_transaction_tag(BATCH_COMMIT_TRANSACTION_TAG)` on the commit path, and\nreal Spanner does record transaction tags on change streams. The evidence\npoints at an emulator gap rather than a code defect, but the local rig cannot\nclose that question. **Confirm the batch-commit skip on GCP dev**, where the\nDataflow job reads a real change stream: run the probe there and require\n`payload_reconciler.batch_commit_skips` to be non-zero.\n\nTwo follow-ups worth filing rather than leaving implicit:\n\n1. There is no e2e test for the batch-commit handoff.\n `tools/integration_tests/test_payload_link_reconciliation.py` covers upload,\n update and delete, but not a transactional batch. That is presumably because\n it cannot pass against the emulator, which is exactly the point: the gap is\n invisible in CI.\n2. If the Java overlay behaves the same way, the emulator is confirmed as the\n cause. That is a cheap check and worth doing before spending GCP time.\n\n---\n\n## GCP dev environment\n\nUse this for real numbers. The dev pipeline is fully provisioned; see\n[GCP Infrastructure](payload-offload-infrastructure.md) for what lives where\nand which repo owns it. That page is the authority on the resources\nthemselves. The diagram below is only the load-testing view: where the\ngenerator sits, what it traverses, and where you go to observe each leg.\n\n```mermaid\nflowchart TB\n subgraph gen[\"Load generator (you create this)\"]\n vm[\"C4D VM, Ubuntu
molotov + load_test.pem
in moz-fx-sync-nonprod\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (shared GKE)\"]\n ts[\"tokenserver
trusts your public JWK\"]\n ss[\"syncserver
needs GCS_PAYLOAD_BUCKET +
OFFLOAD_COLLECTIONS + raised limits\"]\n rec[\"payload-reconciler
cronjob, ~5 min cadence\"]\n end\n\n subgraph v1[\"moz-fx-sync-nonprod-904c (GCPv1, cloudops-infra)\"]\n sp[(\"Spanner syncdb-dev
change stream:
payload_link_changes\")]\n end\n\n subgraph tenant[\"moz-fx-sync-nonprod (tenant, webservices-infra)\"]\n df[\"Dataflow flex template
the REAL Java filter\"]\n ps[\"Pub/Sub
payload-link-changes\"]\n dlq[\"Pub/Sub DLQ
after 5 failed deliveries\"]\n b[(\"GCS
sync-nonprod-dev-
syncstorage-payloads
30d lifecycle rule\")]\n end\n\n vm -->|\"1. self-signed JWT\"| ts\n ts -->|\"2. Hawk token\"| vm\n vm -->|\"3. storage reads + writes\"| ss\n ss -->|\"upload / download\"| b\n ss -->|\"payload_link\"| sp\n sp -->|\"change stream read\"| df\n df -->|publish| ps\n ps -->|\"pull subscription\"| rec\n ps -.-> dlq\n rec -->|\"finalize / delete\"| b\n```\n\nThree things this buys you that the local rig cannot:\n\n| Leg | Why it only exists here |\n| --- | --- |\n| Dataflow | The local rig substitutes a Python publisher. This is the actual Java `isPayloadLinkActionable` filter that ships, plus autoscaling and partition-split behaviour under real load. |\n| Real Spanner | Populates `transaction_tag`, so the batch-commit skip can finally be verified. Also the only place to measure what the change stream costs in Spanner storage, which has never been quantified. |\n| DLQ and lifecycle rule | Neither exists locally. `reconciliation-setup.sh` creates no dead-letter topic, and fake-gcs has no lifecycle policy. |\n\n### Load generator VM\n\nGenerate load from inside GCP, not from a laptop. A laptop's uplink is the\nbottleneck long before the server is, and multi-MiB payloads make that worse.\n\n1. Console -> project `moz-fx-sync-nonprod` -> Compute Engine -> VM instances\n -> Create instance.\n2. Machine type: `C4D` series. OS: Ubuntu, not the Debian default.\n3. The project already has IAP TCP forwarding and the firewall rules. If SSH\n is refused, check the project inherited them; see the SRE space's IAP page.\n\nSet the VM up:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\ngit clone https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\ncurl -sSL https://install.python-poetry.org | python3 -\npoetry install\n```\n\nDocker is an alternative to poetry on the VM, and avoids the Python version\ndance:\n\n```console\ndocker build . --tag syncstorage-loadtest:local\ndocker run --rm -e SERVER_URL -e OAUTH_PRIVATE_KEY_FILE \\\n -e LARGE_PAYLOAD_PROB -e OFFLOAD_COLLECTIONS \\\n -v /path/to/load_test.pem:/keys/load_test.pem:ro \\\n syncstorage-loadtest:local \\\n -c \"molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\"\n```\n\n### Authentication\n\nThree modes, in `storage/auth.py`. For a dev run use self-signed JWTs: no FxA\naccounts to create or clean up, and the token lifetime is yours to set.\n\n```console\npoetry run ./generate-keys.sh\n```\n\nThat writes `load_test.pem` (private, stays on the VM) and `jwk.json`\n(public). The dev tokenserver has to trust that public key, which means\nlanding the JWK in the tokenserver FxA JWK configuration in\n`webservices-infra` under `sync/k8s/`. That is a PR plus an ArgoCD sync, so do\nit before you book time for the run.\n\nThen:\n\n```console\nSERVER_URL=\"https://\" \\\n OAUTH_PRIVATE_KEY_FILE=./load_test.pem \\\n LARGE_PAYLOAD_PROB=1.0 \\\n OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --processes 4 --workers 100 --duration 3600 -v loadtest.py\n```\n\nDirect-access mode (`SERVER_URL` with a `#secret` fragment) bypasses\ntokenserver entirely and is what the local rig uses. It needs the deployed\nmaster secret, so do not use it against a shared environment.\n\n### Server side prerequisites on dev\n\nThe pipeline being provisioned is not the same as offload being on. Offload\ndoes nothing until both of these are set on syncserver, and expanded payloads\ndo nothing until the limits are raised:\n\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET`\n- `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS`\n- `SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES` and the\n `MAX_POST_BYTES` / `MAX_REQUEST_BYTES` / `MAX_TOTAL_BYTES` that go with it\n\nAll four are Helm values in `webservices-infra` under `sync/k8s/sync/`. Raise\nany front-end proxy body-size limit to match, or requests get a 413 before\nthey reach the server.\n\nConfirm what actually took effect by reading it back from the server rather\nthan from the values file:\n\n```console\ncurl -s https:///1.5//info/configuration | jq .\n```\n\nThe load test sizes its batches from exactly this response, so if the limits\nare not in there, the run is not testing expanded payloads no matter what the\nenvironment variables say.\n\n### Verifying a dev run\n\nEverything the local report script checks, plus the parts that only exist on\nGCP:\n\n| What | Where |\n| --- | --- |\n| Reconciler counters (`payload_reconciler.*`) | The metrics pipeline, same as any other service metric |\n| Oldest unacked message age on `payload-link-reconciler-sub` | Pub/Sub console. The most direct measure of finalize latency; should sit in minutes |\n| Anything at all in `payload-link-changes-dlq` | Pub/Sub console. A message only lands there after five failed deliveries, so it always needs a human |\n| Dataflow job lag, autoscaling, worker errors | Dataflow console |\n| Spanner CPU and change-stream storage cost | Spanner console on `moz-fx-sync-nonprod-904c` |\n| Object count and finalize state in the bucket | GCS console, or `gcloud storage ls -L` |\n\nChange-stream storage cost is worth capturing deliberately. It has not been\nquantified, and the plan for prod is to measure it before enabling offload\ntraffic, so a dev run is the cheapest place to get a number.\n\n---\n\n## What to record\n\nStandard long run is 60 to 65 minutes. Log results to the shared documents\nrather than only in a ticket:\n\n- Syncstorage molotov runs: \"Sync Load Test Run Document\"\n- Tokenserver locust runs: the tokenserver load test results document\n\nCapture at minimum: the git SHA under test, molotov arguments, the effective\n`/info/configuration`, the server-side offload collection list, molotov's\nown summary (request counts, failures, RPS), and the reconciler counters plus\nGCS finalize state from the report.\n\n---\n\n## Tokenserver load tests (locust)\n\nSeparate tool, separate target, unrelated to payload offload. Locust rather\nthan molotov, and it has a web UI.\n\n```console\ncd tools/tokenserver/loadtests\npoetry install\npoetry run ./generate-keys.sh\nOAUTH_PRIVATE_KEY_FILE=./load_test.pem poetry run locust # UI on :8089\n```\n\nRunning on a VM, tunnel the UI out over IAP:\n\n```console\ngcloud compute ssh --zone \"\" \"\" --project \"\" \\\n --tunnel-through-iap -- -L 8090:localhost:8089\n```\n\nThen open `localhost:8090`. See `tools/tokenserver/loadtests/README.md` for\nthe rest.\n\n---\n\n## Gaps worth knowing about\n\n- **GCS time can be separated from request time for the collection routes.**\n `storage.gcs.payload.batch` is the wall time one `get_collection` or\n `post_collection` request spent on GCS across all of its objects, tagged with\n `handler`, and the request middleware emits `request.duration` tagged by\n route and method. Dividing `rate(storage.gcs.payload.batch_sum{handler=\"post_collection\"})`\n by the `request.duration` `rate(_count)` for `/1.5/{uid}/storage/{collection}`\n with `method=\"POST\"` (and `get_collection` against `GET` likewise) gives the\n GCS share of an average collection write or read. It is still rough, because\n the denominator counts requests whose payloads stayed inline, under the\n offload threshold. The single-BSO routes (`/{bso}` GET and PUT) emit only\n the per-object `.upload` / `.download` timings, which carry no route tag, so\n they cannot be attributed the same way; measure those with a separate load\n shape. Either way a STOR-629 run can say \"GCS accounted for about N ms of a\n collection request\" rather than only \"the handler got slower\".\n- **No per-collection metric dimension**, so a mixed run cannot separate\n offloaded from inline request latency from the server side either. Working\n around it means either running the two shapes separately and comparing, or\n adding a collection tag.\n\n\n# Documentation for `make_hawk_token.py`\n\n## Summary\n\n`make_hawk_token.py` is a utility script for generating a [Hawk](https://github.com/mozilla/hawk) authentication token and header for use when testing the syncstorage-rs API. It derives a token and signing key from a shared secret and produces either a raw token tuple or a ready-to-use `Authorization: Hawk ...` header.\n\nThe script lives in [`tools/hawk/`](https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/hawk).\n\n---\n\n## Setup\n\n### Prerequisites\n\n- Python ≥ 3.12\n- [Poetry](https://python-poetry.org/docs/#installation) installed\n\n### Install dependencies\n\n```shell\ncd tools/hawk\npoetry install\n```\n\n---\n\n## Usage\n\n```shell\npoetry run python make_hawk_token.py [options]\n```\n\nBy default (no arguments), a token is generated for `http://localhost:8000/1.5/1/storage/col2/` using a built-in test secret.\n\n### Options\n\n| Option | Description | Default |\n|---|---|---|\n| `--uri` | Request URI path | `/1.5/1/storage/col2/` |\n| `--secret` | Shared `SYNC_MASTER_SECRET` value | *(built-in test value)* |\n| `--method` | HTTP method for the request | `GET` |\n| `--as_header` | Print a ready-to-use `Authorization: Hawk ...` header | *(off)* |\n\nUse `-h` for the full option list.\n\n---\n\n## Examples\n\n### Generate a token for a GET request\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET\n```\n\n### Generate a ready-to-use Authorization header\n\n```shell\npoetry run python make_hawk_token.py \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nOutput:\n```\nAuthorization: Hawk id=\"...\", ts=\"...\", nonce=\"...\", mac=\"...\"\n```\n\n### Non-GET methods\n\nFor methods other than GET (e.g. PUT, POST, DELETE), pass `--method`:\n\n```shell\npoetry run python make_hawk_token.py \\\n --method PUT \\\n --uri /1.5/1/storage/meta/global \\\n --secret $SYNC_MASTER_SECRET \\\n --as_header\n```\n\nSee [`tools/examples/put.bash`](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for a full curl example using this output.\n\n---\n\n## Notes\n\n- The `--secret` value must match the `SYNC_MASTER_SECRET` configured in the running syncstorage-rs instance.\n- The generated token is scoped to the exact URI and method — a token generated for `GET /foo` is not valid for `PUT /foo`.\n\n\n# GCS Payload Reconciliation Pipeline\n\n## Summary\n\nWhen syncserver offloads a large BSO payload to Google Cloud Storage\n(`syncserver/src/web/payload_offload.rs`), the GCS object is created\nwith custom metadata `committed=false` and `customTime=now`, and the\nBSO row's `payload_link` column points at it. A separate pipeline must:\n\n- **Finalize** newly-committed objects — flip metadata to\n `committed=true` and pin `customTime` to its maximum value\n (`2200-12-31T23:59:59Z`) so the bucket's lifecycle policy (see\n below) cannot reclaim them.\n- **Garbage-collect orphans** — delete GCS objects whose row's\n `payload_link` was replaced (UPDATE) or removed (DELETE, including\n Spanner row-deletion-policy TTL deletes).\n\nAn object whose write never reached a Spanner commit never receives a\nfinalize, and there are two shapes of that. If the database transaction\nfails or rolls back, syncserver issues an inline best-effort **delete**\nof the objects it uploaded for that request and ignores the result. If an\nupload itself fails, the request is abandoned fail-fast and objects\nalready uploaded for it are not cleaned up inline at all. Either way an\nobject can be left stranded at `committed=false` with its upload-time\n`customTime`.\n\nAnything that slips through is reaped by a GCS **lifecycle policy**\n(`bucket.tf` in `webservices-infra/sync`) that deletes objects whose\n`customTime` is more than **30 days** old. Flipping `customTime` to the\nmax sentinel is what protects committed objects from that policy:\n`daysSinceCustomTime` goes permanently negative once finalized, so the\npolicy cannot touch them regardless of object age.\n\nThe 30 day window is the safety margin for the whole asynchronous arm. It\nhas to comfortably exceed the worst case time from upload to finalize,\nwhich is set by the cronjob cadence plus any Pub/Sub or Dataflow backlog.\nAt a five minute cadence the margin is four orders of magnitude, so the\nnumber only becomes interesting if the pipeline is stopped for weeks.\n\nThis document covers that pipeline. It consumes the\n`payload_link_changes` Spanner change stream defined in\n`syncstorage-spanner/src/schema.ddl`.\n\n---\n\n## Architecture\n\n```mermaid\nflowchart LR\n stream[(\"Spanner change stream\n payload_link_changes\n OLD_AND_NEW_VALUES, 7d retention\")]\n\n subgraph dataflow[\"Custom Dataflow flex template\"]\n filter{\"payload_link NULL\n on both sides?\"}\n end\n\n drop([\"dropped\"])\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n\n subgraph reconciler[\"Reconciler cronjob\"]\n route{\"which side\n carries a link?\"}\n finalize[\"finalize the object\n committed=true\n customTime=MAX\"]\n delete[\"delete the object\"]\n end\n\n stream -->|DataChangeRecord| filter\n filter -->|yes| drop\n filter -->|\"no, publish as JSON\"| topic\n topic -->|pull subscription| route\n topic -.->|\"after 5 failed deliveries\"| dlq\n route -->|new link| finalize\n route -->|\"old link, replaced or removed\"| delete\n```\n\nBoth reconciler actions are idempotent, so a redelivered record is\nharmless.\n\n## Components\n\n### 1. Spanner change stream — `payload_link_changes`\n\nDefined in `syncstorage-spanner/src/schema.ddl`:\n\n```sql\nCREATE CHANGE STREAM payload_link_changes\n FOR bsos(payload_link), batch_bsos(payload_link)\n OPTIONS (\n retention_period = '7d',\n value_capture_type = 'OLD_AND_NEW_VALUES'\n );\n```\n\nColumn-scoped: an UPDATE that does not touch `payload_link` produces no\nrecord. INSERTs and DELETEs always produce a record, even when\n`payload_link` is NULL — those are dropped at the next stage.\n\nThe Spanner DDL is not auto-applied; run `gcloud spanner databases ddl\nupdate` against the target database after merging.\n\n### 2. Custom Dataflow flex template — `tools/payload-link-dataflow/`\n\nA standalone Apache Beam pipeline (Java, Beam 2.60.0) that:\n\n1. Reads `payload_link_changes` via `SpannerIO.readChangeStream()`.\n2. Applies a `Filter.by(isPayloadLinkActionable)` step that drops\n records whose every mod has `payload_link` NULL on both sides.\n Malformed records pass through so the reconciler / DLQ surfaces\n them — not the filter.\n3. Serializes each surviving `DataChangeRecord` to JSON and publishes\n to a Pub/Sub topic.\n\nThe pipeline is **not** vendored from\n[`GoogleCloudPlatform/DataflowTemplates`](https://github.com/GoogleCloudPlatform/DataflowTemplates).\nWe own a small standalone source tree under `src/`; the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` template is referenced for\nintent comparison via `upstream-customization.patch` (documentation\nonly — not a build input).\n\n**Build / publish** (operator runs from `webservices-infra`):\n\n```bash\ndocker build -t /syncserver-payload-link-dataflow: \\\n tools/payload-link-dataflow\ndocker push /syncserver-payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/syncserver-payload-link-dataflow.json \\\n --image /syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n```\n\n**Launch parameters** (full list in\n`tools/payload-link-dataflow/metadata.json`):\n\n- `spannerProjectId`, `spannerInstanceId`, `spannerDatabase` — the\n syncstorage Spanner database.\n- `spannerMetadataInstanceId`, `spannerMetadataDatabase` — where the\n change-stream connector keeps its partition-state table. Dev uses a\n dedicated database, `syncdb-pldf-meta-dev`, so the connector holds no\n write access to the syncstorage database. Every environment should do\n the same.\n- `spannerMetadataTableName=Metadata_payload_link` names the\n partition-state table itself. Optional in the template, but pin it on\n any streaming launch. Left unset, the connector generates a random\n table name inside `expand()` and bakes it into the pipeline graph, so a\n `--update` or a cancel-then-relaunch comes up against a fresh empty\n table, reads from `Timestamp.now()`, and silently drops every record\n committed between the two jobs. The name only has to be a legal Spanner\n identifier and stable for the life of the environment; the connector\n creates the table on first use.\n See \"Job update and restart continuity\" in\n `tools/payload-link-dataflow/README.md`.\n- `changeStreamName=payload_link_changes`.\n- `spannerDatabaseRole=payload_link_reader` — the fine-grained access\n role the job reads the stream through, created by the DDL in\n `syncstorage-spanner/src/schema.ddl`. If the role is absent the job\n fails at startup rather than falling back to broader access.\n- `pubsubTopic=projects//topics/payload-link-changes`.\n\n**Service account requires:**\n\n- `roles/spanner.databaseUser` on the syncstorage database and on the\n metadata database. The one role covers both reading the change stream\n and maintaining the connector's partition metadata, but since the two\n live in separate databases the grant is needed on each. Note the IAM\n grant targets the `-904c` project, so it is applied out of band; see\n [GCP Infrastructure](payload-offload-infrastructure.md#out-of-band-steps).\n- Membership of the `payload_link_reader` database role, which is what\n actually narrows the job to the change stream. The IAM grant alone\n does not let it read BSO rows.\n- `roles/pubsub.publisher` on the destination topic.\n- `roles/storage.objectAdmin` on the Dataflow job bucket, for `staging/`\n and `tmp/`.\n- `roles/dataflow.worker` at the project level.\n\n### 2b. Dev/E2E Python publisher — `tools/payload-link-dataflow/payload-link-publisher-py/`\n\nPlain Python script (not Beam) that polls the\n`READ_payload_link_changes` TVF via `google.cloud.spanner.Client` and\npublishes the same JSON wire format the Java job produces. Sub-second\nstartup, no JVM. **Dev/E2E only** — used by the compose stack\ndescribed below; **Java remains the prod publisher.**\n\nFollows partition splits: reads `_root`, picks up child-partition\ntokens from `child_partitions_record`s, reads each child from its\nadvertised `start_timestamp`, retires parents once their children are\nannounced, and drops any partition that responds `OUT_OF_RANGE`.\nRequired in practice — the emulator routes DataChangeRecords to child\npartitions immediately, so a `_root`-only reader sees zero DCRs. Full\ndetails in the tool's own README.\n\n### 3. Pub/Sub topic + DLQ\n\nProvisioned from `webservices-infra/sync`:\n\n- Topic: `payload-link-changes`\n- Dead-letter topic: `payload-link-changes-dlq`\n- Pull subscription: `payload-link-reconciler-sub`\n - 60s ack deadline\n - 7d message retention\n - DLQ routing after 5 delivery attempts\n\n### 4. Reconciler — `tools/payload-reconciler/`\n\nPython script with one job: pull messages, perform GCS operations, ack.\nSync-pull drain loop with two deployment modes selected by whether\n`RUN_BUDGET_SECONDS` is set:\n\n- **Cronjob mode** (default deployment): `RUN_BUDGET_SECONDS` set (e.g.\n `240`). The script drains the subscription up to that many seconds\n or until the queue idles, then exits 0. K8s cronjob at ~5 min\n cadence.\n- **Long-running mode**: `RUN_BUDGET_SECONDS` unset. The script polls\n forever, never exiting on idle. Deploy as a K8s Deployment when\n finalize-flip latency below the cronjob cadence matters.\n\n**Per-message handling** (`reconcile_payload_links.py:handle_message_body`):\n\nFor each mod in the change record:\n\n- New `payload_link` present: `blob.patch()` sets\n `metadata.committed = \"true\"` and `customTime = \"2200-12-31T23:59:59Z\"`.\n- Old `payload_link` present and not equal to the new value: `blob.delete()`,\n with one exception. A `batch_bsos` row removed under the batch commit\n transaction tag (`transactionTag` equals `batch_commit`, set by syncstorage in\n the Spanner `commit_batch` path) is skipped\n (`payload_reconciler.batch_commit_skips`). On commit the link moves into the\n permanent `bsos` row in the same transaction, so its object must be kept\n (deleting it was the STOR-657 bug). Any other `batch_bsos` removal is a\n genuine delete: `batch_bsos` has no deletion policy of its own, so TTL expiry\n and `user_collections` deletes reach it as cascade deletes with no such tag,\n and their objects are removed. `bsos` changes and `batch_bsos` overwrites are\n unaffected. See STOR-668.\n\nBoth operations tolerate `404 NotFound` as success, see *Failure\nmodes* below.\n\n**Environment**\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | — | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | — | `payload-link-reconciler-sub` in prod. |\n| `GCS_PAYLOAD_BUCKET` | yes | — | Cross-bucket links abort the message. |\n| `RUN_BUDGET_SECONDS` | no | — | Set (e.g. `240`) → cronjob mode; drain up to N seconds then exit 0. Unset → long-running mode; poll forever, never exit on idle. |\n| `SYNC_STATSD_HOST` | no | localhost | statsd metrics server host. |\n| `SYNC_STATSD_PORT` | no | 8125 | statsd metrics server port. |\n\n**Service account requires:**\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`.\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and the `delete` operation).\n\n**Deployment.** Default is a K8s cronjob (~5 min cadence) with\n`RUN_BUDGET_SECONDS` set. When lower finalize-flip latency matters,\ndeploy as a K8s Deployment without `RUN_BUDGET_SECONDS` to run\nlong-running. The cronjob template is\n`sync/k8s/sync/templates/payload-reconciler-cronjob.yaml` in\nwebservices-infra, gated on `payloadReconciler.enabled`. Note it lives in\nthe `sync` chart rather than `sync-jobs`.\n\nFour settings in that manifest are load bearing, and changing any of them\nbreaks an assumption the script relies on:\n\n- `concurrencyPolicy: Forbid`. A run that overruns its window is never\n joined by a second one. The subscription is the queue, so the next tick\n simply picks up where the last left off. Two concurrent drains would\n race on the same messages, which is survivable given idempotency but\n wastes the ack deadline.\n- `backoffLimit: 0`. No in-window retry. Every operation is idempotent\n and unacked messages come back on their own, so the next scheduled run\n *is* the retry. A backoff would just re-drain the same queue sooner.\n- `runBudgetSeconds` strictly less than `activeDeadlineSeconds`. The\n chart fails the render if this is violated. The budget is what makes\n the drain loop exit cleanly on its own; if the deadline fired first the\n pod would be killed mid-message and the run would always look failed.\n- The command override. The image entrypoint is the syncserver binary, so\n the cronjob invokes\n `python3 /app/tools/payload-reconciler/reconcile_payload_links.py`\n explicitly. Invoking by path is also what puts the script's directory\n on `sys.path`, which is how its `import utils` resolves.\n\nCredentials come from workload identity: the pod runs as the tenant GKE\nservice account, which holds the two roles listed above. There is no key\nfile and no secret mounted.\n\n---\n\n## Wire format\n\nEach surviving change record reaches the reconciler as a JSON Pub/Sub\nmessage:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-30T00:00:00.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"transactionTag\": \"\",\n \"isSystemTransaction\": false,\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nMod fields (`keys`, `oldValues`, `newValues`) carry **JSON strings**\nthat the reconciler parses with a second `json.loads`, matching\nSpanner's change-streams wire convention.\n\n`transactionTag` carries the transaction tag Spanner records for the change,\nwhich the reconciler uses to skip batch commit handoffs (see above).\n`isSystemTransaction` is carried too but not currently consumed.\n\n---\n\n## Local e2e compose stack\n\n`make docker_run_reconciliation_e2e_tests` brings up a full-stack\ncompose environment that exercises the entire pipeline against\nemulators:\n\n| Service | Image | Role |\n|---|---|---|\n| `sync-db` | Spanner emulator (existing) | Spanner + change stream |\n| `pubsub-emulator` | `google-cloud-cli:emulators` | Pub/Sub |\n| `fake-gcs` | `fsouza/fake-gcs-server` | GCS |\n| `reconciliation-setup` | one-shot | creates Pub/Sub topic + subscription + bucket |\n| `payload-link-publisher` | Python publisher (default) | polls change stream → publishes to Pub/Sub |\n| `payload-reconciler` | primary image, entrypoint override | drains Pub/Sub → patches/deletes GCS objects |\n| `syncserver` | primary image | offload enabled for all `test_storage.py` collections |\n| `e2e-tests` | primary image | runs `pytest tools/integration_tests/ tools/tokenserver/` |\n\nTwo publisher variants:\n\n- **Python (default)** — `docker-compose.e2e.reconciliation.yaml`.\n Sub-second startup; the day-to-day iteration path.\n- **Java (swap-in)** — layer\n `docker-compose.e2e.reconciliation.java.yaml` on top. Runs the same\n Java flex-template image under Beam's DirectRunner. See\n `tools/payload-link-dataflow/README.md` for the exact invocation.\n Use when you need to reproduce a Java-specific issue.\n\nThe compose stack doubles as regression coverage for `test_storage.py`:\nby opting every collection the storage tests use into GCS offload, each\nBSO write flows through the offload path (upload + `payload_link`\nstorage), and each read flows through `download_payload`. New\nreconciler-specific tests live in\n`tools/integration_tests/test_payload_link_reconciliation.py`, gated\nby a module-level `pytest.mark.skipif` on `GCS_PAYLOAD_BUCKET` so they\nauto-skip in the existing spanner e2e stack (where the env var is\nunset) and un-skip here.\n\n**Emulator fallback.** If a future Spanner emulator upgrade breaks\n`SELECT * FROM READ_payload_link_changes(...)`, the fall-back is the\nJava swap-in overlay above (or running the compose stack against a\nreal dev Spanner instance).\n\n---\n\n## Metrics\n\nBoth halves report. The reconciler emits under the\n`payload_reconciler` statsd namespace, syncserver under\n`storage.gcs.payload`. Between them every GCS operation reports both a\ncount and a latency.\n\n### Reconciler\n\nCounters:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalizes` | Objects flipped to `committed=true` | Tracks offloaded write volume |\n| `orphan_deletes` | Objects deleted because a row stopped pointing at them | Tracks overwrite and delete volume |\n| `gcs_404` `op:finalize` | Finalize target was gone | Low and flat |\n| `gcs_404` `op:delete` | Delete target was already gone | Low and flat, redeliveries are normal |\n| `batch_commit_skips` | A `batch_bsos` removal carrying the `batch_commit` transaction tag was kept rather than deleted | Tracks batch commit volume on offloaded collections |\n| `noop_skips` | A record arrived with nothing to do | Near zero; the Dataflow filter should have dropped it |\n| `errors` `kind:handler` | Handler raised, message left unacked | Zero |\n| `runs` | One per drain, emitted unconditionally | Tracks the cronjob cadence |\n| `messages_processed` | Change records pulled and handled, counted per batch | Tracks pipeline volume |\n| `budget_exhausted` | A run hit `RUN_BUDGET_SECONDS` before the queue idled | Zero |\n\nTimings:\n\n| Metric | Meaning | Healthy shape |\n|---|---|---|\n| `finalize_age` | Spanner commit to finalize, in ms | Seconds to minutes. This is the window the 30 day lifecycle policy is sized against |\n| `gcs_op` | One GCS round trip. Tagged `op` (`finalize` or `delete`) and `result` (`success`, `not_found`, `error`) | Low. Separates our own GCS latency from Dataflow and Pub/Sub delay |\n| `drain_duration` | How long one run took | Comfortably under `RUN_BUDGET_SECONDS` |\n\n`runs` is the only unconditional series. Every other metric here fires\nonly on the event it names, so a quiet pipeline emits almost nothing and\n\"idle\" would otherwise be indistinguishable from \"not running\".\n\n### Syncserver offload path\n\nEmitted from `syncserver/src/web/payload_offload.rs`. All three GCS\noperations report latency, each tagged with a `result`.\n\n| Metric | Type | Tags | Meaning |\n|---|---|---|---|\n| `storage.gcs.payload.upload` | timing | `result` | One object uploaded |\n| `storage.gcs.payload.download` | timing | `result` | One object read back, streaming included |\n| `storage.gcs.payload.cleanup.duration` | timing | `result` | The compensating delete after a failed write transaction |\n| `storage.gcs.payload.batch` | timing | `op`, `handler`, `result` | Total GCS time for one request, across all of its BSOs |\n| `storage.gcs.payload.size` | histogram | `op` | Distribution of individual payload sizes, in bytes |\n| `storage.gcs.payload.upload.bytes` | counter | none | Bytes written, for throughput |\n| `storage.gcs.payload.download.bytes` | counter | none | Bytes read back |\n| `storage.gcs.payload.cleanup` | counter | `handler`, `result`, `reason` | Outcome of the compensating delete |\n\nStatsd timers carry their own count, so there is no separate counter for\nthe number of uploads or downloads.\n\n`storage.gcs.payload.batch` is not the sum of the per-object timings. A\nrequest works on up to `gcs_payload_max_concurrency` objects at once, so\nits total lands somewhere between the slowest single object and the sum\nof all of them. It is the delay the client waits through, and it is\nrecorded even when the batch fails part way, hence the `result` tag: an\nabandoned batch stopped early and is not comparable to a complete one.\n\n`storage.gcs.payload.size` is a histogram rather than a tag on the\ntimings, so the aggregator derives percentiles from it. The byte\ncounters give a total and therefore a mean; the tail is the part that\nmatters on a project about raising a size ceiling.\n\nTags on `storage.gcs.payload.cleanup`:\n\n| Tag | Values | Meaning |\n|---|---|---|\n| `handler` | `put_bso`, `post_collection` | Which write handler issued the cleanup |\n| `result` | `success`, `failure` | Whether the object was deleted |\n| `reason` | `invalid_url`, `gcs_error` | Failure only. `invalid_url` means the `gs://` URL did not parse and no delete was attempted; `gcs_error` means GCS rejected it |\n\nThe cleanup counter firing at all means writes are failing after their\npayload reached GCS, so read it against the write error rate rather than\non its own. `result:failure` is the more interesting cut: those objects\nare now leaked until the 30 day lifecycle policy reaps them. A steady\n`reason:invalid_url` is a code bug rather than an operational problem,\nsince the URL is one syncserver just generated. Note that an unparseable\nURL issues no GCS call and so has no `cleanup.duration` alongside it.\n\nWorth alerting on:\n\n- Anything at all in `payload-link-changes-dlq`. A message only lands\n there after five failed deliveries, so it means a record cannot be\n handled and needs a human. Inspect it through\n `payload-link-changes-dlq-sub`. Note the DLQ is terminal: its\n subscription carries no dead-letter policy of its own, so a message\n sits there until the 7 day retention drops it and no further queue\n catches it. That is the same 7 days as the change stream retention,\n so once a dead-lettered record ages out the underlying change is\n unrecoverable from both. Seven days is a response deadline, not a\n margin.\n- `errors` `kind:handler` sustained above zero.\n- `noop_skips` climbing, which means the Dataflow filter regressed and\n the pipeline is paying Pub/Sub for records it discards.\n- `gcs_404` `op:finalize` rising as a share of `finalizes`, which is the\n signal that objects are being reaped before they get finalized.\n- Oldest unacked message age on `payload-link-reconciler-sub`. This is a\n Pub/Sub metric rather than one of ours, and it is the most direct\n measure of finalize latency. It should sit in minutes. The 30 day\n lifecycle window is the deadline it must never approach.\n- `storage.gcs.payload.cleanup` with `result:failure`, which counts\n objects that leaked because the compensating delete did not land.\n- `finalize_age` climbing. It is the direct measure of how long objects\n sit unprotected, and unlike the Pub/Sub metric above it covers the\n whole path including the GCS patch.\n- `budget_exhausted` sustained above zero, meaning runs keep ending on\n the clock rather than on an idle queue, so the drain is not keeping up.\n- `runs` going flat. Every other reconciler metric is conditional, so\n this is the one that separates a quiet pipeline from a stopped one.\n\n---\n\n## Failure modes\n\n| Symptom | Cause | Behaviour |\n|---|---|---|\n| Spike in `payload_reconciler.noop_skips` | Dataflow filter is letting inert records through | Investigate; should be ~0 if the filter works. Records still ack — no harm but extra Pub/Sub cost. |\n| Sustained `payload_reconciler.gcs_404` with `op:finalize` | Lifecycle rule reclaimed the object before the reconciler finalized it, OR the same message redelivered after a successful prior run (at-least-once tax) | Acceptable up to a low background level. Sharp rise = lifecycle window too aggressive vs. cronjob cadence. |\n| Sustained `payload_reconciler.gcs_404` with `op:delete` | Object was already deleted (redelivery or concurrent cleanup) | Acceptable; idempotent by design. |\n| Messages in `payload-link-changes-dlq` | Repeated handler exceptions on the same message after 5 retries (malformed JSON, cross-bucket link, GCS auth failure) | Inspect the DLQ payload; fix and re-publish or discard. The main subscription continues to drain. |\n| `payload_reconciler.errors` with `kind:handler` non-zero | Same as above before reaching DLQ. | Same. |\n| `storage.gcs.payload.cleanup` with `result:failure` | The compensating delete after a failed write transaction did not land | The object is stranded at `committed=false` and the 30 day lifecycle policy is the only thing left to reap it. Occasional failures are tolerable; sustained means the write path is failing and cleanup is failing with it. |\n| Finalize latency jumps after a Dataflow deploy, with a gap in `finalizes` | The job relaunched against a fresh partition-metadata table and resumed from `Timestamp.now()` | Records committed during the gap are lost for good; the change stream's 7 day retention only helps if you notice in time. Check that `spannerMetadataTableName` is pinned and unchanged. |\n\nA `payload_link` pointing at a bucket other than `GCS_PAYLOAD_BUCKET`\nraises `ValueError` and the message is left unacked — it retries up to\nthe DLQ rather than mutating an unrelated bucket. This is a hard guard.\n\n---\n\n## Keeping the reference patch accurate\n\n`tools/payload-link-dataflow/upstream-customization.patch` is\ndocumentation only — not a build input. If the upstream\n`Cloud_Spanner_Change_Streams_to_PubSub` pipeline evolves in ways that\nchange the conceptual diff, refresh it (see the README in that\ndirectory). This is a doc refresh, not a code change.\n\n\n# Payload Offload Infrastructure\n\n## Summary\n\nThis document is a map of the GCP infrastructure behind offloaded BSO\npayloads: which project each resource lives in, which repo defines it,\nand what applies it. It covers where things run rather than how the\npipeline behaves. For the behaviour of the change stream, Dataflow job,\nPub/Sub topics and reconciler, see\n[Payload Link Reconciler](payload_link_reconciler.md).\n\nTracking epic: STOR-372, \"Expand spanner storage beyond 2.5MB\".\n\n## The two-project split\n\nThe single most confusing thing about this setup is that Spanner and\neverything else live in different GCP projects, under different\ngenerations of Mozilla's cloud tooling.\n\n| | Spanner | Everything else |\n| --- | --- | --- |\n| Project (dev) | `moz-fx-sync-nonprod-904c` | `moz-fx-sync-nonprod` |\n| Generation | GCPv1 (legacy) | GCPv2 (MozCloud tenant) |\n| Managed from | `cloudops-infra` | `webservices-infra` + `global-platform-admin` |\n\nThe Spanner instances stayed in GCPv1 on purpose. Migrating the large\nprod instance is not cheap, so they were left in place while the rest of\nthe service moved to MozCloud. The practical consequences:\n\n1. Terraform in `webservices-infra` cannot configure the Spanner\n instances or databases. Those changes need `cloudops-infra`, which\n means an extra PR and a MozCloud engineer to apply it.\n2. The webservices Terraform admin service account has no\n `setIamPolicy` rights on `-904c`, so IAM grants that target the\n Spanner database have to be applied out of band. See\n [Out-of-band steps](#out-of-band-steps) below.\n\nThe Dataflow job bridges the two projects: it runs in the tenant project\nand reads a change stream in the legacy project.\n\n## Diagram\n\n```mermaid\nflowchart TB\n gha[\"GitHub Actions\n syncstorage-rs\"]\n\n subgraph v1[\"GCPv1: moz-fx-sync-nonprod-904c (cloudops-infra)\"]\n syncdb[\"Spanner syncdb\n instance: sync\n database: syncdb-dev\n change stream: payload_link_changes\n access role: payload_link_reader\"]\n metadb[\"Spanner Dataflow metadata\n instance: sync\n database: syncdb-pldf-meta-dev\"]\n end\n\n subgraph high[\"moz-fx-webservices-high-nonpro (global-platform-admin)\"]\n gke[\"Shared GKE cluster\n syncserver, tokenserver,\n payload-reconciler cronjob\"]\n end\n\n subgraph tenant[\"GCPv2 tenant: moz-fx-sync-nonprod (webservices-infra)\"]\n df[\"Dataflow flex template job\n sync-nonprod-dev-payload-link-dataflow\n dedicated VPC and firewall\"]\n topic[\"Pub/Sub\n payload-link-changes\"]\n dlq[\"Pub/Sub\n payload-link-changes-dlq\"]\n payloads[(\"GCS\n sync-nonprod-dev-syncstorage-payloads\")]\n jobbucket[(\"GCS\n sync-nonprod-dev-payload-link-dataflow\n template spec, staging, tmp\")]\n end\n\n gha -.->|\"publish template spec, via WIF\"| jobbucket\n jobbucket -.->|template spec| df\n syncdb -->|change stream read| df\n df -->|partition state| metadb\n df -->|publish| topic\n topic -->|pull subscription| gke\n topic -.->|\"after 5 failed deliveries\"| dlq\n gke -->|finalize and delete objects| payloads\n gke -->|upload and download payloads| payloads\n gke -->|read and write BSO rows| syncdb\n\n```\n\nNote that the workloads do not run in the tenant project. Sync is\n`risk_level: high`, so its pods run on the shared high risk GKE cluster\nin `moz-fx-webservices-high-nonpro` (nonprod) and\n`moz-fx-webservices-high-prod` (prod). The tenant project holds the\ndata plane resources: buckets, Pub/Sub, secrets, service accounts, DNS,\nand the Dataflow job. Pods reach tenant resources through workload\nidentity on the tenant GKE service account.\n\n## Where each piece is defined\n\n| Resource | Defined in | Repo |\n| --- | --- | --- |\n| Tenant definition (envs, charts, risk rating, project ids) | `tenants/sync.yaml` | global-platform-admin |\n| Tenant GCP projects, enabled APIs, folders, DNS, GAR | `projects/tf/webservices/locals.tf` | global-platform-admin |\n| Shared GKE, GCLB, logging, monitoring | `webservices-high/tf/{nonprod,prod}` | global-platform-admin |\n| GCS payload bucket and lifecycle rule | `sync/tf/dev/bucket.tf` | webservices-infra |\n| Dataflow job, network, firewall, job bucket | `sync/tf/dev/dataflow.tf` | webservices-infra |\n| Dataflow service account and IAM | `sync/tf/dev/dataflow_iam.tf` | webservices-infra |\n| Pub/Sub topics, subscriptions, DLQ routing | `sync/tf/dev/pubsub.tf` | webservices-infra |\n| Spanner instance, database, DDL, database IAM | `projects/sync` | cloudops-infra |\n| Change stream and access role DDL source of truth | `syncstorage-spanner/src/schema.ddl` | syncstorage-rs |\n| Dataflow flex template image and metadata | `tools/payload-link-dataflow/` | syncstorage-rs |\n| Image build and template spec publish | `.github/workflows/mozcloud-publish.yaml` | syncstorage-rs |\n| Helm charts for sync, tokenserver, sync-jobs, sync-test | `sync/k8s/` | webservices-infra |\n\n## Repos and how each is applied\n\n| Repo | Scope | Automation |\n| --- | --- | --- |\n| [global-platform-admin](https://github.com/mozilla/global-platform-admin) | Cross tenant platform: projects, APIs, shared clusters, tenant definitions | Spacelift. The `admin-webservices-projects` stack tracks `main` with autodeploy, so a merge plans and applies without a manual step. |\n| [webservices-infra](https://github.com/mozilla/webservices-infra) | Per tenant resources for the webservices function | Atlantis, autodiscovered per directory. Plan runs on the PR, then `atlantis apply` in a comment. |\n| [cloudops-infra](https://github.com/mozilla-services/cloudops-infra) | Legacy GCPv1 resources, including sync Spanner | Atlantis, applied by a MozCloud engineer |\n| [syncstorage-rs](https://github.com/mozilla-services/syncstorage-rs) | Application code, Spanner DDL, Dataflow template, Helm chart values | GitHub Actions builds and publishes images; ArgoCD deploys |\n\nTerraform state for both global-platform-admin's `projects/tf/webservices`\nand webservices-infra's `sync/tf/*` lives in the same bucket,\n`moz-fx-webservices-terraform-state-global`, accessed by impersonating\n`tf-webservices@moz-fx-websvc-terraform-admin.iam.gserviceaccount.com`.\nThe per environment state is at prefix `projects/sync/`. The project\nlevel state is at `projects/projects/global`, and\n`sync/tf/dev/data.tf` reads it as a remote state data source to resolve\nproject ids and numbers. That prefix is load bearing, do not rename it\nwithout migrating the state object.\n\n## Concrete dev resources\n\nNames are derived from `${application}-${realm}-${environment}` in\n`sync/tf/dev/locals.tf`.\n\n| Thing | Value |\n| --- | --- |\n| Tenant project | `moz-fx-sync-nonprod`, number `960020799362` |\n| Spanner project | `moz-fx-sync-nonprod-904c` |\n| Spanner instance and database | `sync` / `syncdb-dev` |\n| Dataflow metadata database | `sync` / `syncdb-pldf-meta-dev` |\n| Dataflow metadata table | `Metadata_payload_link`, pinned via `spannerMetadataTableName` |\n| Change stream | `payload_link_changes` |\n| Spanner database role | `payload_link_reader` |\n| Payload bucket | `sync-nonprod-dev-syncstorage-payloads`, us-west1 |\n| Dataflow job bucket | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow job | `sync-nonprod-dev-payload-link-dataflow`, us-west1 |\n| Dataflow service account | `sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Template publisher service account | `sync-nonprod-dev-tmpl-pub@moz-fx-sync-nonprod.iam.gserviceaccount.com` |\n| Dataflow VPC | `sync-nonprod-dev-dataflow`, internal ingress on tcp 12345-12346 |\n| Pub/Sub topic | `payload-link-changes` |\n| Pub/Sub DLQ | `payload-link-changes-dlq`, 5 delivery attempts |\n| Reconciler subscription | `payload-link-reconciler-sub`, 60s ack, 7d retention |\n| DLQ inspection subscription | `payload-link-changes-dlq-sub`, 7d retention |\n| Flex template image | `us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow` |\n| Template spec | `gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json` |\n\nOnly dev is built out today.\n\n## Standing up stage or prod\n\nEverything below is per environment. Names derive from\n`${application}-${realm}-${environment}`, so copying the dev Terraform\ninto the target environment directory produces the right names without\nediting them.\n\n1. **Terraform.** Copy `bucket.tf`, `pubsub.tf`, `dataflow.tf` and\n `dataflow_iam.tf` from `sync/tf/dev/` into the target environment\n directory in webservices-infra.\n\n2. **A dedicated Spanner metadata database.** The change stream\n connector keeps its partition state in a Spanner database of its own.\n Each environment should get a separate database, for example\n `syncdb-pldf-meta-stage`, so the connector holds no write access\n to the syncstorage database. This is a cloudops-infra change.\n\n Pin the table inside it as well, with\n `spannerMetadataTableName = \"Metadata_payload_link\"` on the job.\n Dev uses that name; reusing it per environment is fine, since the\n databases are already separate. The parameter is optional and the\n consequence of skipping it is silent, so it is easy to miss: see\n step 6.\n\n3. **DDL.** Apply the change stream and the `payload_link_reader` role to\n the target database. See [Out-of-band steps](#out-of-band-steps).\n\n4. **Cross-project IAM.** A `-904c` admin grants the new environment's\n Dataflow service account `roles/spanner.databaseUser` on the\n syncstorage database and on the metadata database.\n\n5. **Template spec.** Add a `write-payload-link-dataflow-spec-` job\n to `.github/workflows/mozcloud-publish.yaml`, and make sure the\n environment's `-tmpl-pub` service account exists first. The spec has\n to be in the bucket before the Terraform job resource is applied.\n\n6. **Job lifecycle.** Keep `on_delete = \"cancel\"`. This is not a\n dev-only shortcut: the SpannerIO change stream connector does not\n support draining at all, so `\"drain\"` is never the right setting for\n this pipeline. See\n [Draining a change streams pipeline](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining).\n The change stream's 7 day retention is what covers the gap while a\n replacement job comes up.\n\n Cancel is only safe to rely on because the metadata table name is\n pinned. The connector generates that name from a random UUID at\n graph-construction time, so an unpinned job that is cancelled and\n relaunched, or updated in place, comes up against an empty table and\n resumes from `Timestamp.now()` instead of the previous watermarks.\n Nothing errors; the records committed in between are simply never\n published, and the reconciler never learns those objects need\n finalizing or deleting. Set the name once and do not change it.\n\n7. **Reconciler cronjob.** Enable `payloadReconciler` in the\n environment's values file in `sync/k8s/sync/`. The chart requires\n `runBudgetSeconds` to be less than `activeDeadlineSeconds`.\n\n8. **Measure before enabling on prod.** The change stream costs Spanner\n storage that has not been quantified. Turning the stream on ahead of\n any offload traffic is the cheap way to find out.\n\nEnabled APIs need no change. `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so `moz-fx-sync-prod` already has them.\n\nNote that provisioning the pipeline is safe on its own. Offload does\nnothing until `SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET` and\n`SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` are both set on\nsyncserver, so an environment can carry the whole pipeline with zero\ntraffic through it while it is verified.\n\n## Enabled GCP APIs\n\nThe tenant project's enabled APIs are declared in\n`projects/tf/webservices/locals.tf` under the `sync` entry. Enabling an\nAPI there is a PR to global-platform-admin, since tenant teams do not\nhold the IAM to enable services directly.\n\nCurrent list for the sync tenant:\n\n- `spanner.googleapis.com`\n- `secretmanager.googleapis.com`\n- `dataflow.googleapis.com`\n- `datapipelines.googleapis.com`\n- `cloudscheduler.googleapis.com`\n\nThe last three were added in\n[global-platform-admin PR\n#7031](https://github.com/mozilla/global-platform-admin/pull/7031) for\nthis work, tracked by STOR-656 and SREIN-1633.\n`compute.googleapis.com` is enabled by default for every tenant project\nby the shared `google_project` module, which is what Dataflow workers\nneed.\n\nTwo caveats on that list. First, `project_services` in\n`projects/tf/webservices/locals.tf` feeds both the prod and nonprod\nproject modules, so those three APIs are enabled in `moz-fx-sync-prod`\nas well as `moz-fx-sync-nonprod`. There is no per realm override.\nSecond, the current implementation launches\n`google_dataflow_flex_template_job` directly, so nothing in\n`webservices-infra/sync` references Data Pipelines or Cloud Scheduler\nyet. Those two were requested up front in case the job moves to a\nscheduled Data Pipelines definition. They are enabled but unused.\n\n### Security assessment\n\nSync as a whole stays at `risk_level: high` with\n`risk_rating: {confidentiality: 2, integrity: 1, availability: 4}` in\n`tenants/sync.yaml`. The offload itself is low risk.\n\nBSO payloads are encrypted client side with the user's Sync key before\nthey reach syncserver, so GCS holds ciphertext the server cannot\ndecrypt. The offload moves the same class of opaque blob that Spanner\nalready holds, rather than creating a new plaintext exposure. Object\nkeys and metadata do carry identifiers such as fxa_uid, collection and\nbso_id, which is metadata rather than content.\n\nEnabling a service grants no access on its own and creates no\nidentities, so IAM still gates every call. The one item worth care is\nDataflow: a principal with `roles/dataflow.developer` plus `actAs` on\nthe worker service account can run arbitrary pipeline code on GCE VMs\ninside the tenant VPC with that account's permissions. That is why the\njob runs as a dedicated per environment service account holding only\n`roles/dataflow.worker`, `roles/pubsub.publisher` on one topic,\n`roles/storage.objectAdmin` on the job bucket, and\n`roles/spanner.databaseUser` on one database, rather than the Compute\nEngine default account which carries `roles/editor`.\n\nSpanner access is narrowed a second time inside the database. The job\nruns under the `payload_link_reader` database role, which is granted\n`SELECT` on the one change stream and `EXECUTE` on its read function and\nnothing else, so the IAM grant alone does not let the job read BSO rows.\n\n## Out-of-band steps\n\nThree things are not managed by Terraform and have to be done by hand in\nthe right order. Getting this wrong is the usual reason a fresh\nenvironment fails to come up.\n\n1. Apply the change stream DDL to the Spanner database. The DDL in\n `syncstorage-spanner/src/schema.ddl` is not auto-applied. Run\n `gcloud spanner databases ddl update` against the target database\n after merging.\n\n That same DDL block also creates the fine-grained access role the\n Dataflow job reads through:\n\n ```sql\n CREATE ROLE payload_link_reader;\n GRANT SELECT ON CHANGE STREAM payload_link_changes TO ROLE payload_link_reader;\n GRANT EXECUTE ON TABLE FUNCTION READ_payload_link_changes TO ROLE payload_link_reader;\n ```\n\n The job is launched with `spannerDatabaseRole = \"payload_link_reader\"`\n (see `sync/tf/dev/dataflow.tf`), so if the role does not exist the job\n fails at startup rather than falling back to broader access. Apply the\n role and its grants in the same DDL update as the change stream.\n\n2. Grant the Dataflow service account `roles/spanner.databaseUser` on\n the Spanner database. This targets `-904c`, where the webservices\n Terraform admin account has no `setIamPolicy` rights, so a `-904c`\n admin runs it. Tracked by STOR-655.\n\n ```console\n gcloud spanner databases add-iam-policy-binding syncdb-dev \\\n --instance=sync --project=moz-fx-sync-nonprod-904c \\\n --role=roles/spanner.databaseUser \\\n --member=serviceAccount:sync-nonprod-dev-payload-link@moz-fx-sync-nonprod.iam.gserviceaccount.com\n ```\n\n3. Publish the flex template spec into the job bucket before the\n Terraform job resource is applied, or the job has nothing to launch.\n The `write-payload-link-dataflow-spec-dev` job in\n `.github/workflows/mozcloud-publish.yaml` does this on publish.\n\n CI does not use the pipeline service account for this. A separate\n `-tmpl-pub` account exists purely to upload the spec, and the\n syncstorage-rs GitHub Actions workflow impersonates it through\n workload identity federation. Its bucket grant carries an IAM\n condition restricting writes to the `templates/` prefix, so CI cannot\n touch the runtime `staging/` or `tmp/` paths, and it holds nothing on\n Spanner or Pub/Sub. Both accounts are defined in\n `sync/tf/dev/dataflow_iam.tf`.\n\n To publish manually:\n\n ```console\n gcloud dataflow flex-template build \\\n gs://sync-nonprod-dev-payload-link-dataflow/templates/syncserver-payload-link-dataflow.json \\\n --image us-docker.pkg.dev/moz-fx-sync-prod/sync-prod/syncserver-payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n ```\n\n## Consoles and dashboards\n\nSubstitute the project for other environments.\n\n- [Enabled APIs](https://console.cloud.google.com/apis/dashboard?project=moz-fx-sync-nonprod)\n- [Dataflow jobs](https://console.cloud.google.com/dataflow/jobs?project=moz-fx-sync-nonprod)\n- [Pub/Sub topics](https://console.cloud.google.com/cloudpubsub/topic/list?project=moz-fx-sync-nonprod)\n- [GCS buckets](https://console.cloud.google.com/storage/browser?project=moz-fx-sync-nonprod)\n- [Service accounts](https://console.cloud.google.com/iam-admin/serviceaccounts?project=moz-fx-sync-nonprod)\n- [Spanner databases](https://console.cloud.google.com/spanner/instances/sync/databases?project=moz-fx-sync-nonprod-904c)\n\nThe tenant project Terraform is applied by the `admin-webservices-projects`\nstack in Spacelift.\n\n## Reference\n\nMozilla internal documents, not linked here:\n\n- Design document: \"Sync Spanner 10MB Payloads\"\n- \"Syncstorage Payload Expansion Project Plan\"\n- Confluence: \"Bootstrapping a tenant\", under the SRE space\n- Confluence: \"MozCloud Customer Documentation\", under the SRE space\n\n\n# Load Testing Payload Offload\n\nThis provides a guide on running load tests (`tools/syncstorage-loadtest/`) on the\nGCS payload offload path with Molotov in GCP. This focuses on using gcloud cli\ncommands and short scripts as opposed to the UI.\n\nTwo methods:\n\n- **Method A, against dev env.** Real Spanner, real GCS, real change stream and\n reconciler. Payloads capped at whatever dev's limits allow. Use this to test\n the whole pipeline end to end. Note: it requires webservices-infra updates\n should you need to change application limits.\n- **Method B, local syncserver on the VM.** Spanner emulator, real GCS, your own\n limits. No change stream or reconciler. Use this to test expanded payloads\n larger than the server's default ceiling. This allows for easier ad-hoc\n changing of application limits.\n\nBoth use the load tester's **direct access** mode, which creates its own Hawk\ntoken from the master secret and bypasses Tokenserver.\n\n---\n\n## Prerequisites\n\n### Check your gcloud configuration first\n\nIf you have a configuration for the Spanner emulator and it is enabled,\nit has the bad habit of breaking everything in confusing ways. Check:\n\n```console\ngcloud config list\n```\n\nA configuration containing `auth/disable_credentials = true` or\n`api_endpoint_overrides/spanner` sends every request out unauthenticated and\nredirects Spanner calls to localhost. Switch to a normal configuration:\n\n```console\ngcloud config configurations activate default\ngcloud auth login\n```\n\nSwitch back with `gcloud config configurations activate ` when\nyou return to local emulator work.\n\n### Access you need\n\n| For | You need |\n| --- | --- |\n| Creating the VM | `roles/compute.instanceAdmin.v1` on `moz-fx-sync-nonprod`, which `sync/developers` holds. There is no equivalent grant on prod. |\n| Method A | `roles/secretmanager.secretAccessor` on the tenant project, to read the master secret. |\n| Method B | Write access to a GCS bucket you control. |\n\nSync is onboarded to PAM, so anything beyond the standing grants above can be\nrequested just-in-time. List what is available with `gcloud pam entitlements\nsearch`.\n\n---\n\n## Part 1: Create the load generator VM\n\nUse your own project under the org's `developers` folder (get the folder_no in GCP)\nrather than a tenant project.\n\n```console\nPROJ=\n\ngcloud projects create $PROJ --folder=\ngcloud billing accounts list\ngcloud billing projects link $PROJ --billing-account=\ngcloud services enable compute.googleapis.com --project=$PROJ\n```\n\nNew projects have no default VPC, because the org enforces\n`compute.skipDefaultNetworkCreation`. Create one:\n\n```console\ngcloud compute networks create default --subnet-mode=auto --project=$PROJ\n```\n\nFirewall for IAP SSH required. Without this, the SSH in the next step fails\nwhen no firewall rules are set. Org rules forbid opening tcp/22 to `0.0.0.0/0`:\n\n```console\ngcloud compute firewall-rules create allow-iap-ssh \\\n --project=$PROJ --network=default \\\n --direction=INGRESS --action=allow --rules=tcp:22 \\\n --source-ranges=35.235.240.0/20\n```\n\nCreate the instance in **us-west1**, matching where dev runs.\nAlways check region and match to limit latency.\n\n```console\ngcloud compute instances create sync-loadtest \\\n --project=$PROJ \\\n --zone=us-west1-b \\\n --machine-type=n2-standard-8 \\\n --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \\\n --boot-disk-size=200GB\n```\n\n> **Note on Machine family.** New projects get zero quota for newer families, so C4D\n> fails with `Quota 'CPUS_PER_VM_FAMILY' exceeded. Limit: 0.0`. Most families have\n> their own regional quota (`N2_CPUS`, `E2_CPUS`, `C2_CPUS`); only N1 draws on the\n> general `CPUS` pool. Check whichever matches the family you pick, with\n> `gcloud compute regions describe us-west1 --project=$PROJ --flatten=\"quotas[]\"\n> --format=\"table(quotas.metric,quotas.limit,quotas.usage)\"`.\n\nTo Connect via terminal to instance (you'll likely use this command a lot):\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap\n```\n\nInstall sync & load tests on the VM:\n\n```console\nsudo apt update && sudo apt install -y git python3 python3-venv\nmkdir -p ~/src && cd ~/src\ngit clone --depth=1 https://github.com/mozilla-services/syncstorage-rs.git\ncd syncstorage-rs/tools/syncstorage-loadtest\npython3 -m venv venv && source venv/bin/activate\npip install poetry && poetry install\n```\n\n---\n\n## Part 2: Method A - load test against dev\n\n### Fetch the master secret programmatically\n\nDo not copy and paste the secret. It can contain spaces and a paste\nmay result in confusing 401 errors that are just a mismatched token.\n\n```console\ngcloud auth login --no-launch-browser # VM's own service account cannot read it\n\nSECRET=$(gcloud secrets versions access latest --secret=dev-gke-app-secrets \\\n --project=moz-fx-sync-nonprod \\\n | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"SYNC_MASTER_SECRET\"])')\n\nexport SERVER_URL=\"https://dev.sync.nonprod.webservices.mozgcp.net#${SECRET}\"\necho \"${SERVER_URL%%#*}\"\n```\n\nThat last line prints the origin without revealing the secret. If empty,\nstop: an unset `SERVER_URL` results in a fall back to\n`https://token.stage.mozaws.net` and take the OAuth path instead.\n\n### Verify authentication before running\n\n```console\npoetry run python -c \"\nimport asyncio, aiohttp\nfrom storage.client import StorageClient\nasync def m():\n async with aiohttp.ClientSession() as s:\n c = StorageClient(s)\n r, _ = await c.get('/info/collections', (200, 404))\n print(r.status)\nasyncio.run(m())\n\"\n```\n\nUse `/info/collections`, not `/info/configuration`. The latter takes no auth\nextractor at all and returns 200 to anyone, so it cannot tell you whether your\nsecret is right.\n\n### Run\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\n`OFFLOAD_COLLECTIONS` must name a collection the server actually offloads.\nCheck `SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS` in the deployment's\nvalues file.\n\nThen scale up, redirecting output to a file:\n\n```console\nLARGE_PAYLOAD_PROB=1.0 OFFLOAD_COLLECTIONS=bookmarks \\\n nohup poetry run molotov --processes 1 --workers 5 --duration 600 -v loadtest.py \\\n > /tmp/lt.log 2>&1 &\n```\n\nRamp gradually. Dev is small, and a sudden jump to high concurrency will hit\nits ceiling rather quickly.\n\n### Verify offload actually occurred\n\n```console\ngcloud storage ls -r \"gs://sync-nonprod-dev-syncstorage-payloads/**\" \\\n --project=moz-fx-sync-nonprod | grep -v '[/:]$' | tail -5\n```\n\nThen check at least one object was finalized by the reconciler:\n\n```console\ngcloud storage objects describe --project=moz-fx-sync-nonprod\n```\n\nLook for `custom_fields.committed: 'true'` and `custom_time` pinned to the far\nfuture. The gap between `creation_time` and `update_time` is your\nend-to-end reconciliation latency, which can be worth noting.\n\n> Note: the metadata key is `custom_fields` in `gcloud storage` output, not\n> `metadata`.\n\n---\n\n## Part 3: Method B, local syncserver with raised limits\n\nUse this when you need payloads larger than the deployed server allows. There\nis no runtime or client-side override: limits are read from the server's\nenvironment at startup, which you can easily change on the fly.\nIt runs on whatever `/info/configuration`\nis set to. Exceeding said server-side limits returns 413.\n\n### Your own bucket\n\nDo not use a shared environment's bucket. Make one you own:\n\n```console\ngcloud storage buckets create gs:// --project=$PROJ --location=us-west1\n```\n\nAdd a lifecycle rule. These objects are never finalized in this setup, so\nnothing else will clean them up. A 20 minute run can write hundreds of GB, so we\ndon't necessarily want these sticking around:\n\n```console\necho '{\"rule\":[{\"action\":{\"type\":\"Delete\"},\"condition\":{\"age\":1}}]}' > /tmp/lc.json\ngcloud storage buckets update gs:// --lifecycle-file=/tmp/lc.json --project=$PROJ\n```\n\n### Give the VM write access\n\nThe default compute service account is scoped to `devstorage.read_only`, so it\ncannot write, regardless of IAM settings. Note that scopes can only change while the instance\nis stopped:\n\n```console\nSA=$(gcloud compute instances describe sync-loadtest --project=$PROJ \\\n --zone=us-west1-b --format=\"value(serviceAccounts[0].email)\")\n\ngcloud storage buckets add-iam-policy-binding gs:// \\\n --member=serviceAccount:$SA --role=roles/storage.objectAdmin --project=$PROJ\n\ngcloud compute instances stop sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud compute instances set-service-account sync-loadtest \\\n --project=$PROJ --zone=us-west1-b --service-account=$SA \\\n --scopes=https://www.googleapis.com/auth/cloud-platform\ngcloud compute instances start sync-loadtest --project=$PROJ --zone=us-west1-b\n```\n\nDisk contents survive the stop.\n\nConfirm the new scope took effect by writing **from the VM**. Run locally this\nwould use your own credentials and prove nothing about the instance service\naccount, and `$PROJ` is a local shell variable that does not exist on the VM:\n\n```console\ngcloud compute ssh sync-loadtest --project=$PROJ --zone=us-west1-b \\\n --tunnel-through-iap --command \\\n 'echo hello | gcloud storage cp - gs:///write-check.txt && \\\n gcloud storage rm gs:///write-check.txt'\n```\n\nUsing the instance service account this way means no expiring credentials and\nnothing to mount into the container.\n\n### Build the Docker image\n\n```console\nsudo apt install -y docker.io docker-compose-v2 docker-buildx\nsudo usermod -aG docker $USER && newgrp docker\n\ncd ~/src/syncstorage-rs\nDOCKER_BUILDKIT=1 nohup docker build -t app:build . > /tmp/build.log 2>&1 &\n```\n\n> `DOCKER_BUILDKIT=1` is required. The Dockerfile uses BuildKit cache mounts and\n> the legacy builder fails with \"the --mount option requires BuildKit\".\n\nPulling the published image instead is much faster if you have Artifact\nRegistry read, after `gcloud auth configure-docker us-docker.pkg.dev`. Tag it as\n`app:build` either way, because the compose file hardcodes that tag for its\nsetup container.\n\n### Compose override\n\n```console\ncd ~/src/syncstorage-rs\ncat > docker/docker-compose.offload-local.yaml <<'YAML'\nservices:\n syncserver:\n environment:\n SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET: \n SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS: bookmarks\n SYNC_SYNCSTORAGE__LIMITS__MAX_RECORD_PAYLOAD_BYTES: \"20971520\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_POST_BYTES: \"26214400\"\n SYNC_SYNCSTORAGE__LIMITS__MAX_REQUEST_BYTES: \"26218496\"\nYAML\n```\n\nNo nginx runs in this stack, therefore the chart's `nginxClientMaxBodySize` (default\n`2568k`) does not apply. It does apply to any deployed environment, though.\nNote that raising syncserver limits without it gets you a 413 from nginx,\nbefore syncserver sees a request.\n\n### Docker compose\n\n```console\ndocker tag app:build syncstorage-rs:latest\n\ndocker compose -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.offload-local.yaml up -d\n\ndocker compose -f docker/docker-compose.spanner.yaml ps\n```\n\nThe compose default image is `syncstorage-rs:latest`, hence the tag. Otherwise\nset `SYNCSTORAGE_RS_IMAGE=app:build` on every compose.\n\nConfirm the limits are valid:\n\n```console\ncurl -s http://localhost:8000/1.5/1/info/configuration | python3 -m json.tool\n```\n\n`max_record_payload_bytes` must show your raised value.\n\n### Run\n\nThe master secret in this compose config is `secret0`:\n\n```console\ncd ~/src/syncstorage-rs/tools/syncstorage-loadtest\nsource venv/bin/activate\nexport SERVER_URL=\"http://localhost:8000#secret0\"\n\nLARGE_PAYLOAD_PROB=1.0 LARGE_PAYLOAD_SIZE=20971520 OFFLOAD_COLLECTIONS=bookmarks \\\n poetry run molotov --max-runs 5 -cxv loadtest.py\n```\n\nConfirm the object sizes:\n\n```console\ngcloud storage ls -r --long \"gs:///**\" --project=$PROJ | tail -6\n```\n\n### Sampling memory during a run\n\nMemory can be easily measured here, and is a valuable performance metric.\nUse the below command to sample it to a file (rather than watching a streaming display):\n\n```console\nnohup bash -c 'while true; do date -u +%H:%M:%S | tr \"\\n\" \" \"; docker stats --no-stream --format \"{{.MemUsage}} {{.CPUPerc}}\" docker-syncserver-1; sleep 5; done' > /tmp/mem.log 2>&1 &\n```\n\nThen run the load tests. Then `pkill -f \"docker stats\"` and read `/tmp/mem.log`.\n\n---\n\n## Caveats\n\n**Molotov hides its errors.** A failing `setup_session` is counted and the\ntraceback is discarded, so any auth or setup problem shows as\n`SUCCESSES: 0 | FAILURES: 1` with nothing else.\n\nAlso note `_retry` only checks the HTTP status when the caller passes an\nexpected-statuses tuple. The scenario's `/info/configuration` call passes none,\nso a 503 appears as a JSON decode error rather than \"Response 503\".\n\n**Molotov's display can mangle terminals.** To save yourself the possible headache, always\nredirect to a file for anything but a short test, and if your terminal\ngoes blank or stops echoing, run `reset`.\n\n**Always redirect, never pipe.** Piping molotov's output through `head` or\n`grep` produces basically unhelpful output.\n\n**Co-locating the server and the load tester skews results.** Note that to some degree, Method B\nresults in both compete for the same CPU and memory. Use low concurrency, or alternatively a second VM.\n\n**The emulator is not Spanner.** Method B's write throughput and latency numbers\naren't super helpful and indicative of live spanner performance.\n\n**Fastly NGWAF is the edge, in blocking mode.** Dev sets\n`ngwaf_agent_level = \"block\"`, so sustained 403s partway through a run against a\ndeployed environment are worth checking there before blaming the load tester.\nThe `sync-policy` Cloud Armor resource still exists, but neither backend config\nattaches it, so it is not the cause.\n\n**Check where the Spanner metrics live.** Spanner for the sync tenant is in a\nseparate project, `moz-fx-sync-nonprod-904c` (now named `sync-legacy-nonprod`),\nand `sync/developers` has no access to it, so the console is unavailable. Since\nthe move to `sync-legacy` the metrics come from Grafana's `gcp-v2-nonprod`\ndatasource, not `gcp-v1-nonprod`, which no longer returns them.\n\n---\n\n## Cleanup\n\n```console\ngcloud compute instances delete sync-loadtest --project=$PROJ --zone=us-west1-b\ngcloud storage rm -r gs:///** --project=$PROJ\n```\n\nDelete the instance when you are done. An idle load generator bills\nindefinitely!\n\n\n# Documentation for `purge_ttl.py` (PostgreSQL)\n\n## Summary\n\nThe `purge_ttl.py` script is a utility for purging expired Time-To-Live (TTL) records from a Sync storage PostgreSQL database. It deletes expired entries from the `batches` and/or `bsos` tables, with optional filtering by collection ID and a dry-run mode for verifying query logic without modifying data.\n\n> Spanner no longer uses this script — it expires records natively via its built-in [row deletion (TTL) policies](https://cloud.google.com/spanner/docs/ttl). This utility applies to PostgreSQL deployments only.\n\n---\n\n## Status\n- Runs as a regularly scheduled job (e.g. a cron job) in PostgreSQL (\"enterprise\") deployments.\n- The script is bundled in the main `syncserver-postgres` image and invoked from there; there is no separate utility image.\n\n## Specifics\n\n- **Database**: PostgreSQL (accepts a `postgresql://` or `postgres://` DSN).\n- **Tables**:\n - `batches`: Contains batch entries (child `batch_bsos` rows are removed via the schema's foreign-key cascade).\n - `bsos`: Stores Sync Basic Storage Objects (BSO).\n- **Supported Modes**:\n - `batches`: Purges expired entries in the `batches` table.\n - `bsos`: Purges expired entries in the `bsos` table.\n - `both`: Performs purges on both tables.\n- **Expiry Modes**:\n - `now`: Purges entries with `expiry < CURRENT_TIMESTAMP`.\n - `midnight`: Purges entries with `expiry < DATE_TRUNC('day', CURRENT_TIMESTAMP AT TIME ZONE 'UTC')`.\n\nThe script tracks execution duration and rows affected using StatsD metrics for performance monitoring.\n\n---\n\n## Notes\n\n- Ensure the database role used by the DSN has `DELETE` privileges on the target tables.\n- Use the `--dryrun` option to verify query logic before actual purging.\n- Consider setting up automated monitoring for long-running operations or performance issues.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3 is installed.\n2. **Poetry**: Install [Poetry](https://python-poetry.org/docs/#installation).\n3. **Dependencies**: From `tools/postgres`, run `poetry install`.\n4. **Environment Variables:**\n `SYNC_SYNCSTORAGE__DATABASE_URL`: Database connection URL (e.g., `postgresql://user:pass@host:5432/syncstorage`). Used when `--database_url` is not supplied.\n\n### Usage\n\nRun the script using the following command:\n ```bash\n poetry run python purge_ttl.py [options]\n ```\n\n#### Options\n\n| Option | Description | Default |\n|---------------------------------|----------------------------------------------------------------------------------------|----------------------------------|\n| `-u`, `--database_url` | PostgreSQL DSN (`postgresql://...` or `postgres://...`). Required if the env var is unset. | `SYNC_SYNCSTORAGE__DATABASE_URL` |\n| `--collection_ids`, `--ids` | Comma-separated list of collection IDs to purge. | `[]` |\n| `--mode` | Purge mode: `batches`, `bsos`, or `both`. | `both` |\n| `--expiry_mode` | Expiry mode: `now` (current timestamp) or `midnight` (start of current day, UTC). | `midnight` |\n| `--dryrun` | Perform a dry run without making changes to the database. | `False` |\n\n#### Examples\n\n##### Example 1: Basic Purge\nPurge expired entries from both `batches` and `bsos` tables using the DSN from the environment:\n```bash\n poetry run python purge_ttl.py\n```\n\n##### Example 2: Specify the Database URL\n```bash\n poetry run python purge_ttl.py -u postgresql://user:pass@host:5432/syncstorage\n```\n\n##### Example 3: Filter by Collection IDs\nPurge only for specific collection IDs:\n```bash\n poetry run python purge_ttl.py --collection_ids [123,456,789]\n```\n\n##### Example 4: Purge a Single Table\n```bash\n poetry run python purge_ttl.py --mode bsos\n```\n\n##### Example 5: Perform a Dry Run\nTest the script without making actual changes:\n```bash\n poetry run python purge_ttl.py --dryrun\n```\n\n### Detailed Usage\n\n1. **Connecting to PostgreSQL**:\n - The script builds a SQLAlchemy engine from the supplied DSN. A `postgres://` scheme is normalized to `postgresql://` for newer SQLAlchemy versions.\n\n2. **Purge Modes**:\n - `batches`: Deletes expired entries from the `batches` table.\n - `bsos`: Deletes expired BSOs.\n - `both`: Executes purges on both `batches` and `bsos`.\n\n3. **Expiry Conditions**:\n - `now`: Purge entries that have already expired at the current timestamp.\n - `midnight`: Purge entries that expired at or before the start of the current UTC day.\n\n4. **Query Customization**:\n - An optional collection-ID filter is appended via the `add_conditions` helper using bound parameters.\n\n5. **Performance Monitoring**:\n - Metrics for execution duration and rows affected are logged and sent to StatsD for monitoring.\n\n6. **Dry Run**:\n - Enabling the `--dryrun` flag ensures that the queries are constructed and logged without executing them on the database.\n\n\n# Documentation for `purge_old_records.py`\n\n## Summary\n\nThe `purge_old_records.py` script is an administrative utility for managing obsolete user records in Tokenserver. It removes outdated user records from the database and deletes associated data from storage nodes. This process helps reduce storage overhead, improve database performance, and maintain the health of the Tokenserver system.\n\nObsolete records are those replaced by newer records for the same user or marked for deletion if the user has deleted their account. The script can run in batch mode for periodic cleanup and includes options for dry-run testing and forced purging when nodes are down.\n\n---\n\n## Status\n - Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-purge-old-records-0`\n - See YAML configuration when editing each job.\n - See Kubernetes Engine Workload Panel in [sync-prod](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information. \n\n---\n\n## Specifics\n\n- **Primary Functionality**:\n - Deletes obsolete user records.\n - Issues delete requests to user storage nodes to purge related data.\n- **Optional Administrative Task**:\n - The script complements internal record replacement handled by the backend but is not mandatory for system operation.\n- **Batch Processing**:\n - Operates in loops, processing records in batches of a configurable size.\n- **Grace Period**:\n - Provides a grace period to avoid prematurely deleting recently replaced records.\n- **Dry Run**:\n - Offers a non-destructive mode for testing.\n\n---\n\n## Notes\n- **Regular Use**:\n - Running this script regularly can help maintain system performance and reduce storage overhead.\n- **Concurrency**:\n - When running multiple instances of this script, use the `--max-offset` option to reduce collisions.\n- **Forced Deletion**:\n - Use the `--force` option cautiously, especially if storage nodes are down.\n\n---\n\n## Instructions for Running the Script\n\n### Prerequisites\n\n1. **Python Environment**: Ensure Python 3.7+ is installed.\n2. **Dependencies**:\n - Install required Python packages:\n `pip install requests hawkauthlib backoff`\n\n3. **Configuration**:\n - Set up access to the Tokenserver database.\n - Provide necessary metrics and logging configurations.\n\n### Usage\n\nRun the script using the following command:\n```bash\npython purge_old_records.py [options] secret\n```\n\n\n### Options\n\n| Option | Description | Default |\n|-------------------------|-------------------------------------------------------------------------------------------------|--------------|\n| `--purge-interval` | Interval in seconds to sleep between purging runs. | `3600` |\n| `--grace-period` | Grace period in seconds before deleting records. | `86700` (24h5m) |\n| `--max-per-loop` | Maximum number of items to process in each batch. | `10` |\n| `--max-offset` | Random offset to select records for purging, reducing collisions in concurrent tasks. | `0` |\n| `--max-records` | Maximum number of records to purge before exiting. | `0` (no limit) |\n| `--request-timeout` | Timeout in seconds for delete requests to storage nodes. | `60` |\n| `--oneshot` | Perform a single purge run and exit. | Disabled |\n| `--dryrun` | Test the script without making destructive changes. | Disabled |\n| `--force` | Force purging even if the user's storage node is marked as down. | Disabled |\n| `--override_node` | Specify a node to override for deletions if data is copied. | None |\n| `--range_start` | Start of UID range to process. | None |\n| `--range_end` | End of UID range to process. | None |\n| `--human_logs` | Enable human-readable logs. | Disabled |\n| `--last-seen-staging-table` | BigQuery last-seen staging table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-table`. | None |\n| `--last-seen-table` | BigQuery last-seen final table (`PROJECT.DATASET.TABLE`). Requires `--last-seen-staging-table`. | None |\n\n### Examples\n\n#### Example 1: Basic Purge\nPerform a basic purge of obsolete user records:\n```bash\npython purge_old_records.py secret_key\n```\n#### Example 2: Grace Period and Dry Run\nPurge records with a 48-hour grace period in dry-run mode:\n```bash\npython purge_old_records.py --grace-period 172800 --dryrun secret_key\n```\n\n#### Example 3: Specify Range and Offset\nPurge records within a UID range with a random offset:\n\n```bash\npython purge_old_records.py --range_start uid_start --range_end uid_end --max-offset 50 secret_key\n```\n\n#### Example 4: Force Purge on Downed Nodes\nForce the deletion of data on downed nodes:\n```bash\npython purge_old_records.py --force secret_key\n```\n\n---\n\n## Detailed Usage\n\n1. **Batch Processing**:\n - The script processes records in batches defined by the `--max-per-loop` option.\n - Each batch is fetched from the database using random offsets to avoid overlapping with concurrent runs.\n\n2. **Grace Period**:\n - The grace period ensures that recently replaced records are not prematurely deleted.\n\n3. **Storage Node Cleanup**:\n - For each user, the script sends a delete request to their storage node to remove associated data.\n\n4. **Metrics Tracking**:\n - Tracks operations like user record deletions, service data deletions, and errors using metrics integration.\n\n5. **Error Handling**:\n - Uses exponential backoff to retry failed HTTP requests.\n - Detects loops in batch processing and raises exceptions.\n\n6. **Dry Run Mode**:\n - Simulates deletions without modifying the database or storage nodes, useful for testing.\n\n7. **Account Last-Seen Deletion**:\n - With both `--last-seen-*` options, each record retired by an FxA account deletion first has its account's rows deleted from the BigQuery last-seen tables, just before the record itself.\n - If that job fails or times out, the retired record stays, `replaced_at` is bumped, and it is retried after another grace period. Other records continue normally.\n - The default grace period (24 hours 5 minutes) outlasts the one-day OAuth token lifetime plus the upload delay, so an account's last activity normally reaches BigQuery before purge deletes it.\n\n---\n\n\nVerify all required environment variables are set before running integration tests.\n\nYou are a pre-flight assistant for syncstorage-rs integration tests. Your job is to check that the shell environment is correctly configured so tests will actually run, and surface any missing or suspicious values before the user wastes time on a failing test run.\n\nDo NOT print or log the values of any secrets. Use placeholders like `[set]` or `[not set]` when reporting on secret variables.\n\n## Variables to check\n\n### Required for all integration test runs\n\n| Variable | Expected | Notes |\n|---|---|---|\n| `SYNC_MASTER_SECRET` | any non-empty string | secret — report [set] or [not set] only |\n| `SYNC_SYNCSTORAGE__DATABASE_URL` | starts with `mysql://`, `postgresql://`, or `spanner://` | |\n| `SYNC_TOKENSERVER__DATABASE_URL` | starts with `mysql://` or `postgresql://` | |\n| `TOKENSERVER_HOST` | starts with `http://` or `https://` | default: `http://localhost:8000` |\n| `SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL` | starts with `http://` or `https://` | default: `http://localhost:6000` for local |\n| `PYTHONPATH` | contains `tools/` path | required for PyO3 and test imports |\n\n### Required only for real FxA e2e tests (`test_e2e.py`)\n\n| Variable | Notes |\n|---|---|\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N` | secret — report [set] or [not set] only |\n| `SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E` | secret — report [set] or [not set] only |\n\n## Step 1 — Check each variable\n\n```bash\necho \"SYNC_MASTER_SECRET: $([ -n \"$SYNC_MASTER_SECRET\" ] && echo '[set]' || echo '[NOT SET]')\"\necho \"SYNC_SYNCSTORAGE__DATABASE_URL: ${SYNC_SYNCSTORAGE__DATABASE_URL:-[NOT SET]}\"\necho \"SYNC_TOKENSERVER__DATABASE_URL: ${SYNC_TOKENSERVER__DATABASE_URL:-[NOT SET]}\"\necho \"TOKENSERVER_HOST: ${TOKENSERVER_HOST:-[NOT SET - default: http://localhost:8000]}\"\necho \"SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: ${SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL:-[NOT SET - default: http://localhost:6000]}\"\necho \"PYTHONPATH: ${PYTHONPATH:-[NOT SET]}\"\n```\n\n## Step 2 — Validate database URL format\n\nFor `SYNC_SYNCSTORAGE__DATABASE_URL` and `SYNC_TOKENSERVER__DATABASE_URL`:\n- Check the scheme matches a supported backend (`mysql`, `postgresql`, `spanner`)\n- For `postgres://` prefix: note that SQLAlchemy 1.4+ requires `postgresql://` — flag this if present\n- For Spanner: confirm `SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST` is set if the URL contains `test-project` or `emulator`\n\n## Step 3 — Check PYTHONPATH contains tools/\n\n```bash\npython3 -c \"import tools.integration_tests\" 2>&1\n```\n\nIf this fails, the `PYTHONPATH` is not set correctly for test imports.\n\n## Step 4 — Check server reachability (if TOKENSERVER_HOST is set)\n\n```bash\ncurl -sf \"${TOKENSERVER_HOST:-http://localhost:8000}/__heartbeat__\" && echo \"server: reachable\" || echo \"server: NOT REACHABLE\"\n```\n\n## Output format\n\nReport each variable as **OK**, **WARNING**, or **MISSING**:\n\n| Variable | Status | Detail |\n|---|---|---|\n\nThen list any blockers (MISSING required vars, unreachable server) followed by warnings (format issues, defaults in use).\n\nEnd with **Ready to run tests** or **N issues need to be resolved**.\n\n\nExamine git history of files changed in the current branch to identify regressions, re-introduced bugs, or changes that conflict with past fixes.\n\nYou are a senior engineer doing a history-aware code review. Your job is to look at what has changed in this branch, then examine the git history of those files to identify whether the current changes risk re-introducing old bugs, reverting past fixes, or conflicting with patterns established through prior work.\n\n## Step 1 — Get the current diff\n\n```bash\ngit diff main...HEAD\n```\n\n## Step 2 — Get the list of changed files\n\n```bash\ngit diff main...HEAD --name-only\n```\n\n## Step 3 — For each changed file, examine recent commit history\n\n```bash\ngit log --oneline -20 -- \n```\n\n## Step 4 — Inspect relevant commits in full\n\nFor each commit that looks relevant (bug fixes, reverts, security patches, refactors on the same lines), inspect the full diff:\n\n```bash\ngit show \n```\n\n## Step 5 — Look for reverted or fix commits\n\n```bash\ngit log --oneline --all --grep=\"revert\" -- \ngit log --oneline --all --grep=\"fix\" -- \n```\n\nUse your judgment about which historical commits are worth digging into. Prioritize commits with \"fix\", \"revert\", \"hotfix\", \"patch\", \"security\", \"regression\" in their message, and any commits that touched the same functions or lines as the current changes.\n\n## Analysis checklist\n\nFor each changed file, work through the following. Report findings with:\n- **Severity:** High / Medium / Low\n- **Location:** file:line\n- **Issue:** what the historical context reveals\n- **Relevant commit(s):** hash + message\n- **Recommendation:** what to verify or change\n\n### 1. Re-introduced bugs\n- Does the current diff restore code that was previously removed by a bug fix?\n- Are there commits in the history that explicitly fixed logic that the current change modifies or removes?\n\n### 2. Reverted or rolled-back patterns\n- Has this code been reverted before? If so, why — and does the current change repeat the same pattern?\n- Are there \"Revert X\" commits in the history that suggest a prior attempt at this change failed?\n\n### 3. Previously fixed security issues\n- Does the history show security-related fixes on these lines?\n- Does the current change touch or weaken those protections?\n- Pay particular attention to: HAWK/JWT token handling, FxA OAuth validation, gRPC error suppression logic, DB migration constraints.\n\n### 4. Repeated churn\n- Has this file or function been modified many times in a short period? High churn is a signal of instability.\n- Does the current change look like it continues a pattern of patching symptoms rather than fixing the root cause?\n\n### 5. Conflicting intent\n- Do commit messages indicate a deliberate design decision that the current change reverses without explanation?\n- Are there TODOs or FIXMEs introduced by prior commits that the current change should have addressed but didn't?\n\n### 6. Migration or deprecation conflicts\n- Does the history show a migration away from a pattern that the current change re-introduces?\n- Examples: callback → async, legacy auth → FxA OAuth, MySQL-only → multi-backend.\n\n### 7. Test regressions\n- Were tests previously added to cover a bug that the current change might invalidate?\n- Do any historical fix commits include test additions that are now at risk of being bypassed?\n\n## Output format\n\nLead with a summary table:\n\n| Severity | File | Issue | Relevant commit |\n|---|---|---|---|\n\nFollow with per-file detail. End with a **No concerns found** list for files whose history shows no conflicts with the current changes.\n\nIf the git history is shallow or sparse for a file, note that and flag it as low confidence.\n\n\nReview database migration changes against the repo's published-migration policy.\n\nYou are a database migration reviewer for syncstorage-rs. The repo has a hard rule: **never edit a published migration file**. Only new forward migrations with a corresponding rollback are allowed. Your job is to enforce this and catch violations before merge.\n\n## Migration locations\n\n```\nsyncstorage-mysql/migrations/ # up.sql / down.sql per timestamped dir\nsyncstorage-postgres/migrations/\ntokenserver-mysql/migrations/\ntokenserver-postgres/migrations/\nsyncstorage-spanner/src/schema.ddl # Spanner schema — treated as published\nsyncstorage-spanner/src/*.sql # inline Spanner queries\n```\n\nEach migration directory is named `YYYY-MM-DD-hhmmss_description/` and contains `up.sql` and `down.sql`.\n\n## Step 1 — Find changed migration files\n\n```bash\ngit diff main...HEAD --name-only | grep -E \"migrations/|schema\\.ddl|\\.sql\"\n```\n\n## Step 2 — Classify each changed file\n\nFor each changed SQL file:\n\n- **New migration directory** (path did not exist on main) → allowed, proceed to Step 3\n- **Existing migration file modified** (path existed on main and content changed) → **VIOLATION** — flag as a blocker\n- **schema.ddl modified** → **VIOLATION** — flag as a blocker\n- **Inline .sql query file modified** (non-migration) → review for intent, flag if it changes schema\n\nCheck whether a file existed on main:\n```bash\ngit show main:\n```\n\n## Step 3 — Validate new migrations\n\nFor each new migration directory:\n\n1. Confirm both `up.sql` and `down.sql` exist\n2. Confirm `down.sql` is a genuine rollback of `up.sql` (not empty, not a no-op)\n3. Check that the timestamp prefix is newer than all existing migrations in that crate\n4. Check for destructive operations in `up.sql` (`DROP TABLE`, `DROP COLUMN`, `TRUNCATE`) — flag these for explicit review\n5. Check that column types and constraints are consistent with existing schema in the same crate\n\n## Step 4 — Cross-backend consistency\n\nIf migrations exist for both mysql and postgres variants of the same crate, verify the schema changes are semantically equivalent (same columns, types mapped appropriately, same constraints).\n\n## Output format\n\n**Violations (blockers):**\nList any edits to published files with file path and the nature of the change.\n\n**New migrations:**\nFor each new migration: name, what it does, whether down.sql is valid, any destructive ops.\n\n**Warnings:**\nNon-blocking concerns (e.g. missing index, risky destructive op, cross-backend inconsistency).\n\n**Verdict:** Pass / Violations found\n\n\nGenerate a PR title and summary from the current branch diff, following the repo's commit and PR conventions.\n\nYou are a PR description writer for syncstorage-rs. Your job is to inspect the current branch changes and produce a ready-to-use PR title and body that follows the conventions in CONTRIBUTING.md.\n\n## Step 1 — Gather branch context\n\n```bash\ngit diff main...HEAD --name-only\ngit log main...HEAD --oneline\ngit diff main...HEAD --stat\n```\n\n## Step 2 — Read the full diff for context\n\n```bash\ngit diff main...HEAD\n```\n\nFocus on understanding *what changed and why*, not just listing files.\n\n## Step 3 — Classify the change type\n\nBased on changed files, determine the primary `type`:\n\n- `feat` — new functionality added\n- `fix` — bug fix\n- `refactor` — code restructured, no behavior change\n- `test` — test additions or changes only\n- `chore` — build, CI, tooling, dependencies\n- `docs` — documentation only\n- `perf` — performance improvement\n- `style` — formatting, lint fixes only\n\nIf multiple types apply, use the most significant one.\n\n## Step 4 — Determine scope (optional)\n\nIf the change is clearly scoped to one subsystem, add it in parentheses:\n- `tokenserver`, `syncstorage`, `spanner`, `mysql`, `postgres`, `auth`, `migrations`, `ci`, `tools`\n\nExample: `refactor(tokenserver): ...`\n\n## Step 5 — Draft the PR title\n\nRules from CONTRIBUTING.md:\n- Format: `type(scope): subject` or `type: subject`\n- Subject: imperative present tense, no capital first letter, no trailing period\n- Keep it under 70 characters\n\n## Step 6 — Draft the PR body\n\n```markdown\n## Summary\n\n<2–4 bullet points covering what changed and why. Lead with the motivating\nproblem or context, then the solution. Skip obvious restatements of the title.>\n\n## Test plan\n\n\n```\n\nPre-fill the test plan based on file types changed:\n\n- Rust source changed → `- [ ] \\`make clippy_\\` passes`, `- [ ] Unit tests pass (\\`make test\\`)`\n- Python source changed → `- [ ] \\`make ruff-lint\\` and \\`make mypy\\` pass`, `- [ ] Integration tests pass (\\`make run_local_e2e_tests\\`)`\n- SQL migrations added → `- [ ] \\`/migration-review\\` passes`, `- [ ] Migration applies cleanly on a fresh DB`\n- `.github/` changed → `- [ ] CI workflow runs successfully on this branch`\n- `CLAUDE.md` or docs changed → `- [ ] Renders correctly on GitHub`\n\n## Step 7 — Output\n\nPrint the final PR title and body, ready to paste into GitHub. Also include the suggested branch name if the current branch name doesn't follow `type/description-STOR-####` or `type/description-####` format.\n\nNote any breaking changes that should be called out explicitly.\n\n\nRun all Python lint, format, type, and security checks for the tools/ directory.\n\nYou are a Python code quality assistant for the syncstorage-rs tools and integration tests. Your job is to run the full suite of Python checks in the correct order, report findings clearly, and tell the user exactly what needs to be fixed before the PR can merge.\n\nAll checks operate on the `tools/` directory. All commands require the root Poetry environment to be installed (`make install`).\n\n## Step 1 — Check Poetry environment\n\n```bash\npoetry env info\n```\n\nIf no environment is active, tell the user to run `make install` first and stop.\n\n## Step 2 — Format check\n\n```bash\nmake ruff-fmt-chk\n```\n\nIf there are formatting issues, tell the user to run `make ruff-format` to auto-fix, then re-run. Do not proceed past format failures.\n\n## Step 3 — Lint\n\n```bash\nmake ruff-lint\n```\n\nReport each warning/error with file:line and rule code.\n\n## Step 4 — Type checking\n\n```bash\nmake mypy\n```\n\nReport type errors with file:line. Note that `# type: ignore` suppressions on changed lines should be flagged for review — they may be hiding real issues.\n\n## Step 5 — Docstring validation\n\n```bash\nmake pydocstyle\n```\n\nReport missing or malformed docstrings. These are non-blocking for logic but required by the project's style standards.\n\n## Step 6 — Security scan\n\n```bash\nmake bandit\n```\n\nFlag any **HIGH** severity findings as blockers. List **MEDIUM** as warnings.\n\n## Output format\n\nReport each check as **PASS**, **WARNINGS**, or **FAIL**:\n\n| Check | Status | Issues |\n|---|---|---|\n| Format (ruff) | | |\n| Lint (ruff) | | |\n| Types (mypy) | | |\n| Docstrings (pydocstyle) | | |\n| Security (bandit) | | |\n\nList all actionable findings below the table with file:line references.\n\nEnd with **All checks passed** or **N checks need attention** verdict.\n\n\nRun Clippy, fmt check, and cargo audit for a specific Rust backend.\n\nYou are a Rust build and lint assistant for the syncstorage-rs workspace. Your job is to run the correct set of checks for a given backend, report findings clearly, and flag anything that needs attention before the code is merged.\n\n## Step 1 — Determine the backend\n\nAsk the user which backend to check if not already specified: `mysql`, `postgres`, or `spanner`. If the current branch name or open files suggest a backend, infer it and confirm.\n\n## Step 2 — Run format check\n\n```bash\ncargo fmt -- --check\n```\n\nReport any files that are not formatted. If there are formatting issues, stop and tell the user to run `cargo fmt` before proceeding.\n\n## Step 3 — Run Clippy for the chosen backend\n\nUse the exact make target:\n\n```bash\nmake clippy_mysql # for mysql\nmake clippy_postgres # for postgres\nmake clippy_spanner # for spanner\n```\n\n## Step 4 — Run release-mode Clippy\n\n```bash\nmake clippy_release_mysql # for mysql\nmake clippy_release_postgres # for postgres\nmake clippy_release_spanner # for spanner\n```\n\nRelease mode catches dead code and issues only visible with optimizations enabled.\n\n## Step 5 — Run cargo audit\n\n```bash\ncargo audit\n```\n\nFlag any advisories with severity **high** or **critical** as blockers. List medium/low advisories as warnings.\n\n## Output format\n\nSummarize findings in this order:\n\n1. **Format** — pass or list of unformatted files\n2. **Clippy (debug)** — pass or list of warnings/errors with file:line\n3. **Clippy (release)** — pass or list of warnings/errors with file:line\n4. **Audit** — pass or list of advisories (severity, crate, advisory ID)\n\nEnd with a clear **Ready to merge** / **Needs attention** verdict.\n\n\n# Example scripts for working with syncstorage-rs\n\n* `put.bash` - First `chmod +x put.bash` then `./put.bash` to simulate a PUT.\n\n\n# Make a Hawk compatible Auth header\n\n## Dependencies and Environment Setup:\nTo use the syncstorage-rs `make_hawk_token.py` script, you'll need a Python =>3.10 development environment with `Poetry` installed. You can also directly call the script using `Poetry` as described in step 5.\n\nThe easiest solution recommended to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments\nas a way to isolate the dependencies from other directories.\n1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.\n2. Follow the instructions to install the `pyenv-virtualenv` plugin.\nSee the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.\n3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.\n\n Ex:\n ```shell\n export PATH=\"$HOME/.pyenv/bin:$PATH\"\n eval \"$(pyenv init -)\"\n eval \"$(pyenv virtualenv-init -)\"\n ```\n4. Install version, create virtualenv, activate and install dependencies from inside the `hawk/` directory.\n**Note** you can simply install dependencies, not create a virtual environment and invoke the script using `poetry run`.\n\n ```shell\n $ cd syncstorage-rs/tools/hawk\n # pyenv version install\n $ pyenv install 3.14\n\n # creates named, associated virtualenv\n $ pyenv virtualenv 3.14 hawk # or whatever project name you like.\n $ pyenv local hawk # activates virtual env whenever you enter this directory. \n\n # Install dependencies\n $ pip install poetry\n $ poetry install\n ```\n\n5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):\n`poetry run python make_hawk_token.py`\n\n## Create a Token Header:\n\nYou'll need to pass along your `SYNC_MASTER_SECRET` and the uri you'll be testing in order to generate a valid Hawk Id:\n\n`poetry run python make_hawk_token.py --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`\n\n** For testing against uri's using methods other than GET, you'll need to pass along the `--method` flag to generate your token. Ie, `poetry run python make_hawk_token.py --method PUT --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header`. See [examples/put.bash](https://github.com/mozilla-services/syncstorage-rs/blob/master/tools/examples/put.bash) for an example of this.\n\nUse `-h` for help.\n\nBy default, with no passed arguments, a default Hawk Id token will be generated mapping to your localhost:8000 with\nthe path `http://localhost:8000/1.5/1/storage/col2/`.\n\n\n# payload-link-publisher-py\n\nDev/E2E-only Python variant of the payload-link Dataflow publisher.\n\n**Prod uses the Java flex template one directory up.** This variant\nexists so the docker-compose e2e stack can run without a JVM, without\nDataflow, and without minute-scale iteration times.\n\n## Design summary\n\n- **Not Apache Beam.** Python's Beam SDK has no Spanner change-stream\n connector; a Beam pipeline in Python would either require a\n cross-language Java expansion (JVM back in) or a custom Source\n (more code than the whole non-Beam approach). Instead we poll the\n `READ_payload_link_changes` TVF directly via\n `google.cloud.spanner.Client`.\n- **Wire-format identical to the Java job.** `serialize_record`\n produces the exact JSON shape `PayloadLinkChangesToPubSub.serializeRecord`\n produces, so the reconciler consumes either interchangeably.\n- **Partition tracking.** Follows partition splits: reads `_root`,\n picks up child-partition tokens from any `child_partitions_record`,\n reads each child from its advertised `start_timestamp`, and retires\n a parent when a child announces it. Handles `OUT_OF_RANGE` by\n dropping ended partitions. Required in practice — the Spanner\n emulator routes DataChangeRecords to child partitions almost\n immediately, so a `_root`-only reader sees zero DCRs.\n- **In-memory cursor.** On restart, resumes from `now - POLL_OVERLAP_SECONDS`\n on `_root`. Fine for tests (compose brings the pod up alongside the\n stream); would lose events across restarts in prod.\n- **Hardcoded row-shape offsets.** Rather than introspect the\n `READ_` TVF's ChangeRecord / DataChangeRecord /\n ChildPartitionsRecord STRUCT layout at runtime, `publisher.py`\n documents the field positions and accesses them positionally. If a\n future Spanner rev reorders the STRUCT, tests fail loudly and one\n offset map gets updated. Acceptable trade-off for a dev/E2E tool.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-spanner`, `google-cloud-pubsub`, `statsd`. |\n| `publisher.py` | Entry point: poll TVF → filter → serialize → publish. |\n| `utils.py` | `extract_payload_link`, `is_payload_link_actionable` — mirrors the Java filter. |\n| `test_publisher.py` | Unit tests for the pure functions. |\n| `Dockerfile` | `python:3.13-slim` + poetry install. |\n\n## Environment\n\n| Variable | Required | Default | Notes |\n|---|---|---|---|\n| `SPANNER_PROJECT_ID` | yes | — | |\n| `SPANNER_INSTANCE_ID` | yes | — | |\n| `SPANNER_DATABASE_ID` | yes | — | |\n| `PUBSUB_PROJECT_ID` | yes | — | |\n| `PUBSUB_TOPIC` | yes | — | e.g. `payload-link-changes` |\n| `SPANNER_CHANGE_STREAM_NAME` | no | `payload_link_changes` | |\n| `HEARTBEAT_MS` | no | `1000` | TVF heartbeat interval. |\n| `POLL_OVERLAP_SECONDS` | no | `5` | Restart re-reads this window. |\n| `SPANNER_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-spanner`. |\n| `PUBSUB_EMULATOR_HOST` | no | — | Auto-honored by `google-cloud-pubsub`. |\n| `STATSD_HOST` / `STATSD_PORT` | no | — | Standard `statsd.defaults.env` pair. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-link-dataflow/payload-link-publisher-py\npoetry install\npoetry run pytest\npoetry run mypy .\npoetry run ruff check .\n```\n\n`poetry.lock` is generated on first install (not yet checked in).\n\nUnit tests cover `extract_payload_link`, `is_payload_link_actionable`,\n`serialize_record`, and `_to_json_string`. Not covered by unit tests:\n`poll_partition` (needs a live Spanner emulator) and\n`publish_if_actionable` (needs a Pub/Sub emulator). Both are\nexercised end-to-end by the docker-compose reconciliation stack\n(`make docker_run_reconciliation_e2e_tests`).\n\n## Metrics\n\n- `payload_link_publisher.published` — records published to Pub/Sub\n- `payload_link_publisher.filtered` — inert records dropped by\n `is_payload_link_actionable`\n- `payload_link_publisher.errors.publish` — publish failures\n\n## Related\n\n- Java flex template (prod): `tools/payload-link-dataflow/` (parent dir)\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (`payload_link_changes`)\n- Downstream reconciler: `tools/payload-reconciler/`\n- Runbook: `docs/src/tools/payload_link_reconciler.md`\n\n\n# payload-link-dataflow\n\nCustom Dataflow flex template that reads the `payload_link_changes`\nSpanner change stream, drops records whose every mod has `payload_link`\nNULL on both sides, and publishes the remainder to a Pub/Sub topic.\n\nThe downstream consumer is the `payload-reconciler` cronjob\n(`tools/payload-reconciler/`), which finalizes newly-committed GCS\npayload objects (`committed=true`, `customTime=MAX`) and deletes\norphaned ones.\n\n## Directory layout\n\nThe Java source tree under `src/` is a standalone Apache Beam pipeline\nwe own and build directly. It is intentionally minimal: it uses Beam's\nstock `SpannerIO.readChangeStream` and `PubsubIO.writeStrings`, plus the\nfilter and a small JSON serializer. We do **not** vendor or depend on\nthe upstream GoogleCloudPlatform/DataflowTemplates Java tree.\n\n| Path | Purpose |\n|---|---|\n| `pom.xml` | Maven project (Apache Beam + Jackson + SLF4J). |\n| `src/main/java/.../PayloadLinkChangesToPubSub.java` | The pipeline: read change stream → filter → serialize → publish. |\n| `src/main/java/.../PayloadLinkOptions.java` | Flex template parameters. |\n| `src/test/java/.../PayloadLinkChangesToPubSubTest.java` | Unit tests for the filter predicate. |\n| `metadata.json` | Flex template parameter schema (consumed by `gcloud dataflow flex-template build`). |\n| `Dockerfile` | Multi-stage build: Maven → flex template launcher base image. |\n| `upstream-customization.patch` | **Reference documentation only.** Unified diff showing the **filter delta only** against the upstream template; see \"Beyond the filter\" below for the rest. Not a build input. |\n| `upstream.txt` | The upstream commit SHA the reference patch was last reviewed against. Update when re-reviewing intent against upstream. |\n| `generate-full-delta.sh` | On-demand: produces a unified diff of the **complete** delta between upstream (at the pinned SHA) and our impl. Not maintained as a checked-in artifact -- runs against the source tree at execution time. |\n| `payload-link-publisher-py/` | Dev/E2E-only Python variant of this publisher (not Beam, not shipped to prod). See its own README. |\n| `README.md` | This file. |\n\n## Build\n\n```bash\ndocker build -t payload-link-dataflow:local tools/payload-link-dataflow\n```\n\n`mvn package` runs the unit tests, so a change that breaks the filter\ncontract fails the image build.\n\nFor a local non-Docker build:\n\n```bash\ncd tools/payload-link-dataflow\nmvn -B package\n```\n\nThe bundled jar lands at\n`target/payload-link-dataflow-bundled-1.0-SNAPSHOT.jar`.\n\n## Publish + launch (operator workflow; lives in webservices-infra)\n\n```bash\ndocker tag payload-link-dataflow:local /payload-link-dataflow:\ndocker push /payload-link-dataflow:\n\ngcloud dataflow flex-template build \\\n gs:///templates/payload-link-dataflow.json \\\n --image /payload-link-dataflow: \\\n --sdk-language JAVA \\\n --metadata-file tools/payload-link-dataflow/metadata.json\n\ngcloud dataflow flex-template run payload-link-dataflow- \\\n --template-file-gcs-location gs:///templates/payload-link-dataflow.json \\\n --parameters spannerProjectId= \\\n --parameters spannerInstanceId= \\\n --parameters spannerDatabase= \\\n --parameters spannerMetadataInstanceId= \\\n --parameters spannerMetadataDatabase= \\\n --parameters spannerMetadataTableName=Metadata_payload_link \\\n --parameters changeStreamName=payload_link_changes \\\n --parameters pubsubTopic=projects//topics/payload-link-changes\n```\n\n`spannerMetadataTableName` is optional but strongly recommended for\nstreaming launches -- see \"Job update and restart continuity\" below.\n\nThe Dataflow job's service account needs:\n- `roles/spanner.databaseReader` on the syncstorage database\n- `roles/spanner.databaseUser` on the metadata database (the change\n stream connector writes its own partition state there)\n- `roles/pubsub.publisher` on the destination topic\n- `roles/dataflow.worker`\n\n### Fine-grained access control (optional)\n\nPass `spannerDatabaseRole` to read the change stream under a Spanner\ndatabase role instead of a database-wide IAM grant. `payload_link_reader`\nin `syncstorage-spanner/src/schema.ddl` is that role: it holds `SELECT`\non `payload_link_changes` and `EXECUTE` on the generated\n`READ_payload_link_changes` table function, and nothing else.\n\nThe role applies to the change stream read only. Beam's\n`MetadataSpannerConfigFactory` deliberately does not copy `databaseRole`\nonto the connector's metadata database config, so that connection still\nuses the service account's own IAM grants. Two consequences, both from\n[Spanner's FGAC guidance](https://docs.cloud.google.com/spanner/docs/fgac-change-streams):\n\n- `spannerMetadataDatabase` **must** be a different database than\n `spannerDatabase`. The connector needs database-level write access for\n its partition state, and that grant would override the role's\n restrictions on the database being read.\n- the service account must **not** hold `roles/spanner.databaseUser` on\n `spannerDatabase` -- it bypasses fine-grained access control outright.\n Grant `roles/spanner.fineGrainedAccessUser`, plus\n `roles/spanner.databaseRoleUser` conditioned on the role resource,\n instead.\n\nIf the metadata tables share the syncstorage database, setting\n`spannerDatabaseRole` buys nothing security-wise: the role is assumed,\nbut the database-level grant the connector needs supersedes it.\nProvision a dedicated metadata database first.\n\n### Job update and restart continuity\n\nPass `spannerMetadataTableName` on any streaming (unbounded) launch so\n`--update` and cancel-then-restart can resume from the previous job's\nwatermarks. The connector generates the partition metadata table name\ninside `expand()` from a `UUID.randomUUID()` call, and that string\ngets baked into the pipeline graph; if the new job's binary produces\na different UUID, it reinitializes against a fresh empty table and\nreads from `startTimestamp` (defaulting to `Timestamp.now()`),\nsilently dropping records committed between the two jobs.\n\nPick any Spanner-legal identifier -- e.g. `Metadata_payload_link` -- and\nkeep it stable across launches. The connector creates the table on\nfirst initialization if it does not already exist, then reuses it.\nBeam's own recommendation: *\"you have to explicitly set the metadata\ntable name parameter of the new job when updating it. Use the value\nof the `metadataTable` pipeline option from the job you are\nupdating.\"*\n\nNotes on stop modes:\n\n- `--update`: preserves Dataflow's checkpointed state but does **not**\n transfer the metadata table binding -- pin the name or lose state.\n- `--cancel`: leaves the metadata table intact; the next launch with\n the same name resumes cleanly.\n- `--drain`: the change stream connector does not support drain\n (per [Google's guidance](https://docs.cloud.google.com/spanner/docs/change-streams/use-dataflow#draining)).\n Use `--cancel` or `--update` instead.\n\n## Filter behaviour\n\n`PayloadLinkChangesToPubSub.isPayloadLinkActionable` keeps a\n`DataChangeRecord` when any of its mods has a non-null `payload_link`\non either side. It drops only the case where **every** mod has both\nold and new `payload_link` NULL -- the inert INSERT/DELETE noise from\ncolumn-scoped change streams. Malformed records pass through so the\nreconciler / DLQ -- not this filter -- surfaces them.\n\n## Beyond the filter: other differences from upstream\n\n`upstream-customization.patch` expresses only the filter step. Our\npipeline additionally diverges from upstream's\n`Cloud_Spanner_Change_Streams_to_PubSub` template in ways that diff\ndoes not attempt to capture. Run `./generate-full-delta.sh` to see\nthe complete diff against the pinned upstream SHA; the substantive\ndifferences are:\n\n- **Sink swap.** `PubsubIO.writeStrings` + a custom `serializeRecord`,\n not upstream's `FileFormatFactorySpannerChangeStreamsToPubSub`.\n Emits raw JSON strings; no PubsubMessage attributes.\n- **JSON wire format.** Emits only `{commitTimestamp, modType,\n tableName, mods[]}`. Omits fields upstream would emit:\n `serverTransactionId`, `rowType[]`,\n `numberOfRecordsInTransaction`,\n `numberOfPartitionsInTransaction`, `partitionToken`,\n `recordSequence`, `isLastRecordInTransactionInPartition`,\n `valueCaptureType`.\n- **No `UncaughtExceptionLogger.register()`.** Uncaught exceptions\n flow to the runner's default handler.\n- **No `@Template` / `TemplateCategory` annotations.** Upstream uses\n the `com.google.cloud.teleport.metadata` annotation-driven\n registrar for flex-template metadata. We register via\n `metadata.json` + `gcloud dataflow flex-template build` instead --\n simpler, no annotation-processor dependency.\n- **No support for these upstream options:**\n `useSpannerEmulatorHost`, `spannerHost`,\n `spannerChangeStreamTvfNameList`, `outputMessageMetadata`,\n `outputDataFormat` (JSON/Avro switch), `pubsubAPI`. Our\n `PayloadLinkOptions` deliberately has a smaller surface (~12\n options vs. upstream's ~25).\n- **No ValueProvider indirection on options.** Upstream wraps\n several config fields in `ValueProvider` for template\n parameterisation; we take plain strings.\n- **No explicit `streaming=true` mutation.** Beam infers streaming\n from the source (`SpannerIO.readChangeStream` is\n `@UnboundedPerElement`) and `DataflowRunner` flips the flag at\n translation time.\n\nThese deltas were deliberate scope choices, not oversights. The\nfilter-only patch is kept as the primary review artifact because\nit's small enough to eyeball, applies cleanly against the pinned\nSHA, and is insensitive to upstream churn on features we don't use.\n\n`main()` matches upstream on two pipeline-shape invariants that are\nnot deployment choices for this source: it appends `use_runner_v2`\nto `options.experiments` and calls `setEnableStreamingEngine(true)`.\n`BundleFinalizer` (used by `SpannerIO.readChangeStream`) requires the\nDataflow Portable Runner (\"Runner V2\"), and V2 streaming jobs\nrequire Streaming Engine -- so both settings are pinned in code\nrather than left to launch-time flags.\n\n## Output wire format\n\nEach surviving `DataChangeRecord` is serialized to a JSON Pub/Sub\nmessage of the form:\n\n```json\n{\n \"commitTimestamp\": \"2026-06-29T12:34:56.000000000Z\",\n \"modType\": \"UPDATE\",\n \"tableName\": \"bsos\",\n \"mods\": [\n {\n \"keys\": \"{\\\"fxa_uid\\\":\\\"...\\\",\\\"fxa_kid\\\":\\\"...\\\",\\\"collection_id\\\":1,\\\"bso_id\\\":\\\"...\\\"}\",\n \"oldValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-1\\\"}\",\n \"newValues\": \"{\\\"payload_link\\\":\\\"gs://bucket/u/c/b/uuid-2\\\"}\"\n }\n ]\n}\n```\n\nEach mod field carries a **JSON string** (escaped) -- the downstream\nPython reconciler reads them with a second `json.loads` per mod. This\nmatches the Spanner change-streams wire convention.\n\n## Running against the compose e2e stack\n\nThe `docker_run_reconciliation_e2e_tests` Makefile target brings up\nSpanner + Pub/Sub + fake-GCS emulators and a publisher container\nlocally. There are two publisher variants; both produce the exact same\nPub/Sub wire format so the reconciler consumes either\nindistinguishably.\n\n### Default: Python publisher\n\n`docker/docker-compose.e2e.reconciliation.yaml` builds and runs the\nPython variant at `payload-link-publisher-py/`. Sub-second startup,\nno JVM. Use this for day-to-day iteration.\n\n```bash\nmake docker_run_reconciliation_e2e_tests\n```\n\n### Swap-in: this Java publisher\n\n`docker/docker-compose.e2e.reconciliation.java.yaml` overrides the\n`payload-link-publisher` service to build **this** directory's Docker\nimage and run the shaded JAR under Beam's DirectRunner (bypassing the\nflex-template launcher's Dataflow-submission entrypoint). Use when you\nneed to reproduce a prod issue against the actual Java pipeline or\nverify wire-format parity between the two variants.\n\n```bash\ndocker compose \\\n -f docker/docker-compose.spanner.yaml \\\n -f docker/docker-compose.e2e.spanner.yaml \\\n -f docker/docker-compose.e2e.reconciliation.yaml \\\n -f docker/docker-compose.e2e.reconciliation.java.yaml \\\n -f docker/docker-compose.e2e.jwk-cache.yaml \\\n up --exit-code-from e2e-tests --abort-on-container-exit\n```\n\nExpect a ~15-30s JVM+Beam warmup and a heavier memory footprint than\nthe Python variant. If Docker's default resource limits are tight on\nyour dev machine, bump them.\n\n## Keeping the reference patch accurate\n\n`upstream-customization.patch` is reference documentation, not a build\ninput. If upstream's pipeline shape evolves or our pipeline drifts in\nways that change the conceptual diff, regenerate it:\n\n```bash\ngit clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git upstream\ncd upstream\ngit checkout \n# Manually re-author the filter customization against the current\n# upstream SpannerChangeStreamsToPubSub.java (in our src/ we own the\n# whole pipeline; here we only need to express the *delta*).\ngit diff > ../tools/payload-link-dataflow/upstream-customization.patch\necho \"\" > ../tools/payload-link-dataflow/upstream.txt\n```\n\nAfter updating the SHA, run `./generate-full-delta.sh` and skim the\noutput to check that our impl's other divergences (sink swap, JSON\nshape, dropped options -- see \"Beyond the filter\" above) still make\nsense against upstream's new state. If upstream restructured\nsomething we deliberately dropped, update the \"Beyond the filter\"\nbullets accordingly.\n\nThis is purely a documentation refresh -- it does not affect the build.\n\n\n# payload-reconciler\n\nPub/Sub-driven reconciler that closes the loop on GCS payload offload.\n\nReads the `payload-link-changes` Pub/Sub topic (populated by the\n`tools/payload-link-dataflow/` flex template). For each mod inside each\nchange record:\n\n- **New `payload_link` non-null** → patch the referenced GCS object's\n metadata to `committed=true` and `customTime=MAX`. Completes the\n 2-phase commit started in `syncserver/src/web/payload_offload.rs`.\n- **Old `payload_link` non-null and ≠ new** → delete the referenced GCS\n object. It is unreferenced now (the row's `payload_link` moved, was\n cleared, or the row was deleted — including TTL row-deletion-policy\n deletes).\n\nBoth operations are idempotent (re-setting metadata is a no-op; 404 on\ndelete is treated as success), so Pub/Sub at-least-once redelivery is\nsafe and no ordering key is needed.\n\n## Files\n\n| File | Purpose |\n|---|---|\n| `pyproject.toml` | Poetry project: `google-cloud-pubsub`, `google-cloud-storage`, `datadog`. |\n| `reconcile_payload_links.py` | Entry point: sync-pull drain loop + per-message handler. |\n| `utils.py` | `parse_gs_url()` — mirrors the Rust uploader's URL grammar. |\n| `test_utils.py` | Tests for `parse_gs_url`. |\n| `test_reconcile_payload_links.py` | Handler tests with a mocked GCS client. |\n\n## Environment\n\n| Variable | Required | Notes |\n|---|---|---|\n| `PUBSUB_PROJECT_ID` | yes | Project hosting the subscription. |\n| `PUBSUB_SUBSCRIPTION` | yes | e.g. `payload-link-reconciler-sub`. |\n| `GCS_PAYLOAD_BUCKET` | yes | The bucket the syncserver writes payloads into. Cross-bucket links abort the message and route it to the DLQ. |\n| `RUN_BUDGET_SECONDS` | no | Selects deployment mode. **Set** (e.g. `240`): cronjob mode -- drain up to that many seconds then exit 0. **Unset**: long-running mode -- poll forever, never exit on idle; the deployment supervises restarts. Cronjob mode is the default deployment. |\n| `SYNC_STATSD_HOST` | no | statsd server host. |\n| `SYNC_STATSD_PORT` | no | statsd server port. |\n\n## Local install / tests\n\n```bash\ncd tools/payload-reconciler\npoetry install\npoetry run pytest\n```\n\nUnit tests cover `parse_gs_url` and the per-message handler. The\n`drain` loop (sync-pull, budget vs. long-running exit conditions) has\nno unit coverage — exercise it against the Pub/Sub emulator when\nchanging it.\n\n## Metrics\n\nStatsd counters (all under the `payload_reconciler.` prefix):\n\n- `finalizes` — successful `committed=true` + `customTime=MAX` patch\n- `orphan_deletes` — successful delete of an unreferenced object\n- `noop_skips` — message processed with zero ops. Should be ~0 if the\n Dataflow filter is working; non-zero is the alarm bell that the\n filter regressed.\n- `gcs_404` + `op:finalize` / `op:delete` — 404 on the respective op.\n Expected non-zero under at-least-once redelivery (idempotency tax);\n a sharp rise suggests upstream state is gone unexpectedly.\n- `errors` + `kind:handler` — Sustained non-zero values feed the DLQ.\n\n## Deployment\n\n**The reconciler ships in the primary syncserver image**, built from\nthe root `Dockerfile`. Its script and Python deps are baked in via\nthat Dockerfile's per-tool poetry-export / pip-wheel / pip-install\nchain; K8s picks between the syncserver binary and the reconciler\nscript via the pod's `command:`.\n\nDefault: K8s cronjob (definition lives in webservices-infra), cadence\n~5 min, exports `RUN_BUDGET_SECONDS` (e.g. `240`).\n\nAlternative: long-running K8s Deployment. Omit `RUN_BUDGET_SECONDS`;\nthe pod polls forever. Choose this when finalize-flip latency below\nthe cronjob cadence matters (e.g. an operator wants offloaded\npayloads to become `committed=true` within seconds of upload rather\nthan up to one cronjob cycle later).\n\nService account needs the same roles in either mode:\n\n- `roles/pubsub.subscriber` on `payload-link-reconciler-sub`\n- `roles/storage.objectAdmin` on the payload bucket (covers both the\n metadata `patch` and `delete` operations)\n\n## Related\n\n- Producer: `syncserver/src/web/payload_offload.rs`\n- Change stream definition: `syncstorage-spanner/src/schema.ddl`\n (look for `payload_link_changes`)\n- Upstream pipeline: `tools/payload-link-dataflow/`\n\n\n# PostgreSQL Utility Scripts\n\nCurrently there is one script: `purge_ttl.py`.\n\n## purge_ttl.py\n\nPurge expired items (BSOs and batches) from a Sync storage PostgreSQL instance.\nThe script is intended to be invoked regularly, e.g. as a cronjob.\n\n### Prerequisites\n\n- Python 3\n- [Poetry](https://python-poetry.org/docs/#installation)\n\n### Run Script Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run python purge_ttl.py\"\n```\n\nPass `-h` to see the list of options.\n\n### Run Tests Locally\n\n```bash\ncd tools/postgres\npoetry install\nSYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@localhost/syncstorage poetry run pytest test_purge_ttl.py -v\"\n```\n\n### Run with Docker\n\nA Docker image with the PostgreSQL build is not published anywhere at the\nmoment. Once that's available, this readme will be updated to include that\nstep.\n\n```bash\ndocker run --rm \\\n --entrypoint python3 \\\n -e SYNC_SYNCSTORAGE__DATABASE_URL=\"postgresql://user:pass@host:5432/syncstorage\" \\\n syncstorage-postgres-image:tag \\\n /app/tools/postgres/purge_ttl.py\n```\n\n\n# A collection of Sync Tools and utilities\n\nSee each directory for details:\n\n* [hawk](hawk) - a tool for generating test HAWK authorization headers\n* [spanner](spanner) - Google Cloud Platform Spanner tools for maintenance and testing\n* [payload-link-dataflow](payload-link-dataflow) - custom Dataflow flex template feeding the payload-link reconciliation pipeline\n* [payload-reconciler](payload-reconciler) - Pub/Sub-driven reconciler that finalizes and garbage-collects offloaded GCS payload objects\n## Installation\n\nThese tools are mostly written in python. It is recommended that you create a commonly shared virtual environment using something like:\n\n`python3 -m venv venv`\n\nto create a `/venv` directory. To activate this, call `sh /venv/bin/activate`.\n\nScript dependencies can be installed via poetry:\n\n```shell\npip install poetry\npoetry install\n```\n\nfor each tool.\n\n\n"}