{"owner":"dlt-hub","repo":"dlt","hasSkills":true,"totalSkillsCount":67,"totalTokensCount":54017,"categories":["anthropic-skill","root-instruction","claude-rule","subagent-persona","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":[".agents/rules/git-github.md",".claude/rules/add-workspace-command.md",".claude/rules/coding-style.md",".claude/rules/docstrings.md",".claude/rules/documentation.md",".claude/rules/heresy.md",".claude/rules/imports.md",".claude/rules/public-api.md",".claude/rules/testing.md",".claude/rules/worktree.md",".claude/skills/auto-docstring-rules/SKILL.md",".claude/skills/auto-import-rules/SKILL.md",".claude/skills/auto-write-test-rules/SKILL.md",".claude/skills/commit/SKILL.md",".claude/skills/create-worktree/SKILL.md",".claude/skills/dashboard/SKILL.md",".claude/skills/implement-issue/SKILL.md",".claude/skills/optimize-code/SKILL.md",".claude/skills/release-notes/SKILL.md",".claude/skills/review-issues-severity/SKILL.md",".claude/skills/review-pr/SKILL.md",".claude/skills/review-pr/identifier-normalization.md",".claude/skills/review-pr/new-destination-checklist.md",".claude/skills/review-vocabulary/SKILL.md",".claude/skills/review-vocabulary/references/simple-english-checklist.md",".claude/skills/review-vocabulary/references/simple-english-use-cases.md",".claude/skills/review-vocabulary/references/simple-english.md",".claude/skills/review-vocabulary/references/vocabulary/adding-terms.md",".claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md",".claude/skills/review-vocabulary/references/vocabulary/g2-attach-and-foreign-datasets.md",".claude/skills/review-vocabulary/references/vocabulary/g3-transformations-and-materialization.md",".claude/skills/review-vocabulary/references/vocabulary/g4-identifiers-and-sql-generation.md",".claude/skills/review-vocabulary/references/vocabulary/g5-configuration-and-credentials.md",".claude/skills/review-vocabulary/references/vocabulary/g6-merge-strategies-and-merge-conditions.md",".claude/skills/review-vocabulary/references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md",".claude/skills/review-vocabulary/references/vocabulary/g8-agents-loops-and-prompts.md",".claude/skills/review-vocabulary/references/vocabulary/g9-workspace-access-and-tools.md",".claude/skills/switch-worktree/SKILL.md",".claude/skills/worktree-from-issue/SKILL.md",".claude/skills/worktree-make-dev/SKILL.md","AGENTS.md","CLAUDE.md","docs/website/docs/hub/agents/agent-definitions.md","docs/website/docs/hub/agents/index.md","docs/website/docs/hub/agents/job-inspector.md","tests/plugins/dlt_example_plugin/README.md","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/dlthub/agents/e2e/self-report/AGENT.md","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-hidden-rule.md","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-visible-rule.md","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/hidden-skill/SKILL.md","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/visible-skill/SKILL.md","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/broken-code/AGENT.md","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/checked-inspector/AGENT.md","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/error-triage/AGENT.md","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/job-inspector/AGENT.md","tests/workspace/cases/workspaces/agent_workspace/.claude/rules/dlthub-platform-job-resources.md","tests/workspace/cases/workspaces/agent_workspace/.claude/skills/debug-deployment/SKILL.md","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/skills/find-source/SKILL.md","tests/workspace/cli/dlthub/ai/cases/mock_workbench/bootstrap/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/mock_workbench/init/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/skills/find-source/SKILL.md","tests/workspace/cli/dlthub/ai/cases/mock_workbench/sql-database/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/init/.claude-plugin/plugin.json","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/my-toolkit/.claude-plugin/plugin.json","tools/tests/cases/dependency_changes/README.md"],"skills":{".agents/rules/git-github.md":"---\nname: git-github\ndescription: Rules for using git and the GitHub CLI in dlt. Read when committing, creating PRs, or running any git/gh command.\n---\n\n# dlt — Git & GitHub\n\nRules for git and GitHub CLI usage in this repo, for any coding agent. They override the agent's defaults where they differ. See `CONTRIBUTING.md` for the human-facing version.\n\n## When to use\n\nApply whenever running `git` or `gh`, drafting commit messages, or creating/updating PRs.\n\n## Commit messages\n\nFollow [Conventional Commits](https://www.conventionalcommits.org/):\n\n```\n{type}: short imperative subject\n# or, when one area is in focus\n{type}({scope}): short imperative subject\n```\n\n- **Types** (lowercase, matching branch categories): `feat`, `fix`, `docs`, `test`, plus `refactor`/`chore` for non-functional changes.\n- **Scope** is optional — an area like `dashboard`, `cli`, `destinations`, `extract`, `normalize`. Comma-separate multiple.\n- **Subject is lowercase, imperative, no trailing period.**\n- **No emojis** anywhere in the message — not in the subject, not in the body.\n- **Subject line only for the vast majority of commits.**\n- **Max 3 lines total** including any body. If you can't say it in 3 lines, the commit is probably too big — split it.\n- **Code-comment rules apply to commit bodies too.** Don't explain *what* the diff does (the diff shows that). Add a body line only when the *why* is non-obvious and not already in the PR description.\n- **No footers.** No `Co-Authored-By` lines, no \"Generated with ...\" trailers. Treat the agent as a dev tool, not a co-author. This overrides the agent's default.\n- **No references to the task/ticket/caller** in the message unless the user asks (\"as discussed\", \"per review feedback\", etc.).\n\nFormat (HEREDOC):\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n## Cleaning up before a squash-merge\n\nGitHub hides everything after the first line in the commit list, so multi-line\nbodies and footers are easy to overlook — and squashing several such commits\nfloods the merge commit. **When squash-merging, clean the squash message down to\na single Conventional-Commit subject line** (plus a short *why* body only if it\ngenuinely helps). Never let `Co-Authored-By` / \"Generated with\" lines survive\ninto the merge.\n\n## When to commit\n\n- **Only when the user explicitly asks.** Finishing a task is not a trigger; \"looks good\" is not a trigger. Wait for an explicit \"commit this\" / \"make a commit\".\n- `git add -A` is fine — but NEVER stage anything secret-looking (`.env`, `*secrets.toml`, credentials).\n\n## Branches\n\nFrom `CONTRIBUTING.md` — all lowercase, dashes, no underscores:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n# example:\nfeat/4922-add-avro-support\n```\n\n- **Categories**: `feat` (ticket required), `fix` (ticket required), `exp`, `test`, `docs`, `keep`.\n- **Branch from `devel`.** Feature branches and most fixes go to `devel`.\n- Don't create branches unless asked. Don't switch the branch a worktree is on without asking.\n\n## Pull requests\n\n- PRs **target `devel`**. Agents may **NEVER** merge to `master`.\n- **Link the PR to its ticket**, or describe the change clearly enough for someone without prior context.\n- **Creating a PR requires the branch to be pushed already.** If `gh pr create` would need a push, stop and ask the user to push.\n- Confirm the title and body with the user before running `gh pr create`.\n\n## Hard rules for the coding agent (never without an explicit ask)\n\n- **Don't push.** No `git push`/`git push -u`. The user pushes.\n- **Don't merge PRs.** No `gh pr merge`, no auto-merge.\n- **Don't force-push, ever.** If history needs rewriting, describe it and let the user do it.\n- **Don't `--amend` pushed commits**, and never use `--no-verify` or skip hooks/signing.\n- **Don't touch `master`** — it's the release branch (`devel → master` on release day).\n\n## Safe read-only commands (no confirmation needed)\n\n- `git status`, `git diff`, `git log`, `git show`, `git branch`, `git remote -v`\n- `gh pr view`, `gh pr list`, `gh pr checks`, `gh pr diff`, `gh issue view`, `gh api` for GET requests\n\n## Quick checklist before any git/gh action\n\n1. Push, merge, force-push, or amend of a pushed commit? → **Stop**, the user does this.\n2. New commit? → Did the user explicitly ask? If no, **stop**.\n3. Is the subject `type:` / `type(scope):`, lowercase, 1–3 lines total, free of \"what\" narration? If no, rewrite.\n4. Is the commit free of footers (no `Co-Authored-By`, no \"Generated with\") and emojis? If no, trim.\n5. Squash-merging? → Clean the squash message to a single subject line.\n",".claude/rules/add-workspace-command.md":"---\npaths:\n - \"dlt/_workspace/cli/**\"\n - \"dlt/_workspace/mcp/**\"\n - \"dlt/_workspace/helpers/dashboard/**\"\n - \"dlt/_workspace/typing.py\"\n - \"tests/workspace/**\"\n---\n\n# Workspace model/loader/view pattern for commands\n\nWhen adding a new command (cli, mcp or dashboard) `dlt/_workspace` follows a limited MVC pattern so that data models stay independent\nfrom presentation. The same model can be rendered by a CLI print function, returned\nfrom an MCP tool, or displayed in a marimo dashboard.\n\n## Layers\n\n| Layer | Location | Convention |\n|-------|----------|------------|\n| **Model** | `dlt/_workspace/typing.py` | `TypedDict` (T-prefixed), plain data, no I/O |\n| **Loader** | `dlt/_workspace/cli/ai/utils.py`, `cli/utils.py` | `fetch_*()` functions — collect data into a model, no `fmt.echo()` calls |\n| **View** | `cli/*_command.py`, `cli/commands.py` | `_print_*()` or `print_*()` — render a model to CLI via `fmt.echo()` / `fmt.warning()` |\n| **Controller** | argparse handlers, MCP tools, marimo notebooks | Calls loader, then passes model to the appropriate view |\n\nNOTE: loaders are stored in cli module. to be refactored soon\n\n## Rules\n\n1. **Loaders return typed dicts, never print.**\n A loader (`fetch_ai_status`, `fetch_workspace_info`, `fetch_secrets_list`, ...)\n returns a `TypedDict` from `typing.py`. It must not call `fmt.echo()` or any\n display function.\n\n2. **Views accept a model, never load.**\n A view (`_print_ai_status`, `print_workspace_info`, ...) takes the typed dict\n and formats it for human consumption. It must not call loaders or access\n providers/config directly.\n\n3. **Controllers wire loader to view.**\n An argparse command function (decorated with `@track_command`) calls the loader,\n then the view. MCP tools call the loader and return the model (or a\n serialised form). Dashboard cells call the loader and render via marimo UI.\n\n4. **Research existing types**\n * Research other `typing.py` modules and Literal definitions and types to find potentially reusable types.\n * Reuse, reconcile, modify existing types - also from `common` and other namespaces\n * Make **reconciliation** and model to be **clean** part of implementation plan. Request user input if needed\n\n5. **Write good models**\n * Use `Literal` ie. for error or warning cases\n * Do not leak user messages into data model - view should do it\n * OK to store additional context ie. exception messages\n * Use `NotRequired` field to the model (e.g. `mcp_error: NotRequired[str]`) if element may not be present. In most cases that is better than setting to None\n\n\n## Existing examples\n\n- `fetch_workspace_info()` -> `TWorkspaceInfo` -> `print_workspace_info()`\n- `fetch_ai_status()` -> `TAiStatusInfo` -> `_print_ai_status()`\n- `fetch_secrets_list()` -> `List[TLocationInfo]` -> `ai_secrets_list_command()` (inline view)\n\n## When adding new commands\n\nBefore writing a new CLI command, MCP tool, or dashboard cell that shows\nworkspace state, check whether a loader already exists. If not, create the\nmodel in `typing.py` and the loader in the appropriate `utils.py` first,\nthen write the view / controller.\n",".claude/rules/coding-style.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Coding style\n\n## Formatting\n- Line length: 100 chars (black + isort, profile \"black\")\n\n## Code and patterns reuse\n- Obvious utils and patterns are already in code base\n- When in doubt look at the code around the changes you are making - not only style but coding patterns and code organization\n- When using a util function or any other function from the code base - look at existing usage if in doubt how it works\n\n## Comments\n- Inline comments start with lowercase (`# resolve the schema`)\n- NEVER add comments that separate code blocks (`# -- sqlglot_dialect mapping --`)\n- DO NOT state the obvious, do not document your thought process\n\n## Types and data structures\n- Always use full generic parametrization -- `Dict[str, Any]` not `dict`\n- Type aliases use T-prefix PascalCase (`TDataItem`, `TLoaderFileFormat`) -- see @dlt/common/typing.py\n- `TypedDict` for structured dicts, `NamedTuple` for lightweight immutable data\n- Prefer `functools.partial` over `lambda` when currying\n\n## Exceptions\n- Inherit from `DltException`; use `TerminalException`/`TransientException` mixins -- see @dlt/common/exceptions.py\n",".claude/rules/docstrings.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Docstring rules\n\n- DO NOT state the obvious, do not document your thought process\n\nGoogle style docstrings throughout. Three levels of detail:\n\n## Short form (internal methods, properties)\nDescription only -- no Args/Returns. One-liner for simple accessors.\n\n## Full form (public API)\nFull Google style with Args, Returns, Raises. Describe WHAT the function does and its output, never HOW it works internally.\n\n```python\ndef paginate(\n self,\n path: str = \"\",\n method: HTTPMethodBasic = \"GET\",\n params: Optional[Dict[str, Any]] = None,\n) -> Iterator[PageData[Any]]:\n \"\"\"Iterates over paginated API responses, yielding pages of data.\n\n Args:\n path (str): Endpoint path for the request, relative to `base_url`.\n method (HTTPMethodBasic): HTTP method for the request, defaults to 'get'.\n params (Optional[Dict[str, Any]]): Query parameters for the request.\n\n Yields:\n PageData[Any]: A page of data with request and response context.\n\n Raises:\n HTTPError: If the response status code is not a success code.\n \"\"\"\n```\n\n- Public API args use `name (Type):` format; internal methods may omit types\n- Args descriptions are short -- one line unless truly complex\n- Use `Yields:` for generators, `Returns:` for regular functions\n- `Raises:` only for exceptions the caller should handle\n- `Example:` section is optional, add only when usage is non-obvious\n\n## Field docstrings\nInline triple-quoted string on the NEXT line after the field definition.\n\n```python\n@configspec\nclass ItemsNormalizerConfiguration(BaseConfiguration):\n add_dlt_id: bool = True\n \"\"\"When true, items will have `_dlt_id` column added with a unique ID.\"\"\"\n add_dlt_load_id: bool = False\n \"\"\"When true, items will have `_dlt_load_id` column added with current load ID.\"\"\"\n```\n\n- Only document fields where the name + type aren't self-explanatory\n- Keep to one line when possible\n\n## Inline formatting\nDocstrings are rendered as Markdown by most editors. Use only these:\n- Single backticks for parameter names, values, types, paths: `name`, `True`, `None`\n- **bold** for emphasis on key terms\n- `Note:` or `Warning:` prefixes for callouts in prose\n- `>>>` interactive Python style for code examples:\n\n```python\n \"\"\"Load data from a REST API.\n\n Example:\n >>> import dlt\n >>> pipeline = dlt.pipeline(\"my_pipeline\", destination=\"duckdb\")\n >>> pipeline.run(my_source())\n \"\"\"\n```\n\nDo NOT use:\n- Double backticks ``like this`` (reStructuredText literal)\n- `:param:`, `:type:`, `:returns:`, `:rtype:` (Sphinx field list)\n- `.. directive::` blocks (reST directives)\n\n## Anti-patterns -- NEVER do these\n- Do not explain HOW the code works, only WHAT it does and its output\n- Do not state the obvious (`\"\"\"Returns the name\"\"\"` on `get_name`)\n- Do not add docstrings to code you did not change\n- Do not use restructured text/sphinx formatting (see Inline formatting above)\n",".claude/rules/documentation.md":"---\npaths:\n - \"docs/**\"\n---\n\n# Documentation guidelines\n\n## Setup\n- On your first setup, go to directory `docs/` and run `make dev`. This will install the required Python and JS/TS dependencies to build the docs `cd docs && make dev`.\n- This will also install pre-commit hooks using `prek`. This will automatically run linting, type checking, etc. on each `git commit` and `git push`\n\n## Code snippets\n- Inline code snippets will automatically be formatted, linted, and type-checked. To opt-out, add the directive to the codefence `notype` or `nolint`.\n- The `execute` directive means the snippet will be ran during before each commit and on CI. The executed snippets should be lightweight.\n\nFor example\n\n > \\```python notype execute\n >\n > import foo\n > \n > def my_func() -> int: ...\n > \n > \\``` \n \n## Internal links\n- Use relative paths **with** `.md` extension: `[schema contracts](schema-contracts.md)`\n- Link to sections with `#` anchors (auto-generated from headings, lowercase hyphenated): `[merge strategies](merge-loading.md#merge-strategies)`\n- Go up directories as needed: `[adjust a schema](../walkthroughs/adjust-a-schema.md)`\n\n## Markdown formatting\n- `mdsmith` lints `website/docs` on CI (`md-check` hook), configured in `docs/.mdsmith.yml`. After editing a page, run from `docs/`:\n ```sh\n make md-fix # pads and aligns tables, fixes whitespace\n make md-check # must report failures=0\n ```\n- Tables: write them unaligned and let `make md-fix` pad the columns (MDS025). It cannot fix a cell over 35 words (MDS026): shorten the text or move the detail below the table.\n\n## Markdown frontmatter\n- Every doc page needs YAML frontmatter with at least `title`, `description`, and `keywords`:\n ```yaml\n ---\n title: Schema\n description: Schema definition and evolution\n keywords: [schema, dlt schema, yaml]\n ---\n ```\n",".claude/rules/heresy.md":"Heresy: issues in code that not only must be **avoided** but actively **detected and removed**.\n\n## llm heresy\n* llm talks to itself in docstrings instead of writing code comments in relevant places\n* llm talks about some obscure downstream cases in upstream code (ie. base class is full of details of particular implementation or explains particular use case)\n* llm invents obscure terminology and uses it everywhere. does not follow terminology used in docs\n* llm writes extremely detailed and complex sentences to explain simple things\n\n## arithmetic heresy\n* decimals are converted to binary floats and back\n* binary floats are used to construct decimals. ie. `Decimal(1.1)`\n\n## datetime heresy\n* naive datetimes are used without a good reason. the only exception are end user requirements\n* pendulum is used in new code. use the stdlib counterparts: `datetime` with an explicit\n `timezone`/`ZoneInfo`, and the `ensure_datetime*` helpers instead of `ensure_pendulum_datetime*`.\n NOTE: existing pendulum usage MAY be tolerated where it is hard to change\n\n## terms heresy\n* use of following words constitutes heresy: gate/gating, graft\n\n## comments heresy\n* block separator comments as below\n```\n#------------------------------\n# comment\n#------------------------------\n```\nor similar. actually any form",".claude/rules/imports.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Import rules\n\n## Import order\n\nGroups separated by blank line:\n1. **stdlib** (`import os`, `from typing import ...`)\n2. **third-party** (`import pendulum`, `import semver`)\n3. **local dlt** (`from dlt.common import ...`)\n4. (tests only) **test utilities** (`from tests.utils import ...`)\n\n- Stdlib imports are ALWAYS at module level -- never inline in functions. This includes `contextlib`, `io`, `warnings`, `sys`, `os`, `re`, etc.\n- `dlt.common` imports (except `dlt.common.libs`) are module level. They are always available and have no heavy side effects.\n- `dlt.common.libs.*` imports may be inline -- they wrap optional dependencies that may not be installed (see Optional dependencies below).\n- If a module is already imported at the top of the file, additional imports from the same module go at the top too -- never inline a second import from an already-imported module.\n- No relative imports.\n\n## Logging\n\n`from dlt.common import logger`, then `logger.info(...)` -- never `import logging` directly (except in `logger.py` itself or runtime infrastructure).\n\n## TYPE_CHECKING imports\n\n- Optional deps for annotations only: `if TYPE_CHECKING: from X import Y` with `else: Y = Any`\n- Destination factories: client class under `TYPE_CHECKING`, actual import deferred to runtime method\n\n## Optional dependencies\n\ndlt wraps optional deps in `dlt/common/libs/` with fail-fast `MissingDependencyException`. Always import from the wrapper, never from the package directly.\n\n### Canonical patterns\n\n```python\nfrom dlt.common.libs.pandas import pandas\n```\n\n### Rules\n\n- NEVER import **optional package** directly\n- ALWAYS import from `dlt.common.libs.`\n- Import at the top of the module if module already imports **optional package**\n- Import inline in other cases.\n\n**Applies to**: `pyarrow`, `pandas`, `numpy`, `pydantic`, `sqlalchemy`, `ibis`, `deltalake`, `pyiceberg`\n",".claude/rules/public-api.md":"---\npaths:\n - \"dlt/**/*.py\"\n---\n\n# Public API and backward compatibility\n\n## Public interface\n- Everything DEFINED and DIRECTLY IMPORTED in @dlt/__init__.py is PUBLIC INTERFACE. Change here is BREAKING CHANGE.\n- Behaviors described in @docs are considered public and changing them is a BREAKING CHANGE (in outputs of PUBLIC INTERFACES)\n- Always flag BREAKING CHANGES clearly with # BREAKING: comments\n\n## Deprecation warnings\n- Use `DltDeprecationWarning` from `dlt/common/warnings.py`, never raw `DeprecationWarning`\n- Pass the `since` version where the deprecation is introduced\n- Use `Dlt100DeprecationWarning` only for features deprecated since 1.0.0 (hardcodes `since`)\n- To deprecate entire functions or classes, use the `deprecated` decorator from `dlt/common/warnings.py`\n\n## Backward compatibility mechanisms\n- Avoid BREAKING CHANGES with deprecations, configuration options etc.\n- Schema/state migration via `engine_version`, versioned storage layouts, deprecation warnings, `deprecated` decorator\n- Backward compat must be tested -- see `tests/pipeline/test_dlt_versions.py` for e2e version upgrade tests\n",".claude/rules/testing.md":"---\npaths:\n - \"tests/**/*.py\"\n - \"tools/tests/**/*.py\"\n---\n\n# Testing\n\n## Style and structure\n- Module-based tests (plain `def test_*() -> None:` functions), not class-based\n- pytest with fixtures in `conftest.py`, not unittest setUp/tearDown\n\n## Reduce tests sprawl\n- Use `pytest.mark.parametrize` to reduce code duplication. ALWAYS use human readable ids for the test cases\n- It is OK to test many related use cases in one test: ie. when they share common setup or represent a flow of user actions\n\n## Fixtures\n- We have autouse fixtures enabled by `conftest.py`. Always check those before adding common fixtures ie. to preserve env variables or get clean storage for the tests\n- We have A LOT of fixtures in files like @tests/utils.py and @tests/load/utils.py, look there before adding new ones.\n\n## Large files for test cases\n- Do not keep large test cases in Python modules. Use `cases` folder and to keep and share files (ie. schemas, ingest data) across modules. \n\n## Destination tests (tests/load)\n- Tests in @tests/load are ALWAYS run on CI in context of particular destination. Tests that run pipelines MUST use `destinations_configs`, other test modules must filter for a right destination (ie. `skip_if_not_active(\"clickhouse\")`)\n- NOTE: tests outside @tests/load do not use such filtering. Use `duckdb`, `dummy` or `sqlite` or `filesystem` destinations for such tests.\n\n### Test Markers\n- `@pytest.mark.essential` — For critical test paths\n- `@pytest.mark.serial` — Tests that can't run in parallel\n\n## Parallel safety and isolation\n- Tests run in parallel! ALWAYS use test storage (`get_test_storage`), unique pipeline names (`uniq_id`), and worker-aware storage roots (`get_test_storage_root` from `tests/utils.py`). Isolate pipelines with `dev_mode` or random `dataset_name`. Avoid external resources that clash with other tests.\n- Tests must work on Linux, macOS and Windows. Avoid encoding paths and platform-specific code. Local path assertions must be Windows compatible.\n\n## Placement and comments\n- When adding new test ALWAYS try find the right place by looking where code you make changes is tested\n- Docstrings: DESCRIBE test case. SKIP when test name is sufficient\n- Use line comments to describe and separate use cases within a single test\n\n## Running\n* DO NOT start two `pytest` processes at once!\n* when running many tests, use `-p xdist -n auto` to parallelize\n",".claude/rules/worktree.md":"# Working in a git worktree\n\nWhen a worktree is active (created via `/create-worktree`, `/worktree-issue`, or the user says \"work in worktree X\"), apply these rules for ALL subsequent operations.\n\nLet `WORKTREE` = the absolute worktree path (e.g., `/home/user/src/dlt/.worktrees/fix-3562`).\n\n## Shell cwd behavior\n\nClaude Code resets Bash cwd to the project root only when `cd` targets a path **outside** the project root. Paths **within** the project root (including `.worktrees/`) persist across calls.\n\n- **Bash**: run `cd WORKTREE` once to switch; subsequent calls will stay there\n- **git**: `git -C WORKTREE ` also works\n- **Read/Grep/Glob/Edit/Write**: always use absolute paths under `WORKTREE`\n\n## Running Python and tests\n\nAlways use `uv run` in worktrees — never bare `python` or `pytest`:\n\n```\nuv run pytest tests/... -v\nuv run python -c \"...\"\n```\n\n## Worktree location convention\n\nAll worktrees live under `/.worktrees/`.\n\n## Plans created in a worktree\n\nWhen creating a plan while working in a worktree, **always** include the worktree path at the top of the plan:\n\n```markdown\n## Working environment\n- Worktree: \n- Branch: \n- Execute: cd before any work\n```\n\nThis is critical because plan→execute transitions start a new session whose cwd resets to the project root. Without the worktree path in the plan, the execute phase will modify files in the main repo instead of the worktree.\n",".claude/skills/auto-docstring-rules/SKILL.md":"---\nname: auto-docstring-rules\ndescription: Enforces dlt docstring conventions. Use when writing or modifying docstrings, classes, or function signatures in Python files.\nuser-invocable: false\n---\n\nUse @.claude/rules/docstrings.md regardless of path setting",".claude/skills/auto-import-rules/SKILL.md":"---\nname: auto-import-rules\ndescription: Enforces dlt import conventions and ordering. Use when writing or modifying Python imports in dlt/ or tests/.\nuser-invocable: false\n---\n\nUse @.claude/rules/imports.md regardless of path setting",".claude/skills/auto-write-test-rules/SKILL.md":"---\nname: auto-write-test-rules\ndescription: Use when adding or proposing a new test ONLY if the test rules are not present\nuser-invocable: false\n---\n\nUse @.claude/rules/testing.md regardless of path setting",".claude/skills/commit/SKILL.md":"---\nname: commit\ndescription: Stage changes and write a clean Conventional-Commit message that follows the repo's git rules\nargument-hint: [-- ]\n---\n\n# Commit\n\nCreate a commit that follows `.agents/rules/git-github.md`. Invoke as `/commit`, or\nuse these steps whenever you are about to commit.\n\nParse `$ARGUMENTS`: anything after `--` is an optional hint (a subject, a scope, or\n\"only the dashboard files\") — guidance, not the literal message.\n\n## Steps\n\n### 1. Inspect\n\n```\ngit status\ngit diff --staged\ngit diff\n```\n\nDecide what belongs in the commit. If nothing is staged, stage the relevant changes\nwith `git add ` (or `git add -A` when everything belongs together). **NEVER\nstage secret-looking files** (`.env`, `*secrets.toml`, credentials) — they are also\ndenied in `settings.json`.\n\n### 2. Compose the message\n\nApply `.agents/rules/git-github.md`:\n\n- One line: `{type}: subject` or `{type}({scope}): subject` — lowercase, imperative,\n no trailing period. Types: `feat`, `fix`, `docs`, `test`, `refactor`, `chore`.\n- **Subject line only** for the vast majority of commits. Add a body line *only* for a\n non-obvious *why*, max 3 lines total. Never narrate *what* the diff does.\n- **No footers** — no `Co-Authored-By`, no \"Generated with ...\" trailers.\n- **No emojis** anywhere in the message.\n\n### 3. Commit\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n### 4. Confirm\n\n```\ngit log -1 --format=%B\n```\n\nVerify it is a single Conventional-Commit subject (plus an optional why) with no\nfooters. **Do not push and do not amend pushed commits** unless the user explicitly\nasks.\n",".claude/skills/create-worktree/SKILL.md":"---\nname: create-worktree\ndescription: Create or reuse a git worktree for a pull request or branch so reviews and work happen in isolation\nargument-hint: [--pr ] [--branch ]\n---\n\n# Git Worktree\n\nSet up an isolated git worktree.\n\nParse `$ARGUMENTS` to extract:\n- The worktree name (required) — used as the directory name under `.worktrees/`\n- `--pr ` (optional) — checkout a GitHub PR into the worktree using `gh pr checkout`\n- `--branch ` (optional) — create the worktree from a specific branch or ref\n\n## Steps\n\n### 1. Determine worktree path\n\nThe worktree path is: `/.worktrees/`\n\nFor example, if the repo root is `/home/user/src/dlt` and the name is `review-pr-3584`, the worktree path is `/home/user/src/dlt/.worktrees/review-pr-3584`.\n\nCreate the `.worktrees` directory if it does not exist:\n\n```\nmkdir -p /.worktrees\n```\n\n### 2. Create or reuse the worktree\n\nCheck if a worktree already exists at that path:\n\n```\ngit worktree list\n```\n\n- **If a worktree already exists** at the target path, reuse it.\n - If `--pr` was given, run `cd && gh pr checkout ` to make sure it is up to date.\n - If `--branch` was given, run `git -C checkout `.\n- **If no worktree exists** at the target path, create one:\n - If `--pr` was given, first resolve the PR's branch name: `gh pr view --json headRefName -q .headRefName`\n - If `--pr` or `--branch` was given, check `git worktree list` output for any **other** worktree that already has the same branch checked out. If found:\n - Use `AskUserQuestion` to ask the user: \"Branch `` is already checked out in worktree ``. Use that worktree instead?\"\n - If the user says yes, use the existing worktree path (skip creation, proceed to step 3 with the existing path).\n - If the user says no, **STOP** — do not create a new worktree. Print a message explaining that the branch is already checked out elsewhere and exit.\n - If no conflict, create the worktree:\n - If `--branch` was given: `git worktree add .worktrees/ `\n - Otherwise: `git worktree add .worktrees/ --detach`\n - Callers needing a new branch check it out afterwards: `git checkout --no-track -b `\n - If `--pr` was given, then also: `cd && gh pr checkout `\n- `gh pr checkout` handles fetching, fork tracking, and branch setup automatically.\n\n### 3. Switch cwd\n\n```\ncd \n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, stop with an error — the cwd did not persist.\n\n### 4. Report\n\n```\nWorktree ready: \n```\n",".claude/skills/dashboard/SKILL.md":"---\nname: dashboard\ndescription: Use when reading, editing, or creating files in dlt/_workspace/helpers/dashboard/ or tests/workspace/helpers/dashboard/ or tests/e2e/\nuser-invocable: false\n---\n\n# dlt Dashboard Development Guide\n\nConsult this skill whenever working on the dashboard codebase. It tells you where files live, what each module does, and where new code belongs.\n\n## Overview\n\nThe dashboard is a **marimo-based** reactive web UI for inspecting dlt pipelines, schemas, data, and run traces. It lives at `dlt/_workspace/helpers/dashboard/`.\n\n## File Layout\n\n```\ndlt/_workspace/helpers/dashboard/\n├── dlt_dashboard.py # Main marimo app (~30 reactive cells)\n├── dlt_dashboard_styles.css # CSS: colors, badges, timeline, tables\n├── runner.py # Subprocess launcher, CLI integration\n├── config.py # DashboardConfiguration @configspec dataclass\n├── const.py # Constants: port (2718), timeouts, colors, defaults\n├── typing.py # Shared TypedDicts (TTableListItem, TLoadItem, etc.)\n├── strings.py # All user-facing UI text (~350 lines)\n└── utils/\n ├── __init__.py # Package exports\n ├── pipeline.py # get_pipeline(), pipeline_details(), exception handling\n ├── queries.py # SQL execution, caching (BoundedDict), row counts, loads\n ├── schema.py # Table/column lists, schema version fetching\n ├── trace.py # Trace rendering, sanitization\n ├── visualization.py # Execution timeline HTML, load status badges\n ├── home.py # Home page / workspace home rendering\n ├── ui.py # dlt_table(), section_marker(), error_callout(), BoundedDict\n ├── formatters.py # Datetime humanization, string formatting\n └── data_quality.py # dlthub data quality integration\n```\n\n## Architecture\n\n- **Framework**: marimo — a reactive Python notebook that renders as a web app\n- **Cell pattern**: `@app.cell(hide_code=True)` decorators define reactive cells in `dlt_dashboard.py`\n- **Dependency tracking**: Cells automatically re-execute when their parameter dependencies change\n- **Routing**: URL query params (`?pipeline=X&profile=Y`) via `mo.query_params()`\n- **Caching**: `BoundedDict` (LRU, 64/32 entries) in `utils/queries.py` and `utils/schema.py`\n- **Styling**: CSS variables for dlt brand colors (`--dlt-color-lime`, `--dlt-color-aqua`, `--dlt-color-purple`, `--dlt-color-pink`)\n\n## Dashboard Sections (cells in dlt_dashboard.py)\n\n1. **Profile & Pipeline Discovery** — reads workspace profiles, lists local pipelines\n2. **Pipeline Attachment** — `dlt.attach()`, file watcher for auto-refresh\n3. **Overview** — pipeline properties, destination info, state version\n4. **Schema** — table browser with configurable column display\n5. **Browse Data** — SQL editor with cached results and query history\n6. **Trace** — execution timeline, step details, resolved config\n7. **Loads** — load history with row counts and schema versions\n8. **State** — raw JSON state viewer\n9. **Data Quality** — integration with dlthub.data_quality\n10. **Ibis Backend** — read-only Ibis connection\n\n## How the Dashboard is Launched\n\nThree CLI commands all call `runner.run_dashboard()`:\n\n* there are several commands in `dlt` and `dlthub` hosts that launch local dashboard. for example `dlt dashboard` and `dlthub local show`.\n\nEdit mode (`--edit`) ejects `dlt_dashboard.py` and CSS to the cwd for user customization.\n\n## Dependencies\n\nFrom `pyproject.toml` `workspace-deps` group: `marimo>=0.14.5`, `duckdb>=0.9`, `ibis-framework>=12.0.0`, `pyarrow>=16.0.0`.\n\n## Where to Put New Code\n\n| What you're adding | Where it goes |\n|---|---|\n| New dashboard section/cell | `dlt_dashboard.py` — add `@app.cell(hide_code=True)` |\n| New UI text/messages | `strings.py` — add to appropriate section |\n| New data fetching/queries | `utils/queries.py` or new file in `utils/` |\n| Schema-related helpers | `utils/schema.py` |\n| Pipeline metadata helpers | `utils/pipeline.py` |\n| Trace/execution helpers | `utils/trace.py` or `utils/visualization.py` |\n| UI component helpers | `utils/ui.py` |\n| Formatting/display helpers | `utils/formatters.py` |\n| New TypedDicts | `typing.py` |\n| New constants/defaults | `const.py` |\n| Configuration options | `config.py` (`DashboardConfiguration` dataclass) |\n| CSS styling | `dlt_dashboard_styles.css` |\n| Data quality features | `utils/data_quality.py` |\n\n## Coding Conventions\n\n- **All user-facing text** goes in `strings.py`, never inline in cells\n- **Cell-local variables must start with `_`** (underscore prefix). In marimo, variables that are only used within a single cell and should not be shared across cells must be prefixed with `_`. Only variables intended to be consumed by other cells should be unprefixed.\n- **UI helpers**: Use `utils/ui.py` functions (`dlt_table()`, `section()`, `error_callout()`) for consistent UI\n- **CSS classes**: `section-header`, `pipeline-execution-timeline`, `status-badge`, `dlt_table`\n- **Badge classes**: `.badge-grey`, `.badge-yellow`, `.badge-green`, `.badge-red`\n- **Data structures**: TypedDicts in `typing.py` for data passed between cells\n- **Configuration**: `@configspec` dataclass in `config.py`\n\n## Testing\n\n### Test locations\n\n```\ntests/workspace/helpers/dashboard/ # Unit and integration tests\n├── conftest.py # Fixtures: pipeline types and temp directories\n├── test_pipeline_workflow.py # Pipeline data retrieval integration\n├── test_all_cells.py # Cell execution and dependency testing\n├── test_visualization.py # Timeline and badge rendering\n├── test_ui_elements.py # UI component styling\n├── test_table_operations.py # Table listing/filtering\n├── test_trace_operations.py # Trace parsing/display\n├── test_query_operations.py # SQL query execution\n├── test_loads.py # Load history retrieval\n├── test_data_formatting.py # String formatting\n├── test_pipeline_operations.py # Pipeline metadata\n└── test_dashboard_config.py # Configuration resolution\n\ntests/e2e/ # Playwright browser tests\n```\n\n### Running tests\n\n```bash\nmake test-workspace-dashboard # unit/integration tests\nmake test-e2e-dashboard # headless chromium e2e\nmake test-e2e-dashboard-headed # visible browser e2e\nmake create-test-pipelines # create pipelines for manual testing\n```\n\n### Test fixtures\n\nTests use parametrized pipeline fixtures via `ALL_PIPELINES` and `PIPELINES_WITH_LOAD` constants in `conftest.py`. Key fixtures: `success_pipeline_duckdb`, `success_pipeline_filesystem`, `extract_exception_pipeline`, `never_ran_pipeline`, `load_exception_pipeline`.\n\n### Where to add new tests\n\n- **Utils logic**: Add to the matching `test_*.py` file (e.g., new query helper → `test_query_operations.py`)\n- **New section/cell**: Add cell execution test to `test_all_cells.py`, plus a dedicated test file if the section has significant logic\n- **Visual/rendering**: `test_visualization.py` or `test_ui_elements.py`\n- **E2E browser tests**: `tests/e2e/`\n",".claude/skills/implement-issue/SKILL.md":"---\nname: implement-issue\ndescription: Triage, plan, and implement a fix or feature for a GitHub issue end-to-end\nargument-hint: [-- ]\n---\n\n# Implement Issue\n\nParse `$ARGUMENTS` to extract:\n- Everything before the first `--` (or all of it if no `--`) is the **issue number**. Referred to as `ISSUE` below.\n- Everything after the first `--` is **maintainer hints** — domain knowledge, suspected root cause, pointers to relevant code, or constraints. If absent, there are no hints.\n\n## Steps\n\n### 1. Fetch the issue\n\n```\ngh issue view ISSUE --json title,body,comments,labels,state,assignees,author,createdAt,milestone\n```\n\nProduce a concise summary: title, reporter, component/area involved, error message, reproduction steps, expected behavior.\n\n### 2. Determine category and gather hints\n\n#### Category\n\nDerive the branch category from the issue labels:\n\n| Label contains | Category |\n|---|---|\n| `bug` | `fix` |\n| `enhancement`, `feature` | `feat` |\n| `test` | `test` |\n| `documentation` | `docs` |\n\nIf no matching label exists, ask the human for the category.\n\n#### Hints\n\nCollect implementation hints from all available sources, in priority order:\n\n1. **Maintainer hints** from `$ARGUMENTS` (after `--`)\n2. **Issue comments** — look for maintainer/contributor replies with technical guidance, root cause analysis, or implementation suggestions\n3. **Issue body** — the reporter may include debugging details, stack traces, or configuration\n\nIf the combined hints from these sources provide sufficient direction (suspected root cause, relevant code areas, constraints), proceed directly to step 3 without asking the human.\n\nIf hints are insufficient — the issue is vague, the root cause is unclear, or there are multiple possible approaches — ask the human for guidance.\n\n### 3. Research the big picture\n\nBefore jumping to a fix, verify assumptions. This step prevents wasted effort on the wrong solution.\n\n#### 3a. Understand the feature area\n\nUse Explore agents to thoroughly investigate the relevant code paths. Trace the flow from user-facing API to the point of failure. Read the code — don't guess.\n\n#### 3b. Validate the reporter's setup\n\nCheck whether the reporter is using the library correctly:\n\n- Is their configuration valid for the component they're using?\n- Are there special conventions, settings, or patterns they should be using instead?\n- Does the documentation describe the expected behavior differently?\n\n#### 3c. Check external dependencies and integrations\n\nIf the issue involves specific external systems (destinations, sources, file formats, etc.):\n\n- Read the relevant implementation in the codebase (factory, capabilities, client, etc.).\n- Understand how the component interacts with the external system.\n- If needed, consult external documentation via WebFetch/WebSearch.\n\n#### 3d. Determine: bug or user error?\n\nBased on research, conclude one of:\n\n- **Bug**: The library has a defect. Proceed to step 4.\n- **User error**: The reporter's configuration is wrong. Draft a response explaining the correct approach. Ask the human to confirm before posting.\n- **Enhancement**: The library works as designed but could be improved. Discuss scope with the human before proceeding.\n\nReport findings to the human before moving on.\n\n### 4. Set up workspace\n\nBefore planning, create the working environment so plan-mode exploration runs against a clean branch.\n\n#### 4a. Determine branch name\n\nUse the category from step 2 and the issue number:\n```\n{category}/{issue-number}-{short-description}\n```\n\nExample: `fix/3529-dedup-sort-escape`\n\nIf `ISSUE` was already a branch name rather than a number (no ticket exists), use it as given.\n\n#### 4b. Create worktree and dev environment\n\nInvoke the `/worktree-from-issue` skill with the branch name:\n```\n/worktree-from-issue {branch-name}\n```\n\nThis creates the worktree, checks out a new branch from `origin/devel`, and runs `/worktree-make-dev`.\nUse it also when there is no issue number.\n\nNote the worktree path — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 5. Draft implementation plan\n\nEnter plan mode and design a plan that **will pass the `/review-pr` skill**. The review-pr skill checks:\n\n1. **Code quality**: Style conforming to `@CLAUDE.md`, no over-engineering, proper use of existing patterns and utilities.\n2. **Test coverage**: Required scenarios identified, tests in the right location, no duplication, proper parametrization.\n3. **Documentation**: Public API changes need doc updates.\n4. **Backward compatibility**: Breaking changes flagged with deprecation path.\n5. **Dependencies**: Changes to `pyproject.toml` flagged.\n\nThe plan must include:\n\n- **Working environment**: Worktree path, branch name, and `cd` instruction (see `@.claude/rules/worktree.md`).\n- **Root cause**: Precise code location and explanation.\n- **Fix**: Minimal change following existing patterns in surrounding code.\n- **Test**: A test that **fails without the fix and passes with it**. Place tests where similar functionality is already tested. Follow `@CLAUDE.md` test guidelines (parallel safety, platform independence, proper fixtures).\n- **Test scope**: Identify the right test level — `make test-common` for extract/normalize/common work, destination-specific tests for load work. See `@CLAUDE.md` \"Testing strategy\" section.\n- **Verification commands**: Exact selective pytest commands to run (specific test functions, not entire modules).\n- **Files changed**: List with expected line counts.\n\nPresent the plan via ExitPlanMode for human approval.\n\n### 6. Implement the fix\n\nApply the changes from the approved plan. Follow these rules:\n\n- **Read before editing**: Always read the target file first. Never edit blind.\n- **Minimal changes**: Only change what the plan specifies. Don't refactor surrounding code, add comments to unchanged code, or \"improve\" things not in scope.\n- **Follow local patterns**: Match the style of surrounding code exactly — imports, naming, spacing, comment style.\n- **Type hints**: All new function signatures need full type hints with generic parametrization (`Dict[str, Any]` not `dict`).\n- **Test conventions**: `def test_*() -> None:`, pytest fixtures, `@pytest.mark.parametrize` for variants.\n\n### 7. Verify\n\nRun verification in this order. Stop and fix issues before proceeding to the next step.\n\n#### 7a. Run the new test selectively\n\nRun only the new/changed test functions using the commands from the plan. Be selective — target specific test functions, not entire modules.\n\n#### 7b. Regression proof (for external system interactions)\n\nWhen the fix involves an external system (destination, source, API) where you don't have full visibility into the behavior — a \"black box\" — verify experimentally that the test actually catches the bug:\n\n- Temporarily revert the fix\n- Run the new test on a configuration where the bug manifests\n- Confirm it **fails** with an error matching the issue report\n- Restore the fix\n\nThis is valuable when you cannot fully predict how the external system handles the input (e.g., how a database treats unquoted identifiers, how an API responds to edge-case payloads). It proves the test is meaningful and not a no-op.\n\nSkip this step when the code path is fully internal and you can reason about correctness from the code alone.\n\n#### 7c. Format and lint\n\n```\nmake format && make lint\n```\n\nIf you changed pages under `docs/website/docs`, also run `make md-fix && make md-check` in `docs/`.\n\nAll must pass cleanly. Fix any issues before reporting.\n\n### 8. Report\n\nSummarize results:\n\n```\n## Fix for #ISSUE: \n\n**Branch**: {branch-name}\n**Worktree**: WORKTREE\n\n### Changes\n- `<file>`: <description> (<lines> lines)\n- ...\n\n### Verification\n- <test-function> on <config>: passed\n- <test-function> on <config> without fix: FAILED (confirms bug) [if applicable]\n- make format: pass\n- make lint: pass\n\nReady to commit.\n```\n\nWait for the human to confirm before committing.\n",".claude/skills/optimize-code/SKILL.md":"---\nname: optimize-code\ndescription: Analyze and optimize Python code performance in critical paths\nargument-hint: <file-path-or-description> [-- <critical-path-hints>]\n---\n\n# Optimize Code\n\nYou are a Python performance optimization expert. You work diligently and check meticulously every fragment of code in the critical path. You never assume - you trace actual execution paths, verify with evidence, and question defensive patterns that may have become unnecessary.\n\n**ALWAYS benchmark everything.** Use `timeit` oneliners for quick validation, write dedicated benchmarks for optimization variants, and as the final step, benchmark optimized code against the baseline (devel branch).\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **target**: file path, function name, or description of code to optimize\n- Everything after `--` is **critical path hints**: which code paths matter most, what to ignore\n\n## Phase 1: Understand the Critical Path\n\nSKIP if optimizing a single (or 1-2 ) functions where critical path is clear, otherwise INTERROGATE the user to understand the hot path before making any changes.\n\n### 1.1 Identify the target\n\nIf `$ARGUMENTS` contains a file path, read it. Otherwise, ask the user:\n- \"Which file or function should I optimize?\"\n- \"What is the entry point for the hot path?\"\n\n### 1.2 Learn the typical case\n\nAsk about the 90% case to focus optimization effort:\n- \"What data shapes are most common?\" (flat dicts, nested objects, lists)\n- \"Which branches are taken most often?\"\n- \"What can we ignore?\" (e.g., \"ignore pandas/arrow, focus on plain dicts\")\n\n### 1.3 Trace the call chain\n\nFollow the code from entry point through each function call:\n```\nentry_point() → helper_a() → helper_b() → actual_work()\n```\n\nRead each function in the chain. Identify the innermost loop where per-item work happens.\n\n### 1.4 Establish baseline benchmark\n\nBefore any optimization, create a benchmark for the current state:\n```python\nimport timeit\n# Quick validation\npython -c \"import timeit; print(timeit.timeit('target_function()', setup='...', number=1000))\"\n```\n\n## Phase 2: Analyze for Optimization Opportunities\n\nLook for these patterns in the critical path:\n\n### 2.1 Unnecessary work\n\n- **Redundant copies**: `dict(x)` or `list(x)` when reference would suffice\n- **Defensive copies**: Trace data origin - if source returns fresh object, copy is redundant\n- **Repeated computations**: Same value computed multiple times in loop\n\n### 2.2 Function call overhead\n\n- **Hot helper functions**: Consider inlining if called per-item\n- **Uncached method lookups**: `self.method` in loop vs `method = self.method` before loop\n- **Duplicate isinstance/hasattr checks**: Same check in multiple places\n\n### 2.3 Happy path shortcuts\n\n- **Early exit for common case**: Check fast path first, skip expensive work\n- **Identity vs equality**: `x is y` faster than `x == y` when applicable\n\n\n### 2.4 Consider `__slots__` for hot objects\n\nObjects created frequently in hot paths benefit from `__slots__`:\n- 30-40% less memory (no `__dict__`)\n- Faster attribute access (92% of local variable speed vs 59% for regular attributes)\n\n```python\n# Bad: regular class, dict-based attributes\nclass Row:\n def __init__(self, table, data):\n self.table = table\n self.data = data\n\n# Good: slotted class\nclass Row:\n __slots__ = ['table', 'data']\n def __init__(self, table, data):\n self.table = table\n self.data = data\n```\n\n### 2.5 Inner function overhead\n\nInner functions (closures) defined inside loops or hot functions are recreated on every call. This can be a **major** performance hit:\n\n```python\n# Bad: inner function recreated per call\ndef process(items):\n def transform(x): # created every time process() is called\n return x * 2\n return [transform(i) for i in items]\n\n# Good: module-level or method\ndef _transform(x):\n return x * 2\n\ndef process(items):\n return [_transform(i) for i in items]\n\n# Good: inline if simple\ndef process(items):\n return [x * 2 for x in items]\n```\n\nSigns to look for:\n- `def` inside another `def`\n- Lambda inside frequently-called function\n- Closures capturing loop variables\n\n\n### 2.6 Convert recursion to iteration\n\nRecursive functions have overhead per call (stack frame, argument passing) and risk stack overflow on deep structures. Convert to stack-based iteration:\n\n```python\n# Bad: recursive traversal\ndef flatten(obj, path=\"\"):\n if isinstance(obj, dict):\n for k, v in obj.items():\n yield from flatten(v, f\"{path}.{k}\") # recursive call\n else:\n yield path, obj\n\n# Good: stack-based iteration\ndef flatten(obj):\n stack = [(obj, \"\")]\n while stack:\n current, path = stack.pop()\n if isinstance(current, dict):\n for k, v in current.items():\n stack.append((v, f\"{path}.{k}\"))\n else:\n yield path, current\n```\n\nBenefits:\n- No function call overhead per level\n- Predictable memory (heap vs stack)\n- No recursion limit issues\n- Can be easier to add early-exit logic\n\nWARN USER:\n- stack method MAY change the order of processing and ie. yielded elements. ALWAYS suggest to write a test for original code.\n\n## Phase 3: Benchmark Everything\n\n### 3.1 Always benchmark caches\n\nCaches (lru_cache, manual dicts, memoization) can hurt performance if:\n- Cache miss is common (lookup overhead without benefit)\n- Cached computation is cheap (cache overhead exceeds saved work)\n- Memory pressure causes cache eviction\n\n**Benchmark caches even if they already exist** - they may have been added speculatively:\n\n```python\n# Does this cache actually help? Benchmark with and without!\n@lru_cache(maxsize=128)\ndef get_column_type(col_name):\n return self.schema.columns[col_name][\"data_type\"]\n\n# Compare:\n# 1. With cache (current)\n# 2. Without cache (direct lookup)\n# 3. Different cache size\n```\n\n### 3.2 Quick validation with timeit\n\nBefore and after each micro-optimization:\n```bash\npython -c \"import timeit; d={'a':1,'b':2}; print('copy:', timeit.timeit('dict(d)', globals={'d':d}, number=1000000))\"\npython -c \"import timeit; d={'a':1,'b':2}; print('ref:', timeit.timeit('x=d', globals={'d':d}, number=1000000))\"\n```\n\n### 3.3 Write dedicated benchmarks\n\nFor optimization variants, create benchmark scripts in `experiments/` folder:\n```python\n\"\"\"Benchmark: optimization variant comparison\"\"\"\nimport timeit\n\ndef variant_original(): ...\ndef variant_optimized(): ...\n\nprint(\"original:\", timeit.timeit(variant_original, number=10000))\nprint(\"optimized:\", timeit.timeit(variant_optimized, number=10000))\n```\n\n### 3.4 Process isolation and thermal management\n\nRun benchmarks in separate processes with pauses:\n```bash\npython experiments/bench.py devel_case\nsleep 10\npython experiments/bench.py optimized_case\n```\n\n### 3.5 Variance checking\n\nRun each benchmark 3+ times to verify stability:\n```bash\nfor i in 1 2 3; do\n echo \"=== Run $i ===\"\n python experiments/bench.py case_name\n sleep 5\ndone\n```\n\nVariance over 10-15% suggests external factors (thermal throttling, system load).\n\n### 3.6 Use realistic benchmark data\n\nUse real test data from `tests/normalize/cases/`:\n\n| File | Size | Use Case |\n|------|------|----------|\n| `ethereum.blocks.*.json` + `schemas/ethereum.schema.json` | 2MB | Deep nesting, warm/cold path normalizer |\n| `github.events.*.json` | 1.7MB | Dynamic table routing, many event types |\n| `github.issues.*.json` | 526KB | REST API, moderate nesting |\n\nLoad with `from dlt.common.json import json`. For flat rows with ISO timestamps, use `mimesis` (in dev deps) to generate synthetic DB data.\n\n### 3.7 Final benchmark: Optimized vs Devel\n\nAs the last step, compare against baseline branch:\n```bash\n# On devel branch (main repo)\ncd /path/to/main && python experiments/bench.py case_name\n\nsleep 10\n\n# On optimized branch (worktree)\ncd /path/to/worktree && python experiments/bench.py case_name\n```\n\nReport: `devel_time / optimized_time = X.XXx speedup`\n\n## Phase 4: Implement and Verify\n\n### 4.1 Make changes incrementally\n\nOne optimization at a time. Benchmark after each change.\n\n### 4.2 Verify correctness\n\n- Run existing tests: `make test-common`\n- Compare outputs before/after on sample data\n\n### 4.3 Document assumptions\n\nAdd comments explaining why optimization is safe:\n```python\n# columns dict is never mutated after get_table_columns() returns,\n# so we can store reference instead of copying\nself._current_columns = columns\n```\n\n## Phase 5: Report Results\n\nProduce a summary:\n```\n## Optimization Summary\n\n### Changes Made\n1. Replaced dict copy with identity check (buffered.py:100)\n2. Inlined count_rows_in_items for common case (buffered.py:102)\n\n### Benchmark Results\n| Case | Devel | Optimized | Speedup |\n|------|-------|-----------|---------|\n| flat_100 | 10.4s | 2.2s | 4.76x |\n| nested_20 | 4.9s | 2.0s | 2.47x |\n\n### Assumptions\n- columns dict is immutable after creation\n- 90% of items are plain dicts, not arrow/pandas\n```\n",".claude/skills/release-notes/SKILL.md":"---\nname: release-notes\ndescription: Generate release notes between two git tags with categorized PR summaries and author attribution\nargument-hint: <git-tag> [-- <previous-tag>]\n---\n\n# Generate Release Notes\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **release tag** (e.g. `1.22.0`). Referred to as `TAG` below.\n- Everything after `--` is an optional **previous tag** override. If absent, auto-detect the previous tag.\n\n## Step 1: Validate the tag\n\nCheck that `TAG` exists as a git tag:\n\n```\ngit tag -l TAG\n```\n\nIf the tag does not exist, stop with an error: \"Tag `TAG` not found. Please create the tag first and re-run.\"\n\n## Step 2: Determine the previous tag\n\nIf a previous tag was provided in `$ARGUMENTS`, validate it exists and use it as `LOWER`.\n\nOtherwise, find the previous tag automatically:\n\n```\ngit tag --sort=-v:refname\n```\n\nWalk the sorted list to find the tag immediately before `TAG` (by version order). Set that as `LOWER`.\n\nValidate `LOWER` exists. Stop with an error if no previous tag can be determined.\n\n## Step 3: Gather commits\n\nGet all commits between the two tags:\n\n```\ngit log --oneline LOWER..TAG\n```\n\nFilter out:\n- Merge commits (starting with \"Merge pull request\" or \"Merge branch\")\n- Version bump commits (starting with \"bumps to version\" or \"bumps to\")\n\nExtract PR numbers from the remaining commit messages (pattern `#NNNN`). Every PR number must appear in the final release notes — none may be silently dropped.\n\n## Step 4: Fetch PR metadata\n\nFor each PR number, fetch metadata using `gh`. Use Task agents to run in parallel batches for efficiency:\n\n```\ngh pr view NUMBER --repo dlt-hub/dlt --json number,title,labels,body,author\n```\n\nCollect for each PR:\n- **number**: PR number\n- **title**: PR title\n- **author**: `author.login`\n- **labels**: list of label names\n- **body**: PR description (first 500 chars sufficient for categorization)\n\n### Handling low-quality PR titles or descriptions\n\nIf a PR title is vague (e.g. just an issue number, single word) or the body is empty/unhelpful:\n\n1. Check if the PR references an issue (e.g. `fixes #NNN`, `closes #NNN`, `#NNN` in body). If so, fetch the issue:\n ```\n gh issue view NNN --repo dlt-hub/dlt --json title,body\n ```\n2. If still unclear, look at the PR diff to understand what changed:\n ```\n gh pr diff NUMBER --repo dlt-hub/dlt\n ```\n3. Write a clear, accurate summary based on what you find.\n\n## Step 5: Categorize PRs\n\nAssign each PR to exactly one category using these rules (evaluated in order). **Every PR must be classified** — verify the total count matches.\n\n### Breaking Changes\nCollect separately (appears at the top, not a standalone category — the PR also appears in its main category). A PR has breaking changes if:\n- It has a `breaking` label, OR\n- Its body contains \"breaking change\" (case-insensitive), OR\n- Its title contains \"breaking\"\n\nExtract the specific breaking change description from the PR body.\n\n### Highlights\nA PR is a highlight if:\n- It has a `highlight` label, OR\n- It is a significant new feature (title starts with `feat` or `(feat)`) AND the change is impactful — meaning it introduces a new user-facing capability, enables a major use case, or represents substantial effort\n- Use judgment: not every feature is a highlight. A highlight should be something users would be excited about.\n\nHighlighted PRs also appear in their main category (Core Library, etc.) but with a brief entry pointing to the Highlights section.\n\n### Docs\n- Title or body indicates documentation-only changes\n- Files changed are only in `docs/`\n- Has `documentation` label\n\n### Chores\n- CI/CD improvements, test infrastructure, repo maintenance\n- Has labels like `ci full`, `tech-debt`, `ci`\n- Title starts with `chore`, `repo:`, `ci:`, or the change only affects tests/tooling\n\n### Core Library\nEverything else — features, bug fixes, enhancements to the library itself.\n\n## Step 6: Identify new contributors\n\nFor each release author, find their first ever merged PR:\n\n```\ngh search prs --repo dlt-hub/dlt --author AUTHOR --merged --sort created --order asc --limit 1 --json number,createdAt\n```\n\nA contributor is **new** if their first merged PR number falls within the `LOWER..TAG` commit range.\n\n## Step 7: Format the release notes\n\nWrite the notes in this exact structure:\n\n```markdown\n## dlt TAG Release Notes\n\n## Breaking Changes\n\n(Only if there are breaking changes. Numbered list.)\n\n1. **Short description** (#NUMBER @author) — Explanation of what broke and how it affects users.\n\n## Highlights\n\n- **Feature name** (#NUMBER @author) — Description of the feature and why it matters.\n\n## Core Library\n\n- **Feature/fix name** (#NUMBER @author) — Brief description.\n- **Fix: short description** (#NUMBER @author)\n\n## Docs\n\n- Description (#NUMBER @author)\n\n## Chores\n\n- **Title** (#NUMBER @author) — Brief description if non-obvious.\n\n## New Contributors\n\n- @author made their first contribution in #NUMBER\n```\n\n## Formatting rules:\n- PR references use `#NUMBER` format (GitHub auto-links them)\n- Authors use `@login` format (GitHub auto-links them)\n- Features and significant fixes get bold titles with em-dash descriptions\n- Minor fixes can be single-line bold entries without descriptions\n- Omit empty sections entirely (including New Contributors if none)\n- Within Core Library, group features first, then fixes\n\n## Step 8: Present for approval\n\nDisplay the formatted release notes to the user for review. Include a summary:\n- Total PRs analyzed\n- Breakdown by category (with counts)\n- Number of unique contributors (and how many are new)\n- List any PRs where you had to infer the description from issue or diff\n\nAsk the user to approve or request changes before writing.\n\n## Step 9: Write to file\n\nOnly after user approval, write the release notes to `/tmp/release-notes-TAG.md`.\n\nReport the file path to the user.\n",".claude/skills/review-issues-severity/SKILL.md":"---\nname: review-issues-severity\ndescription: Find and prioritize open GitHub issues by severity, community impact, and maintainer abandonment\nargument-hint: [-- <optional filters or focus areas>]\n---\n\n# Review Issues by Severity\n\nParse `$ARGUMENTS` for optional filters:\n- Label filters (e.g., `question`, `bug`)\n- Focus areas (e.g., \"merge disposition\", \"Arrow\", \"ClickHouse\")\n- If absent, perform a broad review across all open issues.\n\n## Steps\n\n### 1. Gather candidate issues\n\nRun the following GitHub API queries **in parallel** to surface issues matching different severity signals:\n\n#### a) Long discussions (high comment count)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&per_page=100&sort=comments&direction=desc&page=1' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### b) Issues with specific labels (if filters provided)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&labels=LABEL&per_page=100&sort=comments&direction=desc' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### c) Recently stale issues (updated 2+ months ago, with maintainer comments)\nLook for issues where `updated_at` is old relative to the current date but had prior maintainer engagement.\n\n### 2. Deep-dive top candidates\n\nFor the top 10-15 most promising candidates, fetch full details using a subagent or parallel `gh issue view` calls:\n\n```bash\ngh issue view -R dlt-hub/dlt NUMBER --json title,body,comments,labels,createdAt,updatedAt,author,assignees\n```\n\nFor each issue, extract:\n- **Participants:** who commented, their association (MEMBER = maintainer, NONE/CONTRIBUTOR = community)\n- **Timeline:** when maintainers last engaged, how long since last response\n- **Problem description:** what breaks and under what conditions\n- **Reproduction quality:** is there a clean repro script?\n- **Current status:** fixed? workaround? stalled? abandoned?\n- **Production severity:** does this cause data loss, crashes, or silent corruption?\n\n### 3. Classify each issue\n\nApply these criteria to each issue:\n\n| Signal | What to look for |\n|--------|-----------------|\n| **Serious problem** | Data corruption, silent data loss, hard crashes, cascading failures, security issues |\n| **Long discussion** | 5+ comments, especially with multiple community reporters hitting the same issue |\n| **Maintainer abandoned** | Maintainer commented but last maintainer response is 2+ months old, no linked PR, reporter follow-ups unanswered |\n| **Question label** | Issue has `question` label — these often represent real bugs initially miscategorized |\n| **Community effort** | Reporter provided detailed repro scripts, root-cause analysis, or proposed fixes that went unacknowledged |\n\n### 4. Prioritize\n\nAssign priority tiers:\n\n- **P0 — Critical:** Silent data corruption, cascading failures, confirmed root cause with no fix. Production systems at risk.\n- **P1 — High:** Hard crashes in common deployment patterns, blocking features for significant user segments, confirmed bugs with stalled fixes.\n- **P2 — Medium:** Broken features in specific environments, intermittent failures, missing warnings that lead users astray.\n- **P3 — Low:** Documentation gaps, environment-specific issues with known workarounds, feature requests with community PRs pending review.\n\nWithin each tier, rank by:\n1. Seriousness of the problem in production (data loss > crash > degraded performance > inconvenience)\n2. Negative community impact if not solved (effort wasted by reporters, users hitting the issue independently)\n3. Amount of effort already invested by reporters (repro scripts, root-cause analysis, proposed PRs)\n\n### 5. Output format\n\nPresent results as:\n\n#### Per-issue analysis (grouped by priority tier)\nFor each issue:\n- **Issue link, title, comment count, last activity date**\n- **Problem:** 2-3 sentence description of what breaks\n- **Why this priority:** key severity signal\n- **Abandonment signal:** when maintainer last engaged, what was left unresolved\n- **Community effort:** what the reporter invested (repro scripts, analysis, PRs)\n\n#### Summary matrix\nA table with columns: Priority | Issue # | Type | Production Impact | Community Effort | Maintainer Status\n\n#### Cross-cutting patterns\nNote any clusters of related issues (e.g., multiple issues sharing a root cause) or systemic patterns in maintainer response.\n",".claude/skills/review-pr/SKILL.md":"---\nname: review-pr\ndescription: Analyze a GitHub pull request including diff, comments, related issues, and local code context\nargument-hint: <pr-url-or-number> [-- <optional instructions>]\n---\n\n# Review Pull Request\n\nParse `$ARGUMENTS` to extract the PR identifier and optional reviewer instructions:\n- Everything before the first `--` (or all of it, if there is no `--`) is the **PR identifier** (URL or number). Referred to as `PR_ID` below.\n- Everything after the first `--` is **reviewer instructions** — additional focus areas or context from the caller. If absent, there are no special instructions.\n\nAnalyze the pull request: PR_ID\n\n## Steps\n\nRun the following steps. Gather data in parallel where possible.\n\n### 1. Set up worktree\n\nResolve the PR number:\n\n```\ngh pr view PR_ID --json number\n```\n\nUse the number to form the worktree name `review-pr-<number>`.\n\nInvoke the `/create-worktree review-pr-<number> --pr <number>` skill to create an isolated worktree for this PR. Note the worktree path from the skill output — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 2. Fetch PR metadata and diff\n\nUse `gh` to collect all PR information in parallel:\n\n- `gh pr view PR_ID --json title,body,author,state,baseRefName,headRefName,number,url,comments,reviews,labels,milestone`\n- `gh pr diff PR_ID`\n\n### 3. Check dependency changes in `pyproject.toml`\n\nIf the PR diff (from step 2) touches `pyproject.toml`, run the automated dependency checker.\n\nFirst, fetch the latest `devel` from origin so the comparison is against the current state:\n\n```\ngit fetch origin devel\n```\n\nThen run the script — always compare against `origin/devel` regardless of the PR's base branch:\n\n```\npython tools/check_dependency_changes.py origin/devel\n```\n\nThe script finds the merge-base, simulates a three-way merge (like GitHub), and reports:\n- **Main dependencies** (`[project.dependencies]`) — displayed as WARNING (these affect every user)\n- **Optional dependencies** (`[project.optional-dependencies]`) — displayed as WARNING (these affect users of specific extras)\n- **Dev/group dependencies** (`[dependency-groups]`) — listed for awareness (no impact on end users)\n- **Merge conflicts** — if `pyproject.toml` would conflict, the script warns and falls back to showing the PR's intended changes\n\nInclude the script output verbatim in the review. If the script exits with code 1 (changes detected), flag dependency changes in the review verdict. If it exits with code 2 (conflicts), flag that `pyproject.toml` needs conflict resolution.\n\nIf `pyproject.toml` is not in the diff, skip this step.\n\n### 4. Check CI lint and mypy status\n\nUse `gh pr checks PR_ID` to retrieve the CI check results. Look for checks matching `lint on all python versions / lint*` — these run mypy, ruff, flake8, bandit, and docstring checks across Python 3.9–3.13 (defined in `.github/workflows/lint.yml`).\n\n- Report the status of each lint check (pass/fail/pending).\n- If any lint check has failed, fetch its logs with `gh run view <run-id> --log-failed` to identify the specific errors (mypy type errors, ruff violations, etc.).\n- Include lint/mypy status in the final review verdict. A failing lint check should be flagged as a required fix.\n\n### 5. Analyze related issues\n\nExtract any issue references from the PR body (e.g. `fixes #1234`, `closes #1234`, `#1234`). For each referenced issue, fetch it with:\n\n- `gh issue view <number> --json title,body,author,state,comments,labels`\n\nUnderstand the original problem being solved.\n\n### 6. Analyze the code\n\nUsing absolute paths under `WORKTREE`:\n\n- Read the changed files in full to understand context beyond the diff.\n- Look at callers/callees of modified functions.\n- Check if similar patterns exist elsewhere that should also be updated.\n- Review the coding style and code patterns to conform with @CLAUDE.md\n- If **reviewer instructions** were provided, pay special attention to the areas or concerns they describe during this analysis.\n- Mention only crucial finding to the user in the initial overview. Avoid code snippets except if critical bugs were found\n\n### 6a. Pay attention to special cases\n\nSome areas of the codebase have rules that are easy to miss in a generic code read. The references below are intentionally not loaded by default — read the linked file ONLY when the PR's diff matches the trigger, and apply its checklist before producing the review.\n\n- **Identifier normalization** — read [identifier-normalization.md](identifier-normalization.md) when the PR's diff includes any of: `dlt/destinations/impl/**`, `dlt/dataset/**`, `dlt/destinations/sql_client*`, code that builds SQL/relations/lineage, or schema-generation paths involving sqlglot or ibis. Covers when raw-string table/column identifiers must pass through the schema's naming normalizer, and how to distinguish first-time normalization from re-normalization.\n\n### 7. Review test coverage\n\n#### 7a. Define the testing scope\n\nBased on the PR changes, identify what needs to be tested:\n\n- What are the distinct behaviors introduced or modified?\n- What are the edge cases and error paths?\n- What are the integration boundaries (e.g., does the change affect how components interact)?\n\nProduce a checklist of required test scenarios before evaluating existing tests.\n\n#### 7b. Evaluate existing tests against the checklist\n\n- Compare the PR's tests against the required scenarios checklist. Flag any gaps.\n- Flag any tests that verify things outside the PR's scope or test obvious/trivial behavior that adds no value.\n\n#### 7c. Verify tests are in the right modules\n\n- Search the test directory for where similar functionality is already tested (grep for related function names, class names, or feature keywords).\n- New tests should live alongside related existing tests, not in new files unless the PR introduces an entirely new module.\n- Flag tests placed in the wrong location.\n\n#### 7d. Check for duplication and over-testing\n\n- Look for tests that cover nearly identical cases and could be merged or parametrized with `@pytest.mark.parametrize`.\n- Look for repeated setup/assertion logic across tests that should be extracted into a helper or fixture.\n- If multiple tests differ only in a single variable (e.g., `True`/`False`, different type values), they must be parametrized into one test function.\n- Flag any copy-paste test code. Minimizing test code maintenance burden is a priority.\n- Flag local path assertions that are not Windows compatible.\n\n#### 7e. Bug Fixes Should Have Regression Tests\nIf the PR fixes a bug:\n- Is there a test that would have caught the bug before the fix?\n- Does the test verify the specific scenario from the bug report?\n\n### 8. Check documentation coverage\n\nIf the PR modifies public API (new parameters, changed behavior, new configuration options, new functions/classes exposed to users):\n\n- Search `WORKTREE/docs/website/docs/` for existing documentation covering the topic of the PR.\n- If the topic is already documented and the PR changes its behavior or adds new options, verify that the PR also updates the relevant doc pages.\n- If documentation updates are missing, flag this in the review as a required change.\n- If the topic is not documented at all, this is not a blocker — don't require docs for previously undocumented internals.\n\n### 9. Produce the review\n\nWrite a structured analysis covering:\n\n- **Summary**: What the PR does and why, referencing the linked issue.\n- **Changes**: Breakdown of changes should be concise, avoid code samples\n- **Dependency Changes**: If `pyproject.toml` was modified, include the warnings and listings from step 3. Main and optional dependency changes must appear as yellow WARNING blocks. Omit this section if no dependency changes were found.\n- **CI Lint & Mypy**: Status of lint checks across Python versions. If any failed, list the specific errors.\n- **Observations & Potential Issues**: Bugs, edge cases, missing error handling, performance concerns, memory issues, security issues, backward compatibility.\n- **Breaking Changes and Public API**: Follow the rules on breaking changes and Public API and report issues.\n- **Test Coverage**: Required scenarios checklist with pass/fail status, misplaced tests, duplication issues, parametrization opportunities.\n- **Documentation**: Whether doc updates are needed and if they were provided.\n- **Reviewer Instructions**: If reviewer instructions were provided, include a dedicated section addressing each point raised. If no instructions were provided, omit this section.\n- **Verdict**: Overall assessment.\n",".claude/skills/review-pr/identifier-normalization.md":"---\nname: Identifier Normalization Check\ndescription: Ensures all table names, column names, and other identifiers pass through the schema naming convention normalizer — the single most common PR review finding in this repo.\n---\n\n# Identifier Normalization Check\n\n## Context\n\nDifferent destinations have incompatible naming rules (e.g., Athena/S3 Tables forbids leading underscores, Weaviate capitalizes identifiers, BigQuery has length limits). All identifiers must go through the schema's naming normalizer — raw string constants are never acceptable for table or column names in runtime code.\n\n## When to Check\n\nThose rules apply to INTERNAL `dlt` code that is intended to work on ANY destination. In that case raw identifiers (string literals) MUST be normalized.\n\n\nYou must known when you normalize RAW identifier that represents a table or column name vs. when you RE-NORMALIZE already normalized identifier - take it from `Schema` instance, `Dataset` or `Relation` and which comes in a Python variable.\n\nNote: nested tables `user__comments` and nested columns `issue__stat_count` are PATHS containing many ALREADY NORMALIZED identifiers.\n\nWhen re-normalizing (and WHEN IN DOUBT)\nUse \n- `normalize_tables_path`\n- `normalize_path`\n\nWhen normalizing a RAW identifier:\n\nUse \n- `normalize_table_identifier`\n- `normalize_identifier`\n\n## What to Check\n\n### 1. Raw String Identifiers\n\nLook for hardcoded table or column name strings used in SQL queries, schema lookups, or API calls without normalization:\n\n```python\n# BAD — raw string identifier\ntable_name = \"my_table\"\nsql = f\"SELECT * FROM {table_name}\"\n\n# GOOD — normalized through schema naming\ntable_name = schema.naming.normalize_table_identifier(\"my_table\")\nsql = f\"SELECT * FROM {sql_client.make_qualified_table_name(table_name)}\"\n```\n\nCode in `dlt/dataset/` that builds queries or accesses tables must normalize identifiers:\n\n```python\n# BAD\ndataset[\"_dlt_loads\"]\n\n# GOOD\ndataset[schema.naming.normalize_table_identifier(\"_dlt_loads\")]\n```\n\nIn `dlt/destinations/impl/*/`:\n\n- Table names passed to SQL must use `sql_client.make_qualified_table_name()`\n- Column names in SQL must be escaped/normalized\n- Internal dlt tables (like `_dlt_loads`, `_dlt_version`, `_dlt_pipeline_state`) must still be normalized\n\n### 2. Re-normalizing identifiers\nUse this path when in doubt!\nWhen re-normalizing identifiers (ie. naming convention changed) or when identifier is a variable you MUST use:\n* `normalize_tables_path` - for table identifiers\n* `normalize_path` - for anything else ie. columns\n\nExample: a pyarrow table in transfer from destination 1 (now a source) to destion 2. Columns are normalized per\ndestination 1 and we re-normalize for destination 2:\n\n```python\ndef get_normalized_arrow_fields_mapping(schema: pyarrow.Schema, naming: NamingConvention) -> StrStr:\n \"\"\"Normalizes schema field names and returns mapping from original to normalized name. Raises on name collisions\"\"\"\n # use normalize_path to be compatible with how regular columns are normalized in dlt.Schema\n norm_f = naming.normalize_path\n```",".claude/skills/review-pr/new-destination-checklist.md":"# New destination review checklist\n\nUse this checklist when reviewing a PR that adds a new dlt destination. These items are in addition to the standard review steps.\n\n## 1. Shared destination test integration\n\n- Check `tests/load/utils.py` `destinations_configs()` — is the new destination included in the right config groups (`default_vector_configs`, `default_sql_configs`, `read_only_sqlclient_configs`, etc.)?\n- Check `tests/load/test_read_interfaces.py` — does the destination need special handling (chunk size skips at `_chunk_size()`, view-only table listing exclusions in `test_ibis_dataset_access`)?\n- Check `tests/load/pipeline/test_restore_state.py` — state sync tests run via the config groups above, verify the destination participates.\n- Check `tests/utils.py` — is the destination in `IMPLEMENTED_DESTINATIONS` and `NON_SQL_DESTINATIONS` (if applicable)?\n\n## 2. Capabilities configuration\n\n- Verify `_raw_capabilities()` in the factory: loader formats, merge strategies, replace strategies, type mapper, nested types, decimal precision, timestamp precision, recommended file size.\n- Compare to the blueprint destination (ducklake, lancedb, filesystem) — are any caps missing or misconfigured?\n- Check `parquet_format` settings if the destination writes parquet with nested types.\n\n## 3. `WithLocalFiles` integration\n\n- If the destination stores data locally (files, embedded databases), the top-level `DestinationClient*Configuration` **must** inherit `WithLocalFiles` — otherwise the pipeline cannot bind `pipeline_name`, `pipeline_working_dir`, `local_dir` into the config.\n- If storage config is nested (e.g. `storage: FilesystemConfiguration`), `on_resolved()` must call `self.storage.attach_from(self)` followed by resolve or `normalize_bucket_url()`. Reference: ducklake pattern in `ducklake/configuration.py`.\n- **Test it**: verify default relative paths resolve to `local_dir`, not cwd. Verify `pipeline_name` and `pipeline_working_dir` propagate to nested storage config.\n\n## 4. SQL client — `WithTableScanners`\n\n- If the destination provides a read-only SQL interface via DuckDB views (like lance, lancedb, filesystem), it should extend `WithTableScanners` (`dlt/destinations/impl/duckdb/sql_client.py`), not `DuckDbSqlClient` directly.\n- `WithTableScanners` provides: in-memory DuckDB management, schema creation, sqlglot-based lazy view loading, `create_views_for_all_tables()` for ibis offload.\n- The subclass implements three abstract methods: `create_view()`, `can_create_view()`, `should_replace_view()`.\n\n## 5. Ibis integration\n\n- Check `dlt/helpers/ibis.py` — is there a branch for the new destination's config class?\n- For `WithTableScanners`-based destinations: use `sql_client.create_views_for_all_tables()` (not manual table iteration).\n- Add the destination to the view-only exclusion list in `test_ibis_dataset_access` if it uses DuckDB views (won't see tables from other schemas).\n- The ibis branch must come before any parent config class check to avoid wrong dispatch via `issubclass`.\n\n## 6. Import rules\n\n- Never import optional packages directly — use `dlt.common.libs.*` wrappers (`pyarrow`, `pandas`, `numpy`, etc.).\n- Private API imports from third-party libs (underscore-prefixed functions) are fragile — flag them.\n",".claude/skills/review-vocabulary/SKILL.md":"---\nname: review-vocabulary\ndescription: Review and rewrite the prose a branch adds — docstrings, comments, user-facing messages, docs — against dlt's fixed vocabulary and Simplified Technical English. Invoke as /review-vocabulary.\nargument-hint: \"[<branch-or-base-ref>] [-- <extra focus or scope notes>]\"\ndisable-model-invocation: true\n---\n\n# Review Vocabulary\n\nRewrite the prose a branch adds so dlt says one thing one way. Two inputs govern every\ndecision: the **fixed vocabulary** in `references/vocabulary/`, indexed below, and the\n**Simplified Technical English** rules in `references/simple-english.md`.\n\n**Never invoke this on your own.** A maintainer runs it with `/review-vocabulary`. It rewrites\ntext across dozens of files and changes CI test ids, so it is not a background cleanup.\n\nParse `$ARGUMENTS`:\n- Everything before the first `--` is the **base ref** to diff against. Defaults to\n `$(git merge-base origin/devel HEAD)`.\n- Everything after the first `--` is extra focus from the maintainer.\n\n## Read first\n\n1. `references/simple-english.md` — the 53 rules. **Cite only rule numbers that exist in that\n file.** The numbering is unintuitive and models invent it. Rule 3.1 is about verb forms, not\n sentence length.\n2. `references/simple-english-use-cases.md` — needed for the error-message shape.\n3. The vocabulary groups the diff touches, picked from the index below. They override the general\n rules where they disagree.\n\n## Scope\n\nDiff with `git diff <base>` — **not** `git diff <base>...HEAD`, which misses uncommitted work\nthat will ship in the same PR.\n\nIn scope, when the branch **added or changed** the line:\n\n| Kind | Where |\n|---|---|\n| Docstrings | `dlt/`, `tests/` |\n| `#` comments | `dlt/`, `tests/` |\n| User-facing messages | `raise X(\"...\")`, `logger.*`, `warnings.warn`, helpers that build a message |\n| Test function names | `tests/` — renames only under the narrow gate below |\n| Documentation | `docs/website/docs/**` |\n| Snippet files a page pulls from | `*snippets.py` — comments as docs, code as a test (see below) |\n\nWhen the maintainer asks for a **whole-file** docs review, pre-existing prose in those files is in\nscope too. Say which mode you are in; the default is added-or-changed lines only.\n\nOut of scope:\n\n- **Pre-existing prose**, in the default mode. Leave it even when it breaks every rule. List what\n you skipped and why.\n- Code, identifiers, signatures, type annotations — except snippet code, which gets a separate\n review against the repo's test rules.\n- SQL keywords and anything already in backticks, CLI commands, file paths, config keys.\n- `pytest.param(id=...)` strings, fixture names, parameter names.\n- Assertions on exact generated SQL.\n- Prose moved verbatim from another file — check the base ref before claiming it is new.\n\n## Classification\n\nGet this right first; every other rule depends on it.\n\n| Text | Mode | Limit | Shape |\n|---|---|---|---|\n| Docstrings, `#` comments | descriptive | 25 words (Rule 6.3) | no imperative, one new fact per sentence (Rule 6.1) |\n| Exception strings, warnings, log lines | **procedural** | **20 words (Rule 5.1)** | what happened (simple past), the cause, then the fix as an imperative |\n| Docs — how-to, steps, install | procedural | 20 words | imperative, condition before command (Rule 5.4) |\n| Docs — concept, reference, architecture | descriptive | 25 words | no imperative |\n\n**Messages keep their remedies.** Many errors end in an instruction — \"Materialize the dataset…\",\n\"Set a permanent `pipeline_salt`…\". Improve the wording: make it imperative, condition first,\nunder 20 words, one instruction per sentence. **Never delete a remedy. Never invent one** where\nthe message has none — a remedy needs facts the string does not interpolate.\n\n**Count the assembled sentence, not the fragment.** Some messages are built by a helper that wraps\na short reason in a frame — `_no_data_location(reason)` in\n`dlt/common/destination/client.py` is the current example. Read the helper first, then judge each\nreason as the clause it becomes inside the finished sentence. A four-word reason can push the\nassembled sentence past 20 words.\n\n**Docs carry their own untouchables:** fenced code blocks, `<!--@@@DLT_SNIPPET ...-->` markers,\nfront-matter, link targets and anchor slugs.\n\n**Headings are anchors.** Renaming one breaks every link to it. Before you rename, grep the whole\ndocs tree for the old slug and update each hit in the same change. When a rename would need an edit\nin a file another agent owns, hand it back rather than reaching across.\n\n### Snippet files are BOTH tests and documentation\n\nA page that uses `<!--@@@DLT_SNIPPET ./x_snippets.py::name-->` pulls real code from a real test.\n`docs/pyproject.toml` collects `*snippets.py` with `*_snippet` functions, so **docs CI executes\nthem**. They therefore get two reviews at once, against two different rule sets:\n\n| Part of the file | Treat as | Rules |\n|---|---|---|\n| the code | a test | `.claude/rules/testing.md`, `coding-style.md`, `imports.md` |\n| the `#` comments | user-facing docs | this vocabulary + STE, classified per comment |\n\nConsequences worth knowing:\n\n- **Comments inside a snippet render to the reader**, so a stale or wrong one is a doc bug, not a\n code-hygiene nit. Judge them as prose. A comment that labels a step is INSTRUCTION; a comment\n that explains behavior is REFERENCE.\n- **The repo comment rule still applies to the code.** Default to no comment. But a snippet is\n teaching material, so a comment that would be redundant in library code can earn its place here.\n Say which of the two you applied when they conflict.\n- **Never change snippet code to satisfy a prose rule.** Changing code changes what CI runs. If the\n code is wrong, report it as a code finding with its own justification.\n- **A fenced code block is NOT a snippet.** It is untested. When a page mixes the two, its fenced\n examples can drift from the API while its snippets cannot. Flag the untested ones as unverified\n rather than assuming they work — and consider recommending they become snippets.\n\n## The fixed vocabulary\n\nOne word, one meaning, one part of speech (Rules 1.11, 9.4). Apply the tables before any other\nrewriting.\n\nThe vocabulary is organised in **groups**, one per area of dlt. A group is self-contained: its\nincluded terms, its excluded terms, and the rulings that are easy to get wrong in that area. Each\ngroup is one file in `references/vocabulary/`. Groups grow independently — a review only needs the\ngroups its diff touches, and adding an area means adding a group, not editing the others.\n\nGroups defined so far:\n\n| Group | Covers | Read when the diff touches |\n|---|---|---|\n| [G1 — Data access and locations](references/vocabulary/g1-data-access-and-locations.md) | destinations, datasets, physical locations, join compatibility | `dlt/dataset/`, `dlt/common/destination/`, `dlt/destinations/`, the dataset-access docs |\n| [G2 — Attach and foreign datasets](references/vocabulary/g2-attach-and-foreign-datasets.md) | cross-destination joins, attach info, catalog aliases | `TAttachInfo`, `ATTACH`, foreign datasets, catalog aliases |\n| [G3 — Transformations and materialization](references/vocabulary/g3-transformations-and-materialization.md) | relations, transformations, model jobs, eager and lazy paths | relations, model jobs, `materialize` in `dlt/dataset/`, `dlt/extract/` or `dlt/load/`, the transformation docs |\n| [G4 — Identifiers and SQL generation](references/vocabulary/g4-identifiers-and-sql-generation.md) | naming conventions, case-folding, query binding | `dlt/common/normalizers/naming/`, `dlt/common/schema/`, sql clients, lineage and query generation |\n| [G5 — Configuration and credentials](references/vocabulary/g5-configuration-and-credentials.md) | configs, credentials, secrets | `dlt/common/configuration/`, credentials classes, `config.toml`, `secrets.toml` |\n| [G6 — Merge strategies and merge conditions](references/vocabulary/g6-merge-strategies-and-merge-conditions.md) | merge strategies, deletes, retirement, source filter, destination scope, placeholders | `dlt/destinations/sql_jobs.py`, a destination's merge job, `merge_key`, `scd2`, `cdc`, `merge-loading.md` |\n| [G7 — Jobs, triggers and the deployment manifest](references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md) | jobs, job runs, launchers, triggers, selectors | `dlt/_workspace/deployment/` outside `agent/`, `dlt/hub/run.py`, `__deployment__.py` files, `dlthub local run` |\n| [G8 — Agents, loops and prompts](references/vocabulary/g8-agents-loops-and-prompts.md) | agent definitions, agent jobs, agent runs, loops, system prompts, user turns, traces | `dlt/_workspace/deployment/agent/`, `launchers/agent.py`, `AGENT.md` files, `run.agent` |\n| [G9 — Workspace access and tools](references/vocabulary/g9-workspace-access-and-tools.md) | the `access` declaration, axes, verbs, tools, feature groups | `access` blocks, `TWorkspaceAccess`, `dlt/_workspace/mcp/`, the tools a loop wires |\n\nRead every group the diff touches before you audit, and give each subagent only the groups its\nfiles map to. G1 to G5 are short: when the mapping is unclear, read them all.\n\nTwo rules apply across every group:\n\n- **`dlt` is the sentence subject in messages.** That is how an error gets active voice with a\n named agent (Rule 3.6): \"dlt cannot join…\", \"dlt cannot determine…\". House style.\n- **A banned word is banned for one meaning, not always.** Every group names its legal\n exceptions. Check them before \"fixing\" a hit.\n\n### Legal technical nouns — never replace, any group (Rules 1.5, 1.8)\n\nattach, attach alias, attach info, attach statement, catalog, config, data location, dataset,\ndestination, destination scope, duckdb, iceberg, materialization, merge condition, model job, partition,\npipeline, placeholder, relation, scanner, snapshot (`cdc` only), source filter, vended.\n\naccess, access axis, agent, agent definition, agent definition reference, agent file, agent job,\nagent output, agent run, agent trace, default trigger, feature group, instructions, job, job definition, job ref, job result,\njob run, launcher, loop, placeholder, runner, runtime, selector, skill, system prompt, tool, toolkit,\ntrigger, turn, user turn, verb.\n\n## Rules this codebase breaks most\n\nThe repo comment rule overrides every row. Default to no comment. Keep at most one short line for\na non-obvious WHY. Prefer a rewrite that shrinks the line count. When a comment only restates the\ncode, delete it.\n\n## Test function names\n\nA name is an identifier. Rename it for one of two reasons only:\n\n- it contradicts the fixed vocabulary\n- it carries a noun chain of more than three words (Rule 2.1)\n\nNever rename for style. Apply two checks first:\n\n- **Does the name mirror the API it tests?** Then keep it. `test_attach_info_built_once_per_relation`\n tracks `Relation._attach_infos()`, and that link beats vocabulary purity.\n- **Every rename changes a CI test id.** Flag each one on its own. Afterwards, grep the repo for\n the old name.\n\n## Workflow\n\n### 1. Audit\n\nPropose. Do not edit.\n\nSplit the diff across subagents by area. **File sets must not overlap** — two agents writing one\nfile corrupt each other.\n\nGive each agent its base ref, its file list, the group files its files map to, the two cross-group\nrules and the classification table. Tell it to read `references/simple-english.md` itself, so rule\nnumbers come from the file.\n\nCollect five groups. Give every finding a **file:line, rule number, current text, rewrite**:\n\n1. Vocabulary violations\n2. Structural violations\n3. Identifier renames\n4. Content corrections — false statements. Run the code before you report one.\n5. Test assertions that break — see the trap below.\n\nWrite the proposal to a file. A few hundred findings do not fit in a chat reply. Report the counts\nand the open decisions.\n\n**Stop. Wait for approval.** When the vocabulary does not settle a decision, ask.\n\n### 2. Apply, in two phases\n\nDocstrings and message strings share files, so they cannot run together.\n\n- **Phase A:** docstrings, comments, renames, assertion fixes, content corrections.\n- **Phase B:** message strings.\n\nTell every Phase B agent that the line numbers moved. It must find each string by content. Pin any\nsubstring a test asserts.\n\nEvery applying agent obeys four rules:\n\n- Apply only your own findings, only in your own files.\n- **Never touch code.** Not a signature, not a return type, not a call site. Agents break this\n rule, and a functionally-equivalent change passes every test. Verify it afterwards.\n- Preserve every f-string placeholder: same names, same count, same order.\n- Keep lines under 100 chars. Black does not reflow a string or a comment.\n\n### 3. Verify\n\nRun every step. Each one caught a real defect.\n\n**a. Prose-only check.** Compare the AST with docstrings stripped, against the **branch tip**.\nComments never reach the AST, so any difference means code or a string changed. Explain every file\nthat appears. Against the merge-base the whole PR appears and the signal is lost.\n\n```python\nimport ast\ndef strip(t):\n for n in ast.walk(t):\n if isinstance(n, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):\n n.body = [x for x in n.body if not (isinstance(x, ast.Expr)\n and isinstance(x.value, ast.Constant) and isinstance(x.value.value, str))]\n return t\n# compare ast.dump(strip(ast.parse(old))) with ast.dump(strip(ast.parse(new)))\n```\n\n**b. The four-step self-check** from `references/simple-english.md`, over the added lines. Add a\nsweep for the excluded vocabulary.\n\n**c. `make format`.** Black reports no change when the agents did their job.\n\n**d. `make lint`.** For pages under `docs/website/docs`, also `make md-fix && make md-check` in\n`docs/`. A rewrite that lengthens a table cell past 35 words fails CI.\n\n**e. Run the tests.** This step is not optional.\n\n## The trap: a message rewrite breaks test assertions\n\nTests match on error text. A grep for the phrases you changed misses some. A parametrized test\nhides its assertion far from the message.\n\nOne pass found them in two rounds. A grep found five. The test run then surfaced eight more. One of\nthose was `match=\"cannot be determined\"`. The frame sentence had become \"dlt cannot determine the\ndata location\".\n\n- Grep `tests/` for `match=`, `in str(exc`, and `in str(reject`.\n- Check the **negative** assertions too. `assert \"can join\" not in ...` still passes against\n \"cannot join\", because no space follows `can`.\n- Run the suites anyway. The grep is a head start, not the check.\n\n## Adding a new term or a new group\n\nNever just record a term. Take the eight steps in `references/vocabulary/adding-terms.md`.\n\nAmend the vocabulary mid-review when you must. Then tell every running agent what inverted. They\naudited under the old table.\n\n## Deliverable\n\n- The proposal file, with counts and your open decisions.\n- After approval: the edits, the verification output, and what you did **not** apply.\n- Never commit. The maintainer commits.\n",".claude/skills/review-vocabulary/references/simple-english-checklist.md":"# Verification checklist\n\nRun this pass on every draft before you deliver it. The checks are ordered from mechanical to judgment.\n\n## Mechanical checks (searchable)\n\nSearch the draft for each pattern. Every hit outside code blocks and quoted text is a violation.\n\n| Search for | Violation | Fix |\n|---|---|---|\n| `'ll`, `'re`, `'ve`, `n't`, `it's` | Contraction (Rule 4.2) | Expand it. |\n| `has been`, `have been`, `had been` | Present/past perfect (Rule 3.4) | Simple past or simple present. |\n| `has` / `have` + past participle | Present perfect (Rule 3.4) | Simple past. |\n| `should`, `would`, `may`, `might`, `could` | Unapproved modal (Rule 3.2) | See the modal ladder in SKILL.md. |\n| `is being`, `are being`, `was being` | Progressive passive (Rules 3.4, 3.5) | Active, simple tense. |\n| `, making`, `, allowing`, `, enabling`, `, ensuring` | \"-ing\" clause as verb (Rule 3.5) | New sentence with a real subject. |\n| `;` | Semicolon (Rule 8.1) | Two sentences. |\n| `e.g.`, `i.e.`, `etc.` | Latin abbreviation (GR-6) | \"for example\", \"that is\", name the items. |\n| `simply`, `easily`, `seamlessly`, `robust` | Filler (no fact) | Delete. |\n| ` if `, ` when ` (mid-sentence) | Trailing condition (Rule 5.4) | Move the condition to the start of the sentence, add a comma. |\n\n## Countable checks\n\n1. **Sentence length.** Count words in each sentence. Procedural limit: 20. Descriptive limit: 25. Notes: 25.\n Backticked commands, numbers with units, and identifiers count as one word each (Rule 8.6).\n2. **Paragraph size.** Maximum six sentences per paragraph (Rule 6.6).\n3. **Multi-word nouns.** Any noun chain over three words → break it with prepositions (Rule 2.1).\n4. **Instructions per sentence.** One, unless the actions are simultaneous (Rule 5.2).\n\n## Judgment checks\n\n5. **Classification.** Is each passage cleanly procedural or descriptive? Procedures in imperative, descriptions never in imperative.\n6. **Voice.** Any passive sentence: is the agent truly unknown, and is the passage descriptive? Otherwise make it active (Rule 3.6).\n7. **Condition placement.** Every \"if/when\" stands before its command, with a comma (Rule 5.4).\n8. **Synonym rotation.** One term per concept across the whole document (Rules 1.11, 9.4). Scan for check/verify/confirm, config/settings, run/execute.\n9. **Warnings.** Command or condition first, risk second (Rules 7.2, 7.3).\n10. **Completeness.** Articles present, \"that\" present after \"make sure\", no telegraph style (Rule 4.2).\n11. **Untouchables intact.** Code, identifiers, quoted errors, and proper nouns are unchanged.\n\n## When reporting violations (check mode)\n\nFor each violation give: the rule number, the offending text, and a compliant rewrite. Cite only rule numbers that appear in rules.md.\nEnd the report with this statement when the user asked for STE compliance: \"No tool can guarantee ASD-STE100 compliance. Final approval rests with the writer. The official standard is a free download at asd-ste100.org.\"\n",".claude/skills/review-vocabulary/references/simple-english-use-cases.md":"# Use cases beyond documentation\n\nSTE was built for aircraft maintenance manuals. The same properties — one meaning per word, short sentences, condition-first commands — transfer to any text where misreading has a cost. By the end of Issue 8, 64% of registered STE users were outside aerospace and defense.\n\nEach case below names the mode and the adaptations.\n\n## Error messages and CLI output\n\nMode: procedural. This is the highest-value target: an error message is a 2 a.m. instruction to a stressed reader.\n\nPattern: state what happened (past simple), state the cause if known, give the command or condition to fix it.\n\n> **Before:** Oops! Something went wrong while attempting to establish a connection. Please ensure your credentials are properly configured and try again.\n> **After:** Connection to the database failed. The password for user `app` was not correct. Set `DB_PASSWORD` and connect again.\n\n## Runbooks and standard operating procedures\n\nMode: strict-leaning procedural. This is STE's home turf — an on-call runbook is a maintenance manual.\n\n- Every step imperative, one instruction per step, conditions first.\n- Warnings before the step, command first, risk second.\n- 20-word limit enforced hard: an operator under pager stress reads each sentence once.\n\n## Incident reports and postmortems\n\nMode: descriptive. Simple past only — a timeline in present perfect (\"we have identified...\") hides when things happened.\n\n> **Before:** We have identified an issue that may have impacted some users' ability to access the service.\n> **After:** Between 14:02 and 14:31 UTC, 12% of requests failed. A deploy at 14:00 removed the cache warmup step.\n\nSTE bans hedges (\"may have impacted\") — the report states what is known and says \"unknown\" for the rest. This reads more honest because it is.\n\n## Commit messages and PR descriptions\n\nMode: descriptive body, imperative subject. Convention already matches STE: imperative subject line, plain past facts in the body. Apply the substitution table and the 25-word limit to the body. Delete \"this PR aims to\".\n\n## API changelogs and release notes\n\nMode: descriptive. One entry, one change, one sentence where possible. \"Breaking:\" entries follow the warning pattern — command first: \"Update your calls to `v2/users`. The `name` field split into `first_name` and `last_name`.\"\n\n## Instructions for AI agents (prompts, AGENTS.md, skills)\n\nMode: procedural. A system prompt is a procedure executed by a reader with no ability to ask questions — the exact reader STE was designed for.\n\n- One instruction per sentence keeps rules independently quotable and hard to half-follow.\n- One word, one meaning prevents the model from treating \"check\", \"verify\", and \"validate\" as three different operations.\n- Condition-first (\"If the build fails, stop\") beats trailing conditions, which models drop.\n- No \"should\" — a model reads \"should\" as optional. Write \"must\" or delete the rule.\n\n## Support macros and status-page updates\n\nMode: descriptive, 25-word limit. Non-native readers are the majority of many user bases. No \"we sincerely apologize for any inconvenience this may have caused\" — \"The API was down for 18 minutes. Uploads made during this time were saved and will process today.\"\n\n## Translation and localization prep\n\nMode: strict. STE's original purpose was making English readable for non-native maintenance crews, and it doubles as pre-editing for machine translation. One meaning per word plus complete grammar (articles, \"that\") removes most translation ambiguity. If your docs get localized, STE cuts the error rate and the cost.\n\n## UI copy and empty states\n\nMode: procedural, hard length limits. Buttons and labels are technical names (exempt). Body copy follows the rules: \"No projects yet. Create a project to start.\" Nothing else survives at this length anyway.\n\n## Where STE does not fit\n\nMarketing pages, launch posts, blog voice, brand writing. STE deletes persuasion on purpose. Write those in your own voice — then use STE for the docs the landing page links to.\n",".claude/skills/review-vocabulary/references/simple-english.md":"---\nname: simple-english\nversion: 1.0.0\ndescription: |\n Write or rewrite technical text with the rules of ASD-STE100 Simplified\n Technical English so it is clear, unambiguous, and free of AI slop. Use for\n documentation, READMEs, runbooks, procedures, error messages, release notes,\n incident reports, and API guides. Also use when the user says \"STE\",\n \"Simplified Technical English\", \"ASD-STE100\", \"de-slop\", \"make this\n readable\", \"write for non-native readers\", or asks for docs that translate\n well. Enforces the standard's 53 rules: 20/25-word sentence limits, one word\n one meaning, simple tenses, active voice, condition before command.\nlicense: MIT\ncompatibility: claude-code cursor codex gemini-cli opencode\nmetadata:\n standard: ASD-STE100 Issue 9 (2025-01-15)\n---\n\n# Simple English: Write Like an Aerospace Manual\n\nWrite technical text with the rules of ASD-STE100 Simplified Technical English. STE is the controlled language that aerospace and defense manufacturers use for maintenance documentation. The rules exist so that a tired reader who is not a native English speaker cannot misread an instruction. They remove the usual signs of AI-generated text as a side effect: long sentences, synonym rotation, hedges, filler, and decorative clauses.\n\nWrite for that tired reader. Each sentence must survive one read.\n\n## Your Task\n\nWhen asked to write or rewrite technical text:\n\n1. **Select the mode** (pragmatic or strict, below).\n2. **Classify each passage** as procedural or descriptive. Every other rule depends on this.\n3. **Fix your vocabulary before drafting.** Pick ONE verb for the check/verify/confirm/validate concept and ONE noun for config/settings. Use no other word for these concepts in the whole document.\n4. **Apply the rules** from the catalog below.\n5. **Run the self-check** before you deliver. This step is not optional.\n6. **Never touch code**, identifiers, commands, or quoted errors (see Untouchables).\n\nWhen asked to CHECK text instead of writing it, report each violation as: rule number, the offending text, a compliant rewrite. Cite only rule numbers that exist in this file. Do not cite rule numbers from memory: the numbering is unintuitive and models invent it (tested — an agent without this file cited \"Rule 3.1: short sentences\"; the real Rule 3.1 is about verb forms).\n\n## Two Modes\n\n| Mode | When | What you apply |\n|---|---|---|\n| **Pragmatic** (default) | Docs, READMEs, error messages — the user wants clear text | All structural rules. Domain words stay (\"idempotent\", \"webhook\"). |\n| **Strict** | The user names STE, ASD-STE100, or compliance | Structural rules + full vocabulary discipline, and tell the user that full compliance needs the official dictionary (free at asd-ste100.org). |\n\n## Step 1: Classify the Text\n\n| | Procedural (instructions) | Descriptive (explanations) |\n|---|---|---|\n| Purpose | Tell the reader what to do | Explain what a thing is or does |\n| Verb form | Imperative: \"Install the pump.\" | Simple present/past/future |\n| Sentence limit | **20 words** (Rule 5.1) | **25 words** (Rule 6.3) |\n| Unit rule | One instruction per sentence (5.2) | One topic per paragraph (6.5), max six sentences per paragraph (6.6) |\n\nDo not mix the two in one passage. A \"Getting started\" section is procedural. An \"Architecture\" section is descriptive. A note inside a procedure is descriptive (25-word limit, no imperative).\n\n## THE RULE CATALOG\n\n53 rules in 9 sections, paraphrased from ASD-STE100 Issue 9 with software examples. The official wording is in the free standard at asd-ste100.org.\n\n### Section 1 — Words (Rules 1.1-1.14)\n\n| Rule | Instruction |\n|---|---|\n| 1.1 | Use only approved words, technical nouns, or technical verbs. |\n| 1.2 | Use an approved word only as its listed part of speech. |\n| 1.3 | Use an approved word only with its approved meaning. |\n| 1.4 | Use only the approved forms of verbs and adjectives. |\n| 1.5 | You can use domain words as technical nouns (\"webhook\", \"commit\", \"endpoint\"). |\n| 1.6 | Use an unapproved word only when it is a technical noun or part of one. |\n| 1.7 | Do not use technical nouns as verbs. |\n| 1.8 | Use the technical nouns of your project or industry. |\n| 1.9 | When you pick a technical noun, pick a short and clear one. |\n| 1.10 | No regional, slang, or jargon words as technical nouns. |\n| 1.11 | One item, one name. Do not call it \"config\" here and \"settings\" there. |\n| 1.12 | You can use domain verbs as technical verbs (\"deploy\", \"compile\", \"merge\"). |\n| 1.13 | Do not use technical verbs as nouns. |\n| 1.14 | Use American English spelling. |\n\nIn pragmatic mode, rules 1.5, 1.8, and 1.12 do the heavy lifting: your domain vocabulary is legal. The ones agents break are 1.7, 1.11, and 1.13.\n\n**Before:** You can webhook the event, then do a deploy.\n**After:** Send the event to the webhook. Then deploy the service.\n\n### Section 2 — Multi-word nouns (Rules 2.1-2.2)\n\n| Rule | Instruction |\n|---|---|\n| 2.1 | Write multi-word nouns of three words or fewer. |\n| 2.2 | When a technical noun needs more than three words, write it in full once, then give a short form or hyphenate the units. |\n\nBreak long noun chains with prepositions (of, on, in, for):\n\n**Before:** the connection pool timeout configuration value\n**After:** the timeout value for the connection pool\n\n### Section 3 — Verbs (Rules 3.1-3.7)\n\n| Rule | Instruction |\n|---|---|\n| 3.1 | Use only the verb forms that the dictionary gives. |\n| 3.2 | Use only: infinitive, imperative, simple present, simple past, simple future, past participle as adjective. |\n| 3.3 | Use the past participle only as an adjective (\"the cached response\"). |\n| 3.4 | No auxiliary verbs for complex constructions. No present perfect, no \"is to be installed\". |\n| 3.5 | Use an \"-ing\" form only as a technical noun or inside one (\"logging\", \"the mounting bracket\") — never as a verb. |\n| 3.6 | Active voice. In descriptive text, passive is legal only when the agent is unknown. |\n| 3.7 | Describe an action with a verb, not a noun (\"compress the file\", not \"perform compression of the file\"). |\n\n**Approved modals: can, will, must. Banned: should, would, may, might, could.**\nThe standard rejects \"could\" even for possibility: write \"an explosion can occur\", never \"could occur\". For \"should\": a requirement becomes \"must\"; a suggestion is stated as fact or deleted. This matters double for agent instructions — models read \"should\" as optional.\n\n**Before:** The migration has completed and the table is being rebuilt.\n**After:** The migration is complete. The database rebuilds the table.\n\n**Before:** The flag can be set in the config file, making restarts unnecessary.\n**After:** You can set the flag in the config file. Then a restart is not necessary.\n\n**Before:** The temperature must be adjusted.\n**After:** Adjust the temperature.\n\n### Section 4 — Sentences (Rules 4.1-4.5)\n\n| Rule | Instruction |\n|---|---|\n| 4.1 | Write short and clear sentences. |\n| 4.2 | Do not omit words or use contractions to shorten sentences. Keep articles, keep \"that\". |\n| 4.3 | Use a vertical list for complex text. |\n| 4.4 | Use connecting words between sentences on related topics (\"Then\", \"As a result\"). |\n| 4.5 | Put an article (the, a, an) or a demonstrative adjective (this, these) before nouns where applicable. |\n\nRule 4.2 is the anti-terseness rule. STE is short sentences with complete grammar, not telegraph style:\n\n**Wrong shortening:** Ensure file exists before running.\n**STE:** Make sure that the file exists before you run the command.\n\n### Section 5 — Procedural writing (Rules 5.1-5.5)\n\n| Rule | Instruction |\n|---|---|\n| 5.1 | Maximum 20 words per sentence. Warnings and cautions included. |\n| 5.2 | One instruction per sentence, unless two actions happen at the same time. |\n| 5.3 | Write instructions in the imperative: \"Run the migration.\" |\n| 5.4 | Put a required condition before the command, divided by a comma: \"If the build fails, read the log.\" |\n| 5.5 | Notes give information, never instructions. Notes get the 25-word limit. |\n\n**Before:** You'll want to grab the API key from the dashboard before configuring the client, which you can do under Settings.\n**After:** Get the API key from the dashboard, under Settings. Then configure the client with this key.\n\n### Section 6 — Descriptive writing (Rules 6.1-6.6)\n\n| Rule | Instruction |\n|---|---|\n| 6.1 | Give information gradually: one new fact per sentence. |\n| 6.2 | Use key words and phrases to give the text a logical structure. |\n| 6.3 | Maximum 25 words per sentence. |\n| 6.4 | Group related information in paragraphs. |\n| 6.5 | One topic per paragraph. |\n| 6.6 | Maximum six sentences per paragraph. |\n\nNo imperative in descriptive text. Descriptions explain; procedures instruct.\n\n### Section 7 — Safety instructions (Rules 7.1-7.3)\n\n| Rule | Instruction |\n|---|---|\n| 7.1 | Use a word that shows the risk level (\"WARNING\" = injury, \"CAUTION\" = damage). |\n| 7.2 | Start with a clear command or condition. |\n| 7.3 | Then give the risk or the possible result. |\n\nNever bury the instruction after the explanation. The pattern transfers directly to destructive CLI flags, irreversible migrations, and dangerous API options.\n\n**Before:** Note that data loss may occur in some circumstances if the destructive flag happens to be enabled when running against production.\n**After:** CAUTION: Do not use the `--force` flag against production. The flag deletes rows that do not match the source.\n\n### Section 8 — Punctuation and word count (Rules 8.1-8.7)\n\n| Rule | Instruction |\n|---|---|\n| 8.1 | All standard punctuation is legal except the semicolon. Write two sentences instead. |\n| 8.2 | Use hyphens to connect words that act as one unit. |\n| 8.3 | Parentheses are legal for references, item numbers, abbreviations, plural forms, explanations, alternatives. |\n| 8.4 | In a vertical list, the lead-in colon ends a sentence for word count. |\n| 8.5 | Text inside parentheses counts as one word. |\n| 8.6 | Count as one word each: numbers, numbers with units, abbreviations, alphanumeric identifiers, quoted text, titles, labels, proper nouns. |\n| 8.7 | A hyphenated word counts as one word. |\n\nRule 8.6 matters for software text: `sqlpipe run --config sqlpipe.yaml` in backticks is quoted text and counts as one word. Long identifiers do not blow your sentence budget.\n\n### Section 9 — Writing practices (Rules 9.1-9.4, GR-1 to GR-8)\n\n| Rule | Instruction |\n|---|---|\n| 9.1 | When a word-for-word replacement does not work, restructure the sentence. |\n| 9.2 | Use each approved word correctly: approved meaning, approved part of speech. |\n| 9.3 | Do not build phrasal verbs (\"go down\" → \"decrease\", \"set up\" → \"install\" or \"configure\"). |\n| 9.4 | Keep one consistent style and terminology through the whole document. |\n\nGeneral recommendations GR-1 to GR-8: keep the conjunction \"that\", be careful with \"with\", give pronouns clear referents, prefer \"this + noun\" over bare \"this\", avoid false friends, avoid Latin abbreviations, use inclusive language, and use the possessive apostrophe form only when you are sure it is correct (GR-8: if unsure, do not use it — non-native readers find it hard).\n\nGR-6 for software docs: \"e.g.\" → \"for example\", \"i.e.\" → \"that is\", and delete \"etc.\" — name the items or write \"and more\".\n\n## VOCABULARY DISCIPLINE\n\nThe official dictionary (~900 approved words, ~1,200 banned words with alternatives) is copyrighted by ASD and is not reproduced here. Its mechanics apply without it: **one word, one meaning, one part of speech.**\n\nKnown part-of-speech rulings, useful as patterns:\n\n| Word | Ruling |\n|---|---|\n| test, check, work | Noun only. \"Do a test\", not \"test the pump\". \"Check that X\" becomes \"make sure that X\". |\n| oil | Noun only as used in STE examples. For the verb, the dictionary gives \"lubricate\". |\n| help | Verb only. For the noun, the dictionary gives \"aid\": \"with the aid of\". |\n| fall | \"To move down by gravity\" only, never \"decrease\". |\n| follow | \"To come after\" only, never \"obey\". Write \"obey the instructions\". |\n| above, below | Physical positions only. For limits write \"more than\", \"less than\". |\n\n### The modal ladder\n\n| You wrote | STE writes |\n|---|---|\n| should (requirement) | must |\n| should (recommendation) | Delete it, or state it as fact: \"X is better because Y.\" |\n| may / might / could (possibility) | can |\n| may (permission) | can |\n| would (hypothetical) | Restructure: \"If X occurs, Y occurs.\" |\n\n### Slop-to-simple substitutions\n\nThis table is ours, not the ASD dictionary. It maps the words AI-generated docs overuse to plain replacements. If the word carries no fact, delete it instead of replacing it.\n\n| Slop | Write instead |\n|---|---|\n| leverage, utilize | use |\n| in order to | to |\n| prior to | before |\n| ensure | make sure that |\n| it is worth noting that | (delete) |\n| it's important to, crucially | (delete — state the fact) |\n| simply, just, easily, seamlessly, effortlessly | (delete) |\n| robust, powerful, comprehensive, performant | (delete, or give the measurable property) |\n| functionality | function, feature |\n| enables you to, allows you to | you can |\n| is designed to, aims to | (delete — say what it does) |\n| facilitate | help, make possible |\n| dive into, delve into | read, examine |\n| when it comes to | for |\n| in the event that | if |\n| due to the fact that | because |\n| as needed, as necessary | (state the condition) |\n| and/or | Pick one, or write \"X, or Y, or both\" |\n| e.g. / i.e. / etc. | for example / that is / (name the items) |\n| gracefully handles | (say what it does: \"retries three times, then stops\") |\n| out of the box | by default |\n| under the hood | internally |\n| blazingly fast, state-of-the-art | fast (give the number) / (delete) |\n| streamline | make simpler, make faster |\n| plethora, myriad | many |\n| addresses the issue, tackles | corrects the fault, removes the error |\n\n### Consistency pass\n\nCollapse these common rotations to one term each (Rules 1.11, 9.4):\n\n- check / verify / confirm / validate / ensure → pick one\n- config / configuration / settings / options → pick one\n- delete / remove / drop / destroy → one per meaning, kept consistent\n- error / issue / problem / failure → \"error\" for errors, \"failure\" for failed operations\n- run / execute / invoke / launch → pick one\n- show / display / render / present → pick one\n\n## Untouchables\n\nThese are technical names (Rules 1.5, 8.6). Leave them exact, even when they break vocabulary rules:\n\n- Code blocks, inline code, identifiers, CLI commands, flags, file paths\n- Quoted error messages and log lines\n- Product names, API endpoint names, config keys\n- Numbers with units — each counts as one word in the sentence limit\n\n## Beyond Documentation\n\nSame rules, different targets. Full adaptations in `references/use-cases.md`:\n\n- **Error messages**: state what happened (simple past), the cause if known, then the fix as an imperative. No \"Oops\", no \"Please ensure\", no apology filler.\n- **Runbooks**: STE's home turf. Imperative steps, conditions first, warnings before the step.\n- **Incident reports**: simple past only. \"We have identified an issue that may have impacted\" becomes \"Between 14:02 and 14:31 UTC, 12% of requests failed.\"\n- **Release notes**: breaking changes follow the warning pattern — command first, risk second.\n- **Agent instructions (prompts, AGENTS.md)**: a system prompt is a procedure for a reader that cannot ask questions. One instruction per sentence, no \"should\", condition first.\n- **Translation prep**: STE's original job. One meaning per word plus complete grammar removes most translation ambiguity.\n\n## Self-Check Before You Deliver\n\nThis step is not optional. Run these four checks on your draft:\n\n1. Count words in your three longest sentences. Over the 20/25 limit → split them.\n2. Search your draft for: `'ll`, `'re`, `'s` (contraction), `has been`, `have been`, `should`, `-ing` verbs after a comma, semicolons.\n3. Search for every `if` and `when`. Each one stands at the START of its sentence, before the command. \"Increase the timeout if the network is slow\" → \"If the network is slow, increase the timeout.\"\n4. Search for the verbs you did NOT pick in Your Task step 3 (the check/verify/confirm set). Replace every hit with your chosen verb.\n\nFix what you find, then deliver. For a full audit, run `references/checklist.md`.\n\n## Full Example\n\n**Before (real unedited AI output):**\n\n> **Connection timeouts.** If sqlpipe hangs or fails with `dial tcp: i/o timeout`, check that the host running sqlpipe can reach the Postgres port (usually 5432) — this is often a security group or firewall rule blocking the connection. If you're connecting to a managed database (RDS, Cloud SQL, etc.), confirm the instance allows connections from sqlpipe's IP. You can also try increasing `source.connect_timeout_seconds` in your config, since a slow network path can trip the default timeout even when the connection eventually succeeds.\n\n**After (classified procedural, verb = \"make sure\", conditions first, one instruction per sentence):**\n\n> **Connection timeouts.** sqlpipe stops with `dial tcp: i/o timeout` when it cannot reach the Postgres port (5432 by default).\n>\n> 1. Make sure that the host that runs sqlpipe can reach the Postgres port. A firewall or security group usually blocks it.\n> 2. If the database is managed (RDS, Cloud SQL), make sure that the instance accepts connections from the IP of sqlpipe.\n> 3. If the network is slow, increase `source.connect_timeout_seconds` in the configuration.\n\nWhat changed: 40-word sentences split under 20; \"you're\" expanded; \"check/confirm\" collapsed to \"make sure that\"; every condition moved before its command; \"etc.\" removed; code and error strings untouched.\n\n## Limits\n\nSTE is for technical facts and instructions. Do not apply it to marketing copy, blog voice, or brand writing — it deletes persuasion by design. When a user asks for STE on marketing text, say so and offer it for the docs instead.\n\nThis skill is an unofficial aid. It is not affiliated with or endorsed by ASD or STEMG, and no tool can guarantee STE compliance. ASD-STE100 is a registered trademark of ASD. The official standard is a free download at asd-ste100.org.\n\n## References\n\n- `references/checklist.md` — full verification pass with searchable patterns, for check mode and final audits\n- `references/use-cases.md` — long-form adaptations: error messages, runbooks, incident reports, commits, UI copy, i18n\n",".claude/skills/review-vocabulary/references/vocabulary/adding-terms.md":"# Adding a new term or a new group\n\nNever just record a term. Take these eight steps.\n\n1. **Pick the group.** No group fits? Add one. Give it the next `G<n>`, a file in this folder named\n after its heading, an index row in `SKILL.md`, and the three parts: included, excluded, rulings.\n A group without excluded terms is not finished.\n2. **Research the usage.** Grep `dlt/`, `tests/` and `docs/` for the word and every synonym. Count\n the hits. Read enough to find the meanings in play. One word often covers two concepts.\n3. **Derive the banned set.** A term is useless without one. For each synonym and inflection,\n decide: banned, or legal with another meaning? Both halves go in the group.\n4. **Check the upstream interface.** The vocabulary must not contradict a public dlt method or\n type name. `access` won because `needs_attach` already said \"accesses its data\". `attach info`\n won because the type is `TAttachInfo`.\n5. **Name the false positives.** `descriptor` is banned for `TAttachInfo` and correct for the\n Python descriptor protocol. Put the exception in the rulings, or the next run \"fixes\" it.\n6. **Fix the part of speech.** Say noun or verb, and ban the other use. `access` is a verb, so\n \"data access\" is a violation.\n7. **Test the ban before you write it.** Grep for the word you intend to exclude. Compliant prose\n already uses it? Then the ban is too wide. Narrow it to the meaning you mean. `inaccessible`\n and `foreign-folded` passed this check and stay legal. `casefold` and `attach instructions`\n failed it and are banned.\n8. **Update the group file.** Add rows to both tables. Add a ruling when the term has a legal\n exception. State what becomes newly compliant and what becomes newly non-compliant. A term that\n flips direction turns compliant prose into findings.\n",".claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md":"# G1 — Data access and locations\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| getting to data (verb) | **access** |\n| the adjective | **accessible**, **inaccessible** |\n| the negative, as a verb | **cannot access** |\n| where the data physically sits | **data location** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| reach, reaches, reachable, in reach of, out of reach, get to | one word for one concept — `access` |\n| physical location, physical destination, physical dataset (in prose) | say **data location**; the method `data_location()` keeps its name |\n\n**Rulings**\n\n- **`access` is a verb for the act, and a noun for what a job may touch.** Keep the verb for the\n act of reading data: \"the engine accesses the data\", not \"data access is one-way\". The noun is\n legal only in the sense G9 defines — the `access` declaration, and the grant that answers it.\n dlt's docs already write \"grant access\", \"read access\" and \"denied access\" 40 times over; that\n usage was always compliant and stays.\n- **`reach` is not always `access`.** \"`SET SESSION` would not reach the cloned sessions\" means\n *propagate to*, not *read data from*. A literal swap changes the meaning — restructure\n (Rule 9.1).\n",".claude/skills/review-vocabulary/references/vocabulary/g2-attach-and-foreign-datasets.md":"# G2 — Attach and foreign datasets\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the `TAttachInfo` object | **attach info** |\n| the `TAttachStatement` object | **attach statement** |\n| the SQL keyword | `` `ATTACH` `` in backticks |\n| the action in prose | **attach** (lowercase, a verb) |\n| the catalog a foreign dataset lands under | **attach alias** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| descriptor (for `TAttachInfo`) | `attach info` matches the type and the method `_attach_infos()` |\n| bare `ATTACH` as a prose noun, `ATTACHed`, \"attaches\" as a plural noun | backtick the keyword, or use the verb |\n| attach instructions | one name — **attach statements** |\n\n**Rulings**\n\n- **`descriptor` is legal for the Python descriptor protocol.** `dlt/common/utils.py` describes a\n real Python descriptor. The ban covers naming the `TAttachInfo` object only.\n- **`attach info` and `attach statement` are different things.** One is the whole descriptor for a\n foreign dataset; the other is a single SQL statement inside it. Do not collapse them.\n",".claude/skills/review-vocabulary/references/vocabulary/g3-transformations-and-materialization.md":"# G3 — Transformations and materialization\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the deferred path | **lazy materialization** |\n| the load job that runs it | **model job** |\n| the immediate path | **eager materialization** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| model extraction | not a thing — it is a **model job** |\n| executed here | say **eager materialization** |\n\n**Rulings**\n\n- **`lazy` and `eager` are legal only for materialization.** dlt has both. Using `lazily` to mean\n *on first use* (memoization) is a second meaning for one word — write \"on the first read\".\n- **A model job is the artifact; lazy materialization is the path.** Use the one you mean.\n",".claude/skills/review-vocabulary/references/vocabulary/g4-identifiers-and-sql-generation.md":"# G4 — Identifiers and SQL generation\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| identifier case handling | **case-fold**, **case-folds**, **case-folding** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| casefold, post-fold (in prose) | one spelling — **case-fold**; the identifier `casefold_identifier` keeps its name |\n\n**Rulings**\n\n- **Hyphenated compounds on the `fold` root are legal** — \"foreign-folded output column\" reads\n correctly and is in use. The ban is on the bare spelling `casefold` in prose.\n",".claude/skills/review-vocabulary/references/vocabulary/g5-configuration-and-credentials.md":"# G5 — Configuration and credentials\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| a resolved settings object | **config** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| configuration (for the object) | `config`; keep \"configuration error\" when it names `ConfigurationValueError` |\n",".claude/skills/review-vocabulary/references/vocabulary/g6-merge-strategies-and-merge-conditions.md":"# G6 — Merge strategies and merge conditions\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| removing destination records (`delete-insert`, `upsert`, `cdc`, hard deletes) | **delete** |\n| closing the validity of an `scd2` record | **retire** (never \"delete\" for `scd2`) |\n| loaded records that `source_filter` does not select | **discard**, **discarded** |\n| writing loaded records to the destination | **insert** (a new record), **update** (an existing one) |\n| a record in the destination that the loaded data does not contain | **absent** |\n| the part of a table a load replaces | **partition** (conceptual, as `merge_key` docs already use it) |\n| the loaded data under `cdc` | **snapshot** |\n| the upstream system whose records `cdc` mirrors | **source system** |\n| the loaded records that the source filter selects (all loaded records without a filter) | **merge source** (a noun) |\n| the `source_filter` option in prose | **source filter** |\n| the `destination_scope` option in prose | **destination scope** |\n| `source_filter`, `destination_scope` together | **merge conditions** |\n| `{table}`, `{staging_table}` | **placeholders**; \"expand\" for substitution |\n| data in docs and messages | **records**; **rows** only in code docstrings and comments, and in \"nested rows\" |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| region, window, bare \"scope\" (for the records a merge can delete or retire) | say **destination scope** for the option, **partition** for the concept |\n| input filter, output filter, merge filter(s), merge scope | old names — say **source filter**, **destination scope**, **merge conditions** |\n| filtered staging (data, rows, records), filtered load, filtered loaded data, bare \"the source\" (for the merge source) | say **merge source** |\n| bare \"the source\", upstream, origin (for the system `cdc` mirrors) | say **source system**; `@dlt.source` keeps its name |\n| owns, is authoritative for (a load and its records) | say \"the records that this load replaces\" |\n| clear, remove, wipe (for a merge delete) | one verb — **delete** |\n| drop, exclude, filter out, land (for loaded records) | **discard** for records the source filter does not select, **insert** for records that are written |\n| supersede, take precedence (between merge conditions and `merge_key`) | say \"dlt ignores `merge_key`\" |\n| strand, migrate (a record that changes partition) | say \"the record moves to another partition\" |\n| knob | say **setting** or name the hint |\n\n**Rulings**\n\n- **The bans are for the merge meaning only.** `region` stays legal for cloud regions and data columns,\n `window` for browser windows and the attribution window in `lag.md`, `scope` for OAuth and config scopes,\n `drop` for dropping tables and the `dlt pipeline drop` command, `remove` outside merge deletes.\n- **\"source\" in \"source filter\" is the merge source**, the loaded data, as in `MERGE ... USING source`. It is\n not `@dlt.source`. Never shorten \"source filter\" to \"the source\", and never write \"destination scope\" as\n \"the scope\".\n- **The merge source is what the merge works on.** Keys, `merge_key` partitions and absent records come from the\n merge source, not from all loaded records. The destination scope does not depend on it.\n- **`loaded data` and `loaded records` stay legal** for everything loaded, before the source filter. Write\n \"merge source\" only where a source filter can apply. Without one, the two are the same.\n- **\"MERGE source\" in SQL context is legal.** `MERGE ... USING` names its input the source; uppercase `MERGE`\n marks that meaning (Databricks comments).\n- **Only strategies that delete or retire absent records have a destination scope**: `delete-insert`,\n `scd2`, `cdc`. `upsert` also deletes (with `hard_delete`), so \"strategies that delete\" is not the rule.\n- **`delete` vs `retire`.** `scd2` never deletes on absence, so \"`scd2` deletes absent records\" is a content\n error, not only a vocabulary one.\n- **`snapshot` is legal only for `cdc` loaded data.** Iceberg and Delta snapshots are a different technical noun.\n Do not write \"snapshot\" for Iceberg `upsert` limits: write \"absent from the loaded data\".\n- **Placeholders differ per destination.** Do not write \"`{table}` expands to the destination table\" without\n naming the destination type: on Delta it expands to `target`.\n\nNewly non-compliant: \"region\", \"window\", bare \"scope\", \"owns\", \"clear\", \"supersede\" in merge prose, and the\nold option names (\"input filter\", \"output filter\", \"merge filters\"), and \"filtered staging\" or bare \"the source\"\nfor the merge source. Newly compliant: \"destination scope\" for the `destination_scope` option, \"merge source\".\n",".claude/skills/review-vocabulary/references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md":"# G7 — Jobs, triggers and the deployment manifest\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the unit a workspace deploys and runs | **job** |\n| one execution of it | **job run** |\n| its identity | **job ref** |\n| its entry in the manifest | **job definition** |\n| the dlt module that starts a job | **launcher** |\n| the machine the platform gives the job | **runner** |\n| the dltHub platform itself | **runtime** |\n| the string that starts a job | **trigger** |\n| the pattern that expands into triggers | **selector** |\n| the trigger a manual run stands in for | **default trigger** |\n| the file that describes every job | **deployment manifest** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| task, workload, script (for a dlt job) | one name — **job**; the runtime's `Script` model keeps its own name |\n| job execution, invocation (for one run) | **job run**, the noun the manifest and the beacon use |\n| primary trigger | **default trigger**, the name of the manifest field |\n| manifest (unqualified, for `AGENT.md`) | **deployment manifest** is the only manifest; see G8 |\n\n**Rulings**\n\n- **`run` is a verb; `job run` is the noun.** \"dlt runs the job\" and \"the job run failed\". The\n command `dlthub local run` is a technical name (Rule 8.6) and stays.\n- **launcher, runner and runtime are three things.** The launcher is dlt code. The runner is a\n machine. The runtime is the platform. Never swap them, and never write \"the runtime launches\".\n- **`task` is legal in two places only:** the work an agent must do (G8), and an Airflow task when\n the sentence says \"Airflow task\". It is never a dlt job.\n",".claude/skills/review-vocabulary/references/vocabulary/g8-agents-loops-and-prompts.md":"# G8 — Agents, loops and prompts\n\nThree nouns carry the feature, and they nest: a definition is written once, a job adds how it\noperates, a run is one execution.\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| what an `AGENT.md` or a decorated function declares: system prompt, inputs, output, tools, skills, rules, access | **agent definition** (`TAgentSpec` in code, `TAgentDefinition` for the block the manifest carries; pydantic-ai's `AgentSpec`) |\n| the file it is read from | **`AGENT.md`**, and **agent file** for the path |\n| the name a definition is referred to by | **agent definition reference**: `<toolkit>:<agent>`, a workspace path, or `<module>:<function>` for a function with no `AGENT.md` behind it |\n| the definition plus the runtime settings that say how it operates: model, limits, trigger, instructions, loop, identity | **agent job** (`run.agent(...)`; what pydantic-ai and claude-agent-sdk call an Agent, and what the web UI lists as one) |\n| one execution of an agent job: inputs in, output and trace out, settings overridable for that run | **agent run** (`Agent.run()` in the frameworks) |\n| what the model returns: `status`, `summary` and the declared fields | **agent output** |\n| the envelope the launcher delivers | **job result** (G7 owns it; an agent job's is `TAgentJobResult`) |\n| the binding to one agent framework | **loop**, or **agent loop** |\n| the text the model gets as its role and task | **system prompt** |\n| the first message of the run | **user turn** |\n| what a person tells this run to do | **instructions** |\n| one model request | **turn** |\n| the record of what the loop ran and did | **agent trace** |\n| `{{ name }}` in a body | **placeholder** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| agent spec (in prose) | **agent definition**; `TAgentSpec` keeps its name as a type |\n| agent reference, agent ref (in prose) | **agent definition reference**; the field `agent_ref` keeps its name |\n| agent (bare) where the sentence needs one of the three | say which: **agent definition**, **agent job** or **agent run** |\n| harness (for the Claude Code CLI) | the **dltHub AI Harness** owns that word; write **the Claude Code CLI** |\n| adapter (for a loop) | dlt's adapters are `bigquery_adapter` and friends; write **loop** |\n| prompt (bare) | say which one: **system prompt** or **user turn** |\n| spec (bare, for an agent) | bare `spec` is the configspec; write **agent definition** |\n| trace (bare, for an agent) | bare `trace` is the pipeline trace; write **agent trace** |\n| manifest (for `AGENT.md`) | **agent file**; the manifest is the deployment manifest (G7) |\n| framework (as a loop's name) | name it: **pydantic-ai**, **claude-agent-sdk** |\n\n**Rulings**\n\n- **Bare `agent` is the actor.** \"The agent reads the logs\", \"the agent returns `aborted`\": the\n model acting during a run. That stays. The moment a sentence is about the file, the `run.agent`\n product or one execution, it names the definition, the job or the run.\n- **`Agent` (capitalised) is the agent job.** That is how the web UI labels it and how pydantic-ai\n and claude-agent-sdk name the same thing. Legal in UI copy and when talking about a framework;\n in dlt prose write **agent job**.\n- **An agent run is a job run.** Use **agent run** when the sentence is about what the agent\n received and produced; **job run** when it is about scheduling, status or logs, as for any job.\n- **`instructions` means two opposite things across the boundary.** In dlt it is the user turn. In\n pydantic-ai, `AgentSpec.instructions` is the system prompt. Qualify every mention of the\n framework field: \"pydantic-ai's `instructions` field (its system prompt)\".\n- **`turn` is one model request.** The **user turn** is the first message. A turn counter counts\n requests. Do not let one word carry both without the qualifier.\n- **`loop` is the agent loop.** In a file that also has Python loops, write **agent loop** once,\n then `loop`.\n- **Model names and aliases are technical nouns** (Rule 1.8): `sonnet`, `opus`, `claude-sonnet-5`.\n",".claude/skills/review-vocabulary/references/vocabulary/g9-workspace-access-and-tools.md":"# G9 — Workspace access and tools\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the declaration of what a job may touch | **access** (the `access` block, `TWorkspaceAccess`) |\n| one of its three keys, `local`, `data` and `context` | **access axis** |\n| a value on an axis | **verb** |\n| what the runtime or a loop does with the declaration | **grant**, **deny** (verbs) |\n| what it did grant | **granted access** |\n| what it did not grant | **denied access** |\n| what a loop supplies beyond the declaration | **over-granted** |\n| what a tool needs before it is served | **required access** |\n| what a model can call | **tool** |\n| the MCP grouping a manifest requests | **feature group** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| permission, entitlement, scope (for the declaration) | **access**, the word dlt's docs already use; `permission` is legal only when naming the SDK's `permission_mode` |\n| narrowed, narrowing | say the fact: **with less access** |\n| grant (as a noun) | **the declared access**; `granted` as an adjective is legal (Rule 3.3) |\n| capability (for a dlt tool) | **tool**; legal only for pydantic-ai `capabilities` |\n| ceiling, floor (as prose metaphor) | say what it does: \"access is the most a loop can wire\" |\n| honor, honour, honored, unhonoured (any spelling) | access is **granted** or **denied**, never honored |\n\n**Rulings**\n\n- **Grant and deny are the verbs.** A job declares access; the runtime and the loop grant what they\n can; the rest is denied. dlt's docs already write \"grant access\" 13 times and \"denied access\"\n twice, so this is the house pairing, not a new one. A loop that supplies more than the\n declaration **over-grants**.\n- **The declaration is a request, not a claim.** A manifest `access` block says what the job wants.\n Nothing in it is granted until the runtime says so. Say \"the job declares\", never \"the job has\".\n- **`access` is the noun, and no synonym joins it.** `permissions`, `scopes`, `capabilities` and\n `entitlements` each name this concept somewhere in the industry; dlt named it `access` before this\n feature existed — the built-in `access` profile is coarse-grained data access, and `profile_for()`\n derives that profile from `access.data`. One concept, one word (Rule 1.11).\n- **`tools:` in an `AGENT.md` holds feature groups, not tools.** Write \"feature groups\" whenever the\n sentence is about that field, or a reader counts 4 tools and gets 19.\n",".claude/skills/switch-worktree/SKILL.md":"---\nname: switch-worktree\ndescription: Switch the current session to work in an existing git worktree\nargument-hint: <worktree-name>\n---\n\n# Switch to worktree\n\nSwitch the current session to an existing worktree.\n\n## Steps\n\n### 1. Resolve the worktree\n\nIf `$ARGUMENTS` is empty, list available worktrees and ask the user to pick one:\n\n```\ngit worktree list\n```\n\nOtherwise, let `NAME` = `$ARGUMENTS`. The worktree path is `<repo-root>/.worktrees/<NAME>`.\n\nVerify the worktree exists in `git worktree list` output. If not, stop with an error.\n\nLet `WORKTREE` = the absolute path to the worktree.\n\n### 2. Switch cwd\n\n```\ncd WORKTREE\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, stop with an error — the cwd did not persist.\n\n### 3. Report\n\n```\nSwitched to worktree: WORKTREE\nBranch: <current branch from git branch --show-current>\n```\n",".claude/skills/worktree-from-issue/SKILL.md":"---\nname: worktree-from-issue\ndescription: Create a git worktree with a new branch for implementing a fix or feature for a GitHub issue\nargument-hint: <category>/<issue-number>-<short-description>\n---\n\n# Git Worktree for Issue Implementation\n\nSet up an isolated git worktree with a new branch to implement a fix or feature for a GitHub issue.\n\nParse `$ARGUMENTS` to extract the branch name in format `<category>/<issue-number>-<short-description>`.\n\n## Branch Naming Convention\n\nBranches must follow the pattern from CONTRIBUTING.md:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n```\n\nCategories: **feat**, **fix**, **exp**, **test**, **docs**, **keep** (all lowercase, dashes, no underscores).\n\nFor `feat` and `fix`, a ticket number is mandatory.\n\nExamples:\n- `fix/3562-clickhouse-drop-sync-perf`\n- `feat/1234-add-new-destination`\n\n## Steps\n\n### 1. Parse and validate\n\nExtract from `$ARGUMENTS`:\n- `branch_name`: Full branch name (e.g., `fix/3562-clickhouse-drop-sync-perf`)\n- `category`: Prefix before `/` (e.g., `fix`)\n- `issue_number`: Number after `/` and before first `-` (e.g., `3562`)\n- `short_name`: Directory-friendly version: `{category}-{issue_number}-{first-few-words}` (e.g., `fix-3562-clickhouse-drop-sync`)\n\nValidate the branch name matches pattern `(feat|fix|exp|test|docs|keep)/[0-9]+-[a-z0-9-]+`.\n\n### 2. Verify the issue exists\n\n```\ngh issue view <issue_number> --json number,title,state\n```\n\nWarn if the issue doesn't exist or is closed, but continue.\n\n### 3. Fetch latest devel\n\nFetch the newest devel from origin so the branch starts from the latest code:\n\n```\ngit fetch origin devel\n```\n\n### 4. Create the worktree\n\nUse `/create-worktree` skill:\n\n```\n/create-worktree <short_name> --branch origin/devel\n```\n\nThen create the feature branch inside the worktree:\n\n```\ngit -C <worktree-path> checkout --no-track -b <branch_name>\n```\n\n`--no-track` leaves the branch without an upstream, so the user's `git push` works.\n\n### 5. Set up dev environment\n\nUse `/worktree-make-dev` skill to set up the development environment in the worktree. This runs `make dev` and copies `secrets.toml`.\n\nNote which secrets source was used — this determines what destinations can be tested.\n\n### 6. Verify cwd\n\nAfter dev setup, verify the cwd is still in the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, run `cd <worktree-path>` and verify again. Stop with an error if the cwd cannot be set.\n\n### 7. Report\n\nReport the worktree info and available test destinations:\n\n```\nWorktree ready: <worktree-path>\nBranch: <branch_name> (based on origin/devel)\nIssue: #<issue_number> - <issue_title>\n```\n\nInclude the test credential availability from `/worktree-make-dev` output — downstream skills need this to know which destinations can be tested.\n",".claude/skills/worktree-make-dev/SKILL.md":"---\nname: worktree-make-dev\ndescription: Set up a dev environment in an existing worktree (make dev + copy secrets)\nargument-hint: <worktree-path>\n---\n\n# Set up dev environment in worktree\n\nPrepare the worktree at `$ARGUMENTS` for development.\n\n## Steps\n\n### 1. Run setup script\n\n```\nbash tools/setup_worktree_env.sh <worktree-path>\n```\n\nThe script runs `make dev` and copies secrets without exposing credentials in the conversation. If it fails, report the error and stop.\n\n### 2. Report\n\nParse the script output for `SECRETS_SOURCE=` to determine what was used:\n\n- **`repo_root`**: Full credentials available — all destinations including remote (snowflake, bigquery, etc.)\n- **`dev_template`**: Local only — duckdb, dummy, filesystem, and container-based destinations (postgres, clickhouse, etc.)\n- **`none`**: No secrets found — warn the user\n\n```\nDev environment ready: <worktree-path>\n- make dev: done\n- secrets.toml: <source>\n```\n","AGENTS.md":"# dlt principles\n- This is a mature library used by thousands of the best engineers in the world\n- dlt is a library, not a platform. users add it to their code. we respect existing workflows and other libraries they use\n- no black boxes: clean, minimal Pythonic interfaces, human readable file formats, no side effects, documentation\n- multiply - don't add: we do more work here so our users do less.\n\n# dlt repo\n- we use `uv` and `uv run`. look in @Makefile\n- we branch from `devel` for feature branches\n\n# Commit messages (all agents)\n- [Conventional Commits](https://www.conventionalcommits.org/): `type:` or `type(scope):`, lowercase imperative subject, no trailing period.\n- Subject line only for the vast majority of commits; add a body line only for a non-obvious *why* (max ~3 lines total).\n- No footers — no `Co-Authored-By`, no \"Generated with ...\". No emojis.\n- When squash-merging, clean the squash message down to a single subject line.\n- Full rules (enforced per-agent via git/gh hooks under `.claude/`, `.cursor/`, `.codex/`): @.agents/rules/git-github.md","CLAUDE.md":"AGENTS.md","docs/website/docs/hub/agents/agent-definitions.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fagent-definitions.md -->","docs/website/docs/hub/agents/index.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Findex.md -->","docs/website/docs/hub/agents/job-inspector.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fjob-inspector.md -->","tests/plugins/dlt_example_plugin/README.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fplugins%2Fdlt_example_plugin%2FREADME.md -->","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/dlthub/agents/e2e/self-report/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_e2e_workspace%2F.claude%2Fdlthub%2Fagents%2Fe2e%2Fself-report%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-hidden-rule.md":"This rule carries the token MARKER-RULE-HIDDEN. An agent asked for its markers reports it.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-visible-rule.md":"This rule carries the token MARKER-RULE-VISIBLE. An agent asked for its markers reports it.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/hidden-skill/SKILL.md":"---\nname: hidden-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-HIDDEN. Copy it into `markers`.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/visible-skill/SKILL.md":"---\nname: visible-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-VISIBLE. Copy it into `markers`.\n","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/broken-code/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fbroken-code%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/checked-inspector/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fchecked-inspector%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/error-triage/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Ferror-triage%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/job-inspector/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fjob-inspector%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/rules/dlthub-platform-job-resources.md":"Resource changes are proposals, never applied automatically.\n","tests/workspace/cases/workspaces/agent_workspace/.claude/skills/debug-deployment/SKILL.md":"---\nname: debug-deployment\ndescription: Debug a failing deployment.\n---\nRead the run record first.\n","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_toolkit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/skills/find-source/SKILL.md":"---\nname: find-source\n---\nFind a source.","tests/workspace/cli/dlthub/ai/cases/mock_workbench/bootstrap/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fbootstrap%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/init/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Frest-api-pipeline%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/skills/find-source/SKILL.md":"---\nname: find-source\ndescription: Find and configure a REST API source\n---\n# Find Source\nInstructions here.","tests/workspace/cli/dlthub/ai/cases/mock_workbench/sql-database/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fsql-database%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/init/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/my-toolkit/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Fmy-toolkit%2F.claude-plugin%2Fplugin.json -->","tools/tests/cases/dependency_changes/README.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tools%2Ftests%2Fcases%2Fdependency_changes%2FREADME.md -->"},"files":{".agents/rules/git-github.md":"---\nname: git-github\ndescription: Rules for using git and the GitHub CLI in dlt. Read when committing, creating PRs, or running any git/gh command.\n---\n\n# dlt — Git & GitHub\n\nRules for git and GitHub CLI usage in this repo, for any coding agent. They override the agent's defaults where they differ. See `CONTRIBUTING.md` for the human-facing version.\n\n## When to use\n\nApply whenever running `git` or `gh`, drafting commit messages, or creating/updating PRs.\n\n## Commit messages\n\nFollow [Conventional Commits](https://www.conventionalcommits.org/):\n\n```\n{type}: short imperative subject\n# or, when one area is in focus\n{type}({scope}): short imperative subject\n```\n\n- **Types** (lowercase, matching branch categories): `feat`, `fix`, `docs`, `test`, plus `refactor`/`chore` for non-functional changes.\n- **Scope** is optional — an area like `dashboard`, `cli`, `destinations`, `extract`, `normalize`. Comma-separate multiple.\n- **Subject is lowercase, imperative, no trailing period.**\n- **No emojis** anywhere in the message — not in the subject, not in the body.\n- **Subject line only for the vast majority of commits.**\n- **Max 3 lines total** including any body. If you can't say it in 3 lines, the commit is probably too big — split it.\n- **Code-comment rules apply to commit bodies too.** Don't explain *what* the diff does (the diff shows that). Add a body line only when the *why* is non-obvious and not already in the PR description.\n- **No footers.** No `Co-Authored-By` lines, no \"Generated with ...\" trailers. Treat the agent as a dev tool, not a co-author. This overrides the agent's default.\n- **No references to the task/ticket/caller** in the message unless the user asks (\"as discussed\", \"per review feedback\", etc.).\n\nFormat (HEREDOC):\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n## Cleaning up before a squash-merge\n\nGitHub hides everything after the first line in the commit list, so multi-line\nbodies and footers are easy to overlook — and squashing several such commits\nfloods the merge commit. **When squash-merging, clean the squash message down to\na single Conventional-Commit subject line** (plus a short *why* body only if it\ngenuinely helps). Never let `Co-Authored-By` / \"Generated with\" lines survive\ninto the merge.\n\n## When to commit\n\n- **Only when the user explicitly asks.** Finishing a task is not a trigger; \"looks good\" is not a trigger. Wait for an explicit \"commit this\" / \"make a commit\".\n- `git add -A` is fine — but NEVER stage anything secret-looking (`.env`, `*secrets.toml`, credentials).\n\n## Branches\n\nFrom `CONTRIBUTING.md` — all lowercase, dashes, no underscores:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n# example:\nfeat/4922-add-avro-support\n```\n\n- **Categories**: `feat` (ticket required), `fix` (ticket required), `exp`, `test`, `docs`, `keep`.\n- **Branch from `devel`.** Feature branches and most fixes go to `devel`.\n- Don't create branches unless asked. Don't switch the branch a worktree is on without asking.\n\n## Pull requests\n\n- PRs **target `devel`**. Agents may **NEVER** merge to `master`.\n- **Link the PR to its ticket**, or describe the change clearly enough for someone without prior context.\n- **Creating a PR requires the branch to be pushed already.** If `gh pr create` would need a push, stop and ask the user to push.\n- Confirm the title and body with the user before running `gh pr create`.\n\n## Hard rules for the coding agent (never without an explicit ask)\n\n- **Don't push.** No `git push`/`git push -u`. The user pushes.\n- **Don't merge PRs.** No `gh pr merge`, no auto-merge.\n- **Don't force-push, ever.** If history needs rewriting, describe it and let the user do it.\n- **Don't `--amend` pushed commits**, and never use `--no-verify` or skip hooks/signing.\n- **Don't touch `master`** — it's the release branch (`devel → master` on release day).\n\n## Safe read-only commands (no confirmation needed)\n\n- `git status`, `git diff`, `git log`, `git show`, `git branch`, `git remote -v`\n- `gh pr view`, `gh pr list`, `gh pr checks`, `gh pr diff`, `gh issue view`, `gh api` for GET requests\n\n## Quick checklist before any git/gh action\n\n1. Push, merge, force-push, or amend of a pushed commit? → **Stop**, the user does this.\n2. New commit? → Did the user explicitly ask? If no, **stop**.\n3. Is the subject `type:` / `type(scope):`, lowercase, 1–3 lines total, free of \"what\" narration? If no, rewrite.\n4. Is the commit free of footers (no `Co-Authored-By`, no \"Generated with\") and emojis? If no, trim.\n5. Squash-merging? → Clean the squash message to a single subject line.\n",".claude/rules/add-workspace-command.md":"---\npaths:\n - \"dlt/_workspace/cli/**\"\n - \"dlt/_workspace/mcp/**\"\n - \"dlt/_workspace/helpers/dashboard/**\"\n - \"dlt/_workspace/typing.py\"\n - \"tests/workspace/**\"\n---\n\n# Workspace model/loader/view pattern for commands\n\nWhen adding a new command (cli, mcp or dashboard) `dlt/_workspace` follows a limited MVC pattern so that data models stay independent\nfrom presentation. The same model can be rendered by a CLI print function, returned\nfrom an MCP tool, or displayed in a marimo dashboard.\n\n## Layers\n\n| Layer | Location | Convention |\n|-------|----------|------------|\n| **Model** | `dlt/_workspace/typing.py` | `TypedDict` (T-prefixed), plain data, no I/O |\n| **Loader** | `dlt/_workspace/cli/ai/utils.py`, `cli/utils.py` | `fetch_*()` functions — collect data into a model, no `fmt.echo()` calls |\n| **View** | `cli/*_command.py`, `cli/commands.py` | `_print_*()` or `print_*()` — render a model to CLI via `fmt.echo()` / `fmt.warning()` |\n| **Controller** | argparse handlers, MCP tools, marimo notebooks | Calls loader, then passes model to the appropriate view |\n\nNOTE: loaders are stored in cli module. to be refactored soon\n\n## Rules\n\n1. **Loaders return typed dicts, never print.**\n A loader (`fetch_ai_status`, `fetch_workspace_info`, `fetch_secrets_list`, ...)\n returns a `TypedDict` from `typing.py`. It must not call `fmt.echo()` or any\n display function.\n\n2. **Views accept a model, never load.**\n A view (`_print_ai_status`, `print_workspace_info`, ...) takes the typed dict\n and formats it for human consumption. It must not call loaders or access\n providers/config directly.\n\n3. **Controllers wire loader to view.**\n An argparse command function (decorated with `@track_command`) calls the loader,\n then the view. MCP tools call the loader and return the model (or a\n serialised form). Dashboard cells call the loader and render via marimo UI.\n\n4. **Research existing types**\n * Research other `typing.py` modules and Literal definitions and types to find potentially reusable types.\n * Reuse, reconcile, modify existing types - also from `common` and other namespaces\n * Make **reconciliation** and model to be **clean** part of implementation plan. Request user input if needed\n\n5. **Write good models**\n * Use `Literal` ie. for error or warning cases\n * Do not leak user messages into data model - view should do it\n * OK to store additional context ie. exception messages\n * Use `NotRequired` field to the model (e.g. `mcp_error: NotRequired[str]`) if element may not be present. In most cases that is better than setting to None\n\n\n## Existing examples\n\n- `fetch_workspace_info()` -> `TWorkspaceInfo` -> `print_workspace_info()`\n- `fetch_ai_status()` -> `TAiStatusInfo` -> `_print_ai_status()`\n- `fetch_secrets_list()` -> `List[TLocationInfo]` -> `ai_secrets_list_command()` (inline view)\n\n## When adding new commands\n\nBefore writing a new CLI command, MCP tool, or dashboard cell that shows\nworkspace state, check whether a loader already exists. If not, create the\nmodel in `typing.py` and the loader in the appropriate `utils.py` first,\nthen write the view / controller.\n",".claude/rules/coding-style.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Coding style\n\n## Formatting\n- Line length: 100 chars (black + isort, profile \"black\")\n\n## Code and patterns reuse\n- Obvious utils and patterns are already in code base\n- When in doubt look at the code around the changes you are making - not only style but coding patterns and code organization\n- When using a util function or any other function from the code base - look at existing usage if in doubt how it works\n\n## Comments\n- Inline comments start with lowercase (`# resolve the schema`)\n- NEVER add comments that separate code blocks (`# -- sqlglot_dialect mapping --`)\n- DO NOT state the obvious, do not document your thought process\n\n## Types and data structures\n- Always use full generic parametrization -- `Dict[str, Any]` not `dict`\n- Type aliases use T-prefix PascalCase (`TDataItem`, `TLoaderFileFormat`) -- see @dlt/common/typing.py\n- `TypedDict` for structured dicts, `NamedTuple` for lightweight immutable data\n- Prefer `functools.partial` over `lambda` when currying\n\n## Exceptions\n- Inherit from `DltException`; use `TerminalException`/`TransientException` mixins -- see @dlt/common/exceptions.py\n",".claude/rules/docstrings.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Docstring rules\n\n- DO NOT state the obvious, do not document your thought process\n\nGoogle style docstrings throughout. Three levels of detail:\n\n## Short form (internal methods, properties)\nDescription only -- no Args/Returns. One-liner for simple accessors.\n\n## Full form (public API)\nFull Google style with Args, Returns, Raises. Describe WHAT the function does and its output, never HOW it works internally.\n\n```python\ndef paginate(\n self,\n path: str = \"\",\n method: HTTPMethodBasic = \"GET\",\n params: Optional[Dict[str, Any]] = None,\n) -> Iterator[PageData[Any]]:\n \"\"\"Iterates over paginated API responses, yielding pages of data.\n\n Args:\n path (str): Endpoint path for the request, relative to `base_url`.\n method (HTTPMethodBasic): HTTP method for the request, defaults to 'get'.\n params (Optional[Dict[str, Any]]): Query parameters for the request.\n\n Yields:\n PageData[Any]: A page of data with request and response context.\n\n Raises:\n HTTPError: If the response status code is not a success code.\n \"\"\"\n```\n\n- Public API args use `name (Type):` format; internal methods may omit types\n- Args descriptions are short -- one line unless truly complex\n- Use `Yields:` for generators, `Returns:` for regular functions\n- `Raises:` only for exceptions the caller should handle\n- `Example:` section is optional, add only when usage is non-obvious\n\n## Field docstrings\nInline triple-quoted string on the NEXT line after the field definition.\n\n```python\n@configspec\nclass ItemsNormalizerConfiguration(BaseConfiguration):\n add_dlt_id: bool = True\n \"\"\"When true, items will have `_dlt_id` column added with a unique ID.\"\"\"\n add_dlt_load_id: bool = False\n \"\"\"When true, items will have `_dlt_load_id` column added with current load ID.\"\"\"\n```\n\n- Only document fields where the name + type aren't self-explanatory\n- Keep to one line when possible\n\n## Inline formatting\nDocstrings are rendered as Markdown by most editors. Use only these:\n- Single backticks for parameter names, values, types, paths: `name`, `True`, `None`\n- **bold** for emphasis on key terms\n- `Note:` or `Warning:` prefixes for callouts in prose\n- `>>>` interactive Python style for code examples:\n\n```python\n \"\"\"Load data from a REST API.\n\n Example:\n >>> import dlt\n >>> pipeline = dlt.pipeline(\"my_pipeline\", destination=\"duckdb\")\n >>> pipeline.run(my_source())\n \"\"\"\n```\n\nDo NOT use:\n- Double backticks ``like this`` (reStructuredText literal)\n- `:param:`, `:type:`, `:returns:`, `:rtype:` (Sphinx field list)\n- `.. directive::` blocks (reST directives)\n\n## Anti-patterns -- NEVER do these\n- Do not explain HOW the code works, only WHAT it does and its output\n- Do not state the obvious (`\"\"\"Returns the name\"\"\"` on `get_name`)\n- Do not add docstrings to code you did not change\n- Do not use restructured text/sphinx formatting (see Inline formatting above)\n",".claude/rules/documentation.md":"---\npaths:\n - \"docs/**\"\n---\n\n# Documentation guidelines\n\n## Setup\n- On your first setup, go to directory `docs/` and run `make dev`. This will install the required Python and JS/TS dependencies to build the docs `cd docs && make dev`.\n- This will also install pre-commit hooks using `prek`. This will automatically run linting, type checking, etc. on each `git commit` and `git push`\n\n## Code snippets\n- Inline code snippets will automatically be formatted, linted, and type-checked. To opt-out, add the directive to the codefence `notype` or `nolint`.\n- The `execute` directive means the snippet will be ran during before each commit and on CI. The executed snippets should be lightweight.\n\nFor example\n\n > \\```python notype execute\n >\n > import foo\n > \n > def my_func() -> int: ...\n > \n > \\``` \n \n## Internal links\n- Use relative paths **with** `.md` extension: `[schema contracts](schema-contracts.md)`\n- Link to sections with `#` anchors (auto-generated from headings, lowercase hyphenated): `[merge strategies](merge-loading.md#merge-strategies)`\n- Go up directories as needed: `[adjust a schema](../walkthroughs/adjust-a-schema.md)`\n\n## Markdown formatting\n- `mdsmith` lints `website/docs` on CI (`md-check` hook), configured in `docs/.mdsmith.yml`. After editing a page, run from `docs/`:\n ```sh\n make md-fix # pads and aligns tables, fixes whitespace\n make md-check # must report failures=0\n ```\n- Tables: write them unaligned and let `make md-fix` pad the columns (MDS025). It cannot fix a cell over 35 words (MDS026): shorten the text or move the detail below the table.\n\n## Markdown frontmatter\n- Every doc page needs YAML frontmatter with at least `title`, `description`, and `keywords`:\n ```yaml\n ---\n title: Schema\n description: Schema definition and evolution\n keywords: [schema, dlt schema, yaml]\n ---\n ```\n",".claude/rules/heresy.md":"Heresy: issues in code that not only must be **avoided** but actively **detected and removed**.\n\n## llm heresy\n* llm talks to itself in docstrings instead of writing code comments in relevant places\n* llm talks about some obscure downstream cases in upstream code (ie. base class is full of details of particular implementation or explains particular use case)\n* llm invents obscure terminology and uses it everywhere. does not follow terminology used in docs\n* llm writes extremely detailed and complex sentences to explain simple things\n\n## arithmetic heresy\n* decimals are converted to binary floats and back\n* binary floats are used to construct decimals. ie. `Decimal(1.1)`\n\n## datetime heresy\n* naive datetimes are used without a good reason. the only exception are end user requirements\n* pendulum is used in new code. use the stdlib counterparts: `datetime` with an explicit\n `timezone`/`ZoneInfo`, and the `ensure_datetime*` helpers instead of `ensure_pendulum_datetime*`.\n NOTE: existing pendulum usage MAY be tolerated where it is hard to change\n\n## terms heresy\n* use of following words constitutes heresy: gate/gating, graft\n\n## comments heresy\n* block separator comments as below\n```\n#------------------------------\n# comment\n#------------------------------\n```\nor similar. actually any form",".claude/rules/imports.md":"---\npaths:\n - \"**/*.py\"\n---\n\n# Import rules\n\n## Import order\n\nGroups separated by blank line:\n1. **stdlib** (`import os`, `from typing import ...`)\n2. **third-party** (`import pendulum`, `import semver`)\n3. **local dlt** (`from dlt.common import ...`)\n4. (tests only) **test utilities** (`from tests.utils import ...`)\n\n- Stdlib imports are ALWAYS at module level -- never inline in functions. This includes `contextlib`, `io`, `warnings`, `sys`, `os`, `re`, etc.\n- `dlt.common` imports (except `dlt.common.libs`) are module level. They are always available and have no heavy side effects.\n- `dlt.common.libs.*` imports may be inline -- they wrap optional dependencies that may not be installed (see Optional dependencies below).\n- If a module is already imported at the top of the file, additional imports from the same module go at the top too -- never inline a second import from an already-imported module.\n- No relative imports.\n\n## Logging\n\n`from dlt.common import logger`, then `logger.info(...)` -- never `import logging` directly (except in `logger.py` itself or runtime infrastructure).\n\n## TYPE_CHECKING imports\n\n- Optional deps for annotations only: `if TYPE_CHECKING: from X import Y` with `else: Y = Any`\n- Destination factories: client class under `TYPE_CHECKING`, actual import deferred to runtime method\n\n## Optional dependencies\n\ndlt wraps optional deps in `dlt/common/libs/` with fail-fast `MissingDependencyException`. Always import from the wrapper, never from the package directly.\n\n### Canonical patterns\n\n```python\nfrom dlt.common.libs.pandas import pandas\n```\n\n### Rules\n\n- NEVER import **optional package** directly\n- ALWAYS import from `dlt.common.libs.<optional package>`\n- Import at the top of the module if module already imports **optional package**\n- Import inline in other cases.\n\n**Applies to**: `pyarrow`, `pandas`, `numpy`, `pydantic`, `sqlalchemy`, `ibis`, `deltalake`, `pyiceberg`\n",".claude/rules/public-api.md":"---\npaths:\n - \"dlt/**/*.py\"\n---\n\n# Public API and backward compatibility\n\n## Public interface\n- Everything DEFINED and DIRECTLY IMPORTED in @dlt/__init__.py is PUBLIC INTERFACE. Change here is BREAKING CHANGE.\n- Behaviors described in @docs are considered public and changing them is a BREAKING CHANGE (in outputs of PUBLIC INTERFACES)\n- Always flag BREAKING CHANGES clearly with # BREAKING: comments\n\n## Deprecation warnings\n- Use `DltDeprecationWarning` from `dlt/common/warnings.py`, never raw `DeprecationWarning`\n- Pass the `since` version where the deprecation is introduced\n- Use `Dlt100DeprecationWarning` only for features deprecated since 1.0.0 (hardcodes `since`)\n- To deprecate entire functions or classes, use the `deprecated` decorator from `dlt/common/warnings.py`\n\n## Backward compatibility mechanisms\n- Avoid BREAKING CHANGES with deprecations, configuration options etc.\n- Schema/state migration via `engine_version`, versioned storage layouts, deprecation warnings, `deprecated` decorator\n- Backward compat must be tested -- see `tests/pipeline/test_dlt_versions.py` for e2e version upgrade tests\n",".claude/rules/testing.md":"---\npaths:\n - \"tests/**/*.py\"\n - \"tools/tests/**/*.py\"\n---\n\n# Testing\n\n## Style and structure\n- Module-based tests (plain `def test_*() -> None:` functions), not class-based\n- pytest with fixtures in `conftest.py`, not unittest setUp/tearDown\n\n## Reduce tests sprawl\n- Use `pytest.mark.parametrize` to reduce code duplication. ALWAYS use human readable ids for the test cases\n- It is OK to test many related use cases in one test: ie. when they share common setup or represent a flow of user actions\n\n## Fixtures\n- We have autouse fixtures enabled by `conftest.py`. Always check those before adding common fixtures ie. to preserve env variables or get clean storage for the tests\n- We have A LOT of fixtures in files like @tests/utils.py and @tests/load/utils.py, look there before adding new ones.\n\n## Large files for test cases\n- Do not keep large test cases in Python modules. Use `cases` folder and to keep and share files (ie. schemas, ingest data) across modules. \n\n## Destination tests (tests/load)\n- Tests in @tests/load are ALWAYS run on CI in context of particular destination. Tests that run pipelines MUST use `destinations_configs`, other test modules must filter for a right destination (ie. `skip_if_not_active(\"clickhouse\")`)\n- NOTE: tests outside @tests/load do not use such filtering. Use `duckdb`, `dummy` or `sqlite` or `filesystem` destinations for such tests.\n\n### Test Markers\n- `@pytest.mark.essential` — For critical test paths\n- `@pytest.mark.serial` — Tests that can't run in parallel\n\n## Parallel safety and isolation\n- Tests run in parallel! ALWAYS use test storage (`get_test_storage`), unique pipeline names (`uniq_id`), and worker-aware storage roots (`get_test_storage_root` from `tests/utils.py`). Isolate pipelines with `dev_mode` or random `dataset_name`. Avoid external resources that clash with other tests.\n- Tests must work on Linux, macOS and Windows. Avoid encoding paths and platform-specific code. Local path assertions must be Windows compatible.\n\n## Placement and comments\n- When adding new test ALWAYS try find the right place by looking where code you make changes is tested\n- Docstrings: DESCRIBE test case. SKIP when test name is sufficient\n- Use line comments to describe and separate use cases within a single test\n\n## Running\n* DO NOT start two `pytest` processes at once!\n* when running many tests, use `-p xdist -n auto` to parallelize\n",".claude/rules/worktree.md":"# Working in a git worktree\n\nWhen a worktree is active (created via `/create-worktree`, `/worktree-issue`, or the user says \"work in worktree X\"), apply these rules for ALL subsequent operations.\n\nLet `WORKTREE` = the absolute worktree path (e.g., `/home/user/src/dlt/.worktrees/fix-3562`).\n\n## Shell cwd behavior\n\nClaude Code resets Bash cwd to the project root only when `cd` targets a path **outside** the project root. Paths **within** the project root (including `.worktrees/`) persist across calls.\n\n- **Bash**: run `cd WORKTREE` once to switch; subsequent calls will stay there\n- **git**: `git -C WORKTREE <command>` also works\n- **Read/Grep/Glob/Edit/Write**: always use absolute paths under `WORKTREE`\n\n## Running Python and tests\n\nAlways use `uv run` in worktrees — never bare `python` or `pytest`:\n\n```\nuv run pytest tests/... -v\nuv run python -c \"...\"\n```\n\n## Worktree location convention\n\nAll worktrees live under `<repo-root>/.worktrees/<name>`.\n\n## Plans created in a worktree\n\nWhen creating a plan while working in a worktree, **always** include the worktree path at the top of the plan:\n\n```markdown\n## Working environment\n- Worktree: <absolute worktree path>\n- Branch: <branch name>\n- Execute: cd <absolute worktree path> before any work\n```\n\nThis is critical because plan→execute transitions start a new session whose cwd resets to the project root. Without the worktree path in the plan, the execute phase will modify files in the main repo instead of the worktree.\n",".claude/skills/auto-docstring-rules/SKILL.md":"---\nname: auto-docstring-rules\ndescription: Enforces dlt docstring conventions. Use when writing or modifying docstrings, classes, or function signatures in Python files.\nuser-invocable: false\n---\n\nUse @.claude/rules/docstrings.md regardless of path setting",".claude/skills/auto-import-rules/SKILL.md":"---\nname: auto-import-rules\ndescription: Enforces dlt import conventions and ordering. Use when writing or modifying Python imports in dlt/ or tests/.\nuser-invocable: false\n---\n\nUse @.claude/rules/imports.md regardless of path setting",".claude/skills/auto-write-test-rules/SKILL.md":"---\nname: auto-write-test-rules\ndescription: Use when adding or proposing a new test ONLY if the test rules are not present\nuser-invocable: false\n---\n\nUse @.claude/rules/testing.md regardless of path setting",".claude/skills/commit/SKILL.md":"---\nname: commit\ndescription: Stage changes and write a clean Conventional-Commit message that follows the repo's git rules\nargument-hint: [-- <optional subject hint or scope>]\n---\n\n# Commit\n\nCreate a commit that follows `.agents/rules/git-github.md`. Invoke as `/commit`, or\nuse these steps whenever you are about to commit.\n\nParse `$ARGUMENTS`: anything after `--` is an optional hint (a subject, a scope, or\n\"only the dashboard files\") — guidance, not the literal message.\n\n## Steps\n\n### 1. Inspect\n\n```\ngit status\ngit diff --staged\ngit diff\n```\n\nDecide what belongs in the commit. If nothing is staged, stage the relevant changes\nwith `git add <paths>` (or `git add -A` when everything belongs together). **NEVER\nstage secret-looking files** (`.env`, `*secrets.toml`, credentials) — they are also\ndenied in `settings.json`.\n\n### 2. Compose the message\n\nApply `.agents/rules/git-github.md`:\n\n- One line: `{type}: subject` or `{type}({scope}): subject` — lowercase, imperative,\n no trailing period. Types: `feat`, `fix`, `docs`, `test`, `refactor`, `chore`.\n- **Subject line only** for the vast majority of commits. Add a body line *only* for a\n non-obvious *why*, max 3 lines total. Never narrate *what* the diff does.\n- **No footers** — no `Co-Authored-By`, no \"Generated with ...\" trailers.\n- **No emojis** anywhere in the message.\n\n### 3. Commit\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n### 4. Confirm\n\n```\ngit log -1 --format=%B\n```\n\nVerify it is a single Conventional-Commit subject (plus an optional why) with no\nfooters. **Do not push and do not amend pushed commits** unless the user explicitly\nasks.\n",".claude/skills/create-worktree/SKILL.md":"---\nname: create-worktree\ndescription: Create or reuse a git worktree for a pull request or branch so reviews and work happen in isolation\nargument-hint: <name> [--pr <number>] [--branch <ref>]\n---\n\n# Git Worktree\n\nSet up an isolated git worktree.\n\nParse `$ARGUMENTS` to extract:\n- The worktree name (required) — used as the directory name under `.worktrees/`\n- `--pr <number>` (optional) — checkout a GitHub PR into the worktree using `gh pr checkout`\n- `--branch <ref>` (optional) — create the worktree from a specific branch or ref\n\n## Steps\n\n### 1. Determine worktree path\n\nThe worktree path is: `<repo-root>/.worktrees/<name>`\n\nFor example, if the repo root is `/home/user/src/dlt` and the name is `review-pr-3584`, the worktree path is `/home/user/src/dlt/.worktrees/review-pr-3584`.\n\nCreate the `.worktrees` directory if it does not exist:\n\n```\nmkdir -p <repo-root>/.worktrees\n```\n\n### 2. Create or reuse the worktree\n\nCheck if a worktree already exists at that path:\n\n```\ngit worktree list\n```\n\n- **If a worktree already exists** at the target path, reuse it.\n - If `--pr` was given, run `cd <worktree-path> && gh pr checkout <number>` to make sure it is up to date.\n - If `--branch` was given, run `git -C <worktree-path> checkout <ref>`.\n- **If no worktree exists** at the target path, create one:\n - If `--pr` was given, first resolve the PR's branch name: `gh pr view <number> --json headRefName -q .headRefName`\n - If `--pr` or `--branch` was given, check `git worktree list` output for any **other** worktree that already has the same branch checked out. If found:\n - Use `AskUserQuestion` to ask the user: \"Branch `<branch>` is already checked out in worktree `<existing-path>`. Use that worktree instead?\"\n - If the user says yes, use the existing worktree path (skip creation, proceed to step 3 with the existing path).\n - If the user says no, **STOP** — do not create a new worktree. Print a message explaining that the branch is already checked out elsewhere and exit.\n - If no conflict, create the worktree:\n - If `--branch` was given: `git worktree add .worktrees/<name> <ref>`\n - Otherwise: `git worktree add .worktrees/<name> --detach`\n - Callers needing a new branch check it out afterwards: `git checkout --no-track -b <new-branch>`\n - If `--pr` was given, then also: `cd <worktree-path> && gh pr checkout <number>`\n- `gh pr checkout` handles fetching, fork tracking, and branch setup automatically.\n\n### 3. Switch cwd\n\n```\ncd <worktree-path>\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, stop with an error — the cwd did not persist.\n\n### 4. Report\n\n```\nWorktree ready: <worktree-path>\n```\n",".claude/skills/dashboard/SKILL.md":"---\nname: dashboard\ndescription: Use when reading, editing, or creating files in dlt/_workspace/helpers/dashboard/ or tests/workspace/helpers/dashboard/ or tests/e2e/\nuser-invocable: false\n---\n\n# dlt Dashboard Development Guide\n\nConsult this skill whenever working on the dashboard codebase. It tells you where files live, what each module does, and where new code belongs.\n\n## Overview\n\nThe dashboard is a **marimo-based** reactive web UI for inspecting dlt pipelines, schemas, data, and run traces. It lives at `dlt/_workspace/helpers/dashboard/`.\n\n## File Layout\n\n```\ndlt/_workspace/helpers/dashboard/\n├── dlt_dashboard.py # Main marimo app (~30 reactive cells)\n├── dlt_dashboard_styles.css # CSS: colors, badges, timeline, tables\n├── runner.py # Subprocess launcher, CLI integration\n├── config.py # DashboardConfiguration @configspec dataclass\n├── const.py # Constants: port (2718), timeouts, colors, defaults\n├── typing.py # Shared TypedDicts (TTableListItem, TLoadItem, etc.)\n├── strings.py # All user-facing UI text (~350 lines)\n└── utils/\n ├── __init__.py # Package exports\n ├── pipeline.py # get_pipeline(), pipeline_details(), exception handling\n ├── queries.py # SQL execution, caching (BoundedDict), row counts, loads\n ├── schema.py # Table/column lists, schema version fetching\n ├── trace.py # Trace rendering, sanitization\n ├── visualization.py # Execution timeline HTML, load status badges\n ├── home.py # Home page / workspace home rendering\n ├── ui.py # dlt_table(), section_marker(), error_callout(), BoundedDict\n ├── formatters.py # Datetime humanization, string formatting\n └── data_quality.py # dlthub data quality integration\n```\n\n## Architecture\n\n- **Framework**: marimo — a reactive Python notebook that renders as a web app\n- **Cell pattern**: `@app.cell(hide_code=True)` decorators define reactive cells in `dlt_dashboard.py`\n- **Dependency tracking**: Cells automatically re-execute when their parameter dependencies change\n- **Routing**: URL query params (`?pipeline=X&profile=Y`) via `mo.query_params()`\n- **Caching**: `BoundedDict` (LRU, 64/32 entries) in `utils/queries.py` and `utils/schema.py`\n- **Styling**: CSS variables for dlt brand colors (`--dlt-color-lime`, `--dlt-color-aqua`, `--dlt-color-purple`, `--dlt-color-pink`)\n\n## Dashboard Sections (cells in dlt_dashboard.py)\n\n1. **Profile & Pipeline Discovery** — reads workspace profiles, lists local pipelines\n2. **Pipeline Attachment** — `dlt.attach()`, file watcher for auto-refresh\n3. **Overview** — pipeline properties, destination info, state version\n4. **Schema** — table browser with configurable column display\n5. **Browse Data** — SQL editor with cached results and query history\n6. **Trace** — execution timeline, step details, resolved config\n7. **Loads** — load history with row counts and schema versions\n8. **State** — raw JSON state viewer\n9. **Data Quality** — integration with dlthub.data_quality\n10. **Ibis Backend** — read-only Ibis connection\n\n## How the Dashboard is Launched\n\nThree CLI commands all call `runner.run_dashboard()`:\n\n* there are several commands in `dlt` and `dlthub` hosts that launch local dashboard. for example `dlt dashboard` and `dlthub local show`.\n\nEdit mode (`--edit`) ejects `dlt_dashboard.py` and CSS to the cwd for user customization.\n\n## Dependencies\n\nFrom `pyproject.toml` `workspace-deps` group: `marimo>=0.14.5`, `duckdb>=0.9`, `ibis-framework>=12.0.0`, `pyarrow>=16.0.0`.\n\n## Where to Put New Code\n\n| What you're adding | Where it goes |\n|---|---|\n| New dashboard section/cell | `dlt_dashboard.py` — add `@app.cell(hide_code=True)` |\n| New UI text/messages | `strings.py` — add to appropriate section |\n| New data fetching/queries | `utils/queries.py` or new file in `utils/` |\n| Schema-related helpers | `utils/schema.py` |\n| Pipeline metadata helpers | `utils/pipeline.py` |\n| Trace/execution helpers | `utils/trace.py` or `utils/visualization.py` |\n| UI component helpers | `utils/ui.py` |\n| Formatting/display helpers | `utils/formatters.py` |\n| New TypedDicts | `typing.py` |\n| New constants/defaults | `const.py` |\n| Configuration options | `config.py` (`DashboardConfiguration` dataclass) |\n| CSS styling | `dlt_dashboard_styles.css` |\n| Data quality features | `utils/data_quality.py` |\n\n## Coding Conventions\n\n- **All user-facing text** goes in `strings.py`, never inline in cells\n- **Cell-local variables must start with `_`** (underscore prefix). In marimo, variables that are only used within a single cell and should not be shared across cells must be prefixed with `_`. Only variables intended to be consumed by other cells should be unprefixed.\n- **UI helpers**: Use `utils/ui.py` functions (`dlt_table()`, `section()`, `error_callout()`) for consistent UI\n- **CSS classes**: `section-header`, `pipeline-execution-timeline`, `status-badge`, `dlt_table`\n- **Badge classes**: `.badge-grey`, `.badge-yellow`, `.badge-green`, `.badge-red`\n- **Data structures**: TypedDicts in `typing.py` for data passed between cells\n- **Configuration**: `@configspec` dataclass in `config.py`\n\n## Testing\n\n### Test locations\n\n```\ntests/workspace/helpers/dashboard/ # Unit and integration tests\n├── conftest.py # Fixtures: pipeline types and temp directories\n├── test_pipeline_workflow.py # Pipeline data retrieval integration\n├── test_all_cells.py # Cell execution and dependency testing\n├── test_visualization.py # Timeline and badge rendering\n├── test_ui_elements.py # UI component styling\n├── test_table_operations.py # Table listing/filtering\n├── test_trace_operations.py # Trace parsing/display\n├── test_query_operations.py # SQL query execution\n├── test_loads.py # Load history retrieval\n├── test_data_formatting.py # String formatting\n├── test_pipeline_operations.py # Pipeline metadata\n└── test_dashboard_config.py # Configuration resolution\n\ntests/e2e/ # Playwright browser tests\n```\n\n### Running tests\n\n```bash\nmake test-workspace-dashboard # unit/integration tests\nmake test-e2e-dashboard # headless chromium e2e\nmake test-e2e-dashboard-headed # visible browser e2e\nmake create-test-pipelines # create pipelines for manual testing\n```\n\n### Test fixtures\n\nTests use parametrized pipeline fixtures via `ALL_PIPELINES` and `PIPELINES_WITH_LOAD` constants in `conftest.py`. Key fixtures: `success_pipeline_duckdb`, `success_pipeline_filesystem`, `extract_exception_pipeline`, `never_ran_pipeline`, `load_exception_pipeline`.\n\n### Where to add new tests\n\n- **Utils logic**: Add to the matching `test_*.py` file (e.g., new query helper → `test_query_operations.py`)\n- **New section/cell**: Add cell execution test to `test_all_cells.py`, plus a dedicated test file if the section has significant logic\n- **Visual/rendering**: `test_visualization.py` or `test_ui_elements.py`\n- **E2E browser tests**: `tests/e2e/`\n",".claude/skills/implement-issue/SKILL.md":"---\nname: implement-issue\ndescription: Triage, plan, and implement a fix or feature for a GitHub issue end-to-end\nargument-hint: <issue-number> [-- <hints or context from maintainer>]\n---\n\n# Implement Issue\n\nParse `$ARGUMENTS` to extract:\n- Everything before the first `--` (or all of it if no `--`) is the **issue number**. Referred to as `ISSUE` below.\n- Everything after the first `--` is **maintainer hints** — domain knowledge, suspected root cause, pointers to relevant code, or constraints. If absent, there are no hints.\n\n## Steps\n\n### 1. Fetch the issue\n\n```\ngh issue view ISSUE --json title,body,comments,labels,state,assignees,author,createdAt,milestone\n```\n\nProduce a concise summary: title, reporter, component/area involved, error message, reproduction steps, expected behavior.\n\n### 2. Determine category and gather hints\n\n#### Category\n\nDerive the branch category from the issue labels:\n\n| Label contains | Category |\n|---|---|\n| `bug` | `fix` |\n| `enhancement`, `feature` | `feat` |\n| `test` | `test` |\n| `documentation` | `docs` |\n\nIf no matching label exists, ask the human for the category.\n\n#### Hints\n\nCollect implementation hints from all available sources, in priority order:\n\n1. **Maintainer hints** from `$ARGUMENTS` (after `--`)\n2. **Issue comments** — look for maintainer/contributor replies with technical guidance, root cause analysis, or implementation suggestions\n3. **Issue body** — the reporter may include debugging details, stack traces, or configuration\n\nIf the combined hints from these sources provide sufficient direction (suspected root cause, relevant code areas, constraints), proceed directly to step 3 without asking the human.\n\nIf hints are insufficient — the issue is vague, the root cause is unclear, or there are multiple possible approaches — ask the human for guidance.\n\n### 3. Research the big picture\n\nBefore jumping to a fix, verify assumptions. This step prevents wasted effort on the wrong solution.\n\n#### 3a. Understand the feature area\n\nUse Explore agents to thoroughly investigate the relevant code paths. Trace the flow from user-facing API to the point of failure. Read the code — don't guess.\n\n#### 3b. Validate the reporter's setup\n\nCheck whether the reporter is using the library correctly:\n\n- Is their configuration valid for the component they're using?\n- Are there special conventions, settings, or patterns they should be using instead?\n- Does the documentation describe the expected behavior differently?\n\n#### 3c. Check external dependencies and integrations\n\nIf the issue involves specific external systems (destinations, sources, file formats, etc.):\n\n- Read the relevant implementation in the codebase (factory, capabilities, client, etc.).\n- Understand how the component interacts with the external system.\n- If needed, consult external documentation via WebFetch/WebSearch.\n\n#### 3d. Determine: bug or user error?\n\nBased on research, conclude one of:\n\n- **Bug**: The library has a defect. Proceed to step 4.\n- **User error**: The reporter's configuration is wrong. Draft a response explaining the correct approach. Ask the human to confirm before posting.\n- **Enhancement**: The library works as designed but could be improved. Discuss scope with the human before proceeding.\n\nReport findings to the human before moving on.\n\n### 4. Set up workspace\n\nBefore planning, create the working environment so plan-mode exploration runs against a clean branch.\n\n#### 4a. Determine branch name\n\nUse the category from step 2 and the issue number:\n```\n{category}/{issue-number}-{short-description}\n```\n\nExample: `fix/3529-dedup-sort-escape`\n\nIf `ISSUE` was already a branch name rather than a number (no ticket exists), use it as given.\n\n#### 4b. Create worktree and dev environment\n\nInvoke the `/worktree-from-issue` skill with the branch name:\n```\n/worktree-from-issue {branch-name}\n```\n\nThis creates the worktree, checks out a new branch from `origin/devel`, and runs `/worktree-make-dev`.\nUse it also when there is no issue number.\n\nNote the worktree path — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 5. Draft implementation plan\n\nEnter plan mode and design a plan that **will pass the `/review-pr` skill**. The review-pr skill checks:\n\n1. **Code quality**: Style conforming to `@CLAUDE.md`, no over-engineering, proper use of existing patterns and utilities.\n2. **Test coverage**: Required scenarios identified, tests in the right location, no duplication, proper parametrization.\n3. **Documentation**: Public API changes need doc updates.\n4. **Backward compatibility**: Breaking changes flagged with deprecation path.\n5. **Dependencies**: Changes to `pyproject.toml` flagged.\n\nThe plan must include:\n\n- **Working environment**: Worktree path, branch name, and `cd` instruction (see `@.claude/rules/worktree.md`).\n- **Root cause**: Precise code location and explanation.\n- **Fix**: Minimal change following existing patterns in surrounding code.\n- **Test**: A test that **fails without the fix and passes with it**. Place tests where similar functionality is already tested. Follow `@CLAUDE.md` test guidelines (parallel safety, platform independence, proper fixtures).\n- **Test scope**: Identify the right test level — `make test-common` for extract/normalize/common work, destination-specific tests for load work. See `@CLAUDE.md` \"Testing strategy\" section.\n- **Verification commands**: Exact selective pytest commands to run (specific test functions, not entire modules).\n- **Files changed**: List with expected line counts.\n\nPresent the plan via ExitPlanMode for human approval.\n\n### 6. Implement the fix\n\nApply the changes from the approved plan. Follow these rules:\n\n- **Read before editing**: Always read the target file first. Never edit blind.\n- **Minimal changes**: Only change what the plan specifies. Don't refactor surrounding code, add comments to unchanged code, or \"improve\" things not in scope.\n- **Follow local patterns**: Match the style of surrounding code exactly — imports, naming, spacing, comment style.\n- **Type hints**: All new function signatures need full type hints with generic parametrization (`Dict[str, Any]` not `dict`).\n- **Test conventions**: `def test_*() -> None:`, pytest fixtures, `@pytest.mark.parametrize` for variants.\n\n### 7. Verify\n\nRun verification in this order. Stop and fix issues before proceeding to the next step.\n\n#### 7a. Run the new test selectively\n\nRun only the new/changed test functions using the commands from the plan. Be selective — target specific test functions, not entire modules.\n\n#### 7b. Regression proof (for external system interactions)\n\nWhen the fix involves an external system (destination, source, API) where you don't have full visibility into the behavior — a \"black box\" — verify experimentally that the test actually catches the bug:\n\n- Temporarily revert the fix\n- Run the new test on a configuration where the bug manifests\n- Confirm it **fails** with an error matching the issue report\n- Restore the fix\n\nThis is valuable when you cannot fully predict how the external system handles the input (e.g., how a database treats unquoted identifiers, how an API responds to edge-case payloads). It proves the test is meaningful and not a no-op.\n\nSkip this step when the code path is fully internal and you can reason about correctness from the code alone.\n\n#### 7c. Format and lint\n\n```\nmake format && make lint\n```\n\nIf you changed pages under `docs/website/docs`, also run `make md-fix && make md-check` in `docs/`.\n\nAll must pass cleanly. Fix any issues before reporting.\n\n### 8. Report\n\nSummarize results:\n\n```\n## Fix for #ISSUE: <title>\n\n**Branch**: {branch-name}\n**Worktree**: WORKTREE\n\n### Changes\n- `<file>`: <description> (<lines> lines)\n- ...\n\n### Verification\n- <test-function> on <config>: passed\n- <test-function> on <config> without fix: FAILED (confirms bug) [if applicable]\n- make format: pass\n- make lint: pass\n\nReady to commit.\n```\n\nWait for the human to confirm before committing.\n",".claude/skills/optimize-code/SKILL.md":"---\nname: optimize-code\ndescription: Analyze and optimize Python code performance in critical paths\nargument-hint: <file-path-or-description> [-- <critical-path-hints>]\n---\n\n# Optimize Code\n\nYou are a Python performance optimization expert. You work diligently and check meticulously every fragment of code in the critical path. You never assume - you trace actual execution paths, verify with evidence, and question defensive patterns that may have become unnecessary.\n\n**ALWAYS benchmark everything.** Use `timeit` oneliners for quick validation, write dedicated benchmarks for optimization variants, and as the final step, benchmark optimized code against the baseline (devel branch).\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **target**: file path, function name, or description of code to optimize\n- Everything after `--` is **critical path hints**: which code paths matter most, what to ignore\n\n## Phase 1: Understand the Critical Path\n\nSKIP if optimizing a single (or 1-2 ) functions where critical path is clear, otherwise INTERROGATE the user to understand the hot path before making any changes.\n\n### 1.1 Identify the target\n\nIf `$ARGUMENTS` contains a file path, read it. Otherwise, ask the user:\n- \"Which file or function should I optimize?\"\n- \"What is the entry point for the hot path?\"\n\n### 1.2 Learn the typical case\n\nAsk about the 90% case to focus optimization effort:\n- \"What data shapes are most common?\" (flat dicts, nested objects, lists)\n- \"Which branches are taken most often?\"\n- \"What can we ignore?\" (e.g., \"ignore pandas/arrow, focus on plain dicts\")\n\n### 1.3 Trace the call chain\n\nFollow the code from entry point through each function call:\n```\nentry_point() → helper_a() → helper_b() → actual_work()\n```\n\nRead each function in the chain. Identify the innermost loop where per-item work happens.\n\n### 1.4 Establish baseline benchmark\n\nBefore any optimization, create a benchmark for the current state:\n```python\nimport timeit\n# Quick validation\npython -c \"import timeit; print(timeit.timeit('target_function()', setup='...', number=1000))\"\n```\n\n## Phase 2: Analyze for Optimization Opportunities\n\nLook for these patterns in the critical path:\n\n### 2.1 Unnecessary work\n\n- **Redundant copies**: `dict(x)` or `list(x)` when reference would suffice\n- **Defensive copies**: Trace data origin - if source returns fresh object, copy is redundant\n- **Repeated computations**: Same value computed multiple times in loop\n\n### 2.2 Function call overhead\n\n- **Hot helper functions**: Consider inlining if called per-item\n- **Uncached method lookups**: `self.method` in loop vs `method = self.method` before loop\n- **Duplicate isinstance/hasattr checks**: Same check in multiple places\n\n### 2.3 Happy path shortcuts\n\n- **Early exit for common case**: Check fast path first, skip expensive work\n- **Identity vs equality**: `x is y` faster than `x == y` when applicable\n\n\n### 2.4 Consider `__slots__` for hot objects\n\nObjects created frequently in hot paths benefit from `__slots__`:\n- 30-40% less memory (no `__dict__`)\n- Faster attribute access (92% of local variable speed vs 59% for regular attributes)\n\n```python\n# Bad: regular class, dict-based attributes\nclass Row:\n def __init__(self, table, data):\n self.table = table\n self.data = data\n\n# Good: slotted class\nclass Row:\n __slots__ = ['table', 'data']\n def __init__(self, table, data):\n self.table = table\n self.data = data\n```\n\n### 2.5 Inner function overhead\n\nInner functions (closures) defined inside loops or hot functions are recreated on every call. This can be a **major** performance hit:\n\n```python\n# Bad: inner function recreated per call\ndef process(items):\n def transform(x): # created every time process() is called\n return x * 2\n return [transform(i) for i in items]\n\n# Good: module-level or method\ndef _transform(x):\n return x * 2\n\ndef process(items):\n return [_transform(i) for i in items]\n\n# Good: inline if simple\ndef process(items):\n return [x * 2 for x in items]\n```\n\nSigns to look for:\n- `def` inside another `def`\n- Lambda inside frequently-called function\n- Closures capturing loop variables\n\n\n### 2.6 Convert recursion to iteration\n\nRecursive functions have overhead per call (stack frame, argument passing) and risk stack overflow on deep structures. Convert to stack-based iteration:\n\n```python\n# Bad: recursive traversal\ndef flatten(obj, path=\"\"):\n if isinstance(obj, dict):\n for k, v in obj.items():\n yield from flatten(v, f\"{path}.{k}\") # recursive call\n else:\n yield path, obj\n\n# Good: stack-based iteration\ndef flatten(obj):\n stack = [(obj, \"\")]\n while stack:\n current, path = stack.pop()\n if isinstance(current, dict):\n for k, v in current.items():\n stack.append((v, f\"{path}.{k}\"))\n else:\n yield path, current\n```\n\nBenefits:\n- No function call overhead per level\n- Predictable memory (heap vs stack)\n- No recursion limit issues\n- Can be easier to add early-exit logic\n\nWARN USER:\n- stack method MAY change the order of processing and ie. yielded elements. ALWAYS suggest to write a test for original code.\n\n## Phase 3: Benchmark Everything\n\n### 3.1 Always benchmark caches\n\nCaches (lru_cache, manual dicts, memoization) can hurt performance if:\n- Cache miss is common (lookup overhead without benefit)\n- Cached computation is cheap (cache overhead exceeds saved work)\n- Memory pressure causes cache eviction\n\n**Benchmark caches even if they already exist** - they may have been added speculatively:\n\n```python\n# Does this cache actually help? Benchmark with and without!\n@lru_cache(maxsize=128)\ndef get_column_type(col_name):\n return self.schema.columns[col_name][\"data_type\"]\n\n# Compare:\n# 1. With cache (current)\n# 2. Without cache (direct lookup)\n# 3. Different cache size\n```\n\n### 3.2 Quick validation with timeit\n\nBefore and after each micro-optimization:\n```bash\npython -c \"import timeit; d={'a':1,'b':2}; print('copy:', timeit.timeit('dict(d)', globals={'d':d}, number=1000000))\"\npython -c \"import timeit; d={'a':1,'b':2}; print('ref:', timeit.timeit('x=d', globals={'d':d}, number=1000000))\"\n```\n\n### 3.3 Write dedicated benchmarks\n\nFor optimization variants, create benchmark scripts in `experiments/` folder:\n```python\n\"\"\"Benchmark: optimization variant comparison\"\"\"\nimport timeit\n\ndef variant_original(): ...\ndef variant_optimized(): ...\n\nprint(\"original:\", timeit.timeit(variant_original, number=10000))\nprint(\"optimized:\", timeit.timeit(variant_optimized, number=10000))\n```\n\n### 3.4 Process isolation and thermal management\n\nRun benchmarks in separate processes with pauses:\n```bash\npython experiments/bench.py devel_case\nsleep 10\npython experiments/bench.py optimized_case\n```\n\n### 3.5 Variance checking\n\nRun each benchmark 3+ times to verify stability:\n```bash\nfor i in 1 2 3; do\n echo \"=== Run $i ===\"\n python experiments/bench.py case_name\n sleep 5\ndone\n```\n\nVariance over 10-15% suggests external factors (thermal throttling, system load).\n\n### 3.6 Use realistic benchmark data\n\nUse real test data from `tests/normalize/cases/`:\n\n| File | Size | Use Case |\n|------|------|----------|\n| `ethereum.blocks.*.json` + `schemas/ethereum.schema.json` | 2MB | Deep nesting, warm/cold path normalizer |\n| `github.events.*.json` | 1.7MB | Dynamic table routing, many event types |\n| `github.issues.*.json` | 526KB | REST API, moderate nesting |\n\nLoad with `from dlt.common.json import json`. For flat rows with ISO timestamps, use `mimesis` (in dev deps) to generate synthetic DB data.\n\n### 3.7 Final benchmark: Optimized vs Devel\n\nAs the last step, compare against baseline branch:\n```bash\n# On devel branch (main repo)\ncd /path/to/main && python experiments/bench.py case_name\n\nsleep 10\n\n# On optimized branch (worktree)\ncd /path/to/worktree && python experiments/bench.py case_name\n```\n\nReport: `devel_time / optimized_time = X.XXx speedup`\n\n## Phase 4: Implement and Verify\n\n### 4.1 Make changes incrementally\n\nOne optimization at a time. Benchmark after each change.\n\n### 4.2 Verify correctness\n\n- Run existing tests: `make test-common`\n- Compare outputs before/after on sample data\n\n### 4.3 Document assumptions\n\nAdd comments explaining why optimization is safe:\n```python\n# columns dict is never mutated after get_table_columns() returns,\n# so we can store reference instead of copying\nself._current_columns = columns\n```\n\n## Phase 5: Report Results\n\nProduce a summary:\n```\n## Optimization Summary\n\n### Changes Made\n1. Replaced dict copy with identity check (buffered.py:100)\n2. Inlined count_rows_in_items for common case (buffered.py:102)\n\n### Benchmark Results\n| Case | Devel | Optimized | Speedup |\n|------|-------|-----------|---------|\n| flat_100 | 10.4s | 2.2s | 4.76x |\n| nested_20 | 4.9s | 2.0s | 2.47x |\n\n### Assumptions\n- columns dict is immutable after creation\n- 90% of items are plain dicts, not arrow/pandas\n```\n",".claude/skills/release-notes/SKILL.md":"---\nname: release-notes\ndescription: Generate release notes between two git tags with categorized PR summaries and author attribution\nargument-hint: <git-tag> [-- <previous-tag>]\n---\n\n# Generate Release Notes\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **release tag** (e.g. `1.22.0`). Referred to as `TAG` below.\n- Everything after `--` is an optional **previous tag** override. If absent, auto-detect the previous tag.\n\n## Step 1: Validate the tag\n\nCheck that `TAG` exists as a git tag:\n\n```\ngit tag -l TAG\n```\n\nIf the tag does not exist, stop with an error: \"Tag `TAG` not found. Please create the tag first and re-run.\"\n\n## Step 2: Determine the previous tag\n\nIf a previous tag was provided in `$ARGUMENTS`, validate it exists and use it as `LOWER`.\n\nOtherwise, find the previous tag automatically:\n\n```\ngit tag --sort=-v:refname\n```\n\nWalk the sorted list to find the tag immediately before `TAG` (by version order). Set that as `LOWER`.\n\nValidate `LOWER` exists. Stop with an error if no previous tag can be determined.\n\n## Step 3: Gather commits\n\nGet all commits between the two tags:\n\n```\ngit log --oneline LOWER..TAG\n```\n\nFilter out:\n- Merge commits (starting with \"Merge pull request\" or \"Merge branch\")\n- Version bump commits (starting with \"bumps to version\" or \"bumps to\")\n\nExtract PR numbers from the remaining commit messages (pattern `#NNNN`). Every PR number must appear in the final release notes — none may be silently dropped.\n\n## Step 4: Fetch PR metadata\n\nFor each PR number, fetch metadata using `gh`. Use Task agents to run in parallel batches for efficiency:\n\n```\ngh pr view NUMBER --repo dlt-hub/dlt --json number,title,labels,body,author\n```\n\nCollect for each PR:\n- **number**: PR number\n- **title**: PR title\n- **author**: `author.login`\n- **labels**: list of label names\n- **body**: PR description (first 500 chars sufficient for categorization)\n\n### Handling low-quality PR titles or descriptions\n\nIf a PR title is vague (e.g. just an issue number, single word) or the body is empty/unhelpful:\n\n1. Check if the PR references an issue (e.g. `fixes #NNN`, `closes #NNN`, `#NNN` in body). If so, fetch the issue:\n ```\n gh issue view NNN --repo dlt-hub/dlt --json title,body\n ```\n2. If still unclear, look at the PR diff to understand what changed:\n ```\n gh pr diff NUMBER --repo dlt-hub/dlt\n ```\n3. Write a clear, accurate summary based on what you find.\n\n## Step 5: Categorize PRs\n\nAssign each PR to exactly one category using these rules (evaluated in order). **Every PR must be classified** — verify the total count matches.\n\n### Breaking Changes\nCollect separately (appears at the top, not a standalone category — the PR also appears in its main category). A PR has breaking changes if:\n- It has a `breaking` label, OR\n- Its body contains \"breaking change\" (case-insensitive), OR\n- Its title contains \"breaking\"\n\nExtract the specific breaking change description from the PR body.\n\n### Highlights\nA PR is a highlight if:\n- It has a `highlight` label, OR\n- It is a significant new feature (title starts with `feat` or `(feat)`) AND the change is impactful — meaning it introduces a new user-facing capability, enables a major use case, or represents substantial effort\n- Use judgment: not every feature is a highlight. A highlight should be something users would be excited about.\n\nHighlighted PRs also appear in their main category (Core Library, etc.) but with a brief entry pointing to the Highlights section.\n\n### Docs\n- Title or body indicates documentation-only changes\n- Files changed are only in `docs/`\n- Has `documentation` label\n\n### Chores\n- CI/CD improvements, test infrastructure, repo maintenance\n- Has labels like `ci full`, `tech-debt`, `ci`\n- Title starts with `chore`, `repo:`, `ci:`, or the change only affects tests/tooling\n\n### Core Library\nEverything else — features, bug fixes, enhancements to the library itself.\n\n## Step 6: Identify new contributors\n\nFor each release author, find their first ever merged PR:\n\n```\ngh search prs --repo dlt-hub/dlt --author AUTHOR --merged --sort created --order asc --limit 1 --json number,createdAt\n```\n\nA contributor is **new** if their first merged PR number falls within the `LOWER..TAG` commit range.\n\n## Step 7: Format the release notes\n\nWrite the notes in this exact structure:\n\n```markdown\n## dlt TAG Release Notes\n\n## Breaking Changes\n\n(Only if there are breaking changes. Numbered list.)\n\n1. **Short description** (#NUMBER @author) — Explanation of what broke and how it affects users.\n\n## Highlights\n\n- **Feature name** (#NUMBER @author) — Description of the feature and why it matters.\n\n## Core Library\n\n- **Feature/fix name** (#NUMBER @author) — Brief description.\n- **Fix: short description** (#NUMBER @author)\n\n## Docs\n\n- Description (#NUMBER @author)\n\n## Chores\n\n- **Title** (#NUMBER @author) — Brief description if non-obvious.\n\n## New Contributors\n\n- @author made their first contribution in #NUMBER\n```\n\n## Formatting rules:\n- PR references use `#NUMBER` format (GitHub auto-links them)\n- Authors use `@login` format (GitHub auto-links them)\n- Features and significant fixes get bold titles with em-dash descriptions\n- Minor fixes can be single-line bold entries without descriptions\n- Omit empty sections entirely (including New Contributors if none)\n- Within Core Library, group features first, then fixes\n\n## Step 8: Present for approval\n\nDisplay the formatted release notes to the user for review. Include a summary:\n- Total PRs analyzed\n- Breakdown by category (with counts)\n- Number of unique contributors (and how many are new)\n- List any PRs where you had to infer the description from issue or diff\n\nAsk the user to approve or request changes before writing.\n\n## Step 9: Write to file\n\nOnly after user approval, write the release notes to `/tmp/release-notes-TAG.md`.\n\nReport the file path to the user.\n",".claude/skills/review-issues-severity/SKILL.md":"---\nname: review-issues-severity\ndescription: Find and prioritize open GitHub issues by severity, community impact, and maintainer abandonment\nargument-hint: [-- <optional filters or focus areas>]\n---\n\n# Review Issues by Severity\n\nParse `$ARGUMENTS` for optional filters:\n- Label filters (e.g., `question`, `bug`)\n- Focus areas (e.g., \"merge disposition\", \"Arrow\", \"ClickHouse\")\n- If absent, perform a broad review across all open issues.\n\n## Steps\n\n### 1. Gather candidate issues\n\nRun the following GitHub API queries **in parallel** to surface issues matching different severity signals:\n\n#### a) Long discussions (high comment count)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&per_page=100&sort=comments&direction=desc&page=1' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### b) Issues with specific labels (if filters provided)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&labels=LABEL&per_page=100&sort=comments&direction=desc' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### c) Recently stale issues (updated 2+ months ago, with maintainer comments)\nLook for issues where `updated_at` is old relative to the current date but had prior maintainer engagement.\n\n### 2. Deep-dive top candidates\n\nFor the top 10-15 most promising candidates, fetch full details using a subagent or parallel `gh issue view` calls:\n\n```bash\ngh issue view -R dlt-hub/dlt NUMBER --json title,body,comments,labels,createdAt,updatedAt,author,assignees\n```\n\nFor each issue, extract:\n- **Participants:** who commented, their association (MEMBER = maintainer, NONE/CONTRIBUTOR = community)\n- **Timeline:** when maintainers last engaged, how long since last response\n- **Problem description:** what breaks and under what conditions\n- **Reproduction quality:** is there a clean repro script?\n- **Current status:** fixed? workaround? stalled? abandoned?\n- **Production severity:** does this cause data loss, crashes, or silent corruption?\n\n### 3. Classify each issue\n\nApply these criteria to each issue:\n\n| Signal | What to look for |\n|--------|-----------------|\n| **Serious problem** | Data corruption, silent data loss, hard crashes, cascading failures, security issues |\n| **Long discussion** | 5+ comments, especially with multiple community reporters hitting the same issue |\n| **Maintainer abandoned** | Maintainer commented but last maintainer response is 2+ months old, no linked PR, reporter follow-ups unanswered |\n| **Question label** | Issue has `question` label — these often represent real bugs initially miscategorized |\n| **Community effort** | Reporter provided detailed repro scripts, root-cause analysis, or proposed fixes that went unacknowledged |\n\n### 4. Prioritize\n\nAssign priority tiers:\n\n- **P0 — Critical:** Silent data corruption, cascading failures, confirmed root cause with no fix. Production systems at risk.\n- **P1 — High:** Hard crashes in common deployment patterns, blocking features for significant user segments, confirmed bugs with stalled fixes.\n- **P2 — Medium:** Broken features in specific environments, intermittent failures, missing warnings that lead users astray.\n- **P3 — Low:** Documentation gaps, environment-specific issues with known workarounds, feature requests with community PRs pending review.\n\nWithin each tier, rank by:\n1. Seriousness of the problem in production (data loss > crash > degraded performance > inconvenience)\n2. Negative community impact if not solved (effort wasted by reporters, users hitting the issue independently)\n3. Amount of effort already invested by reporters (repro scripts, root-cause analysis, proposed PRs)\n\n### 5. Output format\n\nPresent results as:\n\n#### Per-issue analysis (grouped by priority tier)\nFor each issue:\n- **Issue link, title, comment count, last activity date**\n- **Problem:** 2-3 sentence description of what breaks\n- **Why this priority:** key severity signal\n- **Abandonment signal:** when maintainer last engaged, what was left unresolved\n- **Community effort:** what the reporter invested (repro scripts, analysis, PRs)\n\n#### Summary matrix\nA table with columns: Priority | Issue # | Type | Production Impact | Community Effort | Maintainer Status\n\n#### Cross-cutting patterns\nNote any clusters of related issues (e.g., multiple issues sharing a root cause) or systemic patterns in maintainer response.\n",".claude/skills/review-pr/SKILL.md":"---\nname: review-pr\ndescription: Analyze a GitHub pull request including diff, comments, related issues, and local code context\nargument-hint: <pr-url-or-number> [-- <optional instructions>]\n---\n\n# Review Pull Request\n\nParse `$ARGUMENTS` to extract the PR identifier and optional reviewer instructions:\n- Everything before the first `--` (or all of it, if there is no `--`) is the **PR identifier** (URL or number). Referred to as `PR_ID` below.\n- Everything after the first `--` is **reviewer instructions** — additional focus areas or context from the caller. If absent, there are no special instructions.\n\nAnalyze the pull request: PR_ID\n\n## Steps\n\nRun the following steps. Gather data in parallel where possible.\n\n### 1. Set up worktree\n\nResolve the PR number:\n\n```\ngh pr view PR_ID --json number\n```\n\nUse the number to form the worktree name `review-pr-<number>`.\n\nInvoke the `/create-worktree review-pr-<number> --pr <number>` skill to create an isolated worktree for this PR. Note the worktree path from the skill output — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 2. Fetch PR metadata and diff\n\nUse `gh` to collect all PR information in parallel:\n\n- `gh pr view PR_ID --json title,body,author,state,baseRefName,headRefName,number,url,comments,reviews,labels,milestone`\n- `gh pr diff PR_ID`\n\n### 3. Check dependency changes in `pyproject.toml`\n\nIf the PR diff (from step 2) touches `pyproject.toml`, run the automated dependency checker.\n\nFirst, fetch the latest `devel` from origin so the comparison is against the current state:\n\n```\ngit fetch origin devel\n```\n\nThen run the script — always compare against `origin/devel` regardless of the PR's base branch:\n\n```\npython tools/check_dependency_changes.py origin/devel\n```\n\nThe script finds the merge-base, simulates a three-way merge (like GitHub), and reports:\n- **Main dependencies** (`[project.dependencies]`) — displayed as WARNING (these affect every user)\n- **Optional dependencies** (`[project.optional-dependencies]`) — displayed as WARNING (these affect users of specific extras)\n- **Dev/group dependencies** (`[dependency-groups]`) — listed for awareness (no impact on end users)\n- **Merge conflicts** — if `pyproject.toml` would conflict, the script warns and falls back to showing the PR's intended changes\n\nInclude the script output verbatim in the review. If the script exits with code 1 (changes detected), flag dependency changes in the review verdict. If it exits with code 2 (conflicts), flag that `pyproject.toml` needs conflict resolution.\n\nIf `pyproject.toml` is not in the diff, skip this step.\n\n### 4. Check CI lint and mypy status\n\nUse `gh pr checks PR_ID` to retrieve the CI check results. Look for checks matching `lint on all python versions / lint*` — these run mypy, ruff, flake8, bandit, and docstring checks across Python 3.9–3.13 (defined in `.github/workflows/lint.yml`).\n\n- Report the status of each lint check (pass/fail/pending).\n- If any lint check has failed, fetch its logs with `gh run view <run-id> --log-failed` to identify the specific errors (mypy type errors, ruff violations, etc.).\n- Include lint/mypy status in the final review verdict. A failing lint check should be flagged as a required fix.\n\n### 5. Analyze related issues\n\nExtract any issue references from the PR body (e.g. `fixes #1234`, `closes #1234`, `#1234`). For each referenced issue, fetch it with:\n\n- `gh issue view <number> --json title,body,author,state,comments,labels`\n\nUnderstand the original problem being solved.\n\n### 6. Analyze the code\n\nUsing absolute paths under `WORKTREE`:\n\n- Read the changed files in full to understand context beyond the diff.\n- Look at callers/callees of modified functions.\n- Check if similar patterns exist elsewhere that should also be updated.\n- Review the coding style and code patterns to conform with @CLAUDE.md\n- If **reviewer instructions** were provided, pay special attention to the areas or concerns they describe during this analysis.\n- Mention only crucial finding to the user in the initial overview. Avoid code snippets except if critical bugs were found\n\n### 6a. Pay attention to special cases\n\nSome areas of the codebase have rules that are easy to miss in a generic code read. The references below are intentionally not loaded by default — read the linked file ONLY when the PR's diff matches the trigger, and apply its checklist before producing the review.\n\n- **Identifier normalization** — read [identifier-normalization.md](identifier-normalization.md) when the PR's diff includes any of: `dlt/destinations/impl/**`, `dlt/dataset/**`, `dlt/destinations/sql_client*`, code that builds SQL/relations/lineage, or schema-generation paths involving sqlglot or ibis. Covers when raw-string table/column identifiers must pass through the schema's naming normalizer, and how to distinguish first-time normalization from re-normalization.\n\n### 7. Review test coverage\n\n#### 7a. Define the testing scope\n\nBased on the PR changes, identify what needs to be tested:\n\n- What are the distinct behaviors introduced or modified?\n- What are the edge cases and error paths?\n- What are the integration boundaries (e.g., does the change affect how components interact)?\n\nProduce a checklist of required test scenarios before evaluating existing tests.\n\n#### 7b. Evaluate existing tests against the checklist\n\n- Compare the PR's tests against the required scenarios checklist. Flag any gaps.\n- Flag any tests that verify things outside the PR's scope or test obvious/trivial behavior that adds no value.\n\n#### 7c. Verify tests are in the right modules\n\n- Search the test directory for where similar functionality is already tested (grep for related function names, class names, or feature keywords).\n- New tests should live alongside related existing tests, not in new files unless the PR introduces an entirely new module.\n- Flag tests placed in the wrong location.\n\n#### 7d. Check for duplication and over-testing\n\n- Look for tests that cover nearly identical cases and could be merged or parametrized with `@pytest.mark.parametrize`.\n- Look for repeated setup/assertion logic across tests that should be extracted into a helper or fixture.\n- If multiple tests differ only in a single variable (e.g., `True`/`False`, different type values), they must be parametrized into one test function.\n- Flag any copy-paste test code. Minimizing test code maintenance burden is a priority.\n- Flag local path assertions that are not Windows compatible.\n\n#### 7e. Bug Fixes Should Have Regression Tests\nIf the PR fixes a bug:\n- Is there a test that would have caught the bug before the fix?\n- Does the test verify the specific scenario from the bug report?\n\n### 8. Check documentation coverage\n\nIf the PR modifies public API (new parameters, changed behavior, new configuration options, new functions/classes exposed to users):\n\n- Search `WORKTREE/docs/website/docs/` for existing documentation covering the topic of the PR.\n- If the topic is already documented and the PR changes its behavior or adds new options, verify that the PR also updates the relevant doc pages.\n- If documentation updates are missing, flag this in the review as a required change.\n- If the topic is not documented at all, this is not a blocker — don't require docs for previously undocumented internals.\n\n### 9. Produce the review\n\nWrite a structured analysis covering:\n\n- **Summary**: What the PR does and why, referencing the linked issue.\n- **Changes**: Breakdown of changes should be concise, avoid code samples\n- **Dependency Changes**: If `pyproject.toml` was modified, include the warnings and listings from step 3. Main and optional dependency changes must appear as yellow WARNING blocks. Omit this section if no dependency changes were found.\n- **CI Lint & Mypy**: Status of lint checks across Python versions. If any failed, list the specific errors.\n- **Observations & Potential Issues**: Bugs, edge cases, missing error handling, performance concerns, memory issues, security issues, backward compatibility.\n- **Breaking Changes and Public API**: Follow the rules on breaking changes and Public API and report issues.\n- **Test Coverage**: Required scenarios checklist with pass/fail status, misplaced tests, duplication issues, parametrization opportunities.\n- **Documentation**: Whether doc updates are needed and if they were provided.\n- **Reviewer Instructions**: If reviewer instructions were provided, include a dedicated section addressing each point raised. If no instructions were provided, omit this section.\n- **Verdict**: Overall assessment.\n",".claude/skills/review-pr/identifier-normalization.md":"---\nname: Identifier Normalization Check\ndescription: Ensures all table names, column names, and other identifiers pass through the schema naming convention normalizer — the single most common PR review finding in this repo.\n---\n\n# Identifier Normalization Check\n\n## Context\n\nDifferent destinations have incompatible naming rules (e.g., Athena/S3 Tables forbids leading underscores, Weaviate capitalizes identifiers, BigQuery has length limits). All identifiers must go through the schema's naming normalizer — raw string constants are never acceptable for table or column names in runtime code.\n\n## When to Check\n\nThose rules apply to INTERNAL `dlt` code that is intended to work on ANY destination. In that case raw identifiers (string literals) MUST be normalized.\n\n\nYou must known when you normalize RAW identifier that represents a table or column name vs. when you RE-NORMALIZE already normalized identifier - take it from `Schema` instance, `Dataset` or `Relation` and which comes in a Python variable.\n\nNote: nested tables `user__comments` and nested columns `issue__stat_count` are PATHS containing many ALREADY NORMALIZED identifiers.\n\nWhen re-normalizing (and WHEN IN DOUBT)\nUse \n- `normalize_tables_path`\n- `normalize_path`\n\nWhen normalizing a RAW identifier:\n\nUse \n- `normalize_table_identifier`\n- `normalize_identifier`\n\n## What to Check\n\n### 1. Raw String Identifiers\n\nLook for hardcoded table or column name strings used in SQL queries, schema lookups, or API calls without normalization:\n\n```python\n# BAD — raw string identifier\ntable_name = \"my_table\"\nsql = f\"SELECT * FROM {table_name}\"\n\n# GOOD — normalized through schema naming\ntable_name = schema.naming.normalize_table_identifier(\"my_table\")\nsql = f\"SELECT * FROM {sql_client.make_qualified_table_name(table_name)}\"\n```\n\nCode in `dlt/dataset/` that builds queries or accesses tables must normalize identifiers:\n\n```python\n# BAD\ndataset[\"_dlt_loads\"]\n\n# GOOD\ndataset[schema.naming.normalize_table_identifier(\"_dlt_loads\")]\n```\n\nIn `dlt/destinations/impl/*/`:\n\n- Table names passed to SQL must use `sql_client.make_qualified_table_name()`\n- Column names in SQL must be escaped/normalized\n- Internal dlt tables (like `_dlt_loads`, `_dlt_version`, `_dlt_pipeline_state`) must still be normalized\n\n### 2. Re-normalizing identifiers\nUse this path when in doubt!\nWhen re-normalizing identifiers (ie. naming convention changed) or when identifier is a variable you MUST use:\n* `normalize_tables_path` - for table identifiers\n* `normalize_path` - for anything else ie. columns\n\nExample: a pyarrow table in transfer from destination 1 (now a source) to destion 2. Columns are normalized per\ndestination 1 and we re-normalize for destination 2:\n\n```python\ndef get_normalized_arrow_fields_mapping(schema: pyarrow.Schema, naming: NamingConvention) -> StrStr:\n \"\"\"Normalizes schema field names and returns mapping from original to normalized name. Raises on name collisions\"\"\"\n # use normalize_path to be compatible with how regular columns are normalized in dlt.Schema\n norm_f = naming.normalize_path\n```",".claude/skills/review-pr/new-destination-checklist.md":"# New destination review checklist\n\nUse this checklist when reviewing a PR that adds a new dlt destination. These items are in addition to the standard review steps.\n\n## 1. Shared destination test integration\n\n- Check `tests/load/utils.py` `destinations_configs()` — is the new destination included in the right config groups (`default_vector_configs`, `default_sql_configs`, `read_only_sqlclient_configs`, etc.)?\n- Check `tests/load/test_read_interfaces.py` — does the destination need special handling (chunk size skips at `_chunk_size()`, view-only table listing exclusions in `test_ibis_dataset_access`)?\n- Check `tests/load/pipeline/test_restore_state.py` — state sync tests run via the config groups above, verify the destination participates.\n- Check `tests/utils.py` — is the destination in `IMPLEMENTED_DESTINATIONS` and `NON_SQL_DESTINATIONS` (if applicable)?\n\n## 2. Capabilities configuration\n\n- Verify `_raw_capabilities()` in the factory: loader formats, merge strategies, replace strategies, type mapper, nested types, decimal precision, timestamp precision, recommended file size.\n- Compare to the blueprint destination (ducklake, lancedb, filesystem) — are any caps missing or misconfigured?\n- Check `parquet_format` settings if the destination writes parquet with nested types.\n\n## 3. `WithLocalFiles` integration\n\n- If the destination stores data locally (files, embedded databases), the top-level `DestinationClient*Configuration` **must** inherit `WithLocalFiles` — otherwise the pipeline cannot bind `pipeline_name`, `pipeline_working_dir`, `local_dir` into the config.\n- If storage config is nested (e.g. `storage: FilesystemConfiguration`), `on_resolved()` must call `self.storage.attach_from(self)` followed by resolve or `normalize_bucket_url()`. Reference: ducklake pattern in `ducklake/configuration.py`.\n- **Test it**: verify default relative paths resolve to `local_dir`, not cwd. Verify `pipeline_name` and `pipeline_working_dir` propagate to nested storage config.\n\n## 4. SQL client — `WithTableScanners`\n\n- If the destination provides a read-only SQL interface via DuckDB views (like lance, lancedb, filesystem), it should extend `WithTableScanners` (`dlt/destinations/impl/duckdb/sql_client.py`), not `DuckDbSqlClient` directly.\n- `WithTableScanners` provides: in-memory DuckDB management, schema creation, sqlglot-based lazy view loading, `create_views_for_all_tables()` for ibis offload.\n- The subclass implements three abstract methods: `create_view()`, `can_create_view()`, `should_replace_view()`.\n\n## 5. Ibis integration\n\n- Check `dlt/helpers/ibis.py` — is there a branch for the new destination's config class?\n- For `WithTableScanners`-based destinations: use `sql_client.create_views_for_all_tables()` (not manual table iteration).\n- Add the destination to the view-only exclusion list in `test_ibis_dataset_access` if it uses DuckDB views (won't see tables from other schemas).\n- The ibis branch must come before any parent config class check to avoid wrong dispatch via `issubclass`.\n\n## 6. Import rules\n\n- Never import optional packages directly — use `dlt.common.libs.*` wrappers (`pyarrow`, `pandas`, `numpy`, etc.).\n- Private API imports from third-party libs (underscore-prefixed functions) are fragile — flag them.\n",".claude/skills/review-vocabulary/SKILL.md":"---\nname: review-vocabulary\ndescription: Review and rewrite the prose a branch adds — docstrings, comments, user-facing messages, docs — against dlt's fixed vocabulary and Simplified Technical English. Invoke as /review-vocabulary.\nargument-hint: \"[<branch-or-base-ref>] [-- <extra focus or scope notes>]\"\ndisable-model-invocation: true\n---\n\n# Review Vocabulary\n\nRewrite the prose a branch adds so dlt says one thing one way. Two inputs govern every\ndecision: the **fixed vocabulary** in `references/vocabulary/`, indexed below, and the\n**Simplified Technical English** rules in `references/simple-english.md`.\n\n**Never invoke this on your own.** A maintainer runs it with `/review-vocabulary`. It rewrites\ntext across dozens of files and changes CI test ids, so it is not a background cleanup.\n\nParse `$ARGUMENTS`:\n- Everything before the first `--` is the **base ref** to diff against. Defaults to\n `$(git merge-base origin/devel HEAD)`.\n- Everything after the first `--` is extra focus from the maintainer.\n\n## Read first\n\n1. `references/simple-english.md` — the 53 rules. **Cite only rule numbers that exist in that\n file.** The numbering is unintuitive and models invent it. Rule 3.1 is about verb forms, not\n sentence length.\n2. `references/simple-english-use-cases.md` — needed for the error-message shape.\n3. The vocabulary groups the diff touches, picked from the index below. They override the general\n rules where they disagree.\n\n## Scope\n\nDiff with `git diff <base>` — **not** `git diff <base>...HEAD`, which misses uncommitted work\nthat will ship in the same PR.\n\nIn scope, when the branch **added or changed** the line:\n\n| Kind | Where |\n|---|---|\n| Docstrings | `dlt/`, `tests/` |\n| `#` comments | `dlt/`, `tests/` |\n| User-facing messages | `raise X(\"...\")`, `logger.*`, `warnings.warn`, helpers that build a message |\n| Test function names | `tests/` — renames only under the narrow gate below |\n| Documentation | `docs/website/docs/**` |\n| Snippet files a page pulls from | `*snippets.py` — comments as docs, code as a test (see below) |\n\nWhen the maintainer asks for a **whole-file** docs review, pre-existing prose in those files is in\nscope too. Say which mode you are in; the default is added-or-changed lines only.\n\nOut of scope:\n\n- **Pre-existing prose**, in the default mode. Leave it even when it breaks every rule. List what\n you skipped and why.\n- Code, identifiers, signatures, type annotations — except snippet code, which gets a separate\n review against the repo's test rules.\n- SQL keywords and anything already in backticks, CLI commands, file paths, config keys.\n- `pytest.param(id=...)` strings, fixture names, parameter names.\n- Assertions on exact generated SQL.\n- Prose moved verbatim from another file — check the base ref before claiming it is new.\n\n## Classification\n\nGet this right first; every other rule depends on it.\n\n| Text | Mode | Limit | Shape |\n|---|---|---|---|\n| Docstrings, `#` comments | descriptive | 25 words (Rule 6.3) | no imperative, one new fact per sentence (Rule 6.1) |\n| Exception strings, warnings, log lines | **procedural** | **20 words (Rule 5.1)** | what happened (simple past), the cause, then the fix as an imperative |\n| Docs — how-to, steps, install | procedural | 20 words | imperative, condition before command (Rule 5.4) |\n| Docs — concept, reference, architecture | descriptive | 25 words | no imperative |\n\n**Messages keep their remedies.** Many errors end in an instruction — \"Materialize the dataset…\",\n\"Set a permanent `pipeline_salt`…\". Improve the wording: make it imperative, condition first,\nunder 20 words, one instruction per sentence. **Never delete a remedy. Never invent one** where\nthe message has none — a remedy needs facts the string does not interpolate.\n\n**Count the assembled sentence, not the fragment.** Some messages are built by a helper that wraps\na short reason in a frame — `_no_data_location(reason)` in\n`dlt/common/destination/client.py` is the current example. Read the helper first, then judge each\nreason as the clause it becomes inside the finished sentence. A four-word reason can push the\nassembled sentence past 20 words.\n\n**Docs carry their own untouchables:** fenced code blocks, `<!--@@@DLT_SNIPPET ...-->` markers,\nfront-matter, link targets and anchor slugs.\n\n**Headings are anchors.** Renaming one breaks every link to it. Before you rename, grep the whole\ndocs tree for the old slug and update each hit in the same change. When a rename would need an edit\nin a file another agent owns, hand it back rather than reaching across.\n\n### Snippet files are BOTH tests and documentation\n\nA page that uses `<!--@@@DLT_SNIPPET ./x_snippets.py::name-->` pulls real code from a real test.\n`docs/pyproject.toml` collects `*snippets.py` with `*_snippet` functions, so **docs CI executes\nthem**. They therefore get two reviews at once, against two different rule sets:\n\n| Part of the file | Treat as | Rules |\n|---|---|---|\n| the code | a test | `.claude/rules/testing.md`, `coding-style.md`, `imports.md` |\n| the `#` comments | user-facing docs | this vocabulary + STE, classified per comment |\n\nConsequences worth knowing:\n\n- **Comments inside a snippet render to the reader**, so a stale or wrong one is a doc bug, not a\n code-hygiene nit. Judge them as prose. A comment that labels a step is INSTRUCTION; a comment\n that explains behavior is REFERENCE.\n- **The repo comment rule still applies to the code.** Default to no comment. But a snippet is\n teaching material, so a comment that would be redundant in library code can earn its place here.\n Say which of the two you applied when they conflict.\n- **Never change snippet code to satisfy a prose rule.** Changing code changes what CI runs. If the\n code is wrong, report it as a code finding with its own justification.\n- **A fenced code block is NOT a snippet.** It is untested. When a page mixes the two, its fenced\n examples can drift from the API while its snippets cannot. Flag the untested ones as unverified\n rather than assuming they work — and consider recommending they become snippets.\n\n## The fixed vocabulary\n\nOne word, one meaning, one part of speech (Rules 1.11, 9.4). Apply the tables before any other\nrewriting.\n\nThe vocabulary is organised in **groups**, one per area of dlt. A group is self-contained: its\nincluded terms, its excluded terms, and the rulings that are easy to get wrong in that area. Each\ngroup is one file in `references/vocabulary/`. Groups grow independently — a review only needs the\ngroups its diff touches, and adding an area means adding a group, not editing the others.\n\nGroups defined so far:\n\n| Group | Covers | Read when the diff touches |\n|---|---|---|\n| [G1 — Data access and locations](references/vocabulary/g1-data-access-and-locations.md) | destinations, datasets, physical locations, join compatibility | `dlt/dataset/`, `dlt/common/destination/`, `dlt/destinations/`, the dataset-access docs |\n| [G2 — Attach and foreign datasets](references/vocabulary/g2-attach-and-foreign-datasets.md) | cross-destination joins, attach info, catalog aliases | `TAttachInfo`, `ATTACH`, foreign datasets, catalog aliases |\n| [G3 — Transformations and materialization](references/vocabulary/g3-transformations-and-materialization.md) | relations, transformations, model jobs, eager and lazy paths | relations, model jobs, `materialize` in `dlt/dataset/`, `dlt/extract/` or `dlt/load/`, the transformation docs |\n| [G4 — Identifiers and SQL generation](references/vocabulary/g4-identifiers-and-sql-generation.md) | naming conventions, case-folding, query binding | `dlt/common/normalizers/naming/`, `dlt/common/schema/`, sql clients, lineage and query generation |\n| [G5 — Configuration and credentials](references/vocabulary/g5-configuration-and-credentials.md) | configs, credentials, secrets | `dlt/common/configuration/`, credentials classes, `config.toml`, `secrets.toml` |\n| [G6 — Merge strategies and merge conditions](references/vocabulary/g6-merge-strategies-and-merge-conditions.md) | merge strategies, deletes, retirement, source filter, destination scope, placeholders | `dlt/destinations/sql_jobs.py`, a destination's merge job, `merge_key`, `scd2`, `cdc`, `merge-loading.md` |\n| [G7 — Jobs, triggers and the deployment manifest](references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md) | jobs, job runs, launchers, triggers, selectors | `dlt/_workspace/deployment/` outside `agent/`, `dlt/hub/run.py`, `__deployment__.py` files, `dlthub local run` |\n| [G8 — Agents, loops and prompts](references/vocabulary/g8-agents-loops-and-prompts.md) | agent definitions, agent jobs, agent runs, loops, system prompts, user turns, traces | `dlt/_workspace/deployment/agent/`, `launchers/agent.py`, `AGENT.md` files, `run.agent` |\n| [G9 — Workspace access and tools](references/vocabulary/g9-workspace-access-and-tools.md) | the `access` declaration, axes, verbs, tools, feature groups | `access` blocks, `TWorkspaceAccess`, `dlt/_workspace/mcp/`, the tools a loop wires |\n\nRead every group the diff touches before you audit, and give each subagent only the groups its\nfiles map to. G1 to G5 are short: when the mapping is unclear, read them all.\n\nTwo rules apply across every group:\n\n- **`dlt` is the sentence subject in messages.** That is how an error gets active voice with a\n named agent (Rule 3.6): \"dlt cannot join…\", \"dlt cannot determine…\". House style.\n- **A banned word is banned for one meaning, not always.** Every group names its legal\n exceptions. Check them before \"fixing\" a hit.\n\n### Legal technical nouns — never replace, any group (Rules 1.5, 1.8)\n\nattach, attach alias, attach info, attach statement, catalog, config, data location, dataset,\ndestination, destination scope, duckdb, iceberg, materialization, merge condition, model job, partition,\npipeline, placeholder, relation, scanner, snapshot (`cdc` only), source filter, vended.\n\naccess, access axis, agent, agent definition, agent definition reference, agent file, agent job,\nagent output, agent run, agent trace, default trigger, feature group, instructions, job, job definition, job ref, job result,\njob run, launcher, loop, placeholder, runner, runtime, selector, skill, system prompt, tool, toolkit,\ntrigger, turn, user turn, verb.\n\n## Rules this codebase breaks most\n\nThe repo comment rule overrides every row. Default to no comment. Keep at most one short line for\na non-obvious WHY. Prefer a rewrite that shrinks the line count. When a comment only restates the\ncode, delete it.\n\n## Test function names\n\nA name is an identifier. Rename it for one of two reasons only:\n\n- it contradicts the fixed vocabulary\n- it carries a noun chain of more than three words (Rule 2.1)\n\nNever rename for style. Apply two checks first:\n\n- **Does the name mirror the API it tests?** Then keep it. `test_attach_info_built_once_per_relation`\n tracks `Relation._attach_infos()`, and that link beats vocabulary purity.\n- **Every rename changes a CI test id.** Flag each one on its own. Afterwards, grep the repo for\n the old name.\n\n## Workflow\n\n### 1. Audit\n\nPropose. Do not edit.\n\nSplit the diff across subagents by area. **File sets must not overlap** — two agents writing one\nfile corrupt each other.\n\nGive each agent its base ref, its file list, the group files its files map to, the two cross-group\nrules and the classification table. Tell it to read `references/simple-english.md` itself, so rule\nnumbers come from the file.\n\nCollect five groups. Give every finding a **file:line, rule number, current text, rewrite**:\n\n1. Vocabulary violations\n2. Structural violations\n3. Identifier renames\n4. Content corrections — false statements. Run the code before you report one.\n5. Test assertions that break — see the trap below.\n\nWrite the proposal to a file. A few hundred findings do not fit in a chat reply. Report the counts\nand the open decisions.\n\n**Stop. Wait for approval.** When the vocabulary does not settle a decision, ask.\n\n### 2. Apply, in two phases\n\nDocstrings and message strings share files, so they cannot run together.\n\n- **Phase A:** docstrings, comments, renames, assertion fixes, content corrections.\n- **Phase B:** message strings.\n\nTell every Phase B agent that the line numbers moved. It must find each string by content. Pin any\nsubstring a test asserts.\n\nEvery applying agent obeys four rules:\n\n- Apply only your own findings, only in your own files.\n- **Never touch code.** Not a signature, not a return type, not a call site. Agents break this\n rule, and a functionally-equivalent change passes every test. Verify it afterwards.\n- Preserve every f-string placeholder: same names, same count, same order.\n- Keep lines under 100 chars. Black does not reflow a string or a comment.\n\n### 3. Verify\n\nRun every step. Each one caught a real defect.\n\n**a. Prose-only check.** Compare the AST with docstrings stripped, against the **branch tip**.\nComments never reach the AST, so any difference means code or a string changed. Explain every file\nthat appears. Against the merge-base the whole PR appears and the signal is lost.\n\n```python\nimport ast\ndef strip(t):\n for n in ast.walk(t):\n if isinstance(n, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):\n n.body = [x for x in n.body if not (isinstance(x, ast.Expr)\n and isinstance(x.value, ast.Constant) and isinstance(x.value.value, str))]\n return t\n# compare ast.dump(strip(ast.parse(old))) with ast.dump(strip(ast.parse(new)))\n```\n\n**b. The four-step self-check** from `references/simple-english.md`, over the added lines. Add a\nsweep for the excluded vocabulary.\n\n**c. `make format`.** Black reports no change when the agents did their job.\n\n**d. `make lint`.** For pages under `docs/website/docs`, also `make md-fix && make md-check` in\n`docs/`. A rewrite that lengthens a table cell past 35 words fails CI.\n\n**e. Run the tests.** This step is not optional.\n\n## The trap: a message rewrite breaks test assertions\n\nTests match on error text. A grep for the phrases you changed misses some. A parametrized test\nhides its assertion far from the message.\n\nOne pass found them in two rounds. A grep found five. The test run then surfaced eight more. One of\nthose was `match=\"cannot be determined\"`. The frame sentence had become \"dlt cannot determine the\ndata location\".\n\n- Grep `tests/` for `match=`, `in str(exc`, and `in str(reject`.\n- Check the **negative** assertions too. `assert \"can join\" not in ...` still passes against\n \"cannot join\", because no space follows `can`.\n- Run the suites anyway. The grep is a head start, not the check.\n\n## Adding a new term or a new group\n\nNever just record a term. Take the eight steps in `references/vocabulary/adding-terms.md`.\n\nAmend the vocabulary mid-review when you must. Then tell every running agent what inverted. They\naudited under the old table.\n\n## Deliverable\n\n- The proposal file, with counts and your open decisions.\n- After approval: the edits, the verification output, and what you did **not** apply.\n- Never commit. The maintainer commits.\n",".claude/skills/review-vocabulary/references/simple-english-checklist.md":"# Verification checklist\n\nRun this pass on every draft before you deliver it. The checks are ordered from mechanical to judgment.\n\n## Mechanical checks (searchable)\n\nSearch the draft for each pattern. Every hit outside code blocks and quoted text is a violation.\n\n| Search for | Violation | Fix |\n|---|---|---|\n| `'ll`, `'re`, `'ve`, `n't`, `it's` | Contraction (Rule 4.2) | Expand it. |\n| `has been`, `have been`, `had been` | Present/past perfect (Rule 3.4) | Simple past or simple present. |\n| `has` / `have` + past participle | Present perfect (Rule 3.4) | Simple past. |\n| `should`, `would`, `may`, `might`, `could` | Unapproved modal (Rule 3.2) | See the modal ladder in SKILL.md. |\n| `is being`, `are being`, `was being` | Progressive passive (Rules 3.4, 3.5) | Active, simple tense. |\n| `, making`, `, allowing`, `, enabling`, `, ensuring` | \"-ing\" clause as verb (Rule 3.5) | New sentence with a real subject. |\n| `;` | Semicolon (Rule 8.1) | Two sentences. |\n| `e.g.`, `i.e.`, `etc.` | Latin abbreviation (GR-6) | \"for example\", \"that is\", name the items. |\n| `simply`, `easily`, `seamlessly`, `robust` | Filler (no fact) | Delete. |\n| ` if `, ` when ` (mid-sentence) | Trailing condition (Rule 5.4) | Move the condition to the start of the sentence, add a comma. |\n\n## Countable checks\n\n1. **Sentence length.** Count words in each sentence. Procedural limit: 20. Descriptive limit: 25. Notes: 25.\n Backticked commands, numbers with units, and identifiers count as one word each (Rule 8.6).\n2. **Paragraph size.** Maximum six sentences per paragraph (Rule 6.6).\n3. **Multi-word nouns.** Any noun chain over three words → break it with prepositions (Rule 2.1).\n4. **Instructions per sentence.** One, unless the actions are simultaneous (Rule 5.2).\n\n## Judgment checks\n\n5. **Classification.** Is each passage cleanly procedural or descriptive? Procedures in imperative, descriptions never in imperative.\n6. **Voice.** Any passive sentence: is the agent truly unknown, and is the passage descriptive? Otherwise make it active (Rule 3.6).\n7. **Condition placement.** Every \"if/when\" stands before its command, with a comma (Rule 5.4).\n8. **Synonym rotation.** One term per concept across the whole document (Rules 1.11, 9.4). Scan for check/verify/confirm, config/settings, run/execute.\n9. **Warnings.** Command or condition first, risk second (Rules 7.2, 7.3).\n10. **Completeness.** Articles present, \"that\" present after \"make sure\", no telegraph style (Rule 4.2).\n11. **Untouchables intact.** Code, identifiers, quoted errors, and proper nouns are unchanged.\n\n## When reporting violations (check mode)\n\nFor each violation give: the rule number, the offending text, and a compliant rewrite. Cite only rule numbers that appear in rules.md.\nEnd the report with this statement when the user asked for STE compliance: \"No tool can guarantee ASD-STE100 compliance. Final approval rests with the writer. The official standard is a free download at asd-ste100.org.\"\n",".claude/skills/review-vocabulary/references/simple-english-use-cases.md":"# Use cases beyond documentation\n\nSTE was built for aircraft maintenance manuals. The same properties — one meaning per word, short sentences, condition-first commands — transfer to any text where misreading has a cost. By the end of Issue 8, 64% of registered STE users were outside aerospace and defense.\n\nEach case below names the mode and the adaptations.\n\n## Error messages and CLI output\n\nMode: procedural. This is the highest-value target: an error message is a 2 a.m. instruction to a stressed reader.\n\nPattern: state what happened (past simple), state the cause if known, give the command or condition to fix it.\n\n> **Before:** Oops! Something went wrong while attempting to establish a connection. Please ensure your credentials are properly configured and try again.\n> **After:** Connection to the database failed. The password for user `app` was not correct. Set `DB_PASSWORD` and connect again.\n\n## Runbooks and standard operating procedures\n\nMode: strict-leaning procedural. This is STE's home turf — an on-call runbook is a maintenance manual.\n\n- Every step imperative, one instruction per step, conditions first.\n- Warnings before the step, command first, risk second.\n- 20-word limit enforced hard: an operator under pager stress reads each sentence once.\n\n## Incident reports and postmortems\n\nMode: descriptive. Simple past only — a timeline in present perfect (\"we have identified...\") hides when things happened.\n\n> **Before:** We have identified an issue that may have impacted some users' ability to access the service.\n> **After:** Between 14:02 and 14:31 UTC, 12% of requests failed. A deploy at 14:00 removed the cache warmup step.\n\nSTE bans hedges (\"may have impacted\") — the report states what is known and says \"unknown\" for the rest. This reads more honest because it is.\n\n## Commit messages and PR descriptions\n\nMode: descriptive body, imperative subject. Convention already matches STE: imperative subject line, plain past facts in the body. Apply the substitution table and the 25-word limit to the body. Delete \"this PR aims to\".\n\n## API changelogs and release notes\n\nMode: descriptive. One entry, one change, one sentence where possible. \"Breaking:\" entries follow the warning pattern — command first: \"Update your calls to `v2/users`. The `name` field split into `first_name` and `last_name`.\"\n\n## Instructions for AI agents (prompts, AGENTS.md, skills)\n\nMode: procedural. A system prompt is a procedure executed by a reader with no ability to ask questions — the exact reader STE was designed for.\n\n- One instruction per sentence keeps rules independently quotable and hard to half-follow.\n- One word, one meaning prevents the model from treating \"check\", \"verify\", and \"validate\" as three different operations.\n- Condition-first (\"If the build fails, stop\") beats trailing conditions, which models drop.\n- No \"should\" — a model reads \"should\" as optional. Write \"must\" or delete the rule.\n\n## Support macros and status-page updates\n\nMode: descriptive, 25-word limit. Non-native readers are the majority of many user bases. No \"we sincerely apologize for any inconvenience this may have caused\" — \"The API was down for 18 minutes. Uploads made during this time were saved and will process today.\"\n\n## Translation and localization prep\n\nMode: strict. STE's original purpose was making English readable for non-native maintenance crews, and it doubles as pre-editing for machine translation. One meaning per word plus complete grammar (articles, \"that\") removes most translation ambiguity. If your docs get localized, STE cuts the error rate and the cost.\n\n## UI copy and empty states\n\nMode: procedural, hard length limits. Buttons and labels are technical names (exempt). Body copy follows the rules: \"No projects yet. Create a project to start.\" Nothing else survives at this length anyway.\n\n## Where STE does not fit\n\nMarketing pages, launch posts, blog voice, brand writing. STE deletes persuasion on purpose. Write those in your own voice — then use STE for the docs the landing page links to.\n",".claude/skills/review-vocabulary/references/simple-english.md":"---\nname: simple-english\nversion: 1.0.0\ndescription: |\n Write or rewrite technical text with the rules of ASD-STE100 Simplified\n Technical English so it is clear, unambiguous, and free of AI slop. Use for\n documentation, READMEs, runbooks, procedures, error messages, release notes,\n incident reports, and API guides. Also use when the user says \"STE\",\n \"Simplified Technical English\", \"ASD-STE100\", \"de-slop\", \"make this\n readable\", \"write for non-native readers\", or asks for docs that translate\n well. Enforces the standard's 53 rules: 20/25-word sentence limits, one word\n one meaning, simple tenses, active voice, condition before command.\nlicense: MIT\ncompatibility: claude-code cursor codex gemini-cli opencode\nmetadata:\n standard: ASD-STE100 Issue 9 (2025-01-15)\n---\n\n# Simple English: Write Like an Aerospace Manual\n\nWrite technical text with the rules of ASD-STE100 Simplified Technical English. STE is the controlled language that aerospace and defense manufacturers use for maintenance documentation. The rules exist so that a tired reader who is not a native English speaker cannot misread an instruction. They remove the usual signs of AI-generated text as a side effect: long sentences, synonym rotation, hedges, filler, and decorative clauses.\n\nWrite for that tired reader. Each sentence must survive one read.\n\n## Your Task\n\nWhen asked to write or rewrite technical text:\n\n1. **Select the mode** (pragmatic or strict, below).\n2. **Classify each passage** as procedural or descriptive. Every other rule depends on this.\n3. **Fix your vocabulary before drafting.** Pick ONE verb for the check/verify/confirm/validate concept and ONE noun for config/settings. Use no other word for these concepts in the whole document.\n4. **Apply the rules** from the catalog below.\n5. **Run the self-check** before you deliver. This step is not optional.\n6. **Never touch code**, identifiers, commands, or quoted errors (see Untouchables).\n\nWhen asked to CHECK text instead of writing it, report each violation as: rule number, the offending text, a compliant rewrite. Cite only rule numbers that exist in this file. Do not cite rule numbers from memory: the numbering is unintuitive and models invent it (tested — an agent without this file cited \"Rule 3.1: short sentences\"; the real Rule 3.1 is about verb forms).\n\n## Two Modes\n\n| Mode | When | What you apply |\n|---|---|---|\n| **Pragmatic** (default) | Docs, READMEs, error messages — the user wants clear text | All structural rules. Domain words stay (\"idempotent\", \"webhook\"). |\n| **Strict** | The user names STE, ASD-STE100, or compliance | Structural rules + full vocabulary discipline, and tell the user that full compliance needs the official dictionary (free at asd-ste100.org). |\n\n## Step 1: Classify the Text\n\n| | Procedural (instructions) | Descriptive (explanations) |\n|---|---|---|\n| Purpose | Tell the reader what to do | Explain what a thing is or does |\n| Verb form | Imperative: \"Install the pump.\" | Simple present/past/future |\n| Sentence limit | **20 words** (Rule 5.1) | **25 words** (Rule 6.3) |\n| Unit rule | One instruction per sentence (5.2) | One topic per paragraph (6.5), max six sentences per paragraph (6.6) |\n\nDo not mix the two in one passage. A \"Getting started\" section is procedural. An \"Architecture\" section is descriptive. A note inside a procedure is descriptive (25-word limit, no imperative).\n\n## THE RULE CATALOG\n\n53 rules in 9 sections, paraphrased from ASD-STE100 Issue 9 with software examples. The official wording is in the free standard at asd-ste100.org.\n\n### Section 1 — Words (Rules 1.1-1.14)\n\n| Rule | Instruction |\n|---|---|\n| 1.1 | Use only approved words, technical nouns, or technical verbs. |\n| 1.2 | Use an approved word only as its listed part of speech. |\n| 1.3 | Use an approved word only with its approved meaning. |\n| 1.4 | Use only the approved forms of verbs and adjectives. |\n| 1.5 | You can use domain words as technical nouns (\"webhook\", \"commit\", \"endpoint\"). |\n| 1.6 | Use an unapproved word only when it is a technical noun or part of one. |\n| 1.7 | Do not use technical nouns as verbs. |\n| 1.8 | Use the technical nouns of your project or industry. |\n| 1.9 | When you pick a technical noun, pick a short and clear one. |\n| 1.10 | No regional, slang, or jargon words as technical nouns. |\n| 1.11 | One item, one name. Do not call it \"config\" here and \"settings\" there. |\n| 1.12 | You can use domain verbs as technical verbs (\"deploy\", \"compile\", \"merge\"). |\n| 1.13 | Do not use technical verbs as nouns. |\n| 1.14 | Use American English spelling. |\n\nIn pragmatic mode, rules 1.5, 1.8, and 1.12 do the heavy lifting: your domain vocabulary is legal. The ones agents break are 1.7, 1.11, and 1.13.\n\n**Before:** You can webhook the event, then do a deploy.\n**After:** Send the event to the webhook. Then deploy the service.\n\n### Section 2 — Multi-word nouns (Rules 2.1-2.2)\n\n| Rule | Instruction |\n|---|---|\n| 2.1 | Write multi-word nouns of three words or fewer. |\n| 2.2 | When a technical noun needs more than three words, write it in full once, then give a short form or hyphenate the units. |\n\nBreak long noun chains with prepositions (of, on, in, for):\n\n**Before:** the connection pool timeout configuration value\n**After:** the timeout value for the connection pool\n\n### Section 3 — Verbs (Rules 3.1-3.7)\n\n| Rule | Instruction |\n|---|---|\n| 3.1 | Use only the verb forms that the dictionary gives. |\n| 3.2 | Use only: infinitive, imperative, simple present, simple past, simple future, past participle as adjective. |\n| 3.3 | Use the past participle only as an adjective (\"the cached response\"). |\n| 3.4 | No auxiliary verbs for complex constructions. No present perfect, no \"is to be installed\". |\n| 3.5 | Use an \"-ing\" form only as a technical noun or inside one (\"logging\", \"the mounting bracket\") — never as a verb. |\n| 3.6 | Active voice. In descriptive text, passive is legal only when the agent is unknown. |\n| 3.7 | Describe an action with a verb, not a noun (\"compress the file\", not \"perform compression of the file\"). |\n\n**Approved modals: can, will, must. Banned: should, would, may, might, could.**\nThe standard rejects \"could\" even for possibility: write \"an explosion can occur\", never \"could occur\". For \"should\": a requirement becomes \"must\"; a suggestion is stated as fact or deleted. This matters double for agent instructions — models read \"should\" as optional.\n\n**Before:** The migration has completed and the table is being rebuilt.\n**After:** The migration is complete. The database rebuilds the table.\n\n**Before:** The flag can be set in the config file, making restarts unnecessary.\n**After:** You can set the flag in the config file. Then a restart is not necessary.\n\n**Before:** The temperature must be adjusted.\n**After:** Adjust the temperature.\n\n### Section 4 — Sentences (Rules 4.1-4.5)\n\n| Rule | Instruction |\n|---|---|\n| 4.1 | Write short and clear sentences. |\n| 4.2 | Do not omit words or use contractions to shorten sentences. Keep articles, keep \"that\". |\n| 4.3 | Use a vertical list for complex text. |\n| 4.4 | Use connecting words between sentences on related topics (\"Then\", \"As a result\"). |\n| 4.5 | Put an article (the, a, an) or a demonstrative adjective (this, these) before nouns where applicable. |\n\nRule 4.2 is the anti-terseness rule. STE is short sentences with complete grammar, not telegraph style:\n\n**Wrong shortening:** Ensure file exists before running.\n**STE:** Make sure that the file exists before you run the command.\n\n### Section 5 — Procedural writing (Rules 5.1-5.5)\n\n| Rule | Instruction |\n|---|---|\n| 5.1 | Maximum 20 words per sentence. Warnings and cautions included. |\n| 5.2 | One instruction per sentence, unless two actions happen at the same time. |\n| 5.3 | Write instructions in the imperative: \"Run the migration.\" |\n| 5.4 | Put a required condition before the command, divided by a comma: \"If the build fails, read the log.\" |\n| 5.5 | Notes give information, never instructions. Notes get the 25-word limit. |\n\n**Before:** You'll want to grab the API key from the dashboard before configuring the client, which you can do under Settings.\n**After:** Get the API key from the dashboard, under Settings. Then configure the client with this key.\n\n### Section 6 — Descriptive writing (Rules 6.1-6.6)\n\n| Rule | Instruction |\n|---|---|\n| 6.1 | Give information gradually: one new fact per sentence. |\n| 6.2 | Use key words and phrases to give the text a logical structure. |\n| 6.3 | Maximum 25 words per sentence. |\n| 6.4 | Group related information in paragraphs. |\n| 6.5 | One topic per paragraph. |\n| 6.6 | Maximum six sentences per paragraph. |\n\nNo imperative in descriptive text. Descriptions explain; procedures instruct.\n\n### Section 7 — Safety instructions (Rules 7.1-7.3)\n\n| Rule | Instruction |\n|---|---|\n| 7.1 | Use a word that shows the risk level (\"WARNING\" = injury, \"CAUTION\" = damage). |\n| 7.2 | Start with a clear command or condition. |\n| 7.3 | Then give the risk or the possible result. |\n\nNever bury the instruction after the explanation. The pattern transfers directly to destructive CLI flags, irreversible migrations, and dangerous API options.\n\n**Before:** Note that data loss may occur in some circumstances if the destructive flag happens to be enabled when running against production.\n**After:** CAUTION: Do not use the `--force` flag against production. The flag deletes rows that do not match the source.\n\n### Section 8 — Punctuation and word count (Rules 8.1-8.7)\n\n| Rule | Instruction |\n|---|---|\n| 8.1 | All standard punctuation is legal except the semicolon. Write two sentences instead. |\n| 8.2 | Use hyphens to connect words that act as one unit. |\n| 8.3 | Parentheses are legal for references, item numbers, abbreviations, plural forms, explanations, alternatives. |\n| 8.4 | In a vertical list, the lead-in colon ends a sentence for word count. |\n| 8.5 | Text inside parentheses counts as one word. |\n| 8.6 | Count as one word each: numbers, numbers with units, abbreviations, alphanumeric identifiers, quoted text, titles, labels, proper nouns. |\n| 8.7 | A hyphenated word counts as one word. |\n\nRule 8.6 matters for software text: `sqlpipe run --config sqlpipe.yaml` in backticks is quoted text and counts as one word. Long identifiers do not blow your sentence budget.\n\n### Section 9 — Writing practices (Rules 9.1-9.4, GR-1 to GR-8)\n\n| Rule | Instruction |\n|---|---|\n| 9.1 | When a word-for-word replacement does not work, restructure the sentence. |\n| 9.2 | Use each approved word correctly: approved meaning, approved part of speech. |\n| 9.3 | Do not build phrasal verbs (\"go down\" → \"decrease\", \"set up\" → \"install\" or \"configure\"). |\n| 9.4 | Keep one consistent style and terminology through the whole document. |\n\nGeneral recommendations GR-1 to GR-8: keep the conjunction \"that\", be careful with \"with\", give pronouns clear referents, prefer \"this + noun\" over bare \"this\", avoid false friends, avoid Latin abbreviations, use inclusive language, and use the possessive apostrophe form only when you are sure it is correct (GR-8: if unsure, do not use it — non-native readers find it hard).\n\nGR-6 for software docs: \"e.g.\" → \"for example\", \"i.e.\" → \"that is\", and delete \"etc.\" — name the items or write \"and more\".\n\n## VOCABULARY DISCIPLINE\n\nThe official dictionary (~900 approved words, ~1,200 banned words with alternatives) is copyrighted by ASD and is not reproduced here. Its mechanics apply without it: **one word, one meaning, one part of speech.**\n\nKnown part-of-speech rulings, useful as patterns:\n\n| Word | Ruling |\n|---|---|\n| test, check, work | Noun only. \"Do a test\", not \"test the pump\". \"Check that X\" becomes \"make sure that X\". |\n| oil | Noun only as used in STE examples. For the verb, the dictionary gives \"lubricate\". |\n| help | Verb only. For the noun, the dictionary gives \"aid\": \"with the aid of\". |\n| fall | \"To move down by gravity\" only, never \"decrease\". |\n| follow | \"To come after\" only, never \"obey\". Write \"obey the instructions\". |\n| above, below | Physical positions only. For limits write \"more than\", \"less than\". |\n\n### The modal ladder\n\n| You wrote | STE writes |\n|---|---|\n| should (requirement) | must |\n| should (recommendation) | Delete it, or state it as fact: \"X is better because Y.\" |\n| may / might / could (possibility) | can |\n| may (permission) | can |\n| would (hypothetical) | Restructure: \"If X occurs, Y occurs.\" |\n\n### Slop-to-simple substitutions\n\nThis table is ours, not the ASD dictionary. It maps the words AI-generated docs overuse to plain replacements. If the word carries no fact, delete it instead of replacing it.\n\n| Slop | Write instead |\n|---|---|\n| leverage, utilize | use |\n| in order to | to |\n| prior to | before |\n| ensure | make sure that |\n| it is worth noting that | (delete) |\n| it's important to, crucially | (delete — state the fact) |\n| simply, just, easily, seamlessly, effortlessly | (delete) |\n| robust, powerful, comprehensive, performant | (delete, or give the measurable property) |\n| functionality | function, feature |\n| enables you to, allows you to | you can |\n| is designed to, aims to | (delete — say what it does) |\n| facilitate | help, make possible |\n| dive into, delve into | read, examine |\n| when it comes to | for |\n| in the event that | if |\n| due to the fact that | because |\n| as needed, as necessary | (state the condition) |\n| and/or | Pick one, or write \"X, or Y, or both\" |\n| e.g. / i.e. / etc. | for example / that is / (name the items) |\n| gracefully handles | (say what it does: \"retries three times, then stops\") |\n| out of the box | by default |\n| under the hood | internally |\n| blazingly fast, state-of-the-art | fast (give the number) / (delete) |\n| streamline | make simpler, make faster |\n| plethora, myriad | many |\n| addresses the issue, tackles | corrects the fault, removes the error |\n\n### Consistency pass\n\nCollapse these common rotations to one term each (Rules 1.11, 9.4):\n\n- check / verify / confirm / validate / ensure → pick one\n- config / configuration / settings / options → pick one\n- delete / remove / drop / destroy → one per meaning, kept consistent\n- error / issue / problem / failure → \"error\" for errors, \"failure\" for failed operations\n- run / execute / invoke / launch → pick one\n- show / display / render / present → pick one\n\n## Untouchables\n\nThese are technical names (Rules 1.5, 8.6). Leave them exact, even when they break vocabulary rules:\n\n- Code blocks, inline code, identifiers, CLI commands, flags, file paths\n- Quoted error messages and log lines\n- Product names, API endpoint names, config keys\n- Numbers with units — each counts as one word in the sentence limit\n\n## Beyond Documentation\n\nSame rules, different targets. Full adaptations in `references/use-cases.md`:\n\n- **Error messages**: state what happened (simple past), the cause if known, then the fix as an imperative. No \"Oops\", no \"Please ensure\", no apology filler.\n- **Runbooks**: STE's home turf. Imperative steps, conditions first, warnings before the step.\n- **Incident reports**: simple past only. \"We have identified an issue that may have impacted\" becomes \"Between 14:02 and 14:31 UTC, 12% of requests failed.\"\n- **Release notes**: breaking changes follow the warning pattern — command first, risk second.\n- **Agent instructions (prompts, AGENTS.md)**: a system prompt is a procedure for a reader that cannot ask questions. One instruction per sentence, no \"should\", condition first.\n- **Translation prep**: STE's original job. One meaning per word plus complete grammar removes most translation ambiguity.\n\n## Self-Check Before You Deliver\n\nThis step is not optional. Run these four checks on your draft:\n\n1. Count words in your three longest sentences. Over the 20/25 limit → split them.\n2. Search your draft for: `'ll`, `'re`, `'s` (contraction), `has been`, `have been`, `should`, `-ing` verbs after a comma, semicolons.\n3. Search for every `if` and `when`. Each one stands at the START of its sentence, before the command. \"Increase the timeout if the network is slow\" → \"If the network is slow, increase the timeout.\"\n4. Search for the verbs you did NOT pick in Your Task step 3 (the check/verify/confirm set). Replace every hit with your chosen verb.\n\nFix what you find, then deliver. For a full audit, run `references/checklist.md`.\n\n## Full Example\n\n**Before (real unedited AI output):**\n\n> **Connection timeouts.** If sqlpipe hangs or fails with `dial tcp: i/o timeout`, check that the host running sqlpipe can reach the Postgres port (usually 5432) — this is often a security group or firewall rule blocking the connection. If you're connecting to a managed database (RDS, Cloud SQL, etc.), confirm the instance allows connections from sqlpipe's IP. You can also try increasing `source.connect_timeout_seconds` in your config, since a slow network path can trip the default timeout even when the connection eventually succeeds.\n\n**After (classified procedural, verb = \"make sure\", conditions first, one instruction per sentence):**\n\n> **Connection timeouts.** sqlpipe stops with `dial tcp: i/o timeout` when it cannot reach the Postgres port (5432 by default).\n>\n> 1. Make sure that the host that runs sqlpipe can reach the Postgres port. A firewall or security group usually blocks it.\n> 2. If the database is managed (RDS, Cloud SQL), make sure that the instance accepts connections from the IP of sqlpipe.\n> 3. If the network is slow, increase `source.connect_timeout_seconds` in the configuration.\n\nWhat changed: 40-word sentences split under 20; \"you're\" expanded; \"check/confirm\" collapsed to \"make sure that\"; every condition moved before its command; \"etc.\" removed; code and error strings untouched.\n\n## Limits\n\nSTE is for technical facts and instructions. Do not apply it to marketing copy, blog voice, or brand writing — it deletes persuasion by design. When a user asks for STE on marketing text, say so and offer it for the docs instead.\n\nThis skill is an unofficial aid. It is not affiliated with or endorsed by ASD or STEMG, and no tool can guarantee STE compliance. ASD-STE100 is a registered trademark of ASD. The official standard is a free download at asd-ste100.org.\n\n## References\n\n- `references/checklist.md` — full verification pass with searchable patterns, for check mode and final audits\n- `references/use-cases.md` — long-form adaptations: error messages, runbooks, incident reports, commits, UI copy, i18n\n",".claude/skills/review-vocabulary/references/vocabulary/adding-terms.md":"# Adding a new term or a new group\n\nNever just record a term. Take these eight steps.\n\n1. **Pick the group.** No group fits? Add one. Give it the next `G<n>`, a file in this folder named\n after its heading, an index row in `SKILL.md`, and the three parts: included, excluded, rulings.\n A group without excluded terms is not finished.\n2. **Research the usage.** Grep `dlt/`, `tests/` and `docs/` for the word and every synonym. Count\n the hits. Read enough to find the meanings in play. One word often covers two concepts.\n3. **Derive the banned set.** A term is useless without one. For each synonym and inflection,\n decide: banned, or legal with another meaning? Both halves go in the group.\n4. **Check the upstream interface.** The vocabulary must not contradict a public dlt method or\n type name. `access` won because `needs_attach` already said \"accesses its data\". `attach info`\n won because the type is `TAttachInfo`.\n5. **Name the false positives.** `descriptor` is banned for `TAttachInfo` and correct for the\n Python descriptor protocol. Put the exception in the rulings, or the next run \"fixes\" it.\n6. **Fix the part of speech.** Say noun or verb, and ban the other use. `access` is a verb, so\n \"data access\" is a violation.\n7. **Test the ban before you write it.** Grep for the word you intend to exclude. Compliant prose\n already uses it? Then the ban is too wide. Narrow it to the meaning you mean. `inaccessible`\n and `foreign-folded` passed this check and stay legal. `casefold` and `attach instructions`\n failed it and are banned.\n8. **Update the group file.** Add rows to both tables. Add a ruling when the term has a legal\n exception. State what becomes newly compliant and what becomes newly non-compliant. A term that\n flips direction turns compliant prose into findings.\n",".claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md":"# G1 — Data access and locations\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| getting to data (verb) | **access** |\n| the adjective | **accessible**, **inaccessible** |\n| the negative, as a verb | **cannot access** |\n| where the data physically sits | **data location** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| reach, reaches, reachable, in reach of, out of reach, get to | one word for one concept — `access` |\n| physical location, physical destination, physical dataset (in prose) | say **data location**; the method `data_location()` keeps its name |\n\n**Rulings**\n\n- **`access` is a verb for the act, and a noun for what a job may touch.** Keep the verb for the\n act of reading data: \"the engine accesses the data\", not \"data access is one-way\". The noun is\n legal only in the sense G9 defines — the `access` declaration, and the grant that answers it.\n dlt's docs already write \"grant access\", \"read access\" and \"denied access\" 40 times over; that\n usage was always compliant and stays.\n- **`reach` is not always `access`.** \"`SET SESSION` would not reach the cloned sessions\" means\n *propagate to*, not *read data from*. A literal swap changes the meaning — restructure\n (Rule 9.1).\n",".claude/skills/review-vocabulary/references/vocabulary/g2-attach-and-foreign-datasets.md":"# G2 — Attach and foreign datasets\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the `TAttachInfo` object | **attach info** |\n| the `TAttachStatement` object | **attach statement** |\n| the SQL keyword | `` `ATTACH` `` in backticks |\n| the action in prose | **attach** (lowercase, a verb) |\n| the catalog a foreign dataset lands under | **attach alias** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| descriptor (for `TAttachInfo`) | `attach info` matches the type and the method `_attach_infos()` |\n| bare `ATTACH` as a prose noun, `ATTACHed`, \"attaches\" as a plural noun | backtick the keyword, or use the verb |\n| attach instructions | one name — **attach statements** |\n\n**Rulings**\n\n- **`descriptor` is legal for the Python descriptor protocol.** `dlt/common/utils.py` describes a\n real Python descriptor. The ban covers naming the `TAttachInfo` object only.\n- **`attach info` and `attach statement` are different things.** One is the whole descriptor for a\n foreign dataset; the other is a single SQL statement inside it. Do not collapse them.\n",".claude/skills/review-vocabulary/references/vocabulary/g3-transformations-and-materialization.md":"# G3 — Transformations and materialization\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the deferred path | **lazy materialization** |\n| the load job that runs it | **model job** |\n| the immediate path | **eager materialization** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| model extraction | not a thing — it is a **model job** |\n| executed here | say **eager materialization** |\n\n**Rulings**\n\n- **`lazy` and `eager` are legal only for materialization.** dlt has both. Using `lazily` to mean\n *on first use* (memoization) is a second meaning for one word — write \"on the first read\".\n- **A model job is the artifact; lazy materialization is the path.** Use the one you mean.\n",".claude/skills/review-vocabulary/references/vocabulary/g4-identifiers-and-sql-generation.md":"# G4 — Identifiers and SQL generation\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| identifier case handling | **case-fold**, **case-folds**, **case-folding** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| casefold, post-fold (in prose) | one spelling — **case-fold**; the identifier `casefold_identifier` keeps its name |\n\n**Rulings**\n\n- **Hyphenated compounds on the `fold` root are legal** — \"foreign-folded output column\" reads\n correctly and is in use. The ban is on the bare spelling `casefold` in prose.\n",".claude/skills/review-vocabulary/references/vocabulary/g5-configuration-and-credentials.md":"# G5 — Configuration and credentials\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| a resolved settings object | **config** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| configuration (for the object) | `config`; keep \"configuration error\" when it names `ConfigurationValueError` |\n",".claude/skills/review-vocabulary/references/vocabulary/g6-merge-strategies-and-merge-conditions.md":"# G6 — Merge strategies and merge conditions\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| removing destination records (`delete-insert`, `upsert`, `cdc`, hard deletes) | **delete** |\n| closing the validity of an `scd2` record | **retire** (never \"delete\" for `scd2`) |\n| loaded records that `source_filter` does not select | **discard**, **discarded** |\n| writing loaded records to the destination | **insert** (a new record), **update** (an existing one) |\n| a record in the destination that the loaded data does not contain | **absent** |\n| the part of a table a load replaces | **partition** (conceptual, as `merge_key` docs already use it) |\n| the loaded data under `cdc` | **snapshot** |\n| the upstream system whose records `cdc` mirrors | **source system** |\n| the loaded records that the source filter selects (all loaded records without a filter) | **merge source** (a noun) |\n| the `source_filter` option in prose | **source filter** |\n| the `destination_scope` option in prose | **destination scope** |\n| `source_filter`, `destination_scope` together | **merge conditions** |\n| `{table}`, `{staging_table}` | **placeholders**; \"expand\" for substitution |\n| data in docs and messages | **records**; **rows** only in code docstrings and comments, and in \"nested rows\" |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| region, window, bare \"scope\" (for the records a merge can delete or retire) | say **destination scope** for the option, **partition** for the concept |\n| input filter, output filter, merge filter(s), merge scope | old names — say **source filter**, **destination scope**, **merge conditions** |\n| filtered staging (data, rows, records), filtered load, filtered loaded data, bare \"the source\" (for the merge source) | say **merge source** |\n| bare \"the source\", upstream, origin (for the system `cdc` mirrors) | say **source system**; `@dlt.source` keeps its name |\n| owns, is authoritative for (a load and its records) | say \"the records that this load replaces\" |\n| clear, remove, wipe (for a merge delete) | one verb — **delete** |\n| drop, exclude, filter out, land (for loaded records) | **discard** for records the source filter does not select, **insert** for records that are written |\n| supersede, take precedence (between merge conditions and `merge_key`) | say \"dlt ignores `merge_key`\" |\n| strand, migrate (a record that changes partition) | say \"the record moves to another partition\" |\n| knob | say **setting** or name the hint |\n\n**Rulings**\n\n- **The bans are for the merge meaning only.** `region` stays legal for cloud regions and data columns,\n `window` for browser windows and the attribution window in `lag.md`, `scope` for OAuth and config scopes,\n `drop` for dropping tables and the `dlt pipeline drop` command, `remove` outside merge deletes.\n- **\"source\" in \"source filter\" is the merge source**, the loaded data, as in `MERGE ... USING source`. It is\n not `@dlt.source`. Never shorten \"source filter\" to \"the source\", and never write \"destination scope\" as\n \"the scope\".\n- **The merge source is what the merge works on.** Keys, `merge_key` partitions and absent records come from the\n merge source, not from all loaded records. The destination scope does not depend on it.\n- **`loaded data` and `loaded records` stay legal** for everything loaded, before the source filter. Write\n \"merge source\" only where a source filter can apply. Without one, the two are the same.\n- **\"MERGE source\" in SQL context is legal.** `MERGE ... USING` names its input the source; uppercase `MERGE`\n marks that meaning (Databricks comments).\n- **Only strategies that delete or retire absent records have a destination scope**: `delete-insert`,\n `scd2`, `cdc`. `upsert` also deletes (with `hard_delete`), so \"strategies that delete\" is not the rule.\n- **`delete` vs `retire`.** `scd2` never deletes on absence, so \"`scd2` deletes absent records\" is a content\n error, not only a vocabulary one.\n- **`snapshot` is legal only for `cdc` loaded data.** Iceberg and Delta snapshots are a different technical noun.\n Do not write \"snapshot\" for Iceberg `upsert` limits: write \"absent from the loaded data\".\n- **Placeholders differ per destination.** Do not write \"`{table}` expands to the destination table\" without\n naming the destination type: on Delta it expands to `target`.\n\nNewly non-compliant: \"region\", \"window\", bare \"scope\", \"owns\", \"clear\", \"supersede\" in merge prose, and the\nold option names (\"input filter\", \"output filter\", \"merge filters\"), and \"filtered staging\" or bare \"the source\"\nfor the merge source. Newly compliant: \"destination scope\" for the `destination_scope` option, \"merge source\".\n",".claude/skills/review-vocabulary/references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md":"# G7 — Jobs, triggers and the deployment manifest\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the unit a workspace deploys and runs | **job** |\n| one execution of it | **job run** |\n| its identity | **job ref** |\n| its entry in the manifest | **job definition** |\n| the dlt module that starts a job | **launcher** |\n| the machine the platform gives the job | **runner** |\n| the dltHub platform itself | **runtime** |\n| the string that starts a job | **trigger** |\n| the pattern that expands into triggers | **selector** |\n| the trigger a manual run stands in for | **default trigger** |\n| the file that describes every job | **deployment manifest** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| task, workload, script (for a dlt job) | one name — **job**; the runtime's `Script` model keeps its own name |\n| job execution, invocation (for one run) | **job run**, the noun the manifest and the beacon use |\n| primary trigger | **default trigger**, the name of the manifest field |\n| manifest (unqualified, for `AGENT.md`) | **deployment manifest** is the only manifest; see G8 |\n\n**Rulings**\n\n- **`run` is a verb; `job run` is the noun.** \"dlt runs the job\" and \"the job run failed\". The\n command `dlthub local run` is a technical name (Rule 8.6) and stays.\n- **launcher, runner and runtime are three things.** The launcher is dlt code. The runner is a\n machine. The runtime is the platform. Never swap them, and never write \"the runtime launches\".\n- **`task` is legal in two places only:** the work an agent must do (G8), and an Airflow task when\n the sentence says \"Airflow task\". It is never a dlt job.\n",".claude/skills/review-vocabulary/references/vocabulary/g8-agents-loops-and-prompts.md":"# G8 — Agents, loops and prompts\n\nThree nouns carry the feature, and they nest: a definition is written once, a job adds how it\noperates, a run is one execution.\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| what an `AGENT.md` or a decorated function declares: system prompt, inputs, output, tools, skills, rules, access | **agent definition** (`TAgentSpec` in code, `TAgentDefinition` for the block the manifest carries; pydantic-ai's `AgentSpec`) |\n| the file it is read from | **`AGENT.md`**, and **agent file** for the path |\n| the name a definition is referred to by | **agent definition reference**: `<toolkit>:<agent>`, a workspace path, or `<module>:<function>` for a function with no `AGENT.md` behind it |\n| the definition plus the runtime settings that say how it operates: model, limits, trigger, instructions, loop, identity | **agent job** (`run.agent(...)`; what pydantic-ai and claude-agent-sdk call an Agent, and what the web UI lists as one) |\n| one execution of an agent job: inputs in, output and trace out, settings overridable for that run | **agent run** (`Agent.run()` in the frameworks) |\n| what the model returns: `status`, `summary` and the declared fields | **agent output** |\n| the envelope the launcher delivers | **job result** (G7 owns it; an agent job's is `TAgentJobResult`) |\n| the binding to one agent framework | **loop**, or **agent loop** |\n| the text the model gets as its role and task | **system prompt** |\n| the first message of the run | **user turn** |\n| what a person tells this run to do | **instructions** |\n| one model request | **turn** |\n| the record of what the loop ran and did | **agent trace** |\n| `{{ name }}` in a body | **placeholder** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| agent spec (in prose) | **agent definition**; `TAgentSpec` keeps its name as a type |\n| agent reference, agent ref (in prose) | **agent definition reference**; the field `agent_ref` keeps its name |\n| agent (bare) where the sentence needs one of the three | say which: **agent definition**, **agent job** or **agent run** |\n| harness (for the Claude Code CLI) | the **dltHub AI Harness** owns that word; write **the Claude Code CLI** |\n| adapter (for a loop) | dlt's adapters are `bigquery_adapter` and friends; write **loop** |\n| prompt (bare) | say which one: **system prompt** or **user turn** |\n| spec (bare, for an agent) | bare `spec` is the configspec; write **agent definition** |\n| trace (bare, for an agent) | bare `trace` is the pipeline trace; write **agent trace** |\n| manifest (for `AGENT.md`) | **agent file**; the manifest is the deployment manifest (G7) |\n| framework (as a loop's name) | name it: **pydantic-ai**, **claude-agent-sdk** |\n\n**Rulings**\n\n- **Bare `agent` is the actor.** \"The agent reads the logs\", \"the agent returns `aborted`\": the\n model acting during a run. That stays. The moment a sentence is about the file, the `run.agent`\n product or one execution, it names the definition, the job or the run.\n- **`Agent` (capitalised) is the agent job.** That is how the web UI labels it and how pydantic-ai\n and claude-agent-sdk name the same thing. Legal in UI copy and when talking about a framework;\n in dlt prose write **agent job**.\n- **An agent run is a job run.** Use **agent run** when the sentence is about what the agent\n received and produced; **job run** when it is about scheduling, status or logs, as for any job.\n- **`instructions` means two opposite things across the boundary.** In dlt it is the user turn. In\n pydantic-ai, `AgentSpec.instructions` is the system prompt. Qualify every mention of the\n framework field: \"pydantic-ai's `instructions` field (its system prompt)\".\n- **`turn` is one model request.** The **user turn** is the first message. A turn counter counts\n requests. Do not let one word carry both without the qualifier.\n- **`loop` is the agent loop.** In a file that also has Python loops, write **agent loop** once,\n then `loop`.\n- **Model names and aliases are technical nouns** (Rule 1.8): `sonnet`, `opus`, `claude-sonnet-5`.\n",".claude/skills/review-vocabulary/references/vocabulary/g9-workspace-access-and-tools.md":"# G9 — Workspace access and tools\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the declaration of what a job may touch | **access** (the `access` block, `TWorkspaceAccess`) |\n| one of its three keys, `local`, `data` and `context` | **access axis** |\n| a value on an axis | **verb** |\n| what the runtime or a loop does with the declaration | **grant**, **deny** (verbs) |\n| what it did grant | **granted access** |\n| what it did not grant | **denied access** |\n| what a loop supplies beyond the declaration | **over-granted** |\n| what a tool needs before it is served | **required access** |\n| what a model can call | **tool** |\n| the MCP grouping a manifest requests | **feature group** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| permission, entitlement, scope (for the declaration) | **access**, the word dlt's docs already use; `permission` is legal only when naming the SDK's `permission_mode` |\n| narrowed, narrowing | say the fact: **with less access** |\n| grant (as a noun) | **the declared access**; `granted` as an adjective is legal (Rule 3.3) |\n| capability (for a dlt tool) | **tool**; legal only for pydantic-ai `capabilities` |\n| ceiling, floor (as prose metaphor) | say what it does: \"access is the most a loop can wire\" |\n| honor, honour, honored, unhonoured (any spelling) | access is **granted** or **denied**, never honored |\n\n**Rulings**\n\n- **Grant and deny are the verbs.** A job declares access; the runtime and the loop grant what they\n can; the rest is denied. dlt's docs already write \"grant access\" 13 times and \"denied access\"\n twice, so this is the house pairing, not a new one. A loop that supplies more than the\n declaration **over-grants**.\n- **The declaration is a request, not a claim.** A manifest `access` block says what the job wants.\n Nothing in it is granted until the runtime says so. Say \"the job declares\", never \"the job has\".\n- **`access` is the noun, and no synonym joins it.** `permissions`, `scopes`, `capabilities` and\n `entitlements` each name this concept somewhere in the industry; dlt named it `access` before this\n feature existed — the built-in `access` profile is coarse-grained data access, and `profile_for()`\n derives that profile from `access.data`. One concept, one word (Rule 1.11).\n- **`tools:` in an `AGENT.md` holds feature groups, not tools.** Write \"feature groups\" whenever the\n sentence is about that field, or a reader counts 4 tools and gets 19.\n",".claude/skills/switch-worktree/SKILL.md":"---\nname: switch-worktree\ndescription: Switch the current session to work in an existing git worktree\nargument-hint: <worktree-name>\n---\n\n# Switch to worktree\n\nSwitch the current session to an existing worktree.\n\n## Steps\n\n### 1. Resolve the worktree\n\nIf `$ARGUMENTS` is empty, list available worktrees and ask the user to pick one:\n\n```\ngit worktree list\n```\n\nOtherwise, let `NAME` = `$ARGUMENTS`. The worktree path is `<repo-root>/.worktrees/<NAME>`.\n\nVerify the worktree exists in `git worktree list` output. If not, stop with an error.\n\nLet `WORKTREE` = the absolute path to the worktree.\n\n### 2. Switch cwd\n\n```\ncd WORKTREE\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, stop with an error — the cwd did not persist.\n\n### 3. Report\n\n```\nSwitched to worktree: WORKTREE\nBranch: <current branch from git branch --show-current>\n```\n",".claude/skills/worktree-from-issue/SKILL.md":"---\nname: worktree-from-issue\ndescription: Create a git worktree with a new branch for implementing a fix or feature for a GitHub issue\nargument-hint: <category>/<issue-number>-<short-description>\n---\n\n# Git Worktree for Issue Implementation\n\nSet up an isolated git worktree with a new branch to implement a fix or feature for a GitHub issue.\n\nParse `$ARGUMENTS` to extract the branch name in format `<category>/<issue-number>-<short-description>`.\n\n## Branch Naming Convention\n\nBranches must follow the pattern from CONTRIBUTING.md:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n```\n\nCategories: **feat**, **fix**, **exp**, **test**, **docs**, **keep** (all lowercase, dashes, no underscores).\n\nFor `feat` and `fix`, a ticket number is mandatory.\n\nExamples:\n- `fix/3562-clickhouse-drop-sync-perf`\n- `feat/1234-add-new-destination`\n\n## Steps\n\n### 1. Parse and validate\n\nExtract from `$ARGUMENTS`:\n- `branch_name`: Full branch name (e.g., `fix/3562-clickhouse-drop-sync-perf`)\n- `category`: Prefix before `/` (e.g., `fix`)\n- `issue_number`: Number after `/` and before first `-` (e.g., `3562`)\n- `short_name`: Directory-friendly version: `{category}-{issue_number}-{first-few-words}` (e.g., `fix-3562-clickhouse-drop-sync`)\n\nValidate the branch name matches pattern `(feat|fix|exp|test|docs|keep)/[0-9]+-[a-z0-9-]+`.\n\n### 2. Verify the issue exists\n\n```\ngh issue view <issue_number> --json number,title,state\n```\n\nWarn if the issue doesn't exist or is closed, but continue.\n\n### 3. Fetch latest devel\n\nFetch the newest devel from origin so the branch starts from the latest code:\n\n```\ngit fetch origin devel\n```\n\n### 4. Create the worktree\n\nUse `/create-worktree` skill:\n\n```\n/create-worktree <short_name> --branch origin/devel\n```\n\nThen create the feature branch inside the worktree:\n\n```\ngit -C <worktree-path> checkout --no-track -b <branch_name>\n```\n\n`--no-track` leaves the branch without an upstream, so the user's `git push` works.\n\n### 5. Set up dev environment\n\nUse `/worktree-make-dev` skill to set up the development environment in the worktree. This runs `make dev` and copies `secrets.toml`.\n\nNote which secrets source was used — this determines what destinations can be tested.\n\n### 6. Verify cwd\n\nAfter dev setup, verify the cwd is still in the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, run `cd <worktree-path>` and verify again. Stop with an error if the cwd cannot be set.\n\n### 7. Report\n\nReport the worktree info and available test destinations:\n\n```\nWorktree ready: <worktree-path>\nBranch: <branch_name> (based on origin/devel)\nIssue: #<issue_number> - <issue_title>\n```\n\nInclude the test credential availability from `/worktree-make-dev` output — downstream skills need this to know which destinations can be tested.\n",".claude/skills/worktree-make-dev/SKILL.md":"---\nname: worktree-make-dev\ndescription: Set up a dev environment in an existing worktree (make dev + copy secrets)\nargument-hint: <worktree-path>\n---\n\n# Set up dev environment in worktree\n\nPrepare the worktree at `$ARGUMENTS` for development.\n\n## Steps\n\n### 1. Run setup script\n\n```\nbash tools/setup_worktree_env.sh <worktree-path>\n```\n\nThe script runs `make dev` and copies secrets without exposing credentials in the conversation. If it fails, report the error and stop.\n\n### 2. Report\n\nParse the script output for `SECRETS_SOURCE=` to determine what was used:\n\n- **`repo_root`**: Full credentials available — all destinations including remote (snowflake, bigquery, etc.)\n- **`dev_template`**: Local only — duckdb, dummy, filesystem, and container-based destinations (postgres, clickhouse, etc.)\n- **`none`**: No secrets found — warn the user\n\n```\nDev environment ready: <worktree-path>\n- make dev: done\n- secrets.toml: <source>\n```\n","AGENTS.md":"# dlt principles\n- This is a mature library used by thousands of the best engineers in the world\n- dlt is a library, not a platform. users add it to their code. we respect existing workflows and other libraries they use\n- no black boxes: clean, minimal Pythonic interfaces, human readable file formats, no side effects, documentation\n- multiply - don't add: we do more work here so our users do less.\n\n# dlt repo\n- we use `uv` and `uv run`. look in @Makefile\n- we branch from `devel` for feature branches\n\n# Commit messages (all agents)\n- [Conventional Commits](https://www.conventionalcommits.org/): `type:` or `type(scope):`, lowercase imperative subject, no trailing period.\n- Subject line only for the vast majority of commits; add a body line only for a non-obvious *why* (max ~3 lines total).\n- No footers — no `Co-Authored-By`, no \"Generated with ...\". No emojis.\n- When squash-merging, clean the squash message down to a single subject line.\n- Full rules (enforced per-agent via git/gh hooks under `.claude/`, `.cursor/`, `.codex/`): @.agents/rules/git-github.md","CLAUDE.md":"AGENTS.md","docs/website/docs/hub/agents/agent-definitions.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fagent-definitions.md -->","docs/website/docs/hub/agents/index.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Findex.md -->","docs/website/docs/hub/agents/job-inspector.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fjob-inspector.md -->","tests/plugins/dlt_example_plugin/README.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fplugins%2Fdlt_example_plugin%2FREADME.md -->","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/dlthub/agents/e2e/self-report/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_e2e_workspace%2F.claude%2Fdlthub%2Fagents%2Fe2e%2Fself-report%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-hidden-rule.md":"This rule carries the token MARKER-RULE-HIDDEN. An agent asked for its markers reports it.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-visible-rule.md":"This rule carries the token MARKER-RULE-VISIBLE. An agent asked for its markers reports it.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/hidden-skill/SKILL.md":"---\nname: hidden-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-HIDDEN. Copy it into `markers`.\n","tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/visible-skill/SKILL.md":"---\nname: visible-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-VISIBLE. Copy it into `markers`.\n","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/broken-code/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fbroken-code%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/checked-inspector/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fchecked-inspector%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/error-triage/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Ferror-triage%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/job-inspector/AGENT.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fjob-inspector%2FAGENT.md -->","tests/workspace/cases/workspaces/agent_workspace/.claude/rules/dlthub-platform-job-resources.md":"Resource changes are proposals, never applied automatically.\n","tests/workspace/cases/workspaces/agent_workspace/.claude/skills/debug-deployment/SKILL.md":"---\nname: debug-deployment\ndescription: Debug a failing deployment.\n---\nRead the run record first.\n","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_toolkit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_toolkit/skills/find-source/SKILL.md":"---\nname: find-source\n---\nFind a source.","tests/workspace/cli/dlthub/ai/cases/mock_workbench/bootstrap/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fbootstrap%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/init/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Frest-api-pipeline%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/skills/find-source/SKILL.md":"---\nname: find-source\ndescription: Find and configure a REST API source\n---\n# Find Source\nInstructions here.","tests/workspace/cli/dlthub/ai/cases/mock_workbench/sql-database/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fsql-database%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/init/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","tests/workspace/cli/dlthub/ai/cases/versioned_workbench/my-toolkit/.claude-plugin/plugin.json":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Fmy-toolkit%2F.claude-plugin%2Fplugin.json -->","tools/tests/cases/dependency_changes/README.md":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tools%2Ftests%2Fcases%2Fdependency_changes%2FREADME.md -->"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# dlt principles\n- This is a mature library used by thousands of the best engineers in the world\n- dlt is a library, not a platform. users add it to their code. we respect existing workflows and other libraries they use\n- no black boxes: clean, minimal Pythonic interfaces, human readable file formats, no side effects, documentation\n- multiply - don't add: we do more work here so our users do less.\n\n# dlt repo\n- we use `uv` and `uv run`. look in @Makefile\n- we branch from `devel` for feature branches\n\n# Commit messages (all agents)\n- [Conventional Commits](https://www.conventionalcommits.org/): `type:` or `type(scope):`, lowercase imperative subject, no trailing period.\n- Subject line only for the vast majority of commits; add a body line only for a non-obvious *why* (max ~3 lines total).\n- No footers — no `Co-Authored-By`, no \"Generated with ...\". No emojis.\n- When squash-merging, clean the squash message down to a single subject line.\n- Full rules (enforced per-agent via git/gh hooks under `.claude/`, `.cursor/`, `.codex/`): @.agents/rules/git-github.md","isInternal":false,"tokens":275,"sizeBytes":1072},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"AGENTS.md","isInternal":false,"tokens":3,"sizeBytes":9},{"name":"git-github.md","path":".agents/rules/git-github.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.agents/rules/git-github.md","title":"git-github","category":"anthropic-skill","format":"markdown","content":"---\nname: git-github\ndescription: Rules for using git and the GitHub CLI in dlt. Read when committing, creating PRs, or running any git/gh command.\n---\n\n# dlt — Git & GitHub\n\nRules for git and GitHub CLI usage in this repo, for any coding agent. They override the agent's defaults where they differ. See `CONTRIBUTING.md` for the human-facing version.\n\n## When to use\n\nApply whenever running `git` or `gh`, drafting commit messages, or creating/updating PRs.\n\n## Commit messages\n\nFollow [Conventional Commits](https://www.conventionalcommits.org/):\n\n```\n{type}: short imperative subject\n# or, when one area is in focus\n{type}({scope}): short imperative subject\n```\n\n- **Types** (lowercase, matching branch categories): `feat`, `fix`, `docs`, `test`, plus `refactor`/`chore` for non-functional changes.\n- **Scope** is optional — an area like `dashboard`, `cli`, `destinations`, `extract`, `normalize`. Comma-separate multiple.\n- **Subject is lowercase, imperative, no trailing period.**\n- **No emojis** anywhere in the message — not in the subject, not in the body.\n- **Subject line only for the vast majority of commits.**\n- **Max 3 lines total** including any body. If you can't say it in 3 lines, the commit is probably too big — split it.\n- **Code-comment rules apply to commit bodies too.** Don't explain *what* the diff does (the diff shows that). Add a body line only when the *why* is non-obvious and not already in the PR description.\n- **No footers.** No `Co-Authored-By` lines, no \"Generated with ...\" trailers. Treat the agent as a dev tool, not a co-author. This overrides the agent's default.\n- **No references to the task/ticket/caller** in the message unless the user asks (\"as discussed\", \"per review feedback\", etc.).\n\nFormat (HEREDOC):\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n## Cleaning up before a squash-merge\n\nGitHub hides everything after the first line in the commit list, so multi-line\nbodies and footers are easy to overlook — and squashing several such commits\nfloods the merge commit. **When squash-merging, clean the squash message down to\na single Conventional-Commit subject line** (plus a short *why* body only if it\ngenuinely helps). Never let `Co-Authored-By` / \"Generated with\" lines survive\ninto the merge.\n\n## When to commit\n\n- **Only when the user explicitly asks.** Finishing a task is not a trigger; \"looks good\" is not a trigger. Wait for an explicit \"commit this\" / \"make a commit\".\n- `git add -A` is fine — but NEVER stage anything secret-looking (`.env`, `*secrets.toml`, credentials).\n\n## Branches\n\nFrom `CONTRIBUTING.md` — all lowercase, dashes, no underscores:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n# example:\nfeat/4922-add-avro-support\n```\n\n- **Categories**: `feat` (ticket required), `fix` (ticket required), `exp`, `test`, `docs`, `keep`.\n- **Branch from `devel`.** Feature branches and most fixes go to `devel`.\n- Don't create branches unless asked. Don't switch the branch a worktree is on without asking.\n\n## Pull requests\n\n- PRs **target `devel`**. Agents may **NEVER** merge to `master`.\n- **Link the PR to its ticket**, or describe the change clearly enough for someone without prior context.\n- **Creating a PR requires the branch to be pushed already.** If `gh pr create` would need a push, stop and ask the user to push.\n- Confirm the title and body with the user before running `gh pr create`.\n\n## Hard rules for the coding agent (never without an explicit ask)\n\n- **Don't push.** No `git push`/`git push -u`. The user pushes.\n- **Don't merge PRs.** No `gh pr merge`, no auto-merge.\n- **Don't force-push, ever.** If history needs rewriting, describe it and let the user do it.\n- **Don't `--amend` pushed commits**, and never use `--no-verify` or skip hooks/signing.\n- **Don't touch `master`** — it's the release branch (`devel → master` on release day).\n\n## Safe read-only commands (no confirmation needed)\n\n- `git status`, `git diff`, `git log`, `git show`, `git branch`, `git remote -v`\n- `gh pr view`, `gh pr list`, `gh pr checks`, `gh pr diff`, `gh issue view`, `gh api` for GET requests\n\n## Quick checklist before any git/gh action\n\n1. Push, merge, force-push, or amend of a pushed commit? → **Stop**, the user does this.\n2. New commit? → Did the user explicitly ask? If no, **stop**.\n3. Is the subject `type:` / `type(scope):`, lowercase, 1–3 lines total, free of \"what\" narration? If no, rewrite.\n4. Is the commit free of footers (no `Co-Authored-By`, no \"Generated with\") and emojis? If no, trim.\n5. Squash-merging? → Clean the squash message to a single subject line.\n","frontmatter":{"name":"git-github","description":"Rules for using git and the GitHub CLI in dlt. Read when committing, creating PRs, or running any git/gh command."},"isInternal":false,"tokens":1222,"sizeBytes":4647},{"name":"add-workspace-command.md","path":".claude/rules/add-workspace-command.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/add-workspace-command.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"dlt/_workspace/cli/**\"\n - \"dlt/_workspace/mcp/**\"\n - \"dlt/_workspace/helpers/dashboard/**\"\n - \"dlt/_workspace/typing.py\"\n - \"tests/workspace/**\"\n---\n\n# Workspace model/loader/view pattern for commands\n\nWhen adding a new command (cli, mcp or dashboard) `dlt/_workspace` follows a limited MVC pattern so that data models stay independent\nfrom presentation. The same model can be rendered by a CLI print function, returned\nfrom an MCP tool, or displayed in a marimo dashboard.\n\n## Layers\n\n| Layer | Location | Convention |\n|-------|----------|------------|\n| **Model** | `dlt/_workspace/typing.py` | `TypedDict` (T-prefixed), plain data, no I/O |\n| **Loader** | `dlt/_workspace/cli/ai/utils.py`, `cli/utils.py` | `fetch_*()` functions — collect data into a model, no `fmt.echo()` calls |\n| **View** | `cli/*_command.py`, `cli/commands.py` | `_print_*()` or `print_*()` — render a model to CLI via `fmt.echo()` / `fmt.warning()` |\n| **Controller** | argparse handlers, MCP tools, marimo notebooks | Calls loader, then passes model to the appropriate view |\n\nNOTE: loaders are stored in cli module. to be refactored soon\n\n## Rules\n\n1. **Loaders return typed dicts, never print.**\n A loader (`fetch_ai_status`, `fetch_workspace_info`, `fetch_secrets_list`, ...)\n returns a `TypedDict` from `typing.py`. It must not call `fmt.echo()` or any\n display function.\n\n2. **Views accept a model, never load.**\n A view (`_print_ai_status`, `print_workspace_info`, ...) takes the typed dict\n and formats it for human consumption. It must not call loaders or access\n providers/config directly.\n\n3. **Controllers wire loader to view.**\n An argparse command function (decorated with `@track_command`) calls the loader,\n then the view. MCP tools call the loader and return the model (or a\n serialised form). Dashboard cells call the loader and render via marimo UI.\n\n4. **Research existing types**\n * Research other `typing.py` modules and Literal definitions and types to find potentially reusable types.\n * Reuse, reconcile, modify existing types - also from `common` and other namespaces\n * Make **reconciliation** and model to be **clean** part of implementation plan. Request user input if needed\n\n5. **Write good models**\n * Use `Literal` ie. for error or warning cases\n * Do not leak user messages into data model - view should do it\n * OK to store additional context ie. exception messages\n * Use `NotRequired` field to the model (e.g. `mcp_error: NotRequired[str]`) if element may not be present. In most cases that is better than setting to None\n\n\n## Existing examples\n\n- `fetch_workspace_info()` -> `TWorkspaceInfo` -> `print_workspace_info()`\n- `fetch_ai_status()` -> `TAiStatusInfo` -> `_print_ai_status()`\n- `fetch_secrets_list()` -> `List[TLocationInfo]` -> `ai_secrets_list_command()` (inline view)\n\n## When adding new commands\n\nBefore writing a new CLI command, MCP tool, or dashboard cell that shows\nworkspace state, check whether a loader already exists. If not, create the\nmodel in `typing.py` and the loader in the appropriate `utils.py` first,\nthen write the view / controller.\n","frontmatter":{"paths":["dlt/_workspace/cli/**","dlt/_workspace/mcp/**","dlt/_workspace/helpers/dashboard/**","dlt/_workspace/typing.py","tests/workspace/**"]},"isInternal":false,"tokens":777,"sizeBytes":3135},{"name":"coding-style.md","path":".claude/rules/coding-style.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/coding-style.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"**/*.py\"\n---\n\n# Coding style\n\n## Formatting\n- Line length: 100 chars (black + isort, profile \"black\")\n\n## Code and patterns reuse\n- Obvious utils and patterns are already in code base\n- When in doubt look at the code around the changes you are making - not only style but coding patterns and code organization\n- When using a util function or any other function from the code base - look at existing usage if in doubt how it works\n\n## Comments\n- Inline comments start with lowercase (`# resolve the schema`)\n- NEVER add comments that separate code blocks (`# -- sqlglot_dialect mapping --`)\n- DO NOT state the obvious, do not document your thought process\n\n## Types and data structures\n- Always use full generic parametrization -- `Dict[str, Any]` not `dict`\n- Type aliases use T-prefix PascalCase (`TDataItem`, `TLoaderFileFormat`) -- see @dlt/common/typing.py\n- `TypedDict` for structured dicts, `NamedTuple` for lightweight immutable data\n- Prefer `functools.partial` over `lambda` when currying\n\n## Exceptions\n- Inherit from `DltException`; use `TerminalException`/`TransientException` mixins -- see @dlt/common/exceptions.py\n","frontmatter":{"paths":["**/*.py"]},"isInternal":false,"tokens":273,"sizeBytes":1145},{"name":"docstrings.md","path":".claude/rules/docstrings.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/docstrings.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"**/*.py\"\n---\n\n# Docstring rules\n\n- DO NOT state the obvious, do not document your thought process\n\nGoogle style docstrings throughout. Three levels of detail:\n\n## Short form (internal methods, properties)\nDescription only -- no Args/Returns. One-liner for simple accessors.\n\n## Full form (public API)\nFull Google style with Args, Returns, Raises. Describe WHAT the function does and its output, never HOW it works internally.\n\n```python\ndef paginate(\n self,\n path: str = \"\",\n method: HTTPMethodBasic = \"GET\",\n params: Optional[Dict[str, Any]] = None,\n) -> Iterator[PageData[Any]]:\n \"\"\"Iterates over paginated API responses, yielding pages of data.\n\n Args:\n path (str): Endpoint path for the request, relative to `base_url`.\n method (HTTPMethodBasic): HTTP method for the request, defaults to 'get'.\n params (Optional[Dict[str, Any]]): Query parameters for the request.\n\n Yields:\n PageData[Any]: A page of data with request and response context.\n\n Raises:\n HTTPError: If the response status code is not a success code.\n \"\"\"\n```\n\n- Public API args use `name (Type):` format; internal methods may omit types\n- Args descriptions are short -- one line unless truly complex\n- Use `Yields:` for generators, `Returns:` for regular functions\n- `Raises:` only for exceptions the caller should handle\n- `Example:` section is optional, add only when usage is non-obvious\n\n## Field docstrings\nInline triple-quoted string on the NEXT line after the field definition.\n\n```python\n@configspec\nclass ItemsNormalizerConfiguration(BaseConfiguration):\n add_dlt_id: bool = True\n \"\"\"When true, items will have `_dlt_id` column added with a unique ID.\"\"\"\n add_dlt_load_id: bool = False\n \"\"\"When true, items will have `_dlt_load_id` column added with current load ID.\"\"\"\n```\n\n- Only document fields where the name + type aren't self-explanatory\n- Keep to one line when possible\n\n## Inline formatting\nDocstrings are rendered as Markdown by most editors. Use only these:\n- Single backticks for parameter names, values, types, paths: `name`, `True`, `None`\n- **bold** for emphasis on key terms\n- `Note:` or `Warning:` prefixes for callouts in prose\n- `>>>` interactive Python style for code examples:\n\n```python\n \"\"\"Load data from a REST API.\n\n Example:\n >>> import dlt\n >>> pipeline = dlt.pipeline(\"my_pipeline\", destination=\"duckdb\")\n >>> pipeline.run(my_source())\n \"\"\"\n```\n\nDo NOT use:\n- Double backticks ``like this`` (reStructuredText literal)\n- `:param:`, `:type:`, `:returns:`, `:rtype:` (Sphinx field list)\n- `.. directive::` blocks (reST directives)\n\n## Anti-patterns -- NEVER do these\n- Do not explain HOW the code works, only WHAT it does and its output\n- Do not state the obvious (`\"\"\"Returns the name\"\"\"` on `get_name`)\n- Do not add docstrings to code you did not change\n- Do not use restructured text/sphinx formatting (see Inline formatting above)\n","frontmatter":{"paths":["**/*.py"]},"isInternal":false,"tokens":706,"sizeBytes":2955},{"name":"documentation.md","path":".claude/rules/documentation.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/documentation.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"docs/**\"\n---\n\n# Documentation guidelines\n\n## Setup\n- On your first setup, go to directory `docs/` and run `make dev`. This will install the required Python and JS/TS dependencies to build the docs `cd docs && make dev`.\n- This will also install pre-commit hooks using `prek`. This will automatically run linting, type checking, etc. on each `git commit` and `git push`\n\n## Code snippets\n- Inline code snippets will automatically be formatted, linted, and type-checked. To opt-out, add the directive to the codefence `notype` or `nolint`.\n- The `execute` directive means the snippet will be ran during before each commit and on CI. The executed snippets should be lightweight.\n\nFor example\n\n > \\```python notype execute\n >\n > import foo\n > \n > def my_func() -> int: ...\n > \n > \\``` \n \n## Internal links\n- Use relative paths **with** `.md` extension: `[schema contracts](schema-contracts.md)`\n- Link to sections with `#` anchors (auto-generated from headings, lowercase hyphenated): `[merge strategies](merge-loading.md#merge-strategies)`\n- Go up directories as needed: `[adjust a schema](../walkthroughs/adjust-a-schema.md)`\n\n## Markdown formatting\n- `mdsmith` lints `website/docs` on CI (`md-check` hook), configured in `docs/.mdsmith.yml`. After editing a page, run from `docs/`:\n ```sh\n make md-fix # pads and aligns tables, fixes whitespace\n make md-check # must report failures=0\n ```\n- Tables: write them unaligned and let `make md-fix` pad the columns (MDS025). It cannot fix a cell over 35 words (MDS026): shorten the text or move the detail below the table.\n\n## Markdown frontmatter\n- Every doc page needs YAML frontmatter with at least `title`, `description`, and `keywords`:\n ```yaml\n ---\n title: Schema\n description: Schema definition and evolution\n keywords: [schema, dlt schema, yaml]\n ---\n ```\n","frontmatter":{"paths":["docs/**"]},"isInternal":false,"tokens":469,"sizeBytes":1861},{"name":"heresy.md","path":".claude/rules/heresy.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/heresy.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"Heresy: issues in code that not only must be **avoided** but actively **detected and removed**.\n\n## llm heresy\n* llm talks to itself in docstrings instead of writing code comments in relevant places\n* llm talks about some obscure downstream cases in upstream code (ie. base class is full of details of particular implementation or explains particular use case)\n* llm invents obscure terminology and uses it everywhere. does not follow terminology used in docs\n* llm writes extremely detailed and complex sentences to explain simple things\n\n## arithmetic heresy\n* decimals are converted to binary floats and back\n* binary floats are used to construct decimals. ie. `Decimal(1.1)`\n\n## datetime heresy\n* naive datetimes are used without a good reason. the only exception are end user requirements\n* pendulum is used in new code. use the stdlib counterparts: `datetime` with an explicit\n `timezone`/`ZoneInfo`, and the `ensure_datetime*` helpers instead of `ensure_pendulum_datetime*`.\n NOTE: existing pendulum usage MAY be tolerated where it is hard to change\n\n## terms heresy\n* use of following words constitutes heresy: gate/gating, graft\n\n## comments heresy\n* block separator comments as below\n```\n#------------------------------\n# comment\n#------------------------------\n```\nor similar. actually any form","isInternal":false,"tokens":288,"sizeBytes":1307},{"name":"imports.md","path":".claude/rules/imports.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/imports.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"**/*.py\"\n---\n\n# Import rules\n\n## Import order\n\nGroups separated by blank line:\n1. **stdlib** (`import os`, `from typing import ...`)\n2. **third-party** (`import pendulum`, `import semver`)\n3. **local dlt** (`from dlt.common import ...`)\n4. (tests only) **test utilities** (`from tests.utils import ...`)\n\n- Stdlib imports are ALWAYS at module level -- never inline in functions. This includes `contextlib`, `io`, `warnings`, `sys`, `os`, `re`, etc.\n- `dlt.common` imports (except `dlt.common.libs`) are module level. They are always available and have no heavy side effects.\n- `dlt.common.libs.*` imports may be inline -- they wrap optional dependencies that may not be installed (see Optional dependencies below).\n- If a module is already imported at the top of the file, additional imports from the same module go at the top too -- never inline a second import from an already-imported module.\n- No relative imports.\n\n## Logging\n\n`from dlt.common import logger`, then `logger.info(...)` -- never `import logging` directly (except in `logger.py` itself or runtime infrastructure).\n\n## TYPE_CHECKING imports\n\n- Optional deps for annotations only: `if TYPE_CHECKING: from X import Y` with `else: Y = Any`\n- Destination factories: client class under `TYPE_CHECKING`, actual import deferred to runtime method\n\n## Optional dependencies\n\ndlt wraps optional deps in `dlt/common/libs/` with fail-fast `MissingDependencyException`. Always import from the wrapper, never from the package directly.\n\n### Canonical patterns\n\n```python\nfrom dlt.common.libs.pandas import pandas\n```\n\n### Rules\n\n- NEVER import **optional package** directly\n- ALWAYS import from `dlt.common.libs.<optional package>`\n- Import at the top of the module if module already imports **optional package**\n- Import inline in other cases.\n\n**Applies to**: `pyarrow`, `pandas`, `numpy`, `pydantic`, `sqlalchemy`, `ibis`, `deltalake`, `pyiceberg`\n","frontmatter":{"paths":["**/*.py"]},"isInternal":false,"tokens":458,"sizeBytes":1920},{"name":"public-api.md","path":".claude/rules/public-api.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/public-api.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"dlt/**/*.py\"\n---\n\n# Public API and backward compatibility\n\n## Public interface\n- Everything DEFINED and DIRECTLY IMPORTED in @dlt/__init__.py is PUBLIC INTERFACE. Change here is BREAKING CHANGE.\n- Behaviors described in @docs are considered public and changing them is a BREAKING CHANGE (in outputs of PUBLIC INTERFACES)\n- Always flag BREAKING CHANGES clearly with # BREAKING: comments\n\n## Deprecation warnings\n- Use `DltDeprecationWarning` from `dlt/common/warnings.py`, never raw `DeprecationWarning`\n- Pass the `since` version where the deprecation is introduced\n- Use `Dlt100DeprecationWarning` only for features deprecated since 1.0.0 (hardcodes `since`)\n- To deprecate entire functions or classes, use the `deprecated` decorator from `dlt/common/warnings.py`\n\n## Backward compatibility mechanisms\n- Avoid BREAKING CHANGES with deprecations, configuration options etc.\n- Schema/state migration via `engine_version`, versioned storage layouts, deprecation warnings, `deprecated` decorator\n- Backward compat must be tested -- see `tests/pipeline/test_dlt_versions.py` for e2e version upgrade tests\n","frontmatter":{"paths":["dlt/**/*.py"]},"isInternal":false,"tokens":264,"sizeBytes":1117},{"name":"testing.md","path":".claude/rules/testing.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/testing.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"tests/**/*.py\"\n - \"tools/tests/**/*.py\"\n---\n\n# Testing\n\n## Style and structure\n- Module-based tests (plain `def test_*() -> None:` functions), not class-based\n- pytest with fixtures in `conftest.py`, not unittest setUp/tearDown\n\n## Reduce tests sprawl\n- Use `pytest.mark.parametrize` to reduce code duplication. ALWAYS use human readable ids for the test cases\n- It is OK to test many related use cases in one test: ie. when they share common setup or represent a flow of user actions\n\n## Fixtures\n- We have autouse fixtures enabled by `conftest.py`. Always check those before adding common fixtures ie. to preserve env variables or get clean storage for the tests\n- We have A LOT of fixtures in files like @tests/utils.py and @tests/load/utils.py, look there before adding new ones.\n\n## Large files for test cases\n- Do not keep large test cases in Python modules. Use `cases` folder and to keep and share files (ie. schemas, ingest data) across modules. \n\n## Destination tests (tests/load)\n- Tests in @tests/load are ALWAYS run on CI in context of particular destination. Tests that run pipelines MUST use `destinations_configs`, other test modules must filter for a right destination (ie. `skip_if_not_active(\"clickhouse\")`)\n- NOTE: tests outside @tests/load do not use such filtering. Use `duckdb`, `dummy` or `sqlite` or `filesystem` destinations for such tests.\n\n### Test Markers\n- `@pytest.mark.essential` — For critical test paths\n- `@pytest.mark.serial` — Tests that can't run in parallel\n\n## Parallel safety and isolation\n- Tests run in parallel! ALWAYS use test storage (`get_test_storage`), unique pipeline names (`uniq_id`), and worker-aware storage roots (`get_test_storage_root` from `tests/utils.py`). Isolate pipelines with `dev_mode` or random `dataset_name`. Avoid external resources that clash with other tests.\n- Tests must work on Linux, macOS and Windows. Avoid encoding paths and platform-specific code. Local path assertions must be Windows compatible.\n\n## Placement and comments\n- When adding new test ALWAYS try find the right place by looking where code you make changes is tested\n- Docstrings: DESCRIBE test case. SKIP when test name is sufficient\n- Use line comments to describe and separate use cases within a single test\n\n## Running\n* DO NOT start two `pytest` processes at once!\n* when running many tests, use `-p xdist -n auto` to parallelize\n","frontmatter":{"paths":["tests/**/*.py","tools/tests/**/*.py"]},"isInternal":false,"tokens":545,"sizeBytes":2397},{"name":"worktree.md","path":".claude/rules/worktree.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/rules/worktree.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"# Working in a git worktree\n\nWhen a worktree is active (created via `/create-worktree`, `/worktree-issue`, or the user says \"work in worktree X\"), apply these rules for ALL subsequent operations.\n\nLet `WORKTREE` = the absolute worktree path (e.g., `/home/user/src/dlt/.worktrees/fix-3562`).\n\n## Shell cwd behavior\n\nClaude Code resets Bash cwd to the project root only when `cd` targets a path **outside** the project root. Paths **within** the project root (including `.worktrees/`) persist across calls.\n\n- **Bash**: run `cd WORKTREE` once to switch; subsequent calls will stay there\n- **git**: `git -C WORKTREE <command>` also works\n- **Read/Grep/Glob/Edit/Write**: always use absolute paths under `WORKTREE`\n\n## Running Python and tests\n\nAlways use `uv run` in worktrees — never bare `python` or `pytest`:\n\n```\nuv run pytest tests/... -v\nuv run python -c \"...\"\n```\n\n## Worktree location convention\n\nAll worktrees live under `<repo-root>/.worktrees/<name>`.\n\n## Plans created in a worktree\n\nWhen creating a plan while working in a worktree, **always** include the worktree path at the top of the plan:\n\n```markdown\n## Working environment\n- Worktree: <absolute worktree path>\n- Branch: <branch name>\n- Execute: cd <absolute worktree path> before any work\n```\n\nThis is critical because plan→execute transitions start a new session whose cwd resets to the project root. Without the worktree path in the plan, the execute phase will modify files in the main repo instead of the worktree.\n","isInternal":false,"tokens":369,"sizeBytes":1490},{"name":"SKILL.md","path":".claude/skills/auto-docstring-rules/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/auto-docstring-rules/SKILL.md","title":"auto-docstring-rules","category":"anthropic-skill","format":"markdown","content":"---\nname: auto-docstring-rules\ndescription: Enforces dlt docstring conventions. Use when writing or modifying docstrings, classes, or function signatures in Python files.\nuser-invocable: false\n---\n\nUse @.claude/rules/docstrings.md regardless of path setting","frontmatter":{"name":"auto-docstring-rules","description":"Enforces dlt docstring conventions. Use when writing or modifying docstrings, classes, or function signatures in Python files.","user-invocable":false},"isInternal":false,"tokens":57,"sizeBytes":257},{"name":"SKILL.md","path":".claude/skills/auto-import-rules/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/auto-import-rules/SKILL.md","title":"auto-import-rules","category":"anthropic-skill","format":"markdown","content":"---\nname: auto-import-rules\ndescription: Enforces dlt import conventions and ordering. Use when writing or modifying Python imports in dlt/ or tests/.\nuser-invocable: false\n---\n\nUse @.claude/rules/imports.md regardless of path setting","frontmatter":{"name":"auto-import-rules","description":"Enforces dlt import conventions and ordering. Use when writing or modifying Python imports in dlt/ or tests/.","user-invocable":false},"isInternal":false,"tokens":54,"sizeBytes":234},{"name":"SKILL.md","path":".claude/skills/auto-write-test-rules/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/auto-write-test-rules/SKILL.md","title":"auto-write-test-rules","category":"anthropic-skill","format":"markdown","content":"---\nname: auto-write-test-rules\ndescription: Use when adding or proposing a new test ONLY if the test rules are not present\nuser-invocable: false\n---\n\nUse @.claude/rules/testing.md regardless of path setting","frontmatter":{"name":"auto-write-test-rules","description":"Use when adding or proposing a new test ONLY if the test rules are not present","user-invocable":false},"isInternal":false,"tokens":48,"sizeBytes":207},{"name":"SKILL.md","path":".claude/skills/commit/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/commit/SKILL.md","title":"commit","category":"anthropic-skill","format":"markdown","content":"---\nname: commit\ndescription: Stage changes and write a clean Conventional-Commit message that follows the repo's git rules\nargument-hint: [-- <optional subject hint or scope>]\n---\n\n# Commit\n\nCreate a commit that follows `.agents/rules/git-github.md`. Invoke as `/commit`, or\nuse these steps whenever you are about to commit.\n\nParse `$ARGUMENTS`: anything after `--` is an optional hint (a subject, a scope, or\n\"only the dashboard files\") — guidance, not the literal message.\n\n## Steps\n\n### 1. Inspect\n\n```\ngit status\ngit diff --staged\ngit diff\n```\n\nDecide what belongs in the commit. If nothing is staged, stage the relevant changes\nwith `git add <paths>` (or `git add -A` when everything belongs together). **NEVER\nstage secret-looking files** (`.env`, `*secrets.toml`, credentials) — they are also\ndenied in `settings.json`.\n\n### 2. Compose the message\n\nApply `.agents/rules/git-github.md`:\n\n- One line: `{type}: subject` or `{type}({scope}): subject` — lowercase, imperative,\n no trailing period. Types: `feat`, `fix`, `docs`, `test`, `refactor`, `chore`.\n- **Subject line only** for the vast majority of commits. Add a body line *only* for a\n non-obvious *why*, max 3 lines total. Never narrate *what* the diff does.\n- **No footers** — no `Co-Authored-By`, no \"Generated with ...\" trailers.\n- **No emojis** anywhere in the message.\n\n### 3. Commit\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n### 4. Confirm\n\n```\ngit log -1 --format=%B\n```\n\nVerify it is a single Conventional-Commit subject (plus an optional why) with no\nfooters. **Do not push and do not amend pushed commits** unless the user explicitly\nasks.\n","frontmatter":{"name":"commit","description":"Stage changes and write a clean Conventional-Commit message that follows the repo's git rules","argument-hint":["-- <optional subject hint or scope>"]},"isInternal":false,"tokens":452,"sizeBytes":1690},{"name":"SKILL.md","path":".claude/skills/create-worktree/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/create-worktree/SKILL.md","title":"create-worktree","category":"anthropic-skill","format":"markdown","content":"---\nname: create-worktree\ndescription: Create or reuse a git worktree for a pull request or branch so reviews and work happen in isolation\nargument-hint: <name> [--pr <number>] [--branch <ref>]\n---\n\n# Git Worktree\n\nSet up an isolated git worktree.\n\nParse `$ARGUMENTS` to extract:\n- The worktree name (required) — used as the directory name under `.worktrees/`\n- `--pr <number>` (optional) — checkout a GitHub PR into the worktree using `gh pr checkout`\n- `--branch <ref>` (optional) — create the worktree from a specific branch or ref\n\n## Steps\n\n### 1. Determine worktree path\n\nThe worktree path is: `<repo-root>/.worktrees/<name>`\n\nFor example, if the repo root is `/home/user/src/dlt` and the name is `review-pr-3584`, the worktree path is `/home/user/src/dlt/.worktrees/review-pr-3584`.\n\nCreate the `.worktrees` directory if it does not exist:\n\n```\nmkdir -p <repo-root>/.worktrees\n```\n\n### 2. Create or reuse the worktree\n\nCheck if a worktree already exists at that path:\n\n```\ngit worktree list\n```\n\n- **If a worktree already exists** at the target path, reuse it.\n - If `--pr` was given, run `cd <worktree-path> && gh pr checkout <number>` to make sure it is up to date.\n - If `--branch` was given, run `git -C <worktree-path> checkout <ref>`.\n- **If no worktree exists** at the target path, create one:\n - If `--pr` was given, first resolve the PR's branch name: `gh pr view <number> --json headRefName -q .headRefName`\n - If `--pr` or `--branch` was given, check `git worktree list` output for any **other** worktree that already has the same branch checked out. If found:\n - Use `AskUserQuestion` to ask the user: \"Branch `<branch>` is already checked out in worktree `<existing-path>`. Use that worktree instead?\"\n - If the user says yes, use the existing worktree path (skip creation, proceed to step 3 with the existing path).\n - If the user says no, **STOP** — do not create a new worktree. Print a message explaining that the branch is already checked out elsewhere and exit.\n - If no conflict, create the worktree:\n - If `--branch` was given: `git worktree add .worktrees/<name> <ref>`\n - Otherwise: `git worktree add .worktrees/<name> --detach`\n - Callers needing a new branch check it out afterwards: `git checkout --no-track -b <new-branch>`\n - If `--pr` was given, then also: `cd <worktree-path> && gh pr checkout <number>`\n- `gh pr checkout` handles fetching, fork tracking, and branch setup automatically.\n\n### 3. Switch cwd\n\n```\ncd <worktree-path>\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, stop with an error — the cwd did not persist.\n\n### 4. Report\n\n```\nWorktree ready: <worktree-path>\n```\n","frontmatter":{"name":"create-worktree","description":"Create or reuse a git worktree for a pull request or branch so reviews and work happen in isolation","argument-hint":"<name> [--pr <number>] [--branch <ref>]"},"isInternal":false,"tokens":738,"sizeBytes":2703},{"name":"SKILL.md","path":".claude/skills/dashboard/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/dashboard/SKILL.md","title":"dashboard","category":"anthropic-skill","format":"markdown","content":"---\nname: dashboard\ndescription: Use when reading, editing, or creating files in dlt/_workspace/helpers/dashboard/ or tests/workspace/helpers/dashboard/ or tests/e2e/\nuser-invocable: false\n---\n\n# dlt Dashboard Development Guide\n\nConsult this skill whenever working on the dashboard codebase. It tells you where files live, what each module does, and where new code belongs.\n\n## Overview\n\nThe dashboard is a **marimo-based** reactive web UI for inspecting dlt pipelines, schemas, data, and run traces. It lives at `dlt/_workspace/helpers/dashboard/`.\n\n## File Layout\n\n```\ndlt/_workspace/helpers/dashboard/\n├── dlt_dashboard.py # Main marimo app (~30 reactive cells)\n├── dlt_dashboard_styles.css # CSS: colors, badges, timeline, tables\n├── runner.py # Subprocess launcher, CLI integration\n├── config.py # DashboardConfiguration @configspec dataclass\n├── const.py # Constants: port (2718), timeouts, colors, defaults\n├── typing.py # Shared TypedDicts (TTableListItem, TLoadItem, etc.)\n├── strings.py # All user-facing UI text (~350 lines)\n└── utils/\n ├── __init__.py # Package exports\n ├── pipeline.py # get_pipeline(), pipeline_details(), exception handling\n ├── queries.py # SQL execution, caching (BoundedDict), row counts, loads\n ├── schema.py # Table/column lists, schema version fetching\n ├── trace.py # Trace rendering, sanitization\n ├── visualization.py # Execution timeline HTML, load status badges\n ├── home.py # Home page / workspace home rendering\n ├── ui.py # dlt_table(), section_marker(), error_callout(), BoundedDict\n ├── formatters.py # Datetime humanization, string formatting\n └── data_quality.py # dlthub data quality integration\n```\n\n## Architecture\n\n- **Framework**: marimo — a reactive Python notebook that renders as a web app\n- **Cell pattern**: `@app.cell(hide_code=True)` decorators define reactive cells in `dlt_dashboard.py`\n- **Dependency tracking**: Cells automatically re-execute when their parameter dependencies change\n- **Routing**: URL query params (`?pipeline=X&profile=Y`) via `mo.query_params()`\n- **Caching**: `BoundedDict` (LRU, 64/32 entries) in `utils/queries.py` and `utils/schema.py`\n- **Styling**: CSS variables for dlt brand colors (`--dlt-color-lime`, `--dlt-color-aqua`, `--dlt-color-purple`, `--dlt-color-pink`)\n\n## Dashboard Sections (cells in dlt_dashboard.py)\n\n1. **Profile & Pipeline Discovery** — reads workspace profiles, lists local pipelines\n2. **Pipeline Attachment** — `dlt.attach()`, file watcher for auto-refresh\n3. **Overview** — pipeline properties, destination info, state version\n4. **Schema** — table browser with configurable column display\n5. **Browse Data** — SQL editor with cached results and query history\n6. **Trace** — execution timeline, step details, resolved config\n7. **Loads** — load history with row counts and schema versions\n8. **State** — raw JSON state viewer\n9. **Data Quality** — integration with dlthub.data_quality\n10. **Ibis Backend** — read-only Ibis connection\n\n## How the Dashboard is Launched\n\nThree CLI commands all call `runner.run_dashboard()`:\n\n* there are several commands in `dlt` and `dlthub` hosts that launch local dashboard. for example `dlt dashboard` and `dlthub local show`.\n\nEdit mode (`--edit`) ejects `dlt_dashboard.py` and CSS to the cwd for user customization.\n\n## Dependencies\n\nFrom `pyproject.toml` `workspace-deps` group: `marimo>=0.14.5`, `duckdb>=0.9`, `ibis-framework>=12.0.0`, `pyarrow>=16.0.0`.\n\n## Where to Put New Code\n\n| What you're adding | Where it goes |\n|---|---|\n| New dashboard section/cell | `dlt_dashboard.py` — add `@app.cell(hide_code=True)` |\n| New UI text/messages | `strings.py` — add to appropriate section |\n| New data fetching/queries | `utils/queries.py` or new file in `utils/` |\n| Schema-related helpers | `utils/schema.py` |\n| Pipeline metadata helpers | `utils/pipeline.py` |\n| Trace/execution helpers | `utils/trace.py` or `utils/visualization.py` |\n| UI component helpers | `utils/ui.py` |\n| Formatting/display helpers | `utils/formatters.py` |\n| New TypedDicts | `typing.py` |\n| New constants/defaults | `const.py` |\n| Configuration options | `config.py` (`DashboardConfiguration` dataclass) |\n| CSS styling | `dlt_dashboard_styles.css` |\n| Data quality features | `utils/data_quality.py` |\n\n## Coding Conventions\n\n- **All user-facing text** goes in `strings.py`, never inline in cells\n- **Cell-local variables must start with `_`** (underscore prefix). In marimo, variables that are only used within a single cell and should not be shared across cells must be prefixed with `_`. Only variables intended to be consumed by other cells should be unprefixed.\n- **UI helpers**: Use `utils/ui.py` functions (`dlt_table()`, `section()`, `error_callout()`) for consistent UI\n- **CSS classes**: `section-header`, `pipeline-execution-timeline`, `status-badge`, `dlt_table`\n- **Badge classes**: `.badge-grey`, `.badge-yellow`, `.badge-green`, `.badge-red`\n- **Data structures**: TypedDicts in `typing.py` for data passed between cells\n- **Configuration**: `@configspec` dataclass in `config.py`\n\n## Testing\n\n### Test locations\n\n```\ntests/workspace/helpers/dashboard/ # Unit and integration tests\n├── conftest.py # Fixtures: pipeline types and temp directories\n├── test_pipeline_workflow.py # Pipeline data retrieval integration\n├── test_all_cells.py # Cell execution and dependency testing\n├── test_visualization.py # Timeline and badge rendering\n├── test_ui_elements.py # UI component styling\n├── test_table_operations.py # Table listing/filtering\n├── test_trace_operations.py # Trace parsing/display\n├── test_query_operations.py # SQL query execution\n├── test_loads.py # Load history retrieval\n├── test_data_formatting.py # String formatting\n├── test_pipeline_operations.py # Pipeline metadata\n└── test_dashboard_config.py # Configuration resolution\n\ntests/e2e/ # Playwright browser tests\n```\n\n### Running tests\n\n```bash\nmake test-workspace-dashboard # unit/integration tests\nmake test-e2e-dashboard # headless chromium e2e\nmake test-e2e-dashboard-headed # visible browser e2e\nmake create-test-pipelines # create pipelines for manual testing\n```\n\n### Test fixtures\n\nTests use parametrized pipeline fixtures via `ALL_PIPELINES` and `PIPELINES_WITH_LOAD` constants in `conftest.py`. Key fixtures: `success_pipeline_duckdb`, `success_pipeline_filesystem`, `extract_exception_pipeline`, `never_ran_pipeline`, `load_exception_pipeline`.\n\n### Where to add new tests\n\n- **Utils logic**: Add to the matching `test_*.py` file (e.g., new query helper → `test_query_operations.py`)\n- **New section/cell**: Add cell execution test to `test_all_cells.py`, plus a dedicated test file if the section has significant logic\n- **Visual/rendering**: `test_visualization.py` or `test_ui_elements.py`\n- **E2E browser tests**: `tests/e2e/`\n","frontmatter":{"name":"dashboard","description":"Use when reading, editing, or creating files in dlt/_workspace/helpers/dashboard/ or tests/workspace/helpers/dashboard/ or tests/e2e/","user-invocable":false},"isInternal":false,"tokens":1710,"sizeBytes":7396},{"name":"SKILL.md","path":".claude/skills/implement-issue/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/implement-issue/SKILL.md","title":"implement-issue","category":"anthropic-skill","format":"markdown","content":"---\nname: implement-issue\ndescription: Triage, plan, and implement a fix or feature for a GitHub issue end-to-end\nargument-hint: <issue-number> [-- <hints or context from maintainer>]\n---\n\n# Implement Issue\n\nParse `$ARGUMENTS` to extract:\n- Everything before the first `--` (or all of it if no `--`) is the **issue number**. Referred to as `ISSUE` below.\n- Everything after the first `--` is **maintainer hints** — domain knowledge, suspected root cause, pointers to relevant code, or constraints. If absent, there are no hints.\n\n## Steps\n\n### 1. Fetch the issue\n\n```\ngh issue view ISSUE --json title,body,comments,labels,state,assignees,author,createdAt,milestone\n```\n\nProduce a concise summary: title, reporter, component/area involved, error message, reproduction steps, expected behavior.\n\n### 2. Determine category and gather hints\n\n#### Category\n\nDerive the branch category from the issue labels:\n\n| Label contains | Category |\n|---|---|\n| `bug` | `fix` |\n| `enhancement`, `feature` | `feat` |\n| `test` | `test` |\n| `documentation` | `docs` |\n\nIf no matching label exists, ask the human for the category.\n\n#### Hints\n\nCollect implementation hints from all available sources, in priority order:\n\n1. **Maintainer hints** from `$ARGUMENTS` (after `--`)\n2. **Issue comments** — look for maintainer/contributor replies with technical guidance, root cause analysis, or implementation suggestions\n3. **Issue body** — the reporter may include debugging details, stack traces, or configuration\n\nIf the combined hints from these sources provide sufficient direction (suspected root cause, relevant code areas, constraints), proceed directly to step 3 without asking the human.\n\nIf hints are insufficient — the issue is vague, the root cause is unclear, or there are multiple possible approaches — ask the human for guidance.\n\n### 3. Research the big picture\n\nBefore jumping to a fix, verify assumptions. This step prevents wasted effort on the wrong solution.\n\n#### 3a. Understand the feature area\n\nUse Explore agents to thoroughly investigate the relevant code paths. Trace the flow from user-facing API to the point of failure. Read the code — don't guess.\n\n#### 3b. Validate the reporter's setup\n\nCheck whether the reporter is using the library correctly:\n\n- Is their configuration valid for the component they're using?\n- Are there special conventions, settings, or patterns they should be using instead?\n- Does the documentation describe the expected behavior differently?\n\n#### 3c. Check external dependencies and integrations\n\nIf the issue involves specific external systems (destinations, sources, file formats, etc.):\n\n- Read the relevant implementation in the codebase (factory, capabilities, client, etc.).\n- Understand how the component interacts with the external system.\n- If needed, consult external documentation via WebFetch/WebSearch.\n\n#### 3d. Determine: bug or user error?\n\nBased on research, conclude one of:\n\n- **Bug**: The library has a defect. Proceed to step 4.\n- **User error**: The reporter's configuration is wrong. Draft a response explaining the correct approach. Ask the human to confirm before posting.\n- **Enhancement**: The library works as designed but could be improved. Discuss scope with the human before proceeding.\n\nReport findings to the human before moving on.\n\n### 4. Set up workspace\n\nBefore planning, create the working environment so plan-mode exploration runs against a clean branch.\n\n#### 4a. Determine branch name\n\nUse the category from step 2 and the issue number:\n```\n{category}/{issue-number}-{short-description}\n```\n\nExample: `fix/3529-dedup-sort-escape`\n\nIf `ISSUE` was already a branch name rather than a number (no ticket exists), use it as given.\n\n#### 4b. Create worktree and dev environment\n\nInvoke the `/worktree-from-issue` skill with the branch name:\n```\n/worktree-from-issue {branch-name}\n```\n\nThis creates the worktree, checks out a new branch from `origin/devel`, and runs `/worktree-make-dev`.\nUse it also when there is no issue number.\n\nNote the worktree path — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 5. Draft implementation plan\n\nEnter plan mode and design a plan that **will pass the `/review-pr` skill**. The review-pr skill checks:\n\n1. **Code quality**: Style conforming to `@CLAUDE.md`, no over-engineering, proper use of existing patterns and utilities.\n2. **Test coverage**: Required scenarios identified, tests in the right location, no duplication, proper parametrization.\n3. **Documentation**: Public API changes need doc updates.\n4. **Backward compatibility**: Breaking changes flagged with deprecation path.\n5. **Dependencies**: Changes to `pyproject.toml` flagged.\n\nThe plan must include:\n\n- **Working environment**: Worktree path, branch name, and `cd` instruction (see `@.claude/rules/worktree.md`).\n- **Root cause**: Precise code location and explanation.\n- **Fix**: Minimal change following existing patterns in surrounding code.\n- **Test**: A test that **fails without the fix and passes with it**. Place tests where similar functionality is already tested. Follow `@CLAUDE.md` test guidelines (parallel safety, platform independence, proper fixtures).\n- **Test scope**: Identify the right test level — `make test-common` for extract/normalize/common work, destination-specific tests for load work. See `@CLAUDE.md` \"Testing strategy\" section.\n- **Verification commands**: Exact selective pytest commands to run (specific test functions, not entire modules).\n- **Files changed**: List with expected line counts.\n\nPresent the plan via ExitPlanMode for human approval.\n\n### 6. Implement the fix\n\nApply the changes from the approved plan. Follow these rules:\n\n- **Read before editing**: Always read the target file first. Never edit blind.\n- **Minimal changes**: Only change what the plan specifies. Don't refactor surrounding code, add comments to unchanged code, or \"improve\" things not in scope.\n- **Follow local patterns**: Match the style of surrounding code exactly — imports, naming, spacing, comment style.\n- **Type hints**: All new function signatures need full type hints with generic parametrization (`Dict[str, Any]` not `dict`).\n- **Test conventions**: `def test_*() -> None:`, pytest fixtures, `@pytest.mark.parametrize` for variants.\n\n### 7. Verify\n\nRun verification in this order. Stop and fix issues before proceeding to the next step.\n\n#### 7a. Run the new test selectively\n\nRun only the new/changed test functions using the commands from the plan. Be selective — target specific test functions, not entire modules.\n\n#### 7b. Regression proof (for external system interactions)\n\nWhen the fix involves an external system (destination, source, API) where you don't have full visibility into the behavior — a \"black box\" — verify experimentally that the test actually catches the bug:\n\n- Temporarily revert the fix\n- Run the new test on a configuration where the bug manifests\n- Confirm it **fails** with an error matching the issue report\n- Restore the fix\n\nThis is valuable when you cannot fully predict how the external system handles the input (e.g., how a database treats unquoted identifiers, how an API responds to edge-case payloads). It proves the test is meaningful and not a no-op.\n\nSkip this step when the code path is fully internal and you can reason about correctness from the code alone.\n\n#### 7c. Format and lint\n\n```\nmake format && make lint\n```\n\nIf you changed pages under `docs/website/docs`, also run `make md-fix && make md-check` in `docs/`.\n\nAll must pass cleanly. Fix any issues before reporting.\n\n### 8. Report\n\nSummarize results:\n\n```\n## Fix for #ISSUE: <title>\n\n**Branch**: {branch-name}\n**Worktree**: WORKTREE\n\n### Changes\n- `<file>`: <description> (<lines> lines)\n- ...\n\n### Verification\n- <test-function> on <config>: passed\n- <test-function> on <config> without fix: FAILED (confirms bug) [if applicable]\n- make format: pass\n- make lint: pass\n\nReady to commit.\n```\n\nWait for the human to confirm before committing.\n","frontmatter":{"name":"implement-issue","description":"Triage, plan, and implement a fix or feature for a GitHub issue end-to-end","argument-hint":"<issue-number> [-- <hints or context from maintainer>]"},"isInternal":false,"tokens":1841,"sizeBytes":8224},{"name":"SKILL.md","path":".claude/skills/optimize-code/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/optimize-code/SKILL.md","title":"optimize-code","category":"anthropic-skill","format":"markdown","content":"---\nname: optimize-code\ndescription: Analyze and optimize Python code performance in critical paths\nargument-hint: <file-path-or-description> [-- <critical-path-hints>]\n---\n\n# Optimize Code\n\nYou are a Python performance optimization expert. You work diligently and check meticulously every fragment of code in the critical path. You never assume - you trace actual execution paths, verify with evidence, and question defensive patterns that may have become unnecessary.\n\n**ALWAYS benchmark everything.** Use `timeit` oneliners for quick validation, write dedicated benchmarks for optimization variants, and as the final step, benchmark optimized code against the baseline (devel branch).\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **target**: file path, function name, or description of code to optimize\n- Everything after `--` is **critical path hints**: which code paths matter most, what to ignore\n\n## Phase 1: Understand the Critical Path\n\nSKIP if optimizing a single (or 1-2 ) functions where critical path is clear, otherwise INTERROGATE the user to understand the hot path before making any changes.\n\n### 1.1 Identify the target\n\nIf `$ARGUMENTS` contains a file path, read it. Otherwise, ask the user:\n- \"Which file or function should I optimize?\"\n- \"What is the entry point for the hot path?\"\n\n### 1.2 Learn the typical case\n\nAsk about the 90% case to focus optimization effort:\n- \"What data shapes are most common?\" (flat dicts, nested objects, lists)\n- \"Which branches are taken most often?\"\n- \"What can we ignore?\" (e.g., \"ignore pandas/arrow, focus on plain dicts\")\n\n### 1.3 Trace the call chain\n\nFollow the code from entry point through each function call:\n```\nentry_point() → helper_a() → helper_b() → actual_work()\n```\n\nRead each function in the chain. Identify the innermost loop where per-item work happens.\n\n### 1.4 Establish baseline benchmark\n\nBefore any optimization, create a benchmark for the current state:\n```python\nimport timeit\n# Quick validation\npython -c \"import timeit; print(timeit.timeit('target_function()', setup='...', number=1000))\"\n```\n\n## Phase 2: Analyze for Optimization Opportunities\n\nLook for these patterns in the critical path:\n\n### 2.1 Unnecessary work\n\n- **Redundant copies**: `dict(x)` or `list(x)` when reference would suffice\n- **Defensive copies**: Trace data origin - if source returns fresh object, copy is redundant\n- **Repeated computations**: Same value computed multiple times in loop\n\n### 2.2 Function call overhead\n\n- **Hot helper functions**: Consider inlining if called per-item\n- **Uncached method lookups**: `self.method` in loop vs `method = self.method` before loop\n- **Duplicate isinstance/hasattr checks**: Same check in multiple places\n\n### 2.3 Happy path shortcuts\n\n- **Early exit for common case**: Check fast path first, skip expensive work\n- **Identity vs equality**: `x is y` faster than `x == y` when applicable\n\n\n### 2.4 Consider `__slots__` for hot objects\n\nObjects created frequently in hot paths benefit from `__slots__`:\n- 30-40% less memory (no `__dict__`)\n- Faster attribute access (92% of local variable speed vs 59% for regular attributes)\n\n```python\n# Bad: regular class, dict-based attributes\nclass Row:\n def __init__(self, table, data):\n self.table = table\n self.data = data\n\n# Good: slotted class\nclass Row:\n __slots__ = ['table', 'data']\n def __init__(self, table, data):\n self.table = table\n self.data = data\n```\n\n### 2.5 Inner function overhead\n\nInner functions (closures) defined inside loops or hot functions are recreated on every call. This can be a **major** performance hit:\n\n```python\n# Bad: inner function recreated per call\ndef process(items):\n def transform(x): # created every time process() is called\n return x * 2\n return [transform(i) for i in items]\n\n# Good: module-level or method\ndef _transform(x):\n return x * 2\n\ndef process(items):\n return [_transform(i) for i in items]\n\n# Good: inline if simple\ndef process(items):\n return [x * 2 for x in items]\n```\n\nSigns to look for:\n- `def` inside another `def`\n- Lambda inside frequently-called function\n- Closures capturing loop variables\n\n\n### 2.6 Convert recursion to iteration\n\nRecursive functions have overhead per call (stack frame, argument passing) and risk stack overflow on deep structures. Convert to stack-based iteration:\n\n```python\n# Bad: recursive traversal\ndef flatten(obj, path=\"\"):\n if isinstance(obj, dict):\n for k, v in obj.items():\n yield from flatten(v, f\"{path}.{k}\") # recursive call\n else:\n yield path, obj\n\n# Good: stack-based iteration\ndef flatten(obj):\n stack = [(obj, \"\")]\n while stack:\n current, path = stack.pop()\n if isinstance(current, dict):\n for k, v in current.items():\n stack.append((v, f\"{path}.{k}\"))\n else:\n yield path, current\n```\n\nBenefits:\n- No function call overhead per level\n- Predictable memory (heap vs stack)\n- No recursion limit issues\n- Can be easier to add early-exit logic\n\nWARN USER:\n- stack method MAY change the order of processing and ie. yielded elements. ALWAYS suggest to write a test for original code.\n\n## Phase 3: Benchmark Everything\n\n### 3.1 Always benchmark caches\n\nCaches (lru_cache, manual dicts, memoization) can hurt performance if:\n- Cache miss is common (lookup overhead without benefit)\n- Cached computation is cheap (cache overhead exceeds saved work)\n- Memory pressure causes cache eviction\n\n**Benchmark caches even if they already exist** - they may have been added speculatively:\n\n```python\n# Does this cache actually help? Benchmark with and without!\n@lru_cache(maxsize=128)\ndef get_column_type(col_name):\n return self.schema.columns[col_name][\"data_type\"]\n\n# Compare:\n# 1. With cache (current)\n# 2. Without cache (direct lookup)\n# 3. Different cache size\n```\n\n### 3.2 Quick validation with timeit\n\nBefore and after each micro-optimization:\n```bash\npython -c \"import timeit; d={'a':1,'b':2}; print('copy:', timeit.timeit('dict(d)', globals={'d':d}, number=1000000))\"\npython -c \"import timeit; d={'a':1,'b':2}; print('ref:', timeit.timeit('x=d', globals={'d':d}, number=1000000))\"\n```\n\n### 3.3 Write dedicated benchmarks\n\nFor optimization variants, create benchmark scripts in `experiments/` folder:\n```python\n\"\"\"Benchmark: optimization variant comparison\"\"\"\nimport timeit\n\ndef variant_original(): ...\ndef variant_optimized(): ...\n\nprint(\"original:\", timeit.timeit(variant_original, number=10000))\nprint(\"optimized:\", timeit.timeit(variant_optimized, number=10000))\n```\n\n### 3.4 Process isolation and thermal management\n\nRun benchmarks in separate processes with pauses:\n```bash\npython experiments/bench.py devel_case\nsleep 10\npython experiments/bench.py optimized_case\n```\n\n### 3.5 Variance checking\n\nRun each benchmark 3+ times to verify stability:\n```bash\nfor i in 1 2 3; do\n echo \"=== Run $i ===\"\n python experiments/bench.py case_name\n sleep 5\ndone\n```\n\nVariance over 10-15% suggests external factors (thermal throttling, system load).\n\n### 3.6 Use realistic benchmark data\n\nUse real test data from `tests/normalize/cases/`:\n\n| File | Size | Use Case |\n|------|------|----------|\n| `ethereum.blocks.*.json` + `schemas/ethereum.schema.json` | 2MB | Deep nesting, warm/cold path normalizer |\n| `github.events.*.json` | 1.7MB | Dynamic table routing, many event types |\n| `github.issues.*.json` | 526KB | REST API, moderate nesting |\n\nLoad with `from dlt.common.json import json`. For flat rows with ISO timestamps, use `mimesis` (in dev deps) to generate synthetic DB data.\n\n### 3.7 Final benchmark: Optimized vs Devel\n\nAs the last step, compare against baseline branch:\n```bash\n# On devel branch (main repo)\ncd /path/to/main && python experiments/bench.py case_name\n\nsleep 10\n\n# On optimized branch (worktree)\ncd /path/to/worktree && python experiments/bench.py case_name\n```\n\nReport: `devel_time / optimized_time = X.XXx speedup`\n\n## Phase 4: Implement and Verify\n\n### 4.1 Make changes incrementally\n\nOne optimization at a time. Benchmark after each change.\n\n### 4.2 Verify correctness\n\n- Run existing tests: `make test-common`\n- Compare outputs before/after on sample data\n\n### 4.3 Document assumptions\n\nAdd comments explaining why optimization is safe:\n```python\n# columns dict is never mutated after get_table_columns() returns,\n# so we can store reference instead of copying\nself._current_columns = columns\n```\n\n## Phase 5: Report Results\n\nProduce a summary:\n```\n## Optimization Summary\n\n### Changes Made\n1. Replaced dict copy with identity check (buffered.py:100)\n2. Inlined count_rows_in_items for common case (buffered.py:102)\n\n### Benchmark Results\n| Case | Devel | Optimized | Speedup |\n|------|-------|-----------|---------|\n| flat_100 | 10.4s | 2.2s | 4.76x |\n| nested_20 | 4.9s | 2.0s | 2.47x |\n\n### Assumptions\n- columns dict is immutable after creation\n- 90% of items are plain dicts, not arrow/pandas\n```\n","frontmatter":{"name":"optimize-code","description":"Analyze and optimize Python code performance in critical paths","argument-hint":"<file-path-or-description> [-- <critical-path-hints>]"},"isInternal":false,"tokens":2206,"sizeBytes":8945},{"name":"SKILL.md","path":".claude/skills/release-notes/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/release-notes/SKILL.md","title":"release-notes","category":"anthropic-skill","format":"markdown","content":"---\nname: release-notes\ndescription: Generate release notes between two git tags with categorized PR summaries and author attribution\nargument-hint: <git-tag> [-- <previous-tag>]\n---\n\n# Generate Release Notes\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **release tag** (e.g. `1.22.0`). Referred to as `TAG` below.\n- Everything after `--` is an optional **previous tag** override. If absent, auto-detect the previous tag.\n\n## Step 1: Validate the tag\n\nCheck that `TAG` exists as a git tag:\n\n```\ngit tag -l TAG\n```\n\nIf the tag does not exist, stop with an error: \"Tag `TAG` not found. Please create the tag first and re-run.\"\n\n## Step 2: Determine the previous tag\n\nIf a previous tag was provided in `$ARGUMENTS`, validate it exists and use it as `LOWER`.\n\nOtherwise, find the previous tag automatically:\n\n```\ngit tag --sort=-v:refname\n```\n\nWalk the sorted list to find the tag immediately before `TAG` (by version order). Set that as `LOWER`.\n\nValidate `LOWER` exists. Stop with an error if no previous tag can be determined.\n\n## Step 3: Gather commits\n\nGet all commits between the two tags:\n\n```\ngit log --oneline LOWER..TAG\n```\n\nFilter out:\n- Merge commits (starting with \"Merge pull request\" or \"Merge branch\")\n- Version bump commits (starting with \"bumps to version\" or \"bumps to\")\n\nExtract PR numbers from the remaining commit messages (pattern `#NNNN`). Every PR number must appear in the final release notes — none may be silently dropped.\n\n## Step 4: Fetch PR metadata\n\nFor each PR number, fetch metadata using `gh`. Use Task agents to run in parallel batches for efficiency:\n\n```\ngh pr view NUMBER --repo dlt-hub/dlt --json number,title,labels,body,author\n```\n\nCollect for each PR:\n- **number**: PR number\n- **title**: PR title\n- **author**: `author.login`\n- **labels**: list of label names\n- **body**: PR description (first 500 chars sufficient for categorization)\n\n### Handling low-quality PR titles or descriptions\n\nIf a PR title is vague (e.g. just an issue number, single word) or the body is empty/unhelpful:\n\n1. Check if the PR references an issue (e.g. `fixes #NNN`, `closes #NNN`, `#NNN` in body). If so, fetch the issue:\n ```\n gh issue view NNN --repo dlt-hub/dlt --json title,body\n ```\n2. If still unclear, look at the PR diff to understand what changed:\n ```\n gh pr diff NUMBER --repo dlt-hub/dlt\n ```\n3. Write a clear, accurate summary based on what you find.\n\n## Step 5: Categorize PRs\n\nAssign each PR to exactly one category using these rules (evaluated in order). **Every PR must be classified** — verify the total count matches.\n\n### Breaking Changes\nCollect separately (appears at the top, not a standalone category — the PR also appears in its main category). A PR has breaking changes if:\n- It has a `breaking` label, OR\n- Its body contains \"breaking change\" (case-insensitive), OR\n- Its title contains \"breaking\"\n\nExtract the specific breaking change description from the PR body.\n\n### Highlights\nA PR is a highlight if:\n- It has a `highlight` label, OR\n- It is a significant new feature (title starts with `feat` or `(feat)`) AND the change is impactful — meaning it introduces a new user-facing capability, enables a major use case, or represents substantial effort\n- Use judgment: not every feature is a highlight. A highlight should be something users would be excited about.\n\nHighlighted PRs also appear in their main category (Core Library, etc.) but with a brief entry pointing to the Highlights section.\n\n### Docs\n- Title or body indicates documentation-only changes\n- Files changed are only in `docs/`\n- Has `documentation` label\n\n### Chores\n- CI/CD improvements, test infrastructure, repo maintenance\n- Has labels like `ci full`, `tech-debt`, `ci`\n- Title starts with `chore`, `repo:`, `ci:`, or the change only affects tests/tooling\n\n### Core Library\nEverything else — features, bug fixes, enhancements to the library itself.\n\n## Step 6: Identify new contributors\n\nFor each release author, find their first ever merged PR:\n\n```\ngh search prs --repo dlt-hub/dlt --author AUTHOR --merged --sort created --order asc --limit 1 --json number,createdAt\n```\n\nA contributor is **new** if their first merged PR number falls within the `LOWER..TAG` commit range.\n\n## Step 7: Format the release notes\n\nWrite the notes in this exact structure:\n\n```markdown\n## dlt TAG Release Notes\n\n## Breaking Changes\n\n(Only if there are breaking changes. Numbered list.)\n\n1. **Short description** (#NUMBER @author) — Explanation of what broke and how it affects users.\n\n## Highlights\n\n- **Feature name** (#NUMBER @author) — Description of the feature and why it matters.\n\n## Core Library\n\n- **Feature/fix name** (#NUMBER @author) — Brief description.\n- **Fix: short description** (#NUMBER @author)\n\n## Docs\n\n- Description (#NUMBER @author)\n\n## Chores\n\n- **Title** (#NUMBER @author) — Brief description if non-obvious.\n\n## New Contributors\n\n- @author made their first contribution in #NUMBER\n```\n\n## Formatting rules:\n- PR references use `#NUMBER` format (GitHub auto-links them)\n- Authors use `@login` format (GitHub auto-links them)\n- Features and significant fixes get bold titles with em-dash descriptions\n- Minor fixes can be single-line bold entries without descriptions\n- Omit empty sections entirely (including New Contributors if none)\n- Within Core Library, group features first, then fixes\n\n## Step 8: Present for approval\n\nDisplay the formatted release notes to the user for review. Include a summary:\n- Total PRs analyzed\n- Breakdown by category (with counts)\n- Number of unique contributors (and how many are new)\n- List any PRs where you had to infer the description from issue or diff\n\nAsk the user to approve or request changes before writing.\n\n## Step 9: Write to file\n\nOnly after user approval, write the release notes to `/tmp/release-notes-TAG.md`.\n\nReport the file path to the user.\n","frontmatter":{"name":"release-notes","description":"Generate release notes between two git tags with categorized PR summaries and author attribution","argument-hint":"<git-tag> [-- <previous-tag>]"},"isInternal":false,"tokens":1406,"sizeBytes":5868},{"name":"SKILL.md","path":".claude/skills/review-issues-severity/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-issues-severity/SKILL.md","title":"review-issues-severity","category":"anthropic-skill","format":"markdown","content":"---\nname: review-issues-severity\ndescription: Find and prioritize open GitHub issues by severity, community impact, and maintainer abandonment\nargument-hint: [-- <optional filters or focus areas>]\n---\n\n# Review Issues by Severity\n\nParse `$ARGUMENTS` for optional filters:\n- Label filters (e.g., `question`, `bug`)\n- Focus areas (e.g., \"merge disposition\", \"Arrow\", \"ClickHouse\")\n- If absent, perform a broad review across all open issues.\n\n## Steps\n\n### 1. Gather candidate issues\n\nRun the following GitHub API queries **in parallel** to surface issues matching different severity signals:\n\n#### a) Long discussions (high comment count)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&per_page=100&sort=comments&direction=desc&page=1' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### b) Issues with specific labels (if filters provided)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&labels=LABEL&per_page=100&sort=comments&direction=desc' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### c) Recently stale issues (updated 2+ months ago, with maintainer comments)\nLook for issues where `updated_at` is old relative to the current date but had prior maintainer engagement.\n\n### 2. Deep-dive top candidates\n\nFor the top 10-15 most promising candidates, fetch full details using a subagent or parallel `gh issue view` calls:\n\n```bash\ngh issue view -R dlt-hub/dlt NUMBER --json title,body,comments,labels,createdAt,updatedAt,author,assignees\n```\n\nFor each issue, extract:\n- **Participants:** who commented, their association (MEMBER = maintainer, NONE/CONTRIBUTOR = community)\n- **Timeline:** when maintainers last engaged, how long since last response\n- **Problem description:** what breaks and under what conditions\n- **Reproduction quality:** is there a clean repro script?\n- **Current status:** fixed? workaround? stalled? abandoned?\n- **Production severity:** does this cause data loss, crashes, or silent corruption?\n\n### 3. Classify each issue\n\nApply these criteria to each issue:\n\n| Signal | What to look for |\n|--------|-----------------|\n| **Serious problem** | Data corruption, silent data loss, hard crashes, cascading failures, security issues |\n| **Long discussion** | 5+ comments, especially with multiple community reporters hitting the same issue |\n| **Maintainer abandoned** | Maintainer commented but last maintainer response is 2+ months old, no linked PR, reporter follow-ups unanswered |\n| **Question label** | Issue has `question` label — these often represent real bugs initially miscategorized |\n| **Community effort** | Reporter provided detailed repro scripts, root-cause analysis, or proposed fixes that went unacknowledged |\n\n### 4. Prioritize\n\nAssign priority tiers:\n\n- **P0 — Critical:** Silent data corruption, cascading failures, confirmed root cause with no fix. Production systems at risk.\n- **P1 — High:** Hard crashes in common deployment patterns, blocking features for significant user segments, confirmed bugs with stalled fixes.\n- **P2 — Medium:** Broken features in specific environments, intermittent failures, missing warnings that lead users astray.\n- **P3 — Low:** Documentation gaps, environment-specific issues with known workarounds, feature requests with community PRs pending review.\n\nWithin each tier, rank by:\n1. Seriousness of the problem in production (data loss > crash > degraded performance > inconvenience)\n2. Negative community impact if not solved (effort wasted by reporters, users hitting the issue independently)\n3. Amount of effort already invested by reporters (repro scripts, root-cause analysis, proposed PRs)\n\n### 5. Output format\n\nPresent results as:\n\n#### Per-issue analysis (grouped by priority tier)\nFor each issue:\n- **Issue link, title, comment count, last activity date**\n- **Problem:** 2-3 sentence description of what breaks\n- **Why this priority:** key severity signal\n- **Abandonment signal:** when maintainer last engaged, what was left unresolved\n- **Community effort:** what the reporter invested (repro scripts, analysis, PRs)\n\n#### Summary matrix\nA table with columns: Priority | Issue # | Type | Production Impact | Community Effort | Maintainer Status\n\n#### Cross-cutting patterns\nNote any clusters of related issues (e.g., multiple issues sharing a root cause) or systemic patterns in maintainer response.\n","frontmatter":{"name":"review-issues-severity","description":"Find and prioritize open GitHub issues by severity, community impact, and maintainer abandonment","argument-hint":["-- <optional filters or focus areas>"]},"isInternal":false,"tokens":1064,"sizeBytes":4535},{"name":"identifier-normalization.md","path":".claude/skills/review-pr/identifier-normalization.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-pr/identifier-normalization.md","title":"Identifier Normalization Check","category":"anthropic-skill","format":"markdown","content":"---\nname: Identifier Normalization Check\ndescription: Ensures all table names, column names, and other identifiers pass through the schema naming convention normalizer — the single most common PR review finding in this repo.\n---\n\n# Identifier Normalization Check\n\n## Context\n\nDifferent destinations have incompatible naming rules (e.g., Athena/S3 Tables forbids leading underscores, Weaviate capitalizes identifiers, BigQuery has length limits). All identifiers must go through the schema's naming normalizer — raw string constants are never acceptable for table or column names in runtime code.\n\n## When to Check\n\nThose rules apply to INTERNAL `dlt` code that is intended to work on ANY destination. In that case raw identifiers (string literals) MUST be normalized.\n\n\nYou must known when you normalize RAW identifier that represents a table or column name vs. when you RE-NORMALIZE already normalized identifier - take it from `Schema` instance, `Dataset` or `Relation` and which comes in a Python variable.\n\nNote: nested tables `user__comments` and nested columns `issue__stat_count` are PATHS containing many ALREADY NORMALIZED identifiers.\n\nWhen re-normalizing (and WHEN IN DOUBT)\nUse \n- `normalize_tables_path`\n- `normalize_path`\n\nWhen normalizing a RAW identifier:\n\nUse \n- `normalize_table_identifier`\n- `normalize_identifier`\n\n## What to Check\n\n### 1. Raw String Identifiers\n\nLook for hardcoded table or column name strings used in SQL queries, schema lookups, or API calls without normalization:\n\n```python\n# BAD — raw string identifier\ntable_name = \"my_table\"\nsql = f\"SELECT * FROM {table_name}\"\n\n# GOOD — normalized through schema naming\ntable_name = schema.naming.normalize_table_identifier(\"my_table\")\nsql = f\"SELECT * FROM {sql_client.make_qualified_table_name(table_name)}\"\n```\n\nCode in `dlt/dataset/` that builds queries or accesses tables must normalize identifiers:\n\n```python\n# BAD\ndataset[\"_dlt_loads\"]\n\n# GOOD\ndataset[schema.naming.normalize_table_identifier(\"_dlt_loads\")]\n```\n\nIn `dlt/destinations/impl/*/`:\n\n- Table names passed to SQL must use `sql_client.make_qualified_table_name()`\n- Column names in SQL must be escaped/normalized\n- Internal dlt tables (like `_dlt_loads`, `_dlt_version`, `_dlt_pipeline_state`) must still be normalized\n\n### 2. Re-normalizing identifiers\nUse this path when in doubt!\nWhen re-normalizing identifiers (ie. naming convention changed) or when identifier is a variable you MUST use:\n* `normalize_tables_path` - for table identifiers\n* `normalize_path` - for anything else ie. columns\n\nExample: a pyarrow table in transfer from destination 1 (now a source) to destion 2. Columns are normalized per\ndestination 1 and we re-normalize for destination 2:\n\n```python\ndef get_normalized_arrow_fields_mapping(schema: pyarrow.Schema, naming: NamingConvention) -> StrStr:\n \"\"\"Normalizes schema field names and returns mapping from original to normalized name. Raises on name collisions\"\"\"\n # use normalize_path to be compatible with how regular columns are normalized in dlt.Schema\n norm_f = naming.normalize_path\n```","frontmatter":{"name":"Identifier Normalization Check","description":"Ensures all table names, column names, and other identifiers pass through the schema naming convention normalizer — the single most common PR review finding in this repo."},"isInternal":false,"tokens":675,"sizeBytes":3081},{"name":"new-destination-checklist.md","path":".claude/skills/review-pr/new-destination-checklist.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-pr/new-destination-checklist.md","title":"Review-pr Skill","category":"anthropic-skill","format":"markdown","content":"# New destination review checklist\n\nUse this checklist when reviewing a PR that adds a new dlt destination. These items are in addition to the standard review steps.\n\n## 1. Shared destination test integration\n\n- Check `tests/load/utils.py` `destinations_configs()` — is the new destination included in the right config groups (`default_vector_configs`, `default_sql_configs`, `read_only_sqlclient_configs`, etc.)?\n- Check `tests/load/test_read_interfaces.py` — does the destination need special handling (chunk size skips at `_chunk_size()`, view-only table listing exclusions in `test_ibis_dataset_access`)?\n- Check `tests/load/pipeline/test_restore_state.py` — state sync tests run via the config groups above, verify the destination participates.\n- Check `tests/utils.py` — is the destination in `IMPLEMENTED_DESTINATIONS` and `NON_SQL_DESTINATIONS` (if applicable)?\n\n## 2. Capabilities configuration\n\n- Verify `_raw_capabilities()` in the factory: loader formats, merge strategies, replace strategies, type mapper, nested types, decimal precision, timestamp precision, recommended file size.\n- Compare to the blueprint destination (ducklake, lancedb, filesystem) — are any caps missing or misconfigured?\n- Check `parquet_format` settings if the destination writes parquet with nested types.\n\n## 3. `WithLocalFiles` integration\n\n- If the destination stores data locally (files, embedded databases), the top-level `DestinationClient*Configuration` **must** inherit `WithLocalFiles` — otherwise the pipeline cannot bind `pipeline_name`, `pipeline_working_dir`, `local_dir` into the config.\n- If storage config is nested (e.g. `storage: FilesystemConfiguration`), `on_resolved()` must call `self.storage.attach_from(self)` followed by resolve or `normalize_bucket_url()`. Reference: ducklake pattern in `ducklake/configuration.py`.\n- **Test it**: verify default relative paths resolve to `local_dir`, not cwd. Verify `pipeline_name` and `pipeline_working_dir` propagate to nested storage config.\n\n## 4. SQL client — `WithTableScanners`\n\n- If the destination provides a read-only SQL interface via DuckDB views (like lance, lancedb, filesystem), it should extend `WithTableScanners` (`dlt/destinations/impl/duckdb/sql_client.py`), not `DuckDbSqlClient` directly.\n- `WithTableScanners` provides: in-memory DuckDB management, schema creation, sqlglot-based lazy view loading, `create_views_for_all_tables()` for ibis offload.\n- The subclass implements three abstract methods: `create_view()`, `can_create_view()`, `should_replace_view()`.\n\n## 5. Ibis integration\n\n- Check `dlt/helpers/ibis.py` — is there a branch for the new destination's config class?\n- For `WithTableScanners`-based destinations: use `sql_client.create_views_for_all_tables()` (not manual table iteration).\n- Add the destination to the view-only exclusion list in `test_ibis_dataset_access` if it uses DuckDB views (won't see tables from other schemas).\n- The ibis branch must come before any parent config class check to avoid wrong dispatch via `issubclass`.\n\n## 6. Import rules\n\n- Never import optional packages directly — use `dlt.common.libs.*` wrappers (`pyarrow`, `pandas`, `numpy`, etc.).\n- Private API imports from third-party libs (underscore-prefixed functions) are fragile — flag them.\n","isInternal":false,"tokens":736,"sizeBytes":3285},{"name":"SKILL.md","path":".claude/skills/review-pr/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-pr/SKILL.md","title":"review-pr","category":"anthropic-skill","format":"markdown","content":"---\nname: review-pr\ndescription: Analyze a GitHub pull request including diff, comments, related issues, and local code context\nargument-hint: <pr-url-or-number> [-- <optional instructions>]\n---\n\n# Review Pull Request\n\nParse `$ARGUMENTS` to extract the PR identifier and optional reviewer instructions:\n- Everything before the first `--` (or all of it, if there is no `--`) is the **PR identifier** (URL or number). Referred to as `PR_ID` below.\n- Everything after the first `--` is **reviewer instructions** — additional focus areas or context from the caller. If absent, there are no special instructions.\n\nAnalyze the pull request: PR_ID\n\n## Steps\n\nRun the following steps. Gather data in parallel where possible.\n\n### 1. Set up worktree\n\nResolve the PR number:\n\n```\ngh pr view PR_ID --json number\n```\n\nUse the number to form the worktree name `review-pr-<number>`.\n\nInvoke the `/create-worktree review-pr-<number> --pr <number>` skill to create an isolated worktree for this PR. Note the worktree path from the skill output — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 2. Fetch PR metadata and diff\n\nUse `gh` to collect all PR information in parallel:\n\n- `gh pr view PR_ID --json title,body,author,state,baseRefName,headRefName,number,url,comments,reviews,labels,milestone`\n- `gh pr diff PR_ID`\n\n### 3. Check dependency changes in `pyproject.toml`\n\nIf the PR diff (from step 2) touches `pyproject.toml`, run the automated dependency checker.\n\nFirst, fetch the latest `devel` from origin so the comparison is against the current state:\n\n```\ngit fetch origin devel\n```\n\nThen run the script — always compare against `origin/devel` regardless of the PR's base branch:\n\n```\npython tools/check_dependency_changes.py origin/devel\n```\n\nThe script finds the merge-base, simulates a three-way merge (like GitHub), and reports:\n- **Main dependencies** (`[project.dependencies]`) — displayed as WARNING (these affect every user)\n- **Optional dependencies** (`[project.optional-dependencies]`) — displayed as WARNING (these affect users of specific extras)\n- **Dev/group dependencies** (`[dependency-groups]`) — listed for awareness (no impact on end users)\n- **Merge conflicts** — if `pyproject.toml` would conflict, the script warns and falls back to showing the PR's intended changes\n\nInclude the script output verbatim in the review. If the script exits with code 1 (changes detected), flag dependency changes in the review verdict. If it exits with code 2 (conflicts), flag that `pyproject.toml` needs conflict resolution.\n\nIf `pyproject.toml` is not in the diff, skip this step.\n\n### 4. Check CI lint and mypy status\n\nUse `gh pr checks PR_ID` to retrieve the CI check results. Look for checks matching `lint on all python versions / lint*` — these run mypy, ruff, flake8, bandit, and docstring checks across Python 3.9–3.13 (defined in `.github/workflows/lint.yml`).\n\n- Report the status of each lint check (pass/fail/pending).\n- If any lint check has failed, fetch its logs with `gh run view <run-id> --log-failed` to identify the specific errors (mypy type errors, ruff violations, etc.).\n- Include lint/mypy status in the final review verdict. A failing lint check should be flagged as a required fix.\n\n### 5. Analyze related issues\n\nExtract any issue references from the PR body (e.g. `fixes #1234`, `closes #1234`, `#1234`). For each referenced issue, fetch it with:\n\n- `gh issue view <number> --json title,body,author,state,comments,labels`\n\nUnderstand the original problem being solved.\n\n### 6. Analyze the code\n\nUsing absolute paths under `WORKTREE`:\n\n- Read the changed files in full to understand context beyond the diff.\n- Look at callers/callees of modified functions.\n- Check if similar patterns exist elsewhere that should also be updated.\n- Review the coding style and code patterns to conform with @CLAUDE.md\n- If **reviewer instructions** were provided, pay special attention to the areas or concerns they describe during this analysis.\n- Mention only crucial finding to the user in the initial overview. Avoid code snippets except if critical bugs were found\n\n### 6a. Pay attention to special cases\n\nSome areas of the codebase have rules that are easy to miss in a generic code read. The references below are intentionally not loaded by default — read the linked file ONLY when the PR's diff matches the trigger, and apply its checklist before producing the review.\n\n- **Identifier normalization** — read [identifier-normalization.md](identifier-normalization.md) when the PR's diff includes any of: `dlt/destinations/impl/**`, `dlt/dataset/**`, `dlt/destinations/sql_client*`, code that builds SQL/relations/lineage, or schema-generation paths involving sqlglot or ibis. Covers when raw-string table/column identifiers must pass through the schema's naming normalizer, and how to distinguish first-time normalization from re-normalization.\n\n### 7. Review test coverage\n\n#### 7a. Define the testing scope\n\nBased on the PR changes, identify what needs to be tested:\n\n- What are the distinct behaviors introduced or modified?\n- What are the edge cases and error paths?\n- What are the integration boundaries (e.g., does the change affect how components interact)?\n\nProduce a checklist of required test scenarios before evaluating existing tests.\n\n#### 7b. Evaluate existing tests against the checklist\n\n- Compare the PR's tests against the required scenarios checklist. Flag any gaps.\n- Flag any tests that verify things outside the PR's scope or test obvious/trivial behavior that adds no value.\n\n#### 7c. Verify tests are in the right modules\n\n- Search the test directory for where similar functionality is already tested (grep for related function names, class names, or feature keywords).\n- New tests should live alongside related existing tests, not in new files unless the PR introduces an entirely new module.\n- Flag tests placed in the wrong location.\n\n#### 7d. Check for duplication and over-testing\n\n- Look for tests that cover nearly identical cases and could be merged or parametrized with `@pytest.mark.parametrize`.\n- Look for repeated setup/assertion logic across tests that should be extracted into a helper or fixture.\n- If multiple tests differ only in a single variable (e.g., `True`/`False`, different type values), they must be parametrized into one test function.\n- Flag any copy-paste test code. Minimizing test code maintenance burden is a priority.\n- Flag local path assertions that are not Windows compatible.\n\n#### 7e. Bug Fixes Should Have Regression Tests\nIf the PR fixes a bug:\n- Is there a test that would have caught the bug before the fix?\n- Does the test verify the specific scenario from the bug report?\n\n### 8. Check documentation coverage\n\nIf the PR modifies public API (new parameters, changed behavior, new configuration options, new functions/classes exposed to users):\n\n- Search `WORKTREE/docs/website/docs/` for existing documentation covering the topic of the PR.\n- If the topic is already documented and the PR changes its behavior or adds new options, verify that the PR also updates the relevant doc pages.\n- If documentation updates are missing, flag this in the review as a required change.\n- If the topic is not documented at all, this is not a blocker — don't require docs for previously undocumented internals.\n\n### 9. Produce the review\n\nWrite a structured analysis covering:\n\n- **Summary**: What the PR does and why, referencing the linked issue.\n- **Changes**: Breakdown of changes should be concise, avoid code samples\n- **Dependency Changes**: If `pyproject.toml` was modified, include the warnings and listings from step 3. Main and optional dependency changes must appear as yellow WARNING blocks. Omit this section if no dependency changes were found.\n- **CI Lint & Mypy**: Status of lint checks across Python versions. If any failed, list the specific errors.\n- **Observations & Potential Issues**: Bugs, edge cases, missing error handling, performance concerns, memory issues, security issues, backward compatibility.\n- **Breaking Changes and Public API**: Follow the rules on breaking changes and Public API and report issues.\n- **Test Coverage**: Required scenarios checklist with pass/fail status, misplaced tests, duplication issues, parametrization opportunities.\n- **Documentation**: Whether doc updates are needed and if they were provided.\n- **Reviewer Instructions**: If reviewer instructions were provided, include a dedicated section addressing each point raised. If no instructions were provided, omit this section.\n- **Verdict**: Overall assessment.\n","frontmatter":{"name":"review-pr","description":"Analyze a GitHub pull request including diff, comments, related issues, and local code context","argument-hint":"<pr-url-or-number> [-- <optional instructions>]"},"isInternal":false,"tokens":1947,"sizeBytes":8816},{"name":"simple-english-checklist.md","path":".claude/skills/review-vocabulary/references/simple-english-checklist.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/simple-english-checklist.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Verification checklist\n\nRun this pass on every draft before you deliver it. The checks are ordered from mechanical to judgment.\n\n## Mechanical checks (searchable)\n\nSearch the draft for each pattern. Every hit outside code blocks and quoted text is a violation.\n\n| Search for | Violation | Fix |\n|---|---|---|\n| `'ll`, `'re`, `'ve`, `n't`, `it's` | Contraction (Rule 4.2) | Expand it. |\n| `has been`, `have been`, `had been` | Present/past perfect (Rule 3.4) | Simple past or simple present. |\n| `has` / `have` + past participle | Present perfect (Rule 3.4) | Simple past. |\n| `should`, `would`, `may`, `might`, `could` | Unapproved modal (Rule 3.2) | See the modal ladder in SKILL.md. |\n| `is being`, `are being`, `was being` | Progressive passive (Rules 3.4, 3.5) | Active, simple tense. |\n| `, making`, `, allowing`, `, enabling`, `, ensuring` | \"-ing\" clause as verb (Rule 3.5) | New sentence with a real subject. |\n| `;` | Semicolon (Rule 8.1) | Two sentences. |\n| `e.g.`, `i.e.`, `etc.` | Latin abbreviation (GR-6) | \"for example\", \"that is\", name the items. |\n| `simply`, `easily`, `seamlessly`, `robust` | Filler (no fact) | Delete. |\n| ` if `, ` when ` (mid-sentence) | Trailing condition (Rule 5.4) | Move the condition to the start of the sentence, add a comma. |\n\n## Countable checks\n\n1. **Sentence length.** Count words in each sentence. Procedural limit: 20. Descriptive limit: 25. Notes: 25.\n Backticked commands, numbers with units, and identifiers count as one word each (Rule 8.6).\n2. **Paragraph size.** Maximum six sentences per paragraph (Rule 6.6).\n3. **Multi-word nouns.** Any noun chain over three words → break it with prepositions (Rule 2.1).\n4. **Instructions per sentence.** One, unless the actions are simultaneous (Rule 5.2).\n\n## Judgment checks\n\n5. **Classification.** Is each passage cleanly procedural or descriptive? Procedures in imperative, descriptions never in imperative.\n6. **Voice.** Any passive sentence: is the agent truly unknown, and is the passage descriptive? Otherwise make it active (Rule 3.6).\n7. **Condition placement.** Every \"if/when\" stands before its command, with a comma (Rule 5.4).\n8. **Synonym rotation.** One term per concept across the whole document (Rules 1.11, 9.4). Scan for check/verify/confirm, config/settings, run/execute.\n9. **Warnings.** Command or condition first, risk second (Rules 7.2, 7.3).\n10. **Completeness.** Articles present, \"that\" present after \"make sure\", no telegraph style (Rule 4.2).\n11. **Untouchables intact.** Code, identifiers, quoted errors, and proper nouns are unchanged.\n\n## When reporting violations (check mode)\n\nFor each violation give: the rule number, the offending text, and a compliant rewrite. Cite only rule numbers that appear in rules.md.\nEnd the report with this statement when the user asked for STE compliance: \"No tool can guarantee ASD-STE100 compliance. Final approval rests with the writer. The official standard is a free download at asd-ste100.org.\"\n","isInternal":false,"tokens":813,"sizeBytes":2971},{"name":"simple-english-use-cases.md","path":".claude/skills/review-vocabulary/references/simple-english-use-cases.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/simple-english-use-cases.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Use cases beyond documentation\n\nSTE was built for aircraft maintenance manuals. The same properties — one meaning per word, short sentences, condition-first commands — transfer to any text where misreading has a cost. By the end of Issue 8, 64% of registered STE users were outside aerospace and defense.\n\nEach case below names the mode and the adaptations.\n\n## Error messages and CLI output\n\nMode: procedural. This is the highest-value target: an error message is a 2 a.m. instruction to a stressed reader.\n\nPattern: state what happened (past simple), state the cause if known, give the command or condition to fix it.\n\n> **Before:** Oops! Something went wrong while attempting to establish a connection. Please ensure your credentials are properly configured and try again.\n> **After:** Connection to the database failed. The password for user `app` was not correct. Set `DB_PASSWORD` and connect again.\n\n## Runbooks and standard operating procedures\n\nMode: strict-leaning procedural. This is STE's home turf — an on-call runbook is a maintenance manual.\n\n- Every step imperative, one instruction per step, conditions first.\n- Warnings before the step, command first, risk second.\n- 20-word limit enforced hard: an operator under pager stress reads each sentence once.\n\n## Incident reports and postmortems\n\nMode: descriptive. Simple past only — a timeline in present perfect (\"we have identified...\") hides when things happened.\n\n> **Before:** We have identified an issue that may have impacted some users' ability to access the service.\n> **After:** Between 14:02 and 14:31 UTC, 12% of requests failed. A deploy at 14:00 removed the cache warmup step.\n\nSTE bans hedges (\"may have impacted\") — the report states what is known and says \"unknown\" for the rest. This reads more honest because it is.\n\n## Commit messages and PR descriptions\n\nMode: descriptive body, imperative subject. Convention already matches STE: imperative subject line, plain past facts in the body. Apply the substitution table and the 25-word limit to the body. Delete \"this PR aims to\".\n\n## API changelogs and release notes\n\nMode: descriptive. One entry, one change, one sentence where possible. \"Breaking:\" entries follow the warning pattern — command first: \"Update your calls to `v2/users`. The `name` field split into `first_name` and `last_name`.\"\n\n## Instructions for AI agents (prompts, AGENTS.md, skills)\n\nMode: procedural. A system prompt is a procedure executed by a reader with no ability to ask questions — the exact reader STE was designed for.\n\n- One instruction per sentence keeps rules independently quotable and hard to half-follow.\n- One word, one meaning prevents the model from treating \"check\", \"verify\", and \"validate\" as three different operations.\n- Condition-first (\"If the build fails, stop\") beats trailing conditions, which models drop.\n- No \"should\" — a model reads \"should\" as optional. Write \"must\" or delete the rule.\n\n## Support macros and status-page updates\n\nMode: descriptive, 25-word limit. Non-native readers are the majority of many user bases. No \"we sincerely apologize for any inconvenience this may have caused\" — \"The API was down for 18 minutes. Uploads made during this time were saved and will process today.\"\n\n## Translation and localization prep\n\nMode: strict. STE's original purpose was making English readable for non-native maintenance crews, and it doubles as pre-editing for machine translation. One meaning per word plus complete grammar (articles, \"that\") removes most translation ambiguity. If your docs get localized, STE cuts the error rate and the cost.\n\n## UI copy and empty states\n\nMode: procedural, hard length limits. Buttons and labels are technical names (exempt). Body copy follows the rules: \"No projects yet. Create a project to start.\" Nothing else survives at this length anyway.\n\n## Where STE does not fit\n\nMarketing pages, launch posts, blog voice, brand writing. STE deletes persuasion on purpose. Write those in your own voice — then use STE for the docs the landing page links to.\n","isInternal":false,"tokens":857,"sizeBytes":4053},{"name":"simple-english.md","path":".claude/skills/review-vocabulary/references/simple-english.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/simple-english.md","title":"simple-english","category":"anthropic-skill","format":"markdown","content":"---\nname: simple-english\nversion: 1.0.0\ndescription: |\n Write or rewrite technical text with the rules of ASD-STE100 Simplified\n Technical English so it is clear, unambiguous, and free of AI slop. Use for\n documentation, READMEs, runbooks, procedures, error messages, release notes,\n incident reports, and API guides. Also use when the user says \"STE\",\n \"Simplified Technical English\", \"ASD-STE100\", \"de-slop\", \"make this\n readable\", \"write for non-native readers\", or asks for docs that translate\n well. Enforces the standard's 53 rules: 20/25-word sentence limits, one word\n one meaning, simple tenses, active voice, condition before command.\nlicense: MIT\ncompatibility: claude-code cursor codex gemini-cli opencode\nmetadata:\n standard: ASD-STE100 Issue 9 (2025-01-15)\n---\n\n# Simple English: Write Like an Aerospace Manual\n\nWrite technical text with the rules of ASD-STE100 Simplified Technical English. STE is the controlled language that aerospace and defense manufacturers use for maintenance documentation. The rules exist so that a tired reader who is not a native English speaker cannot misread an instruction. They remove the usual signs of AI-generated text as a side effect: long sentences, synonym rotation, hedges, filler, and decorative clauses.\n\nWrite for that tired reader. Each sentence must survive one read.\n\n## Your Task\n\nWhen asked to write or rewrite technical text:\n\n1. **Select the mode** (pragmatic or strict, below).\n2. **Classify each passage** as procedural or descriptive. Every other rule depends on this.\n3. **Fix your vocabulary before drafting.** Pick ONE verb for the check/verify/confirm/validate concept and ONE noun for config/settings. Use no other word for these concepts in the whole document.\n4. **Apply the rules** from the catalog below.\n5. **Run the self-check** before you deliver. This step is not optional.\n6. **Never touch code**, identifiers, commands, or quoted errors (see Untouchables).\n\nWhen asked to CHECK text instead of writing it, report each violation as: rule number, the offending text, a compliant rewrite. Cite only rule numbers that exist in this file. Do not cite rule numbers from memory: the numbering is unintuitive and models invent it (tested — an agent without this file cited \"Rule 3.1: short sentences\"; the real Rule 3.1 is about verb forms).\n\n## Two Modes\n\n| Mode | When | What you apply |\n|---|---|---|\n| **Pragmatic** (default) | Docs, READMEs, error messages — the user wants clear text | All structural rules. Domain words stay (\"idempotent\", \"webhook\"). |\n| **Strict** | The user names STE, ASD-STE100, or compliance | Structural rules + full vocabulary discipline, and tell the user that full compliance needs the official dictionary (free at asd-ste100.org). |\n\n## Step 1: Classify the Text\n\n| | Procedural (instructions) | Descriptive (explanations) |\n|---|---|---|\n| Purpose | Tell the reader what to do | Explain what a thing is or does |\n| Verb form | Imperative: \"Install the pump.\" | Simple present/past/future |\n| Sentence limit | **20 words** (Rule 5.1) | **25 words** (Rule 6.3) |\n| Unit rule | One instruction per sentence (5.2) | One topic per paragraph (6.5), max six sentences per paragraph (6.6) |\n\nDo not mix the two in one passage. A \"Getting started\" section is procedural. An \"Architecture\" section is descriptive. A note inside a procedure is descriptive (25-word limit, no imperative).\n\n## THE RULE CATALOG\n\n53 rules in 9 sections, paraphrased from ASD-STE100 Issue 9 with software examples. The official wording is in the free standard at asd-ste100.org.\n\n### Section 1 — Words (Rules 1.1-1.14)\n\n| Rule | Instruction |\n|---|---|\n| 1.1 | Use only approved words, technical nouns, or technical verbs. |\n| 1.2 | Use an approved word only as its listed part of speech. |\n| 1.3 | Use an approved word only with its approved meaning. |\n| 1.4 | Use only the approved forms of verbs and adjectives. |\n| 1.5 | You can use domain words as technical nouns (\"webhook\", \"commit\", \"endpoint\"). |\n| 1.6 | Use an unapproved word only when it is a technical noun or part of one. |\n| 1.7 | Do not use technical nouns as verbs. |\n| 1.8 | Use the technical nouns of your project or industry. |\n| 1.9 | When you pick a technical noun, pick a short and clear one. |\n| 1.10 | No regional, slang, or jargon words as technical nouns. |\n| 1.11 | One item, one name. Do not call it \"config\" here and \"settings\" there. |\n| 1.12 | You can use domain verbs as technical verbs (\"deploy\", \"compile\", \"merge\"). |\n| 1.13 | Do not use technical verbs as nouns. |\n| 1.14 | Use American English spelling. |\n\nIn pragmatic mode, rules 1.5, 1.8, and 1.12 do the heavy lifting: your domain vocabulary is legal. The ones agents break are 1.7, 1.11, and 1.13.\n\n**Before:** You can webhook the event, then do a deploy.\n**After:** Send the event to the webhook. Then deploy the service.\n\n### Section 2 — Multi-word nouns (Rules 2.1-2.2)\n\n| Rule | Instruction |\n|---|---|\n| 2.1 | Write multi-word nouns of three words or fewer. |\n| 2.2 | When a technical noun needs more than three words, write it in full once, then give a short form or hyphenate the units. |\n\nBreak long noun chains with prepositions (of, on, in, for):\n\n**Before:** the connection pool timeout configuration value\n**After:** the timeout value for the connection pool\n\n### Section 3 — Verbs (Rules 3.1-3.7)\n\n| Rule | Instruction |\n|---|---|\n| 3.1 | Use only the verb forms that the dictionary gives. |\n| 3.2 | Use only: infinitive, imperative, simple present, simple past, simple future, past participle as adjective. |\n| 3.3 | Use the past participle only as an adjective (\"the cached response\"). |\n| 3.4 | No auxiliary verbs for complex constructions. No present perfect, no \"is to be installed\". |\n| 3.5 | Use an \"-ing\" form only as a technical noun or inside one (\"logging\", \"the mounting bracket\") — never as a verb. |\n| 3.6 | Active voice. In descriptive text, passive is legal only when the agent is unknown. |\n| 3.7 | Describe an action with a verb, not a noun (\"compress the file\", not \"perform compression of the file\"). |\n\n**Approved modals: can, will, must. Banned: should, would, may, might, could.**\nThe standard rejects \"could\" even for possibility: write \"an explosion can occur\", never \"could occur\". For \"should\": a requirement becomes \"must\"; a suggestion is stated as fact or deleted. This matters double for agent instructions — models read \"should\" as optional.\n\n**Before:** The migration has completed and the table is being rebuilt.\n**After:** The migration is complete. The database rebuilds the table.\n\n**Before:** The flag can be set in the config file, making restarts unnecessary.\n**After:** You can set the flag in the config file. Then a restart is not necessary.\n\n**Before:** The temperature must be adjusted.\n**After:** Adjust the temperature.\n\n### Section 4 — Sentences (Rules 4.1-4.5)\n\n| Rule | Instruction |\n|---|---|\n| 4.1 | Write short and clear sentences. |\n| 4.2 | Do not omit words or use contractions to shorten sentences. Keep articles, keep \"that\". |\n| 4.3 | Use a vertical list for complex text. |\n| 4.4 | Use connecting words between sentences on related topics (\"Then\", \"As a result\"). |\n| 4.5 | Put an article (the, a, an) or a demonstrative adjective (this, these) before nouns where applicable. |\n\nRule 4.2 is the anti-terseness rule. STE is short sentences with complete grammar, not telegraph style:\n\n**Wrong shortening:** Ensure file exists before running.\n**STE:** Make sure that the file exists before you run the command.\n\n### Section 5 — Procedural writing (Rules 5.1-5.5)\n\n| Rule | Instruction |\n|---|---|\n| 5.1 | Maximum 20 words per sentence. Warnings and cautions included. |\n| 5.2 | One instruction per sentence, unless two actions happen at the same time. |\n| 5.3 | Write instructions in the imperative: \"Run the migration.\" |\n| 5.4 | Put a required condition before the command, divided by a comma: \"If the build fails, read the log.\" |\n| 5.5 | Notes give information, never instructions. Notes get the 25-word limit. |\n\n**Before:** You'll want to grab the API key from the dashboard before configuring the client, which you can do under Settings.\n**After:** Get the API key from the dashboard, under Settings. Then configure the client with this key.\n\n### Section 6 — Descriptive writing (Rules 6.1-6.6)\n\n| Rule | Instruction |\n|---|---|\n| 6.1 | Give information gradually: one new fact per sentence. |\n| 6.2 | Use key words and phrases to give the text a logical structure. |\n| 6.3 | Maximum 25 words per sentence. |\n| 6.4 | Group related information in paragraphs. |\n| 6.5 | One topic per paragraph. |\n| 6.6 | Maximum six sentences per paragraph. |\n\nNo imperative in descriptive text. Descriptions explain; procedures instruct.\n\n### Section 7 — Safety instructions (Rules 7.1-7.3)\n\n| Rule | Instruction |\n|---|---|\n| 7.1 | Use a word that shows the risk level (\"WARNING\" = injury, \"CAUTION\" = damage). |\n| 7.2 | Start with a clear command or condition. |\n| 7.3 | Then give the risk or the possible result. |\n\nNever bury the instruction after the explanation. The pattern transfers directly to destructive CLI flags, irreversible migrations, and dangerous API options.\n\n**Before:** Note that data loss may occur in some circumstances if the destructive flag happens to be enabled when running against production.\n**After:** CAUTION: Do not use the `--force` flag against production. The flag deletes rows that do not match the source.\n\n### Section 8 — Punctuation and word count (Rules 8.1-8.7)\n\n| Rule | Instruction |\n|---|---|\n| 8.1 | All standard punctuation is legal except the semicolon. Write two sentences instead. |\n| 8.2 | Use hyphens to connect words that act as one unit. |\n| 8.3 | Parentheses are legal for references, item numbers, abbreviations, plural forms, explanations, alternatives. |\n| 8.4 | In a vertical list, the lead-in colon ends a sentence for word count. |\n| 8.5 | Text inside parentheses counts as one word. |\n| 8.6 | Count as one word each: numbers, numbers with units, abbreviations, alphanumeric identifiers, quoted text, titles, labels, proper nouns. |\n| 8.7 | A hyphenated word counts as one word. |\n\nRule 8.6 matters for software text: `sqlpipe run --config sqlpipe.yaml` in backticks is quoted text and counts as one word. Long identifiers do not blow your sentence budget.\n\n### Section 9 — Writing practices (Rules 9.1-9.4, GR-1 to GR-8)\n\n| Rule | Instruction |\n|---|---|\n| 9.1 | When a word-for-word replacement does not work, restructure the sentence. |\n| 9.2 | Use each approved word correctly: approved meaning, approved part of speech. |\n| 9.3 | Do not build phrasal verbs (\"go down\" → \"decrease\", \"set up\" → \"install\" or \"configure\"). |\n| 9.4 | Keep one consistent style and terminology through the whole document. |\n\nGeneral recommendations GR-1 to GR-8: keep the conjunction \"that\", be careful with \"with\", give pronouns clear referents, prefer \"this + noun\" over bare \"this\", avoid false friends, avoid Latin abbreviations, use inclusive language, and use the possessive apostrophe form only when you are sure it is correct (GR-8: if unsure, do not use it — non-native readers find it hard).\n\nGR-6 for software docs: \"e.g.\" → \"for example\", \"i.e.\" → \"that is\", and delete \"etc.\" — name the items or write \"and more\".\n\n## VOCABULARY DISCIPLINE\n\nThe official dictionary (~900 approved words, ~1,200 banned words with alternatives) is copyrighted by ASD and is not reproduced here. Its mechanics apply without it: **one word, one meaning, one part of speech.**\n\nKnown part-of-speech rulings, useful as patterns:\n\n| Word | Ruling |\n|---|---|\n| test, check, work | Noun only. \"Do a test\", not \"test the pump\". \"Check that X\" becomes \"make sure that X\". |\n| oil | Noun only as used in STE examples. For the verb, the dictionary gives \"lubricate\". |\n| help | Verb only. For the noun, the dictionary gives \"aid\": \"with the aid of\". |\n| fall | \"To move down by gravity\" only, never \"decrease\". |\n| follow | \"To come after\" only, never \"obey\". Write \"obey the instructions\". |\n| above, below | Physical positions only. For limits write \"more than\", \"less than\". |\n\n### The modal ladder\n\n| You wrote | STE writes |\n|---|---|\n| should (requirement) | must |\n| should (recommendation) | Delete it, or state it as fact: \"X is better because Y.\" |\n| may / might / could (possibility) | can |\n| may (permission) | can |\n| would (hypothetical) | Restructure: \"If X occurs, Y occurs.\" |\n\n### Slop-to-simple substitutions\n\nThis table is ours, not the ASD dictionary. It maps the words AI-generated docs overuse to plain replacements. If the word carries no fact, delete it instead of replacing it.\n\n| Slop | Write instead |\n|---|---|\n| leverage, utilize | use |\n| in order to | to |\n| prior to | before |\n| ensure | make sure that |\n| it is worth noting that | (delete) |\n| it's important to, crucially | (delete — state the fact) |\n| simply, just, easily, seamlessly, effortlessly | (delete) |\n| robust, powerful, comprehensive, performant | (delete, or give the measurable property) |\n| functionality | function, feature |\n| enables you to, allows you to | you can |\n| is designed to, aims to | (delete — say what it does) |\n| facilitate | help, make possible |\n| dive into, delve into | read, examine |\n| when it comes to | for |\n| in the event that | if |\n| due to the fact that | because |\n| as needed, as necessary | (state the condition) |\n| and/or | Pick one, or write \"X, or Y, or both\" |\n| e.g. / i.e. / etc. | for example / that is / (name the items) |\n| gracefully handles | (say what it does: \"retries three times, then stops\") |\n| out of the box | by default |\n| under the hood | internally |\n| blazingly fast, state-of-the-art | fast (give the number) / (delete) |\n| streamline | make simpler, make faster |\n| plethora, myriad | many |\n| addresses the issue, tackles | corrects the fault, removes the error |\n\n### Consistency pass\n\nCollapse these common rotations to one term each (Rules 1.11, 9.4):\n\n- check / verify / confirm / validate / ensure → pick one\n- config / configuration / settings / options → pick one\n- delete / remove / drop / destroy → one per meaning, kept consistent\n- error / issue / problem / failure → \"error\" for errors, \"failure\" for failed operations\n- run / execute / invoke / launch → pick one\n- show / display / render / present → pick one\n\n## Untouchables\n\nThese are technical names (Rules 1.5, 8.6). Leave them exact, even when they break vocabulary rules:\n\n- Code blocks, inline code, identifiers, CLI commands, flags, file paths\n- Quoted error messages and log lines\n- Product names, API endpoint names, config keys\n- Numbers with units — each counts as one word in the sentence limit\n\n## Beyond Documentation\n\nSame rules, different targets. Full adaptations in `references/use-cases.md`:\n\n- **Error messages**: state what happened (simple past), the cause if known, then the fix as an imperative. No \"Oops\", no \"Please ensure\", no apology filler.\n- **Runbooks**: STE's home turf. Imperative steps, conditions first, warnings before the step.\n- **Incident reports**: simple past only. \"We have identified an issue that may have impacted\" becomes \"Between 14:02 and 14:31 UTC, 12% of requests failed.\"\n- **Release notes**: breaking changes follow the warning pattern — command first, risk second.\n- **Agent instructions (prompts, AGENTS.md)**: a system prompt is a procedure for a reader that cannot ask questions. One instruction per sentence, no \"should\", condition first.\n- **Translation prep**: STE's original job. One meaning per word plus complete grammar removes most translation ambiguity.\n\n## Self-Check Before You Deliver\n\nThis step is not optional. Run these four checks on your draft:\n\n1. Count words in your three longest sentences. Over the 20/25 limit → split them.\n2. Search your draft for: `'ll`, `'re`, `'s` (contraction), `has been`, `have been`, `should`, `-ing` verbs after a comma, semicolons.\n3. Search for every `if` and `when`. Each one stands at the START of its sentence, before the command. \"Increase the timeout if the network is slow\" → \"If the network is slow, increase the timeout.\"\n4. Search for the verbs you did NOT pick in Your Task step 3 (the check/verify/confirm set). Replace every hit with your chosen verb.\n\nFix what you find, then deliver. For a full audit, run `references/checklist.md`.\n\n## Full Example\n\n**Before (real unedited AI output):**\n\n> **Connection timeouts.** If sqlpipe hangs or fails with `dial tcp: i/o timeout`, check that the host running sqlpipe can reach the Postgres port (usually 5432) — this is often a security group or firewall rule blocking the connection. If you're connecting to a managed database (RDS, Cloud SQL, etc.), confirm the instance allows connections from sqlpipe's IP. You can also try increasing `source.connect_timeout_seconds` in your config, since a slow network path can trip the default timeout even when the connection eventually succeeds.\n\n**After (classified procedural, verb = \"make sure\", conditions first, one instruction per sentence):**\n\n> **Connection timeouts.** sqlpipe stops with `dial tcp: i/o timeout` when it cannot reach the Postgres port (5432 by default).\n>\n> 1. Make sure that the host that runs sqlpipe can reach the Postgres port. A firewall or security group usually blocks it.\n> 2. If the database is managed (RDS, Cloud SQL), make sure that the instance accepts connections from the IP of sqlpipe.\n> 3. If the network is slow, increase `source.connect_timeout_seconds` in the configuration.\n\nWhat changed: 40-word sentences split under 20; \"you're\" expanded; \"check/confirm\" collapsed to \"make sure that\"; every condition moved before its command; \"etc.\" removed; code and error strings untouched.\n\n## Limits\n\nSTE is for technical facts and instructions. Do not apply it to marketing copy, blog voice, or brand writing — it deletes persuasion by design. When a user asks for STE on marketing text, say so and offer it for the docs instead.\n\nThis skill is an unofficial aid. It is not affiliated with or endorsed by ASD or STEMG, and no tool can guarantee STE compliance. ASD-STE100 is a registered trademark of ASD. The official standard is a free download at asd-ste100.org.\n\n## References\n\n- `references/checklist.md` — full verification pass with searchable patterns, for check mode and final audits\n- `references/use-cases.md` — long-form adaptations: error messages, runbooks, incident reports, commits, UI copy, i18n\n","frontmatter":{"name":"simple-english","version":"1.0.0","description":"Write or rewrite technical text with the rules of ASD-STE100 Simplified\nTechnical English so it is clear, unambiguous, and free of AI slop. Use for\ndocumentation, READMEs, runbooks, procedures, error messages, release notes,\nincident reports, and API guides. Also use when the user says \"STE\",\n\"Simplified Technical English\", \"ASD-STE100\", \"de-slop\", \"make this\nreadable\", \"write for non-native readers\", or asks for docs that translate\nwell. Enforces the standard's 53 rules: 20/25-word sentence limits, one word\none meaning, simple tenses, active voice, condition before command.\n","license":"MIT","compatibility":"claude-code cursor codex gemini-cli opencode","metadata":{"standard":"ASD-STE100 Issue 9 (2025-01-15)"}},"isInternal":false,"tokens":4650,"sizeBytes":18477},{"name":"adding-terms.md","path":".claude/skills/review-vocabulary/references/vocabulary/adding-terms.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/adding-terms.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# Adding a new term or a new group\n\nNever just record a term. Take these eight steps.\n\n1. **Pick the group.** No group fits? Add one. Give it the next `G<n>`, a file in this folder named\n after its heading, an index row in `SKILL.md`, and the three parts: included, excluded, rulings.\n A group without excluded terms is not finished.\n2. **Research the usage.** Grep `dlt/`, `tests/` and `docs/` for the word and every synonym. Count\n the hits. Read enough to find the meanings in play. One word often covers two concepts.\n3. **Derive the banned set.** A term is useless without one. For each synonym and inflection,\n decide: banned, or legal with another meaning? Both halves go in the group.\n4. **Check the upstream interface.** The vocabulary must not contradict a public dlt method or\n type name. `access` won because `needs_attach` already said \"accesses its data\". `attach info`\n won because the type is `TAttachInfo`.\n5. **Name the false positives.** `descriptor` is banned for `TAttachInfo` and correct for the\n Python descriptor protocol. Put the exception in the rulings, or the next run \"fixes\" it.\n6. **Fix the part of speech.** Say noun or verb, and ban the other use. `access` is a verb, so\n \"data access\" is a violation.\n7. **Test the ban before you write it.** Grep for the word you intend to exclude. Compliant prose\n already uses it? Then the ban is too wide. Narrow it to the meaning you mean. `inaccessible`\n and `foreign-folded` passed this check and stay legal. `casefold` and `attach instructions`\n failed it and are banned.\n8. **Update the group file.** Add rows to both tables. Add a ruling when the term has a legal\n exception. State what becomes newly compliant and what becomes newly non-compliant. A term that\n flips direction turns compliant prose into findings.\n","isInternal":false,"tokens":459,"sizeBytes":1818},{"name":"g1-data-access-and-locations.md","path":".claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G1 — Data access and locations\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| getting to data (verb) | **access** |\n| the adjective | **accessible**, **inaccessible** |\n| the negative, as a verb | **cannot access** |\n| where the data physically sits | **data location** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| reach, reaches, reachable, in reach of, out of reach, get to | one word for one concept — `access` |\n| physical location, physical destination, physical dataset (in prose) | say **data location**; the method `data_location()` keeps its name |\n\n**Rulings**\n\n- **`access` is a verb for the act, and a noun for what a job may touch.** Keep the verb for the\n act of reading data: \"the engine accesses the data\", not \"data access is one-way\". The noun is\n legal only in the sense G9 defines — the `access` declaration, and the grant that answers it.\n dlt's docs already write \"grant access\", \"read access\" and \"denied access\" 40 times over; that\n usage was always compliant and stays.\n- **`reach` is not always `access`.** \"`SET SESSION` would not reach the cloned sessions\" means\n *propagate to*, not *read data from*. A literal swap changes the meaning — restructure\n (Rule 9.1).\n","isInternal":false,"tokens":315,"sizeBytes":1212},{"name":"g2-attach-and-foreign-datasets.md","path":".claude/skills/review-vocabulary/references/vocabulary/g2-attach-and-foreign-datasets.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g2-attach-and-foreign-datasets.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G2 — Attach and foreign datasets\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the `TAttachInfo` object | **attach info** |\n| the `TAttachStatement` object | **attach statement** |\n| the SQL keyword | `` `ATTACH` `` in backticks |\n| the action in prose | **attach** (lowercase, a verb) |\n| the catalog a foreign dataset lands under | **attach alias** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| descriptor (for `TAttachInfo`) | `attach info` matches the type and the method `_attach_infos()` |\n| bare `ATTACH` as a prose noun, `ATTACHed`, \"attaches\" as a plural noun | backtick the keyword, or use the verb |\n| attach instructions | one name — **attach statements** |\n\n**Rulings**\n\n- **`descriptor` is legal for the Python descriptor protocol.** `dlt/common/utils.py` describes a\n real Python descriptor. The ban covers naming the `TAttachInfo` object only.\n- **`attach info` and `attach statement` are different things.** One is the whole descriptor for a\n foreign dataset; the other is a single SQL statement inside it. Do not collapse them.\n","isInternal":false,"tokens":271,"sizeBytes":1058},{"name":"g3-transformations-and-materialization.md","path":".claude/skills/review-vocabulary/references/vocabulary/g3-transformations-and-materialization.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g3-transformations-and-materialization.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G3 — Transformations and materialization\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the deferred path | **lazy materialization** |\n| the load job that runs it | **model job** |\n| the immediate path | **eager materialization** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| model extraction | not a thing — it is a **model job** |\n| executed here | say **eager materialization** |\n\n**Rulings**\n\n- **`lazy` and `eager` are legal only for materialization.** dlt has both. Using `lazily` to mean\n *on first use* (memoization) is a second meaning for one word — write \"on the first read\".\n- **A model job is the artifact; lazy materialization is the path.** Use the one you mean.\n","isInternal":false,"tokens":185,"sizeBytes":690},{"name":"g4-identifiers-and-sql-generation.md","path":".claude/skills/review-vocabulary/references/vocabulary/g4-identifiers-and-sql-generation.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g4-identifiers-and-sql-generation.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G4 — Identifiers and SQL generation\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| identifier case handling | **case-fold**, **case-folds**, **case-folding** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| casefold, post-fold (in prose) | one spelling — **case-fold**; the identifier `casefold_identifier` keeps its name |\n\n**Rulings**\n\n- **Hyphenated compounds on the `fold` root are legal** — \"foreign-folded output column\" reads\n correctly and is in use. The ban is on the bare spelling `casefold` in prose.\n","isInternal":false,"tokens":137,"sizeBytes":521},{"name":"g5-configuration-and-credentials.md","path":".claude/skills/review-vocabulary/references/vocabulary/g5-configuration-and-credentials.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g5-configuration-and-credentials.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G5 — Configuration and credentials\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| a resolved settings object | **config** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| configuration (for the object) | `config`; keep \"configuration error\" when it names `ConfigurationValueError` |\n","isInternal":false,"tokens":70,"sizeBytes":287},{"name":"g6-merge-strategies-and-merge-conditions.md","path":".claude/skills/review-vocabulary/references/vocabulary/g6-merge-strategies-and-merge-conditions.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g6-merge-strategies-and-merge-conditions.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G6 — Merge strategies and merge conditions\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| removing destination records (`delete-insert`, `upsert`, `cdc`, hard deletes) | **delete** |\n| closing the validity of an `scd2` record | **retire** (never \"delete\" for `scd2`) |\n| loaded records that `source_filter` does not select | **discard**, **discarded** |\n| writing loaded records to the destination | **insert** (a new record), **update** (an existing one) |\n| a record in the destination that the loaded data does not contain | **absent** |\n| the part of a table a load replaces | **partition** (conceptual, as `merge_key` docs already use it) |\n| the loaded data under `cdc` | **snapshot** |\n| the upstream system whose records `cdc` mirrors | **source system** |\n| the loaded records that the source filter selects (all loaded records without a filter) | **merge source** (a noun) |\n| the `source_filter` option in prose | **source filter** |\n| the `destination_scope` option in prose | **destination scope** |\n| `source_filter`, `destination_scope` together | **merge conditions** |\n| `{table}`, `{staging_table}` | **placeholders**; \"expand\" for substitution |\n| data in docs and messages | **records**; **rows** only in code docstrings and comments, and in \"nested rows\" |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| region, window, bare \"scope\" (for the records a merge can delete or retire) | say **destination scope** for the option, **partition** for the concept |\n| input filter, output filter, merge filter(s), merge scope | old names — say **source filter**, **destination scope**, **merge conditions** |\n| filtered staging (data, rows, records), filtered load, filtered loaded data, bare \"the source\" (for the merge source) | say **merge source** |\n| bare \"the source\", upstream, origin (for the system `cdc` mirrors) | say **source system**; `@dlt.source` keeps its name |\n| owns, is authoritative for (a load and its records) | say \"the records that this load replaces\" |\n| clear, remove, wipe (for a merge delete) | one verb — **delete** |\n| drop, exclude, filter out, land (for loaded records) | **discard** for records the source filter does not select, **insert** for records that are written |\n| supersede, take precedence (between merge conditions and `merge_key`) | say \"dlt ignores `merge_key`\" |\n| strand, migrate (a record that changes partition) | say \"the record moves to another partition\" |\n| knob | say **setting** or name the hint |\n\n**Rulings**\n\n- **The bans are for the merge meaning only.** `region` stays legal for cloud regions and data columns,\n `window` for browser windows and the attribution window in `lag.md`, `scope` for OAuth and config scopes,\n `drop` for dropping tables and the `dlt pipeline drop` command, `remove` outside merge deletes.\n- **\"source\" in \"source filter\" is the merge source**, the loaded data, as in `MERGE ... USING source`. It is\n not `@dlt.source`. Never shorten \"source filter\" to \"the source\", and never write \"destination scope\" as\n \"the scope\".\n- **The merge source is what the merge works on.** Keys, `merge_key` partitions and absent records come from the\n merge source, not from all loaded records. The destination scope does not depend on it.\n- **`loaded data` and `loaded records` stay legal** for everything loaded, before the source filter. Write\n \"merge source\" only where a source filter can apply. Without one, the two are the same.\n- **\"MERGE source\" in SQL context is legal.** `MERGE ... USING` names its input the source; uppercase `MERGE`\n marks that meaning (Databricks comments).\n- **Only strategies that delete or retire absent records have a destination scope**: `delete-insert`,\n `scd2`, `cdc`. `upsert` also deletes (with `hard_delete`), so \"strategies that delete\" is not the rule.\n- **`delete` vs `retire`.** `scd2` never deletes on absence, so \"`scd2` deletes absent records\" is a content\n error, not only a vocabulary one.\n- **`snapshot` is legal only for `cdc` loaded data.** Iceberg and Delta snapshots are a different technical noun.\n Do not write \"snapshot\" for Iceberg `upsert` limits: write \"absent from the loaded data\".\n- **Placeholders differ per destination.** Do not write \"`{table}` expands to the destination table\" without\n naming the destination type: on Delta it expands to `target`.\n\nNewly non-compliant: \"region\", \"window\", bare \"scope\", \"owns\", \"clear\", \"supersede\" in merge prose, and the\nold option names (\"input filter\", \"output filter\", \"merge filters\"), and \"filtered staging\" or bare \"the source\"\nfor the merge source. Newly compliant: \"destination scope\" for the `destination_scope` option, \"merge source\".\n","isInternal":false,"tokens":1157,"sizeBytes":4650},{"name":"g7-jobs-triggers-and-the-deployment-manifest.md","path":".claude/skills/review-vocabulary/references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G7 — Jobs, triggers and the deployment manifest\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the unit a workspace deploys and runs | **job** |\n| one execution of it | **job run** |\n| its identity | **job ref** |\n| its entry in the manifest | **job definition** |\n| the dlt module that starts a job | **launcher** |\n| the machine the platform gives the job | **runner** |\n| the dltHub platform itself | **runtime** |\n| the string that starts a job | **trigger** |\n| the pattern that expands into triggers | **selector** |\n| the trigger a manual run stands in for | **default trigger** |\n| the file that describes every job | **deployment manifest** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| task, workload, script (for a dlt job) | one name — **job**; the runtime's `Script` model keeps its own name |\n| job execution, invocation (for one run) | **job run**, the noun the manifest and the beacon use |\n| primary trigger | **default trigger**, the name of the manifest field |\n| manifest (unqualified, for `AGENT.md`) | **deployment manifest** is the only manifest; see G8 |\n\n**Rulings**\n\n- **`run` is a verb; `job run` is the noun.** \"dlt runs the job\" and \"the job run failed\". The\n command `dlthub local run` is a technical name (Rule 8.6) and stays.\n- **launcher, runner and runtime are three things.** The launcher is dlt code. The runner is a\n machine. The runtime is the platform. Never swap them, and never write \"the runtime launches\".\n- **`task` is legal in two places only:** the work an agent must do (G8), and an Airflow task when\n the sentence says \"Airflow task\". It is never a dlt job.\n","isInternal":false,"tokens":425,"sizeBytes":1618},{"name":"g8-agents-loops-and-prompts.md","path":".claude/skills/review-vocabulary/references/vocabulary/g8-agents-loops-and-prompts.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g8-agents-loops-and-prompts.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G8 — Agents, loops and prompts\n\nThree nouns carry the feature, and they nest: a definition is written once, a job adds how it\noperates, a run is one execution.\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| what an `AGENT.md` or a decorated function declares: system prompt, inputs, output, tools, skills, rules, access | **agent definition** (`TAgentSpec` in code, `TAgentDefinition` for the block the manifest carries; pydantic-ai's `AgentSpec`) |\n| the file it is read from | **`AGENT.md`**, and **agent file** for the path |\n| the name a definition is referred to by | **agent definition reference**: `<toolkit>:<agent>`, a workspace path, or `<module>:<function>` for a function with no `AGENT.md` behind it |\n| the definition plus the runtime settings that say how it operates: model, limits, trigger, instructions, loop, identity | **agent job** (`run.agent(...)`; what pydantic-ai and claude-agent-sdk call an Agent, and what the web UI lists as one) |\n| one execution of an agent job: inputs in, output and trace out, settings overridable for that run | **agent run** (`Agent.run()` in the frameworks) |\n| what the model returns: `status`, `summary` and the declared fields | **agent output** |\n| the envelope the launcher delivers | **job result** (G7 owns it; an agent job's is `TAgentJobResult`) |\n| the binding to one agent framework | **loop**, or **agent loop** |\n| the text the model gets as its role and task | **system prompt** |\n| the first message of the run | **user turn** |\n| what a person tells this run to do | **instructions** |\n| one model request | **turn** |\n| the record of what the loop ran and did | **agent trace** |\n| `{{ name }}` in a body | **placeholder** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| agent spec (in prose) | **agent definition**; `TAgentSpec` keeps its name as a type |\n| agent reference, agent ref (in prose) | **agent definition reference**; the field `agent_ref` keeps its name |\n| agent (bare) where the sentence needs one of the three | say which: **agent definition**, **agent job** or **agent run** |\n| harness (for the Claude Code CLI) | the **dltHub AI Harness** owns that word; write **the Claude Code CLI** |\n| adapter (for a loop) | dlt's adapters are `bigquery_adapter` and friends; write **loop** |\n| prompt (bare) | say which one: **system prompt** or **user turn** |\n| spec (bare, for an agent) | bare `spec` is the configspec; write **agent definition** |\n| trace (bare, for an agent) | bare `trace` is the pipeline trace; write **agent trace** |\n| manifest (for `AGENT.md`) | **agent file**; the manifest is the deployment manifest (G7) |\n| framework (as a loop's name) | name it: **pydantic-ai**, **claude-agent-sdk** |\n\n**Rulings**\n\n- **Bare `agent` is the actor.** \"The agent reads the logs\", \"the agent returns `aborted`\": the\n model acting during a run. That stays. The moment a sentence is about the file, the `run.agent`\n product or one execution, it names the definition, the job or the run.\n- **`Agent` (capitalised) is the agent job.** That is how the web UI labels it and how pydantic-ai\n and claude-agent-sdk name the same thing. Legal in UI copy and when talking about a framework;\n in dlt prose write **agent job**.\n- **An agent run is a job run.** Use **agent run** when the sentence is about what the agent\n received and produced; **job run** when it is about scheduling, status or logs, as for any job.\n- **`instructions` means two opposite things across the boundary.** In dlt it is the user turn. In\n pydantic-ai, `AgentSpec.instructions` is the system prompt. Qualify every mention of the\n framework field: \"pydantic-ai's `instructions` field (its system prompt)\".\n- **`turn` is one model request.** The **user turn** is the first message. A turn counter counts\n requests. Do not let one word carry both without the qualifier.\n- **`loop` is the agent loop.** In a file that also has Python loops, write **agent loop** once,\n then `loop`.\n- **Model names and aliases are technical nouns** (Rule 1.8): `sonnet`, `opus`, `claude-sonnet-5`.\n","isInternal":false,"tokens":1072,"sizeBytes":4046},{"name":"g9-workspace-access-and-tools.md","path":".claude/skills/review-vocabulary/references/vocabulary/g9-workspace-access-and-tools.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/references/vocabulary/g9-workspace-access-and-tools.md","title":"Vocabulary Skill","category":"anthropic-skill","format":"markdown","content":"# G9 — Workspace access and tools\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| the declaration of what a job may touch | **access** (the `access` block, `TWorkspaceAccess`) |\n| one of its three keys, `local`, `data` and `context` | **access axis** |\n| a value on an axis | **verb** |\n| what the runtime or a loop does with the declaration | **grant**, **deny** (verbs) |\n| what it did grant | **granted access** |\n| what it did not grant | **denied access** |\n| what a loop supplies beyond the declaration | **over-granted** |\n| what a tool needs before it is served | **required access** |\n| what a model can call | **tool** |\n| the MCP grouping a manifest requests | **feature group** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| permission, entitlement, scope (for the declaration) | **access**, the word dlt's docs already use; `permission` is legal only when naming the SDK's `permission_mode` |\n| narrowed, narrowing | say the fact: **with less access** |\n| grant (as a noun) | **the declared access**; `granted` as an adjective is legal (Rule 3.3) |\n| capability (for a dlt tool) | **tool**; legal only for pydantic-ai `capabilities` |\n| ceiling, floor (as prose metaphor) | say what it does: \"access is the most a loop can wire\" |\n| honor, honour, honored, unhonoured (any spelling) | access is **granted** or **denied**, never honored |\n\n**Rulings**\n\n- **Grant and deny are the verbs.** A job declares access; the runtime and the loop grant what they\n can; the rest is denied. dlt's docs already write \"grant access\" 13 times and \"denied access\"\n twice, so this is the house pairing, not a new one. A loop that supplies more than the\n declaration **over-grants**.\n- **The declaration is a request, not a claim.** A manifest `access` block says what the job wants.\n Nothing in it is granted until the runtime says so. Say \"the job declares\", never \"the job has\".\n- **`access` is the noun, and no synonym joins it.** `permissions`, `scopes`, `capabilities` and\n `entitlements` each name this concept somewhere in the industry; dlt named it `access` before this\n feature existed — the built-in `access` profile is coarse-grained data access, and `profile_for()`\n derives that profile from `access.data`. One concept, one word (Rule 1.11).\n- **`tools:` in an `AGENT.md` holds feature groups, not tools.** Write \"feature groups\" whenever the\n sentence is about that field, or a reader counts 4 tools and gets 19.\n","isInternal":false,"tokens":642,"sizeBytes":2435},{"name":"SKILL.md","path":".claude/skills/review-vocabulary/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/review-vocabulary/SKILL.md","title":"review-vocabulary","category":"anthropic-skill","format":"markdown","content":"---\nname: review-vocabulary\ndescription: Review and rewrite the prose a branch adds — docstrings, comments, user-facing messages, docs — against dlt's fixed vocabulary and Simplified Technical English. Invoke as /review-vocabulary.\nargument-hint: \"[<branch-or-base-ref>] [-- <extra focus or scope notes>]\"\ndisable-model-invocation: true\n---\n\n# Review Vocabulary\n\nRewrite the prose a branch adds so dlt says one thing one way. Two inputs govern every\ndecision: the **fixed vocabulary** in `references/vocabulary/`, indexed below, and the\n**Simplified Technical English** rules in `references/simple-english.md`.\n\n**Never invoke this on your own.** A maintainer runs it with `/review-vocabulary`. It rewrites\ntext across dozens of files and changes CI test ids, so it is not a background cleanup.\n\nParse `$ARGUMENTS`:\n- Everything before the first `--` is the **base ref** to diff against. Defaults to\n `$(git merge-base origin/devel HEAD)`.\n- Everything after the first `--` is extra focus from the maintainer.\n\n## Read first\n\n1. `references/simple-english.md` — the 53 rules. **Cite only rule numbers that exist in that\n file.** The numbering is unintuitive and models invent it. Rule 3.1 is about verb forms, not\n sentence length.\n2. `references/simple-english-use-cases.md` — needed for the error-message shape.\n3. The vocabulary groups the diff touches, picked from the index below. They override the general\n rules where they disagree.\n\n## Scope\n\nDiff with `git diff <base>` — **not** `git diff <base>...HEAD`, which misses uncommitted work\nthat will ship in the same PR.\n\nIn scope, when the branch **added or changed** the line:\n\n| Kind | Where |\n|---|---|\n| Docstrings | `dlt/`, `tests/` |\n| `#` comments | `dlt/`, `tests/` |\n| User-facing messages | `raise X(\"...\")`, `logger.*`, `warnings.warn`, helpers that build a message |\n| Test function names | `tests/` — renames only under the narrow gate below |\n| Documentation | `docs/website/docs/**` |\n| Snippet files a page pulls from | `*snippets.py` — comments as docs, code as a test (see below) |\n\nWhen the maintainer asks for a **whole-file** docs review, pre-existing prose in those files is in\nscope too. Say which mode you are in; the default is added-or-changed lines only.\n\nOut of scope:\n\n- **Pre-existing prose**, in the default mode. Leave it even when it breaks every rule. List what\n you skipped and why.\n- Code, identifiers, signatures, type annotations — except snippet code, which gets a separate\n review against the repo's test rules.\n- SQL keywords and anything already in backticks, CLI commands, file paths, config keys.\n- `pytest.param(id=...)` strings, fixture names, parameter names.\n- Assertions on exact generated SQL.\n- Prose moved verbatim from another file — check the base ref before claiming it is new.\n\n## Classification\n\nGet this right first; every other rule depends on it.\n\n| Text | Mode | Limit | Shape |\n|---|---|---|---|\n| Docstrings, `#` comments | descriptive | 25 words (Rule 6.3) | no imperative, one new fact per sentence (Rule 6.1) |\n| Exception strings, warnings, log lines | **procedural** | **20 words (Rule 5.1)** | what happened (simple past), the cause, then the fix as an imperative |\n| Docs — how-to, steps, install | procedural | 20 words | imperative, condition before command (Rule 5.4) |\n| Docs — concept, reference, architecture | descriptive | 25 words | no imperative |\n\n**Messages keep their remedies.** Many errors end in an instruction — \"Materialize the dataset…\",\n\"Set a permanent `pipeline_salt`…\". Improve the wording: make it imperative, condition first,\nunder 20 words, one instruction per sentence. **Never delete a remedy. Never invent one** where\nthe message has none — a remedy needs facts the string does not interpolate.\n\n**Count the assembled sentence, not the fragment.** Some messages are built by a helper that wraps\na short reason in a frame — `_no_data_location(reason)` in\n`dlt/common/destination/client.py` is the current example. Read the helper first, then judge each\nreason as the clause it becomes inside the finished sentence. A four-word reason can push the\nassembled sentence past 20 words.\n\n**Docs carry their own untouchables:** fenced code blocks, `<!--@@@DLT_SNIPPET ...-->` markers,\nfront-matter, link targets and anchor slugs.\n\n**Headings are anchors.** Renaming one breaks every link to it. Before you rename, grep the whole\ndocs tree for the old slug and update each hit in the same change. When a rename would need an edit\nin a file another agent owns, hand it back rather than reaching across.\n\n### Snippet files are BOTH tests and documentation\n\nA page that uses `<!--@@@DLT_SNIPPET ./x_snippets.py::name-->` pulls real code from a real test.\n`docs/pyproject.toml` collects `*snippets.py` with `*_snippet` functions, so **docs CI executes\nthem**. They therefore get two reviews at once, against two different rule sets:\n\n| Part of the file | Treat as | Rules |\n|---|---|---|\n| the code | a test | `.claude/rules/testing.md`, `coding-style.md`, `imports.md` |\n| the `#` comments | user-facing docs | this vocabulary + STE, classified per comment |\n\nConsequences worth knowing:\n\n- **Comments inside a snippet render to the reader**, so a stale or wrong one is a doc bug, not a\n code-hygiene nit. Judge them as prose. A comment that labels a step is INSTRUCTION; a comment\n that explains behavior is REFERENCE.\n- **The repo comment rule still applies to the code.** Default to no comment. But a snippet is\n teaching material, so a comment that would be redundant in library code can earn its place here.\n Say which of the two you applied when they conflict.\n- **Never change snippet code to satisfy a prose rule.** Changing code changes what CI runs. If the\n code is wrong, report it as a code finding with its own justification.\n- **A fenced code block is NOT a snippet.** It is untested. When a page mixes the two, its fenced\n examples can drift from the API while its snippets cannot. Flag the untested ones as unverified\n rather than assuming they work — and consider recommending they become snippets.\n\n## The fixed vocabulary\n\nOne word, one meaning, one part of speech (Rules 1.11, 9.4). Apply the tables before any other\nrewriting.\n\nThe vocabulary is organised in **groups**, one per area of dlt. A group is self-contained: its\nincluded terms, its excluded terms, and the rulings that are easy to get wrong in that area. Each\ngroup is one file in `references/vocabulary/`. Groups grow independently — a review only needs the\ngroups its diff touches, and adding an area means adding a group, not editing the others.\n\nGroups defined so far:\n\n| Group | Covers | Read when the diff touches |\n|---|---|---|\n| [G1 — Data access and locations](references/vocabulary/g1-data-access-and-locations.md) | destinations, datasets, physical locations, join compatibility | `dlt/dataset/`, `dlt/common/destination/`, `dlt/destinations/`, the dataset-access docs |\n| [G2 — Attach and foreign datasets](references/vocabulary/g2-attach-and-foreign-datasets.md) | cross-destination joins, attach info, catalog aliases | `TAttachInfo`, `ATTACH`, foreign datasets, catalog aliases |\n| [G3 — Transformations and materialization](references/vocabulary/g3-transformations-and-materialization.md) | relations, transformations, model jobs, eager and lazy paths | relations, model jobs, `materialize` in `dlt/dataset/`, `dlt/extract/` or `dlt/load/`, the transformation docs |\n| [G4 — Identifiers and SQL generation](references/vocabulary/g4-identifiers-and-sql-generation.md) | naming conventions, case-folding, query binding | `dlt/common/normalizers/naming/`, `dlt/common/schema/`, sql clients, lineage and query generation |\n| [G5 — Configuration and credentials](references/vocabulary/g5-configuration-and-credentials.md) | configs, credentials, secrets | `dlt/common/configuration/`, credentials classes, `config.toml`, `secrets.toml` |\n| [G6 — Merge strategies and merge conditions](references/vocabulary/g6-merge-strategies-and-merge-conditions.md) | merge strategies, deletes, retirement, source filter, destination scope, placeholders | `dlt/destinations/sql_jobs.py`, a destination's merge job, `merge_key`, `scd2`, `cdc`, `merge-loading.md` |\n| [G7 — Jobs, triggers and the deployment manifest](references/vocabulary/g7-jobs-triggers-and-the-deployment-manifest.md) | jobs, job runs, launchers, triggers, selectors | `dlt/_workspace/deployment/` outside `agent/`, `dlt/hub/run.py`, `__deployment__.py` files, `dlthub local run` |\n| [G8 — Agents, loops and prompts](references/vocabulary/g8-agents-loops-and-prompts.md) | agent definitions, agent jobs, agent runs, loops, system prompts, user turns, traces | `dlt/_workspace/deployment/agent/`, `launchers/agent.py`, `AGENT.md` files, `run.agent` |\n| [G9 — Workspace access and tools](references/vocabulary/g9-workspace-access-and-tools.md) | the `access` declaration, axes, verbs, tools, feature groups | `access` blocks, `TWorkspaceAccess`, `dlt/_workspace/mcp/`, the tools a loop wires |\n\nRead every group the diff touches before you audit, and give each subagent only the groups its\nfiles map to. G1 to G5 are short: when the mapping is unclear, read them all.\n\nTwo rules apply across every group:\n\n- **`dlt` is the sentence subject in messages.** That is how an error gets active voice with a\n named agent (Rule 3.6): \"dlt cannot join…\", \"dlt cannot determine…\". House style.\n- **A banned word is banned for one meaning, not always.** Every group names its legal\n exceptions. Check them before \"fixing\" a hit.\n\n### Legal technical nouns — never replace, any group (Rules 1.5, 1.8)\n\nattach, attach alias, attach info, attach statement, catalog, config, data location, dataset,\ndestination, destination scope, duckdb, iceberg, materialization, merge condition, model job, partition,\npipeline, placeholder, relation, scanner, snapshot (`cdc` only), source filter, vended.\n\naccess, access axis, agent, agent definition, agent definition reference, agent file, agent job,\nagent output, agent run, agent trace, default trigger, feature group, instructions, job, job definition, job ref, job result,\njob run, launcher, loop, placeholder, runner, runtime, selector, skill, system prompt, tool, toolkit,\ntrigger, turn, user turn, verb.\n\n## Rules this codebase breaks most\n\nThe repo comment rule overrides every row. Default to no comment. Keep at most one short line for\na non-obvious WHY. Prefer a rewrite that shrinks the line count. When a comment only restates the\ncode, delete it.\n\n## Test function names\n\nA name is an identifier. Rename it for one of two reasons only:\n\n- it contradicts the fixed vocabulary\n- it carries a noun chain of more than three words (Rule 2.1)\n\nNever rename for style. Apply two checks first:\n\n- **Does the name mirror the API it tests?** Then keep it. `test_attach_info_built_once_per_relation`\n tracks `Relation._attach_infos()`, and that link beats vocabulary purity.\n- **Every rename changes a CI test id.** Flag each one on its own. Afterwards, grep the repo for\n the old name.\n\n## Workflow\n\n### 1. Audit\n\nPropose. Do not edit.\n\nSplit the diff across subagents by area. **File sets must not overlap** — two agents writing one\nfile corrupt each other.\n\nGive each agent its base ref, its file list, the group files its files map to, the two cross-group\nrules and the classification table. Tell it to read `references/simple-english.md` itself, so rule\nnumbers come from the file.\n\nCollect five groups. Give every finding a **file:line, rule number, current text, rewrite**:\n\n1. Vocabulary violations\n2. Structural violations\n3. Identifier renames\n4. Content corrections — false statements. Run the code before you report one.\n5. Test assertions that break — see the trap below.\n\nWrite the proposal to a file. A few hundred findings do not fit in a chat reply. Report the counts\nand the open decisions.\n\n**Stop. Wait for approval.** When the vocabulary does not settle a decision, ask.\n\n### 2. Apply, in two phases\n\nDocstrings and message strings share files, so they cannot run together.\n\n- **Phase A:** docstrings, comments, renames, assertion fixes, content corrections.\n- **Phase B:** message strings.\n\nTell every Phase B agent that the line numbers moved. It must find each string by content. Pin any\nsubstring a test asserts.\n\nEvery applying agent obeys four rules:\n\n- Apply only your own findings, only in your own files.\n- **Never touch code.** Not a signature, not a return type, not a call site. Agents break this\n rule, and a functionally-equivalent change passes every test. Verify it afterwards.\n- Preserve every f-string placeholder: same names, same count, same order.\n- Keep lines under 100 chars. Black does not reflow a string or a comment.\n\n### 3. Verify\n\nRun every step. Each one caught a real defect.\n\n**a. Prose-only check.** Compare the AST with docstrings stripped, against the **branch tip**.\nComments never reach the AST, so any difference means code or a string changed. Explain every file\nthat appears. Against the merge-base the whole PR appears and the signal is lost.\n\n```python\nimport ast\ndef strip(t):\n for n in ast.walk(t):\n if isinstance(n, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):\n n.body = [x for x in n.body if not (isinstance(x, ast.Expr)\n and isinstance(x.value, ast.Constant) and isinstance(x.value.value, str))]\n return t\n# compare ast.dump(strip(ast.parse(old))) with ast.dump(strip(ast.parse(new)))\n```\n\n**b. The four-step self-check** from `references/simple-english.md`, over the added lines. Add a\nsweep for the excluded vocabulary.\n\n**c. `make format`.** Black reports no change when the agents did their job.\n\n**d. `make lint`.** For pages under `docs/website/docs`, also `make md-fix && make md-check` in\n`docs/`. A rewrite that lengthens a table cell past 35 words fails CI.\n\n**e. Run the tests.** This step is not optional.\n\n## The trap: a message rewrite breaks test assertions\n\nTests match on error text. A grep for the phrases you changed misses some. A parametrized test\nhides its assertion far from the message.\n\nOne pass found them in two rounds. A grep found five. The test run then surfaced eight more. One of\nthose was `match=\"cannot be determined\"`. The frame sentence had become \"dlt cannot determine the\ndata location\".\n\n- Grep `tests/` for `match=`, `in str(exc`, and `in str(reject`.\n- Check the **negative** assertions too. `assert \"can join\" not in ...` still passes against\n \"cannot join\", because no space follows `can`.\n- Run the suites anyway. The grep is a head start, not the check.\n\n## Adding a new term or a new group\n\nNever just record a term. Take the eight steps in `references/vocabulary/adding-terms.md`.\n\nAmend the vocabulary mid-review when you must. Then tell every running agent what inverted. They\naudited under the old table.\n\n## Deliverable\n\n- The proposal file, with counts and your open decisions.\n- After approval: the edits, the verification output, and what you did **not** apply.\n- Never commit. The maintainer commits.\n","frontmatter":{"name":"review-vocabulary","description":"Review and rewrite the prose a branch adds — docstrings, comments, user-facing messages, docs — against dlt's fixed vocabulary and Simplified Technical English. Invoke as /review-vocabulary.","argument-hint":"[<branch-or-base-ref>] [-- <extra focus or scope notes>]","disable-model-invocation":true,"disableModelInvocation":true},"isInternal":false,"tokens":3641,"sizeBytes":15097},{"name":"SKILL.md","path":".claude/skills/switch-worktree/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/switch-worktree/SKILL.md","title":"switch-worktree","category":"anthropic-skill","format":"markdown","content":"---\nname: switch-worktree\ndescription: Switch the current session to work in an existing git worktree\nargument-hint: <worktree-name>\n---\n\n# Switch to worktree\n\nSwitch the current session to an existing worktree.\n\n## Steps\n\n### 1. Resolve the worktree\n\nIf `$ARGUMENTS` is empty, list available worktrees and ask the user to pick one:\n\n```\ngit worktree list\n```\n\nOtherwise, let `NAME` = `$ARGUMENTS`. The worktree path is `<repo-root>/.worktrees/<NAME>`.\n\nVerify the worktree exists in `git worktree list` output. If not, stop with an error.\n\nLet `WORKTREE` = the absolute path to the worktree.\n\n### 2. Switch cwd\n\n```\ncd WORKTREE\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, stop with an error — the cwd did not persist.\n\n### 3. Report\n\n```\nSwitched to worktree: WORKTREE\nBranch: <current branch from git branch --show-current>\n```\n","frontmatter":{"name":"switch-worktree","description":"Switch the current session to work in an existing git worktree","argument-hint":"<worktree-name>"},"isInternal":false,"tokens":235,"sizeBytes":878},{"name":"SKILL.md","path":".claude/skills/worktree-from-issue/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/worktree-from-issue/SKILL.md","title":"worktree-from-issue","category":"anthropic-skill","format":"markdown","content":"---\nname: worktree-from-issue\ndescription: Create a git worktree with a new branch for implementing a fix or feature for a GitHub issue\nargument-hint: <category>/<issue-number>-<short-description>\n---\n\n# Git Worktree for Issue Implementation\n\nSet up an isolated git worktree with a new branch to implement a fix or feature for a GitHub issue.\n\nParse `$ARGUMENTS` to extract the branch name in format `<category>/<issue-number>-<short-description>`.\n\n## Branch Naming Convention\n\nBranches must follow the pattern from CONTRIBUTING.md:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n```\n\nCategories: **feat**, **fix**, **exp**, **test**, **docs**, **keep** (all lowercase, dashes, no underscores).\n\nFor `feat` and `fix`, a ticket number is mandatory.\n\nExamples:\n- `fix/3562-clickhouse-drop-sync-perf`\n- `feat/1234-add-new-destination`\n\n## Steps\n\n### 1. Parse and validate\n\nExtract from `$ARGUMENTS`:\n- `branch_name`: Full branch name (e.g., `fix/3562-clickhouse-drop-sync-perf`)\n- `category`: Prefix before `/` (e.g., `fix`)\n- `issue_number`: Number after `/` and before first `-` (e.g., `3562`)\n- `short_name`: Directory-friendly version: `{category}-{issue_number}-{first-few-words}` (e.g., `fix-3562-clickhouse-drop-sync`)\n\nValidate the branch name matches pattern `(feat|fix|exp|test|docs|keep)/[0-9]+-[a-z0-9-]+`.\n\n### 2. Verify the issue exists\n\n```\ngh issue view <issue_number> --json number,title,state\n```\n\nWarn if the issue doesn't exist or is closed, but continue.\n\n### 3. Fetch latest devel\n\nFetch the newest devel from origin so the branch starts from the latest code:\n\n```\ngit fetch origin devel\n```\n\n### 4. Create the worktree\n\nUse `/create-worktree` skill:\n\n```\n/create-worktree <short_name> --branch origin/devel\n```\n\nThen create the feature branch inside the worktree:\n\n```\ngit -C <worktree-path> checkout --no-track -b <branch_name>\n```\n\n`--no-track` leaves the branch without an upstream, so the user's `git push` works.\n\n### 5. Set up dev environment\n\nUse `/worktree-make-dev` skill to set up the development environment in the worktree. This runs `make dev` and copies `secrets.toml`.\n\nNote which secrets source was used — this determines what destinations can be tested.\n\n### 6. Verify cwd\n\nAfter dev setup, verify the cwd is still in the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, run `cd <worktree-path>` and verify again. Stop with an error if the cwd cannot be set.\n\n### 7. Report\n\nReport the worktree info and available test destinations:\n\n```\nWorktree ready: <worktree-path>\nBranch: <branch_name> (based on origin/devel)\nIssue: #<issue_number> - <issue_title>\n```\n\nInclude the test credential availability from `/worktree-make-dev` output — downstream skills need this to know which destinations can be tested.\n","frontmatter":{"name":"worktree-from-issue","description":"Create a git worktree with a new branch for implementing a fix or feature for a GitHub issue","argument-hint":"<category>/<issue-number>-<short-description>"},"isInternal":false,"tokens":708,"sizeBytes":2772},{"name":"SKILL.md","path":".claude/skills/worktree-make-dev/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/.claude/skills/worktree-make-dev/SKILL.md","title":"worktree-make-dev","category":"anthropic-skill","format":"markdown","content":"---\nname: worktree-make-dev\ndescription: Set up a dev environment in an existing worktree (make dev + copy secrets)\nargument-hint: <worktree-path>\n---\n\n# Set up dev environment in worktree\n\nPrepare the worktree at `$ARGUMENTS` for development.\n\n## Steps\n\n### 1. Run setup script\n\n```\nbash tools/setup_worktree_env.sh <worktree-path>\n```\n\nThe script runs `make dev` and copies secrets without exposing credentials in the conversation. If it fails, report the error and stop.\n\n### 2. Report\n\nParse the script output for `SECRETS_SOURCE=` to determine what was used:\n\n- **`repo_root`**: Full credentials available — all destinations including remote (snowflake, bigquery, etc.)\n- **`dev_template`**: Local only — duckdb, dummy, filesystem, and container-based destinations (postgres, clickhouse, etc.)\n- **`none`**: No secrets found — warn the user\n\n```\nDev environment ready: <worktree-path>\n- make dev: done\n- secrets.toml: <source>\n```\n","frontmatter":{"name":"worktree-make-dev","description":"Set up a dev environment in an existing worktree (make dev + copy secrets)","argument-hint":"<worktree-path>"},"isInternal":false,"tokens":230,"sizeBytes":942},{"name":"e2e-hidden-rule.md","path":"tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-hidden-rule.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-hidden-rule.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"This rule carries the token MARKER-RULE-HIDDEN. An agent asked for its markers reports it.\n","isInternal":false,"tokens":21,"sizeBytes":91},{"name":"e2e-visible-rule.md","path":"tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-visible-rule.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/rules/e2e-visible-rule.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"This rule carries the token MARKER-RULE-VISIBLE. An agent asked for its markers reports it.\n","isInternal":false,"tokens":21,"sizeBytes":92},{"name":"SKILL.md","path":"tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/hidden-skill/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/hidden-skill/SKILL.md","title":"hidden-skill","category":"anthropic-skill","format":"markdown","content":"---\nname: hidden-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-HIDDEN. Copy it into `markers`.\n","frontmatter":{"name":"hidden-skill","description":"Carries a marker token. Load it when asked to report the markers of your skills."},"isInternal":false,"tokens":47,"sizeBytes":195},{"name":"SKILL.md","path":"tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/visible-skill/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/skills/visible-skill/SKILL.md","title":"visible-skill","category":"anthropic-skill","format":"markdown","content":"---\nname: visible-skill\ndescription: Carries a marker token. Load it when asked to report the markers of your skills.\n---\nThis skill carries the token MARKER-SKILL-VISIBLE. Copy it into `markers`.\n","frontmatter":{"name":"visible-skill","description":"Carries a marker token. Load it when asked to report the markers of your skills."},"isInternal":false,"tokens":47,"sizeBytes":197},{"name":"dlthub-platform-job-resources.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/rules/dlthub-platform-job-resources.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/rules/dlthub-platform-job-resources.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"Resource changes are proposals, never applied automatically.\n","isInternal":false,"tokens":9,"sizeBytes":61},{"name":"SKILL.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/skills/debug-deployment/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/skills/debug-deployment/SKILL.md","title":"debug-deployment","category":"anthropic-skill","format":"markdown","content":"---\nname: debug-deployment\ndescription: Debug a failing deployment.\n---\nRead the run record first.\n","frontmatter":{"name":"debug-deployment","description":"Debug a failing deployment."},"isInternal":false,"tokens":21,"sizeBytes":99},{"name":"SKILL.md","path":"tests/workspace/cli/dlthub/ai/cases/mock_toolkit/skills/find-source/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_toolkit/skills/find-source/SKILL.md","title":"find-source","category":"anthropic-skill","format":"markdown","content":"---\nname: find-source\n---\nFind a source.","frontmatter":{"name":"find-source"},"isInternal":false,"tokens":11,"sizeBytes":40},{"name":"SKILL.md","path":"tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/skills/find-source/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/skills/find-source/SKILL.md","title":"find-source","category":"anthropic-skill","format":"markdown","content":"---\nname: find-source\ndescription: Find and configure a REST API source\n---\n# Find Source\nInstructions here.","frontmatter":{"name":"find-source","description":"Find and configure a REST API source"},"isInternal":false,"tokens":24,"sizeBytes":108},{"name":"agent-definitions.md","path":"docs/website/docs/hub/agents/agent-definitions.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/docs/website/docs/hub/agents/agent-definitions.md","title":"Subagent: agent-definitions","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fagent-definitions.md -->","frontmatter":{"title":"Agent definitions","description":"Write a dltHub agent definition as an AGENT.md file or as a decorated Python function","keywords":["dlthub platform","agents","AGENT.md","agent definition","inputs","output","access","tools","skills","rules","run.agent"]},"isInternal":false,"tokens":6797,"sizeBytes":32232},{"name":"index.md","path":"docs/website/docs/hub/agents/index.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/docs/website/docs/hub/agents/index.md","title":"Subagent: index","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Findex.md -->","frontmatter":{"title":"Background agents","description":"Run background agents as jobs on the dltHub platform, for example to diagnose failed job runs","keywords":["dlthub platform","agents","background agents","agent job","AGENT.md","run.agent","job inspector","pydantic-ai","claude-agent-sdk","toolkits"]},"isInternal":false,"tokens":7636,"sizeBytes":36095},{"name":"job-inspector.md","path":"docs/website/docs/hub/agents/job-inspector.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/docs/website/docs/hub/agents/job-inspector.md","title":"Subagent: job-inspector","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=docs%2Fwebsite%2Fdocs%2Fhub%2Fagents%2Fjob-inspector.md -->","frontmatter":{"title":"Job inspector agent","description":"Verified dltHub agent that diagnoses a failed job run and reports a classification with evidence and a proposed fix","keywords":["dlthub platform","agents","job inspector","failed job run","diagnosis","dlthub-platform toolkit","job.fail"]},"isInternal":false,"tokens":3035,"sizeBytes":14432},{"name":"AGENT.md","path":"tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/dlthub/agents/e2e/self-report/AGENT.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_e2e_workspace/.claude/dlthub/agents/e2e/self-report/AGENT.md","title":"self-report","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_e2e_workspace%2F.claude%2Fdlthub%2Fagents%2Fe2e%2Fself-report%2FAGENT.md -->","frontmatter":{"name":"self-report","description":"Reports the tools, skills and rule markers it was given.","tools":["workspace"],"skills":["e2e:visible-skill"],"rules":["e2e:visible-rule"],"access":{"local":["read"]},"inputs":{"type":"object","properties":{"failed_run_id":{"type":"string","description":"The job run to report on.","entity_type":"job-runs"}},"required":["failed_run_id"]},"output":{"type":"object","properties":{"local_tools":{"type":"array","items":{"type":"string"},"description":"Name of every tool in your tool list that is not an MCP tool, copied verbatim."},"mcp_tools":{"type":"array","items":{"type":"string"},"description":"Name of every MCP tool in your tool list, copied verbatim."},"skills":{"type":"array","items":{"type":"string"},"description":"Name of every skill available to you."},"markers":{"type":"array","items":{"type":"string"},"description":"Every MARKER-<WORDS> token you were given, copied exactly."},"workspace_name":{"type":"string","description":"The `name` field `get_workspace_info` returned.","entity_type":"workspace"},"reported_run_id":{"type":"string","description":"The job run you were asked to report on, copied exactly.","entity_type":"job-runs"},"trigger":{"type":"string","description":"The trigger of this run, copied exactly."}},"required":["local_tools","mcp_tools","skills","markers","workspace_name","reported_run_id","trigger"]},"defaults":{"model":"haiku","limits":{"max_turns":12}}},"isInternal":false,"tokens":671,"sizeBytes":2687},{"name":"AGENT.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/broken-code/AGENT.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/broken-code/AGENT.md","title":"broken-code","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fbroken-code%2FAGENT.md -->","frontmatter":{"name":"broken-code","description":"Ships agent code that fails on import.","access":{},"inputs":{"type":"object","properties":{"failed_run_id":{"type":"string","description":"run id of the failed job run to inspect"}},"required":{}},"output":{"type":"object","properties":{"status":{"enum":["succeeded","failed","aborted"]},"summary":{"type":"string"}},"required":["status","summary"]}},"isInternal":false,"tokens":112,"sizeBytes":456},{"name":"AGENT.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/checked-inspector/AGENT.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/checked-inspector/AGENT.md","title":"checked-inspector","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fchecked-inspector%2FAGENT.md -->","frontmatter":{"name":"checked-inspector","description":"Inspects a failed run, with checks its agent.py runs before and after the loop.","access":{},"inputs":{"type":"object","properties":{"failed_run_id":{"type":"string","description":"run id of the failed job run to inspect"}},"required":{}},"output":{"type":"object","properties":{"status":{"enum":["succeeded","failed","aborted"]},"summary":{"type":"string"}},"required":["status","summary"]}},"isInternal":false,"tokens":123,"sizeBytes":503},{"name":"AGENT.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/error-triage/AGENT.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/error-triage/AGENT.md","title":"error-triage","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Ferror-triage%2FAGENT.md -->","frontmatter":{"name":"error-triage","description":"Puts one error message of a failed run into a category.","access":{},"inputs":{"type":"object","properties":{"error_message":{"type":"string","description":"error message of the failed run"}},"required":["error_message"]},"output":{"type":"object","properties":{"status":{"enum":["succeeded","failed","aborted"]},"summary":{"type":"string"},"category":{"enum":["config","data","infra"]}},"required":["status","summary","category"]}},"isInternal":false,"tokens":128,"sizeBytes":524},{"name":"AGENT.md","path":"tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/job-inspector/AGENT.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cases/workspaces/agent_workspace/.claude/dlthub/agents/dlthub-platform/job-inspector/AGENT.md","title":"job-inspector","category":"subagent-persona","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcases%2Fworkspaces%2Fagent_workspace%2F.claude%2Fdlthub%2Fagents%2Fdlthub-platform%2Fjob-inspector%2FAGENT.md -->","frontmatter":{"name":"job-inspector","description":"Inspects a failed job run and reports a diagnosis.","tools":["telemetry"],"skills":["dlthub-platform:debug-deployment"],"rules":["dlthub-platform:job-resources"],"access":{"local":["all"],"data":["read"],"context":["read"]},"inputs":{"type":"object","properties":{"failed_run_id":{"type":"string","description":"explicit run id of the job that failed","entity_type":"job-runs"},"failed_job_ref":{"type":"string","description":"explicit job ref of the failed job","entity_type":"job"}},"required":{}},"output":{"type":"object","properties":{"status":{"enum":["succeeded","failed","aborted"],"description":"Outcome of the task."},"summary":{"type":"string","description":"Markdown describing what was accomplished."},"classification":{"enum":["config","code","unknown"]}},"required":["status","summary"]},"defaults":{"model":"sonnet","limits":{"max_turns":30,"max_tokens":1000000},"loop_run_args":{"retries":1}}},"isInternal":false,"tokens":299,"sizeBytes":1171},{"name":"README.md","path":"tests/plugins/dlt_example_plugin/README.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/plugins/dlt_example_plugin/README.md","title":"dlt_example_plugin Documentation","category":"plugin-manifest","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fplugins%2Fdlt_example_plugin%2FREADME.md -->","isInternal":false,"tokens":55,"sizeBytes":214},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/mock_toolkit/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_toolkit/.claude-plugin/plugin.json","title":"test-toolkit","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_toolkit%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"test-toolkit","version":"0.1.0","description":"Test toolkit"},"isInternal":false,"tokens":25,"sizeBytes":76},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/mock_workbench/bootstrap/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_workbench/bootstrap/.claude-plugin/plugin.json","title":"bootstrap","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fbootstrap%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"bootstrap","version":"0.1.0","description":"Bootstrap toolkit"},"isInternal":false,"tokens":23,"sizeBytes":78},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/mock_workbench/init/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_workbench/init/.claude-plugin/plugin.json","title":"init","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"init","version":"1.0.0","description":"Init toolkit"},"isInternal":false,"tokens":23,"sizeBytes":68},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_workbench/rest-api-pipeline/.claude-plugin/plugin.json","title":"rest-api-pipeline","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Frest-api-pipeline%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"rest-api-pipeline","version":"0.1.0","description":"REST API source pipeline toolkit"},"isInternal":false,"tokens":29,"sizeBytes":101},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/mock_workbench/sql-database/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/mock_workbench/sql-database/.claude-plugin/plugin.json","title":"sql-database","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fmock_workbench%2Fsql-database%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"sql-database","version":"0.1.0","description":"SQL database source toolkit"},"isInternal":false,"tokens":27,"sizeBytes":91},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/versioned_workbench/init/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/versioned_workbench/init/.claude-plugin/plugin.json","title":"init","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Finit%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"init","version":"1.0.0","description":"Init toolkit"},"isInternal":false,"tokens":23,"sizeBytes":68},{"name":"plugin.json","path":"tests/workspace/cli/dlthub/ai/cases/versioned_workbench/my-toolkit/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tests/workspace/cli/dlthub/ai/cases/versioned_workbench/my-toolkit/.claude-plugin/plugin.json","title":"my-toolkit","category":"plugin-manifest","format":"json","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tests%2Fworkspace%2Fcli%2Fdlthub%2Fai%2Fcases%2Fversioned_workbench%2Fmy-toolkit%2F.claude-plugin%2Fplugin.json -->","frontmatter":{"name":"my-toolkit","version":"0.0.0","description":"Test toolkit","keywords":["testing","dlt"],"dependencies":["init"]},"isInternal":false,"tokens":41,"sizeBytes":132},{"name":"README.md","path":"tools/tests/cases/dependency_changes/README.md","rawUrl":"https://raw.githubusercontent.com/dlt-hub/dlt/HEAD/tools/tests/cases/dependency_changes/README.md","title":"dependency_changes Documentation","category":"plugin-manifest","format":"markdown","content":"<!-- Skill available on-demand via /api/skills/dlt-hub/dlt?path=tools%2Ftests%2Fcases%2Fdependency_changes%2FREADME.md -->","isInternal":false,"tokens":319,"sizeBytes":1073}],"systemPromptSnippet":"<yakaai_skills repo=\"dlt-hub/dlt\">\n<!-- File: AGENTS.md (Tokens: ~275 | Category: root-instruction) -->\n# dlt principles\n- This is a mature library used by thousands of the best engineers in the world\n- dlt is a library, not a platform. users add it to their code. we respect existing workflows and other libraries they use\n- no black boxes: clean, minimal Pythonic interfaces, human readable file formats, no side effects, documentation\n- multiply - don't add: we do more work here so our users do less.\n\n# dlt repo\n- we use `uv` and `uv run`. look in @Makefile\n- we branch from `devel` for feature branches\n\n# Commit messages (all agents)\n- [Conventional Commits](https://www.conventionalcommits.org/): `type:` or `type(scope):`, lowercase imperative subject, no trailing period.\n- Subject line only for the vast majority of commits; add a body line only for a non-obvious *why* (max ~3 lines total).\n- No footers — no `Co-Authored-By`, no \"Generated with ...\". No emojis.\n- When squash-merging, clean the squash message down to a single subject line.\n- Full rules (enforced per-agent via git/gh hooks under `.claude/`, `.cursor/`, `.codex/`): @.agents/rules/git-github.md\n\n<!-- File: CLAUDE.md (Tokens: ~3 | Category: claude-rule) -->\nAGENTS.md\n\n<!-- File: .agents/rules/git-github.md (Tokens: ~1222 | Category: anthropic-skill) -->\n---\nname: git-github\ndescription: Rules for using git and the GitHub CLI in dlt. Read when committing, creating PRs, or running any git/gh command.\n---\n\n# dlt — Git & GitHub\n\nRules for git and GitHub CLI usage in this repo, for any coding agent. They override the agent's defaults where they differ. See `CONTRIBUTING.md` for the human-facing version.\n\n## When to use\n\nApply whenever running `git` or `gh`, drafting commit messages, or creating/updating PRs.\n\n## Commit messages\n\nFollow [Conventional Commits](https://www.conventionalcommits.org/):\n\n```\n{type}: short imperative subject\n# or, when one area is in focus\n{type}({scope}): short imperative subject\n```\n\n- **Types** (lowercase, matching branch categories): `feat`, `fix`, `docs`, `test`, plus `refactor`/`chore` for non-functional changes.\n- **Scope** is optional — an area like `dashboard`, `cli`, `destinations`, `extract`, `normalize`. Comma-separate multiple.\n- **Subject is lowercase, imperative, no trailing period.**\n- **No emojis** anywhere in the message — not in the subject, not in the body.\n- **Subject line only for the vast majority of commits.**\n- **Max 3 lines total** including any body. If you can't say it in 3 lines, the commit is probably too big — split it.\n- **Code-comment rules apply to commit bodies too.** Don't explain *what* the diff does (the diff shows that). Add a body line only when the *why* is non-obvious and not already in the PR description.\n- **No footers.** No `Co-Authored-By` lines, no \"Generated with ...\" trailers. Treat the agent as a dev tool, not a co-author. This overrides the agent's default.\n- **No references to the task/ticket/caller** in the message unless the user asks (\"as discussed\", \"per review feedback\", etc.).\n\nFormat (HEREDOC):\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n## Cleaning up before a squash-merge\n\nGitHub hides everything after the first line in the commit list, so multi-line\nbodies and footers are easy to overlook — and squashing several such commits\nfloods the merge commit. **When squash-merging, clean the squash message down to\na single Conventional-Commit subject line** (plus a short *why* body only if it\ngenuinely helps). Never let `Co-Authored-By` / \"Generated with\" lines survive\ninto the merge.\n\n## When to commit\n\n- **Only when the user explicitly asks.** Finishing a task is not a trigger; \"looks good\" is not a trigger. Wait for an explicit \"commit this\" / \"make a commit\".\n- `git add -A` is fine — but NEVER stage anything secret-looking (`.env`, `*secrets.toml`, credentials).\n\n## Branches\n\nFrom `CONTRIBUTING.md` — all lowercase, dashes, no underscores:\n\n```\n{category}/{ticket-id}-description-of-the-branch\n# example:\nfeat/4922-add-avro-support\n```\n\n- **Categories**: `feat` (ticket required), `fix` (ticket required), `exp`, `test`, `docs`, `keep`.\n- **Branch from `devel`.** Feature branches and most fixes go to `devel`.\n- Don't create branches unless asked. Don't switch the branch a worktree is on without asking.\n\n## Pull requests\n\n- PRs **target `devel`**. Agents may **NEVER** merge to `master`.\n- **Link the PR to its ticket**, or describe the change clearly enough for someone without prior context.\n- **Creating a PR requires the branch to be pushed already.** If `gh pr create` would need a push, stop and ask the user to push.\n- Confirm the title and body with the user before running `gh pr create`.\n\n## Hard rules for the coding agent (never without an explicit ask)\n\n- **Don't push.** No `git push`/`git push -u`. The user pushes.\n- **Don't merge PRs.** No `gh pr merge`, no auto-merge.\n- **Don't force-push, ever.** If history needs rewriting, describe it and let the user do it.\n- **Don't `--amend` pushed commits**, and never use `--no-verify` or skip hooks/signing.\n- **Don't touch `master`** — it's the release branch (`devel → master` on release day).\n\n## Safe read-only commands (no confirmation needed)\n\n- `git status`, `git diff`, `git log`, `git show`, `git branch`, `git remote -v`\n- `gh pr view`, `gh pr list`, `gh pr checks`, `gh pr diff`, `gh issue view`, `gh api` for GET requests\n\n## Quick checklist before any git/gh action\n\n1. Push, merge, force-push, or amend of a pushed commit? → **Stop**, the user does this.\n2. New commit? → Did the user explicitly ask? If no, **stop**.\n3. Is the subject `type:` / `type(scope):`, lowercase, 1–3 lines total, free of \"what\" narration? If no, rewrite.\n4. Is the commit free of footers (no `Co-Authored-By`, no \"Generated with\") and emojis? If no, trim.\n5. Squash-merging? → Clean the squash message to a single subject line.\n\n<!-- File: .claude/rules/add-workspace-command.md (Tokens: ~777 | Category: anthropic-skill) -->\n---\npaths:\n - \"dlt/_workspace/cli/**\"\n - \"dlt/_workspace/mcp/**\"\n - \"dlt/_workspace/helpers/dashboard/**\"\n - \"dlt/_workspace/typing.py\"\n - \"tests/workspace/**\"\n---\n\n# Workspace model/loader/view pattern for commands\n\nWhen adding a new command (cli, mcp or dashboard) `dlt/_workspace` follows a limited MVC pattern so that data models stay independent\nfrom presentation. The same model can be rendered by a CLI print function, returned\nfrom an MCP tool, or displayed in a marimo dashboard.\n\n## Layers\n\n| Layer | Location | Convention |\n|-------|----------|------------|\n| **Model** | `dlt/_workspace/typing.py` | `TypedDict` (T-prefixed), plain data, no I/O |\n| **Loader** | `dlt/_workspace/cli/ai/utils.py`, `cli/utils.py` | `fetch_*()` functions — collect data into a model, no `fmt.echo()` calls |\n| **View** | `cli/*_command.py`, `cli/commands.py` | `_print_*()` or `print_*()` — render a model to CLI via `fmt.echo()` / `fmt.warning()` |\n| **Controller** | argparse handlers, MCP tools, marimo notebooks | Calls loader, then passes model to the appropriate view |\n\nNOTE: loaders are stored in cli module. to be refactored soon\n\n## Rules\n\n1. **Loaders return typed dicts, never print.**\n A loader (`fetch_ai_status`, `fetch_workspace_info`, `fetch_secrets_list`, ...)\n returns a `TypedDict` from `typing.py`. It must not call `fmt.echo()` or any\n display function.\n\n2. **Views accept a model, never load.**\n A view (`_print_ai_status`, `print_workspace_info`, ...) takes the typed dict\n and formats it for human consumption. It must not call loaders or access\n providers/config directly.\n\n3. **Controllers wire loader to view.**\n An argparse command function (decorated with `@track_command`) calls the loader,\n then the view. MCP tools call the loader and return the model (or a\n serialised form). Dashboard cells call the loader and render via marimo UI.\n\n4. **Research existing types**\n * Research other `typing.py` modules and Literal definitions and types to find potentially reusable types.\n * Reuse, reconcile, modify existing types - also from `common` and other namespaces\n * Make **reconciliation** and model to be **clean** part of implementation plan. Request user input if needed\n\n5. **Write good models**\n * Use `Literal` ie. for error or warning cases\n * Do not leak user messages into data model - view should do it\n * OK to store additional context ie. exception messages\n * Use `NotRequired` field to the model (e.g. `mcp_error: NotRequired[str]`) if element may not be present. In most cases that is better than setting to None\n\n\n## Existing examples\n\n- `fetch_workspace_info()` -> `TWorkspaceInfo` -> `print_workspace_info()`\n- `fetch_ai_status()` -> `TAiStatusInfo` -> `_print_ai_status()`\n- `fetch_secrets_list()` -> `List[TLocationInfo]` -> `ai_secrets_list_command()` (inline view)\n\n## When adding new commands\n\nBefore writing a new CLI command, MCP tool, or dashboard cell that shows\nworkspace state, check whether a loader already exists. If not, create the\nmodel in `typing.py` and the loader in the appropriate `utils.py` first,\nthen write the view / controller.\n\n<!-- File: .claude/rules/coding-style.md (Tokens: ~273 | Category: anthropic-skill) -->\n---\npaths:\n - \"**/*.py\"\n---\n\n# Coding style\n\n## Formatting\n- Line length: 100 chars (black + isort, profile \"black\")\n\n## Code and patterns reuse\n- Obvious utils and patterns are already in code base\n- When in doubt look at the code around the changes you are making - not only style but coding patterns and code organization\n- When using a util function or any other function from the code base - look at existing usage if in doubt how it works\n\n## Comments\n- Inline comments start with lowercase (`# resolve the schema`)\n- NEVER add comments that separate code blocks (`# -- sqlglot_dialect mapping --`)\n- DO NOT state the obvious, do not document your thought process\n\n## Types and data structures\n- Always use full generic parametrization -- `Dict[str, Any]` not `dict`\n- Type aliases use T-prefix PascalCase (`TDataItem`, `TLoaderFileFormat`) -- see @dlt/common/typing.py\n- `TypedDict` for structured dicts, `NamedTuple` for lightweight immutable data\n- Prefer `functools.partial` over `lambda` when currying\n\n## Exceptions\n- Inherit from `DltException`; use `TerminalException`/`TransientException` mixins -- see @dlt/common/exceptions.py\n\n<!-- File: .claude/rules/docstrings.md (Tokens: ~706 | Category: anthropic-skill) -->\n---\npaths:\n - \"**/*.py\"\n---\n\n# Docstring rules\n\n- DO NOT state the obvious, do not document your thought process\n\nGoogle style docstrings throughout. Three levels of detail:\n\n## Short form (internal methods, properties)\nDescription only -- no Args/Returns. One-liner for simple accessors.\n\n## Full form (public API)\nFull Google style with Args, Returns, Raises. Describe WHAT the function does and its output, never HOW it works internally.\n\n```python\ndef paginate(\n self,\n path: str = \"\",\n method: HTTPMethodBasic = \"GET\",\n params: Optional[Dict[str, Any]] = None,\n) -> Iterator[PageData[Any]]:\n \"\"\"Iterates over paginated API responses, yielding pages of data.\n\n Args:\n path (str): Endpoint path for the request, relative to `base_url`.\n method (HTTPMethodBasic): HTTP method for the request, defaults to 'get'.\n params (Optional[Dict[str, Any]]): Query parameters for the request.\n\n Yields:\n PageData[Any]: A page of data with request and response context.\n\n Raises:\n HTTPError: If the response status code is not a success code.\n \"\"\"\n```\n\n- Public API args use `name (Type):` format; internal methods may omit types\n- Args descriptions are short -- one line unless truly complex\n- Use `Yields:` for generators, `Returns:` for regular functions\n- `Raises:` only for exceptions the caller should handle\n- `Example:` section is optional, add only when usage is non-obvious\n\n## Field docstrings\nInline triple-quoted string on the NEXT line after the field definition.\n\n```python\n@configspec\nclass ItemsNormalizerConfiguration(BaseConfiguration):\n add_dlt_id: bool = True\n \"\"\"When true, items will have `_dlt_id` column added with a unique ID.\"\"\"\n add_dlt_load_id: bool = False\n \"\"\"When true, items will have `_dlt_load_id` column added with current load ID.\"\"\"\n```\n\n- Only document fields where the name + type aren't self-explanatory\n- Keep to one line when possible\n\n## Inline formatting\nDocstrings are rendered as Markdown by most editors. Use only these:\n- Single backticks for parameter names, values, types, paths: `name`, `True`, `None`\n- **bold** for emphasis on key terms\n- `Note:` or `Warning:` prefixes for callouts in prose\n- `>>>` interactive Python style for code examples:\n\n```python\n \"\"\"Load data from a REST API.\n\n Example:\n >>> import dlt\n >>> pipeline = dlt.pipeline(\"my_pipeline\", destination=\"duckdb\")\n >>> pipeline.run(my_source())\n \"\"\"\n```\n\nDo NOT use:\n- Double backticks ``like this`` (reStructuredText literal)\n- `:param:`, `:type:`, `:returns:`, `:rtype:` (Sphinx field list)\n- `.. directive::` blocks (reST directives)\n\n## Anti-patterns -- NEVER do these\n- Do not explain HOW the code works, only WHAT it does and its output\n- Do not state the obvious (`\"\"\"Returns the name\"\"\"` on `get_name`)\n- Do not add docstrings to code you did not change\n- Do not use restructured text/sphinx formatting (see Inline formatting above)\n\n<!-- File: .claude/rules/documentation.md (Tokens: ~469 | Category: anthropic-skill) -->\n---\npaths:\n - \"docs/**\"\n---\n\n# Documentation guidelines\n\n## Setup\n- On your first setup, go to directory `docs/` and run `make dev`. This will install the required Python and JS/TS dependencies to build the docs `cd docs && make dev`.\n- This will also install pre-commit hooks using `prek`. This will automatically run linting, type checking, etc. on each `git commit` and `git push`\n\n## Code snippets\n- Inline code snippets will automatically be formatted, linted, and type-checked. To opt-out, add the directive to the codefence `notype` or `nolint`.\n- The `execute` directive means the snippet will be ran during before each commit and on CI. The executed snippets should be lightweight.\n\nFor example\n\n > \\```python notype execute\n >\n > import foo\n > \n > def my_func() -> int: ...\n > \n > \\``` \n \n## Internal links\n- Use relative paths **with** `.md` extension: `[schema contracts](schema-contracts.md)`\n- Link to sections with `#` anchors (auto-generated from headings, lowercase hyphenated): `[merge strategies](merge-loading.md#merge-strategies)`\n- Go up directories as needed: `[adjust a schema](../walkthroughs/adjust-a-schema.md)`\n\n## Markdown formatting\n- `mdsmith` lints `website/docs` on CI (`md-check` hook), configured in `docs/.mdsmith.yml`. After editing a page, run from `docs/`:\n ```sh\n make md-fix # pads and aligns tables, fixes whitespace\n make md-check # must report failures=0\n ```\n- Tables: write them unaligned and let `make md-fix` pad the columns (MDS025). It cannot fix a cell over 35 words (MDS026): shorten the text or move the detail below the table.\n\n## Markdown frontmatter\n- Every doc page needs YAML frontmatter with at least `title`, `description`, and `keywords`:\n ```yaml\n ---\n title: Schema\n description: Schema definition and evolution\n keywords: [schema, dlt schema, yaml]\n ---\n ```\n\n<!-- File: .claude/rules/heresy.md (Tokens: ~288 | Category: anthropic-skill) -->\nHeresy: issues in code that not only must be **avoided** but actively **detected and removed**.\n\n## llm heresy\n* llm talks to itself in docstrings instead of writing code comments in relevant places\n* llm talks about some obscure downstream cases in upstream code (ie. base class is full of details of particular implementation or explains particular use case)\n* llm invents obscure terminology and uses it everywhere. does not follow terminology used in docs\n* llm writes extremely detailed and complex sentences to explain simple things\n\n## arithmetic heresy\n* decimals are converted to binary floats and back\n* binary floats are used to construct decimals. ie. `Decimal(1.1)`\n\n## datetime heresy\n* naive datetimes are used without a good reason. the only exception are end user requirements\n* pendulum is used in new code. use the stdlib counterparts: `datetime` with an explicit\n `timezone`/`ZoneInfo`, and the `ensure_datetime*` helpers instead of `ensure_pendulum_datetime*`.\n NOTE: existing pendulum usage MAY be tolerated where it is hard to change\n\n## terms heresy\n* use of following words constitutes heresy: gate/gating, graft\n\n## comments heresy\n* block separator comments as below\n```\n#------------------------------\n# comment\n#------------------------------\n```\nor similar. actually any form\n\n<!-- File: .claude/rules/imports.md (Tokens: ~458 | Category: anthropic-skill) -->\n---\npaths:\n - \"**/*.py\"\n---\n\n# Import rules\n\n## Import order\n\nGroups separated by blank line:\n1. **stdlib** (`import os`, `from typing import ...`)\n2. **third-party** (`import pendulum`, `import semver`)\n3. **local dlt** (`from dlt.common import ...`)\n4. (tests only) **test utilities** (`from tests.utils import ...`)\n\n- Stdlib imports are ALWAYS at module level -- never inline in functions. This includes `contextlib`, `io`, `warnings`, `sys`, `os`, `re`, etc.\n- `dlt.common` imports (except `dlt.common.libs`) are module level. They are always available and have no heavy side effects.\n- `dlt.common.libs.*` imports may be inline -- they wrap optional dependencies that may not be installed (see Optional dependencies below).\n- If a module is already imported at the top of the file, additional imports from the same module go at the top too -- never inline a second import from an already-imported module.\n- No relative imports.\n\n## Logging\n\n`from dlt.common import logger`, then `logger.info(...)` -- never `import logging` directly (except in `logger.py` itself or runtime infrastructure).\n\n## TYPE_CHECKING imports\n\n- Optional deps for annotations only: `if TYPE_CHECKING: from X import Y` with `else: Y = Any`\n- Destination factories: client class under `TYPE_CHECKING`, actual import deferred to runtime method\n\n## Optional dependencies\n\ndlt wraps optional deps in `dlt/common/libs/` with fail-fast `MissingDependencyException`. Always import from the wrapper, never from the package directly.\n\n### Canonical patterns\n\n```python\nfrom dlt.common.libs.pandas import pandas\n```\n\n### Rules\n\n- NEVER import **optional package** directly\n- ALWAYS import from `dlt.common.libs.<optional package>`\n- Import at the top of the module if module already imports **optional package**\n- Import inline in other cases.\n\n**Applies to**: `pyarrow`, `pandas`, `numpy`, `pydantic`, `sqlalchemy`, `ibis`, `deltalake`, `pyiceberg`\n\n<!-- File: .claude/rules/public-api.md (Tokens: ~264 | Category: anthropic-skill) -->\n---\npaths:\n - \"dlt/**/*.py\"\n---\n\n# Public API and backward compatibility\n\n## Public interface\n- Everything DEFINED and DIRECTLY IMPORTED in @dlt/__init__.py is PUBLIC INTERFACE. Change here is BREAKING CHANGE.\n- Behaviors described in @docs are considered public and changing them is a BREAKING CHANGE (in outputs of PUBLIC INTERFACES)\n- Always flag BREAKING CHANGES clearly with # BREAKING: comments\n\n## Deprecation warnings\n- Use `DltDeprecationWarning` from `dlt/common/warnings.py`, never raw `DeprecationWarning`\n- Pass the `since` version where the deprecation is introduced\n- Use `Dlt100DeprecationWarning` only for features deprecated since 1.0.0 (hardcodes `since`)\n- To deprecate entire functions or classes, use the `deprecated` decorator from `dlt/common/warnings.py`\n\n## Backward compatibility mechanisms\n- Avoid BREAKING CHANGES with deprecations, configuration options etc.\n- Schema/state migration via `engine_version`, versioned storage layouts, deprecation warnings, `deprecated` decorator\n- Backward compat must be tested -- see `tests/pipeline/test_dlt_versions.py` for e2e version upgrade tests\n\n<!-- File: .claude/rules/testing.md (Tokens: ~545 | Category: anthropic-skill) -->\n---\npaths:\n - \"tests/**/*.py\"\n - \"tools/tests/**/*.py\"\n---\n\n# Testing\n\n## Style and structure\n- Module-based tests (plain `def test_*() -> None:` functions), not class-based\n- pytest with fixtures in `conftest.py`, not unittest setUp/tearDown\n\n## Reduce tests sprawl\n- Use `pytest.mark.parametrize` to reduce code duplication. ALWAYS use human readable ids for the test cases\n- It is OK to test many related use cases in one test: ie. when they share common setup or represent a flow of user actions\n\n## Fixtures\n- We have autouse fixtures enabled by `conftest.py`. Always check those before adding common fixtures ie. to preserve env variables or get clean storage for the tests\n- We have A LOT of fixtures in files like @tests/utils.py and @tests/load/utils.py, look there before adding new ones.\n\n## Large files for test cases\n- Do not keep large test cases in Python modules. Use `cases` folder and to keep and share files (ie. schemas, ingest data) across modules. \n\n## Destination tests (tests/load)\n- Tests in @tests/load are ALWAYS run on CI in context of particular destination. Tests that run pipelines MUST use `destinations_configs`, other test modules must filter for a right destination (ie. `skip_if_not_active(\"clickhouse\")`)\n- NOTE: tests outside @tests/load do not use such filtering. Use `duckdb`, `dummy` or `sqlite` or `filesystem` destinations for such tests.\n\n### Test Markers\n- `@pytest.mark.essential` — For critical test paths\n- `@pytest.mark.serial` — Tests that can't run in parallel\n\n## Parallel safety and isolation\n- Tests run in parallel! ALWAYS use test storage (`get_test_storage`), unique pipeline names (`uniq_id`), and worker-aware storage roots (`get_test_storage_root` from `tests/utils.py`). Isolate pipelines with `dev_mode` or random `dataset_name`. Avoid external resources that clash with other tests.\n- Tests must work on Linux, macOS and Windows. Avoid encoding paths and platform-specific code. Local path assertions must be Windows compatible.\n\n## Placement and comments\n- When adding new test ALWAYS try find the right place by looking where code you make changes is tested\n- Docstrings: DESCRIBE test case. SKIP when test name is sufficient\n- Use line comments to describe and separate use cases within a single test\n\n## Running\n* DO NOT start two `pytest` processes at once!\n* when running many tests, use `-p xdist -n auto` to parallelize\n\n<!-- File: .claude/rules/worktree.md (Tokens: ~369 | Category: anthropic-skill) -->\n# Working in a git worktree\n\nWhen a worktree is active (created via `/create-worktree`, `/worktree-issue`, or the user says \"work in worktree X\"), apply these rules for ALL subsequent operations.\n\nLet `WORKTREE` = the absolute worktree path (e.g., `/home/user/src/dlt/.worktrees/fix-3562`).\n\n## Shell cwd behavior\n\nClaude Code resets Bash cwd to the project root only when `cd` targets a path **outside** the project root. Paths **within** the project root (including `.worktrees/`) persist across calls.\n\n- **Bash**: run `cd WORKTREE` once to switch; subsequent calls will stay there\n- **git**: `git -C WORKTREE <command>` also works\n- **Read/Grep/Glob/Edit/Write**: always use absolute paths under `WORKTREE`\n\n## Running Python and tests\n\nAlways use `uv run` in worktrees — never bare `python` or `pytest`:\n\n```\nuv run pytest tests/... -v\nuv run python -c \"...\"\n```\n\n## Worktree location convention\n\nAll worktrees live under `<repo-root>/.worktrees/<name>`.\n\n## Plans created in a worktree\n\nWhen creating a plan while working in a worktree, **always** include the worktree path at the top of the plan:\n\n```markdown\n## Working environment\n- Worktree: <absolute worktree path>\n- Branch: <branch name>\n- Execute: cd <absolute worktree path> before any work\n```\n\nThis is critical because plan→execute transitions start a new session whose cwd resets to the project root. Without the worktree path in the plan, the execute phase will modify files in the main repo instead of the worktree.\n\n<!-- File: .claude/skills/auto-docstring-rules/SKILL.md (Tokens: ~57 | Category: anthropic-skill) -->\n---\nname: auto-docstring-rules\ndescription: Enforces dlt docstring conventions. Use when writing or modifying docstrings, classes, or function signatures in Python files.\nuser-invocable: false\n---\n\nUse @.claude/rules/docstrings.md regardless of path setting\n\n<!-- File: .claude/skills/auto-import-rules/SKILL.md (Tokens: ~54 | Category: anthropic-skill) -->\n---\nname: auto-import-rules\ndescription: Enforces dlt import conventions and ordering. Use when writing or modifying Python imports in dlt/ or tests/.\nuser-invocable: false\n---\n\nUse @.claude/rules/imports.md regardless of path setting\n\n<!-- File: .claude/skills/auto-write-test-rules/SKILL.md (Tokens: ~48 | Category: anthropic-skill) -->\n---\nname: auto-write-test-rules\ndescription: Use when adding or proposing a new test ONLY if the test rules are not present\nuser-invocable: false\n---\n\nUse @.claude/rules/testing.md regardless of path setting\n\n<!-- File: .claude/skills/commit/SKILL.md (Tokens: ~452 | Category: anthropic-skill) -->\n---\nname: commit\ndescription: Stage changes and write a clean Conventional-Commit message that follows the repo's git rules\nargument-hint: [-- <optional subject hint or scope>]\n---\n\n# Commit\n\nCreate a commit that follows `.agents/rules/git-github.md`. Invoke as `/commit`, or\nuse these steps whenever you are about to commit.\n\nParse `$ARGUMENTS`: anything after `--` is an optional hint (a subject, a scope, or\n\"only the dashboard files\") — guidance, not the literal message.\n\n## Steps\n\n### 1. Inspect\n\n```\ngit status\ngit diff --staged\ngit diff\n```\n\nDecide what belongs in the commit. If nothing is staged, stage the relevant changes\nwith `git add <paths>` (or `git add -A` when everything belongs together). **NEVER\nstage secret-looking files** (`.env`, `*secrets.toml`, credentials) — they are also\ndenied in `settings.json`.\n\n### 2. Compose the message\n\nApply `.agents/rules/git-github.md`:\n\n- One line: `{type}: subject` or `{type}({scope}): subject` — lowercase, imperative,\n no trailing period. Types: `feat`, `fix`, `docs`, `test`, `refactor`, `chore`.\n- **Subject line only** for the vast majority of commits. Add a body line *only* for a\n non-obvious *why*, max 3 lines total. Never narrate *what* the diff does.\n- **No footers** — no `Co-Authored-By`, no \"Generated with ...\" trailers.\n- **No emojis** anywhere in the message.\n\n### 3. Commit\n\n```\ngit commit -m \"$(cat <<'EOF'\nfix(dashboard): keep the selector when no pipeline is selected\nEOF\n)\"\n```\n\n### 4. Confirm\n\n```\ngit log -1 --format=%B\n```\n\nVerify it is a single Conventional-Commit subject (plus an optional why) with no\nfooters. **Do not push and do not amend pushed commits** unless the user explicitly\nasks.\n\n<!-- File: .claude/skills/create-worktree/SKILL.md (Tokens: ~738 | Category: anthropic-skill) -->\n---\nname: create-worktree\ndescription: Create or reuse a git worktree for a pull request or branch so reviews and work happen in isolation\nargument-hint: <name> [--pr <number>] [--branch <ref>]\n---\n\n# Git Worktree\n\nSet up an isolated git worktree.\n\nParse `$ARGUMENTS` to extract:\n- The worktree name (required) — used as the directory name under `.worktrees/`\n- `--pr <number>` (optional) — checkout a GitHub PR into the worktree using `gh pr checkout`\n- `--branch <ref>` (optional) — create the worktree from a specific branch or ref\n\n## Steps\n\n### 1. Determine worktree path\n\nThe worktree path is: `<repo-root>/.worktrees/<name>`\n\nFor example, if the repo root is `/home/user/src/dlt` and the name is `review-pr-3584`, the worktree path is `/home/user/src/dlt/.worktrees/review-pr-3584`.\n\nCreate the `.worktrees` directory if it does not exist:\n\n```\nmkdir -p <repo-root>/.worktrees\n```\n\n### 2. Create or reuse the worktree\n\nCheck if a worktree already exists at that path:\n\n```\ngit worktree list\n```\n\n- **If a worktree already exists** at the target path, reuse it.\n - If `--pr` was given, run `cd <worktree-path> && gh pr checkout <number>` to make sure it is up to date.\n - If `--branch` was given, run `git -C <worktree-path> checkout <ref>`.\n- **If no worktree exists** at the target path, create one:\n - If `--pr` was given, first resolve the PR's branch name: `gh pr view <number> --json headRefName -q .headRefName`\n - If `--pr` or `--branch` was given, check `git worktree list` output for any **other** worktree that already has the same branch checked out. If found:\n - Use `AskUserQuestion` to ask the user: \"Branch `<branch>` is already checked out in worktree `<existing-path>`. Use that worktree instead?\"\n - If the user says yes, use the existing worktree path (skip creation, proceed to step 3 with the existing path).\n - If the user says no, **STOP** — do not create a new worktree. Print a message explaining that the branch is already checked out elsewhere and exit.\n - If no conflict, create the worktree:\n - If `--branch` was given: `git worktree add .worktrees/<name> <ref>`\n - Otherwise: `git worktree add .worktrees/<name> --detach`\n - Callers needing a new branch check it out afterwards: `git checkout --no-track -b <new-branch>`\n - If `--pr` was given, then also: `cd <worktree-path> && gh pr checkout <number>`\n- `gh pr checkout` handles fetching, fork tracking, and branch setup automatically.\n\n### 3. Switch cwd\n\n```\ncd <worktree-path>\n```\n\nThen verify the switch took effect:\n\n```\npwd\n```\n\nIf `pwd` does not show the worktree path, stop with an error — the cwd did not persist.\n\n### 4. Report\n\n```\nWorktree ready: <worktree-path>\n```\n\n<!-- File: .claude/skills/dashboard/SKILL.md (Tokens: ~1710 | Category: anthropic-skill) -->\n---\nname: dashboard\ndescription: Use when reading, editing, or creating files in dlt/_workspace/helpers/dashboard/ or tests/workspace/helpers/dashboard/ or tests/e2e/\nuser-invocable: false\n---\n\n# dlt Dashboard Development Guide\n\nConsult this skill whenever working on the dashboard codebase. It tells you where files live, what each module does, and where new code belongs.\n\n## Overview\n\nThe dashboard is a **marimo-based** reactive web UI for inspecting dlt pipelines, schemas, data, and run traces. It lives at `dlt/_workspace/helpers/dashboard/`.\n\n## File Layout\n\n```\ndlt/_workspace/helpers/dashboard/\n├── dlt_dashboard.py # Main marimo app (~30 reactive cells)\n├── dlt_dashboard_styles.css # CSS: colors, badges, timeline, tables\n├── runner.py # Subprocess launcher, CLI integration\n├── config.py # DashboardConfiguration @configspec dataclass\n├── const.py # Constants: port (2718), timeouts, colors, defaults\n├── typing.py # Shared TypedDicts (TTableListItem, TLoadItem, etc.)\n├── strings.py # All user-facing UI text (~350 lines)\n└── utils/\n ├── __init__.py # Package exports\n ├── pipeline.py # get_pipeline(), pipeline_details(), exception handling\n ├── queries.py # SQL execution, caching (BoundedDict), row counts, loads\n ├── schema.py # Table/column lists, schema version fetching\n ├── trace.py # Trace rendering, sanitization\n ├── visualization.py # Execution timeline HTML, load status badges\n ├── home.py # Home page / workspace home rendering\n ├── ui.py # dlt_table(), section_marker(), error_callout(), BoundedDict\n ├── formatters.py # Datetime humanization, string formatting\n └── data_quality.py # dlthub data quality integration\n```\n\n## Architecture\n\n- **Framework**: marimo — a reactive Python notebook that renders as a web app\n- **Cell pattern**: `@app.cell(hide_code=True)` decorators define reactive cells in `dlt_dashboard.py`\n- **Dependency tracking**: Cells automatically re-execute when their parameter dependencies change\n- **Routing**: URL query params (`?pipeline=X&profile=Y`) via `mo.query_params()`\n- **Caching**: `BoundedDict` (LRU, 64/32 entries) in `utils/queries.py` and `utils/schema.py`\n- **Styling**: CSS variables for dlt brand colors (`--dlt-color-lime`, `--dlt-color-aqua`, `--dlt-color-purple`, `--dlt-color-pink`)\n\n## Dashboard Sections (cells in dlt_dashboard.py)\n\n1. **Profile & Pipeline Discovery** — reads workspace profiles, lists local pipelines\n2. **Pipeline Attachment** — `dlt.attach()`, file watcher for auto-refresh\n3. **Overview** — pipeline properties, destination info, state version\n4. **Schema** — table browser with configurable column display\n5. **Browse Data** — SQL editor with cached results and query history\n6. **Trace** — execution timeline, step details, resolved config\n7. **Loads** — load history with row counts and schema versions\n8. **State** — raw JSON state viewer\n9. **Data Quality** — integration with dlthub.data_quality\n10. **Ibis Backend** — read-only Ibis connection\n\n## How the Dashboard is Launched\n\nThree CLI commands all call `runner.run_dashboard()`:\n\n* there are several commands in `dlt` and `dlthub` hosts that launch local dashboard. for example `dlt dashboard` and `dlthub local show`.\n\nEdit mode (`--edit`) ejects `dlt_dashboard.py` and CSS to the cwd for user customization.\n\n## Dependencies\n\nFrom `pyproject.toml` `workspace-deps` group: `marimo>=0.14.5`, `duckdb>=0.9`, `ibis-framework>=12.0.0`, `pyarrow>=16.0.0`.\n\n## Where to Put New Code\n\n| What you're adding | Where it goes |\n|---|---|\n| New dashboard section/cell | `dlt_dashboard.py` — add `@app.cell(hide_code=True)` |\n| New UI text/messages | `strings.py` — add to appropriate section |\n| New data fetching/queries | `utils/queries.py` or new file in `utils/` |\n| Schema-related helpers | `utils/schema.py` |\n| Pipeline metadata helpers | `utils/pipeline.py` |\n| Trace/execution helpers | `utils/trace.py` or `utils/visualization.py` |\n| UI component helpers | `utils/ui.py` |\n| Formatting/display helpers | `utils/formatters.py` |\n| New TypedDicts | `typing.py` |\n| New constants/defaults | `const.py` |\n| Configuration options | `config.py` (`DashboardConfiguration` dataclass) |\n| CSS styling | `dlt_dashboard_styles.css` |\n| Data quality features | `utils/data_quality.py` |\n\n## Coding Conventions\n\n- **All user-facing text** goes in `strings.py`, never inline in cells\n- **Cell-local variables must start with `_`** (underscore prefix). In marimo, variables that are only used within a single cell and should not be shared across cells must be prefixed with `_`. Only variables intended to be consumed by other cells should be unprefixed.\n- **UI helpers**: Use `utils/ui.py` functions (`dlt_table()`, `section()`, `error_callout()`) for consistent UI\n- **CSS classes**: `section-header`, `pipeline-execution-timeline`, `status-badge`, `dlt_table`\n- **Badge classes**: `.badge-grey`, `.badge-yellow`, `.badge-green`, `.badge-red`\n- **Data structures**: TypedDicts in `typing.py` for data passed between cells\n- **Configuration**: `@configspec` dataclass in `config.py`\n\n## Testing\n\n### Test locations\n\n```\ntests/workspace/helpers/dashboard/ # Unit and integration tests\n├── conftest.py # Fixtures: pipeline types and temp directories\n├── test_pipeline_workflow.py # Pipeline data retrieval integration\n├── test_all_cells.py # Cell execution and dependency testing\n├── test_visualization.py # Timeline and badge rendering\n├── test_ui_elements.py # UI component styling\n├── test_table_operations.py # Table listing/filtering\n├── test_trace_operations.py # Trace parsing/display\n├── test_query_operations.py # SQL query execution\n├── test_loads.py # Load history retrieval\n├── test_data_formatting.py # String formatting\n├── test_pipeline_operations.py # Pipeline metadata\n└── test_dashboard_config.py # Configuration resolution\n\ntests/e2e/ # Playwright browser tests\n```\n\n### Running tests\n\n```bash\nmake test-workspace-dashboard # unit/integration tests\nmake test-e2e-dashboard # headless chromium e2e\nmake test-e2e-dashboard-headed # visible browser e2e\nmake create-test-pipelines # create pipelines for manual testing\n```\n\n### Test fixtures\n\nTests use parametrized pipeline fixtures via `ALL_PIPELINES` and `PIPELINES_WITH_LOAD` constants in `conftest.py`. Key fixtures: `success_pipeline_duckdb`, `success_pipeline_filesystem`, `extract_exception_pipeline`, `never_ran_pipeline`, `load_exception_pipeline`.\n\n### Where to add new tests\n\n- **Utils logic**: Add to the matching `test_*.py` file (e.g., new query helper → `test_query_operations.py`)\n- **New section/cell**: Add cell execution test to `test_all_cells.py`, plus a dedicated test file if the section has significant logic\n- **Visual/rendering**: `test_visualization.py` or `test_ui_elements.py`\n- **E2E browser tests**: `tests/e2e/`\n\n<!-- File: .claude/skills/implement-issue/SKILL.md (Tokens: ~1841 | Category: anthropic-skill) -->\n---\nname: implement-issue\ndescription: Triage, plan, and implement a fix or feature for a GitHub issue end-to-end\nargument-hint: <issue-number> [-- <hints or context from maintainer>]\n---\n\n# Implement Issue\n\nParse `$ARGUMENTS` to extract:\n- Everything before the first `--` (or all of it if no `--`) is the **issue number**. Referred to as `ISSUE` below.\n- Everything after the first `--` is **maintainer hints** — domain knowledge, suspected root cause, pointers to relevant code, or constraints. If absent, there are no hints.\n\n## Steps\n\n### 1. Fetch the issue\n\n```\ngh issue view ISSUE --json title,body,comments,labels,state,assignees,author,createdAt,milestone\n```\n\nProduce a concise summary: title, reporter, component/area involved, error message, reproduction steps, expected behavior.\n\n### 2. Determine category and gather hints\n\n#### Category\n\nDerive the branch category from the issue labels:\n\n| Label contains | Category |\n|---|---|\n| `bug` | `fix` |\n| `enhancement`, `feature` | `feat` |\n| `test` | `test` |\n| `documentation` | `docs` |\n\nIf no matching label exists, ask the human for the category.\n\n#### Hints\n\nCollect implementation hints from all available sources, in priority order:\n\n1. **Maintainer hints** from `$ARGUMENTS` (after `--`)\n2. **Issue comments** — look for maintainer/contributor replies with technical guidance, root cause analysis, or implementation suggestions\n3. **Issue body** — the reporter may include debugging details, stack traces, or configuration\n\nIf the combined hints from these sources provide sufficient direction (suspected root cause, relevant code areas, constraints), proceed directly to step 3 without asking the human.\n\nIf hints are insufficient — the issue is vague, the root cause is unclear, or there are multiple possible approaches — ask the human for guidance.\n\n### 3. Research the big picture\n\nBefore jumping to a fix, verify assumptions. This step prevents wasted effort on the wrong solution.\n\n#### 3a. Understand the feature area\n\nUse Explore agents to thoroughly investigate the relevant code paths. Trace the flow from user-facing API to the point of failure. Read the code — don't guess.\n\n#### 3b. Validate the reporter's setup\n\nCheck whether the reporter is using the library correctly:\n\n- Is their configuration valid for the component they're using?\n- Are there special conventions, settings, or patterns they should be using instead?\n- Does the documentation describe the expected behavior differently?\n\n#### 3c. Check external dependencies and integrations\n\nIf the issue involves specific external systems (destinations, sources, file formats, etc.):\n\n- Read the relevant implementation in the codebase (factory, capabilities, client, etc.).\n- Understand how the component interacts with the external system.\n- If needed, consult external documentation via WebFetch/WebSearch.\n\n#### 3d. Determine: bug or user error?\n\nBased on research, conclude one of:\n\n- **Bug**: The library has a defect. Proceed to step 4.\n- **User error**: The reporter's configuration is wrong. Draft a response explaining the correct approach. Ask the human to confirm before posting.\n- **Enhancement**: The library works as designed but could be improved. Discuss scope with the human before proceeding.\n\nReport findings to the human before moving on.\n\n### 4. Set up workspace\n\nBefore planning, create the working environment so plan-mode exploration runs against a clean branch.\n\n#### 4a. Determine branch name\n\nUse the category from step 2 and the issue number:\n```\n{category}/{issue-number}-{short-description}\n```\n\nExample: `fix/3529-dedup-sort-escape`\n\nIf `ISSUE` was already a branch name rather than a number (no ticket exists), use it as given.\n\n#### 4b. Create worktree and dev environment\n\nInvoke the `/worktree-from-issue` skill with the branch name:\n```\n/worktree-from-issue {branch-name}\n```\n\nThis creates the worktree, checks out a new branch from `origin/devel`, and runs `/worktree-make-dev`.\nUse it also when there is no issue number.\n\nNote the worktree path — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 5. Draft implementation plan\n\nEnter plan mode and design a plan that **will pass the `/review-pr` skill**. The review-pr skill checks:\n\n1. **Code quality**: Style conforming to `@CLAUDE.md`, no over-engineering, proper use of existing patterns and utilities.\n2. **Test coverage**: Required scenarios identified, tests in the right location, no duplication, proper parametrization.\n3. **Documentation**: Public API changes need doc updates.\n4. **Backward compatibility**: Breaking changes flagged with deprecation path.\n5. **Dependencies**: Changes to `pyproject.toml` flagged.\n\nThe plan must include:\n\n- **Working environment**: Worktree path, branch name, and `cd` instruction (see `@.claude/rules/worktree.md`).\n- **Root cause**: Precise code location and explanation.\n- **Fix**: Minimal change following existing patterns in surrounding code.\n- **Test**: A test that **fails without the fix and passes with it**. Place tests where similar functionality is already tested. Follow `@CLAUDE.md` test guidelines (parallel safety, platform independence, proper fixtures).\n- **Test scope**: Identify the right test level — `make test-common` for extract/normalize/common work, destination-specific tests for load work. See `@CLAUDE.md` \"Testing strategy\" section.\n- **Verification commands**: Exact selective pytest commands to run (specific test functions, not entire modules).\n- **Files changed**: List with expected line counts.\n\nPresent the plan via ExitPlanMode for human approval.\n\n### 6. Implement the fix\n\nApply the changes from the approved plan. Follow these rules:\n\n- **Read before editing**: Always read the target file first. Never edit blind.\n- **Minimal changes**: Only change what the plan specifies. Don't refactor surrounding code, add comments to unchanged code, or \"improve\" things not in scope.\n- **Follow local patterns**: Match the style of surrounding code exactly — imports, naming, spacing, comment style.\n- **Type hints**: All new function signatures need full type hints with generic parametrization (`Dict[str, Any]` not `dict`).\n- **Test conventions**: `def test_*() -> None:`, pytest fixtures, `@pytest.mark.parametrize` for variants.\n\n### 7. Verify\n\nRun verification in this order. Stop and fix issues before proceeding to the next step.\n\n#### 7a. Run the new test selectively\n\nRun only the new/changed test functions using the commands from the plan. Be selective — target specific test functions, not entire modules.\n\n#### 7b. Regression proof (for external system interactions)\n\nWhen the fix involves an external system (destination, source, API) where you don't have full visibility into the behavior — a \"black box\" — verify experimentally that the test actually catches the bug:\n\n- Temporarily revert the fix\n- Run the new test on a configuration where the bug manifests\n- Confirm it **fails** with an error matching the issue report\n- Restore the fix\n\nThis is valuable when you cannot fully predict how the external system handles the input (e.g., how a database treats unquoted identifiers, how an API responds to edge-case payloads). It proves the test is meaningful and not a no-op.\n\nSkip this step when the code path is fully internal and you can reason about correctness from the code alone.\n\n#### 7c. Format and lint\n\n```\nmake format && make lint\n```\n\nIf you changed pages under `docs/website/docs`, also run `make md-fix && make md-check` in `docs/`.\n\nAll must pass cleanly. Fix any issues before reporting.\n\n### 8. Report\n\nSummarize results:\n\n```\n## Fix for #ISSUE: <title>\n\n**Branch**: {branch-name}\n**Worktree**: WORKTREE\n\n### Changes\n- `<file>`: <description> (<lines> lines)\n- ...\n\n### Verification\n- <test-function> on <config>: passed\n- <test-function> on <config> without fix: FAILED (confirms bug) [if applicable]\n- make format: pass\n- make lint: pass\n\nReady to commit.\n```\n\nWait for the human to confirm before committing.\n\n<!-- File: .claude/skills/optimize-code/SKILL.md (Tokens: ~2206 | Category: anthropic-skill) -->\n---\nname: optimize-code\ndescription: Analyze and optimize Python code performance in critical paths\nargument-hint: <file-path-or-description> [-- <critical-path-hints>]\n---\n\n# Optimize Code\n\nYou are a Python performance optimization expert. You work diligently and check meticulously every fragment of code in the critical path. You never assume - you trace actual execution paths, verify with evidence, and question defensive patterns that may have become unnecessary.\n\n**ALWAYS benchmark everything.** Use `timeit` oneliners for quick validation, write dedicated benchmarks for optimization variants, and as the final step, benchmark optimized code against the baseline (devel branch).\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **target**: file path, function name, or description of code to optimize\n- Everything after `--` is **critical path hints**: which code paths matter most, what to ignore\n\n## Phase 1: Understand the Critical Path\n\nSKIP if optimizing a single (or 1-2 ) functions where critical path is clear, otherwise INTERROGATE the user to understand the hot path before making any changes.\n\n### 1.1 Identify the target\n\nIf `$ARGUMENTS` contains a file path, read it. Otherwise, ask the user:\n- \"Which file or function should I optimize?\"\n- \"What is the entry point for the hot path?\"\n\n### 1.2 Learn the typical case\n\nAsk about the 90% case to focus optimization effort:\n- \"What data shapes are most common?\" (flat dicts, nested objects, lists)\n- \"Which branches are taken most often?\"\n- \"What can we ignore?\" (e.g., \"ignore pandas/arrow, focus on plain dicts\")\n\n### 1.3 Trace the call chain\n\nFollow the code from entry point through each function call:\n```\nentry_point() → helper_a() → helper_b() → actual_work()\n```\n\nRead each function in the chain. Identify the innermost loop where per-item work happens.\n\n### 1.4 Establish baseline benchmark\n\nBefore any optimization, create a benchmark for the current state:\n```python\nimport timeit\n# Quick validation\npython -c \"import timeit; print(timeit.timeit('target_function()', setup='...', number=1000))\"\n```\n\n## Phase 2: Analyze for Optimization Opportunities\n\nLook for these patterns in the critical path:\n\n### 2.1 Unnecessary work\n\n- **Redundant copies**: `dict(x)` or `list(x)` when reference would suffice\n- **Defensive copies**: Trace data origin - if source returns fresh object, copy is redundant\n- **Repeated computations**: Same value computed multiple times in loop\n\n### 2.2 Function call overhead\n\n- **Hot helper functions**: Consider inlining if called per-item\n- **Uncached method lookups**: `self.method` in loop vs `method = self.method` before loop\n- **Duplicate isinstance/hasattr checks**: Same check in multiple places\n\n### 2.3 Happy path shortcuts\n\n- **Early exit for common case**: Check fast path first, skip expensive work\n- **Identity vs equality**: `x is y` faster than `x == y` when applicable\n\n\n### 2.4 Consider `__slots__` for hot objects\n\nObjects created frequently in hot paths benefit from `__slots__`:\n- 30-40% less memory (no `__dict__`)\n- Faster attribute access (92% of local variable speed vs 59% for regular attributes)\n\n```python\n# Bad: regular class, dict-based attributes\nclass Row:\n def __init__(self, table, data):\n self.table = table\n self.data = data\n\n# Good: slotted class\nclass Row:\n __slots__ = ['table', 'data']\n def __init__(self, table, data):\n self.table = table\n self.data = data\n```\n\n### 2.5 Inner function overhead\n\nInner functions (closures) defined inside loops or hot functions are recreated on every call. This can be a **major** performance hit:\n\n```python\n# Bad: inner function recreated per call\ndef process(items):\n def transform(x): # created every time process() is called\n return x * 2\n return [transform(i) for i in items]\n\n# Good: module-level or method\ndef _transform(x):\n return x * 2\n\ndef process(items):\n return [_transform(i) for i in items]\n\n# Good: inline if simple\ndef process(items):\n return [x * 2 for x in items]\n```\n\nSigns to look for:\n- `def` inside another `def`\n- Lambda inside frequently-called function\n- Closures capturing loop variables\n\n\n### 2.6 Convert recursion to iteration\n\nRecursive functions have overhead per call (stack frame, argument passing) and risk stack overflow on deep structures. Convert to stack-based iteration:\n\n```python\n# Bad: recursive traversal\ndef flatten(obj, path=\"\"):\n if isinstance(obj, dict):\n for k, v in obj.items():\n yield from flatten(v, f\"{path}.{k}\") # recursive call\n else:\n yield path, obj\n\n# Good: stack-based iteration\ndef flatten(obj):\n stack = [(obj, \"\")]\n while stack:\n current, path = stack.pop()\n if isinstance(current, dict):\n for k, v in current.items():\n stack.append((v, f\"{path}.{k}\"))\n else:\n yield path, current\n```\n\nBenefits:\n- No function call overhead per level\n- Predictable memory (heap vs stack)\n- No recursion limit issues\n- Can be easier to add early-exit logic\n\nWARN USER:\n- stack method MAY change the order of processing and ie. yielded elements. ALWAYS suggest to write a test for original code.\n\n## Phase 3: Benchmark Everything\n\n### 3.1 Always benchmark caches\n\nCaches (lru_cache, manual dicts, memoization) can hurt performance if:\n- Cache miss is common (lookup overhead without benefit)\n- Cached computation is cheap (cache overhead exceeds saved work)\n- Memory pressure causes cache eviction\n\n**Benchmark caches even if they already exist** - they may have been added speculatively:\n\n```python\n# Does this cache actually help? Benchmark with and without!\n@lru_cache(maxsize=128)\ndef get_column_type(col_name):\n return self.schema.columns[col_name][\"data_type\"]\n\n# Compare:\n# 1. With cache (current)\n# 2. Without cache (direct lookup)\n# 3. Different cache size\n```\n\n### 3.2 Quick validation with timeit\n\nBefore and after each micro-optimization:\n```bash\npython -c \"import timeit; d={'a':1,'b':2}; print('copy:', timeit.timeit('dict(d)', globals={'d':d}, number=1000000))\"\npython -c \"import timeit; d={'a':1,'b':2}; print('ref:', timeit.timeit('x=d', globals={'d':d}, number=1000000))\"\n```\n\n### 3.3 Write dedicated benchmarks\n\nFor optimization variants, create benchmark scripts in `experiments/` folder:\n```python\n\"\"\"Benchmark: optimization variant comparison\"\"\"\nimport timeit\n\ndef variant_original(): ...\ndef variant_optimized(): ...\n\nprint(\"original:\", timeit.timeit(variant_original, number=10000))\nprint(\"optimized:\", timeit.timeit(variant_optimized, number=10000))\n```\n\n### 3.4 Process isolation and thermal management\n\nRun benchmarks in separate processes with pauses:\n```bash\npython experiments/bench.py devel_case\nsleep 10\npython experiments/bench.py optimized_case\n```\n\n### 3.5 Variance checking\n\nRun each benchmark 3+ times to verify stability:\n```bash\nfor i in 1 2 3; do\n echo \"=== Run $i ===\"\n python experiments/bench.py case_name\n sleep 5\ndone\n```\n\nVariance over 10-15% suggests external factors (thermal throttling, system load).\n\n### 3.6 Use realistic benchmark data\n\nUse real test data from `tests/normalize/cases/`:\n\n| File | Size | Use Case |\n|------|------|----------|\n| `ethereum.blocks.*.json` + `schemas/ethereum.schema.json` | 2MB | Deep nesting, warm/cold path normalizer |\n| `github.events.*.json` | 1.7MB | Dynamic table routing, many event types |\n| `github.issues.*.json` | 526KB | REST API, moderate nesting |\n\nLoad with `from dlt.common.json import json`. For flat rows with ISO timestamps, use `mimesis` (in dev deps) to generate synthetic DB data.\n\n### 3.7 Final benchmark: Optimized vs Devel\n\nAs the last step, compare against baseline branch:\n```bash\n# On devel branch (main repo)\ncd /path/to/main && python experiments/bench.py case_name\n\nsleep 10\n\n# On optimized branch (worktree)\ncd /path/to/worktree && python experiments/bench.py case_name\n```\n\nReport: `devel_time / optimized_time = X.XXx speedup`\n\n## Phase 4: Implement and Verify\n\n### 4.1 Make changes incrementally\n\nOne optimization at a time. Benchmark after each change.\n\n### 4.2 Verify correctness\n\n- Run existing tests: `make test-common`\n- Compare outputs before/after on sample data\n\n### 4.3 Document assumptions\n\nAdd comments explaining why optimization is safe:\n```python\n# columns dict is never mutated after get_table_columns() returns,\n# so we can store reference instead of copying\nself._current_columns = columns\n```\n\n## Phase 5: Report Results\n\nProduce a summary:\n```\n## Optimization Summary\n\n### Changes Made\n1. Replaced dict copy with identity check (buffered.py:100)\n2. Inlined count_rows_in_items for common case (buffered.py:102)\n\n### Benchmark Results\n| Case | Devel | Optimized | Speedup |\n|------|-------|-----------|---------|\n| flat_100 | 10.4s | 2.2s | 4.76x |\n| nested_20 | 4.9s | 2.0s | 2.47x |\n\n### Assumptions\n- columns dict is immutable after creation\n- 90% of items are plain dicts, not arrow/pandas\n```\n\n<!-- File: .claude/skills/release-notes/SKILL.md (Tokens: ~1406 | Category: anthropic-skill) -->\n---\nname: release-notes\ndescription: Generate release notes between two git tags with categorized PR summaries and author attribution\nargument-hint: <git-tag> [-- <previous-tag>]\n---\n\n# Generate Release Notes\n\nParse `$ARGUMENTS` to extract:\n- Everything before `--` is the **release tag** (e.g. `1.22.0`). Referred to as `TAG` below.\n- Everything after `--` is an optional **previous tag** override. If absent, auto-detect the previous tag.\n\n## Step 1: Validate the tag\n\nCheck that `TAG` exists as a git tag:\n\n```\ngit tag -l TAG\n```\n\nIf the tag does not exist, stop with an error: \"Tag `TAG` not found. Please create the tag first and re-run.\"\n\n## Step 2: Determine the previous tag\n\nIf a previous tag was provided in `$ARGUMENTS`, validate it exists and use it as `LOWER`.\n\nOtherwise, find the previous tag automatically:\n\n```\ngit tag --sort=-v:refname\n```\n\nWalk the sorted list to find the tag immediately before `TAG` (by version order). Set that as `LOWER`.\n\nValidate `LOWER` exists. Stop with an error if no previous tag can be determined.\n\n## Step 3: Gather commits\n\nGet all commits between the two tags:\n\n```\ngit log --oneline LOWER..TAG\n```\n\nFilter out:\n- Merge commits (starting with \"Merge pull request\" or \"Merge branch\")\n- Version bump commits (starting with \"bumps to version\" or \"bumps to\")\n\nExtract PR numbers from the remaining commit messages (pattern `#NNNN`). Every PR number must appear in the final release notes — none may be silently dropped.\n\n## Step 4: Fetch PR metadata\n\nFor each PR number, fetch metadata using `gh`. Use Task agents to run in parallel batches for efficiency:\n\n```\ngh pr view NUMBER --repo dlt-hub/dlt --json number,title,labels,body,author\n```\n\nCollect for each PR:\n- **number**: PR number\n- **title**: PR title\n- **author**: `author.login`\n- **labels**: list of label names\n- **body**: PR description (first 500 chars sufficient for categorization)\n\n### Handling low-quality PR titles or descriptions\n\nIf a PR title is vague (e.g. just an issue number, single word) or the body is empty/unhelpful:\n\n1. Check if the PR references an issue (e.g. `fixes #NNN`, `closes #NNN`, `#NNN` in body). If so, fetch the issue:\n ```\n gh issue view NNN --repo dlt-hub/dlt --json title,body\n ```\n2. If still unclear, look at the PR diff to understand what changed:\n ```\n gh pr diff NUMBER --repo dlt-hub/dlt\n ```\n3. Write a clear, accurate summary based on what you find.\n\n## Step 5: Categorize PRs\n\nAssign each PR to exactly one category using these rules (evaluated in order). **Every PR must be classified** — verify the total count matches.\n\n### Breaking Changes\nCollect separately (appears at the top, not a standalone category — the PR also appears in its main category). A PR has breaking changes if:\n- It has a `breaking` label, OR\n- Its body contains \"breaking change\" (case-insensitive), OR\n- Its title contains \"breaking\"\n\nExtract the specific breaking change description from the PR body.\n\n### Highlights\nA PR is a highlight if:\n- It has a `highlight` label, OR\n- It is a significant new feature (title starts with `feat` or `(feat)`) AND the change is impactful — meaning it introduces a new user-facing capability, enables a major use case, or represents substantial effort\n- Use judgment: not every feature is a highlight. A highlight should be something users would be excited about.\n\nHighlighted PRs also appear in their main category (Core Library, etc.) but with a brief entry pointing to the Highlights section.\n\n### Docs\n- Title or body indicates documentation-only changes\n- Files changed are only in `docs/`\n- Has `documentation` label\n\n### Chores\n- CI/CD improvements, test infrastructure, repo maintenance\n- Has labels like `ci full`, `tech-debt`, `ci`\n- Title starts with `chore`, `repo:`, `ci:`, or the change only affects tests/tooling\n\n### Core Library\nEverything else — features, bug fixes, enhancements to the library itself.\n\n## Step 6: Identify new contributors\n\nFor each release author, find their first ever merged PR:\n\n```\ngh search prs --repo dlt-hub/dlt --author AUTHOR --merged --sort created --order asc --limit 1 --json number,createdAt\n```\n\nA contributor is **new** if their first merged PR number falls within the `LOWER..TAG` commit range.\n\n## Step 7: Format the release notes\n\nWrite the notes in this exact structure:\n\n```markdown\n## dlt TAG Release Notes\n\n## Breaking Changes\n\n(Only if there are breaking changes. Numbered list.)\n\n1. **Short description** (#NUMBER @author) — Explanation of what broke and how it affects users.\n\n## Highlights\n\n- **Feature name** (#NUMBER @author) — Description of the feature and why it matters.\n\n## Core Library\n\n- **Feature/fix name** (#NUMBER @author) — Brief description.\n- **Fix: short description** (#NUMBER @author)\n\n## Docs\n\n- Description (#NUMBER @author)\n\n## Chores\n\n- **Title** (#NUMBER @author) — Brief description if non-obvious.\n\n## New Contributors\n\n- @author made their first contribution in #NUMBER\n```\n\n## Formatting rules:\n- PR references use `#NUMBER` format (GitHub auto-links them)\n- Authors use `@login` format (GitHub auto-links them)\n- Features and significant fixes get bold titles with em-dash descriptions\n- Minor fixes can be single-line bold entries without descriptions\n- Omit empty sections entirely (including New Contributors if none)\n- Within Core Library, group features first, then fixes\n\n## Step 8: Present for approval\n\nDisplay the formatted release notes to the user for review. Include a summary:\n- Total PRs analyzed\n- Breakdown by category (with counts)\n- Number of unique contributors (and how many are new)\n- List any PRs where you had to infer the description from issue or diff\n\nAsk the user to approve or request changes before writing.\n\n## Step 9: Write to file\n\nOnly after user approval, write the release notes to `/tmp/release-notes-TAG.md`.\n\nReport the file path to the user.\n\n<!-- File: .claude/skills/review-issues-severity/SKILL.md (Tokens: ~1064 | Category: anthropic-skill) -->\n---\nname: review-issues-severity\ndescription: Find and prioritize open GitHub issues by severity, community impact, and maintainer abandonment\nargument-hint: [-- <optional filters or focus areas>]\n---\n\n# Review Issues by Severity\n\nParse `$ARGUMENTS` for optional filters:\n- Label filters (e.g., `question`, `bug`)\n- Focus areas (e.g., \"merge disposition\", \"Arrow\", \"ClickHouse\")\n- If absent, perform a broad review across all open issues.\n\n## Steps\n\n### 1. Gather candidate issues\n\nRun the following GitHub API queries **in parallel** to surface issues matching different severity signals:\n\n#### a) Long discussions (high comment count)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&per_page=100&sort=comments&direction=desc&page=1' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### b) Issues with specific labels (if filters provided)\n```bash\ngh api 'repos/dlt-hub/dlt/issues?state=open&labels=LABEL&per_page=100&sort=comments&direction=desc' \\\n --jq '.[] | \"\\(.number)\\t\\(.comments)\\t\\(.updated_at | split(\"T\")[0])\\t\\(.created_at | split(\"T\")[0])\\t\\([.labels[].name] | join(\",\"))\\t\\(.title)\"'\n```\n\n#### c) Recently stale issues (updated 2+ months ago, with maintainer comments)\nLook for issues where `updated_at` is old relative to the current date but had prior maintainer engagement.\n\n### 2. Deep-dive top candidates\n\nFor the top 10-15 most promising candidates, fetch full details using a subagent or parallel `gh issue view` calls:\n\n```bash\ngh issue view -R dlt-hub/dlt NUMBER --json title,body,comments,labels,createdAt,updatedAt,author,assignees\n```\n\nFor each issue, extract:\n- **Participants:** who commented, their association (MEMBER = maintainer, NONE/CONTRIBUTOR = community)\n- **Timeline:** when maintainers last engaged, how long since last response\n- **Problem description:** what breaks and under what conditions\n- **Reproduction quality:** is there a clean repro script?\n- **Current status:** fixed? workaround? stalled? abandoned?\n- **Production severity:** does this cause data loss, crashes, or silent corruption?\n\n### 3. Classify each issue\n\nApply these criteria to each issue:\n\n| Signal | What to look for |\n|--------|-----------------|\n| **Serious problem** | Data corruption, silent data loss, hard crashes, cascading failures, security issues |\n| **Long discussion** | 5+ comments, especially with multiple community reporters hitting the same issue |\n| **Maintainer abandoned** | Maintainer commented but last maintainer response is 2+ months old, no linked PR, reporter follow-ups unanswered |\n| **Question label** | Issue has `question` label — these often represent real bugs initially miscategorized |\n| **Community effort** | Reporter provided detailed repro scripts, root-cause analysis, or proposed fixes that went unacknowledged |\n\n### 4. Prioritize\n\nAssign priority tiers:\n\n- **P0 — Critical:** Silent data corruption, cascading failures, confirmed root cause with no fix. Production systems at risk.\n- **P1 — High:** Hard crashes in common deployment patterns, blocking features for significant user segments, confirmed bugs with stalled fixes.\n- **P2 — Medium:** Broken features in specific environments, intermittent failures, missing warnings that lead users astray.\n- **P3 — Low:** Documentation gaps, environment-specific issues with known workarounds, feature requests with community PRs pending review.\n\nWithin each tier, rank by:\n1. Seriousness of the problem in production (data loss > crash > degraded performance > inconvenience)\n2. Negative community impact if not solved (effort wasted by reporters, users hitting the issue independently)\n3. Amount of effort already invested by reporters (repro scripts, root-cause analysis, proposed PRs)\n\n### 5. Output format\n\nPresent results as:\n\n#### Per-issue analysis (grouped by priority tier)\nFor each issue:\n- **Issue link, title, comment count, last activity date**\n- **Problem:** 2-3 sentence description of what breaks\n- **Why this priority:** key severity signal\n- **Abandonment signal:** when maintainer last engaged, what was left unresolved\n- **Community effort:** what the reporter invested (repro scripts, analysis, PRs)\n\n#### Summary matrix\nA table with columns: Priority | Issue # | Type | Production Impact | Community Effort | Maintainer Status\n\n#### Cross-cutting patterns\nNote any clusters of related issues (e.g., multiple issues sharing a root cause) or systemic patterns in maintainer response.\n\n<!-- File: .claude/skills/review-pr/identifier-normalization.md (Tokens: ~675 | Category: anthropic-skill) -->\n---\nname: Identifier Normalization Check\ndescription: Ensures all table names, column names, and other identifiers pass through the schema naming convention normalizer — the single most common PR review finding in this repo.\n---\n\n# Identifier Normalization Check\n\n## Context\n\nDifferent destinations have incompatible naming rules (e.g., Athena/S3 Tables forbids leading underscores, Weaviate capitalizes identifiers, BigQuery has length limits). All identifiers must go through the schema's naming normalizer — raw string constants are never acceptable for table or column names in runtime code.\n\n## When to Check\n\nThose rules apply to INTERNAL `dlt` code that is intended to work on ANY destination. In that case raw identifiers (string literals) MUST be normalized.\n\n\nYou must known when you normalize RAW identifier that represents a table or column name vs. when you RE-NORMALIZE already normalized identifier - take it from `Schema` instance, `Dataset` or `Relation` and which comes in a Python variable.\n\nNote: nested tables `user__comments` and nested columns `issue__stat_count` are PATHS containing many ALREADY NORMALIZED identifiers.\n\nWhen re-normalizing (and WHEN IN DOUBT)\nUse \n- `normalize_tables_path`\n- `normalize_path`\n\nWhen normalizing a RAW identifier:\n\nUse \n- `normalize_table_identifier`\n- `normalize_identifier`\n\n## What to Check\n\n### 1. Raw String Identifiers\n\nLook for hardcoded table or column name strings used in SQL queries, schema lookups, or API calls without normalization:\n\n```python\n# BAD — raw string identifier\ntable_name = \"my_table\"\nsql = f\"SELECT * FROM {table_name}\"\n\n# GOOD — normalized through schema naming\ntable_name = schema.naming.normalize_table_identifier(\"my_table\")\nsql = f\"SELECT * FROM {sql_client.make_qualified_table_name(table_name)}\"\n```\n\nCode in `dlt/dataset/` that builds queries or accesses tables must normalize identifiers:\n\n```python\n# BAD\ndataset[\"_dlt_loads\"]\n\n# GOOD\ndataset[schema.naming.normalize_table_identifier(\"_dlt_loads\")]\n```\n\nIn `dlt/destinations/impl/*/`:\n\n- Table names passed to SQL must use `sql_client.make_qualified_table_name()`\n- Column names in SQL must be escaped/normalized\n- Internal dlt tables (like `_dlt_loads`, `_dlt_version`, `_dlt_pipeline_state`) must still be normalized\n\n### 2. Re-normalizing identifiers\nUse this path when in doubt!\nWhen re-normalizing identifiers (ie. naming convention changed) or when identifier is a variable you MUST use:\n* `normalize_tables_path` - for table identifiers\n* `normalize_path` - for anything else ie. columns\n\nExample: a pyarrow table in transfer from destination 1 (now a source) to destion 2. Columns are normalized per\ndestination 1 and we re-normalize for destination 2:\n\n```python\ndef get_normalized_arrow_fields_mapping(schema: pyarrow.Schema, naming: NamingConvention) -> StrStr:\n \"\"\"Normalizes schema field names and returns mapping from original to normalized name. Raises on name collisions\"\"\"\n # use normalize_path to be compatible with how regular columns are normalized in dlt.Schema\n norm_f = naming.normalize_path\n```\n\n<!-- File: .claude/skills/review-pr/new-destination-checklist.md (Tokens: ~736 | Category: anthropic-skill) -->\n# New destination review checklist\n\nUse this checklist when reviewing a PR that adds a new dlt destination. These items are in addition to the standard review steps.\n\n## 1. Shared destination test integration\n\n- Check `tests/load/utils.py` `destinations_configs()` — is the new destination included in the right config groups (`default_vector_configs`, `default_sql_configs`, `read_only_sqlclient_configs`, etc.)?\n- Check `tests/load/test_read_interfaces.py` — does the destination need special handling (chunk size skips at `_chunk_size()`, view-only table listing exclusions in `test_ibis_dataset_access`)?\n- Check `tests/load/pipeline/test_restore_state.py` — state sync tests run via the config groups above, verify the destination participates.\n- Check `tests/utils.py` — is the destination in `IMPLEMENTED_DESTINATIONS` and `NON_SQL_DESTINATIONS` (if applicable)?\n\n## 2. Capabilities configuration\n\n- Verify `_raw_capabilities()` in the factory: loader formats, merge strategies, replace strategies, type mapper, nested types, decimal precision, timestamp precision, recommended file size.\n- Compare to the blueprint destination (ducklake, lancedb, filesystem) — are any caps missing or misconfigured?\n- Check `parquet_format` settings if the destination writes parquet with nested types.\n\n## 3. `WithLocalFiles` integration\n\n- If the destination stores data locally (files, embedded databases), the top-level `DestinationClient*Configuration` **must** inherit `WithLocalFiles` — otherwise the pipeline cannot bind `pipeline_name`, `pipeline_working_dir`, `local_dir` into the config.\n- If storage config is nested (e.g. `storage: FilesystemConfiguration`), `on_resolved()` must call `self.storage.attach_from(self)` followed by resolve or `normalize_bucket_url()`. Reference: ducklake pattern in `ducklake/configuration.py`.\n- **Test it**: verify default relative paths resolve to `local_dir`, not cwd. Verify `pipeline_name` and `pipeline_working_dir` propagate to nested storage config.\n\n## 4. SQL client — `WithTableScanners`\n\n- If the destination provides a read-only SQL interface via DuckDB views (like lance, lancedb, filesystem), it should extend `WithTableScanners` (`dlt/destinations/impl/duckdb/sql_client.py`), not `DuckDbSqlClient` directly.\n- `WithTableScanners` provides: in-memory DuckDB management, schema creation, sqlglot-based lazy view loading, `create_views_for_all_tables()` for ibis offload.\n- The subclass implements three abstract methods: `create_view()`, `can_create_view()`, `should_replace_view()`.\n\n## 5. Ibis integration\n\n- Check `dlt/helpers/ibis.py` — is there a branch for the new destination's config class?\n- For `WithTableScanners`-based destinations: use `sql_client.create_views_for_all_tables()` (not manual table iteration).\n- Add the destination to the view-only exclusion list in `test_ibis_dataset_access` if it uses DuckDB views (won't see tables from other schemas).\n- The ibis branch must come before any parent config class check to avoid wrong dispatch via `issubclass`.\n\n## 6. Import rules\n\n- Never import optional packages directly — use `dlt.common.libs.*` wrappers (`pyarrow`, `pandas`, `numpy`, etc.).\n- Private API imports from third-party libs (underscore-prefixed functions) are fragile — flag them.\n\n<!-- File: .claude/skills/review-pr/SKILL.md (Tokens: ~1947 | Category: anthropic-skill) -->\n---\nname: review-pr\ndescription: Analyze a GitHub pull request including diff, comments, related issues, and local code context\nargument-hint: <pr-url-or-number> [-- <optional instructions>]\n---\n\n# Review Pull Request\n\nParse `$ARGUMENTS` to extract the PR identifier and optional reviewer instructions:\n- Everything before the first `--` (or all of it, if there is no `--`) is the **PR identifier** (URL or number). Referred to as `PR_ID` below.\n- Everything after the first `--` is **reviewer instructions** — additional focus areas or context from the caller. If absent, there are no special instructions.\n\nAnalyze the pull request: PR_ID\n\n## Steps\n\nRun the following steps. Gather data in parallel where possible.\n\n### 1. Set up worktree\n\nResolve the PR number:\n\n```\ngh pr view PR_ID --json number\n```\n\nUse the number to form the worktree name `review-pr-<number>`.\n\nInvoke the `/create-worktree review-pr-<number> --pr <number>` skill to create an isolated worktree for this PR. Note the worktree path from the skill output — referred to as `WORKTREE` below.\n\nVerify that the cwd is now inside the worktree:\n\n```\npwd\n```\n\nIf `pwd` does not show `WORKTREE`, run `cd WORKTREE` and verify again. Stop with an error if the cwd cannot be set. Once confirmed, subsequent Bash calls will run inside the worktree automatically.\n\n### 2. Fetch PR metadata and diff\n\nUse `gh` to collect all PR information in parallel:\n\n- `gh pr view PR_ID --json title,body,author,state,baseRefName,headRefName,number,url,comments,reviews,labels,milestone`\n- `gh pr diff PR_ID`\n\n### 3. Check dependency changes in `pyproject.toml`\n\nIf the PR diff (from step 2) touches `pyproject.toml`, run the automated dependency checker.\n\nFirst, fetch the latest `devel` from origin so the comparison is against the current state:\n\n```\ngit fetch origin devel\n```\n\nThen run the script — always compare against `origin/devel` regardless of the PR's base branch:\n\n```\npython tools/check_dependency_changes.py origin/devel\n```\n\nThe script finds the merge-base, simulates a three-way merge (like GitHub), and reports:\n- **Main dependencies** (`[project.dependencies]`) — displayed as WARNING (these affect every user)\n- **Optional dependencies** (`[project.optional-dependencies]`) — displayed as WARNING (these affect users of specific extras)\n- **Dev/group dependencies** (`[dependency-groups]`) — listed for awareness (no impact on end users)\n- **Merge conflicts** — if `pyproject.toml` would conflict, the script warns and falls back to showing the PR's intended changes\n\nInclude the script output verbatim in the review. If the script exits with code 1 (changes detected), flag dependency changes in the review verdict. If it exits with code 2 (conflicts), flag that `pyproject.toml` needs conflict resolution.\n\nIf `pyproject.toml` is not in the diff, skip this step.\n\n### 4. Check CI lint and mypy status\n\nUse `gh pr checks PR_ID` to retrieve the CI check results. Look for checks matching `lint on all python versions / lint*` — these run mypy, ruff, flake8, bandit, and docstring checks across Python 3.9–3.13 (defined in `.github/workflows/lint.yml`).\n\n- Report the status of each lint check (pass/fail/pending).\n- If any lint check has failed, fetch its logs with `gh run view <run-id> --log-failed` to identify the specific errors (mypy type errors, ruff violations, etc.).\n- Include lint/mypy status in the final review verdict. A failing lint check should be flagged as a required fix.\n\n### 5. Analyze related issues\n\nExtract any issue references from the PR body (e.g. `fixes #1234`, `closes #1234`, `#1234`). For each referenced issue, fetch it with:\n\n- `gh issue view <number> --json title,body,author,state,comments,labels`\n\nUnderstand the original problem being solved.\n\n### 6. Analyze the code\n\nUsing absolute paths under `WORKTREE`:\n\n- Read the changed files in full to understand context beyond the diff.\n- Look at callers/callees of modified functions.\n- Check if similar patterns exist elsewhere that should also be updated.\n- Review the coding style and code patterns to conform with @CLAUDE.md\n- If **reviewer instructions** were provided, pay special attention to the areas or concerns they describe during this analysis.\n- Mention only crucial finding to the user in the initial overview. Avoid code snippets except if critical bugs were found\n\n### 6a. Pay attention to special cases\n\nSome areas of the codebase have rules that are easy to miss in a generic code read. The references below are intentionally not loaded by default — read the linked file ONLY when the PR's diff matches the trigger, and apply its checklist before producing the review.\n\n- **Identifier normalization** — read [identifier-normalization.md](identifier-normalization.md) when the PR's diff includes any of: `dlt/destinations/impl/**`, `dlt/dataset/**`, `dlt/destinations/sql_client*`, code that builds SQL/relations/lineage, or schema-generation paths involving sqlglot or ibis. Covers when raw-string table/column identifiers must pass through the schema's naming normalizer, and how to distinguish first-time normalization from re-normalization.\n\n### 7. Review test coverage\n\n#### 7a. Define the testing scope\n\nBased on the PR changes, identify what needs to be tested:\n\n- What are the distinct behaviors introduced or modified?\n- What are the edge cases and error paths?\n- What are the integration boundaries (e.g., does the change affect how components interact)?\n\nProduce a checklist of required test scenarios before evaluating existing tests.\n\n#### 7b. Evaluate existing tests against the checklist\n\n- Compare the PR's tests against the required scenarios checklist. Flag any gaps.\n- Flag any tests that verify things outside the PR's scope or test obvious/trivial behavior that adds no value.\n\n#### 7c. Verify tests are in the right modules\n\n- Search the test directory for where similar functionality is already tested (grep for related function names, class names, or feature keywords).\n- New tests should live alongside related existing tests, not in new files unless the PR introduces an entirely new module.\n- Flag tests placed in the wrong location.\n\n#### 7d. Check for duplication and over-testing\n\n- Look for tests that cover nearly identical cases and could be merged or parametrized with `@pytest.mark.parametrize`.\n- Look for repeated setup/assertion logic across tests that should be extracted into a helper or fixture.\n- If multiple tests differ only in a single variable (e.g., `True`/`False`, different type values), they must be parametrized into one test function.\n- Flag any copy-paste test code. Minimizing test code maintenance burden is a priority.\n- Flag local path assertions that are not Windows compatible.\n\n#### 7e. Bug Fixes Should Have Regression Tests\nIf the PR fixes a bug:\n- Is there a test that would have caught the bug before the fix?\n- Does the test verify the specific scenario from the bug report?\n\n### 8. Check documentation coverage\n\nIf the PR modifies public API (new parameters, changed behavior, new configuration options, new functions/classes exposed to users):\n\n- Search `WORKTREE/docs/website/docs/` for existing documentation covering the topic of the PR.\n- If the topic is already documented and the PR changes its behavior or adds new options, verify that the PR also updates the relevant doc pages.\n- If documentation updates are missing, flag this in the review as a required change.\n- If the topic is not documented at all, this is not a blocker — don't require docs for previously undocumented internals.\n\n### 9. Produce the review\n\nWrite a structured analysis covering:\n\n- **Summary**: What the PR does and why, referencing the linked issue.\n- **Changes**: Breakdown of changes should be concise, avoid code samples\n- **Dependency Changes**: If `pyproject.toml` was modified, include the warnings and listings from step 3. Main and optional dependency changes must appear as yellow WARNING blocks. Omit this section if no dependency changes were found.\n- **CI Lint & Mypy**: Status of lint checks across Python versions. If any failed, list the specific errors.\n- **Observations & Potential Issues**: Bugs, edge cases, missing error handling, performance concerns, memory issues, security issues, backward compatibility.\n- **Breaking Changes and Public API**: Follow the rules on breaking changes and Public API and report issues.\n- **Test Coverage**: Required scenarios checklist with pass/fail status, misplaced tests, duplication issues, parametrization opportunities.\n- **Documentation**: Whether doc updates are needed and if they were provided.\n- **Reviewer Instructions**: If reviewer instructions were provided, include a dedicated section addressing each point raised. If no instructions were provided, omit this section.\n- **Verdict**: Overall assessment.\n\n<!-- File: .claude/skills/review-vocabulary/references/simple-english-checklist.md (Tokens: ~813 | Category: anthropic-skill) -->\n# Verification checklist\n\nRun this pass on every draft before you deliver it. The checks are ordered from mechanical to judgment.\n\n## Mechanical checks (searchable)\n\nSearch the draft for each pattern. Every hit outside code blocks and quoted text is a violation.\n\n| Search for | Violation | Fix |\n|---|---|---|\n| `'ll`, `'re`, `'ve`, `n't`, `it's` | Contraction (Rule 4.2) | Expand it. |\n| `has been`, `have been`, `had been` | Present/past perfect (Rule 3.4) | Simple past or simple present. |\n| `has` / `have` + past participle | Present perfect (Rule 3.4) | Simple past. |\n| `should`, `would`, `may`, `might`, `could` | Unapproved modal (Rule 3.2) | See the modal ladder in SKILL.md. |\n| `is being`, `are being`, `was being` | Progressive passive (Rules 3.4, 3.5) | Active, simple tense. |\n| `, making`, `, allowing`, `, enabling`, `, ensuring` | \"-ing\" clause as verb (Rule 3.5) | New sentence with a real subject. |\n| `;` | Semicolon (Rule 8.1) | Two sentences. |\n| `e.g.`, `i.e.`, `etc.` | Latin abbreviation (GR-6) | \"for example\", \"that is\", name the items. |\n| `simply`, `easily`, `seamlessly`, `robust` | Filler (no fact) | Delete. |\n| ` if `, ` when ` (mid-sentence) | Trailing condition (Rule 5.4) | Move the condition to the start of the sentence, add a comma. |\n\n## Countable checks\n\n1. **Sentence length.** Count words in each sentence. Procedural limit: 20. Descriptive limit: 25. Notes: 25.\n Backticked commands, numbers with units, and identifiers count as one word each (Rule 8.6).\n2. **Paragraph size.** Maximum six sentences per paragraph (Rule 6.6).\n3. **Multi-word nouns.** Any noun chain over three words → break it with prepositions (Rule 2.1).\n4. **Instructions per sentence.** One, unless the actions are simultaneous (Rule 5.2).\n\n## Judgment checks\n\n5. **Classification.** Is each passage cleanly procedural or descriptive? Procedures in imperative, descriptions never in imperative.\n6. **Voice.** Any passive sentence: is the agent truly unknown, and is the passage descriptive? Otherwise make it active (Rule 3.6).\n7. **Condition placement.** Every \"if/when\" stands before its command, with a comma (Rule 5.4).\n8. **Synonym rotation.** One term per concept across the whole document (Rules 1.11, 9.4). Scan for check/verify/confirm, config/settings, run/execute.\n9. **Warnings.** Command or condition first, risk second (Rules 7.2, 7.3).\n10. **Completeness.** Articles present, \"that\" present after \"make sure\", no telegraph style (Rule 4.2).\n11. **Untouchables intact.** Code, identifiers, quoted errors, and proper nouns are unchanged.\n\n## When reporting violations (check mode)\n\nFor each violation give: the rule number, the offending text, and a compliant rewrite. Cite only rule numbers that appear in rules.md.\nEnd the report with this statement when the user asked for STE compliance: \"No tool can guarantee ASD-STE100 compliance. Final approval rests with the writer. The official standard is a free download at asd-ste100.org.\"\n\n<!-- File: .claude/skills/review-vocabulary/references/simple-english-use-cases.md (Tokens: ~857 | Category: anthropic-skill) -->\n# Use cases beyond documentation\n\nSTE was built for aircraft maintenance manuals. The same properties — one meaning per word, short sentences, condition-first commands — transfer to any text where misreading has a cost. By the end of Issue 8, 64% of registered STE users were outside aerospace and defense.\n\nEach case below names the mode and the adaptations.\n\n## Error messages and CLI output\n\nMode: procedural. This is the highest-value target: an error message is a 2 a.m. instruction to a stressed reader.\n\nPattern: state what happened (past simple), state the cause if known, give the command or condition to fix it.\n\n> **Before:** Oops! Something went wrong while attempting to establish a connection. Please ensure your credentials are properly configured and try again.\n> **After:** Connection to the database failed. The password for user `app` was not correct. Set `DB_PASSWORD` and connect again.\n\n## Runbooks and standard operating procedures\n\nMode: strict-leaning procedural. This is STE's home turf — an on-call runbook is a maintenance manual.\n\n- Every step imperative, one instruction per step, conditions first.\n- Warnings before the step, command first, risk second.\n- 20-word limit enforced hard: an operator under pager stress reads each sentence once.\n\n## Incident reports and postmortems\n\nMode: descriptive. Simple past only — a timeline in present perfect (\"we have identified...\") hides when things happened.\n\n> **Before:** We have identified an issue that may have impacted some users' ability to access the service.\n> **After:** Between 14:02 and 14:31 UTC, 12% of requests failed. A deploy at 14:00 removed the cache warmup step.\n\nSTE bans hedges (\"may have impacted\") — the report states what is known and says \"unknown\" for the rest. This reads more honest because it is.\n\n## Commit messages and PR descriptions\n\nMode: descriptive body, imperative subject. Convention already matches STE: imperative subject line, plain past facts in the body. Apply the substitution table and the 25-word limit to the body. Delete \"this PR aims to\".\n\n## API changelogs and release notes\n\nMode: descriptive. One entry, one change, one sentence where possible. \"Breaking:\" entries follow the warning pattern — command first: \"Update your calls to `v2/users`. The `name` field split into `first_name` and `last_name`.\"\n\n## Instructions for AI agents (prompts, AGENTS.md, skills)\n\nMode: procedural. A system prompt is a procedure executed by a reader with no ability to ask questions — the exact reader STE was designed for.\n\n- One instruction per sentence keeps rules independently quotable and hard to half-follow.\n- One word, one meaning prevents the model from treating \"check\", \"verify\", and \"validate\" as three different operations.\n- Condition-first (\"If the build fails, stop\") beats trailing conditions, which models drop.\n- No \"should\" — a model reads \"should\" as optional. Write \"must\" or delete the rule.\n\n## Support macros and status-page updates\n\nMode: descriptive, 25-word limit. Non-native readers are the majority of many user bases. No \"we sincerely apologize for any inconvenience this may have caused\" — \"The API was down for 18 minutes. Uploads made during this time were saved and will process today.\"\n\n## Translation and localization prep\n\nMode: strict. STE's original purpose was making English readable for non-native maintenance crews, and it doubles as pre-editing for machine translation. One meaning per word plus complete grammar (articles, \"that\") removes most translation ambiguity. If your docs get localized, STE cuts the error rate and the cost.\n\n## UI copy and empty states\n\nMode: procedural, hard length limits. Buttons and labels are technical names (exempt). Body copy follows the rules: \"No projects yet. Create a project to start.\" Nothing else survives at this length anyway.\n\n## Where STE does not fit\n\nMarketing pages, launch posts, blog voice, brand writing. STE deletes persuasion on purpose. Write those in your own voice — then use STE for the docs the landing page links to.\n\n<!-- File: .claude/skills/review-vocabulary/references/simple-english.md (Tokens: ~4650 | Category: anthropic-skill) -->\n---\nname: simple-english\nversion: 1.0.0\ndescription: |\n Write or rewrite technical text with the rules of ASD-STE100 Simplified\n Technical English so it is clear, unambiguous, and free of AI slop. Use for\n documentation, READMEs, runbooks, procedures, error messages, release notes,\n incident reports, and API guides. Also use when the user says \"STE\",\n \"Simplified Technical English\", \"ASD-STE100\", \"de-slop\", \"make this\n readable\", \"write for non-native readers\", or asks for docs that translate\n well. Enforces the standard's 53 rules: 20/25-word sentence limits, one word\n one meaning, simple tenses, active voice, condition before command.\nlicense: MIT\ncompatibility: claude-code cursor codex gemini-cli opencode\nmetadata:\n standard: ASD-STE100 Issue 9 (2025-01-15)\n---\n\n# Simple English: Write Like an Aerospace Manual\n\nWrite technical text with the rules of ASD-STE100 Simplified Technical English. STE is the controlled language that aerospace and defense manufacturers use for maintenance documentation. The rules exist so that a tired reader who is not a native English speaker cannot misread an instruction. They remove the usual signs of AI-generated text as a side effect: long sentences, synonym rotation, hedges, filler, and decorative clauses.\n\nWrite for that tired reader. Each sentence must survive one read.\n\n## Your Task\n\nWhen asked to write or rewrite technical text:\n\n1. **Select the mode** (pragmatic or strict, below).\n2. **Classify each passage** as procedural or descriptive. Every other rule depends on this.\n3. **Fix your vocabulary before drafting.** Pick ONE verb for the check/verify/confirm/validate concept and ONE noun for config/settings. Use no other word for these concepts in the whole document.\n4. **Apply the rules** from the catalog below.\n5. **Run the self-check** before you deliver. This step is not optional.\n6. **Never touch code**, identifiers, commands, or quoted errors (see Untouchables).\n\nWhen asked to CHECK text instead of writing it, report each violation as: rule number, the offending text, a compliant rewrite. Cite only rule numbers that exist in this file. Do not cite rule numbers from memory: the numbering is unintuitive and models invent it (tested — an agent without this file cited \"Rule 3.1: short sentences\"; the real Rule 3.1 is about verb forms).\n\n## Two Modes\n\n| Mode | When | What you apply |\n|---|---|---|\n| **Pragmatic** (default) | Docs, READMEs, error messages — the user wants clear text | All structural rules. Domain words stay (\"idempotent\", \"webhook\"). |\n| **Strict** | The user names STE, ASD-STE100, or compliance | Structural rules + full vocabulary discipline, and tell the user that full compliance needs the official dictionary (free at asd-ste100.org). |\n\n## Step 1: Classify the Text\n\n| | Procedural (instructions) | Descriptive (explanations) |\n|---|---|---|\n| Purpose | Tell the reader what to do | Explain what a thing is or does |\n| Verb form | Imperative: \"Install the pump.\" | Simple present/past/future |\n| Sentence limit | **20 words** (Rule 5.1) | **25 words** (Rule 6.3) |\n| Unit rule | One instruction per sentence (5.2) | One topic per paragraph (6.5), max six sentences per paragraph (6.6) |\n\nDo not mix the two in one passage. A \"Getting started\" section is procedural. An \"Architecture\" section is descriptive. A note inside a procedure is descriptive (25-word limit, no imperative).\n\n## THE RULE CATALOG\n\n53 rules in 9 sections, paraphrased from ASD-STE100 Issue 9 with software examples. The official wording is in the free standard at asd-ste100.org.\n\n### Section 1 — Words (Rules 1.1-1.14)\n\n| Rule | Instruction |\n|---|---|\n| 1.1 | Use only approved words, technical nouns, or technical verbs. |\n| 1.2 | Use an approved word only as its listed part of speech. |\n| 1.3 | Use an approved word only with its approved meaning. |\n| 1.4 | Use only the approved forms of verbs and adjectives. |\n| 1.5 | You can use domain words as technical nouns (\"webhook\", \"commit\", \"endpoint\"). |\n| 1.6 | Use an unapproved word only when it is a technical noun or part of one. |\n| 1.7 | Do not use technical nouns as verbs. |\n| 1.8 | Use the technical nouns of your project or industry. |\n| 1.9 | When you pick a technical noun, pick a short and clear one. |\n| 1.10 | No regional, slang, or jargon words as technical nouns. |\n| 1.11 | One item, one name. Do not call it \"config\" here and \"settings\" there. |\n| 1.12 | You can use domain verbs as technical verbs (\"deploy\", \"compile\", \"merge\"). |\n| 1.13 | Do not use technical verbs as nouns. |\n| 1.14 | Use American English spelling. |\n\nIn pragmatic mode, rules 1.5, 1.8, and 1.12 do the heavy lifting: your domain vocabulary is legal. The ones agents break are 1.7, 1.11, and 1.13.\n\n**Before:** You can webhook the event, then do a deploy.\n**After:** Send the event to the webhook. Then deploy the service.\n\n### Section 2 — Multi-word nouns (Rules 2.1-2.2)\n\n| Rule | Instruction |\n|---|---|\n| 2.1 | Write multi-word nouns of three words or fewer. |\n| 2.2 | When a technical noun needs more than three words, write it in full once, then give a short form or hyphenate the units. |\n\nBreak long noun chains with prepositions (of, on, in, for):\n\n**Before:** the connection pool timeout configuration value\n**After:** the timeout value for the connection pool\n\n### Section 3 — Verbs (Rules 3.1-3.7)\n\n| Rule | Instruction |\n|---|---|\n| 3.1 | Use only the verb forms that the dictionary gives. |\n| 3.2 | Use only: infinitive, imperative, simple present, simple past, simple future, past participle as adjective. |\n| 3.3 | Use the past participle only as an adjective (\"the cached response\"). |\n| 3.4 | No auxiliary verbs for complex constructions. No present perfect, no \"is to be installed\". |\n| 3.5 | Use an \"-ing\" form only as a technical noun or inside one (\"logging\", \"the mounting bracket\") — never as a verb. |\n| 3.6 | Active voice. In descriptive text, passive is legal only when the agent is unknown. |\n| 3.7 | Describe an action with a verb, not a noun (\"compress the file\", not \"perform compression of the file\"). |\n\n**Approved modals: can, will, must. Banned: should, would, may, might, could.**\nThe standard rejects \"could\" even for possibility: write \"an explosion can occur\", never \"could occur\". For \"should\": a requirement becomes \"must\"; a suggestion is stated as fact or deleted. This matters double for agent instructions — models read \"should\" as optional.\n\n**Before:** The migration has completed and the table is being rebuilt.\n**After:** The migration is complete. The database rebuilds the table.\n\n**Before:** The flag can be set in the config file, making restarts unnecessary.\n**After:** You can set the flag in the config file. Then a restart is not necessary.\n\n**Before:** The temperature must be adjusted.\n**After:** Adjust the temperature.\n\n### Section 4 — Sentences (Rules 4.1-4.5)\n\n| Rule | Instruction |\n|---|---|\n| 4.1 | Write short and clear sentences. |\n| 4.2 | Do not omit words or use contractions to shorten sentences. Keep articles, keep \"that\". |\n| 4.3 | Use a vertical list for complex text. |\n| 4.4 | Use connecting words between sentences on related topics (\"Then\", \"As a result\"). |\n| 4.5 | Put an article (the, a, an) or a demonstrative adjective (this, these) before nouns where applicable. |\n\nRule 4.2 is the anti-terseness rule. STE is short sentences with complete grammar, not telegraph style:\n\n**Wrong shortening:** Ensure file exists before running.\n**STE:** Make sure that the file exists before you run the command.\n\n### Section 5 — Procedural writing (Rules 5.1-5.5)\n\n| Rule | Instruction |\n|---|---|\n| 5.1 | Maximum 20 words per sentence. Warnings and cautions included. |\n| 5.2 | One instruction per sentence, unless two actions happen at the same time. |\n| 5.3 | Write instructions in the imperative: \"Run the migration.\" |\n| 5.4 | Put a required condition before the command, divided by a comma: \"If the build fails, read the log.\" |\n| 5.5 | Notes give information, never instructions. Notes get the 25-word limit. |\n\n**Before:** You'll want to grab the API key from the dashboard before configuring the client, which you can do under Settings.\n**After:** Get the API key from the dashboard, under Settings. Then configure the client with this key.\n\n### Section 6 — Descriptive writing (Rules 6.1-6.6)\n\n| Rule | Instruction |\n|---|---|\n| 6.1 | Give information gradually: one new fact per sentence. |\n| 6.2 | Use key words and phrases to give the text a logical structure. |\n| 6.3 | Maximum 25 words per sentence. |\n| 6.4 | Group related information in paragraphs. |\n| 6.5 | One topic per paragraph. |\n| 6.6 | Maximum six sentences per paragraph. |\n\nNo imperative in descriptive text. Descriptions explain; procedures instruct.\n\n### Section 7 — Safety instructions (Rules 7.1-7.3)\n\n| Rule | Instruction |\n|---|---|\n| 7.1 | Use a word that shows the risk level (\"WARNING\" = injury, \"CAUTION\" = damage). |\n| 7.2 | Start with a clear command or condition. |\n| 7.3 | Then give the risk or the possible result. |\n\nNever bury the instruction after the explanation. The pattern transfers directly to destructive CLI flags, irreversible migrations, and dangerous API options.\n\n**Before:** Note that data loss may occur in some circumstances if the destructive flag happens to be enabled when running against production.\n**After:** CAUTION: Do not use the `--force` flag against production. The flag deletes rows that do not match the source.\n\n### Section 8 — Punctuation and word count (Rules 8.1-8.7)\n\n| Rule | Instruction |\n|---|---|\n| 8.1 | All standard punctuation is legal except the semicolon. Write two sentences instead. |\n| 8.2 | Use hyphens to connect words that act as one unit. |\n| 8.3 | Parentheses are legal for references, item numbers, abbreviations, plural forms, explanations, alternatives. |\n| 8.4 | In a vertical list, the lead-in colon ends a sentence for word count. |\n| 8.5 | Text inside parentheses counts as one word. |\n| 8.6 | Count as one word each: numbers, numbers with units, abbreviations, alphanumeric identifiers, quoted text, titles, labels, proper nouns. |\n| 8.7 | A hyphenated word counts as one word. |\n\nRule 8.6 matters for software text: `sqlpipe run --config sqlpipe.yaml` in backticks is quoted text and counts as one word. Long identifiers do not blow your sentence budget.\n\n### Section 9 — Writing practices (Rules 9.1-9.4, GR-1 to GR-8)\n\n| Rule | Instruction |\n|---|---|\n| 9.1 | When a word-for-word replacement does not work, restructure the sentence. |\n| 9.2 | Use each approved word correctly: approved meaning, approved part of speech. |\n| 9.3 | Do not build phrasal verbs (\"go down\" → \"decrease\", \"set up\" → \"install\" or \"configure\"). |\n| 9.4 | Keep one consistent style and terminology through the whole document. |\n\nGeneral recommendations GR-1 to GR-8: keep the conjunction \"that\", be careful with \"with\", give pronouns clear referents, prefer \"this + noun\" over bare \"this\", avoid false friends, avoid Latin abbreviations, use inclusive language, and use the possessive apostrophe form only when you are sure it is correct (GR-8: if unsure, do not use it — non-native readers find it hard).\n\nGR-6 for software docs: \"e.g.\" → \"for example\", \"i.e.\" → \"that is\", and delete \"etc.\" — name the items or write \"and more\".\n\n## VOCABULARY DISCIPLINE\n\nThe official dictionary (~900 approved words, ~1,200 banned words with alternatives) is copyrighted by ASD and is not reproduced here. Its mechanics apply without it: **one word, one meaning, one part of speech.**\n\nKnown part-of-speech rulings, useful as patterns:\n\n| Word | Ruling |\n|---|---|\n| test, check, work | Noun only. \"Do a test\", not \"test the pump\". \"Check that X\" becomes \"make sure that X\". |\n| oil | Noun only as used in STE examples. For the verb, the dictionary gives \"lubricate\". |\n| help | Verb only. For the noun, the dictionary gives \"aid\": \"with the aid of\". |\n| fall | \"To move down by gravity\" only, never \"decrease\". |\n| follow | \"To come after\" only, never \"obey\". Write \"obey the instructions\". |\n| above, below | Physical positions only. For limits write \"more than\", \"less than\". |\n\n### The modal ladder\n\n| You wrote | STE writes |\n|---|---|\n| should (requirement) | must |\n| should (recommendation) | Delete it, or state it as fact: \"X is better because Y.\" |\n| may / might / could (possibility) | can |\n| may (permission) | can |\n| would (hypothetical) | Restructure: \"If X occurs, Y occurs.\" |\n\n### Slop-to-simple substitutions\n\nThis table is ours, not the ASD dictionary. It maps the words AI-generated docs overuse to plain replacements. If the word carries no fact, delete it instead of replacing it.\n\n| Slop | Write instead |\n|---|---|\n| leverage, utilize | use |\n| in order to | to |\n| prior to | before |\n| ensure | make sure that |\n| it is worth noting that | (delete) |\n| it's important to, crucially | (delete — state the fact) |\n| simply, just, easily, seamlessly, effortlessly | (delete) |\n| robust, powerful, comprehensive, performant | (delete, or give the measurable property) |\n| functionality | function, feature |\n| enables you to, allows you to | you can |\n| is designed to, aims to | (delete — say what it does) |\n| facilitate | help, make possible |\n| dive into, delve into | read, examine |\n| when it comes to | for |\n| in the event that | if |\n| due to the fact that | because |\n| as needed, as necessary | (state the condition) |\n| and/or | Pick one, or write \"X, or Y, or both\" |\n| e.g. / i.e. / etc. | for example / that is / (name the items) |\n| gracefully handles | (say what it does: \"retries three times, then stops\") |\n| out of the box | by default |\n| under the hood | internally |\n| blazingly fast, state-of-the-art | fast (give the number) / (delete) |\n| streamline | make simpler, make faster |\n| plethora, myriad | many |\n| addresses the issue, tackles | corrects the fault, removes the error |\n\n### Consistency pass\n\nCollapse these common rotations to one term each (Rules 1.11, 9.4):\n\n- check / verify / confirm / validate / ensure → pick one\n- config / configuration / settings / options → pick one\n- delete / remove / drop / destroy → one per meaning, kept consistent\n- error / issue / problem / failure → \"error\" for errors, \"failure\" for failed operations\n- run / execute / invoke / launch → pick one\n- show / display / render / present → pick one\n\n## Untouchables\n\nThese are technical names (Rules 1.5, 8.6). Leave them exact, even when they break vocabulary rules:\n\n- Code blocks, inline code, identifiers, CLI commands, flags, file paths\n- Quoted error messages and log lines\n- Product names, API endpoint names, config keys\n- Numbers with units — each counts as one word in the sentence limit\n\n## Beyond Documentation\n\nSame rules, different targets. Full adaptations in `references/use-cases.md`:\n\n- **Error messages**: state what happened (simple past), the cause if known, then the fix as an imperative. No \"Oops\", no \"Please ensure\", no apology filler.\n- **Runbooks**: STE's home turf. Imperative steps, conditions first, warnings before the step.\n- **Incident reports**: simple past only. \"We have identified an issue that may have impacted\" becomes \"Between 14:02 and 14:31 UTC, 12% of requests failed.\"\n- **Release notes**: breaking changes follow the warning pattern — command first, risk second.\n- **Agent instructions (prompts, AGENTS.md)**: a system prompt is a procedure for a reader that cannot ask questions. One instruction per sentence, no \"should\", condition first.\n- **Translation prep**: STE's original job. One meaning per word plus complete grammar removes most translation ambiguity.\n\n## Self-Check Before You Deliver\n\nThis step is not optional. Run these four checks on your draft:\n\n1. Count words in your three longest sentences. Over the 20/25 limit → split them.\n2. Search your draft for: `'ll`, `'re`, `'s` (contraction), `has been`, `have been`, `should`, `-ing` verbs after a comma, semicolons.\n3. Search for every `if` and `when`. Each one stands at the START of its sentence, before the command. \"Increase the timeout if the network is slow\" → \"If the network is slow, increase the timeout.\"\n4. Search for the verbs you did NOT pick in Your Task step 3 (the check/verify/confirm set). Replace every hit with your chosen verb.\n\nFix what you find, then deliver. For a full audit, run `references/checklist.md`.\n\n## Full Example\n\n**Before (real unedited AI output):**\n\n> **Connection timeouts.** If sqlpipe hangs or fails with `dial tcp: i/o timeout`, check that the host running sqlpipe can reach the Postgres port (usually 5432) — this is often a security group or firewall rule blocking the connection. If you're connecting to a managed database (RDS, Cloud SQL, etc.), confirm the instance allows connections from sqlpipe's IP. You can also try increasing `source.connect_timeout_seconds` in your config, since a slow network path can trip the default timeout even when the connection eventually succeeds.\n\n**After (classified procedural, verb = \"make sure\", conditions first, one instruction per sentence):**\n\n> **Connection timeouts.** sqlpipe stops with `dial tcp: i/o timeout` when it cannot reach the Postgres port (5432 by default).\n>\n> 1. Make sure that the host that runs sqlpipe can reach the Postgres port. A firewall or security group usually blocks it.\n> 2. If the database is managed (RDS, Cloud SQL), make sure that the instance accepts connections from the IP of sqlpipe.\n> 3. If the network is slow, increase `source.connect_timeout_seconds` in the configuration.\n\nWhat changed: 40-word sentences split under 20; \"you're\" expanded; \"check/confirm\" collapsed to \"make sure that\"; every condition moved before its command; \"etc.\" removed; code and error strings untouched.\n\n## Limits\n\nSTE is for technical facts and instructions. Do not apply it to marketing copy, blog voice, or brand writing — it deletes persuasion by design. When a user asks for STE on marketing text, say so and offer it for the docs instead.\n\nThis skill is an unofficial aid. It is not affiliated with or endorsed by ASD or STEMG, and no tool can guarantee STE compliance. ASD-STE100 is a registered trademark of ASD. The official standard is a free download at asd-ste100.org.\n\n## References\n\n- `references/checklist.md` — full verification pass with searchable patterns, for check mode and final audits\n- `references/use-cases.md` — long-form adaptations: error messages, runbooks, incident reports, commits, UI copy, i18n\n\n<!-- File: .claude/skills/review-vocabulary/references/vocabulary/adding-terms.md (Tokens: ~459 | Category: anthropic-skill) -->\n# Adding a new term or a new group\n\nNever just record a term. Take these eight steps.\n\n1. **Pick the group.** No group fits? Add one. Give it the next `G<n>`, a file in this folder named\n after its heading, an index row in `SKILL.md`, and the three parts: included, excluded, rulings.\n A group without excluded terms is not finished.\n2. **Research the usage.** Grep `dlt/`, `tests/` and `docs/` for the word and every synonym. Count\n the hits. Read enough to find the meanings in play. One word often covers two concepts.\n3. **Derive the banned set.** A term is useless without one. For each synonym and inflection,\n decide: banned, or legal with another meaning? Both halves go in the group.\n4. **Check the upstream interface.** The vocabulary must not contradict a public dlt method or\n type name. `access` won because `needs_attach` already said \"accesses its data\". `attach info`\n won because the type is `TAttachInfo`.\n5. **Name the false positives.** `descriptor` is banned for `TAttachInfo` and correct for the\n Python descriptor protocol. Put the exception in the rulings, or the next run \"fixes\" it.\n6. **Fix the part of speech.** Say noun or verb, and ban the other use. `access` is a verb, so\n \"data access\" is a violation.\n7. **Test the ban before you write it.** Grep for the word you intend to exclude. Compliant prose\n already uses it? Then the ban is too wide. Narrow it to the meaning you mean. `inaccessible`\n and `foreign-folded` passed this check and stay legal. `casefold` and `attach instructions`\n failed it and are banned.\n8. **Update the group file.** Add rows to both tables. Add a ruling when the term has a legal\n exception. State what becomes newly compliant and what becomes newly non-compliant. A term that\n flips direction turns compliant prose into findings.\n\n<!-- File: .claude/skills/review-vocabulary/references/vocabulary/g1-data-access-and-locations.md (Tokens: ~315 | Category: anthropic-skill) -->\n# G1 — Data access and locations\n\n**Included**\n\n| Concept | Write |\n|---|---|\n| getting to data (verb) | **access** |\n| the adjective | **accessible**, **inaccessible** |\n| the negative, as a verb | **cannot access** |\n| where the data physically sits | **data location** |\n\n**Excluded**\n\n| Never | Because |\n|---|---|\n| reach, reaches, reachable, in reach of, out of reach, get to | one word for one concept — `access` |\n| physical location, physical destination, physical dataset (in prose) | say **data location**; the method `data_location()` keeps its name |\n\n**Rulings**\n\n- **`access` is a verb for the act, and a noun for what a job may touch.** Keep the verb for the\n act of reading data: \"the engine accesses the data\", not \"data access is one-way\". The noun is\n legal only in the sense G9 defines — the `access` declaration, and the grant that answers it.\n dlt's docs already write \"grant access\", \"read access\" and \"denied access\" 40 times over; that\n usage was always compliant and stays.\n- **`reach` is not always `access`.** \"`SET SESSION` would not reach the cloned sessions\" means\n *propagate to*, not *read data from*. A literal swap changes the meaning — restructure\n (Rule 9.1).\n\n<!-- ... +37 more modular skills available via /api/skills/dlt-hub/dlt?path={path} -->\n</yakaai_skills>"}