{"owner":"datahub-project","repo":"datahub","hasSkills":true,"totalSkillsCount":14,"totalTokensCount":19225,"categories":["root-instruction","anthropic-skill","claude-rule","subagent-persona","command-prompt","cursor-rule","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","docs-website/AGENTS.md",".agent-skills/test-review/SKILL.md","datahub-web-react/AGENTS.md","CLAUDE.md","smoke-test/AGENTS.md",".claude/agents/test-quality-analyzer.md","metadata-ingestion/AGENTS.md",".claude/commands/test-review.md",".cursor/rules/datahub-dev.mdc",".cursor/rules/test-review.mdc","contrib/metadata-model-extensions/README.md",".agent-skills/test-review/references/smoke-test-patterns.md","contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md"],"skills":{"AGENTS.md":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests (Python/JS lint lives in lint-jobs.yml / :module:lint)\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotless-check` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is assembled at build time into `docs-website/genDocs/` (gitignored) —\nsee `docs-website/AGENTS.md` for the full pipeline.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n - **Connectors**: File layout, lineage, reporting, and PR-scope conventions for ingestion connectors live in `metadata-ingestion/AGENTS.md`\n - **Smoke tests**: Authoring conventions live in [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n - **Frontend**: Component conventions, theming/color tokens, file layout, and the frontend CI checklist live in `datahub-web-react/AGENTS.md`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory. How to write them: [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md). How to run them: [`smoke-test/README.md`](smoke-test/README.md)\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\nFor pytest smoke tests, see [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n(authoring) and [`smoke-test/README.md`](smoke-test/README.md) (how to run).\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md) —\nincluding test-data seeding via `test.use({ featureName: ... })`.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n","docs-website/AGENTS.md":"# docs-website — Agent Development Guide\n\nThe docs site is a **Docusaurus 2** app. Docusaurus reads from `genDocs/` (gitignored, assembled at\nbuild time). Custom React pages live in `src/pages/`, blog/learn articles in `src/learn/`.\n\n## Build pipeline\n\nThe full build runs as a Gradle task chain. You rarely need to invoke individual steps, but\nunderstanding the pipeline helps when debugging.\n\n```\n./gradlew :docs-website:yarnBuild\n └─ yarnGenerate\n ├─ yarnInstall\n ├─ generateGraphQLSchema → graphql/combined.graphql\n ├─ generateJsonSchema → static/schemas/datahub_ingestion_schema.json\n ├─ :metadata-ingestion:modelDocGen → docs/generated/metamodel/\n └─ :metadata-ingestion:docGen → docs/generated/ingestion/\n```\n\n`yarnGenerate` runs `yarn run generate`, which:\n\n1. Generates GraphQL API docs via the `docusaurus-graphql-plugin`\n2. Generates Python SDK docs via Sphinx (`sphinx/` → Markdown conversion)\n3. Runs `generateDocsDir.ts` to assemble all markdown into `genDocs/`\n\n## `generateDocsDir.ts` — the assembly script\n\nThis TypeScript script is the central piece that discovers all markdown in the repo and writes\nprocessed output to `genDocs/`. Key behaviors:\n\n- **Discovery**: Uses `git ls-files` to find all `.md` files, then filters out excluded paths\n (`.github/`, `node_modules/`, `contrib/`, archived docs, test dirs, etc.)\n- **Frontmatter injection**: Adds `title`, `slug`, `description`, `custom_edit_url` to each file\n- **Link rewriting**: Rewrites relative links and image paths for the hosted site\n- **Inline directives**: `{{ inline /path/to/file.py }}` embeds file contents;\n `{{ command-output cmd }}` embeds command output\n- **Sidebar validation**: Warns if a doc file is not referenced in `sidebars.js` (some dirs are\n exempt: `docs/generated/metamodel`, `docs/generated/ingestion`, `docs/actions/`, etc.)\n\n## `sidebars.js` conventions\n\n- Doc IDs are file paths relative to `genDocs/`, minus the `.md` extension\n- Every hand-authored doc in `docs/` must appear in `sidebars.js` or the build warns\n- Auto-generated directories (`docs/generated/metamodel/`, `docs/generated/ingestion/`) are exempt\n\n## Key files\n\n| File | Purpose |\n| ----------------------- | ------------------------------------------------- |\n| `docusaurus.config.js` | Site config — plugins, navbar, footer, docs path |\n| `sidebars.js` | Sidebar navigation tree |\n| `generateDocsDir.ts` | Markdown discovery, transformation, and assembly |\n| `build.gradle` | Gradle tasks (`yarnGenerate`, `yarnBuild`, etc.) |\n| `sphinx/` | Python SDK doc generation (Sphinx → Markdown) |\n| `graphql/` | GraphQL schema combination + doc generation |\n| `genJsonSchema/` | Combines connector config schemas into one file |\n| `src/pages/` | Custom React pages (not markdown docs) |\n| `src/learn/` | Blog / learning articles (served at `/learn`) |\n| `static/` | Images, logos, static assets |\n\n## Generated directories (gitignored — never edit)\n\n- `genDocs/` — final assembled docs consumed by Docusaurus\n- `graphql/combined.graphql` — concatenated GraphQL schema\n- `static/schemas/datahub_ingestion_schema.json` — combined ingestion config schema\n\n## Logo convention\n\nPlatform logos: `static/img/logos/platforms/{platform_id}.svg`\nDataHub internal connectors: use `img/datahub-logo-color-mark.svg` (shared logo).\n",".agent-skills/test-review/SKILL.md":"# DataHub Test Review\n\nYou are an expert DataHub test reviewer. Your role is to evaluate pytest smoke\ntests against established testing standards, identify issues, and provide\nactionable feedback.\n\nAuthoring how-to: `smoke-test/AGENTS.md`. Review rubric:\n`standards/smoke.md`.\n\n---\n\n## Multi-Agent Compatibility\n\nThis skill is designed to work across multiple coding agents (Claude Code, Cursor, Codex, Copilot, Gemini CLI, Windsurf, and others).\n\n**What works everywhere:**\n\n- All review checklists, standards references, and procedures in this document\n- Bash for running scripts (`detect-test-changes.sh`, `gh` CLI, `git diff`)\n- Reading files, searching code, and generating review reports\n\n**Claude Code-specific features** (other agents can safely ignore these):\n\n- The `/test-review` slash command (`.claude/commands/test-review.md`) loads this skill automatically\n- The `test-quality-analyzer` agent (`.claude/agents/test-quality-analyzer.md`) can be dispatched for parallel analysis -- **fallback instructions are provided inline** for agents that cannot dispatch sub-agents\n- `TaskCreate`/`TaskUpdate` for progress tracking -- if unavailable, simply proceed through the steps sequentially\n\n**Standards file paths:** All standards are in the `standards/` directory alongside this file. All paths below are relative to `.agent-skills/test-review/`.\n\n---\n\n## Quick Start\n\n**Full review?** -> Load standards, gather test files, then launch `test-quality-analyzer` agent (or perform checks directly)\n\n**PR review?** -> Detect changed test files, then analyze only those files\n\n---\n\n## Scope\n\n### In Scope\n\n- `smoke-test/` -- Python pytest smoke tests (API-level tests against a running DataHub instance)\n- `smoke-test/tests/` -- shared test utilities, fixtures, and helpers\n\n### Out of Scope\n\n- `smoke-test/tests/cypress/` -- Cypress UI tests\n- `e2e-test/ui/playwright/` -- Playwright UI tests\n- `metadata-ingestion/tests/integration/` -- ingestion connector tests (covered by `datahub-connector-pr-review`)\n- `metadata-ingestion/tests/unit/` -- unit tests\n- `metadata-ingestion/src/datahub/testing/` -- ingestion testing utilities\n\nThe `detect-test-changes.sh` script lists in-scope Python smoke-test files.\n\n---\n\n## Review Modes\n\n| Mode | Use Case | Scope | Template |\n| ---------------------- | ------------------------------ | ------------------ | ---------------------------- |\n| **Full Review** | Audit all tests in a directory | All in-scope tests | `test-review-report.md` |\n| **Incremental Review** | PR with test changes | Changed files only | `incremental-test-report.md` |\n\n---\n\n## Startup: Load Standards\n\n**On activation, IMMEDIATELY load testing standards** from the `standards/` directory.\n\nRead `.agent-skills/test-review/standards/smoke.md` -- this contains all testing rules.\nAlso skim `smoke-test/AGENTS.md` if isolation or fixture guidance is in dispute.\n\nAfter loading, briefly confirm: \"Loaded test review standards. Ready to review.\"\n\n---\n\n## Progress Tracking with Tasks\n\n**After loading standards**, create a task checklist using TaskCreate:\n\n```\n1. Load testing standards\n2. Detect in-scope smoke test files\n3. Filter out connector-specific and Cypress tests\n4. Analyze smoke tests\n5. Generate review report\n```\n\nIf TaskCreate is not available, proceed through the steps sequentially.\n\n---\n\n## Mode 1: Full Review\n\n### Step 1: Gather Test Files\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\nOr list Python tests under `smoke-test/` excluding `tests/cypress/`.\n\n### Step 2: Load Standards into Context\n\nRead `.agent-skills/test-review/standards/smoke.md` completely.\n\n### Step 3: Launch Test Quality Analyzer\n\n**Claude Code (with Agent tool):**\n\n```\nAgent tool:\n subagent_type: \"test-quality-analyzer\"\n prompt: \"\"\"Analyze the following smoke test files for quality and standards compliance.\n\n\n[Content from .agent-skills/test-review/standards/smoke.md]\n\n\n\n[List of smoke-test Python file paths]\n\n\nFor each file, check all applicable rules from the standards document.\nReport findings with severity (BLOCKER/WARNING/SUGGESTION) and file:line references.\n\"\"\"\n```\n\n**Other agents (sequential fallback):**\n\nIf you cannot dispatch sub-agents, perform the analysis yourself:\n\n1. Read each test file completely\n2. Check against the smoke-test standards (isolation, fixtures, auth, retry, GraphQL/REST, markers, env vars, cleanup, placement, logging)\n3. Scan for anti-patterns listed at the end of the standards file\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/test-review-report.md` template. Fill in:\n\n- Summary table with pass/fail per category\n- All findings organized by severity (BLOCKER > WARNING > SUGGESTION)\n- Checklist results\n- Quality scores (1-10 per dimension)\n- Verdict: APPROVED / NEEDS CHANGES / BLOCKED\n\n**Verdict logic:**\n\n- **APPROVED**: No blockers, no more than 3 warnings\n- **NEEDS CHANGES**: Has warnings or fixable blockers\n- **BLOCKED**: Has fundamental anti-pattern blockers (empty tests, missing cleanup, hardcoded credentials, shared hardcoded URNs)\n\n---\n\n## Mode 2: Incremental Review (CI Mode)\n\n### Step 1: Detect Changed Test Files\n\n**If PR number provided:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh ${PR_NUMBER}\n```\n\n**If running locally:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\n**If no script available:**\n\n```bash\ngh pr diff ${PR_NUMBER} --name-only | grep -E '^smoke-test/' | \\\n grep -E '\\.py$' | grep -v '/tests/cypress/'\n```\n\n### Step 2: Classify Files\n\nParse the output of `detect-test-changes.sh`:\n\n- Lines starting with `smoke:` are smoke test files\n\nIf no in-scope test changes detected (exit code 1), report: \"No in-scope test changes found in this PR.\"\n\n### Step 3: Analyze Changed Files\n\nApply the same analysis as Mode 1, Step 3, but only to changed files.\n\nFor incremental reviews, also check:\n\n- Do new tests follow the same patterns as existing tests in their module?\n- Do modifications preserve existing test behavior?\n- Are unique-name helpers used instead of new shared hardcoded URNs?\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/incremental-test-report.md` template.\n\n---\n\n## CI Invocation\n\nFor non-interactive CI usage via `claude -p`:\n\n```bash\nclaude -p \"Review test changes in PR #${PR_NUMBER} using the test-review skill. \\\n Output the review report in markdown format with a verdict line.\"\n```\n\nThe skill produces deterministic output:\n\n- Structured markdown report\n- Clear verdict: `APPROVED`, `NEEDS CHANGES`, or `BLOCKED`\n- All findings cite file:line references\n\n---\n\n## Standards Reference\n\nAll standards are documented in `standards/smoke.md`. Authoring\nguide: `smoke-test/AGENTS.md`.\n\n1. **Isolation and unique names** -- `unique_suffix()` / unique ingest, no shared hardcoded URNs\n2. **Fixtures and data lifecycle** -- unique-dataset ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique\n3. **Authentication** -- `auth_session` fixture, `make_step_actor_user()`, never inline\n4. **Retry patterns** -- Trace API, `wait_for_writes_to_sync()`, `@with_test_retry()`, no bare `time.sleep()`\n5. **GraphQL / REST** -- `execute_graphql()`, `restli_default_headers`, `ingest_file_via_rest()`\n6. **Markers** -- required `domain(...)`; `p0` only for regressions that must run on every PR; `read_only` only when the test never mutates; `global_policy_mutator` only when mutating shared platform policy\n7. **Environment variables** -- `env_vars.py` registry, no hardcoded URLs\n8. **Guaranteed cleanup** -- fixture `yield` or `try/finally`\n9. **Multi-environment config** -- URLs via env vars, `USE_STATIC_SLEEP` fallback\n10. **Concurrent testing** -- thread-safe `run_concurrent_tests()`\n11. **Placement, logging, quality** -- `tests//`, `logger.info()`, no customer identifiers\n\n### Anti-Patterns (Automatic Blockers)\n\n- Empty/trivial tests\n- Missing cleanup\n- Hardcoded URLs/ports\n- Inline authentication\n- Bare `time.sleep()` for consistency\n- Shared hardcoded URNs / global mutable state\n- Mutating shared platform policy without `global_policy_mutator`\n\n---\n\n## Severity Levels\n\n| Level | Icon | Meaning | Action Required |\n| ---------- | ---- | ------------------ | ---------------------------- |\n| BLOCKER | ---- | Standard violated | Must fix before merge |\n| WARNING | ---- | Should be improved | Should fix before merge |\n| SUGGESTION | ---- | Nice to have | Optional, defer to follow-up |\n\n---\n\n## Templates\n\n- `templates/test-review-report.md` -- Full review report\n- `templates/incremental-test-report.md` -- PR incremental report\n\n---\n\n## Scripts\n\n- `scripts/detect-test-changes.sh` -- Detect in-scope changed smoke-test files\n\n---\n\n## References\n\n- `smoke-test/AGENTS.md` -- Canonical authoring guide\n- `references/smoke-test-patterns.md` -- Additional code examples\n\n---\n\n## Remember\n\n1. **Do not invent rules.** Standards come from `smoke-test/AGENTS.md` and the cited helpers.\n2. **Cypress, Playwright, and connector tests are out of scope.**\n3. **Be conservative with BLOCKERs.** Only flag when a standard is clearly violated. Older files that still use `_ingest_cleanup_data_impl` with unique keys are acceptable.\n4. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n5. **Acknowledge good patterns.** Note when tests follow standards well.\n6. **CI output must be deterministic.** Same input -> same verdict.\n","datahub-web-react/AGENTS.md":"# DataHub Web React — Agent Development Guide\n\nCanonical guide for `datahub-web-react/`. Repo-wide rules (Gradle-based formatting/linting,\ncommit and PR conventions, testing principles, running DataHub via `scripts/dev/datahub-dev.sh`)\nlive in the root [`AGENTS.md`](../AGENTS.md) and are not repeated here.\n\nRelated references:\n\n- React-specific code review rules: [`.cursor/BUGBOT.md`](.cursor/BUGBOT.md)\n- Playwright UI E2E tests: [`../e2e-test/ui/playwright/README.md`](../e2e-test/ui/playwright/README.md)\n\n## Development Commands\n\nFrom the **repo root**:\n\n```bash\n# Hot-reload dev server (preferred)\nscripts/dev/datahub-dev.sh setup frontend # one-time setup\nscripts/dev/datahub-dev.sh frontend # dev server with hot reload\n```\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Dev server proxying to a remote GMS instead of a local one\n../gradlew yarnPreview -Pproxy=\"\"\n\n# Regenerate TypeScript types after changing .graphql files\nyarn generate\n```\n\n### Lint, type-check, and test (before commit / PR)\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Full lint (eslint + prettier + type-check)\n../gradlew :datahub-web-react:yarnLint\n\n# Lint-fix a single file (fast; skips full type-check — run yarnLint before commit)\n../gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (icon stubs needed once per clone)\nnode scripts/generate-lazy-icon-stubs.js\nyarn test src/path/to/file.test.tsx --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in unrelated\nfiles. Focus on errors in files you touched — optional prop calls (`prop?.(arg)`) and\nimport aliases are common culprits.\n\n## File Structure\n\n- Top-level folders: `app` (application components, one folder per top-level nav page;\n nested pages live inside their parent's folder), `graphql` (GraphQL files and generated\n types), `images` (custom images), plus `conf`, `fonts`, `providers`, `utils`.\n- Do not create `index.ts(x)` barrel files.\n- Tests: in `__tests__/` next to the file under test, named `OriginalFile.test.ts(x)`.\n- Utils: used by one file → `SourceFile.utils.ts`; used across a folder →\n `folderName/folderName.utils.ts`.\n- Hooks: helper hooks for one file → `SourceFile.hooks.ts(x)`; standalone reusable hooks →\n `hookName.ts(x)`. Avoid hooks that return JSX unless it clearly makes sense.\n- Types: used once → same file as usage; shared → `folderName/folderName.types.ts`.\n- Helper components: small and single-use → `parent/SourceFile.components.tsx`; shared or\n large without child components → `parent/NewComponent.tsx`; has its own child\n components → `parent/NewComponent/NewComponent.tsx`.\n\n## Components\n\n- Prefer alchemy (`@components`) over `antd` — ESLint `rulesdir/no-antd-imports` enforces\n this. Files that already imported antd on the PR base (`origin/master`) are grandfathered.\n The alchemy library is documented in\n [`src/alchemy-components/README.mdx`](src/alchemy-components/README.mdx); preview\n components with `yarn storybook`.\n- An alchemy component is generalized and reusable anywhere: it must not depend on DataHub\n aspects, data structures, or GraphQL types.\n- Reusable DataHub-specific components (which may take GraphQL types as props) go under\n `src/app/sharedV2/`.\n- Err on the side of more, smaller components: break out a new component when JSX gets\n large or complex, or for any logical/reusable chunk. Same rule for extracting hooks from\n component logic.\n- Contexts: err on the side of not creating them. Use a global context only for truly\n global values (e.g. `AppContext`, `UserContext`). For a small self-contained scope, use\n `folder/Context.ts` (type, default value, helper hooks) plus\n `folder/ContextProvider.tsx` (just the provider).\n- Props: individual values for a small number of props; group into logical objects as they\n grow (e.g. `entityData` rather than one prop per aspect). For components shared between\n OSS and SaaS, SaaS-only props go in a single `acrylProps: { ... }` field.\n- Component file section order: imports, constants (SNAKE_CASE), styled components, props,\n main component.\n\n## Styling, Theming, Icons, and Images\n\n- Use `styled-components` with string CSS, not object CSS:\n\n ```tsx\n // YES\n styled.div`\n border-radius: 2px;\n `;\n\n // NO\n styled.div({ borderRadius: '2px' });\n ```\n\n- Custom components should accept `className` and pass it to the element custom styles\n should apply to — this enables `const CustomButton = styled(Button)` inheritance. If\n multiple inner elements are styleable, expose non-top-level ones as named\n `CSSProperties` props (e.g. `textStyle`, `buttonStyle`).\n\n**Colors — always use semantic color tokens** from `src/conf/theme/colorThemes/types.ts`.\nNever use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy\n`colors.gray[X]` imports.\n\nIn styled-components (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\nIn React component bodies:\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\nFor alchemy components (``, ``, etc.) — do not pass `color`/`colorLevel`\nprops; let them inherit from themed parent styled-components.\n\nDo not import from:\n\n- `src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally\n by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n**Icons**: use the alchemy `` component with phosphor icons only — ant and material\nUI icons are deprecated. Specify color and size via props; if size is unknown, use\n`size=\"inherit\"` and set `font-size` in the parent element. Import phosphor icons from\ntheir individual CSR paths for tree-shaking:\n`import { ArrowRight } from '@phosphor-icons/react/dist/csr/ArrowRight'` — never from\n`@phosphor-icons/react` (ESLint will error). Find icon names at https://phosphoricons.com.\n\n**Images**: use the alchemy `` component (`src` and `alt` are required; it\nhandles loading skeleton and error fallback). For SVGs, set `fill=\"currentColor\"` in the\nSVG definition so the icon inherits color from CSS.\n\n## TypeScript Style\n\nMost style is enforced by ESLint/Prettier. Conventions the tools don't enforce:\n\n- Prefer `type` over `interface` (except when using classes). Type props with TypeScript\n types, never PropTypes.\n- Top-level: named functions (`export default function f() {}`); nested: lambdas\n (`const onClick = () => {}`).\n- Don't annotate variables; do annotate function signatures unless impractical — prefer\n inferred typing over `any`. Prefer a mapper function over unsafe casting; if you must\n cast, do it as early (high) as possible.\n- Always optional-chain array access (`x?.[0]`).\n- Prefer direct imports (`import React, { useState } from 'react'`) over `React.useState`.\n\n## Unit Testing\n\n- Vitest + React Testing Library; tests live in `__tests__/` next to the source file.\n- Wrap components in `TestPageContainer` from `@utils/test-utils/TestPageContainer` — it\n provides the necessary app providers (theme, routing, contexts).\n- Use `MockedProvider` from `@apollo/client/testing` for components that make GraphQL\n queries.\n- Generate icon stubs once per clone before running tests:\n `node scripts/generate-lazy-icon-stubs.js`.\n- Follow the root guide's testing principles: test behavior, not implementation; skip\n tests that only exercise the framework or restate the linter.\n","CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","smoke-test/AGENTS.md":"# Smoke Test Guidelines\n\nCanonical authoring guide for pytest smoke tests against a running DataHub\ninstance. How to run the suite: `[README.md](README.md)`.\n\n## Where tests and helpers live\n\n| Path | What goes here |\n| ---------------------- | ----------------------------------------------------------------------------- |\n| `tests/e2e/` | Live GMS pytest (needs quickstart). Default `pytest` / `smoke.sh` collection. |\n| `tests/unit/` | CPU pytest (no stack). `./gradlew :smoke-test:stackFreePytest`. |\n| `tests/zdu/` | ZDU live E2E pytest (`test_zdu_upgrade.py`) and docs. Not in docker pytest. |\n| `tests/zdu/framework/` | ZDU upgrade framework. Daily workflow: `python -m tests.zdu.framework`. |\n| `tests/oauth/` | OAuth IdP smoke. `.github/workflows/oauth-smoke.yml`. Not in docker pytest. |\n| `utilities/` | Shared helpers, not tests. |\n\nDo not put `test_*.py` next to helpers in `utilities/`. GMS fixtures live in `tests/e2e/conftest.py` only.\n\n## When to add smoke tests\n\nThese are E2E API tests on a **live GMS** (slow, shared stack, xdist). Prefer a\nunit test next to the code, an ingestion test in `metadata-ingestion/tests/`, or a backend unit/integration test **before** adding a smoke test.\n\n**Add one** when the behavior needs GMS: write → search, a public API flow with\nno existing coverage, a production regression to keep, or live-stack authz.\n\n**Do not add** when:\n\n- The logic is a validator, parser, or in-process service.\n- The flow is already covered — extend that module; do not add `test_*_v2.py`.\n- You are stacking positive/negative variants of the same call. One happy path\n is enough. A negative case belongs here only if it needs GMS (e.g. policy\n deny), not invalid input or combinatorial 4xx/case/whitespace.\n\nPut new tests in `tests/e2e//`, not `test_e2e.py`. Connector tests are placed in `metadata-ingestion/tests/`. Generic placeholder names only\n(`my_db.my_schema.events`) — no customer identifiers or ticket IDs.\n\n## Rules\n\n- **Idempotent and order-independent.** Do not assume an empty GMS or another\n test's cleanup. No module-level mutable globals.\n- **Run-unique names** for every entity you create, mutate, or delete (see\n Isolation). Shared hardcoded URNs flake under xdist.\n- **Cleanup even on failure:** fixture `yield`, or `try/finally` inside a test\n (wrap delete in `try/except` so cleanup cannot hide the assertion).\n- **Do not mutate** the admin user, default All Users policies, or other shared\n platform state unless the module is marked `global_policy_mutator`.\n- `logger.info()`, not `print()`.\n- **No** `time.sleep()` for GMS/search/Kafka. Read-after-write:\n `@with_test_retry()`. Bulk ingest/cleanup: `wait_for_writes_to_sync()`\n (`mcp_only` / `mae_only` when only one store matters). Known `trace_id`: Trace\n API. `TestSessionWrapper` already waits on POST/PUT — do not extra-sync unless\n you are asserting search/index.\n- **Auth:** `auth_session` / `graph_client`. Extra users:\n `make_step_actor_user()`. Config: `utilities/env_vars.py`, not `os.getenv` or\n `localhost:8080`. GraphQL: `execute_graphql()` (it already checks `data` /\n `errors`) — assert the fields you care about. Ingest:\n `ingest_file_via_rest()`.\n- **Reuse helpers.** Do not copy-paste `addTag` or roll a local `uuid` when \n `unique_suffix()` exists. Tags/terms/descriptions: \n `utilities/metadata_operations.py`.\n- Extra users: `make_step_actor_user()` in `utilities/multi_user.py`, \n not a one-off signup.\n- `execute_graphql()` already asserts a non-empty body, `data` is not `None`,\n and no `errors` key, only assert the data that the test requires\n\n## Markers\n\n```python\nfrom utilities.domains import Domain\n\npytestmark = pytest.mark.domain(Domain.CATALOG)\n# pytestmark = pytest.mark.domain(Domain.CATALOG, Domain.INGESTION)\n```\n\n| Marker | When |\n| ----------------------- | ---------------------------------------------------------------------------------------------------------- |\n| `domain(...)` | **Required.** `platform`, `observe`, `ingestion`, `ai`, `catalog`. |\n| `p0` | Must run on every PR. |\n| `global_policy_mutator` | Disables default policies / mutates shared platform policy. CI runs these serially after parallel modules. |\n\nKeep `@pytest.mark.dependency()` chains short (ideally ≤3).\n\n## Isolation\n\nModules run in parallel against one GMS. Prefer these over a local `uuid` /\nstring replace:\n\n| Helper | Use |\n| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| `unique_suffix()` | Any entity key (user, domain, PAT, …). |\n| `unique_dataset_urn(name)` | Dataset created in code (GraphQL / SDK). |\n| `materialize_with_unique_name(src, name, dest_dir)` | Fixture rewrite. `name` must appear **only** in URN keys, not descriptions. Returns `(dest_file, unique_name)`. |\n| `materialize_unique_dataset(...)` | Dataset fixture → `(dest_file, dataset_urn)`. |\n| `_ingest_cleanup_unique_dataset_impl(...)` | Rewrite + ingest + yield URN + delete. No pre-delete (URN is new). Default for a shared dataset name in `data.json`. |\n| `_ingest_cleanup_data_impl(...)` | Pre-delete → ingest → cleanup. Only when keys are already unique to this module. |\n\n```python\nfrom conftest import _ingest_cleanup_unique_dataset_impl\n\n@pytest.fixture(scope=\"module\", autouse=True)\ndef dataset_urn(auth_session, graph_client, tmp_path_factory):\n yield from _ingest_cleanup_unique_dataset_impl(\n auth_session, graph_client,\n \"tests/e2e/tags_and_terms/data.json\", \"tags_and_terms\",\n \"test-tags-terms-sample-kafka\",\n tmp_path_factory.mktemp(\"tags_and_terms\"),\n )\n```\n\nTests take `dataset_urn` as a fixture argument. Multi-entity fixtures: rewrite\neach key with `materialize_with_unique_name` — see\n`tests/e2e/containers/containers_test.py`.\n\nMid-test creates:\n\n```python\ndataset_urn = unique_dataset_urn(\"my-feature\")\ntry:\n graph_client.emit(...)\n wait_for_writes_to_sync()\nfinally:\n try:\n delete_urn(graph_client, dataset_urn)\n except Exception:\n logger.warning(\"cleanup failed for %s\", dataset_urn, exc_info=True)\n```\n\n## Other helpers\n\n- `tests/utils.py`: `execute_graphql`, `ingest_file_via_rest`,\n `delete_urn` / `delete_urns` / `delete_urns_from_file`, `with_test_retry`,\n `wait_for_writes_to_sync`.\n- `tests/utilities/concurrent_test_runner.py`: thread-safe\n `run_concurrent_tests` / `run_concurrent_tests_with_args`.\n- `tests/utilities/concurrent_openapi.py`: `run_tests(auth_session, fixture_globs=...)`.\n JSON fixtures: `{request, response}` with DeepDiff `exclude_regex_paths`. Do\n not add OpenAPI fixtures that only re-check a GraphQL path already covered.\n",".claude/agents/test-quality-analyzer.md":"---\nname: test-quality-analyzer\ndescription: |\n Analyze DataHub pytest smoke test files for quality, standards compliance, and anti-patterns. Reports findings with severity levels and file:line references.\n\n \n Context: A PR adds new smoke tests for the incidents feature.\n user: \"Analyze the test quality of smoke-test/tests/incidents/incidents_test.py\"\n assistant: \"I'll use the test-quality-analyzer agent to check fixture usage, data lifecycle, assertions, and anti-patterns.\"\n \n Test quality analysis is a read-only inspection task that triggers this agent.\n \n \nmodel: sonnet\ncolor: blue\ntools:\n - Bash\n - Read\n - Grep\n - Glob\n---\n\n# DataHub Test Quality Analyzer Agent\n\nYou are a test quality analysis agent that inspects DataHub pytest smoke tests for compliance with established testing standards. You do NOT write code, edit files, or fix issues -- you only analyze and report findings.\n\nAuthoring guide: `smoke-test/AGENTS.md`. Review rubric: `.agent-skills/test-review/standards/smoke.md`.\n\n## Core Rules\n\n1. **Read-only analysis.** You have no Write or Edit tools. Inspect test files and report findings.\n2. **Cite file:line for every finding.** Every issue must reference the exact file path and line number.\n3. **Use severity levels consistently.** BLOCKER = must fix, WARNING = should fix, SUGGESTION = nice to have.\n4. **Smoke tests only.** Cypress, Playwright, and `metadata-ingestion/tests/` are out of scope.\n5. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n\n## Input\n\nYou receive:\n\n- A list of test files to analyze\n- The testing standards (embedded in the prompt as `` tags)\n\n## Workflow\n\n### Phase 1: File Classification\n\nFor each test file, determine:\n\n- **Smoke test** (Python pytest): Located under `smoke-test/` excluding `tests/cypress/`\n- **Out of scope**: `smoke-test/tests/cypress/`, `e2e-test/ui/playwright/`, `metadata-ingestion/tests/`\n\nSkip out-of-scope files.\n\n### Phase 2: Smoke Test Analysis\n\nFor each smoke test file, check:\n\n1. **Isolation** -- Run-unique names (`unique_suffix`, `unique_dataset_urn`, `materialize_*`, `_ingest_cleanup_unique_dataset_impl`)? No shared hardcoded URNs?\n2. **Data Lifecycle** -- Fixture `yield` teardown? Unique ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique?\n3. **Authentication** -- `auth_session` fixture, not inline credentials? Extra users via `make_step_actor_user()`?\n4. **Retry Patterns** -- `@with_test_retry()` or `wait_for_writes_to_sync()` instead of bare `time.sleep()`?\n5. **GraphQL / REST** -- `execute_graphql()` with specific field assertions? `ingest_file_via_rest()` / `restli_default_headers`?\n6. **Environment Variables** -- `env_vars.py` instead of `os.getenv`? No hardcoded URLs/ports?\n7. **Markers** -- `domain(...)` declared? `global_policy_mutator` if mutating shared policy? `read_only` tests truly read-only?\n8. **Cleanup** -- Fixture `yield` or `try/finally` for mid-test entities?\n9. **Test Isolation** -- No global mutable state? No cross-test module-level variables?\n10. **Logging** -- `logger.info()`, not `print()`?\n11. **Placement** -- New tests in `tests//`, not `test_e2e.py`?\n12. **Confidentiality** -- No customer identifiers or ticket IDs?\n\n### Phase 3: Anti-Pattern Detection\n\nScan for these automatic BLOCKER triggers:\n\n- Empty/trivial tests (`assert True`, only testing defaults)\n- Bare `time.sleep()` without retry wrapper\n- Hardcoded URLs or ports (should use env_vars registry)\n- Inline credential creation (should use auth_session fixture)\n- Shared hardcoded URNs across modules\n- Cross-test dependencies via global mutable state\n- Missing cleanup for created entities\n- Entities created mid-test without `try/finally`\n- Mutating admin / default policies without `global_policy_mutator`\n- Commented-out test code or breakpoints\n\n## Output Format\n\n```markdown\n# Test Quality Analysis\n\n**Files Analyzed:** {{COUNT}}\n**Smoke Tests:** {{SMOKE_COUNT}}\n\n---\n\n## Findings Summary\n\n| Severity | Count |\n| ---------- | ----- |\n| BLOCKER | {{N}} |\n| WARNING | {{N}} |\n| SUGGESTION | {{N}} |\n\n---\n\n## BLOCKER Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Standard:** {{WHICH_STANDARD_VIOLATED}}\n- **Issue:** {{DESCRIPTION}}\n- **Evidence:** {{CODE_SNIPPET_OR_EXPLANATION}}\n\n---\n\n## WARNING Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Issue:** {{DESCRIPTION}}\n- **Recommendation:** {{FIX}}\n\n---\n\n## SUGGESTION Issues\n\n- `{{FILE}}:{{LINE}}` -- {{DESCRIPTION}}\n\n---\n\n## Positive Observations\n\n- {{GOOD_PATTERN_FOUND}}\n\n---\n\n## Verdict\n\n**{{VERDICT}}** (APPROVED / NEEDS CHANGES / BLOCKED)\n```\n\n## Important Guidelines\n\n- **Be conservative with BLOCKERs.** Only flag as BLOCKER when a standard is clearly violated.\n- **Consider context.** Older test files may still use `_ingest_cleanup_data_impl` with unique keys -- flag new shared hardcoded URNs, not every legacy fixture.\n- **Acknowledge good patterns.** Note when tests follow standards well.\n- **Check the whole file.** Don't stop after finding the first issue.\n","metadata-ingestion/AGENTS.md":"# DataHub Metadata Ingestion Development Guide\n\n## Build and Test Commands\n\n**Using Gradle (slow but reliable):**\n\n```bash\n# Development setup from metadata-ingestion/\n../gradlew :metadata-ingestion:installDev # Setup Python environment\nsource venv/bin/activate # Activate virtual environment\n\n# Linting and formatting\n../gradlew :metadata-ingestion:lint # Run ruff + mypy\n../gradlew :metadata-ingestion:lintFix # Auto-fix linting issues\n\n# Testing\n../gradlew :metadata-ingestion:testQuick # Fast unit tests\n../gradlew :metadata-ingestion:testFull # All tests\n../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_file.py # Single test\n```\n\n**Direct Python commands (when venv is activated):**\n\n```bash\n# Testing\npytest -vv # Run all tests\npytest -m 'not integration' # Unit tests only\npytest -m 'integration' # Integration tests\npytest tests/path/to/file.py # Single test file\npytest tests/path/to/file.py::TestClass # Single test class\npytest tests/path/to/file.py::TestClass::test_method # Single test\n```\n\n## Environment Variables\n\n**Build configuration:**\n\n- `DATAHUB_VENV_USE_COPIES`: Set to `true` to use `--copies` flag when creating Python virtual environments. This copies the Python binary instead of creating a symlink. Useful for Nix environments, immutable filesystems, Windows, or container environments where symlinks don't work correctly. Increases disk usage and setup time, so only enable if needed.\n\n ```bash\n export DATAHUB_VENV_USE_COPIES=true\n ../gradlew :metadata-ingestion:installDev\n ```\n\n## Directory Structure\n\n- `src/datahub/`: Source code for the DataHub CLI and ingestion framework\n- `tests/`: All tests (NOT in `src/` directory)\n- `tests/unit/`: Unit tests\n- `tests/integration/`: Integration tests\n- `scripts/`: Build and development scripts\n- `examples/`: Example ingestion configurations\n- `developing.md`: Detailed development environment information\n\n## Code Style Guidelines\n\n- **Formatting**: Uses ruff, 88 character line length\n- **Imports**: Sorted with ruff.lint.isort, no relative imports, Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.\n- **Types**: Always use type annotations, prefer Protocol for interfaces\n - Avoid `Any` type - use specific types (`Dict[str, int]`, `TypedDict`, or typevars)\n - Use `isinstance` checks instead of `hasattr`\n - Prefer `assert isinstance(...)` over `cast`\n- **Data Structures**: Use dataclasses/pydantic for internal data representation\n - Return dataclasses instead of tuples from methods\n - Centralize utility functions to avoid code duplication\n- **Naming**: Descriptive names, match source system terminology in configs\n- **Error Handling**: Validators throw only ValueError/TypeError/AssertionError\n - Add robust error handling with layers of protection for known failure points\n- **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`\n- **Documentation**: All configs need descriptions\n- **Dependencies**: Avoid version pinning, use ranges with comments\n- **Architecture**: Avoid tall inheritance hierarchies, prefer mixins\n\n## Testing Conventions\n\n- **Location**: Tests go in `tests/` directory alongside `src/`, NOT in `src/`\n- **Structure**: Test files should mirror the source directory structure\n- **Framework**: Use pytest, not unittest\n- **Assertions**: Use `assert` statements, not `self.assertEqual()` or `self.assertIsNone()`\n - Boolean: `assert func()` or `assert not func()`, not `assert func() is True/False`\n - None: `assert result is None` (correct), not `assert result == None`\n - Keep tests concise, avoid verbose repetitive patterns\n- **Classes**: Use regular classes, not `unittest.TestCase`\n- **Imports**: Import `pytest` in test files\n- **Naming**: Test files should be named `test_*.py`\n- **Categories**:\n - Unit tests: `tests/unit/` - fast, no external dependencies\n - Integration tests: `tests/integration/` - may use Docker/external services\n\n## Configuration Guidelines (Pydantic)\n\n- **Naming**: Match terminology of the source system (e.g., `account_id` for Snowflake, not `host_port`)\n- **Descriptions**: All configs must have descriptions\n- **Patterns**: Use AllowDenyPatterns for filtering, named `*_pattern`\n- **Defaults**: Set reasonable defaults, avoid config-driven filtering that should be automatic\n- **Validation**: Single pydantic validator per validation concern\n- **Security**: Use `SecretStr` for passwords, auth tokens, etc.\n- **Deprecation**: Use `pydantic_removed_field` helper for field deprecations\n\n## Connector Conventions\n\n- **Structure**: Split connector code by duty: `constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`, `mapper.py`, and `usage.py` as needed. Match the quality and structure of existing connectors like Power BI, Airbyte, Redshift, BigID, and Grafana.\n- **Constants**: Avoid double-quoted string literals in connector code. Hoist magic strings into module-level constants, and keep regex patterns pre-compiled in the constants file.\n- **Comments**: No top-of-file docstrings in connector modules. Keep comments and docstrings only where they add real context.\n- **Imports**: Do not use `TYPE_CHECKING` in connector code. Lazy imports are fine only for opt-in features the connector does not always depend on.\n- **Syntax**: Do not use the walrus operator in connector code.\n- **Types**: Prefer `Dict` and `List` from `typing` over builtin `dict` and `list` in connector type annotations.\n- **Reporting**: Prefer `self.report.warning(...)` and report counters over bare `logger` warnings for skips, inaccessible objects, and other operator-visible edge cases.\n- **SQL lineage**: Use `SqlParsingAggregator` via `create_lineage_from_sql_statements` with a platform map instead of setting sqlglot dialects per connector. Mirror existing connectors for known-URN mapping, platform instance / env handling, casing, multi-part names, and temp-table handling.\n- **Column lineage**: Resolve upstream and downstream schemas from the DataHub graph when available, load known URNs from the platform / platform instance / env mapping, and match columns case-insensitively.\n- **Progress**: Surface ingestion progress and use explicit ingestion stages, following connectors like Dremio and Snowflake.\n- **New connectors**: A connector usually needs more than Python code: source logo, integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration in `setup.py`, `../gradlew :metadata-ingestion:updateLockFile`, and shared subtypes instead of connector-local subtype definitions.\n- **XML**: Avoid Python's stdlib `xml` parser. Use a safe XML library, following the HANA-related code path.\n- **Validation workflow**: Before committing a new connector, run ingestion locally in debug mode to a local file and inspect the full logs. When testing against customer environments, keep secrets only in temporary paths such as `/tmp/*.env`.\n- **PR scope**: Keep each connector in its own PR. Split shared framework changes such as sqlglot helpers into a separate PR, and keep the PR title and description focused on that connector only.\n\n## Dependencies and Entry Points\n\n`setup.py` is the **source of truth** for all dependencies, extras, and entry points. `pyproject.toml`, `uv.lock`, and `constraints.txt` are **generated from it** and must never be edited manually.\n\n**After any change to `setup.py`** (new dependency, new entry point, version bump), run:\n\n```bash\n../gradlew :metadata-ingestion:updateLockFile\n```\n\nThis regenerates the full chain: `setup.py` → `pyproject.toml` → `uv.lock` → `constraints.txt`.\n\nCI runs `checkLockFile` which calls `scripts/verify_pyproject_equivalence.py` to verify these files are in sync. If you edit `setup.py` without running `updateLockFile`, CI will fail.\n\n**Adding a new ingestion source entry point:**\n\n1. Add the entry to `setup.py` under `entry_points[\"datahub.ingestion.source.plugins\"]`\n2. Run `../gradlew :metadata-ingestion:updateLockFile`\n3. Verify with `../gradlew :metadata-ingestion:checkLockFile`\n\n## Key Files\n\n- `src/datahub/emitter/mcp_builder.py`: Examples of defining various aspect types\n- `setup.py`: Source of truth for dependencies, extras, and entry points\n- `pyproject.toml`: Generated from `setup.py` — do not edit manually\n- `setup.cfg`: Code style configuration\n- `scripts/generate_pyproject_deps.py`: Generates `pyproject.toml` from `setup.py`\n- `scripts/verify_pyproject_equivalence.py`: Validates `setup.py` and `pyproject.toml` are in sync\n- `.github/workflows/metadata-ingestion.yml`: CI workflow configuration\n\n## Connector Documentation\n\nConnector documentation follows a structured format with specific file types and heading conventions.\n\n**Authoring guidelines**: See `docs/sources/AGENTS.md` in this directory for:\n\n- File structure (`_pre.md`, `_post.md`, `_recipe.yml`)\n- Heading level rules (H3 baseline, H2 reserved for module headings)\n- Section ordering (Overview → Concept Mapping → Prerequisites)\n- Content placement guidelines\n- Style conventions and formatting requirements\n\n**Quick reference:**\n\n```bash\n# Regenerate documentation after changes\n../gradlew :metadata-ingestion:docGen\n\n# Preview locally\n../gradlew :docs-website:yarnStart # Opens localhost:3001\n\n# Format markdown before committing\n../gradlew :datahub-web-react:mdPrettierWrite\n```\n",".claude/commands/test-review.md":"# Command: /test-review\n\nPath: `.claude/commands/test-review.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/commands/test-review.md)",".cursor/rules/datahub-dev.mdc":"# Cursor Rule: datahub-dev\n\nPath: `.cursor/rules/datahub-dev.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/datahub-dev.mdc)",".cursor/rules/test-review.mdc":"# Cursor Rule: test-review\n\nPath: `.cursor/rules/test-review.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/test-review.mdc)","contrib/metadata-model-extensions/README.md":"# metadata-model-extensions Documentation\n\nPath: `contrib/metadata-model-extensions/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/README.md)",".agent-skills/test-review/references/smoke-test-patterns.md":"# References Skill\n\nPath: `.agent-skills/test-review/references/smoke-test-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/references/smoke-test-patterns.md)","contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md":"# datahub-demo-dataset-governance-validator Documentation\n\nPath: `contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md)"},"files":{"AGENTS.md":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests (Python/JS lint lives in lint-jobs.yml / :module:lint)\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotless-check` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is assembled at build time into `docs-website/genDocs/` (gitignored) —\nsee `docs-website/AGENTS.md` for the full pipeline.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n - **Connectors**: File layout, lineage, reporting, and PR-scope conventions for ingestion connectors live in `metadata-ingestion/AGENTS.md`\n - **Smoke tests**: Authoring conventions live in [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n - **Frontend**: Component conventions, theming/color tokens, file layout, and the frontend CI checklist live in `datahub-web-react/AGENTS.md`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory. How to write them: [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md). How to run them: [`smoke-test/README.md`](smoke-test/README.md)\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\nFor pytest smoke tests, see [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n(authoring) and [`smoke-test/README.md`](smoke-test/README.md) (how to run).\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md) —\nincluding test-data seeding via `test.use({ featureName: ... })`.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n","docs-website/AGENTS.md":"# docs-website — Agent Development Guide\n\nThe docs site is a **Docusaurus 2** app. Docusaurus reads from `genDocs/` (gitignored, assembled at\nbuild time). Custom React pages live in `src/pages/`, blog/learn articles in `src/learn/`.\n\n## Build pipeline\n\nThe full build runs as a Gradle task chain. You rarely need to invoke individual steps, but\nunderstanding the pipeline helps when debugging.\n\n```\n./gradlew :docs-website:yarnBuild\n └─ yarnGenerate\n ├─ yarnInstall\n ├─ generateGraphQLSchema → graphql/combined.graphql\n ├─ generateJsonSchema → static/schemas/datahub_ingestion_schema.json\n ├─ :metadata-ingestion:modelDocGen → docs/generated/metamodel/\n └─ :metadata-ingestion:docGen → docs/generated/ingestion/\n```\n\n`yarnGenerate` runs `yarn run generate`, which:\n\n1. Generates GraphQL API docs via the `docusaurus-graphql-plugin`\n2. Generates Python SDK docs via Sphinx (`sphinx/` → Markdown conversion)\n3. Runs `generateDocsDir.ts` to assemble all markdown into `genDocs/`\n\n## `generateDocsDir.ts` — the assembly script\n\nThis TypeScript script is the central piece that discovers all markdown in the repo and writes\nprocessed output to `genDocs/`. Key behaviors:\n\n- **Discovery**: Uses `git ls-files` to find all `.md` files, then filters out excluded paths\n (`.github/`, `node_modules/`, `contrib/`, archived docs, test dirs, etc.)\n- **Frontmatter injection**: Adds `title`, `slug`, `description`, `custom_edit_url` to each file\n- **Link rewriting**: Rewrites relative links and image paths for the hosted site\n- **Inline directives**: `{{ inline /path/to/file.py }}` embeds file contents;\n `{{ command-output cmd }}` embeds command output\n- **Sidebar validation**: Warns if a doc file is not referenced in `sidebars.js` (some dirs are\n exempt: `docs/generated/metamodel`, `docs/generated/ingestion`, `docs/actions/`, etc.)\n\n## `sidebars.js` conventions\n\n- Doc IDs are file paths relative to `genDocs/`, minus the `.md` extension\n- Every hand-authored doc in `docs/` must appear in `sidebars.js` or the build warns\n- Auto-generated directories (`docs/generated/metamodel/`, `docs/generated/ingestion/`) are exempt\n\n## Key files\n\n| File | Purpose |\n| ----------------------- | ------------------------------------------------- |\n| `docusaurus.config.js` | Site config — plugins, navbar, footer, docs path |\n| `sidebars.js` | Sidebar navigation tree |\n| `generateDocsDir.ts` | Markdown discovery, transformation, and assembly |\n| `build.gradle` | Gradle tasks (`yarnGenerate`, `yarnBuild`, etc.) |\n| `sphinx/` | Python SDK doc generation (Sphinx → Markdown) |\n| `graphql/` | GraphQL schema combination + doc generation |\n| `genJsonSchema/` | Combines connector config schemas into one file |\n| `src/pages/` | Custom React pages (not markdown docs) |\n| `src/learn/` | Blog / learning articles (served at `/learn`) |\n| `static/` | Images, logos, static assets |\n\n## Generated directories (gitignored — never edit)\n\n- `genDocs/` — final assembled docs consumed by Docusaurus\n- `graphql/combined.graphql` — concatenated GraphQL schema\n- `static/schemas/datahub_ingestion_schema.json` — combined ingestion config schema\n\n## Logo convention\n\nPlatform logos: `static/img/logos/platforms/{platform_id}.svg`\nDataHub internal connectors: use `img/datahub-logo-color-mark.svg` (shared logo).\n",".agent-skills/test-review/SKILL.md":"# DataHub Test Review\n\nYou are an expert DataHub test reviewer. Your role is to evaluate pytest smoke\ntests against established testing standards, identify issues, and provide\nactionable feedback.\n\nAuthoring how-to: `smoke-test/AGENTS.md`. Review rubric:\n`standards/smoke.md`.\n\n---\n\n## Multi-Agent Compatibility\n\nThis skill is designed to work across multiple coding agents (Claude Code, Cursor, Codex, Copilot, Gemini CLI, Windsurf, and others).\n\n**What works everywhere:**\n\n- All review checklists, standards references, and procedures in this document\n- Bash for running scripts (`detect-test-changes.sh`, `gh` CLI, `git diff`)\n- Reading files, searching code, and generating review reports\n\n**Claude Code-specific features** (other agents can safely ignore these):\n\n- The `/test-review` slash command (`.claude/commands/test-review.md`) loads this skill automatically\n- The `test-quality-analyzer` agent (`.claude/agents/test-quality-analyzer.md`) can be dispatched for parallel analysis -- **fallback instructions are provided inline** for agents that cannot dispatch sub-agents\n- `TaskCreate`/`TaskUpdate` for progress tracking -- if unavailable, simply proceed through the steps sequentially\n\n**Standards file paths:** All standards are in the `standards/` directory alongside this file. All paths below are relative to `.agent-skills/test-review/`.\n\n---\n\n## Quick Start\n\n**Full review?** -> Load standards, gather test files, then launch `test-quality-analyzer` agent (or perform checks directly)\n\n**PR review?** -> Detect changed test files, then analyze only those files\n\n---\n\n## Scope\n\n### In Scope\n\n- `smoke-test/` -- Python pytest smoke tests (API-level tests against a running DataHub instance)\n- `smoke-test/tests/` -- shared test utilities, fixtures, and helpers\n\n### Out of Scope\n\n- `smoke-test/tests/cypress/` -- Cypress UI tests\n- `e2e-test/ui/playwright/` -- Playwright UI tests\n- `metadata-ingestion/tests/integration/` -- ingestion connector tests (covered by `datahub-connector-pr-review`)\n- `metadata-ingestion/tests/unit/` -- unit tests\n- `metadata-ingestion/src/datahub/testing/` -- ingestion testing utilities\n\nThe `detect-test-changes.sh` script lists in-scope Python smoke-test files.\n\n---\n\n## Review Modes\n\n| Mode | Use Case | Scope | Template |\n| ---------------------- | ------------------------------ | ------------------ | ---------------------------- |\n| **Full Review** | Audit all tests in a directory | All in-scope tests | `test-review-report.md` |\n| **Incremental Review** | PR with test changes | Changed files only | `incremental-test-report.md` |\n\n---\n\n## Startup: Load Standards\n\n**On activation, IMMEDIATELY load testing standards** from the `standards/` directory.\n\nRead `.agent-skills/test-review/standards/smoke.md` -- this contains all testing rules.\nAlso skim `smoke-test/AGENTS.md` if isolation or fixture guidance is in dispute.\n\nAfter loading, briefly confirm: \"Loaded test review standards. Ready to review.\"\n\n---\n\n## Progress Tracking with Tasks\n\n**After loading standards**, create a task checklist using TaskCreate:\n\n```\n1. Load testing standards\n2. Detect in-scope smoke test files\n3. Filter out connector-specific and Cypress tests\n4. Analyze smoke tests\n5. Generate review report\n```\n\nIf TaskCreate is not available, proceed through the steps sequentially.\n\n---\n\n## Mode 1: Full Review\n\n### Step 1: Gather Test Files\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\nOr list Python tests under `smoke-test/` excluding `tests/cypress/`.\n\n### Step 2: Load Standards into Context\n\nRead `.agent-skills/test-review/standards/smoke.md` completely.\n\n### Step 3: Launch Test Quality Analyzer\n\n**Claude Code (with Agent tool):**\n\n```\nAgent tool:\n subagent_type: \"test-quality-analyzer\"\n prompt: \"\"\"Analyze the following smoke test files for quality and standards compliance.\n\n\n[Content from .agent-skills/test-review/standards/smoke.md]\n\n\n\n[List of smoke-test Python file paths]\n\n\nFor each file, check all applicable rules from the standards document.\nReport findings with severity (BLOCKER/WARNING/SUGGESTION) and file:line references.\n\"\"\"\n```\n\n**Other agents (sequential fallback):**\n\nIf you cannot dispatch sub-agents, perform the analysis yourself:\n\n1. Read each test file completely\n2. Check against the smoke-test standards (isolation, fixtures, auth, retry, GraphQL/REST, markers, env vars, cleanup, placement, logging)\n3. Scan for anti-patterns listed at the end of the standards file\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/test-review-report.md` template. Fill in:\n\n- Summary table with pass/fail per category\n- All findings organized by severity (BLOCKER > WARNING > SUGGESTION)\n- Checklist results\n- Quality scores (1-10 per dimension)\n- Verdict: APPROVED / NEEDS CHANGES / BLOCKED\n\n**Verdict logic:**\n\n- **APPROVED**: No blockers, no more than 3 warnings\n- **NEEDS CHANGES**: Has warnings or fixable blockers\n- **BLOCKED**: Has fundamental anti-pattern blockers (empty tests, missing cleanup, hardcoded credentials, shared hardcoded URNs)\n\n---\n\n## Mode 2: Incremental Review (CI Mode)\n\n### Step 1: Detect Changed Test Files\n\n**If PR number provided:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh ${PR_NUMBER}\n```\n\n**If running locally:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\n**If no script available:**\n\n```bash\ngh pr diff ${PR_NUMBER} --name-only | grep -E '^smoke-test/' | \\\n grep -E '\\.py$' | grep -v '/tests/cypress/'\n```\n\n### Step 2: Classify Files\n\nParse the output of `detect-test-changes.sh`:\n\n- Lines starting with `smoke:` are smoke test files\n\nIf no in-scope test changes detected (exit code 1), report: \"No in-scope test changes found in this PR.\"\n\n### Step 3: Analyze Changed Files\n\nApply the same analysis as Mode 1, Step 3, but only to changed files.\n\nFor incremental reviews, also check:\n\n- Do new tests follow the same patterns as existing tests in their module?\n- Do modifications preserve existing test behavior?\n- Are unique-name helpers used instead of new shared hardcoded URNs?\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/incremental-test-report.md` template.\n\n---\n\n## CI Invocation\n\nFor non-interactive CI usage via `claude -p`:\n\n```bash\nclaude -p \"Review test changes in PR #${PR_NUMBER} using the test-review skill. \\\n Output the review report in markdown format with a verdict line.\"\n```\n\nThe skill produces deterministic output:\n\n- Structured markdown report\n- Clear verdict: `APPROVED`, `NEEDS CHANGES`, or `BLOCKED`\n- All findings cite file:line references\n\n---\n\n## Standards Reference\n\nAll standards are documented in `standards/smoke.md`. Authoring\nguide: `smoke-test/AGENTS.md`.\n\n1. **Isolation and unique names** -- `unique_suffix()` / unique ingest, no shared hardcoded URNs\n2. **Fixtures and data lifecycle** -- unique-dataset ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique\n3. **Authentication** -- `auth_session` fixture, `make_step_actor_user()`, never inline\n4. **Retry patterns** -- Trace API, `wait_for_writes_to_sync()`, `@with_test_retry()`, no bare `time.sleep()`\n5. **GraphQL / REST** -- `execute_graphql()`, `restli_default_headers`, `ingest_file_via_rest()`\n6. **Markers** -- required `domain(...)`; `p0` only for regressions that must run on every PR; `read_only` only when the test never mutates; `global_policy_mutator` only when mutating shared platform policy\n7. **Environment variables** -- `env_vars.py` registry, no hardcoded URLs\n8. **Guaranteed cleanup** -- fixture `yield` or `try/finally`\n9. **Multi-environment config** -- URLs via env vars, `USE_STATIC_SLEEP` fallback\n10. **Concurrent testing** -- thread-safe `run_concurrent_tests()`\n11. **Placement, logging, quality** -- `tests//`, `logger.info()`, no customer identifiers\n\n### Anti-Patterns (Automatic Blockers)\n\n- Empty/trivial tests\n- Missing cleanup\n- Hardcoded URLs/ports\n- Inline authentication\n- Bare `time.sleep()` for consistency\n- Shared hardcoded URNs / global mutable state\n- Mutating shared platform policy without `global_policy_mutator`\n\n---\n\n## Severity Levels\n\n| Level | Icon | Meaning | Action Required |\n| ---------- | ---- | ------------------ | ---------------------------- |\n| BLOCKER | ---- | Standard violated | Must fix before merge |\n| WARNING | ---- | Should be improved | Should fix before merge |\n| SUGGESTION | ---- | Nice to have | Optional, defer to follow-up |\n\n---\n\n## Templates\n\n- `templates/test-review-report.md` -- Full review report\n- `templates/incremental-test-report.md` -- PR incremental report\n\n---\n\n## Scripts\n\n- `scripts/detect-test-changes.sh` -- Detect in-scope changed smoke-test files\n\n---\n\n## References\n\n- `smoke-test/AGENTS.md` -- Canonical authoring guide\n- `references/smoke-test-patterns.md` -- Additional code examples\n\n---\n\n## Remember\n\n1. **Do not invent rules.** Standards come from `smoke-test/AGENTS.md` and the cited helpers.\n2. **Cypress, Playwright, and connector tests are out of scope.**\n3. **Be conservative with BLOCKERs.** Only flag when a standard is clearly violated. Older files that still use `_ingest_cleanup_data_impl` with unique keys are acceptable.\n4. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n5. **Acknowledge good patterns.** Note when tests follow standards well.\n6. **CI output must be deterministic.** Same input -> same verdict.\n","datahub-web-react/AGENTS.md":"# DataHub Web React — Agent Development Guide\n\nCanonical guide for `datahub-web-react/`. Repo-wide rules (Gradle-based formatting/linting,\ncommit and PR conventions, testing principles, running DataHub via `scripts/dev/datahub-dev.sh`)\nlive in the root [`AGENTS.md`](../AGENTS.md) and are not repeated here.\n\nRelated references:\n\n- React-specific code review rules: [`.cursor/BUGBOT.md`](.cursor/BUGBOT.md)\n- Playwright UI E2E tests: [`../e2e-test/ui/playwright/README.md`](../e2e-test/ui/playwright/README.md)\n\n## Development Commands\n\nFrom the **repo root**:\n\n```bash\n# Hot-reload dev server (preferred)\nscripts/dev/datahub-dev.sh setup frontend # one-time setup\nscripts/dev/datahub-dev.sh frontend # dev server with hot reload\n```\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Dev server proxying to a remote GMS instead of a local one\n../gradlew yarnPreview -Pproxy=\"\"\n\n# Regenerate TypeScript types after changing .graphql files\nyarn generate\n```\n\n### Lint, type-check, and test (before commit / PR)\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Full lint (eslint + prettier + type-check)\n../gradlew :datahub-web-react:yarnLint\n\n# Lint-fix a single file (fast; skips full type-check — run yarnLint before commit)\n../gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (icon stubs needed once per clone)\nnode scripts/generate-lazy-icon-stubs.js\nyarn test src/path/to/file.test.tsx --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in unrelated\nfiles. Focus on errors in files you touched — optional prop calls (`prop?.(arg)`) and\nimport aliases are common culprits.\n\n## File Structure\n\n- Top-level folders: `app` (application components, one folder per top-level nav page;\n nested pages live inside their parent's folder), `graphql` (GraphQL files and generated\n types), `images` (custom images), plus `conf`, `fonts`, `providers`, `utils`.\n- Do not create `index.ts(x)` barrel files.\n- Tests: in `__tests__/` next to the file under test, named `OriginalFile.test.ts(x)`.\n- Utils: used by one file → `SourceFile.utils.ts`; used across a folder →\n `folderName/folderName.utils.ts`.\n- Hooks: helper hooks for one file → `SourceFile.hooks.ts(x)`; standalone reusable hooks →\n `hookName.ts(x)`. Avoid hooks that return JSX unless it clearly makes sense.\n- Types: used once → same file as usage; shared → `folderName/folderName.types.ts`.\n- Helper components: small and single-use → `parent/SourceFile.components.tsx`; shared or\n large without child components → `parent/NewComponent.tsx`; has its own child\n components → `parent/NewComponent/NewComponent.tsx`.\n\n## Components\n\n- Prefer alchemy (`@components`) over `antd` — ESLint `rulesdir/no-antd-imports` enforces\n this. Files that already imported antd on the PR base (`origin/master`) are grandfathered.\n The alchemy library is documented in\n [`src/alchemy-components/README.mdx`](src/alchemy-components/README.mdx); preview\n components with `yarn storybook`.\n- An alchemy component is generalized and reusable anywhere: it must not depend on DataHub\n aspects, data structures, or GraphQL types.\n- Reusable DataHub-specific components (which may take GraphQL types as props) go under\n `src/app/sharedV2/`.\n- Err on the side of more, smaller components: break out a new component when JSX gets\n large or complex, or for any logical/reusable chunk. Same rule for extracting hooks from\n component logic.\n- Contexts: err on the side of not creating them. Use a global context only for truly\n global values (e.g. `AppContext`, `UserContext`). For a small self-contained scope, use\n `folder/Context.ts` (type, default value, helper hooks) plus\n `folder/ContextProvider.tsx` (just the provider).\n- Props: individual values for a small number of props; group into logical objects as they\n grow (e.g. `entityData` rather than one prop per aspect). For components shared between\n OSS and SaaS, SaaS-only props go in a single `acrylProps: { ... }` field.\n- Component file section order: imports, constants (SNAKE_CASE), styled components, props,\n main component.\n\n## Styling, Theming, Icons, and Images\n\n- Use `styled-components` with string CSS, not object CSS:\n\n ```tsx\n // YES\n styled.div`\n border-radius: 2px;\n `;\n\n // NO\n styled.div({ borderRadius: '2px' });\n ```\n\n- Custom components should accept `className` and pass it to the element custom styles\n should apply to — this enables `const CustomButton = styled(Button)` inheritance. If\n multiple inner elements are styleable, expose non-top-level ones as named\n `CSSProperties` props (e.g. `textStyle`, `buttonStyle`).\n\n**Colors — always use semantic color tokens** from `src/conf/theme/colorThemes/types.ts`.\nNever use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy\n`colors.gray[X]` imports.\n\nIn styled-components (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\nIn React component bodies:\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\nFor alchemy components (``, ``, etc.) — do not pass `color`/`colorLevel`\nprops; let them inherit from themed parent styled-components.\n\nDo not import from:\n\n- `src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally\n by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n**Icons**: use the alchemy `` component with phosphor icons only — ant and material\nUI icons are deprecated. Specify color and size via props; if size is unknown, use\n`size=\"inherit\"` and set `font-size` in the parent element. Import phosphor icons from\ntheir individual CSR paths for tree-shaking:\n`import { ArrowRight } from '@phosphor-icons/react/dist/csr/ArrowRight'` — never from\n`@phosphor-icons/react` (ESLint will error). Find icon names at https://phosphoricons.com.\n\n**Images**: use the alchemy `` component (`src` and `alt` are required; it\nhandles loading skeleton and error fallback). For SVGs, set `fill=\"currentColor\"` in the\nSVG definition so the icon inherits color from CSS.\n\n## TypeScript Style\n\nMost style is enforced by ESLint/Prettier. Conventions the tools don't enforce:\n\n- Prefer `type` over `interface` (except when using classes). Type props with TypeScript\n types, never PropTypes.\n- Top-level: named functions (`export default function f() {}`); nested: lambdas\n (`const onClick = () => {}`).\n- Don't annotate variables; do annotate function signatures unless impractical — prefer\n inferred typing over `any`. Prefer a mapper function over unsafe casting; if you must\n cast, do it as early (high) as possible.\n- Always optional-chain array access (`x?.[0]`).\n- Prefer direct imports (`import React, { useState } from 'react'`) over `React.useState`.\n\n## Unit Testing\n\n- Vitest + React Testing Library; tests live in `__tests__/` next to the source file.\n- Wrap components in `TestPageContainer` from `@utils/test-utils/TestPageContainer` — it\n provides the necessary app providers (theme, routing, contexts).\n- Use `MockedProvider` from `@apollo/client/testing` for components that make GraphQL\n queries.\n- Generate icon stubs once per clone before running tests:\n `node scripts/generate-lazy-icon-stubs.js`.\n- Follow the root guide's testing principles: test behavior, not implementation; skip\n tests that only exercise the framework or restate the linter.\n","CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","smoke-test/AGENTS.md":"# Smoke Test Guidelines\n\nCanonical authoring guide for pytest smoke tests against a running DataHub\ninstance. How to run the suite: `[README.md](README.md)`.\n\n## Where tests and helpers live\n\n| Path | What goes here |\n| ---------------------- | ----------------------------------------------------------------------------- |\n| `tests/e2e/` | Live GMS pytest (needs quickstart). Default `pytest` / `smoke.sh` collection. |\n| `tests/unit/` | CPU pytest (no stack). `./gradlew :smoke-test:stackFreePytest`. |\n| `tests/zdu/` | ZDU live E2E pytest (`test_zdu_upgrade.py`) and docs. Not in docker pytest. |\n| `tests/zdu/framework/` | ZDU upgrade framework. Daily workflow: `python -m tests.zdu.framework`. |\n| `tests/oauth/` | OAuth IdP smoke. `.github/workflows/oauth-smoke.yml`. Not in docker pytest. |\n| `utilities/` | Shared helpers, not tests. |\n\nDo not put `test_*.py` next to helpers in `utilities/`. GMS fixtures live in `tests/e2e/conftest.py` only.\n\n## When to add smoke tests\n\nThese are E2E API tests on a **live GMS** (slow, shared stack, xdist). Prefer a\nunit test next to the code, an ingestion test in `metadata-ingestion/tests/`, or a backend unit/integration test **before** adding a smoke test.\n\n**Add one** when the behavior needs GMS: write → search, a public API flow with\nno existing coverage, a production regression to keep, or live-stack authz.\n\n**Do not add** when:\n\n- The logic is a validator, parser, or in-process service.\n- The flow is already covered — extend that module; do not add `test_*_v2.py`.\n- You are stacking positive/negative variants of the same call. One happy path\n is enough. A negative case belongs here only if it needs GMS (e.g. policy\n deny), not invalid input or combinatorial 4xx/case/whitespace.\n\nPut new tests in `tests/e2e//`, not `test_e2e.py`. Connector tests are placed in `metadata-ingestion/tests/`. Generic placeholder names only\n(`my_db.my_schema.events`) — no customer identifiers or ticket IDs.\n\n## Rules\n\n- **Idempotent and order-independent.** Do not assume an empty GMS or another\n test's cleanup. No module-level mutable globals.\n- **Run-unique names** for every entity you create, mutate, or delete (see\n Isolation). Shared hardcoded URNs flake under xdist.\n- **Cleanup even on failure:** fixture `yield`, or `try/finally` inside a test\n (wrap delete in `try/except` so cleanup cannot hide the assertion).\n- **Do not mutate** the admin user, default All Users policies, or other shared\n platform state unless the module is marked `global_policy_mutator`.\n- `logger.info()`, not `print()`.\n- **No** `time.sleep()` for GMS/search/Kafka. Read-after-write:\n `@with_test_retry()`. Bulk ingest/cleanup: `wait_for_writes_to_sync()`\n (`mcp_only` / `mae_only` when only one store matters). Known `trace_id`: Trace\n API. `TestSessionWrapper` already waits on POST/PUT — do not extra-sync unless\n you are asserting search/index.\n- **Auth:** `auth_session` / `graph_client`. Extra users:\n `make_step_actor_user()`. Config: `utilities/env_vars.py`, not `os.getenv` or\n `localhost:8080`. GraphQL: `execute_graphql()` (it already checks `data` /\n `errors`) — assert the fields you care about. Ingest:\n `ingest_file_via_rest()`.\n- **Reuse helpers.** Do not copy-paste `addTag` or roll a local `uuid` when \n `unique_suffix()` exists. Tags/terms/descriptions: \n `utilities/metadata_operations.py`.\n- Extra users: `make_step_actor_user()` in `utilities/multi_user.py`, \n not a one-off signup.\n- `execute_graphql()` already asserts a non-empty body, `data` is not `None`,\n and no `errors` key, only assert the data that the test requires\n\n## Markers\n\n```python\nfrom utilities.domains import Domain\n\npytestmark = pytest.mark.domain(Domain.CATALOG)\n# pytestmark = pytest.mark.domain(Domain.CATALOG, Domain.INGESTION)\n```\n\n| Marker | When |\n| ----------------------- | ---------------------------------------------------------------------------------------------------------- |\n| `domain(...)` | **Required.** `platform`, `observe`, `ingestion`, `ai`, `catalog`. |\n| `p0` | Must run on every PR. |\n| `global_policy_mutator` | Disables default policies / mutates shared platform policy. CI runs these serially after parallel modules. |\n\nKeep `@pytest.mark.dependency()` chains short (ideally ≤3).\n\n## Isolation\n\nModules run in parallel against one GMS. Prefer these over a local `uuid` /\nstring replace:\n\n| Helper | Use |\n| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| `unique_suffix()` | Any entity key (user, domain, PAT, …). |\n| `unique_dataset_urn(name)` | Dataset created in code (GraphQL / SDK). |\n| `materialize_with_unique_name(src, name, dest_dir)` | Fixture rewrite. `name` must appear **only** in URN keys, not descriptions. Returns `(dest_file, unique_name)`. |\n| `materialize_unique_dataset(...)` | Dataset fixture → `(dest_file, dataset_urn)`. |\n| `_ingest_cleanup_unique_dataset_impl(...)` | Rewrite + ingest + yield URN + delete. No pre-delete (URN is new). Default for a shared dataset name in `data.json`. |\n| `_ingest_cleanup_data_impl(...)` | Pre-delete → ingest → cleanup. Only when keys are already unique to this module. |\n\n```python\nfrom conftest import _ingest_cleanup_unique_dataset_impl\n\n@pytest.fixture(scope=\"module\", autouse=True)\ndef dataset_urn(auth_session, graph_client, tmp_path_factory):\n yield from _ingest_cleanup_unique_dataset_impl(\n auth_session, graph_client,\n \"tests/e2e/tags_and_terms/data.json\", \"tags_and_terms\",\n \"test-tags-terms-sample-kafka\",\n tmp_path_factory.mktemp(\"tags_and_terms\"),\n )\n```\n\nTests take `dataset_urn` as a fixture argument. Multi-entity fixtures: rewrite\neach key with `materialize_with_unique_name` — see\n`tests/e2e/containers/containers_test.py`.\n\nMid-test creates:\n\n```python\ndataset_urn = unique_dataset_urn(\"my-feature\")\ntry:\n graph_client.emit(...)\n wait_for_writes_to_sync()\nfinally:\n try:\n delete_urn(graph_client, dataset_urn)\n except Exception:\n logger.warning(\"cleanup failed for %s\", dataset_urn, exc_info=True)\n```\n\n## Other helpers\n\n- `tests/utils.py`: `execute_graphql`, `ingest_file_via_rest`,\n `delete_urn` / `delete_urns` / `delete_urns_from_file`, `with_test_retry`,\n `wait_for_writes_to_sync`.\n- `tests/utilities/concurrent_test_runner.py`: thread-safe\n `run_concurrent_tests` / `run_concurrent_tests_with_args`.\n- `tests/utilities/concurrent_openapi.py`: `run_tests(auth_session, fixture_globs=...)`.\n JSON fixtures: `{request, response}` with DeepDiff `exclude_regex_paths`. Do\n not add OpenAPI fixtures that only re-check a GraphQL path already covered.\n",".claude/agents/test-quality-analyzer.md":"---\nname: test-quality-analyzer\ndescription: |\n Analyze DataHub pytest smoke test files for quality, standards compliance, and anti-patterns. Reports findings with severity levels and file:line references.\n\n \n Context: A PR adds new smoke tests for the incidents feature.\n user: \"Analyze the test quality of smoke-test/tests/incidents/incidents_test.py\"\n assistant: \"I'll use the test-quality-analyzer agent to check fixture usage, data lifecycle, assertions, and anti-patterns.\"\n \n Test quality analysis is a read-only inspection task that triggers this agent.\n \n \nmodel: sonnet\ncolor: blue\ntools:\n - Bash\n - Read\n - Grep\n - Glob\n---\n\n# DataHub Test Quality Analyzer Agent\n\nYou are a test quality analysis agent that inspects DataHub pytest smoke tests for compliance with established testing standards. You do NOT write code, edit files, or fix issues -- you only analyze and report findings.\n\nAuthoring guide: `smoke-test/AGENTS.md`. Review rubric: `.agent-skills/test-review/standards/smoke.md`.\n\n## Core Rules\n\n1. **Read-only analysis.** You have no Write or Edit tools. Inspect test files and report findings.\n2. **Cite file:line for every finding.** Every issue must reference the exact file path and line number.\n3. **Use severity levels consistently.** BLOCKER = must fix, WARNING = should fix, SUGGESTION = nice to have.\n4. **Smoke tests only.** Cypress, Playwright, and `metadata-ingestion/tests/` are out of scope.\n5. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n\n## Input\n\nYou receive:\n\n- A list of test files to analyze\n- The testing standards (embedded in the prompt as `` tags)\n\n## Workflow\n\n### Phase 1: File Classification\n\nFor each test file, determine:\n\n- **Smoke test** (Python pytest): Located under `smoke-test/` excluding `tests/cypress/`\n- **Out of scope**: `smoke-test/tests/cypress/`, `e2e-test/ui/playwright/`, `metadata-ingestion/tests/`\n\nSkip out-of-scope files.\n\n### Phase 2: Smoke Test Analysis\n\nFor each smoke test file, check:\n\n1. **Isolation** -- Run-unique names (`unique_suffix`, `unique_dataset_urn`, `materialize_*`, `_ingest_cleanup_unique_dataset_impl`)? No shared hardcoded URNs?\n2. **Data Lifecycle** -- Fixture `yield` teardown? Unique ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique?\n3. **Authentication** -- `auth_session` fixture, not inline credentials? Extra users via `make_step_actor_user()`?\n4. **Retry Patterns** -- `@with_test_retry()` or `wait_for_writes_to_sync()` instead of bare `time.sleep()`?\n5. **GraphQL / REST** -- `execute_graphql()` with specific field assertions? `ingest_file_via_rest()` / `restli_default_headers`?\n6. **Environment Variables** -- `env_vars.py` instead of `os.getenv`? No hardcoded URLs/ports?\n7. **Markers** -- `domain(...)` declared? `global_policy_mutator` if mutating shared policy? `read_only` tests truly read-only?\n8. **Cleanup** -- Fixture `yield` or `try/finally` for mid-test entities?\n9. **Test Isolation** -- No global mutable state? No cross-test module-level variables?\n10. **Logging** -- `logger.info()`, not `print()`?\n11. **Placement** -- New tests in `tests//`, not `test_e2e.py`?\n12. **Confidentiality** -- No customer identifiers or ticket IDs?\n\n### Phase 3: Anti-Pattern Detection\n\nScan for these automatic BLOCKER triggers:\n\n- Empty/trivial tests (`assert True`, only testing defaults)\n- Bare `time.sleep()` without retry wrapper\n- Hardcoded URLs or ports (should use env_vars registry)\n- Inline credential creation (should use auth_session fixture)\n- Shared hardcoded URNs across modules\n- Cross-test dependencies via global mutable state\n- Missing cleanup for created entities\n- Entities created mid-test without `try/finally`\n- Mutating admin / default policies without `global_policy_mutator`\n- Commented-out test code or breakpoints\n\n## Output Format\n\n```markdown\n# Test Quality Analysis\n\n**Files Analyzed:** {{COUNT}}\n**Smoke Tests:** {{SMOKE_COUNT}}\n\n---\n\n## Findings Summary\n\n| Severity | Count |\n| ---------- | ----- |\n| BLOCKER | {{N}} |\n| WARNING | {{N}} |\n| SUGGESTION | {{N}} |\n\n---\n\n## BLOCKER Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Standard:** {{WHICH_STANDARD_VIOLATED}}\n- **Issue:** {{DESCRIPTION}}\n- **Evidence:** {{CODE_SNIPPET_OR_EXPLANATION}}\n\n---\n\n## WARNING Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Issue:** {{DESCRIPTION}}\n- **Recommendation:** {{FIX}}\n\n---\n\n## SUGGESTION Issues\n\n- `{{FILE}}:{{LINE}}` -- {{DESCRIPTION}}\n\n---\n\n## Positive Observations\n\n- {{GOOD_PATTERN_FOUND}}\n\n---\n\n## Verdict\n\n**{{VERDICT}}** (APPROVED / NEEDS CHANGES / BLOCKED)\n```\n\n## Important Guidelines\n\n- **Be conservative with BLOCKERs.** Only flag as BLOCKER when a standard is clearly violated.\n- **Consider context.** Older test files may still use `_ingest_cleanup_data_impl` with unique keys -- flag new shared hardcoded URNs, not every legacy fixture.\n- **Acknowledge good patterns.** Note when tests follow standards well.\n- **Check the whole file.** Don't stop after finding the first issue.\n","metadata-ingestion/AGENTS.md":"# DataHub Metadata Ingestion Development Guide\n\n## Build and Test Commands\n\n**Using Gradle (slow but reliable):**\n\n```bash\n# Development setup from metadata-ingestion/\n../gradlew :metadata-ingestion:installDev # Setup Python environment\nsource venv/bin/activate # Activate virtual environment\n\n# Linting and formatting\n../gradlew :metadata-ingestion:lint # Run ruff + mypy\n../gradlew :metadata-ingestion:lintFix # Auto-fix linting issues\n\n# Testing\n../gradlew :metadata-ingestion:testQuick # Fast unit tests\n../gradlew :metadata-ingestion:testFull # All tests\n../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_file.py # Single test\n```\n\n**Direct Python commands (when venv is activated):**\n\n```bash\n# Testing\npytest -vv # Run all tests\npytest -m 'not integration' # Unit tests only\npytest -m 'integration' # Integration tests\npytest tests/path/to/file.py # Single test file\npytest tests/path/to/file.py::TestClass # Single test class\npytest tests/path/to/file.py::TestClass::test_method # Single test\n```\n\n## Environment Variables\n\n**Build configuration:**\n\n- `DATAHUB_VENV_USE_COPIES`: Set to `true` to use `--copies` flag when creating Python virtual environments. This copies the Python binary instead of creating a symlink. Useful for Nix environments, immutable filesystems, Windows, or container environments where symlinks don't work correctly. Increases disk usage and setup time, so only enable if needed.\n\n ```bash\n export DATAHUB_VENV_USE_COPIES=true\n ../gradlew :metadata-ingestion:installDev\n ```\n\n## Directory Structure\n\n- `src/datahub/`: Source code for the DataHub CLI and ingestion framework\n- `tests/`: All tests (NOT in `src/` directory)\n- `tests/unit/`: Unit tests\n- `tests/integration/`: Integration tests\n- `scripts/`: Build and development scripts\n- `examples/`: Example ingestion configurations\n- `developing.md`: Detailed development environment information\n\n## Code Style Guidelines\n\n- **Formatting**: Uses ruff, 88 character line length\n- **Imports**: Sorted with ruff.lint.isort, no relative imports, Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.\n- **Types**: Always use type annotations, prefer Protocol for interfaces\n - Avoid `Any` type - use specific types (`Dict[str, int]`, `TypedDict`, or typevars)\n - Use `isinstance` checks instead of `hasattr`\n - Prefer `assert isinstance(...)` over `cast`\n- **Data Structures**: Use dataclasses/pydantic for internal data representation\n - Return dataclasses instead of tuples from methods\n - Centralize utility functions to avoid code duplication\n- **Naming**: Descriptive names, match source system terminology in configs\n- **Error Handling**: Validators throw only ValueError/TypeError/AssertionError\n - Add robust error handling with layers of protection for known failure points\n- **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`\n- **Documentation**: All configs need descriptions\n- **Dependencies**: Avoid version pinning, use ranges with comments\n- **Architecture**: Avoid tall inheritance hierarchies, prefer mixins\n\n## Testing Conventions\n\n- **Location**: Tests go in `tests/` directory alongside `src/`, NOT in `src/`\n- **Structure**: Test files should mirror the source directory structure\n- **Framework**: Use pytest, not unittest\n- **Assertions**: Use `assert` statements, not `self.assertEqual()` or `self.assertIsNone()`\n - Boolean: `assert func()` or `assert not func()`, not `assert func() is True/False`\n - None: `assert result is None` (correct), not `assert result == None`\n - Keep tests concise, avoid verbose repetitive patterns\n- **Classes**: Use regular classes, not `unittest.TestCase`\n- **Imports**: Import `pytest` in test files\n- **Naming**: Test files should be named `test_*.py`\n- **Categories**:\n - Unit tests: `tests/unit/` - fast, no external dependencies\n - Integration tests: `tests/integration/` - may use Docker/external services\n\n## Configuration Guidelines (Pydantic)\n\n- **Naming**: Match terminology of the source system (e.g., `account_id` for Snowflake, not `host_port`)\n- **Descriptions**: All configs must have descriptions\n- **Patterns**: Use AllowDenyPatterns for filtering, named `*_pattern`\n- **Defaults**: Set reasonable defaults, avoid config-driven filtering that should be automatic\n- **Validation**: Single pydantic validator per validation concern\n- **Security**: Use `SecretStr` for passwords, auth tokens, etc.\n- **Deprecation**: Use `pydantic_removed_field` helper for field deprecations\n\n## Connector Conventions\n\n- **Structure**: Split connector code by duty: `constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`, `mapper.py`, and `usage.py` as needed. Match the quality and structure of existing connectors like Power BI, Airbyte, Redshift, BigID, and Grafana.\n- **Constants**: Avoid double-quoted string literals in connector code. Hoist magic strings into module-level constants, and keep regex patterns pre-compiled in the constants file.\n- **Comments**: No top-of-file docstrings in connector modules. Keep comments and docstrings only where they add real context.\n- **Imports**: Do not use `TYPE_CHECKING` in connector code. Lazy imports are fine only for opt-in features the connector does not always depend on.\n- **Syntax**: Do not use the walrus operator in connector code.\n- **Types**: Prefer `Dict` and `List` from `typing` over builtin `dict` and `list` in connector type annotations.\n- **Reporting**: Prefer `self.report.warning(...)` and report counters over bare `logger` warnings for skips, inaccessible objects, and other operator-visible edge cases.\n- **SQL lineage**: Use `SqlParsingAggregator` via `create_lineage_from_sql_statements` with a platform map instead of setting sqlglot dialects per connector. Mirror existing connectors for known-URN mapping, platform instance / env handling, casing, multi-part names, and temp-table handling.\n- **Column lineage**: Resolve upstream and downstream schemas from the DataHub graph when available, load known URNs from the platform / platform instance / env mapping, and match columns case-insensitively.\n- **Progress**: Surface ingestion progress and use explicit ingestion stages, following connectors like Dremio and Snowflake.\n- **New connectors**: A connector usually needs more than Python code: source logo, integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration in `setup.py`, `../gradlew :metadata-ingestion:updateLockFile`, and shared subtypes instead of connector-local subtype definitions.\n- **XML**: Avoid Python's stdlib `xml` parser. Use a safe XML library, following the HANA-related code path.\n- **Validation workflow**: Before committing a new connector, run ingestion locally in debug mode to a local file and inspect the full logs. When testing against customer environments, keep secrets only in temporary paths such as `/tmp/*.env`.\n- **PR scope**: Keep each connector in its own PR. Split shared framework changes such as sqlglot helpers into a separate PR, and keep the PR title and description focused on that connector only.\n\n## Dependencies and Entry Points\n\n`setup.py` is the **source of truth** for all dependencies, extras, and entry points. `pyproject.toml`, `uv.lock`, and `constraints.txt` are **generated from it** and must never be edited manually.\n\n**After any change to `setup.py`** (new dependency, new entry point, version bump), run:\n\n```bash\n../gradlew :metadata-ingestion:updateLockFile\n```\n\nThis regenerates the full chain: `setup.py` → `pyproject.toml` → `uv.lock` → `constraints.txt`.\n\nCI runs `checkLockFile` which calls `scripts/verify_pyproject_equivalence.py` to verify these files are in sync. If you edit `setup.py` without running `updateLockFile`, CI will fail.\n\n**Adding a new ingestion source entry point:**\n\n1. Add the entry to `setup.py` under `entry_points[\"datahub.ingestion.source.plugins\"]`\n2. Run `../gradlew :metadata-ingestion:updateLockFile`\n3. Verify with `../gradlew :metadata-ingestion:checkLockFile`\n\n## Key Files\n\n- `src/datahub/emitter/mcp_builder.py`: Examples of defining various aspect types\n- `setup.py`: Source of truth for dependencies, extras, and entry points\n- `pyproject.toml`: Generated from `setup.py` — do not edit manually\n- `setup.cfg`: Code style configuration\n- `scripts/generate_pyproject_deps.py`: Generates `pyproject.toml` from `setup.py`\n- `scripts/verify_pyproject_equivalence.py`: Validates `setup.py` and `pyproject.toml` are in sync\n- `.github/workflows/metadata-ingestion.yml`: CI workflow configuration\n\n## Connector Documentation\n\nConnector documentation follows a structured format with specific file types and heading conventions.\n\n**Authoring guidelines**: See `docs/sources/AGENTS.md` in this directory for:\n\n- File structure (`_pre.md`, `_post.md`, `_recipe.yml`)\n- Heading level rules (H3 baseline, H2 reserved for module headings)\n- Section ordering (Overview → Concept Mapping → Prerequisites)\n- Content placement guidelines\n- Style conventions and formatting requirements\n\n**Quick reference:**\n\n```bash\n# Regenerate documentation after changes\n../gradlew :metadata-ingestion:docGen\n\n# Preview locally\n../gradlew :docs-website:yarnStart # Opens localhost:3001\n\n# Format markdown before committing\n../gradlew :datahub-web-react:mdPrettierWrite\n```\n",".claude/commands/test-review.md":"# Command: /test-review\n\nPath: `.claude/commands/test-review.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/commands/test-review.md)",".cursor/rules/datahub-dev.mdc":"# Cursor Rule: datahub-dev\n\nPath: `.cursor/rules/datahub-dev.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/datahub-dev.mdc)",".cursor/rules/test-review.mdc":"# Cursor Rule: test-review\n\nPath: `.cursor/rules/test-review.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/test-review.mdc)","contrib/metadata-model-extensions/README.md":"# metadata-model-extensions Documentation\n\nPath: `contrib/metadata-model-extensions/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/README.md)",".agent-skills/test-review/references/smoke-test-patterns.md":"# References Skill\n\nPath: `.agent-skills/test-review/references/smoke-test-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/references/smoke-test-patterns.md)","contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md":"# datahub-demo-dataset-governance-validator Documentation\n\nPath: `contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests (Python/JS lint lives in lint-jobs.yml / :module:lint)\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotless-check` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is assembled at build time into `docs-website/genDocs/` (gitignored) —\nsee `docs-website/AGENTS.md` for the full pipeline.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n - **Connectors**: File layout, lineage, reporting, and PR-scope conventions for ingestion connectors live in `metadata-ingestion/AGENTS.md`\n - **Smoke tests**: Authoring conventions live in [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n - **Frontend**: Component conventions, theming/color tokens, file layout, and the frontend CI checklist live in `datahub-web-react/AGENTS.md`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory. How to write them: [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md). How to run them: [`smoke-test/README.md`](smoke-test/README.md)\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\nFor pytest smoke tests, see [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n(authoring) and [`smoke-test/README.md`](smoke-test/README.md) (how to run).\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md) —\nincluding test-data seeding via `test.use({ featureName: ... })`.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n","isInternal":false,"tokens":7279,"sizeBytes":31935},{"name":"AGENTS.md","path":"datahub-web-react/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/datahub-web-react/AGENTS.md","title":"datahub-web-react - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# DataHub Web React — Agent Development Guide\n\nCanonical guide for `datahub-web-react/`. Repo-wide rules (Gradle-based formatting/linting,\ncommit and PR conventions, testing principles, running DataHub via `scripts/dev/datahub-dev.sh`)\nlive in the root [`AGENTS.md`](../AGENTS.md) and are not repeated here.\n\nRelated references:\n\n- React-specific code review rules: [`.cursor/BUGBOT.md`](.cursor/BUGBOT.md)\n- Playwright UI E2E tests: [`../e2e-test/ui/playwright/README.md`](../e2e-test/ui/playwright/README.md)\n\n## Development Commands\n\nFrom the **repo root**:\n\n```bash\n# Hot-reload dev server (preferred)\nscripts/dev/datahub-dev.sh setup frontend # one-time setup\nscripts/dev/datahub-dev.sh frontend # dev server with hot reload\n```\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Dev server proxying to a remote GMS instead of a local one\n../gradlew yarnPreview -Pproxy=\"\"\n\n# Regenerate TypeScript types after changing .graphql files\nyarn generate\n```\n\n### Lint, type-check, and test (before commit / PR)\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Full lint (eslint + prettier + type-check)\n../gradlew :datahub-web-react:yarnLint\n\n# Lint-fix a single file (fast; skips full type-check — run yarnLint before commit)\n../gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (icon stubs needed once per clone)\nnode scripts/generate-lazy-icon-stubs.js\nyarn test src/path/to/file.test.tsx --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in unrelated\nfiles. Focus on errors in files you touched — optional prop calls (`prop?.(arg)`) and\nimport aliases are common culprits.\n\n## File Structure\n\n- Top-level folders: `app` (application components, one folder per top-level nav page;\n nested pages live inside their parent's folder), `graphql` (GraphQL files and generated\n types), `images` (custom images), plus `conf`, `fonts`, `providers`, `utils`.\n- Do not create `index.ts(x)` barrel files.\n- Tests: in `__tests__/` next to the file under test, named `OriginalFile.test.ts(x)`.\n- Utils: used by one file → `SourceFile.utils.ts`; used across a folder →\n `folderName/folderName.utils.ts`.\n- Hooks: helper hooks for one file → `SourceFile.hooks.ts(x)`; standalone reusable hooks →\n `hookName.ts(x)`. Avoid hooks that return JSX unless it clearly makes sense.\n- Types: used once → same file as usage; shared → `folderName/folderName.types.ts`.\n- Helper components: small and single-use → `parent/SourceFile.components.tsx`; shared or\n large without child components → `parent/NewComponent.tsx`; has its own child\n components → `parent/NewComponent/NewComponent.tsx`.\n\n## Components\n\n- Prefer alchemy (`@components`) over `antd` — ESLint `rulesdir/no-antd-imports` enforces\n this. Files that already imported antd on the PR base (`origin/master`) are grandfathered.\n The alchemy library is documented in\n [`src/alchemy-components/README.mdx`](src/alchemy-components/README.mdx); preview\n components with `yarn storybook`.\n- An alchemy component is generalized and reusable anywhere: it must not depend on DataHub\n aspects, data structures, or GraphQL types.\n- Reusable DataHub-specific components (which may take GraphQL types as props) go under\n `src/app/sharedV2/`.\n- Err on the side of more, smaller components: break out a new component when JSX gets\n large or complex, or for any logical/reusable chunk. Same rule for extracting hooks from\n component logic.\n- Contexts: err on the side of not creating them. Use a global context only for truly\n global values (e.g. `AppContext`, `UserContext`). For a small self-contained scope, use\n `folder/Context.ts` (type, default value, helper hooks) plus\n `folder/ContextProvider.tsx` (just the provider).\n- Props: individual values for a small number of props; group into logical objects as they\n grow (e.g. `entityData` rather than one prop per aspect). For components shared between\n OSS and SaaS, SaaS-only props go in a single `acrylProps: { ... }` field.\n- Component file section order: imports, constants (SNAKE_CASE), styled components, props,\n main component.\n\n## Styling, Theming, Icons, and Images\n\n- Use `styled-components` with string CSS, not object CSS:\n\n ```tsx\n // YES\n styled.div`\n border-radius: 2px;\n `;\n\n // NO\n styled.div({ borderRadius: '2px' });\n ```\n\n- Custom components should accept `className` and pass it to the element custom styles\n should apply to — this enables `const CustomButton = styled(Button)` inheritance. If\n multiple inner elements are styleable, expose non-top-level ones as named\n `CSSProperties` props (e.g. `textStyle`, `buttonStyle`).\n\n**Colors — always use semantic color tokens** from `src/conf/theme/colorThemes/types.ts`.\nNever use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy\n`colors.gray[X]` imports.\n\nIn styled-components (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\nIn React component bodies:\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\nFor alchemy components (``, ``, etc.) — do not pass `color`/`colorLevel`\nprops; let them inherit from themed parent styled-components.\n\nDo not import from:\n\n- `src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally\n by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n**Icons**: use the alchemy `` component with phosphor icons only — ant and material\nUI icons are deprecated. Specify color and size via props; if size is unknown, use\n`size=\"inherit\"` and set `font-size` in the parent element. Import phosphor icons from\ntheir individual CSR paths for tree-shaking:\n`import { ArrowRight } from '@phosphor-icons/react/dist/csr/ArrowRight'` — never from\n`@phosphor-icons/react` (ESLint will error). Find icon names at https://phosphoricons.com.\n\n**Images**: use the alchemy `` component (`src` and `alt` are required; it\nhandles loading skeleton and error fallback). For SVGs, set `fill=\"currentColor\"` in the\nSVG definition so the icon inherits color from CSS.\n\n## TypeScript Style\n\nMost style is enforced by ESLint/Prettier. Conventions the tools don't enforce:\n\n- Prefer `type` over `interface` (except when using classes). Type props with TypeScript\n types, never PropTypes.\n- Top-level: named functions (`export default function f() {}`); nested: lambdas\n (`const onClick = () => {}`).\n- Don't annotate variables; do annotate function signatures unless impractical — prefer\n inferred typing over `any`. Prefer a mapper function over unsafe casting; if you must\n cast, do it as early (high) as possible.\n- Always optional-chain array access (`x?.[0]`).\n- Prefer direct imports (`import React, { useState } from 'react'`) over `React.useState`.\n\n## Unit Testing\n\n- Vitest + React Testing Library; tests live in `__tests__/` next to the source file.\n- Wrap components in `TestPageContainer` from `@utils/test-utils/TestPageContainer` — it\n provides the necessary app providers (theme, routing, contexts).\n- Use `MockedProvider` from `@apollo/client/testing` for components that make GraphQL\n queries.\n- Generate icon stubs once per clone before running tests:\n `node scripts/generate-lazy-icon-stubs.js`.\n- Follow the root guide's testing principles: test behavior, not implementation; skip\n tests that only exercise the framework or restate the linter.\n","isInternal":false,"tokens":1896,"sizeBytes":7702},{"name":"AGENTS.md","path":"docs-website/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/docs-website/AGENTS.md","title":"docs-website - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# docs-website — Agent Development Guide\n\nThe docs site is a **Docusaurus 2** app. Docusaurus reads from `genDocs/` (gitignored, assembled at\nbuild time). Custom React pages live in `src/pages/`, blog/learn articles in `src/learn/`.\n\n## Build pipeline\n\nThe full build runs as a Gradle task chain. You rarely need to invoke individual steps, but\nunderstanding the pipeline helps when debugging.\n\n```\n./gradlew :docs-website:yarnBuild\n └─ yarnGenerate\n ├─ yarnInstall\n ├─ generateGraphQLSchema → graphql/combined.graphql\n ├─ generateJsonSchema → static/schemas/datahub_ingestion_schema.json\n ├─ :metadata-ingestion:modelDocGen → docs/generated/metamodel/\n └─ :metadata-ingestion:docGen → docs/generated/ingestion/\n```\n\n`yarnGenerate` runs `yarn run generate`, which:\n\n1. Generates GraphQL API docs via the `docusaurus-graphql-plugin`\n2. Generates Python SDK docs via Sphinx (`sphinx/` → Markdown conversion)\n3. Runs `generateDocsDir.ts` to assemble all markdown into `genDocs/`\n\n## `generateDocsDir.ts` — the assembly script\n\nThis TypeScript script is the central piece that discovers all markdown in the repo and writes\nprocessed output to `genDocs/`. Key behaviors:\n\n- **Discovery**: Uses `git ls-files` to find all `.md` files, then filters out excluded paths\n (`.github/`, `node_modules/`, `contrib/`, archived docs, test dirs, etc.)\n- **Frontmatter injection**: Adds `title`, `slug`, `description`, `custom_edit_url` to each file\n- **Link rewriting**: Rewrites relative links and image paths for the hosted site\n- **Inline directives**: `{{ inline /path/to/file.py }}` embeds file contents;\n `{{ command-output cmd }}` embeds command output\n- **Sidebar validation**: Warns if a doc file is not referenced in `sidebars.js` (some dirs are\n exempt: `docs/generated/metamodel`, `docs/generated/ingestion`, `docs/actions/`, etc.)\n\n## `sidebars.js` conventions\n\n- Doc IDs are file paths relative to `genDocs/`, minus the `.md` extension\n- Every hand-authored doc in `docs/` must appear in `sidebars.js` or the build warns\n- Auto-generated directories (`docs/generated/metamodel/`, `docs/generated/ingestion/`) are exempt\n\n## Key files\n\n| File | Purpose |\n| ----------------------- | ------------------------------------------------- |\n| `docusaurus.config.js` | Site config — plugins, navbar, footer, docs path |\n| `sidebars.js` | Sidebar navigation tree |\n| `generateDocsDir.ts` | Markdown discovery, transformation, and assembly |\n| `build.gradle` | Gradle tasks (`yarnGenerate`, `yarnBuild`, etc.) |\n| `sphinx/` | Python SDK doc generation (Sphinx → Markdown) |\n| `graphql/` | GraphQL schema combination + doc generation |\n| `genJsonSchema/` | Combines connector config schemas into one file |\n| `src/pages/` | Custom React pages (not markdown docs) |\n| `src/learn/` | Blog / learning articles (served at `/learn`) |\n| `static/` | Images, logos, static assets |\n\n## Generated directories (gitignored — never edit)\n\n- `genDocs/` — final assembled docs consumed by Docusaurus\n- `graphql/combined.graphql` — concatenated GraphQL schema\n- `static/schemas/datahub_ingestion_schema.json` — combined ingestion config schema\n\n## Logo convention\n\nPlatform logos: `static/img/logos/platforms/{platform_id}.svg`\nDataHub internal connectors: use `img/datahub-logo-color-mark.svg` (shared logo).\n","isInternal":false,"tokens":842,"sizeBytes":3610},{"name":"AGENTS.md","path":"metadata-ingestion/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/metadata-ingestion/AGENTS.md","title":"metadata-ingestion - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# DataHub Metadata Ingestion Development Guide\n\n## Build and Test Commands\n\n**Using Gradle (slow but reliable):**\n\n```bash\n# Development setup from metadata-ingestion/\n../gradlew :metadata-ingestion:installDev # Setup Python environment\nsource venv/bin/activate # Activate virtual environment\n\n# Linting and formatting\n../gradlew :metadata-ingestion:lint # Run ruff + mypy\n../gradlew :metadata-ingestion:lintFix # Auto-fix linting issues\n\n# Testing\n../gradlew :metadata-ingestion:testQuick # Fast unit tests\n../gradlew :metadata-ingestion:testFull # All tests\n../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_file.py # Single test\n```\n\n**Direct Python commands (when venv is activated):**\n\n```bash\n# Testing\npytest -vv # Run all tests\npytest -m 'not integration' # Unit tests only\npytest -m 'integration' # Integration tests\npytest tests/path/to/file.py # Single test file\npytest tests/path/to/file.py::TestClass # Single test class\npytest tests/path/to/file.py::TestClass::test_method # Single test\n```\n\n## Environment Variables\n\n**Build configuration:**\n\n- `DATAHUB_VENV_USE_COPIES`: Set to `true` to use `--copies` flag when creating Python virtual environments. This copies the Python binary instead of creating a symlink. Useful for Nix environments, immutable filesystems, Windows, or container environments where symlinks don't work correctly. Increases disk usage and setup time, so only enable if needed.\n\n ```bash\n export DATAHUB_VENV_USE_COPIES=true\n ../gradlew :metadata-ingestion:installDev\n ```\n\n## Directory Structure\n\n- `src/datahub/`: Source code for the DataHub CLI and ingestion framework\n- `tests/`: All tests (NOT in `src/` directory)\n- `tests/unit/`: Unit tests\n- `tests/integration/`: Integration tests\n- `scripts/`: Build and development scripts\n- `examples/`: Example ingestion configurations\n- `developing.md`: Detailed development environment information\n\n## Code Style Guidelines\n\n- **Formatting**: Uses ruff, 88 character line length\n- **Imports**: Sorted with ruff.lint.isort, no relative imports, Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.\n- **Types**: Always use type annotations, prefer Protocol for interfaces\n - Avoid `Any` type - use specific types (`Dict[str, int]`, `TypedDict`, or typevars)\n - Use `isinstance` checks instead of `hasattr`\n - Prefer `assert isinstance(...)` over `cast`\n- **Data Structures**: Use dataclasses/pydantic for internal data representation\n - Return dataclasses instead of tuples from methods\n - Centralize utility functions to avoid code duplication\n- **Naming**: Descriptive names, match source system terminology in configs\n- **Error Handling**: Validators throw only ValueError/TypeError/AssertionError\n - Add robust error handling with layers of protection for known failure points\n- **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`\n- **Documentation**: All configs need descriptions\n- **Dependencies**: Avoid version pinning, use ranges with comments\n- **Architecture**: Avoid tall inheritance hierarchies, prefer mixins\n\n## Testing Conventions\n\n- **Location**: Tests go in `tests/` directory alongside `src/`, NOT in `src/`\n- **Structure**: Test files should mirror the source directory structure\n- **Framework**: Use pytest, not unittest\n- **Assertions**: Use `assert` statements, not `self.assertEqual()` or `self.assertIsNone()`\n - Boolean: `assert func()` or `assert not func()`, not `assert func() is True/False`\n - None: `assert result is None` (correct), not `assert result == None`\n - Keep tests concise, avoid verbose repetitive patterns\n- **Classes**: Use regular classes, not `unittest.TestCase`\n- **Imports**: Import `pytest` in test files\n- **Naming**: Test files should be named `test_*.py`\n- **Categories**:\n - Unit tests: `tests/unit/` - fast, no external dependencies\n - Integration tests: `tests/integration/` - may use Docker/external services\n\n## Configuration Guidelines (Pydantic)\n\n- **Naming**: Match terminology of the source system (e.g., `account_id` for Snowflake, not `host_port`)\n- **Descriptions**: All configs must have descriptions\n- **Patterns**: Use AllowDenyPatterns for filtering, named `*_pattern`\n- **Defaults**: Set reasonable defaults, avoid config-driven filtering that should be automatic\n- **Validation**: Single pydantic validator per validation concern\n- **Security**: Use `SecretStr` for passwords, auth tokens, etc.\n- **Deprecation**: Use `pydantic_removed_field` helper for field deprecations\n\n## Connector Conventions\n\n- **Structure**: Split connector code by duty: `constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`, `mapper.py`, and `usage.py` as needed. Match the quality and structure of existing connectors like Power BI, Airbyte, Redshift, BigID, and Grafana.\n- **Constants**: Avoid double-quoted string literals in connector code. Hoist magic strings into module-level constants, and keep regex patterns pre-compiled in the constants file.\n- **Comments**: No top-of-file docstrings in connector modules. Keep comments and docstrings only where they add real context.\n- **Imports**: Do not use `TYPE_CHECKING` in connector code. Lazy imports are fine only for opt-in features the connector does not always depend on.\n- **Syntax**: Do not use the walrus operator in connector code.\n- **Types**: Prefer `Dict` and `List` from `typing` over builtin `dict` and `list` in connector type annotations.\n- **Reporting**: Prefer `self.report.warning(...)` and report counters over bare `logger` warnings for skips, inaccessible objects, and other operator-visible edge cases.\n- **SQL lineage**: Use `SqlParsingAggregator` via `create_lineage_from_sql_statements` with a platform map instead of setting sqlglot dialects per connector. Mirror existing connectors for known-URN mapping, platform instance / env handling, casing, multi-part names, and temp-table handling.\n- **Column lineage**: Resolve upstream and downstream schemas from the DataHub graph when available, load known URNs from the platform / platform instance / env mapping, and match columns case-insensitively.\n- **Progress**: Surface ingestion progress and use explicit ingestion stages, following connectors like Dremio and Snowflake.\n- **New connectors**: A connector usually needs more than Python code: source logo, integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration in `setup.py`, `../gradlew :metadata-ingestion:updateLockFile`, and shared subtypes instead of connector-local subtype definitions.\n- **XML**: Avoid Python's stdlib `xml` parser. Use a safe XML library, following the HANA-related code path.\n- **Validation workflow**: Before committing a new connector, run ingestion locally in debug mode to a local file and inspect the full logs. When testing against customer environments, keep secrets only in temporary paths such as `/tmp/*.env`.\n- **PR scope**: Keep each connector in its own PR. Split shared framework changes such as sqlglot helpers into a separate PR, and keep the PR title and description focused on that connector only.\n\n## Dependencies and Entry Points\n\n`setup.py` is the **source of truth** for all dependencies, extras, and entry points. `pyproject.toml`, `uv.lock`, and `constraints.txt` are **generated from it** and must never be edited manually.\n\n**After any change to `setup.py`** (new dependency, new entry point, version bump), run:\n\n```bash\n../gradlew :metadata-ingestion:updateLockFile\n```\n\nThis regenerates the full chain: `setup.py` → `pyproject.toml` → `uv.lock` → `constraints.txt`.\n\nCI runs `checkLockFile` which calls `scripts/verify_pyproject_equivalence.py` to verify these files are in sync. If you edit `setup.py` without running `updateLockFile`, CI will fail.\n\n**Adding a new ingestion source entry point:**\n\n1. Add the entry to `setup.py` under `entry_points[\"datahub.ingestion.source.plugins\"]`\n2. Run `../gradlew :metadata-ingestion:updateLockFile`\n3. Verify with `../gradlew :metadata-ingestion:checkLockFile`\n\n## Key Files\n\n- `src/datahub/emitter/mcp_builder.py`: Examples of defining various aspect types\n- `setup.py`: Source of truth for dependencies, extras, and entry points\n- `pyproject.toml`: Generated from `setup.py` — do not edit manually\n- `setup.cfg`: Code style configuration\n- `scripts/generate_pyproject_deps.py`: Generates `pyproject.toml` from `setup.py`\n- `scripts/verify_pyproject_equivalence.py`: Validates `setup.py` and `pyproject.toml` are in sync\n- `.github/workflows/metadata-ingestion.yml`: CI workflow configuration\n\n## Connector Documentation\n\nConnector documentation follows a structured format with specific file types and heading conventions.\n\n**Authoring guidelines**: See `docs/sources/AGENTS.md` in this directory for:\n\n- File structure (`_pre.md`, `_post.md`, `_recipe.yml`)\n- Heading level rules (H3 baseline, H2 reserved for module headings)\n- Section ordering (Overview → Concept Mapping → Prerequisites)\n- Content placement guidelines\n- Style conventions and formatting requirements\n\n**Quick reference:**\n\n```bash\n# Regenerate documentation after changes\n../gradlew :metadata-ingestion:docGen\n\n# Preview locally\n../gradlew :docs-website:yarnStart # Opens localhost:3001\n\n# Format markdown before committing\n../gradlew :datahub-web-react:mdPrettierWrite\n```\n","isInternal":false,"tokens":2167,"sizeBytes":9570},{"name":"AGENTS.md","path":"smoke-test/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/smoke-test/AGENTS.md","title":"smoke-test - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Smoke Test Guidelines\n\nCanonical authoring guide for pytest smoke tests against a running DataHub\ninstance. How to run the suite: `[README.md](README.md)`.\n\n## Where tests and helpers live\n\n| Path | What goes here |\n| ---------------------- | ----------------------------------------------------------------------------- |\n| `tests/e2e/` | Live GMS pytest (needs quickstart). Default `pytest` / `smoke.sh` collection. |\n| `tests/unit/` | CPU pytest (no stack). `./gradlew :smoke-test:stackFreePytest`. |\n| `tests/zdu/` | ZDU live E2E pytest (`test_zdu_upgrade.py`) and docs. Not in docker pytest. |\n| `tests/zdu/framework/` | ZDU upgrade framework. Daily workflow: `python -m tests.zdu.framework`. |\n| `tests/oauth/` | OAuth IdP smoke. `.github/workflows/oauth-smoke.yml`. Not in docker pytest. |\n| `utilities/` | Shared helpers, not tests. |\n\nDo not put `test_*.py` next to helpers in `utilities/`. GMS fixtures live in `tests/e2e/conftest.py` only.\n\n## When to add smoke tests\n\nThese are E2E API tests on a **live GMS** (slow, shared stack, xdist). Prefer a\nunit test next to the code, an ingestion test in `metadata-ingestion/tests/`, or a backend unit/integration test **before** adding a smoke test.\n\n**Add one** when the behavior needs GMS: write → search, a public API flow with\nno existing coverage, a production regression to keep, or live-stack authz.\n\n**Do not add** when:\n\n- The logic is a validator, parser, or in-process service.\n- The flow is already covered — extend that module; do not add `test_*_v2.py`.\n- You are stacking positive/negative variants of the same call. One happy path\n is enough. A negative case belongs here only if it needs GMS (e.g. policy\n deny), not invalid input or combinatorial 4xx/case/whitespace.\n\nPut new tests in `tests/e2e//`, not `test_e2e.py`. Connector tests are placed in `metadata-ingestion/tests/`. Generic placeholder names only\n(`my_db.my_schema.events`) — no customer identifiers or ticket IDs.\n\n## Rules\n\n- **Idempotent and order-independent.** Do not assume an empty GMS or another\n test's cleanup. No module-level mutable globals.\n- **Run-unique names** for every entity you create, mutate, or delete (see\n Isolation). Shared hardcoded URNs flake under xdist.\n- **Cleanup even on failure:** fixture `yield`, or `try/finally` inside a test\n (wrap delete in `try/except` so cleanup cannot hide the assertion).\n- **Do not mutate** the admin user, default All Users policies, or other shared\n platform state unless the module is marked `global_policy_mutator`.\n- `logger.info()`, not `print()`.\n- **No** `time.sleep()` for GMS/search/Kafka. Read-after-write:\n `@with_test_retry()`. Bulk ingest/cleanup: `wait_for_writes_to_sync()`\n (`mcp_only` / `mae_only` when only one store matters). Known `trace_id`: Trace\n API. `TestSessionWrapper` already waits on POST/PUT — do not extra-sync unless\n you are asserting search/index.\n- **Auth:** `auth_session` / `graph_client`. Extra users:\n `make_step_actor_user()`. Config: `utilities/env_vars.py`, not `os.getenv` or\n `localhost:8080`. GraphQL: `execute_graphql()` (it already checks `data` /\n `errors`) — assert the fields you care about. Ingest:\n `ingest_file_via_rest()`.\n- **Reuse helpers.** Do not copy-paste `addTag` or roll a local `uuid` when \n `unique_suffix()` exists. Tags/terms/descriptions: \n `utilities/metadata_operations.py`.\n- Extra users: `make_step_actor_user()` in `utilities/multi_user.py`, \n not a one-off signup.\n- `execute_graphql()` already asserts a non-empty body, `data` is not `None`,\n and no `errors` key, only assert the data that the test requires\n\n## Markers\n\n```python\nfrom utilities.domains import Domain\n\npytestmark = pytest.mark.domain(Domain.CATALOG)\n# pytestmark = pytest.mark.domain(Domain.CATALOG, Domain.INGESTION)\n```\n\n| Marker | When |\n| ----------------------- | ---------------------------------------------------------------------------------------------------------- |\n| `domain(...)` | **Required.** `platform`, `observe`, `ingestion`, `ai`, `catalog`. |\n| `p0` | Must run on every PR. |\n| `global_policy_mutator` | Disables default policies / mutates shared platform policy. CI runs these serially after parallel modules. |\n\nKeep `@pytest.mark.dependency()` chains short (ideally ≤3).\n\n## Isolation\n\nModules run in parallel against one GMS. Prefer these over a local `uuid` /\nstring replace:\n\n| Helper | Use |\n| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| `unique_suffix()` | Any entity key (user, domain, PAT, …). |\n| `unique_dataset_urn(name)` | Dataset created in code (GraphQL / SDK). |\n| `materialize_with_unique_name(src, name, dest_dir)` | Fixture rewrite. `name` must appear **only** in URN keys, not descriptions. Returns `(dest_file, unique_name)`. |\n| `materialize_unique_dataset(...)` | Dataset fixture → `(dest_file, dataset_urn)`. |\n| `_ingest_cleanup_unique_dataset_impl(...)` | Rewrite + ingest + yield URN + delete. No pre-delete (URN is new). Default for a shared dataset name in `data.json`. |\n| `_ingest_cleanup_data_impl(...)` | Pre-delete → ingest → cleanup. Only when keys are already unique to this module. |\n\n```python\nfrom conftest import _ingest_cleanup_unique_dataset_impl\n\n@pytest.fixture(scope=\"module\", autouse=True)\ndef dataset_urn(auth_session, graph_client, tmp_path_factory):\n yield from _ingest_cleanup_unique_dataset_impl(\n auth_session, graph_client,\n \"tests/e2e/tags_and_terms/data.json\", \"tags_and_terms\",\n \"test-tags-terms-sample-kafka\",\n tmp_path_factory.mktemp(\"tags_and_terms\"),\n )\n```\n\nTests take `dataset_urn` as a fixture argument. Multi-entity fixtures: rewrite\neach key with `materialize_with_unique_name` — see\n`tests/e2e/containers/containers_test.py`.\n\nMid-test creates:\n\n```python\ndataset_urn = unique_dataset_urn(\"my-feature\")\ntry:\n graph_client.emit(...)\n wait_for_writes_to_sync()\nfinally:\n try:\n delete_urn(graph_client, dataset_urn)\n except Exception:\n logger.warning(\"cleanup failed for %s\", dataset_urn, exc_info=True)\n```\n\n## Other helpers\n\n- `tests/utils.py`: `execute_graphql`, `ingest_file_via_rest`,\n `delete_urn` / `delete_urns` / `delete_urns_from_file`, `with_test_retry`,\n `wait_for_writes_to_sync`.\n- `tests/utilities/concurrent_test_runner.py`: thread-safe\n `run_concurrent_tests` / `run_concurrent_tests_with_args`.\n- `tests/utilities/concurrent_openapi.py`: `run_tests(auth_session, fixture_globs=...)`.\n JSON fixtures: `{request, response}` with DeepDiff `exclude_regex_paths`. Do\n not add OpenAPI fixtures that only re-check a GraphQL path already covered.\n","isInternal":false,"tokens":1690,"sizeBytes":7736},{"name":"smoke-test-patterns.md","path":".agent-skills/test-review/references/smoke-test-patterns.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/references/smoke-test-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `.agent-skills/test-review/references/smoke-test-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/references/smoke-test-patterns.md)","tokens":300,"sizeBytes":241},{"name":"SKILL.md","path":".agent-skills/test-review/SKILL.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/SKILL.md","title":"test-review","category":"anthropic-skill","format":"markdown","content":"# DataHub Test Review\n\nYou are an expert DataHub test reviewer. Your role is to evaluate pytest smoke\ntests against established testing standards, identify issues, and provide\nactionable feedback.\n\nAuthoring how-to: `smoke-test/AGENTS.md`. Review rubric:\n`standards/smoke.md`.\n\n---\n\n## Multi-Agent Compatibility\n\nThis skill is designed to work across multiple coding agents (Claude Code, Cursor, Codex, Copilot, Gemini CLI, Windsurf, and others).\n\n**What works everywhere:**\n\n- All review checklists, standards references, and procedures in this document\n- Bash for running scripts (`detect-test-changes.sh`, `gh` CLI, `git diff`)\n- Reading files, searching code, and generating review reports\n\n**Claude Code-specific features** (other agents can safely ignore these):\n\n- The `/test-review` slash command (`.claude/commands/test-review.md`) loads this skill automatically\n- The `test-quality-analyzer` agent (`.claude/agents/test-quality-analyzer.md`) can be dispatched for parallel analysis -- **fallback instructions are provided inline** for agents that cannot dispatch sub-agents\n- `TaskCreate`/`TaskUpdate` for progress tracking -- if unavailable, simply proceed through the steps sequentially\n\n**Standards file paths:** All standards are in the `standards/` directory alongside this file. All paths below are relative to `.agent-skills/test-review/`.\n\n---\n\n## Quick Start\n\n**Full review?** -> Load standards, gather test files, then launch `test-quality-analyzer` agent (or perform checks directly)\n\n**PR review?** -> Detect changed test files, then analyze only those files\n\n---\n\n## Scope\n\n### In Scope\n\n- `smoke-test/` -- Python pytest smoke tests (API-level tests against a running DataHub instance)\n- `smoke-test/tests/` -- shared test utilities, fixtures, and helpers\n\n### Out of Scope\n\n- `smoke-test/tests/cypress/` -- Cypress UI tests\n- `e2e-test/ui/playwright/` -- Playwright UI tests\n- `metadata-ingestion/tests/integration/` -- ingestion connector tests (covered by `datahub-connector-pr-review`)\n- `metadata-ingestion/tests/unit/` -- unit tests\n- `metadata-ingestion/src/datahub/testing/` -- ingestion testing utilities\n\nThe `detect-test-changes.sh` script lists in-scope Python smoke-test files.\n\n---\n\n## Review Modes\n\n| Mode | Use Case | Scope | Template |\n| ---------------------- | ------------------------------ | ------------------ | ---------------------------- |\n| **Full Review** | Audit all tests in a directory | All in-scope tests | `test-review-report.md` |\n| **Incremental Review** | PR with test changes | Changed files only | `incremental-test-report.md` |\n\n---\n\n## Startup: Load Standards\n\n**On activation, IMMEDIATELY load testing standards** from the `standards/` directory.\n\nRead `.agent-skills/test-review/standards/smoke.md` -- this contains all testing rules.\nAlso skim `smoke-test/AGENTS.md` if isolation or fixture guidance is in dispute.\n\nAfter loading, briefly confirm: \"Loaded test review standards. Ready to review.\"\n\n---\n\n## Progress Tracking with Tasks\n\n**After loading standards**, create a task checklist using TaskCreate:\n\n```\n1. Load testing standards\n2. Detect in-scope smoke test files\n3. Filter out connector-specific and Cypress tests\n4. Analyze smoke tests\n5. Generate review report\n```\n\nIf TaskCreate is not available, proceed through the steps sequentially.\n\n---\n\n## Mode 1: Full Review\n\n### Step 1: Gather Test Files\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\nOr list Python tests under `smoke-test/` excluding `tests/cypress/`.\n\n### Step 2: Load Standards into Context\n\nRead `.agent-skills/test-review/standards/smoke.md` completely.\n\n### Step 3: Launch Test Quality Analyzer\n\n**Claude Code (with Agent tool):**\n\n```\nAgent tool:\n subagent_type: \"test-quality-analyzer\"\n prompt: \"\"\"Analyze the following smoke test files for quality and standards compliance.\n\n\n[Content from .agent-skills/test-review/standards/smoke.md]\n\n\n\n[List of smoke-test Python file paths]\n\n\nFor each file, check all applicable rules from the standards document.\nReport findings with severity (BLOCKER/WARNING/SUGGESTION) and file:line references.\n\"\"\"\n```\n\n**Other agents (sequential fallback):**\n\nIf you cannot dispatch sub-agents, perform the analysis yourself:\n\n1. Read each test file completely\n2. Check against the smoke-test standards (isolation, fixtures, auth, retry, GraphQL/REST, markers, env vars, cleanup, placement, logging)\n3. Scan for anti-patterns listed at the end of the standards file\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/test-review-report.md` template. Fill in:\n\n- Summary table with pass/fail per category\n- All findings organized by severity (BLOCKER > WARNING > SUGGESTION)\n- Checklist results\n- Quality scores (1-10 per dimension)\n- Verdict: APPROVED / NEEDS CHANGES / BLOCKED\n\n**Verdict logic:**\n\n- **APPROVED**: No blockers, no more than 3 warnings\n- **NEEDS CHANGES**: Has warnings or fixable blockers\n- **BLOCKED**: Has fundamental anti-pattern blockers (empty tests, missing cleanup, hardcoded credentials, shared hardcoded URNs)\n\n---\n\n## Mode 2: Incremental Review (CI Mode)\n\n### Step 1: Detect Changed Test Files\n\n**If PR number provided:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh ${PR_NUMBER}\n```\n\n**If running locally:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\n**If no script available:**\n\n```bash\ngh pr diff ${PR_NUMBER} --name-only | grep -E '^smoke-test/' | \\\n grep -E '\\.py$' | grep -v '/tests/cypress/'\n```\n\n### Step 2: Classify Files\n\nParse the output of `detect-test-changes.sh`:\n\n- Lines starting with `smoke:` are smoke test files\n\nIf no in-scope test changes detected (exit code 1), report: \"No in-scope test changes found in this PR.\"\n\n### Step 3: Analyze Changed Files\n\nApply the same analysis as Mode 1, Step 3, but only to changed files.\n\nFor incremental reviews, also check:\n\n- Do new tests follow the same patterns as existing tests in their module?\n- Do modifications preserve existing test behavior?\n- Are unique-name helpers used instead of new shared hardcoded URNs?\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/incremental-test-report.md` template.\n\n---\n\n## CI Invocation\n\nFor non-interactive CI usage via `claude -p`:\n\n```bash\nclaude -p \"Review test changes in PR #${PR_NUMBER} using the test-review skill. \\\n Output the review report in markdown format with a verdict line.\"\n```\n\nThe skill produces deterministic output:\n\n- Structured markdown report\n- Clear verdict: `APPROVED`, `NEEDS CHANGES`, or `BLOCKED`\n- All findings cite file:line references\n\n---\n\n## Standards Reference\n\nAll standards are documented in `standards/smoke.md`. Authoring\nguide: `smoke-test/AGENTS.md`.\n\n1. **Isolation and unique names** -- `unique_suffix()` / unique ingest, no shared hardcoded URNs\n2. **Fixtures and data lifecycle** -- unique-dataset ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique\n3. **Authentication** -- `auth_session` fixture, `make_step_actor_user()`, never inline\n4. **Retry patterns** -- Trace API, `wait_for_writes_to_sync()`, `@with_test_retry()`, no bare `time.sleep()`\n5. **GraphQL / REST** -- `execute_graphql()`, `restli_default_headers`, `ingest_file_via_rest()`\n6. **Markers** -- required `domain(...)`; `p0` only for regressions that must run on every PR; `read_only` only when the test never mutates; `global_policy_mutator` only when mutating shared platform policy\n7. **Environment variables** -- `env_vars.py` registry, no hardcoded URLs\n8. **Guaranteed cleanup** -- fixture `yield` or `try/finally`\n9. **Multi-environment config** -- URLs via env vars, `USE_STATIC_SLEEP` fallback\n10. **Concurrent testing** -- thread-safe `run_concurrent_tests()`\n11. **Placement, logging, quality** -- `tests//`, `logger.info()`, no customer identifiers\n\n### Anti-Patterns (Automatic Blockers)\n\n- Empty/trivial tests\n- Missing cleanup\n- Hardcoded URLs/ports\n- Inline authentication\n- Bare `time.sleep()` for consistency\n- Shared hardcoded URNs / global mutable state\n- Mutating shared platform policy without `global_policy_mutator`\n\n---\n\n## Severity Levels\n\n| Level | Icon | Meaning | Action Required |\n| ---------- | ---- | ------------------ | ---------------------------- |\n| BLOCKER | ---- | Standard violated | Must fix before merge |\n| WARNING | ---- | Should be improved | Should fix before merge |\n| SUGGESTION | ---- | Nice to have | Optional, defer to follow-up |\n\n---\n\n## Templates\n\n- `templates/test-review-report.md` -- Full review report\n- `templates/incremental-test-report.md` -- PR incremental report\n\n---\n\n## Scripts\n\n- `scripts/detect-test-changes.sh` -- Detect in-scope changed smoke-test files\n\n---\n\n## References\n\n- `smoke-test/AGENTS.md` -- Canonical authoring guide\n- `references/smoke-test-patterns.md` -- Additional code examples\n\n---\n\n## Remember\n\n1. **Do not invent rules.** Standards come from `smoke-test/AGENTS.md` and the cited helpers.\n2. **Cypress, Playwright, and connector tests are out of scope.**\n3. **Be conservative with BLOCKERs.** Only flag when a standard is clearly violated. Older files that still use `_ingest_cleanup_data_impl` with unique keys are acceptable.\n4. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n5. **Acknowledge good patterns.** Note when tests follow standards well.\n6. **CI output must be deterministic.** Same input -> same verdict.\n","isInternal":false,"tokens":2271,"sizeBytes":9637},{"name":"datahub-dev.mdc","path":".cursor/rules/datahub-dev.mdc","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/datahub-dev.mdc","title":"Cursor Rule: datahub-dev","category":"cursor-rule","format":"mdc","content":"# Cursor Rule: datahub-dev\n\nPath: `.cursor/rules/datahub-dev.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/datahub-dev.mdc)","tokens":300,"sizeBytes":189},{"name":"test-review.mdc","path":".cursor/rules/test-review.mdc","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/test-review.mdc","title":"Cursor Rule: test-review","category":"cursor-rule","format":"mdc","content":"# Cursor Rule: test-review\n\nPath: `.cursor/rules/test-review.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/test-review.mdc)","tokens":300,"sizeBytes":189},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\n@AGENTS.md\n","isInternal":false,"tokens":11,"sizeBytes":24},{"name":"README.md","path":"contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md","title":"datahub-demo-dataset-governance-validator Documentation","category":"plugin-manifest","format":"markdown","content":"# datahub-demo-dataset-governance-validator Documentation\n\nPath: `contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md)","tokens":300,"sizeBytes":332},{"name":"README.md","path":"contrib/metadata-model-extensions/README.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/README.md","title":"metadata-model-extensions Documentation","category":"plugin-manifest","format":"markdown","content":"# metadata-model-extensions Documentation\n\nPath: `contrib/metadata-model-extensions/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/README.md)","tokens":300,"sizeBytes":232},{"name":"test-quality-analyzer.md","path":".claude/agents/test-quality-analyzer.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/agents/test-quality-analyzer.md","title":"test-quality-analyzer","category":"subagent-persona","format":"markdown","content":"---\nname: test-quality-analyzer\ndescription: |\n Analyze DataHub pytest smoke test files for quality, standards compliance, and anti-patterns. Reports findings with severity levels and file:line references.\n\n \n Context: A PR adds new smoke tests for the incidents feature.\n user: \"Analyze the test quality of smoke-test/tests/incidents/incidents_test.py\"\n assistant: \"I'll use the test-quality-analyzer agent to check fixture usage, data lifecycle, assertions, and anti-patterns.\"\n \n Test quality analysis is a read-only inspection task that triggers this agent.\n \n \nmodel: sonnet\ncolor: blue\ntools:\n - Bash\n - Read\n - Grep\n - Glob\n---\n\n# DataHub Test Quality Analyzer Agent\n\nYou are a test quality analysis agent that inspects DataHub pytest smoke tests for compliance with established testing standards. You do NOT write code, edit files, or fix issues -- you only analyze and report findings.\n\nAuthoring guide: `smoke-test/AGENTS.md`. Review rubric: `.agent-skills/test-review/standards/smoke.md`.\n\n## Core Rules\n\n1. **Read-only analysis.** You have no Write or Edit tools. Inspect test files and report findings.\n2. **Cite file:line for every finding.** Every issue must reference the exact file path and line number.\n3. **Use severity levels consistently.** BLOCKER = must fix, WARNING = should fix, SUGGESTION = nice to have.\n4. **Smoke tests only.** Cypress, Playwright, and `metadata-ingestion/tests/` are out of scope.\n5. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n\n## Input\n\nYou receive:\n\n- A list of test files to analyze\n- The testing standards (embedded in the prompt as `` tags)\n\n## Workflow\n\n### Phase 1: File Classification\n\nFor each test file, determine:\n\n- **Smoke test** (Python pytest): Located under `smoke-test/` excluding `tests/cypress/`\n- **Out of scope**: `smoke-test/tests/cypress/`, `e2e-test/ui/playwright/`, `metadata-ingestion/tests/`\n\nSkip out-of-scope files.\n\n### Phase 2: Smoke Test Analysis\n\nFor each smoke test file, check:\n\n1. **Isolation** -- Run-unique names (`unique_suffix`, `unique_dataset_urn`, `materialize_*`, `_ingest_cleanup_unique_dataset_impl`)? No shared hardcoded URNs?\n2. **Data Lifecycle** -- Fixture `yield` teardown? Unique ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique?\n3. **Authentication** -- `auth_session` fixture, not inline credentials? Extra users via `make_step_actor_user()`?\n4. **Retry Patterns** -- `@with_test_retry()` or `wait_for_writes_to_sync()` instead of bare `time.sleep()`?\n5. **GraphQL / REST** -- `execute_graphql()` with specific field assertions? `ingest_file_via_rest()` / `restli_default_headers`?\n6. **Environment Variables** -- `env_vars.py` instead of `os.getenv`? No hardcoded URLs/ports?\n7. **Markers** -- `domain(...)` declared? `global_policy_mutator` if mutating shared policy? `read_only` tests truly read-only?\n8. **Cleanup** -- Fixture `yield` or `try/finally` for mid-test entities?\n9. **Test Isolation** -- No global mutable state? No cross-test module-level variables?\n10. **Logging** -- `logger.info()`, not `print()`?\n11. **Placement** -- New tests in `tests//`, not `test_e2e.py`?\n12. **Confidentiality** -- No customer identifiers or ticket IDs?\n\n### Phase 3: Anti-Pattern Detection\n\nScan for these automatic BLOCKER triggers:\n\n- Empty/trivial tests (`assert True`, only testing defaults)\n- Bare `time.sleep()` without retry wrapper\n- Hardcoded URLs or ports (should use env_vars registry)\n- Inline credential creation (should use auth_session fixture)\n- Shared hardcoded URNs across modules\n- Cross-test dependencies via global mutable state\n- Missing cleanup for created entities\n- Entities created mid-test without `try/finally`\n- Mutating admin / default policies without `global_policy_mutator`\n- Commented-out test code or breakpoints\n\n## Output Format\n\n```markdown\n# Test Quality Analysis\n\n**Files Analyzed:** {{COUNT}}\n**Smoke Tests:** {{SMOKE_COUNT}}\n\n---\n\n## Findings Summary\n\n| Severity | Count |\n| ---------- | ----- |\n| BLOCKER | {{N}} |\n| WARNING | {{N}} |\n| SUGGESTION | {{N}} |\n\n---\n\n## BLOCKER Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Standard:** {{WHICH_STANDARD_VIOLATED}}\n- **Issue:** {{DESCRIPTION}}\n- **Evidence:** {{CODE_SNIPPET_OR_EXPLANATION}}\n\n---\n\n## WARNING Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Issue:** {{DESCRIPTION}}\n- **Recommendation:** {{FIX}}\n\n---\n\n## SUGGESTION Issues\n\n- `{{FILE}}:{{LINE}}` -- {{DESCRIPTION}}\n\n---\n\n## Positive Observations\n\n- {{GOOD_PATTERN_FOUND}}\n\n---\n\n## Verdict\n\n**{{VERDICT}}** (APPROVED / NEEDS CHANGES / BLOCKED)\n```\n\n## Important Guidelines\n\n- **Be conservative with BLOCKERs.** Only flag as BLOCKER when a standard is clearly violated.\n- **Consider context.** Older test files may still use `_ingest_cleanup_data_impl` with unique keys -- flag new shared hardcoded URNs, not every legacy fixture.\n- **Acknowledge good patterns.** Note when tests follow standards well.\n- **Check the whole file.** Don't stop after finding the first issue.\n","frontmatter":{"name":"test-quality-analyzer","description":"Analyze DataHub pytest smoke test files for quality, standards compliance, and anti-patterns. Reports findings with severity levels and file:line references.\n\n\nContext: A PR adds new smoke tests for the incidents feature.\nuser: \"Analyze the test quality of smoke-test/tests/incidents/incidents_test.py\"\nassistant: \"I'll use the test-quality-analyzer agent to check fixture usage, data lifecycle, assertions, and anti-patterns.\"\n\nTest quality analysis is a read-only inspection task that triggers this agent.\n\n\n","model":"sonnet","color":"blue","tools":["Bash","Read","Grep","Glob"]},"isInternal":false,"tokens":1269,"sizeBytes":5105},{"name":"test-review.md","path":".claude/commands/test-review.md","rawUrl":"https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/commands/test-review.md","title":"Command: /test-review","category":"command-prompt","format":"markdown","content":"# Command: /test-review\n\nPath: `.claude/commands/test-review.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/commands/test-review.md)","tokens":300,"sizeBytes":190}],"systemPromptSnippet":"\n\n# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests (Python/JS lint lives in lint-jobs.yml / :module:lint)\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew lintCheck # Run all lint checks (Python, JS, GraphQL, markdown, Java)\n./gradlew lintFix # Auto-fix all lint issues\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotless-check` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is assembled at build time into `docs-website/genDocs/` (gitignored) —\nsee `docs-website/AGENTS.md` for the full pipeline.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n - **Connectors**: File layout, lineage, reporting, and PR-scope conventions for ingestion connectors live in `metadata-ingestion/AGENTS.md`\n - **Smoke tests**: Authoring conventions live in [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n - **Frontend**: Component conventions, theming/color tokens, file layout, and the frontend CI checklist live in `datahub-web-react/AGENTS.md`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory. How to write them: [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md). How to run them: [`smoke-test/README.md`](smoke-test/README.md)\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\nFor pytest smoke tests, see [`smoke-test/AGENTS.md`](smoke-test/AGENTS.md)\n(authoring) and [`smoke-test/README.md`](smoke-test/README.md) (how to run).\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md) —\nincluding test-data seeding via `test.use({ featureName: ... })`.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n\n\n# DataHub Web React — Agent Development Guide\n\nCanonical guide for `datahub-web-react/`. Repo-wide rules (Gradle-based formatting/linting,\ncommit and PR conventions, testing principles, running DataHub via `scripts/dev/datahub-dev.sh`)\nlive in the root [`AGENTS.md`](../AGENTS.md) and are not repeated here.\n\nRelated references:\n\n- React-specific code review rules: [`.cursor/BUGBOT.md`](.cursor/BUGBOT.md)\n- Playwright UI E2E tests: [`../e2e-test/ui/playwright/README.md`](../e2e-test/ui/playwright/README.md)\n\n## Development Commands\n\nFrom the **repo root**:\n\n```bash\n# Hot-reload dev server (preferred)\nscripts/dev/datahub-dev.sh setup frontend # one-time setup\nscripts/dev/datahub-dev.sh frontend # dev server with hot reload\n```\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Dev server proxying to a remote GMS instead of a local one\n../gradlew yarnPreview -Pproxy=\"\"\n\n# Regenerate TypeScript types after changing .graphql files\nyarn generate\n```\n\n### Lint, type-check, and test (before commit / PR)\n\nFrom **`datahub-web-react/`**:\n\n```bash\n# Full lint (eslint + prettier + type-check)\n../gradlew :datahub-web-react:yarnLint\n\n# Lint-fix a single file (fast; skips full type-check — run yarnLint before commit)\n../gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (icon stubs needed once per clone)\nnode scripts/generate-lazy-icon-stubs.js\nyarn test src/path/to/file.test.tsx --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in unrelated\nfiles. Focus on errors in files you touched — optional prop calls (`prop?.(arg)`) and\nimport aliases are common culprits.\n\n## File Structure\n\n- Top-level folders: `app` (application components, one folder per top-level nav page;\n nested pages live inside their parent's folder), `graphql` (GraphQL files and generated\n types), `images` (custom images), plus `conf`, `fonts`, `providers`, `utils`.\n- Do not create `index.ts(x)` barrel files.\n- Tests: in `__tests__/` next to the file under test, named `OriginalFile.test.ts(x)`.\n- Utils: used by one file → `SourceFile.utils.ts`; used across a folder →\n `folderName/folderName.utils.ts`.\n- Hooks: helper hooks for one file → `SourceFile.hooks.ts(x)`; standalone reusable hooks →\n `hookName.ts(x)`. Avoid hooks that return JSX unless it clearly makes sense.\n- Types: used once → same file as usage; shared → `folderName/folderName.types.ts`.\n- Helper components: small and single-use → `parent/SourceFile.components.tsx`; shared or\n large without child components → `parent/NewComponent.tsx`; has its own child\n components → `parent/NewComponent/NewComponent.tsx`.\n\n## Components\n\n- Prefer alchemy (`@components`) over `antd` — ESLint `rulesdir/no-antd-imports` enforces\n this. Files that already imported antd on the PR base (`origin/master`) are grandfathered.\n The alchemy library is documented in\n [`src/alchemy-components/README.mdx`](src/alchemy-components/README.mdx); preview\n components with `yarn storybook`.\n- An alchemy component is generalized and reusable anywhere: it must not depend on DataHub\n aspects, data structures, or GraphQL types.\n- Reusable DataHub-specific components (which may take GraphQL types as props) go under\n `src/app/sharedV2/`.\n- Err on the side of more, smaller components: break out a new component when JSX gets\n large or complex, or for any logical/reusable chunk. Same rule for extracting hooks from\n component logic.\n- Contexts: err on the side of not creating them. Use a global context only for truly\n global values (e.g. `AppContext`, `UserContext`). For a small self-contained scope, use\n `folder/Context.ts` (type, default value, helper hooks) plus\n `folder/ContextProvider.tsx` (just the provider).\n- Props: individual values for a small number of props; group into logical objects as they\n grow (e.g. `entityData` rather than one prop per aspect). For components shared between\n OSS and SaaS, SaaS-only props go in a single `acrylProps: { ... }` field.\n- Component file section order: imports, constants (SNAKE_CASE), styled components, props,\n main component.\n\n## Styling, Theming, Icons, and Images\n\n- Use `styled-components` with string CSS, not object CSS:\n\n ```tsx\n // YES\n styled.div`\n border-radius: 2px;\n `;\n\n // NO\n styled.div({ borderRadius: '2px' });\n ```\n\n- Custom components should accept `className` and pass it to the element custom styles\n should apply to — this enables `const CustomButton = styled(Button)` inheritance. If\n multiple inner elements are styleable, expose non-top-level ones as named\n `CSSProperties` props (e.g. `textStyle`, `buttonStyle`).\n\n**Colors — always use semantic color tokens** from `src/conf/theme/colorThemes/types.ts`.\nNever use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy\n`colors.gray[X]` imports.\n\nIn styled-components (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\nIn React component bodies:\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\nFor alchemy components (``, ``, etc.) — do not pass `color`/`colorLevel`\nprops; let them inherit from themed parent styled-components.\n\nDo not import from:\n\n- `src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally\n by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n**Icons**: use the alchemy `` component with phosphor icons only — ant and material\nUI icons are deprecated. Specify color and size via props; if size is unknown, use\n`size=\"inherit\"` and set `font-size` in the parent element. Import phosphor icons from\ntheir individual CSR paths for tree-shaking:\n`import { ArrowRight } from '@phosphor-icons/react/dist/csr/ArrowRight'` — never from\n`@phosphor-icons/react` (ESLint will error). Find icon names at https://phosphoricons.com.\n\n**Images**: use the alchemy `` component (`src` and `alt` are required; it\nhandles loading skeleton and error fallback). For SVGs, set `fill=\"currentColor\"` in the\nSVG definition so the icon inherits color from CSS.\n\n## TypeScript Style\n\nMost style is enforced by ESLint/Prettier. Conventions the tools don't enforce:\n\n- Prefer `type` over `interface` (except when using classes). Type props with TypeScript\n types, never PropTypes.\n- Top-level: named functions (`export default function f() {}`); nested: lambdas\n (`const onClick = () => {}`).\n- Don't annotate variables; do annotate function signatures unless impractical — prefer\n inferred typing over `any`. Prefer a mapper function over unsafe casting; if you must\n cast, do it as early (high) as possible.\n- Always optional-chain array access (`x?.[0]`).\n- Prefer direct imports (`import React, { useState } from 'react'`) over `React.useState`.\n\n## Unit Testing\n\n- Vitest + React Testing Library; tests live in `__tests__/` next to the source file.\n- Wrap components in `TestPageContainer` from `@utils/test-utils/TestPageContainer` — it\n provides the necessary app providers (theme, routing, contexts).\n- Use `MockedProvider` from `@apollo/client/testing` for components that make GraphQL\n queries.\n- Generate icon stubs once per clone before running tests:\n `node scripts/generate-lazy-icon-stubs.js`.\n- Follow the root guide's testing principles: test behavior, not implementation; skip\n tests that only exercise the framework or restate the linter.\n\n\n# docs-website — Agent Development Guide\n\nThe docs site is a **Docusaurus 2** app. Docusaurus reads from `genDocs/` (gitignored, assembled at\nbuild time). Custom React pages live in `src/pages/`, blog/learn articles in `src/learn/`.\n\n## Build pipeline\n\nThe full build runs as a Gradle task chain. You rarely need to invoke individual steps, but\nunderstanding the pipeline helps when debugging.\n\n```\n./gradlew :docs-website:yarnBuild\n └─ yarnGenerate\n ├─ yarnInstall\n ├─ generateGraphQLSchema → graphql/combined.graphql\n ├─ generateJsonSchema → static/schemas/datahub_ingestion_schema.json\n ├─ :metadata-ingestion:modelDocGen → docs/generated/metamodel/\n └─ :metadata-ingestion:docGen → docs/generated/ingestion/\n```\n\n`yarnGenerate` runs `yarn run generate`, which:\n\n1. Generates GraphQL API docs via the `docusaurus-graphql-plugin`\n2. Generates Python SDK docs via Sphinx (`sphinx/` → Markdown conversion)\n3. Runs `generateDocsDir.ts` to assemble all markdown into `genDocs/`\n\n## `generateDocsDir.ts` — the assembly script\n\nThis TypeScript script is the central piece that discovers all markdown in the repo and writes\nprocessed output to `genDocs/`. Key behaviors:\n\n- **Discovery**: Uses `git ls-files` to find all `.md` files, then filters out excluded paths\n (`.github/`, `node_modules/`, `contrib/`, archived docs, test dirs, etc.)\n- **Frontmatter injection**: Adds `title`, `slug`, `description`, `custom_edit_url` to each file\n- **Link rewriting**: Rewrites relative links and image paths for the hosted site\n- **Inline directives**: `{{ inline /path/to/file.py }}` embeds file contents;\n `{{ command-output cmd }}` embeds command output\n- **Sidebar validation**: Warns if a doc file is not referenced in `sidebars.js` (some dirs are\n exempt: `docs/generated/metamodel`, `docs/generated/ingestion`, `docs/actions/`, etc.)\n\n## `sidebars.js` conventions\n\n- Doc IDs are file paths relative to `genDocs/`, minus the `.md` extension\n- Every hand-authored doc in `docs/` must appear in `sidebars.js` or the build warns\n- Auto-generated directories (`docs/generated/metamodel/`, `docs/generated/ingestion/`) are exempt\n\n## Key files\n\n| File | Purpose |\n| ----------------------- | ------------------------------------------------- |\n| `docusaurus.config.js` | Site config — plugins, navbar, footer, docs path |\n| `sidebars.js` | Sidebar navigation tree |\n| `generateDocsDir.ts` | Markdown discovery, transformation, and assembly |\n| `build.gradle` | Gradle tasks (`yarnGenerate`, `yarnBuild`, etc.) |\n| `sphinx/` | Python SDK doc generation (Sphinx → Markdown) |\n| `graphql/` | GraphQL schema combination + doc generation |\n| `genJsonSchema/` | Combines connector config schemas into one file |\n| `src/pages/` | Custom React pages (not markdown docs) |\n| `src/learn/` | Blog / learning articles (served at `/learn`) |\n| `static/` | Images, logos, static assets |\n\n## Generated directories (gitignored — never edit)\n\n- `genDocs/` — final assembled docs consumed by Docusaurus\n- `graphql/combined.graphql` — concatenated GraphQL schema\n- `static/schemas/datahub_ingestion_schema.json` — combined ingestion config schema\n\n## Logo convention\n\nPlatform logos: `static/img/logos/platforms/{platform_id}.svg`\nDataHub internal connectors: use `img/datahub-logo-color-mark.svg` (shared logo).\n\n\n# DataHub Metadata Ingestion Development Guide\n\n## Build and Test Commands\n\n**Using Gradle (slow but reliable):**\n\n```bash\n# Development setup from metadata-ingestion/\n../gradlew :metadata-ingestion:installDev # Setup Python environment\nsource venv/bin/activate # Activate virtual environment\n\n# Linting and formatting\n../gradlew :metadata-ingestion:lint # Run ruff + mypy\n../gradlew :metadata-ingestion:lintFix # Auto-fix linting issues\n\n# Testing\n../gradlew :metadata-ingestion:testQuick # Fast unit tests\n../gradlew :metadata-ingestion:testFull # All tests\n../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_file.py # Single test\n```\n\n**Direct Python commands (when venv is activated):**\n\n```bash\n# Testing\npytest -vv # Run all tests\npytest -m 'not integration' # Unit tests only\npytest -m 'integration' # Integration tests\npytest tests/path/to/file.py # Single test file\npytest tests/path/to/file.py::TestClass # Single test class\npytest tests/path/to/file.py::TestClass::test_method # Single test\n```\n\n## Environment Variables\n\n**Build configuration:**\n\n- `DATAHUB_VENV_USE_COPIES`: Set to `true` to use `--copies` flag when creating Python virtual environments. This copies the Python binary instead of creating a symlink. Useful for Nix environments, immutable filesystems, Windows, or container environments where symlinks don't work correctly. Increases disk usage and setup time, so only enable if needed.\n\n ```bash\n export DATAHUB_VENV_USE_COPIES=true\n ../gradlew :metadata-ingestion:installDev\n ```\n\n## Directory Structure\n\n- `src/datahub/`: Source code for the DataHub CLI and ingestion framework\n- `tests/`: All tests (NOT in `src/` directory)\n- `tests/unit/`: Unit tests\n- `tests/integration/`: Integration tests\n- `scripts/`: Build and development scripts\n- `examples/`: Example ingestion configurations\n- `developing.md`: Detailed development environment information\n\n## Code Style Guidelines\n\n- **Formatting**: Uses ruff, 88 character line length\n- **Imports**: Sorted with ruff.lint.isort, no relative imports, Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.\n- **Types**: Always use type annotations, prefer Protocol for interfaces\n - Avoid `Any` type - use specific types (`Dict[str, int]`, `TypedDict`, or typevars)\n - Use `isinstance` checks instead of `hasattr`\n - Prefer `assert isinstance(...)` over `cast`\n- **Data Structures**: Use dataclasses/pydantic for internal data representation\n - Return dataclasses instead of tuples from methods\n - Centralize utility functions to avoid code duplication\n- **Naming**: Descriptive names, match source system terminology in configs\n- **Error Handling**: Validators throw only ValueError/TypeError/AssertionError\n - Add robust error handling with layers of protection for known failure points\n- **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`\n- **Documentation**: All configs need descriptions\n- **Dependencies**: Avoid version pinning, use ranges with comments\n- **Architecture**: Avoid tall inheritance hierarchies, prefer mixins\n\n## Testing Conventions\n\n- **Location**: Tests go in `tests/` directory alongside `src/`, NOT in `src/`\n- **Structure**: Test files should mirror the source directory structure\n- **Framework**: Use pytest, not unittest\n- **Assertions**: Use `assert` statements, not `self.assertEqual()` or `self.assertIsNone()`\n - Boolean: `assert func()` or `assert not func()`, not `assert func() is True/False`\n - None: `assert result is None` (correct), not `assert result == None`\n - Keep tests concise, avoid verbose repetitive patterns\n- **Classes**: Use regular classes, not `unittest.TestCase`\n- **Imports**: Import `pytest` in test files\n- **Naming**: Test files should be named `test_*.py`\n- **Categories**:\n - Unit tests: `tests/unit/` - fast, no external dependencies\n - Integration tests: `tests/integration/` - may use Docker/external services\n\n## Configuration Guidelines (Pydantic)\n\n- **Naming**: Match terminology of the source system (e.g., `account_id` for Snowflake, not `host_port`)\n- **Descriptions**: All configs must have descriptions\n- **Patterns**: Use AllowDenyPatterns for filtering, named `*_pattern`\n- **Defaults**: Set reasonable defaults, avoid config-driven filtering that should be automatic\n- **Validation**: Single pydantic validator per validation concern\n- **Security**: Use `SecretStr` for passwords, auth tokens, etc.\n- **Deprecation**: Use `pydantic_removed_field` helper for field deprecations\n\n## Connector Conventions\n\n- **Structure**: Split connector code by duty: `constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`, `mapper.py`, and `usage.py` as needed. Match the quality and structure of existing connectors like Power BI, Airbyte, Redshift, BigID, and Grafana.\n- **Constants**: Avoid double-quoted string literals in connector code. Hoist magic strings into module-level constants, and keep regex patterns pre-compiled in the constants file.\n- **Comments**: No top-of-file docstrings in connector modules. Keep comments and docstrings only where they add real context.\n- **Imports**: Do not use `TYPE_CHECKING` in connector code. Lazy imports are fine only for opt-in features the connector does not always depend on.\n- **Syntax**: Do not use the walrus operator in connector code.\n- **Types**: Prefer `Dict` and `List` from `typing` over builtin `dict` and `list` in connector type annotations.\n- **Reporting**: Prefer `self.report.warning(...)` and report counters over bare `logger` warnings for skips, inaccessible objects, and other operator-visible edge cases.\n- **SQL lineage**: Use `SqlParsingAggregator` via `create_lineage_from_sql_statements` with a platform map instead of setting sqlglot dialects per connector. Mirror existing connectors for known-URN mapping, platform instance / env handling, casing, multi-part names, and temp-table handling.\n- **Column lineage**: Resolve upstream and downstream schemas from the DataHub graph when available, load known URNs from the platform / platform instance / env mapping, and match columns case-insensitively.\n- **Progress**: Surface ingestion progress and use explicit ingestion stages, following connectors like Dremio and Snowflake.\n- **New connectors**: A connector usually needs more than Python code: source logo, integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration in `setup.py`, `../gradlew :metadata-ingestion:updateLockFile`, and shared subtypes instead of connector-local subtype definitions.\n- **XML**: Avoid Python's stdlib `xml` parser. Use a safe XML library, following the HANA-related code path.\n- **Validation workflow**: Before committing a new connector, run ingestion locally in debug mode to a local file and inspect the full logs. When testing against customer environments, keep secrets only in temporary paths such as `/tmp/*.env`.\n- **PR scope**: Keep each connector in its own PR. Split shared framework changes such as sqlglot helpers into a separate PR, and keep the PR title and description focused on that connector only.\n\n## Dependencies and Entry Points\n\n`setup.py` is the **source of truth** for all dependencies, extras, and entry points. `pyproject.toml`, `uv.lock`, and `constraints.txt` are **generated from it** and must never be edited manually.\n\n**After any change to `setup.py`** (new dependency, new entry point, version bump), run:\n\n```bash\n../gradlew :metadata-ingestion:updateLockFile\n```\n\nThis regenerates the full chain: `setup.py` → `pyproject.toml` → `uv.lock` → `constraints.txt`.\n\nCI runs `checkLockFile` which calls `scripts/verify_pyproject_equivalence.py` to verify these files are in sync. If you edit `setup.py` without running `updateLockFile`, CI will fail.\n\n**Adding a new ingestion source entry point:**\n\n1. Add the entry to `setup.py` under `entry_points[\"datahub.ingestion.source.plugins\"]`\n2. Run `../gradlew :metadata-ingestion:updateLockFile`\n3. Verify with `../gradlew :metadata-ingestion:checkLockFile`\n\n## Key Files\n\n- `src/datahub/emitter/mcp_builder.py`: Examples of defining various aspect types\n- `setup.py`: Source of truth for dependencies, extras, and entry points\n- `pyproject.toml`: Generated from `setup.py` — do not edit manually\n- `setup.cfg`: Code style configuration\n- `scripts/generate_pyproject_deps.py`: Generates `pyproject.toml` from `setup.py`\n- `scripts/verify_pyproject_equivalence.py`: Validates `setup.py` and `pyproject.toml` are in sync\n- `.github/workflows/metadata-ingestion.yml`: CI workflow configuration\n\n## Connector Documentation\n\nConnector documentation follows a structured format with specific file types and heading conventions.\n\n**Authoring guidelines**: See `docs/sources/AGENTS.md` in this directory for:\n\n- File structure (`_pre.md`, `_post.md`, `_recipe.yml`)\n- Heading level rules (H3 baseline, H2 reserved for module headings)\n- Section ordering (Overview → Concept Mapping → Prerequisites)\n- Content placement guidelines\n- Style conventions and formatting requirements\n\n**Quick reference:**\n\n```bash\n# Regenerate documentation after changes\n../gradlew :metadata-ingestion:docGen\n\n# Preview locally\n../gradlew :docs-website:yarnStart # Opens localhost:3001\n\n# Format markdown before committing\n../gradlew :datahub-web-react:mdPrettierWrite\n```\n\n\n# Smoke Test Guidelines\n\nCanonical authoring guide for pytest smoke tests against a running DataHub\ninstance. How to run the suite: `[README.md](README.md)`.\n\n## Where tests and helpers live\n\n| Path | What goes here |\n| ---------------------- | ----------------------------------------------------------------------------- |\n| `tests/e2e/` | Live GMS pytest (needs quickstart). Default `pytest` / `smoke.sh` collection. |\n| `tests/unit/` | CPU pytest (no stack). `./gradlew :smoke-test:stackFreePytest`. |\n| `tests/zdu/` | ZDU live E2E pytest (`test_zdu_upgrade.py`) and docs. Not in docker pytest. |\n| `tests/zdu/framework/` | ZDU upgrade framework. Daily workflow: `python -m tests.zdu.framework`. |\n| `tests/oauth/` | OAuth IdP smoke. `.github/workflows/oauth-smoke.yml`. Not in docker pytest. |\n| `utilities/` | Shared helpers, not tests. |\n\nDo not put `test_*.py` next to helpers in `utilities/`. GMS fixtures live in `tests/e2e/conftest.py` only.\n\n## When to add smoke tests\n\nThese are E2E API tests on a **live GMS** (slow, shared stack, xdist). Prefer a\nunit test next to the code, an ingestion test in `metadata-ingestion/tests/`, or a backend unit/integration test **before** adding a smoke test.\n\n**Add one** when the behavior needs GMS: write → search, a public API flow with\nno existing coverage, a production regression to keep, or live-stack authz.\n\n**Do not add** when:\n\n- The logic is a validator, parser, or in-process service.\n- The flow is already covered — extend that module; do not add `test_*_v2.py`.\n- You are stacking positive/negative variants of the same call. One happy path\n is enough. A negative case belongs here only if it needs GMS (e.g. policy\n deny), not invalid input or combinatorial 4xx/case/whitespace.\n\nPut new tests in `tests/e2e//`, not `test_e2e.py`. Connector tests are placed in `metadata-ingestion/tests/`. Generic placeholder names only\n(`my_db.my_schema.events`) — no customer identifiers or ticket IDs.\n\n## Rules\n\n- **Idempotent and order-independent.** Do not assume an empty GMS or another\n test's cleanup. No module-level mutable globals.\n- **Run-unique names** for every entity you create, mutate, or delete (see\n Isolation). Shared hardcoded URNs flake under xdist.\n- **Cleanup even on failure:** fixture `yield`, or `try/finally` inside a test\n (wrap delete in `try/except` so cleanup cannot hide the assertion).\n- **Do not mutate** the admin user, default All Users policies, or other shared\n platform state unless the module is marked `global_policy_mutator`.\n- `logger.info()`, not `print()`.\n- **No** `time.sleep()` for GMS/search/Kafka. Read-after-write:\n `@with_test_retry()`. Bulk ingest/cleanup: `wait_for_writes_to_sync()`\n (`mcp_only` / `mae_only` when only one store matters). Known `trace_id`: Trace\n API. `TestSessionWrapper` already waits on POST/PUT — do not extra-sync unless\n you are asserting search/index.\n- **Auth:** `auth_session` / `graph_client`. Extra users:\n `make_step_actor_user()`. Config: `utilities/env_vars.py`, not `os.getenv` or\n `localhost:8080`. GraphQL: `execute_graphql()` (it already checks `data` /\n `errors`) — assert the fields you care about. Ingest:\n `ingest_file_via_rest()`.\n- **Reuse helpers.** Do not copy-paste `addTag` or roll a local `uuid` when \n `unique_suffix()` exists. Tags/terms/descriptions: \n `utilities/metadata_operations.py`.\n- Extra users: `make_step_actor_user()` in `utilities/multi_user.py`, \n not a one-off signup.\n- `execute_graphql()` already asserts a non-empty body, `data` is not `None`,\n and no `errors` key, only assert the data that the test requires\n\n## Markers\n\n```python\nfrom utilities.domains import Domain\n\npytestmark = pytest.mark.domain(Domain.CATALOG)\n# pytestmark = pytest.mark.domain(Domain.CATALOG, Domain.INGESTION)\n```\n\n| Marker | When |\n| ----------------------- | ---------------------------------------------------------------------------------------------------------- |\n| `domain(...)` | **Required.** `platform`, `observe`, `ingestion`, `ai`, `catalog`. |\n| `p0` | Must run on every PR. |\n| `global_policy_mutator` | Disables default policies / mutates shared platform policy. CI runs these serially after parallel modules. |\n\nKeep `@pytest.mark.dependency()` chains short (ideally ≤3).\n\n## Isolation\n\nModules run in parallel against one GMS. Prefer these over a local `uuid` /\nstring replace:\n\n| Helper | Use |\n| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| `unique_suffix()` | Any entity key (user, domain, PAT, …). |\n| `unique_dataset_urn(name)` | Dataset created in code (GraphQL / SDK). |\n| `materialize_with_unique_name(src, name, dest_dir)` | Fixture rewrite. `name` must appear **only** in URN keys, not descriptions. Returns `(dest_file, unique_name)`. |\n| `materialize_unique_dataset(...)` | Dataset fixture → `(dest_file, dataset_urn)`. |\n| `_ingest_cleanup_unique_dataset_impl(...)` | Rewrite + ingest + yield URN + delete. No pre-delete (URN is new). Default for a shared dataset name in `data.json`. |\n| `_ingest_cleanup_data_impl(...)` | Pre-delete → ingest → cleanup. Only when keys are already unique to this module. |\n\n```python\nfrom conftest import _ingest_cleanup_unique_dataset_impl\n\n@pytest.fixture(scope=\"module\", autouse=True)\ndef dataset_urn(auth_session, graph_client, tmp_path_factory):\n yield from _ingest_cleanup_unique_dataset_impl(\n auth_session, graph_client,\n \"tests/e2e/tags_and_terms/data.json\", \"tags_and_terms\",\n \"test-tags-terms-sample-kafka\",\n tmp_path_factory.mktemp(\"tags_and_terms\"),\n )\n```\n\nTests take `dataset_urn` as a fixture argument. Multi-entity fixtures: rewrite\neach key with `materialize_with_unique_name` — see\n`tests/e2e/containers/containers_test.py`.\n\nMid-test creates:\n\n```python\ndataset_urn = unique_dataset_urn(\"my-feature\")\ntry:\n graph_client.emit(...)\n wait_for_writes_to_sync()\nfinally:\n try:\n delete_urn(graph_client, dataset_urn)\n except Exception:\n logger.warning(\"cleanup failed for %s\", dataset_urn, exc_info=True)\n```\n\n## Other helpers\n\n- `tests/utils.py`: `execute_graphql`, `ingest_file_via_rest`,\n `delete_urn` / `delete_urns` / `delete_urns_from_file`, `with_test_retry`,\n `wait_for_writes_to_sync`.\n- `tests/utilities/concurrent_test_runner.py`: thread-safe\n `run_concurrent_tests` / `run_concurrent_tests_with_args`.\n- `tests/utilities/concurrent_openapi.py`: `run_tests(auth_session, fixture_globs=...)`.\n JSON fixtures: `{request, response}` with DeepDiff `exclude_regex_paths`. Do\n not add OpenAPI fixtures that only re-check a GraphQL path already covered.\n\n\n# References Skill\n\nPath: `.agent-skills/test-review/references/smoke-test-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.agent-skills/test-review/references/smoke-test-patterns.md)\n\n\n# DataHub Test Review\n\nYou are an expert DataHub test reviewer. Your role is to evaluate pytest smoke\ntests against established testing standards, identify issues, and provide\nactionable feedback.\n\nAuthoring how-to: `smoke-test/AGENTS.md`. Review rubric:\n`standards/smoke.md`.\n\n---\n\n## Multi-Agent Compatibility\n\nThis skill is designed to work across multiple coding agents (Claude Code, Cursor, Codex, Copilot, Gemini CLI, Windsurf, and others).\n\n**What works everywhere:**\n\n- All review checklists, standards references, and procedures in this document\n- Bash for running scripts (`detect-test-changes.sh`, `gh` CLI, `git diff`)\n- Reading files, searching code, and generating review reports\n\n**Claude Code-specific features** (other agents can safely ignore these):\n\n- The `/test-review` slash command (`.claude/commands/test-review.md`) loads this skill automatically\n- The `test-quality-analyzer` agent (`.claude/agents/test-quality-analyzer.md`) can be dispatched for parallel analysis -- **fallback instructions are provided inline** for agents that cannot dispatch sub-agents\n- `TaskCreate`/`TaskUpdate` for progress tracking -- if unavailable, simply proceed through the steps sequentially\n\n**Standards file paths:** All standards are in the `standards/` directory alongside this file. All paths below are relative to `.agent-skills/test-review/`.\n\n---\n\n## Quick Start\n\n**Full review?** -> Load standards, gather test files, then launch `test-quality-analyzer` agent (or perform checks directly)\n\n**PR review?** -> Detect changed test files, then analyze only those files\n\n---\n\n## Scope\n\n### In Scope\n\n- `smoke-test/` -- Python pytest smoke tests (API-level tests against a running DataHub instance)\n- `smoke-test/tests/` -- shared test utilities, fixtures, and helpers\n\n### Out of Scope\n\n- `smoke-test/tests/cypress/` -- Cypress UI tests\n- `e2e-test/ui/playwright/` -- Playwright UI tests\n- `metadata-ingestion/tests/integration/` -- ingestion connector tests (covered by `datahub-connector-pr-review`)\n- `metadata-ingestion/tests/unit/` -- unit tests\n- `metadata-ingestion/src/datahub/testing/` -- ingestion testing utilities\n\nThe `detect-test-changes.sh` script lists in-scope Python smoke-test files.\n\n---\n\n## Review Modes\n\n| Mode | Use Case | Scope | Template |\n| ---------------------- | ------------------------------ | ------------------ | ---------------------------- |\n| **Full Review** | Audit all tests in a directory | All in-scope tests | `test-review-report.md` |\n| **Incremental Review** | PR with test changes | Changed files only | `incremental-test-report.md` |\n\n---\n\n## Startup: Load Standards\n\n**On activation, IMMEDIATELY load testing standards** from the `standards/` directory.\n\nRead `.agent-skills/test-review/standards/smoke.md` -- this contains all testing rules.\nAlso skim `smoke-test/AGENTS.md` if isolation or fixture guidance is in dispute.\n\nAfter loading, briefly confirm: \"Loaded test review standards. Ready to review.\"\n\n---\n\n## Progress Tracking with Tasks\n\n**After loading standards**, create a task checklist using TaskCreate:\n\n```\n1. Load testing standards\n2. Detect in-scope smoke test files\n3. Filter out connector-specific and Cypress tests\n4. Analyze smoke tests\n5. Generate review report\n```\n\nIf TaskCreate is not available, proceed through the steps sequentially.\n\n---\n\n## Mode 1: Full Review\n\n### Step 1: Gather Test Files\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\nOr list Python tests under `smoke-test/` excluding `tests/cypress/`.\n\n### Step 2: Load Standards into Context\n\nRead `.agent-skills/test-review/standards/smoke.md` completely.\n\n### Step 3: Launch Test Quality Analyzer\n\n**Claude Code (with Agent tool):**\n\n```\nAgent tool:\n subagent_type: \"test-quality-analyzer\"\n prompt: \"\"\"Analyze the following smoke test files for quality and standards compliance.\n\n\n[Content from .agent-skills/test-review/standards/smoke.md]\n\n\n\n[List of smoke-test Python file paths]\n\n\nFor each file, check all applicable rules from the standards document.\nReport findings with severity (BLOCKER/WARNING/SUGGESTION) and file:line references.\n\"\"\"\n```\n\n**Other agents (sequential fallback):**\n\nIf you cannot dispatch sub-agents, perform the analysis yourself:\n\n1. Read each test file completely\n2. Check against the smoke-test standards (isolation, fixtures, auth, retry, GraphQL/REST, markers, env vars, cleanup, placement, logging)\n3. Scan for anti-patterns listed at the end of the standards file\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/test-review-report.md` template. Fill in:\n\n- Summary table with pass/fail per category\n- All findings organized by severity (BLOCKER > WARNING > SUGGESTION)\n- Checklist results\n- Quality scores (1-10 per dimension)\n- Verdict: APPROVED / NEEDS CHANGES / BLOCKED\n\n**Verdict logic:**\n\n- **APPROVED**: No blockers, no more than 3 warnings\n- **NEEDS CHANGES**: Has warnings or fixable blockers\n- **BLOCKED**: Has fundamental anti-pattern blockers (empty tests, missing cleanup, hardcoded credentials, shared hardcoded URNs)\n\n---\n\n## Mode 2: Incremental Review (CI Mode)\n\n### Step 1: Detect Changed Test Files\n\n**If PR number provided:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh ${PR_NUMBER}\n```\n\n**If running locally:**\n\n```bash\n.agent-skills/test-review/scripts/detect-test-changes.sh local\n```\n\n**If no script available:**\n\n```bash\ngh pr diff ${PR_NUMBER} --name-only | grep -E '^smoke-test/' | \\\n grep -E '\\.py$' | grep -v '/tests/cypress/'\n```\n\n### Step 2: Classify Files\n\nParse the output of `detect-test-changes.sh`:\n\n- Lines starting with `smoke:` are smoke test files\n\nIf no in-scope test changes detected (exit code 1), report: \"No in-scope test changes found in this PR.\"\n\n### Step 3: Analyze Changed Files\n\nApply the same analysis as Mode 1, Step 3, but only to changed files.\n\nFor incremental reviews, also check:\n\n- Do new tests follow the same patterns as existing tests in their module?\n- Do modifications preserve existing test behavior?\n- Are unique-name helpers used instead of new shared hardcoded URNs?\n\n### Step 4: Generate Report\n\nUse the `.agent-skills/test-review/templates/incremental-test-report.md` template.\n\n---\n\n## CI Invocation\n\nFor non-interactive CI usage via `claude -p`:\n\n```bash\nclaude -p \"Review test changes in PR #${PR_NUMBER} using the test-review skill. \\\n Output the review report in markdown format with a verdict line.\"\n```\n\nThe skill produces deterministic output:\n\n- Structured markdown report\n- Clear verdict: `APPROVED`, `NEEDS CHANGES`, or `BLOCKED`\n- All findings cite file:line references\n\n---\n\n## Standards Reference\n\nAll standards are documented in `standards/smoke.md`. Authoring\nguide: `smoke-test/AGENTS.md`.\n\n1. **Isolation and unique names** -- `unique_suffix()` / unique ingest, no shared hardcoded URNs\n2. **Fixtures and data lifecycle** -- unique-dataset ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique\n3. **Authentication** -- `auth_session` fixture, `make_step_actor_user()`, never inline\n4. **Retry patterns** -- Trace API, `wait_for_writes_to_sync()`, `@with_test_retry()`, no bare `time.sleep()`\n5. **GraphQL / REST** -- `execute_graphql()`, `restli_default_headers`, `ingest_file_via_rest()`\n6. **Markers** -- required `domain(...)`; `p0` only for regressions that must run on every PR; `read_only` only when the test never mutates; `global_policy_mutator` only when mutating shared platform policy\n7. **Environment variables** -- `env_vars.py` registry, no hardcoded URLs\n8. **Guaranteed cleanup** -- fixture `yield` or `try/finally`\n9. **Multi-environment config** -- URLs via env vars, `USE_STATIC_SLEEP` fallback\n10. **Concurrent testing** -- thread-safe `run_concurrent_tests()`\n11. **Placement, logging, quality** -- `tests//`, `logger.info()`, no customer identifiers\n\n### Anti-Patterns (Automatic Blockers)\n\n- Empty/trivial tests\n- Missing cleanup\n- Hardcoded URLs/ports\n- Inline authentication\n- Bare `time.sleep()` for consistency\n- Shared hardcoded URNs / global mutable state\n- Mutating shared platform policy without `global_policy_mutator`\n\n---\n\n## Severity Levels\n\n| Level | Icon | Meaning | Action Required |\n| ---------- | ---- | ------------------ | ---------------------------- |\n| BLOCKER | ---- | Standard violated | Must fix before merge |\n| WARNING | ---- | Should be improved | Should fix before merge |\n| SUGGESTION | ---- | Nice to have | Optional, defer to follow-up |\n\n---\n\n## Templates\n\n- `templates/test-review-report.md` -- Full review report\n- `templates/incremental-test-report.md` -- PR incremental report\n\n---\n\n## Scripts\n\n- `scripts/detect-test-changes.sh` -- Detect in-scope changed smoke-test files\n\n---\n\n## References\n\n- `smoke-test/AGENTS.md` -- Canonical authoring guide\n- `references/smoke-test-patterns.md` -- Additional code examples\n\n---\n\n## Remember\n\n1. **Do not invent rules.** Standards come from `smoke-test/AGENTS.md` and the cited helpers.\n2. **Cypress, Playwright, and connector tests are out of scope.**\n3. **Be conservative with BLOCKERs.** Only flag when a standard is clearly violated. Older files that still use `_ingest_cleanup_data_impl` with unique keys are acceptable.\n4. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n5. **Acknowledge good patterns.** Note when tests follow standards well.\n6. **CI output must be deterministic.** Same input -> same verdict.\n\n\n# Cursor Rule: datahub-dev\n\nPath: `.cursor/rules/datahub-dev.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/datahub-dev.mdc)\n\n\n# Cursor Rule: test-review\n\nPath: `.cursor/rules/test-review.mdc`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.cursor/rules/test-review.mdc)\n\n\n# CLAUDE.md\n\n@AGENTS.md\n\n\n# datahub-demo-dataset-governance-validator Documentation\n\nPath: `contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/datahub-demo-dataset-governance-validator/README.md)\n\n\n# metadata-model-extensions Documentation\n\nPath: `contrib/metadata-model-extensions/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/contrib/metadata-model-extensions/README.md)\n\n\n---\nname: test-quality-analyzer\ndescription: |\n Analyze DataHub pytest smoke test files for quality, standards compliance, and anti-patterns. Reports findings with severity levels and file:line references.\n\n \n Context: A PR adds new smoke tests for the incidents feature.\n user: \"Analyze the test quality of smoke-test/tests/incidents/incidents_test.py\"\n assistant: \"I'll use the test-quality-analyzer agent to check fixture usage, data lifecycle, assertions, and anti-patterns.\"\n \n Test quality analysis is a read-only inspection task that triggers this agent.\n \n \nmodel: sonnet\ncolor: blue\ntools:\n - Bash\n - Read\n - Grep\n - Glob\n---\n\n# DataHub Test Quality Analyzer Agent\n\nYou are a test quality analysis agent that inspects DataHub pytest smoke tests for compliance with established testing standards. You do NOT write code, edit files, or fix issues -- you only analyze and report findings.\n\nAuthoring guide: `smoke-test/AGENTS.md`. Review rubric: `.agent-skills/test-review/standards/smoke.md`.\n\n## Core Rules\n\n1. **Read-only analysis.** You have no Write or Edit tools. Inspect test files and report findings.\n2. **Cite file:line for every finding.** Every issue must reference the exact file path and line number.\n3. **Use severity levels consistently.** BLOCKER = must fix, WARNING = should fix, SUGGESTION = nice to have.\n4. **Smoke tests only.** Cypress, Playwright, and `metadata-ingestion/tests/` are out of scope.\n5. **Do not BLOCK unique-dataset fixtures for missing pre-delete.**\n\n## Input\n\nYou receive:\n\n- A list of test files to analyze\n- The testing standards (embedded in the prompt as `` tags)\n\n## Workflow\n\n### Phase 1: File Classification\n\nFor each test file, determine:\n\n- **Smoke test** (Python pytest): Located under `smoke-test/` excluding `tests/cypress/`\n- **Out of scope**: `smoke-test/tests/cypress/`, `e2e-test/ui/playwright/`, `metadata-ingestion/tests/`\n\nSkip out-of-scope files.\n\n### Phase 2: Smoke Test Analysis\n\nFor each smoke test file, check:\n\n1. **Isolation** -- Run-unique names (`unique_suffix`, `unique_dataset_urn`, `materialize_*`, `_ingest_cleanup_unique_dataset_impl`)? No shared hardcoded URNs?\n2. **Data Lifecycle** -- Fixture `yield` teardown? Unique ingest by default; `_ingest_cleanup_data_impl` only when keys are already unique?\n3. **Authentication** -- `auth_session` fixture, not inline credentials? Extra users via `make_step_actor_user()`?\n4. **Retry Patterns** -- `@with_test_retry()` or `wait_for_writes_to_sync()` instead of bare `time.sleep()`?\n5. **GraphQL / REST** -- `execute_graphql()` with specific field assertions? `ingest_file_via_rest()` / `restli_default_headers`?\n6. **Environment Variables** -- `env_vars.py` instead of `os.getenv`? No hardcoded URLs/ports?\n7. **Markers** -- `domain(...)` declared? `global_policy_mutator` if mutating shared policy? `read_only` tests truly read-only?\n8. **Cleanup** -- Fixture `yield` or `try/finally` for mid-test entities?\n9. **Test Isolation** -- No global mutable state? No cross-test module-level variables?\n10. **Logging** -- `logger.info()`, not `print()`?\n11. **Placement** -- New tests in `tests//`, not `test_e2e.py`?\n12. **Confidentiality** -- No customer identifiers or ticket IDs?\n\n### Phase 3: Anti-Pattern Detection\n\nScan for these automatic BLOCKER triggers:\n\n- Empty/trivial tests (`assert True`, only testing defaults)\n- Bare `time.sleep()` without retry wrapper\n- Hardcoded URLs or ports (should use env_vars registry)\n- Inline credential creation (should use auth_session fixture)\n- Shared hardcoded URNs across modules\n- Cross-test dependencies via global mutable state\n- Missing cleanup for created entities\n- Entities created mid-test without `try/finally`\n- Mutating admin / default policies without `global_policy_mutator`\n- Commented-out test code or breakpoints\n\n## Output Format\n\n```markdown\n# Test Quality Analysis\n\n**Files Analyzed:** {{COUNT}}\n**Smoke Tests:** {{SMOKE_COUNT}}\n\n---\n\n## Findings Summary\n\n| Severity | Count |\n| ---------- | ----- |\n| BLOCKER | {{N}} |\n| WARNING | {{N}} |\n| SUGGESTION | {{N}} |\n\n---\n\n## BLOCKER Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Standard:** {{WHICH_STANDARD_VIOLATED}}\n- **Issue:** {{DESCRIPTION}}\n- **Evidence:** {{CODE_SNIPPET_OR_EXPLANATION}}\n\n---\n\n## WARNING Issues\n\n### 1. {{TITLE}}\n\n- **File:** `{{FILE_PATH}}:{{LINE}}`\n- **Issue:** {{DESCRIPTION}}\n- **Recommendation:** {{FIX}}\n\n---\n\n## SUGGESTION Issues\n\n- `{{FILE}}:{{LINE}}` -- {{DESCRIPTION}}\n\n---\n\n## Positive Observations\n\n- {{GOOD_PATTERN_FOUND}}\n\n---\n\n## Verdict\n\n**{{VERDICT}}** (APPROVED / NEEDS CHANGES / BLOCKED)\n```\n\n## Important Guidelines\n\n- **Be conservative with BLOCKERs.** Only flag as BLOCKER when a standard is clearly violated.\n- **Consider context.** Older test files may still use `_ingest_cleanup_data_impl` with unique keys -- flag new shared hardcoded URNs, not every legacy fixture.\n- **Acknowledge good patterns.** Note when tests follow standards well.\n- **Check the whole file.** Don't stop after finding the first issue.\n\n\n# Command: /test-review\n\nPath: `.claude/commands/test-review.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/datahub-project/datahub/HEAD/.claude/commands/test-review.md)\n"}