{"owner":"datahub-project","repo":"datahub","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","AGENTS.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","AGENTS.md":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests and linting\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotlessJavaCheck` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is served from `docs-website/genDocs/` (gitignored). It is assembled at build time\nfrom multiple hand-authored sources plus several generation steps:\n\n1. **Gradle generation tasks** produce `docs/generated/` (connector docs, entity reference, schemas)\n2. **`generateDocsDir.ts`** discovers all markdown in the repo, applies transformations (frontmatter,\n link rewriting, `{{ inline }}` directives), and writes the result to `genDocs/`\n3. **Docusaurus** serves from `genDocs/`, additionally generating GraphQL API docs and Python SDK docs\n\nSee `docs-website/AGENTS.md` for full pipeline details.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n\n### Frontend Theming (Colors)\n\n**Always use semantic color tokens** from `datahub-web-react/src/conf/theme/colorThemes/types.ts`. Never use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy `colors.gray[X]` imports.\n\n**In styled-components** (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\n**In React component bodies:**\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\n**For alchemy components** (``, ``, etc.) — do not pass `color`/`colorLevel` props. Let them inherit from themed parent styled-components.\n\n**Do not import from:**\n\n- `datahub-web-react/src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md).\n\n### Seeding\n\n`test.use({ featureName: 'my-feature' })` at the `describe` level auto-loads\n`tests/my-feature/fixtures/data.json` via `seeding.fixture.ts` — once per worker per\nfeature per run. Do **not** set `featureName` for suites that create their own data\nvia `apiMock` or direct API calls.\n\n## Frontend CI Checklist\n\nThis checklist is for **commit- or PR-ready** frontend work — i.e. when you're about to\ncommit, push, or hand off changes that are going into a PR. It is **not** required for\nevery intermediate edit: work that is part of a larger task, a work-in-progress branch,\nor scratch experimentation that won't be committed yet can skip it. Run the relevant\ncommands when the change is ready to ship:\n\n```bash\n# Full lint (eslint + prettier src + type-check) for datahub-web-react\n./gradlew :datahub-web-react:yarnLint\n\n# Targeted lint-fix on a single file\n./gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (requires icon stubs — run once per clone)\nnode datahub-web-react/scripts/generate-lazy-icon-stubs.js\ncd datahub-web-react && yarn test src/path/to/file.test.ts --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in\nunrelated files. Focus on errors in files **you touched** — in particular, optional\nprop calls (`prop?.(arg)`) and import aliases.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n\n## Learned User Preferences\n\n- In `metadata-ingestion` connector code, avoid double-quoted string literals: hoist magic strings into module-level constants, and keep all regex in the constants file pre-compiled.\n- Use Pydantic models for structured/internal data; never pass data around as tuples (hard to track).\n- Split connector files by duty (`constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`/`mapper.py`/`usage.py` as needed) and match the quality/patterns of existing connectors (Power BI, Airbyte, Redshift, BigID, Grafana).\n- No \"AI slop\": no top-of-file docstrings, and keep docstrings/comments only where strictly needed.\n- Never use `TYPE_CHECKING` in connector code since the connector controls its own deps (lazy imports are fine only for opt-in features), and don't use the walrus operator.\n- Prefer `self.report.warning(...)` and report counters over bare `logger` for skips and edge cases — the report also writes to the log and surfaces to operators (e.g. warn when `verify_ssl=False`, or when a referenced object is inaccessible).\n- For SQL lineage, use the central `SqlParsingAggregator` (`create_lineage_from_sql_statements`) with a platform map instead of setting sqlglot dialects per-connector; mirror existing connectors for cross-platform known-URN and platform_instance/env/casing mapping, two- vs three-part names, and temp-table handling.\n- For column-level lineage, don't leave edges coarse: resolve upstream/downstream schemas from the DataHub graph when available (as airbyte/bigid/matillion/informatica do), load known URNs from the platform/platform_instance/env mapping, and match columns case-insensitively. Best-effort is fine, but try everything.\n- In connector code, use explicit type annotations rather than `from typing import Any` (Unions are fine when a value genuinely has multiple types), and prefer `Dict`/`List` from `typing` over the builtin `dict`/`list`.\n- Connectors should surface progress during ingestion and use explicit ingestion stages (as in the dremio and snowflake connectors).\n- Before committing a new connector, run its ingestion locally in debug mode to a local file to capture full logs and catch bugs; when testing against a customer environment, push secrets only to a tmp path (e.g. `/tmp/*.env`).\n- When drafting prose or review comments on the user's behalf (e.g. Notion), write in his own direct, human voice — avoid AI tells like \"confirmed these are real gaps\".\n\n## Learned Workspace Facts\n\n- The user is a contributor to the public `datahub-project/datahub` repo and can create and push branches directly on `datahub-public-repo`.\n- A new ingestion connector needs more than Python code: a source logo plus an integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration (`setup.py`/`pyproject.toml`), a refreshed `uv.lock`, and subtypes added to the shared subtypes module rather than defined locally.\n- Avoid Python's stdlib `xml` parser due to a known vulnerability; use a safe XML library (as the HANA-related code does).\n- Keep each connector in its own PR and split shared/framework changes (e.g. sqlglot helpers) into a separate PR; a connector PR's title and description must reference only that connector, not any other connector worked on in the same session.\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","AGENTS.md":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests and linting\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotlessJavaCheck` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is served from `docs-website/genDocs/` (gitignored). It is assembled at build time\nfrom multiple hand-authored sources plus several generation steps:\n\n1. **Gradle generation tasks** produce `docs/generated/` (connector docs, entity reference, schemas)\n2. **`generateDocsDir.ts`** discovers all markdown in the repo, applies transformations (frontmatter,\n link rewriting, `{{ inline }}` directives), and writes the result to `genDocs/`\n3. **Docusaurus** serves from `genDocs/`, additionally generating GraphQL API docs and Python SDK docs\n\nSee `docs-website/AGENTS.md` for full pipeline details.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n\n### Frontend Theming (Colors)\n\n**Always use semantic color tokens** from `datahub-web-react/src/conf/theme/colorThemes/types.ts`. Never use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy `colors.gray[X]` imports.\n\n**In styled-components** (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\n**In React component bodies:**\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\n**For alchemy components** (``, ``, etc.) — do not pass `color`/`colorLevel` props. Let them inherit from themed parent styled-components.\n\n**Do not import from:**\n\n- `datahub-web-react/src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md).\n\n### Seeding\n\n`test.use({ featureName: 'my-feature' })` at the `describe` level auto-loads\n`tests/my-feature/fixtures/data.json` via `seeding.fixture.ts` — once per worker per\nfeature per run. Do **not** set `featureName` for suites that create their own data\nvia `apiMock` or direct API calls.\n\n## Frontend CI Checklist\n\nThis checklist is for **commit- or PR-ready** frontend work — i.e. when you're about to\ncommit, push, or hand off changes that are going into a PR. It is **not** required for\nevery intermediate edit: work that is part of a larger task, a work-in-progress branch,\nor scratch experimentation that won't be committed yet can skip it. Run the relevant\ncommands when the change is ready to ship:\n\n```bash\n# Full lint (eslint + prettier src + type-check) for datahub-web-react\n./gradlew :datahub-web-react:yarnLint\n\n# Targeted lint-fix on a single file\n./gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (requires icon stubs — run once per clone)\nnode datahub-web-react/scripts/generate-lazy-icon-stubs.js\ncd datahub-web-react && yarn test src/path/to/file.test.ts --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in\nunrelated files. Focus on errors in files **you touched** — in particular, optional\nprop calls (`prop?.(arg)`) and import aliases.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n\n## Learned User Preferences\n\n- In `metadata-ingestion` connector code, avoid double-quoted string literals: hoist magic strings into module-level constants, and keep all regex in the constants file pre-compiled.\n- Use Pydantic models for structured/internal data; never pass data around as tuples (hard to track).\n- Split connector files by duty (`constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`/`mapper.py`/`usage.py` as needed) and match the quality/patterns of existing connectors (Power BI, Airbyte, Redshift, BigID, Grafana).\n- No \"AI slop\": no top-of-file docstrings, and keep docstrings/comments only where strictly needed.\n- Never use `TYPE_CHECKING` in connector code since the connector controls its own deps (lazy imports are fine only for opt-in features), and don't use the walrus operator.\n- Prefer `self.report.warning(...)` and report counters over bare `logger` for skips and edge cases — the report also writes to the log and surfaces to operators (e.g. warn when `verify_ssl=False`, or when a referenced object is inaccessible).\n- For SQL lineage, use the central `SqlParsingAggregator` (`create_lineage_from_sql_statements`) with a platform map instead of setting sqlglot dialects per-connector; mirror existing connectors for cross-platform known-URN and platform_instance/env/casing mapping, two- vs three-part names, and temp-table handling.\n- For column-level lineage, don't leave edges coarse: resolve upstream/downstream schemas from the DataHub graph when available (as airbyte/bigid/matillion/informatica do), load known URNs from the platform/platform_instance/env mapping, and match columns case-insensitively. Best-effort is fine, but try everything.\n- In connector code, use explicit type annotations rather than `from typing import Any` (Unions are fine when a value genuinely has multiple types), and prefer `Dict`/`List` from `typing` over the builtin `dict`/`list`.\n- Connectors should surface progress during ingestion and use explicit ingestion stages (as in the dremio and snowflake connectors).\n- Before committing a new connector, run its ingestion locally in debug mode to a local file to capture full logs and catch bugs; when testing against a customer environment, push secrets only to a tmp path (e.g. `/tmp/*.env`).\n- When drafting prose or review comments on the user's behalf (e.g. Notion), write in his own direct, human voice — avoid AI tells like \"confirmed these are real gaps\".\n\n## Learned Workspace Facts\n\n- The user is a contributor to the public `datahub-project/datahub` repo and can create and push branches directly on `datahub-public-repo`.\n- A new ingestion connector needs more than Python code: a source logo plus an integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration (`setup.py`/`pyproject.toml`), a refreshed `uv.lock`, and subtypes added to the shared subtypes module rather than defined locally.\n- Avoid Python's stdlib `xml` parser due to a known vulnerability; use a safe XML library (as the HANA-related code does).\n- Keep each connector in its own PR and split shared/framework changes (e.g. sqlglot helpers) into a separate PR; a connector PR's title and description must reference only that connector, not any other connector worked on in the same session.\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\n@AGENTS.md\n","category":"root","tokens":6},{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# DataHub — Agent Development Guide\n\nThis is the canonical reference for working with the DataHub codebase. It applies to all coding\nagents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.\n\n## Code Navigation (LSP)\n\nPrefer LSP tools over Grep for code navigation tasks:\n\n- Use `goToDefinition` to find where something is defined\n- Use `findReferences` to find all call sites\n- Use `workspaceSymbol` to find symbols by name\n- Use diagnostics after any edit to catch type errors immediately\n\nSee [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.\n\n## Essential Commands\n\n**Build and test:**\n\n```bash\n./gradlew build # Build entire project\n./gradlew check # Run all tests and linting\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n\n# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.\n\n# Java code.\n./gradlew spotlessApply # Java code formatting\n\n# Python code.\n./gradlew :metadata-ingestion:testQuick # Fast Python unit tests\n./gradlew :metadata-ingestion:lint # Python linting (ruff, mypy)\n./gradlew :metadata-ingestion:lintFix # Python linting auto-fix (ruff only)\n\n# Markdown, GraphQL, YAML formatting\n./gradlew :datahub-web-react:mdPrettierWrite # Format markdown files\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL schemas\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions\n```\n\n**IMPORTANT: Verifying Python code changes:**\n\n- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes\n- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors\n- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead\n- lintFix runs ruff formatting and fixing automatically, ensuring code quality\n- For smoke-test changes, the lintFix command will also check those files\n\n## Code Formatting and Linting\n\n**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**\n\n### Available Formatting Tasks\n\n**Format everything:**\n\n```bash\n./gradlew format # Format all code (Java, Markdown, GraphQL, YAML)\n./gradlew formatChanged # Format only changed files (faster)\n```\n\n**Format specific file types:**\n\n```bash\n# Markdown files\n./gradlew :datahub-web-react:mdPrettierWrite # Format all markdown\n./gradlew :datahub-web-react:mdPrettierCheck # Check markdown formatting\n\n# GraphQL schemas\n./gradlew :datahub-web-react:graphqlPrettierWrite # Format GraphQL files\n./gradlew :datahub-web-react:graphqlPrettierCheck # Check GraphQL formatting\n\n# GitHub Actions YAML\n./gradlew :datahub-web-react:githubActionsPrettierWrite # Format workflow files\n./gradlew :datahub-web-react:githubActionsPrettierCheck # Check workflow files\n\n# Java code\n./gradlew spotlessApply # Format Java code\n\n# Python code\n./gradlew :metadata-ingestion:lintFix # Format and fix Python code\n./gradlew :metadata-ingestion:lint # Check Python formatting\n```\n\n### When CI Formatting Checks Fail\n\nIf you see CI failures like:\n\n- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`\n- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`\n- `spotlessJavaCheck` - Use `./gradlew spotlessApply`\n- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`\n\n**Never do this:**\n\n```bash\nnpx prettier --write \"docs/**/*.md\" # WRONG - bypasses Gradle\nyarn prettier --write # WRONG - bypasses Gradle\nnpm run format # WRONG - bypasses Gradle\n```\n\n**Always do this:**\n\n```bash\n./gradlew :datahub-web-react:mdPrettierWrite # CORRECT - uses Gradle\n./gradlew format # CORRECT - formats everything\n```\n\n### Why Use Gradle Tasks?\n\n1. **Consistent configuration**: Gradle tasks use the project's Prettier config\n2. **Pre-commit hook integration**: Gradle tasks match what CI runs\n3. **Dependency management**: Ensures correct tool versions\n4. **Cross-platform**: Works reliably across all environments\n\n**Java SDK v2 integration tests:**\n\nSee [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.\n\n## Architecture Overview\n\nDataHub is a **schema-first, event-driven metadata platform** with three core layers:\n\n### Core Services\n\n- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs\n- **Frontend**: React/TypeScript application consuming GraphQL APIs\n- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources\n- **Event Streaming**: Kafka-based real-time metadata change propagation\n\n### Key Modules\n\n- `metadata-models/`: Avro/PDL schemas defining the metadata model\n- `metadata-service/`: Backend services, APIs, and business logic\n- `datahub-web-react/`: Frontend React application\n- `metadata-ingestion/`: Python ingestion framework and CLI\n- `datahub-graphql-core/`: GraphQL schema and resolvers\n\nMost of the non-frontend modules are written in Java. The modules written in Python are:\n\n- `metadata-ingestion/`\n- `datahub-actions/`\n- `metadata-ingestion-modules/airflow-plugin/`\n- `metadata-ingestion-modules/gx-plugin/`\n- `metadata-ingestion-modules/dagster-plugin/`\n- `metadata-ingestion-modules/prefect-plugin/`\n\nEach Python module has a gradle setup similar to `metadata-ingestion/` (documented above)\n\n### Metadata Model Concepts\n\n- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)\n- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)\n- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)\n- **MCE/MCL**: Metadata Change Events/Logs for updates\n- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)\n\n### Validation Architecture\n\n**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).\n\n- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API\n- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`\n- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`\n- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples\n\n### Authorization Architecture\n\nWhen adding an entity or API:\n\n- Enforce authorization across GraphQL, OpenAPI, and Rest.li\n- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions\n- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs\n- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`\n- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities\n- Add allowed and denied access tests\n\n## Development Flow\n\n1. **Schema changes** in `metadata-models/` trigger code generation across all languages\n2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs\n3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs\n4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs\n\n## Working on Docs\n\nThe docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid\nconflicting with the frontend dev server).\n\n### Quick start\n\n```bash\nscripts/dev/datahub-dev.sh docs # fast start (assumes prior build)\nscripts/dev/datahub-dev.sh docs --build # full rebuild (runs docGen + yarnGenerate first)\n```\n\nOr via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).\n\n### How the docs site is assembled\n\nThe final site is served from `docs-website/genDocs/` (gitignored). It is assembled at build time\nfrom multiple hand-authored sources plus several generation steps:\n\n1. **Gradle generation tasks** produce `docs/generated/` (connector docs, entity reference, schemas)\n2. **`generateDocsDir.ts`** discovers all markdown in the repo, applies transformations (frontmatter,\n link rewriting, `{{ inline }}` directives), and writes the result to `genDocs/`\n3. **Docusaurus** serves from `genDocs/`, additionally generating GraphQL API docs and Python SDK docs\n\nSee `docs-website/AGENTS.md` for full pipeline details.\n\n### Where docs live\n\n| Path | What to edit | Detail guide |\n| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| `docs/` | Hand-authored feature guides, API docs, how-tos | _(this section)_ |\n| `metadata-ingestion/docs/sources//` | Connector docs (`*_pre.md`, `*_post.md`, etc.) | `metadata-ingestion/docs/sources/AGENTS.md` |\n| `metadata-models/docs/entities/` | Entity descriptions (input to `modelDocGen`) | `metadata-models/docs/AGENTS.md` |\n| `docs-website/src/pages/` | Custom React pages (e.g. `/integrations`) | `docs-website/AGENTS.md` |\n| `docs-website/src/learn/` | Blog / learning articles (served at `/learn`) | `docs-website/AGENTS.md` |\n| `docs-website/sidebars.js` | Sidebar navigation tree | `docs-website/AGENTS.md` |\n| `docs-website/static/` | Images, logos, static assets | `docs-website/AGENTS.md` |\n| `docs/generated/` | **Never edit** — auto-generated | |\n| `docs-website/genDocs/` | **Never edit** — assembled output | |\n\n### Adding or editing a hand-authored doc\n\n1. Create/edit the markdown file in `docs/`\n2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)\n3. Run `scripts/dev/datahub-dev.sh docs` to preview\n\nIf `sidebars.js` is missing the entry, the build will warn about an unaccounted file.\n\n### Adding a DataHub Cloud release note\n\nRelease notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_.md`.\n\n**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:\n\n1. Create `docs/managed-datahub/release-notes/v_0_3_.md`\n2. Add `\"docs/managed-datahub/release-notes/v_0_3_\"` as the **first entry** under `\"DataHub Cloud Release History\"` in `docs-website/sidebars.js` (newer releases go at the top)\n\nForgetting step 2 means the release note is published but never appears in the sidebar navigation.\n\n## Code Standards\n\n### General Principles\n\n- This is production code - maintain high quality\n- Follow existing patterns within each module\n- Generate appropriate unit tests\n- Use type annotations everywhere (Python/TypeScript)\n\n### Language-Specific\n\n- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests\n- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs\n - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)\n - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples\n - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code\n - **Error Handling**: Robust error handling with layers of protection for known failure points\n - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc//environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.\n- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library\n\n### Frontend Theming (Colors)\n\n**Always use semantic color tokens** from `datahub-web-react/src/conf/theme/colorThemes/types.ts`. Never use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy `colors.gray[X]` imports.\n\n**In styled-components** (no import needed — `theme` is available via props):\n\n```typescript\nbackground: ${(props) => props.theme.colors.bg};\ncolor: ${(props) => props.theme.colors.text};\nborder: 1px solid ${(props) => props.theme.colors.border};\n```\n\n**In React component bodies:**\n\n```typescript\nimport { useTheme } from 'styled-components';\nconst theme = useTheme();\n\n```\n\n**For alchemy components** (``, ``, etc.) — do not pass `color`/`colorLevel` props. Let them inherit from themed parent styled-components.\n\n**Do not import from:**\n\n- `datahub-web-react/src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally by the theme)\n- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`\n\n### Code Comments\n\nOnly add comments that provide real value beyond what the code already expresses.\n\n**Do NOT** add comments for:\n\n- Obvious operations (`# Get user by ID`, `// Create connection`)\n- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)\n- Restating parameter names or return types already in signatures\n- Basic language constructs (`# Import modules`, `// End of function`)\n\n**DO** add comments for:\n\n- **Why** something is done, especially non-obvious business logic or workarounds\n- **Context** about external constraints, API quirks, or domain knowledge\n- **Warnings** about gotchas, performance implications, or side effects\n- **References** to tickets, RFCs, or external documentation that explain decisions\n- **Complex algorithms** or mathematical formulas that aren't immediately clear\n- **Temporary solutions** with TODOs and context for future improvements\n\nExamples:\n\n```python\n# Good: Explains WHY and provides context\n# Use a 30-second timeout because Snowflake's query API can hang indefinitely\n# on large result sets. See issue #12345.\nconnection_timeout = 30\n\n# Bad: Restates what's obvious from code\n# Set connection timeout to 30 seconds\nconnection_timeout = 30\n```\n\n### Testing Strategy\n\n- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements\n- Java: Tests alongside source in `src/test/`\n- Frontend: Tests in `__tests__/` or `.test.tsx` files\n- Smoke tests go in the `smoke-test/` directory\n\n#### Testing Principles: Focus on Value Over Coverage\n\n**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.\n\n**Focus on behavior, not implementation**:\n\n- Test what the code does (business logic, edge cases that occur in production)\n- Don't test how it does it (implementation details, private fields via reflection)\n- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)\n- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)\n\n**Avoid these specific anti-patterns**:\n\n- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters\n- Verifying exact error message wording (creates brittleness during refactoring)\n- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)\n- Using reflection to verify private implementation details\n- Redundant concurrency testing on `synchronized` methods\n- Testing obvious getter/setter behavior without business logic\n- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic\n- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests\n\n**Appropriate test scope**:\n\n- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests\n - Happy path for each method\n - One example of invalid input per method\n - Edge cases likely to occur in production\n- **Complex business logic**: Test proportional to risk and complexity\n - Integration points and system boundaries\n - Security-critical operations\n - Error handling for realistic failure scenarios\n- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope\n\n**Examples of low-value tests to avoid**:\n\n```java\n// BAD: Testing @Nonnull contract (framework's job)\n@Test\npublic void testNullParameterThrowsException() {\n assertThrows(NullPointerException.class,\n () -> service.process(null)); // parameter is @Nonnull\n}\n\n// BAD: Testing Lombok-generated code\n@Test\npublic void testBuilderSetsAllFields() {\n MyConfig config = MyConfig.builder()\n .field1(\"value1\")\n .field2(\"value2\")\n .build();\n assertEquals(config.getField1(), \"value1\");\n assertEquals(config.getField2(), \"value2\");\n}\n\n// BAD: Testing that annotations exist\n@Test\npublic void testConfigurationAnnotations() {\n assertNotNull(MyConfig.class.getAnnotation(Configuration.class));\n assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));\n}\n// If @Configuration is missing, Spring won't load the context - you don't need a test for this\n\n// BAD: Exact error message (brittle)\nassertEquals(exception.getMessage(),\n \"Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.\");\n\n// BAD: Redundant variations\nassertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"PostgreSQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\nassertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n// ... 10 more case/whitespace variations\n\n// GOOD: Focused behavioral test\n@Test\npublic void testFromString_ValidInputsCaseInsensitive() {\n assertEquals(DatabaseType.fromString(\"postgresql\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\"POSTGRESQL\"), DatabaseType.POSTGRES);\n assertEquals(DatabaseType.fromString(\" postgresql \"), DatabaseType.POSTGRES);\n}\n\n@Test\npublic void testFromString_InvalidInputThrows() {\n assertThrows(IllegalArgumentException.class,\n () -> DatabaseType.fromString(\"oracle\"));\n}\n\n// GOOD: Testing YOUR custom validation logic on a Lombok class\n@Test\npublic void testCustomValidation() {\n assertThrows(IllegalArgumentException.class,\n () -> MyConfig.builder().field1(\"invalid\").build().validate());\n}\n```\n\n**When in doubt**: Ask \"Does this test protect against a realistic regression?\" If not, skip it.\n\n#### Security Testing: Configuration Property Classification\n\n**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`\n\nThis test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).\n\n**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.\n\nThis is a mandatory security guardrail - never disable or skip this test.\n\n### Commits\n\n- Follow Conventional Commits format for commit messages\n- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n### Pull Requests\n\nWhen creating PRs, follow the template in `.github/pull_request_template.md`:\n\n**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):\n\n```\n[optional scope]: \n```\n\nTypes: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`\n\nExample: `feat(parser): add ability to parse arrays`\n\n**Checklist** (verify before submitting):\n\n- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)\n- [ ] Links to related issues (if applicable)\n- [ ] Tests added/updated (if applicable)\n- [ ] Docs added/updated (if applicable)\n- [ ] Breaking changes documented in `docs/how/updating-datahub.md`\n\n### Confidentiality in Committed Code\n\nDataHub is a **public repository**. Never put customer-identifiable or\nenvironment-specific details into committed code, tests, docs, comments, commit\nmessages, or PRs:\n\n- No real database / schema / table / view / column names, and no usernames,\n customer names, host names, account IDs, or URLs from customer environments.\n- No Linear/Jira ticket IDs or links.\n- When reproducing a customer issue in a test, use generic placeholder names\n (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern\n being tested, not the customer's actual identifiers.\n- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,\n but prefer generic names when in doubt.\n- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.\n\n## Starting / Operating DataHub\n\nUse `scripts/dev/datahub-dev.sh` for **ALL** environment operations.\n**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.\n\n### `datahub-dev` CLI Tool\n\nA stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.\n\n**Always use the shell wrapper as the entry point:**\n\n```bash\nscripts/dev/datahub-dev.sh \n```\n\nRun `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,\n`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,\n`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).\n\n### End-to-End Workflow\n\n0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.\n1. **Start**: `scripts/dev/datahub-dev.sh start`\n2. **Code**: Make changes to Java/Python/frontend code\n3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`\n4. **Test**: `scripts/dev/datahub-dev.sh test `\n5. **Iterate**: Repeat steps 2–4\n\n**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).\n\n### Module-to-Container Mapping\n\n| Source directory | Container |\n| --------------------------------- | --------------------------------------------- |\n| `metadata-service/` | `datahub-gms` |\n| `datahub-graphql-core/` | `datahub-gms` |\n| `metadata-io/` | `datahub-gms` |\n| `datahub-frontend/` | `datahub-frontend-react` |\n| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer` |\n| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer` |\n| `metadata-models/` | All (triggers full rebuild + code generation) |\n\n### Environment Variables\n\nSet any env var for DataHub containers via `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set KEY=VALUE\nscripts/dev/datahub-dev.sh env restart # required — changes take effect on restart\nscripts/dev/datahub-dev.sh env list # show current vars and pending_restart status\n```\n\n**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.\n\n**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /\n`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`\nreads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).\n`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.\n\n### Feature Flag Lifecycle\n\n**All flag changes require a container restart.** Use `env set` + `env restart`:\n\n```bash\nscripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true\nscripts/dev/datahub-dev.sh env restart\n```\n\n`flag list` and `flag get` are read-only inspection tools — they show the current live values from\nthe running server but do not change anything.\n\nThe flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**\n(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`\nor after a fresh clone.\n\n### Stopping DataHub\n\n`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.\n\nWhen starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances\nfrom other worktrees/compose projects that occupy the same ports.\n\n### Remote Runners\n\n`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine\n(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.\n\n**Configure a runner** in `~/.datahub/dev/config.json`:\n\n```json\n{\n \"max_local_instances\": 2,\n \"max_remote_instances\": 10,\n \"runner\": \"/path/to/your-runner.sh\"\n}\n```\n\nOr export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.\n\n**Remote lifecycle** (all commands work identically to local once a runner is set):\n\n```bash\n# One-time bootstrap — provisions the remote environment\nscripts/dev/datahub-dev.sh setup --remote\n\n# Start — syncs changed local files, runs quickstartDebug on the remote,\n# then sets up port tunnels so local ports reach the remote instance\nscripts/dev/datahub-dev.sh start\n\n# Stop containers only (remote compute keeps running)\nscripts/dev/datahub-dev.sh stop\n\n# Stop containers AND halt the remote compute (no billing while suspended).\n# 'start' will automatically resume the instance when needed.\nscripts/dev/datahub-dev.sh suspend\n\n# All other commands (status, wait, rebuild, test, flag, env, nuke, …)\n# proxy through the runner transparently — use them exactly as you would locally.\nscripts/dev/datahub-dev.sh status\n```\n\n**Multi-instance management** — each git worktree gets its own isolated instance\n(separate Docker project, volumes, and port assignment):\n\n```bash\n# List all registered instances (local and remote) with their ports and status\nscripts/dev/datahub-dev.sh instances list\n\n# Remove stale entries for worktrees that no longer exist\nscripts/dev/datahub-dev.sh instances clean\n\n# Print export statements for the current instance's CLI environment\neval $(scripts/dev/datahub-dev.sh shell-env)\n# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)\n```\n\n**Port assignment** — each instance gets a slot; ports = base + slot × 1000:\n\n| Slot | GMS | Frontend | Notes |\n| ---- | ----- | -------- | ------------------------- |\n| 0 | 8080 | 9002 | First local instance |\n| 1 | 9080 | 10002 | Second local instance |\n| 2 | 10080 | 11002 | First remote instance |\n| … | … | … | Each worktree is isolated |\n\n**Backwards compatibility / opting out of isolation** — if the new per-worktree\nproject names cause problems (lost data in old volumes, tooling that expects\n`datahub-*` container names, CI environments that don't need isolation), set\n`compose_project` in `~/.datahub/dev/config.json`:\n\n```json\n{ \"compose_project\": \"datahub\" }\n```\n\nThis reverts to the old single-instance behaviour: one `datahub` Docker project,\nsame container names, existing volumes fully accessible. The env var\n`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.\n\n**Runner interface** — a runner is any executable that speaks four verbs:\n\n```bash\nrunner init # one-time environment bootstrap\nrunner sync # push changed local files to the remote\nrunner exec -- [args...] # execute a command in the remote workspace\nrunner tunnel ... # set up port forwarding\nrunner resume # start compute if stopped (no-op if running)\nrunner suspend # stop containers + halt compute\n```\n\nA reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.\n\n### Recovery Escalation\n\n**When to use each:**\n\n- `stop`: Just shut down DataHub — no restart, no data loss\n- `reset`: GMS returns 503 and doesn't recover, frontend shows \"Unable to connect\", tests fail\n with connection errors\n- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it\n- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing\n clean slate, `nuke --keep-data` didn't fix it\n\n### Structured Test Output\n\nSet `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:\n\n```bash\nAGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py\n```\n\n## Common Operations\n\nThese commands work against **any** DataHub instance — local dev, staging, or production.\nProvide connection details via environment variables:\n\n```bash\nexport DATAHUB_GMS_URL=http://localhost:8080 # or your instance URL\nexport DATAHUB_GMS_TOKEN= # omit if auth is not required\n```\n\n### Init (setup authentication)\n\n`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before\nusing any other CLI commands that require authentication.\n\n```bash\n# Quickstart: local instance with default credentials\ndatahub init --username datahub --password datahub\n\n# Full agent best-practices guide (defaults, env vars, all scenarios)\ndatahub init --agent-context\n```\n\n### GraphQL\n\n`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can\nintrospect the live schema to discover available operations.\n\n```bash\n# Discover what's available\ndatahub graphql --list-operations --format json\n\n# Inspect a specific operation's arguments\ndatahub graphql --describe dataset --format json\n\n# Preview a query before executing\ndatahub graphql --query \"{ me { corpUser { urn } } }\" --dry-run\n\n# Execute a query\ndatahub graphql --query \"{ me { corpUser { urn username } } }\" --format json\n```\n\nFor full agent best practices (discovery, dry-run, error codes, common recipes):\n\n```bash\ndatahub graphql --agent-context\n```\n\n## Key Documentation\n\n**Essential reading:**\n\n- `docs/architecture/architecture.md` - System architecture overview\n- `docs/modeling/metadata-model.md` - How metadata is modeled\n- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)\n\n**External docs:**\n\n- https://docs.datahub.com/docs/developers - Official developer guide\n- https://demo.datahub.com/ - Live demo environment\n\n## Playwright UI E2E Tests\n\nFull reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md).\n\n### Seeding\n\n`test.use({ featureName: 'my-feature' })` at the `describe` level auto-loads\n`tests/my-feature/fixtures/data.json` via `seeding.fixture.ts` — once per worker per\nfeature per run. Do **not** set `featureName` for suites that create their own data\nvia `apiMock` or direct API calls.\n\n## Frontend CI Checklist\n\nThis checklist is for **commit- or PR-ready** frontend work — i.e. when you're about to\ncommit, push, or hand off changes that are going into a PR. It is **not** required for\nevery intermediate edit: work that is part of a larger task, a work-in-progress branch,\nor scratch experimentation that won't be committed yet can skip it. Run the relevant\ncommands when the change is ready to ship:\n\n```bash\n# Full lint (eslint + prettier src + type-check) for datahub-web-react\n./gradlew :datahub-web-react:yarnLint\n\n# Targeted lint-fix on a single file\n./gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx\n\n# Vitest unit tests (requires icon stubs — run once per clone)\nnode datahub-web-react/scripts/generate-lazy-icon-stubs.js\ncd datahub-web-react && yarn test src/path/to/file.test.ts --run\n```\n\n`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in\nunrelated files. Focus on errors in files **you touched** — in particular, optional\nprop calls (`prop?.(arg)`) and import aliases.\n\n## Python Virtual Environments\n\nGradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`\n\n## Important Notes\n\n- Entity Registry is defined in YAML, not code (`entity-registry.yml`)\n- All metadata changes flow through the event streaming system\n- GraphQL schema is generated from backend GMS APIs\n\n## Learned User Preferences\n\n- In `metadata-ingestion` connector code, avoid double-quoted string literals: hoist magic strings into module-level constants, and keep all regex in the constants file pre-compiled.\n- Use Pydantic models for structured/internal data; never pass data around as tuples (hard to track).\n- Split connector files by duty (`constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`/`mapper.py`/`usage.py` as needed) and match the quality/patterns of existing connectors (Power BI, Airbyte, Redshift, BigID, Grafana).\n- No \"AI slop\": no top-of-file docstrings, and keep docstrings/comments only where strictly needed.\n- Never use `TYPE_CHECKING` in connector code since the connector controls its own deps (lazy imports are fine only for opt-in features), and don't use the walrus operator.\n- Prefer `self.report.warning(...)` and report counters over bare `logger` for skips and edge cases — the report also writes to the log and surfaces to operators (e.g. warn when `verify_ssl=False`, or when a referenced object is inaccessible).\n- For SQL lineage, use the central `SqlParsingAggregator` (`create_lineage_from_sql_statements`) with a platform map instead of setting sqlglot dialects per-connector; mirror existing connectors for cross-platform known-URN and platform_instance/env/casing mapping, two- vs three-part names, and temp-table handling.\n- For column-level lineage, don't leave edges coarse: resolve upstream/downstream schemas from the DataHub graph when available (as airbyte/bigid/matillion/informatica do), load known URNs from the platform/platform_instance/env mapping, and match columns case-insensitively. Best-effort is fine, but try everything.\n- In connector code, use explicit type annotations rather than `from typing import Any` (Unions are fine when a value genuinely has multiple types), and prefer `Dict`/`List` from `typing` over the builtin `dict`/`list`.\n- Connectors should surface progress during ingestion and use explicit ingestion stages (as in the dremio and snowflake connectors).\n- Before committing a new connector, run its ingestion locally in debug mode to a local file to capture full logs and catch bugs; when testing against a customer environment, push secrets only to a tmp path (e.g. `/tmp/*.env`).\n- When drafting prose or review comments on the user's behalf (e.g. Notion), write in his own direct, human voice — avoid AI tells like \"confirmed these are real gaps\".\n\n## Learned Workspace Facts\n\n- The user is a contributor to the public `datahub-project/datahub` repo and can create and push branches directly on `datahub-public-repo`.\n- A new ingestion connector needs more than Python code: a source logo plus an integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration (`setup.py`/`pyproject.toml`), a refreshed `uv.lock`, and subtypes added to the shared subtypes module rather than defined locally.\n- Avoid Python's stdlib `xml` parser due to a known vulnerability; use a safe XML library (as the HANA-related code does).\n- Keep each connector in its own PR and split shared/framework changes (e.g. sqlglot helpers) into a separate PR; a connector PR's title and description must reference only that connector, not any other connector worked on in the same session.\n","category":"root","tokens":9322}]}