datahub

# CLAUDE.md

@AGENTS.md

# DataHub — Agent Development Guide

This is the canonical reference for working with the DataHub codebase. It applies to all coding
agents (Claude Code, Cursor, Codex CLI, Devin, etc.) and human developers alike.

## Code Navigation (LSP)

Prefer LSP tools over Grep for code navigation tasks:

- Use `goToDefinition` to find where something is defined
- Use `findReferences` to find all call sites
- Use `workspaceSymbol` to find symbols by name
- Use diagnostics after any edit to catch type errors immediately

See [docs/lsp-setup.md](docs/lsp-setup.md) for installation and configuration.

## Essential Commands

**Build and test:**

```bash
./gradlew build           # Build entire project
./gradlew check           # Run all tests and linting
./gradlew format          # Format all code (Java, Markdown, GraphQL, YAML)

# Note that each directory typically has a build.gradle file, but the available tasks follow similar conventions.

# Java code.
./gradlew spotlessApply   # Java code formatting

# Python code.
./gradlew :metadata-ingestion:testQuick     # Fast Python unit tests
./gradlew :metadata-ingestion:lint          # Python linting (ruff, mypy)
./gradlew :metadata-ingestion:lintFix       # Python linting auto-fix (ruff only)

# Markdown, GraphQL, YAML formatting
./gradlew :datahub-web-react:mdPrettierWrite        # Format markdown files
./gradlew :datahub-web-react:graphqlPrettierWrite   # Format GraphQL schemas
./gradlew :datahub-web-react:githubActionsPrettierWrite # Format GitHub Actions
```

**IMPORTANT: Verifying Python code changes:**

- **ALWAYS use `./gradlew :metadata-ingestion:lintFix`** to verify Python code changes
- **NEVER use `python3 -m py_compile`** - it doesn't catch style issues or type errors
- **NEVER use `ruff` or `mypy` commands directly** - use the Gradle task instead
- lintFix runs ruff formatting and fixing automatically, ensuring code quality
- For smoke-test changes, the lintFix command will also check those files

## Code Formatting and Linting

**CRITICAL: Always use Gradle tasks for formatting and linting. Never use npm/yarn/npx commands directly.**

### Available Formatting Tasks

**Format everything:**

```bash
./gradlew format              # Format all code (Java, Markdown, GraphQL, YAML)
./gradlew formatChanged       # Format only changed files (faster)
```

**Format specific file types:**

```bash
# Markdown files
./gradlew :datahub-web-react:mdPrettierWrite        # Format all markdown
./gradlew :datahub-web-react:mdPrettierCheck        # Check markdown formatting

# GraphQL schemas
./gradlew :datahub-web-react:graphqlPrettierWrite   # Format GraphQL files
./gradlew :datahub-web-react:graphqlPrettierCheck   # Check GraphQL formatting

# GitHub Actions YAML
./gradlew :datahub-web-react:githubActionsPrettierWrite   # Format workflow files
./gradlew :datahub-web-react:githubActionsPrettierCheck   # Check workflow files

# Java code
./gradlew spotlessApply       # Format Java code

# Python code
./gradlew :metadata-ingestion:lintFix      # Format and fix Python code
./gradlew :metadata-ingestion:lint         # Check Python formatting
```

### When CI Formatting Checks Fail

If you see CI failures like:

- `markdown_format / markdown_format_check (pull_request)` - Use `./gradlew :datahub-web-react:mdPrettierWrite`
- `graphql_prettier_check` - Use `./gradlew :datahub-web-react:graphqlPrettierWrite`
- `spotlessJavaCheck` - Use `./gradlew spotlessApply`
- Python linting failures - Use `./gradlew :metadata-ingestion:lintFix`

**Never do this:**

```bash
npx prettier --write "docs/**/*.md"    # WRONG - bypasses Gradle
yarn prettier --write                   # WRONG - bypasses Gradle
npm run format                          # WRONG - bypasses Gradle
```

**Always do this:**

```bash
./gradlew :datahub-web-react:mdPrettierWrite      # CORRECT - uses Gradle
./gradlew format                                   # CORRECT - formats everything
```

### Why Use Gradle Tasks?

1. **Consistent configuration**: Gradle tasks use the project's Prettier config
2. **Pre-commit hook integration**: Gradle tasks match what CI runs
3. **Dependency management**: Ensures correct tool versions
4. **Cross-platform**: Works reliably across all environments

**Java SDK v2 integration tests:**

See [metadata-integration/java/datahub-client/CLAUDE.md](metadata-integration/java/datahub-client/CLAUDE.md) for detailed integration test documentation.

## Architecture Overview

DataHub is a **schema-first, event-driven metadata platform** with three core layers:

### Core Services

- **GMS (Generalized Metadata Service)**: Java/Spring backend handling metadata storage and REST/GraphQL APIs
- **Frontend**: React/TypeScript application consuming GraphQL APIs
- **Ingestion Framework**: Python CLI and connectors for extracting metadata from data sources
- **Event Streaming**: Kafka-based real-time metadata change propagation

### Key Modules

- `metadata-models/`: Avro/PDL schemas defining the metadata model
- `metadata-service/`: Backend services, APIs, and business logic
- `datahub-web-react/`: Frontend React application
- `metadata-ingestion/`: Python ingestion framework and CLI
- `datahub-graphql-core/`: GraphQL schema and resolvers

Most of the non-frontend modules are written in Java. The modules written in Python are:

- `metadata-ingestion/`
- `datahub-actions/`
- `metadata-ingestion-modules/airflow-plugin/`
- `metadata-ingestion-modules/gx-plugin/`
- `metadata-ingestion-modules/dagster-plugin/`
- `metadata-ingestion-modules/prefect-plugin/`

Each Python module has a gradle setup similar to `metadata-ingestion/` (documented above)

### Metadata Model Concepts

- **Entities**: Core objects (Dataset, Dashboard, Chart, CorpUser, etc.)
- **Aspects**: Metadata facets (Ownership, Schema, Documentation, etc.)
- **URNs**: Unique identifiers (`urn:li:dataset:(urn:li:dataPlatform:mysql,db.table,PROD)`)
- **MCE/MCL**: Metadata Change Events/Logs for updates
- **Entity Registry**: YAML config defining entity-aspect relationships (`metadata-models/src/main/resources/entity-registry.yml`)

### Validation Architecture

**IMPORTANT**: Validation must work across all APIs (GraphQL, OpenAPI, RestLI).

- **Never add validation in API-specific layers** (GraphQL resolvers, REST controllers) - this only protects one API
- **Always implement AspectPayloadValidators** in `metadata-io/src/main/java/com/linkedin/metadata/aspect/validation/`
- **Register as Spring beans** in `SpringStandardPluginConfiguration.java`
- **Follow existing patterns**: See `SystemPolicyValidator.java` and `PolicyFieldTypeValidator.java` as examples

### Authorization Architecture

When adding an entity or API:

- Enforce authorization across GraphQL, OpenAPI, and Rest.li
- Keep basic entity CRUD permissions alongside any higher-level, entity-specific permissions
- Use `AuthorizationUtils` for GraphQL and `AuthUtil.isAPIAuthorized*` for REST APIs
- Put shared aspect rules in an `AbstractAspectAuthorizationValidator`
- Apply view-based access controls by default; only set `viewUnrestricted: true` for intentionally public entities
- Add allowed and denied access tests

## Development Flow

1. **Schema changes** in `metadata-models/` trigger code generation across all languages
2. **Backend changes** in `metadata-service/` and other Java modules expose new REST/GraphQL APIs
3. **Frontend changes** in `datahub-web-react/` consume GraphQL APIs
4. **Ingestion changes** in `metadata-ingestion/` emit metadata to backend APIs

## Working on Docs

The docs site is a **Docusaurus 2** app in `docs-website/`. It runs on **port 3001** (not 3000, to avoid
conflicting with the frontend dev server).

### Quick start

```bash
scripts/dev/datahub-dev.sh docs            # fast start (assumes prior build)
scripts/dev/datahub-dev.sh docs --build    # full rebuild (runs docGen + yarnGenerate first)
```

Or via Gradle directly: `./gradlew :docs-website:yarnStart` (always does a full build).

### How the docs site is assembled

The final site is served from `docs-website/genDocs/` (gitignored). It is assembled at build time
from multiple hand-authored sources plus several generation steps:

1. **Gradle generation tasks** produce `docs/generated/` (connector docs, entity reference, schemas)
2. **`generateDocsDir.ts`** discovers all markdown in the repo, applies transformations (frontmatter,
   link rewriting, `{{ inline }}` directives), and writes the result to `genDocs/`
3. **Docusaurus** serves from `genDocs/`, additionally generating GraphQL API docs and Python SDK docs

See `docs-website/AGENTS.md` for full pipeline details.

### Where docs live

| Path                                           | What to edit                                    | Detail guide                                |
| ---------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |
| `docs/`                                        | Hand-authored feature guides, API docs, how-tos | _(this section)_                            |
| `metadata-ingestion/docs/sources/<connector>/` | Connector docs (`*_pre.md`, `*_post.md`, etc.)  | `metadata-ingestion/docs/sources/AGENTS.md` |
| `metadata-models/docs/entities/`               | Entity descriptions (input to `modelDocGen`)    | `metadata-models/docs/AGENTS.md`            |
| `docs-website/src/pages/`                      | Custom React pages (e.g. `/integrations`)       | `docs-website/AGENTS.md`                    |
| `docs-website/src/learn/`                      | Blog / learning articles (served at `/learn`)   | `docs-website/AGENTS.md`                    |
| `docs-website/sidebars.js`                     | Sidebar navigation tree                         | `docs-website/AGENTS.md`                    |
| `docs-website/static/`                         | Images, logos, static assets                    | `docs-website/AGENTS.md`                    |
| `docs/generated/`                              | **Never edit** — auto-generated                 |                                             |
| `docs-website/genDocs/`                        | **Never edit** — assembled output               |                                             |

### Adding or editing a hand-authored doc

1. Create/edit the markdown file in `docs/`
2. Add an entry in `docs-website/sidebars.js` (the doc ID is the file path minus `.md`)
3. Run `scripts/dev/datahub-dev.sh docs` to preview

If `sidebars.js` is missing the entry, the build will warn about an unaccounted file.

### Adding a DataHub Cloud release note

Release notes live in `docs/managed-datahub/release-notes/` and follow the naming convention `v_0_3_<N>.md`.

**CRITICAL**: Adding the markdown file alone is not enough — you must also add it to `sidebars.js`:

1. Create `docs/managed-datahub/release-notes/v_0_3_<N>.md`
2. Add `"docs/managed-datahub/release-notes/v_0_3_<N>"` as the **first entry** under `"DataHub Cloud Release History"` in `docs-website/sidebars.js` (newer releases go at the top)

Forgetting step 2 means the release note is published but never appears in the sidebar navigation.

## Code Standards

### General Principles

- This is production code - maintain high quality
- Follow existing patterns within each module
- Generate appropriate unit tests
- Use type annotations everywhere (Python/TypeScript)

### Language-Specific

- **Java**: Use Spotless formatting, Spring Boot patterns, TestNG/JUnit Jupiter for tests
- **Python**: Use ruff for linting/formatting, pytest for testing, pydantic for configs
  - **Type Safety**: Everything must have type annotations, avoid `Any` type, use specific types (`Dict[str, int]`, `TypedDict`)
  - **Data Structures**: Prefer dataclasses/pydantic for internal data, return dataclasses over tuples
  - **Code Quality**: Avoid global state, use named arguments, don't re-export in `__init__.py`, refactor repetitive code
  - **Error Handling**: Robust error handling with layers of protection for known failure points
  - **Security**: Never pass credentials to third-party SDKs via `os.environ`. Use the SDK's programmatic injection mechanism (a settings object, client constructor argument, or credential provider). Writing secrets to the process environment exposes them via `/proc/<pid>/environ` and to any code in the same process. See [`looker_lib_wrapper.py`](metadata-ingestion/src/datahub/ingestion/source/looker/looker_lib_wrapper.py) (`_DataHubLookerApiSettings`) for the canonical pattern.
- **TypeScript**: Use Prettier formatting, strict types (no `any`), React Testing Library

### Frontend Theming (Colors)

**Always use semantic color tokens** from `datahub-web-react/src/conf/theme/colorThemes/types.ts`. Never use hardcoded hex values, `REDESIGN_COLORS`, `ANTD_GRAY`, or direct alchemy `colors.gray[X]` imports.

**In styled-components** (no import needed — `theme` is available via props):

```typescript
background: ${(props) => props.theme.colors.bg};
color: ${(props) => props.theme.colors.text};
border: 1px solid ${(props) => props.theme.colors.border};
```

**In React component bodies:**

```typescript
import { useTheme } from 'styled-components';
const theme = useTheme();
<Icon color={theme.colors.icon} />
```

**For alchemy components** (`<Text>`, `<Icon>`, etc.) — do not pass `color`/`colorLevel` props. Let them inherit from themed parent styled-components.

**Do not import from:**

- `datahub-web-react/src/alchemy-components/theme/foundations/colors.ts` (raw palette, only used internally by the theme)
- `REDESIGN_COLORS` or `ANTD_GRAY` from `entityV2/shared/constants.ts`

### Code Comments

Only add comments that provide real value beyond what the code already expresses.

**Do NOT** add comments for:

- Obvious operations (`# Get user by ID`, `// Create connection`)
- What the code does when it's self-evident (`# Loop through items`, `// Set variable to true`)
- Restating parameter names or return types already in signatures
- Basic language constructs (`# Import modules`, `// End of function`)

**DO** add comments for:

- **Why** something is done, especially non-obvious business logic or workarounds
- **Context** about external constraints, API quirks, or domain knowledge
- **Warnings** about gotchas, performance implications, or side effects
- **References** to tickets, RFCs, or external documentation that explain decisions
- **Complex algorithms** or mathematical formulas that aren't immediately clear
- **Temporary solutions** with TODOs and context for future improvements

Examples:

```python
# Good: Explains WHY and provides context
# Use a 30-second timeout because Snowflake's query API can hang indefinitely
# on large result sets. See issue #12345.
connection_timeout = 30

# Bad: Restates what's obvious from code
# Set connection timeout to 30 seconds
connection_timeout = 30
```

### Testing Strategy

- Python: Tests go in the `tests/` directory alongside `src/`, use `assert` statements
- Java: Tests alongside source in `src/test/`
- Frontend: Tests in `__tests__/` or `.test.tsx` files
- Smoke tests go in the `smoke-test/` directory

#### Testing Principles: Focus on Value Over Coverage

**IMPORTANT**: Quality over quantity. Avoid AI-generated test anti-patterns that create maintenance burden without providing real value.

**Focus on behavior, not implementation**:

- Test what the code does (business logic, edge cases that occur in production)
- Don't test how it does it (implementation details, private fields via reflection)
- Don't test third-party libraries work correctly (Spring, Micrometer, Kafka clients, etc.)
- Don't test Java/Python language features (`synchronized` methods are thread-safe, `@Nonnull` parameters reject nulls)

**Avoid these specific anti-patterns**:

- Testing null inputs on `@Nonnull`/`@NonNull` annotated parameters
- Verifying exact error message wording (creates brittleness during refactoring)
- Testing every possible input variation (case sensitivity x whitespace x special chars = maintenance nightmare)
- Using reflection to verify private implementation details
- Redundant concurrency testing on `synchronized` methods
- Testing obvious getter/setter behavior without business logic
- Testing Lombok-generated code (`@Data`, `@Builder`, `@Value` classes) - you're testing Lombok's code generator, not your logic
- Testing that annotations exist on classes - if required annotations are missing, the framework/compiler will fail at startup, not in your tests

**Appropriate test scope**:

- **Simple utilities** (enums, string parsing, formatters): ~50-100 lines of focused tests
  - Happy path for each method
  - One example of invalid input per method
  - Edge cases likely to occur in production
- **Complex business logic**: Test proportional to risk and complexity
  - Integration points and system boundaries
  - Security-critical operations
  - Error handling for realistic failure scenarios
- **Warning sign**: If tests are 5x+ the size of implementation, reconsider scope

**Examples of low-value tests to avoid**:

```java
// BAD: Testing @Nonnull contract (framework's job)
@Test
public void testNullParameterThrowsException() {
    assertThrows(NullPointerException.class,
        () -> service.process(null)); // parameter is @Nonnull
}

// BAD: Testing Lombok-generated code
@Test
public void testBuilderSetsAllFields() {
    MyConfig config = MyConfig.builder()
        .field1("value1")
        .field2("value2")
        .build();
    assertEquals(config.getField1(), "value1");
    assertEquals(config.getField2(), "value2");
}

// BAD: Testing that annotations exist
@Test
public void testConfigurationAnnotations() {
    assertNotNull(MyConfig.class.getAnnotation(Configuration.class));
    assertNotNull(MyConfig.class.getAnnotation(ComponentScan.class));
}
// If @Configuration is missing, Spring won't load the context - you don't need a test for this

// BAD: Exact error message (brittle)
assertEquals(exception.getMessage(),
    "Unsupported database type 'oracle'. Only PostgreSQL and MySQL variants are supported.");

// BAD: Redundant variations
assertEquals(DatabaseType.fromString("postgresql"), DatabaseType.POSTGRES);
assertEquals(DatabaseType.fromString("PostgreSQL"), DatabaseType.POSTGRES);
assertEquals(DatabaseType.fromString("POSTGRESQL"), DatabaseType.POSTGRES);
assertEquals(DatabaseType.fromString("  postgresql  "), DatabaseType.POSTGRES);
// ... 10 more case/whitespace variations

// GOOD: Focused behavioral test
@Test
public void testFromString_ValidInputsCaseInsensitive() {
    assertEquals(DatabaseType.fromString("postgresql"), DatabaseType.POSTGRES);
    assertEquals(DatabaseType.fromString("POSTGRESQL"), DatabaseType.POSTGRES);
    assertEquals(DatabaseType.fromString("  postgresql  "), DatabaseType.POSTGRES);
}

@Test
public void testFromString_InvalidInputThrows() {
    assertThrows(IllegalArgumentException.class,
        () -> DatabaseType.fromString("oracle"));
}

// GOOD: Testing YOUR custom validation logic on a Lombok class
@Test
public void testCustomValidation() {
    assertThrows(IllegalArgumentException.class,
        () -> MyConfig.builder().field1("invalid").build().validate());
}
```

**When in doubt**: Ask "Does this test protect against a realistic regression?" If not, skip it.

#### Security Testing: Configuration Property Classification

**Critical test**: `metadata-io/src/test/java/com/linkedin/metadata/system_info/collectors/PropertiesCollectorConfigurationTest.java`

This test prevents sensitive data leaks by requiring explicit classification of all configuration properties as either sensitive (redacted) or non-sensitive (visible in system info).

**When adding new configuration properties**: The test will fail with clear instructions on which classification list to add your property to. Refer to the test file's comprehensive documentation for template syntax and examples.

This is a mandatory security guardrail - never disable or skip this test.

### Commits

- Follow Conventional Commits format for commit messages
- Breaking Changes: Always update `docs/how/updating-datahub.md` for breaking changes. Write entries for non-technical audiences, reference the PR number, and focus on what users need to change rather than internal implementation details
- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.

### Pull Requests

When creating PRs, follow the template in `.github/pull_request_template.md`:

**PR Title Format** (from [Contributing Guide](docs/CONTRIBUTING.md#pr-title-format)):

```
<type>[optional scope]: <description>
```

Types: `feat`, `fix`, `refactor`, `docs`, `test`, `perf`, `style`, `build`, `ci`, `chore`

Example: `feat(parser): add ability to parse arrays`

**Checklist** (verify before submitting):

- [ ] PR conforms to the Contributing Guideline (especially PR Title Format)
- [ ] Links to related issues (if applicable)
- [ ] Tests added/updated (if applicable)
- [ ] Docs added/updated (if applicable)
- [ ] Breaking changes documented in `docs/how/updating-datahub.md`

### Confidentiality in Committed Code

DataHub is a **public repository**. Never put customer-identifiable or
environment-specific details into committed code, tests, docs, comments, commit
messages, or PRs:

- No real database / schema / table / view / column names, and no usernames,
  customer names, host names, account IDs, or URLs from customer environments.
- No Linear/Jira ticket IDs or links.
- When reproducing a customer issue in a test, use generic placeholder names
  (e.g. `my_db.my_schema.events`, `col_a`) that preserve the structural pattern
  being tested, not the customer's actual identifiers.
- Vendor/system built-ins (e.g. a platform's standard system tables) are fine,
  but prefer generic names when in doubt.
- **Never bypass git hook failures with `--no-verify`** (or any equivalent skip flag) on commit or push. A failing hook is a signal that something needs attention — stop, report the failure to the user, and confirm how to proceed. Only use `--no-verify` if the user explicitly tells you to for that specific action.

## Starting / Operating DataHub

Use `scripts/dev/datahub-dev.sh` for **ALL** environment operations.
**Do NOT use `./gradlew quickstartDebug` directly** — always use the wrapper script.

### `datahub-dev` CLI Tool

A stdlib-only Python CLI for agent-driven development. No venv needed — runs with system `python3`.

**Always use the shell wrapper as the entry point:**

```bash
scripts/dev/datahub-dev.sh <command>
```

Run `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,
`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,
`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).

### End-to-End Workflow

0. **Setup** (once): `scripts/dev/datahub-dev.sh setup` — installs Python dev environment (provides `datahub` CLI). For frontend work, also run `scripts/dev/datahub-dev.sh setup frontend`.
1. **Start**: `scripts/dev/datahub-dev.sh start`
2. **Code**: Make changes to Java/Python/frontend code
3. **Rebuild**: `scripts/dev/datahub-dev.sh rebuild --wait`
4. **Test**: `scripts/dev/datahub-dev.sh test <test-path>`
5. **Iterate**: Repeat steps 2–4

**Frontend hot-reload:** Run `scripts/dev/datahub-dev.sh frontend` to start the React dev server with hot-reload (instead of rebuilding the frontend container).

### Module-to-Container Mapping

| Source directory                  | Container                                     |
| --------------------------------- | --------------------------------------------- |
| `metadata-service/`               | `datahub-gms`                                 |
| `datahub-graphql-core/`           | `datahub-gms`                                 |
| `metadata-io/`                    | `datahub-gms`                                 |
| `datahub-frontend/`               | `datahub-frontend-react`                      |
| `metadata-jobs/mce-consumer-job/` | `datahub-mce-consumer`                        |
| `metadata-jobs/mae-consumer-job/` | `datahub-mae-consumer`                        |
| `metadata-models/`                | All (triggers full rebuild + code generation) |

### Environment Variables

Set any env var for DataHub containers via `env set` + `env restart`:

```bash
scripts/dev/datahub-dev.sh env set KEY=VALUE
scripts/dev/datahub-dev.sh env restart       # required — changes take effect on restart
scripts/dev/datahub-dev.sh env list           # show current vars and pending_restart status
```

**Do NOT** manually edit `.env` files, use `docker compose -e`, or `export` — always use the wrapper.

**GMS primary storage read pool** (optional, entity aspect DAO only): `EBEAN_READ_POOL_ENABLED` /
`CASSANDRA_READ_POOL_ENABLED` route non-locking reads to a second pool; writes and `forUpdate`
reads stay on PRIMARY. See [docs/deploy/primary-storage-read-pool.md](docs/deploy/primary-storage-read-pool.md).
`DATAHUB_READ_ONLY=true` is separate — it disables writes and does not register the read pool.

### Feature Flag Lifecycle

**All flag changes require a container restart.** Use `env set` + `env restart`:

```bash
scripts/dev/datahub-dev.sh env set SHOW_BROWSE_V2=true
scripts/dev/datahub-dev.sh env restart
```

`flag list` and `flag get` are read-only inspection tools — they show the current live values from
the running server but do not change anything.

The flag manifest at `scripts/generated/flag-classification.json` is **auto-generated**
(gitignored). Run `scripts/dev/datahub-dev.sh sync-flags` after adding fields to `FeatureFlags.java`
or after a fresh clone.

### Stopping DataHub

`scripts/dev/datahub-dev.sh stop` shuts down all containers without restarting.

When starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances
from other worktrees/compose projects that occupy the same ports.

### Remote Runners

`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine
(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.

**Configure a runner** in `~/.datahub/dev/config.json`:

```json
{
  "max_local_instances": 2,
  "max_remote_instances": 10,
  "runner": "/path/to/your-runner.sh"
}
```

Or export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.

**Remote lifecycle** (all commands work identically to local once a runner is set):

```bash
# One-time bootstrap — provisions the remote environment
scripts/dev/datahub-dev.sh setup --remote

# Start — syncs changed local files, runs quickstartDebug on the remote,
#          then sets up port tunnels so local ports reach the remote instance
scripts/dev/datahub-dev.sh start

# Stop containers only (remote compute keeps running)
scripts/dev/datahub-dev.sh stop

# Stop containers AND halt the remote compute (no billing while suspended).
# 'start' will automatically resume the instance when needed.
scripts/dev/datahub-dev.sh suspend

# All other commands (status, wait, rebuild, test, flag, env, nuke, …)
# proxy through the runner transparently — use them exactly as you would locally.
scripts/dev/datahub-dev.sh status
```

**Multi-instance management** — each git worktree gets its own isolated instance
(separate Docker project, volumes, and port assignment):

```bash
# List all registered instances (local and remote) with their ports and status
scripts/dev/datahub-dev.sh instances list

# Remove stale entries for worktrees that no longer exist
scripts/dev/datahub-dev.sh instances clean

# Print export statements for the current instance's CLI environment
eval $(scripts/dev/datahub-dev.sh shell-env)
# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)
```

**Port assignment** — each instance gets a slot; ports = base + slot × 1000:

| Slot | GMS   | Frontend | Notes                     |
| ---- | ----- | -------- | ------------------------- |
| 0    | 8080  | 9002     | First local instance      |
| 1    | 9080  | 10002    | Second local instance     |
| 2    | 10080 | 11002    | First remote instance     |
| …    | …     | …        | Each worktree is isolated |

**Backwards compatibility / opting out of isolation** — if the new per-worktree
project names cause problems (lost data in old volumes, tooling that expects
`datahub-*` container names, CI environments that don't need isolation), set
`compose_project` in `~/.datahub/dev/config.json`:

```json
{ "compose_project": "datahub" }
```

This reverts to the old single-instance behaviour: one `datahub` Docker project,
same container names, existing volumes fully accessible. The env var
`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.

**Runner interface** — a runner is any executable that speaks four verbs:

```bash
runner init                        # one-time environment bootstrap
runner sync                        # push changed local files to the remote
runner exec -- <cmd> [args...]     # execute a command in the remote workspace
runner tunnel <local:remote> ...   # set up port forwarding
runner resume                      # start compute if stopped (no-op if running)
runner suspend                     # stop containers + halt compute
```

A reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.

### Recovery Escalation

**When to use each:**

- `stop`: Just shut down DataHub — no restart, no data loss
- `reset`: GMS returns 503 and doesn't recover, frontend shows "Unable to connect", tests fail
  with connection errors
- `nuke --keep-data`: Containers in restart loops, port conflicts, `reset` didn't fix it
- `nuke`: ES index corruption, MySQL schema issues after model changes, PDL model changes needing
  clean slate, `nuke --keep-data` didn't fix it

### Structured Test Output

Set `AGENT_MODE=1` to get machine-readable JSON test reports at `smoke-test/build/test-report.json`:

```bash
AGENT_MODE=1 scripts/dev/datahub-dev.sh test tests/test_system_info.py
```

## Common Operations

These commands work against **any** DataHub instance — local dev, staging, or production.
Provide connection details via environment variables:

```bash
export DATAHUB_GMS_URL=http://localhost:8080  # or your instance URL
export DATAHUB_GMS_TOKEN=<your-token>         # omit if auth is not required
```

### Init (setup authentication)

`datahub init` writes `~/.datahubenv` with the GMS URL and an access token. Run it once before
using any other CLI commands that require authentication.

```bash
# Quickstart: local instance with default credentials
datahub init --username datahub --password datahub

# Full agent best-practices guide (defaults, env vars, all scenarios)
datahub init --agent-context
```

### GraphQL

`datahub graphql` executes queries and mutations against the DataHub GraphQL API and can
introspect the live schema to discover available operations.

```bash
# Discover what's available
datahub graphql --list-operations --format json

# Inspect a specific operation's arguments
datahub graphql --describe dataset --format json

# Preview a query before executing
datahub graphql --query "{ me { corpUser { urn } } }" --dry-run

# Execute a query
datahub graphql --query "{ me { corpUser { urn username } } }" --format json
```

For full agent best practices (discovery, dry-run, error codes, common recipes):

```bash
datahub graphql --agent-context
```

## Key Documentation

**Essential reading:**

- `docs/architecture/architecture.md` - System architecture overview
- `docs/modeling/metadata-model.md` - How metadata is modeled
- `docs/what-is-datahub/datahub-concepts.md` - Core concepts (URNs, entities, etc.)

**External docs:**

- https://docs.datahub.com/docs/developers - Official developer guide
- https://demo.datahub.com/ - Live demo environment

## Playwright UI E2E Tests

Full reference: [`e2e-test/ui/playwright/README.md`](e2e-test/ui/playwright/README.md).

### Seeding

`test.use({ featureName: 'my-feature' })` at the `describe` level auto-loads
`tests/my-feature/fixtures/data.json` via `seeding.fixture.ts` — once per worker per
feature per run. Do **not** set `featureName` for suites that create their own data
via `apiMock` or direct API calls.

## Frontend CI Checklist

This checklist is for **commit- or PR-ready** frontend work — i.e. when you're about to
commit, push, or hand off changes that are going into a PR. It is **not** required for
every intermediate edit: work that is part of a larger task, a work-in-progress branch,
or scratch experimentation that won't be committed yet can skip it. Run the relevant
commands when the change is ready to ship:

```bash
# Full lint (eslint + prettier src + type-check) for datahub-web-react
./gradlew :datahub-web-react:yarnLint

# Targeted lint-fix on a single file
./gradlew -x yarnInstall -x yarnGenerate yarnLintFix -Pfile=src/path/to/file.tsx

# Vitest unit tests (requires icon stubs — run once per clone)
node datahub-web-react/scripts/generate-lazy-icon-stubs.js
cd datahub-web-react && yarn test src/path/to/file.test.ts --run
```

`yarn type-check` in CI runs repo-wide and will surface pre-existing errors in
unrelated files. Focus on errors in files **you touched** — in particular, optional
prop calls (`prop?.(arg)`) and import aliases.

## Python Virtual Environments

Gradle tasks manage all venvs automatically. Never create, activate, or pip-install into them manually. When running smoke tests outside Gradle: `smoke-test/venv/bin/python -m pytest ...`

## Important Notes

- Entity Registry is defined in YAML, not code (`entity-registry.yml`)
- All metadata changes flow through the event streaming system
- GraphQL schema is generated from backend GMS APIs

## Learned User Preferences

- In `metadata-ingestion` connector code, avoid double-quoted string literals: hoist magic strings into module-level constants, and keep all regex in the constants file pre-compiled.
- Use Pydantic models for structured/internal data; never pass data around as tuples (hard to track).
- Split connector files by duty (`constants.py`, `models.py`, `config.py`, `client.py`, `source.py`, plus `lineage.py`/`mapper.py`/`usage.py` as needed) and match the quality/patterns of existing connectors (Power BI, Airbyte, Redshift, BigID, Grafana).
- No "AI slop": no top-of-file docstrings, and keep docstrings/comments only where strictly needed.
- Never use `TYPE_CHECKING` in connector code since the connector controls its own deps (lazy imports are fine only for opt-in features), and don't use the walrus operator.
- Prefer `self.report.warning(...)` and report counters over bare `logger` for skips and edge cases — the report also writes to the log and surfaces to operators (e.g. warn when `verify_ssl=False`, or when a referenced object is inaccessible).
- For SQL lineage, use the central `SqlParsingAggregator` (`create_lineage_from_sql_statements`) with a platform map instead of setting sqlglot dialects per-connector; mirror existing connectors for cross-platform known-URN and platform_instance/env/casing mapping, two- vs three-part names, and temp-table handling.
- For column-level lineage, don't leave edges coarse: resolve upstream/downstream schemas from the DataHub graph when available (as airbyte/bigid/matillion/informatica do), load known URNs from the platform/platform_instance/env mapping, and match columns case-insensitively. Best-effort is fine, but try everything.
- In connector code, use explicit type annotations rather than `from typing import Any` (Unions are fine when a value genuinely has multiple types), and prefer `Dict`/`List` from `typing` over the builtin `dict`/`list`.
- Connectors should surface progress during ingestion and use explicit ingestion stages (as in the dremio and snowflake connectors).
- Before committing a new connector, run its ingestion locally in debug mode to a local file to capture full logs and catch bugs; when testing against a customer environment, push secrets only to a tmp path (e.g. `/tmp/*.env`).
- When drafting prose or review comments on the user's behalf (e.g. Notion), write in his own direct, human voice — avoid AI tells like "confirmed these are real gaps".

## Learned Workspace Facts

- The user is a contributor to the public `datahub-project/datahub` repo and can create and push branches directly on `datahub-public-repo`.
- A new ingestion connector needs more than Python code: a source logo plus an integrations-page logo, UI form pieces, a `datahub.json` update, entry-point registration (`setup.py`/`pyproject.toml`), a refreshed `uv.lock`, and subtypes added to the shared subtypes module rather than defined locally.
- Avoid Python's stdlib `xml` parser due to a known vulnerability; use a safe XML library (as the HANA-related code does).
- Keep each connector in its own PR and split shared/framework changes (e.g. sqlglot helpers) into a separate PR; a connector PR's title and description must reference only that connector, not any other connector worked on in the same session.