### Csv Relation Types Plan # CSV Import/Export Enhancement for Glossary Term Relations ## Problem Statement Currently, the glossary CSV import/export only captures related term FQNs without the relation type: - **Export**: Only exports FQNs like `Glossary.Term1;Glossary.Term2` - **Import**: Hardcodes all relations to `"relatedTo"` This causes data loss when: 1. A term has `synonym`, `broader`, `narrower`, or custom relation types 2. CSV is exported and re-imported - all relation types become `"relatedTo"` ## Proposed Solution ### New CSV Format **Format**: `relationType:termFQN` pairs separated by semicolons **Examples**: ```csv # New format with relation types relatedTerms synonym:Finance.Revenue;broader:Finance.Income;narrower:Finance.Net Revenue # Backward compatible - no prefix defaults to "relatedTo" relatedTerms Finance.Revenue;Finance.Income # Mixed format (new and legacy) relatedTerms synonym:Finance.Revenue;Finance.Income;broader:Finance.Gross Income ``` ### Parsing Rules 1. If a value contains `:` and the part before `:` is a valid relation type → use that relation type 2. If no `:` or the prefix is not a valid relation type → default to `"relatedTo"` 3. Valid relation types are determined by checking `glossaryTermRelationSettings` or using defaults ### Default Relation Types | Relation Type | Description | |---------------|-------------| | `relatedTo` | Generic related term (default) | | `synonym` | Equivalent term | | `broader` | More general term | | `narrower` | More specific term | | `antonym` | Opposite meaning | | `partOf` | Component of | | `hasPart` | Contains | ## Implementation Plan ### Phase 1: Backend Changes #### 1.1 CsvUtil.java - Export Enhancement **File**: `openmetadata-service/src/main/java/org/openmetadata/csv/CsvUtil.java` **Current** (line 253-263): ```java public static List addTermRelations( List csvRecord, List termRelations) { csvRecord.add( nullOrEmpty(termRelations) ? null : termRelations.stream() .map(tr -> tr.getTerm().getFullyQualifiedName()) .sorted() .collect(Collectors.joining(FIELD_SEPARATOR))); return csvRecord; } ``` **New**: ```java public static List addTermRelations( List csvRecord, List termRelations) { csvRecord.add( nullOrEmpty(termRelations) ? null : termRelations.stream() .map(tr -> { String relationType = tr.getRelationType(); String fqn = tr.getTerm().getFullyQualifiedName(); // Only include relation type prefix if not the default "relatedTo" if (relationType != null && !relationType.equals("relatedTo")) { return relationType + ":" + fqn; } return fqn; }) .sorted() .collect(Collectors.joining(FIELD_SEPARATOR))); return csvRecord; } ``` #### 1.2 GlossaryRepository.java - Import Enhancement **File**: `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/GlossaryRepository.java` **Current** (line 315-327): ```java private List getTermRelationsFromCsv( CSVPrinter printer, CSVRecord csvRecord, int fieldNumber) throws IOException { List entityRefs = getEntityReferences(printer, csvRecord, fieldNumber, GLOSSARY_TERM); if (entityRefs == null) { return null; } List termRelations = new ArrayList<>(); for (EntityReference ref : entityRefs) { termRelations.add(new TermRelation().withTerm(ref).withRelationType("relatedTo")); } return termRelations; } ``` **New**: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### 1.3 Documentation Update **File**: `openmetadata-service/src/main/resources/json/data/glossary/glossaryCsvDocumentation.json` Update the `relatedTerms` field documentation: ```json { "name": "relatedTerms", "required": false, "description": "Related glossary terms with optional relation types. Format: 'relationType:FQN' or just 'FQN'. Multiple values separated by ';'. Valid relation types: relatedTo (default), synonym, broader, narrower, antonym, partOf, hasPart. Example: 'synonym:Glossary.Term1;broader:Glossary.Term2;Glossary.Term3'", "examples": [ "Glossary.Term1;Glossary.Term2", "synonym:Glossary.Term1;broader:Glossary.Term2", "synonym:Glossary.Revenue;Glossary.Income;narrower:Glossary.Net Revenue" ] } ``` ### Phase 2: Testing #### 2.1 Unit Tests **File**: `openmetadata-service/src/test/java/org/openmetadata/csv/CsvUtilTest.java` ```java @Test void testAddTermRelationsWithRelationType() { // Test that relation types are included in export } @Test void testAddTermRelationsDefaultRelationType() { // Test that "relatedTo" terms don't include prefix } ``` #### 2.2 Integration Tests **File**: `openmetadata-service/src/test/java/org/openmetadata/service/resources/glossary/GlossaryTermResourceTest.java` ```java @Test void testGlossaryTermCsvImportWithRelationTypes() { // Test importing CSV with relation type prefixes } @Test void testGlossaryTermCsvExportWithRelationTypes() { // Test exporting terms with various relation types } @Test void testGlossaryTermCsvBackwardCompatibility() { // Test importing old format CSV (no relation types) } @Test void testGlossaryTermCsvRoundTripWithRelationTypes() { // Test that export -> import preserves relation types } ``` ### Phase 3: Edge Cases 1. **FQN contains colon**: Handle cases like `Database:Schema.Term` by validating the prefix against known relation types 2. **Invalid relation type**: If prefix is not a valid relation type, treat entire string as FQN with default `relatedTo` 3. **Empty relation type**: `":Glossary.Term"` should default to `relatedTo` 4. **Custom relation types**: Check against `glossaryTermRelationSettings` for user-defined relation types ### Backward Compatibility | CSV Format | Import Behavior | |------------|----------------| | `Glossary.Term1;Glossary.Term2` | All relations → `relatedTo` | | `synonym:Glossary.Term1;Glossary.Term2` | First → `synonym`, Second → `relatedTo` | | `synonym:Glossary.Term1;broader:Glossary.Term2` | Preserves both relation types | ### Files to Modify | File | Change | |------|--------| | `CsvUtil.java` | Update `addTermRelations()` to include relation type prefix | | `GlossaryRepository.java` | Update `getTermRelationsFromCsv()` to parse relation types | | `glossaryCsvDocumentation.json` | Update field documentation and examples | | `GlossaryTermResourceTest.java` | Add tests for new format | | `CsvUtilTest.java` | Add unit tests for parsing | ### Migration Notes - **No database migration needed**: The database already stores relation types correctly - **Existing CSVs**: Will continue to work (all imported as `relatedTo`) - **New exports**: Will include relation type prefixes for non-default relations ## Summary This enhancement: 1. ✅ Preserves relation types during CSV export/import 2. ✅ Maintains backward compatibility with existing CSVs 3. ✅ Defaults to `relatedTo` when no relation type specified 4. ✅ Follows existing OpenMetadata CSV patterns (`type:value`) 5. ✅ Supports custom relation types via settings --- ### Design Patterns # Design Patterns in this Codebase — a guideline for coding agents The Gang-of-Four patterns that OpenMetadata **actually uses as established conventions**, each with the canonical place to copy the idiom from. This is not a pattern tutorial and not a catalogue of all 23 — it lists only the patterns that recur idiomatically here, so that when you add code you extend the pattern the codebase already relies on instead of inventing a parallel one. ## How to use this - **Follow the local idiom.** When a situation matches one below, mirror the shape of the cited canonical example (same interface, same registration, same lifecycle hooks). Consistency across ~60 entity repositories and ~98 connectors is worth more than a cleverer one-off. - **Don't over-apply.** A pattern earns its place only when it removes real duplication or decouples a real seam. Prefer the simplest thing that works; do not add a factory/registry/strategy layer for a single implementation (see `CLAUDE.md` on avoiding needless abstraction). - **Match the language.** Java leans on interfaces + factories + registries + Lombok; Python on `classmethod` constructors, `singledispatch`, generators, and decorator registries; the UI on `*ClassBase` singletons and React context. Use the host language's mechanism, not a Java-ism in Python. - Paths below are verified and kept fresh by the harness dead-reference check (`make harness-check`). --- ## Java backend (`openmetadata-service`, `common`, `openmetadata-sdk`) ### Creational **Factory Method / Abstract Factory** — construct one of several implementations behind a common type, chosen by config/provider; callers never `new` the concrete class. - *Use it when* you add a provider/engine variant (a secrets backend, a search engine, a connection type): register the impl and let the factory pick it. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java` (switch on provider), `openmetadata-service/src/main/java/org/openmetadata/service/secrets/converter/ClassConverterFactory.java` (map of 40+ converters), `openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepositoryFactory.java` (JDK `ServiceLoader` SPI). **Abstract Factory** (parallel ES vs OS product families) lives under `openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/` and `openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/`. **Builder** — assemble a complex object by named fields instead of a telescoping constructor. - *Use it when* building a schema POJO/DTO (Lombok `@Builder`) or an entity through the SDK (the fluent `*Builder`). - *Here:* Lombok `@Builder` on data objects (112 sites); the hand-written fluent builders in `openmetadata-sdk/src/main/java/org/openmetadata/sdk/fluent/builders/` (32 classes); `openmetadata-service/src/main/java/org/openmetadata/service/util/OpenMetadataConnectionBuilder.java`. **Singleton / Registry** — one shared instance, plus a registry mapping a key (entity name, type) to its handler, populated at startup and resolved at runtime. This is how the backend dispatches polymorphically without giant `switch` statements. - *Use it when* you add an entity type or handler: self-register into the existing registry rather than hard-coding a lookup. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/Entity.java` (`registerEntity` / `getEntityRepository`), `openmetadata-service/src/main/java/org/openmetadata/service/TypeRegistry.java`, `openmetadata-service/src/main/java/org/openmetadata/service/resources/CollectionRegistry.java`. ### Structural **Adapter** — wrap an external/vendor API in the repo's own interface so the rest of the code stays vendor-agnostic. - *Use it when* integrating a second implementation of an external dependency: implement the internal interface, never leak the vendor type upward. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java` (internal API) with `openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java` and `openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchClient.java` adapting the relocated ES/OS SDKs. **Facade** — one entry point over a large subsystem. - *Use it when* you need persistence: go through the DAO facade, not ad-hoc JDBI handles. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java` aggregates 129 sub-DAOs behind one interface. **Proxy (caching)** — a stand-in that transparently adds caching in front of the real object. - *Use it when* reading entities/subjects on hot paths: read through the existing **bounded** caches, don't hit the DB directly (see `CLAUDE.md`: all caches must be bounded). - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java` (the `CACHE_WITH_NAME`/`CACHE_WITH_ID` Guava `LoadingCache`), `openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/SubjectCache.java`. **Composite** — model part-whole trees so a leaf and a container are handled uniformly. - *Use it when* working with hierarchies (teams, glossary terms/domains, FQN containment): reuse the existing recursive helpers rather than re-walking the tree by hand. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/SubjectContext.java` (team tree for RBAC), `openmetadata-service/src/main/java/org/openmetadata/service/util/FullyQualifiedName.java` (service→database→schema→table). **Decorator** — wrap an object to add behavior while keeping the same interface. *Narrow here — one genuine case.* - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/socket/HeaderRequestWrapper.java` (a servlet-request wrapper). Note: the `formatter/decorators/` package is *named* "decorator" but is structurally **Strategy** (one impl per channel, nothing wraps another) — do not treat it as a Decorator. ### Behavioral **Template Method** — a base class fixes the algorithm skeleton and delegates the variable steps to abstract hooks the subclass fills. **The backbone of the backend.** - *Use it when* adding an entity repository or app: extend the base and implement the hooks (`setFields`, `prepare`, `storeEntity`, …); never reimplement the CRUD lifecycle. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java` (~60 repositories), `openmetadata-service/src/main/java/org/openmetadata/service/apps/AbstractNativeApplication.java`. **Strategy** — interchangeable algorithms behind a common interface, selected at runtime. - *Use it when* there is a family of "same operation, different implementation" (auth, secrets backend, per-type conversion): add an impl and register it. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/security/Authorizer.java`, `openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManager.java`, `openmetadata-service/src/main/java/org/openmetadata/service/secrets/converter/ClassConverter.java`. **Observer / Publish-Subscribe** — emit events; decoupled subscribers react. - *Use it when* something should happen on entity change or per request: add a handler/subscription, don't wire the caller directly to the reaction. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/events/EventHandler.java`, the change-event publisher family (`openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/changeEvent/AlertPublisher.java`), `openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java`. **Chain of Responsibility** — an ordered series of handlers, each doing its part and passing control on. - *Use it when* adding a request filter or a migration phase: insert into the existing ordered chain. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/security/DelegatingContainerRequestFilter.java` (JAX-RS filter chain), `openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationWorkflow.java` (Flyway → native → extension). **Command** — encapsulate an action as an object that can be scheduled/queued/run later. - *Use it when* adding a background job or a migration step: implement the job/step interface. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/apps/NativeApplication.java` (Quartz jobs), `openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationProcess.java`. **Visitor (parse-tree)** — separate an operation from the object structure it traverses; here, ANTLR parse trees. - *Use it when* parsing/transforming FQNs, entity links, or JDBC URIs: use the ANTLR listeners, don't hand-roll string splitting. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/util/FullyQualifiedName.java` (`SplitListener` over the tree); grammars in `openmetadata-spec/src/main/antlr4/`. **Iterator (cursor)** — traverse a large/remote collection page-by-page without exposing storage. - *Use it when* processing large entity sets (reindex, insights) or paginating a list API: use cursor-based sources / `ResultList`, never load everything into memory. - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/workflows/interfaces/Source.java` and `EntityRepository` keyset pagination. **State** — allowed transitions depend on an explicit status; illegal transitions are rejected. *Narrow here.* - *Here:* `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseResolutionStatusRepository.java` (incident lifecycle). Broader workflow state is delegated to the Flowable BPMN engine, not hand-rolled — don't build a new state machine when a governance workflow fits. --- ## Python ingestion framework (`ingestion/src/metadata`) **Factory Method — the `create()` contract.** Every Step/Source is built through a `create()` classmethod that validates the service connection and raises `InvalidSourceException` on mismatch; never construct a connector directly. Mandatory for all 135+ connector steps. - *Here:* `ingestion/src/metadata/ingestion/api/step.py` (`Step.create`), `ingestion/src/metadata/ingestion/source/database/postgres/metadata.py`. **Abstract Factory / Registry — the ServiceSpec system.** A per-connector `service_spec.py` manifest declares the classpaths of its source/profiler/sampler; the framework imports them dynamically, so the workflow is decoupled from every connector. - *Use it when* adding a connector: ship a `service_spec.py`, don't wire the class into the framework. - *Here:* `ingestion/src/metadata/utils/service_spec/service_spec.py` (`BaseSpec`, `import_source_class`); 98 `service_spec.py` manifests. **Factory + decorator Registry.** Decorator-based registries map a key (enum value, SQL dialect, class name) to a handler resolved at runtime. - *Use it when* adding a parser/metric/adaptor variant: register it via the shared primitive rather than editing a dispatch `if/elif`. - *Here:* `ingestion/src/metadata/utils/dispatch.py` (`enum_register` / `class_register`), `ingestion/src/metadata/profiler/factory.py`. **Adapter — entity adapters.** Adapt heterogeneous entity shapes (Table/Container/Topic) to one uniform interface, replacing scattered `isinstance` checks. - *Use it when* adding classification/sampling support for a new entity type: add an `@register_adapter` adapter. - *Here:* `ingestion/src/metadata/sampler/entity_adapters.py`. **Template Method.** The base runs the fixed lifecycle (`run` / `execute`) and calls abstract hooks (`_run`/`_iter`, `execute_internal`, the topology `yield_*`/`get_*` methods). - *Use it when* adding a step or workflow: fill the hooks, don't reimplement the lifecycle. - *Here:* `ingestion/src/metadata/ingestion/api/step.py`, `ingestion/src/metadata/ingestion/source/database/database_service.py`, `ingestion/src/metadata/workflow/base.py`. **Strategy via `singledispatch`.** Dispatch on the runtime type of a record instead of an `isinstance` ladder (9 files use `singledispatchmethod`, 22 use `singledispatch`). - *Here:* `ingestion/src/metadata/ingestion/api/topology_runner.py`. **Iterator — generators.** A Source is a generator; records stream lazily and the workflow pulls them with `for record in source.run()`. Produce records with `yield`, don't build a list. - *Here:* `ingestion/src/metadata/ingestion/api/step.py` (`IterStep`), `ingestion/src/metadata/ingestion/api/topology_runner.py`. **Chain / Pipeline — Source→Processor→Stage→Sink.** Each record flows through an ordered tuple of steps that transform/forward/drop it; a workflow composes its own chain in `set_steps()`. - *Here:* `ingestion/src/metadata/workflow/ingestion.py`, `ingestion/src/metadata/ingestion/api/steps.py`. **Observer — Status handler.** Log warnings emitted anywhere during a step's `run()` are observed by a handler and recorded into that step's `Status`, decoupling emitters from the aggregator. - *Here:* `ingestion/src/metadata/ingestion/api/status.py`, `ingestion/src/metadata/ingestion/api/step.py`. --- ## TypeScript UI (`openmetadata-ui/src/main/resources/ui/src`) **Singleton + Factory-Method — the `*ClassBase` seam.** A single instance is exported as the app-wide singleton, and the class is also exported so the enterprise build can subclass and override its factory methods (which return components, field configs, and widgets). This is the deliberate UI plugin/override point — 40 such modules. - *Use it when* adding a UI extension point the enterprise build may override: follow the `*ClassBase` shape. - *Here:* `openmetadata-ui/src/main/resources/ui/src/utils/EntityRightPanelClassBase.ts` (and siblings like `TableClassBase`, `SearchClassBase`). **Factory function.** Build the right component subtree from a discriminant enum. - *Here:* `openmetadata-ui/src/main/resources/ui/src/components/Auth/AuthProviders/AuthProvider.tsx` (switch on auth provider). > React **Context/Provider** (`openmetadata-ui/src/main/resources/ui/src/context`) is the idiomatic > composition/state-sharing seam (24 contexts). It resembles GoF Mediator only loosely — treat it as > the React idiom, not as a GoF pattern to reach for by name. --- ## Cautions - **Naming ≠ structure.** `formatter/decorators/` is Strategy, not Decorator. Verify a candidate's structure, not its class name, before matching it to a pattern. - **Don't force GoF onto React.** Hooks, context, and composition cover most of what Strategy/Observer would in an OO codebase; a `*ClassBase` singleton or a factory function is usually the only GoF shape worth naming on the UI. - **A pattern is a means, not a goal.** If a plain function or a `switch` is clearer and there is only one implementation, use it — and revisit the pattern when a second implementation actually arrives. --- ### Golden Principles # Golden Principles (DRAFT — for human ratification) Candidates drawn from a one-time repository audit's strongest signals: the three most-consistent inter-module/import dependency rules, and the conventions measured at high adherence with a clean number. Nothing here without a **measured number** (the reproducing command is in each row). **Selection rule.** The candidate pool was cut to the intersection of **high adherence AND high cost-of-violation**, capped at ten ("more than ten means preferences"). That drops four low-cost, auto-fixed formatting/hygiene lints — **spotless (100%)**, **ruff format (100%)**, **no-wildcard imports (98.8%)**, **no-console (99.96%)** — which are lints to gate, not principles (see the promotion list). **8 remain.** Ordered by cost-of-violation (highest first). | # | Principle | Failure mode it prevents | Detection command | Adherence (measured) | |---|---|---|---|---| | 1 | **Modules form an acyclic, downward-only dependency graph** | Circular build/reasoning; a change in a "leaf" forcing a rebuild of the "root"; loss of module boundaries | parse each module POM's `org.open-metadata` deps for a back-edge; or `maven-enforcer` `banCircularDependencies` | **0 cycles / 12 modules = 100%** | | 2 | **Every connector satisfies the ServiceSpec plugin contract** (`{__init__,service_spec,metadata,connection}.py` + a `Source` subclass of a `Common*`/`DatabaseServiceSource` base + `create()` raising `InvalidSourceException` + a top-level `ServiceSpec`) | A connector that silently fails to load/register at runtime | per connector dir: `find … -name service_spec.py`; assert `ServiceSpec` importable + the four files present | **~97 connectors, all registered = 100%**; **94/95 `metadata.py` carry `create()`+`InvalidSourceException` = 98.9%** | | 3 | **Generated code is a pure sink; source imports it only as types** (no generated file imports app/domain code; no runtime `import metadata.generated…`) | Hand-edits / runtime coupling that the next `make generate` overwrites → drift, CI reverts, broken builds | `grep -rlE "from '(\.\./)+(components\|pages\|rest\|utils\|hooks\|context)/" openmetadata-ui/.../src/generated` → **0**; `grep -rn "^import metadata.generated" ingestion/src` → **0** | **0 app-imports (100% sink)**; source→generated **1,736/1,738 = 99.9%** type-only · *now hook-enforced (edit-block)* | | 4 | **No new type errors** (basedpyright ratchet — the baseline only shrinks) | Type regressions shipping as runtime bugs | `make static-checks` (nox → basedpyright `--baselinemode=discard`) | ratchet over **11,927 baselined findings**; **0 new** required; CI-gated | | 5 | **No bare `except:`** (catch a specific type, or at minimum name the exception) | Swallowing `KeyboardInterrupt`/`SystemExit`; hiding every error | `grep -rnE 'except\s*:' ingestion/src/metadata`; or ruff `E722` | **2,074/2,075 = 99.95%** (1 file) | | 6 | **Functional React components only** (no class components) | Hooks-incompatible components; two divergent component models | `grep -rlE 'extends (React\.)?(Component\|PureComponent)\b' openmetadata-ui/.../src` | **0 violations = 100%** | | 7 | **Apache-2.0 license header on every new source file** | OSS licensing / compliance gaps | `license-check-and-add check` (UI); header grep on newly-added files | **99.75%** (12/4,751 miss; 9 are generated `.js`) · *now hook-enforced for new UI files* | | 8 | **Parameterized logging, never string concatenation** (`LOG.x("… {}", var)`) | Log-injection; needless string building; unstructured logs | `grep -rnE '(log\|LOG\|logger)\.(info\|warn\|error\|debug\|trace)\([^;]*"\s*\+' …/src/main/java` | **0/5,989 = 100%** — the lowest-cost of the eight; flagged for the ratifier | ## Conflicts to resolve before ratifying (surfaced, not decided) - **#3 is both a strength and the biggest debt.** The `generated/` tree obeys the sink rule (imports nothing from the app), yet the app violates the *inbound* side massively — 1,292 components/pages import generated types directly vs 93 in `rest/`. The principle as stated ("generated imports nothing") holds at 100%; a stricter version ("app imports generated only through an API layer") is at ~7%. **Decide which one you're ratifying.** - **Principles 5 and 8 are Python/Java-local.** The sibling rule "avoid broad `except Exception`" is a non-principle — 75.5% of Python handlers are broad, by design. Do not generalize #5 into "no broad catch" across languages; Python sanctions it. - **#4 is a ratchet, not an invariant.** It reads as a principle but the code has 11,927 baselined findings. Ratifying "type-clean" would misrepresent the tree; ratify "no *new* type errors." ## Explicitly NOT principles (measured, but disqualified) None had both high adherence and a clean measurement: - **Nobody follows it:** avoid broad `except Exception` (24.5% — the deliberate ingestion idiom). - **Low adherence / not cleanly measurable:** avoid Ant Design (81.7% antd-free, migration stalled), no `any` (90.2%), `.component.tsx` naming (36.4% proxy), wrap JSX strings in `t()` (unmeasurable), non-en locale translation (~90–94%), comments explain-why (unmeasurable), 90% coverage (ungated). --- ### Impersonation Design # Bot Impersonation Feature - Design Document ## 1. Overview ### 1.1 Purpose Enable bots in OpenMetadata to impersonate users when performing actions, ensuring proper attribution of changes to the actual user while maintaining audit trail of bot involvement. ### 1.2 Background Currently, when bots perform actions (e.g., ingestion pipelines, automation workflows), the `updatedBy` field shows the bot's name. This obscures who actually initiated the action. With impersonation, we can track: - **updatedBy**: The actual user who initiated the action - **impersonatedBy**: The bot that executed the action on behalf of the user ### 1.3 Goals - Allow bots to act on behalf of users with proper authorization - Maintain complete audit trail showing both user and bot - Use impersonated user's permissions for authorization checks - Provide secure, policy-based control over impersonation capabilities - Display impersonation context in UI and activity feeds ### 1.4 Non-Goals - User-to-user impersonation (only bot-to-user) - Impersonation for authentication purposes - Bypassing authorization checks ## 2. Design Principles 1. **Security First**: Only authorized bots can impersonate, with strict validation 2. **Transparency**: All impersonated actions are clearly visible in audit logs and UI 3. **Permission Inheritance**: Use impersonated user's permissions, not bot's 4. **Backward Compatibility**: Existing entities without impersonation continue to work 5. **Auditability**: Complete trail of who did what via which bot ## 3. Architecture ### 3.1 Token Exchange Flow (On-Demand Impersonation) Since JWT tokens are pre-generated and bots cannot dynamically create tokens with different `impersonatedUser` claims, we need a **token exchange API** that allows bots to obtain short-lived impersonation tokens on-demand. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 3.2 Component Details #### 3.2.1 JWT Token Structure **Standard Token (No Impersonation)** ```json { "sub": "alice", "email": "alice@example.com", "isBot": false, "exp": 1234567890 } ``` **Impersonation Token** ```json { "sub": "ingestion-bot", "email": "ingestion-bot@example.com", "isBot": true, "impersonatedUser": "alice", "exp": 1234567890 } ``` #### 3.2.2 Security Context **CatalogSecurityContext Extension** ```java public record CatalogSecurityContext( Principal principal, // The bot String scheme, String authenticationScheme, Set userRoles, boolean isBot, String impersonatedUser // NEW: The actual user ) implements SecurityContext ``` **SubjectContext Enhancement** ```java public record SubjectContext( User user, // The impersonated user (or actual user) String impersonatedBy // NEW: The bot name (if impersonating) ) ``` ## 4. Authorization Model ### 4.1 Three-Level Security Check #### Level 1: Bot Capability Flag ```java // User entity schema { "name": "ingestion-bot", "isBot": true, "allowImpersonation": true // NEW: Must be explicitly enabled } ``` #### Level 2: Impersonation Permission ```java // New MetadataOperation enum value public enum MetadataOperation { // ... existing operations IMPERSONATE // NEW: Permission to impersonate users } ``` #### Level 3: Policy-Based Control ```json { "name": "IngestionBotImpersonationPolicy", "description": "Allow ingestion bot to impersonate users in Engineering domain", "rules": [ { "name": "ImpersonateEngineeringUsers", "resources": ["user"], "operations": ["Impersonate"], "effect": "allow", "condition": "hasDomain('Engineering')" } ] } ``` ### 4.2 Authorization Flow ```java // Pseudo-code for validation in JwtFilter void validateImpersonation(String botName, String targetUser) { // 1. Bot must have isBot=true (already validated) // 2. Check bot has allowImpersonation flag User bot = Entity.getEntityByName(USER, botName, "allowImpersonation", NON_DELETED); if (!Boolean.TRUE.equals(bot.getAllowImpersonation())) { throw new AuthorizationException("Bot not authorized to impersonate"); } // 3. Check policy grants IMPERSONATE permission SubjectContext botContext = SubjectContext.getSubjectContext(botName); User targetUserEntity = Entity.getEntityByName(USER, targetUser, "", NON_DELETED); ResourceContext resourceContext = new ResourceContext(USER, targetUserEntity.getId(), null); OperationContext operationContext = new OperationContext(USER, MetadataOperation.IMPERSONATE); // This throws AuthorizationException if not permitted PolicyEvaluator.hasPermission(botContext, resourceContext, operationContext); // 4. Validation passed - impersonation allowed } ``` ### 4.3 Permission Evaluation When a bot impersonates a user, authorization checks use the **impersonated user's permissions**, not the bot's: ```java // In DefaultAuthorizer public static SubjectContext getSubjectContext(SecurityContext securityContext) { CatalogSecurityContext catalogContext = (CatalogSecurityContext) securityContext; // Use impersonated user if present, otherwise use principal String userName = catalogContext.impersonatedUser() != null ? catalogContext.impersonatedUser() : catalogContext.getUserPrincipal().getName(); User user = Entity.getEntityByName(Entity.USER, userName, USER_FIELDS, NON_DELETED); // Track who is impersonating (if applicable) String impersonatedBy = catalogContext.impersonatedUser() != null ? catalogContext.getUserPrincipal().getName() : null; return new SubjectContext(user, impersonatedBy); } ``` **Example**: If `ingestion-bot` impersonates `alice`: - CRUD operations are authorized using alice's roles/teams/policies - If alice lacks permission to update a table, the operation fails - This prevents privilege escalation ### 4.4 Bot Impersonation Grants and RBAC Scoping (v1.1 — June 2026) This section supersedes the parts of 4.1/4.2 that the initial implementation simplified, and addresses two issues: - **OpenMetadata #28043**: custom bots cannot enable impersonation. The `allowImpersonation` flag exists only on the `user` entity, is not part of `createUser.json`/`createBot.json`, and PATCHing it on the bot user fails (500 in 1.12.x). Only application bots (`CreateApp.allowBotImpersonation`) could ever receive the flag. - **Collate #3581**: bots with impersonation enabled can impersonate **admin** users. Admins cannot be restricted via RBAC, so this is privilege escalation. Blocking it outright breaks workflows that legitimately need it (e.g. AskCollate), so the control must be configurable. #### 4.4.1 Design Decision: Single Flag + RBAC Scope The split of responsibility is: | Question | Mechanism | |----------|-----------| | *Can this bot impersonate at all?* | `allowImpersonation` boolean on the bot user — an admin-granted capability, set at bot creation | | *Who can this bot impersonate?* | Standard policy evaluation of the `Impersonate` operation with the **target user as the resource** | A second boolean (`allowAdminImpersonation`) was considered and rejected: target scoping belongs in policies/rules, where it is already expressive (conditions, deny rules, teams, domains) and admin-manageable without code changes. #### 4.4.1.1 Why a dedicated flag and not RBAC alone This question recurs ("`Impersonate` is already an operation — drop the flag, grant it through a role like everything else"). It has been evaluated and **rejected**. The flag and RBAC answer two different questions, and collapsing them opens a real privilege-escalation gap: - **The flag is the enablement-authority gate**: *may this bot impersonate at all?* — admin-only, set at bot creation, enforced in `DefaultAuthorizer` **independently of RBAC**. - **RBAC is the target-scope gate**: *whom may it impersonate?* — policies, conditions, deny rules. If enablement were pure RBAC, "who can enable impersonation" would collapse into "who can grant the `Impersonate` operation" — i.e. anyone with `EditRoles` on a bot user or `EditPolicy`+`Create` on policies. **That permission set is routinely delegated to non-admins** (orgs delegate RBAC management). Impersonation — which lets a bot act as any user, including bypassing controls admins can't otherwise be restricted by — would then become grantable by non-admins. The flag keeps enablement admin-only **regardless of how RBAC editing is delegated**. The flag is also the *robust* control, not merely an extra layer. Under RBAC-only, the `Impersonate` grant can arrive through **many** paths — a direct role, a team's `defaultRoles` the bot inherits, an inherited policy, or a broad `["All"]`-operations policy (subsumption). Guaranteeing "admin-only" would mean guarding every one of those write paths. The flag is instead a **single chokepoint**: one admin-only, creation-time write, checked before any policy evaluation. It is belt-and-suspenders — **even if `Impersonate` leaks into a bot's effective policy, no flag means no impersonation.** Two facts bound the surface and explain what the flag does and does not cover: - `JwtFilter` rejects the `X-Impersonate-User` header for any non-bot principal (`"Only bot users can impersonate other users"`). So a **regular user self-granting** `Impersonate` achieves nothing — they are not a bot. By default they also cannot edit RBAC. This case is already closed without the flag. - The case the flag uniquely closes is a **non-admin RBAC delegate enabling impersonation on a bot**. RBAC-only would permit it; the flag does not. Conclusion: the flag is not redundant with RBAC — it is the admin-only, single-chokepoint enablement gate that RBAC delegation cannot widen. Do not remove it without replacing this property. The only redundancy ever identified was cosmetic: `BotImpersonationRole` is auto-attached when the flag is set, so the two travel together for UX. That convenience does not make the flag itself redundant. #### 4.4.2 Granting the Capability (`createBot.json`) `createBot.json` gains an optional `allowImpersonation` boolean with **tri-state semantics** (no schema default, so an absent field is `null`): - `null` (absent) — keep the bot user's current value. Critical for PUT-based upserts (ingestion re-applies bots via `PUT /v1/bots`); an absent field must not silently revoke the grant. - `true` — grant. **Admin-only**, and **only when the bot is being created**. Enabling impersonation on an existing bot is rejected with 400: an existing bot's token is already distributed, and flipping the flag would silently upgrade every holder of that token. Granting at creation forces a new bot + new token + deliberate admin action. - `false` — revoke. Admin-only, allowed at any time (privilege reduction). When the grant is applied, `BotResource` propagates the flag to the **bot user** entity (single source of truth — the authorizer reads it from the user) and attaches the seeded `BotImpersonationRole` to the bot user so the capability works out of the box. Revoking removes the flag and detaches that role. `User.allowImpersonation` becomes effectively read-only on the user APIs: - `PATCH /v1/users` changing it → 400 `user attribute allowImpersonation can't be modified` (replaces the unhandled 500 from #28043). - `PUT /v1/users` always carries over the stored value (fixes a latent bug where any PUT on a bot user nulled the flag, because `CreateUser` has no such field). #### 4.4.3 Scoping Who Can Be Impersonated (policies) `checkImpersonationAuthorization` in `DefaultAuthorizer` evaluates, on every impersonated request: 1. The impersonating principal is a bot with `allowImpersonation=true` (capability gate, unchanged). 2. `PolicyEvaluator.hasPermission(botSubjectContext, targetUserResourceContext, OperationContext(user, IMPERSONATE))` — full policy evaluation, with the target user as the resource. This replaces the previous flat scan of role policies for the `Impersonate` operation, which ignored rule `effect` (a `deny` rule counted as allow) and could not discriminate targets. Because this is the standard evaluation path, deny-overrides-allow, SpEL conditions, and compiled-rule caching all apply. Two new condition functions make target discrimination possible: | Function | True when | |----------|-----------| | `isAdminUser()` | the resource (target user) is an admin | | `isBotUser()` | the resource (target user) is a bot | Existing functions compose for finer scoping: `matchTeam()` (team-scoped impersonation), `hasDomain()`, `matchAnyTag()`. The `Impersonate` operation is registered as a `user`-resource operation, so the policy editor offers it on the `user` resource and the functions appear in the condition dropdown (`/v1/policies/functions`). `Impersonate` is **explicit-grant-only**: `CompiledRule.matchOperation` excludes it from `ALL`/`EditAll`/`ViewAll` subsumption. A broad god-mode policy (`operations: ["All"]`) therefore neither grants nor *appears* to grant impersonation; only a rule naming `Impersonate` does. With the flag in place this is defense-in-depth (the flag already gates enablement), but it keeps permission listings honest and prevents impersonation from silently riding along with broad policies. #### 4.4.4 Seed Policies and Defaults - **`BotImpersonationPolicy`** (new, the default): allow `Impersonate` on `user`, **no deny rules**. **Backward-compatible by default**: a freshly granted bot can impersonate any user, *including admins*, matching the pre-existing `ApplicationBotImpersonationPolicy` (`All`/allow) behavior. This is deliberate — silently denying admin impersonation would break workflows (e.g. AskCollate) that rely on it. - **`BotImpersonationRole`** (new): `DefaultBotPolicy` + `BotImpersonationPolicy`. Auto-attached at grant time. - **`BotNonAdminImpersonationPolicy`** (new, opt-in restriction): allow `Impersonate` on `user` + **deny `isAdminUser()`**. Admins who want to prevent a bot from impersonating admins attach this instead of the permissive default. - **`BotNonAdminImpersonationRole`** (new): `DefaultBotPolicy` + `BotNonAdminImpersonationPolicy`. The directly-assignable form of the opt-in restriction — swap a bot's `BotImpersonationRole` for this to deny admin targets. - **`ApplicationBotImpersonationPolicy`** (existing, `All`/allow) is unchanged: application bots created with `allowBotImpersonation` (e.g. AskCollate) keep their current behavior, including admin impersonation. Restriction is therefore **opt-in**, not default — the "configurable policy control" Collate #3581 asks for, delivered without breaking any bot that currently impersonates admins. The `isAdminUser()` / `isBotUser()` condition functions and deny-overrides-allow semantics let admins author any narrower scope (per-team, per-domain, deny-bot, etc.). **No migration needed.** All four impersonation seeds (`BotImpersonationPolicy`, `BotNonAdminImpersonationPolicy`, and their roles) are new in this change — no released version has them. Seed loading is insert-if-missing, so a fresh install and any upgrade from a prior release both get the current (permissive) JSON seeded directly. There is no prior on-disk state to convert, so no Flyway migration or startup reconciliation is required. #### 4.4.5 Threat-Model Delta | Threat | Before | After | |--------|--------|-------| | Bot impersonates admin | Allowed for any impersonation bot | Allowed by default (backward-compatible); deniable opt-in via `BotNonAdminImpersonationPolicy` (`isAdminUser()` deny) | | Bot impersonates another bot (token laundering) | Allowed | Allowed by default; deniable via an `isBotUser()` deny rule in a custom policy | | Flag flipped on live bot via PATCH | 500, undefined behavior (works on newer builds) | 400, read-only; grant is creation-time via `createBot` | | Flag silently wiped by PUT on bot user | Yes (latent bug) | PUT carries over stored value | | `deny` rule with `Impersonate` op | Counted as **allow** | Honored (standard evaluator) | | Non-admin RBAC delegate enables impersonation on a bot | n/a | Blocked — enablement is the admin-only flag, independent of RBAC editing (see 4.4.1.1) | | Broad `["All"]` policy grants impersonation by subsumption | Would grant | `Impersonate` excluded from `ALL`/`EditAll`/`ViewAll` subsumption; flag still required | ## 5. Data Model Changes ### 5.1 JSON Schema Updates #### 5.1.1 Base Type Definition ```json // openmetadata-spec/src/main/resources/json/schema/type/basic.json { "definitions": { "impersonatedBy": { "description": "Bot user that performed the action on behalf of the actual user.", "type": "string" } } } ``` #### 5.1.2 Entity Schema Pattern ```json // Example: openmetadata-spec/src/main/resources/json/schema/entity/data/table.json { "properties": { "updatedBy": { "description": "User who made the update.", "type": "string" }, "impersonatedBy": { "description": "Bot that performed the update on behalf of updatedBy user.", "$ref": "../../type/basic.json#/definitions/impersonatedBy" }, "updatedAt": { "$ref": "../../type/basic.json#/definitions/timestamp" } } } ``` #### 5.1.3 User Schema for Bot Capability ```json // openmetadata-spec/src/main/resources/json/schema/entity/teams/user.json { "properties": { "isBot": { "description": "When true, indicates this is a bot user.", "type": "boolean", "default": false }, "allowImpersonation": { "description": "When true, this bot is allowed to impersonate users (subject to policy checks).", "type": "boolean", "default": false } } } ``` ### 5.2 Java Entity Interface ```java // openmetadata-spec/src/main/java/org/openmetadata/schema/EntityInterface.java public interface EntityInterface { // ... existing methods String getUpdatedBy(); void setUpdatedBy(String updatedBy); // NEW: Impersonation tracking default String getImpersonatedBy() { return null; } default void setImpersonatedBy(String botName) { /* no-op implementation to be overridden */ } } ``` ### 5.3 Database Schema Migration #### MySQL Migration ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### PostgreSQL Migration ```sql -- bootstrap/sql/migrations/native/1.7.0/postgres/schemaChanges.sql -- Same as MySQL with appropriate syntax ALTER TABLE table_entity ADD COLUMN IF NOT EXISTS impersonatedBy VARCHAR(256); -- ... etc ``` ### 5.4 Elasticsearch/OpenSearch Index Mapping ```json // Update index mappings to include impersonatedBy { "mappings": { "properties": { "updatedBy": { "type": "keyword" }, "impersonatedBy": { "type": "keyword" }, "updatedAt": { "type": "date" } } } } ``` ## 6. Implementation Details ### 6.1 Token Exchange API (New Endpoint) **API Specification:** ```yaml POST /api/v1/users/impersonate Authorization: Bearer Content-Type: application/json Request Body: { "targetUser": "alice", "expirySeconds": 3600 // Optional, defaults to 1 hour, max 24 hours } Response: { "accessToken": "eyJhbGciOiJSUzI1NiIsIn...", "tokenType": "Bearer", "expiresIn": 3600, "impersonatedUser": "alice" } ``` **Implementation:** ``` /* Detailed source-code truncated for AI context efficiency. */ ``` **New Request Schema:** ```json // openmetadata-spec/src/main/resources/json/schema/auth/impersonationRequest.json { "$id": "https://open-metadata.org/schema/auth/impersonationRequest.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "ImpersonationRequest", "description": "Request to generate an impersonation token", "type": "object", "javaType": "org.openmetadata.schema.auth.ImpersonationRequest", "properties": { "targetUser": { "description": "Username of the user to impersonate", "type": "string" }, "expirySeconds": { "description": "Token expiry in seconds (default 3600, max 86400)", "type": "integer", "minimum": 60, "maximum": 86400, "default": 3600 } }, "required": ["targetUser"], "additionalProperties": false } ``` ### 6.2 JWTTokenGenerator Enhancement Add new method to generate impersonation tokens: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 6.3 JwtFilter Enhancement ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 6.3 DefaultAuthorizer Update ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 6.4 EntityRepository Update ```java // openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java public abstract class EntityRepository { protected void prepareInternal(T entity, CreateEntity request, String updatedBy) { // ... existing code ... entity.setUpdatedBy(updatedBy); entity.setUpdatedAt(System.currentTimeMillis()); // NEW: Set impersonatedBy from SubjectContext SubjectContext subjectContext = SubjectContext.getSubjectContext(updatedBy); if (subjectContext.impersonatedBy() != null) { entity.setImpersonatedBy(subjectContext.impersonatedBy()); } // ... rest of existing code ... } public final PutResponse update(UriInfo uriInfo, T original, T updated, String updatedBy) { setFieldsInternal(original, putFields); updated.setUpdatedBy(updatedBy); updated.setUpdatedAt(System.currentTimeMillis()); // NEW: Set impersonatedBy SubjectContext subjectContext = SubjectContext.getSubjectContext(updatedBy); if (subjectContext.impersonatedBy() != null) { updated.setImpersonatedBy(subjectContext.impersonatedBy()); } // ... rest of existing code ... } } ``` ## 7. Frontend Changes ### 7.1 Type Definitions (Auto-generated) ```typescript // Generated from schema export interface EntityReference { id: string; name: string; fullyQualifiedName?: string; // ... other fields } export interface Table extends EntityInterface { // ... other fields updatedBy?: string; impersonatedBy?: string; // NEW updatedAt?: number; } ``` ### 7.2 UI Components #### Entity Header Component ```tsx // Display impersonation badge const EntityHeader = ({ entity }: { entity: EntityInterface }) => { return (
{entity.impersonatedBy ? ( Updated by {entity.updatedBy} via {entity.impersonatedBy} ) : ( Updated by {entity.updatedBy} )} {formatDateTime(entity.updatedAt)}
); }; ``` #### Activity Feed ```tsx // Show impersonation in activity feed const ActivityFeedItem = ({ activity }: { activity: ChangeEvent }) => { const displayName = activity.impersonatedBy ? `${activity.userName} (via ${activity.impersonatedBy})` : activity.userName; return (
{displayName} {activity.action} {activity.entityType}
); }; ``` ### 7.3 Admin Settings UI ```tsx // Bot configuration page const BotSettings = ({ bot }: { bot: User }) => { const [allowImpersonation, setAllowImpersonation] = useState(bot.allowImpersonation); return (

Impersonation Settings

{ updateBotSettings(bot.id, { allowImpersonation: checked }); setAllowImpersonation(checked); }} />

Allow this bot to impersonate users (subject to policy restrictions)

{allowImpersonation && ( This bot can perform actions on behalf of users. Ensure appropriate impersonation policies are configured. )}
); }; ``` ## 8. Usage Examples ### 8.1 Python SDK Usage ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 8.2 Ingestion Workflow Example ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 8.3 REST API Examples #### Get Impersonation Token ```bash curl -X POST https://localhost:8585/api/v1/users/impersonate \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "targetUser": "alice", "expirySeconds": 3600 }' # Response: { "accessToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik...", "tokenType": "Bearer", "expiresIn": 3600, "impersonatedUser": "alice", "jwtTokenExpiry": { "expiresAt": 1699123456789 } } ``` #### Use Impersonation Token ```bash # Update table using impersonation token IMPERSONATION_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik..." curl -X PATCH https://localhost:8585/api/v1/tables/{table-id} \ -H "Authorization: Bearer $IMPERSONATION_TOKEN" \ -H "Content-Type: application/json-patch+json" \ -d '[{ "op": "add", "path": "/description", "value": "Updated via impersonation" }]' # Response includes: { "id": "...", "name": "dim_customer", "updatedBy": "alice", "impersonatedBy": "ingestion-bot", "updatedAt": 1699120000000 } ``` #### Verify Impersonation in Activity Feed ```bash curl -X GET "https://localhost:8585/api/v1/feed?entityLink=<#E::table::..." \ -H "Authorization: Bearer " # Response shows: { "data": [ { "id": "...", "type": "entityUpdated", "userName": "alice", "impersonatedBy": "ingestion-bot", "timestamp": 1699120000000, "changeDescription": { "fieldsUpdated": [ { "name": "description", "oldValue": "...", "newValue": "Updated via impersonation" } ] } } ] } ``` ### 8.4 Error Scenarios #### Bot Without Impersonation Permission ```bash curl -X POST https://localhost:8585/api/v1/users/impersonate \ -H "Authorization: Bearer " \ -d '{"targetUser": "alice"}' # Response: 403 Forbidden { "code": 403, "message": "Bot 'my-bot' is not authorized to impersonate users", "exceptionType": "AuthorizationException" } ``` #### Bot Without allowImpersonation Flag ```bash # Response: 403 Forbidden { "code": 403, "message": "Bot 'my-bot' is not authorized to impersonate users", "exceptionType": "AuthorizationException" } ``` #### Policy Restriction Violation ```bash # Bot tries to impersonate user from different domain curl -X POST https://localhost:8585/api/v1/users/impersonate \ -H "Authorization: Bearer " \ -d '{"targetUser": "bob-from-marketing"}' # Response: 403 Forbidden { "code": 403, "message": "Bot 'engineering-bot' is not authorized to impersonate user 'bob-from-marketing'", "exceptionType": "AuthorizationException" } ``` #### Non-Bot User Attempt ```bash # Regular user tries to get impersonation token curl -X POST https://localhost:8585/api/v1/users/impersonate \ -H "Authorization: Bearer " \ -d '{"targetUser": "alice"}' # Response: 403 Forbidden { "code": 403, "message": "Only bot users can generate impersonation tokens", "exceptionType": "AuthorizationException" } ``` ## 9. API Changes ### 9.1 REST API Behavior All existing CRUD endpoints remain unchanged. Impersonation is handled transparently through JWT token claims. ### 9.2 New Endpoint Summary | Method | Endpoint | Description | Auth Required | |--------|----------|-------------|---------------| | POST | `/api/v1/users/impersonate` | Request impersonation token | Bot with `allowImpersonation=true` | ### 8.2 Search and Filter Support ```bash # Search for entities updated by specific user via bot GET /api/v1/search/query?q=impersonatedBy:ingestion-bot # Search for specific user's actions (including impersonated) GET /api/v1/search/query?q=updatedBy:alice # Activity feed with impersonation filter GET /api/v1/feed?filterType=impersonated&bot=ingestion-bot ``` ## 9. Security Considerations ### 9.1 Threat Model | Threat | Mitigation | |--------|-----------| | **Unauthorized Impersonation** | Three-level validation: isBot + allowImpersonation flag + policy permission | | **Privilege Escalation** | Use impersonated user's permissions, not bot's | | **Token Theft** | Standard JWT security + short expiry + token validation | | **Audit Trail Tampering** | Immutable audit logs with both user and bot tracked | | **Policy Bypass** | Impersonation permission checked on every request | ### 9.2 Security Best Practices 1. **Least Privilege**: Only enable `allowImpersonation` for bots that absolutely need it 2. **Policy Restrictions**: Use domain/team-based policies to limit scope 3. **Token Expiry**: Impersonation tokens should have shorter expiry (e.g., 1 hour) 4. **Audit Monitoring**: Alert on impersonation usage patterns 5. **Regular Review**: Periodic audit of which bots have impersonation enabled ### 9.3 Attack Scenarios and Defenses #### Scenario 1: Malicious Bot Attempts Impersonation ``` Attack: Bot without allowImpersonation tries to include impersonatedUser claim Defense: JwtFilter validates allowImpersonation flag, rejects request Result: 403 Forbidden - "Bot not authorized to impersonate" ``` #### Scenario 2: Bot Tries to Impersonate Admin ``` Attack: Bot with impersonation tries to impersonate admin user Defense: Policy checks if bot can impersonate that specific user Result: 403 Forbidden - "Bot 'X' not authorized to impersonate user 'admin'" ``` #### Scenario 3: Stolen Impersonation Token ``` Attack: Attacker obtains valid impersonation token Defense: - Short token expiry (1 hour) - Token bound to specific user - Audit logs show suspicious activity patterns - Can revoke bot's allowImpersonation flag Result: Limited blast radius, quick detection and mitigation ``` ## 10. Testing Strategy ### 10.1 Unit Tests ```java // JwtFilterTest.java @Test public void testImpersonationValidation_Success() { // Given: Bot with allowImpersonation=true and valid token String token = generateImpersonationToken("test-bot", "alice"); // When: Filter processes request CatalogSecurityContext context = jwtFilter.getCatalogSecurityContext(token); // Then: Context contains impersonation info assertEquals("test-bot", context.getUserPrincipal().getName()); assertEquals("alice", context.impersonatedUser()); } @Test public void testImpersonationValidation_BotNotAuthorized() { // Given: Bot with allowImpersonation=false User bot = createBot("test-bot", false); String token = generateImpersonationToken("test-bot", "alice"); // When/Then: Validation fails assertThrows(AuthorizationException.class, () -> { jwtFilter.validateJwtAndGetClaims(token); }); } @Test public void testNonBotCannotImpersonate() { // Given: Regular user tries to impersonate String token = JWT.create() .withSubject("alice") .withClaim("isBot", false) .withClaim("impersonatedUser", "bob") .sign(algorithm); // When/Then: Validation fails assertThrows(AuthorizationException.class, () -> { jwtFilter.validateJwtAndGetClaims(token); }); } ``` ```java // EntityRepositoryTest.java @Test public void testEntityUpdate_WithImpersonation() { // Given: Bot impersonating user SubjectContext context = new SubjectContext( createUser("alice"), "ingestion-bot" // impersonatedBy ); Table table = createTable(); // When: Update entity tableRepository.update(null, table, updatedTable, "alice"); // Then: Both fields are set assertEquals("alice", updatedTable.getUpdatedBy()); assertEquals("ingestion-bot", updatedTable.getImpersonatedBy()); } ``` ### 10.2 Integration Tests ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 10.3 Security Tests ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 10.4 Performance Tests ```java @Test public void testImpersonationOverhead() { // Measure overhead of impersonation validation // Baseline: Normal request long baseline = measureRequestTime(() -> updateTableNormal()); // With impersonation long withImpersonation = measureRequestTime(() -> updateTableWithImpersonation()); // Assert: Overhead < 10% double overhead = (withImpersonation - baseline) / (double) baseline; assertTrue(overhead < 0.10, "Impersonation overhead should be < 10%"); } ``` ## 11. Migration Plan ### Phase 1: Schema and Backend (Week 1-2) 1. Add JSON schema definitions 2. Add database migrations 3. Update EntityInterface 4. Regenerate Java/Python/TypeScript models 5. Update JwtFilter, CatalogSecurityContext, SubjectContext 6. Deploy to dev environment ### Phase 2: Authorization (Week 2-3) 7. Add IMPERSONATE to MetadataOperation enum 8. Update DefaultAuthorizer with validation logic 9. Update EntityRepository to populate impersonatedBy 10. Add unit tests and integration tests 11. Deploy to staging environment ### Phase 3: Bot Support (Week 3-4) 12. Add allowImpersonation to User schema 13. Update bot creation APIs 14. Update Python SDK to support impersonation tokens 15. Update ingestion framework 16. Test with sample ingestion workflows ### Phase 4: Frontend (Week 4-5) 17. Update UI components to display impersonation 18. Add bot configuration page 19. Update activity feed 20. Add filtering by impersonatedBy ### Phase 5: Documentation and Rollout (Week 5-6) 21. Write user documentation 22. Write admin guide for policy configuration 23. Create migration guide for bot users 24. Gradual rollout to production 25. Monitor and adjust policies ## 12. Monitoring and Observability ### 12.1 Metrics ```java // Metrics to track - impersonation.requests.total (counter) - impersonation.requests.by_bot (counter, labeled by bot name) - impersonation.requests.by_target_user (counter, labeled by user) - impersonation.validation.failures (counter, labeled by reason) - impersonation.authorization.duration (histogram) ``` ### 12.2 Logging ```java // Log examples LOG.info("Impersonation validated: bot={}, target={}, operation={}", botName, targetUser, operation); LOG.warn("Impersonation attempt blocked: bot={}, target={}, reason={}", botName, targetUser, reason); LOG.debug("Impersonation token processed: bot={}, user={}, entity={}", botName, userName, entityType); ``` ### 12.3 Alerts ```yaml # Alert on suspicious patterns - name: HighImpersonationFailureRate condition: impersonation.validation.failures > 10/min severity: warning - name: UnauthorizedImpersonationAttempts condition: impersonation.requests{reason="unauthorized"} > 5/min severity: critical - name: CrossDomainImpersonationSpike condition: impersonation.requests{cross_domain="true"} > 20/min severity: warning ``` ## 13. Documentation ### 13.1 User Guide **For Bot Users:** - How to enable impersonation for your bot - How to generate impersonation tokens - Examples in Python SDK and REST API - Troubleshooting common issues **For Admins:** - How to configure impersonation policies - Best practices for security - How to audit impersonation usage - How to investigate suspicious activity ### 13.2 API Documentation Update OpenAPI specification: ```yaml components: securitySchemes: BotImpersonation: type: http scheme: bearer bearerFormat: JWT description: | JWT token with impersonation claim. Bot must have allowImpersonation=true and policy permission to impersonate the target user. Token claims: - isBot: true (required) - impersonatedUser: string (target username) ``` ## 14. Future Enhancements ### 14.1 Impersonation Audit Report - UI dashboard showing all impersonated actions - Exportable audit reports - Anomaly detection for unusual patterns ### 14.2 Temporary Impersonation Tokens - Time-limited impersonation grants - Automatic expiry after specific duration - Revocable impersonation sessions ### 14.3 Impersonation Approval Workflow - Require admin approval for sensitive impersonations - Approval via UI or API - Notification to impersonated user ### 14.4 Delegation Instead of Impersonation - User explicitly delegates actions to bot - Delegation scope limited to specific operations/entities - User can revoke delegation anytime ## 15. Appendix ### 15.1 Related Work - AWS IAM Role Assumption - Kubernetes Service Account Impersonation - Google Cloud Service Account Impersonation ### 15.2 Security Standards - OWASP API Security Top 10 - NIST 800-53 Access Control Guidelines - SOC 2 Audit Requirements ### 15.3 Glossary - **Impersonation**: Bot acting on behalf of a user - **Subject**: The entity performing authorization checks (user or bot) - **Principal**: The authenticated entity in security context - **Policy**: Rule defining what operations are allowed - **Resource Context**: Entity being accessed - **Operation Context**: Type of action being performed --- **Document Version**: 1.1 **Last Updated**: 2026-06-12 **Authors**: OpenMetadata Engineering **v1.1**: Custom-bot impersonation grants via `createBot.json`, RBAC target scoping (`isAdminUser()`/`isBotUser()` conditions), read-only `User.allowImpersonation` on user APIs, `Impersonate` excluded from `ALL` subsumption. Default `BotImpersonationPolicy` is **permissive/backward-compatible** (admin impersonation stays allowed); admin restriction is opt-in via `BotNonAdminImpersonationPolicy`/`BotNonAdminImpersonationRole`. All seeds are new, so no migration is needed. See section 4.4. **Status**: Draft for Review --- ### Index # Documentation Index The map to OpenMetadata's written knowledge — design docs, plans, generated references, and the reference docs that live *outside* `docs/` (in the UI, ingestion, and bootstrap trees). Use it to find an existing doc before reverse-engineering the code or guessing. **Freshness** was verified against the working tree on **2026-07-24** (not inferred from age — each verdict cites an artifact that was checked to still exist): - **CURRENT** — its load-bearing references still exist and match the code. - **STALE** — it references paths/classes/flags that no longer exist. - **SUPERSEDED** — a newer doc or implementation replaced it (named in the row / footnotes). - **⚠** — CURRENT overall, but with a specific caveat in the footnotes below. - **by construction** — regenerated from source and CI-gated, so it cannot drift from its source. > This index lists one file each; it does not move or relink anything (migration is out of scope). ## Start here — root guides (not under `docs/`) | Guide | What it is | |---|---| | `CLAUDE.md` | Always-loaded session guidance; the pointer index to `.claude/rules/*` and skills | | `ARCHITECTURE.md` | System map — modules, the request/ingestion/search paths, the invariants that hold | | `DEVELOPER.md` | How to build, test, and add an entity or connector (end-to-end checklists) | | `AGENTS.md` | Codex entry doc — **carries known contradictions** (Webpack→Vite, antd, Python ceiling); see `docs/tech-debt.md` #6 | ## Backend & platform design docs (`docs/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `docs/impersonation-design.md` | Bot→user impersonation: `updatedBy`=user / `impersonatedBy`=bot, gated by `allowImpersonation` + RBAC `Impersonate` policy scoping | Touching bot impersonation auth — the flag, `BotImpersonationPolicy` seeds, `checkImpersonationAuthorization` (§4.4 is authoritative) | 2026-06-16 | CURRENT ⚠¹ | | `docs/session-management-multi-node-design.md` | Shipped multi-node server-side session + websocket system: shared JDBC/Redis store, `OM_SESSION` cookie, session-bound JWTs, CAS refresh | Working on login/refresh/logout across pods, `SessionService`/`SessionStore`, JWT session validation, websocket handshake | 2026-06-03 | CURRENT | | `docs/streamable-logs.md` | S3/MinIO-backed streamable ingestion logs: HTTP append/close, `partial.txt`→`logs.txt`, SSE live tail, abandoned-run sweeper | Working on ingestion log storage/streaming — `S3LogStorage`, `LogStorageInterface`, `/logs/{fqn}/{runId}` endpoints | 2026-05-15 | CURRENT | | `docs/ingestion-log-streaming.md` | Live log tail over SSE: `LogStreamEvent` schema, resume cursors, one shared reader per run, and the limits that bound every stream | Building or debugging a client that tails ingestion logs live — `/logs/{fqn}/stream/{runId}`, `IngestionLogTailer`, `LogStreamSettings` | 2026-08-10 | CURRENT | | `docs/rdf-local-development.md` | Run RDF/knowledge-graph support locally with Apache Jena Fuseki — startup scripts, env vars, `rdf.*` config, `/api/v1/rdf/*`, `RdfIndexApp` | Setting up or debugging local RDF/Fuseki development | 2026-07-16 | CURRENT | | `docs/rdf-production-setup.md` | Production sizing/tuning for a remote Fuseki triple store — TDB2 heap vs page-cache, batch/timeout, weekly recreate, compaction | Sizing, scheduling, or troubleshooting a prod Fuseki deployment; tuning RDF bulk-write | 2026-07-16 | CURRENT | | `docs/csv-relation-types-plan.md` | Carry glossary term relation types through CSV export/import via a `relationType:termFQN` prefix (default `relatedTo`) | Modifying glossary CSV round-tripping — `CsvUtil.addTermRelations`, `GlossaryRepository.getTermRelationsFromCsv` | 2026-03-17 | CURRENT | | `docs/auto-classification/add-support-for-another-entity.md` | Step-by-step: extend auto-classification (PII + sample data) to a new entity across schema/Java/Python/UI via the `EntityAdapter` registry | Adding auto-classification/sample-data support for a new entity type | 2026-05-19 | CURRENT | | `docs/perf/cdn-deployment-guide.md` | AWS design proposal: per-customer/per-release UI bundles from one CloudFront + S3 via an embedded CloudFront Function router (no Lambda@Edge) | Planning/reviewing CDN delivery of the UI bundle + per-customer version pinning — infra design, not existing code | 2026-05-25 | CURRENT (unimplemented proposal) | ## Plans & specs (`docs/plans/`, `docs/superpowers/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `docs/plans/2026-01-27-search-indexing-stats-redesign.md` | Rework `SearchIndexingApp` stats into a per-stage pipeline model (`StageStatsTracker`/`StageCounter`), index-alias promotion, vector bulk processor | Before touching search reindex stats, `search_index_server_stats`, index promotion, vector indexing | 2026-01-29 | CURRENT (shipped design) | | `docs/plans/2026-06-22-bulk-deletion-redesign.md` | Fast, orphan-free, resumable service-level recursive hard-deletion by id-set; self-audits what landed on `main` vs remaining gaps | Before working on recursive/bulk deletion, `entity_relationship` orphan cleanup, the deletion-lock gate | 2026-06-26 | CURRENT (shipped design) | | `docs/superpowers/specs/2026-06-22-logviewer-modal-design.md` | Design spec for the reusable `LogViewerModal` (dark terminal modal over `@melloware/react-logviewer`); self-marked "Implemented; revised 2026-06-24" | Before modifying `LogViewerModal`, its log-level parser, theming, or streaming container/hook | 2026-07-16 | CURRENT | | `docs/superpowers/plans/2026-06-22-logviewer-modal.md` | Original TDD build plan for `LogViewerModal` (built-in LazyLog search, `CopyToClipboardButton`) | Historical context only — the shipped component follows the revised spec, not this plan | 2026-07-16 | SUPERSEDED ² | ## Generated references (`docs/generated/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `docs/generated/entity-index.md` | Auto-generated: 81 first-class entities → schema JSON, Java POJO, Python model, TS type, REST resource class | Locate every codegen artifact / the REST resource for an entity without grepping four trees | 2026-07-24 | CURRENT (by construction) ³ | | `docs/generated/api-reference.md` | Auto-generated: all 1748 REST endpoints (method + path + `@Operation` summary) grouped by resource package | Find an endpoint's exact path/method, or enumerate a package's routes without reading JAX-RS classes | 2026-07-24 | CURRENT (by construction) ³ | ## Repo audit & quality (`docs/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `docs/golden-principles.md` | 8 candidate repo-wide invariants (DRAFT for ratification), each with a measured adherence number + reproducing command | Cite/enforce an invariant (acyclic modules, ServiceSpec contract, generated-as-sink, no bare `except:`, …) | 2026-07-24 | CURRENT | | `docs/tech-debt.md` | Prioritized (impact÷size, 3 tiers) ledger of 21 audit findings, each with location, size, and agent-fixability | Pick up a bounded cleanup, or understand a known structural debt before working near it | 2026-07-24 | CURRENT | | `docs/quality.md` | One evidence-cited quality grade (A–C / Not assessed) per Maven module | Gauge a module's structural health / known debt before a large change | 2026-07-24 | CURRENT | ## Assets (`docs/assets/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `docs/assets/` (4 PNGs) | Architecture/marketing diagrams for the "Open Context Layer for AI" — hero, architecture, context graph, memory-primitives | Editing the root `README.md` visuals | 2026-06-10 | CURRENT (sole consumer: root `README.md`) | ## UI reference docs (outside `docs/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `openmetadata-ui/src/main/resources/ui/specs/` | **Machine-readable design system** (41 files). `README.md` declares two stacks — **go-forward = UntitledUI + Tailwind (`tw:`)**, **legacy (deprecated) = Ant Design + Less** — plus `foundations/*` (color, spacing, typography, radius, elevation, motion), `tokens/*` (Tailwind-utility + master token reference), `untitled/*` (go-forward component specs), and legacy `components/*` | **Before writing or modifying any UI code** — start at `specs/README.md`, then the `foundations`/`tokens` and the `untitled/.md` (or legacy `components/*`) spec for what you touch | 2026-07-27 | CURRENT ⁶ | | `openmetadata-ui/src/main/resources/ui/docs/colors.md` | Semantic color-token system (`tw:bg-primary`, `tw:text-fg-*`, `tw:border-*`) with light/dark values + the mandatory `ring`→`border` migration (§2.3.1) | Before writing/reviewing any Tailwind color class or dark-mode styling, or when tempted to use `ring-*` or a raw hex | 2026-07-23 | CURRENT | | `openmetadata-ui/src/main/resources/ui/docs/formutils.md` | The modern react-hook-form + react-aria form stack (`FieldProp`, `getField`/`FormFields`/`HookForm`) vs the legacy antd `@utils/formUtils` API | Before building/modifying any UI form — which API to use, and wiring to `useFormDrawerWithHook` + a pure transform | 2026-07-15 | CURRENT | | `openmetadata-ui/src/main/resources/ui/playwright/docs/` | Auto-generated E2E test-coverage catalog (`README.md` + Discovery/Governance/Integration/Observability/Platform) mapping component → spec file → scenarios | Check what UI behavior is already E2E-covered before writing Playwright tests or reasoning about gaps | 2026-03-09 – 2026-07-16 | CURRENT (by construction) ³ | ## Ingestion & bootstrap reference (outside `docs/`) | Doc | Purpose | Read when | Modified | Freshness | |---|---|---|---|---| | `ingestion/docs/design/ingestion-diagnostics.md` | DEBUG-gated ingestion diagnostics subsystem (operation registry, watchdog, heartbeat, memory tracker, HTTP introspection, stage backpressure, signal dumps) | Understand why/how the diagnostics subsystem works before instrumenting ingestion hangs/OOMs | 2026-05-20 | CURRENT ⚠⁴ | | `bootstrap/MIGRATION_SYSTEM.md` | Hybrid DB-migration architecture — Flyway→native→extension execution order, `SERVER_CHANGE_LOG` tracking, file layout (Flyway *parsers* only, not the Flyway runner) | Before adding/debugging a migration under `bootstrap/sql/migrations/`, or reasoning about ordering / tracking tables / MySQL+Postgres dual paths | 2025-10-28 | CURRENT ⚠⁵ | ## Caveats (verification notes) 1. **impersonation-design** — §4.4 (v1.1) is authoritative and matches shipped code (`allowImpersonation` in `createBot.json`/`user.json`, `checkImpersonationAuthorization` in `DefaultAuthorizer.java`, the four policy/role seeds, `BotImpersonationIT`). The earlier §4.1/4.2 `POST /users/impersonate` token-exchange endpoint was **never shipped** — impersonation uses the `X-Impersonate-User` header (`JwtFilter.java`), which §4.4 supersedes 4.1/4.2 to reflect. Read §4.4. 2. **logviewer-modal plan** — superseded by the **2026-06-24 revision** of the design spec (row above): the plan's built-in LazyLog search + `CopyToClipboardButton` were reversed (search moved to the header, a footer status bar added). The shipped dir (`LogViewerModal.utils.tsx`, `LogsViewerModalContainer.tsx`, `useLogStream.ts`) follows the revised spec, not this plan. 3. **by construction** — `docs/generated/*` are regenerated by `make generate-reference-docs` and gated by the reference-docs freshness CI job; the Playwright docs by `playwright/doc-generator/generate.js` + the `playwright-docs-check.yml` workflow. They cannot drift from their source on the watched paths. (One benign lag: the Playwright `README.md` roll-up footer reads `2026-03-09` while `Governance.md` regenerated `2026-07-16`; self-corrects on the next spec-touching PR.) 4. **ingestion-diagnostics** — the design **shipped** (`ingestion/src/metadata/ingestion/diagnostics/`, activated at `loggerLevel == DEBUG`, installed in `metadata/workflow/base.py`), but the doc's §5/§7 *flat file map drifted*: files now live under `collectors/`, `monitors/`, `samplers/`; there is no standalone `heartbeat.py`; wire-in is `base.py`, not the doc's `base_workflow.py`. Treat the file map as design-era, the behavior as current. 5. **MIGRATION_SYSTEM** — accurate (the `flyway/`+`native/` layout, `SERVER_CHANGE_LOG`, `MigrationWorkflow`/`FlywayMigrationFile`, and `conf/openmetadata.yaml` `flywayPath`/`nativePath`/`extensionPath` all verified) except: the runner class is `MigrationProcessImpl` (doc says `MigrationProcess`), and the `extensions/` dir is not materialized on disk (`extensionPath: ""`) though the path is still supported. 6. **ui/specs** — tracked (41 files) and CURRENT: all four audit commands its `README.md` cites resolve to real `package.json` scripts (`tw-audit`, `tw-audit:report`, `tw-guard`, `token-audit`), and the two-stack policy it states (go-forward UntitledUI+Tailwind, legacy Antd+Less deprecated) matches the enforced rules — `tw-guard` blocks new `antd` imports / new `.less` files, and `.claude/rules/frontend-styling.md` routes agents into `specs/README.md`. --- *Not indexed:* `docs/harness-audit/` exists in the working tree but is **untracked** (working audit notes, intentionally out of version control), so it is not part of the committed knowledge base. --- ### Ingestion Log Streaming # Live Ingestion Log Streaming (SSE) How a client tails an ingestion pipeline's logs in real time over Server-Sent Events instead of polling `/logs/{id}/last` on a timer. For where the log bytes are stored and how they get there, read [streamable-logs.md](streamable-logs.md). This document is about the **read path a UI uses while a run is in progress**. ## Why streaming instead of polling The paginated endpoint (`GET /logs/{id}/last?after=`) forces every viewer into a poll loop: pick an interval, re-issue the request, diff the cursor, repeat until the run ends. That has three costs the streaming endpoint removes: | Polling | Streaming | |---|---| | Every open tab is its own poll loop against S3 / Airflow. | All viewers of a run share **one** server-side reader. | | Latency is the poll interval. | New content is pushed as soon as the shared reader sees it. | | The client decides when the run is over (or never stops). | The server sends an explicit `complete` event and closes. | | A "fetch everything" loop can walk pages without bound. | Reads per tick, bytes per stream, and stream lifetime are all capped. | ## The endpoint ### `GET /api/v1/services/ingestionPipelines/logs/{fqn}/stream/{runId}` One endpoint for every deployment. `{fqn}` is the pipeline's **fullyQualifiedName or Id (UUID)**, like `/logs/{id}/last`. `{runId}` is the run to follow — a UUID when the run's logs are in object storage, or the pipeline service's own run identifier (Airflow's `scheduled__…`) otherwise. | Query parameter | Default | Meaning | |---|---|---| | `after` | beginning of the log | Resume cursor from a previous event. Content up to the cursor is not re-sent. | To tail the newest run, read its `runId` from `pipelineStatuses` first — the same field `/logs/{id}/last` resolves internally. Response: `text/event-stream`. Every frame is one JSON [`LogStreamEvent`](../openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/logStreamEvent.json) on an unnamed SSE event, so a plain `EventSource.onmessage` receives all of them. ```jsonc // eventType: "logs" — new content {"eventType":"logs","runId":"a1b2…","logs":"[2026-08-10 …] INFO Ingesting table x","after":"4211","replay":false,"truncated":false} // eventType: "complete" — the server is closing the stream {"eventType":"complete","runId":"a1b2…","after":"4680","reason":"runFinished"} // eventType: "error" — the stream cannot be served; it is closed right after {"eventType":"error","runId":"a1b2…","message":"The server is already streaming the maximum number of pipeline runs. …"} ``` | Field | Meaning | |---|---| | `eventType` | `logs`, `complete` or `error`. | | `runId` | Run the content belongs to. | | `logs` | Content appended since the previous event. Absent on `complete` / `error`. | | `after` | Cursor pointing just past `logs`. **Store it**: it is what you pass to `?after=` to reconnect. | | `replay` | `true` when the chunk came from the server's replay buffer because the stream was already running when you connected. | | `truncated` | `true` on the first event when the server could not work out exactly what you are missing. **Treat it as a reset**: clear the viewer, render the replayed block that follows, and backfill earlier history from `GET /logs/{id}/last` or the download endpoint. | | `reason` | Why the stream ended. See the table below. | | `message` | Human-readable detail on `error`, and on a `complete` that ended early. | Between events the server sends SSE heartbeat comments (`: heartbeat`) every 25 s so proxies do not drop an idle connection. `EventSource` ignores them. **End-of-stream reasons** | `reason` | What happened | What a client should do | |---|---|---| | `runFinished` | The run reached a terminal state and its log went quiet — or the server has no status row for it and it stayed quiet for a minute. | Nothing. The log is complete. | | `idleTimeout` | No new content for 5 minutes and the run never reported a terminal state. | Reconnect with `?after=` if you still care about the run. | | `maxDuration` | The stream hit its 1 hour lifetime cap. | Reconnect with `?after=`. | | `maxBytes` | The stream delivered 32 MB. | Use `GET /logs/{id}/last/download` for the rest. | A stream that ends **without** a `complete` event was cut short — the client stopped draining the socket and crossed its backlog ceiling, the server went away, or the network dropped. Treat a closed body with no `complete` as "reconnect with `?after=`". **Back off between reconnects.** The causes above are often persistent: a viewer that cannot keep up crosses its backlog ceiling again on the next attempt, and each reconnect re-fetches the replay backlog. Reconnecting immediately in a loop turns one struggling client into a load generator. Use a capped exponential delay, and give up after a few consecutive failures rather than retrying forever. **Status codes** | Code | When | |---|---| | 200 | Stream opened. It is committed immediately — the response never waits for the run to finish. | | 404 | No such pipeline. | Everything that goes wrong *after* the pipeline resolves is reported as an `error` event on the stream, not as an HTTP status: no log backend configured, the server at its stream capacity, a viewer past the connection cap. A client therefore needs one error path (handle `eventType: "error"`), not two. ## Using it from a browser `EventSource` cannot set an `Authorization` header, so use `fetch` with a streaming reader: ```ts const controller = new AbortController(); let cursor: string | undefined; const tail = async (fqn: string, runId: string) => { const url = new URL( `${getBasePath()}/api/v1/services/ingestionPipelines/logs/${getEncodedFqn( fqn )}/stream/${encodeURIComponent(runId)}`, window.location.origin ); if (cursor) { url.searchParams.set('after', cursor); } const response = await fetch(url, { headers: { Authorization: `Bearer ${await getOidcToken()}` }, signal: controller.signal, }); const reader = response.body!.getReader(); const decoder = new TextDecoder(); let buffer = ''; for (;;) { const { done, value } = await reader.read(); if (done) { break; } buffer += decoder.decode(value, { stream: true }); const frames = buffer.split('\n'); buffer = frames.pop() ?? ''; for (const frame of frames) { if (!frame.startsWith('data: ')) { continue; // heartbeat comment or blank separator } const event = JSON.parse(frame.slice(6)); cursor = event.after ?? cursor; if (event.eventType === 'logs') { appendToViewer(event.logs); } else if (event.eventType === 'complete') { onStreamEnd(event.reason); // reconnect here for a non-runFinished reason } else { onStreamError(event.message); } } } }; ``` Reconnecting is always `?after=`. The cursor is opaque: it means "line offset" for object storage and "chunk offset" for Airflow, and the two are not interchangeable, so never construct one by hand. If the run is still being tailed for someone else when you reconnect, the server resumes you from the shared reader's buffer: it replays exactly the chunks issued after your cursor. When your cursor is older than that buffer — or came from another server behind the load balancer — the server cannot tell what sits in between, so it replays what it has with `truncated: true`. That is the one case where a client must reset its viewer rather than append. ## Using it from the command line ```bash curl -N -H "Authorization: Bearer $OM_TOKEN" \ "http://localhost:8585/api/v1/services/ingestionPipelines/logs/my.pipeline.fqn/stream/$RUN_ID" ``` `-N` disables curl's buffering, which is what makes the live tail visible. ## How it works ``` GET /logs/{fqn}/stream/{runId} │ ▼ IngestionPipelineResource ──▶ IngestionLogStreamFactory ──▶ picks the source for where the bytes are │ ├─ StorageLogTailSource (S3, line cursor) │ └─ PipelineServiceLogTailSource (Airflow/Argo, chunk cursor) ▼ IngestionLogStreamManager ──▶ one IngestionLogTailer per (pipeline, run) │ │ polls every 2 s on a shared scheduler │ │ keeps a bounded replay buffer │ └▶ fans each chunk out to every viewer └─ SseConnectionRegistry: connection cap + 25 s heartbeat + disconnect sweep ``` **One reader per run.** The tailer is keyed by `(storage backend, pipeline FQN, run)`. Opening the same run in ten tabs creates ten SSE connections but still exactly one reader against S3 or Airflow. The last viewer to disconnect stops the reader, so an unwatched run is not read at all. **Cursors are per backend.** Object storage paginates a run's log by line offset. The pipeline service paginates it in fixed-size chunks and keeps appending to the last chunk while the task runs, so a chunk-index cursor alone would re-deliver a growing chunk on every poll — the cursor is `chunk:charactersAlreadyDelivered` and only the growth is emitted. The source also never asks for a chunk index the backend did not itself report, because Airflow answers an out-of-range chunk with a 400. **Knowing when to stop.** The run's state is read from its pipeline-status rows, at most once every 10 s and never again once terminal — no call to the pipeline service is made for this. Once the run is terminal the stream stays open until the log has been quiet for 10 s, so the final flush of a just-finished run is still delivered. Only the last few runs of a pipeline keep a status row, so a run the server finds no row for is **unknown**, not finished — that describes a run that aged out of the window *and* one that was triggered a second ago and has not written anything yet. An unknown run gets a full minute of silence before the stream closes, which is what stops a freshly triggered pipeline from being reported as already over while Airflow is still starting the task. **Nothing is unbounded.** Every limit below is enforced by `LogStreamSettings`: | Limit | Default | What it protects | |---|---|---| | `pollSeconds` | 2 | Read rate against the log backend, per run. | | `linesPerRead` | 1000 | Page size held in heap at any moment. | | `maxReadsPerTick` | 20 | Burst of backend calls when catching up on a backlog. | | `maxBytesPerTick` | 1 MB | Content pushed at a client in one tick. | | `maxStreamBytes` | 32 MB | Total a single stream will deliver. | | `maxStreamSeconds` | 3600 | Lifetime of a forgotten browser tab. | | `maxIdleSeconds` | 300 | Runs that die without reporting a terminal state. | | `finishGraceSeconds` | 10 | Silence after a confirmed-terminal run before closing. | | `unknownRunGraceSeconds` | 60 | Silence before closing a run with no status row — a just-triggered run needs time to start writing. | | `maxReplayBytes` | 256 KB | Backlog kept per run for late joiners. | | `maxPendingBytesPerClient` | 4 MB | Memory a stalled browser can pin. | | `maxActiveRuns` | 200 | Runs tailed concurrently per server. | | `maxActiveConnections` | 500 | Open log stream connections per server. | A request past `maxActiveRuns` or `maxActiveConnections` gets an `error` event and a closed stream rather than being queued. Polling runs on a shared scheduler sized from `maxActiveRuns` (one thread per 25 runs, clamped to 2–16). A poll is a network read, so a slow backend shows up as a longer effective poll interval for everyone rather than as a stalled stream — and a poll that fails for any reason closes its own stream and gives the run's tail slot back instead of leaving a dead reader behind. ## Multi-server deployments A tailer is per server. Two servers behind a load balancer that both have viewers for the same run each keep one reader, which is the intended trade: reads are cheap and idempotent, and no cross-server coordination is needed. The write path's sticky-session requirement (see [streamable-logs.md](streamable-logs.md#multi-server-topology)) does not apply here — reading `partial.txt` from S3 works from any instance. ## Source files - `openmetadata-service/src/main/java/org/openmetadata/service/logstorage/stream/` — the streaming engine - `openmetadata-service/src/main/java/org/openmetadata/service/sse/SseConnectionRegistry.java` — connection cap and heartbeat - `openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/logStreamEvent.json` — the event schema - `openmetadata-service/src/test/java/org/openmetadata/service/logstorage/stream/` — unit tests - `openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/IngestionPipelineLogStreamIT.java` — end-to-end test --- ### Quality # Module Quality Grades One grade per module, **each citing the specific measurement behind it**. No grade without evidence — where the audit didn't measure a module, it is marked **Not assessed** rather than guessed. The measurements come from a one-time repository audit: the Maven module graph, per-language package/import censuses, and convention-adherence counts (each reproducible by the commands in `golden-principles.md`). Scale: **A** exemplary · **B** solid with bounded debt · **C** works but carries structural debt · **Not assessed** = insufficient evidence. | Module | Grade | Evidence | |---|---|---| | ingestion | **B+** | strongest architectural discipline measured | | openmetadata-spec | **B** (evidence-limited) | clean codegen foundation; minor POM hygiene | | openmetadata-service | **C+** | pristine surface hygiene, badly tangled internals | | openmetadata-ui | **C** | disciplined component model, heavy architectural + i18n debt | | openmetadata-mcp | **Not assessed** | only its DAG position is known | | openmetadata-sdk | **Not assessed** | only its DAG position is known | --- ## ingestion — B+ **For:** the ServiceSpec plugin contract holds at ~100% (**~97 connectors all registered**; **94/95 `metadata.py` carry `create()`+`InvalidSourceException` = 98.9%**); generated-import discipline is **99.9% type-only** (1,736/1,738; single ANTLR runtime exception in `spline`); ruff formatting **100%**; bare-except **99.95%**. It is the module that best keeps the repo's golden principles. **Against:** the broad-`except Exception` idiom is **75.5%** of handlers with **~86 silent swallows** (sanctioned, but the silent subset is real debt); a **11,927-finding** basedpyright baseline; one circular sibling import `mssql↔azuresql`. **Net:** the cleanest architecture in the repo; debts are idiom-scoped and incremental, not structural. ## openmetadata-spec — B (evidence-limited) **For:** it is the schema-first source of truth that drives all codegen, so Golden Principle #3 ("generated is a pure sink") *depends on it* and holds at 100%. Clean position in the Maven graph. **Against:** POM hygiene — declares its `common` dependency **twice**, and the reactor `` list puts `spec` (#1) *before* its own dependency `common` (#3). **Evidence limit:** the convention-adherence pass measured **no** lint metrics for spec (it is JSON Schemas + generated POJOs, not hand-written linted source), so this grade rests on the structural (module-graph + POM) findings only — stated so it isn't mistaken for a lint-backed grade. ## openmetadata-service — C+ **For:** surface hygiene is excellent — spotless **100%**, parameterized logging **100%**, no-wildcard **98.8%**, and boundary validation is systematic via `EntityResource` inheritance. **Against (the reason it's not a B):** it is the **most internally tangled module in the repo**. The package-import census found `resources ↔ jdbi3` is a mutual cycle (**130/99**) and **18 of 21** package pairs are cyclic — only `security/` is even a partial sink. This directly violates the repo's own #1 golden principle *inside* the core backend. `resources/ai/` additionally grew a service/seed-loader tier fused into the REST layer. **Surfaced conflict:** lint-clean ≠ well-layered. By the lint metrics this module looks pristine; by the package-import census it is the least-layered. The grade weights the architecture (harder to fix, higher blast radius) over the formatting. ## openmetadata-ui — C **For:** the component model is disciplined — **100%** functional components, 0 class components; lint hygiene is high — no-console **99.96%**, license header **99.75%**. **Against:** heavy architectural debt — a **130-module `components↔utils` SCC** (28 cyclic SCCs, 50 direct 2-cycles) with no import-boundary tooling; **generated-type leakage** of 1,292 direct component/page importers vs 93 in `rest/` (13.9:1); the **antd migration is stalled** at 864 files, 68.5% edited in the last 90 days; and **~250–396 untranslated English strings per non-en locale**. `any` at 90.2% is mid. **Surfaced conflict:** clean on the component axis (functional-only) but only 36.4% on the file-naming axis — "UI is disciplined" is true for the model, not the file conventions. ## openmetadata-mcp — Not assessed **Only evidence:** the Maven module graph places it correctly (`mcp → openmetadata-service`, compile; no back-edge, part of the acyclic graph). **The package-layering, cycle, and convention-adherence passes did not sample mcp** — no such measurement exists for it. Per "no grade without evidence," it is **not graded**; the single known fact is a clean DAG position. ## openmetadata-sdk — Not assessed **Only evidence:** the Maven graph — `sdk → openmetadata-spec` (compile); consumed by `openmetadata-integration-tests`, **not** by `openmetadata-service` (a clean client/leaf position); no cycle. **No adherence data and no deep internal analysis.** Not graded, for the same reason as mcp. --- ### Note on the two "Not assessed" modules This is a **coverage gap in the audit**, not a statement that mcp/sdk are low quality. The deep passes deliberately focused on the three largest surfaces (service, ui, ingestion). Grading mcp/sdk would require a comparable pass (package layering + a convention-adherence sample). Until then, any grade would be invention — which the "no grade without evidence" rule forbids. --- ### Rdf Local Development # RDF/Apache Jena Local Development Guide This guide documents how to set up RDF/Knowledge Graph support for local development with OpenMetadata and Apache Jena Fuseki. For production sizing, tuning, compaction, scheduling, and monitoring, see [Setting up Apache Jena Fuseki efficiently](rdf-production-setup.md). ## Overview OpenMetadata supports RDF (Resource Description Framework) for knowledge graph capabilities using Apache Jena Fuseki as the triple store. This enables: - SPARQL queries against metadata - JSON-LD serialization of entities - Semantic search and graph exploration ## Architecture ``` ┌─────────────────────┐ ┌─────────────────────┐ │ OpenMetadata │ │ Apache Jena │ │ Server (IntelliJ) │────▶│ Fuseki (Docker) │ │ Port: 8585 │ │ Port: 3030 │ └─────────────────────┘ └─────────────────────┘ ``` ## Prerequisites - Docker and Docker Compose installed - IntelliJ IDEA with the project imported - MySQL or PostgreSQL running (for OpenMetadata backend) - Elasticsearch running (for search) ## Quick Start ### Step 1: Choose the Right Startup Mode The standard local Docker flow does not enable RDF or start Fuseki: ```bash cd /path/to/OpenMetadata ./docker/run_local_docker.sh -d mysql ``` For PostgreSQL-based development: ```bash ./docker/run_local_docker.sh -d postgresql ``` Use the RDF-specific startup script when you want the full Docker stack with Fuseki enabled: ```bash ./docker/run_local_docker_rdf.sh -d mysql ``` For PostgreSQL-based RDF development: ```bash ./docker/run_local_docker_rdf.sh -d postgresql ``` This RDF startup path starts OpenMetadata, the backing database, search, ingestion services, and Fuseki with: - **Port**: 3030 - **Admin Password**: admin - **Dataset**: openmetadata - **Memory**: 2-4GB allocated ### Step 2: Verify Fuseki is Running ```bash # Check Fuseki health curl -s http://localhost:3030/$/ping # Access Fuseki UI in browser open http://localhost:3030 ``` The Fuseki web UI is available at `http://localhost:3030` with credentials: - Username: `admin` - Password: `admin` ### Step 3: Configure IntelliJ Run Configuration If you are running the full RDF Docker stack with `run_local_docker_rdf.sh`, the Docker services already receive the RDF environment variables automatically. If you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker, start Fuseki separately: ```bash docker compose -f docker/development/docker-compose.yml -f docker/development/docker-compose-fuseki.yml up -d fuseki ``` If your local backend uses PostgreSQL, swap `docker-compose.yml` for `docker-compose-postgres.yml`. Create or modify your IntelliJ run configuration for `OpenMetadataApplication` with these environment variables only when you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker: ``` RDF_ENABLED=true RDF_STORAGE_TYPE=FUSEKI RDF_BASE_URI=https://open-metadata.org/ RDF_ENDPOINT=http://localhost:3030/openmetadata RDF_REMOTE_USERNAME=admin RDF_REMOTE_PASSWORD=admin RDF_DATASET=openmetadata ``` #### Setting Environment Variables in IntelliJ: 1. Open **Run** → **Edit Configurations** 2. Select your `OpenMetadataApplication` configuration 3. Click on **Modify options** → **Environment variables** 4. Add the environment variables above (semicolon-separated or using the dialog) Example environment variables string: ``` RDF_ENABLED=true;RDF_STORAGE_TYPE=FUSEKI;RDF_BASE_URI=https://open-metadata.org/;RDF_ENDPOINT=http://localhost:3030/openmetadata;RDF_REMOTE_USERNAME=admin;RDF_REMOTE_PASSWORD=admin;RDF_DATASET=openmetadata ``` ### Step 4: Start OpenMetadata Server Run `OpenMetadataApplication` from IntelliJ. On startup, you should see in the logs: ``` INFO [main] o.o.s.OpenMetadataApplication - RDF knowledge graph support initialized ``` ### Step 5: Verify RDF is Enabled ```bash # Check RDF status curl http://localhost:8585/api/v1/rdf/status # Expected response: # {"enabled": true} ``` ## Configuration Reference ### Server Configuration (conf/openmetadata.yaml) The RDF configuration section in `openmetadata.yaml`: ```yaml rdf: enabled: ${RDF_ENABLED:-false} baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"} storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"} remoteEndpoint: ${RDF_ENDPOINT:-${RDF_REMOTE_ENDPOINT:-"http://localhost:3030/openmetadata"}} connectTimeoutMs: ${RDF_CONNECT_TIMEOUT_MS:-2000} requestTimeoutMs: ${RDF_REQUEST_TIMEOUT_MS:-60000} bulkEntityBatchSize: ${RDF_BULK_ENTITY_BATCH_SIZE:-100} bulkRelationshipSourceBatchSize: ${RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE:-100} bulkLineageEdgeBatchSize: ${RDF_BULK_LINEAGE_EDGE_BATCH_SIZE:-50} username: ${RDF_REMOTE_USERNAME:-"admin"} password: ${RDF_REMOTE_PASSWORD:-"admin"} dataset: ${RDF_DATASET:-"openmetadata"} inferenceEnabled: ${RDF_INFERENCE_ENABLED:-false} ``` ### Environment Variables | Variable | Description | Default | |----------|-------------|---------| | `RDF_ENABLED` | Enable/disable RDF support | `false` | | `RDF_STORAGE_TYPE` | Storage backend type | `FUSEKI` | | `RDF_BASE_URI` | Base URI for RDF resources | `https://open-metadata.org/` | | `RDF_ENDPOINT` | Fuseki SPARQL endpoint URL | `http://localhost:3030/openmetadata` | | `RDF_REMOTE_ENDPOINT` | Deprecated fallback when `RDF_ENDPOINT` is unset | unset | | `RDF_CONNECT_TIMEOUT_MS` | Fuseki connection timeout | `2000` | | `RDF_REQUEST_TIMEOUT_MS` | Per-request timeout | `60000` | | `RDF_BULK_ENTITY_BATCH_SIZE` | Entity models per bulk write | `100` | | `RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE` | Relationship sources per bulk write | `100` | | `RDF_BULK_LINEAGE_EDGE_BATCH_SIZE` | Detailed lineage edges per bulk write | `50` | | `RDF_REMOTE_USERNAME` | Fuseki admin username | `admin` | | `RDF_REMOTE_PASSWORD` | Fuseki admin password | `admin` | | `RDF_DATASET` | Fuseki dataset name | `openmetadata` | | `RDF_INFERENCE_ENABLED` | Enable in-process full-graph inference | `false` | ### Docker Compose Configuration The Fuseki container (`docker/development/docker-compose-fuseki.yml`): ```yaml services: fuseki: build: context: ../rdf-store dockerfile: Dockerfile image: openmetadata-fuseki:5.6.0 container_name: openmetadata-fuseki ports: - "3030:3030" environment: - FUSEKI_ADMIN_PASSWORD=admin - FUSEKI_OPENMETADATA_PASSWORD=openmetadata-secret - JVM_ARGS=-Xmx1500m -Xms256m volumes: - fuseki-tdb2-data:/fuseki-data ``` ## API Endpoints Once RDF is enabled, these endpoints are available: ### Check RDF Status ```bash GET /api/v1/rdf/status ``` ### Get Entity as RDF ```bash # Get entity in JSON-LD format (default) GET /api/v1/rdf/entity/{entityType}/{id} # Get entity in Turtle format GET /api/v1/rdf/entity/{entityType}/{id}?format=turtle # Get entity in RDF/XML format GET /api/v1/rdf/entity/{entityType}/{id}?format=rdfxml # Get entity in N-Triples format GET /api/v1/rdf/entity/{entityType}/{id}?format=ntriples ``` ### Execute SPARQL Query ```bash POST /api/v1/rdf/sparql Content-Type: application/json { "query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10" } ``` ### Get Glossary Term Relationship Graph ```bash # Get the full glossary term graph GET /api/v1/rdf/glossary/graph # Filter primary terms to a glossary GET /api/v1/rdf/glossary/graph?glossaryId= # Filter to a glossary term and its direct incoming/outgoing neighbors GET /api/v1/rdf/glossary/graph?glossaryTermId= # Require the selected term to belong to a glossary, while still returning # direct cross-glossary neighbors when relationships cross glossary boundaries GET /api/v1/rdf/glossary/graph?glossaryId=&glossaryTermId= ``` Optional query parameters: | Parameter | Description | |-----------|-------------| | `glossaryId` | Filter primary terms to a glossary. | | `glossaryTermId` | Filter to a selected glossary term and its direct incoming/outgoing glossary-term relations. | | `relationTypes` | Comma-separated relation types to include. | | `limit` | Maximum number of terms to return. Default: `500`. | | `offset` | Pagination offset. Default: `0`. | | `includeIsolated` | Include terms without relations. Default: `true`. | ### Example Queries ```bash # Check if RDF is enabled curl -s http://localhost:8585/api/v1/rdf/status | jq # Get a table entity as JSON-LD curl -s -H "Authorization: Bearer " \ "http://localhost:8585/api/v1/rdf/entity/table/" | jq # Execute a SPARQL query curl -s -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"}' \ http://localhost:8585/api/v1/rdf/sparql | jq # Get a selected glossary term graph curl -s -H "Authorization: Bearer " \ "http://localhost:8585/api/v1/rdf/glossary/graph?glossaryId=&glossaryTermId=" | jq ``` ## Indexing Entities to RDF ### Manual Reindexing Trigger the RDF indexing application to populate the triple store with existing entities: ```bash curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"entities": [], "recreateIndex": true, "batchSize": 100}' \ http://localhost:8585/api/v1/apps/trigger/RdfIndexApp ``` ### Automatic Indexing When RDF is enabled, new entities are automatically indexed to the triple store on create/update/delete operations. ## Fuseki Web UI The Fuseki web interface provides: - **Dataset Management**: View and manage datasets at `http://localhost:3030/#/manage` - **SPARQL Query Interface**: Execute queries at `http://localhost:3030/#/dataset/openmetadata/query` - **Data Upload**: Upload RDF data at `http://localhost:3030/#/dataset/openmetadata/upload` ## Troubleshooting ### Fuseki Connection Issues 1. Verify Fuseki is running: ```bash docker ps | grep fuseki curl http://localhost:3030/$/ping ``` 2. Check Fuseki logs: ```bash docker logs openmetadata-fuseki ``` 3. Ensure the dataset exists: ```bash curl -u admin:admin http://localhost:3030/$/datasets ``` ### RDF Not Enabled in Server 1. Verify environment variables are set correctly in IntelliJ 2. Check server logs for RDF initialization message 3. Confirm configuration in `openmetadata.yaml` ### SPARQL Query Errors 1. Check Fuseki is accessible from OpenMetadata server 2. Verify the dataset name matches (`openmetadata`) 3. Check Fuseki logs for query errors ### Reset Fuseki Data To clear all RDF data and start fresh: ```bash # Stop Fuseki docker compose -f docker/development/docker-compose-fuseki.yml down # Remove volume docker volume rm openmetadata_fuseki-data # Restart Fuseki docker compose -f docker/development/docker-compose-fuseki.yml up -d ``` ## Full Stack with Docker Script For a complete local environment with RDF enabled (server running in Docker, not IntelliJ): ```bash ./docker/run_local_docker_rdf.sh -m ui -d mysql -f true ``` Options: - `-m ui|no-ui` - Include UI or not - `-d mysql|postgresql` - Database type - `-f true|false` - Start Fuseki for RDF support - `-s true|false` - Skip Maven build - `-x true|false` - Enable JVM debug on port 5005 ## Related Files - **Docker Compose**: `docker/development/docker-compose-fuseki.yml` - **Server Config**: `conf/openmetadata.yaml` - **RDF Java Code**: `openmetadata-service/src/main/java/org/openmetadata/service/rdf/` - **Ontology**: `openmetadata-spec/src/main/resources/rdf/ontology/openmetadata.ttl` - **RDF Index App**: `openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/RdfIndexApp.java` --- ### Rdf Production Setup # Setting up Apache Jena Fuseki efficiently OpenMetadata stores its RDF knowledge graph in a remote Apache Jena Fuseki dataset. Production sizing must account for two different memory consumers: the Fuseki JVM heap and the operating-system page cache used by TDB2's memory-mapped indexes. Giving the JVM all container memory starves the page cache and usually reduces throughput. ## Capacity planning TDB2 storage varies with URI and literal width. A useful planning range for OpenMetadata graphs is 150-250 bytes per triple before compaction headroom. | Live triples | Approximate live TDB2 data | Fuseki heap | Suggested total RAM | Suggested persistent disk | | ---: | ---: | ---: | ---: | ---: | | 1 million | 0.15-0.25 GB | 2 GB | 4-8 GB | 2 GB minimum | | 10 million | 1.5-2.5 GB | 4 GB | 8-16 GB | 8 GB minimum | | 50 million | 7.5-12.5 GB | 8 GB | 24-48 GB | 40 GB minimum | Persistent storage should be at least 2-3 times the expected live dataset, plus room for the journal. TDB2 compaction builds a replacement dataset before deleting the old one, so a volume sized only for live data can run out of space during compaction. The 10 GiB PVC in `docker/rdf-store/kubernetes/fuseki-deployment.yaml` is a development default, not a production recommendation. Keep the Fuseki heap smaller than the container memory limit. Memory beyond `-Xmx` is useful: TDB2 memory-maps its indexes and relies heavily on the OS page cache. The OpenMetadata server does not need additional heap for normal RDF queries because built-in lineage and semantic expansion execute as property-path queries inside Fuseki. ## Write throughput TDB2 is a single-writer store. More OpenMetadata indexing threads can prepare and read batches concurrently, but Fuseki ultimately serializes write transactions. Throughput therefore depends primarily on the number and size of SPARQL UPDATE transactions, not the number of HTTP clients. An RDF indexing run with `recreateIndex: true` clears the graph first and then uses insert-only writes. It does not perform per-entity or per-relationship reconciliation against the empty graph. Entity, relationship, and lineage data are grouped into bounded updates. Incremental runs continue to reconcile existing values. RDF indexing hydrates every field supported by each entity repository because the RDF mapper preserves fields even when they have no explicit JSON-LD context mapping. It omits only `changeDescription`, `votes`, and the embedded `testCaseResult`, which the mapper intentionally does not emit as triples. Start with the defaults and tune one setting at a time: | Environment variable | Default | Purpose | | --- | ---: | --- | | `RDF_BULK_ENTITY_BATCH_SIZE` | `100` | Entity models per SPARQL update. | | `RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE` | `100` | Source entities reconciled per relationship update. | | `RDF_BULK_LINEAGE_EDGE_BATCH_SIZE` | `50` | Detailed lineage edges per update. | | `RDF_REQUEST_TIMEOUT_MS` | `60000` | Maximum time for one RDF request. | Larger batches reduce transaction and journal overhead but increase request size, parse time, and retry cost. If a larger batch approaches `RDF_REQUEST_TIMEOUT_MS`, either reduce the batch or raise the timeout. Wide tables can produce tens of megabytes of N-Triples per 100-entity batch, so validate changes against representative catalogs. ## Scheduling The recommended RDF schedule is a weekly recreate on Saturday at midnight: ```text 0 0 * * 6 ``` Search indexing defaults to Sunday at 00:30: ```text 30 0 * * 0 ``` Keep the jobs separated manually. Both scan the metadata database and hydrate entity relationships, so running them together increases database pressure. OpenMetadata does not provide cross-application mutual exclusion between RDF and search indexing. Upgrades migrate an RDF app that still has the former exact daily default (`0 0 * * *`) to the weekly schedule. Custom schedules and applications with scheduling disabled are not changed. ## Compaction and disk growth OpenMetadata requests Fuseki compaction after clearing a recreate run and after every successful indexing run. Compaction is best-effort: an indexing run can succeed even if disk reclamation fails. To compact manually: ```bash curl -u admin: -X POST \ 'http://localhost:3030/$/compact/openmetadata?deleteOld=true' ``` Fuseki returns an asynchronous task identifier. Inspect active and completed tasks at `/$/tasks` and confirm the data volume has enough space for both the old and replacement datasets. Unexpected journal growth usually indicates failed or skipped compaction, a write-heavy incremental workload, or a volume that filled before compaction completed. ## Configuration reference The OpenMetadata server reads the following settings from `conf/openmetadata.yaml`: | Environment variable | Default | | --- | --- | | `RDF_ENABLED` | `false` | | `RDF_BASE_URI` | `https://open-metadata.org/` | | `RDF_STORAGE_TYPE` | `FUSEKI` | | `RDF_ENDPOINT` | `http://localhost:3030/openmetadata` | | `RDF_REMOTE_ENDPOINT` | unset (deprecated fallback) | | `RDF_CONNECT_TIMEOUT_MS` | `2000` | | `RDF_REQUEST_TIMEOUT_MS` | `60000` | | `RDF_WRITE_MAX_RETRIES` | `2` | | `RDF_WRITE_RETRY_INITIAL_BACKOFF_MS` | `250` | | `RDF_WRITE_RETRY_MAX_BACKOFF_MS` | `2000` | | `RDF_BULK_ENTITY_BATCH_SIZE` | `100` | | `RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE` | `100` | | `RDF_BULK_LINEAGE_EDGE_BATCH_SIZE` | `50` | | `RDF_REMOTE_USERNAME` | `admin` | | `RDF_REMOTE_PASSWORD` | `admin` | | `RDF_DATASET` | `openmetadata` | | `RDF_INFERENCE_ENABLED` | `false` | Use `RDF_ENDPOINT` for new deployments. `RDF_REMOTE_ENDPOINT` remains a deprecated fallback for backward compatibility, and `RDF_ENDPOINT` takes precedence when both are set. Override the development credentials in every production deployment. ## Monitoring and failure diagnosis Useful Fuseki administration endpoints are: - `/$/ping` for liveness and readiness. - `/$/stats` for dataset and operation statistics. - `/$/tasks` for compaction and other asynchronous administration work. Monitor indexing records per second, SPARQL update latency, container RSS, page-cache availability, persistent-volume usage, and journal growth. OpenMetadata logs `RDF circuit breaker is open` after repeated connection failures or request timeouts; check Fuseki health, request latency, credentials, and network reachability before increasing retries. ## Inference In-process inference is disabled by default. Enabling `RDF_INFERENCE_ENABLED=true` causes general queries to construct the entire graph in the OpenMetadata JVM and should be limited to small graphs. The SPARQL console retains its explicit `inference` option for administrative experiments, but it has the same small-graph constraint. Use SPARQL 1.1 property paths for transitive lineage, inverse ownership, and domain or glossary inheritance. These execute inside Fuseki without copying the graph into the OpenMetadata process. Deployments that require persistent RDFS or OWL semantics should configure a Fuseki assembler dataset with an appropriate Jena reasoner and size it independently; server-side materialization or reasoning is operationally safer than per-request full-graph inference in OpenMetadata. The complete-lineage endpoint intentionally does not impose a row limit on its property-path query. Size `RDF_REQUEST_TIMEOUT_MS`, Fuseki resources, and client response handling for the largest lineage graph operators can request. Semantic search is not an exhaustive traversal: each seed expands at most 100 related graph candidates before reranking to the caller's requested result limit. Use the complete-lineage endpoint or direct SPARQL for exhaustive graph traversal. For local startup and API examples, see [RDF/Apache Jena Local Development Guide](rdf-local-development.md). --- ### Session Management Multi Node Design # Multi-Node Session and WebSocket Session Management Design ## 1. Status This document describes the current server-side session and websocket session design for OpenMetadata issue [#21971](https://github.com/open-metadata/OpenMetadata/issues/21971). The implementation is centered on these files: - `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionService.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStore.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/session/JdbcSessionStore.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/session/RedisSessionStore.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStoreFactory.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/JwtFilter.java` - `openmetadata-service/src/main/java/org/openmetadata/service/security/jwt/JWTTokenGenerator.java` - `openmetadata-service/src/main/java/org/openmetadata/service/socket/SocketAddressFilter.java` - `openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java` - `openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java` - `openmetadata-service/src/main/java/org/openmetadata/service/cache/CacheBundle.java` ## 2. Problem Server-managed login state used to depend on pod-local servlet state. That breaks in a multi-node deployment because login, callback, refresh, logout, and websocket reconnects can land on different pods. The key failure modes are: - A login or OIDC/SAML callback starts on one node and completes on another. - Refresh state is unavailable when the request is routed to a different node. - Logout or session revocation is not visible to all nodes. - A websocket can remain connected after the browser session is revoked. - A secure websocket handshake can be spoofed if the server trusts a client-supplied `userId`. ## 3. Goals 1. Store server-managed user sessions in a shared, authoritative backend. 2. Support both JDBC-backed sessions and Redis-backed sessions. 3. Bind newly issued browser access JWTs to the server-side session that issued them. 4. Keep provider refresh tokens and OpenMetadata refresh tokens server-side. 5. Make refresh safe under cross-node concurrency. 6. Make logout and revocation visible to API and websocket paths. 7. Reject websocket handshakes whose token principal, cookie session, or requested socket user do not match. 8. Close websocket connections for revoked or expired sessions. ## 4. Non-Goals 1. Changing personal access token or bot token semantics. 2. Replacing the browser-managed public OIDC flow. 3. Making legacy JWTs without a `sessionId` claim retroactively session-bound. 4. Guaranteeing instant cross-node websocket disconnect without pub/sub. Without pub/sub, remote sockets are closed by periodic validation. ## 5. Architecture ### 5.1 Component Model | Component | Responsibility | | --- | --- | | `SessionService` | Creates, activates, refreshes, revokes, expires, and prunes sessions. Owns the Caffeine near-cache and revocation listeners. | | `SessionStore` | Shared persistence contract used by both JDBC and Redis stores. | | `JdbcSessionStore` | Default store backed by the `user_session` table through `SessionRepository`. | | `RedisSessionStore` | Optional Redis store with key TTLs, per-user status indexes, and Lua compare-and-set on `version` plus index maintenance. | | `SessionStoreFactory` | Selects Redis when Redis cache is configured and available; otherwise uses JDBC. Refuses Redis-to-JDBC fallback when Redis is configured but unavailable. | | `SessionCookieUtil` | Reads, writes, validates, and clears the opaque `OM_SESSION` cookie. | | `JWTTokenGenerator` | Issues OpenMetadata JWTs and can include the `sessionId` claim for session-backed auth flows. | | `JwtFilter` | Validates JWTs. If the JWT has `sessionId`, it reloads the session from the store and requires an active, unexpired, username-matching session. | | `SocketAddressFilter` | Validates websocket handshake identity and session state before Socket.IO sees the connection. | | `WebSocketManager` | Tracks sockets by user and by session, sends events, and disconnects revoked or inactive session sockets. | | `OpenMetadataApplication` | Wires `SessionService` into auth handlers, websockets, revocation listeners, and the websocket session validator. | | `CacheBundle` | Handles cache invalidation pub/sub. Session invalidation messages also disconnect sockets on remote pods. | ### 5.2 Storage Selection `SessionStoreFactory` chooses the store at application startup: - If `cache.provider = redis` and Redis is available, sessions use `RedisSessionStore`. - If Redis is configured but unavailable, startup fails closed. - If Redis is not configured, sessions use `JdbcSessionStore`. The system does not fail over live from Redis to JDBC. Mixing stores would split active sessions across backends and make revocation unpredictable. ### 5.3 Session Cache Each pod keeps a local Caffeine near-cache: - maximum size: `10_000` - expire after access: `10s` The cache is a performance optimization, not a correctness boundary. Security-sensitive checks use fresh reloads where revocation must be observed immediately. ## 6. User Session Management ### 6.1 Session ID `UserSession.id` is an opaque bearer secret carried in the `OM_SESSION` cookie. It is generated by `SessionIdGenerator` from secure random bytes and base64url encoded without padding. It is not a UUID. ### 6.2 Session Types Current session types: - `AUTH`: browser or interactive user auth session. - `MCP`: reserved for future interactive MCP session support. ### 6.3 Session Status `SessionStatus` values: - `PENDING`: login started, callback not completed. - `ACTIVE`: usable session. - `REFRESHING`: one node holds the refresh lease. - `REVOKED`: logout or session-limit revocation. - `EXPIRED`: timeout reached. ### 6.4 Session Fields The important logical fields are: ```json { "id": "opaque-session-id", "type": "AUTH", "provider": "openmetadata", "status": "ACTIVE", "userId": "uuid", "username": "alice", "email": "alice@example.com", "omRefreshToken": "fernet:encrypted-token", "providerRefreshToken": "fernet:encrypted-provider-token", "redirectUri": "https://ui.example.com/callback", "state": "oidc-state", "nonce": "oidc-nonce", "pkceVerifier": "pkce-verifier", "version": 7, "refreshLeaseUntil": 1741300000000, "createdAt": 1741200000000, "updatedAt": 1741200005000, "lastAccessedAt": 1741200005000, "expiresAt": 1743792000000, "idleExpiresAt": 1741804800000 } ``` Refresh tokens are encrypted before persistence with `Fernet.encryptIfApplies(...)`. If the Fernet key is not configured, session creation fails instead of writing plaintext refresh tokens. ### 6.5 Session Creation Basic, LDAP, and OpenMetadata login create an `ACTIVE` session directly: 1. Validate credentials. 2. Resolve the provisioned OpenMetadata user. 3. Persist or receive the OpenMetadata refresh token. 4. Create an `ACTIVE` `AUTH` session. 5. Encrypt and store the refresh token in the session. 6. Write the `OM_SESSION` cookie. 7. Return an OpenMetadata-signed JWT with a `sessionId` claim. If user lookup or session creation fails after a refresh token is created, the refresh token is deleted. ### 6.6 Pending Session Activation SAML and confidential OIDC use pending sessions: 1. Login creates a `PENDING` `AUTH` session containing redirect state, OIDC state, nonce, and PKCE verifier when applicable. 2. The callback loads the pending session from the shared store. 3. The user is created or updated. 4. The OpenMetadata refresh token is inserted. 5. `activatePendingSession` expires the pending session. 6. A brand-new active session ID is generated and stored. 7. The active session cookie replaces the pending cookie. 8. The browser receives an OpenMetadata-signed JWT with the active session ID. Issuing a new active session ID during activation is the session fixation defense. The pre-auth cookie value is never reused for the authenticated session. If activation fails, the newly inserted refresh token is deleted and no JWT is issued. ### 6.7 Refresh Refresh is guarded by an optimistic lease: 1. Load the session from `OM_SESSION`. 2. Reject missing, expired, pending, revoked, or already expired sessions. 3. If another node holds a non-stale `REFRESHING` lease, return retry guidance through `SessionRefreshInProgressException`. 4. Acquire the lease by writing `REFRESHING`, setting `refreshLeaseUntil`, and incrementing `version` with compare-and-set. 5. The winning node decrypts the stored refresh token. 6. The provider or OpenMetadata refresh token is rotated as needed. 7. `completeRefresh` writes the refreshed session back to `ACTIVE`, clears the lease, updates idle expiry without extending beyond the absolute session expiry, and increments `version`. 8. The response contains a new OpenMetadata-signed JWT bound to the same session ID. Lease duration is currently `15s`. ### 6.8 Logout and Revocation Logout calls `SessionService.revokeSession(request, response)`: 1. Read `OM_SESSION`. 2. Reload the session from the authoritative store. 3. Write `REVOKED` with compare-and-set. 4. Clear `refreshLeaseUntil`. 5. Clear the `OM_SESSION` cookie. 6. Notify local revocation listeners. Session limit enforcement also uses `revokeSession` for least-recently-used active sessions. The limit is configured by `authenticationConfiguration.maxActiveSessionsPerUser`, exposed through `AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER` in `openmetadata.yaml`. The default is `5`; values below `1` fall back to the default. ### 6.9 Expiration and Cleanup `SessionService` runs cleanup every `15m`: - mark expired sessions as `EXPIRED` - prune `REVOKED` and `EXPIRED` rows after `7d` - process in bounded batches For Redis, primary keys have TTLs and cleanup methods are no-ops. Session correctness still relies on in-process status and expiry checks. Default timeouts: - pending session timeout: `10m` - authenticated session expiry: `authenticationConfiguration.sessionExpiry`, default `7d` - refresh lease: `15s` - cleanup retention: `7d` The `OM_SESSION` cookie max age is rewritten during refresh lease acquisition and is capped at the remaining effective session lifetime. ## 7. Session-Bound JWTs Server-managed auth flows return OpenMetadata-signed access JWTs with: ```json { "sub": "alice", "tokenType": "OM_USER", "sessionId": "opaque-session-id" } ``` `JwtFilter` handles the claim as follows: 1. Validate the JWT signature, expiry, token type, principal, and token-specific rules. 2. If there is no `sessionId` claim, preserve existing stateless behavior. 3. If `sessionId` exists, call `SessionService.getFreshSessionById(sessionId)`. 4. Require: - session exists - status is `ACTIVE` - session is not expired - session username matches the JWT principal 5. Reject the token when any check fails. This means session-backed browser API requests now consult the shared session store. That is an intentional tradeoff in the current implementation: revocation is observed on the next request instead of waiting for access-token expiry. PATs, bot tokens, and legacy JWTs without `sessionId` remain stateless. ## 8. WebSocket Session Management ### 8.1 Handshake Validation `SocketAddressFilter` runs before the Socket.IO server receives the connection. When secure websocket connections are enabled: 1. Extract and validate the `Authorization` header. 2. Resolve the token principal from JWT claims. 3. Resolve the principal's user UUID server-side. 4. Reject the request if the query `userId` is present and does not match the resolved user UUID. 5. Inject the server-resolved `UserId` header for `WebSocketManager`. 6. If the JWT has `sessionId`, inject a `SessionId` header. 7. Validate `OM_SESSION` when present: - reload the session fresh - require `ACTIVE` - require not expired - require session username to match the token principal - require cookie session ID to match token `sessionId` when both are present If no `OM_SESSION` cookie is present: - session-bound JWTs are accepted because `JwtFilter` has already validated the session ID against `SessionService` - legacy secure JWTs without `sessionId` are rejected with `401 Session is required` - non-secure websocket mode remains compatible with existing query-based behavior The filter no longer forwards trust from the user-supplied `userId` query parameter when secure mode is enabled. ### 8.2 Socket Tracking `WebSocketManager` maintains two local maps per pod: - `activityFeedEndpoints`: `userId -> socketId -> SocketIoSocket` - `socketSessionIds`: `socketId -> sessionId` On connection: 1. Read `UserId` from the injected header, falling back to query only for legacy/non-secure paths. 2. Read `SessionId` from the injected header, falling back to query `sessionId` only for legacy paths. 3. Store the socket in the user's local socket map. 4. Store the socket-to-session mapping when a session ID is available. On disconnect, both maps are cleaned up. The connection log records user and remote address only. It does not log initial headers, so bearer tokens are not written to logs. ### 8.3 Revocation-Driven Disconnect `SessionService` exposes revocation listeners. `OpenMetadataApplication` registers a listener that: 1. Converts the revoked session's `userId` to UUID. 2. Calls `WebSocketManager.disconnectForSession(userId, sessionId)` on the local pod. 3. Publishes a `"session"` invalidation message through cache invalidation pub/sub when available. `CacheBundle` handles remote `"session"` invalidation messages: - if the message has a session ID, call `disconnectForSession(userId, sessionId)` - if no session ID is present, fall back to `disconnectAllForUser(userId)` for backward compatibility This gives targeted disconnects. Logging out one browser session does not force-close other sessions for the same user. ### 8.4 Periodic WebSocket Validation `OpenMetadataApplication.WebSocketSessionValidator` runs every `60s` by default. Operators can tune the interval with the `openmetadata.websocketSessionValidationIntervalSeconds` system property or the `WEBSOCKET_SESSION_VALIDATION_INTERVAL_SECONDS` environment variable. Values below `15s` are clamped to `15s`. Each run calls `WebSocketManager.disconnectInactiveSessions(sessionService, intervalMillis)`, which: 1. Iterates local sockets with known `sessionId`. 2. Reloads a socket's session fresh through `SessionService.getFreshSessionById` only when that socket's revalidation interval is due. 3. Disconnects sockets whose session is missing, not `ACTIVE`, expired, or owned by a different user. This is the fallback when there is no cross-pod pub/sub. With JDBC and no pub/sub, a socket on a remote node is closed within the validator interval instead of immediately. ## 9. End-to-End Flow ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## 10. Consistency Model ### 10.1 API Requests For tokens with `sessionId`, the session store is authoritative. A revoked or expired session is rejected on the next API request that uses that token. For tokens without `sessionId`, existing stateless behavior is preserved. ### 10.2 Refresh Refresh uses optimistic compare-and-set on `version`, so only one node can hold the refresh lease for a session at a time. JDBC implements this through the session repository update path. Redis implements it with a Lua CAS script over the stored session JSON. The Redis script also removes the session ID from all non-terminal per-user status indexes and adds it to the target non-terminal index before returning, so the JSON write and index movement succeed or fail together. ### 10.3 WebSockets Websocket consistency has two layers: - event-driven disconnect through local revocation listeners and optional cache invalidation pub/sub - polling-based validation with a `60s` default interval and `15s` minimum The event path is immediate when revocation occurs on the same pod or pub/sub delivers the remote event. The polling path bounds staleness when pub/sub is unavailable, and each socket is fresh loaded at most once per validation interval. ## 11. Operational Characteristics | Path | Store behavior | | --- | --- | | Login | create active or pending session | | OIDC/SAML callback | fresh load pending session, expire pending session, create active session | | Session-bound API request | fresh load session by `sessionId` | | Refresh | load session, acquire CAS lease, complete CAS update | | Logout | fresh load session, CAS revoke, clear cookie | | WebSocket handshake | validate JWT, optionally fresh load cookie session | | WebSocket validator | throttled fresh load for each tracked socket with `sessionId` | Redis deployments should monitor Redis availability as auth-critical infrastructure. When Redis is configured for sessions, the service refuses to start without it. ## 12. Security Properties 1. `OM_SESSION` is opaque and high entropy. 2. `OM_SESSION` is written as an HTTP-only cookie. 3. Provider refresh tokens and OpenMetadata refresh tokens are encrypted at rest. 4. Refresh tokens are not returned to the browser by server-managed auth flows. 5. Pending-session activation issues a brand-new active session ID. 6. Session-bound JWTs are invalid once the backing session is revoked, expired, deleted, or owned by a different user. 7. Secure websocket mode derives socket user identity from the JWT principal, not from query params. 8. Websocket logs do not include initial headers or bearer tokens. 9. Revocation targets the revoked session instead of disconnecting every socket for the user. ## 13. Test Coverage Relevant unit coverage includes: - `SessionServiceTest` - `SessionCookieUtilTest` - `SessionTimeoutResolverTest` - `SessionStoreContractTest` - `RedisSessionStoreTest` - `JwtFilterTest` - `BasicAuthServletHandlerTest` - `LdapAuthServletHandlerTest` - `SamlAuthServletHandlerTest` - `AuthenticationCodeFlowHandlerTest` - `SocketAddressFilterTest` - `WebSocketManagerTest` Relevant integration coverage includes: - `SessionMultiNodeIT` - `SessionRedisMultiNodeIT` - `SessionMultiNodeCluster` Important scenarios covered or expected from this suite: - login on one node and refresh/logout on another - pending OIDC/SAML callback state loaded from shared session storage - refresh lease contention - stale cache behavior after revocation - Redis-backed cross-node sessions - websocket principal binding - per-session websocket disconnect - session-bound JWT rejection for revoked sessions ## 14. Tradeoff Resolutions 1. Session-bound browser API requests intentionally reload session state on JWT validation. This is the chosen correctness boundary: logout and revocation are observed on the next browser API request instead of waiting for access-token expiry. 2. Tokens without `sessionId` remain on existing JWT semantics. This preserves PAT, bot, public OIDC, and rolling-upgrade compatibility. New server-managed auth responses include `sessionId`. 3. Non-secure websocket mode remains query-param based only for backward compatibility. Production deployments should keep secure websocket connections enabled so `SocketAddressFilter` derives the socket user from the JWT principal and records the session ID. 4. The active-session cap is now configurable with `authenticationConfiguration.maxActiveSessionsPerUser` and `AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER`; the default remains `5`. 5. Secure, session-managed websocket handshakes now record a session ID from the JWT claim or `OM_SESSION` cookie. The validator checks those sockets on a configurable interval with a `60s` default and `15s` minimum; sockets without session IDs are legacy/non-secure compatibility cases. 6. Cross-pod websocket revocation has two paths: cache invalidation pub/sub for immediate targeted disconnects when available, and the configurable websocket validator as the bounded-staleness fallback for JDBC-only deployments. --- ### Streamable Logs # Streamable Ingestion Logs This document describes the end-to-end design of OpenMetadata's streamable ingestion-pipeline log system: how logs flow from a running connector to durable S3 storage, how the UI reads them while a run is in progress, and how the system handles long idle gaps, restarts, and abandoned runs. ## Overview Ingestion pipelines (metadata, profiler, lineage, usage, dbt, etc.) emit logs as they run. Operators need to: - Watch logs **live** while a pipeline is running, including for long-running connectors that can take hours. - Read logs **after the run ends**, with a single canonical artifact per run. - Recover gracefully from server restarts, network blips, and connector idle gaps. OpenMetadata addresses this with a server-side log storage abstraction backed by S3 (or any S3-compatible store like MinIO). The connector pushes log batches over HTTP; the server persists them and serves both live and post-run reads. ## Architecture ``` ┌──────────────────────┐ │ Python ingestion │ POST /logs/{fqn}/{runId} (append) │ connector │ POST /logs/{fqn}/{runId}/close (finalize) │ (logs_mixin.py) │ └──────────┬───────────┘ │ HTTP ▼ ┌──────────────────────┐ │ OpenMetadata server │ │ IngestionPipeline │ │ Resource │ └──────────┬───────────┘ │ LogStorageInterface ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ S3LogStorage │────────▶│ S3 / MinIO bucket │ │ (streaming, in-mem │ │ partial.txt │ │ buffers, sweeper) │ │ logs.txt │ └──────────┬───────────┘ └──────────────────────┘ │ SSE / GET (paginated / download) ▼ ┌──────────────────────┐ │ OpenMetadata UI │ │ (live tail + history)│ └──────────────────────┘ ``` The `LogStorageInterface` abstraction supports multiple backends: | Backend | Purpose | |---------|---------| | `S3LogStorage` | Production: stores logs durably in S3 / MinIO. The focus of this document. | | `DefaultLogStorage` | Backward-compat: delegates to the pipeline service client (Airflow / Argo). No first-class storage. | This document covers the `S3LogStorage` implementation. ## Storage Layout Each pipeline run is identified by a `(fqn, runId)` tuple. On S3 the layout is: ``` {bucket}/{prefix}/ # prefix defaults to "pipeline-logs" {sanitizedFQN}/{runId}/ partial.txt # readable view during the run logs.txt # final artifact, materialized at /close .active/{sanitizedFQN}/{runId}/{serverId} # heartbeat marker ``` **`partial.txt`** is the durable, readable view of an in-progress run. It is updated periodically as the connector appends batches. It carries durable offset state in S3 user-defined metadata: | Metadata key | Purpose | |--------------|---------| | `x-amz-meta-last-flushed-line` | Logical line counter at the moment of this PUT. Drives retry idempotency and post-restart recovery. | | `x-amz-meta-total-bytes` | Cross-check on body size; helps detect drift. | | `x-amz-meta-writer-epoch` | Bumped each time a fresh OM-server instance picks up the stream after a restart. | | `x-amz-meta-writer-version` | Identifies the writer code version. Useful during migration windows. | **`logs.txt`** is the canonical post-run artifact. It is created **only** at `/close` (or by the abandoned-run sweeper), as a server-side S3 copy of the final `partial.txt`. Content matches `partial.txt` exactly at the moment of close. **`.active/...`** markers are dropped as a side effect of `appendLogs`. They have no functional role in correctness; they are operational hints for diagnostics ("which OM-server instance most recently saw this run"). A bucket lifecycle policy ensures cleanup: - `expirationDays` (default 30) on the `pipeline-logs/` prefix expires all logs after the retention window. ## Run Lifecycle ### 1. Connector emits a batch The Python ingestion runner buffers log lines and POSTs batches to the server: ``` POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId} Content-Type: application/json "" OR { "logs": "", "connectorId": "...", "compressed": true } ``` `IngestionPipelineResource.writePipelineLogs` decodes the body and calls `repository.appendLogs(fqn, runId, content)`, which delegates to `S3LogStorage.appendLogs`. ### 2. Server-side append `S3LogStorage.appendLogs` does five things, all in memory, all under a per-stream `ReentrantLock`: 1. **Increments `totalLinesAppended`**, the monotonic logical line counter that anchors retry idempotency. 2. **Appends to `SimpleLogBuffer`** (in-memory ring, capacity 1000 lines). This is the source for the SSE/WebSocket live-tail UI experience. It is bounded; oldest lines evict on overflow. It is **not** load-bearing for durability. 3. **Appends to `pendingFlush`** (in-memory queue, no fixed cap, byte-tracked). This is the durable-pending-write queue and survives until the next successful PUT. 4. **Notifies SSE listeners**, fanning out the new lines to any open live-tail HTTP connections. 5. **Schedules an early flush** if `pendingFlush` exceeds `earlyFlushWatermarkBytes` (default 5 MB). This protects against memory bloat under bursty writes. A single-threaded `cleanupExecutor` schedules the periodic flush, the abandoned-run sweeper, and metrics updates. ### 3. Periodic flush to `partial.txt` Every `partialFlushIntervalMinutes` (default 2) and on demand from the early-flush watermark, `writePartialLogsForStream` runs under the per-stream lock: 1. Snapshot `pendingFlush` and clear it. 2. If empty, no-op (idle streams cost nothing). 3. `GetObject partial.txt` → reads `Content-Length` and metadata from the response headers. On 404, treat as empty. 4. Build new metadata (`last-flushed-line`, `total-bytes`, `writer-epoch`, `writer-version`). 5. **If existing body < 5 MB** — read the body, build merged body = existing + `\n`-joined snapshot, `PutObject` atomically. 6. **If existing body ≥ 5 MB** — abort the body stream and concatenate server-side via Multipart Upload: `CreateMultipartUpload`, `UploadPartCopy` (existing body as part 1), `UploadPart` (new content as part 2, the last part has no 5 MB minimum), `CompleteMultipartUpload`. The merged body never enters JVM heap and is not re-uploaded. 7. On failure, abort any in-flight multipart upload, re-merge the snapshot to the head of `pendingFlush`, and try again next tick. No data loss. Because `pendingFlush` is unbounded by the `SimpleLogBuffer` cap, no line is ever evicted before being flushed. ### 4. Live read while running The UI's "live logs" view does two things in parallel: - **HTTP GET** `/logs/{fqn}/{runId}?after={cursor}` for paginated history. The server reads `partial.txt` from S3 and concatenates the in-memory `pendingFlush` snapshot for the most-recent-tail bytes that haven't yet been flushed. The cursor is a line offset. - **Server-Sent Events (SSE)** for live tail. The endpoint registers a `LogStreamListener` against the stream key and pushes new lines as `notifyListeners` fires from each `appendLogs`. This gives the user "everything written so far" via GET and "everything written in real time from now on" via SSE. ### 5. `/close` finalization When the connector terminates (success, graceful failure, or graceful abort), it calls: ``` POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId}/close ``` `S3LogStorage.closeStream` runs under the per-stream lock: 1. **Final flush**: drain remaining `pendingFlush` to `partial.txt` (same path as the periodic flush). 2. **Server-side copy** `partial.txt` → `logs.txt`. Bytes do not transit through OM. Cheap and constant-time regardless of log size. 3. **Delete `partial.txt`**. 4. **Best-effort delete** the `.active/{fqn}/{runId}/{serverId}` marker. 5. Drop in-memory state for the stream (`activeStreams`, `pendingFlush`, `totalLinesAppended`, `recentLogsCache`, the per-stream lock). `/close` is idempotent. A second call finds no `partial.txt` and no in-memory state; it is a graceful no-op. A `/close` that arrives after the abandoned-run sweeper already finalized the stream behaves the same way. ### 6. Post-`/close` reads Once `/close` completes, `logs.txt` is the canonical artifact. `getLogs(fqn, runId)` reads it directly. Pagination is by line offset; the response includes `after` (next cursor) and `total` (total bytes / lines). There is also a download endpoint that streams the full file (or composes from segments / partial in legacy fallbacks). ## Read Paths | Endpoint | Pre-`/close` | Post-`/close` | |----------|-------------|---------------| | `GET /logs/{fqn}/{runId}` | Reads `partial.txt` + appends `pendingFlush` snapshot. Apply cursor pagination. | Reads `logs.txt`. | | `GET /logs/{fqn}/{runId}/download` | Streams `partial.txt`. | Streams `logs.txt`. | | `GET /logs/{fqn}/stream/{runId}` (SSE) | Live tail with resume cursors and an explicit end-of-stream event. One shared reader per run. | Streams the finished log, then closes with `reason: runFinished`. | | `GET /logs/{fqn}/stream/{runId}` (SSE) | Same engine, but each frame is one raw log line with no cursor. Legacy shape. | Same. | The SSE read path is documented on its own in [ingestion-log-streaming.md](ingestion-log-streaming.md) — event schema, resume cursors, and the limits that bound it. Legacy `partial.txt` files written by older code (without S3 metadata) read normally; the new flush logic treats them as "no prior offset" and merges any new content correctly. ## Abandoned-Run Recovery Connectors can die without calling `/close` — process killed, OOM, network partition, infrastructure failure. To bound resource use and still produce a final `logs.txt`, a sweeper runs periodically: - **Schedule**: every `cleanupIntervalMinutes` (default 60). - **Threshold**: `streamTimeoutMinutes` since last `appendLogs` (default 1440 = 24h). For each expired stream, the sweeper does the same finalization steps as `/close` (final flush, copy to `logs.txt`, delete `partial.txt`, drop in-memory state). The end result is identical: an abandoned run produces a finalized `logs.txt` artifact that the UI can read, just delayed. The 24h default is intentionally lenient: typical idle gaps in slow connectors (waiting on source queries, batch boundaries, queues) are minutes-to-hours, not days. Operators can tune the threshold downward in deployments where memory pressure from many parallel runs requires more aggressive reclamation. ## Failure Modes & Recovery | Failure | Recovery | |---------|----------| | S3 PUT fails during periodic flush | `pendingFlush` snapshot is restored under the lock. Next tick retries. No data loss. | | OM-server restart mid-run | All in-memory state lost. `partial.txt` on S3 retains all previously-flushed content. The next `appendLogs` re-creates state; the first flush after restart reads `partial.txt` (with metadata) and resumes from `last-flushed-line`. Worst-case loss: lines that were in `pendingFlush` at restart time, bounded above by `partialFlushIntervalMinutes`. | | Connector dies without `/close` | Abandoned-run sweeper finalizes the run after `streamTimeoutHours`. `logs.txt` is materialized from the most recent `partial.txt`. | | `/close` retries after partial success | All steps are idempotent. Second call finds no `partial.txt` and no in-memory state; no-op. | | Concurrent `appendLogs` and cleanup | The per-stream lock serializes them. Cleanup finds the stream "fresh" again and skips it next tick. | | Bucket lifecycle expires `partial.txt` mid-run | Should not happen at default `expirationDays = 30`. If misconfigured (very low retention), the next flush would treat it as a fresh `partial.txt` and start over. Recommended floor: 7 days. | ## Configuration All settings live under `LogStorageConfiguration` in `openmetadata.yaml`: | Field | Default | Description | |-------|---------|-------------| | `bucketName` | (required) | S3 bucket for log storage. | | `prefix` | `pipeline-logs` | Key prefix within the bucket. | | `enableServerSideEncryption` | `true` | Apply SSE on every PUT. | | `sseAlgorithm` | `AES_256` | Or `AWS_KMS` (requires `kmsKeyId`). | | `storageClass` | `STANDARD_IA` | S3 storage class for log objects. | | `expirationDays` | 30 | Bucket lifecycle: expire all logs after this many days. | | `streamTimeoutMinutes` | 1440 | Idle threshold (in minutes) before the abandoned-run sweeper finalizes a stream. | | `cleanupIntervalMinutes` | 60 | How often the sweeper wakes up to check for abandoned streams. | | `partialFlushIntervalMinutes` | 2 | Periodic `pendingFlush` → `partial.txt` cadence. | | `earlyFlushWatermarkBytes` | 5242880 (5 MB) | Triggers an out-of-band flush when `pendingFlush` exceeds this size. | | `pendingFlushAlertAfterFailures` | 10 | Emit an alerting metric after this many consecutive failed flushes for a stream. | | `maxConcurrentStreams` | 100 | Bound on in-flight pipeline runs per OM-server instance. | | `awsConfig.*` | — | AWS credentials / region / endpoint (also supports IAM role + custom endpoints for MinIO). | ## Concurrency Model Coordination is a per-stream lock keyed by `streamKey = fqn + "/" + runId`. The lock is held for the duration of `appendLogs`, periodic flush, abandoned-run cleanup, and `/close`. Locks are backed by a Guava `Striped` with a fixed stripe count, so memory does not grow with completed-run accumulation; the same key always maps to the same lock instance, eliminating the acquire-vs-remove race that a per-key map would have. False contention across stripes is bounded by `maxConcurrentStreams << stripe count`. A single-threaded `ScheduledExecutorService` (`cleanupExecutor`) drives: - Periodic flushes (`writePartialLogs`) - Abandoned-run sweeper (`cleanupAbandonedStreams`) - Metrics updates (`updateStreamMetrics`) - One-shot early flushes scheduled by the watermark trigger Under sustained burst load, scheduled tasks queue on this single thread. This is intentional: it bounds resource use and avoids unbounded thread creation under spikes. If a deployment regularly sees queue backlog, the watermark or flush interval can be tuned. ## Observability Key metrics exposed by `StreamableLogsMetrics`: - `om_streamable_logs_log_shipment_*` — distribution of append latencies. - `om_streamable_logs_logs_sent` / `logs_failed` — counter of successful and failed appends. - `om_streamable_logs_batch_size` — distribution of lines per batch. - `om_streamable_logs_s3_*` — distribution of S3 read/write latencies and counters of S3 errors. - `om_streamable_logs_pending_part_uploads` — gauge for monitoring queue backlog (legacy, will be retired with multipart removal). - `om_streamable_logs_multipart_uploads` — gauge for active multipart uploads (legacy, will be retired). - `om_streamable_logs_pending_flush_bytes` — gauge for in-memory `pendingFlush` size per stream (new). - `om_streamable_logs_consecutive_flush_failures` — gauge per stream (new). Recommended alerts: - `pending_flush_bytes` > 50 MB sustained → memory pressure or persistent S3 failures. - `consecutive_flush_failures` ≥ 10 → S3 connectivity or auth issue. - `s3_errors` rate > 1/min → S3 health degradation. ## Multi-Server Topology The design assumes single-writer-per-run: an ALB / load balancer enforces sticky sessions for `(fqn, runId)` via the `PIPELINE_SESSION` cookie set on the first `appendLogs` response. All subsequent requests for the same run land on the same OM-server instance for the lifetime of the run. If stickiness is broken (cookie stripped by a proxy, multi-cluster routing without coordination), two OM-server instances could write to the same `partial.txt` and clobber each other. This is **out of scope** for the current design. A future iteration could move offset state to the database for cross-server coordination. ## References - Source files: - `openmetadata-service/src/main/java/org/openmetadata/service/logstorage/S3LogStorage.java` - `openmetadata-service/src/main/java/org/openmetadata/service/logstorage/LogStorageFactory.java` - `openmetadata-spec/src/main/java/org/openmetadata/service/logstorage/LogStorageInterface.java` - `openmetadata-service/src/main/java/org/openmetadata/service/resources/services/ingestionpipelines/IngestionPipelineResource.java` - `ingestion/src/metadata/utils/streamable_logger.py` - `ingestion/src/metadata/ingestion/ometa/mixins/logs_mixin.py` - Related PRs: #23590, #24198, #24287, #24410 --- ### Tech Debt # Tech Debt Ledger Findings from a one-time repository audit worth fixing. **Ordered by impact ÷ size** (biggest payoff per unit of effort first) — so small fixes that unblock a golden principle or close a hazard rank above large architectural refactors. **Agent?** = can an agent fix it *unsupervised* (mechanical, verifiable) vs. supervised (needs judgment/review) vs. no (human/native-speaker/architectural). ## Tier 1 — Quick wins (high impact, tiny size) — do first | # | Finding | Location | Why it matters | Size | Agent? | |---|---|---|---|---|---| | 1 | 1 file with a bare `except:` | `ingestion/.../database/hive/custom_hive_connection.py:181` | The only blocker to gating ruff `E722` (Golden Principle #5) | 1 line | **unsupervised** | | 2 | 3 hand-written UI files missing the Apache header | `LineageTable.interface.ts`, `mocks/rests/applicationAPI.mock.ts`, `EntityLineage/.../LineageLayers.interface.ts` | Trips the license gate; `yarn license-header-fix` fixes it | trivial | **unsupervised** | | 3 | 21 Java files with wildcard imports | `attachments/AzureAssetService.java` (3), `PersonaResource.java`, `StoredProcedureResource.java`, … | Unblocks making no-wildcard blocking; violations are in files edited *this month* (will regress) | ~21 files, mechanical | **unsupervised** | | 4 | POM hygiene: duplicate deps + declared-order inversion | `openmetadata-spec/pom.xml` (declares `common` twice), `openmetadata-dist/pom.xml` (declares `openmetadata-ui` twice); `` lists `spec` before its dep `common` | Minor, but `spec→common` contradicts the "foundation is first" reading of the reactor list | tiny | **unsupervised** | | 5 | Stale MCP namespace in a skill | `.claude/skills/playwright-validation/SKILL.md:59,63,67,68` — `mcp__playwright__*` (real server is `mcp__playwright-test__*`) | The only reference that fails **at tool-call time**, not just read time | 4 lines | **unsupervised** | | 6 | `AGENTS.md` still carries contradictions corrected in CLAUDE.md | `AGENTS.md:12` (Webpack→Vite), `:105/:223` (use Ant Design), `:13` (Python 3.10-3.12) | CLAUDE.md was fixed; AGENTS.md (Codex's entry doc) still misleads | small doc edit | **unsupervised** | | 7 | `openmetadata-shaded-deps` has no "do not edit" marker | module has only POMs + a build artifact; not in any instruction doc | An agent "upgrading" it breaks the `es.*`/`os.*` relocation → cascading compile failures | 1 rule/note | **unsupervised** | ## Tier 2 — Medium (real value, bounded but not trivial) | # | Finding | Location | Why it matters | Size | Agent? | |---|---|---|---|---|---| | 8 | No secret scanner + no key-extension gitignore | CI (`.github/workflows`), `.gitignore` | Nothing catches a committed secret; fixture dirs are Snyk-excluded — an agent could commit a real key undetected | medium | **supervised** (workflow edit needs auth) | | 9 | Migration append-only is unenforced | `bootstrap/sql/migrations/**`; no CI check, no runtime checksum-abort | Editing an applied migration silently no-ops on existing DBs / drifts on fresh installs, with no error | medium | **supervised** (workflow edit needs auth) | | 10 | CodeQL runs on `workflow_dispatch` only | `.github/workflows/codeql.yml` | SAST does not run on PRs — a whole class of checks never gates contributions | small config | **supervised** (workflow edit needs auth) | | 11 | ~86 broad-excepts that silently swallow | `ingestion/src/metadata/**` (the ~5.5% with no log/re-raise) | The genuinely risky subset of the (sanctioned) broad-except idiom | ~86 sites, per-site judgment | **supervised** | | 12 | `mssql ↔ azuresql` circular sibling import | `database/mssql/connection.py:44` ↔ `database/azuresql/connection.py` | The one connector import cycle; the rest are clean (`timescale→postgres` is legit subclassing) | small | **supervised** | | 13 | `any` outside `generated/` | 461 files (top-5 are generated) | Erodes type safety; blocked from becoming a blocking lint until `generated/` is excluded and re-measured | large but incremental | **supervised** | | 14 | basedpyright baseline of 11,927 findings | `.basedpyright/baseline.json` (926 files) | The ratchet holds (no new errors) but the debt is large; each burn-down shrinks it | very large, incremental | **supervised** | ## Tier 3 — Large / architectural (high impact, large size) — plan, don't rush | # | Finding | Location | Why it matters | Size | Agent? | |---|---|---|---|---|---| | 15 | Java package cycles: `resources ↔ jdbi3` (130/99), 18/21 pairs cyclic | `openmetadata-service/src/main/java/org/openmetadata/service/**` | The repo's **own** #1 principle (acyclic layering) is violated *inside* the core backend; much is misplaced value types (`resources.feeds.MessageParser.EntityLink`) that belong in a shared package | large refactor | **no** (architectural) | | 16 | Frontend `components ↔ utils` cycle (130-module SCC) | `openmetadata-ui/.../src` (28 cyclic SCCs, 50 direct 2-cycles) | No import-boundary tooling; barrel/`*.interface.ts` mutual references + `utils` importing components | large | **no** (architectural) | | 17 | Generated-type leakage: 1,292 direct importers vs 93 in `rest/` | `openmetadata-ui/.../src/components`,`pages` | No anti-corruption layer; the stricter form of Golden Principle #3 sits at ~7% | large | **no** (architectural) | | 18 | Ant Design migration stalled | 864 UI files (18.3%), 68.5% edited ≤90d | The stated "use ui-core-components" direction isn't progressing; can't be a blocking rule until it does | very large, ongoing | **no** (program of work) | | 19 | ~250–396 untranslated English strings per non-en locale × 19 | `openmetadata-ui/.../src/locale/languages/*.json` | Reviewable defect CLAUDE.md flags; CI checks key-sync, not translation | large | **no** (native speakers) | | 20 | `py-tests` required check ≈ 69 min work-median | CI | The dominant PR wall-clock (ranked #1 by cost in the CI run-history analysis); a candidate for sharding/scoping | large CI work | **supervised** | | 21 | `playwright-postgresql-e2e` 82% pass (least-reliable required check) | CI | The required check most likely to fail for non-code reasons; re-run flakiness is ~0-measurable (fixed by new commits), so it hides as a low pass rate | investigation | **no** | ## Conflicts surfaced (not resolved here) - **"Lint-clean" ≠ "well-layered."** `openmetadata-service` is 100% on spotless/logging and ~99% on wildcard imports, yet is the **most** internally tangled module (18/21 cyclic package pairs). The two signals disagree about the module's health — see `quality.md`. - **Generated is model citizen and worst offender at once.** #3 (pure sink, 100%) and #17 (leakage, ~7%) are about the same `generated/` tree from opposite directions. Fixing #17 is a large refactor; #3 is already true. Don't let "#3 holds" imply the generated boundary is clean. - **Broad-except: Python idiom vs Java ban.** #11 (fix silent swallows) must not become "ban broad except" — that idiom is 75.5% of Python handlers, by design. --- ### Ui Code Quality Gate # UI code-quality gate How frontend quality is enforced on every `openmetadata-ui` PR, and what you must configure in SonarCloud and branch protection to complete it. The governing rule: **new code must be clean; existing debt is fixed gradually.** Every gate below is scoped to what a change *adds*, never to the whole file or the whole repo, so the backlog can never block a PR. ## The two sides Almost all new UI code is AI-generated, so a check that only runs in CI arrives too late — the model already chose the wrong pattern. Every rule therefore exists twice. | | Generation-time (agent is writing) | Review-time (CI) | |---|---|---| | **Knowledge** | `.claude/rules/*.md`, auto-loaded by `paths:` glob | — | | **Enforcement** | `.claude/settings.json` hooks | `ui-checkstyle` job | | **Self-serve** | `make ui-checkstyle-changed` | required status check | **One toolchain, three call sites.** The same ESLint config runs in the agent hook, in `make ui-checkstyle-changed`, and in the `ui-checkstyle` CI job. Prefer an off-the-shelf ESLint plugin over a bespoke script: a plugin matches the AST rather than diff text, gives live editor feedback, and composes with `--fix` and `eslint-disable`. Reach for a script only when a rule genuinely cannot be expressed in ESLint — `tw-guard` qualifies, because its antd/`.less` backlog (864 and 449 files) makes added-lines-only scoping unavoidable. Depth behind the rules lives in `skills/vendor/` — `react-best-practices`, `web-design-guidelines` and `composition-patterns`, vendored verbatim from [vercel-labs/agent-skills](https://github.com/vercel-labs/agent-skills) (MIT) so they need no install step. Skills load only when invoked, which is why the load-bearing subset is distilled into `.claude/rules/frontend-performance.md` and `.claude/rules/frontend-a11y.md`, which auto-load on matching files. **Checkstyle is the enforcement point — not pre-commit.** New gates are deliberately kept out of `.pre-commit-config.yaml`: every hook there is paid on every single commit. Committing must stay fast. `ui-checkstyle` is the one place a gate has to hold, and `make ui-checkstyle-changed` is how you get that answer locally before pushing. ## Run it locally ```bash make ui-checkstyle-changed # exactly what CI runs, on just your changed files ``` This is the command to trust — it runs the fixing steps (organize-imports, eslint, prettier, license headers, i18n sync, app-docs) **and** the audit gates (`tw-audit`, `tw-guard`). The gates are collected rather than short-circuited, so one failure does not hide the others. In CI, warnings appear in the sticky GitHub Actions comment on the PR titled **UI Checkstyle passed — lint findings in changed files**. The comment groups findings by rule and includes file, line, column, and message details for the PR's changed files. The same output is available in **Actions → UI Checkstyle → checkstyle → ESLint + Prettier + Organise Imports (src)**. Warnings remain non-blocking; ESLint errors and formatting changes still fail `ui-checkstyle`. ## What each gate enforces | Gate | Scope | Fails when | |---|---|---| | ESLint + Prettier + organize-imports | changed files | output differs from committed form | | Licence header | changed files | Apache-2.0 header missing/stale | | i18n key-sync | all locales | locale files out of sync with `en-us.json` | | `tw-audit` | changed files | hardcoded Tailwind value that maps to a design token | | `tw-guard` | **added lines** | new `antd` import or new `.less` file | | `jsx-a11y` (ESLint) | changed files | one of 19 zero-backlog accessibility rules trips | | SonarJS (ESLint) | changed files | one of 16 zero-backlog correctness rules trips | | OpenMetadata performance (ESLint) | changed files | eager route page import, unguarded lazy component, or unbounded module cache | | OpenMetadata import architecture (ESLint) | changed files | warning-only architecture, cycle, barrel, or request fan-out finding | | SonarCloud gate | **new code** | complexity, duplication, or new issues on lines this PR added | ## Component reuse — guidance, not a gate `.claude/rules/component-library.md` carries the table of what to import instead of hand-rolling (`Select` rather than `
`, and so on). **No linter knows this design system**, so that table is guidance and review, not an automated check. A bespoke `reuse-audit` script was built for this and then removed. It reimplemented, badly, what ESLint already does well: matching regexes against raw diff lines produced false positives on `data-role=`, on `[role="menu"]` selectors and inside comments, and its hand-rolled git handling could report "clean" when the diff had failed to load. Its one real advantage — inspecting only added lines — existed to tolerate a backlog of **16** instances, small enough not to justify ~430 lines of bespoke code and its own test suite. What CI enforces instead is that a hand-rolled widget must at least be *accessible*: `jsx-a11y` rejects an invalid `role`, a role missing its required `aria-*` props, and an unusable tab order. Using the library component is the easy way to satisfy that. ## Two severity tiers, chosen by measurement Every rule is on. Severity is decided by the rule's **measured backlog**, never by taste, because ESLint reports per *file* rather than per added line — an `error` rule with existing violations would fail PRs for code they merely touched. | Tier | Meaning | Today | |---|---|---| | `error` | zero measured backlog — blocking | 16 SonarJS + 19 jsx-a11y + 3 OpenMetadata performance | | `warn` | has a backlog — visible in the editor and CI output, not blocking | 21 SonarJS, 15 jsx-a11y, 4 React, 10 OpenMetadata import rules, `react-hooks/exhaustive-deps`, `i18next/no-literal-string`, `@typescript-eslint/no-non-null-assertion` | Repo-wide today: **0 errors, 10120 warnings** across 2228 files. The warnings *are* the backlog, made visible instead of hidden — the target is zero, reached rule by rule. `i18next/no-literal-string` was disabled with a `TODO: re-enable when the plugin supports ESLint 9`. That incompatibility no longer reproduces; it runs fine and reports a large backlog, so it is back on at `warn`. The repo convention is no user-facing string literals, so it should reach `error`. ## Repository-specific performance rules `openmetadata-ui/src/main/resources/ui/eslint-rules/openmetadata-performance.mjs` contains three reporting-only rules. Their test suite is run with `yarn test:eslint-rules`. All three were enabled at `error` only after a full `src/` scan reached zero findings. - `no-eager-page-imports` applies to `src/components/AppRouter/**` and rejects runtime static imports whose path contains `pages/`. Type-only imports remain valid. - `require-suspense-fallback` recognizes `lazy` and `React.lazy` only when imported from React. It accepts a component passed directly or subsequently to an approved helper imported from `components/AppRouter/withSuspenseFallback`, or a local variable-dependency path from the lazy binding to JSX rendered or passed beneath a real React `Suspense` boundary with an explicit `fallback` prop. An unrelated boundary elsewhere in the module does not satisfy the rule. - `no-unbounded-module-cache` checks module-level `Map` and `Set` bindings with cache-like names. A cache needs an explicit numeric or uppercase named size comparison whose guarded `if` branch or `while` body evicts from the same binding using `delete` or `clear`. The rules intentionally do not autofix because introducing a loading boundary, choosing an eviction policy, and deciding which route dependency should remain eager require runtime context. ## Import architecture and request warnings `openmetadata-ui/src/main/resources/ui/eslint-rules/openmetadata-imports.mjs` contains ten reporting-only rules. They are enabled at `warn`, do not autofix, and therefore do not fail CI while their measured backlog is reduced. | Rule | What it reports | Baseline findings / files | |---|---|---:| | `no-impure-pure-utils` | React/JSX or upward UI, state, page, hook, or REST dependencies in `*PureUtils` | 62 / 23 | | `no-lower-layer-page-imports` | page imports outside pages and the AppRouter owner | 291 / 271 | | `no-cross-page-imports` | one page feature statically importing another page feature | 43 / 32 | | `no-rest-ui-imports` | REST clients depending on components, pages, hooks, context, or stores | 55 / 37 | | `no-hook-ui-imports` | hooks depending on components or pages | 10 / 6 | | `no-circular-imports` | runtime imports/re-exports that participate in a cycle; type-only imports are ignored | 295 / 164 | | `no-internal-barrel-imports` | runtime imports resolving to an app-internal `index` barrel; type-only imports are allowed | 143 / 134 | | `no-lodash-default-import` | default or namespace imports from the Lodash package root | 1 / 1 | | `no-api-calls-in-iteration` | REST calls inside loops or dynamic iteration callbacks such as `map` | 28 / 21 | | `review-sequential-api-calls` | a second or later directly awaited REST call in one function, for dependency review | 207 / 105 | The last rule is intentionally phrased as a review: static analysis cannot prove whether the second request depends on the first. Keep legitimate sequencing; parallelize independent requests. Promote each deterministic rule to `error` only after its repo-wide baseline reaches zero. Re-evaluate the request-review rule separately before making it mandatory. **Rules deliberately still off**, and why: - `react/jsx-no-useless-fragment` — auto-fixes, so at any severity `eslint --fix` rewrites files and hard-fails the git-diff check. Land a one-time repo-wide autofix commit, then add it at `error`. - `sonarjs/file-header`, `arrow-function-convention`, `shorthand-property-grouping`, `elseif-without-else` and similar — stylistic, and they conflict with Prettier and the repo's existing conventions. Turning them on would add thousands of warnings nobody intends to fix, which devalues every other warning. - `sonarjs/no-reference-error`, `no-implicit-dependencies` — need resolver/global configuration this config does not supply; without it they are almost entirely false positives. **Promotion path:** clear a rule's backlog, re-measure, move it to `error`. The counts live in `eslint.config.mjs` next to each rule so the next person can see what it costs. > **Before adding any rule to the `warn` tier, check whether it auto-fixes.** `ui-checkstyle` runs > `eslint --fix` and fails on the resulting git diff, so an auto-fixing rule at `warn` would silently > rewrite files and hard-fail the gate. Every current `warn` rule reports `fixable: none` or > suggestions-only. `react-hooks/exhaustive-deps` declares `fixable: 'code'` but was verified > empirically not to rewrite a dependency array under `--fix` — which matters twice over, since > auto-adding an effect dependency changes runtime behaviour. ## SonarJS in ESLint — the fast half of Sonar `eslint-plugin-sonarjs` is the *same engine and the same `Sxxxx` rule ids* as the SonarCloud analysis that already runs on every UI PR. A finding in your editor is the finding Sonar will report. The high-backlog SonarJS rules are also enforced *blockingly* by SonarCloud, whose Clean-as-You-Code model scopes them to **new lines** — something ESLint fundamentally cannot express. So `cognitive-complexity` and `no-duplicate-string` warn locally and block on new code in the PR gate. `eslint-plugin-sonarjs` is pinned **exactly** (`4.2.0`). SonarCloud upgrades its analyzer server-side on its own schedule and that drift is silent. ## The Sonar lens in your editor — SonarQube for IDE This is the third place the same rules appear, and the only one that shows the *server's* profile verbatim rather than a local approximation. 1. Install **SonarQube for IDE** (formerly SonarLint) — available for VS Code, IntelliJ, and others. 2. Bind the workspace in **Connected Mode** to SonarCloud, organization `open-metadata`, project `open-metadata-ui`. 3. Connected Mode pulls the project's quality profile, so the editor flags exactly what the PR gate will flag — including the high-backlog rules ESLint holds back, marked against new code. Without Connected Mode the plugin uses its own defaults and will disagree with CI. Bind it, or rely on `make ui-checkstyle-changed`. ## SonarCloud configuration (admin — must be done once) Project `open-metadata-ui`, organization `open-metadata`, scanned by `.github/workflows/yarn-coverage.yml`. ### Quality profile Create a custom profile whose active rules mirror the set enabled in `eslint.config.mjs`, and set rule parameters explicitly on both sides — do not rely on two defaults agreeing (`cognitive-complexity` threshold 15 in both places). ### Quality gate — conditions on New Code ONLY Gate name: **OpenMetadata UI — Clean as You Code**. Set it as the project's default gate. | Condition (New Code) | Operator | Value | |---|---|---| | Coverage | is less than | **90.0%** | | Issues | is greater than | **0** | | Security Hotspots Reviewed | is less than | **100%** | | Duplicated Lines (%) | is greater than | **3.0%** | | *any condition on Overall Code* | — | **none** | **Never attach a condition to Overall Code.** It would fail on legacy debt from day one and break the entire "new code only" contract. Every condition above is evaluated solely against the lines a PR adds or modifies. > **90% coverage on new code is the strictest condition here** — above Sonar way's default of 80%, > and the UI currently has no coverage floor at all (`jest.config.js` sets `collectCoverageFrom` but > no `coverageThreshold`). Expect this to be the condition that fails most PRs at first: any new > component, hook or util needs tests landing in the same PR. That is the intent — new code is held > to a standard the backlog is not — but it is a real change in what "done" means for a UI PR, and > teams should hear it before the gate turns on rather than from a red check. > > Two mechanical consequences worth knowing: > - A PR that only **moves or reformats** code can still register those lines as new and uncovered. > - Coverage comes from `sonar.typescript.lcov.reportPaths` (`src/test/unit/coverage/lcov.info`), so > if the Jest run fails or the lcov is missing, new-code coverage reads as 0% and the gate fails. > Fix the test run, not the gate. ### New Code definition *Reference branch = `main`* for branches; *Previous version* (or 30 days) for `main` itself. Set in project settings, not on the gate. ### Branch protection Mark these three required on `main`: | Required check | Enforces | |---|---| | `ui-checkstyle` | lint (incl. SonarJS + jsx-a11y), prettier, licence, i18n, `tw-audit`, `tw-guard` | | `ui-coverage` | Jest run completed | | **`ui-sonar-gate`** | the Clean-as-You-Code quality gate, incl. 90% coverage on new code | Mark **`ui-sonar-gate`**, **not** SonarCloud's own check. The scan is gated behind `dorny/paths-filter` and the `safe to test` label, so a PR with no UI changes never produces that check and would block forever waiting on it. `ui-sonar-gate` always runs and passes when the scan was legitimately skipped. The gate result comes from the scanner itself: the PR scan passes `-Dsonar.qualitygate.wait=true` (timeout 600s), so SonarCloud decides and the scanner exits non-zero on failure. `ui-sonar-gate` turns that outcome into the check contributors see. This is the supported mechanism — do not reintroduce polling of `/api/qualitygates/project_status`, which races the asynchronous report processing and silently passes when it times out. ## Two behaviours to expect **Modified lines count as new code.** Editing a line in a messy legacy file pulls that line's issues into gate scope. This is the mechanism that retires debt gradually — you clean what you touch — but it reads as "the gate failed on code I didn't write". It isn't; the line is in your diff. **New-code attribution needs confirming once.** The PR scan passes `-Dsonar.scm.disabled=true` (the push scan does not). New code for a PR comes from the `sonar.pullrequest.*` parameters, so this is probably fine — but on the first gated PR, check that Sonar's **New Code** tab shows only the diff and not whole files. If it shows whole files, drop that flag from the PR scan step. ## Tracking the backlog The gate is blind to old code by design, so it will never tell you whether debt is shrinking. Track `sqale_index`, `code_smells` and `duplicated_lines_density` on *overall* code monthly: ``` GET https://sonarcloud.io/api/measures/search_history ?component=open-metadata-ui&metrics=cognitive_complexity,duplicated_lines_density,code_smells,sqale_index,ncloc ``` Expectation: flat or falling on a growing `ncloc`. Two consecutive months climbing is the signal to schedule targeted cleanup — Clean as You Code only retires debt where people happen to edit. ---