Csv Relation Types Plan Copy Section

CSV Import/Export Enhancement for Glossary Term Relations

Problem Statement Copy Chapter

Currently, the glossary CSV import/export only captures related term FQNs without the relation type:
- Export: Only exports FQNs like Glossary.Term1;Glossary.Term2
- Import: Hardcodes all relations to "relatedTo"

This causes data loss when:
1. A term has synonym, broader, narrower, or custom relation types
2. CSV is exported and re-imported - all relation types become "relatedTo"

Proposed Solution Copy Chapter

New CSV Format Copy Section

Format: relationType:termFQN pairs separated by semicolons

Examples:

New format with relation types

relatedTerms
synonym:Finance.Revenue;broader:Finance.Income;narrower:Finance.Net Revenue
Backward compatible - no prefix defaults to "relatedTo"

relatedTerms
Finance.Revenue;Finance.Income
Mixed format (new and legacy)

relatedTerms
synonym:Finance.Revenue;Finance.Income;broader:Finance.Gross Income

Parsing Rules Copy Section

1. If a value contains : and the part before : is a valid relation type → use that relation type
2. If no : or the prefix is not a valid relation type → default to "relatedTo"
3. Valid relation types are determined by checking glossaryTermRelationSettings or using defaults

Default Relation Types Copy Section

| Relation Type | Description |
|---------------|-------------|
| relatedTo | Generic related term (default) |
| synonym | Equivalent term |
| broader | More general term |
| narrower | More specific term |
| antonym | Opposite meaning |
| partOf | Component of |
| hasPart | Contains |

Implementation Plan Copy Chapter

Phase 1: Backend Changes Copy Section

#### 1.1 CsvUtil.java - Export Enhancement

File: openmetadata-service/src/main/java/org/openmetadata/csv/CsvUtil.java

Current (line 253-263):

public static List<String> addTermRelations(
    List<String> csvRecord, List<TermRelation> termRelations) {
  csvRecord.add(
      nullOrEmpty(termRelations)
          ? null
          : termRelations.stream()
              .map(tr -> tr.getTerm().getFullyQualifiedName())
              .sorted()
              .collect(Collectors.joining(FIELD_SEPARATOR)));
  return csvRecord;
}

New:

public static List<String> addTermRelations(
    List<String> csvRecord, List<TermRelation> termRelations) {
  csvRecord.add(
      nullOrEmpty(termRelations)
          ? null
          : termRelations.stream()
              .map(tr -> {
                String relationType = tr.getRelationType();
                String fqn = tr.getTerm().getFullyQualifiedName();
                // Only include relation type prefix if not the default "relatedTo"
                if (relationType != null && !relationType.equals("relatedTo")) {
                  return relationType + ":" + fqn;
                }
                return fqn;
              })
              .sorted()
              .collect(Collectors.joining(FIELD_SEPARATOR)));
  return csvRecord;
}

#### 1.2 GlossaryRepository.java - Import Enhancement

File: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/GlossaryRepository.java

Current (line 315-327):

private List<TermRelation> getTermRelationsFromCsv(
    CSVPrinter printer, CSVRecord csvRecord, int fieldNumber) throws IOException {
  List<EntityReference> entityRefs =
      getEntityReferences(printer, csvRecord, fieldNumber, GLOSSARY_TERM);
  if (entityRefs == null) {
    return null;
  }
  List<TermRelation> termRelations = new ArrayList<>();
  for (EntityReference ref : entityRefs) {
    termRelations.add(new TermRelation().withTerm(ref).withRelationType("relatedTo"));
  }
  return termRelations;
}

New:

/ Detailed source-code truncated for AI context efficiency. /

#### 1.3 Documentation Update

File: openmetadata-service/src/main/resources/json/data/glossary/glossaryCsvDocumentation.json

Update the relatedTerms field documentation:

{
  "name": "relatedTerms",
  "required": false,
  "description": "Related glossary terms with optional relation types. Format: 'relationType:FQN' or just 'FQN'. Multiple values separated by ';'. Valid relation types: relatedTo (default), synonym, broader, narrower, antonym, partOf, hasPart. Example: 'synonym:Glossary.Term1;broader:Glossary.Term2;Glossary.Term3'",
  "examples": [
    "Glossary.Term1;Glossary.Term2",
    "synonym:Glossary.Term1;broader:Glossary.Term2",
    "synonym:Glossary.Revenue;Glossary.Income;narrower:Glossary.Net Revenue"
  ]
}

Phase 2: Testing Copy Section

#### 2.1 Unit Tests

File: openmetadata-service/src/test/java/org/openmetadata/csv/CsvUtilTest.java

@Test
void testAddTermRelationsWithRelationType() {
  // Test that relation types are included in export
}
@Test
void testAddTermRelationsDefaultRelationType() {
  // Test that "relatedTo" terms don't include prefix
}

#### 2.2 Integration Tests

File: openmetadata-service/src/test/java/org/openmetadata/service/resources/glossary/GlossaryTermResourceTest.java

@Test
void testGlossaryTermCsvImportWithRelationTypes() {
  // Test importing CSV with relation type prefixes
}
@Test
void testGlossaryTermCsvExportWithRelationTypes() {
  // Test exporting terms with various relation types
}
@Test
void testGlossaryTermCsvBackwardCompatibility() {
  // Test importing old format CSV (no relation types)
}
@Test
void testGlossaryTermCsvRoundTripWithRelationTypes() {
  // Test that export -> import preserves relation types
}

Phase 3: Edge Cases Copy Section

1. FQN contains colon: Handle cases like Database:Schema.Term by validating the prefix against known relation types
2. Invalid relation type: If prefix is not a valid relation type, treat entire string as FQN with default relatedTo
3. Empty relation type: ":Glossary.Term" should default to relatedTo
4. Custom relation types: Check against glossaryTermRelationSettings for user-defined relation types

Backward Compatibility Copy Section

| CSV Format | Import Behavior |
|------------|----------------|
| Glossary.Term1;Glossary.Term2 | All relations → relatedTo |
| synonym:Glossary.Term1;Glossary.Term2 | First → synonym, Second → relatedTo |
| synonym:Glossary.Term1;broader:Glossary.Term2 | Preserves both relation types |

Files to Modify Copy Section

| File | Change |
|------|--------|
| CsvUtil.java | Update addTermRelations() to include relation type prefix |
| GlossaryRepository.java | Update getTermRelationsFromCsv() to parse relation types |
| glossaryCsvDocumentation.json | Update field documentation and examples |
| GlossaryTermResourceTest.java | Add tests for new format |
| CsvUtilTest.java | Add unit tests for parsing |

Migration Notes Copy Section

- No database migration needed: The database already stores relation types correctly
- Existing CSVs: Will continue to work (all imported as relatedTo)
- New exports: Will include relation type prefixes for non-default relations

Summary Copy Chapter

This enhancement:
1. ✅ Preserves relation types during CSV export/import
2. ✅ Maintains backward compatibility with existing CSVs
3. ✅ Defaults to relatedTo when no relation type specified
4. ✅ Follows existing OpenMetadata CSV patterns (type:value)
5. ✅ Supports custom relation types via settings

---

Design Patterns Copy Section

Design Patterns in this Codebase — a guideline for coding agents

The Gang-of-Four patterns that OpenMetadata actually uses as established conventions, each with
the canonical place to copy the idiom from. This is not a pattern tutorial and not a catalogue of all
23 — it lists only the patterns that recur idiomatically here, so that when you add code you extend
the pattern the codebase already relies on instead of inventing a parallel one.

How to use this Copy Chapter

- Follow the local idiom. When a situation matches one below, mirror the shape of the cited
canonical example (same interface, same registration, same lifecycle hooks). Consistency across
~60 entity repositories and ~98 connectors is worth more than a cleverer one-off.
- Don't over-apply. A pattern earns its place only when it removes real duplication or decouples a
real seam. Prefer the simplest thing that works; do not add a factory/registry/strategy layer for a
single implementation (see CLAUDE.md on avoiding needless abstraction).
- Match the language. Java leans on interfaces + factories + registries + Lombok; Python on
classmethod constructors, singledispatch, generators, and decorator registries; the UI on
*ClassBase singletons and React context. Use the host language's mechanism, not a Java-ism in
Python.
- Paths below are verified and kept fresh by the harness dead-reference check (make harness-check).

---

Java backend (openmetadata-service, common, openmetadata-sdk) openmetadata-service, common, openmetadata-sdk)')" title="Copy chapter prompt for LLMs"> Copy Chapter

Creational Copy Section

Factory Method / Abstract Factory — construct one of several implementations behind a common
type, chosen by config/provider; callers never new the concrete class.
- Use it when you add a provider/engine variant (a secrets backend, a search engine, a connection
type): register the impl and let the factory pick it.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManagerFactory.java
(switch on provider), openmetadata-service/src/main/java/org/openmetadata/service/secrets/converter/ClassConverterFactory.java
(map of 40+ converters), openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepositoryFactory.java
(JDK ServiceLoader SPI). Abstract Factory (parallel ES vs OS product families) lives under
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ and
openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/.

Builder — assemble a complex object by named fields instead of a telescoping constructor.
- Use it when building a schema POJO/DTO (Lombok @Builder) or an entity through the SDK (the
fluent *Builder).
- Here: Lombok @Builder on data objects (112 sites); the hand-written fluent builders in
openmetadata-sdk/src/main/java/org/openmetadata/sdk/fluent/builders/ (32 classes);
openmetadata-service/src/main/java/org/openmetadata/service/util/OpenMetadataConnectionBuilder.java.

Singleton / Registry — one shared instance, plus a registry mapping a key (entity name, type) to
its handler, populated at startup and resolved at runtime. This is how the backend dispatches
polymorphically without giant switch statements.
- Use it when you add an entity type or handler: self-register into the existing registry rather
than hard-coding a lookup.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/Entity.java (registerEntity /
getEntityRepository), openmetadata-service/src/main/java/org/openmetadata/service/TypeRegistry.java,
openmetadata-service/src/main/java/org/openmetadata/service/resources/CollectionRegistry.java.

Structural Copy Section

Adapter — wrap an external/vendor API in the repo's own interface so the rest of the code stays
vendor-agnostic.
- Use it when integrating a second implementation of an external dependency: implement the internal
interface, never leak the vendor type upward.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/search/SearchClient.java
(internal API) with openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java
and openmetadata-service/src/main/java/org/openmetadata/service/search/opensearch/OpenSearchClient.java
adapting the relocated ES/OS SDKs.

Facade — one entry point over a large subsystem.
- Use it when you need persistence: go through the DAO facade, not ad-hoc JDBI handles.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java
aggregates 129 sub-DAOs behind one interface.

Proxy (caching) — a stand-in that transparently adds caching in front of the real object.
- Use it when reading entities/subjects on hot paths: read through the existing bounded caches,
don't hit the DB directly (see CLAUDE.md: all caches must be bounded).
- Here: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java
(the CACHE_WITH_NAME/CACHE_WITH_ID Guava LoadingCache),
openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/SubjectCache.java.

Composite — model part-whole trees so a leaf and a container are handled uniformly.
- Use it when working with hierarchies (teams, glossary terms/domains, FQN containment): reuse the
existing recursive helpers rather than re-walking the tree by hand.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/SubjectContext.java
(team tree for RBAC), openmetadata-service/src/main/java/org/openmetadata/service/util/FullyQualifiedName.java
(service→database→schema→table).

Decorator — wrap an object to add behavior while keeping the same interface. *Narrow here — one
genuine case.*
- Here: openmetadata-service/src/main/java/org/openmetadata/service/socket/HeaderRequestWrapper.java
(a servlet-request wrapper). Note: the formatter/decorators/ package is named "decorator" but is
structurally Strategy (one impl per channel, nothing wraps another) — do not treat it as a
Decorator.

Behavioral Copy Section

Template Method — a base class fixes the algorithm skeleton and delegates the variable steps to
abstract hooks the subclass fills. The backbone of the backend.
- Use it when adding an entity repository or app: extend the base and implement the hooks
(setFields, prepare, storeEntity, …); never reimplement the CRUD lifecycle.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java
(~60 repositories), openmetadata-service/src/main/java/org/openmetadata/service/apps/AbstractNativeApplication.java.

Strategy — interchangeable algorithms behind a common interface, selected at runtime.
- Use it when there is a family of "same operation, different implementation" (auth, secrets
backend, per-type conversion): add an impl and register it.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/security/Authorizer.java,
openmetadata-service/src/main/java/org/openmetadata/service/secrets/SecretsManager.java,
openmetadata-service/src/main/java/org/openmetadata/service/secrets/converter/ClassConverter.java.

Observer / Publish-Subscribe — emit events; decoupled subscribers react.
- Use it when something should happen on entity change or per request: add a handler/subscription,
don't wire the caller directly to the reaction.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/events/EventHandler.java,
the change-event publisher family (openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/changeEvent/AlertPublisher.java),
openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java.

Chain of Responsibility — an ordered series of handlers, each doing its part and passing control on.
- Use it when adding a request filter or a migration phase: insert into the existing ordered chain.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/security/DelegatingContainerRequestFilter.java
(JAX-RS filter chain), openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationWorkflow.java
(Flyway → native → extension).

Command — encapsulate an action as an object that can be scheduled/queued/run later.
- Use it when adding a background job or a migration step: implement the job/step interface.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/apps/NativeApplication.java
(Quartz jobs), openmetadata-service/src/main/java/org/openmetadata/service/migration/api/MigrationProcess.java.

Visitor (parse-tree) — separate an operation from the object structure it traverses; here, ANTLR
parse trees.
- Use it when parsing/transforming FQNs, entity links, or JDBC URIs: use the ANTLR listeners, don't
hand-roll string splitting.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/util/FullyQualifiedName.java
(SplitListener over the tree); grammars in openmetadata-spec/src/main/antlr4/.

Iterator (cursor) — traverse a large/remote collection page-by-page without exposing storage.
- Use it when processing large entity sets (reindex, insights) or paginating a list API: use
cursor-based sources / ResultList, never load everything into memory.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/workflows/interfaces/Source.java
and EntityRepository keyset pagination.

State — allowed transitions depend on an explicit status; illegal transitions are rejected.
Narrow here.
- Here: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseResolutionStatusRepository.java
(incident lifecycle). Broader workflow state is delegated to the Flowable BPMN engine, not
hand-rolled — don't build a new state machine when a governance workflow fits.

---

Python ingestion framework (ingestion/src/metadata) ingestion/src/metadata)')" title="Copy chapter prompt for LLMs"> Copy Chapter

Factory Method — the create() contract. Every Step/Source is built through a create()
classmethod that validates the service connection and raises InvalidSourceException on mismatch;
never construct a connector directly. Mandatory for all 135+ connector steps.
- Here: ingestion/src/metadata/ingestion/api/step.py (Step.create),
ingestion/src/metadata/ingestion/source/database/postgres/metadata.py.

Abstract Factory / Registry — the ServiceSpec system. A per-connector service_spec.py manifest
declares the classpaths of its source/profiler/sampler; the framework imports them dynamically, so the
workflow is decoupled from every connector.
- Use it when adding a connector: ship a service_spec.py, don't wire the class into the framework.
- Here: ingestion/src/metadata/utils/service_spec/service_spec.py (BaseSpec,
import_source_class); 98 service_spec.py manifests.

Factory + decorator Registry. Decorator-based registries map a key (enum value, SQL dialect, class
name) to a handler resolved at runtime.
- Use it when adding a parser/metric/adaptor variant: register it via the shared primitive rather
than editing a dispatch if/elif.
- Here: ingestion/src/metadata/utils/dispatch.py (enum_register / class_register),
ingestion/src/metadata/profiler/factory.py.

Adapter — entity adapters. Adapt heterogeneous entity shapes (Table/Container/Topic) to one
uniform interface, replacing scattered isinstance checks.
- Use it when adding classification/sampling support for a new entity type: add an
@register_adapter adapter.
- Here: ingestion/src/metadata/sampler/entity_adapters.py.

Template Method. The base runs the fixed lifecycle (run / execute) and calls abstract hooks
(_run/_iter, execute_internal, the topology yield_/get_ methods).
- Use it when adding a step or workflow: fill the hooks, don't reimplement the lifecycle.
- Here: ingestion/src/metadata/ingestion/api/step.py,
ingestion/src/metadata/ingestion/source/database/database_service.py,
ingestion/src/metadata/workflow/base.py.

Strategy via singledispatch. Dispatch on the runtime type of a record instead of an isinstance
ladder (9 files use singledispatchmethod, 22 use singledispatch).
- Here: ingestion/src/metadata/ingestion/api/topology_runner.py.

Iterator — generators. A Source is a generator; records stream lazily and the workflow pulls them
with for record in source.run(). Produce records with yield, don't build a list.
- Here: ingestion/src/metadata/ingestion/api/step.py (IterStep),
ingestion/src/metadata/ingestion/api/topology_runner.py.

Chain / Pipeline — Source→Processor→Stage→Sink. Each record flows through an ordered tuple of
steps that transform/forward/drop it; a workflow composes its own chain in set_steps().
- Here: ingestion/src/metadata/workflow/ingestion.py,
ingestion/src/metadata/ingestion/api/steps.py.

Observer — Status handler. Log warnings emitted anywhere during a step's run() are observed by a
handler and recorded into that step's Status, decoupling emitters from the aggregator.
- Here: ingestion/src/metadata/ingestion/api/status.py,
ingestion/src/metadata/ingestion/api/step.py.

---

TypeScript UI (openmetadata-ui/src/main/resources/ui/src) openmetadata-ui/src/main/resources/ui/src)')" title="Copy chapter prompt for LLMs"> Copy Chapter

Singleton + Factory-Method — the *ClassBase seam. A single instance is exported as the app-wide
singleton, and the class is also exported so the enterprise build can subclass and override its
factory methods (which return components, field configs, and widgets). This is the deliberate UI
plugin/override point — 40 such modules.
- Use it when adding a UI extension point the enterprise build may override: follow the *ClassBase
shape.
- Here: openmetadata-ui/src/main/resources/ui/src/utils/EntityRightPanelClassBase.ts (and siblings
like TableClassBase, SearchClassBase).

Factory function. Build the right component subtree from a discriminant enum.
- Here: openmetadata-ui/src/main/resources/ui/src/components/Auth/AuthProviders/AuthProvider.tsx
(switch on auth provider).

React Context/Provider (openmetadata-ui/src/main/resources/ui/src/context) is the idiomatic

composition/state-sharing seam (24 contexts). It resembles GoF Mediator only loosely — treat it as

the React idiom, not as a GoF pattern to reach for by name.

---

Cautions Copy Chapter

- Naming ≠ structure. formatter/decorators/ is Strategy, not Decorator. Verify a candidate's
structure, not its class name, before matching it to a pattern.
- Don't force GoF onto React. Hooks, context, and composition cover most of what Strategy/Observer
would in an OO codebase; a *ClassBase singleton or a factory function is usually the only GoF shape
worth naming on the UI.
- A pattern is a means, not a goal. If a plain function or a switch is clearer and there is only
one implementation, use it — and revisit the pattern when a second implementation actually arrives.

---

Golden Principles Copy Section

Golden Principles (DRAFT — for human ratification)

Candidates drawn from a one-time repository audit's strongest signals: the three most-consistent
inter-module/import dependency rules, and the conventions measured at high adherence with a clean
number. Nothing here without a measured number (the reproducing command is in each row).

Selection rule. The candidate pool was cut to the intersection of high adherence AND high
cost-of-violation, capped at ten ("more than ten means preferences"). That drops four low-cost,
auto-fixed formatting/hygiene lints — spotless (100%), ruff format (100%), no-wildcard
imports (98.8%), no-console (99.96%) — which are lints to gate, not principles (see the promotion
list). 8 remain. Ordered by cost-of-violation (highest first).

| # | Principle | Failure mode it prevents | Detection command | Adherence (measured) |
|---|---|---|---|---|
| 1 | Modules form an acyclic, downward-only dependency graph | Circular build/reasoning; a change in a "leaf" forcing a rebuild of the "root"; loss of module boundaries | parse each module POM's org.open-metadata deps for a back-edge; or maven-enforcer banCircularDependencies | 0 cycles / 12 modules = 100% |
| 2 | Every connector satisfies the ServiceSpec plugin contract ({__init__,service_spec,metadata,connection}.py + a Source subclass of a Common*/DatabaseServiceSource base + create() raising InvalidSourceException + a top-level ServiceSpec) | A connector that silently fails to load/register at runtime | per connector dir: find … -name service_spec.py; assert ServiceSpec importable + the four files present | ~97 connectors, all registered = 100%; 94/95 metadata.py carry create()+InvalidSourceException = 98.9% |
| 3 | Generated code is a pure sink; source imports it only as types (no generated file imports app/domain code; no runtime import metadata.generated…) | Hand-edits / runtime coupling that the next make generate overwrites → drift, CI reverts, broken builds | grep -rlE "from '(\.\./)+(components\|pages\|rest\|utils\|hooks\|context)/" openmetadata-ui/.../src/generated → 0; grep -rn "^import metadata.generated" ingestion/src → 0 | 0 app-imports (100% sink); source→generated 1,736/1,738 = 99.9% type-only · now hook-enforced (edit-block) |
| 4 | No new type errors (basedpyright ratchet — the baseline only shrinks) | Type regressions shipping as runtime bugs | make static-checks (nox → basedpyright --baselinemode=discard) | ratchet over 11,927 baselined findings; 0 new required; CI-gated |
| 5 | No bare except: (catch a specific type, or at minimum name the exception) | Swallowing KeyboardInterrupt/SystemExit; hiding every error | grep -rnE 'except\s*:' ingestion/src/metadata; or ruff E722 | 2,074/2,075 = 99.95% (1 file) |
| 6 | Functional React components only (no class components) | Hooks-incompatible components; two divergent component models | grep -rlE 'extends (React\.)?(Component\|PureComponent)\b' openmetadata-ui/.../src | 0 violations = 100% |
| 7 | Apache-2.0 license header on every new source file | OSS licensing / compliance gaps | license-check-and-add check (UI); header grep on newly-added files | 99.75% (12/4,751 miss; 9 are generated .js) · now hook-enforced for new UI files |
| 8 | Parameterized logging, never string concatenation (LOG.x("… {}", var)) | Log-injection; needless string building; unstructured logs | grep -rnE '(log\|LOG\|logger)\.(info\|warn\|error\|debug\|trace)\([^;]"\s\+' …/src/main/java | 0/5,989 = 100% — the lowest-cost of the eight; flagged for the ratifier |

Conflicts to resolve before ratifying (surfaced, not decided) Copy Chapter

Explicitly NOT principles (measured, but disqualified) Copy Chapter

---

Impersonation Design Copy Section

Bot Impersonation Feature - Design Document

1. Overview Copy Chapter

1.1 Purpose Copy Section

1.2 Background Copy Section

1.3 Goals Copy Section

1.4 Non-Goals Copy Section

2. Design Principles Copy Chapter

1. Security First: Only authorized bots can impersonate, with strict validation
2. Transparency: All impersonated actions are clearly visible in audit logs and UI
3. Permission Inheritance: Use impersonated user's permissions, not bot's
4. Backward Compatibility: Existing entities without impersonation continue to work
5. Auditability: Complete trail of who did what via which bot

3. Architecture Copy Chapter

3.1 Token Exchange Flow (On-Demand Impersonation) Copy Section

Since JWT tokens are pre-generated and bots cannot dynamically create tokens with different impersonatedUser claims, we need a token exchange API that allows bots to obtain short-lived impersonation tokens on-demand.

/ Detailed source-code truncated for AI context efficiency. /

3.2 Component Details Copy Section

#### 3.2.1 JWT Token Structure

Standard Token (No Impersonation)

{
  "sub": "alice",
  "email": "[email protected]",
  "isBot": false,
  "exp": 1234567890
}

Impersonation Token

{
  "sub": "ingestion-bot",
  "email": "[email protected]",
  "isBot": true,
  "impersonatedUser": "alice",
  "exp": 1234567890
}

#### 3.2.2 Security Context

CatalogSecurityContext Extension

public record CatalogSecurityContext(
    Principal principal,           // The bot
    String scheme,
    String authenticationScheme,
    Set<String> userRoles,
    boolean isBot,
    String impersonatedUser        // NEW: The actual user
) implements SecurityContext

SubjectContext Enhancement

public record SubjectContext(
    User user,                     // The impersonated user (or actual user)
    String impersonatedBy          // NEW: The bot name (if impersonating)
)

4. Authorization Model Copy Chapter

4.1 Three-Level Security Check Copy Section

#### Level 1: Bot Capability Flag

// User entity schema
{
  "name": "ingestion-bot",
  "isBot": true,
  "allowImpersonation": true  // NEW: Must be explicitly enabled
}

#### Level 2: Impersonation Permission

// New MetadataOperation enum value
public enum MetadataOperation {
    // ... existing operations
    IMPERSONATE  // NEW: Permission to impersonate users
}

#### Level 3: Policy-Based Control

{
  "name": "IngestionBotImpersonationPolicy",
  "description": "Allow ingestion bot to impersonate users in Engineering domain",
  "rules": [
    {
      "name": "ImpersonateEngineeringUsers",
      "resources": ["user"],
      "operations": ["Impersonate"],
      "effect": "allow",
      "condition": "hasDomain('Engineering')"
    }
  ]
}

4.2 Authorization Flow Copy Section

// Pseudo-code for validation in JwtFilter
void validateImpersonation(String botName, String targetUser) {
    // 1. Bot must have isBot=true (already validated)
    // 2. Check bot has allowImpersonation flag
    User bot = Entity.getEntityByName(USER, botName, "allowImpersonation", NON_DELETED);
    if (!Boolean.TRUE.equals(bot.getAllowImpersonation())) {
        throw new AuthorizationException("Bot not authorized to impersonate");
    }
    // 3. Check policy grants IMPERSONATE permission
    SubjectContext botContext = SubjectContext.getSubjectContext(botName);
    User targetUserEntity = Entity.getEntityByName(USER, targetUser, "", NON_DELETED);
    ResourceContext resourceContext = new ResourceContext(USER, targetUserEntity.getId(), null);
    OperationContext operationContext = new OperationContext(USER, MetadataOperation.IMPERSONATE);
    // This throws AuthorizationException if not permitted
    PolicyEvaluator.hasPermission(botContext, resourceContext, operationContext);
    // 4. Validation passed - impersonation allowed
}

4.3 Permission Evaluation Copy Section

When a bot impersonates a user, authorization checks use the impersonated user's permissions, not the bot's:

// In DefaultAuthorizer
public static SubjectContext getSubjectContext(SecurityContext securityContext) {
    CatalogSecurityContext catalogContext = (CatalogSecurityContext) securityContext;
    // Use impersonated user if present, otherwise use principal
    String userName = catalogContext.impersonatedUser() != null
        ? catalogContext.impersonatedUser()
        : catalogContext.getUserPrincipal().getName();
    User user = Entity.getEntityByName(Entity.USER, userName, USER_FIELDS, NON_DELETED);
    // Track who is impersonating (if applicable)
    String impersonatedBy = catalogContext.impersonatedUser() != null
        ? catalogContext.getUserPrincipal().getName()
        : null;
    return new SubjectContext(user, impersonatedBy);
}

Example: If ingestion-bot impersonates alice:
- CRUD operations are authorized using alice's roles/teams/policies
- If alice lacks permission to update a table, the operation fails
- This prevents privilege escalation

4.4 Bot Impersonation Grants and RBAC Scoping (v1.1 — June 2026) Copy Section

This section supersedes the parts of 4.1/4.2 that the initial implementation simplified, and addresses two issues:

- OpenMetadata #28043: custom bots cannot enable impersonation. The allowImpersonation flag exists only on the user entity, is not part of createUser.json/createBot.json, and PATCHing it on the bot user fails (500 in 1.12.x). Only application bots (CreateApp.allowBotImpersonation) could ever receive the flag.
- Collate #3581: bots with impersonation enabled can impersonate admin users. Admins cannot be restricted via RBAC, so this is privilege escalation. Blocking it outright breaks workflows that legitimately need it (e.g. AskCollate), so the control must be configurable.

#### 4.4.1 Design Decision: Single Flag + RBAC Scope

The split of responsibility is:

| Question | Mechanism |
|----------|-----------|
| Can this bot impersonate at all? | allowImpersonation boolean on the bot user — an admin-granted capability, set at bot creation |
| Who can this bot impersonate? | Standard policy evaluation of the Impersonate operation with the target user as the resource |

A second boolean (allowAdminImpersonation) was considered and rejected: target scoping belongs in policies/rules, where it is already expressive (conditions, deny rules, teams, domains) and admin-manageable without code changes.

#### 4.4.1.1 Why a dedicated flag and not RBAC alone

This question recurs ("Impersonate is already an operation — drop the flag, grant it through a role like everything else"). It has been evaluated and rejected. The flag and RBAC answer two different questions, and collapsing them opens a real privilege-escalation gap:

- The flag is the enablement-authority gate: may this bot impersonate at all? — admin-only, set at bot creation, enforced in DefaultAuthorizer independently of RBAC.
- RBAC is the target-scope gate: whom may it impersonate? — policies, conditions, deny rules.

If enablement were pure RBAC, "who can enable impersonation" would collapse into "who can grant the Impersonate operation" — i.e. anyone with EditRoles on a bot user or EditPolicy+Create on policies. That permission set is routinely delegated to non-admins (orgs delegate RBAC management). Impersonation — which lets a bot act as any user, including bypassing controls admins can't otherwise be restricted by — would then become grantable by non-admins. The flag keeps enablement admin-only regardless of how RBAC editing is delegated.

The flag is also the robust control, not merely an extra layer. Under RBAC-only, the Impersonate grant can arrive through many paths — a direct role, a team's defaultRoles the bot inherits, an inherited policy, or a broad ["All"]-operations policy (subsumption). Guaranteeing "admin-only" would mean guarding every one of those write paths. The flag is instead a single chokepoint: one admin-only, creation-time write, checked before any policy evaluation. It is belt-and-suspenders — even if Impersonate leaks into a bot's effective policy, no flag means no impersonation.

Two facts bound the surface and explain what the flag does and does not cover:

- JwtFilter rejects the X-Impersonate-User header for any non-bot principal ("Only bot users can impersonate other users"). So a regular user self-granting Impersonate achieves nothing — they are not a bot. By default they also cannot edit RBAC. This case is already closed without the flag.
- The case the flag uniquely closes is a non-admin RBAC delegate enabling impersonation on a bot. RBAC-only would permit it; the flag does not.

Conclusion: the flag is not redundant with RBAC — it is the admin-only, single-chokepoint enablement gate that RBAC delegation cannot widen. Do not remove it without replacing this property. The only redundancy ever identified was cosmetic: BotImpersonationRole is auto-attached when the flag is set, so the two travel together for UX. That convenience does not make the flag itself redundant.

#### 4.4.2 Granting the Capability (createBot.json)

createBot.json gains an optional allowImpersonation boolean with tri-state semantics (no schema default, so an absent field is null):

- null (absent) — keep the bot user's current value. Critical for PUT-based upserts (ingestion re-applies bots via PUT /v1/bots); an absent field must not silently revoke the grant.
- true — grant. Admin-only, and only when the bot is being created. Enabling impersonation on an existing bot is rejected with 400: an existing bot's token is already distributed, and flipping the flag would silently upgrade every holder of that token. Granting at creation forces a new bot + new token + deliberate admin action.
- false — revoke. Admin-only, allowed at any time (privilege reduction).

When the grant is applied, BotResource propagates the flag to the bot user entity (single source of truth — the authorizer reads it from the user) and attaches the seeded BotImpersonationRole to the bot user so the capability works out of the box. Revoking removes the flag and detaches that role.

User.allowImpersonation becomes effectively read-only on the user APIs:

- PATCH /v1/users changing it → 400 user attribute allowImpersonation can't be modified (replaces the unhandled 500 from #28043).
- PUT /v1/users always carries over the stored value (fixes a latent bug where any PUT on a bot user nulled the flag, because CreateUser has no such field).

#### 4.4.3 Scoping Who Can Be Impersonated (policies)

checkImpersonationAuthorization in DefaultAuthorizer evaluates, on every impersonated request:

1. The impersonating principal is a bot with allowImpersonation=true (capability gate, unchanged).
2. PolicyEvaluator.hasPermission(botSubjectContext, targetUserResourceContext, OperationContext(user, IMPERSONATE)) — full policy evaluation, with the target user as the resource. This replaces the previous flat scan of role policies for the Impersonate operation, which ignored rule effect (a deny rule counted as allow) and could not discriminate targets.

Because this is the standard evaluation path, deny-overrides-allow, SpEL conditions, and compiled-rule caching all apply. Two new condition functions make target discrimination possible:

| Function | True when |
|----------|-----------|
| isAdminUser() | the resource (target user) is an admin |
| isBotUser() | the resource (target user) is a bot |

Existing functions compose for finer scoping: matchTeam() (team-scoped impersonation), hasDomain(), matchAnyTag().

The Impersonate operation is registered as a user-resource operation, so the policy editor offers it on the user resource and the functions appear in the condition dropdown (/v1/policies/functions).

Impersonate is explicit-grant-only: CompiledRule.matchOperation excludes it from ALL/EditAll/ViewAll subsumption. A broad god-mode policy (operations: ["All"]) therefore neither grants nor appears to grant impersonation; only a rule naming Impersonate does. With the flag in place this is defense-in-depth (the flag already gates enablement), but it keeps permission listings honest and prevents impersonation from silently riding along with broad policies.

#### 4.4.4 Seed Policies and Defaults

- BotImpersonationPolicy (new, the default): allow Impersonate on user, no deny rules. Backward-compatible by default: a freshly granted bot can impersonate any user, including admins, matching the pre-existing ApplicationBotImpersonationPolicy (All/allow) behavior. This is deliberate — silently denying admin impersonation would break workflows (e.g. AskCollate) that rely on it.
- BotImpersonationRole (new): DefaultBotPolicy + BotImpersonationPolicy. Auto-attached at grant time.
- BotNonAdminImpersonationPolicy (new, opt-in restriction): allow Impersonate on user + deny isAdminUser(). Admins who want to prevent a bot from impersonating admins attach this instead of the permissive default.
- BotNonAdminImpersonationRole (new): DefaultBotPolicy + BotNonAdminImpersonationPolicy. The directly-assignable form of the opt-in restriction — swap a bot's BotImpersonationRole for this to deny admin targets.
- ApplicationBotImpersonationPolicy (existing, All/allow) is unchanged: application bots created with allowBotImpersonation (e.g. AskCollate) keep their current behavior, including admin impersonation.

Restriction is therefore opt-in, not default — the "configurable policy control" Collate #3581 asks for, delivered without breaking any bot that currently impersonates admins. The isAdminUser() / isBotUser() condition functions and deny-overrides-allow semantics let admins author any narrower scope (per-team, per-domain, deny-bot, etc.).

No migration needed. All four impersonation seeds (BotImpersonationPolicy, BotNonAdminImpersonationPolicy, and their roles) are new in this change — no released version has them. Seed loading is insert-if-missing, so a fresh install and any upgrade from a prior release both get the current (permissive) JSON seeded directly. There is no prior on-disk state to convert, so no Flyway migration or startup reconciliation is required.

#### 4.4.5 Threat-Model Delta

| Threat | Before | After |
|--------|--------|-------|
| Bot impersonates admin | Allowed for any impersonation bot | Allowed by default (backward-compatible); deniable opt-in via BotNonAdminImpersonationPolicy (isAdminUser() deny) |
| Bot impersonates another bot (token laundering) | Allowed | Allowed by default; deniable via an isBotUser() deny rule in a custom policy |
| Flag flipped on live bot via PATCH | 500, undefined behavior (works on newer builds) | 400, read-only; grant is creation-time via createBot |
| Flag silently wiped by PUT on bot user | Yes (latent bug) | PUT carries over stored value |
| deny rule with Impersonate op | Counted as allow | Honored (standard evaluator) |
| Non-admin RBAC delegate enables impersonation on a bot | n/a | Blocked — enablement is the admin-only flag, independent of RBAC editing (see 4.4.1.1) |
| Broad ["All"] policy grants impersonation by subsumption | Would grant | Impersonate excluded from ALL/EditAll/ViewAll subsumption; flag still required |

5. Data Model Changes Copy Chapter

5.1 JSON Schema Updates Copy Section

#### 5.1.1 Base Type Definition

// openmetadata-spec/src/main/resources/json/schema/type/basic.json
{
  "definitions": {
    "impersonatedBy": {
      "description": "Bot user that performed the action on behalf of the actual user.",
      "type": "string"
    }
  }
}

#### 5.1.2 Entity Schema Pattern

// Example: openmetadata-spec/src/main/resources/json/schema/entity/data/table.json
{
  "properties": {
    "updatedBy": {
      "description": "User who made the update.",
      "type": "string"
    },
    "impersonatedBy": {
      "description": "Bot that performed the update on behalf of updatedBy user.",
      "$ref": "../../type/basic.json#/definitions/impersonatedBy"
    },
    "updatedAt": {
      "$ref": "../../type/basic.json#/definitions/timestamp"
    }
  }
}

#### 5.1.3 User Schema for Bot Capability

// openmetadata-spec/src/main/resources/json/schema/entity/teams/user.json
{
  "properties": {
    "isBot": {
      "description": "When true, indicates this is a bot user.",
      "type": "boolean",
      "default": false
    },
    "allowImpersonation": {
      "description": "When true, this bot is allowed to impersonate users (subject to policy checks).",
      "type": "boolean",
      "default": false
    }
  }
}

5.2 Java Entity Interface Copy Section

// openmetadata-spec/src/main/java/org/openmetadata/schema/EntityInterface.java
public interface EntityInterface {
    // ... existing methods
    String getUpdatedBy();
    void setUpdatedBy(String updatedBy);
    // NEW: Impersonation tracking
    default String getImpersonatedBy() {
        return null;
    }
    default void setImpersonatedBy(String botName) {
        / no-op implementation to be overridden /
    }
}

5.3 Database Schema Migration Copy Section

#### MySQL Migration

/ Detailed source-code truncated for AI context efficiency. /

#### PostgreSQL Migration

-- bootstrap/sql/migrations/native/1.7.0/postgres/schemaChanges.sql
-- Same as MySQL with appropriate syntax
ALTER TABLE table_entity ADD COLUMN IF NOT EXISTS impersonatedBy VARCHAR(256);
-- ... etc

5.4 Elasticsearch/OpenSearch Index Mapping Copy Section

// Update index mappings to include impersonatedBy
{
  "mappings": {
    "properties": {
      "updatedBy": {
        "type": "keyword"
      },
      "impersonatedBy": {
        "type": "keyword"
      },
      "updatedAt": {
        "type": "date"
      }
    }
  }
}

6. Implementation Details Copy Chapter

6.1 Token Exchange API (New Endpoint) Copy Section

API Specification:

POST /api/v1/users/impersonate
Authorization: Bearer <bot-token>
Content-Type: application/json
Request Body:
{
  "targetUser": "alice",
  "expirySeconds": 3600  // Optional, defaults to 1 hour, max 24 hours
}
Response:
{
  "accessToken": "eyJhbGciOiJSUzI1NiIsIn...",
  "tokenType": "Bearer",
  "expiresIn": 3600,
  "impersonatedUser": "alice"
}

Implementation:

/ Detailed source-code truncated for AI context efficiency. /

New Request Schema:

// openmetadata-spec/src/main/resources/json/schema/auth/impersonationRequest.json
{
  "$id": "https://open-metadata.org/schema/auth/impersonationRequest.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ImpersonationRequest",
  "description": "Request to generate an impersonation token",
  "type": "object",
  "javaType": "org.openmetadata.schema.auth.ImpersonationRequest",
  "properties": {
    "targetUser": {
      "description": "Username of the user to impersonate",
      "type": "string"
    },
    "expirySeconds": {
      "description": "Token expiry in seconds (default 3600, max 86400)",
      "type": "integer",
      "minimum": 60,
      "maximum": 86400,
      "default": 3600
    }
  },
  "required": ["targetUser"],
  "additionalProperties": false
}

6.2 JWTTokenGenerator Enhancement Copy Section

Add new method to generate impersonation tokens:

/ Detailed source-code truncated for AI context efficiency. /

6.3 JwtFilter Enhancement Copy Section

/ Detailed source-code truncated for AI context efficiency. /

6.3 DefaultAuthorizer Update Copy Section

/ Detailed source-code truncated for AI context efficiency. /

6.4 EntityRepository Update Copy Section

// openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java
public abstract class EntityRepository<T extends EntityInterface> {
    protected void prepareInternal(T entity, CreateEntity request, String updatedBy) {
        // ... existing code ...
        entity.setUpdatedBy(updatedBy);
        entity.setUpdatedAt(System.currentTimeMillis());
        // NEW: Set impersonatedBy from SubjectContext
        SubjectContext subjectContext = SubjectContext.getSubjectContext(updatedBy);
        if (subjectContext.impersonatedBy() != null) {
            entity.setImpersonatedBy(subjectContext.impersonatedBy());
        }
        // ... rest of existing code ...
    }
    public final PutResponse<T> update(UriInfo uriInfo, T original, T updated, String updatedBy) {
        setFieldsInternal(original, putFields);
        updated.setUpdatedBy(updatedBy);
        updated.setUpdatedAt(System.currentTimeMillis());
        // NEW: Set impersonatedBy
        SubjectContext subjectContext = SubjectContext.getSubjectContext(updatedBy);
        if (subjectContext.impersonatedBy() != null) {
            updated.setImpersonatedBy(subjectContext.impersonatedBy());
        }
        // ... rest of existing code ...
    }
}

7. Frontend Changes Copy Chapter

7.1 Type Definitions (Auto-generated) Copy Section

// Generated from schema
export interface EntityReference {
  id: string;
  name: string;
  fullyQualifiedName?: string;
  // ... other fields
}
export interface Table extends EntityInterface {
  // ... other fields
  updatedBy?: string;
  impersonatedBy?: string;  // NEW
  updatedAt?: number;
}

7.2 UI Components Copy Section

#### Entity Header Component

// Display impersonation badge
const EntityHeader = ({ entity }: { entity: EntityInterface }) => {
  return (
    <div className="entity-header">
      <div className="updated-info">
        {entity.impersonatedBy ? (
          <Tooltip
            title={Action performed by ${entity.updatedBy} via bot ${entity.impersonatedBy}}
          >
            <span>
              Updated by <strong>{entity.updatedBy}</strong>
              <Tag color="blue" className="impersonation-tag">
                via {entity.impersonatedBy}
              </Tag>
            </span>
          </Tooltip>
        ) : (
          <span>Updated by <strong>{entity.updatedBy}</strong></span>
        )}
        <span className="timestamp">{formatDateTime(entity.updatedAt)}</span>
      </div>
    </div>
  );
};

#### Activity Feed

// Show impersonation in activity feed
const ActivityFeedItem = ({ activity }: { activity: ChangeEvent }) => {
  const displayName = activity.impersonatedBy
    ? ${activity.userName} (via ${activity.impersonatedBy})
    : activity.userName;
  return (
    <div className="activity-item">
      <Avatar name={activity.userName} />
      <div>
        <strong>{displayName}</strong> {activity.action} {activity.entityType}
        <time>{formatRelativeTime(activity.timestamp)}</time>
      </div>
    </div>
  );
};

7.3 Admin Settings UI Copy Section

// Bot configuration page
const BotSettings = ({ bot }: { bot: User }) => {
  const [allowImpersonation, setAllowImpersonation] = useState(bot.allowImpersonation);
  return (
    <div className="bot-settings">
      <h3>Impersonation Settings</h3>
      <Switch
        checked={allowImpersonation}
        onChange={(checked) => {
          updateBotSettings(bot.id, { allowImpersonation: checked });
          setAllowImpersonation(checked);
        }}
      />
      <p className="help-text">
        Allow this bot to impersonate users (subject to policy restrictions)
      </p>
      {allowImpersonation && (
        <Alert type="warning">
          This bot can perform actions on behalf of users. Ensure appropriate
          impersonation policies are configured.
        </Alert>
      )}
    </div>
  );
};

8. Usage Examples Copy Chapter

8.1 Python SDK Usage Copy Section

/ Detailed source-code truncated for AI context efficiency. /

8.2 Ingestion Workflow Example Copy Section

/ Detailed source-code truncated for AI context efficiency. /

8.3 REST API Examples Copy Section

#### Get Impersonation Token

curl -X POST https://localhost:8585/api/v1/users/impersonate \
  -H "Authorization: Bearer <bot-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "targetUser": "alice",
    "expirySeconds": 3600
  }'
Response:

{
  "accessToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik...",
  "tokenType": "Bearer",
  "expiresIn": 3600,
  "impersonatedUser": "alice",
  "jwtTokenExpiry": {
    "expiresAt": 1699123456789
  }
}

#### Use Impersonation Token

Update table using impersonation token

IMPERSONATION_TOKEN="eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ik..."
curl -X PATCH https://localhost:8585/api/v1/tables/{table-id} \
  -H "Authorization: Bearer $IMPERSONATION_TOKEN" \
  -H "Content-Type: application/json-patch+json" \
  -d '[{
    "op": "add",
    "path": "/description",
    "value": "Updated via impersonation"
  }]'
Response includes:

{
  "id": "...",
  "name": "dim_customer",
  "updatedBy": "alice",
  "impersonatedBy": "ingestion-bot",
  "updatedAt": 1699120000000
}

#### Verify Impersonation in Activity Feed

curl -X GET "https://localhost:8585/api/v1/feed?entityLink=<#E::table::..." \
  -H "Authorization: Bearer <token>"
Response shows:

{
  "data": [
    {
      "id": "...",
      "type": "entityUpdated",
      "userName": "alice",
      "impersonatedBy": "ingestion-bot",
      "timestamp": 1699120000000,
      "changeDescription": {
        "fieldsUpdated": [
          {
            "name": "description",
            "oldValue": "...",
            "newValue": "Updated via impersonation"
          }
        ]
      }
    }
  ]
}

8.4 Error Scenarios Copy Section

#### Bot Without Impersonation Permission

curl -X POST https://localhost:8585/api/v1/users/impersonate \
  -H "Authorization: Bearer <bot-token-without-permission>" \
  -d '{"targetUser": "alice"}'
Response: 403 Forbidden

{
  "code": 403,
  "message": "Bot 'my-bot' is not authorized to impersonate users",
  "exceptionType": "AuthorizationException"
}

#### Bot Without allowImpersonation Flag

Response: 403 Forbidden

{
  "code": 403,
  "message": "Bot 'my-bot' is not authorized to impersonate users",
  "exceptionType": "AuthorizationException"
}

#### Policy Restriction Violation

Bot tries to impersonate user from different domain

curl -X POST https://localhost:8585/api/v1/users/impersonate \
  -H "Authorization: Bearer <bot-token>" \
  -d '{"targetUser": "bob-from-marketing"}'
Response: 403 Forbidden

{
  "code": 403,
  "message": "Bot 'engineering-bot' is not authorized to impersonate user 'bob-from-marketing'",
  "exceptionType": "AuthorizationException"
}

#### Non-Bot User Attempt

Regular user tries to get impersonation token

curl -X POST https://localhost:8585/api/v1/users/impersonate \
  -H "Authorization: Bearer <regular-user-token>" \
  -d '{"targetUser": "alice"}'
Response: 403 Forbidden

{
  "code": 403,
  "message": "Only bot users can generate impersonation tokens",
  "exceptionType": "AuthorizationException"
}

9. API Changes Copy Chapter

9.1 REST API Behavior Copy Section

All existing CRUD endpoints remain unchanged. Impersonation is handled transparently through JWT token claims.

9.2 New Endpoint Summary Copy Section

| Method | Endpoint | Description | Auth Required |
|--------|----------|-------------|---------------|
| POST | /api/v1/users/impersonate | Request impersonation token | Bot with allowImpersonation=true |

8.2 Search and Filter Support Copy Section

Search for entities updated by specific user via bot

GET /api/v1/search/query?q=impersonatedBy:ingestion-bot
Search for specific user's actions (including impersonated)

GET /api/v1/search/query?q=updatedBy:alice
Activity feed with impersonation filter

GET /api/v1/feed?filterType=impersonated&bot=ingestion-bot

9. Security Considerations Copy Chapter

9.1 Threat Model Copy Section

| Threat | Mitigation |
|--------|-----------|
| Unauthorized Impersonation | Three-level validation: isBot + allowImpersonation flag + policy permission |
| Privilege Escalation | Use impersonated user's permissions, not bot's |
| Token Theft | Standard JWT security + short expiry + token validation |
| Audit Trail Tampering | Immutable audit logs with both user and bot tracked |
| Policy Bypass | Impersonation permission checked on every request |

9.2 Security Best Practices Copy Section

1. Least Privilege: Only enable allowImpersonation for bots that absolutely need it
2. Policy Restrictions: Use domain/team-based policies to limit scope
3. Token Expiry: Impersonation tokens should have shorter expiry (e.g., 1 hour)
4. Audit Monitoring: Alert on impersonation usage patterns
5. Regular Review: Periodic audit of which bots have impersonation enabled

9.3 Attack Scenarios and Defenses Copy Section

#### Scenario 1: Malicious Bot Attempts Impersonation

Attack: Bot without allowImpersonation tries to include impersonatedUser claim
Defense: JwtFilter validates allowImpersonation flag, rejects request
Result: 403 Forbidden - "Bot not authorized to impersonate"

#### Scenario 2: Bot Tries to Impersonate Admin

Attack: Bot with impersonation tries to impersonate admin user
Defense: Policy checks if bot can impersonate that specific user
Result: 403 Forbidden - "Bot 'X' not authorized to impersonate user 'admin'"

#### Scenario 3: Stolen Impersonation Token

Attack: Attacker obtains valid impersonation token
Defense:
  - Short token expiry (1 hour)
  - Token bound to specific user
  - Audit logs show suspicious activity patterns
  - Can revoke bot's allowImpersonation flag
Result: Limited blast radius, quick detection and mitigation

10. Testing Strategy Copy Chapter

10.1 Unit Tests Copy Section

// JwtFilterTest.java
@Test
public void testImpersonationValidation_Success() {
    // Given: Bot with allowImpersonation=true and valid token
    String token = generateImpersonationToken("test-bot", "alice");
    // When: Filter processes request
    CatalogSecurityContext context = jwtFilter.getCatalogSecurityContext(token);
    // Then: Context contains impersonation info
    assertEquals("test-bot", context.getUserPrincipal().getName());
    assertEquals("alice", context.impersonatedUser());
}
@Test
public void testImpersonationValidation_BotNotAuthorized() {
    // Given: Bot with allowImpersonation=false
    User bot = createBot("test-bot", false);
    String token = generateImpersonationToken("test-bot", "alice");
    // When/Then: Validation fails
    assertThrows(AuthorizationException.class, () -> {
        jwtFilter.validateJwtAndGetClaims(token);
    });
}
@Test
public void testNonBotCannotImpersonate() {
    // Given: Regular user tries to impersonate
    String token = JWT.create()
        .withSubject("alice")
        .withClaim("isBot", false)
        .withClaim("impersonatedUser", "bob")
        .sign(algorithm);
    // When/Then: Validation fails
    assertThrows(AuthorizationException.class, () -> {
        jwtFilter.validateJwtAndGetClaims(token);
    });
}

// EntityRepositoryTest.java
@Test
public void testEntityUpdate_WithImpersonation() {
    // Given: Bot impersonating user
    SubjectContext context = new SubjectContext(
        createUser("alice"),
        "ingestion-bot"  // impersonatedBy
    );
    Table table = createTable();
    // When: Update entity
    tableRepository.update(null, table, updatedTable, "alice");
    // Then: Both fields are set
    assertEquals("alice", updatedTable.getUpdatedBy());
    assertEquals("ingestion-bot", updatedTable.getImpersonatedBy());
}

10.2 Integration Tests Copy Section

/ Detailed source-code truncated for AI context efficiency. /

10.3 Security Tests Copy Section

/ Detailed source-code truncated for AI context efficiency. /

10.4 Performance Tests Copy Section

@Test
public void testImpersonationOverhead() {
    // Measure overhead of impersonation validation
    // Baseline: Normal request
    long baseline = measureRequestTime(() -> updateTableNormal());
    // With impersonation
    long withImpersonation = measureRequestTime(() -> updateTableWithImpersonation());
    // Assert: Overhead < 10%
    double overhead = (withImpersonation - baseline) / (double) baseline;
    assertTrue(overhead < 0.10, "Impersonation overhead should be < 10%");
}

11. Migration Plan Copy Chapter

Phase 1: Schema and Backend (Week 1-2) Copy Section

Phase 2: Authorization (Week 2-3) Copy Section

Phase 3: Bot Support (Week 3-4) Copy Section

Phase 4: Frontend (Week 4-5) Copy Section

Phase 5: Documentation and Rollout (Week 5-6) Copy Section

12. Monitoring and Observability Copy Chapter

12.1 Metrics Copy Section

// Metrics to track
- impersonation.requests.total (counter)
- impersonation.requests.by_bot (counter, labeled by bot name)
- impersonation.requests.by_target_user (counter, labeled by user)
- impersonation.validation.failures (counter, labeled by reason)
- impersonation.authorization.duration (histogram)

12.2 Logging Copy Section

// Log examples
LOG.info("Impersonation validated: bot={}, target={}, operation={}",
    botName, targetUser, operation);
LOG.warn("Impersonation attempt blocked: bot={}, target={}, reason={}",
    botName, targetUser, reason);
LOG.debug("Impersonation token processed: bot={}, user={}, entity={}",
    botName, userName, entityType);

12.3 Alerts Copy Section

Alert on suspicious patterns

- name: HighImpersonationFailureRate
  condition: impersonation.validation.failures > 10/min
  severity: warning
- name: UnauthorizedImpersonationAttempts
  condition: impersonation.requests{reason="unauthorized"} > 5/min
  severity: critical
- name: CrossDomainImpersonationSpike
  condition: impersonation.requests{cross_domain="true"} > 20/min
  severity: warning

13. Documentation Copy Chapter

13.1 User Guide Copy Section

For Bot Users:
- How to enable impersonation for your bot
- How to generate impersonation tokens
- Examples in Python SDK and REST API
- Troubleshooting common issues

For Admins:
- How to configure impersonation policies
- Best practices for security
- How to audit impersonation usage
- How to investigate suspicious activity

13.2 API Documentation Copy Section

Update OpenAPI specification:

components:
  securitySchemes:
    BotImpersonation:
      type: http
      scheme: bearer
      bearerFormat: JWT
      description: |
        JWT token with impersonation claim. Bot must have allowImpersonation=true
        and policy permission to impersonate the target user.
        Token claims:
        - isBot: true (required)
        - impersonatedUser: string (target username)

14. Future Enhancements Copy Chapter

14.1 Impersonation Audit Report Copy Section

14.2 Temporary Impersonation Tokens Copy Section

14.3 Impersonation Approval Workflow Copy Section

14.4 Delegation Instead of Impersonation Copy Section

15. Appendix Copy Chapter

15.1 Related Work Copy Section

15.2 Security Standards Copy Section

15.3 Glossary Copy Section

---

Document Version: 1.1
Last Updated: 2026-06-12
Authors: OpenMetadata Engineering
v1.1: Custom-bot impersonation grants via createBot.json, RBAC target scoping (isAdminUser()/isBotUser() conditions), read-only User.allowImpersonation on user APIs, Impersonate excluded from ALL subsumption. Default BotImpersonationPolicy is permissive/backward-compatible (admin impersonation stays allowed); admin restriction is opt-in via BotNonAdminImpersonationPolicy/BotNonAdminImpersonationRole. All seeds are new, so no migration is needed. See section 4.4.
Status: Draft for Review

---

Index Copy Section

Documentation Index

The map to OpenMetadata's written knowledge — design docs, plans, generated references, and the
reference docs that live outside docs/ (in the UI, ingestion, and bootstrap trees). Use it to
find an existing doc before reverse-engineering the code or guessing.

Freshness was verified against the working tree on 2026-07-24 (not inferred from age — each
verdict cites an artifact that was checked to still exist):

- CURRENT — its load-bearing references still exist and match the code.
- STALE — it references paths/classes/flags that no longer exist.
- SUPERSEDED — a newer doc or implementation replaced it (named in the row / footnotes).
- ⚠ — CURRENT overall, but with a specific caveat in the footnotes below.
- by construction — regenerated from source and CI-gated, so it cannot drift from its source.

This index lists one file each; it does not move or relink anything (migration is out of scope).

Start here — root guides (not under docs/) docs/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Guide | What it is |
|---|---|
| CLAUDE.md | Always-loaded session guidance; the pointer index to .claude/rules/* and skills |
| ARCHITECTURE.md | System map — modules, the request/ingestion/search paths, the invariants that hold |
| DEVELOPER.md | How to build, test, and add an entity or connector (end-to-end checklists) |
| AGENTS.md | Codex entry doc — carries known contradictions (Webpack→Vite, antd, Python ceiling); see docs/tech-debt.md #6 |

Backend & platform design docs (docs/) docs/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| docs/impersonation-design.md | Bot→user impersonation: updatedBy=user / impersonatedBy=bot, gated by allowImpersonation + RBAC Impersonate policy scoping | Touching bot impersonation auth — the flag, BotImpersonationPolicy seeds, checkImpersonationAuthorization (§4.4 is authoritative) | 2026-06-16 | CURRENT ⚠¹ |
| docs/session-management-multi-node-design.md | Shipped multi-node server-side session + websocket system: shared JDBC/Redis store, OM_SESSION cookie, session-bound JWTs, CAS refresh | Working on login/refresh/logout across pods, SessionService/SessionStore, JWT session validation, websocket handshake | 2026-06-03 | CURRENT |
| docs/streamable-logs.md | S3/MinIO-backed streamable ingestion logs: HTTP append/close, partial.txt→logs.txt, SSE live tail, abandoned-run sweeper | Working on ingestion log storage/streaming — S3LogStorage, LogStorageInterface, /logs/{fqn}/{runId} endpoints | 2026-05-15 | CURRENT |
| docs/ingestion-log-streaming.md | Live log tail over SSE: LogStreamEvent schema, resume cursors, one shared reader per run, and the limits that bound every stream | Building or debugging a client that tails ingestion logs live — /logs/{fqn}/stream/{runId}, IngestionLogTailer, LogStreamSettings | 2026-08-10 | CURRENT |
| docs/rdf-local-development.md | Run RDF/knowledge-graph support locally with Apache Jena Fuseki — startup scripts, env vars, rdf. config, /api/v1/rdf/, RdfIndexApp | Setting up or debugging local RDF/Fuseki development | 2026-07-16 | CURRENT |
| docs/rdf-production-setup.md | Production sizing/tuning for a remote Fuseki triple store — TDB2 heap vs page-cache, batch/timeout, weekly recreate, compaction | Sizing, scheduling, or troubleshooting a prod Fuseki deployment; tuning RDF bulk-write | 2026-07-16 | CURRENT |
| docs/csv-relation-types-plan.md | Carry glossary term relation types through CSV export/import via a relationType:termFQN prefix (default relatedTo) | Modifying glossary CSV round-tripping — CsvUtil.addTermRelations, GlossaryRepository.getTermRelationsFromCsv | 2026-03-17 | CURRENT |
| docs/auto-classification/add-support-for-another-entity.md | Step-by-step: extend auto-classification (PII + sample data) to a new entity across schema/Java/Python/UI via the EntityAdapter registry | Adding auto-classification/sample-data support for a new entity type | 2026-05-19 | CURRENT |
| docs/perf/cdn-deployment-guide.md | AWS design proposal: per-customer/per-release UI bundles from one CloudFront + S3 via an embedded CloudFront Function router (no Lambda@Edge) | Planning/reviewing CDN delivery of the UI bundle + per-customer version pinning — infra design, not existing code | 2026-05-25 | CURRENT (unimplemented proposal) |

Plans & specs (docs/plans/, docs/superpowers/) docs/plans/, docs/superpowers/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| docs/plans/2026-01-27-search-indexing-stats-redesign.md | Rework SearchIndexingApp stats into a per-stage pipeline model (StageStatsTracker/StageCounter), index-alias promotion, vector bulk processor | Before touching search reindex stats, search_index_server_stats, index promotion, vector indexing | 2026-01-29 | CURRENT (shipped design) |
| docs/plans/2026-06-22-bulk-deletion-redesign.md | Fast, orphan-free, resumable service-level recursive hard-deletion by id-set; self-audits what landed on main vs remaining gaps | Before working on recursive/bulk deletion, entity_relationship orphan cleanup, the deletion-lock gate | 2026-06-26 | CURRENT (shipped design) |
| docs/superpowers/specs/2026-06-22-logviewer-modal-design.md | Design spec for the reusable LogViewerModal (dark terminal modal over @melloware/react-logviewer); self-marked "Implemented; revised 2026-06-24" | Before modifying LogViewerModal, its log-level parser, theming, or streaming container/hook | 2026-07-16 | CURRENT |
| docs/superpowers/plans/2026-06-22-logviewer-modal.md | Original TDD build plan for LogViewerModal (built-in LazyLog search, CopyToClipboardButton) | Historical context only — the shipped component follows the revised spec, not this plan | 2026-07-16 | SUPERSEDED ² |

Generated references (docs/generated/) docs/generated/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| docs/generated/entity-index.md | Auto-generated: 81 first-class entities → schema JSON, Java POJO, Python model, TS type, REST resource class | Locate every codegen artifact / the REST resource for an entity without grepping four trees | 2026-07-24 | CURRENT (by construction) ³ |
| docs/generated/api-reference.md | Auto-generated: all 1748 REST endpoints (method + path + @Operation summary) grouped by resource package | Find an endpoint's exact path/method, or enumerate a package's routes without reading JAX-RS classes | 2026-07-24 | CURRENT (by construction) ³ |

Repo audit & quality (docs/) docs/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| docs/golden-principles.md | 8 candidate repo-wide invariants (DRAFT for ratification), each with a measured adherence number + reproducing command | Cite/enforce an invariant (acyclic modules, ServiceSpec contract, generated-as-sink, no bare except:, …) | 2026-07-24 | CURRENT |
| docs/tech-debt.md | Prioritized (impact÷size, 3 tiers) ledger of 21 audit findings, each with location, size, and agent-fixability | Pick up a bounded cleanup, or understand a known structural debt before working near it | 2026-07-24 | CURRENT |
| docs/quality.md | One evidence-cited quality grade (A–C / Not assessed) per Maven module | Gauge a module's structural health / known debt before a large change | 2026-07-24 | CURRENT |

Assets (docs/assets/) docs/assets/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| docs/assets/ (4 PNGs) | Architecture/marketing diagrams for the "Open Context Layer for AI" — hero, architecture, context graph, memory-primitives | Editing the root README.md visuals | 2026-06-10 | CURRENT (sole consumer: root README.md) |

UI reference docs (outside docs/) docs/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| openmetadata-ui/src/main/resources/ui/specs/ | Machine-readable design system (41 files). README.md declares two stacks — go-forward = UntitledUI + Tailwind (tw:), legacy (deprecated) = Ant Design + Less — plus foundations/ (color, spacing, typography, radius, elevation, motion), tokens/ (Tailwind-utility + master token reference), untitled/ (go-forward component specs), and legacy components/ | Before writing or modifying any UI code — start at specs/README.md, then the foundations/tokens and the untitled/<component>.md (or legacy components/*) spec for what you touch | 2026-07-27 | CURRENT ⁶ |
| openmetadata-ui/src/main/resources/ui/docs/colors.md | Semantic color-token system (tw:bg-primary, tw:text-fg-, tw:border-) with light/dark values + the mandatory ring→border migration (§2.3.1) | Before writing/reviewing any Tailwind color class or dark-mode styling, or when tempted to use ring-* or a raw hex | 2026-07-23 | CURRENT |
| openmetadata-ui/src/main/resources/ui/docs/formutils.md | The modern react-hook-form + react-aria form stack (FieldProp, getField/FormFields/HookForm) vs the legacy antd @utils/formUtils API | Before building/modifying any UI form — which API to use, and wiring to useFormDrawerWithHook + a pure transform | 2026-07-15 | CURRENT |
| openmetadata-ui/src/main/resources/ui/playwright/docs/ | Auto-generated E2E test-coverage catalog (README.md + Discovery/Governance/Integration/Observability/Platform) mapping component → spec file → scenarios | Check what UI behavior is already E2E-covered before writing Playwright tests or reasoning about gaps | 2026-03-09 – 2026-07-16 | CURRENT (by construction) ³ |

Ingestion & bootstrap reference (outside docs/) docs/)')" title="Copy chapter prompt for LLMs"> Copy Chapter

| Doc | Purpose | Read when | Modified | Freshness |
|---|---|---|---|---|
| ingestion/docs/design/ingestion-diagnostics.md | DEBUG-gated ingestion diagnostics subsystem (operation registry, watchdog, heartbeat, memory tracker, HTTP introspection, stage backpressure, signal dumps) | Understand why/how the diagnostics subsystem works before instrumenting ingestion hangs/OOMs | 2026-05-20 | CURRENT ⚠⁴ |
| bootstrap/MIGRATION_SYSTEM.md | Hybrid DB-migration architecture — Flyway→native→extension execution order, SERVER_CHANGE_LOG tracking, file layout (Flyway parsers only, not the Flyway runner) | Before adding/debugging a migration under bootstrap/sql/migrations/, or reasoning about ordering / tracking tables / MySQL+Postgres dual paths | 2025-10-28 | CURRENT ⚠⁵ |

Caveats (verification notes) Copy Chapter

1. impersonation-design — §4.4 (v1.1) is authoritative and matches shipped code (allowImpersonation in createBot.json/user.json, checkImpersonationAuthorization in DefaultAuthorizer.java, the four policy/role seeds, BotImpersonationIT). The earlier §4.1/4.2 POST /users/impersonate token-exchange endpoint was never shipped — impersonation uses the X-Impersonate-User header (JwtFilter.java), which §4.4 supersedes 4.1/4.2 to reflect. Read §4.4.
2. logviewer-modal plan — superseded by the 2026-06-24 revision of the design spec (row above): the plan's built-in LazyLog search + CopyToClipboardButton were reversed (search moved to the header, a footer status bar added). The shipped dir (LogViewerModal.utils.tsx, LogsViewerModalContainer.tsx, useLogStream.ts) follows the revised spec, not this plan.
3. by construction — docs/generated/* are regenerated by make generate-reference-docs and gated by the reference-docs freshness CI job; the Playwright docs by playwright/doc-generator/generate.js + the playwright-docs-check.yml workflow. They cannot drift from their source on the watched paths. (One benign lag: the Playwright README.md roll-up footer reads 2026-03-09 while Governance.md regenerated 2026-07-16; self-corrects on the next spec-touching PR.)
4. ingestion-diagnostics — the design shipped (ingestion/src/metadata/ingestion/diagnostics/, activated at loggerLevel == DEBUG, installed in metadata/workflow/base.py), but the doc's §5/§7 flat file map drifted: files now live under collectors/, monitors/, samplers/; there is no standalone heartbeat.py; wire-in is base.py, not the doc's base_workflow.py. Treat the file map as design-era, the behavior as current.
5. MIGRATION_SYSTEM — accurate (the flyway/+native/ layout, SERVER_CHANGE_LOG, MigrationWorkflow/FlywayMigrationFile, and conf/openmetadata.yaml flywayPath/nativePath/extensionPath all verified) except: the runner class is MigrationProcessImpl (doc says MigrationProcess), and the extensions/ dir is not materialized on disk (extensionPath: "") though the path is still supported.
6. ui/specs — tracked (41 files) and CURRENT: all four audit commands its README.md cites resolve to real package.json scripts (tw-audit, tw-audit:report, tw-guard, token-audit), and the two-stack policy it states (go-forward UntitledUI+Tailwind, legacy Antd+Less deprecated) matches the enforced rules — tw-guard blocks new antd imports / new .less files, and .claude/rules/frontend-styling.md routes agents into specs/README.md.

---

Not indexed: docs/harness-audit/ exists in the working tree but is untracked (working audit notes, intentionally out of version control), so it is not part of the committed knowledge base.

---

Ingestion Log Streaming Copy Section

Live Ingestion Log Streaming (SSE)

How a client tails an ingestion pipeline's logs in real time over Server-Sent Events instead of
polling /logs/{id}/last on a timer.

For where the log bytes are stored and how they get there, read
streamable-logs.md. This document is about the read path a UI uses while a
run is in progress.

Why streaming instead of polling Copy Chapter

The paginated endpoint (GET /logs/{id}/last?after=<cursor>) forces every viewer into a poll loop:
pick an interval, re-issue the request, diff the cursor, repeat until the run ends. That has three
costs the streaming endpoint removes:

| Polling | Streaming |
|---|---|
| Every open tab is its own poll loop against S3 / Airflow. | All viewers of a run share one server-side reader. |
| Latency is the poll interval. | New content is pushed as soon as the shared reader sees it. |
| The client decides when the run is over (or never stops). | The server sends an explicit complete event and closes. |
| A "fetch everything" loop can walk pages without bound. | Reads per tick, bytes per stream, and stream lifetime are all capped. |

The endpoint Copy Chapter

GET /api/v1/services/ingestionPipelines/logs/{fqn}/stream/{runId} GET /api/v1/services/ingestionPipelines/logs/{fqn}/stream/{runId}')" title="Copy section prompt for LLMs"> Copy Section

One endpoint for every deployment. {fqn} is the pipeline's fullyQualifiedName or Id (UUID),
like /logs/{id}/last. {runId} is the run to follow — a UUID when the run's logs are in object
storage, or the pipeline service's own run identifier (Airflow's scheduled__…) otherwise.

| Query parameter | Default | Meaning |
|---|---|---|
| after | beginning of the log | Resume cursor from a previous event. Content up to the cursor is not re-sent. |

To tail the newest run, read its runId from pipelineStatuses first — the same field
/logs/{id}/last resolves internally.

Response: text/event-stream. Every frame is one JSON
LogStreamEvent
on an unnamed SSE event, so a plain EventSource.onmessage receives all of them.

// eventType: "logs" — new content
{"eventType":"logs","runId":"a1b2…","logs":"[2026-08-10 …] INFO Ingesting table x","after":"4211","replay":false,"truncated":false}
// eventType: "complete" — the server is closing the stream
{"eventType":"complete","runId":"a1b2…","after":"4680","reason":"runFinished"}
// eventType: "error" — the stream cannot be served; it is closed right after
{"eventType":"error","runId":"a1b2…","message":"The server is already streaming the maximum number of pipeline runs. …"}

| Field | Meaning |
|---|---|
| eventType | logs, complete or error. |
| runId | Run the content belongs to. |
| logs | Content appended since the previous event. Absent on complete / error. |
| after | Cursor pointing just past logs. Store it: it is what you pass to ?after= to reconnect. |
| replay | true when the chunk came from the server's replay buffer because the stream was already running when you connected. |
| truncated | true on the first event when the server could not work out exactly what you are missing. Treat it as a reset: clear the viewer, render the replayed block that follows, and backfill earlier history from GET /logs/{id}/last or the download endpoint. |
| reason | Why the stream ended. See the table below. |
| message | Human-readable detail on error, and on a complete that ended early. |

Between events the server sends SSE heartbeat comments (: heartbeat) every 25 s so proxies do not
drop an idle connection. EventSource ignores them.

End-of-stream reasons

| reason | What happened | What a client should do |
|---|---|---|
| runFinished | The run reached a terminal state and its log went quiet — or the server has no status row for it and it stayed quiet for a minute. | Nothing. The log is complete. |
| idleTimeout | No new content for 5 minutes and the run never reported a terminal state. | Reconnect with ?after= if you still care about the run. |
| maxDuration | The stream hit its 1 hour lifetime cap. | Reconnect with ?after=. |
| maxBytes | The stream delivered 32 MB. | Use GET /logs/{id}/last/download for the rest. |

A stream that ends without a complete event was cut short — the client stopped draining the
socket and crossed its backlog ceiling, the server went away, or the network dropped. Treat a closed
body with no complete as "reconnect with ?after=<last cursor>".

Back off between reconnects. The causes above are often persistent: a viewer that cannot keep up
crosses its backlog ceiling again on the next attempt, and each reconnect re-fetches the replay
backlog. Reconnecting immediately in a loop turns one struggling client into a load generator. Use a
capped exponential delay, and give up after a few consecutive failures rather than retrying forever.

Status codes

| Code | When |
|---|---|
| 200 | Stream opened. It is committed immediately — the response never waits for the run to finish. |
| 404 | No such pipeline. |

Everything that goes wrong after the pipeline resolves is reported as an error event on the
stream, not as an HTTP status: no log backend configured, the server at its stream capacity, a
viewer past the connection cap. A client therefore needs one error path (handle eventType:
"error"), not two.

Using it from a browser Copy Chapter

EventSource cannot set an Authorization header, so use fetch with a streaming reader:

const controller = new AbortController();
let cursor: string | undefined;
const tail = async (fqn: string, runId: string) => {
  const url = new URL(
    ${getBasePath()}/api/v1/services/ingestionPipelines/logs/${getEncodedFqn(
      fqn
    )}/stream/${encodeURIComponent(runId)},
    window.location.origin
  );
  if (cursor) {
    url.searchParams.set('after', cursor);
  }
  const response = await fetch(url, {
    headers: { Authorization: Bearer ${await getOidcToken()} },
    signal: controller.signal,
  });
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  for (;;) {
    const { done, value } = await reader.read();
    if (done) {
      break;
    }
    buffer += decoder.decode(value, { stream: true });
    const frames = buffer.split('\n');
    buffer = frames.pop() ?? '';
    for (const frame of frames) {
      if (!frame.startsWith('data: ')) {
        continue; // heartbeat comment or blank separator
      }
      const event = JSON.parse(frame.slice(6));
      cursor = event.after ?? cursor;
      if (event.eventType === 'logs') {
        appendToViewer(event.logs);
      } else if (event.eventType === 'complete') {
        onStreamEnd(event.reason); // reconnect here for a non-runFinished reason
      } else {
        onStreamError(event.message);
      }
    }
  }
};

Reconnecting is always ?after=<last cursor you saw>. The cursor is opaque: it means "line offset"
for object storage and "chunk offset" for Airflow, and the two are not interchangeable, so never
construct one by hand.

If the run is still being tailed for someone else when you reconnect, the server resumes you from
the shared reader's buffer: it replays exactly the chunks issued after your cursor. When your cursor
is older than that buffer — or came from another server behind the load balancer — the server cannot
tell what sits in between, so it replays what it has with truncated: true. That is the one case
where a client must reset its viewer rather than append.

Using it from the command line Copy Chapter

curl -N -H "Authorization: Bearer $OM_TOKEN" \
  "http://localhost:8585/api/v1/services/ingestionPipelines/logs/my.pipeline.fqn/stream/$RUN_ID"

-N disables curl's buffering, which is what makes the live tail visible.

How it works Copy Chapter

GET /logs/{fqn}/stream/{runId}
        │
        ▼
IngestionPipelineResource ──▶ IngestionLogStreamFactory ──▶ picks the source for where the bytes are
        │                                                     ├─ StorageLogTailSource        (S3, line cursor)
        │                                                     └─ PipelineServiceLogTailSource (Airflow/Argo, chunk cursor)
        ▼
IngestionLogStreamManager ──▶ one IngestionLogTailer per (pipeline, run)
        │                          │  polls every 2 s on a shared scheduler
        │                          │  keeps a bounded replay buffer
        │                          └▶ fans each chunk out to every viewer
        └─ SseConnectionRegistry: connection cap + 25 s heartbeat + disconnect sweep

One reader per run. The tailer is keyed by (storage backend, pipeline FQN, run). Opening the
same run in ten tabs creates ten SSE connections but still exactly one reader against S3 or Airflow.
The last viewer to disconnect stops the reader, so an unwatched run is not read at all.

Cursors are per backend. Object storage paginates a run's log by line offset. The pipeline
service paginates it in fixed-size chunks and keeps appending to the last chunk while the task runs,
so a chunk-index cursor alone would re-deliver a growing chunk on every poll — the cursor is
chunk:charactersAlreadyDelivered and only the growth is emitted. The source also never asks for a
chunk index the backend did not itself report, because Airflow answers an out-of-range chunk with a
400.

Knowing when to stop. The run's state is read from its pipeline-status rows, at most once every
10 s and never again once terminal — no call to the pipeline service is made for this. Once the run
is terminal the stream stays open until the log has been quiet for 10 s, so the final flush of a
just-finished run is still delivered.

Only the last few runs of a pipeline keep a status row, so a run the server finds no row for is
unknown, not finished — that describes a run that aged out of the window and one that was
triggered a second ago and has not written anything yet. An unknown run gets a full minute of
silence before the stream closes, which is what stops a freshly triggered pipeline from being
reported as already over while Airflow is still starting the task.

Nothing is unbounded. Every limit below is enforced by LogStreamSettings:

| Limit | Default | What it protects |
|---|---|---|
| pollSeconds | 2 | Read rate against the log backend, per run. |
| linesPerRead | 1000 | Page size held in heap at any moment. |
| maxReadsPerTick | 20 | Burst of backend calls when catching up on a backlog. |
| maxBytesPerTick | 1 MB | Content pushed at a client in one tick. |
| maxStreamBytes | 32 MB | Total a single stream will deliver. |
| maxStreamSeconds | 3600 | Lifetime of a forgotten browser tab. |
| maxIdleSeconds | 300 | Runs that die without reporting a terminal state. |
| finishGraceSeconds | 10 | Silence after a confirmed-terminal run before closing. |
| unknownRunGraceSeconds | 60 | Silence before closing a run with no status row — a just-triggered run needs time to start writing. |
| maxReplayBytes | 256 KB | Backlog kept per run for late joiners. |
| maxPendingBytesPerClient | 4 MB | Memory a stalled browser can pin. |
| maxActiveRuns | 200 | Runs tailed concurrently per server. |
| maxActiveConnections | 500 | Open log stream connections per server. |

A request past maxActiveRuns or maxActiveConnections gets an error event and a closed stream
rather than being queued.

Polling runs on a shared scheduler sized from maxActiveRuns (one thread per 25 runs, clamped to
2–16). A poll is a network read, so a slow backend shows up as a longer effective poll interval for
everyone rather than as a stalled stream — and a poll that fails for any reason closes its own
stream and gives the run's tail slot back instead of leaving a dead reader behind.

Multi-server deployments Copy Chapter

A tailer is per server. Two servers behind a load balancer that both have viewers for the same run
each keep one reader, which is the intended trade: reads are cheap and idempotent, and no
cross-server coordination is needed. The write path's sticky-session requirement (see
streamable-logs.md) does not apply here — reading
partial.txt from S3 works from any instance.

Source files Copy Chapter

- openmetadata-service/src/main/java/org/openmetadata/service/logstorage/stream/ — the streaming engine
- openmetadata-service/src/main/java/org/openmetadata/service/sse/SseConnectionRegistry.java — connection cap and heartbeat
- openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/logStreamEvent.json — the event schema
- openmetadata-service/src/test/java/org/openmetadata/service/logstorage/stream/ — unit tests
- openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/IngestionPipelineLogStreamIT.java — end-to-end test

---

Quality Copy Section

Module Quality Grades

One grade per module, each citing the specific measurement behind it. No grade without evidence —
where the audit didn't measure a module, it is marked Not assessed rather than guessed. The
measurements come from a one-time repository audit: the Maven module graph, per-language package/import
censuses, and convention-adherence counts (each reproducible by the commands in golden-principles.md).

Scale: A exemplary · B solid with bounded debt · C works but carries structural debt ·
Not assessed = insufficient evidence.

| Module | Grade | Evidence |
|---|---|---|
| ingestion | B+ | strongest architectural discipline measured |
| openmetadata-spec | B (evidence-limited) | clean codegen foundation; minor POM hygiene |
| openmetadata-service | C+ | pristine surface hygiene, badly tangled internals |
| openmetadata-ui | C | disciplined component model, heavy architectural + i18n debt |
| openmetadata-mcp | Not assessed | only its DAG position is known |
| openmetadata-sdk | Not assessed | only its DAG position is known |

---

ingestion — B+ Copy Chapter

openmetadata-spec — B (evidence-limited) Copy Chapter

openmetadata-service — C+ Copy Chapter

openmetadata-ui — C Copy Chapter

openmetadata-mcp — Not assessed Copy Chapter

openmetadata-sdk — Not assessed Copy Chapter

---

Note on the two "Not assessed" modules Copy Section

---

Rdf Local Development Copy Section

RDF/Apache Jena Local Development Guide

This guide documents how to set up RDF/Knowledge Graph support for local development with OpenMetadata and Apache Jena Fuseki.

For production sizing, tuning, compaction, scheduling, and monitoring, see Setting up Apache Jena Fuseki efficiently.

Overview Copy Chapter

OpenMetadata supports RDF (Resource Description Framework) for knowledge graph capabilities using Apache Jena Fuseki as the triple store. This enables:
- SPARQL queries against metadata
- JSON-LD serialization of entities
- Semantic search and graph exploration

Architecture Copy Chapter

┌─────────────────────┐     ┌─────────────────────┐
│   OpenMetadata      │     │   Apache Jena       │
│   Server (IntelliJ) │────▶│   Fuseki (Docker)   │
│   Port: 8585        │     │   Port: 3030        │
└─────────────────────┘     └─────────────────────┘

Prerequisites Copy Chapter

- Docker and Docker Compose installed
- IntelliJ IDEA with the project imported
- MySQL or PostgreSQL running (for OpenMetadata backend)
- Elasticsearch running (for search)

Quick Start Copy Chapter

Step 1: Choose the Right Startup Mode Copy Section

The standard local Docker flow does not enable RDF or start Fuseki:

cd /path/to/OpenMetadata
./docker/run_local_docker.sh -d mysql

For PostgreSQL-based development:

./docker/run_local_docker.sh -d postgresql

Use the RDF-specific startup script when you want the full Docker stack with Fuseki enabled:

./docker/run_local_docker_rdf.sh -d mysql

For PostgreSQL-based RDF development:

./docker/run_local_docker_rdf.sh -d postgresql

This RDF startup path starts OpenMetadata, the backing database, search, ingestion services, and Fuseki with:
- Port: 3030
- Admin Password: admin
- Dataset: openmetadata
- Memory: 2-4GB allocated

Step 2: Verify Fuseki is Running Copy Section

Check Fuseki health

curl -s http://localhost:3030/$/ping
Access Fuseki UI in browser

open http://localhost:3030

The Fuseki web UI is available at http://localhost:3030 with credentials:
- Username: admin
- Password: admin

Step 3: Configure IntelliJ Run Configuration Copy Section

If you are running the full RDF Docker stack with run_local_docker_rdf.sh, the Docker services already receive the RDF environment variables automatically.

If you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker, start Fuseki separately:

docker compose -f docker/development/docker-compose.yml -f docker/development/docker-compose-fuseki.yml up -d fuseki

If your local backend uses PostgreSQL, swap docker-compose.yml for docker-compose-postgres.yml.

Create or modify your IntelliJ run configuration for OpenMetadataApplication with these environment variables only when you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker:

RDF_ENABLED=true
RDF_STORAGE_TYPE=FUSEKI
RDF_BASE_URI=https://open-metadata.org/
RDF_ENDPOINT=http://localhost:3030/openmetadata
RDF_REMOTE_USERNAME=admin
RDF_REMOTE_PASSWORD=admin
RDF_DATASET=openmetadata

#### Setting Environment Variables in IntelliJ:

1. Open Run → Edit Configurations
2. Select your OpenMetadataApplication configuration
3. Click on Modify options → Environment variables
4. Add the environment variables above (semicolon-separated or using the dialog)

Example environment variables string:

RDF_ENABLED=true;RDF_STORAGE_TYPE=FUSEKI;RDF_BASE_URI=https://open-metadata.org/;RDF_ENDPOINT=http://localhost:3030/openmetadata;RDF_REMOTE_USERNAME=admin;RDF_REMOTE_PASSWORD=admin;RDF_DATASET=openmetadata

Step 4: Start OpenMetadata Server Copy Section

Run OpenMetadataApplication from IntelliJ. On startup, you should see in the logs:

INFO  [main] o.o.s.OpenMetadataApplication - RDF knowledge graph support initialized

Step 5: Verify RDF is Enabled Copy Section

Check RDF status

curl http://localhost:8585/api/v1/rdf/status
Expected response:

{"enabled": true}

Configuration Reference Copy Chapter

Server Configuration (conf/openmetadata.yaml) Copy Section

The RDF configuration section in openmetadata.yaml:

rdf:
  enabled: ${RDF_ENABLED:-false}
  baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"}
  storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"}
  remoteEndpoint: ${RDF_ENDPOINT:-${RDF_REMOTE_ENDPOINT:-"http://localhost:3030/openmetadata"}}
  connectTimeoutMs: ${RDF_CONNECT_TIMEOUT_MS:-2000}
  requestTimeoutMs: ${RDF_REQUEST_TIMEOUT_MS:-60000}
  bulkEntityBatchSize: ${RDF_BULK_ENTITY_BATCH_SIZE:-100}
  bulkRelationshipSourceBatchSize: ${RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE:-100}
  bulkLineageEdgeBatchSize: ${RDF_BULK_LINEAGE_EDGE_BATCH_SIZE:-50}
  username: ${RDF_REMOTE_USERNAME:-"admin"}
  password: ${RDF_REMOTE_PASSWORD:-"admin"}
  dataset: ${RDF_DATASET:-"openmetadata"}
  inferenceEnabled: ${RDF_INFERENCE_ENABLED:-false}

Environment Variables Copy Section

| Variable | Description | Default |
|----------|-------------|---------|
| RDF_ENABLED | Enable/disable RDF support | false |
| RDF_STORAGE_TYPE | Storage backend type | FUSEKI |
| RDF_BASE_URI | Base URI for RDF resources | https://open-metadata.org/ |
| RDF_ENDPOINT | Fuseki SPARQL endpoint URL | http://localhost:3030/openmetadata |
| RDF_REMOTE_ENDPOINT | Deprecated fallback when RDF_ENDPOINT is unset | unset |
| RDF_CONNECT_TIMEOUT_MS | Fuseki connection timeout | 2000 |
| RDF_REQUEST_TIMEOUT_MS | Per-request timeout | 60000 |
| RDF_BULK_ENTITY_BATCH_SIZE | Entity models per bulk write | 100 |
| RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE | Relationship sources per bulk write | 100 |
| RDF_BULK_LINEAGE_EDGE_BATCH_SIZE | Detailed lineage edges per bulk write | 50 |
| RDF_REMOTE_USERNAME | Fuseki admin username | admin |
| RDF_REMOTE_PASSWORD | Fuseki admin password | admin |
| RDF_DATASET | Fuseki dataset name | openmetadata |
| RDF_INFERENCE_ENABLED | Enable in-process full-graph inference | false |

Docker Compose Configuration Copy Section

The Fuseki container (docker/development/docker-compose-fuseki.yml):

services:
  fuseki:
    build:
      context: ../rdf-store
      dockerfile: Dockerfile
    image: openmetadata-fuseki:5.6.0
    container_name: openmetadata-fuseki
    ports:
      - "3030:3030"
    environment:
      - FUSEKI_ADMIN_PASSWORD=admin
      - FUSEKI_OPENMETADATA_PASSWORD=openmetadata-secret
      - JVM_ARGS=-Xmx1500m -Xms256m
    volumes:
      - fuseki-tdb2-data:/fuseki-data

API Endpoints Copy Chapter

Once RDF is enabled, these endpoints are available:

Check RDF Status Copy Section

GET /api/v1/rdf/status

Get Entity as RDF Copy Section

Get entity in JSON-LD format (default)

GET /api/v1/rdf/entity/{entityType}/{id}
Get entity in Turtle format

GET /api/v1/rdf/entity/{entityType}/{id}?format=turtle
Get entity in RDF/XML format

GET /api/v1/rdf/entity/{entityType}/{id}?format=rdfxml
Get entity in N-Triples format

GET /api/v1/rdf/entity/{entityType}/{id}?format=ntriples

Execute SPARQL Query Copy Section

POST /api/v1/rdf/sparql
Content-Type: application/json
{
  "query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"
}

Get Glossary Term Relationship Graph Copy Section

Get the full glossary term graph

GET /api/v1/rdf/glossary/graph
Filter primary terms to a glossary

GET /api/v1/rdf/glossary/graph?glossaryId=<glossary-id>
Filter to a glossary term and its direct incoming/outgoing neighbors

GET /api/v1/rdf/glossary/graph?glossaryTermId=<glossary-term-id>
Require the selected term to belong to a glossary, while still returning

direct cross-glossary neighbors when relationships cross glossary boundaries

GET /api/v1/rdf/glossary/graph?glossaryId=<glossary-id>&glossaryTermId=<glossary-term-id>

Optional query parameters:

| Parameter | Description |
|-----------|-------------|
| glossaryId | Filter primary terms to a glossary. |
| glossaryTermId | Filter to a selected glossary term and its direct incoming/outgoing glossary-term relations. |
| relationTypes | Comma-separated relation types to include. |
| limit | Maximum number of terms to return. Default: 500. |
| offset | Pagination offset. Default: 0. |
| includeIsolated | Include terms without relations. Default: true. |

Example Queries Copy Section

Check if RDF is enabled

curl -s http://localhost:8585/api/v1/rdf/status | jq
Get a table entity as JSON-LD

curl -s -H "Authorization: Bearer <token>" \
  "http://localhost:8585/api/v1/rdf/entity/table/<table-id>" | jq
Execute a SPARQL query

curl -s -X POST \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"}' \
  http://localhost:8585/api/v1/rdf/sparql | jq
Get a selected glossary term graph

curl -s -H "Authorization: Bearer <token>" \
  "http://localhost:8585/api/v1/rdf/glossary/graph?glossaryId=<glossary-id>&glossaryTermId=<glossary-term-id>" | jq

Indexing Entities to RDF Copy Chapter

Manual Reindexing Copy Section

Trigger the RDF indexing application to populate the triple store with existing entities:

curl -X POST \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"entities": [], "recreateIndex": true, "batchSize": 100}' \
  http://localhost:8585/api/v1/apps/trigger/RdfIndexApp

Automatic Indexing Copy Section

When RDF is enabled, new entities are automatically indexed to the triple store on create/update/delete operations.

Fuseki Web UI Copy Chapter

The Fuseki web interface provides:

- Dataset Management: View and manage datasets at http://localhost:3030/#/manage
- SPARQL Query Interface: Execute queries at http://localhost:3030/#/dataset/openmetadata/query
- Data Upload: Upload RDF data at http://localhost:3030/#/dataset/openmetadata/upload

Troubleshooting Copy Chapter

Fuseki Connection Issues Copy Section

1. Verify Fuseki is running:

docker ps | grep fuseki
   curl http://localhost:3030/$/ping

2. Check Fuseki logs:

docker logs openmetadata-fuseki

3. Ensure the dataset exists:

curl -u admin:admin http://localhost:3030/$/datasets

RDF Not Enabled in Server Copy Section

1. Verify environment variables are set correctly in IntelliJ
2. Check server logs for RDF initialization message
3. Confirm configuration in openmetadata.yaml

SPARQL Query Errors Copy Section

1. Check Fuseki is accessible from OpenMetadata server
2. Verify the dataset name matches (openmetadata)
3. Check Fuseki logs for query errors

Reset Fuseki Data Copy Section

To clear all RDF data and start fresh:

Stop Fuseki

docker compose -f docker/development/docker-compose-fuseki.yml down
Remove volume

docker volume rm openmetadata_fuseki-data
Restart Fuseki

docker compose -f docker/development/docker-compose-fuseki.yml up -d

Full Stack with Docker Script Copy Chapter

For a complete local environment with RDF enabled (server running in Docker, not IntelliJ):

./docker/run_local_docker_rdf.sh -m ui -d mysql -f true

Options:
- -m ui|no-ui - Include UI or not
- -d mysql|postgresql - Database type
- -f true|false - Start Fuseki for RDF support
- -s true|false - Skip Maven build
- -x true|false - Enable JVM debug on port 5005

Related Files Copy Chapter

- Docker Compose: docker/development/docker-compose-fuseki.yml
- Server Config: conf/openmetadata.yaml
- RDF Java Code: openmetadata-service/src/main/java/org/openmetadata/service/rdf/
- Ontology: openmetadata-spec/src/main/resources/rdf/ontology/openmetadata.ttl
- RDF Index App: openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/RdfIndexApp.java

---

Rdf Production Setup Copy Section

Setting up Apache Jena Fuseki efficiently

OpenMetadata stores its RDF knowledge graph in a remote Apache Jena Fuseki dataset. Production sizing must account for two different memory consumers: the Fuseki JVM heap and the operating-system page cache used by TDB2's memory-mapped indexes. Giving the JVM all container memory starves the page cache and usually reduces throughput.

Capacity planning Copy Chapter

TDB2 storage varies with URI and literal width. A useful planning range for OpenMetadata graphs is 150-250 bytes per triple before compaction headroom.

| Live triples | Approximate live TDB2 data | Fuseki heap | Suggested total RAM | Suggested persistent disk |
| ---: | ---: | ---: | ---: | ---: |
| 1 million | 0.15-0.25 GB | 2 GB | 4-8 GB | 2 GB minimum |
| 10 million | 1.5-2.5 GB | 4 GB | 8-16 GB | 8 GB minimum |
| 50 million | 7.5-12.5 GB | 8 GB | 24-48 GB | 40 GB minimum |

Persistent storage should be at least 2-3 times the expected live dataset, plus room for the journal. TDB2 compaction builds a replacement dataset before deleting the old one, so a volume sized only for live data can run out of space during compaction. The 10 GiB PVC in docker/rdf-store/kubernetes/fuseki-deployment.yaml is a development default, not a production recommendation.

Keep the Fuseki heap smaller than the container memory limit. Memory beyond -Xmx is useful: TDB2 memory-maps its indexes and relies heavily on the OS page cache. The OpenMetadata server does not need additional heap for normal RDF queries because built-in lineage and semantic expansion execute as property-path queries inside Fuseki.

Write throughput Copy Chapter

TDB2 is a single-writer store. More OpenMetadata indexing threads can prepare and read batches concurrently, but Fuseki ultimately serializes write transactions. Throughput therefore depends primarily on the number and size of SPARQL UPDATE transactions, not the number of HTTP clients.

An RDF indexing run with recreateIndex: true clears the graph first and then uses insert-only writes. It does not perform per-entity or per-relationship reconciliation against the empty graph. Entity, relationship, and lineage data are grouped into bounded updates. Incremental runs continue to reconcile existing values.

RDF indexing hydrates every field supported by each entity repository because the RDF mapper preserves fields even when they have no explicit JSON-LD context mapping. It omits only changeDescription, votes, and the embedded testCaseResult, which the mapper intentionally does not emit as triples.

Start with the defaults and tune one setting at a time:

| Environment variable | Default | Purpose |
| --- | ---: | --- |
| RDF_BULK_ENTITY_BATCH_SIZE | 100 | Entity models per SPARQL update. |
| RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE | 100 | Source entities reconciled per relationship update. |
| RDF_BULK_LINEAGE_EDGE_BATCH_SIZE | 50 | Detailed lineage edges per update. |
| RDF_REQUEST_TIMEOUT_MS | 60000 | Maximum time for one RDF request. |

Larger batches reduce transaction and journal overhead but increase request size, parse time, and retry cost. If a larger batch approaches RDF_REQUEST_TIMEOUT_MS, either reduce the batch or raise the timeout. Wide tables can produce tens of megabytes of N-Triples per 100-entity batch, so validate changes against representative catalogs.

Scheduling Copy Chapter

The recommended RDF schedule is a weekly recreate on Saturday at midnight:

0 0   6

Search indexing defaults to Sunday at 00:30:

30 0   0

Keep the jobs separated manually. Both scan the metadata database and hydrate entity relationships, so running them together increases database pressure. OpenMetadata does not provide cross-application mutual exclusion between RDF and search indexing.

Upgrades migrate an RDF app that still has the former exact daily default (0 0 *) to the weekly schedule. Custom schedules and applications with scheduling disabled are not changed.

Compaction and disk growth Copy Chapter

OpenMetadata requests Fuseki compaction after clearing a recreate run and after every successful indexing run. Compaction is best-effort: an indexing run can succeed even if disk reclamation fails.

To compact manually:

curl -u admin:<password> -X POST \
  'http://localhost:3030/$/compact/openmetadata?deleteOld=true'

Fuseki returns an asynchronous task identifier. Inspect active and completed tasks at /$/tasks and confirm the data volume has enough space for both the old and replacement datasets. Unexpected journal growth usually indicates failed or skipped compaction, a write-heavy incremental workload, or a volume that filled before compaction completed.

Configuration reference Copy Chapter

The OpenMetadata server reads the following settings from conf/openmetadata.yaml:

| Environment variable | Default |
| --- | --- |
| RDF_ENABLED | false |
| RDF_BASE_URI | https://open-metadata.org/ |
| RDF_STORAGE_TYPE | FUSEKI |
| RDF_ENDPOINT | http://localhost:3030/openmetadata |
| RDF_REMOTE_ENDPOINT | unset (deprecated fallback) |
| RDF_CONNECT_TIMEOUT_MS | 2000 |
| RDF_REQUEST_TIMEOUT_MS | 60000 |
| RDF_WRITE_MAX_RETRIES | 2 |
| RDF_WRITE_RETRY_INITIAL_BACKOFF_MS | 250 |
| RDF_WRITE_RETRY_MAX_BACKOFF_MS | 2000 |
| RDF_BULK_ENTITY_BATCH_SIZE | 100 |
| RDF_BULK_RELATIONSHIP_SOURCE_BATCH_SIZE | 100 |
| RDF_BULK_LINEAGE_EDGE_BATCH_SIZE | 50 |
| RDF_REMOTE_USERNAME | admin |
| RDF_REMOTE_PASSWORD | admin |
| RDF_DATASET | openmetadata |
| RDF_INFERENCE_ENABLED | false |

Use RDF_ENDPOINT for new deployments. RDF_REMOTE_ENDPOINT remains a deprecated fallback for backward compatibility, and RDF_ENDPOINT takes precedence when both are set. Override the development credentials in every production deployment.

Monitoring and failure diagnosis Copy Chapter

Useful Fuseki administration endpoints are:

- /$/ping for liveness and readiness.
- /$/stats for dataset and operation statistics.
- /$/tasks for compaction and other asynchronous administration work.

Monitor indexing records per second, SPARQL update latency, container RSS, page-cache availability, persistent-volume usage, and journal growth. OpenMetadata logs RDF circuit breaker is open after repeated connection failures or request timeouts; check Fuseki health, request latency, credentials, and network reachability before increasing retries.

Inference Copy Chapter

In-process inference is disabled by default. Enabling RDF_INFERENCE_ENABLED=true causes general queries to construct the entire graph in the OpenMetadata JVM and should be limited to small graphs. The SPARQL console retains its explicit inference option for administrative experiments, but it has the same small-graph constraint.

Use SPARQL 1.1 property paths for transitive lineage, inverse ownership, and domain or glossary inheritance. These execute inside Fuseki without copying the graph into the OpenMetadata process. Deployments that require persistent RDFS or OWL semantics should configure a Fuseki assembler dataset with an appropriate Jena reasoner and size it independently; server-side materialization or reasoning is operationally safer than per-request full-graph inference in OpenMetadata.

The complete-lineage endpoint intentionally does not impose a row limit on its property-path query. Size RDF_REQUEST_TIMEOUT_MS, Fuseki resources, and client response handling for the largest lineage graph operators can request. Semantic search is not an exhaustive traversal: each seed expands at most 100 related graph candidates before reranking to the caller's requested result limit. Use the complete-lineage endpoint or direct SPARQL for exhaustive graph traversal.

For local startup and API examples, see RDF/Apache Jena Local Development Guide.

---

Session Management Multi Node Design Copy Section

Multi-Node Session and WebSocket Session Management Design

1. Status Copy Chapter

This document describes the current server-side session and websocket session design for
OpenMetadata issue #21971.

The implementation is centered on these files:

- openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionService.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStore.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/session/JdbcSessionStore.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/session/RedisSessionStore.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStoreFactory.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/JwtFilter.java
- openmetadata-service/src/main/java/org/openmetadata/service/security/jwt/JWTTokenGenerator.java
- openmetadata-service/src/main/java/org/openmetadata/service/socket/SocketAddressFilter.java
- openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java
- openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java
- openmetadata-service/src/main/java/org/openmetadata/service/cache/CacheBundle.java

2. Problem Copy Chapter

Server-managed login state used to depend on pod-local servlet state. That breaks in a multi-node
deployment because login, callback, refresh, logout, and websocket reconnects can land on different
pods.

The key failure modes are:

- A login or OIDC/SAML callback starts on one node and completes on another.
- Refresh state is unavailable when the request is routed to a different node.
- Logout or session revocation is not visible to all nodes.
- A websocket can remain connected after the browser session is revoked.
- A secure websocket handshake can be spoofed if the server trusts a client-supplied userId.

3. Goals Copy Chapter

1. Store server-managed user sessions in a shared, authoritative backend.
2. Support both JDBC-backed sessions and Redis-backed sessions.
3. Bind newly issued browser access JWTs to the server-side session that issued them.
4. Keep provider refresh tokens and OpenMetadata refresh tokens server-side.
5. Make refresh safe under cross-node concurrency.
6. Make logout and revocation visible to API and websocket paths.
7. Reject websocket handshakes whose token principal, cookie session, or requested socket user do not match.
8. Close websocket connections for revoked or expired sessions.

4. Non-Goals Copy Chapter

1. Changing personal access token or bot token semantics.
2. Replacing the browser-managed public OIDC flow.
3. Making legacy JWTs without a sessionId claim retroactively session-bound.
4. Guaranteeing instant cross-node websocket disconnect without pub/sub. Without pub/sub, remote
sockets are closed by periodic validation.

5. Architecture Copy Chapter

5.1 Component Model Copy Section

| Component | Responsibility |
| --- | --- |
| SessionService | Creates, activates, refreshes, revokes, expires, and prunes sessions. Owns the Caffeine near-cache and revocation listeners. |
| SessionStore | Shared persistence contract used by both JDBC and Redis stores. |
| JdbcSessionStore | Default store backed by the user_session table through SessionRepository. |
| RedisSessionStore | Optional Redis store with key TTLs, per-user status indexes, and Lua compare-and-set on version plus index maintenance. |
| SessionStoreFactory | Selects Redis when Redis cache is configured and available; otherwise uses JDBC. Refuses Redis-to-JDBC fallback when Redis is configured but unavailable. |
| SessionCookieUtil | Reads, writes, validates, and clears the opaque OM_SESSION cookie. |
| JWTTokenGenerator | Issues OpenMetadata JWTs and can include the sessionId claim for session-backed auth flows. |
| JwtFilter | Validates JWTs. If the JWT has sessionId, it reloads the session from the store and requires an active, unexpired, username-matching session. |
| SocketAddressFilter | Validates websocket handshake identity and session state before Socket.IO sees the connection. |
| WebSocketManager | Tracks sockets by user and by session, sends events, and disconnects revoked or inactive session sockets. |
| OpenMetadataApplication | Wires SessionService into auth handlers, websockets, revocation listeners, and the websocket session validator. |
| CacheBundle | Handles cache invalidation pub/sub. Session invalidation messages also disconnect sockets on remote pods. |

5.2 Storage Selection Copy Section

SessionStoreFactory chooses the store at application startup:

- If cache.provider = redis and Redis is available, sessions use RedisSessionStore.
- If Redis is configured but unavailable, startup fails closed.
- If Redis is not configured, sessions use JdbcSessionStore.

The system does not fail over live from Redis to JDBC. Mixing stores would split active sessions
across backends and make revocation unpredictable.

5.3 Session Cache Copy Section

Each pod keeps a local Caffeine near-cache:

- maximum size: 10_000
- expire after access: 10s

The cache is a performance optimization, not a correctness boundary. Security-sensitive checks use
fresh reloads where revocation must be observed immediately.

6. User Session Management Copy Chapter

6.1 Session ID Copy Section

UserSession.id is an opaque bearer secret carried in the OM_SESSION cookie.

It is generated by SessionIdGenerator from secure random bytes and base64url encoded without
padding. It is not a UUID.

6.2 Session Types Copy Section

Current session types:

- AUTH: browser or interactive user auth session.
- MCP: reserved for future interactive MCP session support.

6.3 Session Status Copy Section

SessionStatus values:

- PENDING: login started, callback not completed.
- ACTIVE: usable session.
- REFRESHING: one node holds the refresh lease.
- REVOKED: logout or session-limit revocation.
- EXPIRED: timeout reached.

6.4 Session Fields Copy Section

The important logical fields are:

{
  "id": "opaque-session-id",
  "type": "AUTH",
  "provider": "openmetadata",
  "status": "ACTIVE",
  "userId": "uuid",
  "username": "alice",
  "email": "[email protected]",
  "omRefreshToken": "fernet:encrypted-token",
  "providerRefreshToken": "fernet:encrypted-provider-token",
  "redirectUri": "https://ui.example.com/callback",
  "state": "oidc-state",
  "nonce": "oidc-nonce",
  "pkceVerifier": "pkce-verifier",
  "version": 7,
  "refreshLeaseUntil": 1741300000000,
  "createdAt": 1741200000000,
  "updatedAt": 1741200005000,
  "lastAccessedAt": 1741200005000,
  "expiresAt": 1743792000000,
  "idleExpiresAt": 1741804800000
}

Refresh tokens are encrypted before persistence with Fernet.encryptIfApplies(...). If the Fernet
key is not configured, session creation fails instead of writing plaintext refresh tokens.

6.5 Session Creation Copy Section

Basic, LDAP, and OpenMetadata login create an ACTIVE session directly:

1. Validate credentials.
2. Resolve the provisioned OpenMetadata user.
3. Persist or receive the OpenMetadata refresh token.
4. Create an ACTIVE AUTH session.
5. Encrypt and store the refresh token in the session.
6. Write the OM_SESSION cookie.
7. Return an OpenMetadata-signed JWT with a sessionId claim.

If user lookup or session creation fails after a refresh token is created, the refresh token is
deleted.

6.6 Pending Session Activation Copy Section

SAML and confidential OIDC use pending sessions:

1. Login creates a PENDING AUTH session containing redirect state, OIDC state, nonce, and PKCE
verifier when applicable.
2. The callback loads the pending session from the shared store.
3. The user is created or updated.
4. The OpenMetadata refresh token is inserted.
5. activatePendingSession expires the pending session.
6. A brand-new active session ID is generated and stored.
7. The active session cookie replaces the pending cookie.
8. The browser receives an OpenMetadata-signed JWT with the active session ID.

Issuing a new active session ID during activation is the session fixation defense. The pre-auth
cookie value is never reused for the authenticated session.

If activation fails, the newly inserted refresh token is deleted and no JWT is issued.

6.7 Refresh Copy Section

Refresh is guarded by an optimistic lease:

1. Load the session from OM_SESSION.
2. Reject missing, expired, pending, revoked, or already expired sessions.
3. If another node holds a non-stale REFRESHING lease, return retry guidance through
SessionRefreshInProgressException.
4. Acquire the lease by writing REFRESHING, setting refreshLeaseUntil, and incrementing
version with compare-and-set.
5. The winning node decrypts the stored refresh token.
6. The provider or OpenMetadata refresh token is rotated as needed.
7. completeRefresh writes the refreshed session back to ACTIVE, clears the lease, updates idle
expiry without extending beyond the absolute session expiry, and increments version.
8. The response contains a new OpenMetadata-signed JWT bound to the same session ID.

Lease duration is currently 15s.

6.8 Logout and Revocation Copy Section

Logout calls SessionService.revokeSession(request, response):

1. Read OM_SESSION.
2. Reload the session from the authoritative store.
3. Write REVOKED with compare-and-set.
4. Clear refreshLeaseUntil.
5. Clear the OM_SESSION cookie.
6. Notify local revocation listeners.

Session limit enforcement also uses revokeSession for least-recently-used active sessions.

The limit is configured by authenticationConfiguration.maxActiveSessionsPerUser, exposed through
AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER in openmetadata.yaml. The default is 5; values
below 1 fall back to the default.

6.9 Expiration and Cleanup Copy Section

SessionService runs cleanup every 15m:

- mark expired sessions as EXPIRED
- prune REVOKED and EXPIRED rows after 7d
- process in bounded batches

For Redis, primary keys have TTLs and cleanup methods are no-ops. Session correctness still relies
on in-process status and expiry checks.

Default timeouts:

- pending session timeout: 10m
- authenticated session expiry: authenticationConfiguration.sessionExpiry, default 7d
- refresh lease: 15s
- cleanup retention: 7d

The OM_SESSION cookie max age is rewritten during refresh lease acquisition and is capped at the
remaining effective session lifetime.

7. Session-Bound JWTs Copy Chapter

Server-managed auth flows return OpenMetadata-signed access JWTs with:

{
  "sub": "alice",
  "tokenType": "OM_USER",
  "sessionId": "opaque-session-id"
}

JwtFilter handles the claim as follows:

1. Validate the JWT signature, expiry, token type, principal, and token-specific rules.
2. If there is no sessionId claim, preserve existing stateless behavior.
3. If sessionId exists, call SessionService.getFreshSessionById(sessionId).
4. Require:
- session exists
- status is ACTIVE
- session is not expired
- session username matches the JWT principal
5. Reject the token when any check fails.

This means session-backed browser API requests now consult the shared session store. That is an
intentional tradeoff in the current implementation: revocation is observed on the next request
instead of waiting for access-token expiry. PATs, bot tokens, and legacy JWTs without sessionId
remain stateless.

8. WebSocket Session Management Copy Chapter

8.1 Handshake Validation Copy Section

SocketAddressFilter runs before the Socket.IO server receives the connection.

When secure websocket connections are enabled:

1. Extract and validate the Authorization header.
2. Resolve the token principal from JWT claims.
3. Resolve the principal's user UUID server-side.
4. Reject the request if the query userId is present and does not match the resolved user UUID.
5. Inject the server-resolved UserId header for WebSocketManager.
6. If the JWT has sessionId, inject a SessionId header.
7. Validate OM_SESSION when present:
- reload the session fresh
- require ACTIVE
- require not expired
- require session username to match the token principal
- require cookie session ID to match token sessionId when both are present

If no OM_SESSION cookie is present:

- session-bound JWTs are accepted because JwtFilter has already validated the session ID
against SessionService
- legacy secure JWTs without sessionId are rejected with 401 Session is required
- non-secure websocket mode remains compatible with existing query-based behavior

The filter no longer forwards trust from the user-supplied userId query parameter when secure
mode is enabled.

8.2 Socket Tracking Copy Section

WebSocketManager maintains two local maps per pod:

- activityFeedEndpoints: userId -> socketId -> SocketIoSocket
- socketSessionIds: socketId -> sessionId

On connection:

1. Read UserId from the injected header, falling back to query only for legacy/non-secure paths.
2. Read SessionId from the injected header, falling back to query sessionId only for legacy
paths.
3. Store the socket in the user's local socket map.
4. Store the socket-to-session mapping when a session ID is available.

On disconnect, both maps are cleaned up.

The connection log records user and remote address only. It does not log initial headers, so bearer
tokens are not written to logs.

8.3 Revocation-Driven Disconnect Copy Section

SessionService exposes revocation listeners. OpenMetadataApplication registers a listener that:

1. Converts the revoked session's userId to UUID.
2. Calls WebSocketManager.disconnectForSession(userId, sessionId) on the local pod.
3. Publishes a "session" invalidation message through cache invalidation pub/sub when available.

CacheBundle handles remote "session" invalidation messages:

- if the message has a session ID, call disconnectForSession(userId, sessionId)
- if no session ID is present, fall back to disconnectAllForUser(userId) for backward
compatibility

This gives targeted disconnects. Logging out one browser session does not force-close other
sessions for the same user.

8.4 Periodic WebSocket Validation Copy Section

OpenMetadataApplication.WebSocketSessionValidator runs every 60s by default. Operators can tune
the interval with the openmetadata.websocketSessionValidationIntervalSeconds system property or
the WEBSOCKET_SESSION_VALIDATION_INTERVAL_SECONDS environment variable. Values below 15s are
clamped to 15s.

Each run calls WebSocketManager.disconnectInactiveSessions(sessionService, intervalMillis), which:

1. Iterates local sockets with known sessionId.
2. Reloads a socket's session fresh through SessionService.getFreshSessionById only when that
socket's revalidation interval is due.
3. Disconnects sockets whose session is missing, not ACTIVE, expired, or owned by a different
user.

This is the fallback when there is no cross-pod pub/sub. With JDBC and no pub/sub, a socket on a
remote node is closed within the validator interval instead of immediately.

9. End-to-End Flow Copy Chapter

/ Detailed source-code truncated for AI context efficiency. /

10. Consistency Model Copy Chapter

10.1 API Requests Copy Section

For tokens with sessionId, the session store is authoritative. A revoked or expired session is
rejected on the next API request that uses that token.

For tokens without sessionId, existing stateless behavior is preserved.

10.2 Refresh Copy Section

Refresh uses optimistic compare-and-set on version, so only one node can hold the refresh lease
for a session at a time.

JDBC implements this through the session repository update path. Redis implements it with a Lua CAS
script over the stored session JSON. The Redis script also removes the session ID from all
non-terminal per-user status indexes and adds it to the target non-terminal index before returning,
so the JSON write and index movement succeed or fail together.

10.3 WebSockets Copy Section

Websocket consistency has two layers:

- event-driven disconnect through local revocation listeners and optional cache invalidation pub/sub
- polling-based validation with a 60s default interval and 15s minimum

The event path is immediate when revocation occurs on the same pod or pub/sub delivers the remote
event. The polling path bounds staleness when pub/sub is unavailable, and each socket is fresh
loaded at most once per validation interval.

11. Operational Characteristics Copy Chapter

| Path | Store behavior |
| --- | --- |
| Login | create active or pending session |
| OIDC/SAML callback | fresh load pending session, expire pending session, create active session |
| Session-bound API request | fresh load session by sessionId |
| Refresh | load session, acquire CAS lease, complete CAS update |
| Logout | fresh load session, CAS revoke, clear cookie |
| WebSocket handshake | validate JWT, optionally fresh load cookie session |
| WebSocket validator | throttled fresh load for each tracked socket with sessionId |

Redis deployments should monitor Redis availability as auth-critical infrastructure. When Redis is
configured for sessions, the service refuses to start without it.

12. Security Properties Copy Chapter

1. OM_SESSION is opaque and high entropy.
2. OM_SESSION is written as an HTTP-only cookie.
3. Provider refresh tokens and OpenMetadata refresh tokens are encrypted at rest.
4. Refresh tokens are not returned to the browser by server-managed auth flows.
5. Pending-session activation issues a brand-new active session ID.
6. Session-bound JWTs are invalid once the backing session is revoked, expired, deleted, or owned by
a different user.
7. Secure websocket mode derives socket user identity from the JWT principal, not from query params.
8. Websocket logs do not include initial headers or bearer tokens.
9. Revocation targets the revoked session instead of disconnecting every socket for the user.

13. Test Coverage Copy Chapter

Relevant unit coverage includes:

- SessionServiceTest
- SessionCookieUtilTest
- SessionTimeoutResolverTest
- SessionStoreContractTest
- RedisSessionStoreTest
- JwtFilterTest
- BasicAuthServletHandlerTest
- LdapAuthServletHandlerTest
- SamlAuthServletHandlerTest
- AuthenticationCodeFlowHandlerTest
- SocketAddressFilterTest
- WebSocketManagerTest

Relevant integration coverage includes:

- SessionMultiNodeIT
- SessionRedisMultiNodeIT
- SessionMultiNodeCluster

Important scenarios covered or expected from this suite:

- login on one node and refresh/logout on another
- pending OIDC/SAML callback state loaded from shared session storage
- refresh lease contention
- stale cache behavior after revocation
- Redis-backed cross-node sessions
- websocket principal binding
- per-session websocket disconnect
- session-bound JWT rejection for revoked sessions

14. Tradeoff Resolutions Copy Chapter

1. Session-bound browser API requests intentionally reload session state on JWT validation. This is
the chosen correctness boundary: logout and revocation are observed on the next browser API
request instead of waiting for access-token expiry.
2. Tokens without sessionId remain on existing JWT semantics. This preserves PAT, bot, public OIDC,
and rolling-upgrade compatibility. New server-managed auth responses include sessionId.
3. Non-secure websocket mode remains query-param based only for backward compatibility. Production
deployments should keep secure websocket connections enabled so SocketAddressFilter derives the
socket user from the JWT principal and records the session ID.
4. The active-session cap is now configurable with
authenticationConfiguration.maxActiveSessionsPerUser and
AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER; the default remains 5.
5. Secure, session-managed websocket handshakes now record a session ID from the JWT claim or
OM_SESSION cookie. The validator checks those sockets on a configurable interval with a 60s
default and 15s minimum; sockets without session IDs are legacy/non-secure compatibility cases.
6. Cross-pod websocket revocation has two paths: cache invalidation pub/sub for immediate targeted
disconnects when available, and the configurable websocket validator as the bounded-staleness
fallback for JDBC-only deployments.

---

Streamable Logs Copy Section

Streamable Ingestion Logs

This document describes the end-to-end design of OpenMetadata's streamable ingestion-pipeline log system: how logs flow from a running connector to durable S3 storage, how the UI reads them while a run is in progress, and how the system handles long idle gaps, restarts, and abandoned runs.

Overview Copy Chapter

Ingestion pipelines (metadata, profiler, lineage, usage, dbt, etc.) emit logs as they run. Operators need to:

- Watch logs live while a pipeline is running, including for long-running connectors that can take hours.
- Read logs after the run ends, with a single canonical artifact per run.
- Recover gracefully from server restarts, network blips, and connector idle gaps.

OpenMetadata addresses this with a server-side log storage abstraction backed by S3 (or any S3-compatible store like MinIO). The connector pushes log batches over HTTP; the server persists them and serves both live and post-run reads.

Architecture Copy Chapter

┌──────────────────────┐
│ Python ingestion     │  POST /logs/{fqn}/{runId}        (append)
│ connector            │  POST /logs/{fqn}/{runId}/close  (finalize)
│ (logs_mixin.py)      │
└──────────┬───────────┘
           │ HTTP
           ▼
┌──────────────────────┐
│ OpenMetadata server  │
│ IngestionPipeline    │
│ Resource             │
└──────────┬───────────┘
           │ LogStorageInterface
           ▼
┌──────────────────────┐         ┌──────────────────────┐
│ S3LogStorage         │────────▶│ S3 / MinIO bucket    │
│ (streaming, in-mem   │         │  partial.txt         │
│  buffers, sweeper)   │         │  logs.txt            │
└──────────┬───────────┘         └──────────────────────┘
           │ SSE / GET (paginated / download)
           ▼
┌──────────────────────┐
│ OpenMetadata UI      │
│ (live tail + history)│
└──────────────────────┘

The LogStorageInterface abstraction supports multiple backends:

| Backend | Purpose |
|---------|---------|
| S3LogStorage | Production: stores logs durably in S3 / MinIO. The focus of this document. |
| DefaultLogStorage | Backward-compat: delegates to the pipeline service client (Airflow / Argo). No first-class storage. |

This document covers the S3LogStorage implementation.

Storage Layout Copy Chapter

Each pipeline run is identified by a (fqn, runId) tuple. On S3 the layout is:

{bucket}/{prefix}/                          # prefix defaults to "pipeline-logs"
  {sanitizedFQN}/{runId}/
    partial.txt                             # readable view during the run
    logs.txt                                # final artifact, materialized at /close
  .active/{sanitizedFQN}/{runId}/{serverId} # heartbeat marker

partial.txt is the durable, readable view of an in-progress run. It is updated periodically as the connector appends batches. It carries durable offset state in S3 user-defined metadata:

| Metadata key | Purpose |
|--------------|---------|
| x-amz-meta-last-flushed-line | Logical line counter at the moment of this PUT. Drives retry idempotency and post-restart recovery. |
| x-amz-meta-total-bytes | Cross-check on body size; helps detect drift. |
| x-amz-meta-writer-epoch | Bumped each time a fresh OM-server instance picks up the stream after a restart. |
| x-amz-meta-writer-version | Identifies the writer code version. Useful during migration windows. |

logs.txt is the canonical post-run artifact. It is created only at /close (or by the abandoned-run sweeper), as a server-side S3 copy of the final partial.txt. Content matches partial.txt exactly at the moment of close.

.active/... markers are dropped as a side effect of appendLogs. They have no functional role in correctness; they are operational hints for diagnostics ("which OM-server instance most recently saw this run").

A bucket lifecycle policy ensures cleanup:
- expirationDays (default 30) on the pipeline-logs/ prefix expires all logs after the retention window.

Run Lifecycle Copy Chapter

1. Connector emits a batch Copy Section

The Python ingestion runner buffers log lines and POSTs batches to the server:

POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId}
Content-Type: application/json
"<raw log content>"          OR
{
  "logs": "<base64-gzipped log content>",
  "connectorId": "...",
  "compressed": true
}

IngestionPipelineResource.writePipelineLogs decodes the body and calls repository.appendLogs(fqn, runId, content), which delegates to S3LogStorage.appendLogs.

2. Server-side append Copy Section

S3LogStorage.appendLogs does five things, all in memory, all under a per-stream ReentrantLock:

1. Increments totalLinesAppended, the monotonic logical line counter that anchors retry idempotency.
2. Appends to SimpleLogBuffer (in-memory ring, capacity 1000 lines). This is the source for the SSE/WebSocket live-tail UI experience. It is bounded; oldest lines evict on overflow. It is not load-bearing for durability.
3. Appends to pendingFlush (in-memory queue, no fixed cap, byte-tracked). This is the durable-pending-write queue and survives until the next successful PUT.
4. Notifies SSE listeners, fanning out the new lines to any open live-tail HTTP connections.
5. Schedules an early flush if pendingFlush exceeds earlyFlushWatermarkBytes (default 5 MB). This protects against memory bloat under bursty writes.

A single-threaded cleanupExecutor schedules the periodic flush, the abandoned-run sweeper, and metrics updates.

3. Periodic flush to partial.txt partial.txt')" title="Copy section prompt for LLMs"> Copy Section

Every partialFlushIntervalMinutes (default 2) and on demand from the early-flush watermark, writePartialLogsForStream runs under the per-stream lock:

1. Snapshot pendingFlush and clear it.
2. If empty, no-op (idle streams cost nothing).
3. GetObject partial.txt → reads Content-Length and metadata from the response headers. On 404, treat as empty.
4. Build new metadata (last-flushed-line, total-bytes, writer-epoch, writer-version).
5. If existing body < 5 MB — read the body, build merged body = existing + \n-joined snapshot, PutObject atomically.
6. If existing body ≥ 5 MB — abort the body stream and concatenate server-side via Multipart Upload: CreateMultipartUpload, UploadPartCopy (existing body as part 1), UploadPart (new content as part 2, the last part has no 5 MB minimum), CompleteMultipartUpload. The merged body never enters JVM heap and is not re-uploaded.
7. On failure, abort any in-flight multipart upload, re-merge the snapshot to the head of pendingFlush, and try again next tick. No data loss.

Because pendingFlush is unbounded by the SimpleLogBuffer cap, no line is ever evicted before being flushed.

4. Live read while running Copy Section

The UI's "live logs" view does two things in parallel:

- HTTP GET /logs/{fqn}/{runId}?after={cursor} for paginated history. The server reads partial.txt from S3 and concatenates the in-memory pendingFlush snapshot for the most-recent-tail bytes that haven't yet been flushed. The cursor is a line offset.
- Server-Sent Events (SSE) for live tail. The endpoint registers a LogStreamListener against the stream key and pushes new lines as notifyListeners fires from each appendLogs.

This gives the user "everything written so far" via GET and "everything written in real time from now on" via SSE.

5. /close finalization /close finalization')" title="Copy section prompt for LLMs"> Copy Section

When the connector terminates (success, graceful failure, or graceful abort), it calls:

POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId}/close

S3LogStorage.closeStream runs under the per-stream lock:

1. Final flush: drain remaining pendingFlush to partial.txt (same path as the periodic flush).
2. Server-side copy partial.txt → logs.txt. Bytes do not transit through OM. Cheap and constant-time regardless of log size.
3. Delete partial.txt.
4. Best-effort delete the .active/{fqn}/{runId}/{serverId} marker.
5. Drop in-memory state for the stream (activeStreams, pendingFlush, totalLinesAppended, recentLogsCache, the per-stream lock).

/close is idempotent. A second call finds no partial.txt and no in-memory state; it is a graceful no-op. A /close that arrives after the abandoned-run sweeper already finalized the stream behaves the same way.

6. Post-/close reads /close reads')" title="Copy section prompt for LLMs"> Copy Section

Once /close completes, logs.txt is the canonical artifact. getLogs(fqn, runId) reads it directly. Pagination is by line offset; the response includes after (next cursor) and total (total bytes / lines).

There is also a download endpoint that streams the full file (or composes from segments / partial in legacy fallbacks).

Read Paths Copy Chapter

| Endpoint | Pre-/close | Post-/close |
|----------|-------------|---------------|
| GET /logs/{fqn}/{runId} | Reads partial.txt + appends pendingFlush snapshot. Apply cursor pagination. | Reads logs.txt. |
| GET /logs/{fqn}/{runId}/download | Streams partial.txt. | Streams logs.txt. |
| GET /logs/{fqn}/stream/{runId} (SSE) | Live tail with resume cursors and an explicit end-of-stream event. One shared reader per run. | Streams the finished log, then closes with reason: runFinished. |
| GET /logs/{fqn}/stream/{runId} (SSE) | Same engine, but each frame is one raw log line with no cursor. Legacy shape. | Same. |

The SSE read path is documented on its own in
ingestion-log-streaming.md — event schema, resume cursors, and the
limits that bound it.

Legacy partial.txt files written by older code (without S3 metadata) read normally; the new flush logic treats them as "no prior offset" and merges any new content correctly.

Abandoned-Run Recovery Copy Chapter

Connectors can die without calling /close — process killed, OOM, network partition, infrastructure failure. To bound resource use and still produce a final logs.txt, a sweeper runs periodically:

- Schedule: every cleanupIntervalMinutes (default 60).
- Threshold: streamTimeoutMinutes since last appendLogs (default 1440 = 24h).

For each expired stream, the sweeper does the same finalization steps as /close (final flush, copy to logs.txt, delete partial.txt, drop in-memory state). The end result is identical: an abandoned run produces a finalized logs.txt artifact that the UI can read, just delayed.

The 24h default is intentionally lenient: typical idle gaps in slow connectors (waiting on source queries, batch boundaries, queues) are minutes-to-hours, not days. Operators can tune the threshold downward in deployments where memory pressure from many parallel runs requires more aggressive reclamation.

Failure Modes & Recovery Copy Chapter

| Failure | Recovery |
|---------|----------|
| S3 PUT fails during periodic flush | pendingFlush snapshot is restored under the lock. Next tick retries. No data loss. |
| OM-server restart mid-run | All in-memory state lost. partial.txt on S3 retains all previously-flushed content. The next appendLogs re-creates state; the first flush after restart reads partial.txt (with metadata) and resumes from last-flushed-line. Worst-case loss: lines that were in pendingFlush at restart time, bounded above by partialFlushIntervalMinutes. |
| Connector dies without /close | Abandoned-run sweeper finalizes the run after streamTimeoutHours. logs.txt is materialized from the most recent partial.txt. |
| /close retries after partial success | All steps are idempotent. Second call finds no partial.txt and no in-memory state; no-op. |
| Concurrent appendLogs and cleanup | The per-stream lock serializes them. Cleanup finds the stream "fresh" again and skips it next tick. |
| Bucket lifecycle expires partial.txt mid-run | Should not happen at default expirationDays = 30. If misconfigured (very low retention), the next flush would treat it as a fresh partial.txt and start over. Recommended floor: 7 days. |

Configuration Copy Chapter

All settings live under LogStorageConfiguration in openmetadata.yaml:

| Field | Default | Description |
|-------|---------|-------------|
| bucketName | (required) | S3 bucket for log storage. |
| prefix | pipeline-logs | Key prefix within the bucket. |
| enableServerSideEncryption | true | Apply SSE on every PUT. |
| sseAlgorithm | AES_256 | Or AWS_KMS (requires kmsKeyId). |
| storageClass | STANDARD_IA | S3 storage class for log objects. |
| expirationDays | 30 | Bucket lifecycle: expire all logs after this many days. |
| streamTimeoutMinutes | 1440 | Idle threshold (in minutes) before the abandoned-run sweeper finalizes a stream. |
| cleanupIntervalMinutes | 60 | How often the sweeper wakes up to check for abandoned streams. |
| partialFlushIntervalMinutes | 2 | Periodic pendingFlush → partial.txt cadence. |
| earlyFlushWatermarkBytes | 5242880 (5 MB) | Triggers an out-of-band flush when pendingFlush exceeds this size. |
| pendingFlushAlertAfterFailures | 10 | Emit an alerting metric after this many consecutive failed flushes for a stream. |
| maxConcurrentStreams | 100 | Bound on in-flight pipeline runs per OM-server instance. |
| awsConfig.* | — | AWS credentials / region / endpoint (also supports IAM role + custom endpoints for MinIO). |

Concurrency Model Copy Chapter

Coordination is a per-stream lock keyed by streamKey = fqn + "/" + runId. The lock is held for the duration of appendLogs, periodic flush, abandoned-run cleanup, and /close. Locks are backed by a Guava Striped<Lock> with a fixed stripe count, so memory does not grow with completed-run accumulation; the same key always maps to the same lock instance, eliminating the acquire-vs-remove race that a per-key map would have. False contention across stripes is bounded by maxConcurrentStreams << stripe count.

A single-threaded ScheduledExecutorService (cleanupExecutor) drives:
- Periodic flushes (writePartialLogs)
- Abandoned-run sweeper (cleanupAbandonedStreams)
- Metrics updates (updateStreamMetrics)
- One-shot early flushes scheduled by the watermark trigger

Under sustained burst load, scheduled tasks queue on this single thread. This is intentional: it bounds resource use and avoids unbounded thread creation under spikes. If a deployment regularly sees queue backlog, the watermark or flush interval can be tuned.

Observability Copy Chapter

Key metrics exposed by StreamableLogsMetrics:

- om_streamable_logs_log_shipment_* — distribution of append latencies.
- om_streamable_logs_logs_sent / logs_failed — counter of successful and failed appends.
- om_streamable_logs_batch_size — distribution of lines per batch.
- om_streamable_logs_s3_* — distribution of S3 read/write latencies and counters of S3 errors.
- om_streamable_logs_pending_part_uploads — gauge for monitoring queue backlog (legacy, will be retired with multipart removal).
- om_streamable_logs_multipart_uploads — gauge for active multipart uploads (legacy, will be retired).
- om_streamable_logs_pending_flush_bytes — gauge for in-memory pendingFlush size per stream (new).
- om_streamable_logs_consecutive_flush_failures — gauge per stream (new).

Recommended alerts:
- pending_flush_bytes > 50 MB sustained → memory pressure or persistent S3 failures.
- consecutive_flush_failures ≥ 10 → S3 connectivity or auth issue.
- s3_errors rate > 1/min → S3 health degradation.

Multi-Server Topology Copy Chapter

The design assumes single-writer-per-run: an ALB / load balancer enforces sticky sessions for (fqn, runId) via the PIPELINE_SESSION cookie set on the first appendLogs response. All subsequent requests for the same run land on the same OM-server instance for the lifetime of the run.

If stickiness is broken (cookie stripped by a proxy, multi-cluster routing without coordination), two OM-server instances could write to the same partial.txt and clobber each other. This is out of scope for the current design. A future iteration could move offset state to the database for cross-server coordination.

References Copy Chapter

- Source files:
- openmetadata-service/src/main/java/org/openmetadata/service/logstorage/S3LogStorage.java
- openmetadata-service/src/main/java/org/openmetadata/service/logstorage/LogStorageFactory.java
- openmetadata-spec/src/main/java/org/openmetadata/service/logstorage/LogStorageInterface.java
- openmetadata-service/src/main/java/org/openmetadata/service/resources/services/ingestionpipelines/IngestionPipelineResource.java
- ingestion/src/metadata/utils/streamable_logger.py
- ingestion/src/metadata/ingestion/ometa/mixins/logs_mixin.py
- Related PRs: #23590, #24198, #24287, #24410

---

Tech Debt Copy Section

Tech Debt Ledger

Findings from a one-time repository audit worth fixing. Ordered by impact ÷ size (biggest payoff
per unit of effort first) — so small fixes that unblock a golden principle or close a hazard rank above
large architectural refactors.

Agent? = can an agent fix it unsupervised (mechanical, verifiable) vs. supervised (needs
judgment/review) vs. no (human/native-speaker/architectural).

Tier 1 — Quick wins (high impact, tiny size) — do first Copy Chapter

| # | Finding | Location | Why it matters | Size | Agent? |
|---|---|---|---|---|---|
| 1 | 1 file with a bare except: | ingestion/.../database/hive/custom_hive_connection.py:181 | The only blocker to gating ruff E722 (Golden Principle #5) | 1 line | unsupervised |
| 2 | 3 hand-written UI files missing the Apache header | LineageTable.interface.ts, mocks/rests/applicationAPI.mock.ts, EntityLineage/.../LineageLayers.interface.ts | Trips the license gate; yarn license-header-fix fixes it | trivial | unsupervised |
| 3 | 21 Java files with wildcard imports | attachments/AzureAssetService.java (3), PersonaResource.java, StoredProcedureResource.java, … | Unblocks making no-wildcard blocking; violations are in files edited this month (will regress) | ~21 files, mechanical | unsupervised |
| 4 | POM hygiene: duplicate deps + declared-order inversion | openmetadata-spec/pom.xml (declares common twice), openmetadata-dist/pom.xml (declares openmetadata-ui twice); <modules> lists spec before its dep common | Minor, but spec→common contradicts the "foundation is first" reading of the reactor list | tiny | unsupervised |
| 5 | Stale MCP namespace in a skill | .claude/skills/playwright-validation/SKILL.md:59,63,67,68 — mcp__playwright__ (real server is mcp__playwright-test__) | The only reference that fails at tool-call time, not just read time | 4 lines | unsupervised |
| 6 | AGENTS.md still carries contradictions corrected in CLAUDE.md | AGENTS.md:12 (Webpack→Vite), :105/:223 (use Ant Design), :13 (Python 3.10-3.12) | CLAUDE.md was fixed; AGENTS.md (Codex's entry doc) still misleads | small doc edit | unsupervised |
| 7 | openmetadata-shaded-deps has no "do not edit" marker | module has only POMs + a build artifact; not in any instruction doc | An agent "upgrading" it breaks the es./os. relocation → cascading compile failures | 1 rule/note | unsupervised |

Tier 2 — Medium (real value, bounded but not trivial) Copy Chapter

| # | Finding | Location | Why it matters | Size | Agent? |
|---|---|---|---|---|---|
| 8 | No secret scanner + no key-extension gitignore | CI (.github/workflows), .gitignore | Nothing catches a committed secret; fixture dirs are Snyk-excluded — an agent could commit a real key undetected | medium | supervised (workflow edit needs auth) |
| 9 | Migration append-only is unenforced | bootstrap/sql/migrations/; no CI check, no runtime checksum-abort | Editing an applied migration silently no-ops on existing DBs / drifts on fresh installs, with no error | medium | supervised (workflow edit needs auth) |
| 10 | CodeQL runs on workflow_dispatch only | .github/workflows/codeql.yml | SAST does not run on PRs — a whole class of checks never gates contributions | small config | supervised (workflow edit needs auth) |
| 11 | ~86 broad-excepts that silently swallow | ingestion/src/metadata/ (the ~5.5% with no log/re-raise) | The genuinely risky subset of the (sanctioned) broad-except idiom | ~86 sites, per-site judgment | supervised |
| 12 | mssql ↔ azuresql circular sibling import | database/mssql/connection.py:44 ↔ database/azuresql/connection.py | The one connector import cycle; the rest are clean (timescale→postgres is legit subclassing) | small | supervised |
| 13 | any outside generated/ | 461 files (top-5 are generated) | Erodes type safety; blocked from becoming a blocking lint until generated/ is excluded and re-measured | large but incremental | supervised |
| 14 | basedpyright baseline of 11,927 findings | .basedpyright/baseline.json (926 files) | The ratchet holds (no new errors) but the debt is large; each burn-down shrinks it | very large, incremental | supervised |

Tier 3 — Large / architectural (high impact, large size) — plan, don't rush Copy Chapter

| # | Finding | Location | Why it matters | Size | Agent? |
|---|---|---|---|---|---|
| 15 | Java package cycles: resources ↔ jdbi3 (130/99), 18/21 pairs cyclic | openmetadata-service/src/main/java/org/openmetadata/service/ | The repo's own #1 principle (acyclic layering) is violated inside the core backend; much is misplaced value types (resources.feeds.MessageParser.EntityLink) that belong in a shared package | large refactor | no (architectural) |
| 16 | Frontend components ↔ utils cycle (130-module SCC) | openmetadata-ui/.../src (28 cyclic SCCs, 50 direct 2-cycles) | No import-boundary tooling; barrel/*.interface.ts mutual references + utils importing components | large | no (architectural) |
| 17 | Generated-type leakage: 1,292 direct importers vs 93 in rest/ | openmetadata-ui/.../src/components,pages | No anti-corruption layer; the stricter form of Golden Principle #3 sits at ~7% | large | no (architectural) |
| 18 | Ant Design migration stalled | 864 UI files (18.3%), 68.5% edited ≤90d | The stated "use ui-core-components" direction isn't progressing; can't be a blocking rule until it does | very large, ongoing | no (program of work) |
| 19 | ~250–396 untranslated English strings per non-en locale × 19 | openmetadata-ui/.../src/locale/languages/*.json | Reviewable defect CLAUDE.md flags; CI checks key-sync, not translation | large | no (native speakers) |
| 20 | py-tests required check ≈ 69 min work-median | CI | The dominant PR wall-clock (ranked #1 by cost in the CI run-history analysis); a candidate for sharding/scoping | large CI work | supervised |
| 21 | playwright-postgresql-e2e 82% pass (least-reliable required check) | CI | The required check most likely to fail for non-code reasons; re-run flakiness is ~0-measurable (fixed by new commits), so it hides as a low pass rate | investigation | no |

Conflicts surfaced (not resolved here) Copy Chapter

---

Ui Code Quality Gate Copy Section

UI code-quality gate

How frontend quality is enforced on every openmetadata-ui PR, and what you must configure in
SonarCloud and branch protection to complete it.

The governing rule: new code must be clean; existing debt is fixed gradually. Every gate below is
scoped to what a change adds, never to the whole file or the whole repo, so the backlog can never
block a PR.

The two sides Copy Chapter

Almost all new UI code is AI-generated, so a check that only runs in CI arrives too late — the model
already chose the wrong pattern. Every rule therefore exists twice.

| | Generation-time (agent is writing) | Review-time (CI) |
|---|---|---|
| Knowledge | .claude/rules/*.md, auto-loaded by paths: glob | — |
| Enforcement | .claude/settings.json hooks | ui-checkstyle job |
| Self-serve | make ui-checkstyle-changed | required status check |

One toolchain, three call sites. The same ESLint config runs in the agent hook, in
make ui-checkstyle-changed, and in the ui-checkstyle CI job. Prefer an off-the-shelf ESLint plugin
over a bespoke script: a plugin matches the AST rather than diff text, gives live editor feedback,
and composes with --fix and eslint-disable. Reach for a script only when a rule genuinely cannot
be expressed in ESLint — tw-guard qualifies, because its antd/.less backlog (864 and 449 files)
makes added-lines-only scoping unavoidable.

Depth behind the rules lives in skills/vendor/ — react-best-practices, web-design-guidelines
and composition-patterns, vendored verbatim from
vercel-labs/agent-skills (MIT) so they need no install
step. Skills load only when invoked, which is why the load-bearing subset is distilled into
.claude/rules/frontend-performance.md and .claude/rules/frontend-a11y.md, which auto-load on
matching files.

Checkstyle is the enforcement point — not pre-commit. New gates are deliberately kept out of
.pre-commit-config.yaml: every hook there is paid on every single commit. Committing must stay
fast. ui-checkstyle is the one place a gate has
to hold, and make ui-checkstyle-changed is how you get that answer locally before pushing.

Run it locally Copy Chapter

make ui-checkstyle-changed     # exactly what CI runs, on just your changed files

This is the command to trust — it runs the fixing steps (organize-imports, eslint, prettier, license
headers, i18n sync, app-docs) and the audit gates (tw-audit, tw-guard).
The gates are collected rather than short-circuited, so one failure does not hide the others.

In CI, warnings appear in the sticky GitHub Actions comment on the PR titled
UI Checkstyle passed — lint findings in changed files. The comment groups findings by rule and
includes file, line, column, and message details for the PR's changed files. The same output is
available in Actions → UI Checkstyle → checkstyle → ESLint + Prettier + Organise Imports (src).
Warnings remain non-blocking; ESLint errors and formatting changes still fail ui-checkstyle.

What each gate enforces Copy Chapter

| Gate | Scope | Fails when |
|---|---|---|
| ESLint + Prettier + organize-imports | changed files | output differs from committed form |
| Licence header | changed files | Apache-2.0 header missing/stale |
| i18n key-sync | all locales | locale files out of sync with en-us.json |
| tw-audit | changed files | hardcoded Tailwind value that maps to a design token |
| tw-guard | added lines | new antd import or new .less file |
| jsx-a11y (ESLint) | changed files | one of 19 zero-backlog accessibility rules trips |
| SonarJS (ESLint) | changed files | one of 16 zero-backlog correctness rules trips |
| OpenMetadata performance (ESLint) | changed files | eager route page import, unguarded lazy component, or unbounded module cache |
| OpenMetadata import architecture (ESLint) | changed files | warning-only architecture, cycle, barrel, or request fan-out finding |
| SonarCloud gate | new code | complexity, duplication, or new issues on lines this PR added |

Component reuse — guidance, not a gate Copy Chapter

.claude/rules/component-library.md carries the table of what to import instead of hand-rolling
(Select rather than <div role="listbox">, and so on). No linter knows this design system, so
that table is guidance and review, not an automated check.

A bespoke reuse-audit script was built for this and then removed. It reimplemented, badly, what
ESLint already does well: matching regexes against raw diff lines produced false positives on
data-role=, on [role="menu"] selectors and inside comments, and its hand-rolled git handling
could report "clean" when the diff had failed to load. Its one real advantage — inspecting only
added lines — existed to tolerate a backlog of 16 instances, small enough not to justify ~430
lines of bespoke code and its own test suite.

What CI enforces instead is that a hand-rolled widget must at least be accessible: jsx-a11y
rejects an invalid role, a role missing its required aria-* props, and an unusable tab order.
Using the library component is the easy way to satisfy that.

Two severity tiers, chosen by measurement Copy Chapter

Every rule is on. Severity is decided by the rule's measured backlog, never by taste, because
ESLint reports per file rather than per added line — an error rule with existing violations
would fail PRs for code they merely touched.

| Tier | Meaning | Today |
|---|---|---|
| error | zero measured backlog — blocking | 16 SonarJS + 19 jsx-a11y + 3 OpenMetadata performance |
| warn | has a backlog — visible in the editor and CI output, not blocking | 21 SonarJS, 15 jsx-a11y, 4 React, 10 OpenMetadata import rules, react-hooks/exhaustive-deps, i18next/no-literal-string, @typescript-eslint/no-non-null-assertion |

Repo-wide today: 0 errors, 10120 warnings across 2228 files. The warnings are the backlog, made
visible instead of hidden — the target is zero, reached rule by rule.

i18next/no-literal-string was disabled with a TODO: re-enable when the plugin supports ESLint 9.
That incompatibility no longer reproduces; it runs fine and reports a large backlog, so it is back on
at warn. The repo convention is no user-facing string literals, so it should reach error.

Repository-specific performance rules Copy Chapter

openmetadata-ui/src/main/resources/ui/eslint-rules/openmetadata-performance.mjs contains three
reporting-only rules. Their test suite is run with yarn test:eslint-rules. All three were enabled at
error only after a full src/ scan reached zero findings.

- no-eager-page-imports applies to src/components/AppRouter/ and rejects runtime static imports
whose path contains pages/. Type-only imports remain valid.
- require-suspense-fallback recognizes lazy and React.lazy only when imported from React. It
accepts a component passed directly or subsequently to an approved helper imported from
components/AppRouter/withSuspenseFallback, or a local variable-dependency path from the lazy
binding to JSX rendered or passed beneath a real React Suspense boundary with an explicit
fallback prop. An unrelated boundary elsewhere in the module does not satisfy the rule.
- no-unbounded-module-cache checks module-level Map and Set bindings with cache-like names. A
cache needs an explicit numeric or uppercase named size comparison whose guarded if branch or
while body evicts from the same binding using delete or clear.

The rules intentionally do not autofix because introducing a loading boundary, choosing an eviction
policy, and deciding which route dependency should remain eager require runtime context.

Import architecture and request warnings Copy Chapter

openmetadata-ui/src/main/resources/ui/eslint-rules/openmetadata-imports.mjs contains ten
reporting-only rules. They are enabled at warn, do not autofix, and therefore do not fail CI while
their measured backlog is reduced.

| Rule | What it reports | Baseline findings / files |
|---|---|---:|
| no-impure-pure-utils | React/JSX or upward UI, state, page, hook, or REST dependencies in *PureUtils | 62 / 23 |
| no-lower-layer-page-imports | page imports outside pages and the AppRouter owner | 291 / 271 |
| no-cross-page-imports | one page feature statically importing another page feature | 43 / 32 |
| no-rest-ui-imports | REST clients depending on components, pages, hooks, context, or stores | 55 / 37 |
| no-hook-ui-imports | hooks depending on components or pages | 10 / 6 |
| no-circular-imports | runtime imports/re-exports that participate in a cycle; type-only imports are ignored | 295 / 164 |
| no-internal-barrel-imports | runtime imports resolving to an app-internal index barrel; type-only imports are allowed | 143 / 134 |
| no-lodash-default-import | default or namespace imports from the Lodash package root | 1 / 1 |
| no-api-calls-in-iteration | REST calls inside loops or dynamic iteration callbacks such as map | 28 / 21 |
| review-sequential-api-calls | a second or later directly awaited REST call in one function, for dependency review | 207 / 105 |

The last rule is intentionally phrased as a review: static analysis cannot prove whether the second
request depends on the first. Keep legitimate sequencing; parallelize independent requests. Promote
each deterministic rule to error only after its repo-wide baseline reaches zero. Re-evaluate the
request-review rule separately before making it mandatory.

Rules deliberately still off, and why:

- react/jsx-no-useless-fragment — auto-fixes, so at any severity eslint --fix rewrites files and
hard-fails the git-diff check. Land a one-time repo-wide autofix commit, then add it at error.
- sonarjs/file-header, arrow-function-convention, shorthand-property-grouping,
elseif-without-else and similar — stylistic, and they conflict with Prettier and the repo's
existing conventions. Turning them on would add thousands of warnings nobody intends to fix, which
devalues every other warning.
- sonarjs/no-reference-error, no-implicit-dependencies — need resolver/global configuration this
config does not supply; without it they are almost entirely false positives.

Promotion path: clear a rule's backlog, re-measure, move it to error. The counts live in
eslint.config.mjs next to each rule so the next person can see what it costs.

Before adding any rule to the warn tier, check whether it auto-fixes. ui-checkstyle runs

eslint --fix and fails on the resulting git diff, so an auto-fixing rule at warn would silently

rewrite files and hard-fail the gate. Every current warn rule reports fixable: none or

suggestions-only. react-hooks/exhaustive-deps declares fixable: 'code' but was verified

empirically not to rewrite a dependency array under --fix — which matters twice over, since

auto-adding an effect dependency changes runtime behaviour.

SonarJS in ESLint — the fast half of Sonar Copy Chapter

eslint-plugin-sonarjs is the same engine and the same Sxxxx rule ids as the SonarCloud analysis
that already runs on every UI PR. A finding in your editor is the finding Sonar will report.

The high-backlog SonarJS rules are also enforced blockingly by SonarCloud, whose Clean-as-You-Code
model scopes them to new lines — something ESLint fundamentally cannot express. So
cognitive-complexity and no-duplicate-string warn locally and block on new code in the PR gate.

eslint-plugin-sonarjs is pinned exactly (4.2.0). SonarCloud upgrades its analyzer server-side
on its own schedule and that drift is silent.

The Sonar lens in your editor — SonarQube for IDE Copy Chapter

This is the third place the same rules appear, and the only one that shows the server's profile
verbatim rather than a local approximation.

1. Install SonarQube for IDE (formerly SonarLint) — available for VS Code, IntelliJ, and others.
2. Bind the workspace in Connected Mode to SonarCloud, organization open-metadata, project
open-metadata-ui.
3. Connected Mode pulls the project's quality profile, so the editor flags exactly what the PR gate
will flag — including the high-backlog rules ESLint holds back, marked against new code.

Without Connected Mode the plugin uses its own defaults and will disagree with CI. Bind it, or rely
on make ui-checkstyle-changed.

SonarCloud configuration (admin — must be done once) Copy Chapter

Project open-metadata-ui, organization open-metadata, scanned by .github/workflows/yarn-coverage.yml.

Quality profile Copy Section

Create a custom profile whose active rules mirror the set enabled in eslint.config.mjs, and set rule
parameters explicitly on both sides — do not rely on two defaults agreeing (cognitive-complexity
threshold 15 in both places).

Quality gate — conditions on New Code ONLY Copy Section

Gate name: OpenMetadata UI — Clean as You Code. Set it as the project's default gate.

| Condition (New Code) | Operator | Value |
|---|---|---|
| Coverage | is less than | 90.0% |
| Issues | is greater than | 0 |
| Security Hotspots Reviewed | is less than | 100% |
| Duplicated Lines (%) | is greater than | 3.0% |
| any condition on Overall Code | — | none |

Never attach a condition to Overall Code. It would fail on legacy debt from day one and break the
entire "new code only" contract. Every condition above is evaluated solely against the lines a PR
adds or modifies.

90% coverage on new code is the strictest condition here — above Sonar way's default of 80%,

and the UI currently has no coverage floor at all (jest.config.js sets collectCoverageFrom but

no coverageThreshold). Expect this to be the condition that fails most PRs at first: any new

component, hook or util needs tests landing in the same PR. That is the intent — new code is held

to a standard the backlog is not — but it is a real change in what "done" means for a UI PR, and

teams should hear it before the gate turns on rather than from a red check.

> Two mechanical consequences worth knowing:

- A PR that only moves or reformats code can still register those lines as new and uncovered.

- Coverage comes from sonar.typescript.lcov.reportPaths (src/test/unit/coverage/lcov.info), so

if the Jest run fails or the lcov is missing, new-code coverage reads as 0% and the gate fails.

Fix the test run, not the gate.

New Code definition Copy Section

Reference branch = main for branches; Previous version (or 30 days) for main itself. Set in
project settings, not on the gate.

Branch protection Copy Section

Mark these three required on main:

| Required check | Enforces |
|---|---|
| ui-checkstyle | lint (incl. SonarJS + jsx-a11y), prettier, licence, i18n, tw-audit, tw-guard |
| ui-coverage | Jest run completed |
| ui-sonar-gate | the Clean-as-You-Code quality gate, incl. 90% coverage on new code |

Mark ui-sonar-gate, not SonarCloud's own check. The scan is gated behind
dorny/paths-filter and the safe to test label, so a PR with no UI changes never produces that
check and would block forever waiting on it. ui-sonar-gate always runs and passes when the scan
was legitimately skipped.

The gate result comes from the scanner itself: the PR scan passes -Dsonar.qualitygate.wait=true
(timeout 600s), so SonarCloud decides and the scanner exits non-zero on failure. ui-sonar-gate
turns that outcome into the check contributors see. This is the supported mechanism — do not
reintroduce polling of /api/qualitygates/project_status, which races the asynchronous report
processing and silently passes when it times out.

Two behaviours to expect Copy Chapter

Modified lines count as new code. Editing a line in a messy legacy file pulls that line's issues
into gate scope. This is the mechanism that retires debt gradually — you clean what you touch — but
it reads as "the gate failed on code I didn't write". It isn't; the line is in your diff.

New-code attribution needs confirming once. The PR scan passes -Dsonar.scm.disabled=true
(the push scan does not). New code for a PR comes from the sonar.pullrequest.* parameters, so this
is probably fine — but on the first gated PR, check that Sonar's New Code tab shows only the diff
and not whole files. If it shows whole files, drop that flag from the PR scan step.

Tracking the backlog Copy Chapter

The gate is blind to old code by design, so it will never tell you whether debt is shrinking. Track
sqale_index, code_smells and duplicated_lines_density on overall code monthly:

GET https://sonarcloud.io/api/measures/search_history
    ?component=open-metadata-ui&metrics=cognitive_complexity,duplicated_lines_density,code_smells,sqale_index,ncloc

Expectation: flat or falling on a growing ncloc. Two consecutive months climbing is the signal to
schedule targeted cleanup — Clean as You Code only retires debt where people happen to edit.

---