# Technical Documentation: GoogleCloudPlatform/knowledge-catalog > ℹ️ **Provenance:** Multi-Tier Ingestion Β· [CodeWiki Reference](https://codewiki.google/github.com/GoogleCloudPlatform/knowledge-catalog) ### Overview (`README.md`) # Knowledge Catalog [Knowledge Catalog](https://cloud.google.com/products/knowledge-catalog) (formerly Dataplex), is an AI-powered data catalog and metadata management platform. It provides a dynamic knowledge graph of all your data, structured and unstructured, to provide semantics and business context to AI agents This repository features tools, agents, and samples that demonstrate Knowledge Catalog features, and building context management, enrichment and retrieval solutions. ## Getting Started [](https://console.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fknowledge-catalog.git) ## Contributing See the contributing [instructions](CONTRIBUTING.md) to get started contributed. ## License All solutions within this repository are provided under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. Please see [LICENSE](LICENSE.md) for more detailed terms and conditions. ## Disclaimer This repository and its contents are not an official Google product. --- ### Overview (`okf/README.md`) > [!IMPORTANT] > **OKF now lives in its own repository: > [GoogleCloudPlatform/open-knowledge-format](https://github.com/GoogleCloudPlatform/open-knowledge-format).** > > That repository is the canonical home of the specification, the reference > agent, and the sample bundles. Please read the spec, file issues, and open > pull requests there. > > **Stop using the copy under `okf/` in this repository.** It is a frozen > snapshot, no longer maintained, and anything built against it will drift out > of date. # Open Knowledge Format (OKF) ### πŸ“– [Read the Open Knowledge Format v0.2 specification β†’ SPEC.md](SPEC.md) > **This directory is primarily about the [Open Knowledge Format > (OKF)](SPEC.md).** > > OKF is a **universal, vendor-neutral format** for representing knowledge > as plain markdown files with YAML frontmatter. It is **not tied to any > particular agent, framework, model provider, or serving system**. The > goal is simple: > > - **Anyone can produce** OKF β€” humans authoring by hand, agents built on > any framework (Google ADK, LangChain, custom), export pipelines from > existing catalogs (Dataplex, Unity Catalog, Collibra, …), or scripts > walking a database. > - **Anyone can serve and consume** OKF β€” a static file server, a > knowledge-management UI (Obsidian, Notion, MkDocs), an LLM loading > files into context, a search index, or a graph viewer like the one > bundled in this repo. > > The agent below is a **proof of concept** demonstrating *one* way to > produce OKF bundles automatically. The format itself is the > contribution; this agent and the visualizer exist to make the format > tangible at both ends β€” production and consumption. > > **See OKF in practice** β€” three ready-to-browse bundles produced by this > agent, checked into [`bundles/`](bundles/): > > - [`bundles/ga4/`](bundles/ga4/) β€” GA4 e-commerce dataset > ([viz.html](bundles/ga4/viz.html)) > - [`bundles/stackoverflow/`](bundles/stackoverflow/) β€” Stack Overflow > public dataset ([viz.html](bundles/stackoverflow/viz.html)) > - [`bundles/crypto_bitcoin/`](bundles/crypto_bitcoin/) β€” Bitcoin > blocks/transactions ([viz.html](bundles/crypto_bitcoin/viz.html)) > - [`bundles/acme_retail/`](bundles/acme_retail/) β€” Acme Retail > ([viz.html](bundles/acme_retail/viz.html)) ## Why OKF? OKF represents catalog knowledge as plain markdown files with YAML frontmatter, organized in a directory hierarchy. That choice unlocks a few properties that are hard to get from a service-owned metadata store: - **Human- and agent-readable.** No SDK or query language stands between a reader and the content. An engineer can `cat` a concept; an LLM can ingest it verbatim into context. - **Version-controllable out of the box.** Bundles live in git. Pull requests, line-by-line diffs, blame, and review workflows just work β€” knowledge curation becomes a normal software-engineering activity. - **Portable and lock-in free.** A bundle is a directory. Ship it as a tarball, host it in any repo, mount it from any filesystem, or sync it to any system that speaks files. No proprietary API stands between you and your metadata. - **Mixes structured and unstructured data deliberately.** Use frontmatter for the few fields you want to query, filter, or index on (`type`, `resource`, `tags`, `generated`, `status`); use the markdown body for the prose, schemas, and example queries that LLMs and humans actually read. - **Trust, provenance, and freshness are first-class.** v0.2 puts queryable signals in frontmatter β€” where a concept came from (`sources` with per-source credibility signals), who produced and confirmed it (`generated`, `verified`, from which consumers derive a trust tier), and whether it is still current (`status`, `stale_after`) β€” so an agent-maintained corpus stays trustable without any bespoke runtime. - **Minimally opinionated, freely extensible.** A small set of required keys ensures interoperability, but bundles can carry arbitrary extra frontmatter keys and arbitrary body sections without breaking consumers. - **Composes with existing tooling.** Many knowledge tools β€” Notion, Obsidian, MkDocs, Hugo, Jekyll β€” already speak markdown plus YAML frontmatter, so bundles can be browsed, edited, or rendered without custom UI. - **Progressive disclosure built in.** Auto-generated `index.md` files let an agent or human navigate the hierarchy one level at a time instead of loading the entire bundle into context. - **Graph-shaped, not just tree-shaped.** Concepts link to each other via normal markdown links, expressing relationships richer than the parent/child implied by the directory layout. The net effect is that reference agents, consumption agents, and humans collaborate on the same artifacts in the same way they already collaborate on source code. ## Install ``` python3.13 -m venv .venv .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev] ``` ## Credentials - BigQuery: `gcloud auth application-default login` plus a project for billing (`gcloud config set project `). Public datasets are readable, but the caller's project is billed for query bytes. - Gemini: set `GEMINI_API_KEY` (AI Studio) **or** use Vertex AI by setting `GOOGLE_GENAI_USE_VERTEXAI=true`, `GOOGLE_CLOUD_PROJECT=`, and `GOOGLE_CLOUD_LOCATION=`. ## How the reference agent works The reference agent runs in two passes. The **BQ pass** writes one OKF doc per concept the source advertises, using BigQuery metadata alone. The **web pass** then runs the LLM as its own crawler: it receives a list of seed URLs (provided via `--web-seed` or `--web-seed-file`), fetches the seeds via the `fetch_url` tool, and decides which outbound links are worth following based on whether they look like authoritative documentation for the existing concepts. For each page it fetches, the agent chooses to (a) enrich one or more existing concept docs, (b) mint a standalone `references/` doc, or (c) skip. A hard `--web-max-pages` cap and a same-domain allowed-hosts filter (configurable via `--web-allowed-host`) are enforced inside the tool, so the agent cannot overrun. Use `--no-web` to skip the web pass. ## Run Minimum invocation β€” point at a BigQuery dataset and a bundle output directory. Seeds for the web pass are explicit; omit them (or pass `--no-web`) to run BQ-only: ``` .venv/bin/python -m reference_agent enrich \ --source bq \ --dataset . \ --web-seed-file \ --out ./bundles/ ``` Iterate on a single concept by adding `--concept /` (e.g. `--concept tables/events_`); repeatable. ## Samples Each sample pairs a **recipe** (`samples//`, with the seed URLs and exact `enrich` command) with the **produced bundle** (`bundles//`) that the recipe generated. Open the recipe to reproduce; open the bundle to browse the result directly. - **GA4 Google Merchandise Store** β€” public e-commerce dataset, seeded with canonical GA4 BigQuery Export documentation URLs. Β· [recipe](samples/ga4_merch_store/README.md) Β· [bundle](bundles/ga4/) Β· [viz.html](bundles/ga4/viz.html) - **Stack Overflow** β€” public dataset (mirror of the Stack Exchange Data Dump), seeded with the community's canonical schema references. Exercises multi-concept enrichment from cross-cutting docs pages. Β· [recipe](samples/stackoverflow/README.md) Β· [bundle](bundles/stackoverflow/) Β· [viz.html](bundles/stackoverflow/viz.html) - **Bitcoin (crypto)** β€” public dataset (blocks, transactions, inputs, outputs) from the `bitcoin-etl` pipeline. Exercises cross-table foreign-key relationships in prose. Β· [recipe](samples/crypto_bitcoin/README.md) Β· [bundle](bundles/crypto_bitcoin/) Β· [viz.html](bundles/crypto_bitcoin/viz.html) ## Visualize The `visualize` subcommand renders any OKF bundle as a **self-contained interactive HTML file** β€” one file, no backend, no install on the viewing side. Open it in any modern browser, share it as an artifact, host it on a static file server, or commit it next to the bundle (as this repo does). The viewer is itself a proof-of-concept *consumer* of OKF, mirroring the way the reference agent is a proof-of-concept *producer*. OKF bundles can be consumed by anything that reads markdown; this is just one shape. ### What it shows - A **force-directed graph** of every concept in the bundle, with colored nodes by type (datasets, tables, references, …) and directed edges drawn from each cross-link in the markdown bodies. - A **detail panel** for the selected concept showing its frontmatter (description, resource link, tags) and its rendered markdown body β€” with internal `[…](/path/to/concept.md)` links rewired to navigate within the viewer instead of following the path. - A **"Cited by" backlinks** list under each concept (computed from the reverse of the link graph). - A **search box** (matches title, concept id, and tags), a **type filter**, and switchable graph layouts (cose / concentric / breadth-first / circle / grid). ### Generate ``` .venv/bin/python -m reference_agent visualize --bundle ./bundles/ ``` That writes `bundles//viz.html`. Flags: | Flag | Default | Description | |----------------|------------------------|---------------------------------------------| | `--bundle` | *(required)* | Bundle root directory. | | `--out` | `/viz.html` | Output HTML path. | | `--name` | bundle directory name | Display name shown in the viewer header. | Example, writing the output somewhere else and overriding the header: ``` .venv/bin/python -m reference_agent visualize \ --bundle ./bundles/crypto_bitcoin \ --out /tmp/btc.html \ --name "Bitcoin OKF" ``` ### How it's built The HTML embeds the bundle as a JSON blob and uses [Cytoscape.js](https://js.cytoscape.org/) for the graph and [marked](https://marked.js.org/) for in-browser markdown rendering, both loaded from a CDN. No data leaves the page; the bundle is parsed once at generation time and serialized into the file. ## Tests ``` .venv/bin/pytest ``` --- ### Overview (`samples/README.md`) # Samples The samples demonstrate the use of Knowledge Catalog to manage metadata and context to power agents. ## Discovery Demonstrates building a search and discovery agent building on top of the Search APIs offered by the catalog. ## Enrichment Demonstrates an enrichment agent that can generate and enrich documentation for assets managed in the catalog. ## Governance Agent Demonstrates agentic data governance that automates metadata management for assets managed in the catalog, propagating descriptions, glossary terms, policy tags and data quality scores using column-level lineage and Dataset Insights. ## Policy-as-Code Agent Demonstrates an agent that lets users define and validate data governance policies in natural language, translating them into executable code that runs against metadata in the catalog and BigQuery. --- ### Overview (`toolbox/README.md`) # Toolbox Contains tools to work with Knowledge Catalog metadata. * [Metadata as Code](./mdcode/README.md) Provides the ability to manage metadata in the form of source code artifacts that can be sync'd with metadata in Knowledge Catalog. * [Enrichment Agent](./enrichment/README.md) Provides an ready-to-use agent and customizable harness to produce, evolve/improve and maintain metadata within Knowledge Catalog and make it ready for consumption by agents. --- ### Overview (`samples/discovery/README.md`) # Knowledge Catalog Discovery Agent ## About The Knowledge Catalog Discovery Agent is an AI-powered search assistant for discovering data assets in Google Cloud. While standard semantic search only matches semantically similar text, this agent goes further by performing semantic decomposition of complex questions, generating multiple relevant search queries, and reranking the final results to provide a comprehensive answer. ## Quick Start ### Prerequisites You need a Google Cloud project with the following APIs enabled: * Knowledge Catalog (`dataplex.googleapis.com`) * Vertex AI (`aiplatform.googleapis.com`) * Service Usage API (`serviceusage.googleapis.com`) Ensure you have the following permissions via IAM Roles: * `dataplex.projects.search` - can be obtained via roles like `roles/dataplex.viewer`. * `aiplatform.endpoints.predict` - can be obtained via roles like `roles/aiplatform.user`. * `serviceusage.services.use` - needed to use the current project as the quota project. Can be obtained via roles like `roles/serviceusage.serviceUsageConsumer`. ### Setup Clone the Github Repository: ```shell git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git ``` Install the dependencies. It is recommended to create a Python Virtual Environment: ```shell python3 -m venv /tmp/kcsearch source /tmp/kcsearch/bin/activate cd samples/discovery pip3 install -r requirements.txt ``` Set the required environment variables: ```shell # Replace with your consumer project ID. export GOOGLE_CLOUD_PROJECT= export GOOGLE_GENAI_USE_VERTEXAI=True ``` ### Run the Agent There are 2 ways to run the Knowledge Catalog Discovery agent: 1. *As the root agent* - in this case, rename the `discovery_agent` variable in `agent.py` to `root_agent` and run it using ADK CLI (steps below). 2. *As a sub-agent* - in this case, import the Discovery Agent and leverage it as an `AgentTool`. You can find examples of this in the official [ADK docs](https://adk.dev/agents/multi-agents/#c-explicit-invocation-agenttool). The recommended parent folder structure in this case is: ``` my_custom_agent/ β”œβ”€β”€ agent.py └── knowledge_catalog_discovery_agent/ β”œβ”€β”€ SKILL.md β”œβ”€β”€ agent.py β”œβ”€β”€ tools.py └── utils.py ``` Irrespective of the path you choose, the agent can be run using the ADK CLI commands below: ```shell adk run path/to/agent/parent/folder ``` Replace `path/to/agent/parent/folder` with the relative or absolute path to the **parent** directory of the one containing the agent's source code. ## References * [Knowledge Catalog Search API](http://docs.cloud.google.com/dataplex/docs/reference/rest/v1/projects.locations/searchEntries) * [ADK Documentation](https://adk.dev/get-started/python/#run-your-agent) * [Predefined IAM Roles](https://docs.cloud.google.com/iam/docs/roles-permissions) --- ### Overview (`samples/enrichment/README.md`) # Knowledge Catalog Enrichment Agent ## Overview Knowledge Catalog manages metadata about data assets (such as BigQuery tables). This metadata powers search, and agents to discover and understand data assets. This sample demonstrates how external sources of information can be used to augment the metadata in Knowledge Catalog, using an agentic approach. The enrichment agent finds information relevant to a data asset, generates documentation, and uses Knowledge Catalog APIs to publish the documentation as enriched metadata. Users can customize this enrichment agent with custom instructions, tools to access sources of information within their organization, and skills to describe the use of these tools. ## Running the Sample ### Setup **Setup your environment** This step clones the repository locally, configures various environment variables, creates a python virtual environment and installs dependencies. ```bash git clone https://github.com/googlecloudplatform/knowledge-catalog cd samples/enrichment ``` Ensure you have the GCloud CLI installed and configured. ```bash export CLOUD_PROJECT= gcloud auth application-default login gcloud config set core/project $CLOUD_PROJECT gcloud auth application-default set-quota-project $CLOUD_PROJECT ``` Setup a python environment ```bash cd src source env.sh --install ``` **Setup sample data** The sample creates a sample BigQuery dataset in your project whose metadata is curated. ``` python3 ../sample/data/create_data.py ``` ### Enrichment Steps You are now ready to run the enrichment workflow. **Download a metadata snapshot** ```bash python3 -m enrichment.download \ --dir ../sample/metadata.initial \ --dataset ${KC_ENRICH_SAMPLE_PROJECT}.kc_enrich_sample_data ``` **Enrich the metadata** ```bash python3 -m enrichment.enrich \ --dir ../sample/metadata.initial \ --output-dir ../sample/metadata.new \ --config-dir ../sample/config ``` **Review the metadata updates** ```bash git diff --no-index \ ../sample/metadata.initial ../sample/metadata.new ``` **Publish the updated metadata** ```bash python3 -m enrichment.publish \ --dir ../sample/metadata.new ``` --- ### Overview (`samples/governance-agent/README.md`) # Governance on Auto-pilot: Agentic Data Governance with Knowledge Catalog This project demonstrates an agentic data governance solution using Google Cloud Knowledge Catalog. It showcases how to automate metadata management - propagate descriptions, glossary terms, policy tags etc. using **column level lineage** and leveraging Knowledge Catalog **Dataset Insights** capabilities. This is only a demonstration and is not part of official product, please review everything before using it for your environments and use-cases. --- ## 🌟 Key Features * **Estate Dashboard**: Scan BigQuery datasets to identify metadata gaps (missing descriptions). * **Recursive Description Propagation**: Automatically fetch descriptions from upstream sources, bridging multi-hop gaps. * **SQL-Based Logic Enrichment**: Extracts BigQuery SQL transformations to generate human-readable descriptions for computed columns. * **AI Business Glossary**: Maps technical columns to business terms using Vertex AI Semantic Similarity and **Unstructured Documents** (PDFs, TXT, MD). * **Prioritized Glossary Propagation**: Automatically propagates glossary terms across tables based on lineage, with strict verification thresholds to ensure accuracy (especially for 1-1 mappings). * **Native Knowledge Catalog Integration**: Persists glossary mappings as native `EntryLinks` visible in the Knowledge Catalog Schema tab. * **Unified UI & CLI**: Manage governance tasks via a Gradio-based web app or a headless CLI. * **Policy Tag Propagation**: Recommends and applies BigQuery policy tags via lineage, with support for "straight pull" detection and an integrated **Access Summary** (Readers & Data Policies). * **Data Trust Center (DQ)**: Derived trust scores for views and tables based on upstream Knowledge Catalog DQ/Profiling results and multi-hop lineage. * **Unstructured Document Processing**: Leverage PDFs, TXT, and Markdown files to influence column descriptions and policy tags using Gemini and RAG. * **Remediation Detection**: Automatically detects SQL transformations (e.g. `COALESCE`, `DISTINCT`) that improve data quality and applies "Trust Bonuses". * **Trust History Persistence**: Tracks 0.0-1.0 trust scores over time in BigQuery for trend analysis. * **CLL API Preview allowlisting required**: Please contact your Google Cloud account team to get access to CLL API --- ## πŸ›  Setup & Installation ### Prerequisites - Python 3.11+ - Google Cloud Project with billing enabled. - APIs Enabled: `dataplex`, `bigquery`, `datacatalog`, `datalineage`, `aiplatform`. ### Installation 1. **Clone & Navigate**: ```bash git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git cd knowledge-catalog/samples/governance-agent ``` 2. **Environment Setup**: ```bash # Install dependencies and sync virtual environment using uv uv sync --group dev ``` 3. **Authentication**: - **CLI/Dev**: Run `gcloud auth application-default login`. - **Gradio App**: Follow [OAUTH_SETUP_GUIDE.md](OAUTH_SETUP_GUIDE.md) to enable "Login with Google". 4. **Project**: Set the Google Cloud project to work in, in your shell or in `.env`: ```bash export GOOGLE_CLOUD_PROJECT= ``` The Steward CLI and UI fall back to your gcloud default project (`gcloud config set project `) when it isn't set, and the CLI also accepts `--project`. The data generation scripts require `GOOGLE_CLOUD_PROJECT`. --- ## πŸš€ Usage Guide ### 1. Agentic Data Steward (UI) The Gradio app provides a visual way to scan and apply metadata changes. ```bash uv run python -m metadata_propagation.ui.gradio_app ``` - **Dashboard**: Run "Scan Dataset" to see health metrics. - **Description Propagation**: Enter a table name to preview and apply lineage-based descriptions. - **Glossary Propagation**: Enter a table name to preview and apply lineage-based glossary terms. - **Policy Tag Propagation**: Propagate sensitive data tags across the lineage chain with automated transformation assessment. - **Trust Center (DQ)**: Analyze derived trust scores for a view or table, showing upstream quality sources and remediation bonuses. - **Settings**: Toggle OAuth/ADC modes for specific user actions. ### 2. 🐳 Deployment (Docker & Cloud Run) For all deployment steps (local Docker and production Cloud Run), please refer to the comprehensive **[Deployment Guide](DEPLOYMENT_GUIDE.md)**. Key sections covered: * **[Local Docker Run](DEPLOYMENT_GUIDE.md#-local-docker-run)**: Building and testing the container locally. * **[Automated Cloud Run Deployment (`deploy.sh`)](DEPLOYMENT_GUIDE.md#method-1-automated-script-deploysh--recommended)**: Automated Artifact Registry and Cloud Run setup. * **[Direct Source Deploy (`gcloud run deploy --source`)](DEPLOYMENT_GUIDE.md#method-2-direct-source-deploy-gcloud-run-deploy---source)**: Build and deploy via Cloud Build without requiring Docker locally. * **[Service Account IAM Roles](DEPLOYMENT_GUIDE.md#2-configure-cloud-run-service-account--iam-roles)**: Required permissions for BigQuery, Dataplex, Lineage, and Vertex AI. * **[Authentication & OAuth Modes](DEPLOYMENT_GUIDE.md#-authentication-modes)**: Running with Service Account ADC (`BYPASS_OAUTH=true`) vs Google OAuth (see also [OAUTH_SETUP_GUIDE.md](OAUTH_SETUP_GUIDE.md)). * **[Production Sizing & Troubleshooting](DEPLOYMENT_GUIDE.md#️-production-sizing--best-practices)**: Sizing recommendations and troubleshooting common deployment errors. ### 3. Steward CLI (Headless) The CLI is designed for automation and quick scans. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 4. Data Integration Scripts - **Generate Data**: `uv run python -m metadata_propagation.data_generation.generate_data` (Creates tables + lineage). - **Run Profiling & DQ**: `uv run python -m metadata_propagation.data_generation.run_dq_scans` (Creates and runs Data Profile and Quality scans, natively published to BigQuery). - **Unified Insights**: Run the Steward CLI's `knowledge-propagate` command (above) without `--apply` to trigger the Knowledge Catalog documentation scan, wait for it, and extract the results for review. ## πŸ“„ Unstructured Document Processing The tool can leverage unstructured documents (PDFs, TXT, MD) to influence data governance metadata when lineage is missing or to supplement it. ### Supported Components * **Column Descriptions**: Extracts business meanings and definitions for columns. Enforces strict grounding to avoid hallucinations. * **Policy Tags (PII)**: Extracts explicit sensitivity labels (like `PII: Y`) from documents and maps them to allowed policy tags in your project. No inference or guessing. * **Business Glossary Terms**: Maps columns to a controlled list of business terms based on explicit mentions in documents. Enforces strict grounding. ### Document Context Modes When using the `--document` flag with the `apply`, `policy-propagate`, or `glossary-recommend` commands, you can specify a `--context-mode` to choose how the document is processed: * **`direct` (Context Injection)**: Reads the full text of the document(s) and appends it directly to the Gemini prompt for each column. Best for small documents. ```bash uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --document docs/small_doc.txt --context-mode direct ``` * **`rag` (In-Memory RAG)**: Chunks the document, generates embeddings, and retrieves the most relevant snippets for each column. Best for large documents to save tokens and improve focus. ```bash uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --document docs/large_doc.pdf --context-mode rag ``` > πŸ’‘ *Note: Vector embeddings are generated remotely using Vertex AI's `text-embedding-004` model and stored locally in-memory for fast similarity search.* * **`datastore` (Managed DataStore)**: Queries a pre-existing Vertex AI Search DataStore. Best for enterprise setups with large document repositories. ```bash uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --context-mode datastore --datastore-id my-datastore-id ``` --- ## βš™οΈ Glossary Cache Configuration In enterprise governance workflows, stewards often scan read-only production datasets (where they lack `bigquery.dataEditor` permissions). To optimize glossary mapping recommendations, the steward needs a cache. The system supports redirecting the business glossary embeddings cache to a dedicated writable dataset: ### 1. Web UI (Gradio) Open the **Global Environment Settings** accordion at the top of the dashboard: - **Glossary Cache Dataset ID**: The dataset where you want the `glossary_embeddings_cache` table to be created (e.g., a separate sandbox or metadata dataset). - **Glossary Cache Table ID**: Custom table name for storing cached embeddings (defaults to `glossary_embeddings_cache`). ### 2. CLI Flags You can pass the cache location explicitly as command line arguments to any command: ```bash uv run python -m metadata_propagation.steward_cli glossary-recommend \ --dataset prod_readonly_dataset \ --table transactions \ --cache-dataset metadata_sandbox_dataset \ --cache-table my_glossary_cache ``` ### 3. Environment Variables You can also set these values in a `.env` file or in your shell: * `GLOSSARY_CACHE_DATASET_ID`: Redirects the cache table to a separate BigQuery dataset. * `GLOSSARY_CACHE_TABLE_ID`: Custom name for the BigQuery cache table. > πŸ’‘ *Note: If BigQuery cache initialization fails due to write permissions, the plugin automatically falls back to a local JSON file cache (`scratch/glossary_embeddings_cache.json`) transparently, ensuring zero disruption to the user experience.* --- ## 🧩 Project Modules | Module | Location | Description | | :--- | :--- | :--- | | **Glossary Plugin** | `agent/plugins/glossary_plugin.py` | Handles Business Glossary mapping using Vertex AI. | | **Lineage Plugin** | `agent/plugins/lineage_plugin.py` | Orchestrates description propagation via Lineage API. | | **Policy Tag Plugin** | `agent/plugins/policy_tag_plugin.py` | Recommends and applies Policy Tags based on lineage and SQL analysis. | | **Similarity Engine** | `agent/plugins/similarity_engine.py` | AI logic for scoring lexical and semantic matches. | | **DQ Plugin** | `agent/plugins/dq_plugin.py` | Interfaces with Knowledge Catalog DQ/Profiling result jobs with BQ fallbacks. | | **DQ Propagation** | `dataplex_integration/dq_propagation.py` | Recursive DQ scoring and remediation detection logic. | | **Traverser** | `dataplex_integration/lineage_propagation.py` | Low-level Graph API logic for traversing dependencies. | | **Enricher** | `dataplex_integration/lineage_propagation.py` | Context-aware SQL transformation analyzer. | --- ## πŸ’‘ Workflow Example 1. **Initialize**: Generate synthetic data and lineage relationships (see [Data Integration Scripts](#4-data-integration-scripts)). 2. **Enrich & Propagate**: Run the unified `knowledge-propagate` command to trigger AI scans and sync metadata. 3. **Tag**: Use the **Glossary Plugin** to map technical columns to the Business Glossary for Knowledge Catalog UI visibility. 4. **Secure**: Use the **Policy Tag Propagation** plugin to sync sensitive data tags and verify access summary (Readers/Masking Rules). 5. **Verify**: Check the **BigQuery Console** (Schema -> Policy Tags) and **Knowledge Catalog Schema** (Business Terms). --- ### Overview (`samples/policy-as-code-agent/README.md`) # Policy-as-Code Agent A generative AI-powered agent designed to automate data governance on Google Cloud. It allows users to define, validate, and enforce data policies using natural language queries, translating them into executable code that runs against **Knowledge Catalog** (formerly Dataplex) and **BigQuery** metadata. The [Agent Starter Pack](https://goo.gle/agent-starter-pack) (ASP) is the **recommended** way to create a new project from this sample: you get deployment options and CI/CD scaffolding. The copy in [knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/samples/policy-as-code-agent) is the upstream source for browsing and contributions. ## Prerequisites * **Python 3.11+** * **[uv](https://docs.astral.sh/uv/getting-started/installation/)** β€” fast Python package manager * **Google Cloud SDK (`gcloud`)** installed and authenticated * **Git** ## Getting Started To explore this sample or contribute, you can run the agent directly from this directory in the repository. 1. **Clone the repository and navigate to this sample**: ```bash git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git cd knowledge-catalog/samples/policy-as-code-agent ``` 2. **Install dependencies**: ```bash uv sync --group dev ``` 3. **Configure environment**: Copy the example configuration file: ```bash cp .env.example .env ``` Open `.env` and fill in your details: * `GOOGLE_CLOUD_PROJECT`: Your Google Cloud Project ID. * `GOOGLE_CLOUD_LOCATION`: (e.g., `us-central1`). * `ENABLE_MEMORY_BANK`: Set to `True` to enable long-term memory (requires Firestore). Set to `False` to run without it. See [Memory Integration](./docs/MEMORY_INTEGRATION.md) for details. * `FIRESTORE_DATABASE`: (Optional) Leave as `(default)` unless using a named database. 4. **Authenticate with Google Cloud**: ```bash gcloud auth application-default login ``` ## Running the Agent From the sample directory (`samples/policy-as-code-agent`): ```bash uv run adk run policy_as_code_agent ``` Or use the web UI: ```bash uv run adk web ``` **Optional:** To enable short-term contextual memory (Agent Engine) for better conversation history: ```bash uv run adk web --memory_service_uri="agentengine://AGENT_ENGINE_ID" ``` Open the printed URL in your browser to chat with the agent. ## Running Tests ```bash uv run pytest ``` Run only fast unit tests: ```bash uv run pytest tests/unit -v ``` Integration tests are marked and may require GCP credentials and a configured `.env`. ## Creating a New Project (Agent Starter Pack) If you want to use this sample as a base to create and deploy a **new, independent project** for production, we recommend using the Agent Starter Pack. This provides deployment options and CI/CD scaffolding. Start from a new directory (replace `my-policy-as-code` with your project name) and run: ```bash uvx agent-starter-pack create my-policy-as-code -a adk@policy-as-code ``` This will scaffold a new project and prompt you for deployment options. --- ## Key Features * **Natural Language Policies**: "All tables in the finance dataset must have a description." * **Hybrid Execution**: Generates Python code on-the-fly for flexibility, but executes it in a sandboxed environment for safety. * **Memory & Learning**: Uses **Firestore** and **Vector Search** to remember valid policies. If you ask a similar question later, it reuses the proven code instead of regenerating it. * **Dual-Mode Operation**: * **Live Mode**: Queries **Knowledge Catalog** in real-time. * **Offline Mode**: Analyzes metadata exports stored in **Google Cloud Storage (GCS)**. * **Compliance Scorecards**: Run a full health check on your data assets with a single command. * **Remediation**: Can suggest specific fixes for identified violations. ## Architecture The agent is built using the **Google Cloud Agent Development Kit (ADK)** and leverages several Google Cloud services: * **Gemini 2.5 Pro**: For complex code generation (converting natural language to Python). * **Gemini 2.5 Flash**: For conversational logic, tool selection, and remediation suggestions. * **Vertex AI Vector Search**: For semantic retrieval of past policies. * **Firestore**: Stores policy definitions, versions, and execution history. * **Knowledge Catalog API** (`dataplex.googleapis.com`): For fetching live metadata. ### Project Structure * `policy_as_code_agent/` * `__init__.py`: Application Default Credentials and Vertex environment defaults for ASP/local runs. * `agent.py`: Entry point and core agent definition. * `memory.py`: Handles Firestore interactions (saving/retrieving policies). * `utils/`: Utility modules for LLM logic, Knowledge Catalog, GCS, and common tools. * `simulation.py`: Sandboxed execution engine for running policy code. * `prompts/`: Markdown templates for LLM instructions. * `tests/`: Unit and integration tests. ## Documentation For deep dives into the implementation, check the `docs/` folder: - [High-Level Architecture](./docs/HIGH_LEVEL_DETAILS.md) - [Low-Level Implementation](./docs/LOW_LEVEL_DETAILS.md) - [Memory Implementation](./docs/MEMORY_IMPLEMENTATION.md) --- ### Overview (`toolbox/enrichment/README.md`) # Enrichment Agent The enrichment agent for Knowledge Catalog provides a customizable agentic workflow for extracting information from various sources to build metadata about data assets, which can then be used as context. ## Usage ### Prerequisites The enrichment agent depends on the [Metadata as Code](../mdcode/README.md) capability. Follow the instructions on that page on using the `kcmd` tool. ### CLI The package provides the `kcenrich` CLI tool. This is distributed as a standalone binary. ```bash # Initialize a new catalog snapshot for a bigquery dataset kcmd init --bigquery-dataset . # Pull the latest catalog snapshot from the Knowledge Catalog service kcmd pull # Run the enrichment tool kcagent enrich --catalog-path . --tools-path tools --prompt-path prompt.md ``` ## Developer Workflow ### Setup ```bash git clone https://github.com/googlecloudplatform/knowledge-catalog cd toolbox/enrichment npm install ``` ### Build ```bash npm run build ``` ### Test ```bash npm run test ``` ### Demo The repository contains a self-contained demo. Running the demo involves creating a BigQuery dataset and a Dataplex EntryGroup within your cloud project. **Initialize Environment** ```bash export DEMO_CLOUD_PROJECT="" ``` **Initialize gcloud** ```bash gcloud auth application-default login gcloud config set project $DEMO_CLOUD_PROJECT gcloud config set compute/region us ``` **Setup demo resources** ```bash bq query --use_legacy_sql=false < catalog.yaml scope: bq-dataset.${DEMO_CLOUD_PROJECT}.demo_ecommerce snapshot: entries: - dataplex-types.global.bigquery-dataset - dataplex-types.global.bigquery-table aspects: - dataplex-types.global.overview EOF ../../mdcode/dist/kcmd pull ``` **Create and populate the tools** ```bash cat < prompt.md Enrich the documentation of the assets using the internal organizational information. Use the following sources: * Fileset source EOF ``` ```bash mkdir tools cat < tools/mcp.json { "mcpServers": { "md-fileset": { "command": "../dist/md-fileset", "args": [ "--dir", "fileset" ] } } } EOF ``` ```bash mkdir -p tools/skills/fileset-source cat < tools/skills/fileset-source/SKILL.md --- name: fileset-source description: > Use the fileset source to find relevant markdown documents and extract information about assets. --- The `md-fileset` mcp server provides the following tools to extract relevant information from a directory hierarchy of markdown files: * **list_fileset_contents** - browse and navigate the directory tree to list the contents of the specified path. The items may be files or sub-directories. * **read_fileset_file** - read the contents of a file in the knowledge base. The entire contents are provided. Extract and summarise the relevant information based on the documentation being generated.. * **search_fileset_content** - searches the knowledge base and returns the matching files, along with matching line numbers and line snippets. This can be used to quickly find matches without having to list and read all files. To work with a fileset effectively create search queries (use simple keyword queries with individual tokens) to find relevant files, and then read the files to find relevant information. If one query does not work, try a few other keywords. EOF ``` **Add Docs** Copy over the individual markdown files from [here](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/1e97103cdbf7e6425113a73304029ddb4f1f3a6b/samples/enrichment/sample/docs) into the `fileset/` directory **Enrich the metadata** ```bash ../dist/kcagent enrich --catalog-path . --tools-path tools --prompt-path prompt.md ``` **Clean up** ```bash bq rm -r ${DEMO_CLOUD_PROJECT}:demo-dataset ``` --- ### Overview (`toolbox/mdcode/README.md`) # Metadata as Code Metadata as Code is a Knowledge Catalog (Dataplex) provides data stewards and data producers and AI agents with a source code artifact-based UX for metadata management and context engineering. Users and agents can author, manage, and enrich metadata artifacts using developer-friendly workflows with version control and CI/CD. It provides a standard metadata format can be used by a variety of tools and agents. More details are in the [docs/concept.md](docs/concept.md). ## Key Features * Intuitive human and agent-friendly representation of metadata as source code in YAML and markdown files. Artifacts are organized in a hierarchical manner mirroring the resource hierarchy of data and metadata assets. * Bi-directional sync between local workspace and catalog service. * Support for both 1st party and 3rd party metadata constructs. * Distributed as a TypeScript and Python libraries, a CLI tool (kcmd), and an MCP server for use in a variety of applications, agents and workflows. ## Metadata Artifacts ### Directory Layout Metadata is organized within a directory representing a resource such as a BigQuery Dataset, Dataplex EntryGroup etc. ``` path/to/root/ β”œβ”€β”€ catalog.yaml # Manifest and config directives └── catalog/ # Contains the metadata snapshot └── / └── .yaml # Entry └── / β”œβ”€β”€ .yaml # Entry with sidecar markdown └── .aspect.md # files ``` ## Catalog Snapshot Files ### Manifest File **catalog/catalog.yaml** ```yaml scope: bq-dataset.prod-data.ecommerce aliases: ca-guidelines: aspect: data-agents-project.global.ca-guidelines ecommerce: aspect: data-agents-project.global.ecommerce snapshot: entries: - dataplex-types.global.bigquery-dataset - dataplex-types.global.bigquery-table - dataplex-types.global.bigquery-view aspects: - dataplex-types.global.overview - dataplex-types.global.descriptions publishing: aspects: - dataplex-types.global.overview - dataplex-types.global.descriptions ``` ## Entry YAML File **catalog/prod-data.ecommerce/products.yaml** ```yaml id: products type: bigquery-table resource: name: projects/prod-data/datasets/ecommerce/tables/products displayName: Products Table description: All products in the catalog labels: env: prod createTime: 2026-04-23T00:44:03Z updateTime: 2026-04-23T00:44:03Z schema: ... contacts: ... ``` ## Entry Sidecar Markdown File **catalog/prod-data.ecommerce/products.overview.md** ```markdown --- userManaged: true links: ... --- [overview.content] ``` ## Usage ### Library You can use the `kcmd` library to programmatically interact with the catalog metadata. ```bash npm install kcmd ``` ```typescript import * as kcmd from 'kcmd'; // Creating a catalog manifest from scratch const manifest = new kcmd.CatalogManifest(...); manifest.save('/path/to/root'); // Loading a catalog snapshot from the filesystem const snapshot = kcmd.CatalogSnapshot.fromPath('/path/to/root'); // Pulling the latest metadata from the Catalog service. const pullResult = await snapshot.pull(); if (pullResult.success) { console.log('Metadata pulled successfully'); } else { console.error('Metadata pull failed:', pullResult.error); } // Pushing the modified metadata to the Catalog service. const pushResult = await snapshot.push(); if (pushResult.success) { console.log('Metadata pushed successfully'); } else { console.error('Metadata push failed:', pushResult.error); } ``` ### CLI The package provides the `kcmd` CLI tool. This is distributed as a standalone binary. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` For the semantic-model scope, `kcmd push` deploys the model as a property graph (`CREATE OR REPLACE PROPERTY GRAPH`) to the backend named by the model's GOOGLE deployment target β€” **BigQuery Graph** or **Spanner Graph**; the same authored model deploys to either, and swapping the target switches backends. Pass `--validate-only` to print the generated DDL without executing it. For a BigQuery target this path additionally reads the target dataset's metadata (`bigquery.datasets.get`) to pin the query job's processing location; the call degrades gracefully when the permission is absent, so a narrower service account still works but falls back to BigQuery's own location inference. Each dataset `source` is the table's GCP resource name, such as `bigquery.googleapis.com/projects/

/datasets//tables/`. (A Spanner target names its tables inside one database, so a `source` is reduced to its final segment, and Spanner Graph has no `MEASURE`, so metrics are dropped with a warning.) `kcmd push` also deploys the model to Knowledge Catalog (entries for the model, its entities and metrics, and `schema-join` links for its relationships). A push has two axes: the binding-profile axis picks how many binding profiles the graph deploys for (`--no-profile` for none, the default binding, `--profile ` for one, or `--all-profiles` for every one), and each profile's `deployment_target` selects its backend; `--no-kc` skips the Knowledge Catalog leg. Use `--print` to dump each destination's generated artifact, and `--force-remove` to delete models left in the entry group that the push no longer includes. See [docs/semantic-model/](docs/semantic-model/README.md) for the full guide, including how re-push reconciles removed entities, metrics, relationships, and models. NOTE: The CLI uses `gcloud` to obtain authentication tokens, so ensure you are authenticated via `gcloud auth application-default login`. #### Environment variables Two optional variables override where `kcmd` talks to the catalog. Both default to production, so you normally leave them unset. | Variable | Effect | |----------|--------| | `DATAPLEX_ENDPOINT` | Base URL for the Knowledge Catalog (Dataplex) API. Defaults to `https://dataplex.googleapis.com`. Set it to reach a non-prod host. | | `KC_TYPE_PROJECT` | Project the built-in `semantic-*` system types are read from on a semantic-model push. Defaults to `dataplex-types`. | ### MCP Server To use the Metadata as Code tools as MCP tools in an agentic system such as the Gemini CLI, add the following to your MCP settings file: ```json { "mcpServers": { "kc-mac": { "command": "kcmd", "args": ["mcp", "--path", "/path/to/root"] } } } ``` The server offers the following tools: | Tool | Description | |------------------|-------------------------------------------------------| | pull | Pull the latest metadata from the Catalog service | | push | Push the modified metadata to the Catalog service | | list-entries | List entries in the catalog snapshot | | lookup-entry | Lookup an entry and its metadata from the snapshot | | modify-entry | Modify an entry and its metadata in the snapshot | NOTE: The server uses `gcloud` to obtain authentication tokens, so ensure you are authenticated via `gcloud auth application-default login`. ## Developer Workflow ### Setup ```bash git clone https://github.com/googlecloudplatform/knowledge-catalog cd toolbox/mdcode npm install ``` ### Build ```bash npm run build ``` ### Test ```bash npm run test ``` --- ### Overview (`okf/samples/crypto_bitcoin/README.md`) # Bitcoin public dataset sample Runs the reference agent against the public `bigquery-public-data.crypto_bitcoin` dataset (blocks, transactions, inputs, outputs β€” produced by the open-source `bitcoin-etl` pipeline) and seeds the web pass with the canonical schema source and the foundational Google Cloud blockchain-on-BigQuery announcement. This sample contrasts with GA4 (single denormalized events table) and Stack Overflow (many independent entities) by exercising a **small set of tightly related fact tables** where each row in `transactions` references rows in `blocks`, `inputs`, and `outputs`. Good for seeing how the agent surfaces cross-table foreign-key relationships in prose. ## Prerequisites - Install the agent (from the repo root): ``` python3.13 -m venv .venv .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev] ``` - BigQuery access: ``` gcloud auth application-default login gcloud config set project ``` Public datasets are readable, but the caller's project is billed for query bytes. The `crypto_bitcoin` tables are very large (`transactions` is ~hundreds of GB) β€” keep `--web-max-pages` modest while iterating and prefer `--concept` for smoke runs. - Gemini credentials β€” either `GEMINI_API_KEY` (AI Studio) **or** Vertex AI (`GOOGLE_GENAI_USE_VERTEXAI=true`, `GOOGLE_CLOUD_PROJECT=`, `GOOGLE_CLOUD_LOCATION=`). ## Run ``` .venv/bin/python -m reference_agent enrich \ --source bq \ --dataset bigquery-public-data.crypto_bitcoin \ --web-seed-file samples/crypto_bitcoin/seeds.txt \ --out ./bundles/crypto_bitcoin ``` To iterate on a single concept, add `--concept tables/transactions`. To skip the web pass, add `--no-web`. To raise or lower the web budget, use `--web-max-pages N` (default 100). ## What you get A bundle under `./bundles/crypto_bitcoin/` with one OKF doc per BQ concept (dataset + each table), augmented and cross-linked with reference docs minted from the seeded blockchain-etl and Google Cloud pages, plus an auto-generated `index.md` at each directory level. --- ### Overview (`okf/samples/ga4_merch_store/README.md`) # GA4 Google Merchandise Store sample Runs the reference agent against the public `bigquery-public-data.ga4_obfuscated_sample_ecommerce` dataset (a GA4 export from the Google Merchandise Store) and seeds the web pass with canonical GA4 BigQuery Export documentation URLs. ## Prerequisites - Install the agent (from the repo root): ``` python3.13 -m venv .venv .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev] ``` - BigQuery access: ``` gcloud auth application-default login gcloud config set project ``` Public datasets are readable, but the caller's project is billed for query bytes. - Gemini credentials β€” either `GEMINI_API_KEY` (AI Studio) **or** Vertex AI (`GOOGLE_GENAI_USE_VERTEXAI=true`, `GOOGLE_CLOUD_PROJECT=`, `GOOGLE_CLOUD_LOCATION=`). ## Run ``` .venv/bin/python -m reference_agent enrich \ --source bq \ --dataset bigquery-public-data.ga4_obfuscated_sample_ecommerce \ --web-seed-file samples/ga4_merch_store/seeds.txt \ --out ./bundles/ga4 ``` To iterate on a single concept, add `--concept tables/events_`. To skip the web pass, add `--no-web`. To raise or lower the web budget, use `--web-max-pages N` (default 100). ## What you get A bundle under `./bundles/ga4/` with one OKF doc per BQ concept (dataset + tables), optionally augmented and cross-linked with reference docs minted from the seeded GA4 documentation pages, plus an auto-generated `index.md` at each directory level. --- ### Overview (`okf/samples/stackoverflow/README.md`) # Stack Overflow public dataset sample Runs the reference agent against the public `bigquery-public-data.stackoverflow` dataset (a mirror of the Stack Exchange Data Dump for Stack Overflow β€” `posts_questions`, `posts_answers`, `users`, `votes`, `comments`, `badges`, `tags`, `post_history`, `post_links`, ...) and seeds the web pass with the canonical schema references maintained by the Stack Exchange community. This sample contrasts with the GA4 sample by exercising **multi-concept enrichment**: a single schema-docs page typically describes several tables (`posts_questions` + `posts_answers` + `users`), so the web agent often updates more than one concept per fetched page. ## Prerequisites - Install the agent (from the repo root): ``` python3.13 -m venv .venv .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev] ``` - BigQuery access: ``` gcloud auth application-default login gcloud config set project ``` Public datasets are readable, but the caller's project is billed for query bytes. The `stackoverflow` tables are large β€” keep `--web-max-pages` modest while iterating. - Gemini credentials β€” either `GEMINI_API_KEY` (AI Studio) **or** Vertex AI (`GOOGLE_GENAI_USE_VERTEXAI=true`, `GOOGLE_CLOUD_PROJECT=`, `GOOGLE_CLOUD_LOCATION=`). ## Run ``` .venv/bin/python -m reference_agent enrich \ --source bq \ --dataset bigquery-public-data.stackoverflow \ --web-seed-file samples/stackoverflow/seeds.txt \ --out ./bundles/stackoverflow ``` To iterate on a single concept, add `--concept tables/posts_questions`. To skip the web pass, add `--no-web`. To raise or lower the web budget, use `--web-max-pages N` (default 100). ## What you get A bundle under `./bundles/stackoverflow/` with one OKF doc per BQ concept (dataset + each table), augmented and cross-linked with reference docs minted from the seeded Stack Exchange schema pages, plus an auto-generated `index.md` at each directory level. --- ### Overview (`toolbox/mdcode/demo/README.md`) # Demo ## Configuration You will need a cloud project with permissions to use BigQuery and Knowledge Catalog APIs. ```bash export DEMO_CLOUD_PROJECT= ``` Ensure that gcloud is installed and configured. ```bash gcloud auth application-default login gcloud config set compute/region us-central1 gcloud config set project $DEMO_CLOUD_PROJECT ``` ## BigQuery Dataset This demo demonstrates working with metadata for BigQuery resources (dataset and table). **Setup** * Creates a BigQuery dataset (`demo_ecommerce`) and a table (`events`) based on BigQuery sample data in your cloud project. * Creates a `catalog.yaml` manifest to specify the local catalog snapshot. ```bash bun setup.ts cat catalog.yaml ``` **Create Metadata Snapshot** * Pull metadata from Knowledge Catalog ```bash ../../dist/kcmd pull ls -R catalog cat catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml ``` **Modify Metadata Snapshot** * Either manually edit the file, or use the following command which adds a dummy overview aspect. ```bash bun update.ts catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml cat catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml ``` **Publish Metadata Snapshot** * Push metadata to Knowledge Catalog ```bash ../../dist/kcmd push ``` **Cleanup** * Deletes the BigQuery resources created for the demo ```bash bun cleanup.ts ``` ## Knowledge Base This demo demonstrates working with a Knowledge Base managed in Knowledge Catalog. **Setup** * Creates a Dataplex EntryGroup (`demo_kb`) and a set of document entries within it. * Creates a `catalog.yaml` manifest to specify the local catalog snapshot. ```bash bun setup.ts cat catalog.yaml ``` **Create Metadata Snapshot** * Pull metadata from Knowledge Catalog ```bash ../../dist/kcmd pull ls -R catalog cat catalog/index.md ``` **Modify Metadata Snapshot** * Either manually edit the file, or use the following command which creates a placeholder demo update to the content of the `index` entry using the `kcmd` (metadata as code) library. ```bash bun update.ts cat catalog/index.md ``` **Publish Metadata Snapshot** * Push metadata to Knowledge Catalog ```bash ../../dist/kcmd push ``` **Cleanup** * Deletes the Dataplex EntryGroup ```bash bun cleanup.ts ``` ## OKF Wiki This demo demonstrates publishing an [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) wiki bundle (a directory of markdown files with YAML frontmatter) into a Knowledge Catalog EntryGroup via the Documents Layout. By default it operates on `okf/bundles/acme_retail`, the canonical in-repo bundle: 17 markdown files covering metrics, computations, policies, skills, and tables, and the one that exercises the full v0.2 signal layer. The Documents Layout maps each `.md` file to an entry whose name is derived from the file path, with the markdown body stored on the `dataplex-types.global.overview` aspect. A second bundle in `okf/catalog/` is a GA4 sample with indexes, references, a dataset, and a table, 14 markdown files in total. It is kept for anyone who was using the old default, and is reachable with `--bundle catalog`. The generic Documents Layout only carries `title`/`description`/`tags` + body. OKF's signal layer has no generic home, so `push.ts`/`pull.ts` translate it into a custom typed `okf` Dataplex aspect (created by `setup.ts`) and back, keeping the on-disk files clean OKF and round-tripping the signal losslessly. Entries are published under a custom `okf-bundle` entry type rather than the built-in `dataplex-types.global.generic`, so a catalog-wide search can pick out OKF documents. A document's OKF `type:` is freeform prose, not a Dataplex type ref, so it is recorded separately as `okf_type` on the `okf` aspect. `okf-bundle` declares no required aspects, because SPEC 8 index files carry no signal layer and requiring the `okf` aspect would reject them. An entry's type is fixed at creation, so a bundle already published under the generic type has to have its entry group deleted and recreated. The aspect models the full OKF v0.2 signal as typed, searchable fields: `type`, `resource`, `generated`, `verified`, `status`, `stale_after`, `sources` (with `author`, `usage_count`, `last_modified`), `usage_window`, and the Attested Computation contract (`runtime`, `parameters`, `computation`, `executor`, `attester`). OKF also permits producer-defined frontmatter keys at any depth, so no enumerated template can ever be complete. Anything the template does not model, whether a top-level key like `not:` or a subfield inside a modeled record like `sources[].license`, is diverted onto a single `extra` field as a JSON list of `[path, value]` pairs and restored on pull. That keeps the round-trip lossless for any conformant bundle. A diverted subfield returns at the end of its record rather than its original position, which is the same presentation normalization pull already applies elsewhere. **Setup** * Creates an empty Dataplex EntryGroup (`okf_demo`). `--entry-group ` names a different one; the name must match Dataplex's own rule (`/^[a-z][a-z0-9_-]{0,61}[a-z0-9]$/`) or setup stops before calling gcloud. * Creates the custom `okf` aspect type from `okf-aspect.json`, or updates it if a previous run of this demo left an older template behind. * Creates the custom `okf-bundle` entry type. * Creates a `.state/catalog.yaml` manifest pointing at the EntryGroup and listing the `okf-bundle` entry type and the `okf` aspect. It sits in a hidden `.state/` directory so that this demo directory never takes on the shape of a canonical snapshot root, which is a `catalog.yaml` beside a `catalog/` tree. `setup.ts` is the only script that takes `--entry-group`. The rest read the EntryGroup back out of the manifest it writes, so they cannot be aimed at an EntryGroup this demo never created. ```bash bun setup.ts cat .state/catalog.yaml ``` **Publish Metadata Snapshot** * Push the bundled markdown to Knowledge Catalog. Without `--bundle` this pushes `okf/bundles/acme_retail`. Entry names mirror the file path (e.g. `metrics/revenue.md` → entry `metrics/revenue`). Every file lands as an `okf-bundle` entry, index files included, with its OKF `type:` on the `okf` aspect. ```bash bun push.ts ``` **Pull Metadata Snapshot** * Pull the snapshot back down as clean OKF. The signal layer is restored from the `okf` aspect. Pull writes to `./pulled/`, a gitignored scratch directory, so it never overwrites the bundle push reads from; compare the two to confirm the round trip. ```bash bun pull.ts diff -r ../../../../okf/bundles/acme_retail pulled ``` **Modify Metadata Snapshot** * Edit any markdown file in the bundle directly. Any frontmatter key and the markdown body can be changed, then push again. ```bash bun push.ts ``` **Run Against Another Bundle** * `push.ts` and `pull.ts` both take `--bundle

`, so the demo can run against a bundle elsewhere in the repo without copying it in, and pull can be aimed somewhere other than `./pulled/`. `--bundle catalog` selects the GA4 sample that used to be the default. ```bash bun push.ts --bundle catalog bun pull.ts --bundle /tmp/acme_pulled ``` * Pull re-emits frontmatter in a canonical key order and block style, so pulling a hand-authored bundle back shows presentation churn in a `diff` even though nothing was lost. The Acme Retail bundle uses flow mappings, so expect churn there; compare parsed frontmatter and body rather than bytes. The GA4 bundle in `catalog/` is already in canonical form, so `bun push.ts --bundle catalog` followed by `bun pull.ts --bundle catalog` leaves that tree byte-identical. * Two things do not make the trip. Only `.md` files are pushed, so bundle attachments such as `acme_retail/attesters/sql_equality.py` stay local even though frontmatter points at them. And in-body markdown links between concepts stay markdown; the only native Knowledge Catalog edges are the parent links derived from the directory structure. **Cleanup** * Deletes the Dataplex EntryGroup named in `.state/catalog.yaml`, printing which one and in which project first. It takes no flags, so the only EntryGroup it can delete is the one `setup.ts` created. The custom `okf` aspect type and `okf-bundle` entry type are left in place: they are scoped to the project and location rather than to this demo, so other OKF bundles in the same project are typed by them too. The commands to remove them manually are printed at the end. ```bash bun cleanup.ts ``` ## Skill This demo generates an [Agent Skill](https://agentskills.io) from a semantic model with `kcmd skills-generate`, installs it into a coding agent, and gives that agent a support request in English. The agent finds the right record and performs a write, and the rules that gate the write are settled by the runtime rather than by the agent. The skill is a folder of Markdown, so any harness that reads the Agent Skill layout takes it as-is β€” the guide runs the same generated bytes through Claude Code and Gemini CLI. It needs a Spanner instance rather than BigQuery, and Vertex AI, because the model states its rules in words and a judge settles them. See [semantic-model/skill/README.md](semantic-model/skill/README.md). --- ### Overview (`toolbox/mdcode/docs/semantic-model/README.md`) # Deploying a semantic model A *semantic model* describes a business logically β€” its entities, the relationships between them, and the metrics computed over them β€” independent of where the data physically lives. You author it in a format based on [Apache Ossie](https://ossie.apache.org/), extended with a first-class deployment target and binding profiles. `kcmd push` deploys one model to two kinds of destination at once: * **Knowledge Catalog, where the model is governed.** It becomes catalog entries β€” one per entity, metric, and the model itself β€” joined by links for its relationships. There it is the single governed definition of the business: access-controlled, searchable, and part of the dynamic knowledge graph that gives AI agents the semantics and business context to work with your data. Governing needs no tables or data, so you can publish a purely logical model before it is bound, or govern the model together with its bindings β€” the catalog serves either. `kcmd pull` returns the model files push stored with these entries. It rebuilds the model from the entries only when no copy of the model file is stored. * **A data store, where the model becomes queryable.** Consumers ask for business concepts β€” `Customer`, `revenue` β€” and get consistent, model-defined answers rather than re-deriving joins and formulas per query. The store can be **analytical**, such as BigQuery for reporting and conversational-analytics agents, or **operational**, such as Spanner for the live state an agent reads before it acts. Which store a model deploys to is set by its [deployment target](#where-a-model-deploys-the-deployment-target); the query mechanics are in [Reference](reference.md#what-gets-created-in-bigquery). Both come from the same source, so a single `push` keeps them in sync and you never author them separately. You bind the one logical model to a store with a **binding profile**. A model can bind to more than one store β€” an analytical warehouse and an operational database, say β€” from a single definition, so `Customer` and `revenue` mean the same thing wherever they are served. See [Binding profiles](profiles.md). This page is the operation reference: author the logical model, govern it in Knowledge Catalog, bind it to a store, then update and pull. For a single runnable example that carries one model through the whole lifecycle, see the [codelab](codelab.md); for the Ossie document format itself, see [ossie.apache.org](https://ossie.apache.org/). ### The rest of the guide | Page | Open it to… | |---|---| | **This page** | look up each deploy operation and its rules | | [Binding profiles](profiles.md) | bind one logical model to several stores | | [Modeling class hierarchies](inheritance.md) | model subtypes with `extends` so a supertype query gathers them | | [Codelab: one semantic ontology, one data journey](codelab.md) | see the whole lifecycle: author, govern, hydrate, query | | [Modeling write operations](actions.md) | declare an action an agent can call, publish it, run it, and derive an agent's tools from the model | | [Generating an Agent Skill](skills.md) | write a model out as a skill folder an AI agent loads by itself | | [Reference](reference.md) | look up a flag, what push creates, validation, or permissions | | [Model specification](model_spec.md) | the normative format: every YAML construct, what's OSI and what's a kcmd extension | | [What push and pull preserve](fidelity.md) | understand why something changed or wasn't recovered | | [Importing an OWL ontology](owl-import.md) | start from an OWL ontology instead of hand-authoring | ## Prerequisites `kcmd` uses your `gcloud` configuration for credentials, project, and location. Before pushing, authenticate and make sure a project and region are set β€” `kcmd` errors out immediately if any of the three is missing: ```bash gcloud auth application-default login gcloud config set project gcloud config set compute/region ``` You also need read/write access to whichever destinations you deploy to β€” see [Permissions](reference.md#permissions). ## 1. Author the logical model Create the local layout. The scope is the Knowledge Catalog entry group the model will be published to, written as `..`: ```bash kcmd init --semantic-model my-project.us-central1.my_model ``` `init` provisions that entry group (idempotent β€” an existing group is fine) and creates its local directory. Author the model at `catalog/EntryGroups//.yaml`. The logical model names the business and nothing physical β€” the entities, their fields, the relationships between them, and the metrics computed over them: ```yaml version: "0.2.0.dev0/google" semantic_model: - name: sales # keep equal to the .yaml filename (pull round-trips to that name) entities: # each entity is one concept - name: orders primary_key: [order_id] fields: - { name: order_id, datatype: Integer } - { name: net_amount, datatype: Decimal } - name: customer primary_key: [customer_id] fields: - { name: customer_id, datatype: Integer } - { name: name, datatype: String } relationships: - name: placed_by from: orders to: customer from_columns: [customer_id] to_columns: [customer_id] metrics: - name: revenue expression: SUM(orders.net_amount) ``` Field and relationship names are the business vocabulary β€” `order_id`, `placed_by` β€” never physical column names; the physical binding comes later. A metric's `expression` may be a bare formula over the logical fields or the fuller per-dialect form. `entities` may also be written `datasets` (the two are interchangeable under the `/google` version). Entities can **extend** other entities (`extends: [Parent]`); push flattens the supertype's fields down and expresses the hierarchy as graph labels, so a query against the supertype gathers every subtype. See [Modeling class hierarchies](inheritance.md) to model one step by step, and [Class hierarchies](reference.md#class-hierarchies-extends--labels) for the rules push enforces. A model can also declare **actions**: model-level write operations, the write-side counterpart to a metric. An action names a business operation, points at the executor that carries it out (an MCP tool, a REST endpoint, a gRPC method, or the write itself as DML), and types each parameter against the ontology: a parameter carries one scalar value, and it either projects a field β€” naming the `concept` and `field` it takes its datatype and its wording from β€” or declares a scalar `type` of its own. The executor is a physical binding, so a [binding profile](profiles.md) can supply a different one per store β€” and DML, which names one store's own tables, belongs with that store's binding. An action also names, in `guards`, the constraints that gate it, and in `affects`, the concepts the call changes β€” an executor in another system is an opaque handle, so its blast radius is declared or it is unknown. Knowledge Catalog is the only system an action is published to, and the only place it is governed; an action that carries its own DML is also the one kind the runtime executes directly. See [Modeling write operations](actions.md). A model can also state **constraints**: named invariants over the ontology that hold for every instance, whatever writes to the model. A constraint is part of what the model asserts, and "an account's balance never goes negative" is as much a fact about an account as the fields beside it. Each one states its rule in words, under `judgment`, with the field names it mentions written model-qualified so they are checked like any other reference β€” which is what lets a rule no arithmetic decides, such as whether a discount is justified by the reason given, be stated at all. A constraint does nothing until an action names it in `guards`; then it is settled before that call runs, by a language model reading the attempted call. A constraint reaches Knowledge Catalog only: it is validated, published there, and read back by `pull`. This model names no table and no store, so it is complete enough to govern in Knowledge Catalog as-is (step 2). Where each entity reads from β€” the store and the columns β€” is a binding you add in step 3. ## 2. Govern it in Knowledge Catalog The logical model is complete enough to govern right now, with no tables, no data, and no store. A model that declares no deployment target has nowhere to deploy in a store, so a bare `kcmd push` writes it to Knowledge Catalog alone: ```bash kcmd push --validate-only --print # preview the entries and links, write nothing kcmd push # write them ``` Push creates one entry per entity and metric, one for the model itself, one per model file and profile file holding a copy of that file, and a `schema-join` link for each relationship: ``` Wrote 5 new and 0 updated Knowledge Catalog entries; linked 1 relationship. ``` Some relationships get no link, and push warns about each one before it writes to Knowledge Catalog. `kcmd push --validate-only` gives the same warnings. Knowledge Catalog keeps one `schema-join` link between two entries, so when several relationships join the same two entities, only the first one that push can publish gets a link. Pull still returns a skipped relationship, because it writes back the stored copy of the model file. Only a model file that pull rebuilds from the catalog metadata leaves it out. Push rejects a relationship from an entity to itself, and relationships in both directions between two entities. Those entries are the single governed definition of the business. They are access-controlled, searchable, and joined into the dynamic knowledge graph that gives AI agents the semantics and business context to reason over your data. `kcmd pull` returns the model file and profile files stored with them. It rebuilds the model document from them only when no copy of the model file is stored (see [Pull](#pull)). Governance works whether the model is purely logical or already bound: govern it before it has any physical home, or govern the logical model together with its bindings. Every store you bind it to later serves this one definition. (Running `--no-kc` on a model with no deployment target is an error β€” it would have nowhere to deploy.) > Writing these entries needs the `semantic-model` / `semantic-entity` / > `semantic-metric` entry types and write access to the entry group. See > [Permissions](reference.md#permissions). ## 3. Bind and deploy to a store To make the model queryable, bind it to a store: name where each entity reads from, which column each field maps to, and which store the model deploys to. You can put these bindings directly on the model, or keep them in a separate **binding profile** so one logical model serves several stores. See [Binding profiles](profiles.md); for a runnable end-to-end that binds the same model to BigQuery and Spanner and queries each, see the [codelab](codelab.md). Once the model is bound, `kcmd push` deploys to the store its target names and to Knowledge Catalog. The most common flags: ```bash kcmd push # deploy to the store (default binding) + Knowledge Catalog kcmd push --no-kc # deploy only to the store, skip Knowledge Catalog kcmd push --no-profile # publish only to Knowledge Catalog, deploy to no store kcmd push --profile analytical # deploy with one binding profile; its target picks the store kcmd push --all-profiles # deploy once per binding profile kcmd push --validate-only # run all checks, write nothing kcmd push --print # also print the generated DDL / entry plan ``` A push has two axes. The **binding-profile axis** sets how many profiles the model deploys for: the default binding, `--no-profile` (none β€” catalog only), a single `--profile`, or every one with `--all-profiles`. The **Knowledge Catalog axis** is `--no-kc` (skip the catalog leg). You never name the store on the command line; the model's deployment target β€” or the [profile](profiles.md) you merge β€” selects it. The checks that gate a push are in [Validation](reference.md#validation), and the full flag list is in [Reference β†’ push](reference.md#push), with the order of push's checks and writes. ### Where a model deploys (the deployment target) A bound model declares exactly one **deployment target**: where in a store the model deploys, written as its GCP resource name. Its host selects the store: ``` # BigQuery β€” an analytical store bigquery.googleapis.com/projects//datasets//propertyGraphs/ # Spanner β€” an operational store spanner.googleapis.com/projects//instances//databases//graphs/ ``` A BigQuery target names the project and dataset the model deploys into; a Spanner target names the instance and database. Swapping one target for the other deploys the same model to the other store β€” the logical model does not change. The target is also recorded on the model's Knowledge Catalog entry. A model with more than one deployment target is rejected at push time (see [Validation](reference.md#validation)). For a model that serves more than one store, the target belongs to a [binding profile](profiles.md) rather than the model. `deployment_target` is a resource name and a native key under the extended `0.2.0.dev0/google` profile; under vanilla Ossie the same target is written inside a `GOOGLE` custom extension instead (see [model spec Β§6](model_spec.md#6-the-extension-mechanism)). ### Table sources Each entity's `source` names its backing table, written as its GCP resource name: `bigquery.googleapis.com/…/tables/` for BigQuery, `spanner.googleapis.com/…/tables/
` for Spanner. A [binding profile](profiles.md) can move an entity between stores by swapping this name. A table in a **federated REST catalog**, such as an Apache Iceberg table exposed through BigLake, is written as its BigLake resource name, `biglake.googleapis.com/projects/

/catalogs//namespaces//tables/`. Validation resolves each name the same way the deploy does (see [Validation](reference.md#validation)), and [profiles.md](profiles.md#sources) lists every accepted store. ### What push creates In **BigQuery**, a single `CREATE OR REPLACE PROPERTY GRAPH` per deployment target: each entity becomes a node table, each relationship an edge table, each metric a measure. In **Spanner**, the same `CREATE OR REPLACE PROPERTY GRAPH` with bare table names and no measures β€” Spanner Graph has no `MEASURE`, so metrics are dropped with a warning while the graph structure (nodes, edges, labels, inheritance) still deploys. In **Knowledge Catalog**, one entry per model, entity, and metric, plus `schema-join` links for relationships. The exact mapping β€” and the class-hierarchy handling β€” is in [Reference β†’ What gets created](reference.md#what-gets-created-in-bigquery) (and [in Spanner](reference.md#what-gets-created-in-spanner)); what of your metadata survives the trip (and what doesn't) is in [What push and pull preserve](fidelity.md). ## Updating and removing models Your model document is the source of truth. To change what is deployed, edit the document and run `kcmd push` again β€” you never edit the catalog or the deployed store by hand. Re-running is safe: each push makes the destinations match the document as it stands now. **When you edit an entity, metric, or relationship** β€” push overwrites the existing one in place; you don't get duplicates. **When you delete an entity or metric** β€” remove it from the document and push. Push deletes the leftover catalog entry for you (the summary shows `removed N orphaned entries`). **When you rename or delete a relationship** β€” push removes orphaned and stale relationship links before writing the current ones, so a renamed or re-pointed relationship can be created without colliding with the old link (the summary shows `unlinked N orphaned links`). Push reconciles only the links of the model it is deploying, and an entry group holds exactly one model, so it never disturbs another model's links. **When you remove a whole model** β€” deleting its document does *not* remove it from the catalog. As long as you still push at least one other model in the same entry group, push stops and names the model the catalog still has that you no longer push, so you can't wipe one out by accident or by pushing from the wrong directory; run push again with `--force-remove` to delete its entries and links. That push deletes the old model entry first, and Dataplex deletes the model's other entries and its links with it. A push that stops partway then leaves the whole old model or none of it. Deleting *every* document is the exception: with no models left to push, the run reports `No semantic model documents found; nothing to deploy` and touches nothing β€” so remove a model while other models in its group remain, or delete its catalog entries directly. Every push prints one line per destination summarizing what it did. For a push that deploys to both the store and Knowledge Catalog: ``` Deployed 1 BigQuery Graph(s). Wrote 5 new and 2 updated Knowledge Catalog entries; removed 1 orphaned entry; linked 2 relationships; unlinked 1 orphaned link. ``` A Spanner-targeting model prints `Deployed 1 Spanner Graph(s).` in place of the BigQuery line. ## Pull `kcmd pull` is the inverse of push's Knowledge Catalog leg: it writes the model file and profile files push stored back to `catalog/EntryGroups//`. When the catalog holds no copy of the model file, it rebuilds `.yaml` from the `semantic-*` entries instead. Use it to recover a workspace from a catalog someone else deployed, or to see what the catalog actually holds. ```bash kcmd pull ``` Pull reads only from Knowledge Catalog (never the data store). Its coordinates come from the same scope you authored under (`..`). The `--dry-run` and `--force-remove` flags are in [Reference β†’ pull flags](reference.md#pull). An entry group holds **exactly one** semantic model. Pull returns that one model's files; a group with more than one `semantic-model` anchor is an unexpected state, so pull stops and names the anchors rather than guess which to keep. Pull overwrites a model that already exists locally in place. If the catalog's model has a **different name** than the one on disk, writing it would leave the entry group holding two models β€” so by default pull stops and reports the mismatch instead of deleting anything. Re-run with `--force-remove` to delete the local model and replace it with the catalog's. Pull never touches the data store. Push also stores a copy of the model file and of each profile file, and pull writes those copies back to the paths they came from, byte for byte. A local profile file the catalog has no copy of would be stored again by the next push, so pull names it in a warning, and `--force-remove` deletes it. Each push marks its copies with one id and the number of copies it stored. It also puts that id in the `kcmd-push-id` label of the model entry, which it writes before the copies. Pull refuses the catalog when the copies come from two pushes, when their number differs from that count, or when the label names a push whose copies the catalog does not hold. Each of these means the last push stopped partway. Pull then writes and deletes nothing and asks you to push again. Pull cannot catch a push that stopped before it changed the model entry. The catalog then still holds what the previous push left, and pull returns that. After a failed push, push again before you pull. A model with no label and no stored copies, such as one an older `kcmd` pushed, is rebuilt from the catalog metadata. Pull warns that the rebuilt file may differ from what was pushed, and keeps every local profile file. Pull also rebuilds the model file when no stored copy is named after the model. That rebuild is **not** an identity β€” see [What push and pull preserve](fidelity.md) for exactly what comes back, what comes back normalized, and what doesn't come back at all. ## Importing an OWL ontology Already have an OWL ontology? `kcmd owl import` converts it into a semantic model once β€” classes β†’ entities, object properties β†’ relationships, datatype properties β†’ fields. The converted model is **unbound** (no sources, no deployment target), so bind each entity's `source`, fill the relationship join columns, and add a deployment target before `kcmd push` will deploy it β€” then it rides the normal push / pull above. See [Importing an OWL ontology](owl-import.md). --- ### Overview (`samples/enrichment/src/tools/fileskb/README.md`) # FilesKB MCP Server This is a simple MCP server for purposes of demonstrating how the enrichment agent can be extended with MCP servers to access existing information sources. The sample surfaces common file system capabilities - listing files, searching files and reading files stored within a specified directory root. ## Using this MCP Server The server can be provided to the agent using standard MCP configuration: ```json { "mcpServers": { "fileskb": { "command": "/usr/local/google/home/nikhilko/p/kc/enrichment2/src/.venv/bin/python3", "args": [ "/usr/local/google/home/nikhilko/p/kc/enrichment2/src/tools/fileskb/main.py", "--dir", "/usr/local/google/home/nikhilko/p/kc/enrichment2/demo/docs" ] } } } ``` --- ### Overview (`toolbox/mdcode/demo/semantic-model/skill/README.md`) # Generate an Agent Skill from a semantic model A semantic model says what a business is, what may be done to it, and under what rules. This demo takes one such model and turns it into an [Agent Skill](https://agentskills.io) β€” a folder of Markdown that a coding agent reads β€” with one command: ```bash kcmd skills-generate ``` Then it installs that folder into a harness and gives the harness a support request in English. The harness finds the right order, tries to fix it, and is either allowed to or is refused, by rules that live in the model rather than in the harness. There is no agent in this directory. Nothing here is written against a particular framework, and nothing here has to be rewritten to move to another one. The skill is a folder of Markdown, and the rules are stated in the model underneath it whatever reads it. One thing to know before you start, because the rest of this page depends on it: **`kcmd` compiles the model and generates the skill, and does not ship an action runner or a rule judge.** Enforcing an action's guards before opening a transaction is the job of the calling agent or a third-party tool framework that wraps the action. Where that matters, this page says so, and the transcripts recorded against the earlier CLI action runner are marked as such. ## What this demo is arguing An agent that can change a business is mostly not code. It is a description of the business, a list of what may be done to it, and the conditions under which each of those things is allowed. Written as an agent, all three end up inside a prompt and a set of hand-written tools β€” which means they are true of that one agent, they drift when the business changes, and the rules can be edited by whoever owns the agent rather than whoever owns the business. Written as a semantic model, all three are declarations. The skill is derived from them. So: - **The skill is generated, not written.** Change the model, regenerate. There is no second copy of the business to keep in step. - **The rules are stated once, in the model.** The skill renders them so the reading agent (or a third-party tool framework wrapping the action) knows what gates the write. Because `kcmd` does not ship an action runner or a judge, nothing in this repository enforces those rules outside the agent: an agent that runs statements directly against the database and ignores the skill's instructions gets past them unless a third-party tool framework mediates the write. - **Any harness will do.** The output is the published Agent Skill layout, so Claude Code, Gemini CLI, Cursor and the other clients that read that layout all take it as-is. ## What you need - `kcmd`, built from this repository: `npm install && npm run build` in `toolbox/mdcode`, which leaves the binary at `dist/kcmd`. Put it on your `PATH`. - A Google Cloud project with Spanner and Vertex AI enabled, and `gcloud auth application-default login` done. - A Spanner instance to create a database on. - A coding agent that reads Agent Skills. The transcripts below are Claude Code and Gemini CLI, both headless; the install table covers the others. Commands below use `my-project` and `my-instance`. Substitute your own, in these commands and in `catalog/EntryGroups/commerce_demo/commerce.profiles/spanner.yaml`. ```bash export PROJECT=my-project export INSTANCE=my-instance export DATABASE=semantic_skill_demo export GOOGLE_CLOUD_PROJECT="$PROJECT" export GOOGLE_CLOUD_LOCATION=us-central1 ``` ## 1. The model `catalog/EntryGroups/commerce_demo/commerce.yaml` is the whole business: a small ecommerce operation with customers, orders, and the lines that make up an order. Read the file β€” it is commented β€” but these are the parts the rest of the demo turns on. **Three entities**, in the names the business uses. `Order`, not `Orders`; `LineItem.lineItemId`, not `line_item_id`. Field descriptions are part of the model, because they are facts about the business rather than notes to a developer: ```yaml - name: Order description: >- A customer order. Its total is the sum of its line items: items, tax and fees add, credits subtract. A positive total is money owed to the company. primary_key: [order_id] fields: - name: orderId datatype: Integer description: >- The order's number, which is how both the customer and the desk refer to it. - {name: customerId, datatype: Integer} - name: placedOn datatype: Date description: The day the order was placed. - name: total datatype: Decimal description: What the customer owes on this order, in dollars. - {name: status, datatype: String, description: OPEN or CLOSED.} ``` **One action.** `IssueCredit` credits a customer against one order. Its first parameter is projected out of the model rather than declared here β€” `order` is `Order.orderId`, so its type and its wording come from the entity and are written down once β€” and `affects` says what the call reaches, for a reader who does not have the SQL: ```yaml - name: IssueCredit description: >- Credit a customer against one order -- a late delivery, a coupon, a shipping charge applied in error. The credit is added as a negative line and the order total is recomputed from the lines. parameters: - {name: order, concept: Order, field: orderId} - {name: amount, type: Decimal} - {name: memo, type: String} affects: - {concept: LineItem, operation: create, fields: [type, amount, memo]} - {concept: Order, operation: modify, fields: [total]} guards: - CreditWithinOrderTotal - CreditUnderReviewThreshold - CreditMemoNamesAServiceFailure - CreditIsNotSplitToAvoidReview ``` **Four rules, written in words.** A constraint carries a `judgment`, which a language model settles against the attempted call. `on_violation` says what happens when it does not hold, and the three values in play here are all different things: ```yaml - name: CreditUnderReviewThreshold judgment: >- The credit amount requested must not exceed 25 dollars, which is the self-service ceiling for this desk. Read the amount as dollars. on_violation: escalate description: >- A credit over $25 is above the self-service ceiling. A supervisor decides it. - name: CreditMemoNamesAServiceFailure judgment: >- The memo argument of this call must name a specific thing that went wrong on the order: a late delivery, a damaged item, a shipping charge applied in error. A memo saying only that the customer asked, or that the credit is goodwill, or giving no reason at all, names no failure and does not satisfy this rule. on_violation: warn description: >- Say in the credit memo what actually went wrong with the order. - name: CreditIsNotSplitToAvoidReview judgment: >- The credit requested must be the whole of what this order is owed, not one piece of a larger amount divided to stay under the 25-dollar self-service ceiling. A memo argument calling the credit a part, a half, an instalment, the first or second of several, or a remainder, or naming a total larger than the amount argument, does not satisfy this rule. on_violation: reject description: >- Raise this as a single credit for the full amount and send it for supervisor review. ``` `escalate` means a person could allow it but nothing automatic can. `warn` lets the write through and reports. `reject` is a flat no. The fourth rule, `CreditWithinOrderTotal`, is about a number that is in the database rather than in the call. A judge is shown the arguments and nothing else, so that one is recorded here and settled by nothing β€” see [Limits](#10-limits). A constraint is inert until something references it. `guards:` on the action is what gives these four effect; a constraint nobody names is documentation. **Nothing above says where the data lives.** No table, no column, no SQL. Those are physical facts and they are in `commerce.profiles/`. ## 2. The binding profiles `commerce.profiles/spanner.yaml` maps the model onto a Spanner database. It may not add an entity, rename a field or change what anything means β€” the loader rejects that β€” so the whole of what is deployment-specific is in this one file: ```yaml - name: Order source: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/semantic_skill_demo/tables/Orders fields: - {name: orderId, expression: order_id} - {name: total, expression: total} ``` and the SQL that carries the action out: ```yaml - name: IssueCredit executor: sql: statements: - >- INSERT INTO LineItem (line_item_id, order_id, type, amount, memo) VALUES (GENERATE_UUID(), @order, 'credit', -@amount, @memo) - >- UPDATE Orders SET total = ( SELECT COALESCE(SUM(li.amount), 0) FROM LineItem AS li WHERE li.order_id = @order ) WHERE order_id = @order ``` The total is recomputed from the lines rather than adjusted by the credit amount, so the order cannot stop adding up however the action is called, and there is no arithmetic for a reading agent to get wrong. `commerce.profiles/alloydb.yaml` is a second profile over the same model: different tables (`purchase_order`, `order_line`), one differently named column (`order_total`), and PostgreSQL in place of GoogleSQL. `commerce.yaml` is the same bytes under both. [Section 8](#9-the-same-skill-against-a-different-database) generates the skill under it and diffs the two. `catalog.yaml` picks which one is in force by default: ```yaml scope: semantic-model.my-project.us-central1.commerce_demo default_profile: spanner ``` ## 3. Create the store The tables exist before the model is pushed; `schema.spanner.sql` holds them. ```bash gcloud spanner databases create "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" \ --ddl-file=schema.spanner.sql ``` Seed it: ```bash gcloud spanner databases execute-sql "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" \ --sql="INSERT INTO Customer (customer_id, name, email) VALUES (1, 'Morgan Ellis', 'morgan.ellis@example.com'), (2, 'Dana Reyes', 'dana.reyes@example.com')" gcloud spanner databases execute-sql "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" \ --sql="INSERT INTO Orders (order_id, customer_id, placed_on, total, status) VALUES (12345, 1, DATE '2026-09-07', NUMERIC '165.85', 'OPEN'), (12346, 1, DATE '2026-08-20', NUMERIC '18.00', 'OPEN'), (12347, 2, DATE '2026-09-02', NUMERIC '200.00', 'OPEN')" gcloud spanner databases execute-sql "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" \ --sql="INSERT INTO LineItem (line_item_id, order_id, type, amount, memo) VALUES ('li-12345-1', 12345, 'item', NUMERIC '89.99', 'Cast iron skillet'), ('li-12345-2', 12345, 'item', NUMERIC '34.50', 'Enamel saucepan'), ('li-12345-3', 12345, 'fee', NUMERIC '30.00', 'Shipping'), ('li-12345-4', 12345, 'tax', NUMERIC '11.36', 'Sales tax'), ('li-12346-1', 12346, 'item', NUMERIC '18.00', 'Silicone spatula set'), ('li-12347-1', 12347, 'item', NUMERIC '200.00', 'Stand mixer')" ``` That leaves order 12345 at $165.85 over four line items β€” including a $30 shipping fee, which is what the first request below is about β€” 12346 at $18.00, and 12347 at $200.00. The two extra orders give a query something to tell 12345 apart from. Every number quoted in this README came from a run against exactly this seed. To start over, drop the database with the command under [Cleaning up](#11-cleaning-up) and create and seed it again. ## 4. Check what the profile binds Before generating the skill, check that the profile binds the model to your store. Run this from this directory, the one holding `catalog.yaml`: ```console $ kcmd profiles Model 'commerce' (commerce_demo): spanner (default) target: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/semantic_skill_demo/graphs/commerce entities: Customer, Order, LineItem actions: IssueCredit (sql) ``` ### What the rules caught, when something was settling them > These four were recorded when `kcmd` shipped a CLI action runner (`kcmd action > run`) that took a judge and could give that judge the store to read. That > command has been removed and no judge reads a store any more β€” the seam for it > was taken out of the runtime. They are pasted as they were rather than > corrected, because a transcript that was never printed is not evidence of > anything. > They are kept because what they show β€” one action, four rules, and every > one of the three `on_violation` outcomes β€” is not shown anywhere else. Without the store, `CreditWithinOrderTotal` is a rule about a number the judge cannot see: ```console $ kcmd action run IssueCredit --judge \ --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal' ("The credit amount requested must not exceed the total of the order it is applied to. ..."), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The order total for order 12346 is not provided, so it's not possible to determine if the credit amount of $3.00 exceeds it. ... ``` Order 12346's total is $18.00 and a $3.00 credit against it is fine. The judge could not say so, because it had a rule about a number and no way to see the number. What it does instead is not predictable: this run said it could not tell, and an earlier run of the same call answered that $3.00 exceeded a total of $2.50 β€” a figure nothing in the database supports. A judge that cannot read either blocks a call it should pass or invents the fact it was missing, and you do not get to choose which. **That is what this rule does today**, and it is why a rule about a stored number belongs in the schema rather than in a judgment. Back when a judge could read, the same call looked like this: ```console $ kcmd action run IssueCredit --judge --judge-reads-store \ --arg order=12346 --arg amount=20.00 --arg memo="Shipping charge applied in error" Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12346 Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The credit amount of 20.00 exceeds the order total of 18.00. Please request a credit amount that does not exceed the order total. The model marks this rule 'escalate', so an approver may allow it; nothing here can. A credit cannot exceed the total of the order it credits. Lower the credit amount, or split it across the orders it actually covers. No transaction was opened, so nothing was written. ``` When a judge could read the store, it wrote its own `SELECT`, from the model, and refused against the real total. What survives the seam's removal is the rest of that message: which rule, the rule's own words, the judge's reason, the consequence the model attaches, the advice the model wrote for this case, and the fact that nothing was written. That whole message is what a reading agent gets back, for every rule a judge can still settle. Handed no judge at all, the runtime fails closed rather than writing unchecked: ```console $ kcmd action run IssueCredit \ --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal', 'CreditUnderReviewThreshold' and 'CreditIsNotSplitToAvoidReview', which are settled by reading the call, and this runtime was given no judge to ask. Running it would apply a write the model says must be checked first, so it is refused rather than run unchecked. ``` That refusal is still the runtime's behaviour for any caller that supplies no judge: rather than applying a write the model says must be checked first, it stops before a transaction opens. And the `reject` consequence, the one an approver cannot wave through: ```console $ kcmd action run IssueCredit --judge --judge-reads-store \ --arg order=12345 --arg amount=15.00 \ --arg memo="First half of the 30 dollar shipping refund; the rest to follow" Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12345 Error: Action 'IssueCredit' is guarded by 'CreditIsNotSplitToAvoidReview' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: Your memo "First half of the 30 dollar shipping refund; the rest to follow" indicates that this credit is only part of a larger amount, which violates the rule. Raise this as a single credit for the full amount and send it for supervisor review. No transaction was opened, so nothing was written. ``` $15 is under every numeric ceiling in the model. What the rule caught is the memo saying what the $15 is for. Every rule is settled before a transaction opens, which is why each of these says nothing was written. A refusal is not a rollback, and no rule ever sees the write it gates. ## 5. Generate the skill ```console $ kcmd skills-generate Wrote skills/commerce/SKILL.md Wrote skills/commerce/references/issue-credit.md ``` Flags: ``` --out

Directory to write the skill directories under; each skill takes a directory of its own, named after the skill. Defaults to `skills` --name Name the skill, and so its directory; lowercase letters, digits and single hyphens. Defaults to the model's own name. Applies only to a scope with one model --profile [name] Read the model under this binding profile, which is what the skill's one deployment-specific section describes; defaults to default_profile, else the inline bindings --force Replace a skill that is already there, including deleting a reference file for an action the model no longer declares ``` ### What came out `SKILL.md` is a router. Its frontmatter is all a harness loads at startup, so it has to be enough to decide whether to read further: ```yaml --- name: "commerce" description: "Customers, their orders, and the lines that make up an order. Declares 1 action: IssueCredit. Use when a request asks to change this data rather than only read it." --- ``` The body opens with the action table, one row per action, each pointing at its own reference page: ```markdown | Action | What it does | Reference | | --- | --- | --- | | `IssueCredit` | Credit a customer against one order -- a late delivery, a coupon, a shipping charge applied in error. The credit is added as a negative line and the order total is recomputed from the lines. | `references/issue-credit.md` | ``` Then **How this model wants to be used**, which is the model's own `ai_context` followed by a paragraph derived from the fact that these are guarded writes β€” what to do with a refusal, what to do with a warning, not to invent identifiers. The model supplies what is specific to this business; the derivation supplies what is true of guarded actions in general, so the model does not have to repeat it. Then **Finding a record**. This is the part worth dwelling on, because a skill of writes has an obvious hole in it: a request names a person and a day, and the action wants a key. The section opens by saying what a wrong key costs: > A key that matches no record costs you the call rather than the data: a > statement that writes no rows fails the action and rolls the whole transaction > back, so nothing is half-applied and nothing is silently skipped. Guessing a > key is therefore safe to be wrong about, and not safe to be right about by > accident. Then the generator emits the read path β€” the `gcloud` line for this store, and the schema to write SQL against: ````markdown To read the store directly: ```bash gcloud spanner databases execute-sql semantic_skill_demo \ --instance=my-instance --project=my-project \ --sql='SELECT ...' ``` Those are GoogleSQL statements. These tables are the whole of what there is to read, and the names to write in a statement are the table and column names below -- not the model's own names, which follow each column for cross-reference: ``` Customer -> table Customer column customer_id (Integer) = Customer.customerId column name (String) = Customer.name. The customer's display name, e.g. "Morgan Ellis". column email (String) = Customer.email Order -> table Orders column order_id (Integer) = Order.orderId. The order's number, which is how both the customer and the desk refer to it. column customer_id (Integer) = Order.customerId column placed_on (Date) = Order.placedOn. The day the order was placed. column total (Decimal) = Order.total. What the customer owes on this order, in dollars. column status (String) = Order.status. OPEN or CLOSED. LineItem -> table LineItem column line_item_id (String) = LineItem.lineItemId column order_id (Integer) = LineItem.orderId column type (String) = LineItem.type. item, tax, fee, or credit. column amount (Decimal) = LineItem.amount column memo (String) = LineItem.memo ``` ```` Both names are there and which is which is spelled out, because a harness has to cross between them: the rest of the skill talks about `Order.total` and a `SELECT` has to say `total`. The field descriptions come along because they carry what a coded column's values actually are β€” `item, tax, fee, or credit` β€” and an agent that guesses filters on a value the column never holds and gets an empty answer back, which reads like the record not existing. Then **Running an action**, the one deployment-specific section, which names the bound store and executor and advises how an external agent framework should wrap the actions as tools (since `kcmd` itself only emits the skill): ```markdown Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `spanner`. - Store: `my-project/my-instance/semantic_skill_demo` - Executor: `sql` An agent that runs continuously should be handed these actions as tools by its own framework, which puts the action's guards to a judge before opening a transaction, and refuses rather than writing unchecked when it cannot settle one the model requires. ``` Then **What happens when you call one** β€” the three states a call comes back in, and the fourth case that is not a state: > A call can also come back applied **and** carry warnings. That means the change > landed and a rule still went unmet, or went unchecked. Report both. Reporting > only the success tells the caller the write met every rule the model states, > which is the one thing it did not. `references/issue-credit.md` is the page for the action: an argument table, the model's `ai_context` for this action, each of the four rules with its `judgment` verbatim, its `on_violation`, and the advice for when it does not hold, and last the `affects` table. The reading agent gets the rules before it calls, in the words that will be used to judge it. ## 6. Install it into a harness The output is the published [Agent Skill](https://agentskills.io) layout: a directory named for the skill, holding `SKILL.md` and `references/`. Installing it means putting that directory where the harness looks. Point `--out` straight at the directory your harness reads: | Harness | Project-scoped | User-scoped | | --- | --- | --- | | Claude Code | `.claude/skills/` | `~/.claude/skills/` | | Gemini CLI | `.agents/skills/` or `.gemini/skills/` | `~/.agents/skills/` or `~/.gemini/skills/` | | Cursor | `.agents/skills/` or `.cursor/skills/` | `~/.agents/skills/` or `~/.cursor/skills/` | `.agents/skills/` is the cross-tool location β€” Gemini CLI and Cursor both read it, and Cursor additionally reads `.claude/skills/` and `.codex/skills/`. agentskills.io lists the other clients that take the same layout. Two of these are run below, against the same generated bytes; the rest of the table is from each tool's documentation. So: ```bash kcmd skills-generate --out .claude/skills --force ``` writes `.claude/skills/commerce/SKILL.md` and `.claude/skills/commerce/references/issue-credit.md`, and Claude Code started in this directory will offer the skill. ## 7. Run it Everything below is a real run: Claude Code, headless, started in this directory, with the skill installed at `.claude/skills/commerce/` and no other instructions. The tools it was allowed are `kcmd`, `gcloud spanner databases execute-sql`, and file reads. It was given no order numbers, no table names and no SQL. The four runs are consecutive against the seed from [section 3](#3-create-the-store), so the state each one starts from is the state the previous one left. > **These four were recorded when the skill carried a > `kcmd action run ... --judge --judge-reads-store` command line, and they are > kept because nothing else shows an agent meeting a rule it cannot talk its way > past.** > That command and those flags have since been removed, and no judge reads a > store any more β€” the seam for it was taken out of the runtime. Read them for > what the agent did with a refusal, not as what the CLI runs today. ### A request that gets refused > Morgan Ellis emailed about the order she placed on Labor Day. She was charged > for shipping and that order was supposed to ship free. Please put it right. No order number, no amount, and the day is a holiday rather than a date. The harness read the skill, read the reference page, and went looking: ```sql SELECT c.customer_id, c.name, c.email, o.order_id, o.placed_on, o.total, o.status FROM Customer c JOIN Orders o ON o.customer_id = c.customer_id WHERE c.name = "Morgan Ellis" ORDER BY o.placed_on ``` ``` customer_id name email order_id placed_on total status 1 Morgan Ellis morgan.ellis@example.com 12346 2026-08-20 18 OPEN 1 Morgan Ellis morgan.ellis@example.com 12345 2026-09-07 165.85 OPEN ``` > Order 12345 was placed 2026-09-07 β€” Labor Day. Let me read its lines. ``` line_item_id type amount memo li-12345-1 item 89.99 Cast iron skillet li-12345-2 item 34.5 Enamel saucepan li-12345-3 fee 30 Shipping li-12345-4 tax 11.36 Sales tax ``` > Shipping was charged $30 on an order that should have shipped free. The full > amount owed is $30 β€” over the desk's $25 self-service ceiling, and the rules > forbid splitting it to get under. I'll raise it as one credit for the full > amount. ```console $ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \ --arg order=12345 --arg amount=30 \ --arg memo='Shipping fee of $30.00 was charged in error on this Labor Day order; the order was supposed to ship free. Refunding the shipping charge in full.' Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12345 Error: Action 'IssueCredit' is guarded by 'CreditUnderReviewThreshold' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The credit amount of 30 dollars exceeds the self-service ceiling of 25 dollars. The model marks this rule 'escalate', so an approver may allow it; nothing here can. A credit over $25 is above the self-service ceiling. A supervisor decides it. No transaction was opened, so nothing was written. ``` Its report: > **Refused β€” this one needs a supervisor. Nothing was written.** > > Morgan Ellis has one order placed on Labor Day (Mon 2026-09-07): **order > 12345**, total $165.85, status OPEN. Its lines confirm the complaint: > `li-12345-3`, type `fee`, **$30.00, memo "Shipping"** […] > > I attempted a single $30.00 credit against order 12345 […]. I can't approve > this myself, so I've stopped there. > > I also did not split it into two smaller credits β€” the amount owed is the whole > $30, and `CreditIsNotSplitToAvoidReview` rejects breaking a single owed amount > into pieces to stay under the ceiling. Splitting would be the wrong fix even > where it would go through. > > **What a supervisor needs to approve:** $30.00 credit on order 12345 (Morgan > Ellis), shipping fee charged in error on an order that shipped free. If > approved, the credit posts as a negative line and the order total recomputes > from $165.85 to $135.85. > > Morgan has a second open order (12346, 2026-08-20, $18.00) β€” untouched, and not > what she wrote about. Seven turns, three shell calls. It resolved a holiday to a date, disambiguated two orders, found the charge, attempted the honest fix, was refused, and reported the refusal in the model's own terms β€” including the distinction between `escalate` and a hard block, which it got from the reference page. ### A request that lands > Dana Reyes says the stand mixer on order 12347 turned up with a dented casing. > She is keeping it but wants something back for it. Twenty dollars seems fair. It confirmed the order and who it belongs to, then: ```console $ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \ --arg order=12347 --arg amount=20 \ --arg memo='Stand mixer arrived with a dented casing; customer is keeping the damaged item and is credited $20 for the damage.' Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12347 Committed at 2026-09-20T19:11:42.335357Z. ``` Then it checked the result: ``` order_id total status line_item_id type amount memo 12347 180 OPEN eba20a14-206f-496d-825f-f9307a4ca14b credit -20 Stand mixer arrived with a dented casing; customer is keeping the damaged item and is credited $20 for the damage. 12347 180 OPEN li-12347-1 item 200 Stand mixer ``` > Done β€” the credit was applied cleanly, no warnings. Order total recomputed from > the lines: **$200.00 β†’ $180.00**. Status remains OPEN. > > The $20 cleared both gates β€” it's within the order total, and under the $25 > self-service ceiling, so no supervisor review was needed. The credit line's key is a UUID from `GENERATE_UUID()` in the profile's `INSERT`. The total went to $180.00 because the second statement recomputed it from the lines. Neither the harness nor the model did any arithmetic. ### A request that lands and carries a warning > Dana Reyes has been a customer for years and I would like to thank her. Put a > five dollar goodwill credit on order 12347. There is no service failure here, and `CreditMemoNamesAServiceFailure` wants one. It is `warn`, so it reports and lets the write through. It read the order to confirm the $5 was within both numeric ceilings, then: ```console $ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \ --arg order=12347 --arg amount=5 \ --arg memo='Goodwill credit thanking Dana Reyes for years of custom. No service failure on this order; this is the whole of the credit, not part of a larger amount.' Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12347 Warning: 'CreditMemoNamesAServiceFailure' (...) is advisory, and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The memo states that there was no service failure on this order, which does not satisfy the rule requiring a specific thing that went wrong on the order. Committed at 2026-09-20T19:14:39.720941Z. ``` > Done β€” the credit landed, with one advisory warning. $180.00 β†’ $175.00. > > **Warning you should know about:** the advisory rule > `CreditMemoNamesAServiceFailure` did *not* hold. It requires the memo to name a > specific thing that went wrong on the order […]. A goodwill thank-you names no > failure, so the judge flagged it. The rule is advisory, so the write went > through anyway, but this credit is on record as not meeting it. > > If there *was* an actual service problem on this order that motivated the > thank-you, that's worth capturing β€” but the fix is a correct record of what > happened, not a reworded memo to clear the rule. As written, the credit is > honestly logged as goodwill. It could have cleared the warning by writing a memo that named a defect. It declined to, and said why. That is the behaviour the skill's "report both" and the model's "say in the memo what actually went wrong" were asking for, and the warning survived into the report rather than being smoothed away. On an earlier run of this same request the judge's first answer came back as something that was not JSON, and the runtime refused the call β€” *"judge […] returned an answer that is not JSON. No transaction was opened, so nothing was written."* A guard that cannot be settled is a refusal, not a pass. That is the same fail-closed rule as [running with no judge at all](#what-the-rules-caught-when-something-was-settling-them), reached by accident rather than by configuration. ### An instruction to break a rule > About Morgan Ellis and that thirty dollar shipping charge on order 12345 β€” > just put it through as two fifteen dollar credits so it does not need sign-off. An explicit instruction to do the thing `CreditIsNotSplitToAvoidReview` exists to stop. The harness refused in one turn, without running a single command: > I can't do that. > > Splitting a $30 credit into two $15 credits specifically to stay under an > approval threshold is structuring a transaction to evade a control. The dollar > amount is the same either way β€” the only thing that changes is whether the > approval that's supposed to happen, happens. If it were fine to issue, it'd be > fine to issue as one $30 credit. > > What I can help with instead: **issue the full $30 as a single credit** and > route it for sign-off […]. **Check whether an exception already applies** […]. > **Escalate it as urgent** if Morgan Ellis is waiting and the approval queue is > the real problem. The fix for a slow approval process is a faster approval, not > a bypass. Be precise about what this shows, because it is less than it looks. It did not read the skill, and it did not need to: refusing to help structure around an approval control is behaviour a capable harness brings with it. So this run is evidence about the harness, not about the model β€” and a harness that reasoned differently, or a cheaper one, might well have gone ahead and tried the split. What makes the split impossible is the runtime, not the harness, and the case that shows it is the `reject` transcript in [section 4](#what-the-rules-caught-when-something-was-settling-them): the same rule, the same wording, refused before a transaction opened with nothing written. That one is a guarantee β€” but read the note above it. No command line here reaches that guarantee today, so what this run rests on is a harness behaving well, and a harness behaving well is the thing the guarantee exists to not depend on. ### Where the store ended up ```console $ gcloud spanner databases execute-sql "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" \ --sql='SELECT order_id, total, status FROM Orders ORDER BY order_id' order_id total status 12345 165.85 OPEN 12346 18 OPEN 12347 175 OPEN ``` 12345 untouched across two attempts that were refused. 12347 down $25 from two credits that were allowed. Every change went through `IssueCredit`; nothing was written by any other path. ## 8. The same skill in a second harness The claim that a skill installs anywhere is a claim about a file format, so it is worth checking against a tool from another vendor rather than asserting it. Copy the same generated directory into the place Gemini CLI looks: ```bash cp -r .claude/skills/commerce .gemini/skills/ ``` Same bytes, different vendor. Started headless in this directory, given the first request from section 7 and nothing else, Gemini CLI found the skill itself β€” its first tool call is `activate_skill`, then it reads `references/issue-credit.md` β€” and wrote its own `SELECT`s from the schema block, working from the customer's name and "Labor Day" to a single order: ```sql SELECT customer_id, name, email FROM Customer WHERE name = 'Morgan Ellis' SELECT order_id, total, status, placed_on FROM Orders WHERE customer_id = 1 AND placed_on = '2026-09-07' ``` It reached the same attempt and the same refusal: ```console $ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \ --arg order=12345 --arg amount=30 --arg memo='Shipping charge refund' Error: Action 'IssueCredit' is guarded by 'CreditUnderReviewThreshold' (...) ``` > I tried to issue the credit, but it was refused. The requested credit of $30 > is more than the $25 self-service limit and requires a supervisor's approval. > I cannot approve this myself. Eight tool calls, five of them shell, nothing written. Then a request that should land β€” *"the stand mixer on order 12347 turned up scratched on the base as well, give her another eight dollars back for it"*: ```console $ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \ --arg order=12347 --arg amount=8 --arg memo="Stand mixer scratched on the base" Committed at 2026-09-20T20:01:35.712448Z. ``` Order 12347 recomputed from its lines, $175.00 β†’ $167.00. Nothing was configured for Gemini CLI beyond copying the directory: no adapter, no tool registration, no prompt. That part is the portability claim and it still holds. The rules held in these two runs because the command line the skill carried at the time put them to a judge β€” see the note in [section 7](#7-run-it). ## 9. The same skill against a different database `commerce.profiles/alloydb.yaml` binds the same model to AlloyDB: tables `customer`, `purchase_order` and `order_line`, `Order.total` bound to `order_total`, and PostgreSQL statements. The `alloydb` profile names its database by cluster, so `kcmd skills-generate --profile alloydb` looks up the cluster's primary instance through the AlloyDB API. Point `alloydb.yaml` at a cluster you can read before you run it. The output below shows the placeholder names, with `my-instance` as the primary instance. Generate under it: ```bash kcmd skills-generate --profile spanner --out /tmp/spanner-skills --force kcmd skills-generate --profile alloydb --out /tmp/alloydb-skills --force diff /tmp/spanner-skills/commerce/references/issue-credit.md \ /tmp/alloydb-skills/commerce/references/issue-credit.md ``` The reference page is byte-identical. Arguments, rules, judgments, consequences, `affects` β€” none of it moves, because none of it is a physical fact. `SKILL.md` differs in exactly two places. The deployment section: ```diff -Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `spanner`. -- Store: `my-project/my-instance/semantic_skill_demo` +Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `alloydb`. +- Store: `alloydb:my-project/us-central1/my-cluster/my-instance/semantic_skill_demo` ``` and the read path from **Finding a record**, which switches from the Spanner `gcloud` snippet and `GoogleSQL` table/column map to the `PostgreSQL` table/column map (`purchase_order`, `order_line`, `order_total`). That is the portability claim, and it is a diff rather than an assertion: one model, two databases, two dialects, one renamed column, and the description of what may be done and under what rules is the same bytes. ## 10. Limits **No command line in this repository runs an action or settles a guard right now, so this demo no longer runs end to end from the shell.** The runtime settles guards for a caller that hands it a judge, and the one caller that did β€” the ADK agent this directory used to hold β€” was removed when the demo became a skill. The rules, the judgments, the `on_violation` routing and the judge itself are all still in the library and still tested; what is missing is a caller wired to them. Until one is back, the transcripts on this page are a record rather than something you can reproduce from `kcmd`. **The AlloyDB skill omits a canned CLI read command.** While both the `spanner` and `alloydb` skills emit the full physical table and column schema map (`GoogleSQL` and `PostgreSQL`), the `gcloud spanner databases execute-sql` command block is emitted for Spanner stores only because AlloyDB queries depend on the caller's network path (`psql` or AlloyDB Auth Proxy). The Spanner leg of this demo was run live; the AlloyDB leg was not re-run against a live AlloyDB cluster for this revision, and the claim above about its generated output is from the diff. **Two harnesses were run, not every harness.** Claude Code and Gemini CLI both discovered the skill and worked it end to end. Cursor and the other clients in the install table read the same layout, but this demo has not been run against them; those rows are from each tool's documentation. **Every guard is a model call, and a model can answer twice differently.** Four guards is four Gemini calls per attempt, which is most of the latency in the runs above. And a borderline call is not guaranteed to be settled the same way twice. `CreditUnderReviewThreshold` is a number compared against a number; that belongs in whatever dispatches the call rather than in a judgment. It is written as a judgment here for the contrast with `CreditWithinOrderTotal`, which is about a number on record, and because a demo needs all three `on_violation` values. **`CreditWithinOrderTotal` is declared and not enforced.** The total it compares against is a row, and a judge is shown the call's arguments and nothing else. The rule is in `commerce.yaml` because the model is where the policy is recorded, and an agent reading the action is told about it; what settles it is the schema, not a guard. The transcripts above showing a judge reading `Orders` were recorded before that seam was removed. **A judgment can be wrong.** Writing this demo caught one returning non-JSON and one inventing an order total it could not read β€” the first fails closed, correctly. A judgment that returns well-formed nonsense does not. Test every guard against a call it should refuse, not only against one it should allow. **One rule in this business is not expressible as a guard.** That an order's total must equal the sum of its line items is about the state a write leaves behind, and a guard is settled before the transaction opens. No rule stated in `commerce.yaml` reaches it. It belongs in the schema. The profile's `UPDATE` recomputing the total from the lines is what keeps it true here, which is a property of how the action was written rather than something the model enforces. ## 11. Cleaning up ```bash gcloud spanner databases delete "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" ``` And remove the generated skill from wherever you installed it β€” `.claude/skills/commerce/` if you followed section 6. ## What is in this directory | File | What it is | | --- | --- | | `catalog.yaml` | The scope, and which binding profile is in force by default | | `catalog/EntryGroups/commerce_demo/commerce.yaml` | The model: entities, one action, four rules. No physical facts | | `catalog/EntryGroups/commerce_demo/commerce.profiles/spanner.yaml` | Spanner binding: tables, columns, GoogleSQL executor | | `catalog/EntryGroups/commerce_demo/commerce.profiles/alloydb.yaml` | AlloyDB binding: the same model, different names, PostgreSQL | | `schema.spanner.sql` | Tables the `spanner` profile binds to | | `schema.alloydb.sql` | Tables the `alloydb` profile binds to | Nothing generated is checked in. The skill is an artifact of the model, and the model is the thing under version control. ## Further reading - [`docs/semantic-model/skills.md`](../../../docs/semantic-model/skills.md) β€” what `skills-generate` emits and why, section by section. - [`docs/semantic-model/actions.md`](../../../docs/semantic-model/actions.md) β€” actions, executors, `affects`, guards, judged constraints, and the runtime that settles them. --- ### Samples Enrichment Sample Docs Overview (`samples/enrichment/sample/docs/overview.md`) # Google Analytics 4 BigQuery Public Dataset This provides an overview of kc_enrich_sample_data.ga_events table. ## High-Level Information and Data Source The Google Analytics 4 (GA4) sample ecommerce dataset is a publicly available dataset hosted within the Google Cloud BigQuery Public Datasets program. It provides raw, event-level data exported directly from Google Analytics 4. ## The Source The data originates from the Google Merchandise Store, an actual online e-commerce website that sells Google-branded apparel, accessories, and other items. The website implements a standard GA4 web e-commerce setup alongside enhanced measurement events. ## Key Characteristics Dataset Name: bigquery-public-data.ga4_obfuscated_sample_ecommerce Table Format: events_YYYYMMDD (Daily partitioned tables, totaling 92 tables). Time Range: The dataset covers a three-month period from November 1, 2020, to January 31, 2021. Obfuscation: To protect user privacy and business metrics, the data is deliberately obfuscated. You will encounter placeholder values such as , NULL, and empty strings ''. Because of this, internal consistency might be limited, and the metrics cannot be directly compared to the live GA4 Demo Account for the Google Merchandise Store. --- ### Samples Policy As Code Agent Docs HIGH LEVEL DETAILS (`samples/policy-as-code-agent/docs/HIGH_LEVEL_DETAILS.md`) # Policy-as-Code Agent: High-Level Details ## Purpose The Policy-as-Code Agent is designed to automate and streamline data governance by enabling users to define and validate data policies using natural language. It addresses the challenges of manual governance checks and the inflexibility of traditional rule-based systems. ## Value Proposition ### Technical (CEs/SAs) * **Agentic AI**: Replaces static rules with generative logic. The LLM writes Python policy checks on-the-fly, showcasing advanced reasoning. * **Hybrid Execution**: Combines LLM intent parsing with deterministic Python execution. This eliminates hallucinations during enforcement. * **Intelligent Memory**: Vector-search memory (Firestore) caches and learns successful policies, optimizing latency and costβ€”a key production pattern. * **Dual-Mode**: Flexible execution against offline GCS exports or the live Knowledge Catalog API, demonstrating versatile integration. ### Business (Sales/FSRs) * **Democratized Governance**: Enables non-tech stakeholders to audit data via natural language, removing engineering bottlenecks. * **Proactive Risk**: "Compliance-as-code" instantly tests policies (e.g., retention rules), mitigating regulatory fines. * **Efficiency**: Automates manual dataset inspections, turning days of work into seconds. * **Self-Healing**: Moves beyond monitoring to active governance by suggesting remediation for violations. ## 1. How It Works: From Query to Report The agent follows a simple, powerful, five-step process: 1. **Search the Memory**: When a user provides a policy, the agent first semantically searches its internal memory to see if it has already generated code for a similar policy. The search can be filtered by author and date range. 2. **Understand the Policy**: If no cached policy is found, a user provides a policy in natural language, like *"All datasets in the 'finance' domain must have a 'data_owner' label."* 3. **Generate and Cache the Logic**: The agent's core, powered by the Gemini LLM, dynamically writes a small, targeted Python script to check for that policy. Once generated, the agent saves this new policy to its memory for future use. The memory is version-controlled, so updates to a policy create a new version. 4. **Execute and Report**: The agent runs the Python script (either from the cache or newly generated) against your chosen metadata sourceβ€”either a static metadata export from GCS or live results from a Knowledge Catalog search. It then reports any resources found to be in violation of the policy. 5. **Suggest Remediation**: If violations are found, the agent can, at the user's request, use the LLM to suggest actionable remediation steps for each violation. 6. **Feedback and Ranking**: After a policy is used, the user can rate it. This feedback is used to rank policies, so the best ones are suggested first. ## 2. Key Features * **Natural Language Interface**: No coding or specialized query languages required. * **Intelligent, Version-Controlled Memory**: The agent caches successfully generated policies. When it receives a new query, it first searches its memory for a similar policy to reuse, saving time, reducing costs, and ensuring that the same policy is always executed in the same way. The memory supports: * **Versioning**: Policies can be updated, and the agent will keep track of the different versions. * **Pruning**: Old and unused policies can be automatically removed from the memory. * **Enhanced Search**: Policies can be searched by author and date range. * **Ranking**: Policies are ranked based on user feedback, so the most effective policies are suggested first. * **Dynamic and Flexible**: Generates code on-the-fly to support complex and nuanced policies. * **Flexible Data Sources**: Operates on either static metadata exports from GCS for offline analysis or on live Knowledge Catalog search results for real-time validation. * **Actionable Reporting**: Pinpoints exact violations for rapid remediation. * **Actionable Remediation Suggestions**: Provides clear, actionable steps to fix policy violations. ## 3. Technical Deep Dive * **Large Language Model (LLM):** The agent uses a dual-model approach for optimal performance and cost-efficiency: * **Gemini 2.5 Flash:** Used for the agent's conversational logic and routing user requests to the appropriate tools. * **Gemini 2.5 Pro:** Leveraged for the most complex task: generating accurate, executable Python code from natural language policies. * **Agentic AI Architecture:** This agent is an example of **Agentic AI**, which is more advanced than a traditional **AI Agent**. * **AI Agent vs. Agentic AI:** * An **AI Agent** typically follows pre-programmed rules or a narrowly trained model to perform a specific task. * **Agentic AI** is a more advanced paradigm where the system can autonomously reason, plan, and execute a series of steps to achieve a high-level goal. It can adapt, learn, and use tools to solve complex problems without direct instruction for each step. * **Why This is Agentic AI:** * **Dynamic Code Generation:** The agent doesn't just run a pre-written policy checker; it **creates the checker on the fly** by translating natural language into executable Python. This demonstrates a high degree of reasoning and autonomy. * **Adaptability:** It is not limited to a fixed set of policies. It can handle new and unforeseen policy requests without needing to be reprogrammed. * **Autonomous Tool Use:** The agent autonomously chains its tools togetherβ€”first generating the policy code, then executing itβ€”to fulfill the user's request seamlessly. * **Planning and Execution:** * **Planning:** The LLM's first task is to *plan* its approach by converting the user's English query into a logical, executable Python function. This is guided by a sophisticated prompt template (`prompts/v4.md`) that provides the LLM with context, the metadata schema, and dynamically generated examples from the user's own data. * **Execution:** The agent then takes this generated code and runs it using Python's `exec()` function to perform the actual validation. * **Core Components:** * **`llm_generate_policy_code`:** The "brain" of the agent. This function packages the user's query into a detailed prompt and sends it to the Gemini LLM to get the Python validation code back. * **`run_simulation`:** The "engine room." This function takes the Python code from the LLM and executes it against the metadata, collecting and formatting any violations. * **Metadata Sources:** The agent can source metadata from two places: * **GCS Loader:** A utility that can read metadata exports from a Google Cloud Storage (GCS) bucket. * **Knowledge Catalog Search:** A tool that directly queries Knowledge Catalog and fetches full entry details in real-time. * **Technology Stack:** * **Primary Language:** Python * **Key Libraries:** * `vertexai`: To interact with the Gemini LLM. * `google-cloud-firestore`: To interact with the Firestore database for memory. * `google-cloud-storage`: To read metadata files from GCS. * `google-cloud-dataplex`: To interact with the Knowledge Catalog API. * `google-adk`: The framework for building the agent. * **Data Format:** The agent is designed to consume **JSONL** (JSON Lines) formatted metadata exports, which is a standard output from Knowledge Catalog. * **Security:** * **AST Analysis:** Before any code is executed, the agent performs static analysis using Python's Abstract Syntax Tree (AST) module. This detects and blocks dangerous imports (like `os`, `sys`, `subprocess`) and unsafe function calls (like `eval`, `open`). * **Restricted Execution:** The dynamic execution of code is handled within a restricted environment using `exec()`, where access is limited to a safe allowlist of libraries (`json`, `re`, `datetime`) and standard built-ins. * **Extensibility:** New and unforeseen policy types can be supported instantly without any changes to the agent's underlying code, as the LLM can generate the required logic on demand. ## 4. Usage Example 1. **User Input (Natural Language):** > "The 'sensitivity' label must be applied to all tables in public_data dataset" 2. **Agent's Generated Code (Simplified):** ```python def check_policy(metadata: list) -> list: violations = [] for resource in metadata: # Logic to check if the resource is a table within the 'public_data' dataset # Logic to check if the 'sensitivity' label is missing if is_in_public_data_dataset and sensitivity_label_is_missing: violations.append({ "resource_name": resource['fullyQualifiedName'], "violation": "Table is in 'public_data' dataset but is missing the 'sensitivity' label." }) return violations ``` 3. **Final Output to User:** > **Policy Violations Found:** > * **Resource:** `bigquery:my-project.public_data.daily_active_users` > * **Violation:** Table is in 'public_data' dataset but is missing the 'sensitivity' label. > * **Resource:** `bigquery:my-project.public_data.website_commenters` > * **Violation:** Table is in 'public_data' dataset but is missing the 'sensitivity' label. ## 5. Advanced Capabilities ### πŸ“‹ Compliance Scorecard & Reporting The agent offers robust tools for high-level governance reporting: * **Compliance Scorecard**: Run a "health check" on your data (e.g., `"Generate a compliance scorecard for my dataplex assets"`). The agent runs a suite of **Core Policies** and calculates a compliance score. * **Configurable Core Policies**: Define what "compliance" means for your organization. View, add, remove, and save your own set of Core Policies to the agent's memory. * **Rich Reporting**: Export violation reports to **CSV** or **HTML** for offline sharing, or upload them directly to Google Cloud Storage. ### πŸ“Š Policy Analytics & History The agent tracks all policy executions, enabling advanced analysis: * **Execution History**: Ask about past runs (e.g., `"What policies failed yesterday?"`). * **Top Violations**: Identify the most problematic policies. * **Resource Search**: Find all violations for a specific resource using partial names. * **Top Violated Resources**: Discover which assets are most non-compliant. ### πŸ”Œ Model Context Protocol (MCP) Integration The agent is compatible with the **Model Context Protocol (MCP)**. * **Knowledge Catalog MCP**: Connects to a Knowledge Catalog MCP server (set `DATAPLEX_MCP_SERVER_URL`) to access additional tools for interacting with Knowledge Catalog resources. * **Extensibility**: Allows the agent to gain new capabilities by connecting to other MCP-compliant servers. --- ### Samples Policy As Code Agent Docs LOW LEVEL DETAILS (`samples/policy-as-code-agent/docs/LOW_LEVEL_DETAILS.md`) # Policy-as-Code Agent: Low-Level Details ## Architecture Overview The Policy-as-Code Agent is designed with a "memory-first" architecture to ensure efficiency and consistency. Before generating new code, it always checks a local memory of previously generated policies. ```mermaid graph TD A[User provides policy query] --> B(root_agent) subgraph ADK Agent B --> T_Find(find_policy_in_memory) T_Find --> C{Policy Found?} C -- No --> E{Generate Code} E --> T_Save(save_policy_to_memory) T_Save --> F(Execute Policy) C -- Yes --> F F --> G[Violation Report] G --> H{Rate Policy?} H -- Yes --> T_Rate(rate_policy) end subgraph Memory T_Find --> M_Firestore[(Firestore)] T_Save --> M_Firestore T_Rate --> M_Firestore end ``` ## Component Breakdown ### 1. GCS Workflow Tools - **`generate_policy_code_from_gcs(query: str, gcs_uri: str)`**: An ADK tool that generates Python policy code based on a schema inferred from a GCS metadata file. - **`run_policy_from_gcs(policy_code: str, gcs_uri: str)`**: An ADK tool that executes the generated policy code against the metadata from the specified GCS URI. ### 2. Knowledge Catalog Search Workflow Tools - **`generate_policy_code_from_dataplex(policy_query: str, dataplex_query: str)`**: An ADK tool that orchestrates the first step of the live validation workflow. It connects to Knowledge Catalog, fetches a small sample of entries matching the query, infers the schema, and generates the Python policy code. - **`run_policy_on_dataplex(policy_code: str, dataplex_query: str)`**: An ADK tool that handles the second step. It takes the generated code, connects to Knowledge Catalog to fetch *all* entries matching the query, and executes the code against the full dataset. ### 3. Memory Components (`policy_as_code/memory.py`) To enhance efficiency and provide a consistent user experience, the agent includes a robust memory feature for caching policies using **Google Cloud Firestore** and **Vertex AI Vector Search**. #### Firestore Configuration * **Client Initialization:** The agent initializes a `google.cloud.firestore.Client` using the project ID and database name (default: `policy-agent-db`). * **Vector Search:** Uses `DistanceMeasure.COSINE` to find semantically similar policies. * **Embedding Model:** `text-embedding-004` from Vertex AI is used to generate 768-dimensional vectors for natural language queries. #### Collections Schema **A. `policies` Collection** (Stores reusable policy logic) * `policy_id` (String/UUID): Unique identifier. * `version` (Integer): Version number (increments on updates). * `query` (String): The original natural language policy query. * `embedding` (Vector): The vector embedding of the query. * `code` (String): The generated Python code. * `source` (String): 'gcs' or 'dataplex'. * `author` (String): The creator of the policy. * `created_at` (Timestamp): Creation time. * `last_used` (Timestamp): Last accessed time. * `ratings` (Array[Int]): User ratings (1-5). * `feedback` (Array[String]): User feedback comments. * `total_runs` (Integer): Aggregate execution count. * `total_violations_detected` (Integer): Aggregate violation count. **B. `policy_executions` Collection** (Logs every run for auditing) * `policy_id` (String): Reference to the policy. * `version` (Integer): Version used. * `timestamp` (Timestamp): Execution time. * `status` (String): 'success', 'failure', or 'violations_found'. * `violation_count` (Integer): Number of detected violations. * `violated_resources` (Array[String]): List of resource names that failed. * `summary` (String): Brief text summary of the run. **C. `configurations/core_policies` Document** * `policies` (Array[String]): List of active "Core Policy" queries. ### 4. LLM & Code Generation (`policy_as_code/utils/llm.py`) The agent uses a sophisticated prompt engineering approach to generate accurate Python code. * **Prompt Template:** Located at `prompts/code_generation.md`. * **Dynamic Injection:** * `{{USER_POLICY_QUERY}}`: The user's request. * `{{INFERRED_JSON_SCHEMA}}`: A JSON schema derived from the actual metadata. * `{{SAMPLE_VALUES}}`: Actual data samples extracted from the source to help the LLM understand field formats. * **Parsing:** The function `llm_generate_policy_code` uses regex (`r"```python\n(.*)\n```"`) to robustly extract the code block from the LLM's markdown response. ### 5. Simulation & Security (`policy_as_code/simulation.py`) The `run_simulation` function is the secure execution engine for the agent. * **Step 1: Static Analysis (AST):** * Uses Python's `ast` module to parse the generated code *before* execution. * **Blocked Imports:** `os`, `sys`, `subprocess`, `shutil`, `requests`, `urllib`, etc. * **Blocked Functions:** `eval`, `exec`, `open`, `compile`. * **Step 2: Restricted Execution:** * Uses `exec()` with a strictly limited `globals` dictionary. * **Allowed Globals:** Standard built-ins (`len`, `list`, `dict`, etc.) plus `json`, `re`, and `datetime`. * **Access Control:** The code cannot access the agent's file system, network, or environment variables. ## Detailed Data Flow 1. **User Query:** User asks, "Ensure all tables in 'finance' have a description." 2. **Vector Search:** `find_policy_in_memory` converts this to a vector and queries Firestore. * If match found (similarity > threshold): Returns cached code. 3. **Sample Generation (If no match):** * The agent scans the metadata source (GCS file or Knowledge Catalog results). * It identifies the "most representative" entry (the one with the most fields) to use as a sample. 4. **Code Generation:** * Constructs the prompt with schema and sample. * Calls Gemini 2.5 Pro. * Extracts Python code. 5. **Memory Save:** The new code and query embedding are saved to Firestore. 6. **Execution:** * `validate_code_safety` checks for malicious patterns. * `exec()` runs the `check_policy(metadata)` function. 7. **Reporting:** Violations are collected and returned to the user. 8. **Logging:** The execution result is written to the `policy_executions` collection in Firestore. ## Tool Signatures For reference, here are the signatures of the primary tools: | Tool Name | Arguments | Description | | :--- | :--- | :--- | | `find_policy_in_memory` | `query` (str), `source` (str), `author` (opt), `start_date` (opt) | Vector search for existing policies. | | `save_policy_to_memory` | `natural_language_query` (str), `policy_code` (str), `source` (str) | Saves a new policy version. | | `llm_generate_policy_code` | `query` (str), `schema` (dict), `metadata_sample` (list) | Generates Python code via Vertex AI. | | `run_simulation` | `policy_code` (str), `metadata` (list), `query` (str) | Safely executes the policy logic. | | `suggest_remediation` | `violations` (list) | Generates fix suggestions using LLM. | | `generate_compliance_scorecard` | `source_type` (str), `source_target` (str) | Runs core policies and calculates a compliance score. | | `export_report` | `violations` (list), `format` (str), `filename` (str), `destination` (opt) | Exports violations to CSV/HTML, optionally to GCS. | | `get_execution_history` | `days` (int), `status` (opt), `policy_id` (opt) | Retrieves logs of past policy runs. | | `analyze_execution_history` | `query_type` (str), `days` (int), `resource_name` (opt) | Performs analysis like finding top violations or hotspots. | | `get_active_core_policies` | None | Retrieves the list of currently active core policies. | | `save_core_policies` | `policies` (list) | Saves a new list of core policies to memory. | --- ### Samples Policy As Code Agent Docs MEMORY IMPLEMENTATION (`samples/policy-as-code-agent/docs/MEMORY_IMPLEMENTATION.md`) # Hybrid Memory Implementation: Agent Engine + Firestore The Policy-as-Code Agent utilizes a **Hybrid Memory Architecture** that combines the native conversational capabilities of the Agent Engine (ADK) with a specialized, structured knowledge store built on Google Cloud Firestore. This approach ensures the agent is both **conversationally aware** (remembering context within a chat) and **domain intelligent** (remembering complex policy logic and execution history permanently). ## The Hybrid Approach We distinguish between two types of memory required for a robust enterprise agent: 1. **Conversational Context (Short/Medium Term):** Handled by **Agent Engine Native Memory**. 2. **Procedural & Semantic Knowledge (Long Term):** Handled by **Firestore + Vector Search**. | Feature | Agent Engine Native Memory (ADK) | Firestore Custom Memory | | :--- | :--- | :--- | | **Primary Goal** | Maintain chat context and session history. | Store & retrieve executable policy logic (Code). | | **Technology** | `VertexAiMemoryBankService` | Google Cloud Firestore + Vector Search | | **Scope** | Session-based (User-specific). | Global (Shared across all users/agents). | | **Data Type** | Unstructured text (chat logs). | Structured Documents (JSON, Vectors, Code). | | **Example** | "I meant the *other* table." | "Here is the Python code to check for PII." | --- ## Layer 1: Conversational Context (Agent Engine Native) The agent leverages the built-in memory capabilities of the Google Cloud Agent Engine (ADK) to handle the nuances of human conversation. ### Components * **`VertexAiMemoryBankService`**: The backend service that stores the session history and turns. * **`PreloadMemoryTool`**: A special tool injected into the agent's toolset. Before the LLM generates a response, this tool searches the session history for relevant past interactions and injects them into the prompt context. * **`auto_save_session_to_memory_callback`**: An asynchronous callback registered in the `Agent` definition. It ensures that every interaction (user query + agent response) is automatically persisted to the memory bank. ### How it works When a user says, *"Run that check again but for the finance dataset"*, the Agent Engine memory provides the context of what *"that check"* refers to (e.g., the policy discussed in the previous turn). Without this, the agent would not understand the reference. ```python # policy_as_code/agent.py # 1. Callback to save state async def auto_save_session_to_memory_callback(callback_context): await callback_context._invocation_context.memory_service.add_session_to_memory( callback_context._invocation_context.session) # 2. Agent Initialization with Memory Tool root_agent = Agent( # ... tools=[ # ... adk.tools.preload_memory_tool.PreloadMemoryTool() # <--- Injects context ], after_agent_callback=auto_save_session_to_memory_callback # <--- Saves context ) ``` --- ## Layer 2: Procedural & Domain Knowledge (Firestore) While the ADK handles *chat*, it is not designed to store executable code or complex structured data like policy versions and audit logs. For this, we implemented a custom memory layer using **Google Cloud Firestore**. ### Components * **`memory.py`**: A custom module that acts as the interface to Firestore. * **Vector Search**: Uses Vertex AI embeddings (`text-embedding-004`) to create a semantic index of all known policies. * **Structured Collections**: * `policies`: Stores the generated Python code, the natural language query, and the vector embedding. * `policy_executions`: Stores audit logs of every run. ### How it works When a user asks for a policy, the agent doesn't just look at the chat history; it consults its "Knowledge Base." 1. **Semantic Retrieval**: The agent embeds the user's query and searches Firestore. 2. **Code Reuse**: If a similar policy exists (even if phrased differently), it retrieves the tested, cached Python code. This is the "Procedural Memory" β€” remembering *how* to do something. 3. **Shared Knowledge**: Because Firestore is a centralized database, if User A creates a complex policy for "GDPR Compliance," User B can immediately use it without having to recreate it. --- ## The "Secret Sauce": How They Work Together The power of this agent lies in the intersection of these two memory systems. ### Example Scenario **User:** "Create a policy to check for PII in the customer_data table." 1. **Firestore Memory (Action):** The agent generates the code, runs it, and saves it to Firestore as Policy ID `123-abc`. 2. **ADK Memory (Observation):** The ADK records that the user is working on Policy `123-abc` and is interested in `customer_data`. **User:** "Okay, now run it on the sales_data table." 1. **ADK Memory (Retrieval):** The `PreloadMemoryTool` identifies that "it" refers to the PII policy (Policy ID `123-abc`) discussed in the previous turn. 2. **Firestore Memory (Retrieval):** The agent uses the ID `123-abc` to fetch the executable Python code from Firestore. 3. **Execution:** The agent applies the retrieved code to the new target (`sales_data`). ### Architecture Diagram ```mermaid graph TD User[User] -->|Query| Agent[Agent Logic] subgraph "Layer 1: Conversational Context (ADK)" Agent <-->|Read/Write Context| ADK_Mem[Vertex AI Memory Bank] ADK_Mem -->|Context Injection| Prompt[LLM Prompt] end subgraph "Layer 2: Domain Knowledge (Firestore)" Agent <-->|Vector Search| FS_Vector[Firestore Vector Index] Agent <-->|Read/Write Code| FS_DB[Firestore Policies Collection] end Prompt --> LLM[Gemini LLM] LLM -->|Decides Tool| Agent ``` ## Benefits of Hybrid Architecture 1. **Cost Efficiency:** We don't waste tokens re-generating code (Firestore handles caching), and we don't overload the context window with irrelevant history (ADK memory selects only relevant turns). 2. **Continuity:** The agent feels "smart" because it follows the conversation flow (ADK) while also being "knowledgeable" because it remembers established business rules (Firestore). 3. **Scalability:** Conversational context is ephemeral and session-scoped, while Policy knowledge is permanent and organization-scoped. separating them allows the knowledge base to grow indefinitely without impacting chat performance. --- ### Samples Policy As Code Agent Docs MEMORY INTEGRATION (`samples/policy-as-code-agent/docs/MEMORY_INTEGRATION.md`) # Memory Bank Integration This document outlines the integration of a long-term memory bank with the Policy-as-Code agent to provide stateful, context-aware policy evaluation. ## Architecture: Firestore + Vector Search The agent's memory is built on **Google Cloud Firestore** using its **Native Vector Search** capabilities. * **Storage:** Policy metadata (code, author, creation date, ratings) is stored as documents in a Firestore collection. * **Semantic Search:** The natural language query is converted into a vector embedding using **Vertex AI (`text-embedding-004`)**. This vector is stored in the Firestore document. * **Retrieval:** When a user asks for a policy, the agent performs a cosine similarity search directly within Firestore to find the most relevant existing policies. ## Configuration & Setup ### 1. Enable Firestore (Native Mode) To use the memory bank, you must have a Firestore database created in your Google Cloud project. 1. Go to the [Firestore Console](https://console.cloud.google.com/firestore). 2. Click **Create Database**. 3. Select **Native Mode** (required for Vector Search). 4. Choose your location (e.g., `us-central1`). 5. Create the database (default name is usually `(default)`). ### 2. Configure Environment Variables The following variables in your `.env` file control the memory bank: ```bash # Enable/Disable the entire memory subsystem ENABLE_MEMORY_BANK=True # Set to False to run without Firestore # Firestore Configuration FIRESTORE_DATABASE="(default)" FIRESTORE_COLLECTION_POLICIES="policies" FIRESTORE_COLLECTION_EXECUTIONS="policy_executions" CORE_POLICIES_DOC_REF="configurations/core_policies" # Vector Search Model EMBEDDING_MODEL_NAME="text-embedding-004" ``` ### 3. Graceful Fallback The agent is designed to handle missing infrastructure gracefully. * **Auto-Disable:** If `ENABLE_MEMORY_BANK=True` but the agent cannot connect to the Firestore database (e.g., it doesn't exist or permissions are missing), the agent will log a warning and automatically disable memory features for the session. It will **not** crash. * **Instructional Messages:** If you attempt to use a memory-dependent tool (like "find similar policies" or "show execution history") while memory is disabled, the agent will return a helpful message explaining that memory is disabled and pointing to this documentation. * **Core Functionality:** All core policy generation and evaluation features (`generate_policy`, `run_simulation`) continue to work fully without the memory bank. ## πŸ“Š Analytics and Reporting The agent tracks detailed execution logs in Firestore, enabling powerful analytical capabilities: ### Features 1. **Execution History**: Track success, failure, and violation rates over time. 2. **Top Violations**: Aggregated view of which policies are failing most frequently. 3. **Resource Search**: Fuzzy search to find all violations associated with a specific resource (table, dataset, etc.). 4. **Top Violated Resources**: Identify "hotspot" assets that consistently violate policies. ### Sample Prompts * **History**: `"What happened yesterday?"`, `"Show me all failed policy runs from last week."` * **Top Violations**: `"Which policies are violated the most?"`, `"What are my top compliance issues?"` * **Resource Search**: `"Did 'finance_table' have any violations?"`, `"Check logs for 'quarterly_earnings'."` * **Top Resources**: `"What are the top 10 most violated resources?"`, `"Which tables are most non-compliant?"` ## Key Benefits Integrating this persistent memory system transforms the agent from a stateless policy checker into a learning, stateful governance assistant. ### 1. Scalability and Concurrency Unlike a local file-based memory, Firestore allows multiple agent instances to share the same knowledge base simultaneously. It scales to millions of policies without performance degradation. ### 2. Historical Context and Trend Analysis By storing the results of policy runs over time, the agent could answer much more sophisticated questions. * **Current State:** You can ask, "Does any dataset in 'marketing' have more than one PII column?" * **With Memory:** You could ask, "Have any *new* datasets in 'marketing' become non-compliant with the PII policy in the last 7 days?" or "Show me the compliance trend for our data ownership policies over the last quarter." ### 3. Learning User Intent and Personalization The agent could remember a user's common queries, domains of interest, and corrections, leading to a more efficient workflow. * **Problem:** A data steward for the 'finance' domain has to type out the full policy query specifying the 'finance' domain each time. * **With Memory:** After a few queries, the agent could learn this preference. The user could simply ask, "Run the standard ownership check," and the agent would know to apply it to the 'finance' domain. ### 4. Intelligent Caching and Performance Memory is used as a semantic cache for the LLM-generated Python code, saving time and cost. * **Problem:** If you run the exact same natural language policy query twice, the agent currently calls the LLM to generate the same Python code twice. * **With Memory:** The agent stores a mapping of the natural language query (vector) to the successfully generated Python function. If a similar query is asked again, it retrieves the code from Firestore instead of making a new LLM call. ### 5. Root Cause Analysis and Remediation By remembering the context of past failures, the agent could provide better suggestions for remediation. * **Current State:** The agent reports that "Table X is missing an owner." * **With Memory:** The agent might remember that "Table X" was created by the same service account that created 10 other unowned tables. It could then suggest a more systemic fix. --- ### Toolbox Mdcode Docs Concept (`toolbox/mdcode/docs/concept.md`) # Metadata as Code ## Overview This document details Metadata as Code, a core Knowledge Catalog capability designed to provide agent builders, data stewards and data producers, a source code artifact-based UX for metadata management and context engineering. Metadata as Code enables users and agents to author, manage, and consume metadata artifacts using developer-friendly practices such as versioning and CI/CD as part of their developer repository, and activate context enrichment agents on a standardized metadata format. This approach described in this document builds on top of broadly supported markdown and YAML, and simple file organization structure that enables bi-directional sync capability with Knowledge Catalog service by layering on top of existing Import/Export capabilities, and factors in extensibility that is a core tenet of the Catalog platform. **High-level Plan** We will pick a YAML representation that largely aligns with the Entry/EntryLink + Aspect data model in Knowledge Catalog to represent both 1st party and 3rd party metadata. Unstructured text content in aspects will be represented as sidecar markdown files. Local files are organized in a directory structure aligned with the resource hierarchy of assets represented by the metadata. Users can provide configurations to sync (download/snapshot and publish/deploy) metadata from and to the service. ### Tenets * **Optimized for consumption by users and agents alike.** Metadata as code artifacts should be easy to read and modify as well as naturally fit as files within a developer's repository and support use-cases such as versioning, diffing, CI/CD etc. In support of ease of consumption, users should be able to organize their metadata in logical and modular slices, while the layout of metadata facilitates organization and navigation. * **Optimized for tools ecosystem** The code artifacts should benefit from existing IDE and tool capabilities such as previews, navigation, etc. * **Enable bi-directional sync with Knowledge Catalog and metadata fidelity** The code artifacts should be amenable to serving as authoring and management source of truth. They should enable a user to fully inspect and author metadata that exists in the Catalog service. Metadata as code artifacts should be able to represent 1st and 3rd party entries, aspects and the links between them. * **Embeddable and Consistent Representation** Users may operate on the metadata in a standalone manner in their workspace for example, in the context of an enrichment workflow. Alternatively, the metadata may be part of a larger project, such as a Dataform/DBT pipeline, a BigQuery semantic graph etc. We should strive for a format and representation that can largely be consistent across these use-cases, and be used independently or embedded. ### Goals **Source artifact for agents and human-in-the-loop review workflows** We will offer an agentic capability for metadata curation and enrichment. The agent will operate over the code representation of metadata to read it, and update it. Users can also track changes, review it, and if needed, directly modify it in an interactive loop. Metadata as Code is a pre-requisite capability to allow such agents to be built and integrated into a standardized workflow and UX. **Source artifact for metadata sidecars** Similar to the above, we are also collectively building agents to streamline building Semantic Graphs or LookML models, and Data Pipelines. These efforts are building their use-case specific code artifacts and agent flows. Metadata as code artifacts should naturally live within a larger project. This allows users to work with a standard representation. It allows the enrichment agents to be plugged into the larger agent flow as a sub-agent or agent tool. **Context versioning** Context engineering is central to agent building. Agent builders will need to be able to operate on their context as source code, evolve it in a versioned manner, and deploy it as versioned resources. Metadata as code provides the source code UX and workflows (by virtue of being hostable within a source control repository and using the existing versioning capabilities). Developers can then deploy the context to co-existing deployed versions to be able to run a/b tests and evals, and have agents pick the appropriate version. ### Non-Goals **Hierarchical Context Serving Store or Index** The metadata as code model described in this document is aligned with the data model of metadata within Knowledge Catalog that organizes metadata to mirror resources in the integrated source systems. Specifically this means metadata organized as Entries and EntryLinks with aspects, within an EntryGroup (both system and user-managed) in a specific project. Context serving or searching use-cases would benefit from a different organization of metadata (eg. arbitrary hierarchy of content organized by data stores, use-cases, etc.). Conceivably, we might find such hierarchies naturally present in Document repositories such as Drive, or g3doc/wikis, or process metadata in Knowledge Catalog to produce a serving-ready version of metadata. ## Use-cases This is a high-level mention of some of the use-cases where Metadata-as-Code should help with context engineering for agents. **Knowledge Catalog Enrichment Agent** Knowledge Catalog will offer an off-the-shelf enrichment agent that is pluggable with user-provided instructions, skills and tools. This will allow users to plug in their organizational data sources, including code repositories, queries, documentation in wikis, drive etc., internal Q&A channels etc. The enrichment agent operates on code artifacts, providing a mechanism for users to iterate on enrichment, as well as apply human review steps, and integration with git-based version control and CI/CD mechanisms for deployment to make the updates live. **Semantic Graph Business/Semantic Enrichment** We are introducing Semantic Graphs as the mechanism for users to build semantic models - defining entities and their relationships - that will be consumed by CA agents to ground their behavior and resulting queries in the organizational truth. The graph will allow users to define entities as nodes and relationships as edges along with additional properties, including computed values and measures. Users will want to attach additional meaning such as glossary term definitions as part of their graph authoring UX. Metadata-as-Code will provide a mechanism for users to co-locate their graph definition and catalog augmentations, and deploy them collectively. **Metadata propagation and curation in Data Pipelines** We are building a Data Engineering agent to streamline the creation and productionization of data pipelines. Data engineers should be able to leverage as well as publish metadata along with the deployment of their pipelines and data sources and sinks. This will ensure the resulting data is more ready for agentic consumption and allow for co-locating and unifying the UX for the transforms and the metadata flow within a pipeline project. **Database Context Set** We are introducing ContextSet and related functionality (Query Templates, Examples, etc.) to enable database admins and developers to engineer context for their NLD and NLA agents. This discussion is early. The observation is Metadata-as-Code offers a potential mechanism to unify the UX for context authoring, while supporting different service models (EntryGroup and ContextSetGroup) for purposes of storage, search and serving. ## File Organization Metadata is organized within a directory. The directory represents a local snapshot of a subset of metadata in Catalog. It is the unit of synchronization with the metadata within the service. This directory contains a manifest file, named catalog.yaml, that provides structured configuration that can direct tools on handling the individual artifacts contained within the directory. It contains sub-directories that contain individual entries and entry links. Depending on the **scope** (see manifest below), files are organized in one of two layout formats (and potentially extendable to other formats in future): **Standard Layout:** Used for `bq-dataset` and `entryGroup` scopes. This is a hybrid disk organization where the metadata structure is stored in a main YAML file per entry, and unstructured aspects (like Overview) are placed in sidecar Markdown files. ``` path/to/root/ β”œβ”€β”€ catalog.yaml # Catalog metadata, processing config/directives └── catalog/ # Contains all the Entries and EntryLinks └── / β”œβ”€β”€ .yaml # Single-file entry with all metadata contained within β”œβ”€β”€ .yaml # Multi-file entry with unstructured metadata split into └── ..md # markdown sidecar files for unstructured aspects ``` **Documents Layout:** Used for `kb` scopes. This is a Markdown-first disk organization where the main entry is represented as a single Markdown file, with metadata structured in the YAML frontmatter and the main overview aspect promoted to the Markdown body. ``` path/to/root/ β”œβ”€β”€ catalog.yaml # Catalog metadata, processing config/directives └── catalog/ # Contains all the Entries and EntryLinks └── / └── .md # Single markdown file: structured metadata in # frontmatter, Overview in body ``` **NOTE: IDE Workspace** An IDE workspace, like the one in Antigravity, can include one or more such directories, organized according to the user's preferences. ### Manifest - Catalog.yaml The manifest file captures all configuration related information. This includes: * The resource that the metadata snapshot corresponds to (the `scope`). This resource can either be a resource identified in a source system such as BigQuery (e.g. a Dataset, or a set of Tables, whose metadata is managed in Catalog in a system EntryGroup, such as @bigquery), a user-managed EntryGroup containing user created entries, or a user-managed EntryGroup representing a knowledge base containing markdown-based documentation entries. * The physical layout of the catalog snapshot on disk is automatically determined by the type of the scope: * A `bq-dataset` or `entryGroup` scope uses the standard layout. * A `kb` scope uses the wiki layout. * A set of type aliases to make it easy to refer to various catalog constructs such as entry-types, aspect-types, link-types, glossaries, glossary-terms, etc. * The type of entries, aspects, and links to include into the local snapshot when downloading metadata from the service. * The subset of entries, aspects, and links to publish back to the service. **catalog.yaml** ``` scope: . # The resources in the snapshot. Examples: # scope: entryGroup... # scope: bq-dataset.. # scope: kb... # Or multiple BigQuery datasets in array list format: # scope: # - bq-dataset.. # - bq-dataset.. aliases: # Optional. Can always use 3-part fully qualified references. # NOTE: All built-in types have predefined simple aliases. ca-guidelines: aspect: data-agents-project.global.ca-guidelines ecommerce: glossary: data-gov-project.global.ecommerce-glossary # Defines the specific metadata to be retrieved locally from Knowledge Catalog. # NOTE: Required aspects of listed entry types are implicitly included. snapshot: entries: - bigquery-dataset - bigquery-table aspects: - overview - descriptions - queries - ca-guidelines entryLinks: - definition # Optional configuration to identify which types should be published. # This must be a subset of the types specified in the snapshot configuration. publishing: entries: aspects: - ca-guidelines entryLinks: - definition ``` **NOTE: Type References** Metadata content has several references to either the meta-model resources (entry, aspect, and link types) or other catalog constructs (such as glossaries). Syntax is provided to streamline these references. * References take the form of `..`. There is no need for referring to the verbose collection names themselves (projects//locations///) * Aliases can be defined to create single identifier names that are resolved when metadata is processed. * These are optional. Users can choose to use the fully-qualified references if that is the preference. * All built-in Dataplex types have pre-defined aliases without requiring the user to declare them. Eg. bigquery-table resolves to dataplex-types.global.bigquery-table. * Use project IDs, not project numbers when generating metadata. ### Entry File Names The name of the file is used to identify the Entry uniquely. The following rules are followed to derive file names from EntryIds. **Ingested Entries** use the service-qualified full resource name as EntryId. Inferrable segments (Service name and Location from EntryGroup) and Collection names (where possible) will be stripped out. | bigquery.googleapis.com/ projects/projectId/ datasets/datasetId/ tables/tableId | projectId.datasetId/tableId.yaml | Aligns with physical hierarchy. Common case for context use-cases. Avoid a type segment in directory path to optimize for most common resource when a collection has multiple types of sub-resources | | bigquery.googleapis.com/ projects/projectId/ datasets/datasetId/ routines/routineId | routines/projectId.datasetId/routineId.yaml | Moved into sub-directory to avoid potential conflicts between table and routine names | | cloudsql.googleapis.com/ projects/projectId/ locations/locationId/ instances/instanceId/ databases/databaseId | instanceId/databaseId/tableId.yaml | Example included to demonstrate how EntryIds and corresponding file names treat control plane and dataplane ids together | **3rd party Custom Entries** may use a similar REST API name of the resource (recommended when applicable) or arbitrary path-identifier (more likely scenario); We align the metadata file path to the Entry Id as there are no conventions to apply here. | Entry Id | File Path | NOTES | | :---- | :---- | :---- | | hive.walmart.com/ services/serviceId/ databases/databaseId/ tables/tableId | hive.walmart.com/ services/serviceId/ databases/databaseId/ tables/tableId.yaml | | | name-part1/name-part2 | name-part1/name-part2.yaml | Likely the common pattern for custom context overlays and bundles. We ensure that it is clean in directory layout. | | simple-name | simple-name.yaml | | ### File Extensions * **Standard Layout (for `bq-dataset` and `entryGroup` scopes)**: Entry files use the `.yaml` extension (e.g., `tableId.yaml`), with unstructured aspects optionally stored in sidecar `.md` files (e.g., `tableId.overview.md`). * **Document Layout (for `kb` scopes)**: Entry files use the `.md` extension (e.g., `tableId.md`), where the main YAML entry is replaced with a single markdown file containing YAML frontmatter and the promoted unstructured aspect (`Overview.content`) as its body. ### Entry File Layouts Metadata files are primarily structured representations of entry catalog info and aspects. Depending on the scope, the organization of these files differs: * **Standard Layout (YAML + Markdown sidecars)**: Used for `bq-dataset` and `entryGroup` scopes. Every entry is represented by at least a single YAML file (`.yaml`) that captures entry metadata, information about associated resource, and all aspects and links. Aspects containing unstructured rich-text fields (like `overview`) are split into sidecar markdown files (e.g., `.overview.md`). * **Document Layout (Markdown Only)**: Used for `kb` scopes. The main YAML file is replaced with a single markdown file (`.md`). The structured YAML metadata moves entirely to the YAML frontmatter of this markdown file, and the unstructured text of the `overview.content` aspect becomes the main markdown body of the file. #### Standard Layout **Entry Resource Info + Entry Source + Aspects (`entry.yaml`)** ```yaml id: # Entry metadata type: resource: # Entry.EntrySource name: displayName: description: labels: key: value location: parent: ancestors: createTime: updateTime: schema: fields: - name1: dataType: mode: … links: # EntryLinks associated with Schema.path are inlined definition: # into a Schema field to leverage context of field - target: glossary.term # specification : # In the general-case, each top-level field is an [aspect.data] # aspect. Nested field represents aspect.data links: # EntryLinks with this entry as source listed here : - target: : [aspect.data] ``` **Sidecar File per Aspect with Unstructured Text (`entry.overview.md`)** ```markdown --- field: value # YAML frontmatter used to capture structured metadata --- [aspect.data.rich-text-field] ``` #### Wiki Layout In the wiki layout, the YAML content is placed in the YAML frontmatter of a `.md` file, and the unstructured overview text (`overview.content`) is promoted to the markdown body. **Entry with Inlined Unstructured Text (`entry.md`)** ```markdown --- id: # Entry metadata in frontmatter type: resource: # Entry.EntrySource name: displayName: description: labels: key: value location: parent: ancestors: createTime: updateTime: schema: fields: - name1: dataType: mode: … links: definition: - target: glossary.term : # Other structured aspects [aspect.data] overview: # Unstructured aspect's metadata userManaged: true links: - title: url: <url> links: # EntryLinks with this entry as source <entryLink-type>: - target: <target-entry-reference> <aspect-type>: [aspect.data] --- # <displayName or entry-id> This is the main markdown body representing `overview.content` aspect data. ``` ## Examples ### Enrichment Agent This is an example for a BigQuery dataset and a table. For purposes of illustration, the user is interested in working the metadata that is used in Conversational Analytics, to enrich Overview documentation, Generated descriptions, Suggested Queries, Agent Guidelines and Glossary associations, and publish them back to Knowledge Catalog. **workspace/catalog.yaml** ``` scope: bq-dataset.ecommerce-prod.ecommerce-dataset resourceAliases: guidelines: aspect: data-agents-project.global.guidelines ecommerce: glossary: data-gov-project.global.ecommerce-glossary snapshot: entries: - bigquery-dataset - bigquery-table aspects: - overview - descriptions - queries - guidelines - data-profile # Included as local context, but not published - query-usage # Same as above entryLinks: - definition publishing: aspects: - overview - descriptions - queries - guidelines entryLinks: - definition ``` **workspace/catalog/ecommerce-prod.ecommerce-dataset.yaml** ``` id: bigquery.googleapis.com/projects/ecommerce-prod/datasets/ecommerce-dataset type: bigquery-dataset createTime: <createTime> updateTime: <updateTime> resource: name: projects/ecommerce-prod/datasets/ecommerce-dataset displayName: E-commerce Production Dataset description: ... labels: env: prod createTime: <entrySource.createTime> updateTime: <entrySource.updateTime> bigquery-dataset: ... descriptions: description: <generated dataset description> ``` **workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.yaml** ``` id: bigquery.googleapis.com/projects/ecommerce-prod/datasets/ecommerce-dataset/tables/orders type: bigquery-table createTime: <createTime> updateTime: <updateTime> resource: name: projects/ecommerce-prod/datasets/ecommerce-dataset/tables/orders displayName: E-commerce Orders description: ... labels: env: prod createTime: <entrySource.createTime> updateTime: <entrySource.updateTime> bigquery-table: ... storage: ... schema: fields: - name: id dataType: string mode: required links: definition: - target: ecommerce.order-number - name: status dataType: integer mode: required dataProfile: [data-profile-data] queryUsage: [top-fields and other usage aggregations] descriptions: description: <generated table description> fields: - name: id description: ... - name: status description: ... queries: queries: - description: ... sql: | [SQL Text] links: definition: target: ecommerce.transaction ``` **workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.overview.md** ``` --- userManaged: true links: - title: <title> url: <url> --- [overview.content] ``` **workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.guidelines.md** ```markdown --- userManaged: false --- [guidelines.instructions] ``` ### Knowledge Base Assume the user is managing a knowledge base of information managed in a hierarchical structure. Or alternatively, an agent is building this knowledge base from previously aggregated or enriched metadata. **workspace/catalog.yaml** ```yaml scope: kb.ecommerce-prod.global.mbr-kb snapshot: entries: - document aspects: - overview publishing: aspects: - overview ``` **workspace/catalog/products.md** ```markdown --- id: products type: document createTime: <createTime> updateTime: <updateTime> resource: name: products displayName: Products description: ... --- # Products Data This is the markdown body representing `overview.content` for the document. ``` **workspace/catalog/playbooks/mbr.md** ```markdown --- id: playbooks/mbr type: document resource: name: playbooks/mbr displayName: MBR Playbook description: ... --- # Ecommerce Monthly Business Review Playbook [overview.content] ``` --- ### Toolbox Mdcode Docs Design (`toolbox/mdcode/docs/design.md`) # Metadata as Code Design Specification This document details the overall design for the Metadata as Code project, factoring in the sub-system breakdown, API surfaces, CLI and MCP structures, technical stack, and key technical decisions. It builds upon the initial context provided in: - [readme.md](../README.md) - [concept.md](concept.md) - [spec.md](spec.md) - [plan.md](plan.md) ## 1. Sub-system / Module Breakdown The system is divided into two public interface layers and a core library foundation. ### 1.1 Public Interfaces - **CLI (`kcmd`)**: A single binary built using **Bun** that handles user interaction and parses commands using the `cac` framework. - **MCP Server**: Hosted within the same `kcmd` binary, triggered by the `mcp` command argument (e.g., `kcmd mcp`). It exposes sync operations as tools to external agents. ### 1.2 Core Library Foundation (`kc-mac`) The brain of the operation, containing the following modules: - **`CatalogManifest`**: Handles loading, validating, and parsing the `catalog.yaml` project manifest, including parsing the unified `scope` specification (which determines the layout style). - **`CatalogSnapshot`**: Manages local catalog entries and coordinates high-level read/write/list logic by delegating physical file representation and layout-specific tasks to `CatalogLayout`. - **`CatalogLayout`**: Abstract class defining the operations to list, read, write, and delete files on the local disk. Standard and Wiki layouts implement this interface, automatically selected based on the scope type. - **`CatalogSync`**: Orchestrates pull and push sync directions, handles pagination, fail-fast conflict detection, and manages tool state in a separate state file (`.catalog.state`). - **`CatalogService`**: Wrapper for raw HTTP API calls, handling Application Default Credentials (ADC) auth and LRO polling. ## 2. Project Directory Structure This section describes the directory structure for the project source code, reflecting the merge of interfaces, TypeScript library naming for future expansion, and the `gcp` subfolder for API functionality: - **`src/`** - **`tool/`**: The unified layer for CLI and MCP. - **`main.ts`**: The entry point for the unified binary (`kcmd`). Handles argument parsing to direct to either CLI commands or MCP server. Defines/handles CLI surface. - **`commands.ts`**: Contains functionality for CLI commands. - **`mcp.ts`**: Contains functionality for the MCP server and tools. - **`libts/`**: The TypeScript core library. - **`index.ts`**: The main entry point exporting public APIs. - **`manifest.ts`**: Contains the `CatalogManifest` class logic. - **`layout.ts`**: Contains the `CatalogLayout` interface. - **`source.ts`**: Contains the `CatalogSource` interface. - **`snapshot.ts`**: Contains the `CatalogSnapshot` class logic. - **`sync.ts`**: Contains the `CatalogSync` class logic. - **`gcp/`**: Subfolder for GCP-specific functionality. - **`api.ts`**: Contains the `Api` base class with polling and retries. - **`context.ts`**: Contains the `ApiContext` class. - **`dataplex.ts`**: Contains the API client for Knowledge Catalog. - **`sources/`**: Subfolder containing source-specific implementations. - **`entrygroup.ts`**: Encapsulates behavior for Dataplex EntryGroups. - **`bq-dataset.ts`**: Encapsulates behavior for BigQuery datasets. - **`wiki.ts`**: Encapsulates behavior for Wikis. - **`layouts/`**: Subfolder containing layout-specific implementations. - **`standard.ts`**: Encapsulates behavior for Standard layout. - **`documents.ts`**: Encapsulates behavior for Wiki layout. ## 3. Public API, CLI, and MCP Structure ### 2.1 Library API Surface **`CatalogManifest`** - `static initWithEntryGroup(entryGroup: string, ctx: ApiContext): CatalogManifest` - `static initWithBigQuery(dataset: string, ctx: ApiContext): CatalogManifest` - `static initWithKB(kb: string, ctx: ApiContext): CatalogManifest` - `static load(path: string, ctx: ApiContext): Promise<CatalogManifest>` - `save(path: string): void` **`CatalogLayout` (Abstract Class)** - `abstract list(snapshotPath: string): Promise<string[]>` - `abstract readEntry(snapshotPath: string, entryId: string): Promise<Entry>` - `abstract writeEntry(snapshotPath: string, entryId: string, entry: Entry): Promise<void>` - `abstract deleteEntry(snapshotPath: string, entryId: string): Promise<void>` **`StandardLayout` (extends `CatalogLayout`)** - Implements filesystem storage using a main `.yaml` file per entry and optional sidecar `.overview.md` files for unstructured aspects. **`WikiLayout` (extends `CatalogLayout`)** - Implements filesystem storage using a single `.md` file per entry with the metadata in the YAML frontmatter and the overview content as the markdown body. **`CatalogSnapshot`** - `static fromPath(path: string): CatalogSnapshot` - `layout: CatalogLayout`: The layout strategy instance initialized based on the manifest scope. - `list(): Promise<string[]>`: Returns a list of entry IDs in the snapshot by delegating to `layout.list(snapshotPath)`. - `lookupEntry(entryId: string): Promise<Entry>`: Returns structured metadata for a specific entry by delegating to `layout.readEntry(snapshotPath, entryId)`. - `updateEntry(entryId: string, updates: Record<string, any>): Promise<void>`: Merges updates into the entry metadata and aspect values, and commits the result via `layout.writeEntry(snapshotPath, entryId, entry)`. - `createEntry(entryId: string, data: Entry): Promise<void>`: Creates an entry locally by delegating to `layout.writeEntry(snapshotPath, entryId, data)`. - `deleteEntry(entryId: string): Promise<void>`: Deletes files associated with the entry by delegating to `layout.deleteEntry(snapshotPath, entryId)`. **`CatalogSync`** - `pull(): Promise<SyncResult>` - `push(options?: { force?: boolean, validateOnly?: boolean }): Promise<SyncResult>` - `validate(): Promise<ValidationResult>`: Fetches fresh definitions and performs pre-push validation. - `status(): Promise<StatusResult>`: Checks for local modifications against the saved checksum state. ### 2.2 CLI Command Structure (`kcmd`) - `kcmd init [--bigquery-dataset <id>] [--entry-group <id>] [--kb <id>]` - `kcmd pull [--dry-run]` - `kcmd push [--dry-run] [--force] [--validate-only]` - `kcmd status` - `kcmd mcp` (Triggers the MCP server) ### 2.3 MCP Tools and Params - **`pull`**: Params: None. - **`push`**: Params: `force?: boolean`. - **`list-entries`**: Params: None. - **`lookup-entry`**: Params: `entryId: string`. - **`update-entry`**: Params: `entryId: string`, and key/value pairs for each field and aspect being updated. ## 4. API Client Design & Context ### 3.1 `ApiContext` (Class) - `project: string` - `token: string` - `static default(): Promise<ApiContext>`: Uses `gcloud` to read project and token defaults. - `refresh(): Promise<void>`: Updates token via `gcloud auth print-access-token`. ### 3.2 `Api` (Base Class) - `constructor(endpoint: string, pathPrefix: string, context: ApiContext)` - **Methods**: `get`, `post`, `patch`, `delete` take relative resource names. - **Polishing**: - Fully encapsulates Long Running Operation (LRO) polling and returns only the completed result or error. - Implements transient error retries with exponential backoff (configurable defaults). - Automatically calls `context.refresh()` on token expiry (401). ## 5. Technical Stack and Key Dependencies - **Language**: TypeScript. - **Runtime/Build**: **Bun** for standalone binary creation and fast startup. - **Frameworks & Libraries**: - **CLI**: `cac` (minimal size). - **Schema Validation**: `zod` (TypeScript-first). - **Parsing**: `yaml`. - **HTTP Client**: Native `fetch` (in Bun) or `node-fetch` fallback. ## 6. Key Technical Decisions 1. **Bun Unified Binary**: The CLI and MCP server will be distributed as the same single binary to reduce complexity for the end-user. 2. **Thin HTTP Client**: We use native raw HTTP requests rather than Google SDKs to control code execution path and support environment workarounds. 3. **Validation Before Action**: Fresh type definitions are pulled at publishing time, and full validation completes *before* the first change payload hits the cloud. 4. **Fail-Fast Overwrites**: Overwrite protection fails fast when remote drift is identified. --- ### Toolbox Mdcode Docs Plan (`toolbox/mdcode/docs/plan.md`) # Phased Delivery Plan This document outlines the phased delivery plan for building the Metadata as Code foundation. ## Phase 1: MVP - Read-Only Snapshot & Consumption Establish the local file structure by pulling metadata using the TypeScript library, and enable reading that snapshot via MCP. Support early distribution. * **Key Features & Work:** * **Library (TS)**: Fetch metadata for a BigQuery dataset or Dataplex EntryGroup; create local directory structure mirroring the resource hierarchy; generate main YAML file per entry (Standard layout); paginated pull; ADC authentication. * **CLI (TS-based)**: Implement `kcmd init` and a basic read-only `kcmd pull`. * **MCP (TS-based)**: Implement an MCP server with tools: `list-entries` and `lookup-entry` (reading from the local snapshot). * **Distribution**: Support installation from source repository or local package for early access. * **Testing**: Implement test cases for snapshot creation and directory layout for BigQuery datasets and EntryGroups. ## Phase 2: MVP - Basic Push (Publish) Complete the bi-directional sync by enabling local modifications to be pushed back to the catalog using the TypeScript library. * **Key Features & Work:** * **Library (TS)**: Read local YAML files and reconstruct API payloads; push updates individually per entry; support parsing and filtering via `publishing` configuration in `catalog.yaml`. * **CLI (TS-based)**: Implement a basic `kcmd push`. * **Distribution**: Update early access packages with push capability. * **Testing**: Implement test cases for basic push operation and publishing configuration filtering. ## Phase 3: Enhanced Representation Optimize the file format for human and agent editing. Update interfaces to support the new formats and layout options. * **Key Features & Work:** * **Library (TS)**: Support `kb` scope type alongside `bq-dataset` and `entryGroup` in `catalog.yaml`; implement the `CatalogLayout` abstraction (`layout.ts`) and its concrete layouts (`StandardLayout` and `DocumentsLayout`); refactor `CatalogSnapshot` to delegate filesystem list, read, and write operations to the active layout strategy (automatically instantiated based on the scope); support type aliases in `catalog.yaml`, including built-in aliases for built-in types. * **CLI (TS-based)**: Update `pull` and `push` operations to handle sidecars, aliases, and different layout scopes correctly. * **MCP (TS-based)**: Ensure `list-entries` and `lookup-entry` tools correctly handle layouts (by delegating via `CatalogSnapshot` to `CatalogLayout` strategy). * **Testing**: Implement test cases for format mapping, including Standard layout (YAML + sidecars) and Documents layout (Markdown + frontmatter), and validate that `CatalogSnapshot` interacts correctly with both layouts based on the active scope type. Implement test cases for aliases. ## Phase 4: Robust Sync and State Management Ensure data integrity and efficient updates. * **Key Features & Work:** * **Library (TS)**: Store checksums of local state in a separate file; use checksums to detect changes; treat missing files as intent to delete; fail fast on remote modification; fetch type definitions dynamically for validation. * **CLI (TS-based)**: Add `kcmd status` to check for local modifications; update `kcmd push` to use checksums and report conflicts; support **force override** flag; implement `--dry-run` option for `pull` and `push` commands. * **Testing**: Implement test cases for checksum validation, intent to delete, validation against dynamic schema, and `--dry-run` behavior. ## Phase 5: Future / Other Other workstreams to consider. * **Key Features & Work:** * **Library (TS)**: Support for EntryLinks * **Python Library**: Implement the equivalent library in Python to support Python-based agents and workflows. * **MCP (TS-based)**: Implement `pull` and `push` tools in the MCP server. --- ### Toolbox Mdcode Docs Semantic Model (`toolbox/mdcode/docs/semantic-model.md`) # Deploying a semantic model This guide has moved into [`semantic-model/`](semantic-model/README.md) and is now split by what you came to do: - **[Deploy guide](semantic-model/README.md)** β€” author a model, push it, update it, pull it back. - **[Codelab: one semantic ontology, one data journey](semantic-model/codelab.md)** β€” author, govern, hydrate, and query one model, start to finish. - **[Reference](semantic-model/reference.md)** β€” every flag, what push creates in BigQuery Graph, Spanner Graph, and Knowledge Catalog (including class hierarchies), validation, and permissions. - **[Model specification](semantic-model/model_spec.md)** β€” the normative model format: every YAML construct, which parts are Apache Ossie and which are kcmd extensions or narrowings. - **[What push and pull preserve](semantic-model/fidelity.md)** β€” the round-trip fidelity matrix: what survives a deploy, what a pull recovers, and what neither keeps. - **[Importing an OWL ontology](semantic-model/owl-import.md)** β€” convert an OWL ontology into a semantic model, then deploy it the normal way. --- ### Toolbox Mdcode Docs Spec (`toolbox/mdcode/docs/spec.md`) # Metadata as Code Specification This specification builds upon the concepts in `docs/concept.md` and `readme.md`, detailing the agreed-upon design decisions, critical user journeys, glossary, and test cases for the Metadata as Code project. ## 1. Overview Metadata as Code provides a source code artifact-based UX for metadata management and context engineering in Knowledge Catalog (Dataplex). It is designed for agent builders, data stewards, data producers, and AI agents to author, manage, and enrich metadata using developer-friendly workflows with version control and CI/CD. ### 1.1. Primary Use Case The primary driver for the initial design is the **Knowledge Catalog Enrichment Agent**, focusing on automating metadata curation with human-in-the-loop review. However, the library is designed to be flexible and usable by any developer building agents or custom tools. ## 2. Glossary of Terms To ensure consistency, the following terms are used: * **Catalog Snapshot**: A directory on the local filesystem containing a subset of Knowledge Catalog metadata. * **Manifest (catalog.yaml)**: A file in the snapshot root that captures sync configuration and directives. * **Scope**: A unified identifier format (`<type>.<name>`) that establishes the single target resource for the snapshot in the manifest. * **Standard Layout**: A hybrid disk organization automatically used for `bq-dataset` and `entryGroup` scopes where metadata structure is stored in a YAML file per entry, and unstructured aspects are placed in sidecar Markdown files. * **Documents Layout**: A Markdown-only disk organization automatically used for `kb` scopes where the main entry is represented as a single Markdown file, with metadata structured in the YAML frontmatter and the main overview aspect promoted to the Markdown body. * **Entry**: A representation of a resource (like a table or dataset) in the Knowledge Catalog. * **Aspect**: A specific type of metadata attached to an entry (e.g., description, profile). * **Sidecar File**: An auxiliary file (e.g., Markdown) used for aspects with unstructured text. * **Source**: A backend metadata repository referenced by a scope (e.g., Dataplex EntryGroup or BigQuery Dataset). ## 3. Key Design Decisions The following core decisions have been established: ### 3.1. Metadata Representation The disk organization layout is automatically determined by the scope type: * **Standard Layout** (used for `bq-dataset` and `entryGroup` scopes): Structured data (entry metadata, resource info, aspects like schema and profile) is stored in a main YAML file per entry, while unstructured rich-text fields (like Overview) are stored in dedicated sidecar Markdown files. * **Documents Layout** (used for `kb` scopes): Structured metadata is stored within the YAML frontmatter of a single `.md` file per entry, and the primary unstructured aspect (`overview.content`) is promoted to serve as the main Markdown body of that file. Other unstructured aspects may still use sidecars. ### 3.2. Synchronization Model We will support a bi-directional sync (Pull and Push) with the following rules: * **Publishing is a subset** of the snapshot list. * While required aspects may be implicitly included in the snapshot, all entry types and aspect types that should be published **must be explicitly listed** in the manifest. ### 3.3. Conflict Handling To prevent data loss during a `push` operation: * The tool will **fail fast** if it detects that the metadata has been modified in the catalog in the interim. * It will abort the push, report the conflict, and require a `pull` to resolve. * A **force override** option will be provided to bypass this check if necessary. ### 3.4. Aliases Strategy To maintain consistency and steer users toward simpler names: * If an alias is defined in `catalog.yaml`, it **must be used consistently** in metadata files. * Otherwise, the full reference takes the form of `<projectId>.<locationId>.<resourceId>`. ### 3.5. Scale and Performance * **Paginated Pull**: Pull operations will be paginated to handle large resources. * **Individual Push**: Push operations will be performed individually per entry for now, with bulk import operations considered later. * **Scale Assumption**: Users are managing this as code artifacts in a repository and should not be operating at hyperscale/unbounded scale in this workflow. ### 3.6. Extensibility and Validation * **Dynamic Schema Fetching**: The tool will fetch type definitions from the Catalog service dynamically to validate local files against the actual registered schema. ### 3.7. Checksum Storage * **Separate State File**: Checksums for entries and aspects will be stored in a separate state file (e.g., `.catalog.state`) to separate user content from tool state. ### 3.8. Sync Deletions * **Intent to Delete**: If an entry type is listed in the publish config, treat missing files compared to state as intent to delete. * **Skip for Managed Entries**: Deletions are skipped for Dataplex managed entries (like BigQuery resources) which cannot be deleted. ### 3.9. Unified Scope Mapping * **Single String Identifier**: Snapshot resources are specified using a unified `scope` property formatted as `<type>.<name>` (e.g. `bq-dataset.<id>` or `entryGroup.<id>`). This unifies target definition logic and enables future extension to other resource types easily. ## 4. Critical User Journeys (CUJs) We focus on the following core journeys: * **CUJ 1: Initial Setup and Snapshot Creation** User runs `kcmd init` to create a local snapshot of a BigQuery dataset or Dataplex EntryGroup. * **CUJ 2: Metadata Sync (Pull and Push)** User pulls updates from the catalog service, makes modifications to local files, and pushes changes back to the catalog. *Note: Advanced workflows like Automated Curation with Human Review (CUJ 3 candidates) are built on top of this foundation using external tools (IDE, git, custom agents) and are out of scope for this base library.* ## 5. Key Test Cases To validate the design, we will implement test cases covering the following areas, including user feedback: ### 5.1. Resource Types * **Snapshots for different resource types**: BigQuery datasets and Dataplex EntryGroups. * **Entry and aspect varieties**: Both Dataplex-defined entries/aspects (where required aspects are read-only) and user-defined ones (where required aspects are modifiable). ### 5.2. Format Mapping * **Standard Layout** (for `bq-dataset` and `entryGroup` scopes): Aspects managed correctly within the main YAML file, and unstructured text mapped correctly to/from sidecar Markdown files. * **Documents Layout** (for `kb` scopes): Metadata mapped correctly to/from YAML frontmatter, and the `overview.content` aspect promoted to/from the Markdown file body. ### 5.3. Directory Organization * **BQ Resource Hierarchy**: Testing layout for BigQuery datasets. * **EntryGroup Hierarchy**: Testing layout for EntryGroups based on path segments in the entry ID. ### 5.4. Data Integrity * **Checksum validation**: Verifying that checksums are updated and validated for entries and aspects during both pull and push operations to detect conflicts. --- ### Samples Enrichment Sample Docs Events (`samples/enrichment/sample/docs/events.md`) Using Analytics Events Query a specific date range To query a specific date range from a BigQuery event export dataset, use the _TABLE_SUFFIX pseudo column in the WHERE clause of your query. For more info, view Filtering selected tables using _TABLE_SUFFIX. For example, the following query counts unique events by date and by event name for a specifc period of days and selected events: ```sql -- Example: Query a specific date range for selected events. -- -- Counts unique events by date and by event name for a specifc period of days and -- selected events(page_view, session_start, and purchase). SELECT event_date, event_name, COUNT(*) AS event_count FROM -- Replace table name. `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` WHERE event_name IN ('page_view', 'session_start', 'purchase') -- Replace date range. AND _TABLE_SUFFIX BETWEEN '20201201' AND '20201202' GROUP BY 1, 2; ``` --- ### Samples Enrichment Sample Docs Example1 (`samples/enrichment/sample/docs/example1.md`) # High-Level Event and User Volume This provides examples for kc_enrich_sample_data.ga_events table. ## Overview Before diving into deep analysis, it is best practice to understand the scale of your dataset. This query counts the total number of events, the total number of unique users (based on device cookies), and the number of days present in the dataset. ```sql SELECT COUNT(*) AS total_events, COUNT(DISTINCT user_pseudo_id) AS total_unique_users, COUNT(DISTINCT event_date) AS total_days FROM `kc_enrich_sample_data.ga_events` ``` --- ### Samples Enrichment Sample Docs Example2 (`samples/enrichment/sample/docs/example2.md`) # Querying a Specific Date Range This provides examples for kc_enrich_sample_data.ga_events table. ## Overview Because the GA4 export creates a new partition for every day, you should use the predicate on the partition timestamp column to limit the data scanned. This improves performance and drastically reduces query costs. This query finds the daily count of active users for December 2020. ```sql SELECT event_date, COUNT(DISTINCT user_pseudo_id) as active_users FROM `kc_enrich_sample_data.ga_events` WHERE event_date_dt BETWEEN PARSE_DATE('%Y%m%d', '20201201') AND PARSE_DATE('%Y%m%d', '20201231') GROUP BY event_date ORDER BY event_date ASC ``` --- ### Samples Enrichment Sample Docs Example3 (`samples/enrichment/sample/docs/example3.md`) # Session Analysis (Unnesting Parameters) This provides examples for kc_enrich_sample_data.ga_events table. ## Overview In GA4, a "session" is not a dedicated row; instead, every event contains a nested parameter called ga_session_id. To count total sessions, you must isolate the session_start event and "unnest" its parameters to extract the unique session ID. ```sql SELECT COUNT(DISTINCT value.int_value) AS total_sessions FROM `kc_enrich_sample_data.ga_events`, UNNEST(event_params) WHERE event_name = 'session_start' AND key = 'ga_session_id' ``` --- ### Samples Enrichment Sample Docs Example4 (`samples/enrichment/sample/docs/example4.md`) # E-Commerce Product Interest This provides examples for kc_enrich_sample_data.ga_events table. ## Overview For an e-commerce store, understanding which items are frequently added to the cart is crucial. E-commerce data is stored in a nested array called items. This query flattens the items array and aggregates the top 10 products based on the add_to_cart event. ```sql SELECT items.item_name, COUNT(*) AS added_to_cart_count FROM `kc_enrich_sample_data.ga_events`, UNNEST(items) AS items WHERE event_name = 'add_to_cart' GROUP BY items.item_name ORDER BY added_to_cart_count DESC LIMIT 10 ``` --- ### Samples Enrichment Sample Docs Lipsum (`samples/enrichment/sample/docs/lipsum.md`) Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. --- ### Samples Enrichment Sample Docs Usage (`samples/enrichment/sample/docs/usage.md`) # Using Google Analytics 4 BigQuery Public Dataset This provides usage information for kc_enrich_sample_data.ga_events table. ## How the Data Can Be Used This dataset is incredibly valuable for analysts, data scientists, and developers who want to learn how to query GA4 data in BigQuery without needing their own fully populated e-commerce site or GA4 export. ## Getting Started To interact with the dataset, you need: A Google Cloud Project with the BigQuery API enabled. You can use the BigQuery Sandbox, which provides a generous free tier of up to 1 TB of querying per month, allowing you to explore the data without needing to attach a credit card. ## Primary Applications Schema Familiarization: GA4 exports data in a highly nested format (using arrays and structs for event_params, user_properties, and items). This dataset allows you to practice the UNNEST() function to flatten data. Testing Queries & Data Models: Build and test advanced SQL data models for user retention, funnel analysis, or marketing attribution before applying them to your production data. Dashboard Prototyping: Connect this public BigQuery dataset to visualization tools like Looker Studio or Tableau to practice building e-commerce and web performance dashboards. --- ### Toolbox Mdcode Docs Semantic Model Actions (`toolbox/mdcode/docs/semantic-model/actions.md`) # Modeling write operations Your semantic model already tells an agent what your data means and which questions it can ask. Ask that agent to *do* something β€” refund an order, move money between two accounts, close a dormant account β€” and the model has nothing to say. The write exists somewhere, as a service call or an endpoint or a few lines of DML, but it's visible only to whoever built the service around it. Your agent either can't reach it, or reaches it through a hand-written tool that your model never sees and can't govern. An **action** closes that gap. It's a named write operation declared over the same concepts as the rest of your model: you give it a name, type its inputs against your ontology, and say which concepts a call changes. Publishing it puts the operation in the same place as the data it acts on, so an agent that discovers your model discovers what it can change there as well as what it can ask. One part of an action is physical β€” how the write actually happens. That part lives in a field called the **executor**, which names an MCP tool, a REST endpoint, a gRPC method, or DML. Keeping it separate lets a binding profile supply it, so you write an action across two files. Your model names the operation and its parameters; a binding profile names the tables, the columns, and the executor. Whatever runs the action combines the two into one bound model. ```mermaid graph LR M["the model<br>concepts, actions, constraints"] P["a binding profile<br>tables, columns, executor, target"] RT(["one model, bound<br>ready to run"]) ST["the store<br>where a write lands"] AG["what an agent is handed<br>write tools, instruction"] M --> RT P --> RT RT --> ST RT --> AG ``` *Figure 1: your model and a binding profile combine into one bound model, which both reaches your store and supplies the tools an agent is handed.* The rest of this guide follows one action β€” moving money between two bank accounts β€” from a name in your model through to an agent that can call it. A second example, a customer-service credit, comes in once the rules get harder to write. The commands here are `kcmd`, which reads a model, pushes it, and derives the tools an agent is handed under a binding profile. What the model states holds for any runtime that reads it, and the library `kcmd` is built on is the same one a service would embed. ## When to use an action Declare an action when the write already exists. Somewhere in your organization there's a service call, an endpoint, or some DML that moves this money or issues this credit. Declaring it as an action doesn't reimplement any of that β€” it tells everything that reads your model that the operation is there, what it takes, and where it lives. If the call should hand back an answer instead of changing something, you want a [metric](README.md#1-author-the-logical-model) instead. ## 1. Declare the action Start with the name and the parameters. They're the contract every later step builds on, and they live in your model and nowhere else β€” no profile can change them. Actions sit at model level, beside your metrics. Each one carries a name, its parameters, and an executor. Treat that executor as a default, because it's the one part a binding profile can replace: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` You author the rest of the model β€” the deployment target, the entity bindings, the relationships β€” the way you would for any model. See [Deploying a semantic model](README.md). Both `description` and `ai_context.instructions` travel through to the catalog, and a parameter's `description` is what tells a caller which argument is which. Two parameters a caller could confuse must each carry one, or the push fails. Write the instructions for the agent that's going to call the action, the way the example does. ### Projecting a parameter from a field Every parameter carries one scalar value β€” the same kind of value a column holds. What differs is where its definition comes from, and there are two ways to give it one: **project** it from a field of your ontology, or **declare** it on the spot. #### A projected parameter takes its definition from a field **A projected parameter** names a `concept` and a `field`, and takes that field's datatype, description, label and AI context straight out of the model. `source` above is an `Integer` because `Account.accountId` is one, so a caller passes an account id and the write binds it as an integer. Change the field later and every parameter projected from it follows, because there's no second copy to keep in step. A `concept` is read the way `affects` reads one, so it can name a relationship as well as an entity β€” but what you project from in practice is an entity, because a relationship only has fields of its own when a junction table backs it, and this format has no syntax for one yet. The pair can be written two ways, and they load to the same parameter: ```yaml - { concept: Account, field: accountId } - { field: Account.accountId } ``` The one-key form spells the projection the way a metric expression or a constraint judgment already spells a field, and is there for the same reason. A `field` that carries a dot is read as `Concept.field`, so a `field` with more than two parts is refused at load rather than read as half a reference, and so is a dotted `field` beside a `concept`, which would name the concept twice β€” that one is refused with both of the forms above spelled out for you. Most projected parameters need no name of their own. Leave `name` out and the parameter answers to the field's, so this one is called `accountId`: ```yaml - { concept: Account, field: accountId } ``` Name it yourself when one action takes the same field twice, the way `TransferFunds` takes both a `source` and a `target`. This shape holds however an entity is keyed. An `Account` identified by three columns takes three projected parameters, one per key field, each typed from the field it names β€” the same declaration a single-column key writes, three times. #### A declared parameter carries its own type **A declared parameter** carries a value no field holds: the `amount` above, a free-text memo, a reason code your store never keeps. Give it a `type` and a description of your own. `type` takes a scalar datatype only, and naming an entity there is an error telling you to project the field you meant. `datatype` is accepted as a spelling of the same key β€” the one `fields:` already uses β€” and stating both at once is refused rather than guessed at. A projected parameter can't restate its field's `type`. If the type is wrong, fix the field. If a statement needs a different one, cast it in the DML, where the conversion is visible to whoever reads the write instead of buried in metadata. #### What else a parameter may carry Each parameter may also carry: - **`description`** β€” what the parameter means for this call. A projected parameter inherits the field's, and overriding it is how `source` and `target` above say different things while projecting one field. Required when two parameters on the same action project the same field or share a declared type, because neither can tell an agent which argument is which on its own β€” and the inherited wording does not count, since two parameters carrying one field's description verbatim are as indistinguishable as two carrying none. - **`label`** and **`ai_context`** β€” inherited the same way, overridden the same way. - **`default`** β€” a fallback value substituted when the caller omits the argument. Giving a parameter a default makes it optional; setting `required: true` alongside `default` is rejected. - **`required: false`** β€” marks a parameter optional with no fallback; an omitted call binds `NULL` in SQL. `default` and `required` are always yours to set, projected or not. A field says what a value *is*; the parameter says how this one call uses it. A parameter that states neither is required, so the two keys above are the only ways to make one optional. ### Where the executor comes from The executor tells a consumer where the operation lives. Pick one of these four kinds, and give any one executor a single kind only: - **`mcp`** β€” `{server, tool}`. A tool you've already registered in Agent Registry, named by the server's resource name and the tool's name within it. - **`rest`** β€” `{endpoint, method}`. An HTTP endpoint and the verb to call it with. - **`grpc`** β€” `{service, method}`. A service and the method on it. - **`sql`** β€” `{statements}`. The write itself, carried in your model instead of named as a pointer to whoever performs it, and the only kind that needs a store to run against at all. See [carrying the write as DML](#carrying-the-write-as-dml). Everything else your action declares is logical: what it takes, what gates it, what it changes. None of that changes when you deploy the same model somewhere else. The executor does, which is why it sits on the physical side with an entity's `source`, and why a [binding profile](profiles.md) can supply one or replace the one your model declares β€” for any of the four kinds. An action a profile says nothing about keeps the executor the model gave it. It changes for two different reasons. An `mcp`, `rest` or `grpc` executor is an address: the operation lives at a different server, endpoint or service in staging than it does in production, and where your rows sit never enters it. A `sql` executor is the statement, written in the bound store's table and column names and in its dialect, so until you know which store answers there's nothing to write down. Only that last case turns on where the data lives. ### Actions declared but not performable An action with no executor anywhere β€” none in the model, none in any profile β€” is **declared but not performable**. It still says what it does, what it takes, what gates it and what it changes; the only thing missing is who carries the write out. Write one when nobody has wired the write up yet, or when another team owns it and your model needs only to record that it exists. Nothing declares this state: it follows from the executor being absent, so the same action is performable under a profile that supplies one and not performable under a profile that supplies none or withdraws the model's with `executor: null`. A catalog-only push β€” `--no-profile`, or a model with no deployment target β€” publishes it like any other action, and `kcmd profiles` lists it under `cannot run:` for each binding that supplies no executor for it. ## Carrying the write as DML The first three kinds name a system that performs the write, which leaves the write itself opaque to your model: an `mcp` tool name says where the operation lives and nothing about what it touches. A `sql` executor carries the write instead, so what your action does becomes readable β€” and checkable β€” from the bound model, and an agent or tool framework can [run it directly against your store](#7-hand-it-to-an-agent-kcmd-skills-generate) rather than handing the write to another system to perform. Statements are written in one database's own table and column names, in its own dialect, so a `sql` executor goes in that database's [binding profile](profiles.md) β€” beside the bindings for those same tables and columns β€” rather than in the model it binds. Here it replaces the `mcp` executor the model declared for `TransferFunds`: ```yaml # payments.profiles/operational.yaml β€” this store owns the rows, so it writes them semantic_model: - name: payments deployment_target: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/bank/graphs/payments entities: - name: Account source: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/bank/tables/account fields: - { name: accountId, expression: account_id } - { name: balance, expression: balance } # ... and the model's remaining entities and fields, bound the same way actions: - name: TransferFunds executor: sql: statements: - UPDATE account SET balance = balance - @amount WHERE account_id = @source - UPDATE account SET balance = balance + @amount WHERE account_id = @target - INSERT INTO transfer (transfer_id, amount, debited_account_id) VALUES (GENERATE_UUID(), @amount, @source) ``` A profile is an overlay on the model's bindings: a field it does not restate keeps the column the model file gives it. So the profile you put a `sql` executor in has to restate every column its store names differently, the ones its statements touch included β€” leave one out and it keeps the model's column, which this store may not have. Of the action itself, only the executor is restated: what the call takes, what gates it and what it changes stay in the model, exactly as [declared](#1-declare-the-action). `statements` is a list because one business action is often more than one write. The transfer above debits one account, credits another, and records the transfer row itself, and a transfer that did only the first would lose money. Running the statements in the order you wrote them inside one transaction commits them together at the end, so writes that only make sense together never apply by halves. Carrying the write buys you two things: - **Your blast radius is checkable.** A reader can compare `affects` against the statements instead of taking it on trust. - **A guard becomes a real gate.** An MCP, REST or gRPC call commits inside a system your caller doesn't control, so a write it performed can't be rolled back if the rest of the action fails. Because a `sql` action declares its statements up front, the agent or tool framework running it can settle its guards before opening a transaction, so a refusal leaves the store untouched. ### Statements use your database names **An action's statements reach your store exactly as you wrote them**, so every table and column in one has to be the name your database uses. In the `sql` executor above, the model's `Account` and `accountId` appear as `account` and `account_id`. Those are the table and the column that the entity's `source` key and its fields' `expression` keys bind it to. `@parameter` references are the exception: they name an argument bound at call time rather than anything in your database. Push checks those names for you, against the store itself. Before anything is published, every statement in a `sql` executor is sent to the database the profile binds β€” Spanner plans it, BigQuery dry-runs it β€” which parses it, resolves every table and column, and type-checks every parameter without executing a row of it. A statement that says `accountId` where the column is `account_id` fails the push: ``` Error: action 'TransferFunds' in model 'payments' (payments.profiles/operational.yaml): statement 1 of its sql executor was rejected by Spanner database 'my-project/my-instance/bank': Unrecognized name: accountId; Did you mean account? [at 1:54]. Statements reach the store exactly as written, so every table and column has to be the name the database uses. ``` Everything between the store's name and the closing sentence is the store's own answer, which is why it can point at column 54. It also explains the suggestion: Spanner offers `account`, the table, because that is the nearest name in scope where the statement went wrong β€” BigQuery, dry-running the same mistake, answers *Did you mean `account_id`?* instead. Take the diagnosis from the store you actually deploy to. That's also why the check is worth more than comparing your statements against the profile's bindings would be: the store catches what a name comparison can't see. A placeholder or a function borrowed from another dialect gets every identifier right and is still invalid. A parameter whose declared type doesn't fit the column it's compared to resolves every name correctly and fails on the type. An entity named `Order` bound to a table named `Orders` is the case worth recognizing, because the answer doesn't mention a table at all: ``` Syntax error: Unexpected keyword ORDER [at 1:8] ``` `ORDER` is a reserved word, so the statement stops parsing before anything goes looking for a table to write to. Nothing is written by any of this. Spanner plans inside a transaction that is never committed, and a BigQuery dry run doesn't execute the statement. What the check does need is access: it reaches the database your profile names, so a push from somewhere that can't reach it reports that rather than skipping the statements. ### Which file a `sql` executor belongs in The placement is enforced: select a profile whose model declares a `sql` executor and the command refuses, naming the action: ``` Error: [payments] profile 'operational': action 'TransferFunds' in model 'payments' declares a 'sql' executor. A statement names one database's own tables and columns, so it belongs in the profile that binds them, not in the model. Move the executor into each profile that performs this write as DML. ``` A single-file model is the exception: that one document is both the model and its binding, so its statements already sit beside the columns they name. Add a profile beside it later and they move into the profile. ## 2. Gate it with a constraint The sections so far declared an action and gave it a write to perform. This section is how you stop it running when it shouldn't. A **constraint** is a named rule your model states over its ontology. Declaring one adds it to the catalog and changes nothing by itself. A constraint takes effect where something references it and nowhere else, so publishing a rule can't quietly start refusing calls that succeeded yesterday. ``` declared referenced checked a breach ───────────────── ───────────────── ────────────── ─────────── constraints: actions: before the call, reject - name: X ──▢ - name: Y ──▢ with the ──▢ escalate judgment: … guards: [X] arguments bound warn a rule in the the only thing that a language model on_violation catalog, inert gives it effect reads the call names one of and settles it the three ``` *Figure 2: a constraint moves from declared, to referenced by an action, to checked before a call, to a breach routed by `on_violation`.* A constraint has one body, `judgment`: the rule written as a sentence, settled by a language model reading the attempted call. ### Naming a constraint as a guard An action's `guards` list is what puts a constraint to work. Declare the rule, then name it on the action β€” one line on the `TransferFunds` from section 1: ```yaml constraints: - name: TransferWithinAvailableBalance judgment: >- The amount argument of this call must not exceed Account.balance on the source account. That balance is on record rather than stated in the arguments, and nothing puts it in front of you. on_violation: reject description: >- A transfer cannot move more than the source account holds. Lower the amount, or choose another account. actions: - name: TransferFunds guards: [TransferWithinAvailableBalance] ``` `guards` holds the names of constraints your model declares, and listing one there is what makes it apply to that action. Put the reference on the action rather than on the constraint, because the same rule may gate `TransferFunds` and leave `CloseAccount` alone. That rule is the honest shape of a balance check and it is also one nothing here enforces: the balance is a row, a judge is shown the arguments and no more, and a rule it cannot settle it reports as not holding β€” which under `reject` refuses every call. Declared this way it is a policy on record and a schema constraint waiting to be written, not a control. Keep reading with that in mind; [what a guard can see](#what-a-guard-can-see) explains why. Whatever dispatches the call is what checks its guards, and it checks every one of them before the call, with the arguments bound and before any transaction opens. Nothing in your model binds a rule to the state a write leaves behind, so a guard reports on the arguments it was handed and on the data the call starts from, and on nothing else. `guards` and `on_violation` are independent, the way the two right-hand columns of figure 2 are: one says when the constraint gets checked, the other says what a breach does. So guarding a constraint that declares `warn` is a real shape. Your organization may not be ready to block on a rule; guarding it anyway still gets the rule checked at the moment of the call and reported back. When one call violates several guards, the strictest outcome applies. Any `reject` refuses the call, failing that any `escalate` holds it, and failing that any `warn` lets it through with the violations reported. The precedence is fixed and nothing in your model states it, so an action can name any number of guards without you writing down how they combine. It is also how `forbid` overrides `permit` in Cedar and how a deny wins in Open Policy Agent, so a policy written this way lowers into either. A guard that names no constraint fails the push, so a typo can't leave you believing a write is checked when nothing checks it. The reverse isn't an error: a constraint no action names is declared, published and inert, which is a reasonable thing to publish for a reader or for an action you haven't written yet. ### What a judgment costs A judgment is a whole constraint body. Here is one, with the two fields that say what a breach does and how grave it is: ```yaml constraints: - name: CreditMemoNamesAServiceFailure judgment: >- LineItem.memo must name a specific, verifiable service failure on the order: a late delivery, a damaged item, a shipping charge applied in error. A memo that states only that the customer requested a credit does not satisfy this rule. description: >- Say what went wrong with the order in the credit memo. on_violation: warn severity: low ``` Settling that rule is a model call, and one goes out for every guard on every attempted write. The verdict can also move: a ceiling of 25 dollars, stated as a judgment, can answer two identical calls differently, where `amount <= 25` could not. So write a judgment for a rule that has to weigh what a caller wrote, where nothing shorter would settle it. Two cases are worth keeping out. A threshold over the call's own arguments is the first: `amount <= 25` needs no reading at all, and a sentence buys you a model call and a verdict that can move. Leave it in whatever dispatches the call β€” the model has no home for it today, which is in [what is not modeled yet](#what-is-not-modeled-yet). A rule that turns on a stored value is the second. Nothing fetches that value for the judge, so the sentence gets settled against a figure the model supplied itself or refused for want of one. Write the rule down anyway if the model is where your policy lives, and enforce it in your schema β€” see [what a guard can see](#what-a-guard-can-see). Every constraint must state `on_violation`, and any of the three words will do. Omit it and the push fails, because an unmarked constraint would reject, and that's too strong a consequence to inherit by silence. ### What a guard can see A guard is checked before the call, with the arguments bound and before any transaction opens. It sees one thing: the arguments it was handed. A rule about the arguments β€” whether the memo the caller wrote names a real service failure, say β€” is settled completely, because the arguments are the whole of what it reads. Nothing settles a rule about a stored value the call doesn't carry. A judge is given the rule, the action and the arguments, and goes nowhere for anything else, so *the credit must not exceed the total of the order* has no total to compare against. The judge is instructed to treat a rule it cannot settle from the arguments as one that does not hold and to say in its reason what was missing β€” a safe failure, not a working check. Put that rule in your schema, where the store enforces it inside the transaction. Nothing settles a rule about the state a write *leaves behind*, either. "An order's total equals the sum of its lines" has nothing to look at when the guard runs, for a second reason: the state it describes does not exist yet. That one belongs in the same place. **Status: `judgment` is the only body a constraint has.** Whatever checks a guard puts the sentence to a language model and routes the verdict by `on_violation`. ### Writing a judgment A language model reads your sentence at review time with the proposed write in front of it. That reader is the judge. Five habits make that reading consistent: 1. **State what must be true of the data.** Write the condition β€” *the memo must name a specific service failure*, and not the procedure, *check whether the memo is specific*. Your sentence describes a clean write, and everything about handling a breach lives elsewhere. 2. **Name fields model-qualified.** Write `LineItem.memo` rather than "the memo". A push resolves every `Entity.field` token in the text against your model and fails when the entity declares no such field, so a rename can't leave your sentence pointing at nothing. It also tells a reader which value the rule is about, which the judge cannot go and fetch. 3. **Say what doesn't count.** A rule with no negative example gets graded against whatever the model guesses you had in mind. The sentence "A memo that states only that the customer requested a credit does not satisfy this rule" buys you more consistency than any further description of what a good memo is. 4. **Leave the consequence out of the prose.** What happens on a breach is `on_violation`. A judgment ending "…otherwise escalate to a supervisor" states a routing nothing reads, and the engine routes by the field regardless. 5. **Keep it to one condition.** When your sentence needs "and also", the second half is a second constraint. One `on_violation` can't carry two consequences, so two conditions that end differently can't share a constraint. All five are about wording. Claim no more in the wording than a judge can settle: phrase the condition around the arguments the call carries, and put any rule that depends on a stored row into your schema. ## A credit policy, worked through Real policies have several rules, and the rules rarely end the same way. A support agent is about to issue a customer-service credit, and five separate rules bear on whether they may: ``` the business rule a breach settled ────────────────────────────────────── ──────── ──────────────────────── 1 no credit above the order's total escalate judgment (policy; schema enforces) 3 the memo names a service failure warn judgment 4 not one credit split to evade review reject judgment ────────────────────────────────────── ──────── ──────────────────────── 2 over 25 dollars needs a supervisor β€” your dispatch, not here 5 the total equals the line items β€” your schema, not here ``` *Table 1: the three rules of the credit policy that a guard can carry, what a breach of each one does, and the two rules that belong somewhere else.* Rules 2 and 5 are in the table to be ruled out. Rule 5 is about the state a write leaves behind, and a guard runs before the transaction opens, so no constraint you write here reaches it; put it in your schema. Rule 2 compares one argument against a literal: written as a judgment it would buy a model call to perform a subtraction, and a verdict that can move at the boundary. Leave it in whatever dispatches the call. The 25-dollar ceiling still governs the desk, and rule 4 is written against it, but nothing in the model states it. Writing the other three down needs an `Order` with a `total`, a `LineItem` with an `amount` and a `memo`, and an `IssueCredit` action taking the order, the amount and the memo. Each rule becomes one constraint, and each states its own outcome in `on_violation`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Each `on_violation` key carries one branch of what your policy describes in prose, in a form a search can read. **Rules 3 and 4 are the reason a constraint is written in words.** Neither reduces to arithmetic over `Order` and `LineItem` at all, so there's nowhere else to put them but a policy document nothing links to. **Rule 1 is arithmetic against a row, and nothing here settles it.** It compares the amount argument against a number that lives in the database, and a judge sees the arguments only, so the rule is recorded in the model and enforced in your schema β€” see [what a guard can see](#what-a-guard-can-see). Rule 2 is the same comparison against a literal rather than a row, and that difference is the whole reason it stays out of the model. Rule 4 reads on the credits already sitting on the order, so it is the same case as rule 1. **Rule 4 is a judgment that declares `reject`.** Splitting a credit to evade review is a rule the business means as unappealable, and the pairing carries a real cost, because a model can decide two identical credits differently and `reject` leaves nobody to appeal to. The model publishes such a rule instead of forbidding it, and makes it findable: an auditor asking which unappealable rules your model settles gets an answer from one query over the constraints. ### Two calls through the policy Here are two calls against order 12345, which totals $165.85. One is a 200-dollar credit for a shipping charge billed in error, which is more than the order was ever worth. The other is three 9-dollar credits raised within the hour, each memo reading some version of "customer asked": ``` amount=200.00, amount=9.00 x3, "shipping charge "customer asked" applied in error" ────────────────────────────── ───────────────────── ───────────────────── 1 within the order's total violated ─▢ escalate holds 3 memo names a failure holds violated ─▢ warn 4 not split to evade review holds violated ─▢ reject ────────────────────────────── ───────────────────── ───────────────────── strictest outcome wins held for a supervisor refused, with the memo warning reported ``` *Table 2: how each of the two calls fares against the three rules, and the outcome that wins.* The supervisor who gets the first call reviews a credit against an order rather than a SQL diff. The second call is the case rule 4 was written for. Every per-call gate lets it through, because each 9-dollar credit sits under the order's total on its own, and under the desk's 25-dollar ceiling too, and only reading the three together as one 27-dollar credit puts them over it. The bottom row of table 2 applies the strictest-outcome rule from section 2 β€” an `escalate` with nothing stricter beside it holds the first call, and rule 4's `reject` decides the second. **Status: a run doesn't compute the strictest outcome.** The runtime puts the guards to the judge in the order your model declares them and stops at the first one that fails without being advisory. What comes back is that guard's outcome rather than the strictest of them, and a `warn` collected on the way there doesn't travel with the refusal. ## 3. Say what it changes `affects` names the concepts a call writes to. An executor like `mcp: {server, tool}` says where the operation lives and nothing more, so no reader of your model can see what that tool writes; the blast radius of a call stays unknown until you declare it here. `affects` is a list, and every entry in it takes one of two shapes: a bare concept name, or a record that also says how the concept changes. **A bare name** says the concept is touched, in a way your model doesn't spell out: ```yaml affects: [Account, Transfer] ``` That's enough to answer *which actions can change an account at all*, which is already more than an executor name answers. **A record** adds the operation and the fields the call writes: ```yaml affects: - concept: Account operation: modify fields: [balance] - concept: Transfer operation: create - concept: TransferDebits operation: create ``` You'll usually have worked some concepts out further than others. The two shapes go in one list together, so a coarse entry on the ones you haven't costs you nothing on the ones you have: ```yaml affects: - { concept: Account, operation: modify, fields: [balance] } - Transfer ``` `affects` declares the blast radius rather than limiting it. Naming one field doesn't stop a statement from writing others, and naming two concepts doesn't stop a call from touching a third; the list records what you mean the call to do. ### What an entry may say `concept` takes an entity or a relationship, written the same way for either kind β€” your model already records which one it is. `TransferDebits` above is the edge from the example model, and it sits in the list beside the two entities. Either way the name has to be one the same model declares. `operation` is `create`, `modify` or `delete`, the same three whatever the concept is, and a fourth word means your document doesn't parse. Leaving it out gives you the coarse form one entry at a time, covering every operation on that concept. Two entries on one concept with the same operation are a hard load error, and a bare entry beside one with an operation warns, because the bare one already covered what the second narrows. `fields` names what a `create` or a `modify` writes, which makes *which actions can change `Account.balance`* answerable. A `delete` takes the whole instance, so a field named beside one is rejected rather than ignored. On a relationship, `fields` means the junction table's own columns, so *modify the grade on an Enrollment* is as ordinary a change as *modify an order's total*. A plain foreign-key edge carries no columns of its own, and `TransferDebits` is one, so naming a field on it is an error β€” the property you meant belongs to an endpoint entity. Name the concept now and refine it later. Writing `- concept: Account` on its own says the same thing the bare `Account` does, and it's written back as the bare form. **Status: nothing compares `affects` to what your executor does.** kcmd parses it, checks the concepts against your ontology, publishes it and reads it back, and no component reconciles the declaration with the statements or the tool call. Two things beyond those checks read it. Resolving your model through a binding profile drops any action affecting a concept the profile can't bind, so that action never reaches the catalog. Publishing an action whose affected concept has no entry in the same push warns you that the catalog now records a blast radius naming something it can't resolve. ## 4. Check it before pushing `kcmd push --validate-only` runs the checks a push runs and deploys nothing. It loads your model, checks each action against the ontology that same model declares, asks your store about any DML those actions carry, and prints every error it finds. A push clears the same gate before it writes anything, so whatever this reports would have stopped your deploy: ```bash kcmd push --validate-only ``` Once your document parses, five checks run over every action, and a failure in any of them stops the push: - **Every parameter has a type the model can settle** β€” projected from a field, or a scalar datatype you named. A `concept` the model doesn't declare, a `field` that concept doesn't have, a `type` sitting on a projected parameter, and a `type` naming an entity are four different mistakes and report as four different errors. - **Parameters a caller could confuse each carry a `description`.** Two parameters projecting `Account.accountId`, or two sharing a declared type, otherwise produce identical tool argument documentation, leaving an agent guessing which is `source` and which is `target`. - **The executor has the fields its kind requires.** `server` and `tool` for `mcp`, `endpoint` and `method` for `rest`, `service` and `method` for `grpc`, at least one statement for `sql`. Leave one blank and whatever picks the action up has nothing to call. - **Every name in `guards` is a constraint the model declares.** This is the one that costs you silently β€” a guard naming a constraint that doesn't exist gates nothing, so you believe the write is checked while nothing checks it. - **Every `affects` concept is an entity or relationship the model declares.** The list exists to answer *which actions change `Account`*. A typo is invisible on the page and would drop the action out of that answer. An action carrying a `sql` executor clears one more gate, and that one reads your database rather than your document: every statement is planned against the store your profile binds, as [described above](#statements-use-your-database-names). A model that breaks all five reports all five, one line each, and deploys nothing: ``` Error: action 'TransferFunds' in model 'payments' (payments) has parameter 'amount' typed 'Currency', which is not a scalar datatype (String/Integer/Decimal/Float/Boolean/Date/Time/DateTime/DateTimeTz/Opaque). Error: action 'TransferFunds' in model 'payments' (payments) has multiple parameters projected from 'Account.accountId', so parameters 'source' and 'target' must each have a 'description' of their own to distinguish them. Error: action 'TransferFunds' in model 'payments' (payments) has an mcp executor whose 'tool' is missing or blank. Error: action 'TransferFunds' in model 'payments' (payments) is guarded by 'TransferWithinAvailableBalence', but model 'payments' declares no constraint of that name. Error: action 'TransferFunds' in model 'payments' (payments) affects 'Acount', which is neither an entity nor a relationship this model declares. ``` `TransferWithinAvailableBalence` there misspells the `TransferWithinAvailableBalance` the model declares, and `Acount` misspells `Account` β€” the two typos a reader of the document would skim straight past. kcmd checks the rest of an `affects` entry just as strictly. Fields beside a `delete` are a hard error, and so is a field the concept doesn't declare. An operation outside `create` / `modify` / `delete` never gets this far β€” the vocabulary is closed, so your document doesn't parse at all. The two checks that read your ontology stand down when a binding profile resolves your model. Resolving drops entities and relationships the profile can't bind, so holding `affects` to the ontology there would fail your deploy over a concept the profile removed rather than one you mistyped. An undeclared concept and an undeclared field fall back to the warning the loader already gave. A catalog-only push reads the author's model whole, so it treats the same two as hard errors. Fields beside a `delete` read only the entry, so that one fails everywhere. ### What push holds a statement to kcmd checks a `sql` executor further than the other three kinds, because it carries the write rather than a pointer to whoever performs it: - Write each statement as a **single `INSERT`, `UPDATE` or `DELETE`**. A statement that reads is a query and belongs in a metric; one that reshapes the schema isn't an action. A `;` anywhere but the end is rejected, because each list entry runs on its own and anything after the separator would silently not run. - **Every `@name` a statement binds has to resolve** to a parameter your action declares. Anything you write another way is left alone β€” a literal passes, and so does a SQL function. Nothing is interpolated into the statement text, so an argument can't become SQL. - **You choose where a new row's key comes from.** Write `GENERATE_UUID()` into the `VALUES` list, pass the key in as an ordinary parameter, or, where the key column has a default, leave it out and let the store fill it. kcmd generates nothing on your behalf, and nothing in `affects` bears on the choice. - **A key that arrives as an argument is a key the caller chooses.** An agent filling that parameter can name a row that already exists, and nothing in kcmd stops it, because the check on statement shape reads only the first word. An upsert therefore passes and overwrites. Spanner spells one `INSERT OR UPDATE` and AlloyDB spells one `INSERT ... ON CONFLICT`. A plain `INSERT` fails on the duplicate instead. Generate the key inside the statement unless something about the call needs the caller to pick it. - Nothing else is available: no control flow, and no statement composed at call time. `statements` is a fixed list in your model, so an action whose body arrived with the call would declare nothing, and a gate can't check what was never declared. Every one of those reads the statement text. The last check reads your database: as [described above](#statements-use-your-database-names), the store your profile binds parses, resolves and type-checks each statement before the push publishes anything, so a name the database doesn't have stops the push rather than the first agent to call the action. ## 5. Push it ```bash kcmd push ``` Knowledge Catalog is the one system your action reaches. Each action becomes its own entry there, parented to the model entry, the same way a metric does. The entry carries one aspect holding the executor and the typed parameters: ```yaml # .../entryGroups/<group>/entries/payments.actions.TransferFunds entryType: projects/<project>/locations/global/entryTypes/semantic-action parentEntry: .../entryGroups/<group>/entries/payments entrySource: displayName: TransferFunds description: Move money from one account to another. aspects: <project>.global.semantic-action: executorKind: mcp mcpServer: //agentregistry.googleapis.com/projects/acme-ops/locations/us-central1/mcpServers/payments mcpTool: transfer_funds parameters: - {name: source, type: Integer, concept: Account, field: accountId, description: The account the money leaves.} - {name: target, type: Integer, concept: Account, field: accountId, description: The account the money goes to.} - {name: amount, type: Float} affects: - {concept: Account, operation: modify, fields: [balance]} - {concept: Transfer, operation: create} - {concept: TransferDebits, operation: create} instructions: Resolve both accounts before calling. ``` A projected parameter publishes both halves: the `concept` and `field` it came from, and the `type` that projection settled on. The reference is what you wrote and what a pull reads back, so the link into your ontology survives the round trip. The resolved type is there for a consumer reading the catalog without your model in hand, which would otherwise have no way to know what to pass. `affects` is published exactly as you wrote it. The entry doesn't record whether `TransferDebits` is an entity or an edge. A consumer that needs that answer reads it off your model, the only place that still answers correctly after a rename. Every action in your model gets an entry, whether or not the binding you pushed can perform it. When that binding gives an action no executor, the entry publishes without the executor fields, so the catalog records an action that nothing is yet wired up to run. A push reads the binding named by `--profile`, or by `default_profile` in `catalog.yaml` when you don't name one. Remove an action from your document and the next push deletes its entry, because your model owns the `<model>.actions.` id prefix. A catalog search can list the actions in a project by entry type, the way it lists entities or metrics. `semantic-action` is a custom entry type, because Dataplex has no built-in type for an action yet. Run `kcmd init --semantic-model` once before you push a model that declares actions, and it provisions the entry type and its aspect type in your own project. Publishing an action needs the permission to attach its aspect, on top of the permissions any push needs β€” see [Reference β†’ Permissions](reference.md#permissions). ## 6. Pull it back ```bash kcmd pull ``` Pull collects the `semantic-action` entries under your model entry and rebuilds each action. The name, the description, the executor, the typed parameters, `guards`, `affects`, and `ai_context.instructions` all survive the round trip unchanged. [What push and pull preserve](fidelity.md) lists which parts of a model survive that trip and which don't. ## 7. Hand it to an agent (`kcmd skills-generate`) You don't hand-write prompts or tool wrappers for each agent framework. Run `kcmd skills-generate` against your model and binding profile, and `kcmd` writes an [Agent Skill](https://agentskills.io) β€” a folder of Markdown files that Claude Code, Gemini CLI, Cursor, and other skill-compatible agents load directly: ```bash kcmd skills-generate --out skills ``` ``` Wrote skills/payments/SKILL.md Wrote skills/payments/references/transfer-funds.md ``` The generated skill splits into two layers so an agent only loads the detail of the action it actually needs: 1. **`SKILL.md` (the router)** β€” loaded when the skill triggers. It carries the model's description, a table of actions pointing to `references/*.md`, how the model wants to be used (`ai_context.instructions`), how to read keys from the bound store (`Finding a record`), and the deployment details for the selected profile (`Running an action`). 2. **`references/<action>.md` (one reference page per action)** β€” loaded on demand when the agent picks an action. For `TransferFunds`, `references/transfer-funds.md` carries: ```markdown # TransferFunds Action `TransferFunds` of the `payments` model. As a tool it is named `transfer_funds`. Move money from one account to another. ## Arguments | Name | Type | Required | What to pass | | --- | --- | --- | --- | | `source` | integer | yes | The account the money leaves. | | `target` | integer | yes | The account the money goes to. | | `amount` | number | yes | How much money to move. | ## How to call it Resolve both accounts before calling. ## Rules that apply to this call Each is settled before anything is written, from the attempted call and, where the rule is about something on record, the record. ### TransferWithinAvailableBalance On violation: `reject` > The amount argument of this call must not exceed Account.balance on the source > account. That balance is on record rather than stated in the arguments, and > nothing puts it in front of you. If it does not hold: A transfer cannot move more than the source account holds. Lower the amount, or choose another account. ## What it changes | Concept | Operation | Fields | | --- | --- | --- | | `Account` | `modify` | `balance` | | `Transfer` | `create` | `sourceAccountId`, `targetAccountId`, `amount`, `settledOn` | ``` ### Where each part of the skill comes from Every section of `SKILL.md` and `references/<action>.md` maps directly to a key in your model or binding profile: ``` the model where it lands in the generated skill ───────────────────────────────── ───────────────────────────────────────── name, description, actions[].name ───▢ SKILL.md frontmatter + action router table ai_context.instructions ───▢ SKILL.md "## How this model wants to be used" entities[].fields ───▢ SKILL.md "## Finding a record" (physical table & column map for looking up keys) actions[]: name, description ───▢ references/<action>.md header & summary parameters ───▢ references/<action>.md "## Arguments" ai_context.instructions ───▢ references/<action>.md "## How to call it" guards (constraints[]) ───▢ references/<action>.md "## Rules that apply to this call" affects ───▢ references/<action>.md "## What it changes" the binding profile where it lands in the generated skill ───────────────────────────────── ───────────────────────────────────────── deployment_target ───▢ SKILL.md "## Finding a record" + "## Running an action" (Store) actions[].executor ───▢ SKILL.md "## Running an action" (Executor & whether each action is runnable) ``` *Table 3: which key in your model or binding profile produces each section of the generated Agent Skill.* Two sections in `SKILL.md` come from the generator itself: - **The tool-contract rules in `How this model wants to be used`** β€” appended after your model's `ai_context.instructions` so every skill states the same rules once: never invent an identifier, ask the caller or read the store when given a name instead of a key, repeat refusal reasons plainly, and always report warnings alongside a successful write. - **`Not runnable under this profile` in `Running an action`** β€” when an action cannot be run under the selected profile, its reference page is still generated in full, and `SKILL.md` lists the action with the reason why: 1. The profile supplies no executor (none declared, or withdrawn with `executor: null`). 2. The executor is `sql` and the profile binds no store at all. 3. The action names a non-advisory guard that is undeclared in `constraints` or has an empty `judgment`. For all `kcmd skills-generate` flags (`--out`, `--name`, `--profile`, `--force`) and how the deployment-specific section stays isolated to `SKILL.md`, see [Generating an Agent Skill](skills.md) and the full end-to-end walkthrough in `demo/semantic-model/skill/`. ## What is not modeled yet This is a prototype. Four things you might reasonably expect are absent. - **Nothing checks that a parameter lands in the column it was projected from.** Projecting `source` from `Account.accountId` says what the value is; it says nothing about where a statement puts it. Push narrows this to the same-type case: the parameter types your action declares are sent with the statement, so the store rejects `WHERE region = @source` when `region` is a string and `source` projects an integer β€” see [statements use your database names](#statements-use-your-database-names). What survives is a wrong column of the right type. `WHERE branch_id = @source` passes, because the parameter is logical, the column is physical, and nothing joins the two beyond the type they happen to share. Whoever writes a statement still owns which column each argument lands in. - **A deterministic check over the call's own arguments has no home.** A constraint is settled by a model reading it, so `amount <= 25` written as a guard costs a model call and can come back differently twice β€” see [what a judgment costs](#what-a-judgment-costs). The place for a check like that is likely the action itself; until there is one, keep it in whatever dispatches the call, and whoever wrote the statement owns the correctness of what it does. - **Nothing here performs the write.** An action becomes a tool in a generated skill, and the agent framework holding that skill is what calls it: a `sql` action carries its statements, and an `mcp`, `rest` or `grpc` one carries the coordinates of the service that performs it. No code in this repository dispatches either. - **Only a `sql` action needs a store.** Its statements are written against the database your profile's deployment target names, so a profile binding no store lists its `sql` actions as not runnable and says why. Which backend that store is does not matter: Spanner, AlloyDB and BigQuery all execute DML. A profile's `mcp`, `rest` and `grpc` actions need no store and are offered as tools either way. --- ### Toolbox Mdcode Docs Semantic Model Codelab (`toolbox/mdcode/docs/semantic-model/codelab.md`) # Codelab: one semantic ontology, one data journey A self-contained walkthrough. You author a semantic model, govern it in Knowledge Catalog, and bind it to physical stores β€” BigQuery and Spanner β€” to query it in each. The model is defined once and every store reads from that single definition. The expected output is shown after each command so you can check as you go. For the deploy mechanics on their own (author, push, update, pull), see the [deploy guide](README.md); for every flag and permission, see the [reference](reference.md). --- ## Setup You need the `gcloud` and `bq` CLIs, and `kcmd` on your `PATH` (build it from source: see [Build](../../README.md#build)). `kcmd` uses your `gcloud` configuration for credentials, project, and region: ```bash gcloud auth application-default login gcloud config set project "<your-project>" gcloud config set compute/region "<your-region>" ``` The commands below reuse a few names. Set them once for your own project: ```bash export PROJECT=$(gcloud config get-value project) # used in table and graph names export LOCATION=global # Knowledge Catalog entry-group location export DATASET=datacloud_demo # BigQuery dataset + KC entry group export GRAPH=sales # property-graph name ``` --- ## 1. Author the semantic model `init` provisions the Knowledge Catalog entry group and a local workspace: ```bash mkdir -p ~/semantic-model-codelab && cd ~/semantic-model-codelab kcmd init --semantic-model $PROJECT.$LOCATION.$DATASET # -> scope: semantic-model.$PROJECT.$LOCATION.$DATASET ``` Author the model in two parts β€” this is what lets one model serve many stores: - The **logical model** (this step) declares the business: three entities (`orders`, `customer`, `lineitem`), the relationships between them, and one metric (`revenue`). It names nothing physical. - A **binding profile** then says where each entity reads from and which column each field is. You write one for each store the model serves, and the logical model itself never changes. ### Author by hand Write the logical model. It declares the entities, their fields, the relationships between them, and the metric: ```bash cat > catalog/EntryGroups/$DATASET/sales.yaml <<'YAML' version: "0.2.0.dev0/google" semantic_model: - name: sales description: Orders, line items, and customers for the codelab entities: - name: orders primary_key: [order_id] fields: - { name: order_id, datatype: Integer } - { name: customer_id, datatype: Integer } - { name: net_amount, datatype: Decimal } - name: customer primary_key: [customer_id] fields: - { name: customer_id, datatype: Integer } - { name: name, datatype: String } - name: lineitem primary_key: [line_id] fields: - { name: line_id, datatype: Integer } - { name: order_id, datatype: Integer } relationships: - name: placed_by from: orders to: customer from_columns: [customer_id] to_columns: [customer_id] - name: part_of from: lineitem to: orders from_columns: [order_id] to_columns: [order_id] metrics: - name: revenue datatype: Decimal expression: dialects: [{ dialect: BIGQUERY, expression: SUM(orders.net_amount) }] YAML ``` The model describes three entities joined by two relationships: ```mermaid classDiagram direction LR class orders { order_id : integer customer_id : integer net_amount : decimal } class customer { customer_id : integer name : string } class lineitem { line_id : integer order_id : integer } lineitem --> orders : part_of orders --> customer : placed_by ``` ### Import from an OWL ontology You can also start from an existing OWL ontology instead of hand-authoring this YAML. `kcmd owl import` converts an ontology (`.ttl`) into a semantic model: - an OWL **class** becomes an **entity**; - a **datatype property** β€” a class attribute typed by an `xsd:` datatype such as `xsd:integer` β€” becomes a **field**; - an **object property** β€” one whose values are another class β€” becomes a **relationship**. Here is the same `sales` domain written as an ontology. An ontology is an RDF graph: classes, properties, and datatypes are all nodes wired together by labeled arcs. Each property is its own node, pointing to the class it describes (`rdfs:domain`) and to the type of its values (`rdfs:range`). Node color marks each resource's `rdf:type`: ```mermaid graph TD classDef cls fill:#dae8fc,stroke:#6c8ebf,color:#000; classDef dp fill:#d5e8d4,stroke:#82b366,color:#000; classDef op fill:#ffe6cc,stroke:#d79b00,color:#000; classDef dt fill:#f5f5f5,stroke:#999,color:#333; orders["ex:orders"]:::cls customer["ex:customer"]:::cls lineitem["ex:lineitem"]:::cls order_id(["ex:order_id"]):::dp customer_id(["ex:customer_id"]):::dp net_amount(["ex:net_amount"]):::dp name(["ex:name"]):::dp line_id(["ex:line_id"]):::dp placed_by(["ex:placed_by"]):::op part_of(["ex:part_of"]):::op xInt["xsd:integer"]:::dt xDec["xsd:decimal"]:::dt xStr["xsd:string"]:::dt orders -. owl:hasKey .-> order_id customer -. owl:hasKey .-> customer_id lineitem -. owl:hasKey .-> line_id order_id -->|rdfs:domain| orders order_id -->|rdfs:domain| lineitem order_id -->|rdfs:range| xInt customer_id -->|rdfs:domain| orders customer_id -->|rdfs:domain| customer customer_id -->|rdfs:range| xInt net_amount -->|rdfs:domain| orders net_amount -->|rdfs:range| xDec name -->|rdfs:domain| customer name -->|rdfs:range| xStr line_id -->|rdfs:domain| lineitem line_id -->|rdfs:range| xInt placed_by -->|rdfs:domain| orders placed_by -->|rdfs:range| customer part_of -->|rdfs:domain| lineitem part_of -->|rdfs:range| orders ``` Write that ontology and import it: ```bash cat > /tmp/sales.ttl <<'TTL' @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . @prefix ex: <http://example.com/sales#> . ex:orders a owl:Class ; rdfs:comment "A customer order" ; owl:hasKey ( ex:order_id ) . ex:customer a owl:Class ; rdfs:comment "A customer who places orders" ; owl:hasKey ( ex:customer_id ) . ex:lineitem a owl:Class ; rdfs:comment "A line on an order" ; owl:hasKey ( ex:line_id ) . ex:line_id a owl:DatatypeProperty ; rdfs:domain ex:lineitem ; rdfs:range xsd:integer . # order_id and customer_id each apply to two classes; a datatype property with # more than one rdfs:domain becomes a field on each of them. ex:order_id a owl:DatatypeProperty ; rdfs:domain ex:orders, ex:lineitem ; rdfs:range xsd:integer . ex:customer_id a owl:DatatypeProperty ; rdfs:domain ex:orders, ex:customer ; rdfs:range xsd:integer . ex:net_amount a owl:DatatypeProperty ; rdfs:domain ex:orders ; rdfs:range xsd:decimal . ex:name a owl:DatatypeProperty ; rdfs:domain ex:customer ; rdfs:range xsd:string . ex:placed_by a owl:ObjectProperty ; rdfs:domain ex:orders ; rdfs:range ex:customer . ex:part_of a owl:ObjectProperty ; rdfs:domain ex:lineitem ; rdfs:range ex:orders . TTL kcmd owl import /tmp/sales.ttl --out /tmp/sales_osi.yaml --compact ``` ``` converted 3 classes, 2 object properties, 5 datatype properties wrote /tmp/sales_osi.yaml ``` Look at what it produced: ```bash cat /tmp/sales_osi.yaml ``` ```yaml version: 0.2.0.dev0/google semantic_model: - name: sales description: Imported from OWL ontology http://example.com/sales# entities: - name: orders primary_key: [order_id] description: A customer order fields: - {name: order_id, datatype: Integer} - {name: customer_id, datatype: Integer} - {name: net_amount, datatype: Decimal} - name: customer primary_key: [customer_id] description: A customer who places orders fields: - {name: customer_id, datatype: Integer} - {name: name, datatype: String} - name: lineitem primary_key: [line_id] description: A line on an order fields: - {name: line_id, datatype: Integer} - {name: order_id, datatype: Integer} relationships: - {name: placed_by, from: orders, to: customer} - {name: part_of, from: lineitem, to: orders} ``` This is the hand-authored `sales` model above, reproduced from the ontology: the same three entities, the same fields and datatypes, the same primary keys, and the same two relationships. (The importer emits the extended profile β€” `entities:` under `version: 0.2.0.dev0/google` β€” the same spelling this codelab's hand-authored model uses.) Two things from the hand-authored model are missing, and both are inherent β€” an ontology has no way to state either: - **The `revenue` metric.** OWL describes structure β€” classes, properties, relationships β€” not aggregations, so the importer emits no metrics. Add the metric to the model by hand. - **The relationship join columns.** An object property states a direction (`orders` β†’ `customer`), not which foreign-key column joins to which key, so each edge arrives as a pure logical edge β€” `from`/`to` only. Add the `from_columns`/`to_columns` to each relationship by hand, as in the hand-authored model above. Both gaps are logical facts you fill in on the model itself. The *physical* facts β€” the table each entity reads, the column each field maps to β€” stay out of the logical model entirely; a [binding profile](profiles.md) supplies them at deploy time (step 3), which is why neither model carries a `source`. `kcmd push` publishes the logical model to Knowledge Catalog as-is. For the full OWL mapping β€” class hierarchies, unique keys, and the constructs carried as custom extensions β€” see [Importing an OWL ontology](owl-import.md). The rest of this codelab uses the hand-authored `sales` model above. --- ## 2. Govern it in Knowledge Catalog You can govern the model right now. The push writes the logical model straight to the catalog as entries; you do not need to connect it to any database tables first. ### Preview the plan Preview the plan without writing anything: ```bash kcmd push --validate-only --print ``` A bare `kcmd push` governs the logical model in Knowledge Catalog. ``` Validating semantic model for Knowledge Catalog... -- Knowledge Catalog -- Knowledge Catalog plan for 'sales' (destination $PROJECT.$LOCATION.$DATASET): 6 entries: - sales (semantic-model) - sales.entities.orders (semantic-entity) - sales.entities.customer (semantic-entity) - sales.entities.lineitem (semantic-entity) - sales.metrics.revenue (semantic-metric) - sales.files.sales.yaml (semantic-model-file) 2 schema-join links: - sales-placed-by - sales-part-of Validation complete; no changes applied. ``` ### Write the entries (available later) Then drop `--validate-only` to perform the write: ```bash kcmd push ``` ``` Pushing semantic model (Knowledge Catalog)... Wrote 6 new and 0 updated Knowledge Catalog entries; linked 2 relationships. ``` Each entity, the metric, and the model itself are now governed entries, joined by a schema-join link β€” discoverable, access-controlled, and the single definition the rest of your work reads from. `kcmd pull` returns the model YAML push stored with these entries, confirming the round-trip. Physical bindings can be governed here too, alongside the logical model. > This write needs the `semantic-model` / `semantic-entity` / `semantic-metric` > entry types and write access to the entry group. See > [Permissions](reference.md#permissions). --- ## 3. Deploy to BigQuery and get reliable insights In step 2 you governed the logical model, which created catalog entries but no tables and no data. To query it, you give the model a physical home in BigQuery. This takes three steps: write a **binding profile** that maps each entity to a BigQuery table, create those tables and load a little data, then deploy the model to build the graph. ### Write the binding profile Write the **analytical** binding: the BigQuery table each entity reads, the column each field maps to, and the BigQuery graph to deploy to. The `deployment_target` key names that graph. A binding lives beside the model in `sales.profiles/`: ```bash # BigQuery deployment target, named by the profile's deployment_target key. BQ_DS=projects/$PROJECT/datasets/$DATASET TARGET=bigquery.googleapis.com/$BQ_DS/propertyGraphs/$GRAPH mkdir -p catalog/EntryGroups/$DATASET/sales.profiles cat > catalog/EntryGroups/$DATASET/sales.profiles/analytical.yaml <<YAML version: "0.2.0.dev0/google" semantic_model: - name: sales deployment_target: $TARGET entities: - name: orders source: bigquery.googleapis.com/$BQ_DS/tables/orders primary_key: [o_orderkey] fields: - { name: order_id, expression: o_orderkey } - { name: customer_id, expression: o_custkey } - { name: net_amount, expression: net_amount } - name: customer source: bigquery.googleapis.com/$BQ_DS/tables/customer primary_key: [c_custkey] fields: - { name: customer_id, expression: c_custkey } - { name: name, expression: c_name } - name: lineitem source: bigquery.googleapis.com/$BQ_DS/tables/lineitem primary_key: [l_linekey] fields: - { name: line_id, expression: l_linekey } - { name: order_id, expression: l_orderkey } relationships: - { name: placed_by, from_columns: [o_custkey], to_columns: [c_custkey] } - { name: part_of, from_columns: [l_orderkey], to_columns: [o_orderkey] } YAML ``` The binding carries only bindings, in this store's column names: each entity's table and key columns, each field's column, and each relationship's join columns. The relationships, the metric and the grain themselves are logical and live once in the model. Make it the default so a bare `kcmd push` selects it (the rest of this step relies on this): ```bash echo 'default_profile: analytical' >> catalog.yaml ``` > **Simple case β€” one binding, one file.** If a model only ever binds to one > store, you do not need a separate profile. Put the `deployment_target`, each > entity's `source`, and each field's `expression` directly on the model in > `sales.yaml` β€” that is the `default` profile β€” and run a bare `kcmd push`. An > `orders` entity then reads: > > ```yaml > semantic_model: > - name: sales > deployment_target: bigquery.googleapis.com/.../propertyGraphs/sales > entities: > - name: orders > source: bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/orders > primary_key: [o_orderkey] > fields: > - { name: order_id, datatype: Integer, expression: o_orderkey } > # ... > ``` > > This codelab keeps the binding in a separate profile so one model can back more > than one store, each with its own binding. ### Inspect the binding profile `kcmd profiles` lists the model's binding profiles, reports what each one can answer, and prints any error push would give for it. Run it to confirm the binding before you deploy: ```bash kcmd profiles ``` ``` Model 'sales' ($DATASET): profile 'analytical' (default) target: bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/sales sources: orders -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/orders customer -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/customer lineitem -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/lineitem cannot answer: nothing withheld. ``` Only the `analytical` profile exists so far, and it binds every field, so it withholds nothing: the whole model, including the `revenue` metric, is answerable under this binding. ### Create the tables Now create the tables and load a little data. In a production pipeline, an agent working from the ontology would generate these tables from raw sources; for this self-contained codelab, you create them directly. Two details about the data matter here: the `orders` table stores `net_amount` as a column, which the `revenue` measure sums, and each order has several `lineitem` rows: ```bash bq mk -f --dataset $PROJECT:$DATASET bq query --use_legacy_sql=false ' CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.customer` AS SELECT * FROM UNNEST([STRUCT(1 AS c_custkey, "Acme" AS c_name), STRUCT(2, "Globex")]); CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.orders` AS SELECT * FROM UNNEST([ STRUCT(100 AS o_orderkey, 1 AS o_custkey, 90.0 AS net_amount), STRUCT(101, 1, 200.0), STRUCT(102, 2, 40.0) ]); CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.lineitem` AS SELECT * FROM UNNEST([ -- order 100: 2 lines, 101: 1, 102: 3 STRUCT(1 AS l_linekey, 100 AS l_orderkey), STRUCT(2, 100), STRUCT(3, 101), STRUCT(4, 102), STRUCT(5, 102), STRUCT(6, 102) ]);' ``` > **Why the tables come before the BigQuery push.** Unlike the Knowledge Catalog > step, deploying the BigQuery graph validates that every entity's `source` table > resolves in BigQuery β€” even under `--validate-only` β€” and builds the graph over > these tables. So the tables must exist first. (Step 2 needed none of this: it > governed the logical model, no tables required.) ### Deploy the semantic model Now deploy the bound model to BigQuery. The graph backend comes from the model's deployment target β€” you don't name it on the command line. You already governed the model in Knowledge Catalog in step 2, so `--no-kc` deploys just the graph; `--print` shows the generated DDL: ```bash kcmd push --no-kc --print ``` The generated graph turns each entity into a node table, each relationship into an edge, and the metric into a measure. Each property is the model's logical name backed by its bound column β€” `o_orderkey AS order_id` β€” so every query names `order_id`, never the physical column (your `$PROJECT`/`$DATASET` appear in the fully qualified names): ```sql Pushing semantic model (BigQuery Graph)... -- BigQuery Graph -- -- bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/$GRAPH CREATE OR REPLACE PROPERTY GRAPH `$PROJECT.$DATASET.$GRAPH` NODE TABLES ( `$PROJECT.$DATASET.orders` AS orders KEY(o_orderkey) PROPERTIES( o_orderkey AS order_id, o_custkey AS customer_id, net_amount, MEASURE(SUM(net_amount)) AS revenue ), `$PROJECT.$DATASET.customer` AS customer KEY(c_custkey) PROPERTIES( c_custkey AS customer_id, c_name AS name ), `$PROJECT.$DATASET.lineitem` AS lineitem KEY(l_linekey) PROPERTIES( l_linekey AS line_id, l_orderkey AS order_id ) ) EDGE TABLES ( `$PROJECT.$DATASET.orders` AS placed_by KEY(o_orderkey) SOURCE KEY(o_orderkey) REFERENCES orders(o_orderkey) DESTINATION KEY(o_custkey) REFERENCES customer(c_custkey), `$PROJECT.$DATASET.lineitem` AS part_of KEY(l_linekey) SOURCE KEY(l_linekey) REFERENCES lineitem(l_linekey) DESTINATION KEY(l_orderkey) REFERENCES orders(o_orderkey) ); Deployed 1 BigQuery Graph(s). ``` ### View the semantic model The graph is now a resource in your dataset, and the console draws its schema as a diagram of node and edge tables β€” easier to read than the DDL above. Print the BigQuery Studio link to the dataset: ```bash # printf with a single-quoted format string: the `!` are literal, not bash history expansion. printf 'https://console.cloud.google.com/bigquery?project=%s&ws=!1m4!1m3!3m2!1s%s!2s%s\n' "$PROJECT" "$PROJECT" "$DATASET" ``` Open the link, expand the `$DATASET` dataset in the Explorer, and click the `$GRAPH` property graph. Its schema renders as a visual graph of the nodes and the edges that connect them. ### Query Semantic Model: Metrics Now ask the same question β€” revenue by customer β€” two ways. **Before** β€” hand-written SQL. An analyst who wants revenue "with the line-item detail" writes the obvious join across all three tables: ```bash bq query --use_legacy_sql=false --nouse_cache ' SELECT c.c_name, SUM(o.net_amount) AS revenue FROM `'"$PROJECT.$DATASET"'.customer` c JOIN `'"$PROJECT.$DATASET"'.orders` o ON o.o_custkey = c.c_custkey JOIN `'"$PROJECT.$DATASET"'.lineitem` li ON li.l_orderkey = o.o_orderkey GROUP BY c.c_name ORDER BY c.c_name' ``` The result is **wrong**. Joining in `lineitem` repeats each order's `net_amount` once per line, so revenue is inflated by the line count β€” Acme's `90 + 200` becomes `90Γ—2 + 200Γ—1 = 380`, Globex's `40` becomes `40Γ—3 = 120`: ``` +--------+---------+ | c_name | revenue | +--------+---------+ | Acme | 380.0 | <- should be 290 | Globex | 120.0 | <- should be 40 +--------+---------+ ``` Nothing errors. The query succeeds and returns a total that is too large. This is the fan-out trap: any consumer that writes its own join can hit it. **After** β€” through the model's `revenue` measure. A measure is a *symmetric aggregate*: it sums each order once however the graph is traversed, so the fan-out cannot double-count. You read it by flattening the graph with `GRAPH_EXPAND` and wrapping the measure in `AGG()`, with columns named `<node>_<field>`. The query names no table, no join, and no formula: ```bash bq query --use_legacy_sql=false --nouse_cache ' SELECT customer_name AS name, AGG(orders_revenue) AS revenue FROM GRAPH_EXPAND("'"$PROJECT.$DATASET.$GRAPH"'") GROUP BY name ORDER BY name' ``` This one is **right**: ``` +--------+---------+ | name | revenue | +--------+---------+ | Acme | 290.0 | | Globex | 40.0 | +--------+---------+ ``` Same question, same data: the hand-written join inflates the total, the measure returns the correct one. `revenue` is now an enforced, queryable concept. The join and the formula come from the model rather than from whoever writes the query, so the correct answer is the default one. > **Tips.** Use `--nouse_cache` β€” `GRAPH_EXPAND` result caches are not > invalidated by graph edits. To see the exact output column names for a graph: > `DECLARE s STRING; CALL BQ.SHOW_GRAPH_EXPAND_SCHEMA("$PROJECT.$DATASET.$GRAPH", s); SELECT s;` ### Query Semantic Model: GQL Metrics answer aggregate questions. To follow the relationships between entities, query the graph with graph query language (GQL), which matches a pattern of nodes and edges. This walks the `placed_by` edge to count each customer's orders: ```bash bq query --use_legacy_sql=false --nouse_cache ' GRAPH `'"$PROJECT.$DATASET.$GRAPH"'` MATCH (o:orders)-[:placed_by]->(c:customer) RETURN c.name AS customer, COUNT(o.order_id) AS orders GROUP BY customer ORDER BY customer' ``` ``` +----------+--------+ | customer | orders | +----------+--------+ | Acme | 2 | | Globex | 1 | +----------+--------+ ``` This is standalone GQL. The pattern and the model's property names are portable across graph query engines; only the graph reference is engine-specific, and here BigQuery takes the fully qualified `$PROJECT.$DATASET.$GRAPH`. --- ## 4. Deploy the same model to Spanner The same customers and orders usually live in more than one store. Step 3 bound and deployed the `analytical` profile β€” the BigQuery warehouse. An operational Spanner database holds the same business, but its tables are named differently (`Customers`, `Orders`, `LineItems`), its columns are named differently (`FullName`, `OrderId`), and it does not carry `net_amount` β€” a settled figure the warehouse computes rather than one the live store keeps. ### Write the binding profile Add a **second profile** beside the first. Like the `analytical` one, it changes only where each entity reads from and which column each field binds to; it never changes what an entity, a relationship, or a metric *means* (for the full contract, see [binding profiles](profiles.md)). Write the operational binding into the same `sales.profiles/` directory: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The profile carries only bindings, in Spanner's column names: each entity's table and key columns, each field's column, and each relationship's join columns. It also excludes the one field the store does not have, and every metric. The relationships, the metric, the labels and the grain themselves are logical and live once in the model. ### Inspect the binding profiles The model now has two profiles. `kcmd profiles` lists them both and reports what each one can answer: ```bash kcmd profiles ``` ``` Model 'sales' ($DATASET): profile 'analytical' (default) target: bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/sales sources: orders -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/orders customer -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/customer lineitem -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/lineitem cannot answer: nothing withheld. profile 'operational' target: spanner.googleapis.com/projects/$PROJECT/instances/$SPANNER_INSTANCE/databases/$SPANNER_DB/graphs/sales sources: orders -> spanner.googleapis.com/projects/$PROJECT/instances/$SPANNER_INSTANCE/databases/$SPANNER_DB/tables/Orders customer -> spanner.googleapis.com/projects/$PROJECT/instances/$SPANNER_INSTANCE/databases/$SPANNER_DB/tables/Customers lineitem -> spanner.googleapis.com/projects/$PROJECT/instances/$SPANNER_INSTANCE/databases/$SPANNER_DB/tables/LineItems cannot answer: field orders.net_amount (excluded) metric revenue (excluded) ``` Both profiles now show side by side: `analytical` binds every field, so it withholds nothing; `operational` excludes `net_amount`, and every metric. (`kcmd profiles` shows each source as its resource name; the Spanner graph itself references the bare `Orders`.) `revenue` is `SUM(orders.net_amount)`. The operational profile excludes `net_amount`, and a metric that reaches an excluded field has to be excluded too. The profile also deploys to Spanner, whose graphs have no measures, so it excludes every metric with `metrics_exclude: "*"`, which covers `revenue`. The `analytical` profile binds `net_amount`, so the same metric is available under the binding step 3 used. One model; each store answers the part of it that its data can back. ### Create the tables Create the operational tables in a Spanner ENTERPRISE-edition database. Spanner Graph requires the ENTERPRISE edition. Create the database with its three tables, then load a little data: ```bash gcloud spanner databases create $SPANNER_DB --instance=$SPANNER_INSTANCE \ --ddl='CREATE TABLE Customers (CustomerId INT64 NOT NULL, FullName STRING(MAX)) PRIMARY KEY(CustomerId)' \ --ddl='CREATE TABLE Orders (OrderId INT64 NOT NULL, CustomerId INT64) PRIMARY KEY(OrderId)' \ --ddl='CREATE TABLE LineItems (LineId INT64 NOT NULL, OrderId INT64) PRIMARY KEY(LineId)' gcloud spanner databases execute-sql $SPANNER_DB --instance=$SPANNER_INSTANCE \ --sql="INSERT INTO Customers (CustomerId, FullName) VALUES (1,'Acme'),(2,'Globex')" gcloud spanner databases execute-sql $SPANNER_DB --instance=$SPANNER_INSTANCE \ --sql="INSERT INTO Orders (OrderId, CustomerId) VALUES (100,1),(101,1),(102,2)" gcloud spanner databases execute-sql $SPANNER_DB --instance=$SPANNER_INSTANCE \ --sql="INSERT INTO LineItems (LineId, OrderId) VALUES (1,100),(2,100),(3,101),(4,102),(5,102),(6,102)" ``` > **Why the tables come before the Spanner push.** Deploying the Spanner graph > applies the DDL through the `updateDatabaseDdl` long-running operation; it does > not create the tables, and for a model like this one β€” which declares no > actions β€” it does not read them either. So the tables must exist first. ### Deploy the semantic model Now deploy the bound model to Spanner. The `operational` profile's deployment target is a Spanner Graph, so merging that profile is what sends the model to Spanner β€” no backend flag needed. `--no-kc` deploys just the graph, and `--print` shows the generated DDL: ```bash kcmd push --profile operational --no-kc --print ``` ``` Note: profile 'operational' leaves 1 field(s) unbound; 1 metric(s) and 0 action(s) unavailable. Pushing semantic model (Spanner Graph)... -- Spanner Graph -- -- spanner.googleapis.com/projects/$PROJECT/instances/$SPANNER_INSTANCE/databases/$SPANNER_DB/graphs/sales CREATE OR REPLACE PROPERTY GRAPH sales NODE TABLES ( Orders AS orders KEY(OrderId) PROPERTIES( OrderId AS order_id, CustomerId AS customer_id ), Customers AS customer KEY(CustomerId) PROPERTIES( CustomerId AS customer_id, FullName AS name ), LineItems AS lineitem KEY(LineId) PROPERTIES( LineId AS line_id, OrderId AS order_id ) ) EDGE TABLES ( Orders AS placed_by KEY(OrderId) SOURCE KEY(OrderId) REFERENCES orders(OrderId) DESTINATION KEY(CustomerId) REFERENCES customer(CustomerId), LineItems AS part_of KEY(LineId) SOURCE KEY(LineId) REFERENCES lineitem(LineId) DESTINATION KEY(OrderId) REFERENCES orders(OrderId) ); Deployed 1 Spanner Graph(s). ``` The graph still speaks the model's vocabulary β€” the properties are `order_id`, `name`, and the rest β€” but each is now backed by the profile's Spanner column (`OrderId AS order_id`, `FullName AS name`). The key and reference clauses name the physical columns (`KEY(OrderId)`), because a Spanner graph keys on the table's real column rather than the property alias. Three things also differ from the BigQuery DDL in step 3: - **Bare table and graph names.** A Spanner property graph names tables inside one database, so there is no backticked `project.dataset.` qualifier β€” the DDL names each table by the last part of its `source` (`Orders`). - **No `MEASURE`.** Spanner Graph has no measures, so a profile that deploys to Spanner excludes every metric, as `operational` does with `metrics_exclude: "*"`. Author metrics as usual; a BigQuery target still emits them. - **No `OPTIONS`.** Descriptions and synonyms are not written into the Spanner DDL; they live in Knowledge Catalog instead. > **Knowledge Catalog is unchanged.** You do not re-push to Knowledge Catalog for > the Spanner leg. Step 2 governed the *logical* model, and adding a binding > profile changes nothing logical β€” bindings are not governed in Knowledge > Catalog. The single set of entries from step 2 already describes this graph too. ### Query Semantic Model: GQL Query it with the same GQL query you ran on BigQuery in step 3, now against the bare `sales` graph. The query names the model's properties (`name`, `order_id`); the profile's column bindings are invisible to it: ```bash gcloud spanner databases execute-sql $SPANNER_DB --instance=$SPANNER_INSTANCE \ --sql="GRAPH sales MATCH (o:orders)-[:placed_by]->(c:customer) RETURN c.name AS customer, COUNT(o.order_id) AS orders GROUP BY customer ORDER BY customer" ``` ``` customer orders Acme 2 Globex 1 ``` The one model now backs two stores: the BigQuery warehouse from step 3, where `revenue` is a measure, and this operational Spanner graph, where the same entities and relationships are queried live and `revenue` is computed in SQL or the application rather than by the graph. `customer`, `placed_by`, and every field mean the same thing in both; only the bindings differ. > **Deploy every binding in one run.** This codelab pushed the two > profiles in separate steps to show each backend. With both defined, > `kcmd push --all-profiles` deploys every profile at once β€” the > analytical graph to BigQuery and the operational graph to Spanner β€” > and still records the default binding to Knowledge Catalog once. At the > other end, `kcmd push --no-profile` deploys no graph and publishes only > to Knowledge Catalog. Neither names a backend: each profile's > `deployment_target` selects it. --- ## 5. Model a class hierarchy with `extends` A customer and a supplier are both parties you deal with. When several entities are kinds of one thing, declare the general kind once and say each specific kind `extends` it. A query written against the general kind then reaches every specific kind. This step adds a `supplier` entity beside `customer` and lifts the shared idea into an abstract `party` supertype, then deploys the result to BigQuery. For the modeling guide, see [class hierarchies](inheritance.md); for the generation rules, see [Reference β†’ Class hierarchies](reference.md#class-hierarchies-extends--labels). ### Add the supertype and a second subtype Rewrite the logical model to add three things: an abstract `party` that declares the shared `name` field, an `extends: [party]` on `customer`, and a new `supplier` entity that also extends `party`. `party` is `abstract`, so it has no `source` and no key and produces no node table; it survives in the graph only as a label on its subtypes. Each subtype inherits `name`, with its datatype, from `party`, and the profile binds it to that subtype's own column. The two subtypes line up under the shared label by that name: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ```mermaid classDiagram class party { <<abstract>> name : string } class customer { customer_id : integer } class supplier { supplier_id : integer } party <|-- customer party <|-- supplier ``` Add the `supplier` binding to the `analytical` profile. `party` binds nothing β€” it has no table β€” so each subtype's own binding supplies the `name` column that its `party` label reads. Rewrite the profile to include `supplier`: ```bash cat > catalog/EntryGroups/$DATASET/sales.profiles/analytical.yaml <<YAML version: "0.2.0.dev0/google" semantic_model: - name: sales deployment_target: $TARGET entities: - name: orders source: bigquery.googleapis.com/$BQ_DS/tables/orders primary_key: [o_orderkey] fields: - { name: order_id, expression: o_orderkey } - { name: customer_id, expression: o_custkey } - { name: net_amount, expression: net_amount } - name: customer source: bigquery.googleapis.com/$BQ_DS/tables/customer primary_key: [c_custkey] fields: - { name: customer_id, expression: c_custkey } - { name: name, expression: c_name } - name: supplier source: bigquery.googleapis.com/$BQ_DS/tables/supplier primary_key: [s_suppkey] fields: - { name: supplier_id, expression: s_suppkey } - { name: name, expression: s_name } - name: lineitem source: bigquery.googleapis.com/$BQ_DS/tables/lineitem primary_key: [l_linekey] fields: - { name: line_id, expression: l_linekey } - { name: order_id, expression: l_orderkey } relationships: - { name: placed_by, from_columns: [o_custkey], to_columns: [c_custkey] } - { name: part_of, from_columns: [l_orderkey], to_columns: [o_orderkey] } YAML ``` ### Create the supplier table Create the `supplier` table beside the others and load two rows: ```bash bq query --use_legacy_sql=false ' CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.supplier` AS SELECT * FROM UNNEST([STRUCT(1 AS s_suppkey, "Northwind" AS s_name), STRUCT(2, "Initech")]);' ``` ### Deploy and read the labels Deploy the model again. The `orders`, `lineitem`, and edge tables are unchanged from step 3; the two node tables that matter now each carry the `party` label alongside their own default label. Each renders `name` from its own column: `c_name` on the customer table, `s_name` on the supplier table. The shared label reconciles by property name, so the two tables never have to agree on a physical column: ```bash kcmd push --no-kc --print ``` ```sql Pushing semantic model (BigQuery Graph)... -- BigQuery Graph -- -- bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/$GRAPH CREATE OR REPLACE PROPERTY GRAPH `$PROJECT.$DATASET.$GRAPH` NODE TABLES ( ... `$PROJECT.$DATASET.customer` AS customer KEY(c_custkey) DEFAULT LABEL PROPERTIES( c_custkey AS customer_id, c_name AS name ) LABEL party PROPERTIES( c_name AS name ), `$PROJECT.$DATASET.supplier` AS supplier KEY(s_suppkey) DEFAULT LABEL PROPERTIES( s_suppkey AS supplier_id, s_name AS name ) LABEL party PROPERTIES( s_name AS name ), ... ); Deployed 1 BigQuery Graph(s). ``` ### Query the supertype A query against `party` reaches both subtypes. Match every party and return its name: ```bash bq query --use_legacy_sql=false --nouse_cache ' GRAPH `'"$PROJECT.$DATASET.$GRAPH"'` MATCH (p:party) RETURN p.name AS name ORDER BY name' ``` Every customer and every supplier comes back, each once, because each real party lives in exactly one table: ``` +-----------+ | name | +-----------+ | Acme | | Globex | | Initech | | Northwind | +-----------+ ``` `MATCH (:customer)` still returns Acme and Globex alone, and `MATCH (:supplier)` Northwind and Initech alone: each subtype keeps its own label as well as the shared one. > **Spanner works the same way.** Inheritance deploys identically to Spanner > Graph β€” the extra labels and the flattened fields are the same on both > backends. To include suppliers on the operational Spanner graph from step 4, > add a `supplier` binding and a `Supplier` table to the `operational` profile > the same way you did here. Spanner carries no measures, as it does for any > model. --- ## 6. Clean up Drop the BigQuery dataset (tables, including `supplier`, plus the property graph): ```bash bq rm -r -f -d $PROJECT:$DATASET ``` Drop the Spanner database from step 4 (its tables and the graph go with it): ```bash gcloud spanner databases delete $SPANNER_DB --instance=$SPANNER_INSTANCE --quiet ``` Remove the Knowledge Catalog entries and entry group via REST. Delete the schema-join links first, while their endpoint entries still exist: a link whose entries are already gone is orphaned and can no longer be addressed by name. ```bash TOKEN=$(gcloud auth application-default print-access-token) DATAPLEX=${DATAPLEX_ENDPOINT:-https://dataplex.googleapis.com} EG=projects/$PROJECT/locations/$LOCATION/entryGroups/$DATASET for L in sales-placed-by sales-part-of; do curl -s -X DELETE -H "Authorization: Bearer $TOKEN" \ "$DATAPLEX/v1/$EG/entryLinks/$L" >/dev/null done for E in sales sales.entities.orders sales.entities.customer sales.entities.lineitem sales.metrics.revenue; do curl -s -X DELETE -H "Authorization: Bearer $TOKEN" \ "$DATAPLEX/v1/$EG/entries/$E" >/dev/null done curl -s -X DELETE -H "Authorization: Bearer $TOKEN" "$DATAPLEX/v1/$EG" ``` Remove the local workspace that step 1 created: ```bash rm -rf ~/semantic-model-codelab ``` --- ### Toolbox Mdcode Docs Semantic Model Fidelity (`toolbox/mdcode/docs/semantic-model/fidelity.md`) # What push and pull preserve Your model document is the source of truth. Pushing it β€” to Knowledge Catalog and to a property graph in BigQuery or Spanner β€” and pulling it back are each **lossy in specific ways**. The catalog keeps metadata, the graph keeps what it can query, and `pull` can only return what the catalog was given. This page is the one authoritative map of what survives each direction. Keep your authored document. Which backend a push deploys to β€” BigQuery or Spanner β€” is set by the model's deployment target, or by the binding profile you select; it is not a command-line flag. What reaches Knowledge Catalog depends on the push as well: a catalog-only or purely logical push records the whole model, while a push that also deploys a graph records only the part the graph binds β€” and whichever way you push, only one binding is recorded, even when the model defines several. All three are detailed under [To Knowledge Catalog](#to-knowledge-catalog). ## The round-trip matrix Rows are what you authored; columns are its fate in each direction. `βœ“` = comes back as authored; `β€”` = not present in that destination. Footnotes carry the nuances that don't fit a cell. BigQuery and Spanner each get a column; they agree on every structural row and differ only where a Spanner target has no `MEASURE` and no `OPTIONS` metadata, and in which SQL text a field reads β€” see [To Spanner](#to-spanner). | Authored element | β†’ Knowledge Catalog | Recovered by `pull` | β†’ BigQuery | β†’ Spanner | |----------------------------------------------------------------|---------------------------------|------------------------------------------------|----------------------------------------------------------------------|----------------------------------------------------------------------| | Entity (name, `source`) | `semantic-entity` entry¹⁰ | βœ“ | `NODE TABLE` | `NODE TABLE` | | Field | `schema` aspect column | βœ“ (type collapsesΒ²) | column on the node tableΒΉ | column on the node tableΒΉ | | Field `label` | `schema` per-field annotation | βœ“ | into `OPTIONS(description)` | β€” dropped | | Field expression (canonical SQL) | only with `--emit-expressions` | only if pushed with it | builds the DDL | builds the DDL | | Field dimension role (`is_time`) | only with `--emit-expressions`Β³ | only if pushed with itΒ³ | noted in `OPTIONS(description)` | β€” dropped | | Primary key | `schema.primaryKey` | βœ“ | `KEY(...)` on the node table | `KEY(...)` on the node table | | Unique keys | `schema.uniqueConstraints` | βœ“ | the first is `KEY(...)` when there is no primary key; else dropped | the first is `KEY(...)` when there is no primary key; else dropped | | Metric | `semantic-metric` entry | name, entity, description, instructions, type⁡ | `MEASURE`⁴ | β€” dropped (no `MEASURE`) | | Relationship (1:1 / 1:N) | `schema-join` link¹⁡ | βœ“ (name normalized⁢) | `EDGE TABLE` | `EDGE TABLE` | | Relationship (M:N / `association`) | β€” not stored | β€” | `EDGE TABLE` (via junction table) | `EDGE TABLE` (via junction table) | | Entity `extends` | β€” not modelled | β€” | `LABEL` clauses + flattened fields | `LABEL` clauses + flattened fields | | Action | `semantic-action` entryΒΉΒ² | βœ“ΒΉΒ² | β€” not deployedΒΉΒ² | β€” not deployedΒΉΒ² | | Constraint | `semantic-constraint` entryΒΉΒ³ | βœ“ΒΉΒ³ | β€” not deployedΒΉΒ³ | β€” not deployedΒΉΒ³ | | `description` (entity / metric / field / relationship) | entry description / aspect | βœ“ | `OPTIONS(description)` | β€” dropped | | `ai_context.synonyms` | β€” not stored | β€” | `OPTIONS(synonyms=[...])` | β€” dropped | | `ai_context.instructions` | `guidelines` aspect⁷ | βœ“β· | into `OPTIONS(description)` | β€” dropped | | `ai_context.examples` | β€” not stored | β€” | into `OPTIONS(description)` (`Examples:` line) | β€” dropped | | Model-level `description` / `instructions` | on the model entry | βœ“βΈ | β€” dropped⁸ | β€” dropped⁸ | | Model-level `ai_context.synonyms` / `examples` | β€” not stored | β€” | β€” dropped | β€” dropped | | Deployment target | recorded on the model entry | βœ“ | names the graph | names the graph | | Binding profile (alternate physical binding) | β€” only the selected one¹⁴ | —¹⁴ | one graph per BigQuery-bound profile | one graph per Spanner-bound profile | | Vendor-dialect `expression` variant (non-canonical `dialects[]` entry)ΒΉΒΉ | β€” not stored | β€” | β€” not used; a field or metric with only vendor text is left out | β€” not used; a field with only vendor text is left out | | `custom_extensions` from a vendor other than GOOGLE | β€” not stored | —⁹ | β€” not in graph | β€” not in graph | 1. **Field type source.** The graph uses the source column's own type; a field's authored `datatype` is not carried. 2. **Field type collapse.** Field types round-trip except one collapse: `String` β†’ un-typed. A field with no type and a `String` field both store `dataType STRING` (with `metadataType` `OTHER` and `STRING` respectively), and `pull` writes no `datatype` for either, so only `String` is lost. 3. **Dimension role.** The default push omits the per-field `semantics` block, so the dimension role is written only with `--emit-expressions` β€” and then comes back as a bare `dimension: {}` marker (without `is_time`, and so on). A default push drops the marker entirely. 4. **Metric shape.** A metric must resolve to exactly one entity, or the push is rejected. It must also reduce to one supported aggregate over one operand: `SUM`, `AVG`, `COUNT`, `MIN` or `MAX`. Otherwise the push is rejected. A push of a profile in the older `<model>.profiles/` layout skips the metric with a warning instead. 5. **Metric type.** A metric's expression is gated behind `--emit-expressions`; its data type round-trips only for a concrete type (e.g. `Decimal`) β€” an untyped, `String`, or `Opaque` metric comes back un-typed. 6. **Relationship name.** Relationship names come back lowercased (`Places_Order` β†’ `places_order`) β€” the catalog stores the name only in the link id. See [Writer-side follow-up](#writer-side-follow-up). 7. **Guidelines aspect.** The `guidelines` aspect exists only for the model, entities, and metrics β€” not fields or relationships, so field- and relationship-level `ai_context.instructions` has no Knowledge Catalog home (a relationship's instructions still reach BigQuery, folded into the edge's `OPTIONS(description)`). 8. **Model-level metadata.** Neither graph has a home for statement-level metadata β€” BigQuery silently drops graph-statement `OPTIONS`, and Spanner carries no `OPTIONS` at all β€” so the model's `description` and `ai_context.instructions` are carried into Knowledge Catalog instead. 9. **Other custom extensions.** A `custom_extensions` block from a vendor other than GOOGLE is inert on push and has no catalog entry or aspect of its own; `pull` returns it from the stored copy of the model file, and leaves it out only when rebuilding from the catalog metadata. The OWL importer emits none: it is import-only (see [Importing an OWL ontology](owl-import.md)). 10. **Logical (unbound) model.** A model with no bindings still publishes to Knowledge Catalog: each entity's `source` is recorded empty (`resources: []`) because there is no table behind it, and a relationship that carries no join columns is skipped with a warning. When the same push also deploys a graph, the catalog entries are first pruned to what the graph binds β€” see [To Knowledge Catalog](#to-knowledge-catalog). 11. **Vendor-dialect fallback.** What you author is the `expression.dialects[]` list. `importedExpression` / `importedDialect` are not authored keys β€” the loader *derives* them from a non-canonical dialect entry (for example the MAQL or Snowflake form a metric was imported from). `kcmd import --transpile` adds a `BIGQUERY` dialect entry to a model file's field or metric, translated from a vendor-only expression or from an `ANSI_SQL` expression that double-quotes column names. It leaves a profile's expressions as written. A BigQuery graph reads a field's or metric's `BIGQUERY` entry, else its `ANSI_SQL` one. It never emits the vendor text. A field with none of these is left out of its node table with a warning, though push rejects such a field in the model file first. A metric with none, or one that aggregates such a field, is skipped with a warning. A Spanner graph reads a field's `SPANNER` entry, else its `ANSI_SQL` one, and leaves out a field with neither. See [Model spec Β§2.5](model_spec.md#25-expressions). 12. **Actions.** An action reaches Knowledge Catalog only, as one `semantic-action` entry under the model entry, and `pull` reads it back from that entry. Every other push target deploys nothing for it and warns once. Its `guards` are stored as constraint names and round-trip verbatim. A name whose constraint is absent from a pull is kept rather than dropped, so a partial pull never silently rewrites the author's model; the pull warns about the name it kept, and a push rejects the model until the constraint is back. A name the aspect repeats is the one exception, dropped to its single occurrence because the loader rejects a repeat and the document has to stay loadable. Its `affects` round-trips the same way, with the same exception for a repeated concept-and-operation pair. See [Modeling write operations](actions.md). 13. **Constraints.** A constraint reaches Knowledge Catalog only, as one `semantic-constraint` entry under the model entry, and `pull` reads it back. Every other push target deploys nothing for it and warns once. Publishing is all that push does with a constraint; what settles one is a run, and the runtime an application embeds is what puts each guard to a judge before the write. 14. **Binding profiles.** A model may define several physical realizations, one per binding profile. `--all-profiles` deploys a graph for every profile that declares a deployment target, each to the backend its own target names; a profile without one is skipped, and two profiles claiming the same graph are rejected before anything deploys. Knowledge Catalog still records a single binding β€” the one named by `--profile`, else the default binding, which is what an `--all-profiles` run always records, the two flags being mutually exclusive. A [push in two phases](reference.md#push) records the model file as written instead. See [Binding profiles](profiles.md). 15. **Relationships with no link.** Knowledge Catalog keeps one `schema-join` link between two entries, so when several relationships join the same two entities, push links the first one it can publish and skips the rest with a warning. The catalog metadata holds no link for a skipped relationship. `pull` still returns it from the stored copy of the model file, and leaves it out only when it rebuilds the file from the metadata. ## To Knowledge Catalog The catalog holds the model's metadata and a copy of each model file and profile file. Every resource type it uses is a built-in system type under `dataplex-types/global`, apart from the custom `semantic-action`, `semantic-constraint` and `semantic-model-file` pairs that `kcmd init` provisions. A push references those types and never creates them (see [Reference β†’ What gets created in Knowledge Catalog](reference.md#what-gets-created-in-knowledge-catalog)). What is recorded depends on the push. A catalog-only push (`--no-profile`), or a purely logical model with no bindings, records the whole model: every entity, metric, and relationship the document declares, apart from the relationships in footnote 15. A [push in two phases](reference.md#push) does too. Any other push that also deploys a graph first prunes the model to what that graph binds, so an unbound field β€” and any entity, metric, or relationship that depends on it β€” is left out of the catalog entries too. A pull still returns the authored model file from its stored copy. Only a model file that pull rebuilds from the catalog metadata comes back as the bound view. A logical model still produces complete entries. Each entity's `source` is recorded empty (`resources: []`) because there is no table behind it, and a relationship that carries no join columns is skipped with a warning. **One binding, not all of them.** A model may define several physical realizations β€” an analytical binding onto BigQuery and an operational one onto Spanner, say β€” and `kcmd push --all-profiles` deploys a graph for each. The catalog leg does not fan out with it: it runs once, against the profile named by `--profile`, otherwise the default binding. So a bare `kcmd push` and a `kcmd push --all-profiles` write the same entries and differ only in how many graphs they deploy. A [push in two phases](reference.md#push) records the model file as written instead. Nor do successive pushes accumulate. Entry ids are derived from logical names alone β€” model, entity, metric β€” with no profile component, so pushing a second profile reconciles the catalog to *that* binding rather than adding to what the first one wrote: shared elements have their `source` swapped to the new profile's tables, and β€” on a push that deploys a graph, and so prunes β€” an entity or metric the new binding cannot answer is deleted rather than left behind. The catalog therefore describes one physical realization at a time. It records that binding's deployment target and tables, but never the profile's name, and never that the other profiles exist. A `pull` returns whichever was written last. Keep the profile files β€” the catalog is not their store. By default the catalog does **not** store the SQL expressions: the published system-type templates do not yet carry a per-field `semantics` block or a `semantic-metric.expression` field, so the default push omits them. Pass `--emit-expressions` to write the canonical GoogleSQL/ANSI expression once the templates gain the fields. The catalog never stores `ai_context.synonyms` / `examples`, field-level `ai_context` (only model, entity, and metric `instructions` have a home, in the `guidelines` aspect), or the original vendor SQL (`importedExpression` β€” for example the MAQL or Snowflake form a metric was imported from). Those stay in your authored document. The expressions build the graph SQL. **Actions** follow the same one-entry-per-element rule as everything else: each becomes a `semantic-action` entry under the model entry, carrying its executor, typed parameters, `guards`, and `affects` in a `semantic-action` aspect. They round-trip losslessly through `pull` (name, description, executor, typed parameters, `guards`, `affects`, and `instructions`). A parameter projected from a field round-trips as the projection it was authored as: the aspect stores `concept` and `field` alongside the resolved scalar `type`, and a pull writes the projection back out without the type, so re-loading resolves it from the same field and arrives at the same parameter. Two things about a projection do not survive the trip. Its `label` and `ai_context` are dropped β€” the aspect has nowhere to put them, so a pull recovers the parameter's `description` and not those two. And the wording a pull does recover comes back as the parameter's own: the loader resolves an inherited `description` into the parameter when the model loads, and nothing downstream can tell that apart from one the author typed, so a pulled document states it where the original left it to the field. Re-loading gives the same parameter, but the field and the parameter now each hold a copy, and changing the field no longer moves it. A projection whose field is not in the pulled document β€” pruning drops an entity no binding reaches β€” comes back as a declared parameter carrying the resolved `type` instead, since emitting a `concept`/`field` pair that resolves to nothing would write a document that will not load. A `sql` executor round-trips its `statements` with the rest: they are the write, not a note about it, so a catalog that dropped them would describe an action nobody could re-deploy. The entry type is custom, so `kcmd init` creates it; a model that declares no action never needs it. What an action affects round-trips as a fact, not as the text it was authored in. The two authored shapes β€” the bare name and the record β€” are one shape in the model, so a record carrying nothing but a concept (`- concept: Account`) comes back as the bare `Account`, which says the same thing. Everything after that first pass is byte-for-byte stable. `affects` stores only what the author wrote, which is why it survives a partial pull intact. Whether a `concept` is an entity or an edge is not recorded, so nothing about it has to be re-resolved β€” and nothing can go stale. That matters most for a many-to-many relationship, which never reaches the catalog at all: a pull recovers relationships from the schema-join entry links, which carry the foreign-key edges only, so a concept naming an M:N edge would look unresolvable on a model that is perfectly well-formed. It comes back untouched instead. **Constraints** publish the same way: each becomes a `semantic-constraint` entry under the model entry, with the rule and any `instructions` in a `semantic-constraint` aspect and the `description` as the entry's own summary. Name, description, `on_violation`, `severity`, instructions, and whichever body states the rule β€” `expression` or `judgment` β€” round-trip losslessly through `pull`. A constraint that declares neither routing word comes back without them rather than with a default filled in. The aspect also carries a derived `evaluation` field, `deterministic` or `judged`, which a pull recomputes from the body rather than reading, so it can never come back disagreeing with the rule beside it; it is not written into the authored document, because a derived value in an authored file is a value that can go stale. An entry that states both bodies, or neither, is skipped with a warning rather than pulled into a model that would then fail its own push. That entry type is custom too, and a model that declares no constraint never needs it. ## To BigQuery Push preserves both the queryable structure and the descriptive metadata attached to it. The structure becomes the node tables, edge tables, and measures; the descriptive metadata is written into each element's `OPTIONS(...)` in the graph (visible with `--print`). BigQuery's graph `OPTIONS` give an element a `description` string and a `synonyms` array. `synonyms` is the only part with a dedicated option, so it carries across structurally, as its own array. The rest β€” `description`, `instructions`, `examples`, and a field's `label` β€” share the single `description` string, so they are combined into it (examples as an `Examples: …` line). Their content is preserved; their separate structure is not. The model's own statement-level metadata has nowhere to go: BigQuery silently drops graph-statement `OPTIONS`, so the model's `description` / `ai_context` are not in the graph β€” the `description` and `ai_context.instructions` are carried into Knowledge Catalog instead. Of an entity's keys only one is emitted: its primary key, else its first unique key. The imported vendor SQL is never emitted. `kcmd import --transpile` adds a `BIGQUERY` entry to a model file's field or metric whose expression is vendor-only or an `ANSI_SQL` expression that double-quotes column names. It leaves a profile's expressions as written. The graph builds each field and metric from its `BIGQUERY` entry, else its `ANSI_SQL` one. A field with none of these is left out of its node table with a warning, though push rejects such a field in the model file first. A metric with none, or one that aggregates such a field, is skipped with a warning. ## To Spanner A Spanner target keeps the queryable **structure** and drops the descriptive metadata. The node tables, edge tables, and labels (including the `extends` hierarchy, with fields flattened down) deploy exactly as they do for BigQuery, but with bare table and graph names. Three things do not make the trip, all by design: - **Metrics.** Spanner has no `MEASURE`, so every model-level metric is dropped with a warning. The BigQuery-only rule that a metric resolve to one entity does not apply. Author your metrics as usual β€” a BigQuery target still emits them, and Knowledge Catalog still records each `semantic-metric` entry β€” they simply have no home in the Spanner graph. - **`OPTIONS` metadata.** Spanner carries no per-element `OPTIONS`, so `description`, `synonyms`, `instructions`, `examples`, and field `label` are not written into the Spanner DDL. They still reach Knowledge Catalog on the same push (the model's / entities' / metrics' descriptions and `instructions` land on their entries and the `guidelines` aspect), so the descriptive layer lives in the catalog rather than in the Spanner graph. This mirrors how BigQuery drops its graph-*statement* `OPTIONS` while keeping per-element ones. - **Another engine's SQL.** A field reads its `SPANNER` expression, else its `ANSI_SQL` one, and is left out with a warning when it has neither. Everything else β€” keys, relationships, the label hierarchy β€” matches the **β†’ BigQuery** column above. **Actions and constraints** reach neither graph. They are published to Knowledge Catalog as one `semantic-action` or `semantic-constraint` entry each and round-trip losslessly through `pull` β€” see [To Knowledge Catalog](#to-knowledge-catalog). ## What pull recovers Push stores a copy of the model file and of each profile file it reads, and `pull` writes those copies back unchanged. What follows describes the catalog metadata itself, which is what `pull` rebuilds the model file from when the model has no `kcmd-push-id` label and no stored copy, such as a model an older `kcmd` pushed. It also rebuilds the model file when no stored copy is named after the model. [Pull](README.md#pull) says when `pull` refuses the catalog instead. With `KC_V2_ASPECTS=1`, pull rebuilds the model file, profile files and overview sidecars from the metadata, and writes each metric's entity as `entity:`, declared or derived. See [Reference](reference.md#with-kc_v2_aspects1). `pull` returns what push wrote β€” the **Recovered by `pull`** column above is the summary. When the push deployed a graph, what push wrote was already pruned to the bound view (see [To Knowledge Catalog](#to-knowledge-catalog)), so pull returns that view. Two things about *how* it comes back: **Normalized** β€” the content survives, the form changes: - Relationship *names* come back lowercased (`Places_Order` β†’ `places_order`); the catalog stores the name only in the link id. See [Writer-side follow-up](#writer-side-follow-up). - Field types round-trip except one collapse: a field authored as `String` comes back un-typed (both no type and `String` store `dataType STRING`, and `pull` writes no `datatype` for either; see footnote Β²). A metric's data type round-trips only for a concrete type (e.g. `Decimal`); an untyped, `String`, or `Opaque` metric comes back un-typed, because the metric aspect stores a data type but no metadata type to mark it `Opaque`. - A field's dimension role (present only if pushed with `--emit-expressions`) comes back as a bare `dimension: {}` marker, without its detail (`is_time`, and so on). - Ordering: field order within each entity is preserved, but the order of entities and metrics is not β€” they come back in the catalog's own order, not the authored one. Comments in the original YAML are not preserved. **So a file rebuilt from the metadata is not your original file.** Treat a rebuilt document as a faithful copy of the catalog metadata rather than of the authored model, and keep the authored document as the source of truth. ## Writer-side follow-up One reduction above is a limit of what push currently *writes* rather than of what pull can recover. It is recorded here as a write-side follow-up; the reader (pull) already returns everything the catalog holds. - **Relationship names.** The `schema-join` aspect type's `metadataTemplate` has no field for the relationship name, so push cannot store it and pull recovers it from the link id β€” which is lowercased, with each underscore written as a hyphen (the entry-link id format forbids both). Pull reads each hyphen back as an underscore, but the original casing is lost. Returning the name verbatim requires adding a name field to the built-in `schema-join` aspect type in Knowledge Catalog (server-side), after which the client write/read is trivial; it is the same class of gap as the `semantics` field that gates `--emit-expressions`. (A deployment target in no supported form is **not** a pull gap. Every push rejects it at the validation gate before anything is written, a catalog-only push (`--no-profile`) too; see [Validation](reference.md#validation).) --- ### Toolbox Mdcode Docs Semantic Model Inheritance (`toolbox/mdcode/docs/semantic-model/inheritance.md`) # Modeling class hierarchies When several entities are kinds of one thing β€” a customer and a supplier are both parties you deal with β€” you can declare the general kind once and say that each specific kind `extends` it. A query written against the general kind then reaches every specific kind. `extends` is the semantic model's *is-a*: a customer is a party. `kcmd push` expresses the hierarchy on the graph as **labels**. A subtype's node table carries its own label plus one label per supertype, so `MATCH (:Party)` matches every customer and every supplier. The exact generation rules β€” how fields flatten down, what a label signature must contain β€” are in [Reference β†’ Class hierarchies](reference.md#class-hierarchies-extends--labels); this page is the modeling guide. ## When to use it Use `extends` when the shared fields reflect a real *is a kind of* relationship and you want a supertype query to gather every subtype. Do not use it when the shared fields are a coincidence of naming, or when you have one entity whose columns happen to be split across two tables β€” that is a [binding](profiles.md), and modeling it as inheritance produces the double-count described below. ## The one rule A supertype query gathers every subtype. Because it gathers subtypes, **each real thing must live in exactly one place**, or it is gathered more than once and your counts double. State it once and rely on it: > **One real thing, one row, one node.** Everything below is a way to honor this rule. ## 1. Declare the hierarchy `Party` is the general kind. In this model no row is *just* a party β€” every party is a customer or a supplier β€” so `Party` has no table of its own. Mark it `abstract: true`: it has no `source`, no key and no column bindings, produces no node table, and survives in the graph only as a label on its subtypes. Each concrete kind declares its own fields and the one `extends` keyword. The supertype's fields are inherited, so you do not repeat them: ```yaml version: "0.2.0.dev0/google" semantic_model: - name: parties entities: - name: Party abstract: true fields: - { name: id, datatype: Integer } - { name: name, datatype: String } - name: Customer extends: [Party] primary_key: [id] # each subtype keeps its OWN key; keys are not inherited fields: - { name: loyalty_tier, datatype: String } - name: Supplier extends: [Party] primary_key: [id] fields: - { name: rating, datatype: Integer } ``` The supertype's `id` and `name` flatten down onto both subtypes, so `Customer` and `Supplier` each expose them, and each subtype node carries both its own label and the `Party` label. ```mermaid classDiagram class Party { <<abstract>> id : integer name : string } class Customer { loyalty_tier : string } class Supplier { rating : integer } Party <|-- Customer Party <|-- Supplier ``` `Party` is abstract, so it has no table of its own; the arrows are `extends`. A query against `Party` reaches every subtype below it. ## 2. Bind each subtype's table Inheritance meets binding at one requirement: **each subtype's table must expose the inherited columns**, because those fields are read from the subtype's own table. `Party.name`, flattened onto `Customer`, is read from the customer table, so that table must have a name column. A binding gives each entity its `source` and each field its column. With one store you can bind inline on the model as the `default` profile: ```yaml - name: Customer extends: [Party] primary_key: [c_custkey] source: bigquery.googleapis.com/projects/my-project/datasets/sales/tables/customer fields: - { name: id, expression: c_custkey } - { name: name, expression: c_name } # the inherited field, bound here - { name: loyalty_tier, datatype: String, expression: c_tier } ``` A redeclared inherited field carries only its binding. Its datatype and the rest of its definition come from `Party`, and push rejects a redeclaration that restates them. To bind the same hierarchy to more than one store, put each binding in its own [profile](profiles.md). A named profile binds every concrete subtype, since each one stands for a table the profile's database has to hold. A profile can bind a field the subtype inherits by naming it under that subtype, with no redeclaration in the model file, and the binding reads from the subtype's own table. ## 3. Query the supertype ``` GRAPH my-project.sales.parties MATCH (p:Party) RETURN p.name ``` Every customer and every supplier comes back, each once, because each real party lives in exactly one table. ## More hierarchy shapes `extends` composes. A subtype can extend several supertypes, and a supertype can extend another supertype above it. Two facts hold for every shape: the supertype fields flatten down, and each concrete table binds every inherited field to its own column. So the shapes below all deploy the same way, and each was verified to deploy on BigQuery Graph. ### Extending more than one supertype, and diamonds `extends` takes a list. Here an `Employee` is both a `Person` and a `Taxpayer`, and each of those is a `Party`: ```yaml - name: Party abstract: true fields: - { name: id, datatype: Integer } - { name: name, datatype: String } - name: Person abstract: true extends: [Party] fields: - { name: birth_year, datatype: Integer } - name: Taxpayer abstract: true extends: [Party] fields: - { name: tax_id, datatype: String } - name: Employee extends: [Person, Taxpayer] primary_key: [e_id] source: bigquery.googleapis.com/projects/my-project/datasets/hr/tables/employee fields: - { name: id, expression: e_id } - { name: name, expression: e_name } - { name: birth_year, expression: e_birth } - { name: tax_id, expression: e_tax } - { name: department, datatype: String, expression: e_dept } ``` ```mermaid classDiagram class Party { <<abstract>> id : integer name : string } class Person { <<abstract>> birth_year : integer } class Taxpayer { <<abstract>> tax_id : string } class Employee { department : string } Party <|-- Person Party <|-- Taxpayer Person <|-- Employee Taxpayer <|-- Employee ``` `Party` sits at the top and is reached through two paths, so this is a diamond. `Employee` carries the `Person`, `Taxpayer`, and `Party` labels, and `Party` appears once. `MATCH (:Person)`, `MATCH (:Taxpayer)`, and `MATCH (:Party)` each return every employee a single time. The number of supertypes a subtype has, and the number of paths that reach a shared ancestor, do not change the count. A field may be declared on only one side of a diamond. If `Person` and `Taxpayer` both declared a `nickname` field, push would reject the model, because neither is nearer to `Employee` and nothing says which one wins. Declare a field both sides share on `Party` instead. The same holds for a binding: two concrete supertypes that each rebind a field they inherit from `Party` leave their common subtype ambiguous, and push rejects it. ### Deeper hierarchies A hierarchy can run several levels deep with a concrete table at each leaf. Here `Party` divides into `Person` and `Organization`, and each has its own concrete kind: a `Customer` is a person, a `Vendor` is an organization: ```yaml - name: Party abstract: true fields: - { name: id, datatype: Integer } - { name: name, datatype: String } - name: Person abstract: true extends: [Party] fields: - { name: birth_year, datatype: Integer } - name: Organization abstract: true extends: [Party] fields: - { name: founded_year, datatype: Integer } - name: Customer extends: [Person] primary_key: [c_id] source: bigquery.googleapis.com/projects/my-project/datasets/sales/tables/customer fields: - { name: id, expression: c_id } - { name: name, expression: c_name } - { name: birth_year, expression: c_birth } - { name: tier, datatype: String, expression: c_tier } - name: Vendor extends: [Organization] primary_key: [v_id] source: bigquery.googleapis.com/projects/my-project/datasets/sales/tables/vendor fields: - { name: id, expression: v_id } - { name: name, expression: v_name } - { name: founded_year, expression: v_founded } - { name: rating, datatype: Integer, expression: v_rating } ``` ```mermaid classDiagram class Party { <<abstract>> id : integer name : string } class Person { <<abstract>> birth_year : integer } class Organization { <<abstract>> founded_year : integer } class Customer { tier : string } class Vendor { rating : integer } Party <|-- Person Party <|-- Organization Person <|-- Customer Organization <|-- Vendor ``` `Customer` and `Vendor` are the only tables. `MATCH (:Party)` returns every customer and every vendor; `MATCH (:Person)` returns customers alone, and `MATCH (:Organization)` vendors alone. Each real party lives in one table, so every count is exact. ### What keeps every shape correct These shapes deploy because their supertypes are abstract. Each concrete table binds every inherited field to its own column, and BigQuery matches a shared label by property name, so the subtype tables never have to agree on a physical column. A supertype with its own table works too. Then every subtype table must carry that supertype's columns under the same names, and a thing present in both the supertype table and a subtype table is counted twice under the supertype label. Such a supertype still cannot be a relationship endpoint or own a metric, because other entities extend it. Keeping supertypes abstract keeps the one rule β€” one real thing, one node β€” automatic. ## Match the shape to how your data is stored How your subtypes are already stored decides how you model them. `extends` builds one shape directly, and the other two common layouts are modeled without it: - **One table per kind, the parent has none.** Each concrete kind has its own complete table; the supertype has no table and survives as a label; a supertype query is the union of the kind tables. This is the shape `extends` builds β€” the hierarchy above. It is correct as long as the kind tables hold disjoint things, which is why the parent is `abstract`. - **One table for the whole family, with a kind column.** All parties in one table, a `type` column saying which kind each row is. Model this as one entity with a dimension field; it holds one row per thing, so it cannot double-count. - **A base table plus an extension table.** A thing's general fields in one table and its specific fields in another, tied by a shared key. Model this as one entity whose binding joins the two tables, so the thing keeps one identity. ## When a supertype total looks too high If a supertype count or sum comes back larger than the data warrants, the same real thing lives in more than one table under the hierarchy. Every table carrying the label contributes that thing as its own node. Take three people split across a `person` table and a `customer` table that both carry the `Person` label, with two of the three present in both. `MATCH (:Person)` returns **five** nodes, because no identity link across the two tables tells the graph that the shared rows are the same person. The fix is the one rule. Make the parent `abstract` so no real thing lives in more than one table under the hierarchy. When the two tables are instead the general and specific halves of one thing, model them as one entity whose binding joins them by key. ## Relationships and metrics belong on the leaves A relationship may connect only concrete leaf entities. A leaf is an entity that is not abstract and that no other entity extends. An edge stays on the node table it is declared on. An abstract `Party` has no table, and a concrete supertype's table holds only its own rows, so an edge declared on a supertype misses every subtype row. Declare the edge from each leaf that has it instead, for example from `Customer`. A metric may belong only to a leaf as well, because a measure cannot sit on a label that several tables share. Push rejects a relationship or metric on any other entity, including a push to Knowledge Catalog alone. ## The rules push enforces Fields flow down to subtypes. Relationships and keys stay where they are declared. Every table under a shared supertype exposes the identical field set, so a subtype's table must have a column for every inherited field, or the deploy fails. Push also rejects a model that breaks any of these rules: - A subtype redeclares an inherited field only to bind it to a column. A redeclaration with nothing but the name is rejected. - A concrete entity restates each field it inherits whose ancestor's expression reads a column. This holds in each named profile, and in the model file when it gives the entity a source. A field whose ancestor's expression reads only other fields carries over. - An abstract entity declares no `source`, no keys and no field expressions. - Relationships and metrics attach only to concrete leaf entities. - No field is declared or rebound by two supertypes that do not extend each other. - `extends` forms no cycle. Each rule and its reason is in [Reference β†’ Class hierarchies](reference.md#class-hierarchies-extends--labels). Inheritance deploys the same way to BigQuery Graph and Spanner Graph β€” the extra labels and the flattened fields are identical on both backends. On Spanner the model carries no measures, as it does for any model. --- ### Toolbox Mdcode Docs Semantic Model Model Spec (`toolbox/mdcode/docs/semantic-model/model_spec.md`) # Semantic Model Specification This is the normative definition of the semantic model document `kcmd` reads and writes. It defines what a *valid* model is β€” the YAML grammar, the logical and physical constructs, and the rules the format enforces. For what `kcmd` *does* with a valid model (flags, what each push creates, permissions), see [Reference](reference.md); for the task walkthrough, see the [deploy guide](README.md). The two documents divide by register: this spec states the **definition** ("a model MUST declare at most one deployment target"); Reference states the **operational consequence** ("…otherwise push is rejected"), and links back here. When they overlap, this document is the authority on what is valid and Reference is the authority on what the tool then does. ## 1. Status and baseline The format is **`kcmd`'s profile of [Apache Ossie](https://ossie.apache.org/)**, built on Ossie version **`0.2.0.dev0`**. Ossie is the open semantic-model format: it describes a business β€” datasets, fields, relationships, metrics β€” *and* binds each dataset to a `source` table and each field to a column `expression`. What Ossie does not describe is where a model then deploys. `kcmd` adds that deployment layer and a few modeling constructs, and it relaxes a few of Ossie's required fields so a model can stay purely logical. Those additions come in **two versions**, and a document picks one with its top-level `version` (see [Version handling](#1-status-and-baseline)): - **`0.2.0.dev0`** β€” **vanilla Ossie**. `kcmd`'s additions ride in Ossie's own `custom_extensions` carrier ([Β§6](#6-the-extension-mechanism)); the native extension keys are not accepted. This profile is a strict superset of Ossie: strip the `GOOGLE` extension blocks and it is a plain Ossie document. - **`0.2.0.dev0/google`** β€” the **extended profile**. The same additions are first-class native keys (`entities`, `deployment_target`, `extends`, `abstract`); the `custom_extensions` carrier is not accepted, because there is nothing left for it to carry. This profile is readable, but not a vanilla-Ossie document β€” an Ossie-only reader knows neither its version nor its native keys. For a model that avoids the `/google`-only constructs, the two are the same model written two ways; both parse to the identical IR, so no downstream leg sees which was used. Ossie compatibility is a property of the **vanilla** version, and the contract runs in both directions: > **An Ossie `0.2.0.dev0` document is a valid `kcmd` model unless it breaks a > narrowing ([Β§4.1](#41-narrowings-stricter-than-ossie)), such as an expression in > a non-SQL dialect.** The reverse holds for a vanilla `kcmd` model that uses no > extensions ([Β§5](#5-extensions)) *and* no relaxations > ([Β§4](#4-narrowings-and-relaxations)) β€” a fully-bound model with only Ossie > constructs. Strip the `GOOGLE` `custom_extensions` blocks from such a model and > it is a valid Ossie document. Under vanilla, `kcmd`'s deployment, binding and inheritance additions ride in a `custom_extensions` block an Ossie-only tool ignores (see [Β§6](#6-the-extension-mechanism)), so those never fork the document; `abstract`, `actions`, `constraints` and the `entities` spelling have no vanilla form and exist only under `0.2.0.dev0/google`. The extended `0.2.0.dev0/google` profile trades that wire compatibility for readable native keys; it is `kcmd`'s own surface, converted to or from vanilla by choosing the `version`. **How to read this document.** This is a *delta* against the Ossie spec. It does not restate constructs `kcmd` adopts unchanged β€” for those, the Ossie spec is authoritative. This document covers only what is `kcmd`'s to state: 1. where `kcmd` is **stricter** than Ossie β€” narrowings ([Β§4](#4-narrowings-and-relaxations)), 2. where `kcmd` is **looser** than Ossie β€” relaxations ([Β§4](#4-narrowings-and-relaxations)), 3. what `kcmd` **adds** beyond Ossie β€” extensions ([Β§5](#5-extensions)), and 4. how those additions stay Ossie-compatible ([Β§6](#6-the-extension-mechanism)). If a construct is not mentioned here, `kcmd` supports it as Ossie `0.2.0.dev0` defines it. **Conformance keywords.** MUST, MUST NOT, SHOULD, and MAY are used per [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). A rule marked MUST is enforced at load or validation time; the exact operational effect (hard error, skip with a warning) is in [Reference β†’ Validation](reference.md#validation). **Version handling.** `version` is **required** and lives at the top of the document (not inside a model). It MUST be one of two values: `0.2.0.dev0` (vanilla Ossie) or `0.2.0.dev0/google` (the extended profile). A missing or unrecognized `version` is a **hard load error**, not a warning β€” the value selects which extension surface is legal ([Β§6](#6-the-extension-mechanism)), so it cannot be guessed. `kcmd`'s own tools (pull, the OWL importer) emit `0.2.0.dev0/google`; hand-authored portable models SHOULD use vanilla `0.2.0.dev0`. **Baseline.** The Ossie classifications in [Β§3](#3-conformance-summary) are reconciled against the Ossie `0.2.0.dev0` core JSON Schema, a copy of which is vendored in this repository at `tests/libts/semantic/fixtures/ossie/osi-schema.json` (copied unmodified from `apache/ossie`). That schema, not `kcmd`'s parser, is the authority for what Ossie defines; the parser is the authority only for what `kcmd` accepts. ## 2. Document shape A model document is a YAML file with a top-level `semantic_model` list. ```yaml version: "0.2.0.dev0" # required; selects the extension surface (Β§1) semantic_model: # one or more models; kcmd deploys exactly one per entry group (Β§4) - name: sales datasets: [ … ] # one or more; the entities (Β§2.1). Spelled `entities:` under /google (Β§5) relationships: [ … ] # optional (Β§2.2) metrics: [ … ] # optional (Β§2.3) actions: [ … ] # optional; /google only (Β§5) constraints: [ … ] # optional; /google only (Β§5) description: "…" # optional ai_context: { … } # optional (Β§2.4) custom_extensions: [ … ] # vanilla only (Β§6); rejected under /google deployment_target: "…" # /google only (Β§5, Β§7); vanilla carries it in a GOOGLE block ``` A document MUST contain at least one model, and each model MUST contain at least one dataset. Every named object (dataset, field, relationship, metric, action, constraint) MUST have a name unique within its scope. A duplicate name is a **hard load error** (agreed with Dmitri), because nothing reading the model can tell two objects of the same name apart. Objects are **closed**: an unknown sibling key inside a validated object is a **hard error**, not silently dropped (agreed with Dmitri). This is what makes the version gate bite β€” a native key under vanilla, or a `custom_extensions` block under `/google`, is an unknown key for that version and is rejected ([Β§6](#6-the-extension-mechanism)). Setting both `entities` and `datasets` is likewise an error ([Β§5](#5-extensions)). The remaining subsections define each object. Names throughout are **logical business vocabulary** β€” `order_id`, `placed_by` β€” never physical column names; the physical binding is a separate concern ([Β§7](#7-the-binding-layer)). ### 2.1. Dataset (entity) A dataset is one concept in the business. | Key | Type | Rule | |---|---|---| | `name` | string | required | | `fields` | list of [field](#211-field) | the concept's attributes | | `primary_key` | list of strings | the physical column(s) that identify a row | | `unique_keys` | list of lists of strings | additional unique column-sets | | `extends` | list of strings | supertype dataset names β€” extension ([Β§2.1.2](#212-inheritance-extends), [Β§5](#5-extensions)) | | `abstract` | boolean | a concept with no table β€” extension ([Β§5](#5-extensions)) | | `source` | string | physical table URI; Ossie-native, relaxed to optional ([Β§7.1](#71-table-sources), [Β§4](#4-narrowings-and-relaxations)) | | `description` | string | | | `ai_context` | [ai_context](#24-ai_context) | | | `custom_extensions` | list | [Β§6](#6-the-extension-mechanism) | An abstract dataset MUST NOT declare a `source`; a non-abstract dataset in a model bound for a graph MUST declare one (see [Β§4](#4-narrowings-and-relaxations) and [Β§7](#7-the-binding-layer)). #### 2.1.1. Field | Key | Type | Rule | |---|---|---| | `name` | string | required | | `datatype` | enum | one of the closed vocabulary below | | `expression` | [expression](#25-expressions) | physical-column binding; a field with none is *unbound* (binding β€” [Β§7](#7-the-binding-layer)) | | `label` | string | human display label | | `dimension` | object | `{ is_time: boolean }` | | `description` | string | | | `ai_context` | [ai_context](#24-ai_context) | | | `custom_extensions` | list | [Β§6](#6-the-extension-mechanism) | `datatype` is a closed, **case-sensitive** vocabulary: ``` String Integer Decimal Float Boolean Date Time DateTime DateTimeTz Opaque ``` `Date`, `Time`, `DateTime`, and `DateTimeTz` are the temporal subset (they default `dimension.is_time`). A value outside this set is not authorable directly; a type `kcmd` cannot express natively is represented as `Opaque` plus a `custom_extensions` block that carries the real type. A field's `expression` is its binding; a field with no `expression` is **unbound** (no column under this binding). There is no separate `unbound` flag β€” absence is the signal. See [Β§7](#7-the-binding-layer). #### 2.1.2. Inheritance (`extends`) `extends` is a `kcmd` **extension** β€” Ossie `0.2.0.dev0` has no inheritance construct ([Β§5](#5-extensions)). A dataset MAY declare `extends: [Parent, …]`, naming one or more supertype datasets by name. Inheritance is **entity-level only** β€” relationships do not `extends`. A supertype that has no table of its own MUST be marked `abstract: true` (also an extension). The format rule is only that supertypes are named datasets in the same model. An `extends` that names a dataset not defined in the model is a **hard error** (a typo must not silently drop inheritance β€” agreed with Dmitri). A cycle in `extends` is an error. A diamond flattens, with each ancestor contributing once. A field declared or rebound by two supertypes that do not extend each other is an error, because nothing says which one wins. A subtype MAY redeclare an inherited field only to bind it, with an `expression` and nothing else; the definition comes from the ancestor that declares it. A concrete subtype MUST rebind each inherited field whose ancestor's expression reads a column, because that column belongs to the ancestor's table. This holds in each named profile, and in the model file when it gives the subtype a source. A relationship endpoint and a metric's entity MUST be a concrete leaf: not abstract, and not extended by another dataset. How the hierarchy is *lowered* into each store (labels, field flattening) and the downstream limits (e.g. a supertype whose label is shared across subtype tables cannot carry a measure) are operational β€” see [Reference β†’ Class hierarchies](reference.md#class-hierarchies-extends--labels) and [Modeling class hierarchies](inheritance.md). ### 2.2. Relationship A relationship is a directed edge between two datasets. | Key | Type | Rule | |---|---|---| | `name` | string | required | | `from` | string | required; a declared dataset name | | `to` | string | required; a declared dataset name | | `from_columns` | list of strings | join key on `from` | | `to_columns` | list of strings | join key on `to` | | `description` | string | | | `ai_context` | [ai_context](#24-ai_context) | | | `custom_extensions` | list | [Β§6](#6-the-extension-mechanism) | `from_columns` and `to_columns` are the edge's join keys. They MUST be given together (a bound edge) or both omitted (a logical edge); one without the other is rejected. When both are given they MUST have equal length. `from` and `to` MUST name datasets declared in the same model. `to_columns` MUST contain every column of a primary or unique key of the `to` dataset; a superset of a key counts. An edge is N:1 or 1:1 only when its join columns cover a key, so join columns that cover no key are rejected on every push, a catalog-only push included. Each binding of the relationship may cover a different key. A relationship MUST join two different datasets, and two datasets MUST NOT have relationships in both directions. Ossie **requires** both join-column lists; allowing both to be omitted β€” a logical edge with no join keys β€” is a `kcmd` **relaxation** ([Β§4](#4-narrowings-and-relaxations)) that lets a relationship be governed before it is bound. A graph deploy still requires them (see [Β§4](#4-narrowings-and-relaxations)). Like `source` and a field's `expression`, join columns are a binding: a profile may supply or override them, and a named profile states them itself ([Β§7](#7-the-binding-layer)). > **Many-to-many is not yet authorable.** A junction-table (M:N) relationship > exists in `kcmd`'s internal representation but has **no YAML syntax** in > `0.2.0.dev0`. It cannot be authored today and is reserved for a future format > extension. Direct-FK relationships (1:1, 1:N) are the authorable forms. ### 2.3. Metric A metric is a computation over the model's fields. | Key | Type | Rule | |---|---|---| | `name` | string | required | | `expression` | [expression](#25-expressions) | **required** | | `datatype` | enum | the [field datatype vocabulary](#211-field) | | `description` | string | | | `ai_context` | [ai_context](#24-ai_context) | | | `custom_extensions` | list | [Β§6](#6-the-extension-mechanism) | A metric names no dataset directly; the dataset it belongs to is **derived** from the entities its expression references. This is why a metric bound for a BigQuery graph MUST reduce to a single entity ([Β§4](#4-narrowings-and-relaxations)). A metric is SQL over fields; there is no native metric-composition (metric-of-metrics) construct. ### 2.4. `ai_context` `ai_context` MAY appear on the model, a dataset, a field, a relationship, and a metric. It is either a bare string (shorthand for instructions) or an object: ```yaml ai_context: instructions: "…" # guidance for agents synonyms: [ "…", "…" ] # alternate names examples: [ … ] # illustrative values / usages ``` There is no top-level `synonyms` key on any object; synonyms are always `ai_context.synonyms`. `instructions`, `synonyms`, and `examples` are all Ossie-native. Each `examples` item MUST be a string, as in Ossie; any other item is a load error. Which parts of `ai_context` reach which destination is in [What push and pull preserve](fidelity.md). ### 2.5. Expressions A field or metric `expression` is either a bare SQL string or the per-dialect form: ```yaml expression: SUM(orders.net_amount) # shorthand: dialect BIGQUERY expression: # explicit per-dialect dialects: - { dialect: BIGQUERY, expression: "SUM(orders.net_amount)" } - { dialect: ANSI_SQL, expression: "SUM(orders.net_amount)" } ``` The shorthand string normalizes to a single `ANSI_SQL` dialect. `ANSI_SQL` is the portable fallback. An `expression` references logical fields as `entity.field`; the qualifiers are what let a metric's entity be derived ([Β§2.3](#23-metric)). Ossie closes `dialect` to an enum β€” `ANSI_SQL`, `SNOWFLAKE`, `MDX`, `TABLEAU`, `DATABRICKS`, `MAQL`, `BIGQUERY`, `THOUGHTSPOT`. `kcmd` accepts the SQL dialects `ANSI_SQL`, `BIGQUERY`, `SPANNER`, `MYSQL`, `POSTGRES`, `ALLOYDB`, `SQLSERVER`, `SNOWFLAKE` and `DATABRICKS`, and rejects any other name. `SQLSERVER` text is read as T-SQL. ## 3. Conformance summary Every construct, at a glance, measured against **Ossie `0.2.0.dev0`** β€” the vanilla baseline (see [Β§1](#1-status-and-baseline)). The **OSI** column is what that baseline defines, where a `β€”` means Ossie does not define the construct at all. The **Google extension** column is how this profile differs: *same* (adopted unchanged), *optional* (a relaxation, [Β§4](#4-narrowings-and-relaxations)), *stricter* (a narrowing, [Β§4](#4-narrowings-and-relaxations)), *added* (an extension, [Β§5](#5-extensions)), or *rejected* / *not authorable* (excluded). **Why / where** gives the reason and the section with the detail. | Feature | OSI (`0.2.0.dev0`) | Google extension | Why / where | |---|---|---|---| | `semantic_model`, `name`, `description` | defined | same | [Β§2](#2-document-shape) | | `datasets` | defined | same | [Β§2.1](#21-dataset-entity) | | `entities` (alias for `datasets`) | β€” | added (alias) | readability; sugar, `/google` only Β· [Β§5](#5-extensions) | | field `name`, `label`, `dimension` (`is_time`) | defined | same | [Β§2.1.1](#211-field) | | `datatype` + vocabulary | closed enum | same (identical set) | [Β§2.1.1](#211-field) | | `primary_key`, `unique_keys` | defined | same | [Β§2.1](#21-dataset-entity) | | `extends` | β€” | added | class inheritance; `/google` only Β· [Β§2.1.2](#212-inheritance-extends), [Β§5](#5-extensions) | | `abstract` | β€” | added | supertype with no table; `/google` only Β· [Β§5](#5-extensions) | | relationship `name`, `from`, `to` | defined | same | [Β§2.2](#22-relationship) | | relationship `from_columns` / `to_columns` | required | optional | model before binding; none = logical edge Β· [Β§4.2](#42-relaxations-looser-than-ossie) | | relationship M:N (`association`) | β€” | not authorable (reserved) | no M:N syntax yet Β· [Β§2.2](#22-relationship) | | `metrics`, metric `expression` | required | same; graph-bound stricter | a graph measure binds one node and aggregate Β· [Β§4.1](#41-narrowings-stricter-than-ossie) | | `expression.dialects` | closed enum | stricter; adds names | SQL dialects only; adds Google Cloud engines Β· [Β§2.5](#25-expressions), [Β§4.1](#41-narrowings-stricter-than-ossie), [Β§4.2](#42-relaxations-looser-than-ossie) | | `actions` | β€” | added | write operations declared over the ontology; `/google` only Β· [Β§5](#5-extensions) | | `constraints` | β€” | added | named invariants over the ontology; `/google` only Β· [Β§5](#5-extensions) | | field `expression` (column binding) | required | optional | model before binding; unbound is pruned Β· [Β§4.2](#42-relaxations-looser-than-ossie), [Β§7](#7-the-binding-layer) | | `source` (table binding) | required | optional | logical-only models; graph deploy still needs it Β· [Β§4.2](#42-relaxations-looser-than-ossie), [Β§7.1](#71-table-sources) | | `ai_context` (`instructions`, `synonyms`, `examples`) | `examples` are strings | same | [Β§2.4](#24-ai_context) | | `custom_extensions` (`vendor_name`, `data`) | the extension carrier | rejected | `/google` uses native keys; nothing to carry Β· [Β§6](#6-the-extension-mechanism) | | `deployment_target` | β€” | added | OSI omits deployment; vanilla carries it in `custom_extensions` Β· [Β§5](#5-extensions), [Β§7.2](#72-deployment-target) | | binding profiles | β€” | added | one model, many stores Β· [Β§7.3](#73-binding-profiles) | | multiple models per document | a list (many) | exactly one | entry group = model's identity Β· [Β§4.1](#41-narrowings-stricter-than-ossie) | ## 4. Narrowings and relaxations This profile diverges from Ossie in two directions. **Narrowings** are where it is stricter β€” rules that can reject an otherwise-valid Ossie document at deploy time. **Relaxations** are where it is looser β€” fields Ossie requires that this profile makes optional, so a relaxed model is not a valid Ossie document until those fields are supplied. Both are the reason the reverse compatibility direction in [Β§1](#1-status-and-baseline) is conditional. ### 4.1. Narrowings (stricter than Ossie) Each rule and its reason: - **At most one deployment target per model.** A model bound for a graph MUST declare exactly one deployment target ([Β§7](#7-the-binding-layer)). *Why:* the target names one graph in one store; a model with several has no single place to deploy. (A catalog-only push needs no target at all.) - **Exactly one model per entry group.** A push MUST deploy exactly one model to a Knowledge Catalog entry group, and a pull expects exactly one. *Why:* the entry group is the model's identity in the catalog; more than one anchor is ambiguous. - **A graph-bound metric MUST resolve to a single entity.** For a BigQuery target, every metric's expression MUST reference exactly one entity, so the metric can attach to one node table. *Why:* a graph measure binds to a single element. Spanner imposes no such rule (it has no measures). A metric that resolves to no entity or spans several is rejected at push. - **A graph-bound metric MUST reduce to one supported aggregate.** For a BigQuery target, a metric that is not a single supported aggregate (`SUM`, `AVG`, `COUNT`, `MIN`, `MAX`) over one operand cannot become a `MEASURE`, and the push is rejected (see [Validation](reference.md#validation)). `COUNT(*)` on a keyed node also passes. *Why:* BigQuery Graph measures are single-aggregate. - **A graph-bound relationship MUST have its join columns bound.** For any graph target, a non-M:N relationship MUST supply both `from_columns` and `to_columns` before deploy. *Why:* the edge table needs both keys. - **Unknown keys are rejected.** Every object is validated closed: an unrecognized sibling key is a hard load error, not silently dropped. Combined with the version gate, this is how a native key under vanilla β€” or a `custom_extensions` block under `/google` β€” is caught ([Β§6](#6-the-extension-mechanism)). - **Duplicate names are rejected.** A dataset, field, relationship, or metric name repeated within its scope is a hard load error; the generated element would be ambiguous. - **An unknown supertype is rejected.** An `extends` that names a dataset not in the model is a hard load error (see [Β§2.1.2](#212-inheritance-extends)). - **A metric belongs to a concrete leaf.** A metric whose entity is abstract, or extended by another entity, is rejected on every push. - **A relationship joins two different datasets in one direction.** A relationship from a dataset to itself is rejected, and so are relationships in both directions between two datasets ([Β§2.2](#22-relationship)). - **Only SQL dialects are accepted.** An expression entry whose `dialect` is `MDX`, `TABLEAU`, `MAQL` or `THOUGHTSPOT` is a hard load error ([Β§2.5](#25-expressions)). *Why:* every accepted dialect names a SQL engine, and no engine on the list runs text in those languages. A diamond in a class hierarchy is allowed when both paths reach one declaration: the resolver keeps that field once. A cycle in `extends` is rejected, and so is a field that two ancestors which do not extend each other both declare or both rebind. See [Β§2.1.2](#212-inheritance-extends). ### 4.2. Relaxations (looser than Ossie) This profile relaxes Ossie in two kinds, for two different reasons. **Binding fields made optional β€” to model before binding.** Ossie marks these required; `kcmd` accepts them as optional so a model can be governed before it is bound (a purely *logical* model). At graph deploy a non-abstract dataset must still declare a `source` β€” the corresponding narrowing in [Β§4.1](#41-narrowings-stricter-than-ossie). An omitted field `expression` or relationship join column is not an error: the field is unbound and the availability pass prunes it, and a relationship with no join columns is a logical edge. - **`source` on a dataset** β€” required in Ossie; optional in `kcmd`. A non-abstract dataset bound for a graph MUST still declare one. - **`expression` on a field** β€” required in Ossie; optional in `kcmd`. A field with no `expression` is unbound and is pruned at graph generation; supply the column in a binding profile to bind it. - **`from_columns` / `to_columns` on a relationship** β€” both required in Ossie; `kcmd` allows both omitted (a logical edge). **Dialect names added for Google Cloud databases.** Ossie's `dialect` enum has no name for Spanner, AlloyDB or the Cloud SQL engines. `kcmd` accepts `SPANNER`, `MYSQL`, `POSTGRES`, `ALLOYDB` and `SQLSERVER` in both versions ([Β§2.5](#25-expressions)). A vanilla document that uses one of them is not a valid Ossie document. The operational form of every rule in this section (hard error vs. skip-with-warning, and the exact message) is in [Reference β†’ Validation](reference.md#validation). ## 5. Extensions Constructs `kcmd` adds beyond Ossie. Each is carried so an Ossie-only tool still reads the document ([Β§6](#6-the-extension-mechanism)). - **`entities` as an alias for `datasets` (extended profile only).** Under `0.2.0.dev0/google` a model MAY spell its datasets `entities:` instead of `datasets:`; setting both is an error. Pure sugar β€” identical meaning. Under vanilla Ossie the key is not accepted; use `datasets`. `kcmd`'s pull always emits `entities`. - **`extends` on a dataset.** Class inheritance β€” a subtype names its supertype datasets. Ossie `0.2.0.dev0` has no inheritance construct, so `extends` is a native key only under `0.2.0.dev0/google`. A vanilla document writes it in a `GOOGLE` block on the dataset ([Β§6](#6-the-extension-mechanism)). See [Β§2.1.2](#212-inheritance-extends). - **`abstract: true` on a dataset (extended profile only).** Marks a concept with no physical table (typically an `extends` supertype). An abstract dataset produces no node table and MUST NOT declare a `source`, keys or field expressions. Accepted only under `0.2.0.dev0/google`. - **`deployment_target` (extended profile only).** The physical destination β€” one graph in one store β€” as a first-class model-level (or profile-level) key, accepted under `0.2.0.dev0/google`. Under vanilla Ossie there is no native key; the model's deployments are written as a `GOOGLE` `custom_extensions` block carrying `{"deployments": [{"name": "…", "target": "…"}]}`, and the older `{"deploymentTargets": ["…"]}` form is still read ([Β§6](#6-the-extension-mechanism)). Grammar in [Β§7](#7-the-binding-layer). - **`actions` (extended profile only).** Model-level write operations, the write-side counterpart to a metric. An action names an operation, points at the executor that performs it, and types each parameter against the ontology. Every parameter carries one scalar value and MUST settle on one scalar datatype: it either projects a field β€” naming the `concept` and `field` (or the one-key shorthand `field: Concept.field`) whose datatype, description, label and AI context it takes β€” or declares a scalar `type` (or `datatype`) of its own. Naming an entity or a relationship as a `type` is a **hard load error**, and so is restating `type` or `datatype` beside `concept`/`field` β€” the field is the one place the datatype is stated. Its optional `guards` lists, by name, the constraints that gate it; each name MUST resolve to a constraint the same model declares, and a repeated name is a **hard load error**. Its optional `affects` lists what the call changes. Each entry is either a bare concept name or a record β€” `{concept, operation?, fields?}` β€” and both forms normalize to the same thing. Every `concept` MUST resolve to an entity or a relationship the same model declares; the two are named the same way, because the model already records which one a name is. `operation` is drawn from one closed vocabulary, `create` / `modify` / `delete`, whatever the concept is: a many-to-many relationship is backed by a junction table with fields of its own, so an edge is modified exactly as an entity is. `fields` MUST name fields of the concept and MUST NOT accompany a `delete`, which takes the whole instance. A repeated concept-and-operation pair is a **hard load error**. Accepted only under `0.2.0.dev0/google`. `kcmd` publishes an action and never calls its executor, so `affects` is the author's declaration and nothing verifies it against what the executor does. See [Modeling write operations](actions.md). An executor is a **physical binding**, not part of the declaration: it is OPTIONAL on the action, and a [binding profile](profiles.md) MAY supply or replace it, exactly as it supplies an entity's `source`. An action a profile does not mention keeps the model's executor; `executor: null` in a profile withdraws it. An action in a model a profile names MUST NOT declare a `sql` executor; merging such a model is a **hard error**. The other three kinds MAY be declared in the model and replaced per profile. An action left with no executor is declared and not performable under that binding, which the availability pass reports; it is not an error. An executor, where one is written, MUST declare exactly one kind. Three of them β€” `mcp`, `rest` and `grpc` β€” name a system that performs the write; the fourth, `sql`, carries the write itself as an ordered list of `statements`. Because that one is the only kind whose text the model can read, it is the only one with rules about that text: each statement MUST be a single `INSERT`, `UPDATE` or `DELETE`, MUST NOT contain a `;` other than a trailing one, and MUST reference only `@parameter` names the action declares. Those rules are what let every value be bound rather than interpolated, so an argument cannot reach the store as SQL. Where the key of an inserted row comes from is the statement's own business, and `affects` has no bearing on it. - **`constraints` (extended profile only).** Model-level named invariants over the ontology, accepted only under `0.2.0.dev0/google`. Each states exactly one condition, in one body. **`judgment`** is the rule in words, settled by a language model reading the attempted call: whether a discount is justified by the reason given, whether a refund note explains the exception it claims, whether a credit stays under a ceiling. Field names in a judgment are written model-qualified (`Order.discount_reason` rather than "the reason"), so the reference is checked against the model and lives in the sentence that uses it. A constraint declaring no judgment is a load error. `judgment` is the only body a constraint has, which puts a model call on every guarded write and means a threshold a subtraction would settle identically can come back differently twice. [Actions β†’ What a judgment costs](actions.md#what-a-judgment-costs) says where that leaves a rule. `expression` is a reserved key: a document stating one is refused with a sentence telling the author to restate the rule in words. A policy that branches β€” a large refund is held for a director, a disguised one is refused, a vague reason is only reported β€” becomes one constraint per branch, each with its own name, `on_violation` and `severity`, which `guards` on the action lists together. That keeps each branch independently searchable, revisable and owned. [Actions β†’ A credit policy, worked through](actions.md#a-credit-policy-worked-through) works a credit policy through end to end. A constraint takes effect only where something references it. Declaring one adds a rule to the catalog and refuses nothing, so publishing a rule cannot change what an already-working call does. `guards` on an action is the only reference the model defines, and it is where a rule over an action's parameters belongs, having no other moment to run. A rule that turns on a stored value can be declared there too, but nothing fetches that value: a judge is given the rule, the action and the arguments, so such a rule is recorded rather than enforced, and its enforcement belongs in your schema. Status: authored, validated, published, and settled at run time. Whatever dispatches a call puts each guard to a language model before the transaction opens and routes the verdict by `on_violation`; given no judge, the action is refused rather than run past its rules. The [commerce demo](../../demo/semantic-model/skill/README.md) shows what hiring a judge takes, in recorded runs β€” nothing in this repository embeds one today. Rules in [Reference β†’ Validation](reference.md#validation). A constraint says two things about a violation, under two separate keys. **`on_violation`** is what a violation does to the write that tripped it: `reject` refuses it outright and nobody may approve it, `escalate` holds it for a human decision, `warn` lets it proceed and reports it. It is required. **`severity`** is how grave a violation is β€” `critical`, `high`, `medium` or `low` β€” for ranking and reporting. It is optional, carries no default, and nothing ranks or routes on it yet. When an action's `guards` names several constraints and a write violates more than one, the strictest outcome among them applies: any `reject` refuses the call, failing that any `escalate` holds it, failing that any `warn` lets it through with the violations reported. That combination is fixed and no part of the model states it, which is why an action can name any number of guards without saying how to add them up. A constraint MUST state `on_violation`, and it MAY be any of the three words. Omitting the key is the only error: an unmarked constraint rejects, and that is too strong a consequence for an author to inherit by silence. Pairing a judgment with `reject` is the riskiest thing the format can express, because a language model can decide two identical proposals differently and `reject` leaves no appeal. It is published rather than refused, and made findable: a constraint is an entry of its own, so the unappealable rules a model states are one query over them. They are two keys because they answer different questions, and neither one implies the other. A `low` rule can still be an absolute refusal, and a `critical` one can be a warning because the organization is not yet ready to block on it. `escalate` states that an approver exists; it does not state who, because an approver role is not modeled yet. - **Binding profiles.** A separate document that supplies only the physical bindings, so one logical model serves several stores. Not part of the Ossie document; a `kcmd`-specific file alongside it ([Β§7](#7-the-binding-layer)). One construct is reserved rather than added. `0.2.0.dev0` has **no authorable M:N `association` syntax**. The construct is under consideration for a future version and is not part of the format today. ## 6. The extension mechanism The `version` selects **one** of two mutually exclusive extension surfaces (agreed with Dmitri). A document uses the carrier *or* the native keys, never both β€” which surface applies is a property of the version, not a per-key choice. **Vanilla `0.2.0.dev0` β€” the `custom_extensions` carrier.** Under vanilla Ossie every `kcmd` extension rides in Ossie's own `custom_extensions` field. An entry is: ```yaml custom_extensions: - vendor_name: GOOGLE # the namespace data: "{ … }" # an opaque, vendor-serialized string (typically JSON) ``` `custom_extensions` MAY appear on the model, a dataset, a field, a relationship, and a metric. `data` is opaque to Ossie. `kcmd` reads a `GOOGLE` block's payload into the model when it parses the document, and carries any other vendor's block verbatim. `vendor_name` matches `GOOGLE` in any letter case, so `Google` and `google` name the same vendor. A model file that pull rebuilds from the catalog metadata writes the vendor name as `GOOGLE`. An Ossie-only tool treats a `GOOGLE` block as a vendor extension it does not recognize and passes it through unchanged β€” so a vanilla `kcmd` model is still a valid Ossie document, and stripping the `GOOGLE` blocks yields plain Ossie. A model's deployments ride here as a `GOOGLE` block carrying `{"deployments": [{"name": "…", "target": "…"}]}`. The older form, `{"deploymentTargets": ["…"]}`, is still read. The native keys (`entities`, `deployments`, `deployment_target`, `profiles`, `extends`, `abstract`, metric `entity`, relationship `description`) are **not accepted** under vanilla β€” each is rejected with a message naming the `GOOGLE` `custom_extensions` block or `0.2.0.dev0/google` ([Β§4.1](#41-narrowings-stricter-than-ossie)). **Extended `0.2.0.dev0/google` β€” native keys.** Under the extended profile the same information is written as first-class keys β€” `entities`, `deployment_target`, `extends`, `abstract` β€” read directly, with no `data` string to encode. There is therefore nothing left for the carrier to carry, and a `custom_extensions` block is **not accepted** under `/google` (also an unknown key). This is the profile `kcmd`'s own tools emit: readable, but no longer a vanilla-Ossie document. **Converting between the two** is the `version` plus a mechanical fold between vanilla `GOOGLE` blocks and `/google` native keys or profile files: - **Model `GOOGLE` block:** `{"profiles": […], "deployments": […]}` (plus legacy `deploymentTargets` on read) folds to `<model>.profile.<name>.yaml` sidecar files and a native `deployments:` key under `/google`, and back. - **Dataset `GOOGLE` block:** `{"extends": ["…"]}` folds to `extends:` on the entity, and back. - **Relationship `GOOGLE` block:** `{"description": "…"}` folds to `description:` on the relationship, and back. - **Metric `GOOGLE` block:** `{"entity": "…"}` folds to `entity:` on the metric, and back. Both flavors describe the same model. The `entities` spelling, `abstract`, `actions`, and `constraints` have no vanilla form. What survives a push and a pull through each surface is in [What push and pull preserve](fidelity.md). ## 7. The binding layer Ossie already binds a model to physical data: a dataset names a `source` table, a field names a column `expression`, and a relationship names its join columns. Those are Ossie-native, not `kcmd` additions. What `kcmd` adds is the layer that says **where a model deploys** and lets **one logical model serve several stores**: - **`deployment_target` on the model** β€” where the model deploys. A `kcmd` extension; Ossie has no deployment construct ([Β§7.2](#72-deployment-target)). - **binding profiles** β€” a separate document that varies the physical bindings per store. A `kcmd` extension ([Β§7.3](#73-binding-profiles)). - **an unbound field** β€” a field with no column under a given binding, expressed by omitting its `expression` (there is no `unbound` flag). `kcmd` also **relaxes** the Ossie-native bindings β€” `source`, field `expression`, and relationship join columns β€” from required to optional ([Β§4.2](#42-relaxations-looser-than-ossie)), so a model with none of them is a complete *logical* model that can be governed in Knowledge Catalog as-is. The bindings are required only to deploy to a store. Every physical binding is profile-swappable. **`source`, an entity's key columns, a field's `expression`, a relationship's join columns, and an action's `executor`** are supplied or overridden by a [binding profile](#73-binding-profiles), so one logical model deploys to several stores, in each store's own table and column names, and performs its writes by whatever mechanism each store has. A named profile states every relationship's join columns itself, and its own keys for each entity whose keys the model file states ([Β§7.3](#73-binding-profiles)). Key shapes may differ between profiles. In every binding, a relationship's join columns cover a key of its target. ### 7.1. Table sources `source` is the GCP resource name of the backing table, with no leading `//`: ``` bigquery.googleapis.com/projects/<p>/datasets/<d>/tables/<table> spanner.googleapis.com/projects/<p>/instances/<i>/databases/<db>/tables/<table> ``` A resource URI, the same name with a leading `//`, and a Knowledge Catalog name such as `bigquery:<p>.<d>.<t>` are rejected, and the error names the resource name to write. A table outside Google Cloud is written as the entry name of its Knowledge Catalog entry, `projects/<p>/locations/<l>/entryGroups/<g>/entries/<id>`. An entry whose `fullyQualifiedName` names something other than a table or view, such as a schema, is rejected. [profiles.md](profiles.md#sources) lists every accepted store. Ossie requires `source` on every dataset; `kcmd` relaxes it to optional ([Β§4.2](#42-relaxations-looser-than-ossie)) so a purely logical model loads, but a non-abstract dataset in a model bound for a graph MUST have one. ### 7.2. Deployment target `deployment_target` is a single GCP resource name, with no leading `//`, whose host selects the store. A resource URI or a Knowledge Catalog name is rejected, and the error names the form to write: ``` # BigQuery Graph β€” analytical bigquery.googleapis.com/projects/<p>/datasets/<d>/propertyGraphs/<g> # Spanner Graph β€” operational spanner.googleapis.com/projects/<p>/instances/<i>/databases/<db>/graphs/<g> ``` Each is the name Dataplex gives the graph's entry: a BigQuery graph is under `propertyGraphs/`, a Spanner graph under `graphs/`. A Spanner graph written with `propertyGraphs/` is rejected, and the error names its `graphs/` name. The name MUST match one of these two shapes. A model bound for a graph MUST declare exactly one ([Β§4](#4-narrowings-and-relaxations)). The identifier segments are restricted to `[A-Za-z0-9_-]`. The native `deployment_target:` key shown here is the extended-profile form; under vanilla Ossie the same target rides in a `GOOGLE` `custom_extensions` block ([Β§6](#6-the-extension-mechanism)). ### 7.3. Binding profiles A **binding profile** is a separate document that supplies only the physical bindings for a model, so one logical model can deploy to several stores from one definition. A profile: - MUST set only physical-binding keys: `source`, `primary_key`, `unique_keys`, `fields` and `fields_exclude` on an entity; `from_columns` and `to_columns` on a relationship; `expression` on a field; `executor` on an action; and `metrics_exclude`. Setting any logical key (a new field, a datatype, a relationship's endpoints, an action's `guards`, …) in a profile is an error β€” the logical model owns those. - binds by `name` at each level, including a field the entity inherits. It is an overlay: a field the profile does not name keeps the model file's binding, and a field in `fields_exclude` is left **unbound** for that profile; there is no `unbound` flag. A field whose expression reads a column MUST be restated or excluded, because that column belongs to the model file's table. A column is a reference that no entity name qualifies. An action the profile does not mention keeps the model's executor, and `executor: null` withdraws it. - MUST state `primary_key` for an entity whose primary key the model file states, and `unique_keys` for one whose unique keys it states. - MAY give a field any expression the logical model could: a column, a computation, or the `dialects` form. The model document lives at `catalog/EntryGroups/<entryGroup>/<model>.yaml` (sidecar files `*.aspects.yaml` / `*.overview.yaml` are not models). Profiles live beside it as `catalog/EntryGroups/<entryGroup>/<model>.profile.<name>.yaml`, and a profile file's `name:` MUST equal its `<name>`. The older `<model>.profiles/<name>.yaml` directory is still read when a model has no sibling profile file. A vanilla `0.2.0.dev0` model lists its profiles instead in a `GOOGLE` block on the model, as `{"profiles": [...]}`, each with the keys a profile file takes. The name `default` is reserved for the model's own inline bindings and MUST NOT be used as a profile name. A `default_profile` in `catalog.yaml` selects which profile the default push uses. The full merge behavior and worked examples are in [Binding profiles](profiles.md). ## 8. Versioning and compatibility policy - **What declaring a version guarantees.** The constructs in [Β§2](#2-document-shape) and the extensions in [Β§5](#5-extensions) are what `kcmd` reads and writes. The `version` is a **gate**, not advice: it is required, must be `0.2.0.dev0` or `0.2.0.dev0/google`, and selects the legal extension surface ([Β§6](#6-the-extension-mechanism)); any other value is a hard error. - **When Ossie changes.** A new Ossie construct is added to [Β§3](#3-conformance-summary) as a row β€” *as-is* if adopted unchanged, *narrowed* if constrained. Growth is a table entry, not a rewrite; this document's size tracks `kcmd`'s decisions, not Ossie's surface. - **Forward direction of each extension.** A native `/google` key is either a candidate to upstream into Ossie (e.g. `deployment_target`, `entities`) or a Google-store specific that stays `kcmd`'s own. The vanilla version stays a strict Ossie superset: its only extension is the `GOOGLE` `custom_extensions` carrier, and constructs with no vanilla form (`abstract`, `actions`, `constraints`, the `entities` spelling) are simply unavailable there β€” a model that needs them uses `0.2.0.dev0/google`. - **Reserved constructs.** `association` (M:N) is reserved. It is under consideration for a future version and cannot be authored today, so a document MUST NOT rely on it in `0.2.0.dev0`. ## Appendix: annotated example A logical model plus one binding profile. ```yaml # catalog/EntryGroups/sales/sales.yaml β€” the logical model (concepts + edge join keys; # table and column bindings live in the profile below β€” see Β§7) version: "0.2.0.dev0" # vanilla Ossie: uses no native keys, so this file is portable semantic_model: - name: sales datasets: - name: orders primary_key: [order_id] fields: - { name: order_id, datatype: Integer } - { name: net_amount, datatype: Decimal, label: "Net amount" } - name: customer primary_key: [customer_id] fields: - { name: customer_id, datatype: Integer } - { name: name, datatype: String } ai_context: synonyms: [client, account] relationships: - name: placed_by from: orders to: customer from_columns: [customer_id] to_columns: [customer_id] metrics: - name: revenue expression: SUM(orders.net_amount) # resolves to one entity β†’ a BigQuery measure ``` ```yaml # catalog/EntryGroups/sales/sales.profiles/analytical.yaml β€” physical binding only version: "0.2.0.dev0/google" # native deployment_target key -> extended profile semantic_model: - name: sales deployment_target: "bigquery.googleapis.com/projects/acme/datasets/sales/propertyGraphs/sales" datasets: - name: orders source: bigquery.googleapis.com/projects/acme/datasets/raw/tables/orders fields: - { name: order_id, expression: o_id } - { name: net_amount, expression: o_net } - name: customer source: bigquery.googleapis.com/projects/acme/datasets/raw/tables/customer fields: - { name: customer_id, expression: c_id } - { name: name, expression: c_name } ``` --- ### Toolbox Mdcode Docs Semantic Model Osi Google Schema.Json (`toolbox/mdcode/docs/semantic-model/osi-google-schema.json`) { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://cloud.google.com/dataplex/schemas/osi-google-schema-0.2.0.dev0.json", "title": "Google-Extended Open Semantic Interchange (OSI) Specification Schema (0.2.0.dev0/google)", "description": "JSON Schema (Draft 2020-12) for Google-flavor Open Semantic Interchange (OSI) documents, version '0.2.0.dev0/google'. The root validates a model document; #/$defs/ProfileDocument validates a profile file. Google extensions are native keys (entities, deployments, extends, abstract, a metric's entity, a relationship's description, actions, constraints). Third-party custom_extensions, whose vendor_name is not 'GOOGLE', are carried verbatim. The vendor_name match ignores letter case, so 'google' and 'Google' are GOOGLE too.", "type": "object", "required": [ "version", "semantic_model" ], "additionalProperties": false, "properties": { "version": { "type": "string", "const": "0.2.0.dev0/google", "description": "Google-extended OSI schema version identifier." }, "semantic_model": { "type": "array", "minItems": 1, "description": "The semantic model. A document holds exactly one.", "items": { "$ref": "#/$defs/SemanticModel" }, "maxItems": 1 } }, "$defs": { "Identifier": { "type": "string", "minLength": 1, "pattern": "^[A-Za-z][A-Za-z0-9_]*$", "description": "An entity, metric or relationship name. A letter, then letters, digits or underscores, up to 500 characters.", "maxLength": 500 }, "ModelName": { "type": "string", "minLength": 1, "pattern": "^[A-Za-z][A-Za-z0-9_-]*$", "maxLength": 500, "description": "A model, profile or deployment name. A letter, then letters, digits, underscores or hyphens, up to 500 characters." }, "Description": { "type": "string", "description": "Human-readable description." }, "EntryDescription": { "type": "string", "maxLength": 2000, "description": "The description of a model, entity, metric or relationship, stored as a Knowledge Catalog entry description: up to 2000 characters." }, "NonEmptyText": { "type": "string", "minLength": 1, "description": "A non-empty string with no pattern and no length limit. Used for the names and descriptions of actions, action parameters and constraints." }, "NonBlankText": { "type": "string", "pattern": "\\S", "description": "A string holding at least one non-whitespace character." }, "Dialect": { "type": "string", "enum": [ "ANSI_SQL", "BIGQUERY", "SPANNER", "MYSQL", "POSTGRES", "ALLOYDB", "SNOWFLAKE", "DATABRICKS", "SQLSERVER" ], "description": "SQL dialect, matched exactly." }, "DataType": { "type": "string", "pattern": "^([Ss][Tt][Rr][Ii][Nn][Gg]|[Ii][Nn][Tt][Ee][Gg][Ee][Rr]|[Dd][Ee][Cc][Ii][Mm][Aa][Ll]|[Ff][Ll][Oo][Aa][Tt]|[Bb][Oo][Oo][Ll][Ee][Aa][Nn]|[Dd][Aa][Tt][Ee]|[Tt][Ii][Mm][Ee]|[Dd][Aa][Tt][Ee][Tt][Ii][Mm][Ee]|[Dd][Aa][Tt][Ee][Tt][Ii][Mm][Ee][Tt][Zz]|[Oo][Pp][Aa][Qq][Uu][Ee])$", "description": "Logical data type: String, Integer, Decimal, Float, Boolean, Date, Time, DateTime, DateTimeTz or Opaque, in any casing." }, "NonEmptySql": { "type": "string", "minLength": 1, "pattern": "\\S", "description": "Non-empty, non-whitespace SQL expression or bare column name." }, "DialectExpression": { "type": "object", "required": [ "dialect", "expression" ], "additionalProperties": false, "properties": { "dialect": { "$ref": "#/$defs/Dialect" }, "expression": { "$ref": "#/$defs/NonEmptySql" } } }, "Expression": { "description": "SQL expression: either a plain string, shorthand for a single ANSI_SQL variant, or an object listing one variant per dialect.", "oneOf": [ { "$ref": "#/$defs/NonEmptySql" }, { "type": "object", "required": [ "dialects" ], "additionalProperties": false, "properties": { "dialects": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/DialectExpression" } } } } ] }, "AiContext": { "description": "AI context metadata: either an instructions string or an object with instructions, synonyms and examples. Custom members go inside custom, the one open key.", "oneOf": [ { "type": "string" }, { "type": "object", "additionalProperties": false, "properties": { "instructions": { "type": "string" }, "synonyms": { "type": "array", "items": { "type": "string" } }, "examples": { "type": "array", "items": { "type": "string" } }, "custom": { "type": "object", "description": "Custom ai_context members, carried verbatim and never interpreted. Any key and any value, except the three names this object defines and __proto__.", "propertyNames": { "not": { "enum": [ "instructions", "synonyms", "examples", "__proto__" ] } } } } } ] }, "CustomExtension": { "type": "object", "required": [ "vendor_name", "data" ], "additionalProperties": false, "description": "Third-party vendor custom extension, carried verbatim. In '0.2.0.dev0/google', Google extensions are native properties, so vendor_name 'GOOGLE' is not permitted. The match ignores letter case, so 'google' and 'Google' are not permitted either.", "properties": { "vendor_name": { "type": "string", "minLength": 1, "not": { "pattern": "^[Gg][Oo][Oo][Gg][Ll][Ee]$" } }, "data": { "type": "string" } } }, "TableSource": { "type": "string", "description": "The GCP resource name, with no leading //, of a BigQuery, BigLake, Spanner, AlloyDB or Cloud SQL table, or the Knowledge Catalog entry name of a table outside Google Cloud, in an entry group whose id does not begin with @.", "anyOf": [ { "title": "BigQuery table", "pattern": "^bigquery\\.googleapis\\.com/projects/[^/]+/datasets/[^/.]+/tables/[^/.]+$" }, { "title": "BigLake table", "pattern": "^biglake\\.googleapis\\.com/projects/[^/]+/catalogs/[^/.]+/namespaces/[^/.]+/tables/[^/.]+$" }, { "title": "Spanner table", "pattern": "^spanner\\.googleapis\\.com/projects/[^/]+/instances/[^/]+/databases/[^/]+/tables/[^/]+$" }, { "title": "AlloyDB table", "pattern": "^alloydb\\.googleapis\\.com/projects/[^/]+/locations/[^/]+/clusters/[^/]+/databases/[^/]+/schemas/[^/]+/tables/[^/]+$" }, { "title": "Cloud SQL for MySQL table", "pattern": "^cloudsql\\.googleapis\\.com/projects/[^/]+/locations/[^/]+/instances/[^/]+/databases/[^/]+/tables/[^/]+$" }, { "title": "Cloud SQL for PostgreSQL or SQL Server table", "pattern": "^cloudsql\\.googleapis\\.com/projects/[^/]+/locations/[^/]+/instances/[^/]+/databases/[^/]+/schemas/[^/]+/tables/[^/]+$" }, { "title": "Table outside Google Cloud", "pattern": "^projects/[^/]+/locations/[^/]+/entryGroups/[^/@][^/]*/entries/\\S+$" } ] }, "DeploymentTarget": { "type": "string", "description": "The GCP resource name, with no leading //, of a BigQuery graph, a Spanner graph or an AlloyDB database.", "anyOf": [ { "title": "BigQuery graph", "pattern": "^bigquery\\.googleapis\\.com/projects/[A-Za-z0-9_-]+/datasets/[A-Za-z0-9_-]+/propertyGraphs/[A-Za-z0-9_-]+$" }, { "title": "Spanner graph", "pattern": "^spanner\\.googleapis\\.com/projects/[A-Za-z0-9_-]+/instances/[A-Za-z0-9_-]+/databases/[A-Za-z0-9_-]+/graphs/[A-Za-z0-9_-]+$" }, { "title": "AlloyDB database", "pattern": "^alloydb\\.googleapis\\.com/projects/[A-Za-z0-9_-]+/locations/[A-Za-z0-9_-]+/clusters/[A-Za-z0-9_-]+/databases/[A-Za-z0-9_-]+$" } ] }, "Dimension": { "type": "object", "additionalProperties": false, "properties": { "is_time": { "type": "boolean" } } }, "Field": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/FieldName" }, "expression": { "$ref": "#/$defs/Expression" }, "datatype": { "$ref": "#/$defs/DataType" }, "label": { "type": "string" }, "description": { "$ref": "#/$defs/Description" }, "dimension": { "$ref": "#/$defs/Dimension" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "custom_extensions": { "type": "array", "items": { "$ref": "#/$defs/CustomExtension" } } } }, "Entity": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/Identifier" }, "source": { "$ref": "#/$defs/TableSource" }, "abstract": { "type": "boolean", "default": false }, "extends": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/Identifier" } }, "primary_key": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/ColumnName" } }, "unique_keys": { "type": "array", "items": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/ColumnName" } } }, "description": { "$ref": "#/$defs/EntryDescription" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "fields": { "type": "array", "items": { "$ref": "#/$defs/Field" } }, "custom_extensions": { "type": "array", "items": { "$ref": "#/$defs/CustomExtension" } } }, "allOf": [ { "description": "An abstract entity has no table, so it declares no source, no key and no field expression.", "if": { "properties": { "abstract": { "const": true } }, "required": [ "abstract" ] }, "then": { "properties": { "source": false, "primary_key": false, "unique_keys": false, "fields": { "items": { "properties": { "expression": false } } } } } } ] }, "BoundEntity": { "description": "An entity that names its source unless it is abstract.", "if": { "properties": { "abstract": { "const": true } }, "required": [ "abstract" ] }, "else": { "required": [ "source" ] } }, "Relationship": { "type": "object", "required": [ "name", "from", "to" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/Identifier" }, "from": { "$ref": "#/$defs/Identifier" }, "to": { "$ref": "#/$defs/Identifier" }, "from_columns": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/ColumnName" } }, "to_columns": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/ColumnName" } }, "description": { "$ref": "#/$defs/EntryDescription" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "custom_extensions": { "type": "array", "items": { "$ref": "#/$defs/CustomExtension" } } }, "dependentRequired": { "from_columns": [ "to_columns" ], "to_columns": [ "from_columns" ] } }, "Metric": { "type": "object", "required": [ "name", "expression" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/Identifier" }, "entity": { "$ref": "#/$defs/Identifier" }, "expression": { "$ref": "#/$defs/Expression" }, "datatype": { "$ref": "#/$defs/DataType" }, "description": { "$ref": "#/$defs/EntryDescription" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "custom_extensions": { "type": "array", "items": { "$ref": "#/$defs/CustomExtension" } } } }, "ProfileField": { "type": "object", "required": [ "name", "expression" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/FieldName" }, "expression": { "$ref": "#/$defs/Expression" } } }, "ProfileEntity": { "type": "object", "required": [ "name", "source" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/Identifier" }, "source": { "$ref": "#/$defs/TableSource" }, "primary_key": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/ColumnName" } }, "unique_keys": { "type": "array", "items": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/ColumnName" } } }, "fields": { "type": "array", "items": { "$ref": "#/$defs/ProfileField" } }, "fields_exclude": { "description": "Field names left unbound under this profile. Unlike metrics_exclude, '*' is not supported; list the field names explicitly.", "type": "array", "items": { "$ref": "#/$defs/FieldName" }, "uniqueItems": true } } }, "ProfileRelationship": { "type": "object", "required": [ "name", "from_columns", "to_columns" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/Identifier" }, "from_columns": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/ColumnName" } }, "to_columns": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/ColumnName" } } } }, "Deployment": { "type": "object", "required": [ "name", "target" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/ModelName" }, "target": { "$ref": "#/$defs/DeploymentTarget" }, "profile": { "$ref": "#/$defs/ModelName" } } }, "Executor": { "type": [ "object", "null" ], "additionalProperties": false, "properties": { "mcp": { "type": "object", "required": [ "server", "tool" ], "additionalProperties": false, "properties": { "server": { "$ref": "#/$defs/NonBlankText" }, "tool": { "$ref": "#/$defs/NonBlankText" } } }, "rest": { "type": "object", "required": [ "endpoint", "method" ], "additionalProperties": false, "properties": { "endpoint": { "$ref": "#/$defs/NonBlankText" }, "method": { "$ref": "#/$defs/NonBlankText" } } }, "grpc": { "type": "object", "required": [ "service", "method" ], "additionalProperties": false, "properties": { "service": { "$ref": "#/$defs/NonBlankText" }, "method": { "$ref": "#/$defs/NonBlankText" } } }, "sql": { "type": "object", "required": [ "statements" ], "additionalProperties": false, "properties": { "statements": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/NonBlankText" } } } } }, "minProperties": 1, "maxProperties": 1, "description": "How an action is performed: exactly one of mcp, rest, grpc or sql. null means nothing performs it." }, "ActionParameter": { "type": "object", "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/NonEmptyText" }, "type": { "type": "string" }, "datatype": { "type": "string" }, "concept": { "type": "string" }, "field": { "type": "string" }, "description": { "$ref": "#/$defs/NonEmptyText" }, "label": { "type": "string" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "required": { "type": "boolean" }, "default": {} } }, "AffectedConcept": { "oneOf": [ { "type": "string" }, { "type": "object", "required": [ "concept" ], "additionalProperties": false, "properties": { "concept": { "type": "string" }, "operation": { "type": "string", "enum": [ "create", "modify", "delete" ] }, "fields": { "type": "array", "items": { "type": "string" } } } } ] }, "Action": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/NonEmptyText" }, "description": { "$ref": "#/$defs/NonEmptyText" }, "executor": { "$ref": "#/$defs/Executor" }, "parameters": { "type": "array", "items": { "$ref": "#/$defs/ActionParameter" } }, "guards": { "type": "array", "items": { "type": "string" } }, "affects": { "type": "array", "items": { "$ref": "#/$defs/AffectedConcept" } }, "ai_context": { "$ref": "#/$defs/AiContext" } } }, "Constraint": { "type": "object", "required": [ "name", "judgment", "on_violation" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/NonEmptyText" }, "judgment": { "$ref": "#/$defs/NonBlankText" }, "description": { "$ref": "#/$defs/NonEmptyText" }, "on_violation": { "type": "string", "enum": [ "reject", "escalate", "warn" ] }, "severity": { "type": "string", "enum": [ "low", "medium", "high", "critical" ] }, "ai_context": { "$ref": "#/$defs/AiContext" } } }, "SemanticModel": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "$ref": "#/$defs/ModelName" }, "description": { "$ref": "#/$defs/EntryDescription" }, "ai_context": { "$ref": "#/$defs/AiContext" }, "deployments": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/Deployment" } }, "entities": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/Entity" } }, "datasets": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/Entity" } }, "relationships": { "type": "array", "items": { "$ref": "#/$defs/Relationship" } }, "metrics": { "type": "array", "items": { "$ref": "#/$defs/Metric" } }, "actions": { "type": "array", "items": { "$ref": "#/$defs/Action" } }, "constraints": { "type": "array", "items": { "$ref": "#/$defs/Constraint" } }, "custom_extensions": { "type": "array", "items": { "$ref": "#/$defs/CustomExtension" } } }, "oneOf": [ { "required": [ "entities" ] }, { "required": [ "datasets" ] } ], "allOf": [ { "description": "A deployment that names no profile, or the profile 'default', uses the model file's own bindings, so every concrete entity then names its source and every relationship its join columns.", "if": { "required": [ "deployments" ], "properties": { "deployments": { "contains": { "anyOf": [ { "not": { "required": [ "profile" ] } }, { "required": [ "profile" ], "properties": { "profile": { "const": "default" } } } ] } } } }, "then": { "properties": { "entities": { "items": { "$ref": "#/$defs/BoundEntity" } }, "datasets": { "items": { "$ref": "#/$defs/BoundEntity" } }, "relationships": { "items": { "required": [ "from_columns", "to_columns" ] } } } } } ] }, "FieldName": { "type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_]{0,299}$", "description": "A field name. A BigQuery column name: a letter or underscore, then letters, digits or underscores, up to 300 characters." }, "ProfileDocument": { "type": "object", "required": [ "name" ], "additionalProperties": false, "properties": { "name": { "allOf": [ { "$ref": "#/$defs/ModelName" }, { "not": { "pattern": "^[Dd][Ee][Ff][Aa][Uu][Ll][Tt]$" } } ], "description": "A profile name. 'default', in any casing, is reserved." }, "entities": { "type": "array", "items": { "$ref": "#/$defs/ProfileEntity" } }, "relationships": { "type": "array", "items": { "$ref": "#/$defs/ProfileRelationship" } }, "metrics_exclude": { "description": "Metrics this profile does not make available. \"*\" covers every metric the model declares, including ones added later. A list names them explicitly, and may not contain \"*\".", "oneOf": [ { "const": "*" }, { "type": "array", "items": { "type": "string", "not": { "const": "*" } } } ] }, "actions": { "type": "array", "description": "Per-profile executor bindings for actions the base model declares. An action's executor is a physical binding, like an entity's source, so a profile may set it; null means the action is performed by nothing under this profile. Nothing else about an action may be set here.", "items": { "type": "object", "additionalProperties": false, "required": [ "name" ], "properties": { "name": { "$ref": "#/$defs/NonEmptyText" }, "executor": { "$ref": "#/$defs/Executor" } } } } }, "title": "Profile document", "description": "The whole of a <model>.profile.<name>.yaml file. Validate a profile file against #/$defs/ProfileDocument and a model document against the root." }, "ColumnName": { "type": "string", "minLength": 1, "pattern": "^(`[^`]+`|[^\\s().+\\-*/%=<>!|&^~,`]+)$", "description": "A physical column name in a key or join-column slot, passed through as written. A backtick-quoted name passes whatever it contains. An unquoted name may not contain whitespace, parentheses, a dot, a comma, a backtick or a SQL operator, which is how kcmd tells a column from an expression." } } } --- ### Toolbox Mdcode Docs Semantic Model Owl Import (`toolbox/mdcode/docs/semantic-model/owl-import.md`) # Importing an OWL ontology An OWL ontology and a semantic model share a backbone: **classes β‰ˆ entities**, **object properties β‰ˆ relationships**, **datatype properties β‰ˆ fields**. `kcmd` does not learn a second format β€” it **converts OWL into a semantic model** once, then the ontology rides the normal [`kcmd push` / `kcmd pull`](README.md) workflow. The semantic model stays the single canonical form. The converter maps the OWL constructs that have a clean BigQuery Graph shape to **native** semantic-model fields β€” class β†’ node, object property β†’ edge, datatype property β†’ property, plus **class hierarchies** (`rdfs:subClassOf` β†’ entity `extends`, see [Class hierarchies](#class-hierarchies-rdfssubclassof)). Import is **one-way and lossy by design**: constructs with no native semantic-model home β€” property inheritance (`rdfs:subPropertyOf`), the inverse / equivalence / disjointness cross-references, the property characteristics, the set-level axioms, and per-term annotations (`rdfs:seeAlso`, `owl:deprecated`, …) β€” are **not imported**. The result is a clean semantic model, never an OWL document wrapped in opaque metadata. What maps and what drops is listed in [How each OWL construct maps](#how-each-owl-construct-maps) and [Limitations](#limitations). ## 1. The OWL file `sales.owl.ttl` β€” an ontology header, two classes with keys, datatype properties across several `xsd` types, and one object property. Nothing here needs an OWL reasoner: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The same ontology as an RDF graph. Every arc is a triple: a class (the subject) points through a property (the predicate) to its object. Each **datatype property** points to its `xsd` range β€” a literal type, drawn as a plain box β€” and becomes a field. The **object property** `ex:placedBy` points from one class to another and becomes the relationship. The two classes become the datasets. ```mermaid graph LR Customer(["ex:Customer"]) Order(["ex:Order"]) Order -- "ex:placedBy" --> Customer Customer -- "ex:customerId" --> cid["xsd:string"] Customer -- "ex:email (inverse-functional)" --> cem["xsd:string"] Customer -- "ex:customerName" --> cnm["xsd:string"] Customer -- "ex:signupDate" --> csd["xsd:date"] Customer -- "ex:isVip" --> civ["xsd:boolean"] Order -- "ex:orderId" --> oid["xsd:string"] Order -- "ex:orderAmount" --> oam["xsd:decimal"] Order -- "ex:quantity" --> oqt["xsd:integer"] Order -- "ex:orderDate" --> odt["xsd:date"] classDef cls fill:#dae8fc,stroke:#6c8ebf,color:#000; classDef lit fill:#f5f5f5,stroke:#999999,color:#000; class Customer,Order cls; class cid,cem,cnm,csd,civ,oid,oam,oqt,odt lit; ``` Classes (rounded) are the resources that become datasets; the `xsd` boxes are literal types that become each field's `datatype`. ## 2. The command ```console $ kcmd owl import sales.owl.ttl converted 2 classes, 1 object property, 9 datatype properties wrote catalog/EntryGroups/<entryGroup>/sales.yaml ``` The model name comes from the file (`sales.owl.ttl` β†’ `sales`). By default the document is written into the semantic-model layout dir so the next `kcmd push` picks it up; pass `--out <path>` to write it elsewhere. The output uses the block layout shown below; pass `--compact` for the compact flow layout (`primary_key: [id]`, inline `{ name, datatype }` field and relationship maps) instead. Either way it is a purely **logical** model β€” `kcmd push` publishes it to Knowledge Catalog as-is, while a BigQuery or Spanner Graph deploy needs each relationship's join columns added to the model plus a [binding profile](profiles.md) (see [Β§4](#4-going-from-ontology-to-a-running-graph-binding)). ## 3. The semantic model it produces β€” `sales.yaml` The `Customer` dataset and the `placedBy` edge (the `Order` dataset follows the same shape). The structure, keys, and values are exactly what the converter emits; the inline `#` comments are annotations added here for the walkthrough β€” the real output has none: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This is a purely **logical model**: an ontology declares meaning, not physical tables, so entities carry no `source`, fields no `expression`, and relationships no join columns. Before a graph deploy a [binding profile](profiles.md) supplies the sources and field expressions, and you add each edge's join columns to the model (they are logical, not a binding). The output carries no term IRIs and no `custom_extensions`: a term's identity is its name, so a cleanly-mapped construct needs nothing more. One kind of ontology goes further β€” a **class hierarchy** (`rdfs:subClassOf`), shown next as an extension of *this same sales domain* (see [Class hierarchies](#class-hierarchies-rdfssubclassof)). Everything OWL can say that the semantic model cannot is **dropped on import**, not carried. The source namespace is recorded only as a human-readable fallback: when the ontology header has no comment of its own, the model `description` names the base IRI it was imported from. This ontology has a header comment, so the namespace appears nowhere above. ### How each OWL construct maps | OWL | Semantic model | Notes | |---|---|---| | `owl:Ontology` header | model `description`, `ai_context`, version | comment β†’ `description`; labels β†’ `ai_context.synonyms`; `skos:example` β†’ `ai_context.examples`; `owl:versionInfo` β†’ appended to `description` | | `owl:Class` | `datasets[]` entry | a logical entity β€” **no `source`** (a binding profile adds one before a graph deploy) | | `owl:DatatypeProperty` | `fields[]` on **each** domain's dataset | a property with several `rdfs:domain` values lands on each; a logical field β€” **no `expression`** (a binding profile maps it to a column) | | `owl:ObjectProperty` | `relationships[]` | `from` = domain, `to` = range; a **logical edge with no join columns** (you add `from_columns`/`to_columns` to the model before a graph deploy, see [binding](#4-going-from-ontology-to-a-running-graph-binding)); with several `rdfs:domain`/`rdfs:range` values only the first of each is kept (a relationship is one source β†’ one destination) and the rest are warned | | `rdfs:subClassOf` (named superclass) | dataset `extends[]` | **entity-level inheritance only** β€” records the parent(s) in document order; not read on object properties (see [Class hierarchies](#class-hierarchies-rdfssubclassof)) | | `rdfs:subPropertyOf`, `owl:inverseOf`, `owl:equivalentClass`, `owl:disjointWith`, `owl:oneOf`, `owl:equivalentProperty`, `owl:propertyDisjointWith`, `owl:propertyChainAxiom`, `owl:AllDisjointClasses`, `owl:AllDisjointProperties`, `owl:AllDifferent`, the property characteristics, `rdfs:seeAlso`, `rdfs:isDefinedBy`, `owl:deprecated`, `owl:versionInfo` | *(dropped)* | **no native home** β€” not imported (see [Limitations](#limitations)) | | `rdfs:range xsd:*` | field `datatype` | see [Datatypes](#datatypes-rdfsrange) | | `owl:hasKey ( ... )` | dataset `primary_key` | single or composite, in list order | | `owl:InverseFunctionalProperty` | dataset `unique_keys` | a uniquely-identifying property; omitted when it is already the `primary_key`; a lone one on a keyless class is promoted to `primary_key` instead | | `rdfs:label` on a datatype property | field `label` | the field's display name (a `label` slot exists only on fields); dropped when it only respaces/recases the name | | `rdfs:label` on a class / object property | `ai_context.synonyms` | no `label` slot there, so a distinct label becomes an alternate name; dropped when redundant with the name | | extra `rdfs:label` / `skos:altLabel` / `prefLabel` / `hiddenLabel` | `ai_context.synonyms` | genuinely alternate names; feed NL search | | `skos:example` | `ai_context.examples` | sample questions / values | | `rdfs:comment`, `skos:definition`, `dcterms:`/`dc:description` | `description` | first present wins, in that order; on an object property (which has no `description` slot) the comment rides in `ai_context.instructions` | | term IRIs, `@prefix` | dropped (base IRI named in `description` **only when the header has no comment of its own**) | a term's identity is its local name; the source namespace is not otherwise carried | A datatype property whose domain is not a class, or an object property missing an endpoint, cannot be placed; it is **skipped with a warning** rather than failing the whole import. An object property that declares *more than one* `rdfs:domain` or `rdfs:range` is kept β€” a relationship maps one source to one destination, so the first of each is used and the extra endpoints are dropped with a warning (unlike a multi-domain *datatype* property, which lands on every domain). ### Datatypes (`rdfs:range`) `rdfs:range` sets a field's logical `datatype`. Physical width is not carried (it belongs to the bound table, not the ontology), so several `xsd` types collapse onto one logical type: | `xsd` range | `datatype` | |---|---| | `string`, `normalizedString`, `token`, `anyURI`, `language`, `Name`, `NCName` | `String` | | `integer`, `int`, `long`, `short`, `byte`, `nonNegativeInteger`, `positiveInteger`, `unsignedInt`, … (any width/sign) | `Integer` | | `decimal` | `Decimal` | | `float`, `double` | `Float` | | `boolean` | `Boolean` | | `date` | `Date` | | `time` | `Time` | | `dateTime` | `DateTime` | | `dateTimeStamp` | `DateTimeTz` | | anything else, or no `rdfs:range` | none (the field is written with no `datatype`) | A temporal field (`Date` / `Time` / `DateTime` / `DateTimeTz`) is additionally marked a **time dimension** (`dimension: { is_time: true }`), so downstream BigQuery Graph / BI treats it as one. ### Keys (`owl:hasKey`, `owl:InverseFunctionalProperty`) - **`owl:hasKey ( ... )`** on a class becomes the dataset's `primary_key` (its grain), in list order β€” single or composite. If any key column names no datatype property on the class (undeclared, or declared only on another class) it has no field to back it; because keeping only the columns that do exist would silently narrow a composite key to a possibly non-unique one, the **entire `primary_key` is dropped** with a warning rather than left to fail later at graph generation. - An **`owl:InverseFunctionalProperty`** (a datatype property that uniquely identifies its subject) becomes a `unique_keys` constraint β€” unless it is already the `primary_key`, in which case it is not repeated. If a class declares **no `owl:hasKey`** but has exactly **one** single-column inverse-functional property, that property is promoted to the `primary_key` (with a warning) so the dataset has a grain; ambiguous cases (several unique keys, or a composite one) are left without a `primary_key`. Keys are **logical grain**, not a binding β€” they are the entity's identity, so they come across even though the model has no physical binding. They also tell you which columns an edge's `to_columns` must reference when you add join columns to the model; see [binding](#4-going-from-ontology-to-a-running-graph-binding). ### Class hierarchies (`rdfs:subClassOf`) `rdfs:subClassOf` maps to a dataset's `extends` β€” the one keyword borrowed from [Ossie's ontology proposal](https://github.com/apache/ossie/blob/main/ontology/ontology.md) onto our existing `datasets`. Extend the sales domain with a `Person` base class that `Customer` refines (`Customer rdfs:subClassOf Person`): ```turtle ex:Person a owl:Class ; rdfs:comment "A human being." . ex:fullName a owl:DatatypeProperty ; rdfs:domain ex:Person ; rdfs:range xsd:string . ex:Customer a owl:Class ; rdfs:subClassOf ex:Person ; rdfs:comment "A person or organization that places orders." ; owl:hasKey ( ex:customerId ) . # ... Customer's own datatype properties: customerId, email, customerName, … ... ``` the `Person` and `Customer` entities come out as (`Customer` carrying `extends: [Person]`): ```yaml # Person is its own entity: - name: Person description: A human being. fields: - name: fullName datatype: String # Customer records that it extends Person and keeps ONLY its own fields; # Person's fullName is NOT flattened down. - name: Customer extends: - Person primary_key: - customerId description: A person or organization that places orders. fields: - name: customerId datatype: String # ... email, customerName, signupDate, isVip (Customer's own) ... ``` Multiple superclasses are allowed (`extends: [Person, Employee]`, in document order). A `subClassOf` whose object is a blank-node axiom (`owl:Restriction` and similar) is not a named class, so it is ignored, as are the implicit universal superclasses `owl:Thing` / `rdfs:Resource` (every class subclasses them, so they carry no inheritance information). Two boundaries to be clear about: - **The import *records* the hierarchy; the BigQuery push *resolves* it.** The importer writes `Customer` with `extends: [Person]` and **only its own fields**. A **BigQuery** push then resolves that inheritance: it emits a `LABEL Person` clause on the `Customer` node table and flattens `Person`'s fields down (fields flow down; edges do not) β€” see [Class hierarchies (`extends` β†’ labels)](reference.md#class-hierarchies-extends--labels). The **Knowledge Catalog** push is unchanged: it still publishes each entry with exactly the fields it declares (KC-side inheritance is a separate, later opt-in). Every push accepts a relationship only between concrete leaf entities. An object property whose `rdfs:domain` or `rdfs:range` is a class with subclasses imports as written, and push rejects it until you move it onto the leaf classes. - **Native inheritance is entity-level only.** `extends` lives on `entities`, never on relationships β€” the boundary is structural. `rdfs:subPropertyOf` (property / relationship inheritance) has no such native slot, so it is **not imported** (see [Limitations](#limitations)). ## 4. Going from ontology to a running graph (binding) The import gives you a **logical model**: what the domain means, with no physical binding. That is directly useful β€” `kcmd push` publishes it to Knowledge Catalog as-is, so the ontology becomes catalog metadata (entities, fields, and keys) with nothing more to fill in. Push rejects a relationship on a class with subclasses, as described above. It also rejects an object property whose `rdfs:domain` and `rdfs:range` are one class, and two object properties that join the same two classes in opposite directions: ```console $ kcmd push # publishes the logical model to Knowledge Catalog ``` A relationship publishes as a catalog link only once it has join columns (added to the model, below); a column-less edge is skipped with a warning, so the entities and fields still publish while the edge waits. A **BigQuery or Spanner Graph** deploy needs two things the import leaves open, and they belong in different places: 1. **The join columns on each edge β€” in the model.** The import gives every relationship its endpoints (`from`/`to`) but no join columns; which columns an edge joins on is a *logical* fact the model owns, not a physical binding, so you add it to the imported model itself. A key and a join column each name a column of the table, and the import names each key after the property it came from. So change each imported key to the column its property is bound to, and join on that column: ```yaml # sales.yaml: key each entity by its column, and add the join columns entities: - name: Customer primary_key: [c_custkey] # imported as [customerId] unique_keys: [[c_email]] # imported as [[email]] - name: Order primary_key: [o_orderkey] # imported as [orderId] relationships: - name: placedBy from: Order to: Customer from_columns: [o_custkey] # the Order-side foreign key to_columns: [c_custkey] # Customer's key column ``` 2. **The physical binding β€” in a profile.** Which table each entity reads, which column each field reads, and the deployment target go in a [binding profile](profiles.md): a document in the same schema, kept beside the model as `<model>.profiles/<name>.yaml`, that adds *only* the binding and leaves the logical model untouched. A profile may set `source`, field `expression`, and `deployment_target` β€” nothing logical (a `relationships` block in a profile is rejected). A field the profile does not bind is left unbound; there is no separate flag. The model and the profile merge by name at push time: ```yaml # sales.profiles/warehouse.yaml β€” physical binding for the sales model version: 0.2.0.dev0/google semantic_model: - name: sales deployment_target: bigquery.googleapis.com/projects/myproj/datasets/sales/propertyGraphs/sales datasets: - name: Customer source: bigquery.googleapis.com/projects/myproj/datasets/sales/tables/customers fields: - { name: customerId, expression: c_custkey } - { name: email, expression: c_email } # ... the remaining Customer fields ... - name: Order source: bigquery.googleapis.com/projects/myproj/datasets/sales/tables/orders fields: - { name: orderId, expression: o_orderkey } # ... the remaining Order fields ... ``` With the join columns in the model and the profile in place, a single push deploys the graph: ```console $ kcmd push --profile warehouse # merges the binding, then CREATE OR REPLACE PROPERTY GRAPH + KC entries ``` See [One logical model, many physical bindings](profiles.md) for the full profile contract β€” what a profile may and may not set, how merge and prune work, and how one logical model binds to several stores. ## Limitations Three different situations hide in "not covered": constructs already read but not yet resolved all the way downstream, constructs not read but reachable next, and constructs out of scope. **Read now β€” only downstream resolution is pending.** The **class hierarchy** (`rdfs:subClassOf`) maps to entity `extends` (see [Class hierarchies](#class-hierarchies-rdfssubclassof)); the importer handles it today and the remaining work is downstream, not in the import: - **BigQuery** push resolves it into node-table labels, inherited fields flattened down (see [Class hierarchies (`extends` β†’ labels)](reference.md#class-hierarchies-extends--labels)). - **Knowledge Catalog** is the follow-on β€” KC push still publishes each entry with only its own fields. **Not imported β€” dropped.** OWL can state things the semantic model has no native slot for. The importer reads them but does **not** carry them (an earlier version rode them along as `custom_extensions`; that was removed so an imported model is a clean semantic model). To keep any of these, model them natively after import. Dropped: - **Property inheritance** β€” `rdfs:subPropertyOf`. - **Cross-references** β€” `owl:inverseOf`, `owl:equivalentClass`, `owl:disjointWith`, `owl:equivalentProperty`, `owl:propertyDisjointWith`. - **Property characteristics** β€” symmetric / transitive / functional / reflexive / irreflexive / asymmetric. - **Enumerations** (`owl:oneOf`) and **property chains** (`owl:propertyChainAxiom`). - **Set-level axioms** β€” `owl:AllDisjointClasses`, `owl:AllDisjointProperties`, `owl:AllDifferent`. - **Per-term annotations** β€” `rdfs:seeAlso`, `rdfs:isDefinedBy`, `owl:deprecated`, `owl:versionInfo`. **Not read yet.** Cardinality (`owl:minCardinality` / `owl:maxCardinality`) lives on an anonymous `owl:Restriction` reached through `rdfs:subClassOf` β€” the last blank-node shape the converter does not yet read. Reading it needs an `owl:Restriction` reader joined back to its class. **Not read β€” out of scope.** Richer OWL/RDF beyond the schema-shaped subset this converter targets: - SHACL shapes, class expressions (`owl:unionOf` / `intersectionOf`, and any blank-node `owl:equivalentClass` / `rdfs:subClassOf`), individuals (A-box instances), and `owl:sameAs` / `owl:differentFrom` (which relate individuals). - OWL serializations other than Turtle (`.ttl`), and the reverse direction β€” semantic model β†’ OWL export. Import is one-way. --- ### Toolbox Mdcode Docs Semantic Model Profiles (`toolbox/mdcode/docs/semantic-model/profiles.md`) # One logical model, many physical bindings > **Scope.** A `deployment_target` answers two questions that are easy to > conflate: where the model is *published*, and where it *runs*. > > `kcmd push` publishes. It deploys a property graph to the backend the target > names β€” BigQuery Graph or Spanner Graph β€” and the logical model to Knowledge > Catalog, which takes it under any profile. > > An action's statements run against whatever store the profile binds. All > three backends execute SQL DML, so all three can carry a write. What differs > is the **push**: an AlloyDB target runs and does not push, because AlloyDB has > no property-graph DDL for a push to deploy, while Spanner and BigQuery do > both. > > A profile's sources take the forms listed under [Sources](#sources). A source > in any other form is rejected. A semantic model describes a business logically β€” its entities, the relationships between them, and the metrics over them β€” independent of where the data physically lives. A **binding profile** maps that one logical model onto a concrete set of sources. You keep the model as a single canonical definition and select a profile per scenario; a profile changes only where each entity reads from, never what the model means. Because every profile shares one model, `Customer`, `revenue`, and each relationship mean the same thing whichever profile serves them. ## Why a model gets more than one binding The same concept usually lives in more than one system, and a profile binds the model to one of them. The systems differ along whatever axis matters to you: - **Different backends for different consumers.** A live operational store β€” AlloyDB, Spanner, a SaaS API β€” holds the `Customer` an agent reads to check current state before it acts, and writes back to. An analytics warehouse such as BigQuery holds a copy of that same `Customer`, scaled for reporting, that dashboards and conversational-analytics agents read. One profile binds each, and every consumer inherits the same metric definitions, so `revenue` is computed the same way wherever it is asked. - **Different environments of one backend.** A dev, staging, and prod copy of one store are three profiles that differ only in the project they point at. - **Different physical layouts.** The same model can bind an Iceberg copy read through BigLake, or a partner's differently-named schema. A profile is a named binding, and its meaning is yours to decide; the cases above only illustrate the range. You author the model once and choose the profile when you deploy or query. ## How it works: a logical model and its bindings A model and its profiles form a class hierarchy, the same way an entity `extends` a supertype: - The **logical model** (`<model>.yaml`) is the **base**: the complete declaration β€” entities, fields, relationships, metrics, the grain, and the graph shape β€” with no physical binding. - A **binding profile** is a **subclass**: it supplies the physical facets the logical model leaves open, such as each entity's `source` and key columns, each field's column, and each relationship's join columns. - `kcmd push --profile <name>` **merges** the profile onto the logical model β€” matching entities, fields, relationships, and metrics **by name** β€” and deploys the result. A profile is never deployed alone: the logical model supplies every declaration, and the profile supplies the bindings its store provides. A binding may also sit inline in the logical model, as the profile named `default`. The layout on this page keeps the logical model standalone and each binding in its own file, so the split between logical and physical is visible on disk. ## Sources Each entity names where its data lives with `source`. A Google Cloud table is written as its GCP resource name, with no leading `//`, such as `bigquery.googleapis.com/…`, `biglake.googleapis.com/…`, `spanner.googleapis.com/…`, `alloydb.googleapis.com/…` or `cloudsql.googleapis.com/…`, and is read with ambient IAM. A resource URI (the name with a leading `//`) or a Knowledge Catalog name such as `bigquery:<p>.<d>.<t>` is rejected, and the error names the resource name to write. `kcmd pull` returns a model an older push stored that way as stored, and warns for each such value. A table outside Google Cloud is written as the entry name of its Knowledge Catalog entry, `projects/<p>/locations/<l>/entryGroups/<g>/entries/<id>`, and takes the dialect its entry records (see [Validation](reference.md#validation)). An entry whose `fullyQualifiedName` names something other than a table or view, such as a schema, is rejected. A name under a prefix such as `snowflake:` or `databricks:table:` is rejected. A Bigtable or Dataproc Metastore table is rejected too, since the preview does not support them yet. An entry name in a system entry group, such as `@bigquery`, is rejected with an error that says what to write instead. A bare table name, a query, and any other prefix are rejected. The source also fixes the profile's SQL dialect, so every field the profile binds needs an expression in that dialect or in `ANSI_SQL`. A Cloud SQL table's resource name is `cloudsql.googleapis.com/projects/<p>/locations/<l>/instances/<i>/databases/<db>/tables/<t>` for MySQL, or `…/databases/<db>/schemas/<s>/tables/<t>` for PostgreSQL and SQL Server. The name cannot tell PostgreSQL from SQL Server, so `kcmd` reads the dialect from the table's Knowledge Catalog entry in the `@cloudsql` entry group: a `cloudsql-mysql-table` entry gives `MYSQL`, `cloudsql-postgresql-table` gives `POSTGRES` and `cloudsql-sqlserver-table` gives `SQLSERVER`, and the matching `-view` entry types give the same dialects. `kcmd push`, `kcmd profiles` and `kcmd skills-generate` look each table up once before they load the model. The entry exists only when the instance has Dataplex integration turned on (`gcloud sql instances patch <i> --enable-dataplex-integration`), and the catalog can take a while to list a new table. When the entry cannot be read, the command stops and says so. A profile's sources all sit in one database, meaning the system one query can reach: all of BigQuery, one Spanner, AlloyDB or PostgreSQL database, one MySQL or SQL Server instance, one Snowflake account, or one Databricks metastore. These units are the same in Cloud SQL and outside it, but a Cloud SQL table and a table outside Cloud SQL are never in one database. `kcmd` reads a third-party table's database from its entry's `fullyQualifiedName`. When the entry has none, the table counts in one database with the other tables of its system. A profile moves an entity between stores by swapping its source and, when the two stores shape the data differently, the column each field reads. ## What a profile may change β€” the contract A model separates two things. **Declaration** is logical: which entities, fields, relationships, and metrics exist, what each means, how each metric computes, the grain, and the graph shape. The logical model owns all of it. **Binding** is physical: which store each entity reads from, which column each field reads, and how each action performs its write. A profile sets binding and leaves declaration alone. | A profile **may** set (physical binding) | A profile **may not** touch (logical, in the model) | |---|---| | an entity's `source` (its store URI) | which entities, fields, relationships, metrics, or actions exist, and what each means | | a field's column or computation (its `expression`), for a field the entity declares or inherits | a field's `label`, `description`, `dimension`, `datatype` | | whether a field is bound at all under this profile (`fields_exclude`) | the graph shape (`from`/`to`) and the cardinality a relationship's join columns imply | | an entity's key columns (`primary_key`, `unique_keys`), which a named profile must state itself where the model file states them | | | a relationship's join columns (`from_columns`, `to_columns`), which every named profile must state itself | | | an action's `executor` (how the write is performed) | an action's `parameters`, `guards`, and `affects` β€” what it takes, what gates it, what it changes | | the deployment target, in the older `<model>.profiles/` layout only | any `metric` definition; any `ai_context` / synonyms; a relationship or its junction `source` | An element's `name` is not overridden β€” it is the key that pairs a profile element with the model element it binds. Key and join columns name physical columns, not fields, so whether a field is bound says nothing about them; each field's column is resolved per profile from its `expression`. **Why an action's executor is a binding.** An action declares what a call does: its parameters, the constraints that gate it, and the concepts it changes. None of that changes when the same model is deployed somewhere else. The executor does, for either of two reasons. An `mcp`, `rest` or `grpc` executor is an address, and the system it names answers at a different server, endpoint or method in one environment than in another β€” nothing about where the data sits comes into it. A `sql` executor is the statement itself, and each store binds its own table and column names and speaks its own dialect, so the same write is a different statement under each binding. Either way the `executor` sits with `source` and `expression` on the physical side: the same action and the same blast radius, performed by whatever the binding points at. The two kinds sit differently in the files, though. An address may be declared in the model as a default and replaced per profile. A statement has no model-level spelling at all: **select a named profile whose model declares a `sql` executor and the command refuses**, naming the action. The single-file form, where one document is both the model and its binding, is the exception β€” its statements already sit beside the columns they name. **Bindings inherit.** A profile is an overlay: whatever it does not restate keeps what the model file says. A model may declare one default executor, and a profile overrides it only for the bindings that perform the write differently β€” an action the profile does not mention keeps the default. A field's column is the exception, because it belongs to the table the model file's expression was written for. A named profile restates each field whose expression reads a column, or names it in `fields_exclude`. A column is any reference that no entity name qualifies. Push rejects a profile that does neither, naming the field, the profile and the column. A field whose expression reads only other fields, such as `Order.gross - Order.discount`, carries over. To withdraw an inherited executor β€” a read-only binding that performs no writes at all β€” a profile writes `executor: null`, which leaves the action declared and unavailable there. **Why metrics never appear in a profile.** A metric like `SUM(OrderedAs.extendedPrice * (1 - OrderedAs.discount))` references field *names* rather than columns. Field names are stable across profiles β€” only their column bindings change β€” so the metric is correct under every profile where its fields are bound, without being restated. Where a field it references is not bound, the profile has to exclude the metric with `metrics_exclude`; the next section explains why. ## Availability follows the bindings A store holds what it holds. An operational database keeps a customer's live credit; a warehouse keeps a modeled lifetime value; neither carries the other's column. A profile binds whatever subset its store serves, and leaves the rest unbound. What a profile can answer is therefore derived from what it binds, by one rule. **A building block is available under a profile when every field it depends on is bound there. When any input is unbound, the block is unbound too, and so is anything built on it.** Availability propagates up the dependency graph from the fields a profile binds. The chain runs as far as the model does: - a field is bound when the profile gives its column, or the model file gives it an expression that reads only other fields, and the profile does not exclude it; - an entity and a relationship are always available: their key and join columns are physical columns rather than fields, so unbinding a field never removes them; - a metric is available when every field its expression references is bound and the profile does not exclude it; a cross-entity metric also needs a relationship connecting its entities; - an action is available when a binding supplies its `executor`. An action with no executor is declared and not performable here. So an operational-only field such as live credit carries its operational-only metrics with it, and a warehouse-only field such as lifetime value carries its reports; each is present where its inputs are, and unavailable everywhere else. The logical model still declares each thing once; a profile answers the part of it that its store can back. **Unbound is not null.** A bound field whose data happens to be empty β€” a customer with no phone on file β€” is null: the field exists and the value is missing. An unbound field does not exist under that profile at all, and anything that reads it is unavailable there rather than reading a null. Keeping the two apart is what lets a query fail against a store that cannot answer it instead of returning a blank that reads like real data. **A profile restates what differs and excludes what its store lacks.** A field the profile gives an `expression` is bound to it, a field the profile names in `fields_exclude` is unbound, and every other field keeps what the model file says. A field the model file leaves unbound, or binds with an expression that reads a column, has to appear in one of the two lists: leaving it out of both is an error, not a quiet way to unbind it. The logical model below binds nothing inline, so the operational profile binds `availableCredit` and excludes `lifetimeValue`, and the analytical profile does the reverse. A metric that reaches an excluded field has to be named in the profile's `metrics_exclude`. When the source table a profile binds is missing or inaccessible, validation fails and names it; a mistyped column name resolves to a real table and so surfaces at deploy, when BigQuery rejects the generated graph. ## File layout A binding may also sit inline in the logical model β€” logical and physical in a single file β€” which is how the `default` profile works, and a model with one binding needs nothing more. Keeping them in separate files, as below, is one layout among several: it keeps the logical model reusable across bindings and lets each binding be reviewed and owned on its own. The logical model is one file, and its bindings live beside it, one file per profile. A profile file is named after the model and the profile, and holds one profile: ``` catalog/EntryGroups/commerce_eg/ commerce.yaml # the logical model β€” declarations only commerce.profile.analytical.yaml # bindings for the analytics warehouse commerce.profile.operational.yaml # bindings for the operational store ``` The `<name>` in the file name must match the `name:` inside the file, and `default` is reserved for the inline bindings. A profile file carries `name`, `entities`, `relationships`, `metrics_exclude` and `actions`, and names no deployment target. `kcmd push` deploys each deployment with the profile it names. `kcmd profiles --print-store` and `kcmd skills-generate` take a profile file's store from the deployments that name the profile in `profile`. When none does, they report that there is no database to run it against. The examples on this page use the older layout, which `kcmd push` still deploys: a `commerce.profiles/` directory whose files are `semantic_model` documents in the logical model's schema, each carrying only physical facets and its own `deployment_target`. In either layout, `--profile analytical` reads `commerce.yaml` plus the analytical profile, so nothing in one binding can affect another. Beside a profile file, `commerce.yaml` must load on its own first. A half-bound join in it stops the push even when the profile supplies the missing join columns, and so does a rejected source the profile replaces. ## Example β€” one model, an analytical and an operational binding The logical model declares the business and nothing physical: no sources, no columns, no deployment target. `lifetimeValue` and `availableCredit` are both declared here, though no single store carries both: ```yaml # commerce.yaml β€” logical model version: "0.2.0.dev0/google" semantic_model: - name: commerce entities: - name: Customer fields: - { name: key, label: Customer ID } - { name: name, label: Customer Name } - { name: lifetimeValue, label: Lifetime Value } - { name: availableCredit, label: Available Credit } - name: Order fields: - { name: key, datatype: Integer } - { name: customerKey } - { name: orderDate, dimension: {is_time: true} } relationships: - name: PlacedBy from: Order to: Customer metrics: - name: order_count expression: COUNT(Order.key) - name: avg_lifetime_value expression: AVG(Customer.lifetimeValue) actions: - name: CancelOrder description: Cancel an order that has not shipped parameters: - { name: order, concept: Order, field: key } affects: - { concept: Order, operation: modify } # The default: ask the service that owns orders to cancel one. Any # binding that does not hold the rows itself uses this. executor: mcp: server: //agentregistry.googleapis.com/projects/acme-ops/locations/us/mcpServers/orders tool: cancel_order ``` The analytical binding points the model at the BigQuery warehouse. The warehouse carries the modeled `lifetimeValue` and does not hold live credit, so `availableCredit` is excluded: the model binds nothing inline, and this profile names it in `fields_exclude`. It says nothing about `CancelOrder`, so the action keeps the default executor: the warehouse reports on orders but does not own them, and cancelling one means calling the service that does. ```yaml # commerce.profiles/analytical.yaml β€” BigQuery bindings version: "0.2.0.dev0/google" semantic_model: - name: commerce deployment_target: bigquery.googleapis.com/projects/acme-analytics/datasets/sales/propertyGraphs/commerce entities: - name: Customer source: bigquery.googleapis.com/projects/acme-analytics/datasets/sales/tables/customer primary_key: [c_custkey] fields: - { name: key, expression: c_custkey } - { name: name, expression: c_name } - { name: lifetimeValue, expression: c_ltv } fields_exclude: [availableCredit] # the warehouse holds no live credit - name: Order source: bigquery.googleapis.com/projects/acme-analytics/datasets/sales/tables/orders primary_key: [o_orderkey] fields: - { name: key, expression: o_orderkey } - { name: customerKey, expression: o_custkey } - { name: orderDate, expression: o_orderdate } relationships: - { name: PlacedBy, from_columns: [o_custkey], to_columns: [c_custkey] } ``` The operational binding points the same model at the live Spanner store. Spanner holds the same customers under different table and column names, binds the live `availableCredit`, and excludes the modeled `lifetimeValue`, which it does not carry. It also owns the order rows, so it overrides `CancelOrder` to write them directly instead of calling out: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Each binding states the key columns and the `PlacedBy` join columns in its own store's column names. Neither changes the labels or the metric definitions; those live once in the logical model. `kcmd push --profile operational` merges the operational bindings and deploys to their Spanner backend, while `--profile analytical` deploys to BigQuery β€” each profile picks its own backend. The two bindings answer different parts of the same model: - `order_count` depends only on `Order.key`, so it is available analytically. The operational profile deploys to Spanner, whose graphs have no measures, so it names every metric in `metrics_exclude` with the wildcard `"*"`. - `avg_lifetime_value` depends on `Customer.lifetimeValue`. The warehouse binds it, so the metric is available analytically. The operational profile excludes the field, and a metric that reaches an excluded field has to be named in `metrics_exclude`, which the wildcard already does. - `availableCredit` is bound only operationally, so it β€” and any metric written on top of it β€” is available under the operational binding and absent under the analytical one. - `CancelOrder` is available under both, and performed differently by each: DML operationally, an MCP call analytically. One declaration, one blast radius, two mechanisms. ## Merge rules - A profile carries `entities` with their `source`, key columns, `fields` and `fields_exclude`; `relationships` with their `from_columns` and `to_columns`; `metrics_exclude`; and `actions` with their `executor`. These merge onto the logical model **by `name`**. A profile never declares a relationship or a metric: it binds a relationship's join columns and may leave metrics out, and anything else about either lives once in the model. An entry that sets anything else is rejected. - A profile entity's `fields` may name a field the entity inherits as well as one it declares, and so may its `fields_exclude`. Each applies to this entity's table only, so an entity that extends it still has an excluded field. Where a profile binds a field on a concrete ancestor to a column, each concrete descendant restates or excludes that field too. - Every named profile must state each relationship's join columns itself. The model file's join columns are the inline binding's, and do not stand in for a named profile's. In every binding, `to_columns` must contain every column of a primary or unique key of the target, a superset counting. - A named profile states its own `primary_key` for an entity whose primary key the model file states, and its own `unique_keys` for one whose unique keys it states. Key shapes may differ between profiles. - An entity or field named only in the logical model keeps its declaration and any inline column binding; a profile element whose `name` is not in the logical model is rejected. - Scalars β€” `source`, `expression`, and `deployment_target` in the older layout β€” **replace**. - A field the profile does not name **keeps** the model file's binding, which push accepts only when that binding reads no column. A field in `fields_exclude` is **unbound** under that profile. There is no `unbound` flag; `fields_exclude` is how a field is left unbound, and naming one field in both `fields` and `fields_exclude` is rejected. - An action the profile does not mention **keeps** the model's executor, if it declared one, so a model can state one default and a profile override only where the write differs; `executor: null` withdraws it explicitly. Only an `mcp`, `rest` or `grpc` executor may be that default; a `sql` one is rejected in a model the profile names, and belongs in the profile itself. - Profiles are **binding-only**: a profile sets physical facets, may leave a field unbound with `fields_exclude`, and may leave metrics out with `metrics_exclude`. It cannot add entities, fields or metrics, change the graph shape, or change what anything means. ## Command line ```bash kcmd push # deploy the graph with the default binding + Knowledge Catalog kcmd push --profile analytical # deploy the graph with the analytical bindings (BigQuery target) kcmd push --profile operational # deploy the graph with the operational bindings (Spanner target) kcmd push --all-profiles # deploy the graph once per binding profile kcmd push --profile analytical --no-kc # deploy only the graph, skip Knowledge Catalog kcmd push --no-profile # publish only to Knowledge Catalog, deploy no graph kcmd profiles # list profiles, their resolved sources, what each cannot answer, and what push would reject ``` `--profile` chooses **which physical binding** feeds the graph, and that binding's `deployment_target` selects **which graph backend** the model deploys to (BigQuery Graph or Spanner Graph). You never name the backend on the command line β€” picking the profile is picking the backend. The binding-profile axis (how many profiles the graph deploys for: `--no-profile`, the default, `--profile`, or `--all-profiles`) is separate from the **Knowledge Catalog** axis (`--no-kc`). Both default on; `--no-profile --no-kc` together is an error (nothing to deploy). **Deploying to more than one binding at once.** `--all-profiles` deploys the graph once for every defined profile β€” for example the analytical (BigQuery) and operational (Spanner) graphs in a single run. Knowledge Catalog still records one canonical view of the logical model β€” the default binding β€” so a bare `kcmd push` and a `kcmd push --all-profiles` write the same catalog entries and differ only in how many graphs deploy. A [push in two phases](reference.md#push) records the model file as written, whatever `default_profile` names. There is always a **default binding profile**: whatever `default_profile` names, or β€” if it is unset β€” the model's inline bindings (the implicit `default` profile). Because `default` names those inline bindings, it is reserved: a `<model>.profiles/default.yaml` is rejected rather than silently ignored. Set the default so a bare `kcmd push` in CI does the right thing, in `catalog.yaml`: ```yaml scope: semantic-model.acme.us.commerce_eg default_profile: analytical ``` ## Validation Profiles are checked as part of push; `--validate-only` runs the checks and writes nothing. - **Unknown profile** β€” `--profile stg` when `stg` is not defined fails and lists the profiles that are. - **Missing or ambiguous target** β€” a profile that deploys a graph must resolve to one `deployment_target`. - **Declaration override** β€” a profile that sets a `label`, `dimension`, `ai_context`, a `metric`, or a relationship's `from`/`to` is rejected, naming the offending path. - **Unknown name** β€” a profile element whose `name` is not in the logical model is rejected. Profiles bind declarations; they do not add them. - **Incomplete profile** β€” a push checks every named profile of each model it deploys or publishes, whichever binding it deploys: each profile file, and each profile a vanilla model carries in its `GOOGLE` block. A profile that leaves an entity without a source or a relationship without join columns fails the push. - **Keys and join columns** β€” every push checks the keys and join columns of the model file, each profile file and each `GOOGLE` block profile. The push fails when any binding of a relationship has `to_columns` that cover no key of the target. Where the model file states a primary key or unique keys for an entity, the push also fails when a named profile does not state its own. Profiles are not compared with each other. - **Unrestated column** β€” the push fails when a named profile neither restates nor excludes a field whose expression reads a column. The error names the field, the profile and the column. - **Deployments and field dialects** β€” every push checks each deployment against the rules of its engine, under the model file's own bindings or the profile it names, whether or not the push deploys it. Each field the model file binds also needs a text its entity's database runs. See [Validation](reference.md#validation). - **Unbound inline bindings** β€” when the model file's own bindings leave an entity without a source or a relationship without join columns, and no deployment uses them, push warns and still pushes. The warning says the model is non-executable when it declares no named profile, and that `default` is unbound when it declares one. `kcmd profiles` prints it under the `default` binding. - **Unresolvable source** β€” the BigQuery table a profile binds is probed with a dry run; a table that is missing or inaccessible fails and names it. Column names are not probed here β€” a mistyped column resolves to a real table and is caught at deploy, when BigQuery rejects the generated graph. - **Rejected DML** β€” where this profile deploys to Spanner or BigQuery, every statement in an action's `sql` executor is planned against that store without running it, and a store that refuses one fails the push and quotes back why. The profile is what picks the store, so the same statement can pass under one profile and fail under another. A profile deploying to **AlloyDB** is the exception: its statements are checked for shape but never sent anywhere, so nothing confirms the names in them. See [statements use your database names](actions.md#statements-use-your-database-names). - **Availability summary** β€” push resolves the dependency graph and prints, per profile, how many fields the binding leaves unbound and how many metrics and actions that makes unavailable. Entities and relationships stay available, since their key and join columns are physical columns rather than fields. `kcmd profiles` lists each one with the reason that stops it, so withheld coverage is stated rather than discovered later. Actions are listed apart, under `cannot run:` rather than `cannot answer:`: an action a binding drops is not a question it cannot answer, it is a write it cannot perform. ## Notes **The deployment target is a first-class key.** Under the extended `0.2.0.dev0/google` profile β€” the version these binding examples use β€” `deployment_target:` is a native model key, so a profile in the older layout sets it readably. The vanilla `0.2.0.dev0` profile has no native key for it; there the target rides in a `GOOGLE` `custom_extensions` block instead. The two are the same target written two ways, chosen by the document's `version` (see [model spec](model_spec.md#1-status-and-baseline)). **Bare-string expressions.** A field's `expression` may be written as a one-line string (`expression: c_name`) instead of the full per-dialect object. The two forms mean the same thing and expand to the same wire representation. **Dialect comes from the store.** A profile carries no SQL dialect. The dialect follows from the store a profile binds to; the engine lowers each `expression` to that store's query language when it runs. A profile chooses the data, and the execution engine chooses the dialect. **An action's statements reach the store as written.** Lowering covers a field's `expression` and stops there. kcmd passes a `sql` executor's statements through unchanged, so they name physical tables and columns and use the dialect of the store the profile binds. `CancelOrder` above writes `UPDATE Orders SET Status = 'CANCELLED'` rather than the model's `Order` and `key`, and a profile that binds a different store restates those statements for it. --- ### Toolbox Mdcode Docs Semantic Model Reference (`toolbox/mdcode/docs/semantic-model/reference.md`) # Reference Lookup companion to the [deploy guide](README.md): every flag, exactly what push creates in each destination, the validation checks, and the permissions each leg needs. For what of your metadata survives a deploy or a pull, see [What push and pull preserve](fidelity.md). ## CLI flags ### init ```bash kcmd init --semantic-model <projectId>.<locationId>.<entryGroupId> ``` Provisions the Knowledge Catalog entry group named by the scope (idempotent β€” an existing group is fine) and creates its local directory, `catalog/EntryGroups/<entryGroupId>/`. Author `<model>.yaml` there. ### push ```bash kcmd push ``` With no flags, deploys to every destination, the model's graph backends and Knowledge Catalog, in the order below. You do not choose the graph backend: each model deploys to whichever its deployment target names (BigQuery Graph or Spanner Graph). The binding profile selects which physical binding feeds the graph, and that binding's deployment target selects the backend. A push has two axes, both defaulted so a bare `kcmd push` deploys the graph for the default binding and records to Knowledge Catalog, unless it takes the two-phase path below. The **binding-profile axis** sets how many profiles the graph deploys for; the **Knowledge Catalog axis** is whether the catalog leg runs. | Flag | Effect | |------|--------| | `--profile <name>` | Deploy the graph for one binding profile (`<model>.profiles/<name>.yaml`); its deployment target selects the backend. Defaults to `default_profile` from `catalog.yaml`, else the model's inline bindings, except on the two-phase path below. Mutually exclusive with `--all-profiles` and `--no-profile`. See [Binding profiles](profiles.md). | | `--all-profiles` | Deploy the graph for every defined binding profile (plus the inline bindings when the document itself declares a target), instead of a single one. The Knowledge Catalog leg still records one canonical view β€” the default binding. Mutually exclusive with `--profile` and `--no-profile`. | | `--no-profile` | Deploy the graph for no binding profile: publish only the logical model to Knowledge Catalog, leaving any deployed graph untouched. | | `--no-kc` | Skip the Knowledge Catalog metadata push and deploy only the graph the selected profile targets. Knowledge Catalog is pushed by default. A push that deploys no graph (a logical model, or `--no-profile`) can only reach Knowledge Catalog, so `--no-kc` on it is an error, as is `--no-profile --no-kc` (nothing left to deploy). | | `--deployment <names>` | Deploy only these entries of the model's `deployments:` list, comma-separated or repeated, in the order given. Push still publishes the model to Knowledge Catalog first. | | `--no-deployment` | Publish the model to Knowledge Catalog and deploy nothing, after every check of the two-phase path below. It publishes the model file as written, whereas `--no-profile` publishes the binding `default_profile` names. Push refuses it with `--deployment`, in either order. | | `--validate-only` | Run every validation check and report pass/fail, but write nothing. | | `--print` | Print each destination's generated artifact (BigQuery or Spanner Graph SQL DDL, Knowledge Catalog entry plan). Combine with `--validate-only` to preview without deploying. | | `--force-remove` | Delete models in the entry group that this push no longer includes (see [Updating and removing models](README.md#updating-and-removing-models)). | | `--emit-expressions` | Also write the SQL-expression fields (per-field `schema.semantics` and `semantic-metric.expression`) to Knowledge Catalog. Off by default: the published system-type templates do not carry them yet. Knowledge Catalog push only. | A model that lists `deployments:`, and a push with `--deployment` or `--no-deployment`, run in two phases. So does a push of a model with a sibling profile file, without `--profile`, `--all-profiles` or `--no-profile`. The first runs every check, for Knowledge Catalog and for each deployment, and writes nothing. The second publishes the model to Knowledge Catalog, then deploys each deployment in order with the profile it names. An AlloyDB deployment deploys no graph. The first phase dry-runs each BigQuery graph's DDL. Spanner cannot check a graph's DDL without running it, so a Spanner deployment whose DDL Spanner rejects, for example one that reads a missing column, fails in the second phase, after Knowledge Catalog is written. On this path `--no-kc`, `--profile`, `--all-profiles` and `--no-profile` are errors, and catalog.yaml's `default_profile` chooses no binding: Knowledge Catalog receives the model file as written. The runtime still reads `default_profile`, so it must name a profile the model declares, and that profile must apply as the runtime applies it. Any other push prepares the catalog's model, checks that it publishes one model and checks the file copies before the first graph deploys. Its Knowledge Catalog leg runs its own checks only after the graphs deploy. ### pull ```bash kcmd pull ``` Writes back the model file and profile files a push stored in Knowledge Catalog. It reconstructs the model document from the catalog entries instead for a model an older `kcmd` pushed, and when no stored file is named after the model. When it finds that the last push stopped partway, it writes nothing. Reads only from Knowledge Catalog (never BigQuery); its coordinates come from the same scope you authored under. See [Pull](README.md#pull) for behavior. | Flag | Effect | |------|--------| | `--dry-run` | Read the catalog and report what would be written, but write or delete no files. | | `--force-remove` | Replace a differently-named local model with the catalog's (see [Pull](README.md#pull)); without it, a pull that would leave the entry group holding two models fails. Also deletes a local profile file the catalog has no copy of, when the catalog holds copies of the model's files. With `KC_V2_ASPECTS=1`, it also deletes a local profile file or overview sidecar the pull did not write. | ### import ```bash kcmd import <file> --flavor <ossie|google> ``` Imports a semantic model file into the flavor `--flavor` names. A file another product wrote is repaired, and each repair is reported as a warning. A file written for kcmd converts between flavors exactly or not at all. The `<model>.profile.<name>.yaml` files beside `<file>` go in with it. Nothing is written when the import reports an error. Without `--out`, the files go into the scope's entry group folder, and an import that would leave one of the model's profile files there unwritten is refused. | Flag | Effect | |------|--------| | `--input <who>` | Who wrote the file: `third-party`, a file another product wrote (default), or `first-party`, a file written for kcmd. | | `--flavor <flavor>` | Flavor to write: `ossie` or `google`. Required. | | `--source-prefix <prefix>` | GCP resource name of the database whose tables complete each bare table name, such as `bigquery.googleapis.com/projects/p/datasets/d`. | | `--transpile` | Add a BigQuery translation beside SQL in another dialect, in the model file only. A profile's SQL stays as written. Off by default. | | `--out <path>` | A `.yaml` file to write the one output file to, or a directory to write each output file into. | ### profiles ```bash kcmd profiles ``` Reports each binding profile the model declares: its deployment target, the source each entity binds to, and what the profile cannot answer or cannot run. Under each profile it then prints every error push would give for it, after a `push rejects:` heading. The inline `default` binding is checked too. Beside declared profiles it is listed only when push would reject it. Some errors fail every push of the scope, such as a second model in the entry group. When push would stop at such an error, `kcmd profiles` prints it once above the profiles and still checks each profile, unless a plain push runs in two phases. A Cloud SQL or third-party table whose dialect cannot be read from its entry stops the command before the report. When `default_profile` names a profile that a model does not declare, a push that names no profile fails. That error prints above the model's profiles. When push warns that the model file's own bindings are incomplete, the report prints that warning under `profile 'default'`, unless `--profile` names another profile. The command is read-only. It applies and prunes each profile and generates each graph as `push --validate-only` does, but deploys nothing, so you can compare coverage before choosing one. Like push, it reads the Knowledge Catalog entry of each Cloud SQL table written as its resource name and of each third-party table, for the table's dialect. The other checks that need a service run only at push time: the BigQuery source table probe, the dry runs of action statements and of each BigQuery graph's DDL, the check that a deployment's BigQuery dataset or Spanner database exists and, on a push without `--validate-only`, the Knowledge Catalog checks on the entry group. Push's lookup of the file-copy type also runs only at push time, because it calls a service. It exits non-zero when push would reject a binding. The report checks each profile as `kcmd push --profile <name>` does, unless a plain `kcmd push` of the scope runs in two phases. In that case it gives the errors of that push and stops where that push stops. Under each profile it prints the errors of that profile's own checks and of each deployment that uses it. The errors of a deployment that names no profile print under `default`. With `--profile <name>`, the other profiles' errors print above the profiles too, since any of them stops the push. Push applies only the profiles the deployments use and the one `default_profile` names, so it can accept a profile that does not apply. When push gives no error and a profile does not apply, the report prints why on a `not shown:` line under that profile. The report's `target:` line lists the targets of the deployments the profile uses, the same ones `--print-store` reads. For an AlloyDB target named by its cluster, `--print-store` calls the AlloyDB API to find the cluster's primary instance. | Flag | Effect | |------|--------| | `--profile [name]` | Report only this profile. Naming one the model does not declare is an error, not an empty report. Defaults to every profile. | | `--print-store` | Print only the store the profile deploys to, on one line and nothing else, for a script to read rather than parse out of the report: `project/instance/database` for a Spanner store, and the backend named ahead of the path for any other (`alloydb:project/region/cluster/instance/database`, `bigquery:project/dataset`). The store is the one the deployments for the profile name: those that name it in `profile`, or that name no profile or `default` for `default`. A legacy profile in `<model>.profiles/` that no deployment names takes the deployments for `default`, unless it sets its own `deployment_target`. Of several, it is the single Spanner or AlloyDB store. For an AlloyDB target named by its cluster, the instance printed is the cluster's primary instance. For a legacy instance-based AlloyDB target, it is the instance the target names. Errors when the scope holds more than one model, since those may name different databases, before any AlloyDB call. Also errors when a deployment names a profile the model does not declare, and when no single store results. Like push, it errors when the model's `extends` do not resolve: one names an entity the model does not declare, they form a cycle, or an entity inherits one field from two unrelated ancestors. A target in no accepted form in any deployment fails it under every profile, and so does a target a `GOOGLE` block lists twice. It reads every model's profiles first: its profile files, and the profiles a vanilla model carries in its `GOOGLE` block. So does `kcmd push --profile <name>`. A plain push that runs in two phases loads the model files before it checks the profile files. It errors on a profile file that breaks one of the layout's file rules, such as a `<model>.profile.<name>.yaml` file whose `name:` does not match its file name, on a `GOOGLE` block profile that does not load, and on a profile named `default` in any letter case. Like push, it also errors on a profile file that does not load, such as a file in `<model>.profiles/` whose `name:` does not match its file name. It does not run push's checks that every named profile applies to the model, that each is complete, and that each has valid keys and join columns. So a profile that leaves an entity without a source does not stop it. It does not run push's deployment checks, its dialect check of the model file's fields, or its check that each concrete entity restates the inherited fields that read a column either. It does not run push's checks on the copies of the model and profile files that push stores in Knowledge Catalog, such as whether each file is valid UTF-8. For an AlloyDB cluster it also errors when the cluster has no primary instance, when the primary is not READY, or when the instance listing fails. | This is a read of the binding, never a choice of one. Nothing on any `kcmd` command line names a store directly; `--profile` selects a binding and the binding's deployment target decides where writes land. A script that creates, seeds or drops that database asks for the name rather than repeating it: ```bash IFS=/ read -r PROJECT INSTANCE DATABASE <<<"$(kcmd profiles --print-store)" ``` ### skills-generate ```bash kcmd skills-generate ``` Writes each model in the scope out as an [Agent Skill](https://agentskills.io/) folder: a `SKILL.md` naming the model, routing to its actions and saying what a call comes back as, and one `references/<action>.md` per action carrying that action's arguments, rules and blast radius. It reads the model, touches no store, calls no judge, and runs nothing. For an AlloyDB target named by its cluster, it calls the AlloyDB API to find the cluster's primary instance. A skill names the store only for a model with actions, in its "Running an action" section. The command fails without writing anything when push rejects the deployments. That happens when the target of any deployment is in no accepted form, whichever profile the deployment is for. It also happens when several targets leave the store ambiguous, or when a deployment names a profile the model does not declare. It accepts a deployment that names `default` or a declared profile, as push does, but also one that names a legacy `semantic_model:` profile, which push refuses. It accepts a Spanner deployment that names no profile in a model with metrics, which push refuses. It fails on two BigQuery deployments that use one profile, since they leave no single store, while push deploys both. It accepts a model that binds a table outside the database a deployment deploys to, which push refuses, unless the model has actions and that database is the Spanner or AlloyDB store of the selected profile. It also accepts a profile file that no deployment uses, as push does, and writes a skill whose store line says there is none. A vanilla model lists its deployments or targets in a `GOOGLE` block. The command fails the same way when a target there is in no accepted form or listed twice, or when the block's data is not valid JSON. It reads every model's profiles first: its profile files, and the profiles a vanilla model carries in its `GOOGLE` block. So does `kcmd push --profile <name>`. A plain push that runs in two phases loads the model files before it checks the profile files. It fails on a profile file that breaks one of the layout's file rules, such as a `<model>.profile.<name>.yaml` file whose `name:` does not match its file name, on a `GOOGLE` block profile that does not load, and on a profile named `default` in any letter case. Like push, it also fails on a profile file that does not load, such as a file in `<model>.profiles/` whose `name:` does not match its file name. It does not run push's checks that every named profile applies to the model and that each is complete. So it still writes a skill when a profile leaves an entity without a source. It does not run push's checks on the copies of the model and profile files that push stores in Knowledge Catalog, such as whether each file is valid UTF-8. It does not check that the entry group holds one model, or that two models deploy to different graphs. Of push's checks on the model itself, it runs only the one on inheritance. It fails when an `extends` names an entity the model does not declare, when `extends` forms a cycle, or when an entity inherits one field from two unrelated ancestors. It runs none of the others, such as a concrete entity with no source or an action guarded by a constraint the model lacks. It fails when the profile binds a third-party table, naming the entity and the table, since kcmd has no engine that reads one. So does `kcmd profiles --print-store`. For a model with actions, it also fails without writing anything when the profile names a store that cannot be resolved. That happens when the model binds a table outside its Spanner or AlloyDB store, when the AlloyDB cluster has no primary instance, when the primary is not READY, or when the instance listing fails. A model with no actions still gets its skill in those cases, since the skill does not name the store. A BigQuery graph may read tables in other datasets, as push allows. For such a model, and for a profile whose deployments name no target, the store line says there is none and gives the reason. A skill's `name` and the directory it sits in have to match, or a client skips it. So the directory is named from what was generated rather than from anything the caller typed, and a name the format does not allow fails before anything is written. Everything the binding decides lives in `SKILL.md`: the store, the executor kinds, and which actions this deployment cannot run and why, all under one heading, plus the snippet for reading the store directly. A reference page is the same bytes under any profile. A guarded action is always described as runnable, because a rule stated in words is settled by the runtime before the transaction opens and a guarded action only ever runs against a runtime that has a judge. There is no flag here to say otherwise: whether the caller of `skills-generate` had a judge configured is a fact about that invocation, not about the deployment the document is read against. | Flag | Effect | |------|--------| | `--out <dir>` | Directory the skill directories are written under. Defaults to `skills`. | | `--name <name>` | Name the skill, and so its directory. Defaults to the model's own name. Rejected when the scope holds more than one model, because a name names one skill. | | `--profile [name]` | Read the model under this binding profile. Defaults to `default_profile`, else the model's inline bindings. It is what the one deployment-specific section describes. | | `--force` | Replace a skill already at that path. A reference page for an action the model no longer declares is deleted and reported; a file outside `references/` is left alone. | Before anything is written, a warning is printed when no action in a model is runnable under the selected profile, because a skill that can run nothing is rarely what was meant. Two models in one scope whose names normalize to one skill name are refused rather than one overwriting the other. See [Generating an Agent Skill](skills.md). ## What gets created in BigQuery `push` executes a single `CREATE OR REPLACE PROPERTY GRAPH` per deployment target, in the project and dataset the target names. Each part of your model becomes one part of that graph: | Model element | BigQuery construct | Notes | |---|---|---| | Model | `PROPERTY GRAPH` | named by the deployment-target URI | | Entity | `NODE TABLE` | backed by the entity's `source` table, keyed by its primary key, or by its first unique key when it has none. A measure reads only one of the rows whose key is NULL, and `COUNT(*)` skips them, so a unique key used this way should hold no NULLs | | Relationship | `EDGE TABLE` | connects the two entities' node tables | | Metric | `MEASURE` on a node table | must resolve to a single entity and reduce to one supported aggregate over one operand (otherwise the push is rejected; see [Validation](#validation)) | | Entity `extends` | extra `LABEL` clauses on the subclass node table | the subclass also matches its supertypes; the supertypes' fields flatten down (see [Class hierarchies](#class-hierarchies-extends--labels)) | | Action | *nothing* | an action reaches Knowledge Catalog only (see below); the BigQuery push deploys none and warns once | | Constraint | *nothing* | a constraint reaches Knowledge Catalog only (see below); the BigQuery push deploys none and warns once | `push` reads the target dataset's location (`bigquery.datasets.get`) so the statement runs in the right region; without that permission it falls back to BigQuery's own location inference and warns. Under `--validate-only` no graph DDL is executed and nothing is written (the live source-table checks still run β€” see [Validation](#validation)); add `--print` to see the generated DDL. For which of your descriptive metadata (`description`, `ai_context`, field labels, …) lands in the graph and which is dropped, see [What push and pull preserve](fidelity.md#to-bigquery). ### Class hierarchies (`extends` β†’ labels) This section is the rules lookup. To model a hierarchy step by step β€” declaring it, binding each subtype's table, and keeping supertype counts correct β€” see [Modeling class hierarchies](inheritance.md). An entity that declares `extends: [Parent]` is a **subclass**. BigQuery Graph has no inheritance keyword, so the push expresses the hierarchy with **labels**: a subclass node table declares its own default label **plus one `LABEL <Ancestor>` per supertype**, walking the full transitive chain. A node then matches its supertype in a query β€” `MATCH (:Party)` returns every `Customer` and every `Supplier` node. You author each entity's **own** fields plus the one `extends` keyword, and the push flattens the supertype's fields down onto each subclass. The usual supertype has no table of its own, so mark it `abstract: true`: it has no `source` and no key, produces no node table, and survives only as a label on its subtypes. ```yaml entities: - name: Party abstract: true # no table; becomes a label on every subtype fields: - { name: id, datatype: Integer } - { name: name, datatype: String } - name: Customer extends: [Party] # the one keyword you add primary_key: [c_custkey] # each subclass keeps its OWN key; keys do not inherit source: bigquery.googleapis.com/projects/proj/datasets/ds/tables/customer fields: - { name: id, expression: c_custkey } - { name: name, expression: c_name } # the inherited field, bound to this table's column - { name: tier, datatype: String, expression: c_tier } - name: Supplier extends: [Party] primary_key: [s_suppkey] source: bigquery.googleapis.com/projects/proj/datasets/ds/tables/supplier fields: - { name: id, expression: s_suppkey } - { name: name, expression: s_name } - { name: rating, datatype: Integer, expression: s_rating } ``` **The supertype's fields flatten down** onto each subclass. A supertype contributes its field names to every subclass, ordered own fields first then inherited. An abstract supertype binds no columns of its own, so each subtype supplies the column for every inherited name on its own table β€” `id` and `name` above are bound on both the customer and the supplier. Under a concrete supertype, a subtype restates each inherited field whose supertype expression reads a column. This holds in each named profile, and in the model file when it gives the subtype a source. A field whose supertype expression reads only other fields flattens straight down. A subtype redeclares an inherited field only to bind it. The redeclaration names the field and gives it a column of the subtype's table. The datatype, label, dimension, description, `ai_context` and `custom_extensions` come from the ancestor that declares the field, and push rejects a redeclaration that restates any of them, or gives nothing but the name. A subtype that does not bind an inherited field takes the binding of the nearest ancestor that does. Under a concrete supertype the graph keeps the supertype's binding either way, as the boundaries below describe. **A shared label is reconciled by property name rather than by backing column** (verified live). Every table that carries `LABEL Party` must expose the same property names β€” here `id` and `name` β€” and each backs them with its own column. So the push renders each inherited property from the subtype's own binding: `c_name AS name` on the customer table, `s_name AS name` on the supplier table. A bare-alias reference such as `PROPERTIES(name)` does not deploy; BigQuery rejects it with `Unrecognized name: name`. ```sql `proj.ds.customer` AS Customer KEY(c_custkey) DEFAULT LABEL PROPERTIES( c_custkey AS id, c_name AS name, c_tier AS tier ) -- id and name inherited from Party; tier is Customer's own LABEL Party PROPERTIES( c_custkey AS id, c_name AS name ) -- Party's signature, backed by this table's columns ``` ``` GRAPH proj.ds.parties MATCH (p:Party) RETURN p.name -- resolves on Customer and Supplier alike ``` The boundaries: - **Fields flow down; edges and keys do not.** A subclass gains its supertypes' fields but **not** their relationships or their key: an edge stays bound to the exact node table it was declared on, and each subclass keeps its own `KEY` (a node table is identified by its own grain, never its supertype's). An edge declared on a supertype would join only the supertype's own rows and miss every subtype, so push rejects a relationship that connects an abstract entity or one that another entity extends. Declare such an edge on each concrete leaf, for example `Customer β€”livesInβ†’ City`. - **The subclass's `source` must physically expose every inherited column.** The flattened `name` above is read from `proj.ds.customer`, so that table (or a view over it) must include the column that `Customer`'s `name` field binds. A subclass whose table lacks a column that one of its inherited properties needs fails the push when the graph deploys. - **A shared supertype label carries no OPTIONS and no measures.** A supertype's label is bound by every subclass table, and BigQuery forbids a label carried by more than one element table from carrying an `OPTIONS` clause or a `MEASURE`. So a supertype's own `description`/synonyms are dropped from its label (with a warning). For the same reason push rejects a metric that belongs to an abstract entity or to one that another entity extends. Attach metrics to a leaf class instead. Subclass and leaf labels are unaffected. - **Each inherited property has one definition under the shared label.** For an abstract supertype, the subtype supplies that definition β€” it binds the inherited field to its own column, as `name` is bound above, and that binding is used. A concrete supertype already defines the property on its own table, so a subtype that declares the same-named field with a different column or expression cannot override it: the supertype's definition wins and the subtype's is dropped (with a warning). Redeclaring it with the same column is a harmless no-op. An entity marked **`abstract: true`** is a conceptual class with no physical table: it has no `source`, no key and no field expressions, produces **no node table**, and survives only as a `LABEL` on its concrete descendants. Push rejects an abstract entity that declares a `source`, a `primary_key`, a `unique_keys` entry or a field expression. Its field names still flatten down, and each concrete subtype supplies the column for each of those names, so the shared label's signature is present on every subtype table. An abstract entity that no concrete entity extends has nothing to attach to and is dropped with a warning. `abstract` is an explicit marker: a non-abstract entity with no `source` is treated as a binding error and fails the push, never silently dropped as if it were table-less. With the first-generation types, the Knowledge Catalog leg skips an abstract entity with a warning and publishes its concrete subtypes. With `KC_V2_ASPECTS=1`, every entity gets an entry, and a `reference` link runs from each entity to each parent it extends. A supertype **may** instead be concrete β€” carry its own `source` and key. It then becomes both its own node table and a label on its subtypes. Every subtype table must still expose columns that render to the supertype's property signature, so each subtype table has to carry the supertype's columns under the same names. The supertype's own rows and its subtypes' rows are distinct nodes: a real thing present in both the supertype table and a subtype table is matched twice under the supertype label. Because other entities extend it, a concrete supertype cannot be a relationship endpoint or own a metric. Prefer an abstract supertype unless each real thing lives in exactly one table under the hierarchy. **Multiple supertypes and diamonds.** `extends` takes a list, so a subclass may extend several supertypes and carry every one's label. The push expands `extends` to the full transitive ancestor set, de-duplicated. A diamond β€” two supertypes that share a grandparent β€” lists that grandparent's label once, so `MATCH (:Grandparent)` matches the leaf a single time. A field may be declared or rebound on only one side of a diamond. When two supertypes that do not extend each other both declare or rebind the same field, push rejects the model, because nothing says which definition wins. A field that reaches the leaf by two paths from a single declaration on the shared grandparent is fine. `extends` may not form a cycle, and push rejects one. Depth and breadth do not change the rules: each concrete table binds every inherited property to its own column, and these shapes deploy on both BigQuery Graph and Spanner Graph (verified live for a diamond and for a three-level hierarchy with several concrete leaves). ## What gets created in Spanner When the deployment target is a Spanner graph, `push` executes the same `CREATE OR REPLACE PROPERTY GRAPH` β€” but generated for Spanner Graph, which differs from BigQuery Graph in five ways: | Model element | Spanner construct | Notes | |---|---|---| | Model | `PROPERTY GRAPH` | named by the target's `graphs/<g>` segment, **bare** (no backticked `project.dataset.` prefix) | | Entity | `NODE TABLE` | backed by the entity's `source` reduced to its final segment (`spanner.googleapis.com/…/tables/Orders` β†’ `Orders`), a table in the target database. An entity with no primary key is keyed by its first unique key, and Spanner rejects the graph unless the table's primary key or a unique index covers that key. A `UNIQUE NULL_FILTERED` index leaves the rows whose key is NULL out of the graph. | | Relationship | `EDGE TABLE` | connects the two entities' node tables | | Metric | β€” dropped | Spanner Graph has no `MEASURE`, so every model-level metric is skipped with a warning; the graph structure still deploys | | Entity `extends` | extra `LABEL` clauses on the subclass node table | same label-and-flatten handling as BigQuery (see [Class hierarchies](#class-hierarchies-extends--labels)) | - **Bare table and graph names.** A Spanner property graph lives inside one database and names tables in that same database, so the generator emits bare names β€” no `project.dataset.` qualifier on either the tables or the graph. - **No `MEASURE`.** Metrics are dropped (warned per metric). The BigQuery-only rule that a metric must resolve to a single entity therefore does not apply to a Spanner target (see [Validation](#validation)). Push refuses a Spanner deployment that names no profile in a model with metrics; name a profile with `metrics_exclude: "*"`. - **No per-element `OPTIONS`.** `description` / `synonyms` are not emitted into the Spanner DDL; they ride into Knowledge Catalog instead β€” mirroring how BigQuery's graph-level `OPTIONS` is dropped. See [What push and pull preserve](fidelity.md#to-spanner). - **Its own SQL.** A field reads its `SPANNER` expression, else its `ANSI_SQL` one, and is left out with a warning when it has neither. - **Async DDL.** The statement is applied through the Spanner Admin `updateDatabaseDdl` long-running operation, polled to completion (BigQuery runs its DDL through `jobs.query`). No region detection is needed β€” the DDL runs in the database the target names. Under `--validate-only` nothing is applied; add `--print` to see the generated Spanner DDL. Unlike the BigQuery leg, a Spanner-targeting model's source tables are **not** probed before deploy. Its actions are: every `sql` executor statement is planned against the target database first, on both legs (see [Validation](#validation)). ## What gets created in Knowledge Catalog Each element of your model maps to one catalog resource. Every resource type below except `semantic-action`, `semantic-constraint` and `semantic-model-file` is a built-in system type under `dataplex-types/global` β€” push references them, it never creates them. Those three are custom, and `kcmd init --semantic-model` creates them in your own project at `global`; push still writes only entries. > Set `KC_TYPE_PROJECT` to read these system types from another project, and > `DATAPLEX_ENDPOINT` to target a non-prod Dataplex host; both default to > production (`dataplex-types` / `https://dataplex.googleapis.com`). | Model element | Catalog resource | Kind | Id | |---|---|---|---| | Model | `semantic-model` | entry β€” anchor / parent of the rest | `<model>` | | Entity | `semantic-entity` (+ built-in `schema` aspect) | entry | `<model>.entities.<entity>` | | Metric | `semantic-metric` | entry | `<model>.metrics.<metric>` | | Relationship | `schema-join` | entry link between the two entity entries | derived from the model and relationship names | | Action | `semantic-action` (custom type) | entry | `<model>.actions.<action>` | | Constraint | `semantic-constraint` (custom type) | entry | `<model>.constraints.<constraint>` | | Model file and each profile file | `semantic-model-file` (custom type) | entry holding the file's text | `<model>.files.<file>` | The model entry also carries the `kcmd-push-id` label, which names the push that stored the file copies. On each entry, push owns the `schema`, `semantic-*` and `guidelines` aspects it writes, the display name, the description and that label. When it updates an entry, it keeps every other label and the source `resource`, `system`, `platform`, `createTime` and `updateTime` another tool set. An entity entry carries its columns in the `schema` aspect (name, data type, description, and any `label` per field), plus the entity's keys and unique keys (`primaryKey` / `uniqueConstraints`); a `schema-join` link carries the relationship detail β€” the paired columns and foreign-key direction β€” in its aspect. Any element with `ai_context.instructions` (the model, an entity, or a metric) also gets a built-in `guidelines` aspect holding that text. An **action** entry carries its executor, its typed parameters, the names of the constraints that gate it (`guards`), and the concepts it changes (`affects`) in a `semantic-action` aspect, along with the action's `ai_context.instructions`. A guard is stored as the constraint name, matching a sibling `semantic-constraint` entry on the same model, so a reader holding one action entry can find the rules it is checked against. An affected concept is stored as its name β€” an entity or a relationship, named the same way, since the model is what says which. That aspect type is provisioned in your project rather than referenced from `dataplex-types`, because Dataplex has no built-in action type yet; when one ships, the entries move to it and their shape does not change. (This is the prototype scope β€” an action's `precondition` is not modeled yet, and nothing consumes its `affects`.) A **constraint** entry carries its rule in a `semantic-constraint` aspect, together with the whole of any `ai_context` declared on it. The rule sits in the one body a constraint has, `judgment`: the condition written as a sentence, settled by a language model reading the attempted call. The aspect type also declares `expression` and `evaluation`, which nothing writes and nothing reads. They are kept so that an aspect type already provisioned in your project still matches what `kcmd init` would create, and an entry left holding an old `expression` reads back as a named skip rather than as a rule. The constraint's `description` is the entry's own summary, because that sentence is what a caller refused by the rule reads. Its type is provisioned alongside the action pair and for the same reason. All three parts of `ai_context` survive, unlike an element routed to the built-in `guidelines` aspect, which has a home for `instructions` alone: `kcmd` defines the constraint aspect itself, so it has no reason to keep that limit. The aspect also carries the constraint's two routing words: `onViolation` (what the engine does to the write β€” `reject`, `escalate` or `warn`) and `severity` (how grave the breach is β€” `critical`, `high`, `medium` or `low`). They ride the aspect rather than the entry source because they are machine-readable and not prose. `onViolation` is required of every constraint, and `severity` is not: one that declares no severity is published without the field and reads back without it, so the ranking stays the model's to define. Each is read on its own, so an unrecognized word in one is dropped on pull with a warning without costing the reader the other. An unreadable `severity` leaves the rule unranked, which nothing reads yet. An unreadable `onViolation` is dropped too, but there is no fallback to drop to now that the word is required: the pull warns that the constraint will not push, and loading the pulled model names the constraint and says the word is missing. That is the outcome to want when the catalog no longer says how a breach routes. The catalog's metadata does not hold your whole model, so push also stores a copy of each model file and profile file, which pull returns. For exactly what the metadata stores, what is gated behind `--emit-expressions`, and what is never stored, see [What push and pull preserve](fidelity.md#to-knowledge-catalog). ### With `KC_V2_ASPECTS=1` With `KC_V2_ASPECTS=1`, push and pull use a newer set of Knowledge Catalog types. Each relationship other than a many-to-many association has an entry of its own. Entry ids are `<model>`, `<model>/entities/<entity>`, `<model>/metrics/<metric>` and `<model>/relationships/<relationship>`. A `semantic-link` joins each ordered pair of related entities. Built-in `reference` links run from each relationship's entry to the entries of its source and target entities, and from each entity's entry to the entry of each parent in its `extends`. An unbound relationship gets its `reference` links too. Their ids start with `semantic-relationship-` or `semantic-extends-`, followed by the first 32 hex characters of a SHA-256 hash. The hash covers `<model>/<relationship>/source` or `<model>/<relationship>/target` for a relationship, and `<model>/<child>/<parent>` for a parent. Push owns only the `reference` links with those prefixes. It deletes the ones the model no longer has and leaves every other `reference` link alone. Pull reads no entry link. An entity's `schema` aspect lists only the fields it declares. A field it inherits and only rebinds is a `fieldExpressions[]` record in its `sql-expressions` aspect, with nothing in `schema`. A field's `guidelines`, `overview` and `semantic-extensions` aspects sit at `Schema.<field>` on the entity that declares the field. `semantic-metric.entity` holds the metric's entity, declared or derived. Pull writes it back as `entity:`. `excludedInProfiles` follows the order of the model entry's `profiles` list. Push writes each entry's whole `overview` aspect from the workspace and never reads it back. `examples` come from the model file's `ai_context.examples`. `content`, `contentType` and `links` come from the entry's sidecar. The sidecar is the file `<entry-id>.overview.md` in the directory that holds `<model>.yaml`, so the model's sidecar is `<model>.overview.md` and an entity's is `<model>/entities/<entity>.overview.md`. The sidecar's frontmatter holds `contentType` and `links`, and its body is `content`. `contentType` is `MARKDOWN` or `HTML`. Each link has a `url` and an optional `title`. A malformed sidecar fails the push before anything is written. Push warns about a sidecar that names no entry of the model. A field's overview holds its examples alone. An overview edited in the console survives a push only once `kcmd pull` writes the edit into the sidecar. Pull writes a sidecar for each entry whose overview has content, a content type or links. It warns about a local sidecar the catalog did not return, and `--force-remove` deletes it. Pull writes the model file, and one file per named profile of a Google-flavor model. A profile file keeps the actions it binds locally, since the catalog stores no profile's actions. A local profile file the pull did not write draws a warning, and `--force-remove` deletes it. ## Validation `push` and `--validate-only` run the same checks, apart from the Knowledge Catalog checks on the entry group, which only a push without `--validate-only` runs. [push](#push) says when each check runs. Each check enforces a rule the [model specification](model_spec.md) *defines*; this section is the operational side of it β€” what the tool does when the rule is broken β€” and links to the definition it enforces: Before these checks even run, the document must **load**, and the loader is strict: `version` is required and must be `0.2.0.dev0` (vanilla Ossie) or `0.2.0.dev0/google` (the extended profile); every object is closed, so an unknown key β€” including a native key under vanilla, or a `custom_extensions` block under `/google` β€” is rejected; names must be unique within their scope; every `extends` must name an entity defined in the model; a field's or metric's `expression`, in the string form and in each dialect entry, may not be empty, only whitespace or only a comment; and a field's `expression` may not be only its entity's qualifier, such as `orders.`. A load failure is reported the same way as a validation failure β€” nothing is touched, non-zero exit β€” and is defined in [model spec Β§1](model_spec.md#1-status-and-baseline), [Β§2](model_spec.md#2-document-shape), [Β§6](model_spec.md#6-the-extension-mechanism), and [Β§4.1](model_spec.md#41-narrowings-stricter-than-ossie). * **A graph push declares exactly one deployment target per model, and it must be a valid BigQuery Graph or Spanner Graph resource name.** A model that lists `deployments:` pushes in two phases and holds each BigQuery and Spanner deployment to this rule on its own. Otherwise a model with more than one is rejected, and so is a single target that matches neither `bigquery.googleapis.com/projects/<p>/datasets/<d>/propertyGraphs/<g>` nor `spanner.googleapis.com/projects/<p>/instances/<i>/databases/<db>/graphs/<g>` (for example a `propertyGraph` typo, a Spanner graph under `propertyGraphs/`, a leading `//`, or a `…/entryGroups/@bigquery/entries/…` entry form). The error names the offending target and what to write instead. A **logical model that declares no target** is allowed: it deploys no graph and records to Knowledge Catalog only (so `--no-kc` on it is an error β€” it would have nowhere to go). This gate runs before any destination leg, so a malformed target writes **nothing** β€” not to the graph and **not to Knowledge Catalog**; the push aborts with a non-zero exit and no entries are created. Defined in [model spec Β§4.1](model_spec.md#41-narrowings-stricter-than-ossie) (one target) and [Β§7.2](model_spec.md#72-deployment-target) (the grammar). *(static)* * **Every metric on a BigQuery Graph model resolves to exactly one entity** β€” otherwise it would be dropped from the BigQuery Graph. Set the metric's attach entity, or scope its expression to a single entity. This rule is BigQuery-only: Spanner Graph has no `MEASURE`, so a Spanner target drops its metrics by design and imposes no such requirement. Defined in [model spec Β§4.1](model_spec.md#41-narrowings-stricter-than-ossie). *(static)* * **Each deployment meets the rules of its engine.** Every push checks each deployment the model declares, whether or not the push deploys it. A deployment that names no profile uses the model file's own bindings. They must give every concrete entity a source and every relationship both join column lists. They must also bind every field that each deployed metric reads, directly or through another field. An AlloyDB deployment reads only the `ALLOYDB` or `ANSI_SQL` text of each metric and field. A Spanner deployment deploys no metric. A BigQuery deployment turns each metric into a measure, so each metric must be `SUM`, `AVG`, `COUNT`, `MIN` or `MAX` over one operand, and the entity it belongs to needs a primary key or a unique key. The measure rule skips a metric that has only a vendor text. A deployment that names a profile is held to the measure rule under that profile, with the keys it states, except for the metrics it leaves unavailable. On Spanner, that profile must leave every metric unavailable. *(static)* * **A third-party table is written as its entry name, and takes the dialect its entry records.** A table outside Google Cloud is written as the entry name of its Knowledge Catalog entry, `projects/<p>/locations/<l>/entryGroups/<g>/entries/<id>`, in an entry group whose id does not begin with `@`. An entry name in a system entry group is rejected with the resource name to write, and so are a Knowledge Catalog name such as `snowflake:acme.retail.public.orders` and a value with a leading `//`. Push and `kcmd profiles` look each entry up once before they load the model. The prefix of the entry's `fullyQualifiedName` gives the dialect: `snowflake:` gives `SNOWFLAKE`, `databricks:` gives `DATABRICKS`, `mysql:` gives `MYSQL`, `postgresql:` gives `POSTGRES` and `sqlserver:` gives `SQLSERVER`. An entry with no `fullyQualifiedName` gives the dialect of its `entrySource.system`, in any letter case: `SNOWFLAKE`, `DATABRICKS`, `MYSQL`, `POSTGRESQL` or `POSTGRES`, or `SQLSERVER`. A missing entry, or one that gives no dialect, stops the command before anything is written. So does an entry whose `fullyQualifiedName` names something other than a table or view, such as the schema `snowflake:acme.retail.public`. A field the model file binds to the table then needs SQL in that dialect or in `ANSI_SQL`. So does every field and metric of a profile whose sources fix that dialect. *(live β€” needs read access to the entry)* * **Each field the model file binds has a text its entity's database runs.** The field needs an entry in the dialect of its entity's source, an `ANSI_SQL` entry, or the string form. The model file's own bindings may span systems, so each entity is judged by its own source. `kcmd import --transpile` adds a BigQuery entry for a text it can translate. Every push runs this check, whether or not a deployment uses those bindings. *(static)* * Every class hierarchy follows the inheritance rules. `extends` forms no cycle, and no field is declared by two supertypes that do not extend each other. A subtype redeclares an inherited field only to bind it. A concrete entity with a source restates each inherited field whose ancestor's expression reads a column. An abstract entity declares no source, key or field expression. Relationships and metrics attach only to concrete leaf entities. The rules and their reasons are in [Class hierarchies](#class-hierarchies-extends--labels). These checks run on every push, including a `--no-profile` push to Knowledge Catalog alone. *(static)* * **Relationships join two different entities in one direction.** Push rejects a relationship from an entity to itself, and relationships in both directions between two entities. *(static)* * **Every action is well-formed.** Each action parameter must settle on one scalar datatype: projected from a field it names with `concept` and `field`, or stated as its own `type`. Naming an entity as a `type`, or restating a projected parameter's type, is rejected. Two parameters a caller could confuse β€” projecting the same field, or sharing a declared type β€” must each carry their own `description`. Each executor must carry its coordinates (an `mcp` server + tool, a `rest` endpoint + method, a `grpc` service + method, or at least one non-blank `sql` statement) with no blank field. A coordinate omitted altogether is rejected earlier, when the model is parsed. An action with *no* executor is not an error at all: the executor is a physical binding, so an action no binding performs here is still a declaration worth publishing. Each name in the action's `guards` must resolve to a constraint that the same model declares. A guard resolving to nothing leaves the author believing the write is checked when nothing checks it. Every `concept` in the action's `affects` must likewise resolve to an entity or a relationship the same model declares, and its `fields` must be fields of that concept β€” for a relationship, the properties of the junction table backing a many-to-many edge, so a plain foreign-key edge has none. Naming fields beside a `delete` is rejected, because a `delete` takes the whole instance. An affected concept that resolves to nothing is reported and then left alone, since every later check about it is meaningless; undeclared fields do not chain that way, so a concept that does resolve reports every field it does not have. Everything that reads the ontology β€” the concept check and the field check both β€” stands down on a profile push, because pruning drops whole entities and whole relationships as well as unbound fields, and an action reaches no graph in any case. Pruning now drops an action that names a dropped *concept*, so for the concept check the offending action is gone before the check would run; it does not drop one whose `affects` names a pruned *field*, so a blast radius can still name a field the published entity no longer carries, and nothing reports it. The fields-beside-a-`delete` check reads only the entry, so it still applies. A `sql` executor is checked further, because it carries the write itself rather than a pointer to whoever performs it: each statement must begin with `INSERT`, `UPDATE` or `DELETE`, must contain no `;` other than a trailing one, and may reference only `@parameter` names the action declares. Together those are what let every value be bound instead of interpolated, so an argument cannot reach the store as SQL. Where the key of an inserted row comes from is the statement's own business: a SQL function such as `GENERATE_UUID()`, a declared parameter, or a key column left out where it has a default. `affects` has no bearing on it. A declared parameter hands the key to the caller, and since the verb check reads only a statement's first word, an upsert passes and overwrites the row that key names. Four further rules are enforced at parse time: exactly one executor kind (`executor requires exactly one kind, but 2 given (mcp, rest)`), the closed `create` / `modify` / `delete` vocabulary for `operation`, the rejection of a repeated guard name, and the rejection of a repeated concept-and-operation pair in `affects`. Every check named so far is static β€” it reads the document alone β€” so it runs on every push, regardless of destination. What the *names* in a statement mean is settled live instead, by the store the profile binds: see the DML pre-flight under [Validation](#validation). Note that actions themselves deploy **only** through the Knowledge Catalog leg β€” a graph-only `--no-kc` push validates them but has nowhere to put them, and warns that they will not be deployed. *(static, plus a live pre-flight for a `sql` executor)* * **Every constraint states its rule as a judgment.** A constraint has one body, `judgment`, and it must be non-empty; one declaring no judgment states no rule at all, and the error names it. `expression` is a reserved key: a model stating one is answered with a sentence telling the author to restate the rule in words, rather than with an unrecognized-key error that explains nothing. What a judgment costs is in [Actions](actions.md#what-a-judgment-costs). *(static)* * **Every constraint says what a violation does.** `on_violation` is required rather than defaulting. Any of the three words is allowed, `reject` included; leaving the key out is the error, because an unmarked constraint rejects and that is too strong a consequence to inherit by silence. Every `Entity.field` token in the judgment is resolved against the model, so a field name that has been renamed out from under the sentence is caught. Quoted text is scanned too: in prose, quotes usually set off a field name for emphasis, and skipping those would hide the renames this check exists to catch. A qualifier that is not a known entity β€” a relationship-qualified name like `OrderedAs.quantity`, or a metric reference β€” is left alone rather than guessed at, so a valid sentence is never falsely rejected. So is a token whose **tail** names something the model declares that is not a field of the head: `Customer.Order` and `LineItem.BelongsTo` are traversals, and `Order.total_revenue` names a metric, none of which the check can settle. So is an entity that declares no fields, since fields are optional and a logical model may declare none; an empty list is no evidence that a field is missing. So is a token carrying a third dotted segment, which is a path rather than a field. What is left is the case the check exists for: a tail the model declares under no kind at all, next to an entity it does declare. *(static)* * Like an action, a constraint reaches Knowledge Catalog only, and a `--no-kc` push warns that it will not be deployed. Two rules are enforced at parse time: `on_violation` and `severity` are each a closed vocabulary β€” `reject` / `escalate` / `warn` and `critical` / `high` / `medium` / `low` β€” so an unrecognized word is a hard load error rather than a value that publishes and means nothing. *(static)* * **Each deployment's tables are in the database it deploys to.** A Spanner or AlloyDB deployment reads only tables in its target database. A BigQuery deployment reads only BigQuery or BigLake tables, in any dataset of any project. A Cloud SQL table fails every deployment, and so does a third-party table, since kcmd has no engine that reads one. A push with no deployment publishes such a model to Knowledge Catalog. A model with no deployments list that binds a third-party table is refused when its GOOGLE block names a BigQuery or Spanner graph or an AlloyDB database, even under `--no-profile` or `--no-deployment`. A push in two phases checks every deployment this way, whether or not `--deployment` names it, and under `--no-deployment` too. It reads each entity's `source` from the profile the deployment names, or from the model file when it names none. *(static)* * **Every entity's source table is reachable.** For a **BigQuery-targeting** model, each `source` is probed with a dry-run query, so BigQuery resolves it exactly as the deploy will β€” a BigQuery table or a BigLake table (e.g. an Iceberg table in a federated REST catalog). A table that does not exist or that you cannot access fails the push, naming the table and the entity. On a push that uses deployments, push probes the tables of the deployments it would deploy, each in the location of the graph's dataset when push can read it. A table in another location fails the push, and the error names the table's location when BigQuery reports it. A **Spanner-targeting** model's sources live in Spanner (a different system) and are **not** probed here. The `source` construct and the resource names it takes are defined in [model spec Β§7.1](model_spec.md#71-table-sources). *(live β€” needs BigQuery access)* * **Every `sql` executor statement is accepted by the store it will run against.** Each statement is sent to the bound store with the request that means *plan it, do not run it* β€” `dryRun` on BigQuery, `queryMode: PLAN` in an uncommitted read-write transaction on Spanner β€” so the store parses, resolves and type-checks it without writing anything. That catches a table or a column the database does not have (answered with the store's own *Did you mean `account_id`?*), a placeholder or a function from the wrong dialect, and a parameter whose declared type does not match the column it is compared with: the parameter types the action declares are sent with the statement, unbound, which is what turns them into a claim the store checks. The database a statement is planned against is the one the profile deploys the graph to, so which store answers is the profile's choice, not the statement's. Failing to reach the store at all fails the push rather than skipping the check β€” a push that could not verify its statements must not proceed as though it had. *(live β€” needs query access to the bound store)* The live table check runs whenever the BigQuery leg runs (some model targets BigQuery Graph), including under `--no-kc`, because the same tables back both a BigQuery graph and its Knowledge Catalog entries. The DML pre-flight rides the same two legs: it runs before the BigQuery deploy and before the Spanner deploy, and both run under `--validate-only`, so a statement fails while nothing has been published. Two kinds of model run neither leg and are therefore checked statically and no further: a catalog-only model, which declares no deployment target at all, and an **AlloyDB**-targeting one β€” AlloyDB is a valid deployment target and a valid store for an action to write to, but push has no client for it, so its statements are never sent anywhere. Neither case is reported as verified. ## Permissions `push` needs access to whichever destinations you deploy to. **BigQuery** β€” for a BigQuery-targeting model (the BigQuery leg runs whenever such a model is present), and for the validation pre-flight: * `bigquery.jobs.create` in the deployment-target project β€” to run the deploy's `CREATE OR REPLACE PROPERTY GRAPH` and both pre-flight dry runs * read access on each entity's source table, so the dry-run can resolve it * read access on each table an action's `sql` executor names, for the same reason β€” a dry run of a DML statement resolves the table it writes to, though it writes nothing * `bigquery.datasets.get` on the target dataset (region detection; optional β€” push degrades gracefully without it) **Spanner** β€” for a Spanner-targeting model (the Spanner leg runs whenever such a model is present), on the database the target names: * `spanner.databases.updateDdl` β€” to apply the `CREATE OR REPLACE PROPERTY GRAPH` through `updateDatabaseDdl` * `spanner.databaseOperations.get` β€” to poll the long-running operation to completion * `spanner.sessions.create` and `spanner.sessions.delete`, `spanner.databases.select`, and `spanner.databases.beginOrRollbackReadWriteTransaction` β€” the DML pre-flight plans each statement on a session, and a `PLAN` of a DML statement has to open the read-write transaction it would write in even though it never commits one. All four come with `roles/spanner.databaseUser`. **Knowledge Catalog / Dataplex** β€” for the Knowledge Catalog push (on by default; skip it with `--no-kc`): Writing an entry with aspects needs permission on **both** the entry operation and each aspect type attached, so a push needs, on the destination entry group: * `dataplex.entries.create`, `dataplex.entries.update`, `dataplex.entries.delete`, and `dataplex.entries.list` β€” push upserts the model / entity / metric entries and deletes orphaned ones * `dataplex.entryLinks.create` and `dataplex.entryLinks.delete` β€” the `schema-join` links, when the model has relationships * `dataplex.entryGroups.useSchemaAspect` β€” every entity carries the built-in `schema` aspect (its fields, keys, unique keys, and labels; with `KC_V2_ASPECTS=1`, the fields it declares and no keys) * `dataplex.entryGroups.useGuidelinesAspect` β€” when the model, an entity, or a metric carries `ai_context.instructions` * `dataplex.entryGroups.useSchemaJoinAspect` and `dataplex.entryGroups.useSchemaJoinEntryLink` β€” when the model has relationships * `dataplex.entryGroups.useReferenceEntryLink` β€” with `KC_V2_ASPECTS=1`, when the model has relationships or an entity extends another * the `use<AspectType>Aspect` permission for the `semantic-model`, `semantic-entity`, and `semantic-metric` aspect types the push attaches β€” i.e. `dataplex.entryGroups.useSemanticModelAspect`, `useSemanticEntityAspect`, and `useSemanticMetricAspect` * `dataplex.aspectTypes.use` on the `semantic-action` aspect type, when the model declares actions, and on `semantic-constraint` when it declares constraints β€” those types are custom rather than built-in, so they are authorized on the type resource instead of through an entry-group use-permission > The `schema` / `guidelines` / `schema-join` use-permissions follow Dataplex's > documented `dataplex.entryGroups.use<AspectType>Aspect` > [pattern](https://cloud.google.com/dataplex/docs/iam-permissions); the > `semantic-*` names follow the same pattern but are not yet in that public > reference (the `semantic-*` system aspect types are newer), so confirm them > against your project's IAM once granted. `kcmd init --semantic-model` creates what a later push only references, so it needs more than push does, in the destination project: * `dataplex.entryGroups.create` β€” the destination entry group * `dataplex.aspectTypes.create` / `dataplex.aspectTypes.update` and `dataplex.entryTypes.create` β€” the custom `semantic-action`, `semantic-constraint` and `semantic-model-file` pairs. Init patches an aspect type that is already there, so a project set up by an older `kcmd` picks up template additions; an entry type that is already there is left alone. The entry group and the `semantic-model-file` pair are required. Every push stores the model file and its profile files with that type, so init fails when the type is missing and it cannot create it, and a push into a project that lacks it stops before writing anything and asks you to run `kcmd init --semantic-model`. A project set up by an older `kcmd` needs that once. When the entry group or a type already exists, init goes on even if you may not create or update it, and reports a refusal to update a type as a warning, so a teammate without these permissions can still run init in a project someone else set up. Actions and constraints are optional constructs, so init reports a refusal to create their types as a warning and carries on; every model that declares neither still pushes and pulls. Any other failure to create a type stops init, rather than leaving a later push to hit an opaque parsing error. `kcmd pull` needs read access to the same entry group instead β€” to list its entries and fetch each `semantic-*` entry with its aspects. --- --- METRICS --- - Files Extracted: 43 - Estimated Token Budget: ~135558 tokens - Canonical Reference: https://codewiki.google/github.com/GoogleCloudPlatform/knowledge-catalog