knowledge-catalog Documentation

Technical Documentation: GoogleCloudPlatform/knowledge-catalog

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Knowledge Catalog

Knowledge Catalog (formerly Dataplex), is an AI-powered data catalog and metadata management platform. It provides a dynamic knowledge graph of all your data, structured and unstructured, to provide semantics and business context to AI agents

This repository features tools, agents, and samples that demonstrate Knowledge Catalog features, and building context management, enrichment and retrieval solutions.

Getting Started Copy Chapter

Contributing Copy Chapter

See the contributing instructions to get started contributed.

License Copy Chapter

All solutions within this repository are provided under the Apache 2.0 license. Please see LICENSE for more detailed terms and conditions.

Disclaimer Copy Chapter

This repository and its contents are not an official Google product.

Overview (`okf/README.md`) Copy Section

[!IMPORTANT]
OKF now lives in its own repository:
GoogleCloudPlatform/open-knowledge-format.

That repository is the canonical home of the specification, the reference
agent, and the sample bundles. Please read the spec, file issues, and open
pull requests there.

Stop using the copy under okf/ in this repository. It is a frozen
snapshot, no longer maintained, and anything built against it will drift out
of date.

Open Knowledge Format (OKF)

📖 [Read the Open Knowledge Format v0.2 specification → SPEC.md](SPEC.md) Copy Section

This directory is primarily about the Open Knowledge Format
(OKF).

OKF is a universal, vendor-neutral format for representing knowledge
as plain markdown files with YAML frontmatter. It is not tied to any
particular agent, framework, model provider, or serving system. The
goal is simple:

• Anyone can produce OKF — humans authoring by hand, agents built on
  any framework (Google ADK, LangChain, custom), export pipelines from
  existing catalogs (Dataplex, Unity Catalog, Collibra, …), or scripts
  walking a database.
• Anyone can serve and consume OKF — a static file server, a
  knowledge-management UI (Obsidian, Notion, MkDocs), an LLM loading
  files into context, a search index, or a graph viewer like the one
  bundled in this repo.

The agent below is a proof of concept demonstrating one way to
produce OKF bundles automatically. The format itself is the
contribution; this agent and the visualizer exist to make the format
tangible at both ends — production and consumption.

See OKF in practice — three ready-to-browse bundles produced by this
agent, checked into bundles/:

• bundles/ga4/ — GA4 e-commerce dataset
  (viz.html)
• bundles/stackoverflow/ — Stack Overflow
  public dataset (viz.html)
• bundles/crypto_bitcoin/ — Bitcoin
  blocks/transactions (viz.html)
• bundles/acme_retail/ — Acme Retail
  (viz.html)

Why OKF? Copy Chapter

OKF represents catalog knowledge as plain markdown files with YAML
frontmatter, organized in a directory hierarchy. That choice unlocks a few
properties that are hard to get from a service-owned metadata store:

• Human- and agent-readable. No SDK or query language stands between a
  reader and the content. An engineer can cat a concept; an LLM can ingest
  it verbatim into context.
• Version-controllable out of the box. Bundles live in git. Pull
  requests, line-by-line diffs, blame, and review workflows just work —
  knowledge curation becomes a normal software-engineering activity.
• Portable and lock-in free. A bundle is a directory. Ship it as a
  tarball, host it in any repo, mount it from any filesystem, or sync it to
  any system that speaks files. No proprietary API stands between you and
  your metadata.
• Mixes structured and unstructured data deliberately. Use frontmatter
  for the few fields you want to query, filter, or index on (type,
  resource, tags, generated, status); use the markdown body for the
  prose, schemas, and example queries that LLMs and humans actually read.
• Trust, provenance, and freshness are first-class. v0.2 puts queryable
  signals in frontmatter — where a concept came from (sources with per-source
  credibility signals), who produced and confirmed it (generated, verified,
  from which consumers derive a trust tier), and whether it is still current
  (status, stale_after) — so an agent-maintained corpus stays trustable
  without any bespoke runtime.
• Minimally opinionated, freely extensible. A small set of required
  keys ensures interoperability, but bundles can carry arbitrary extra
  frontmatter keys and arbitrary body sections without breaking
  consumers.
• Composes with existing tooling. Many knowledge tools — Notion,
  Obsidian, MkDocs, Hugo, Jekyll — already speak markdown plus YAML
  frontmatter, so bundles can be browsed, edited, or rendered without
  custom UI.
• Progressive disclosure built in. Auto-generated index.md files
  let an agent or human navigate the hierarchy one level at a time
  instead of loading the entire bundle into context.
• Graph-shaped, not just tree-shaped. Concepts link to each other via
  normal markdown links, expressing relationships richer than the
  parent/child implied by the directory layout.

The net effect is that reference agents, consumption agents, and humans
collaborate on the same artifacts in the same way they already collaborate
on source code.

Install Copy Chapter

python3.13 -m venv .venv
.venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev]

Credentials Copy Chapter

• BigQuery: gcloud auth application-default login plus a project for billing
  (gcloud config set project ). Public datasets are readable, but the
  caller's project is billed for query bytes.
• Gemini: set GEMINI_API_KEY (AI Studio) or use Vertex AI by setting
  GOOGLE_GENAI_USE_VERTEXAI=true, GOOGLE_CLOUD_PROJECT=, and
  GOOGLE_CLOUD_LOCATION=<region>.

How the reference agent works Copy Chapter

The reference agent runs in two passes. The BQ pass writes one OKF
doc per concept the source advertises, using BigQuery metadata alone.
The web pass then runs the LLM as its own crawler: it receives a
list of seed URLs (provided via --web-seed or --web-seed-file),
fetches the seeds via the fetch_url tool, and decides which outbound
links are worth following based on whether they look like authoritative
documentation for the existing concepts. For each page it fetches, the
agent chooses to (a) enrich one or more existing concept docs, (b) mint
a standalone references/<slug> doc, or (c) skip. A hard
--web-max-pages cap and a same-domain allowed-hosts filter
(configurable via --web-allowed-host) are enforced inside the tool,
so the agent cannot overrun. Use --no-web to skip the web pass.

Run Copy Chapter

Minimum invocation — point at a BigQuery dataset and a bundle output
directory. Seeds for the web pass are explicit; omit them (or pass
--no-web) to run BQ-only:

.venv/bin/python -m reference_agent enrich \
    --source bq \
    --dataset <project>.<dataset> \
    --web-seed-file <path/to/seeds.txt> \
    --out ./bundles/<name>

Iterate on a single concept by adding --concept <type>/<name> (e.g.
--concept tables/events_); repeatable.

Samples Copy Chapter

Each sample pairs a recipe (samples/<name>/, with the seed URLs and
exact enrich command) with the produced bundle (bundles/<name>/)
that the recipe generated. Open the recipe to reproduce; open the bundle
to browse the result directly.

• GA4 Google Merchandise Store — public e-commerce dataset, seeded
  with canonical GA4 BigQuery Export documentation URLs.
  · recipe
  · bundle
  · viz.html
• Stack Overflow — public dataset (mirror of the Stack Exchange Data
  Dump), seeded with the community's canonical schema references.
  Exercises multi-concept enrichment from cross-cutting docs pages.
  · recipe
  · bundle
  · viz.html
• Bitcoin (crypto) — public dataset (blocks, transactions, inputs,
  outputs) from the bitcoin-etl pipeline. Exercises cross-table
  foreign-key relationships in prose.
  · recipe
  · bundle
  · viz.html

Visualize Copy Chapter

The visualize subcommand renders any OKF bundle as a self-contained
interactive HTML file — one file, no backend, no install on the
viewing side. Open it in any modern browser, share it as an artifact,
host it on a static file server, or commit it next to the bundle (as
this repo does).

The viewer is itself a proof-of-concept consumer of OKF, mirroring
the way the reference agent is a proof-of-concept producer. OKF
bundles can be consumed by anything that reads markdown; this is just
one shape.

What it shows Copy Section

• A force-directed graph of every concept in the bundle, with
  colored nodes by type (datasets, tables, references, …) and directed
  edges drawn from each cross-link in the markdown bodies.
• A detail panel for the selected concept showing its frontmatter
  (description, resource link, tags) and its rendered markdown body —
  with internal […](/path/to/concept.md) links rewired to navigate
  within the viewer instead of following the path.
• A "Cited by" backlinks list under each concept (computed from the
  reverse of the link graph).
• A search box (matches title, concept id, and tags), a type
  filter, and switchable graph layouts (cose / concentric /
  breadth-first / circle / grid).

Generate Copy Section

.venv/bin/python -m reference_agent visualize --bundle ./bundles/<name>

That writes bundles/<name>/viz.html. Flags:

Example, writing the output somewhere else and overriding the header:

.venv/bin/python -m reference_agent visualize \
    --bundle ./bundles/crypto_bitcoin \
    --out /tmp/btc.html \
    --name "Bitcoin OKF"

How it's built Copy Section

The HTML embeds the bundle as a JSON blob and uses
Cytoscape.js for the graph and
marked for in-browser markdown rendering,
both loaded from a CDN. No data leaves the page; the bundle is parsed
once at generation time and serialized into the file.

Tests Copy Chapter

.venv/bin/pytest

Overview (`samples/README.md`) Copy Section

Samples

The samples demonstrate the use of Knowledge Catalog to manage metadata and
context to power agents.

Discovery Copy Chapter

Demonstrates building a search and discovery agent building on top of the
Search APIs offered by the catalog.

Enrichment Copy Chapter

Demonstrates an enrichment agent that can generate and enrich documentation
for assets managed in the catalog.

Governance Agent Copy Chapter

Demonstrates agentic data governance that automates metadata management for
assets managed in the catalog, propagating descriptions, glossary terms, policy
tags and data quality scores using column-level lineage and Dataset Insights.

Policy-as-Code Agent Copy Chapter

Demonstrates an agent that lets users define and validate data governance
policies in natural language, translating them into executable code that runs
against metadata in the catalog and BigQuery.

Overview (`toolbox/README.md`) Copy Section

Toolbox

Contains tools to work with Knowledge Catalog metadata.

• Metadata as Code
  Provides the ability to manage metadata in the form of source code artifacts that
  can be sync'd with metadata in Knowledge Catalog.

• Enrichment Agent
  Provides an ready-to-use agent and customizable harness to produce, evolve/improve
  and maintain metadata within Knowledge Catalog and make it ready for consumption by
  agents.

Overview (`samples/discovery/README.md`) Copy Section

Knowledge Catalog Discovery Agent

About Copy Chapter

The Knowledge Catalog Discovery Agent is an AI-powered search assistant for discovering data assets in Google Cloud. While standard semantic search only matches semantically similar text, this agent goes further by performing semantic decomposition of complex questions, generating multiple relevant search queries, and reranking the final results to provide a comprehensive answer.

Quick Start Copy Chapter

Prerequisites Copy Section

You need a Google Cloud project with the following APIs enabled:

• Knowledge Catalog (dataplex.googleapis.com)
• Vertex AI (aiplatform.googleapis.com)
• Service Usage API (serviceusage.googleapis.com)

Ensure you have the following permissions via IAM Roles:

• dataplex.projects.search - can be obtained via roles like roles/dataplex.viewer.
• aiplatform.endpoints.predict - can be obtained via roles like roles/aiplatform.user.
• serviceusage.services.use - needed to use the current project as the quota project. Can be obtained via roles like roles/serviceusage.serviceUsageConsumer.

Setup Copy Section

Clone the Github Repository:

git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git

Install the dependencies. It is recommended to create a Python Virtual Environment:

python3 -m venv /tmp/kcsearch  
source /tmp/kcsearch/bin/activate  

cd samples/discovery
pip3 install -r requirements.txt

Set the required environment variables:

# Replace <PROJECT_ID> with your consumer project ID.
export GOOGLE_CLOUD_PROJECT=<PROJECT_ID>
export GOOGLE_GENAI_USE_VERTEXAI=True

Run the Agent Copy Section

There are 2 ways to run the Knowledge Catalog Discovery agent:

1. As the root agent - in this case, rename the discovery_agent variable in agent.py to root_agent and run it using ADK CLI (steps below).

2. As a sub-agent - in this case, import the Discovery Agent and leverage it as an AgentTool. You can find examples of this in the official ADK docs. The recommended parent folder structure in this case is:

my_custom_agent/
├── agent.py
└── knowledge_catalog_discovery_agent/
    ├── SKILL.md
    ├── agent.py
    ├── tools.py
    └── utils.py

Irrespective of the path you choose, the agent can be run using the ADK CLI commands below:

adk run path/to/agent/parent/folder

Replace path/to/agent/parent/folder with the relative or absolute path to the parent directory of the one containing the agent's source code.

References Copy Chapter

• Knowledge Catalog Search API
• ADK Documentation
• Predefined IAM Roles

Overview (`samples/enrichment/README.md`) Copy Section

Knowledge Catalog Enrichment Agent

Overview Copy Chapter

Knowledge Catalog manages metadata about data assets (such as BigQuery tables).
This metadata powers search, and agents to discover and understand data assets.

This sample demonstrates how external sources of information can be used to
augment the metadata in Knowledge Catalog, using an agentic approach. The
enrichment agent finds information relevant to a data asset, generates
documentation, and uses Knowledge Catalog APIs to publish the documentation
as enriched metadata.

Users can customize this enrichment agent with custom instructions, tools to
access sources of information within their organization, and skills to
describe the use of these tools.

Running the Sample Copy Chapter

Setup Copy Section

Setup your environment

This step clones the repository locally, configures various environment variables,
creates a python virtual environment and installs dependencies.

git clone https://github.com/googlecloudplatform/knowledge-catalog
cd samples/enrichment

Ensure you have the GCloud CLI installed and configured.

export CLOUD_PROJECT=<cloud-project-id>

gcloud auth application-default login
gcloud config set core/project $CLOUD_PROJECT
gcloud auth application-default set-quota-project $CLOUD_PROJECT

Setup a python environment

cd src
source env.sh --install

Setup sample data

The sample creates a sample BigQuery dataset in your project whose metadata is
curated.

python3 ../sample/data/create_data.py

Enrichment Steps Copy Section

You are now ready to run the enrichment workflow.

Download a metadata snapshot

python3 -m enrichment.download \
  --dir ../sample/metadata.initial \
  --dataset ${KC_ENRICH_SAMPLE_PROJECT}.kc_enrich_sample_data

Enrich the metadata

python3 -m enrichment.enrich \
  --dir ../sample/metadata.initial \
  --output-dir ../sample/metadata.new \
  --config-dir ../sample/config

Review the metadata updates

git diff --no-index \
  ../sample/metadata.initial ../sample/metadata.new

Publish the updated metadata

python3 -m enrichment.publish \
  --dir ../sample/metadata.new

Overview (`samples/governance-agent/README.md`) Copy Section

Governance on Auto-pilot: Agentic Data Governance with Knowledge Catalog

This project demonstrates an agentic data governance solution using Google Cloud Knowledge Catalog. It showcases how to automate metadata management - propagate descriptions, glossary terms, policy tags etc. using column level lineage and leveraging Knowledge Catalog Dataset Insights capabilities. This is only a demonstration and is not part of official product, please review everything before using it for your environments and use-cases.

🌟 Key Features Copy Chapter

• Estate Dashboard: Scan BigQuery datasets to identify metadata gaps (missing descriptions).
• Recursive Description Propagation: Automatically fetch descriptions from upstream sources, bridging multi-hop gaps.
• SQL-Based Logic Enrichment: Extracts BigQuery SQL transformations to generate human-readable descriptions for computed columns.
• AI Business Glossary: Maps technical columns to business terms using Vertex AI Semantic Similarity and Unstructured Documents (PDFs, TXT, MD).
• Prioritized Glossary Propagation: Automatically propagates glossary terms across tables based on lineage, with strict verification thresholds to ensure accuracy (especially for 1-1 mappings).
• Native Knowledge Catalog Integration: Persists glossary mappings as native EntryLinks visible in the Knowledge Catalog Schema tab.
• Unified UI & CLI: Manage governance tasks via a Gradio-based web app or a headless CLI.
• Policy Tag Propagation: Recommends and applies BigQuery policy tags via lineage, with support for "straight pull" detection and an integrated Access Summary (Readers & Data Policies).
• Data Trust Center (DQ): Derived trust scores for views and tables based on upstream Knowledge Catalog DQ/Profiling results and multi-hop lineage.
• Unstructured Document Processing: Leverage PDFs, TXT, and Markdown files to influence column descriptions and policy tags using Gemini and RAG.
• Remediation Detection: Automatically detects SQL transformations (e.g. COALESCE, DISTINCT) that improve data quality and applies "Trust Bonuses".
• Trust History Persistence: Tracks 0.0-1.0 trust scores over time in BigQuery for trend analysis.
• CLL API Preview allowlisting required: Please contact your Google Cloud account team to get access to CLL API

🛠 Setup & Installation Copy Chapter

Prerequisites Copy Section

• Python 3.11+
• Google Cloud Project with billing enabled.
• APIs Enabled: dataplex, bigquery, datacatalog, datalineage, aiplatform.

Installation Copy Section

1. Clone & Navigate:
   bash Copy

   git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git
   cd knowledge-catalog/samples/governance-agent

2. Environment Setup:
   bash Copy

   # Install dependencies and sync virtual environment using uv
   uv sync --group dev

3. Authentication:
   • CLI/Dev: Run gcloud auth application-default login.
   • Gradio App: Follow OAUTH_SETUP_GUIDE.md to enable "Login with Google".
4. Project: Set the Google Cloud project to work in, in your shell or in .env:
   bash Copy

   export GOOGLE_CLOUD_PROJECT=<PROJECT_ID>

   The Steward CLI and UI fall back to your gcloud default project (gcloud config set project ) when it isn't set, and the CLI also accepts --project. The data generation scripts require GOOGLE_CLOUD_PROJECT.

🚀 Usage Guide Copy Chapter

1. Agentic Data Steward (UI) Copy Section

The Gradio app provides a visual way to scan and apply metadata changes.

uv run python -m metadata_propagation.ui.gradio_app

• Dashboard: Run "Scan Dataset" to see health metrics.
• Description Propagation: Enter a table name to preview and apply lineage-based descriptions.
• Glossary Propagation: Enter a table name to preview and apply lineage-based glossary terms.
• Policy Tag Propagation: Propagate sensitive data tags across the lineage chain with automated transformation assessment.
• Trust Center (DQ): Analyze derived trust scores for a view or table, showing upstream quality sources and remediation bonuses.
• Settings: Toggle OAuth/ADC modes for specific user actions.

2. 🐳 Deployment (Docker & Cloud Run) Copy Section

For all deployment steps (local Docker and production Cloud Run), please refer to the comprehensive Deployment Guide.

Key sections covered:

• Local Docker Run: Building and testing the container locally.
• Automated Cloud Run Deployment (deploy.sh): Automated Artifact Registry and Cloud Run setup.
• Direct Source Deploy (gcloud run deploy --source): Build and deploy via Cloud Build without requiring Docker locally.
• Service Account IAM Roles: Required permissions for BigQuery, Dataplex, Lineage, and Vertex AI.
• Authentication & OAuth Modes: Running with Service Account ADC (BYPASS_OAUTH=true) vs Google OAuth (see also OAUTH_SETUP_GUIDE.md).
• Production Sizing & Troubleshooting: Sizing recommendations and troubleshooting common deployment errors.

3. Steward CLI (Headless) Copy Section

The CLI is designed for automation and quick scans.

/* Detailed source-code truncated for AI context efficiency. */

4. Data Integration Scripts Copy Section

• Generate Data: uv run python -m metadata_propagation.data_generation.generate_data (Creates tables + lineage).
• Run Profiling & DQ: uv run python -m metadata_propagation.data_generation.run_dq_scans (Creates and runs Data Profile and Quality scans, natively published to BigQuery).
• Unified Insights: Run the Steward CLI's knowledge-propagate command (above) without --apply to trigger the Knowledge Catalog documentation scan, wait for it, and extract the results for review.

📄 Unstructured Document Processing Copy Chapter

The tool can leverage unstructured documents (PDFs, TXT, MD) to influence data governance metadata when lineage is missing or to supplement it.

Supported Components Copy Section

• Column Descriptions: Extracts business meanings and definitions for columns. Enforces strict grounding to avoid hallucinations.
• Policy Tags (PII): Extracts explicit sensitivity labels (like PII: Y) from documents and maps them to allowed policy tags in your project. No inference or guessing.
• Business Glossary Terms: Maps columns to a controlled list of business terms based on explicit mentions in documents. Enforces strict grounding.

Document Context Modes Copy Section

When using the --document flag with the apply, policy-propagate, or glossary-recommend commands, you can specify a --context-mode to choose how the document is processed:

• direct (Context Injection): Reads the full text of the document(s) and appends it directly to the Gemini prompt for each column. Best for small documents.
  bash Copy

  uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --document docs/small_doc.txt --context-mode direct

• rag (In-Memory RAG): Chunks the document, generates embeddings, and retrieves the most relevant snippets for each column. Best for large documents to save tokens and improve focus.
  bash Copy

  uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --document docs/large_doc.pdf --context-mode rag

  💡 Note: Vector embeddings are generated remotely using Vertex AI's text-embedding-004 model and stored locally in-memory for fast similarity search.

• datastore (Managed DataStore): Queries a pre-existing Vertex AI Search DataStore. Best for enterprise setups with large document repositories.
  bash Copy

  uv run python -m metadata_propagation.steward_cli apply --dataset retail_syn_data --table transactions --context-mode datastore --datastore-id my-datastore-id

⚙️ Glossary Cache Configuration Copy Chapter

In enterprise governance workflows, stewards often scan read-only production datasets (where they lack bigquery.dataEditor permissions). To optimize glossary mapping recommendations, the steward needs a cache. The system supports redirecting the business glossary embeddings cache to a dedicated writable dataset:

1. Web UI (Gradio) Copy Section

Open the Global Environment Settings accordion at the top of the dashboard:

• Glossary Cache Dataset ID: The dataset where you want the glossary_embeddings_cache table to be created (e.g., a separate sandbox or metadata dataset).
• Glossary Cache Table ID: Custom table name for storing cached embeddings (defaults to glossary_embeddings_cache).

2. CLI Flags Copy Section

You can pass the cache location explicitly as command line arguments to any command:

uv run python -m metadata_propagation.steward_cli glossary-recommend \
  --dataset prod_readonly_dataset \
  --table transactions \
  --cache-dataset metadata_sandbox_dataset \
  --cache-table my_glossary_cache

3. Environment Variables Copy Section

You can also set these values in a .env file or in your shell:

• GLOSSARY_CACHE_DATASET_ID: Redirects the cache table to a separate BigQuery dataset.
• GLOSSARY_CACHE_TABLE_ID: Custom name for the BigQuery cache table.

💡 Note: If BigQuery cache initialization fails due to write permissions, the plugin automatically falls back to a local JSON file cache (scratch/glossary_embeddings_cache.json) transparently, ensuring zero disruption to the user experience.

🧩 Project Modules Copy Chapter

💡 Workflow Example Copy Chapter

1. Initialize: Generate synthetic data and lineage relationships (see Data Integration Scripts).
2. Enrich & Propagate: Run the unified knowledge-propagate command to trigger AI scans and sync metadata.
3. Tag: Use the Glossary Plugin to map technical columns to the Business Glossary for Knowledge Catalog UI visibility.
4. Secure: Use the Policy Tag Propagation plugin to sync sensitive data tags and verify access summary (Readers/Masking Rules).
5. Verify: Check the BigQuery Console (Schema -> Policy Tags) and Knowledge Catalog Schema (Business Terms).

Overview (`samples/policy-as-code-agent/README.md`) Copy Section

Policy-as-Code Agent

A generative AI-powered agent designed to automate data governance on Google Cloud. It allows users to define, validate, and enforce data policies using natural language queries, translating them into executable code that runs against Knowledge Catalog (formerly Dataplex) and BigQuery metadata.

The Agent Starter Pack (ASP) is the recommended way to create a new project from this sample: you get deployment options and CI/CD scaffolding. The copy in knowledge-catalog is the upstream source for browsing and contributions.

Prerequisites Copy Chapter

• Python 3.11+
• uv — fast Python package manager
• Google Cloud SDK (gcloud) installed and authenticated
• Git

Getting Started Copy Chapter

To explore this sample or contribute, you can run the agent directly from this directory in the repository.

1. Clone the repository and navigate to this sample:

   bash Copy

   git clone https://github.com/GoogleCloudPlatform/knowledge-catalog.git
   cd knowledge-catalog/samples/policy-as-code-agent

2. Install dependencies:

   bash Copy

   uv sync --group dev

3. Configure environment:
   Copy the example configuration file:

   bash Copy

   cp .env.example .env

   Open .env and fill in your details:

   • GOOGLE_CLOUD_PROJECT: Your Google Cloud Project ID.
   • GOOGLE_CLOUD_LOCATION: (e.g., us-central1).
   • ENABLE_MEMORY_BANK: Set to True to enable long-term memory (requires Firestore). Set to False to run without it. See Memory Integration for details.
   • FIRESTORE_DATABASE: (Optional) Leave as (default) unless using a named database.

4. Authenticate with Google Cloud:

   bash Copy

   gcloud auth application-default login

Running the Agent Copy Chapter

From the sample directory (samples/policy-as-code-agent):

uv run adk run policy_as_code_agent

Or use the web UI:

uv run adk web

Optional: To enable short-term contextual memory (Agent Engine) for better conversation history:

uv run adk web --memory_service_uri="agentengine://AGENT_ENGINE_ID"

Open the printed URL in your browser to chat with the agent.

Running Tests Copy Chapter

uv run pytest

Run only fast unit tests:

uv run pytest tests/unit -v

Integration tests are marked and may require GCP credentials and a configured .env.

Creating a New Project (Agent Starter Pack) Copy Chapter

If you want to use this sample as a base to create and deploy a new, independent project for production, we recommend using the Agent Starter Pack. This provides deployment options and CI/CD scaffolding.

Start from a new directory (replace my-policy-as-code with your project name) and run:

uvx agent-starter-pack create my-policy-as-code -a adk@policy-as-code

This will scaffold a new project and prompt you for deployment options.

Key Features Copy Chapter

• Natural Language Policies: "All tables in the finance dataset must have a description."
• Hybrid Execution: Generates Python code on-the-fly for flexibility, but executes it in a sandboxed environment for safety.
• Memory & Learning: Uses Firestore and Vector Search to remember valid policies. If you ask a similar question later, it reuses the proven code instead of regenerating it.
• Dual-Mode Operation:
  • Live Mode: Queries Knowledge Catalog in real-time.
  • Offline Mode: Analyzes metadata exports stored in Google Cloud Storage (GCS).
• Compliance Scorecards: Run a full health check on your data assets with a single command.
• Remediation: Can suggest specific fixes for identified violations.

Architecture Copy Chapter

The agent is built using the Google Cloud Agent Development Kit (ADK) and leverages several Google Cloud services:

• Gemini 2.5 Pro: For complex code generation (converting natural language to Python).
• Gemini 2.5 Flash: For conversational logic, tool selection, and remediation suggestions.
• Vertex AI Vector Search: For semantic retrieval of past policies.
• Firestore: Stores policy definitions, versions, and execution history.
• Knowledge Catalog API (dataplex.googleapis.com): For fetching live metadata.

Project Structure Copy Section

• policy_as_code_agent/
  • __init__.py: Application Default Credentials and Vertex environment defaults for ASP/local runs.
  • agent.py: Entry point and core agent definition.
  • memory.py: Handles Firestore interactions (saving/retrieving policies).
  • utils/: Utility modules for LLM logic, Knowledge Catalog, GCS, and common tools.
  • simulation.py: Sandboxed execution engine for running policy code.
  • prompts/: Markdown templates for LLM instructions.
• tests/: Unit and integration tests.

Documentation Copy Chapter

For deep dives into the implementation, check the docs/ folder:

• High-Level Architecture
• Low-Level Implementation
• Memory Implementation

Overview (`toolbox/enrichment/README.md`) Copy Section

Enrichment Agent

The enrichment agent for Knowledge Catalog provides a customizable agentic
workflow for extracting information from various sources to build metadata
about data assets, which can then be used as context.

Usage Copy Chapter

Prerequisites Copy Section

The enrichment agent depends on the Metadata as Code capability.
Follow the instructions on that page on using the kcmd tool.

CLI Copy Section

The package provides the kcenrich CLI tool. This is distributed as a standalone binary.

# Initialize a new catalog snapshot for a bigquery dataset
kcmd init --bigquery-dataset <projectId>.<datasetId>

# Pull the latest catalog snapshot from the Knowledge Catalog service
kcmd pull

# Run the enrichment tool
kcagent enrich --catalog-path . --tools-path tools --prompt-path prompt.md

Developer Workflow Copy Chapter

Setup Copy Section

git clone https://github.com/googlecloudplatform/knowledge-catalog
cd toolbox/enrichment
npm install

Build Copy Section

npm run build

Test Copy Section

npm run test

Demo Copy Section

The repository contains a self-contained demo. Running the demo involves creating a BigQuery dataset and a Dataplex EntryGroup within your cloud project.

Initialize Environment

export DEMO_CLOUD_PROJECT="<your-gcp-project-id>"

Initialize gcloud

gcloud auth application-default login
gcloud config set project $DEMO_CLOUD_PROJECT
gcloud config set compute/region us

Setup demo resources

bq query --use_legacy_sql=false <<EOF
CREATE SCHEMA IF NOT EXISTS \`${DEMO_CLOUD_PROJECT}.demo_ecommerce\`
OPTIONS (
  location = 'US',
  labels = [('usage', 'demo')]
);

CREATE TABLE IF NOT EXISTS \`${DEMO_CLOUD_PROJECT}.demo_ecommerce.events\`
PARTITION BY event_date_dt
AS
SELECT
  *,
  PARSE_DATE('%Y%m%d', event_date) AS event_date_dt
FROM
  \`bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*\`;
EOF

Create and populate a catalog snapshot

mkdir -p demo
cd demo
cat <<EOF > catalog.yaml
scope: bq-dataset.${DEMO_CLOUD_PROJECT}.demo_ecommerce

snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
    - dataplex-types.global.bigquery-table
  aspects:
    - dataplex-types.global.overview
EOF

../../mdcode/dist/kcmd pull

Create and populate the tools

cat <<EOF > prompt.md
Enrich the documentation of the assets using the internal organizational information.
Use the following sources:

* Fileset source
EOF

mkdir tools
cat <<EOF > tools/mcp.json
{
  "mcpServers": {
    "md-fileset": {
      "command": "../dist/md-fileset",
      "args": [ "--dir", "fileset" ]
    }
  }
}
EOF

mkdir -p tools/skills/fileset-source
cat <<EOF > tools/skills/fileset-source/SKILL.md
---
name: fileset-source
description: >
  Use the fileset source to find relevant markdown documents and extract information
  about assets.
---

The `md-fileset` mcp server provides the following tools to extract relevant
information from a directory hierarchy of markdown files:

* **list_fileset_contents** - browse and navigate the directory tree to list the
  contents of the specified path. The items may be files or sub-directories.

* **read_fileset_file** - read the contents of a file in the knowledge base. The entire
  contents are provided. Extract and summarise the relevant information based on
  the documentation being generated..

* **search_fileset_content** - searches the knowledge base and returns the matching files,
  along with matching line numbers and line snippets. This can be used to quickly
  find matches without having to list and read all files.

To work with a fileset effectively create search queries (use simple keyword queries
with individual tokens) to find relevant files, and then read the files to find relevant information. If one query does not work, try a few other keywords.
EOF

Add Docs
Copy over the individual markdown files from here into the fileset/ directory

Enrich the metadata

../dist/kcagent enrich --catalog-path . --tools-path tools --prompt-path prompt.md

Clean up

bq rm -r ${DEMO_CLOUD_PROJECT}:demo-dataset

Overview (`toolbox/mdcode/README.md`) Copy Section

Metadata as Code

Metadata as Code is a Knowledge Catalog (Dataplex) provides data stewards and data producers and AI agents with a source code artifact-based UX for metadata management and context engineering.

Users and agents can author, manage, and enrich metadata artifacts using developer-friendly workflows with version control and CI/CD. It provides a standard metadata format can be used by a variety of tools and agents.

More details are in the docs/concept.md.

Key Features Copy Chapter

• Intuitive human and agent-friendly representation of metadata as source code in YAML and markdown files. Artifacts are organized in a hierarchical manner mirroring the resource hierarchy of data and metadata assets.
• Bi-directional sync between local workspace and catalog service.
• Support for both 1st party and 3rd party metadata constructs.
• Distributed as a TypeScript and Python libraries, a CLI tool (kcmd), and an MCP server for use in a variety of applications, agents and workflows.

Metadata Artifacts Copy Chapter

Directory Layout Copy Section

Metadata is organized within a directory representing a resource such as a BigQuery Dataset, Dataplex EntryGroup etc.

path/to/root/
├── catalog.yaml                       # Manifest and config directives
└── catalog/                           # Contains the metadata snapshot
    └── <dir1>/
        └── <entry-id1>.yaml           # Entry
        └── <dir2>/
            ├── <entry-id2>.yaml       # Entry with sidecar markdown
            └── <entry-id2>.aspect.md  # files

Catalog Snapshot Files Copy Chapter

Manifest File Copy Section

catalog/catalog.yaml

scope: bq-dataset.prod-data.ecommerce

aliases:
  ca-guidelines:
    aspect: data-agents-project.global.ca-guidelines
  ecommerce:
    aspect: data-agents-project.global.ecommerce

snapshot:
  entries:
    - dataplex-types.global.bigquery-dataset
    - dataplex-types.global.bigquery-table
    - dataplex-types.global.bigquery-view
  aspects:
    - dataplex-types.global.overview
    - dataplex-types.global.descriptions

publishing:
  aspects:
    - dataplex-types.global.overview
    - dataplex-types.global.descriptions

Entry YAML File Copy Chapter

catalog/prod-data.ecommerce/products.yaml

id: products
type: bigquery-table

resource:
  name: projects/prod-data/datasets/ecommerce/tables/products
  displayName: Products Table
  description: All products in the catalog
  labels:
    env: prod
  createTime: 2026-04-23T00:44:03Z
  updateTime: 2026-04-23T00:44:03Z

schema:
  ...

contacts:
  ...

Entry Sidecar Markdown File Copy Chapter

catalog/prod-data.ecommerce/products.overview.md

---
userManaged: true
links:
  ...
---
[overview.content]

Usage Copy Chapter

Library Copy Section

You can use the kcmd library to programmatically interact with the catalog metadata.

npm install kcmd

import * as kcmd from 'kcmd';

// Creating a catalog manifest from scratch
const manifest = new kcmd.CatalogManifest(...);
manifest.save('/path/to/root');

// Loading a catalog snapshot from the filesystem
const snapshot = kcmd.CatalogSnapshot.fromPath('/path/to/root');

// Pulling the latest metadata from the Catalog service.
const pullResult = await snapshot.pull();
if (pullResult.success) {
  console.log('Metadata pulled successfully');
}
else {
  console.error('Metadata pull failed:', pullResult.error);
}

// Pushing the modified metadata to the Catalog service.
const pushResult = await snapshot.push();
if (pushResult.success) {
  console.log('Metadata pushed successfully');
}
else {
  console.error('Metadata push failed:', pushResult.error);
}

CLI Copy Section

The package provides the kcmd CLI tool. This is distributed as a standalone binary.

/* Detailed source-code truncated for AI context efficiency. */

For the semantic-model scope, kcmd push deploys the model as a property graph
(CREATE OR REPLACE PROPERTY GRAPH) to the backend named by the model's GOOGLE
deployment target — BigQuery Graph or Spanner Graph; the same authored
model deploys to either, and swapping the target switches backends. Pass
--validate-only to print the generated DDL without executing it. For a BigQuery
target this path additionally reads the target dataset's metadata
(bigquery.datasets.get) to pin the query job's processing location; the call
degrades gracefully when the permission is absent, so a narrower service account
still works but falls back to BigQuery's own location inference. Each dataset
source is the table's GCP resource name, such as
bigquery.googleapis.com/projects//datasets/<d>/tables/<t>.
(A Spanner target names its tables inside one database, so a source is reduced
to its final segment, and Spanner Graph has no MEASURE, so metrics are dropped
with a warning.)

kcmd push also deploys the model to Knowledge Catalog (entries for the model,
its entities and metrics, and schema-join links for its relationships). A push
has two axes: the binding-profile axis picks how many binding profiles the graph
deploys for (--no-profile for none, the default binding, --profile <name> for
one, or --all-profiles for every one), and each profile's deployment_target
selects its backend; --no-kc skips the Knowledge Catalog leg. Use --print to
dump each destination's generated artifact, and --force-remove to delete models
left in the entry group that the push no longer includes. See
docs/semantic-model/ for the full guide,
including how re-push reconciles removed entities, metrics, relationships, and
models.

NOTE: The CLI uses gcloud to obtain authentication tokens, so ensure you are authenticated via gcloud auth application-default login.

Environment variables

Two optional variables override where kcmd talks to the catalog. Both default
to production, so you normally leave them unset.

MCP Server Copy Section

To use the Metadata as Code tools as MCP tools in an agentic system such as the Gemini CLI, add the following to your MCP settings file:

{
  "mcpServers": {
    "kc-mac": {
      "command": "kcmd",
      "args": ["mcp", "--path", "/path/to/root"]
    }
  }
}

The server offers the following tools:

NOTE: The server uses gcloud to obtain authentication tokens, so ensure you are authenticated via gcloud auth application-default login.

Developer Workflow Copy Chapter

Setup Copy Section

git clone https://github.com/googlecloudplatform/knowledge-catalog
cd toolbox/mdcode
npm install

Build Copy Section

npm run build

Test Copy Section

npm run test

Overview (`okf/samples/crypto_bitcoin/README.md`) Copy Section

Bitcoin public dataset sample

Runs the reference agent against the public
bigquery-public-data.crypto_bitcoin dataset (blocks, transactions,
inputs, outputs — produced by the open-source bitcoin-etl pipeline)
and seeds the web pass with the canonical schema source and the
foundational Google Cloud blockchain-on-BigQuery announcement.

This sample contrasts with GA4 (single denormalized events table) and
Stack Overflow (many independent entities) by exercising a small set
of tightly related fact tables where each row in transactions
references rows in blocks, inputs, and outputs. Good for seeing
how the agent surfaces cross-table foreign-key relationships in prose.

Prerequisites Copy Chapter

• Install the agent (from the repo root):
  text Copy

  python3.13 -m venv .venv
  .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev]

• BigQuery access:
  text Copy

  gcloud auth application-default login
  gcloud config set project <your-billing-project>

  Public datasets are readable, but the caller's project is billed for
  query bytes. The crypto_bitcoin tables are very large
  (transactions is ~hundreds of GB) — keep --web-max-pages modest
  while iterating and prefer --concept for smoke runs.
• Gemini credentials — either GEMINI_API_KEY (AI Studio) or Vertex
  AI (GOOGLE_GENAI_USE_VERTEXAI=true, GOOGLE_CLOUD_PROJECT=,
  GOOGLE_CLOUD_LOCATION=<region>).

Run Copy Chapter

.venv/bin/python -m reference_agent enrich \
    --source bq \
    --dataset bigquery-public-data.crypto_bitcoin \
    --web-seed-file samples/crypto_bitcoin/seeds.txt \
    --out ./bundles/crypto_bitcoin

To iterate on a single concept, add --concept tables/transactions.
To skip the web pass, add --no-web. To raise or lower the web budget,
use --web-max-pages N (default 100).

What you get Copy Chapter

A bundle under ./bundles/crypto_bitcoin/ with one OKF doc per BQ
concept (dataset + each table), augmented and cross-linked with
reference docs minted from the seeded blockchain-etl and Google Cloud
pages, plus an auto-generated index.md at each directory level.

Overview (`okf/samples/ga4_merch_store/README.md`) Copy Section

GA4 Google Merchandise Store sample

Runs the reference agent against the public
bigquery-public-data.ga4_obfuscated_sample_ecommerce dataset (a GA4 export
from the Google Merchandise Store) and seeds the web pass with canonical GA4
BigQuery Export documentation URLs.

Prerequisites Copy Chapter

• Install the agent (from the repo root):
  text Copy

  python3.13 -m venv .venv
  .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev]

• BigQuery access:
  text Copy

  gcloud auth application-default login
  gcloud config set project <your-billing-project>

  Public datasets are readable, but the caller's project is billed for
  query bytes.
• Gemini credentials — either GEMINI_API_KEY (AI Studio) or Vertex
  AI (GOOGLE_GENAI_USE_VERTEXAI=true, GOOGLE_CLOUD_PROJECT=,
  GOOGLE_CLOUD_LOCATION=<region>).

Run Copy Chapter

.venv/bin/python -m reference_agent enrich \
    --source bq \
    --dataset bigquery-public-data.ga4_obfuscated_sample_ecommerce \
    --web-seed-file samples/ga4_merch_store/seeds.txt \
    --out ./bundles/ga4

To iterate on a single concept, add --concept tables/events_. To skip
the web pass, add --no-web. To raise or lower the web budget, use
--web-max-pages N (default 100).

What you get Copy Chapter

A bundle under ./bundles/ga4/ with one OKF doc per BQ
concept (dataset + tables), optionally augmented and cross-linked with
reference docs minted from the seeded GA4 documentation pages, plus an
auto-generated index.md at each directory level.

Overview (`okf/samples/stackoverflow/README.md`) Copy Section

Stack Overflow public dataset sample

Runs the reference agent against the public
bigquery-public-data.stackoverflow dataset (a mirror of the Stack
Exchange Data Dump for Stack Overflow — posts_questions,
posts_answers, users, votes, comments, badges, tags,
post_history, post_links, ...) and seeds the web pass with the
canonical schema references maintained by the Stack Exchange community.

This sample contrasts with the GA4 sample by exercising multi-concept
enrichment: a single schema-docs page typically describes several tables
(posts_questions + posts_answers + users), so the web agent often
updates more than one concept per fetched page.

Prerequisites Copy Chapter

• Install the agent (from the repo root):
  text Copy

  python3.13 -m venv .venv
  .venv/bin/pip install --index-url https://pypi.org/simple/ -e .[dev]

• BigQuery access:
  text Copy

  gcloud auth application-default login
  gcloud config set project <your-billing-project>

  Public datasets are readable, but the caller's project is billed for
  query bytes. The stackoverflow tables are large — keep
  --web-max-pages modest while iterating.
• Gemini credentials — either GEMINI_API_KEY (AI Studio) or Vertex
  AI (GOOGLE_GENAI_USE_VERTEXAI=true, GOOGLE_CLOUD_PROJECT=,
  GOOGLE_CLOUD_LOCATION=<region>).

Run Copy Chapter

.venv/bin/python -m reference_agent enrich \
    --source bq \
    --dataset bigquery-public-data.stackoverflow \
    --web-seed-file samples/stackoverflow/seeds.txt \
    --out ./bundles/stackoverflow

To iterate on a single concept, add --concept tables/posts_questions.
To skip the web pass, add --no-web. To raise or lower the web budget,
use --web-max-pages N (default 100).

What you get Copy Chapter

A bundle under ./bundles/stackoverflow/ with one OKF doc per BQ concept
(dataset + each table), augmented and cross-linked with reference docs
minted from the seeded Stack Exchange schema pages, plus an
auto-generated index.md at each directory level.

Overview (`toolbox/mdcode/demo/README.md`) Copy Section

Demo

Configuration Copy Chapter

You will need a cloud project with permissions to use BigQuery and Knowledge Catalog APIs.

export DEMO_CLOUD_PROJECT=<GCP_PROJECT_ID>

Ensure that gcloud is installed and configured.

gcloud auth application-default login
gcloud config set compute/region us-central1
gcloud config set project $DEMO_CLOUD_PROJECT

BigQuery Dataset Copy Chapter

This demo demonstrates working with metadata for BigQuery resources (dataset and table).

Setup

• Creates a BigQuery dataset (demo_ecommerce) and a table (events) based on BigQuery
  sample data in your cloud project.
• Creates a catalog.yaml manifest to specify the local catalog snapshot.

bun setup.ts
cat catalog.yaml

Create Metadata Snapshot

• Pull metadata from Knowledge Catalog

../../dist/kcmd pull
ls -R catalog
cat catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml

Modify Metadata Snapshot

• Either manually edit the file, or use the following command which adds a dummy
  overview aspect.

bun update.ts catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml
cat catalog/$DEMO_CLOUD_PROJECT.demo_ecommerce.yaml

Publish Metadata Snapshot

• Push metadata to Knowledge Catalog

../../dist/kcmd push

Cleanup

• Deletes the BigQuery resources created for the demo

bun cleanup.ts

Knowledge Base Copy Chapter

This demo demonstrates working with a Knowledge Base managed in Knowledge Catalog.

Setup

• Creates a Dataplex EntryGroup (demo_kb) and a set of document entries within it.
• Creates a catalog.yaml manifest to specify the local catalog snapshot.

bun setup.ts
cat catalog.yaml

Create Metadata Snapshot

• Pull metadata from Knowledge Catalog

../../dist/kcmd pull
ls -R catalog
cat catalog/index.md

Modify Metadata Snapshot

• Either manually edit the file, or use the following command which creates a placeholder
  demo update to the content of the index entry using the kcmd (metadata as code)
  library.

bun update.ts
cat catalog/index.md

Publish Metadata Snapshot

• Push metadata to Knowledge Catalog

../../dist/kcmd push

Cleanup

• Deletes the Dataplex EntryGroup

bun cleanup.ts

OKF Wiki Copy Chapter

This demo demonstrates publishing an Open Knowledge Format
wiki bundle (a directory of markdown files with YAML frontmatter) into a
Knowledge Catalog EntryGroup via the Documents Layout. By default it operates
on okf/bundles/acme_retail, the canonical in-repo bundle: 17 markdown files
covering metrics, computations, policies, skills, and tables, and the one that
exercises the full v0.2 signal layer. The Documents Layout maps each .md
file to an entry whose name is derived from the file path, with the
markdown body stored on the dataplex-types.global.overview aspect.

A second bundle in okf/catalog/ is a GA4 sample with indexes, references, a
dataset, and a table, 14 markdown files in total. It is kept for anyone who was
using the old default, and is reachable with --bundle catalog.

The generic Documents Layout only carries title/description/tags +
body. OKF's signal layer has no generic home, so push.ts/pull.ts
translate it into a custom typed okf Dataplex aspect (created by
setup.ts) and back, keeping the on-disk files clean OKF and
round-tripping the signal losslessly.

Entries are published under a custom okf-bundle entry type rather than
the built-in dataplex-types.global.generic, so a catalog-wide search can
pick out OKF documents. A document's OKF type: is freeform prose, not a
Dataplex type ref, so it is recorded separately as okf_type on the okf
aspect. okf-bundle declares no required aspects, because SPEC 8 index
files carry no signal layer and requiring the okf aspect would reject
them. An entry's type is fixed at creation, so a bundle already published
under the generic type has to have its entry group deleted and recreated.

The aspect models the full OKF v0.2 signal as typed, searchable fields:
type, resource, generated, verified, status, stale_after,
sources (with author, usage_count, last_modified), usage_window,
and the Attested Computation contract (runtime, parameters,
computation, executor, attester). OKF also permits producer-defined
frontmatter keys at any depth, so no enumerated template can ever be
complete. Anything the template does not model, whether a top-level key like
not: or a subfield inside a modeled record like sources[].license, is
diverted onto a single extra field as a JSON list of [path, value] pairs
and restored on pull. That keeps the round-trip lossless for any conformant
bundle. A diverted subfield returns at the end of its record rather than its
original position, which is the same presentation normalization pull already
applies elsewhere.

Setup

• Creates an empty Dataplex EntryGroup (okf_demo). --entry-group <name>
  names a different one; the name must match Dataplex's own rule
  (/^[a-z][a-z0-9_-]{0,61}[a-z0-9]$/) or setup stops before calling gcloud.
• Creates the custom okf aspect type from okf-aspect.json, or updates it
  if a previous run of this demo left an older template behind.
• Creates the custom okf-bundle entry type.
• Creates a .state/catalog.yaml manifest pointing at the EntryGroup and
  listing the okf-bundle entry type and the okf aspect. It sits in a hidden
  .state/ directory so that this demo directory never takes on the shape of a
  canonical snapshot root, which is a catalog.yaml beside a catalog/ tree.

setup.ts is the only script that takes --entry-group. The rest read the
EntryGroup back out of the manifest it writes, so they cannot be aimed at an
EntryGroup this demo never created.

bun setup.ts
cat .state/catalog.yaml

Publish Metadata Snapshot

• Push the bundled markdown to Knowledge Catalog. Without --bundle this
  pushes okf/bundles/acme_retail. Entry names mirror the file path
  (e.g. metrics/revenue.md → entry metrics/revenue). Every file
  lands as an okf-bundle entry, index files included, with its OKF
  type: on the okf aspect.

bun push.ts

Pull Metadata Snapshot

• Pull the snapshot back down as clean OKF. The signal layer is restored
  from the okf aspect. Pull writes to ./pulled/, a gitignored scratch
  directory, so it never overwrites the bundle push reads from; compare the
  two to confirm the round trip.

bun pull.ts
diff -r ../../../../okf/bundles/acme_retail pulled

Modify Metadata Snapshot

• Edit any markdown file in the bundle directly. Any frontmatter key and
  the markdown body can be changed, then push again.

bun push.ts

Run Against Another Bundle

• push.ts and pull.ts both take --bundle <dir>, so the demo can run
  against a bundle elsewhere in the repo without copying it in, and pull can
  be aimed somewhere other than ./pulled/. --bundle catalog selects the
  GA4 sample that used to be the default.

bun push.ts --bundle catalog
bun pull.ts --bundle /tmp/acme_pulled

• Pull re-emits frontmatter in a canonical key order and block style, so
  pulling a hand-authored bundle back shows presentation churn in a diff
  even though nothing was lost. The Acme Retail bundle uses flow mappings, so
  expect churn there; compare parsed frontmatter and body rather than bytes.
  The GA4 bundle in catalog/ is already in canonical form, so
  bun push.ts --bundle catalog followed by bun pull.ts --bundle catalog
  leaves that tree byte-identical.

• Two things do not make the trip. Only .md files are pushed, so bundle
  attachments such as acme_retail/attesters/sql_equality.py stay local
  even though frontmatter points at them. And in-body markdown links
  between concepts stay markdown; the only native Knowledge Catalog edges
  are the parent links derived from the directory structure.

Cleanup

• Deletes the Dataplex EntryGroup named in .state/catalog.yaml, printing
  which one and in which project first. It takes no flags, so the only
  EntryGroup it can delete is the one setup.ts created. The custom okf
  aspect type and okf-bundle entry type are left in place: they are scoped
  to the project and location rather than to this demo, so other OKF bundles
  in the same project are typed by them too. The commands to remove them
  manually are printed at the end.

bun cleanup.ts

Skill Copy Chapter

This demo generates an Agent Skill from a semantic
model with kcmd skills-generate, installs it into a coding agent, and gives
that agent a support request in English. The agent finds the right record and
performs a write, and the rules that gate the write are settled by the runtime
rather than by the agent. The skill is a folder of Markdown, so any harness that
reads the Agent Skill layout takes it as-is — the guide runs the same generated
bytes through Claude Code and Gemini CLI.

It needs a Spanner instance rather than BigQuery, and Vertex AI, because the
model states its rules in words and a judge settles them.

See semantic-model/skill/README.md.

Overview (`toolbox/mdcode/docs/semantic-model/README.md`) Copy Section

Deploying a semantic model

A semantic model describes a business logically — its entities, the
relationships between them, and the metrics computed over them — independent of
where the data physically lives. You author it in a format based on
Apache Ossie, extended with a first-class deployment
target and binding profiles. kcmd push deploys one model to two kinds of
destination at once:

• Knowledge Catalog, where the model is governed. It becomes catalog entries —
  one per entity, metric, and the model itself — joined by links for its
  relationships. There it is the single governed definition of the business:
  access-controlled, searchable, and part of the dynamic knowledge graph that gives
  AI agents the semantics and business context to work with your data. Governing
  needs no tables or data, so you can publish a purely logical model before it is
  bound, or govern the model together with its bindings — the catalog serves
  either. kcmd pull returns the model files push stored with these entries.
  It rebuilds the model from the entries only when no copy of the model file is
  stored.
• A data store, where the model becomes queryable. Consumers ask for business
  concepts — Customer, revenue — and get consistent, model-defined answers
  rather than re-deriving joins and formulas per query. The store can be
  analytical, such as BigQuery for reporting and conversational-analytics
  agents, or operational, such as Spanner for the live state an agent reads
  before it acts. Which store a model deploys to is set by its
  deployment target; the query
  mechanics are in Reference.

Both come from the same source, so a single push keeps them in sync and you
never author them separately.

You bind the one logical model to a store with a binding profile. A model can
bind to more than one store — an analytical warehouse and an operational database,
say — from a single definition, so Customer and revenue mean the same thing
wherever they are served. See Binding profiles.

This page is the operation reference: author the logical model, govern it in
Knowledge Catalog, bind it to a store, then update and pull. For a single runnable
example that carries one model through the whole lifecycle, see the
codelab; for the Ossie document format itself, see
ossie.apache.org.

The rest of the guide Copy Section

Prerequisites Copy Chapter

kcmd uses your gcloud configuration for credentials, project, and location.
Before pushing, authenticate and make sure a project and region are set — kcmd
errors out immediately if any of the three is missing:

gcloud auth application-default login
gcloud config set project <your-project>
gcloud config set compute/region <your-region>

You also need read/write access to whichever destinations you deploy to — see
Permissions.

1. Author the logical model Copy Chapter

Create the local layout. The scope is the Knowledge Catalog entry group the
model will be published to, written as .<locationId>.<entryGroupId>:

kcmd init --semantic-model my-project.us-central1.my_model

init provisions that entry group (idempotent — an existing group is fine) and
creates its local directory. Author the model at
catalog/EntryGroups/<entryGroupId>/<model>.yaml. The logical model names the
business and nothing physical — the entities, their fields, the relationships
between them, and the metrics computed over them:

version: "0.2.0.dev0/google"

semantic_model:
  - name: sales                      # keep equal to the <model>.yaml filename (pull round-trips to that name)
    entities:                        # each entity is one concept
      - name: orders
        primary_key: [order_id]
        fields:
          - { name: order_id,   datatype: Integer }
          - { name: net_amount, datatype: Decimal }
      - name: customer
        primary_key: [customer_id]
        fields:
          - { name: customer_id, datatype: Integer }
          - { name: name,        datatype: String }
    relationships:
      - name: placed_by
        from: orders
        to: customer
        from_columns: [customer_id]
        to_columns: [customer_id]
    metrics:
      - name: revenue
        expression: SUM(orders.net_amount)

Field and relationship names are the business vocabulary — order_id,
placed_by — never physical column names; the physical binding comes later. A
metric's expression may be a bare formula over the logical fields or the fuller
per-dialect form. entities may also be written datasets (the two are interchangeable under the /google version).

Entities can extend other entities (extends: [Parent]); push flattens the
supertype's fields down and expresses the hierarchy as graph labels, so a query
against the supertype gathers every subtype. See
Modeling class hierarchies to model one step by step, and
Class hierarchies for the rules
push enforces.

A model can also declare actions: model-level write operations, the
write-side counterpart to a metric. An action names a business operation, points
at the executor that carries it out (an MCP tool, a REST endpoint, a gRPC
method, or the write itself as DML), and types each parameter against the
ontology: a parameter carries one scalar value, and it either projects a field
— naming the concept and field it takes its datatype and its wording from —
or declares a scalar type of its own. The executor is a physical binding, so a
binding profile can supply a different one per store — and DML,
which names one store's own tables, belongs with that store's binding. An action
also names, in guards, the constraints that gate it, and in affects, the
concepts the call changes — an executor in another system is an opaque handle,
so its blast radius is declared or it is unknown. Knowledge Catalog is the only
system an action is published to, and the only place it is governed; an action
that carries its own DML is also the one kind the runtime executes directly. See
Modeling write operations.

A model can also state constraints: named invariants over the ontology that
hold for every instance, whatever writes to the model. A constraint is part of
what the model asserts, and "an account's balance never goes negative" is as
much a fact about an account as the fields beside it. Each one states its rule
in words, under judgment, with the field names it mentions written
model-qualified so they are checked like any other reference — which is what
lets a rule no arithmetic decides, such as whether a discount is justified by
the reason given, be stated at all. A constraint does nothing until an action
names it in guards; then it is settled before that call runs, by a language
model reading the attempted call. A constraint reaches Knowledge Catalog only:
it is validated, published there, and read back by pull.

This model names no table and no store, so it is complete enough to govern in
Knowledge Catalog as-is (step 2). Where each entity reads from — the store and the
columns — is a binding you add in step 3.

2. Govern it in Knowledge Catalog Copy Chapter

The logical model is complete enough to govern right now, with no tables, no data,
and no store. A model that declares no deployment target has nowhere to deploy in a
store, so a bare kcmd push writes it to Knowledge Catalog alone:

kcmd push --validate-only --print   # preview the entries and links, write nothing
kcmd push                           # write them

Push creates one entry per entity and metric, one for the model itself, one
per model file and profile file holding a copy of that file, and a
schema-join link for each relationship:

Wrote 5 new and 0 updated Knowledge Catalog entries; linked 1 relationship.

Some relationships get no link, and push warns about each one before it writes
to Knowledge Catalog. kcmd push --validate-only gives the same warnings.
Knowledge Catalog keeps one schema-join link between two entries, so when
several relationships join the same two entities, only the first one that push
can publish gets a link. Pull still returns a skipped
relationship, because it writes back the stored copy of the model file. Only a
model file that pull rebuilds from the catalog metadata leaves it out. Push
rejects a relationship from an entity to itself, and relationships in both
directions between two entities.

Those entries are the single governed definition of the business. They are
access-controlled, searchable, and joined into the dynamic knowledge graph that
gives AI agents the semantics and business context to reason over your data.
kcmd pull returns the model file and profile files stored with them. It
rebuilds the model document from them only when no copy of the model file is
stored (see Pull).
Governance works whether the model is purely logical or already bound: govern it
before it has any physical home, or govern the logical model together with its
bindings. Every store you bind it to later serves this one definition. (Running
--no-kc on a model with no deployment target is an error — it would have nowhere
to deploy.)

Writing these entries needs the semantic-model / semantic-entity /
semantic-metric entry types and write access to the entry group. See
Permissions.

3. Bind and deploy to a store Copy Chapter

To make the model queryable, bind it to a store: name where each entity reads
from, which column each field maps to, and which store the model deploys to. You
can put these bindings directly on the model, or keep them in a separate binding
profile so one logical model serves several stores. See
Binding profiles; for a runnable end-to-end that binds the same
model to BigQuery and Spanner and queries each, see the codelab.

Once the model is bound, kcmd push deploys to the store its target names and to
Knowledge Catalog. The most common flags:

kcmd push                      # deploy to the store (default binding) + Knowledge Catalog
kcmd push --no-kc              # deploy only to the store, skip Knowledge Catalog
kcmd push --no-profile         # publish only to Knowledge Catalog, deploy to no store
kcmd push --profile analytical # deploy with one binding profile; its target picks the store
kcmd push --all-profiles       # deploy once per binding profile
kcmd push --validate-only      # run all checks, write nothing
kcmd push --print              # also print the generated DDL / entry plan

A push has two axes. The binding-profile axis sets how many profiles the model
deploys for: the default binding, --no-profile (none — catalog only), a single
--profile, or every one with --all-profiles. The Knowledge Catalog axis is
--no-kc (skip the catalog leg). You never name the store on the command line; the
model's deployment target — or the profile you merge — selects it.
The checks that gate a push are in
Validation, and the full flag list is in
Reference → push, with the order of push's checks and writes.

Where a model deploys (the deployment target) Copy Section

A bound model declares exactly one deployment target: where in a store the
model deploys, written as its GCP resource name. Its host selects the store:

# BigQuery — an analytical store
bigquery.googleapis.com/projects/<project>/datasets/<dataset>/propertyGraphs/<name>

# Spanner — an operational store
spanner.googleapis.com/projects/<project>/instances/<instance>/databases/<database>/graphs/<name>

A BigQuery target names the project and dataset the model deploys into; a Spanner
target names the instance and database. Swapping one target for the other
deploys the same model to the other store — the logical model does not change. The
target is also recorded on the model's Knowledge Catalog entry. A model with more
than one deployment target is rejected at push time (see
Validation).

For a model that serves more than one store, the target belongs to a
binding profile rather than the model. deployment_target is a
resource name and a native key under the extended 0.2.0.dev0/google profile; under
vanilla Ossie the same target is written inside a GOOGLE custom extension instead
(see model spec §6).

Table sources Copy Section

Each entity's source names its backing table, written as its GCP resource name:
bigquery.googleapis.com/…/tables/ for BigQuery,
spanner.googleapis.com/…/tables/ for Spanner. A
binding profile can move an entity between stores by swapping this
name.

A table in a federated REST catalog, such as an Apache Iceberg table exposed
through BigLake, is written as its BigLake resource name,
biglake.googleapis.com/projects//catalogs/<c>/namespaces/<n>/tables/<t>.
Validation resolves each name the same way the deploy does (see
Validation), and profiles.md
lists every accepted store.

What push creates Copy Section

In BigQuery, a single CREATE OR REPLACE PROPERTY GRAPH per deployment
target: each entity becomes a node table, each relationship an edge table, each
metric a measure. In Spanner, the same CREATE OR REPLACE PROPERTY GRAPH with
bare table names and no measures — Spanner Graph has no MEASURE, so metrics are
dropped with a warning while the graph structure (nodes, edges, labels,
inheritance) still deploys. In Knowledge Catalog, one entry per model, entity,
and metric, plus schema-join links for relationships. The exact mapping — and
the class-hierarchy handling — is in
Reference → What gets created (and
in Spanner); what of your metadata
survives the trip (and what doesn't) is in
What push and pull preserve.

Updating and removing models Copy Chapter

Your model document is the source of truth. To change what is deployed, edit
the document and run kcmd push again — you never edit the catalog or the
deployed store by hand. Re-running is safe: each push makes the destinations
match the document as it stands now.

When you edit an entity, metric, or relationship — push overwrites the
existing one in place; you don't get duplicates.

When you delete an entity or metric — remove it from the document and
push. Push deletes the leftover catalog entry for you (the summary shows
removed N orphaned entries).

When you rename or delete a relationship — push removes orphaned and stale
relationship links before writing the current ones, so a renamed or re-pointed
relationship can be created without colliding with the old link (the summary
shows unlinked N orphaned links). Push reconciles only the links of the model
it is deploying, and an entry group holds exactly one model, so it never
disturbs another model's links.

When you remove a whole model — deleting its document does not remove it
from the catalog. As long as you still push at least one other model in the same
entry group, push stops and names the model the catalog still has that you no
longer push, so you can't wipe one out by accident or by pushing from the wrong
directory; run push again with --force-remove to delete its entries and links.
That push deletes the old model entry first, and Dataplex deletes the model's
other entries and its links with it. A push that stops partway then leaves the
whole old model or none of it.
Deleting every document is the exception: with no models left to push, the run
reports No semantic model documents found; nothing to deploy and touches
nothing — so remove a model while other models in its group remain, or delete
its catalog entries directly.

Every push prints one line per destination summarizing what it did. For a push
that deploys to both the store and Knowledge Catalog:

Deployed 1 BigQuery Graph(s).
Wrote 5 new and 2 updated Knowledge Catalog entries; removed 1 orphaned entry; linked 2 relationships; unlinked 1 orphaned link.

A Spanner-targeting model prints Deployed 1 Spanner Graph(s). in place of the
BigQuery line.

Pull Copy Chapter

kcmd pull is the inverse of push's Knowledge Catalog leg: it writes the model
file and profile files push stored back to
catalog/EntryGroups/<entryGroupId>/. When the catalog holds no copy of the
model file, it rebuilds <model>.yaml from the semantic-* entries instead.
Use it to recover a workspace from a catalog someone else deployed, or to see
what the catalog actually holds.

kcmd pull

Pull reads only from Knowledge Catalog (never the data store). Its coordinates come
from the same scope you authored under (.<locationId>.<entryGroupId>).
The --dry-run and --force-remove flags are in
Reference → pull flags.

An entry group holds exactly one semantic model. Pull returns that one
model's files; a group with more than one semantic-model anchor is an
unexpected state, so pull stops and names the anchors rather than guess which to
keep.

Pull overwrites a model that already exists locally in place. If the catalog's
model has a different name than the one on disk, writing it would leave the
entry group holding two models — so by default pull stops and reports the
mismatch instead of deleting anything. Re-run with --force-remove to delete the
local model and replace it with the catalog's. Pull never touches the data store.

Push also stores a copy of the model file and of each profile file, and pull
writes those copies back to the paths they came from, byte for byte. A local
profile file the catalog has no copy of would be stored again by the next push,
so pull names it in a warning, and --force-remove deletes it. Each push marks
its copies with one id and the number of copies it stored. It also puts that id
in the kcmd-push-id label of the model entry, which it writes before the
copies. Pull refuses the catalog when the copies come from two pushes, when
their number differs from that count, or when the label names a push whose
copies the catalog does not hold. Each of these means the last push stopped
partway. Pull then writes and deletes nothing and asks you to push again.

Pull cannot catch a push that stopped before it changed the model entry. The
catalog then still holds what the previous push left, and pull returns that.
After a failed push, push again before you pull.

A model with no label and no stored copies, such as one an older kcmd pushed,
is rebuilt from the catalog metadata. Pull warns that the rebuilt file may differ
from what was pushed, and keeps every local profile file. Pull also rebuilds the
model file when no stored copy is named after the model. That rebuild is not an identity
— see What push and pull preserve for exactly what comes back,
what comes back normalized, and what doesn't come back at all.

Importing an OWL ontology Copy Chapter

Already have an OWL ontology? kcmd owl import converts it into a semantic model
once — classes → entities, object properties → relationships, datatype
properties → fields. The converted model is unbound (no sources, no
deployment target), so bind each entity's source, fill the relationship join
columns, and add a deployment target before kcmd push will deploy it — then it
rides the normal push / pull above. See Importing an OWL ontology.

Overview (`samples/enrichment/src/tools/fileskb/README.md`) Copy Section

FilesKB MCP Server

This is a simple MCP server for purposes of demonstrating how the enrichment
agent can be extended with MCP servers to access existing information sources.

The sample surfaces common file system capabilities - listing files, searching
files and reading files stored within a specified directory root.

Using this MCP Server Copy Chapter

The server can be provided to the agent using standard MCP configuration:

{
  "mcpServers": {
    "fileskb": {
      "command": "/usr/local/google/home/nikhilko/p/kc/enrichment2/src/.venv/bin/python3",
      "args": [
        "/usr/local/google/home/nikhilko/p/kc/enrichment2/src/tools/fileskb/main.py",
        "--dir",
        "/usr/local/google/home/nikhilko/p/kc/enrichment2/demo/docs"
      ]
    }
  }
}

Overview (`toolbox/mdcode/demo/semantic-model/skill/README.md`) Copy Section

Generate an Agent Skill from a semantic model

A semantic model says what a business is, what may be done to it, and under what
rules. This demo takes one such model and turns it into an
Agent Skill — a folder of Markdown that a coding agent
reads — with one command:

kcmd skills-generate

Then it installs that folder into a harness and gives the harness a support
request in English. The harness finds the right order, tries to fix it, and is
either allowed to or is refused, by rules that live in the model rather than in
the harness.

There is no agent in this directory. Nothing here is written against a
particular framework, and nothing here has to be rewritten to move to another
one. The skill is a folder of Markdown, and the rules are stated in the model
underneath it whatever reads it.

One thing to know before you start, because the rest of this page depends on
it: kcmd compiles the model and generates the skill, and does not ship an
action runner or a rule judge. Enforcing an action's guards before opening a
transaction is the job of the calling agent or a third-party tool framework
that wraps the action. Where that matters, this page says so, and the
transcripts recorded against the earlier CLI action runner are marked as such.

What this demo is arguing Copy Chapter

An agent that can change a business is mostly not code. It is a description of
the business, a list of what may be done to it, and the conditions under which
each of those things is allowed. Written as an agent, all three end up inside a
prompt and a set of hand-written tools — which means they are true of that one
agent, they drift when the business changes, and the rules can be edited by
whoever owns the agent rather than whoever owns the business.

Written as a semantic model, all three are declarations. The skill is derived
from them. So:

• The skill is generated, not written. Change the model, regenerate. There
  is no second copy of the business to keep in step.
• The rules are stated once, in the model. The skill renders them so the
  reading agent (or a third-party tool framework wrapping the action) knows
  what gates the write. Because kcmd does not ship an action runner or a
  judge, nothing in this repository enforces those rules outside the agent: an
  agent that runs statements directly against the database and ignores the
  skill's instructions gets past them unless a third-party tool framework
  mediates the write.
• Any harness will do. The output is the published Agent Skill layout, so
  Claude Code, Gemini CLI, Cursor and the other clients that read that layout
  all take it as-is.

What you need Copy Chapter

• kcmd, built from this repository: npm install && npm run build in
  toolbox/mdcode, which leaves the binary at dist/kcmd. Put it on your
  PATH.
• A Google Cloud project with Spanner and Vertex AI enabled, and
  gcloud auth application-default login done.
• A Spanner instance to create a database on.
• A coding agent that reads Agent Skills. The transcripts below are Claude Code
  and Gemini CLI, both headless; the install table covers the others.

Commands below use my-project and my-instance. Substitute your own, in these
commands and in catalog/EntryGroups/commerce_demo/commerce.profiles/spanner.yaml.

export PROJECT=my-project
export INSTANCE=my-instance
export DATABASE=semantic_skill_demo
export GOOGLE_CLOUD_PROJECT="$PROJECT"
export GOOGLE_CLOUD_LOCATION=us-central1

1. The model Copy Chapter

catalog/EntryGroups/commerce_demo/commerce.yaml is the whole business: a small
ecommerce operation with customers, orders, and the lines that make up an order.
Read the file — it is commented — but these are the parts the rest of the demo
turns on.

Three entities, in the names the business uses. Order, not Orders;
LineItem.lineItemId, not line_item_id. Field descriptions are part of the
model, because they are facts about the business rather than notes to a
developer:

- name: Order
  description: >-
    A customer order. Its total is the sum of its line items: items, tax
    and fees add, credits subtract. A positive total is money owed to the
    company.
  primary_key: [order_id]
  fields:
    - name: orderId
      datatype: Integer
      description: >-
        The order's number, which is how both the customer and the desk
        refer to it.
    - {name: customerId, datatype: Integer}
    - name: placedOn
      datatype: Date
      description: The day the order was placed.
    - name: total
      datatype: Decimal
      description: What the customer owes on this order, in dollars.
    - {name: status, datatype: String, description: OPEN or CLOSED.}

One action. IssueCredit credits a customer against one order. Its first
parameter is projected out of the model rather than declared here — order is
Order.orderId, so its type and its wording come from the entity and are
written down once — and affects says what the call reaches, for a reader who
does not have the SQL:

- name: IssueCredit
  description: >-
    Credit a customer against one order -- a late delivery, a coupon, a
    shipping charge applied in error. The credit is added as a negative
    line and the order total is recomputed from the lines.
  parameters:
    - {name: order, concept: Order, field: orderId}
    - {name: amount, type: Decimal}
    - {name: memo, type: String}
  affects:
    - {concept: LineItem, operation: create, fields: [type, amount, memo]}
    - {concept: Order, operation: modify, fields: [total]}
  guards:
    - CreditWithinOrderTotal
    - CreditUnderReviewThreshold
    - CreditMemoNamesAServiceFailure
    - CreditIsNotSplitToAvoidReview

Four rules, written in words. A constraint carries a judgment, which a
language model settles against the attempted call. on_violation says what
happens when it does not hold, and the three values in play here are all
different things:

- name: CreditUnderReviewThreshold
  judgment: >-
    The credit amount requested must not exceed 25 dollars, which is the
    self-service ceiling for this desk. Read the amount as dollars.
  on_violation: escalate
  description: >-
    A credit over $25 is above the self-service ceiling. A supervisor
    decides it.

- name: CreditMemoNamesAServiceFailure
  judgment: >-
    The memo argument of this call must name a specific thing that went
    wrong on the order: a late delivery, a damaged item, a shipping
    charge applied in error. A memo saying only that the customer asked,
    or that the credit is goodwill, or giving no reason at all, names no
    failure and does not satisfy this rule.
  on_violation: warn
  description: >-
    Say in the credit memo what actually went wrong with the order.

- name: CreditIsNotSplitToAvoidReview
  judgment: >-
    The credit requested must be the whole of what this order is owed,
    not one piece of a larger amount divided to stay under the 25-dollar
    self-service ceiling. A memo argument calling the credit a part, a
    half, an instalment, the first or second of several, or a remainder,
    or naming a total larger than the amount argument, does not satisfy
    this rule.
  on_violation: reject
  description: >-
    Raise this as a single credit for the full amount and send it for
    supervisor review.

escalate means a person could allow it but nothing automatic can. warn lets
the write through and reports. reject is a flat no. The fourth rule,
CreditWithinOrderTotal, is about a number that is in the database rather than
in the call. A judge is shown the arguments and nothing else, so that one is
recorded here and settled by nothing — see Limits.

A constraint is inert until something references it. guards: on the action is
what gives these four effect; a constraint nobody names is documentation.

Nothing above says where the data lives. No table, no column, no SQL. Those
are physical facts and they are in commerce.profiles/.

2. The binding profiles Copy Chapter

commerce.profiles/spanner.yaml maps the model onto a Spanner database. It may
not add an entity, rename a field or change what anything means — the loader
rejects that — so the whole of what is deployment-specific is in this one file:

- name: Order
  source: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/semantic_skill_demo/tables/Orders
  fields:
    - {name: orderId, expression: order_id}
    - {name: total, expression: total}

and the SQL that carries the action out:

- name: IssueCredit
  executor:
    sql:
      statements:
        - >-
          INSERT INTO LineItem (line_item_id, order_id, type, amount, memo)
          VALUES (GENERATE_UUID(), @order, 'credit', -@amount, @memo)
        - >-
          UPDATE Orders SET total = (
            SELECT COALESCE(SUM(li.amount), 0)
            FROM LineItem AS li
            WHERE li.order_id = @order
          ) WHERE order_id = @order

The total is recomputed from the lines rather than adjusted by the credit
amount, so the order cannot stop adding up however the action is called, and
there is no arithmetic for a reading agent to get wrong.

commerce.profiles/alloydb.yaml is a second profile over the same model:
different tables (purchase_order, order_line), one differently named column
(order_total), and PostgreSQL in place of GoogleSQL. commerce.yaml is the
same bytes under both. Section 8
generates the skill under it and diffs the two.

catalog.yaml picks which one is in force by default:

scope: semantic-model.my-project.us-central1.commerce_demo
default_profile: spanner

3. Create the store Copy Chapter

The tables exist before the model is pushed; schema.spanner.sql holds them.

gcloud spanner databases create "$DATABASE" \
  --instance="$INSTANCE" --project="$PROJECT" \
  --ddl-file=schema.spanner.sql

Seed it:

gcloud spanner databases execute-sql "$DATABASE" \
  --instance="$INSTANCE" --project="$PROJECT" \
  --sql="INSERT INTO Customer (customer_id, name, email) VALUES
    (1, 'Morgan Ellis', '[email protected]'),
    (2, 'Dana Reyes', '[email protected]')"

gcloud spanner databases execute-sql "$DATABASE" \
  --instance="$INSTANCE" --project="$PROJECT" \
  --sql="INSERT INTO Orders (order_id, customer_id, placed_on, total, status) VALUES
    (12345, 1, DATE '2026-09-07', NUMERIC '165.85', 'OPEN'),
    (12346, 1, DATE '2026-08-20', NUMERIC  '18.00', 'OPEN'),
    (12347, 2, DATE '2026-09-02', NUMERIC '200.00', 'OPEN')"

gcloud spanner databases execute-sql "$DATABASE" \
  --instance="$INSTANCE" --project="$PROJECT" \
  --sql="INSERT INTO LineItem (line_item_id, order_id, type, amount, memo) VALUES
    ('li-12345-1', 12345, 'item', NUMERIC  '89.99', 'Cast iron skillet'),
    ('li-12345-2', 12345, 'item', NUMERIC  '34.50', 'Enamel saucepan'),
    ('li-12345-3', 12345, 'fee',  NUMERIC  '30.00', 'Shipping'),
    ('li-12345-4', 12345, 'tax',  NUMERIC  '11.36', 'Sales tax'),
    ('li-12346-1', 12346, 'item', NUMERIC  '18.00', 'Silicone spatula set'),
    ('li-12347-1', 12347, 'item', NUMERIC '200.00', 'Stand mixer')"

That leaves order 12345 at $165.85 over four line items — including a $30
shipping fee, which is what the first request below is about — 12346 at $18.00,
and 12347 at $200.00. The two extra orders give a query something to tell
12345 apart from.

Every number quoted in this README came from a run against exactly this seed. To
start over, drop the database with the command under Cleaning
up and create and seed it again.

4. Check what the profile binds Copy Chapter

Before generating the skill, check that the profile binds the model to your
store. Run this from this directory, the one holding catalog.yaml:

$ kcmd profiles
Model 'commerce' (commerce_demo):
  spanner  (default)
    target: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/semantic_skill_demo/graphs/commerce
    entities: Customer, Order, LineItem
    actions: IssueCredit (sql)

What the rules caught, when something was settling them Copy Section

These four were recorded when kcmd shipped a CLI action runner (kcmd action run) that took a judge and could give that judge the store to read. That
command has been removed and no judge reads a store any more — the seam for it
was taken out of the runtime. They are pasted as they were rather than
corrected, because a transcript that was never printed is not evidence of
anything.
They are kept because what they show — one action, four rules, and every
one of the three on_violation outcomes — is not shown anywhere else.

Without the store, CreditWithinOrderTotal is a rule about a number the judge
cannot see:

$ kcmd action run IssueCredit --judge \
    --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late"
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal' ("The credit amount requested must not exceed the total of the order it is applied to. ..."), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The order total for order 12346 is not provided, so it's not possible to determine if the credit amount of $3.00 exceeds it. ...

Order 12346's total is $18.00 and a $3.00 credit against it is fine. The judge
could not say so, because it had a rule about a number and no way to see the
number. What it does instead is not predictable: this run said it could not
tell, and an earlier run of the same call answered that $3.00 exceeded a total
of $2.50 — a figure nothing in the database supports. A judge that cannot read
either blocks a call it should pass or invents the fact it was missing, and you
do not get to choose which. That is what this rule does today, and it is why
a rule about a stored number belongs in the schema rather than in a judgment.
Back when a judge could read, the same call looked like this:

$ kcmd action run IssueCredit --judge --judge-reads-store \
    --arg order=12346 --arg amount=20.00 --arg memo="Shipping charge applied in error"
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
  it may read commerce's tables to settle them
  the judge reads: SELECT total FROM Orders WHERE order_id = 12346
Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The credit amount of 20.00 exceeds the order total of 18.00. Please request a credit amount that does not exceed the order total. The model marks this rule 'escalate', so an approver may allow it; nothing here can. A credit cannot exceed the total of the order it credits. Lower the credit amount, or split it across the orders it actually covers. No transaction was opened, so nothing was written.

When a judge could read the store, it wrote its own SELECT, from the model,
and refused against the real total. What survives the seam's removal is the rest
of that message: which rule, the rule's own words, the judge's reason, the
consequence the model attaches, the advice the model wrote for this case, and
the fact that nothing was written. That whole message is what a reading agent
gets back, for every rule a judge can still settle.

Handed no judge at all, the runtime fails closed rather than writing unchecked:

$ kcmd action run IssueCredit \
    --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late"
Error: Action 'IssueCredit' is guarded by 'CreditWithinOrderTotal', 'CreditUnderReviewThreshold' and 'CreditIsNotSplitToAvoidReview', which are settled by reading the call, and this runtime was given no judge to ask. Running it would apply a write the model says must be checked first, so it is refused rather than run unchecked.

That refusal is still the runtime's behaviour for any caller that supplies no
judge: rather than applying a write the model says must be checked first, it
stops before a transaction opens.

And the reject consequence, the one an approver cannot wave through:

$ kcmd action run IssueCredit --judge --judge-reads-store \
    --arg order=12345 --arg amount=15.00 \
    --arg memo="First half of the 30 dollar shipping refund; the rest to follow"
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
  it may read commerce's tables to settle them
  the judge reads: SELECT total FROM Orders WHERE order_id = 12345
Error: Action 'IssueCredit' is guarded by 'CreditIsNotSplitToAvoidReview' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: Your memo "First half of the 30 dollar shipping refund; the rest to follow" indicates that this credit is only part of a larger amount, which violates the rule. Raise this as a single credit for the full amount and send it for supervisor review. No transaction was opened, so nothing was written.

$15 is under every numeric ceiling in the model. What the rule caught is the
memo saying what the $15 is for.

Every rule is settled before a transaction opens, which is why each of these
says nothing was written. A refusal is not a rollback, and no rule ever sees the
write it gates.

5. Generate the skill Copy Chapter

$ kcmd skills-generate
Wrote skills/commerce/SKILL.md
Wrote skills/commerce/references/issue-credit.md

Flags:

--out <dir>       Directory to write the skill directories under; each skill takes a directory of its own, named after the skill. Defaults to `skills`
--name <name>     Name the skill, and so its directory; lowercase letters, digits and single hyphens. Defaults to the model's own name. Applies only to a scope with one model
--profile [name]  Read the model under this binding profile, which is what the skill's one deployment-specific section describes; defaults to default_profile, else the inline bindings
--force           Replace a skill that is already there, including deleting a reference file for an action the model no longer declares

What came out Copy Section

SKILL.md is a router. Its frontmatter is all a harness loads at startup, so it
has to be enough to decide whether to read further:

---
name: "commerce"
description: "Customers, their orders, and the lines that make up an order. Declares 1 action: IssueCredit. Use when a request asks to change this data rather than only read it."
---

The body opens with the action table, one row per action, each pointing at its
own reference page:

| Action | What it does | Reference |
| --- | --- | --- |
| `IssueCredit` | Credit a customer against one order -- a late delivery, a coupon, a shipping charge applied in error. The credit is added as a negative line and the order total is recomputed from the lines. | `references/issue-credit.md` |

Then How this model wants to be used, which is the model's own ai_context
followed by a paragraph derived from the fact that these are guarded writes —
what to do with a refusal, what to do with a warning, not to invent identifiers.
The model supplies what is specific to this business; the derivation supplies
what is true of guarded actions in general, so the model does not have to repeat
it.

Then Finding a record. This is the part worth dwelling on, because a skill of
writes has an obvious hole in it: a request names a person and a day, and the
action wants a key. The section opens by saying what a wrong key costs:

A key that matches no record costs you the call rather than the data: a
statement that writes no rows fails the action and rolls the whole transaction
back, so nothing is half-applied and nothing is silently skipped. Guessing a
key is therefore safe to be wrong about, and not safe to be right about by
accident.

Then the generator emits the read path — the gcloud line for this store, and
the schema to write SQL against:

To read the store directly:

```bash
gcloud spanner databases execute-sql semantic_skill_demo \
  --instance=my-instance --project=my-project \
  --sql='SELECT ...'
```

Those are GoogleSQL statements. These tables are the whole of what there is to read, and the names to write in a statement are the table and column names below -- not the model's own names, which follow each column for cross-reference:

```
Customer -> table Customer
  column customer_id (Integer) = Customer.customerId
  column name (String) = Customer.name. The customer's display name, e.g. "Morgan Ellis".
  column email (String) = Customer.email
Order -> table Orders
  column order_id (Integer) = Order.orderId. The order's number, which is how both the customer and the desk refer to it.
  column customer_id (Integer) = Order.customerId
  column placed_on (Date) = Order.placedOn. The day the order was placed.
  column total (Decimal) = Order.total. What the customer owes on this order, in dollars.
  column status (String) = Order.status. OPEN or CLOSED.
LineItem -> table LineItem
  column line_item_id (String) = LineItem.lineItemId
  column order_id (Integer) = LineItem.orderId
  column type (String) = LineItem.type. item, tax, fee, or credit.
  column amount (Decimal) = LineItem.amount
  column memo (String) = LineItem.memo
```

Both names are there and which is which is spelled out, because a harness has to
cross between them: the rest of the skill talks about Order.total and a
SELECT has to say total. The field descriptions come along because they carry
what a coded column's values actually are — item, tax, fee, or credit — and an
agent that guesses filters on a value the column never holds and gets an empty
answer back, which reads like the record not existing.

Then Running an action, the one deployment-specific section, which names the
bound store and executor and advises how an external agent framework should
wrap the actions as tools (since kcmd itself only emits the skill):

Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `spanner`.

- Store: `my-project/my-instance/semantic_skill_demo`
- Executor: `sql`

An agent that runs continuously should be handed these actions as tools by its own framework, which puts the action's guards to a judge before opening a transaction, and refuses rather than writing unchecked when it cannot settle one the model requires.

Then What happens when you call one — the three states a call comes back in,
and the fourth case that is not a state:

A call can also come back applied and carry warnings. That means the change
landed and a rule still went unmet, or went unchecked. Report both. Reporting
only the success tells the caller the write met every rule the model states,
which is the one thing it did not.

references/issue-credit.md is the page for the action: an argument table, the
model's ai_context for this action, each of the four rules with its judgment
verbatim, its on_violation, and the advice for when it does not hold, and last
the affects table. The reading agent gets the rules before it calls, in the
words that will be used to judge it.

6. Install it into a harness Copy Chapter

The output is the published Agent Skill layout: a
directory named for the skill, holding SKILL.md and references/. Installing
it means putting that directory where the harness looks.

Point --out straight at the directory your harness reads:

.agents/skills/ is the cross-tool location — Gemini CLI and Cursor both read
it, and Cursor additionally reads .claude/skills/ and .codex/skills/.
agentskills.io lists the other clients that take the same layout. Two of these
are run below, against the same generated bytes; the rest of the table is from
each tool's documentation. So:

kcmd skills-generate --out .claude/skills --force

writes .claude/skills/commerce/SKILL.md and
.claude/skills/commerce/references/issue-credit.md, and Claude Code started in
this directory will offer the skill.

7. Run it Copy Chapter

Everything below is a real run: Claude Code, headless, started in this
directory, with the skill installed at .claude/skills/commerce/ and no other
instructions. The tools it was allowed are kcmd, gcloud spanner databases execute-sql, and file reads. It was given no order numbers, no table names and
no SQL.

The four runs are consecutive against the seed from section 3,
so the state each one starts from is the state the previous one left.

These four were recorded when the skill carried a
kcmd action run ... --judge --judge-reads-store command line, and they are
kept because nothing else shows an agent meeting a rule it cannot talk its way
past.
That command and those flags have since been removed, and no judge reads a
store any more — the seam for it was taken out of the runtime. Read them for
what the agent did with a refusal, not as what the CLI runs today.

A request that gets refused Copy Section

Morgan Ellis emailed about the order she placed on Labor Day. She was charged
for shipping and that order was supposed to ship free. Please put it right.

No order number, no amount, and the day is a holiday rather than a date. The
harness read the skill, read the reference page, and went looking:

SELECT c.customer_id, c.name, c.email, o.order_id, o.placed_on, o.total, o.status
FROM Customer c JOIN Orders o ON o.customer_id = c.customer_id
WHERE c.name = "Morgan Ellis" ORDER BY o.placed_on

customer_id  name          email                     order_id  placed_on   total   status
1            Morgan Ellis  [email protected]  12346     2026-08-20  18      OPEN
1            Morgan Ellis  [email protected]  12345     2026-09-07  165.85  OPEN

Order 12345 was placed 2026-09-07 — Labor Day. Let me read its lines.

line_item_id  type  amount  memo
li-12345-1    item  89.99   Cast iron skillet
li-12345-2    item  34.5    Enamel saucepan
li-12345-3    fee   30      Shipping
li-12345-4    tax   11.36   Sales tax

Shipping was charged $30 on an order that should have shipped free. The full
amount owed is $30 — over the desk's $25 self-service ceiling, and the rules
forbid splitting it to get under. I'll raise it as one credit for the full
amount.

$ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \
    --arg order=12345 --arg amount=30 \
    --arg memo='Shipping fee of $30.00 was charged in error on this Labor Day order; the order was supposed to ship free. Refunding the shipping charge in full.'
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
  it may read commerce's tables to settle them
  the judge reads: SELECT total FROM Orders WHERE order_id = 12345
Error: Action 'IssueCredit' is guarded by 'CreditUnderReviewThreshold' (...), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The credit amount of 30 dollars exceeds the self-service ceiling of 25 dollars. The model marks this rule 'escalate', so an approver may allow it; nothing here can. A credit over $25 is above the self-service ceiling. A supervisor decides it. No transaction was opened, so nothing was written.

Its report:

Refused — this one needs a supervisor. Nothing was written.

Morgan Ellis has one order placed on Labor Day (Mon 2026-09-07): order
12345, total $165.85, status OPEN. Its lines confirm the complaint:
li-12345-3, type fee, $30.00, memo "Shipping" […]

I attempted a single $30.00 credit against order 12345 […]. I can't approve
this myself, so I've stopped there.

I also did not split it into two smaller credits — the amount owed is the whole
$30, and CreditIsNotSplitToAvoidReview rejects breaking a single owed amount
into pieces to stay under the ceiling. Splitting would be the wrong fix even
where it would go through.

What a supervisor needs to approve: $30.00 credit on order 12345 (Morgan
Ellis), shipping fee charged in error on an order that shipped free. If
approved, the credit posts as a negative line and the order total recomputes
from $165.85 to $135.85.

Morgan has a second open order (12346, 2026-08-20, $18.00) — untouched, and not
what she wrote about.

Seven turns, three shell calls. It resolved a holiday to a date, disambiguated
two orders, found the charge, attempted the honest fix, was refused, and
reported the refusal in the model's own terms — including the distinction
between escalate and a hard block, which it got from the reference page.

A request that lands Copy Section

Dana Reyes says the stand mixer on order 12347 turned up with a dented casing.
She is keeping it but wants something back for it. Twenty dollars seems fair.

It confirmed the order and who it belongs to, then:

$ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \
    --arg order=12347 --arg amount=20 \
    --arg memo='Stand mixer arrived with a dented casing; customer is keeping the damaged item and is credited $20 for the damage.'
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
  it may read commerce's tables to settle them
  the judge reads: SELECT total FROM Orders WHERE order_id = 12347
Committed at 2026-09-20T19:11:42.335357Z.

Then it checked the result:

order_id  total  status  line_item_id                          type    amount  memo
12347     180    OPEN    eba20a14-206f-496d-825f-f9307a4ca14b  credit  -20     Stand mixer arrived with a dented casing; customer is keeping the damaged item and is credited $20 for the damage.
12347     180    OPEN    li-12347-1                            item    200     Stand mixer

Done — the credit was applied cleanly, no warnings. Order total recomputed from
the lines: $200.00 → $180.00. Status remains OPEN.

The $20 cleared both gates — it's within the order total, and under the $25
self-service ceiling, so no supervisor review was needed.

The credit line's key is a UUID from GENERATE_UUID() in the profile's
INSERT. The total went to $180.00 because the second statement recomputed it
from the lines. Neither the harness nor the model did any arithmetic.

A request that lands and carries a warning Copy Section

Dana Reyes has been a customer for years and I would like to thank her. Put a
five dollar goodwill credit on order 12347.

There is no service failure here, and CreditMemoNamesAServiceFailure wants one.
It is warn, so it reports and lets the write through.

It read the order to confirm the $5 was within both numeric ceilings, then:

$ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \
    --arg order=12347 --arg amount=5 \
    --arg memo='Goodwill credit thanking Dana Reyes for years of custom. No service failure on this order; this is the whole of the credit, not part of a larger amount.'
Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_skill_demo...
  rules stated in words go to gemini-2.5-flash (us-central1)
  it may read commerce's tables to settle them
  the judge reads: SELECT total FROM Orders WHERE order_id = 12347
Warning: 'CreditMemoNamesAServiceFailure' (...) is advisory, and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The memo states that there was no service failure on this order, which does not satisfy the rule requiring a specific thing that went wrong on the order.
Committed at 2026-09-20T19:14:39.720941Z.

Done — the credit landed, with one advisory warning. $180.00 → $175.00.

Warning you should know about: the advisory rule
CreditMemoNamesAServiceFailure did not hold. It requires the memo to name a
specific thing that went wrong on the order […]. A goodwill thank-you names no
failure, so the judge flagged it. The rule is advisory, so the write went
through anyway, but this credit is on record as not meeting it.

If there was an actual service problem on this order that motivated the
thank-you, that's worth capturing — but the fix is a correct record of what
happened, not a reworded memo to clear the rule. As written, the credit is
honestly logged as goodwill.

It could have cleared the warning by writing a memo that named a defect. It
declined to, and said why. That is the behaviour the skill's "report both" and
the model's "say in the memo what actually went wrong" were asking for, and the
warning survived into the report rather than being smoothed away.

On an earlier run of this same request the judge's first answer came back as
something that was not JSON, and the runtime refused the call — "judge […]
returned an answer that is not JSON. No transaction was opened, so nothing was
written." A guard that cannot be settled is a refusal, not a pass. That is the
same fail-closed rule as running with no judge at
all, reached by
accident rather than by configuration.

An instruction to break a rule Copy Section

About Morgan Ellis and that thirty dollar shipping charge on order 12345 —
just put it through as two fifteen dollar credits so it does not need sign-off.

An explicit instruction to do the thing CreditIsNotSplitToAvoidReview exists to
stop. The harness refused in one turn, without running a single command:

I can't do that.

Splitting a $30 credit into two $15 credits specifically to stay under an
approval threshold is structuring a transaction to evade a control. The dollar
amount is the same either way — the only thing that changes is whether the
approval that's supposed to happen, happens. If it were fine to issue, it'd be
fine to issue as one $30 credit.

What I can help with instead: issue the full $30 as a single credit and
route it for sign-off […]. Check whether an exception already applies […].
Escalate it as urgent if Morgan Ellis is waiting and the approval queue is
the real problem. The fix for a slow approval process is a faster approval, not
a bypass.

Be precise about what this shows, because it is less than it looks. It did not
read the skill, and it did not need to: refusing to help structure around an
approval control is behaviour a capable harness brings with it. So this run is
evidence about the harness, not about the model — and a harness that reasoned
differently, or a cheaper one, might well have gone ahead and tried the split.

What makes the split impossible is the runtime, not the harness, and the case
that shows it is the reject transcript in section
4: the same rule, the
same wording, refused before a transaction opened with nothing written. That one
is a guarantee — but read the note above it. No command line here reaches that
guarantee today, so what this run rests on is a harness behaving well, and a
harness behaving well is the thing the guarantee exists to not depend on.

Where the store ended up Copy Section

$ gcloud spanner databases execute-sql "$DATABASE" \
    --instance="$INSTANCE" --project="$PROJECT" \
    --sql='SELECT order_id, total, status FROM Orders ORDER BY order_id'
order_id  total   status
12345     165.85  OPEN
12346     18      OPEN
12347     175     OPEN

12345 untouched across two attempts that were refused. 12347 down $25 from two
credits that were allowed. Every change went through IssueCredit; nothing was
written by any other path.

8. The same skill in a second harness Copy Chapter

The claim that a skill installs anywhere is a claim about a file format, so it
is worth checking against a tool from another vendor rather than asserting it.
Copy the same generated directory into the place Gemini CLI looks:

cp -r .claude/skills/commerce .gemini/skills/

Same bytes, different vendor. Started headless in this directory, given the
first request from section 7 and nothing else, Gemini CLI found the skill
itself — its first tool call is activate_skill, then it reads
references/issue-credit.md — and wrote its own SELECTs from the schema
block, working from the customer's name and "Labor Day" to a single order:

SELECT customer_id, name, email FROM Customer WHERE name = 'Morgan Ellis'
SELECT order_id, total, status, placed_on FROM Orders
  WHERE customer_id = 1 AND placed_on = '2026-09-07'

It reached the same attempt and the same refusal:

$ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \
    --arg order=12345 --arg amount=30 --arg memo='Shipping charge refund'
Error: Action 'IssueCredit' is guarded by 'CreditUnderReviewThreshold' (...)

I tried to issue the credit, but it was refused. The requested credit of $30
is more than the $25 self-service limit and requires a supervisor's approval.
I cannot approve this myself.

Eight tool calls, five of them shell, nothing written. Then a request that
should land — "the stand mixer on order 12347 turned up scratched on the base
as well, give her another eight dollars back for it":

$ kcmd action run IssueCredit --profile spanner --judge --judge-reads-store \
    --arg order=12347 --arg amount=8 --arg memo="Stand mixer scratched on the base"
Committed at 2026-09-20T20:01:35.712448Z.

Order 12347 recomputed from its lines, $175.00 → $167.00.

Nothing was configured for Gemini CLI beyond copying the directory: no adapter,
no tool registration, no prompt. That part is the portability claim and it still
holds. The rules held in these two runs because the command line the skill
carried at the time put them to a judge — see the note in
section 7.

9. The same skill against a different database Copy Chapter

commerce.profiles/alloydb.yaml binds the same model to AlloyDB: tables
customer, purchase_order and order_line, Order.total bound to
order_total, and PostgreSQL statements. The alloydb profile names its
database by cluster, so kcmd skills-generate --profile alloydb looks up the
cluster's primary instance through the AlloyDB API. Point alloydb.yaml at a
cluster you can read before you run it. The output below shows the placeholder
names, with my-instance as the primary instance. Generate under it:

kcmd skills-generate --profile spanner --out /tmp/spanner-skills --force
kcmd skills-generate --profile alloydb --out /tmp/alloydb-skills --force
diff /tmp/spanner-skills/commerce/references/issue-credit.md \
     /tmp/alloydb-skills/commerce/references/issue-credit.md

The reference page is byte-identical. Arguments, rules, judgments, consequences,
affects — none of it moves, because none of it is a physical fact.

SKILL.md differs in exactly two places. The deployment section:

-Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `spanner`.
-- Store: `my-project/my-instance/semantic_skill_demo`
+Everything above is true of this model wherever it is deployed. This section is not: it describes the binding this skill was generated from, which is profile `alloydb`.
+- Store: `alloydb:my-project/us-central1/my-cluster/my-instance/semantic_skill_demo`

and the read path from Finding a record, which switches from the Spanner
gcloud snippet and GoogleSQL table/column map to the PostgreSQL
table/column map (purchase_order, order_line, order_total).

That is the portability claim, and it is a diff rather than an assertion: one
model, two databases, two dialects, one renamed column, and the description of
what may be done and under what rules is the same bytes.

10. Limits Copy Chapter

No command line in this repository runs an action or settles a guard right
now, so this demo no longer runs end to end from the shell. The runtime
settles guards for a caller that hands it a judge, and the one caller that did —
the ADK agent this directory used to hold — was removed when the demo became a
skill. The rules, the judgments, the on_violation routing and the judge itself
are all still in the library and still tested; what is missing is a caller wired
to them. Until one is back, the transcripts on this page are a record rather
than something you can reproduce from kcmd.

The AlloyDB skill omits a canned CLI read command. While both the spanner
and alloydb skills emit the full physical table and column schema map (GoogleSQL
and PostgreSQL), the gcloud spanner databases execute-sql command block is
emitted for Spanner stores only because AlloyDB queries depend on the caller's
network path (psql or AlloyDB Auth Proxy). The Spanner leg of this demo was
run live; the AlloyDB leg was not re-run against a live AlloyDB cluster for this
revision, and the claim above about its generated output is from the diff.

Two harnesses were run, not every harness. Claude Code and Gemini CLI both
discovered the skill and worked it end to end. Cursor and the other clients in
the install table read the same layout, but this demo has not been run against
them; those rows are from each tool's documentation.

Every guard is a model call, and a model can answer twice differently. Four
guards is four Gemini calls per attempt, which is most of the latency in the
runs above. And a borderline call is not guaranteed to be settled the same way
twice. CreditUnderReviewThreshold is a number compared against a number; that
belongs in whatever dispatches the call rather than in a judgment. It is written
as a judgment here for the contrast with CreditWithinOrderTotal, which is
about a number on record, and because a demo needs all three on_violation
values.

CreditWithinOrderTotal is declared and not enforced. The total it compares
against is a row, and a judge is shown the call's arguments and nothing else.
The rule is in commerce.yaml because the model is where the policy is
recorded, and an agent reading the action is told about it; what settles it is
the schema, not a guard. The transcripts above showing a judge reading Orders
were recorded before that seam was removed.

A judgment can be wrong. Writing this demo caught one returning non-JSON and
one inventing an order total it could not read — the first fails closed,
correctly. A judgment that returns well-formed nonsense does not.
Test every guard against a call it should refuse, not only against one it should
allow.

One rule in this business is not expressible as a guard. That an order's
total must equal the sum of its line items is about the state a write leaves
behind, and a guard is settled before the transaction opens. No rule stated in
commerce.yaml reaches it. It belongs in the schema. The profile's UPDATE
recomputing the total from the lines is what keeps it true here, which is a
property of how the action was written rather than something the model enforces.

11. Cleaning up Copy Chapter

gcloud spanner databases delete "$DATABASE" \
  --instance="$INSTANCE" --project="$PROJECT"

And remove the generated skill from wherever you installed it —
.claude/skills/commerce/ if you followed section 6.

What is in this directory Copy Chapter

Nothing generated is checked in. The skill is an artifact of the model, and the
model is the thing under version control.

Further reading Copy Chapter

• docs/semantic-model/skills.md —
  what skills-generate emits and why, section by section.
• docs/semantic-model/actions.md —
  actions, executors, affects, guards, judged constraints, and the runtime that
  settles them.

Samples Enrichment Sample Docs Overview (`samples/enrichment/sample/docs/overview.md`) Copy Section

Google Analytics 4 BigQuery Public Dataset

This provides an overview of kc_enrich_sample_data.ga_events table.

High-Level Information and Data Source Copy Chapter

The Google Analytics 4 (GA4) sample ecommerce dataset is a publicly available dataset hosted within the Google Cloud BigQuery Public Datasets program. It provides raw, event-level data exported directly from Google Analytics 4.

The Source Copy Chapter

The data originates from the Google Merchandise Store, an actual online e-commerce website that sells Google-branded apparel, accessories, and other items. The website implements a standard GA4 web e-commerce setup alongside enhanced measurement events.

Key Characteristics Copy Chapter

Dataset Name: bigquery-public-data.ga4_obfuscated_sample_ecommerce

Table Format: events_YYYYMMDD (Daily partitioned tables, totaling 92 tables).

Time Range: The dataset covers a three-month period from November 1, 2020, to January 31, 2021.

Obfuscation: To protect user privacy and business metrics, the data is deliberately obfuscated. You will encounter placeholder values such as , NULL, and empty strings ''. Because of this, internal consistency might be limited, and the metrics cannot be directly compared to the live GA4 Demo Account for the Google Merchandise Store.

Samples Policy As Code Agent Docs HIGH LEVEL DETAILS (`samples/policy-as-code-agent/docs/HIGH_LEVEL_DETAILS.md`) Copy Section

Policy-as-Code Agent: High-Level Details

Purpose Copy Chapter

The Policy-as-Code Agent is designed to automate and streamline data governance by enabling users to define and validate data policies using natural language. It addresses the challenges of manual governance checks and the inflexibility of traditional rule-based systems.

Value Proposition Copy Chapter

Technical (CEs/SAs) Copy Section

• Agentic AI: Replaces static rules with generative logic. The LLM writes Python policy checks on-the-fly, showcasing advanced reasoning.
• Hybrid Execution: Combines LLM intent parsing with deterministic Python execution. This eliminates hallucinations during enforcement.
• Intelligent Memory: Vector-search memory (Firestore) caches and learns successful policies, optimizing latency and cost—a key production pattern.
• Dual-Mode: Flexible execution against offline GCS exports or the live Knowledge Catalog API, demonstrating versatile integration.

Business (Sales/FSRs) Copy Section

• Democratized Governance: Enables non-tech stakeholders to audit data via natural language, removing engineering bottlenecks.
• Proactive Risk: "Compliance-as-code" instantly tests policies (e.g., retention rules), mitigating regulatory fines.
• Efficiency: Automates manual dataset inspections, turning days of work into seconds.
• Self-Healing: Moves beyond monitoring to active governance by suggesting remediation for violations.

1. How It Works: From Query to Report Copy Chapter

The agent follows a simple, powerful, five-step process:

1. Search the Memory: When a user provides a policy, the agent first semantically searches its internal memory to see if it has already generated code for a similar policy. The search can be filtered by author and date range.
2. Understand the Policy: If no cached policy is found, a user provides a policy in natural language, like "All datasets in the 'finance' domain must have a 'data_owner' label."
3. Generate and Cache the Logic: The agent's core, powered by the Gemini LLM, dynamically writes a small, targeted Python script to check for that policy. Once generated, the agent saves this new policy to its memory for future use. The memory is version-controlled, so updates to a policy create a new version.
4. Execute and Report: The agent runs the Python script (either from the cache or newly generated) against your chosen metadata source—either a static metadata export from GCS or live results from a Knowledge Catalog search. It then reports any resources found to be in violation of the policy.
5. Suggest Remediation: If violations are found, the agent can, at the user's request, use the LLM to suggest actionable remediation steps for each violation.
6. Feedback and Ranking: After a policy is used, the user can rate it. This feedback is used to rank policies, so the best ones are suggested first.

2. Key Features Copy Chapter

• Natural Language Interface: No coding or specialized query languages required.
• Intelligent, Version-Controlled Memory: The agent caches successfully generated policies. When it receives a new query, it first searches its memory for a similar policy to reuse, saving time, reducing costs, and ensuring that the same policy is always executed in the same way. The memory supports:
  • Versioning: Policies can be updated, and the agent will keep track of the different versions.
  • Pruning: Old and unused policies can be automatically removed from the memory.
  • Enhanced Search: Policies can be searched by author and date range.
  • Ranking: Policies are ranked based on user feedback, so the most effective policies are suggested first.
• Dynamic and Flexible: Generates code on-the-fly to support complex and nuanced policies.
• Flexible Data Sources: Operates on either static metadata exports from GCS for offline analysis or on live Knowledge Catalog search results for real-time validation.
• Actionable Reporting: Pinpoints exact violations for rapid remediation.
• Actionable Remediation Suggestions: Provides clear, actionable steps to fix policy violations.

3. Technical Deep Dive Copy Chapter

• Large Language Model (LLM): The agent uses a dual-model approach for optimal performance and cost-efficiency:
  • Gemini 2.5 Flash: Used for the agent's conversational logic and routing user requests to the appropriate tools.
  • Gemini 2.5 Pro: Leveraged for the most complex task: generating accurate, executable Python code from natural language policies.
• Agentic AI Architecture: This agent is an example of Agentic AI, which is more advanced than a traditional AI Agent.
  • AI Agent vs. Agentic AI:
    • An AI Agent typically follows pre-programmed rules or a narrowly trained model to perform a specific task.
    • Agentic AI is a more advanced paradigm where the system can autonomously reason, plan, and execute a series of steps to achieve a high-level goal. It can adapt, learn, and use tools to solve complex problems without direct instruction for each step.
  • Why This is Agentic AI:
    • Dynamic Code Generation: The agent doesn't just run a pre-written policy checker; it creates the checker on the fly by translating natural language into executable Python. This demonstrates a high degree of reasoning and autonomy.
    • Adaptability: It is not limited to a fixed set of policies. It can handle new and unforeseen policy requests without needing to be reprogrammed.
    • Autonomous Tool Use: The agent autonomously chains its tools together—first generating the policy code, then executing it—to fulfill the user's request seamlessly.
  • Planning and Execution:
    • Planning: The LLM's first task is to plan its approach by converting the user's English query into a logical, executable Python function. This is guided by a sophisticated prompt template (prompts/v4.md) that provides the LLM with context, the metadata schema, and dynamically generated examples from the user's own data.
    • Execution: The agent then takes this generated code and runs it using Python's exec() function to perform the actual validation.
• Core Components:
  • llm_generate_policy_code: The "brain" of the agent. This function packages the user's query into a detailed prompt and sends it to the Gemini LLM to get the Python validation code back.
  • run_simulation: The "engine room." This function takes the Python code from the LLM and executes it against the metadata, collecting and formatting any violations.
  • Metadata Sources: The agent can source metadata from two places:
    • GCS Loader: A utility that can read metadata exports from a Google Cloud Storage (GCS) bucket.
    • Knowledge Catalog Search: A tool that directly queries Knowledge Catalog and fetches full entry details in real-time.
• Technology Stack:
  • Primary Language: Python
  • Key Libraries:
    • vertexai: To interact with the Gemini LLM.
    • google-cloud-firestore: To interact with the Firestore database for memory.
    • google-cloud-storage: To read metadata files from GCS.
    • google-cloud-dataplex: To interact with the Knowledge Catalog API.
    • google-adk: The framework for building the agent.
  • Data Format: The agent is designed to consume JSONL (JSON Lines) formatted metadata exports, which is a standard output from Knowledge Catalog.
• Security:
  • AST Analysis: Before any code is executed, the agent performs static analysis using Python's Abstract Syntax Tree (AST) module. This detects and blocks dangerous imports (like os, sys, subprocess) and unsafe function calls (like eval, open).
  • Restricted Execution: The dynamic execution of code is handled within a restricted environment using exec(), where access is limited to a safe allowlist of libraries (json, re, datetime) and standard built-ins.
• Extensibility: New and unforeseen policy types can be supported instantly without any changes to the agent's underlying code, as the LLM can generate the required logic on demand.

4. Usage Example Copy Chapter

1. User Input (Natural Language):

   "The 'sensitivity' label must be applied to all tables in public_data dataset"

2. Agent's Generated Code (Simplified):

   python Copy

   def check_policy(metadata: list) -> list:
       violations = []
       for resource in metadata:
           # Logic to check if the resource is a table within the 'public_data' dataset
           # Logic to check if the 'sensitivity' label is missing
           if is_in_public_data_dataset and sensitivity_label_is_missing:
               violations.append({
                   "resource_name": resource['fullyQualifiedName'],
                   "violation": "Table is in 'public_data' dataset but is missing the 'sensitivity' label."
               })
       return violations

3. Final Output to User:

   Policy Violations Found:

   • Resource: bigquery:my-project.public_data.daily_active_users
     • Violation: Table is in 'public_data' dataset but is missing the 'sensitivity' label.
   • Resource: bigquery:my-project.public_data.website_commenters
     • Violation: Table is in 'public_data' dataset but is missing the 'sensitivity' label.

5. Advanced Capabilities Copy Chapter

📋 Compliance Scorecard & Reporting Copy Section

The agent offers robust tools for high-level governance reporting:

• Compliance Scorecard: Run a "health check" on your data (e.g., "Generate a compliance scorecard for my dataplex assets"). The agent runs a suite of Core Policies and calculates a compliance score.
• Configurable Core Policies: Define what "compliance" means for your organization. View, add, remove, and save your own set of Core Policies to the agent's memory.
• Rich Reporting: Export violation reports to CSV or HTML for offline sharing, or upload them directly to Google Cloud Storage.

📊 Policy Analytics & History Copy Section

The agent tracks all policy executions, enabling advanced analysis:

• Execution History: Ask about past runs (e.g., "What policies failed yesterday?").
• Top Violations: Identify the most problematic policies.
• Resource Search: Find all violations for a specific resource using partial names.
• Top Violated Resources: Discover which assets are most non-compliant.

🔌 Model Context Protocol (MCP) Integration Copy Section

The agent is compatible with the Model Context Protocol (MCP).

• Knowledge Catalog MCP: Connects to a Knowledge Catalog MCP server (set DATAPLEX_MCP_SERVER_URL) to access additional tools for interacting with Knowledge Catalog resources.
• Extensibility: Allows the agent to gain new capabilities by connecting to other MCP-compliant servers.

Samples Policy As Code Agent Docs LOW LEVEL DETAILS (`samples/policy-as-code-agent/docs/LOW_LEVEL_DETAILS.md`) Copy Section

Policy-as-Code Agent: Low-Level Details

Architecture Overview Copy Chapter

The Policy-as-Code Agent is designed with a "memory-first" architecture to ensure efficiency and consistency. Before generating new code, it always checks a local memory of previously generated policies.

graph TD
    A[User provides policy query] --> B(root_agent)

    subgraph ADK Agent
        B --> T_Find(find_policy_in_memory)
        T_Find --> C{Policy Found?}
        C -- No --> E{Generate Code}
        E --> T_Save(save_policy_to_memory)
        T_Save --> F(Execute Policy)
        C -- Yes --> F
        F --> G[Violation Report]
        G --> H{Rate Policy?}
        H -- Yes --> T_Rate(rate_policy)
    end

    subgraph Memory
        T_Find --> M_Firestore[(Firestore)]
        T_Save --> M_Firestore
        T_Rate --> M_Firestore
    end

Component Breakdown Copy Chapter

1. GCS Workflow Tools Copy Section

• generate_policy_code_from_gcs(query: str, gcs_uri: str): An ADK tool that generates Python policy code based on a schema inferred from a GCS metadata file.
• run_policy_from_gcs(policy_code: str, gcs_uri: str): An ADK tool that executes the generated policy code against the metadata from the specified GCS URI.

2. Knowledge Catalog Search Workflow Tools Copy Section

• generate_policy_code_from_dataplex(policy_query: str, dataplex_query: str): An ADK tool that orchestrates the first step of the live validation workflow. It connects to Knowledge Catalog, fetches a small sample of entries matching the query, infers the schema, and generates the Python policy code.
• run_policy_on_dataplex(policy_code: str, dataplex_query: str): An ADK tool that handles the second step. It takes the generated code, connects to Knowledge Catalog to fetch all entries matching the query, and executes the code against the full dataset.

3. Memory Components (`policy_as_code/memory.py`) Copy Section

To enhance efficiency and provide a consistent user experience, the agent includes a robust memory feature for caching policies using Google Cloud Firestore and Vertex AI Vector Search.

Firestore Configuration

• Client Initialization: The agent initializes a google.cloud.firestore.Client using the project ID and database name (default: policy-agent-db).
• Vector Search: Uses DistanceMeasure.COSINE to find semantically similar policies.
• Embedding Model: text-embedding-004 from Vertex AI is used to generate 768-dimensional vectors for natural language queries.

Collections Schema

A. policies Collection (Stores reusable policy logic)

• policy_id (String/UUID): Unique identifier.
• version (Integer): Version number (increments on updates).
• query (String): The original natural language policy query.
• embedding (Vector): The vector embedding of the query.
• code (String): The generated Python code.
• source (String): 'gcs' or 'dataplex'.
• author (String): The creator of the policy.
• created_at (Timestamp): Creation time.
• last_used (Timestamp): Last accessed time.
• ratings (Array[Int]): User ratings (1-5).
• feedback (Array[String]): User feedback comments.
• total_runs (Integer): Aggregate execution count.
• total_violations_detected (Integer): Aggregate violation count.

B. policy_executions Collection (Logs every run for auditing)

• policy_id (String): Reference to the policy.
• version (Integer): Version used.
• timestamp (Timestamp): Execution time.
• status (String): 'success', 'failure', or 'violations_found'.
• violation_count (Integer): Number of detected violations.
• violated_resources (Array[String]): List of resource names that failed.
• summary (String): Brief text summary of the run.

C. configurations/core_policies Document

• policies (Array[String]): List of active "Core Policy" queries.

4. LLM & Code Generation (`policy_as_code/utils/llm.py`) Copy Section

The agent uses a sophisticated prompt engineering approach to generate accurate Python code.

• Prompt Template: Located at prompts/code_generation.md.
• Dynamic Injection:
  • {{USER_POLICY_QUERY}}: The user's request.
  • {{INFERRED_JSON_SCHEMA}}: A JSON schema derived from the actual metadata.
  • {{SAMPLE_VALUES}}: Actual data samples extracted from the source to help the LLM understand field formats.
• Parsing: The function llm_generate_policy_code uses regex (r"```python\n(.*)\n```") to robustly extract the code block from the LLM's markdown response.

5. Simulation & Security (`policy_as_code/simulation.py`) Copy Section

The run_simulation function is the secure execution engine for the agent.

• Step 1: Static Analysis (AST):
  • Uses Python's ast module to parse the generated code before execution.
  • Blocked Imports: os, sys, subprocess, shutil, requests, urllib, etc.
  • Blocked Functions: eval, exec, open, compile.
• Step 2: Restricted Execution:
  • Uses exec() with a strictly limited globals dictionary.
  • Allowed Globals: Standard built-ins (len, list, dict, etc.) plus json, re, and datetime.
  • Access Control: The code cannot access the agent's file system, network, or environment variables.

Detailed Data Flow Copy Chapter

1. User Query: User asks, "Ensure all tables in 'finance' have a description."
2. Vector Search: find_policy_in_memory converts this to a vector and queries Firestore.
   • If match found (similarity > threshold): Returns cached code.
3. Sample Generation (If no match):
   • The agent scans the metadata source (GCS file or Knowledge Catalog results).
   • It identifies the "most representative" entry (the one with the most fields) to use as a sample.
4. Code Generation:
   • Constructs the prompt with schema and sample.
   • Calls Gemini 2.5 Pro.
   • Extracts Python code.
5. Memory Save: The new code and query embedding are saved to Firestore.
6. Execution:
   • validate_code_safety checks for malicious patterns.
   • exec() runs the check_policy(metadata) function.
7. Reporting: Violations are collected and returned to the user.
8. Logging: The execution result is written to the policy_executions collection in Firestore.

Tool Signatures Copy Chapter

For reference, here are the signatures of the primary tools:

Samples Policy As Code Agent Docs MEMORY IMPLEMENTATION (`samples/policy-as-code-agent/docs/MEMORY_IMPLEMENTATION.md`) Copy Section

Hybrid Memory Implementation: Agent Engine + Firestore

The Policy-as-Code Agent utilizes a Hybrid Memory Architecture that combines the native conversational capabilities of the Agent Engine (ADK) with a specialized, structured knowledge store built on Google Cloud Firestore.

This approach ensures the agent is both conversationally aware (remembering context within a chat) and domain intelligent (remembering complex policy logic and execution history permanently).

The Hybrid Approach Copy Chapter

We distinguish between two types of memory required for a robust enterprise agent:

1. Conversational Context (Short/Medium Term): Handled by Agent Engine Native Memory.
2. Procedural & Semantic Knowledge (Long Term): Handled by Firestore + Vector Search.

Layer 1: Conversational Context (Agent Engine Native) Copy Chapter

The agent leverages the built-in memory capabilities of the Google Cloud Agent Engine (ADK) to handle the nuances of human conversation.

Components Copy Section

• VertexAiMemoryBankService: The backend service that stores the session history and turns.
• PreloadMemoryTool: A special tool injected into the agent's toolset. Before the LLM generates a response, this tool searches the session history for relevant past interactions and injects them into the prompt context.
• auto_save_session_to_memory_callback: An asynchronous callback registered in the Agent definition. It ensures that every interaction (user query + agent response) is automatically persisted to the memory bank.

How it works Copy Section

When a user says, "Run that check again but for the finance dataset", the Agent Engine memory provides the context of what "that check" refers to (e.g., the policy discussed in the previous turn). Without this, the agent would not understand the reference.

# policy_as_code/agent.py

# 1. Callback to save state
async def auto_save_session_to_memory_callback(callback_context):
    await callback_context._invocation_context.memory_service.add_session_to_memory(
        callback_context._invocation_context.session)    

# 2. Agent Initialization with Memory Tool
root_agent = Agent(
    # ...
    tools=[
        # ...
        adk.tools.preload_memory_tool.PreloadMemoryTool() # <--- Injects context
    ],
    after_agent_callback=auto_save_session_to_memory_callback # <--- Saves context
)

Layer 2: Procedural & Domain Knowledge (Firestore) Copy Chapter

While the ADK handles chat, it is not designed to store executable code or complex structured data like policy versions and audit logs. For this, we implemented a custom memory layer using Google Cloud Firestore.

Components Copy Section

• memory.py: A custom module that acts as the interface to Firestore.
• Vector Search: Uses Vertex AI embeddings (text-embedding-004) to create a semantic index of all known policies.
• Structured Collections:
  • policies: Stores the generated Python code, the natural language query, and the vector embedding.
  • policy_executions: Stores audit logs of every run.

How it works Copy Section

When a user asks for a policy, the agent doesn't just look at the chat history; it consults its "Knowledge Base."

1. Semantic Retrieval: The agent embeds the user's query and searches Firestore.
2. Code Reuse: If a similar policy exists (even if phrased differently), it retrieves the tested, cached Python code. This is the "Procedural Memory" — remembering how to do something.
3. Shared Knowledge: Because Firestore is a centralized database, if User A creates a complex policy for "GDPR Compliance," User B can immediately use it without having to recreate it.

The "Secret Sauce": How They Work Together Copy Chapter

The power of this agent lies in the intersection of these two memory systems.

Example Scenario Copy Section

User: "Create a policy to check for PII in the customer_data table."

1. Firestore Memory (Action): The agent generates the code, runs it, and saves it to Firestore as Policy ID 123-abc.
2. ADK Memory (Observation): The ADK records that the user is working on Policy 123-abc and is interested in customer_data.

User: "Okay, now run it on the sales_data table."

1. ADK Memory (Retrieval): The PreloadMemoryTool identifies that "it" refers to the PII policy (Policy ID 123-abc) discussed in the previous turn.
2. Firestore Memory (Retrieval): The agent uses the ID 123-abc to fetch the executable Python code from Firestore.
3. Execution: The agent applies the retrieved code to the new target (sales_data).

Architecture Diagram Copy Section

graph TD
    User[User] -->|Query| Agent[Agent Logic]

    subgraph "Layer 1: Conversational Context (ADK)"
        Agent <-->|Read/Write Context| ADK_Mem[Vertex AI Memory Bank]
        ADK_Mem -->|Context Injection| Prompt[LLM Prompt]
    end

    subgraph "Layer 2: Domain Knowledge (Firestore)"
        Agent <-->|Vector Search| FS_Vector[Firestore Vector Index]
        Agent <-->|Read/Write Code| FS_DB[Firestore Policies Collection]
    end

    Prompt --> LLM[Gemini LLM]
    LLM -->|Decides Tool| Agent

Benefits of Hybrid Architecture Copy Chapter

1. Cost Efficiency: We don't waste tokens re-generating code (Firestore handles caching), and we don't overload the context window with irrelevant history (ADK memory selects only relevant turns).
2. Continuity: The agent feels "smart" because it follows the conversation flow (ADK) while also being "knowledgeable" because it remembers established business rules (Firestore).
3. Scalability: Conversational context is ephemeral and session-scoped, while Policy knowledge is permanent and organization-scoped. separating them allows the knowledge base to grow indefinitely without impacting chat performance.

Samples Policy As Code Agent Docs MEMORY INTEGRATION (`samples/policy-as-code-agent/docs/MEMORY_INTEGRATION.md`) Copy Section

Memory Bank Integration

This document outlines the integration of a long-term memory bank with the Policy-as-Code agent to provide stateful, context-aware policy evaluation.

Architecture: Firestore + Vector Search Copy Chapter

The agent's memory is built on Google Cloud Firestore using its Native Vector Search capabilities.

• Storage: Policy metadata (code, author, creation date, ratings) is stored as documents in a Firestore collection.
• Semantic Search: The natural language query is converted into a vector embedding using Vertex AI (text-embedding-004). This vector is stored in the Firestore document.
• Retrieval: When a user asks for a policy, the agent performs a cosine similarity search directly within Firestore to find the most relevant existing policies.

Configuration & Setup Copy Chapter

1. Enable Firestore (Native Mode) Copy Section

To use the memory bank, you must have a Firestore database created in your Google Cloud project.

1. Go to the Firestore Console.
2. Click Create Database.
3. Select Native Mode (required for Vector Search).
4. Choose your location (e.g., us-central1).
5. Create the database (default name is usually (default)).

2. Configure Environment Variables Copy Section

The following variables in your .env file control the memory bank:

# Enable/Disable the entire memory subsystem
ENABLE_MEMORY_BANK=True  # Set to False to run without Firestore

# Firestore Configuration
FIRESTORE_DATABASE="(default)" 
FIRESTORE_COLLECTION_POLICIES="policies"
FIRESTORE_COLLECTION_EXECUTIONS="policy_executions"
CORE_POLICIES_DOC_REF="configurations/core_policies"

# Vector Search Model
EMBEDDING_MODEL_NAME="text-embedding-004"

3. Graceful Fallback Copy Section

The agent is designed to handle missing infrastructure gracefully.

• Auto-Disable: If ENABLE_MEMORY_BANK=True but the agent cannot connect to the Firestore database (e.g., it doesn't exist or permissions are missing), the agent will log a warning and automatically disable memory features for the session. It will not crash.
• Instructional Messages: If you attempt to use a memory-dependent tool (like "find similar policies" or "show execution history") while memory is disabled, the agent will return a helpful message explaining that memory is disabled and pointing to this documentation.
• Core Functionality: All core policy generation and evaluation features (generate_policy, run_simulation) continue to work fully without the memory bank.

📊 Analytics and Reporting Copy Chapter

The agent tracks detailed execution logs in Firestore, enabling powerful analytical capabilities:

Features Copy Section

1. Execution History: Track success, failure, and violation rates over time.
2. Top Violations: Aggregated view of which policies are failing most frequently.
3. Resource Search: Fuzzy search to find all violations associated with a specific resource (table, dataset, etc.).
4. Top Violated Resources: Identify "hotspot" assets that consistently violate policies.

Sample Prompts Copy Section

• History: "What happened yesterday?", "Show me all failed policy runs from last week."
• Top Violations: "Which policies are violated the most?", "What are my top compliance issues?"
• Resource Search: "Did 'finance_table' have any violations?", "Check logs for 'quarterly_earnings'."
• Top Resources: "What are the top 10 most violated resources?", "Which tables are most non-compliant?"

Key Benefits Copy Chapter

Integrating this persistent memory system transforms the agent from a stateless policy checker into a learning, stateful governance assistant.

1. Scalability and Concurrency Copy Section

Unlike a local file-based memory, Firestore allows multiple agent instances to share the same knowledge base simultaneously. It scales to millions of policies without performance degradation.

2. Historical Context and Trend Analysis Copy Section

By storing the results of policy runs over time, the agent could answer much more sophisticated questions.

• Current State: You can ask, "Does any dataset in 'marketing' have more than one PII column?"
• With Memory: You could ask, "Have any new datasets in 'marketing' become non-compliant with the PII policy in the last 7 days?" or "Show me the compliance trend for our data ownership policies over the last quarter."

3. Learning User Intent and Personalization Copy Section

The agent could remember a user's common queries, domains of interest, and corrections, leading to a more efficient workflow.

• Problem: A data steward for the 'finance' domain has to type out the full policy query specifying the 'finance' domain each time.
• With Memory: After a few queries, the agent could learn this preference. The user could simply ask, "Run the standard ownership check," and the agent would know to apply it to the 'finance' domain.

4. Intelligent Caching and Performance Copy Section

Memory is used as a semantic cache for the LLM-generated Python code, saving time and cost.

• Problem: If you run the exact same natural language policy query twice, the agent currently calls the LLM to generate the same Python code twice.
• With Memory: The agent stores a mapping of the natural language query (vector) to the successfully generated Python function. If a similar query is asked again, it retrieves the code from Firestore instead of making a new LLM call.

5. Root Cause Analysis and Remediation Copy Section

By remembering the context of past failures, the agent could provide better suggestions for remediation.

• Current State: The agent reports that "Table X is missing an owner."
• With Memory: The agent might remember that "Table X" was created by the same service account that created 10 other unowned tables. It could then suggest a more systemic fix.

Toolbox Mdcode Docs Concept (`toolbox/mdcode/docs/concept.md`) Copy Section

Metadata as Code

Overview Copy Chapter

This document details Metadata as Code, a core Knowledge Catalog capability
designed to provide agent builders, data stewards and data producers, a source
code artifact-based UX for metadata management and context engineering. Metadata
as Code enables users and agents to author, manage, and consume metadata
artifacts using developer-friendly practices such as versioning and CI/CD as
part of their developer repository, and activate context enrichment agents on a
standardized metadata format.

This approach described in this document builds on top of broadly supported
markdown and YAML, and simple file organization structure that enables
bi-directional sync capability with Knowledge Catalog service by layering on top
of existing Import/Export capabilities, and factors in extensibility that is a
core tenet of the Catalog platform.

High-level Plan
We will pick a YAML representation that largely aligns with the Entry/EntryLink

• Aspect data model in Knowledge Catalog to represent both 1st party and 3rd
  party metadata. Unstructured text content in aspects will be represented as
  sidecar markdown files. Local files are organized in a directory structure
  aligned with the resource hierarchy of assets represented by the metadata. Users
  can provide configurations to sync (download/snapshot and publish/deploy)
  metadata from and to the service.

Tenets Copy Section

• Optimized for consumption by users and agents alike.
  Metadata as code artifacts should be easy to read and modify as well as
  naturally fit as files within a developer's repository and support use-cases
  such as versioning, diffing, CI/CD etc. In support of ease of consumption,
  users should be able to organize their metadata in logical and modular slices,
  while the layout of metadata facilitates organization and navigation.

• Optimized for tools ecosystem
  The code artifacts should benefit from existing IDE and tool capabilities such
  as previews, navigation, etc.

• Enable bi-directional sync with Knowledge Catalog and metadata fidelity
  The code artifacts should be amenable to serving as authoring and management
  source of truth. They should enable a user to fully inspect and author
  metadata that exists in the Catalog service. Metadata as code artifacts should
  be able to represent 1st and 3rd party entries, aspects and the links between
  them.

• Embeddable and Consistent Representation
  Users may operate on the metadata in a standalone manner in their workspace
  for example, in the context of an enrichment workflow. Alternatively, the
  metadata may be part of a larger project, such as a Dataform/DBT pipeline,
  a BigQuery semantic graph etc. We should strive for a format and
  representation that can largely be consistent across these use-cases, and be
  used independently or embedded.

Goals Copy Section

Source artifact for agents and human-in-the-loop review workflows
We will offer an agentic capability for metadata curation and enrichment. The
agent will operate over the code representation of metadata to read it, and
update it. Users can also track changes, review it, and if needed, directly
modify it in an interactive loop.

Metadata as Code is a pre-requisite capability to allow such agents to be built
and integrated into a standardized workflow and UX.

Source artifact for metadata sidecars
Similar to the above, we are also collectively building agents to streamline
building Semantic Graphs or LookML models, and Data Pipelines. These efforts are
building their use-case specific code artifacts and agent flows.

Metadata as code artifacts should naturally live within a larger project. This
allows users to work with a standard representation. It allows the enrichment
agents to be plugged into the larger agent flow as a sub-agent or agent tool.

Context versioning
Context engineering is central to agent building. Agent builders will need to be
able to operate on their context as source code, evolve it in a versioned
manner, and deploy it as versioned resources.

Metadata as code provides the source code UX and workflows (by virtue of being
hostable within a source control repository and using the existing versioning
capabilities). Developers can then deploy the context to co-existing deployed
versions to be able to run a/b tests and evals, and have agents pick the
appropriate version.

Non-Goals Copy Section

Hierarchical Context Serving Store or Index
The metadata as code model described in this document is aligned with the data
model of metadata within Knowledge Catalog that organizes metadata to mirror
resources in the integrated source systems. Specifically this means metadata
organized as Entries and EntryLinks with aspects, within an EntryGroup (both
system and user-managed) in a specific project.

Context serving or searching use-cases would benefit from a different
organization of metadata (eg. arbitrary hierarchy of content organized by data
stores, use-cases, etc.). Conceivably, we might find such hierarchies naturally
present in Document repositories such as Drive, or g3doc/wikis, or process
metadata in Knowledge Catalog to produce a serving-ready version of metadata.

Use-cases Copy Chapter

This is a high-level mention of some of the use-cases where Metadata-as-Code
should help with context engineering for agents.

Knowledge Catalog Enrichment Agent
Knowledge Catalog will offer an off-the-shelf enrichment agent that is pluggable
with user-provided instructions, skills and tools. This will allow users to plug
in their organizational data sources, including code repositories, queries,
documentation in wikis, drive etc., internal Q&A channels etc.

The enrichment agent operates on code artifacts, providing a mechanism for users
to iterate on enrichment, as well as apply human review steps, and integration
with git-based version control and CI/CD mechanisms for deployment to make the
updates live.

Semantic Graph Business/Semantic Enrichment
We are introducing Semantic Graphs as the mechanism for users to build semantic
models - defining entities and their relationships - that will be consumed by CA
agents to ground their behavior and resulting queries in the organizational
truth.

The graph will allow users to define entities as nodes and relationships as
edges along with additional properties, including computed values and measures.
Users will want to attach additional meaning such as glossary term definitions
as part of their graph authoring UX. Metadata-as-Code will provide a mechanism
for users to co-locate their graph definition and catalog augmentations, and
deploy them collectively.

Metadata propagation and curation in Data Pipelines
We are building a Data Engineering agent to streamline the creation and
productionization of data pipelines. Data engineers should be able to leverage
as well as publish metadata along with the deployment of their pipelines and
data sources and sinks. This will ensure the resulting data is more ready for
agentic consumption and allow for co-locating and unifying the UX for the
transforms and the metadata flow within a pipeline project.

Database Context Set
We are introducing ContextSet and related functionality (Query Templates,
Examples, etc.) to enable database admins and developers to engineer context for
their NLD and NLA agents.

This discussion is early. The observation is Metadata-as-Code offers a potential
mechanism to unify the UX for context authoring, while supporting different
service models (EntryGroup and ContextSetGroup) for purposes of storage, search
and serving.

File Organization Copy Chapter

Metadata is organized within a directory. The directory represents a local
snapshot of a subset of metadata in Catalog. It is the unit of synchronization
with the metadata within the service.

This directory contains a manifest file, named catalog.yaml, that provides
structured configuration that can direct tools on handling the individual
artifacts contained within the directory. It contains sub-directories that
contain individual entries and entry links.

Depending on the scope (see manifest below), files are organized in one of
two layout formats (and potentially extendable to other formats in future):

Standard Layout:
Used for bq-dataset and entryGroup scopes. This is a hybrid disk
organization where the metadata structure is stored in a main YAML file per
entry, and unstructured aspects (like Overview) are placed in sidecar Markdown
files.

path/to/root/
├── catalog.yaml                      # Catalog metadata, processing config/directives
└── catalog/                          # Contains all the Entries and EntryLinks
    └── <dir1>/<dir2>
        ├── <entry-id1>.yaml          # Single-file entry with all metadata contained within
        ├── <entry-id2>.yaml          # Multi-file entry with unstructured metadata split into
        └── <entry-id2>.<aspect>.md   # markdown sidecar files for unstructured aspects

Documents Layout:
Used for kb scopes. This is a Markdown-first disk organization where the
main entry is represented as a single Markdown file, with metadata structured in
the YAML frontmatter and the main overview aspect promoted to the Markdown body.

path/to/root/
├── catalog.yaml                      # Catalog metadata, processing config/directives
└── catalog/                          # Contains all the Entries and EntryLinks
    └── <dir1>/<dir2>
        └── <entry-id1>.md            # Single markdown file: structured metadata in
                                      # frontmatter, Overview in body

NOTE: IDE Workspace
An IDE workspace, like the one in Antigravity, can include one or more such
directories, organized according to the user's preferences.

Manifest - Catalog.yaml Copy Section

The manifest file captures all configuration related information. This includes:

• The resource that the metadata snapshot corresponds to (the scope). This
  resource can either be a resource identified in a source system such as BigQuery
  (e.g. a Dataset, or a set of Tables, whose metadata is managed in Catalog in a
  system EntryGroup, such as @bigquery), a user-managed EntryGroup containing
  user created entries, or a user-managed EntryGroup representing a knowledge base containing markdown-based documentation entries.

• The physical layout of the catalog snapshot on disk is automatically determined
  by the type of the scope:

  • A bq-dataset or entryGroup scope uses the standard layout.
  • A kb scope uses the wiki layout.

• A set of type aliases to make it easy to refer to various catalog constructs
  such as entry-types, aspect-types, link-types, glossaries, glossary-terms, etc.

• The type of entries, aspects, and links to include into the local snapshot
  when downloading metadata from the service.

• The subset of entries, aspects, and links to publish back to the service.

catalog.yaml

scope: <type>.<name>              # The resources in the snapshot. Examples:
                                  # scope: entryGroup.<projectId>.<locationId>.<entryGroupId>
                                  # scope: bq-dataset.<projectId>.<datasetId>
                                  # scope: kb.<projectId>.<locationId>.<entryGroupId>
                                  # Or multiple BigQuery datasets in array list format:
                                  # scope:
                                  #   - bq-dataset.<projectId>.<datasetId1>
                                  #   - bq-dataset.<projectId>.<datasetId2>


aliases:                          # Optional. Can always use 3-part fully qualified references.
                                  # NOTE: All built-in types have predefined simple aliases.
  ca-guidelines:
    aspect: data-agents-project.global.ca-guidelines
  ecommerce:
    glossary: data-gov-project.global.ecommerce-glossary

# Defines the specific metadata to be retrieved locally from Knowledge Catalog.
# NOTE: Required aspects of listed entry types are implicitly included.
snapshot:
  entries:
  - bigquery-dataset
  - bigquery-table
  aspects:
  - overview
  - descriptions
  - queries
  - ca-guidelines
  entryLinks:
  - definition

# Optional configuration to identify which types should be published.
# This must be a subset of the types specified in the snapshot configuration.
publishing:
  entries:
  aspects:
  - ca-guidelines
  entryLinks:
  - definition

NOTE: Type References
Metadata content has several references to either the meta-model resources
(entry, aspect, and link types) or other catalog constructs (such as
glossaries). Syntax is provided to streamline these references.

• References take the form of .<locationId>.<resourceId>. There is
  no need for referring to the verbose collection names themselves
  (projects//locations///)
• Aliases can be defined to create single identifier names that are resolved
  when metadata is processed.
  • These are optional. Users can choose to use the fully-qualified references
    if that is the preference.
  • All built-in Dataplex types have pre-defined aliases without requiring the
    user to declare them. Eg. bigquery-table resolves to
    dataplex-types.global.bigquery-table.
• Use project IDs, not project numbers when generating metadata.

Entry File Names Copy Section

The name of the file is used to identify the Entry uniquely. The following rules
are followed to derive file names from EntryIds.

Ingested Entries use the service-qualified full resource name as EntryId.
Inferrable segments (Service name and Location from EntryGroup) and Collection
names (where possible) will be stripped out.

| bigquery.googleapis.com/ projects/projectId/ datasets/datasetId/ tables/tableId | projectId.datasetId/tableId.yaml | Aligns with physical hierarchy. Common case for context use-cases. Avoid a type segment in directory path to optimize for most common resource when a collection has multiple types of sub-resources |
| bigquery.googleapis.com/ projects/projectId/ datasets/datasetId/ routines/routineId | routines/projectId.datasetId/routineId.yaml | Moved into sub-directory to avoid potential conflicts between table and routine names |
| cloudsql.googleapis.com/ projects/projectId/ locations/locationId/ instances/instanceId/ databases/databaseId | instanceId/databaseId/tableId.yaml | Example included to demonstrate how EntryIds and corresponding file names treat control plane and dataplane ids together |

3rd party Custom Entries may use a similar REST API name of the resource
(recommended when applicable) or arbitrary path-identifier (more likely
scenario); We align the metadata file path to the Entry Id as there are no
conventions to apply here.

File Extensions Copy Section

• Standard Layout (for bq-dataset and entryGroup scopes): Entry files
  use the .yaml extension (e.g., tableId.yaml), with unstructured aspects
  optionally stored in sidecar .md files (e.g., tableId.overview.md).

• Document Layout (for kb scopes): Entry files use the .md extension
  (e.g., tableId.md), where the main YAML entry is replaced with a single
  markdown file containing YAML frontmatter and the promoted unstructured aspect
  (Overview.content) as its body.

Entry File Layouts Copy Section

Metadata files are primarily structured representations of entry catalog info
and aspects. Depending on the scope, the organization of these files differs:

• Standard Layout (YAML + Markdown sidecars): Used for bq-dataset and
  entryGroup scopes. Every entry is represented by at least a single YAML file
  (<entry-id>.yaml) that captures entry metadata, information about associated
  resource, and all aspects and links. Aspects containing unstructured rich-text
  fields (like overview) are split into sidecar markdown files
  (e.g., <entry-id>.overview.md).

• Document Layout (Markdown Only): Used for kb scopes. The main YAML file is
  replaced with a single markdown file (<entry-id>.md). The structured YAML
  metadata moves entirely to the YAML frontmatter of this markdown file, and the
  unstructured text of the overview.content aspect becomes the main markdown
  body of the file.

Standard Layout

Entry Resource Info + Entry Source + Aspects (entry.yaml)

id: <id>                                # Entry metadata
type: <entryType>
resource:                               # Entry.EntrySource
  name: <entrySource.name>
  displayName: <entrySource.displayName>
  description: <entrySource.description>
  labels:
    key: value
  location: <entrySource.location>
  parent: <entry.parent>
  ancestors: <entrySource.ancestors>
  createTime: <entrySource.createTime>
  updateTime: <entrySource.updateTime>

schema:
  fields:
  - name1: <schemaField.name>
    dataType: <schemaField.dataType>
    mode: <schemaField.mode>
    …
    links:                              # EntryLinks associated with Schema.path are inlined
      definition:                       # into a Schema field to leverage context of field
      - target: glossary.term           # specification

<aspect-type>:                          # In the general-case, each top-level field is an
  [aspect.data]                         # aspect. Nested field represents aspect.data

links:                                  # EntryLinks with this entry as source listed here
  <entryLink-type>:
  - target: <target-entry-reference>
    <aspect-type>:
      [aspect.data]

Sidecar File per Aspect with Unstructured Text (entry.overview.md)

---
field: value                            # YAML frontmatter used to capture structured metadata
---
[aspect.data.rich-text-field]

Wiki Layout

In the wiki layout, the YAML content is placed in the YAML frontmatter
of a .md file, and the unstructured overview text (overview.content) is
promoted to the markdown body.

Entry with Inlined Unstructured Text (entry.md)

---
id: <id>                                # Entry metadata in frontmatter
type: <entryType>
resource:                               # Entry.EntrySource
  name: <entrySource.name>
  displayName: <entrySource.displayName>
  description: <entrySource.description>
  labels:
    key: value
  location: <entrySource.location>
  parent: <entry.parent>
  ancestors: <entrySource.ancestors>
  createTime: <entrySource.createTime>
  updateTime: <entrySource.updateTime>

schema:
  fields:
  - name1: <schemaField.name>
    dataType: <schemaField.dataType>
    mode: <schemaField.mode>
    …
    links:
      definition:
      - target: glossary.term

<aspect-type>:                          # Other structured aspects
  [aspect.data]

overview:                               # Unstructured aspect's metadata
  userManaged: true
  links:
  - title: <title>
    url: <url>

links:                                  # EntryLinks with this entry as source
  <entryLink-type>:
  - target: <target-entry-reference>
    <aspect-type>:
      [aspect.data]
---
# <displayName or entry-id>

This is the main markdown body representing `overview.content` aspect data.

Examples Copy Chapter

Enrichment Agent Copy Section

This is an example for a BigQuery dataset and a table. For purposes of
illustration, the user is interested in working the metadata that is used in
Conversational Analytics, to enrich Overview documentation, Generated
descriptions, Suggested Queries, Agent Guidelines and Glossary associations, and
publish them back to Knowledge Catalog.

workspace/catalog.yaml

scope: bq-dataset.ecommerce-prod.ecommerce-dataset

resourceAliases:
  guidelines:
    aspect: data-agents-project.global.guidelines
  ecommerce:
    glossary: data-gov-project.global.ecommerce-glossary

snapshot:
  entries:
  - bigquery-dataset
  - bigquery-table
  aspects:
  - overview
  - descriptions
  - queries
  - guidelines
  - data-profile            # Included as local context, but not published
  - query-usage             # Same as above
  entryLinks:
  - definition

publishing:
  aspects:
  - overview
  - descriptions
  - queries
  - guidelines
  entryLinks:
  - definition

workspace/catalog/ecommerce-prod.ecommerce-dataset.yaml

id: bigquery.googleapis.com/projects/ecommerce-prod/datasets/ecommerce-dataset
type: bigquery-dataset
createTime: <createTime>
updateTime: <updateTime>

resource:
  name: projects/ecommerce-prod/datasets/ecommerce-dataset
  displayName: E-commerce Production Dataset
  description: ...
  labels:
    env: prod
  createTime: <entrySource.createTime>
  updateTime: <entrySource.updateTime>

bigquery-dataset:
  ...

descriptions:
  description: <generated dataset description>

workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.yaml

id: bigquery.googleapis.com/projects/ecommerce-prod/datasets/ecommerce-dataset/tables/orders
type: bigquery-table
createTime: <createTime>
updateTime: <updateTime>

resource:
  name: projects/ecommerce-prod/datasets/ecommerce-dataset/tables/orders
  displayName: E-commerce Orders
  description: ...
  labels:
    env: prod
  createTime: <entrySource.createTime>
  updateTime: <entrySource.updateTime>

bigquery-table:
  ...
storage:
  ...

schema:
  fields:
  - name: id
    dataType: string
    mode: required
    links:
      definition:
      - target: ecommerce.order-number
  - name: status
    dataType: integer
    mode: required

dataProfile:
  [data-profile-data]

queryUsage:
  [top-fields and other usage aggregations]

descriptions:
  description: <generated table description>
  fields:
  - name: id
    description: ...
  - name: status
    description: ...

queries:
  queries:
  - description: ...
    sql: |
      [SQL Text]

links:
  definition:
    target: ecommerce.transaction

workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.overview.md

---
userManaged: true
links:
- title: <title>
  url: <url>
---
[overview.content]

workspace/catalog/ecommerce-prod.ecommerce-dataset/orders.guidelines.md

---
userManaged: false
---
[guidelines.instructions]

Knowledge Base Copy Section

Assume the user is managing a knowledge base of information managed in a hierarchical
structure. Or alternatively, an agent is building this knowledge base from previously
aggregated or enriched metadata.

workspace/catalog.yaml

scope: kb.ecommerce-prod.global.mbr-kb

snapshot:
  entries:
  - document
  aspects:
  - overview

publishing:
  aspects:
  - overview

workspace/catalog/products.md

---
id: products
type: document
createTime: <createTime>
updateTime: <updateTime>

resource:
  name: products
  displayName: Products
  description: ...
---
# Products Data

This is the markdown body representing `overview.content` for the document.

workspace/catalog/playbooks/mbr.md

---
id: playbooks/mbr
type: document

resource:
  name: playbooks/mbr
  displayName: MBR Playbook
  description: ...
---
# Ecommerce Monthly Business Review Playbook

[overview.content]

Toolbox Mdcode Docs Design (`toolbox/mdcode/docs/design.md`) Copy Section

Metadata as Code Design Specification

This document details the overall design for the Metadata as Code project, factoring in the sub-system breakdown, API surfaces, CLI and MCP structures, technical stack, and key technical decisions. It builds upon the initial context provided in:

• readme.md
• concept.md
• spec.md
• plan.md

1. Sub-system / Module Breakdown Copy Chapter

The system is divided into two public interface layers and a core library foundation.

1.1 Public Interfaces Copy Section

• CLI (kcmd): A single binary built using Bun that handles user interaction and parses commands using the cac framework.
• MCP Server: Hosted within the same kcmd binary, triggered by the mcp command argument (e.g., kcmd mcp). It exposes sync operations as tools to external agents.

1.2 Core Library Foundation (`kc-mac`) Copy Section

The brain of the operation, containing the following modules:

• CatalogManifest: Handles loading, validating, and parsing the catalog.yaml project manifest, including parsing the unified scope specification (which determines the layout style).
• CatalogSnapshot: Manages local catalog entries and coordinates high-level read/write/list logic by delegating physical file representation and layout-specific tasks to CatalogLayout.
• CatalogLayout: Abstract class defining the operations to list, read, write, and delete files on the local disk. Standard and Wiki layouts implement this interface, automatically selected based on the scope type.
• CatalogSync: Orchestrates pull and push sync directions, handles pagination, fail-fast conflict detection, and manages tool state in a separate state file (.catalog.state).
• CatalogService: Wrapper for raw HTTP API calls, handling Application Default Credentials (ADC) auth and LRO polling.

2. Project Directory Structure Copy Chapter

This section describes the directory structure for the project source code, reflecting the merge of interfaces, TypeScript library naming for future expansion, and the gcp subfolder for API functionality:

• src/
  • tool/: The unified layer for CLI and MCP.
    • main.ts: The entry point for the unified binary (kcmd). Handles argument parsing to direct to either CLI commands or MCP server. Defines/handles CLI surface.
    • commands.ts: Contains functionality for CLI commands.
    • mcp.ts: Contains functionality for the MCP server and tools.
  • libts/: The TypeScript core library.
    • index.ts: The main entry point exporting public APIs.
    • manifest.ts: Contains the CatalogManifest class logic.
    • layout.ts: Contains the CatalogLayout interface.
    • source.ts: Contains the CatalogSource interface.
    • snapshot.ts: Contains the CatalogSnapshot class logic.
    • sync.ts: Contains the CatalogSync class logic.
    • gcp/: Subfolder for GCP-specific functionality.
      • api.ts: Contains the Api base class with polling and retries.
      • context.ts: Contains the ApiContext class.
      • dataplex.ts: Contains the API client for Knowledge Catalog.
    • sources/: Subfolder containing source-specific implementations.
      • entrygroup.ts: Encapsulates behavior for Dataplex EntryGroups.
      • bq-dataset.ts: Encapsulates behavior for BigQuery datasets.
      • wiki.ts: Encapsulates behavior for Wikis.
    • layouts/: Subfolder containing layout-specific implementations.
      • standard.ts: Encapsulates behavior for Standard layout.
      • documents.ts: Encapsulates behavior for Wiki layout.

3. Public API, CLI, and MCP Structure Copy Chapter

2.1 Library API Surface Copy Section

CatalogManifest

• static initWithEntryGroup(entryGroup: string, ctx: ApiContext): CatalogManifest
• static initWithBigQuery(dataset: string, ctx: ApiContext): CatalogManifest
• static initWithKB(kb: string, ctx: ApiContext): CatalogManifest
• static load(path: string, ctx: ApiContext): Promise<CatalogManifest>
• save(path: string): void

CatalogLayout (Abstract Class)

• abstract list(snapshotPath: string): Promise<string[]>
• abstract readEntry(snapshotPath: string, entryId: string): Promise<Entry>
• abstract writeEntry(snapshotPath: string, entryId: string, entry: Entry): Promise<void>
• abstract deleteEntry(snapshotPath: string, entryId: string): Promise<void>

StandardLayout (extends CatalogLayout)

• Implements filesystem storage using a main .yaml file per entry and optional sidecar .overview.md files for unstructured aspects.

WikiLayout (extends CatalogLayout)

• Implements filesystem storage using a single .md file per entry with the metadata in the YAML frontmatter and the overview content as the markdown body.

CatalogSnapshot

• static fromPath(path: string): CatalogSnapshot
• layout: CatalogLayout: The layout strategy instance initialized based on the manifest scope.
• list(): Promise<string[]>: Returns a list of entry IDs in the snapshot by delegating to layout.list(snapshotPath).
• lookupEntry(entryId: string): Promise<Entry>: Returns structured metadata for a specific entry by delegating to layout.readEntry(snapshotPath, entryId).
• updateEntry(entryId: string, updates: Record<string, any>): Promise<void>: Merges updates into the entry metadata and aspect values, and commits the result via layout.writeEntry(snapshotPath, entryId, entry).
• createEntry(entryId: string, data: Entry): Promise<void>: Creates an entry locally by delegating to layout.writeEntry(snapshotPath, entryId, data).
• deleteEntry(entryId: string): Promise<void>: Deletes files associated with the entry by delegating to layout.deleteEntry(snapshotPath, entryId).

CatalogSync

• pull(): Promise<SyncResult>
• push(options?: { force?: boolean, validateOnly?: boolean }): Promise<SyncResult>
• validate(): Promise<ValidationResult>: Fetches fresh definitions and performs pre-push validation.
• status(): Promise<StatusResult>: Checks for local modifications against the saved checksum state.

2.2 CLI Command Structure (`kcmd`) Copy Section

• kcmd init [--bigquery-dataset ] [--entry-group ] [--kb ]
• kcmd pull [--dry-run]
• kcmd push [--dry-run] [--force] [--validate-only]
• kcmd status
• kcmd mcp (Triggers the MCP server)

2.3 MCP Tools and Params Copy Section

• pull: Params: None.
• push: Params: force?: boolean.
• list-entries: Params: None.
• lookup-entry: Params: entryId: string.
• update-entry: Params: entryId: string, and key/value pairs for each field and aspect being updated.

4. API Client Design & Context Copy Chapter

3.1 `ApiContext` (Class) Copy Section

• project: string
• token: string
• static default(): Promise<ApiContext>: Uses gcloud to read project and token defaults.
• refresh(): Promise<void>: Updates token via gcloud auth print-access-token.

3.2 `Api` (Base Class) Copy Section

• constructor(endpoint: string, pathPrefix: string, context: ApiContext)
• Methods: get, post, patch, delete take relative resource names.
• Polishing:
  • Fully encapsulates Long Running Operation (LRO) polling and returns only the completed result or error.
  • Implements transient error retries with exponential backoff (configurable defaults).
  • Automatically calls context.refresh() on token expiry (401).

5. Technical Stack and Key Dependencies Copy Chapter

• Language: TypeScript.
• Runtime/Build: Bun for standalone binary creation and fast startup.
• Frameworks & Libraries:
  • CLI: cac (minimal size).
  • Schema Validation: zod (TypeScript-first).
  • Parsing: yaml.
  • HTTP Client: Native fetch (in Bun) or node-fetch fallback.

6. Key Technical Decisions Copy Chapter

1. Bun Unified Binary: The CLI and MCP server will be distributed as the same single binary to reduce complexity for the end-user.
2. Thin HTTP Client: We use native raw HTTP requests rather than Google SDKs to control code execution path and support environment workarounds.
3. Validation Before Action: Fresh type definitions are pulled at publishing time, and full validation completes before the first change payload hits the cloud.
4. Fail-Fast Overwrites: Overwrite protection fails fast when remote drift is identified.

Toolbox Mdcode Docs Plan (`toolbox/mdcode/docs/plan.md`) Copy Section

Phased Delivery Plan

This document outlines the phased delivery plan for building the Metadata as Code foundation.

Phase 1: MVP - Read-Only Snapshot & Consumption Copy Chapter

Establish the local file structure by pulling metadata using the TypeScript library, and enable reading that snapshot via MCP. Support early distribution.

• Key Features & Work:
  • Library (TS): Fetch metadata for a BigQuery dataset or Dataplex EntryGroup; create local directory structure mirroring the resource hierarchy; generate main YAML file per entry (Standard layout); paginated pull; ADC authentication.
  • CLI (TS-based): Implement kcmd init and a basic read-only kcmd pull.
  • MCP (TS-based): Implement an MCP server with tools: list-entries and lookup-entry (reading from the local snapshot).
  • Distribution: Support installation from source repository or local package for early access.
  • Testing: Implement test cases for snapshot creation and directory layout for BigQuery datasets and EntryGroups.

Phase 2: MVP - Basic Push (Publish) Copy Chapter

Complete the bi-directional sync by enabling local modifications to be pushed back to the catalog using the TypeScript library.

• Key Features & Work:
  • Library (TS): Read local YAML files and reconstruct API payloads; push updates individually per entry; support parsing and filtering via publishing configuration in catalog.yaml.
  • CLI (TS-based): Implement a basic kcmd push.
  • Distribution: Update early access packages with push capability.
  • Testing: Implement test cases for basic push operation and publishing configuration filtering.

Phase 3: Enhanced Representation Copy Chapter

Optimize the file format for human and agent editing. Update interfaces to support the new formats and layout options.

• Key Features & Work:
  • Library (TS): Support kb scope type alongside bq-dataset and entryGroup in catalog.yaml; implement the CatalogLayout abstraction (layout.ts) and its concrete layouts (StandardLayout and DocumentsLayout); refactor CatalogSnapshot to delegate filesystem list, read, and write operations to the active layout strategy (automatically instantiated based on the scope); support type aliases in catalog.yaml, including built-in aliases for built-in types.
  • CLI (TS-based): Update pull and push operations to handle sidecars, aliases, and different layout scopes correctly.
  • MCP (TS-based): Ensure list-entries and lookup-entry tools correctly handle layouts (by delegating via CatalogSnapshot to CatalogLayout strategy).
  • Testing: Implement test cases for format mapping, including Standard layout (YAML + sidecars) and Documents layout (Markdown + frontmatter), and validate that CatalogSnapshot interacts correctly with both layouts based on the active scope type. Implement test cases for aliases.

Phase 4: Robust Sync and State Management Copy Chapter

Ensure data integrity and efficient updates.

• Key Features & Work:
  • Library (TS): Store checksums of local state in a separate file; use checksums to detect changes; treat missing files as intent to delete; fail fast on remote modification; fetch type definitions dynamically for validation.
  • CLI (TS-based): Add kcmd status to check for local modifications; update kcmd push to use checksums and report conflicts; support force override flag; implement --dry-run option for pull and push commands.
  • Testing: Implement test cases for checksum validation, intent to delete, validation against dynamic schema, and --dry-run behavior.

Phase 5: Future / Other Copy Chapter

Other workstreams to consider.

• Key Features & Work:
  • Library (TS): Support for EntryLinks
  • Python Library: Implement the equivalent library in Python to support Python-based agents and workflows.
  • MCP (TS-based): Implement pull and push tools in the MCP server.

Toolbox Mdcode Docs Semantic Model (`toolbox/mdcode/docs/semantic-model.md`) Copy Section

Deploying a semantic model

This guide has moved into semantic-model/ and is
now split by what you came to do:

• Deploy guide — author a model, push it, update
  it, pull it back.
• Codelab: one semantic ontology, one data journey —
  author, govern, hydrate, and query one model, start to finish.
• Reference — every flag, what push creates in
  BigQuery Graph, Spanner Graph, and Knowledge Catalog (including class
  hierarchies), validation, and permissions.
• Model specification — the normative model
  format: every YAML construct, which parts are Apache Ossie and which are kcmd
  extensions or narrowings.
• What push and pull preserve — the
  round-trip fidelity matrix: what survives a deploy, what a pull recovers, and
  what neither keeps.
• Importing an OWL ontology — convert an OWL
  ontology into a semantic model, then deploy it the normal way.

Toolbox Mdcode Docs Spec (`toolbox/mdcode/docs/spec.md`) Copy Section

Metadata as Code Specification

This specification builds upon the concepts in docs/concept.md and readme.md, detailing the agreed-upon design decisions, critical user journeys, glossary, and test cases for the Metadata as Code project.

1. Overview Copy Chapter

Metadata as Code provides a source code artifact-based UX for metadata management and context engineering in Knowledge Catalog (Dataplex). It is designed for agent builders, data stewards, data producers, and AI agents to author, manage, and enrich metadata using developer-friendly workflows with version control and CI/CD.

1.1. Primary Use Case Copy Section

The primary driver for the initial design is the Knowledge Catalog Enrichment Agent, focusing on automating metadata curation with human-in-the-loop review. However, the library is designed to be flexible and usable by any developer building agents or custom tools.

2. Glossary of Terms Copy Chapter

To ensure consistency, the following terms are used:

• Catalog Snapshot: A directory on the local filesystem containing a subset of Knowledge Catalog metadata.
• Manifest (catalog.yaml): A file in the snapshot root that captures sync configuration and directives.
• Scope: A unified identifier format (<type>.<name>) that establishes the single target resource for the snapshot in the manifest.
• Standard Layout: A hybrid disk organization automatically used for bq-dataset and entryGroup scopes where metadata structure is stored in a YAML file per entry, and unstructured aspects are placed in sidecar Markdown files.
• Documents Layout: A Markdown-only disk organization automatically used for kb scopes where the main entry is represented as a single Markdown file, with metadata structured in the YAML frontmatter and the main overview aspect promoted to the Markdown body.
• Entry: A representation of a resource (like a table or dataset) in the Knowledge Catalog.
• Aspect: A specific type of metadata attached to an entry (e.g., description, profile).
• Sidecar File: An auxiliary file (e.g., Markdown) used for aspects with unstructured text.
• Source: A backend metadata repository referenced by a scope (e.g., Dataplex EntryGroup or BigQuery Dataset).

3. Key Design Decisions Copy Chapter

The following core decisions have been established:

3.1. Metadata Representation Copy Section

The disk organization layout is automatically determined by the scope type:

• Standard Layout (used for bq-dataset and entryGroup scopes): Structured data (entry metadata, resource info, aspects like schema and profile) is stored in a main YAML file per entry, while unstructured rich-text fields (like Overview) are stored in dedicated sidecar Markdown files.
• Documents Layout (used for kb scopes): Structured metadata is stored within the YAML frontmatter of a single .md file per entry, and the primary unstructured aspect (overview.content) is promoted to serve as the main Markdown body of that file. Other unstructured aspects may still use sidecars.

3.2. Synchronization Model Copy Section

We will support a bi-directional sync (Pull and Push) with the following rules:

• Publishing is a subset of the snapshot list.
• While required aspects may be implicitly included in the snapshot, all entry types and aspect types that should be published must be explicitly listed in the manifest.

3.3. Conflict Handling Copy Section

To prevent data loss during a push operation:

• The tool will fail fast if it detects that the metadata has been modified in the catalog in the interim.
• It will abort the push, report the conflict, and require a pull to resolve.
• A force override option will be provided to bypass this check if necessary.

3.4. Aliases Strategy Copy Section

To maintain consistency and steer users toward simpler names:

• If an alias is defined in catalog.yaml, it must be used consistently in metadata files.
• Otherwise, the full reference takes the form of .<locationId>.<resourceId>.

3.5. Scale and Performance Copy Section

• Paginated Pull: Pull operations will be paginated to handle large resources.
• Individual Push: Push operations will be performed individually per entry for now, with bulk import operations considered later.
• Scale Assumption: Users are managing this as code artifacts in a repository and should not be operating at hyperscale/unbounded scale in this workflow.

3.6. Extensibility and Validation Copy Section

• Dynamic Schema Fetching: The tool will fetch type definitions from the Catalog service dynamically to validate local files against the actual registered schema.

3.7. Checksum Storage Copy Section

• Separate State File: Checksums for entries and aspects will be stored in a separate state file (e.g., .catalog.state) to separate user content from tool state.

3.8. Sync Deletions Copy Section

• Intent to Delete: If an entry type is listed in the publish config, treat missing files compared to state as intent to delete.
• Skip for Managed Entries: Deletions are skipped for Dataplex managed entries (like BigQuery resources) which cannot be deleted.

3.9. Unified Scope Mapping Copy Section

• Single String Identifier: Snapshot resources are specified using a unified scope property formatted as <type>.<name> (e.g. bq-dataset. or entryGroup.). This unifies target definition logic and enables future extension to other resource types easily.

4. Critical User Journeys (CUJs) Copy Chapter

We focus on the following core journeys:

• CUJ 1: Initial Setup and Snapshot Creation
  User runs kcmd init to create a local snapshot of a BigQuery dataset or Dataplex EntryGroup.
• CUJ 2: Metadata Sync (Pull and Push)
  User pulls updates from the catalog service, makes modifications to local files, and pushes changes back to the catalog.

Note: Advanced workflows like Automated Curation with Human Review (CUJ 3 candidates) are built on top of this foundation using external tools (IDE, git, custom agents) and are out of scope for this base library.

5. Key Test Cases Copy Chapter

To validate the design, we will implement test cases covering the following areas, including user feedback:

5.1. Resource Types Copy Section

• Snapshots for different resource types: BigQuery datasets and Dataplex EntryGroups.
• Entry and aspect varieties: Both Dataplex-defined entries/aspects (where required aspects are read-only) and user-defined ones (where required aspects are modifiable).

5.2. Format Mapping Copy Section

• Standard Layout (for bq-dataset and entryGroup scopes): Aspects managed correctly within the main YAML file, and unstructured text mapped correctly to/from sidecar Markdown files.
• Documents Layout (for kb scopes): Metadata mapped correctly to/from YAML frontmatter, and the overview.content aspect promoted to/from the Markdown file body.

5.3. Directory Organization Copy Section

• BQ Resource Hierarchy: Testing layout for BigQuery datasets.
• EntryGroup Hierarchy: Testing layout for EntryGroups based on path segments in the entry ID.

5.4. Data Integrity Copy Section

• Checksum validation: Verifying that checksums are updated and validated for entries and aspects during both pull and push operations to detect conflicts.

Samples Enrichment Sample Docs Events (`samples/enrichment/sample/docs/events.md`) Copy Section

Using Analytics Events

Query a specific date range

To query a specific date range from a BigQuery event export dataset, use the _TABLE_SUFFIX pseudo column in the WHERE clause of your query. For more info, view Filtering selected tables using _TABLE_SUFFIX.

For example, the following query counts unique events by date and by event name for a specifc period of days and selected events:

-- Example: Query a specific date range for selected events.
--
-- Counts unique events by date and by event name for a specifc period of days and
-- selected events(page_view, session_start, and purchase).

SELECT
  event_date,
  event_name,
  COUNT(*) AS event_count
FROM
  -- Replace table name.
  `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*`
WHERE
  event_name IN ('page_view', 'session_start', 'purchase')
  -- Replace date range.
  AND _TABLE_SUFFIX BETWEEN '20201201' AND '20201202'
GROUP BY 1, 2;

Samples Enrichment Sample Docs Example1 (`samples/enrichment/sample/docs/example1.md`) Copy Section

High-Level Event and User Volume

This provides examples for kc_enrich_sample_data.ga_events table.

Overview Copy Chapter

Before diving into deep analysis, it is best practice to understand the scale of your dataset. This query counts the total number of events, the total number of unique users (based on device cookies), and the number of days present in the dataset.

SELECT 
  COUNT(*) AS total_events, 
  COUNT(DISTINCT user_pseudo_id) AS total_unique_users, 
  COUNT(DISTINCT event_date) AS total_days
FROM 
  `kc_enrich_sample_data.ga_events`

Samples Enrichment Sample Docs Example2 (`samples/enrichment/sample/docs/example2.md`) Copy Section

Querying a Specific Date Range

This provides examples for kc_enrich_sample_data.ga_events table.

Overview Copy Chapter

Because the GA4 export creates a new partition for every day, you should use the predicate on the partition timestamp column to limit the data scanned. This improves performance and drastically reduces query costs. This query finds the daily count of active users for December 2020.

SELECT 
  event_date, 
  COUNT(DISTINCT user_pseudo_id) as active_users 
FROM 
  `kc_enrich_sample_data.ga_events` 
WHERE 
  event_date_dt BETWEEN PARSE_DATE('%Y%m%d', '20201201') AND PARSE_DATE('%Y%m%d', '20201231') 
GROUP BY 
  event_date 
ORDER BY 
  event_date ASC

Samples Enrichment Sample Docs Example3 (`samples/enrichment/sample/docs/example3.md`) Copy Section

Session Analysis (Unnesting Parameters)

This provides examples for kc_enrich_sample_data.ga_events table.

Overview Copy Chapter

In GA4, a "session" is not a dedicated row; instead, every event contains a nested parameter called ga_session_id. To count total sessions, you must isolate the session_start event and "unnest" its parameters to extract the unique session ID.

SELECT 
  COUNT(DISTINCT value.int_value) AS total_sessions
FROM 
  `kc_enrich_sample_data.ga_events`, 
  UNNEST(event_params) 
WHERE 
  event_name = 'session_start' 
  AND key = 'ga_session_id'

Samples Enrichment Sample Docs Example4 (`samples/enrichment/sample/docs/example4.md`) Copy Section

E-Commerce Product Interest

This provides examples for kc_enrich_sample_data.ga_events table.

Overview Copy Chapter

For an e-commerce store, understanding which items are frequently added to the cart is crucial. E-commerce data is stored in a nested array called items. This query flattens the items array and aggregates the top 10 products based on the add_to_cart event.

SELECT 
  items.item_name, 
  COUNT(*) AS added_to_cart_count 
FROM 
  `kc_enrich_sample_data.ga_events`, 
  UNNEST(items) AS items
WHERE 
  event_name = 'add_to_cart' 
GROUP BY 
  items.item_name 
ORDER BY 
  added_to_cart_count DESC 
LIMIT 10

Samples Enrichment Sample Docs Lipsum (`samples/enrichment/sample/docs/lipsum.md`) Copy Section

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Samples Enrichment Sample Docs Usage (`samples/enrichment/sample/docs/usage.md`) Copy Section

Using Google Analytics 4 BigQuery Public Dataset

This provides usage information for kc_enrich_sample_data.ga_events table.

How the Data Can Be Used Copy Chapter

This dataset is incredibly valuable for analysts, data scientists, and developers who want to learn how to query GA4 data in BigQuery without needing their own fully populated e-commerce site or GA4 export.

Getting Started Copy Chapter

To interact with the dataset, you need:

A Google Cloud Project with the BigQuery API enabled.

You can use the BigQuery Sandbox, which provides a generous free tier of up to 1 TB of querying per month, allowing you to explore the data without needing to attach a credit card.

Primary Applications Copy Chapter

Schema Familiarization: GA4 exports data in a highly nested format (using arrays and structs for event_params, user_properties, and items). This dataset allows you to practice the UNNEST() function to flatten data.

Testing Queries & Data Models: Build and test advanced SQL data models for user retention, funnel analysis, or marketing attribution before applying them to your production data.

Dashboard Prototyping: Connect this public BigQuery dataset to visualization tools like Looker Studio or Tableau to practice building e-commerce and web performance dashboards.

Toolbox Mdcode Docs Semantic Model Actions (`toolbox/mdcode/docs/semantic-model/actions.md`) Copy Section

Modeling write operations

Your semantic model already tells an agent what your data means and which
questions it can ask. Ask that agent to do something — refund an order, move
money between two accounts, close a dormant account — and the model has nothing
to say. The write exists somewhere, as a service call or an endpoint or a few
lines of DML, but it's visible only to whoever built the service around it. Your
agent either can't reach it, or reaches it through a hand-written tool that your
model never sees and can't govern.

An action closes that gap. It's a named write operation declared over the
same concepts as the rest of your model: you give it a name, type its inputs
against your ontology, and say which concepts a call changes. Publishing it puts
the operation in the same place as the data it acts on, so an agent that
discovers your model discovers what it can change there as well as what it can
ask.

One part of an action is physical — how the write actually happens. That part
lives in a field called the executor, which names an MCP tool, a REST
endpoint, a gRPC method, or DML. Keeping it separate lets a binding profile
supply it, so you write an action across two files. Your model names the
operation and its parameters; a binding profile names the tables, the columns,
and the executor. Whatever runs the action combines the two into one bound
model.

graph LR
    M["the model<br>concepts, actions, constraints"]
    P["a binding profile<br>tables, columns, executor, target"]
    RT(["one model, bound<br>ready to run"])
    ST["the store<br>where a write lands"]
    AG["what an agent is handed<br>write tools, instruction"]

    M --> RT
    P --> RT
    RT --> ST
    RT --> AG

Figure 1: your model and a binding profile combine into one bound model, which
both reaches your store and supplies the tools an agent is handed.

The rest of this guide follows one action — moving money between two bank
accounts — from a name in your model through to an agent that can call it. A
second example, a customer-service credit, comes in once the rules get harder to
write.

The commands here are kcmd, which reads a model, pushes it, and derives the
tools an agent is handed under a binding profile. What the model states holds
for any runtime that reads it, and the library kcmd is built on is the same
one a service would embed.

When to use an action Copy Chapter

Declare an action when the write already exists. Somewhere in your organization
there's a service call, an endpoint, or some DML that moves this money or issues
this credit. Declaring it as an action doesn't reimplement any of that — it
tells everything that reads your model that the operation is there, what it
takes, and where it lives.

If the call should hand back an answer instead of changing something, you want a
metric instead.

1. Declare the action Copy Chapter

Start with the name and the parameters. They're the contract every later step
builds on, and they live in your model and nowhere else — no profile can change
them.

Actions sit at model level, beside your metrics. Each one carries a name, its
parameters, and an executor. Treat that executor as a default, because it's the
one part a binding profile can replace:

/* Detailed source-code truncated for AI context efficiency. */

You author the rest of the model — the deployment target, the entity bindings,
the relationships — the way you would for any model. See
Deploying a semantic model.

Both description and ai_context.instructions travel through to the catalog,
and a parameter's description is what tells a caller which argument is which.
Two parameters a caller could confuse must each carry one, or the push fails.
Write the instructions for the agent that's going to call the action, the way
the example does.

Projecting a parameter from a field Copy Section

Every parameter carries one scalar value — the same kind of value a column
holds. What differs is where its definition comes from, and there are two ways
to give it one: project it from a field of your ontology, or declare it
on the spot.

A projected parameter takes its definition from a field

A projected parameter names a concept and a field, and takes that
field's datatype, description, label and AI context straight out of the model.
source above is an Integer because Account.accountId is one, so a caller
passes an account id and the write binds it as an integer. Change the field
later and every parameter projected from it follows, because there's no second
copy to keep in step. A concept is read the way affects reads one, so it can
name a relationship as well as an entity — but what you project from in practice
is an entity, because a relationship only has fields of its own when a junction
table backs it, and this format has no syntax for one yet.

The pair can be written two ways, and they load to the same parameter:

- { concept: Account, field: accountId }
          - { field: Account.accountId }

The one-key form spells the projection the way a metric expression or a
constraint judgment already spells a field, and is there for the same reason.
A field that carries a dot is read as Concept.field, so a field with more
than two parts is refused at load rather than read as half a reference, and so
is a dotted field beside a concept, which would name the concept twice —
that one is refused with both of the forms above spelled out for you.

Most projected parameters need no name of their own. Leave name out and the
parameter answers to the field's, so this one is called accountId:

- { concept: Account, field: accountId }

Name it yourself when one action takes the same field twice, the way
TransferFunds takes both a source and a target.

This shape holds however an entity is keyed. An Account identified by three
columns takes three projected parameters, one per key field, each typed from
the field it names — the same declaration a single-column key writes, three
times.

A declared parameter carries its own type

A declared parameter carries a value no field holds: the amount above, a
free-text memo, a reason code your store never keeps. Give it a type and a
description of your own. type takes a scalar datatype only, and naming an
entity there is an error telling you to project the field you meant. datatype
is accepted as a spelling of the same key — the one fields: already uses —
and stating both at once is refused rather than guessed at.

A projected parameter can't restate its field's type. If the type is wrong,
fix the field. If a statement needs a different one, cast it in the DML, where
the conversion is visible to whoever reads the write instead of buried in
metadata.

What else a parameter may carry

Each parameter may also carry:

• description — what the parameter means for this call. A projected
  parameter inherits the field's, and overriding it is how source and
  target above say different things while projecting one field. Required when
  two parameters on the same action project the same field or share a declared
  type, because neither can tell an agent which argument is which on its own —
  and the inherited wording does not count, since two parameters carrying one
  field's description verbatim are as indistinguishable as two carrying none.
• label and ai_context — inherited the same way, overridden the
  same way.
• default — a fallback value substituted when the caller omits the
  argument. Giving a parameter a default makes it optional; setting
  required: true alongside default is rejected.
• required: false — marks a parameter optional with no fallback; an
  omitted call binds NULL in SQL.

default and required are always yours to set, projected or not. A field
says what a value is; the parameter says how this one call uses it. A
parameter that states neither is required, so the two keys above are the only
ways to make one optional.

Where the executor comes from Copy Section

The executor tells a consumer where the operation lives. Pick one of these four
kinds, and give any one executor a single kind only:

• mcp — {server, tool}. A tool you've already registered in Agent
  Registry, named by the server's resource name and the tool's name within it.
• rest — {endpoint, method}. An HTTP endpoint and the verb to call it
  with.
• grpc — {service, method}. A service and the method on it.
• sql — {statements}. The write itself, carried in your model instead
  of named as a pointer to whoever performs it, and the only kind that needs a
  store to run against at all.
  See carrying the write as DML.

Everything else your action declares is logical: what it takes, what gates it,
what it changes. None of that changes when you deploy the same model somewhere
else. The executor does, which is why it sits on the physical side with an
entity's source, and why a binding profile can supply one or
replace the one your model declares — for any of the four kinds. An action a
profile says nothing about keeps the executor the model gave it.

It changes for two different reasons. An mcp, rest or grpc executor is an
address: the operation lives at a different server, endpoint or service in
staging than it does in production, and where your rows sit never enters it. A
sql executor is the statement, written in the bound store's table and column
names and in its dialect, so until you know which store answers there's nothing
to write down. Only that last case turns on where the data lives.

Actions declared but not performable Copy Section

An action with no executor anywhere — none in the model, none in any profile —
is declared but not performable. It still says what it does, what it takes,
what gates it and what it changes; the only thing missing is who carries the
write out. Write one when nobody has wired the write up yet, or when another
team owns it and your model needs only to record that it exists.

Nothing declares this state: it follows from the executor being absent, so the
same action is performable under a profile that supplies one and not
performable under a profile that supplies none or withdraws the model's with
executor: null. A catalog-only push — --no-profile,
or a model with no deployment target — publishes it like any other action, and
kcmd profiles lists it under cannot run: for each binding that supplies no
executor for it.

Carrying the write as DML Copy Chapter

The first three kinds name a system that performs the write, which leaves the
write itself opaque to your model: an mcp tool name says where the operation
lives and nothing about what it touches. A sql executor carries the write
instead, so what your action does becomes readable — and checkable — from the
bound model, and an agent or tool framework can
run it directly against your store
rather than handing the write to another system to perform.

Statements are written in one database's own table and column names, in its own
dialect, so a sql executor goes in that database's binding
profile — beside the bindings for those same tables and columns —
rather than in the model it binds. Here it replaces the mcp executor the model
declared for TransferFunds:

# payments.profiles/operational.yaml — this store owns the rows, so it writes them
semantic_model:
  - name: payments
    deployment_target: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/bank/graphs/payments
    entities:
      - name: Account
        source: spanner.googleapis.com/projects/my-project/instances/my-instance/databases/bank/tables/account
        fields:
          - { name: accountId, expression: account_id }
          - { name: balance,   expression: balance }
          # ... and the model's remaining entities and fields, bound the same way
    actions:
      - name: TransferFunds
        executor:
          sql:
            statements:
              - UPDATE account SET balance = balance - @amount WHERE account_id = @source
              - UPDATE account SET balance = balance + @amount WHERE account_id = @target
              - INSERT INTO transfer (transfer_id, amount, debited_account_id) VALUES (GENERATE_UUID(), @amount, @source)

A profile is an overlay on the model's bindings: a field it does not restate
keeps the column the model file gives it. So the profile you put a sql
executor in has to restate every column its store names differently, the ones
its statements touch included — leave one out and it keeps the model's column,
which this store may not have. Of the action itself, only the executor is
restated: what the call takes, what gates it and what it changes stay in the
model, exactly as declared.

statements is a list because one business action is often more than one write.
The transfer above debits one account, credits another, and records the
transfer row itself, and a transfer that did only the first would lose money.
Running the statements in the order you wrote them inside one transaction
commits them together at the end, so writes that only make sense together never
apply by halves.

Carrying the write buys you two things:

• Your blast radius is checkable. A reader can compare affects against the
  statements instead of taking it on trust.
• A guard becomes a real gate. An MCP, REST or gRPC call commits inside a
  system your caller doesn't control, so a write it performed can't be rolled
  back if the rest of the action fails. Because a sql action declares its
  statements up front, the agent or tool framework running it can settle its
  guards before opening a transaction, so a refusal leaves the store untouched.

Statements use your database names Copy Section

An action's statements reach your store exactly as you wrote them, so every
table and column in one has to be the name your database uses. In the sql
executor above, the model's Account and accountId appear as account and
account_id. Those are the table and the column that the entity's source key
and its fields' expression keys bind it to.

@parameter references are the exception: they name an argument bound at call
time rather than anything in your database.

Push checks those names for you, against the store itself. Before anything is
published, every statement in a sql executor is sent to the database the
profile binds — Spanner plans it, BigQuery dry-runs it — which parses it,
resolves every table and column, and type-checks every parameter without
executing a row of it. A statement that says accountId where the column is
account_id fails the push:

Error: action 'TransferFunds' in model 'payments'
(payments.profiles/operational.yaml): statement 1 of its sql executor was
rejected by Spanner database 'my-project/my-instance/bank': Unrecognized name:
accountId; Did you mean account? [at 1:54]. Statements reach the store exactly
as written, so every table and column has to be the name the database uses.

Everything between the store's name and the closing sentence is the store's own
answer, which is why it can point at column 54. It also explains the suggestion:
Spanner offers account, the table, because that is the nearest name in scope
where the statement went wrong — BigQuery, dry-running the same mistake, answers
Did you mean account_id? instead. Take the diagnosis from the store you
actually deploy to. That's also why the check is
worth more than comparing your statements against the profile's bindings would
be: the store catches what a name comparison can't see. A placeholder or a
function borrowed from another dialect gets every identifier right and is still
invalid. A parameter whose declared type doesn't fit the column it's compared to
resolves every name correctly and fails on the type.

An entity named Order bound to a table named Orders is the case worth
recognizing, because the answer doesn't mention a table at all:

Syntax error: Unexpected keyword ORDER [at 1:8]

ORDER is a reserved word, so the statement stops parsing before anything goes
looking for a table to write to.

Nothing is written by any of this. Spanner plans inside a transaction that is
never committed, and a BigQuery dry run doesn't execute the statement. What the
check does need is access: it reaches the database your profile names, so a
push from somewhere that can't reach it reports that rather than skipping the
statements.

Which file a `sql` executor belongs in Copy Section

The placement is enforced: select a profile whose model declares a sql
executor and the command refuses, naming the action:

Error: [payments] profile 'operational': action 'TransferFunds' in model
'payments' declares a 'sql' executor. A statement names one database's own
tables and columns, so it belongs in the profile that binds them, not in the
model. Move the executor into each profile that performs this write as DML.

A single-file model is the exception: that one document is both the model and
its binding, so its statements already sit beside the columns they name. Add a
profile beside it later and they move into the profile.

2. Gate it with a constraint Copy Chapter

The sections so far declared an action and gave it a write to perform. This
section is how you stop it running when it shouldn't. A constraint is a
named rule your model states over its ontology. Declaring one adds it to the
catalog and changes nothing by itself. A constraint takes effect where
something references it and nowhere else, so publishing a rule can't quietly
start refusing calls that succeeded yesterday.

declared                referenced             checked            a breach
  ─────────────────       ─────────────────      ──────────────     ───────────
  constraints:            actions:               before the call,   reject
    - name: X       ──▶     - name: Y      ──▶   with the      ──▶  escalate
      judgment: …             guards: [X]        arguments bound    warn

  a rule in the           the only thing that    a language model   on_violation
  catalog, inert          gives it effect        reads the call     names one of
                                                 and settles it     the three

Figure 2: a constraint moves from declared, to referenced by an action, to
checked before a call, to a breach routed by on_violation.

A constraint has one body, judgment: the rule written as a sentence, settled
by a language model reading the attempted call.

Naming a constraint as a guard Copy Section

An action's guards list is what puts a constraint to work. Declare the rule,
then name it on the action — one line on the TransferFunds from section 1:

constraints:
      - name: TransferWithinAvailableBalance
        judgment: >-
          The amount argument of this call must not exceed Account.balance on
          the source account. That balance is on record rather than stated in
          the arguments, and nothing puts it in front of you.
        on_violation: reject
        description: >-
          A transfer cannot move more than the source account holds. Lower the
          amount, or choose another account.
    actions:
      - name: TransferFunds
        guards: [TransferWithinAvailableBalance]

guards holds the names of constraints your model declares, and listing one
there is what makes it apply to that action. Put the reference on the action
rather than on the constraint, because the same rule may gate TransferFunds
and leave CloseAccount alone.

That rule is the honest shape of a balance check and it is also one nothing here
enforces: the balance is a row, a judge is shown the arguments and no more, and
a rule it cannot settle it reports as not holding — which under reject refuses
every call. Declared this way it is a policy on record and a schema constraint
waiting to be written, not a control. Keep reading with that in mind; what a
guard can see explains why.

Whatever dispatches the call is what checks its guards, and it checks every one
of them before the call, with the arguments bound and before any transaction
opens. Nothing in your model binds a rule to the state a write leaves behind, so
a guard reports on the arguments it was handed and on the data the call starts
from, and on nothing else.

guards and on_violation are independent, the way the two right-hand columns
of figure 2 are: one says when the constraint gets checked, the other says what
a breach does. So guarding a constraint that declares warn is a real shape.
Your organization may not be ready to block on a rule; guarding it anyway still
gets the rule checked at the moment of the call and reported back.

When one call violates several guards, the strictest outcome applies. Any
reject refuses the call, failing that any escalate holds it, and failing
that any warn lets it through with the violations reported. The precedence is
fixed and nothing in your model states it, so an action can name any number of
guards without you writing down how they combine. It is also how forbid
overrides permit in Cedar and how a deny wins in Open Policy Agent, so a
policy written this way lowers into either.

A guard that names no constraint fails the push, so a typo can't leave you
believing a write is checked when nothing checks it. The reverse isn't an
error: a constraint no action names is declared, published and inert, which is
a reasonable thing to publish for a reader or for an action you haven't written
yet.

What a judgment costs Copy Section

A judgment is a whole constraint body. Here is one, with the two fields that
say what a breach does and how grave it is:

constraints:
      - name: CreditMemoNamesAServiceFailure
        judgment: >-
          LineItem.memo must name a specific, verifiable service failure on the
          order: a late delivery, a damaged item, a shipping charge applied in
          error. A memo that states only that the customer requested a credit
          does not satisfy this rule.
        description: >-
          Say what went wrong with the order in the credit memo.
        on_violation: warn
        severity: low

Settling that rule is a model call, and one goes out for every guard on every
attempted write. The verdict can also move: a ceiling of 25 dollars, stated as a
judgment, can answer two identical calls differently, where amount <= 25 could
not.

So write a judgment for a rule that has to weigh what a caller wrote, where
nothing shorter would settle it. Two cases are worth keeping out. A threshold
over the call's own arguments is the first: amount <= 25 needs no reading at
all, and a sentence buys you a model call and a verdict that can move.
Leave it in whatever dispatches the call — the model has no home for it today,
which is in what is not modeled yet.

A rule that turns on a stored value is the second. Nothing fetches that value
for the judge, so the sentence gets settled against a figure the model supplied
itself or refused for want of one. Write the rule down anyway if the model is
where your policy lives, and enforce it in your schema — see
what a guard can see.

Every constraint must state on_violation, and any of the three words will do.
Omit it and the push fails, because an unmarked constraint would reject, and
that's too strong a consequence to inherit by silence.

What a guard can see Copy Section

A guard is checked before the call, with the arguments bound and before any
transaction opens. It sees one thing: the arguments it was handed. A rule about
the arguments — whether the memo the caller wrote names a real service failure,
say — is settled completely, because the arguments are the whole of what it
reads.

Nothing settles a rule about a stored value the call doesn't carry. A judge is
given the rule, the action and the arguments, and goes nowhere for anything
else, so the credit must not exceed the total of the order has no total to
compare against. The judge is instructed to treat a rule it cannot settle from
the arguments as one that does not hold and to say in its reason what was
missing — a safe failure, not a working check. Put that rule in your schema,
where the store enforces it inside the transaction.

Nothing settles a rule about the state a write leaves behind, either. "An
order's total equals the sum of its lines" has nothing to look at when the guard
runs, for a second reason: the state it describes does not exist yet. That one
belongs in the same place.

Status: judgment is the only body a constraint has. Whatever checks a
guard puts the sentence to a language model and routes the verdict by
on_violation.

Writing a judgment Copy Section

A language model reads your sentence at review time with the proposed write in
front of it. That reader is the judge. Five habits make that reading
consistent:

1. State what must be true of the data. Write the condition — the memo
   must name a specific service failure, and not the procedure, check whether
   the memo is specific. Your sentence describes a clean write, and everything
   about handling a breach lives elsewhere.
2. Name fields model-qualified. Write LineItem.memo rather than "the
   memo". A push resolves every Entity.field token in the text against your
   model and fails when the entity declares no such field, so a rename can't
   leave your sentence pointing at nothing. It also tells a reader which value
   the rule is about, which the judge cannot go and fetch.
3. Say what doesn't count. A rule with no negative example gets graded
   against whatever the model guesses you had in mind. The sentence "A memo
   that states only that the customer requested a credit does not satisfy this
   rule" buys you more consistency than any further description of what a good
   memo is.
4. Leave the consequence out of the prose. What happens on a breach is
   on_violation. A judgment ending "…otherwise escalate to a supervisor"
   states a routing nothing reads, and the engine routes by the field
   regardless.
5. Keep it to one condition. When your sentence needs "and also", the
   second half is a second constraint. One on_violation can't carry two
   consequences, so two conditions that end differently can't share a
   constraint.

All five are about wording. Claim no more in the wording than a judge can
settle: phrase the condition around the arguments the call carries, and put any
rule that depends on a stored row into your schema.

A credit policy, worked through Copy Chapter

Real policies have several rules, and the rules rarely end the same way. A
support agent is about to issue a customer-service credit, and five separate
rules bear on whether they may:

the business rule                        a breach   settled
  ──────────────────────────────────────   ────────   ────────────────────────
  1  no credit above the order's total     escalate   judgment (policy; schema enforces)
  3  the memo names a service failure      warn       judgment
  4  not one credit split to evade review  reject     judgment
  ──────────────────────────────────────   ────────   ────────────────────────
  2  over 25 dollars needs a supervisor    —          your dispatch, not here
  5  the total equals the line items       —          your schema, not here

Table 1: the three rules of the credit policy that a guard can carry, what a
breach of each one does, and the two rules that belong somewhere else.

Rules 2 and 5 are in the table to be ruled out. Rule 5 is about the state a
write leaves behind, and a guard runs before the transaction opens, so no
constraint you write here reaches it; put it in your schema. Rule 2 compares one
argument against a literal: written as a judgment it would buy a model call to
perform a subtraction, and a verdict that can move at the boundary. Leave it in
whatever dispatches the call. The 25-dollar ceiling still governs the desk, and
rule 4 is written against it, but nothing in the model states it.

Writing the other three down needs an Order with a total, a LineItem with
an amount and a memo, and an IssueCredit action taking the order, the
amount and the memo. Each rule becomes one constraint, and each states its own
outcome in on_violation:

/* Detailed source-code truncated for AI context efficiency. */

Each on_violation key carries one branch of what your policy describes in
prose, in a form a search can read.

Rules 3 and 4 are the reason a constraint is written in words. Neither
reduces to arithmetic over Order and LineItem at all, so there's nowhere
else to put them but a policy document nothing links to.

Rule 1 is arithmetic against a row, and nothing here settles it. It compares
the amount argument against a number that lives in the database, and a judge
sees the arguments only, so the rule is recorded in the model and enforced in
your schema — see
what a guard can see. Rule 2 is the
same comparison against a literal rather than a row, and that difference is the
whole reason it stays out of the model. Rule 4 reads on the credits already
sitting on the order, so it is the same case as rule 1.

Rule 4 is a judgment that declares reject. Splitting a credit to evade
review is a rule the business means as unappealable, and the pairing carries a
real cost, because a model can decide two identical credits differently and
reject leaves nobody to appeal to. The model publishes such a rule instead of
forbidding it, and makes it findable: an auditor asking which unappealable rules
your model settles gets an answer from one query over the constraints.

Two calls through the policy Copy Section

Here are two calls against order 12345, which totals $165.85. One is a
200-dollar credit for a shipping charge billed in error, which is more than the
order was ever worth. The other is three 9-dollar credits raised within the
hour, each memo reading some version of "customer asked":

amount=200.00,         amount=9.00 x3,
                                  "shipping charge       "customer asked"
                                   applied in error"
  ──────────────────────────────  ─────────────────────  ─────────────────────
  1  within the order's total     violated ─▶ escalate   holds
  3  memo names a failure         holds                  violated ─▶ warn
  4  not split to evade review    holds                  violated ─▶ reject
  ──────────────────────────────  ─────────────────────  ─────────────────────
  strictest outcome wins          held for a supervisor  refused, with the
                                                         memo warning reported

Table 2: how each of the two calls fares against the three rules, and the
outcome that wins.

The supervisor who gets the first call reviews a credit against an order rather
than a SQL diff. The second call is the case rule 4 was written for. Every
per-call gate lets it through, because each 9-dollar credit sits under the
order's total on its own, and under the desk's 25-dollar ceiling too, and only
reading the three together as one 27-dollar credit puts them over it.

The bottom row of table 2 applies the strictest-outcome rule from section 2 —
an escalate with nothing stricter beside it holds the first call, and rule 4's
reject decides the second.

Status: a run doesn't compute the strictest outcome. The runtime puts the
guards to the judge in the order your model declares them and stops at the
first one that fails without being advisory. What comes back is that guard's
outcome rather than the strictest of them, and a warn collected on the way
there doesn't travel with the refusal.

3. Say what it changes Copy Chapter

affects names the concepts a call writes to. An executor like
mcp: {server, tool} says where the operation lives and nothing more, so no
reader of your model can see what that tool writes; the blast radius of a call
stays unknown until you declare it here.

affects is a list, and every entry in it takes one of two shapes: a bare
concept name, or a record that also says how the concept changes.

A bare name says the concept is touched, in a way your model doesn't spell
out:

affects: [Account, Transfer]

That's enough to answer which actions can change an account at all, which is
already more than an executor name answers.

A record adds the operation and the fields the call writes:

affects:
          - concept: Account
            operation: modify
            fields: [balance]
          - concept: Transfer
            operation: create
          - concept: TransferDebits
            operation: create

You'll usually have worked some concepts out further than others. The two
shapes go in one list together, so a coarse entry on the ones you haven't costs
you nothing on the ones you have:

affects:
          - { concept: Account, operation: modify, fields: [balance] }
          - Transfer

affects declares the blast radius rather than limiting it. Naming one field
doesn't stop a statement from writing others, and naming two concepts doesn't
stop a call from touching a third; the list records what you mean the call to
do.

What an entry may say Copy Section

concept takes an entity or a relationship, written the same way for either
kind — your model already records which one it is. TransferDebits above is
the edge from the example model, and it sits in the list beside the two
entities. Either way the name has to be one the same model declares.

operation is create, modify or delete, the same three whatever the
concept is, and a fourth word means your document doesn't parse. Leaving it out
gives you the coarse form one entry at a time, covering every operation on that
concept. Two entries on one concept with the same operation are a hard load
error, and a bare entry beside one with an operation warns, because the bare
one already covered what the second narrows.

fields names what a create or a modify writes, which makes which actions
can change Account.balance answerable. A delete takes the whole instance,
so a field named beside one is rejected rather than ignored.

On a relationship, fields means the junction table's own columns, so modify
the grade on an Enrollment is as ordinary a change as modify an order's
total. A plain foreign-key edge carries no columns of its own, and
TransferDebits is one, so naming a field on it is an error — the property you
meant belongs to an endpoint entity.

Name the concept now and refine it later. Writing - concept: Account on its
own says the same thing the bare Account does, and it's written back as the
bare form.

Status: nothing compares affects to what your executor does. kcmd parses
it, checks the concepts against your ontology, publishes it and reads it back,
and no component reconciles the declaration with the statements or the tool
call. Two things beyond those checks read it. Resolving your model through a
binding profile drops any action affecting a concept the profile can't bind, so
that action never reaches the catalog. Publishing an action whose affected
concept has no entry in the same push warns you that the catalog now records a
blast radius naming something it can't resolve.

4. Check it before pushing Copy Chapter

kcmd push --validate-only runs the checks a push runs and deploys nothing. It
loads your model, checks each action against the ontology that same model
declares, asks your store about any DML those actions carry, and prints every
error it finds. A push clears the same gate before it writes anything, so
whatever this reports would have stopped your deploy:

kcmd push --validate-only

Once your document parses, five checks run over every action, and a failure in
any of them stops the push:

• Every parameter has a type the model can settle — projected from a field,
  or a scalar datatype you named. A concept the model doesn't declare, a
  field that concept doesn't have, a type sitting on a projected parameter,
  and a type naming an entity are four different mistakes and report as four
  different errors.
• Parameters a caller could confuse each carry a description. Two
  parameters projecting Account.accountId, or two sharing a declared type,
  otherwise produce identical tool argument documentation, leaving an agent
  guessing which is source and which is target.
• The executor has the fields its kind requires. server and tool for
  mcp, endpoint and method for rest, service and method for grpc,
  at least one statement for sql. Leave one blank and whatever picks the
  action up has nothing to call.
• Every name in guards is a constraint the model declares. This is the
  one that costs you silently — a guard naming a constraint that doesn't exist
  gates nothing, so you believe the write is checked while nothing checks it.
• Every affects concept is an entity or relationship the model declares.
  The list exists to answer which actions change Account. A typo is
  invisible on the page and would drop the action out of that answer.

An action carrying a sql executor clears one more gate, and that one reads
your database rather than your document: every statement is planned against the
store your profile binds, as described above.

A model that breaks all five reports all five, one line each, and deploys
nothing:

Error: action 'TransferFunds' in model 'payments' (payments) has parameter
'amount' typed 'Currency', which is not a scalar datatype
(String/Integer/Decimal/Float/Boolean/Date/Time/DateTime/DateTimeTz/Opaque).
Error: action 'TransferFunds' in model 'payments' (payments) has multiple
parameters projected from 'Account.accountId', so parameters 'source' and
'target' must each have a 'description' of their own to distinguish them.
Error: action 'TransferFunds' in model 'payments' (payments) has an mcp
executor whose 'tool' is missing or blank.
Error: action 'TransferFunds' in model 'payments' (payments) is guarded by
'TransferWithinAvailableBalence', but model 'payments' declares no constraint
of that name.
Error: action 'TransferFunds' in model 'payments' (payments) affects 'Acount',
which is neither an entity nor a relationship this model declares.

TransferWithinAvailableBalence there misspells the
TransferWithinAvailableBalance the model declares, and Acount misspells
Account — the two typos a reader of the document would skim straight past.

kcmd checks the rest of an affects entry just as strictly. Fields beside a
delete are a hard error, and so is a field the concept doesn't declare. An
operation outside create / modify / delete never gets this far — the
vocabulary is closed, so your document doesn't parse at all.

The two checks that read your ontology stand down when a binding profile
resolves your model. Resolving drops entities and relationships the profile
can't bind, so holding affects to the ontology there would fail your deploy
over a concept the profile removed rather than one you mistyped. An undeclared
concept and an undeclared field fall back to the warning the loader already
gave. A catalog-only push reads the author's model whole, so it treats the same
two as hard errors. Fields beside a delete read only the entry, so that one
fails everywhere.

What push holds a statement to Copy Section

kcmd checks a sql executor further than the other three kinds, because it
carries the write rather than a pointer to whoever performs it:

• Write each statement as a single INSERT, UPDATE or DELETE. A
  statement that reads is a query and belongs in a metric; one that reshapes the
  schema isn't an action. A ; anywhere but the end is rejected, because each
  list entry runs on its own and anything after the separator would silently not
  run.
• Every @name a statement binds has to resolve to a parameter your
  action declares. Anything you write another way is left alone — a literal
  passes, and so does a SQL function. Nothing is interpolated into the
  statement text, so an argument can't become SQL.
• You choose where a new row's key comes from. Write GENERATE_UUID()
  into the VALUES list, pass the key in as an ordinary parameter, or, where
  the key column has a default, leave it out and let the store fill it. kcmd
  generates nothing on your behalf, and nothing in affects bears on the
  choice.
• A key that arrives as an argument is a key the caller chooses. An agent
  filling that parameter can name a row that already exists, and nothing in
  kcmd stops it, because the check on statement shape reads only the first
  word. An upsert therefore passes and overwrites. Spanner spells one
  INSERT OR UPDATE and AlloyDB spells one INSERT ... ON CONFLICT. A plain
  INSERT fails on the duplicate instead. Generate the key inside the
  statement unless something about the call needs the caller to pick it.
• Nothing else is available: no control flow, and no statement composed at call
  time. statements is a fixed list in your model, so an action whose body
  arrived with the call would declare nothing, and a gate can't check what was
  never declared.

Every one of those reads the statement text. The last check reads your database:
as described above, the store your
profile binds parses, resolves and type-checks each statement before the push
publishes anything, so a name the database doesn't have stops the push rather
than the first agent to call the action.

5. Push it Copy Chapter

kcmd push

Knowledge Catalog is the one system your action reaches. Each action becomes its
own entry there, parented to the model entry, the same way a metric does. The
entry carries one aspect holding the executor and the typed parameters:

# .../entryGroups/<group>/entries/payments.actions.TransferFunds
entryType: projects/<project>/locations/global/entryTypes/semantic-action
parentEntry: .../entryGroups/<group>/entries/payments
entrySource:
  displayName: TransferFunds
  description: Move money from one account to another.
aspects:
  <project>.global.semantic-action:
    executorKind: mcp
    mcpServer: //agentregistry.googleapis.com/projects/acme-ops/locations/us-central1/mcpServers/payments
    mcpTool: transfer_funds
    parameters:
      - {name: source, type: Integer, concept: Account, field: accountId, description: The account the money leaves.}
      - {name: target, type: Integer, concept: Account, field: accountId, description: The account the money goes to.}
      - {name: amount, type: Float}
    affects:
      - {concept: Account, operation: modify, fields: [balance]}
      - {concept: Transfer, operation: create}
      - {concept: TransferDebits, operation: create}
    instructions: Resolve both accounts before calling.

A projected parameter publishes both halves: the concept and field it came
from, and the type that projection settled on. The reference is what you
wrote and what a pull reads back, so the link into your ontology survives the
round trip. The resolved type is there for a consumer reading the catalog
without your model in hand, which would otherwise have no way to know what to
pass.

affects is published exactly as you wrote it. The entry doesn't record
whether TransferDebits is an entity or an edge. A consumer that needs that
answer reads it off your model, the only place that still answers correctly
after a rename.

Every action in your model gets an entry, whether or not the binding you pushed
can perform it. When that binding gives an action no executor, the entry
publishes without the executor fields, so the catalog records an action that
nothing is yet wired up to run. A push reads the binding named by --profile,
or by default_profile in catalog.yaml when you don't name one.

Remove an action from your document and the next push deletes its entry, because
your model owns the <model>.actions. id prefix. A catalog search can list the
actions in a project by entry type, the way it lists entities or metrics.

semantic-action is a custom entry type, because Dataplex has no built-in type
for an action yet. Run kcmd init --semantic-model once before you push a model
that declares actions, and it provisions the entry type and its aspect type in
your own project.

Publishing an action needs the permission to attach its aspect, on top of the
permissions any push needs — see
Reference → Permissions.

6. Pull it back Copy Chapter

kcmd pull

Pull collects the semantic-action entries under your model entry and rebuilds
each action. The name, the description, the executor, the typed parameters,
guards, affects, and ai_context.instructions all survive the round trip
unchanged. What push and pull preserve lists which parts of a
model survive that trip and which don't.

7. Hand it to an agent (`kcmd skills-generate`) Copy Chapter

You don't hand-write prompts or tool wrappers for each agent framework. Run
kcmd skills-generate against your model and binding profile, and kcmd
writes an Agent Skill — a folder of Markdown files
that Claude Code, Gemini CLI, Cursor, and other skill-compatible agents load
directly:

kcmd skills-generate --out skills

Wrote skills/payments/SKILL.md
Wrote skills/payments/references/transfer-funds.md

The generated skill splits into two layers so an agent only loads the detail of
the action it actually needs:

1. SKILL.md (the router) — loaded when the skill triggers. It carries the
   model's description, a table of actions pointing to references/*.md, how
   the model wants to be used (ai_context.instructions), how to read keys
   from the bound store (Finding a record), and the deployment details for the
   selected profile (Running an action).
2. references/<action>.md (one reference page per action) — loaded on
   demand when the agent picks an action. For TransferFunds,
   references/transfer-funds.md carries:

# TransferFunds

Action `TransferFunds` of the `payments` model. As a tool it is named `transfer_funds`.

Move money from one account to another.

## Arguments

| Name | Type | Required | What to pass |
| --- | --- | --- | --- |
| `source` | integer | yes | The account the money leaves. |
| `target` | integer | yes | The account the money goes to. |
| `amount` | number | yes | How much money to move. |

## How to call it

Resolve both accounts before calling.

## Rules that apply to this call

Each is settled before anything is written, from the attempted call and, where
the rule is about something on record, the record.

### TransferWithinAvailableBalance

On violation: `reject`

> The amount argument of this call must not exceed Account.balance on the source
> account. That balance is on record rather than stated in the arguments, and
> nothing puts it in front of you.

If it does not hold: A transfer cannot move more than the source account holds.
Lower the amount, or choose another account.

## What it changes

| Concept | Operation | Fields |
| --- | --- | --- |
| `Account` | `modify` | `balance` |
| `Transfer` | `create` | `sourceAccountId`, `targetAccountId`, `amount`, `settledOn` |

Where each part of the skill comes from Copy Section

Every section of SKILL.md and references/<action>.md maps directly to a key
in your model or binding profile:

the model                              where it lands in the generated skill
  ─────────────────────────────────      ─────────────────────────────────────────
  name, description, actions[].name ───▶ SKILL.md frontmatter + action router table
  ai_context.instructions           ───▶ SKILL.md "## How this model wants to be used"
  entities[].fields                 ───▶ SKILL.md "## Finding a record" (physical
                                           table & column map for looking up keys)
  actions[]:
    name, description               ───▶ references/<action>.md header & summary
    parameters                      ───▶ references/<action>.md "## Arguments"
    ai_context.instructions         ───▶ references/<action>.md "## How to call it"
    guards (constraints[])          ───▶ references/<action>.md "## Rules that
                                           apply to this call"
    affects                         ───▶ references/<action>.md "## What it changes"

  the binding profile                    where it lands in the generated skill
  ─────────────────────────────────      ─────────────────────────────────────────
  deployment_target                 ───▶ SKILL.md "## Finding a record" +
                                           "## Running an action" (Store)
  actions[].executor                ───▶ SKILL.md "## Running an action" (Executor
                                           & whether each action is runnable)

Table 3: which key in your model or binding profile produces each section of
the generated Agent Skill.

Two sections in SKILL.md come from the generator itself:

• The tool-contract rules in How this model wants to be used — appended
  after your model's ai_context.instructions so every skill states the same
  rules once: never invent an identifier, ask the caller or read the store when
  given a name instead of a key, repeat refusal reasons plainly, and always
  report warnings alongside a successful write.
• Not runnable under this profile in Running an action — when an action
  cannot be run under the selected profile, its reference page is still
  generated in full, and SKILL.md lists the action with the reason why:
  1. The profile supplies no executor (none declared, or withdrawn with
     executor: null).
  2. The executor is sql and the profile binds no store at all.
  3. The action names a non-advisory guard that is undeclared in constraints
     or has an empty judgment.

For all kcmd skills-generate flags (--out, --name, --profile, --force)
and how the deployment-specific section stays isolated to SKILL.md, see
Generating an Agent Skill and the full end-to-end walkthrough in
demo/semantic-model/skill/.

What is not modeled yet Copy Chapter

This is a prototype. Four things you might reasonably expect are absent.

• Nothing checks that a parameter lands in the column it was projected
  from. Projecting source from Account.accountId says what the value is;
  it says nothing about where a statement puts it. Push narrows this to the
  same-type case: the parameter types your action declares are sent with the
  statement, so the store rejects WHERE region = @source when region is a
  string and source projects an integer — see
  statements use your database names.
  What survives is a wrong column of the right type. WHERE branch_id = @source
  passes, because the parameter is logical, the column is physical, and nothing
  joins the two beyond the type they happen to share. Whoever writes a statement
  still owns which column each argument lands in.
• A deterministic check over the call's own arguments has no home. A
  constraint is settled by a model reading it, so amount <= 25 written as a
  guard costs a model call and can come back differently twice — see
  what a judgment costs. The place for a check like
  that is likely the action itself; until there is one, keep it in whatever
  dispatches the call, and whoever wrote the statement owns the correctness of
  what it does.
• Nothing here performs the write. An action becomes a tool in a generated
  skill, and the agent framework holding that skill is what calls it: a sql
  action carries its statements, and an mcp, rest or grpc one carries the
  coordinates of the service that performs it. No code in this repository
  dispatches either.
• Only a sql action needs a store. Its statements are written against the
  database your profile's deployment target names, so a profile binding no
  store lists its sql actions as not runnable and says why. Which backend
  that store is does not matter: Spanner, AlloyDB and BigQuery all execute DML.
  A profile's mcp, rest and grpc actions need no store and are offered as
  tools either way.

Toolbox Mdcode Docs Semantic Model Codelab (`toolbox/mdcode/docs/semantic-model/codelab.md`) Copy Section

Codelab: one semantic ontology, one data journey

A self-contained walkthrough. You author a semantic model, govern it in Knowledge
Catalog, and bind it to physical stores — BigQuery and Spanner — to query it in
each. The model is defined once and every store reads from that single
definition. The expected output is shown after each command so you can check as
you go.

For the deploy mechanics on their own (author, push, update, pull), see the
deploy guide; for every flag and permission, see the
reference.

Setup Copy Chapter

You need the gcloud and bq CLIs, and kcmd on your PATH (build it from
source: see Build). kcmd uses your gcloud
configuration for credentials, project, and region:

gcloud auth application-default login
gcloud config set project "<your-project>"
gcloud config set compute/region "<your-region>"

The commands below reuse a few names. Set them once for your own project:

export PROJECT=$(gcloud config get-value project)  # used in table and graph names
export LOCATION=global                              # Knowledge Catalog entry-group location
export DATASET=datacloud_demo                       # BigQuery dataset + KC entry group
export GRAPH=sales                                  # property-graph name

1. Author the semantic model Copy Chapter

init provisions the Knowledge Catalog entry group and a local workspace:

mkdir -p ~/semantic-model-codelab && cd ~/semantic-model-codelab
kcmd init --semantic-model $PROJECT.$LOCATION.$DATASET
# -> scope: semantic-model.$PROJECT.$LOCATION.$DATASET

Author the model in two parts — this is what lets one model serve many stores:

• The logical model (this step) declares the business: three entities
  (orders, customer, lineitem), the relationships between them, and one
  metric (revenue). It names nothing physical.
• A binding profile then says where each entity reads from and which column
  each field is. You write one for each store the model serves, and the logical
  model itself never changes.

Author by hand Copy Section

Write the logical model. It declares the entities, their fields, the
relationships between them, and the metric:

cat > catalog/EntryGroups/$DATASET/sales.yaml <<'YAML'
version: "0.2.0.dev0/google"
semantic_model:
  - name: sales
    description: Orders, line items, and customers for the codelab
    entities:
      - name: orders
        primary_key: [order_id]
        fields:
          - { name: order_id,    datatype: Integer }
          - { name: customer_id, datatype: Integer }
          - { name: net_amount,  datatype: Decimal }
      - name: customer
        primary_key: [customer_id]
        fields:
          - { name: customer_id, datatype: Integer }
          - { name: name,        datatype: String }
      - name: lineitem
        primary_key: [line_id]
        fields:
          - { name: line_id,  datatype: Integer }
          - { name: order_id, datatype: Integer }
    relationships:
      - name: placed_by
        from: orders
        to: customer
        from_columns: [customer_id]
        to_columns: [customer_id]
      - name: part_of
        from: lineitem
        to: orders
        from_columns: [order_id]
        to_columns: [order_id]
    metrics:
      - name: revenue
        datatype: Decimal
        expression:
          dialects: [{ dialect: BIGQUERY, expression: SUM(orders.net_amount) }]
YAML

The model describes three entities joined by two relationships:

classDiagram
    direction LR
    class orders {
        order_id : integer
        customer_id : integer
        net_amount : decimal
    }
    class customer {
        customer_id : integer
        name : string
    }
    class lineitem {
        line_id : integer
        order_id : integer
    }
    lineitem --> orders : part_of
    orders --> customer : placed_by

Import from an OWL ontology Copy Section

You can also start from an existing OWL ontology instead of hand-authoring this
YAML. kcmd owl import converts an ontology (.ttl) into a semantic model:

• an OWL class becomes an entity;
• a datatype property — a class attribute typed by an xsd: datatype such
  as xsd:integer — becomes a field;
• an object property — one whose values are another class — becomes a
  relationship.

Here is the same sales domain written as an ontology. An ontology is an RDF
graph: classes, properties, and datatypes are all nodes wired together by
labeled arcs. Each property is its own node, pointing to the class it describes
(rdfs:domain) and to the type of its values (rdfs:range). Node color marks
each resource's rdf:type:

graph TD
    classDef cls fill:#dae8fc,stroke:#6c8ebf,color:#000;
    classDef dp fill:#d5e8d4,stroke:#82b366,color:#000;
    classDef op fill:#ffe6cc,stroke:#d79b00,color:#000;
    classDef dt fill:#f5f5f5,stroke:#999,color:#333;

    orders["ex:orders"]:::cls
    customer["ex:customer"]:::cls
    lineitem["ex:lineitem"]:::cls

    order_id(["ex:order_id"]):::dp
    customer_id(["ex:customer_id"]):::dp
    net_amount(["ex:net_amount"]):::dp
    name(["ex:name"]):::dp
    line_id(["ex:line_id"]):::dp

    placed_by(["ex:placed_by"]):::op
    part_of(["ex:part_of"]):::op

    xInt["xsd:integer"]:::dt
    xDec["xsd:decimal"]:::dt
    xStr["xsd:string"]:::dt

    orders -. owl:hasKey .-> order_id
    customer -. owl:hasKey .-> customer_id
    lineitem -. owl:hasKey .-> line_id

    order_id -->|rdfs:domain| orders
    order_id -->|rdfs:domain| lineitem
    order_id -->|rdfs:range| xInt
    customer_id -->|rdfs:domain| orders
    customer_id -->|rdfs:domain| customer
    customer_id -->|rdfs:range| xInt
    net_amount -->|rdfs:domain| orders
    net_amount -->|rdfs:range| xDec
    name -->|rdfs:domain| customer
    name -->|rdfs:range| xStr
    line_id -->|rdfs:domain| lineitem
    line_id -->|rdfs:range| xInt

    placed_by -->|rdfs:domain| orders
    placed_by -->|rdfs:range| customer
    part_of -->|rdfs:domain| lineitem
    part_of -->|rdfs:range| orders

Write that ontology and import it:

cat > /tmp/sales.ttl <<'TTL'
@prefix owl:  <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .
@prefix ex:   <http://example.com/sales#> .

ex:orders a owl:Class ;
    rdfs:comment "A customer order" ;
    owl:hasKey ( ex:order_id ) .
ex:customer a owl:Class ;
    rdfs:comment "A customer who places orders" ;
    owl:hasKey ( ex:customer_id ) .
ex:lineitem a owl:Class ;
    rdfs:comment "A line on an order" ;
    owl:hasKey ( ex:line_id ) .

ex:line_id a owl:DatatypeProperty ;
    rdfs:domain ex:lineitem ;
    rdfs:range xsd:integer .
# order_id and customer_id each apply to two classes; a datatype property with
# more than one rdfs:domain becomes a field on each of them.
ex:order_id a owl:DatatypeProperty ;
    rdfs:domain ex:orders, ex:lineitem ;
    rdfs:range xsd:integer .
ex:customer_id a owl:DatatypeProperty ;
    rdfs:domain ex:orders, ex:customer ;
    rdfs:range xsd:integer .
ex:net_amount a owl:DatatypeProperty ;
    rdfs:domain ex:orders ;
    rdfs:range xsd:decimal .
ex:name a owl:DatatypeProperty ;
    rdfs:domain ex:customer ;
    rdfs:range xsd:string .

ex:placed_by a owl:ObjectProperty ;
    rdfs:domain ex:orders ;
    rdfs:range ex:customer .
ex:part_of a owl:ObjectProperty ;
    rdfs:domain ex:lineitem ;
    rdfs:range ex:orders .
TTL

kcmd owl import /tmp/sales.ttl --out /tmp/sales_osi.yaml --compact

converted 3 classes, 2 object properties, 5 datatype properties
wrote /tmp/sales_osi.yaml

Look at what it produced:

cat /tmp/sales_osi.yaml

version: 0.2.0.dev0/google
semantic_model:
  - name: sales
    description: Imported from OWL ontology http://example.com/sales#
    entities:
      - name: orders
        primary_key: [order_id]
        description: A customer order
        fields:
          - {name: order_id, datatype: Integer}
          - {name: customer_id, datatype: Integer}
          - {name: net_amount, datatype: Decimal}
      - name: customer
        primary_key: [customer_id]
        description: A customer who places orders
        fields:
          - {name: customer_id, datatype: Integer}
          - {name: name, datatype: String}
      - name: lineitem
        primary_key: [line_id]
        description: A line on an order
        fields:
          - {name: line_id, datatype: Integer}
          - {name: order_id, datatype: Integer}
    relationships:
      - {name: placed_by, from: orders, to: customer}
      - {name: part_of, from: lineitem, to: orders}

This is the hand-authored sales model above, reproduced from the ontology: the
same three entities, the same fields and datatypes, the same primary keys, and
the same two relationships. (The importer emits the extended profile — entities:
under version: 0.2.0.dev0/google — the same spelling this codelab's hand-authored
model uses.)

Two things from the hand-authored model are missing, and both are inherent —
an ontology has no way to state either:

• The revenue metric. OWL describes structure — classes, properties,
  relationships — not aggregations, so the importer emits no metrics. Add the
  metric to the model by hand.
• The relationship join columns. An object property states a direction
  (orders → customer), not which foreign-key column joins to which key, so
  each edge arrives as a pure logical edge — from/to only. Add the
  from_columns/to_columns to each relationship by hand, as in the
  hand-authored model above.

Both gaps are logical facts you fill in on the model itself. The physical
facts — the table each entity reads, the column each field maps to — stay out of
the logical model entirely; a binding profile supplies them at
deploy time (step 3), which is why neither model carries a source. kcmd push
publishes the logical model to Knowledge Catalog as-is. For the full
OWL mapping — class hierarchies, unique keys, and the constructs carried as
custom extensions — see Importing an OWL ontology.

The rest of this codelab uses the hand-authored sales model above.

2. Govern it in Knowledge Catalog Copy Chapter

You can govern the model right now. The push writes the logical model straight
to the catalog as entries; you do not need to connect it to any database tables
first.

Preview the plan Copy Section

Preview the plan without writing anything:

kcmd push --validate-only --print

A bare kcmd push governs the logical model in Knowledge Catalog.

Validating semantic model for Knowledge Catalog...
-- Knowledge Catalog --
Knowledge Catalog plan for 'sales' (destination $PROJECT.$LOCATION.$DATASET):
  6 entries:
    - sales (semantic-model)
    - sales.entities.orders (semantic-entity)
    - sales.entities.customer (semantic-entity)
    - sales.entities.lineitem (semantic-entity)
    - sales.metrics.revenue (semantic-metric)
    - sales.files.sales.yaml (semantic-model-file)
  2 schema-join links:
    - sales-placed-by
    - sales-part-of
Validation complete; no changes applied.

Write the entries (available later) Copy Section

Then drop --validate-only to perform the write:

kcmd push

Pushing semantic model (Knowledge Catalog)...
Wrote 6 new and 0 updated Knowledge Catalog entries; linked 2 relationships.

Each entity, the metric,
and the model itself are now governed entries, joined by a schema-join link —
discoverable, access-controlled, and the single definition the rest of your work
reads from. kcmd pull returns the model YAML push stored with these entries,
confirming the round-trip. Physical bindings can be governed here too, alongside
the logical model.

This write needs the semantic-model / semantic-entity / semantic-metric
entry types and write access to the entry group. See
Permissions.

3. Deploy to BigQuery and get reliable insights Copy Chapter

In step 2 you governed the logical model, which created catalog entries but no
tables and no data. To query it, you give the model a physical home in BigQuery.
This takes three steps: write a binding profile that maps each entity to a
BigQuery table, create those tables and load a little data, then deploy the model
to build the graph.

Write the binding profile Copy Section

Write the analytical binding: the BigQuery table each entity reads, the
column each field maps to, and the BigQuery graph to deploy to. The
deployment_target key names that graph. A binding lives beside the model in
sales.profiles/:

# BigQuery deployment target, named by the profile's deployment_target key.
BQ_DS=projects/$PROJECT/datasets/$DATASET
TARGET=bigquery.googleapis.com/$BQ_DS/propertyGraphs/$GRAPH

mkdir -p catalog/EntryGroups/$DATASET/sales.profiles
cat > catalog/EntryGroups/$DATASET/sales.profiles/analytical.yaml <<YAML
version: "0.2.0.dev0/google"
semantic_model:
  - name: sales
    deployment_target: $TARGET
    entities:
      - name: orders
        source: bigquery.googleapis.com/$BQ_DS/tables/orders
        primary_key: [o_orderkey]
        fields:
          - { name: order_id,    expression: o_orderkey }
          - { name: customer_id, expression: o_custkey }
          - { name: net_amount,  expression: net_amount }
      - name: customer
        source: bigquery.googleapis.com/$BQ_DS/tables/customer
        primary_key: [c_custkey]
        fields:
          - { name: customer_id, expression: c_custkey }
          - { name: name,        expression: c_name }
      - name: lineitem
        source: bigquery.googleapis.com/$BQ_DS/tables/lineitem
        primary_key: [l_linekey]
        fields:
          - { name: line_id,  expression: l_linekey }
          - { name: order_id, expression: l_orderkey }
    relationships:
      - { name: placed_by, from_columns: [o_custkey],  to_columns: [c_custkey] }
      - { name: part_of,   from_columns: [l_orderkey], to_columns: [o_orderkey] }
YAML

The binding carries only bindings, in this store's column names: each entity's
table and key columns, each field's column, and each relationship's join
columns. The relationships, the metric and the grain themselves are logical and
live once in the model. Make it the default so a bare kcmd push selects it (the
rest of this step relies on this):

echo 'default_profile: analytical' >> catalog.yaml

Simple case — one binding, one file. If a model only ever binds to one
store, you do not need a separate profile. Put the deployment_target, each
entity's source, and each field's expression directly on the model in
sales.yaml — that is the default profile — and run a bare kcmd push. An
orders entity then reads:

yaml Copy

semantic_model:
  - name: sales
    deployment_target: bigquery.googleapis.com/.../propertyGraphs/sales
    entities:
      - name: orders
        source: bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/orders
        primary_key: [o_orderkey]
        fields:
          - { name: order_id, datatype: Integer, expression: o_orderkey }
          # ...

This codelab keeps the binding in a separate profile so one model can back more
than one store, each with its own binding.

Inspect the binding profile Copy Section

kcmd profiles lists the model's binding profiles, reports what each one can
answer, and prints any error push would give for it. Run it to confirm the
binding before you deploy:

kcmd profiles

Model 'sales' ($DATASET):
  profile 'analytical' (default)
    target: bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/sales
    sources:
      orders -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/orders
      customer -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/customer
      lineitem -> bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/tables/lineitem
    cannot answer: nothing withheld.

Only the analytical profile exists so far, and it binds every field, so it
withholds nothing: the whole model, including the revenue metric, is answerable
under this binding.

Create the tables Copy Section

Now create the tables and load a little data. In a production pipeline, an agent
working from the ontology would generate these tables from raw sources; for this
self-contained codelab, you create them directly. Two details about the data
matter here: the orders table stores net_amount as a column, which the
revenue measure sums, and each order has several lineitem rows:

bq mk -f --dataset $PROJECT:$DATASET

bq query --use_legacy_sql=false '
CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.customer` AS
SELECT * FROM UNNEST([STRUCT(1 AS c_custkey, "Acme" AS c_name),
                      STRUCT(2, "Globex")]);
CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.orders` AS
SELECT * FROM UNNEST([
  STRUCT(100 AS o_orderkey, 1 AS o_custkey,  90.0 AS net_amount),
  STRUCT(101, 1, 200.0),
  STRUCT(102, 2,  40.0)
]);
CREATE OR REPLACE TABLE `'"$PROJECT.$DATASET"'.lineitem` AS
SELECT * FROM UNNEST([                 -- order 100: 2 lines, 101: 1, 102: 3
  STRUCT(1 AS l_linekey, 100 AS l_orderkey), STRUCT(2, 100),
  STRUCT(3, 101),
  STRUCT(4, 102), STRUCT(5, 102), STRUCT(6, 102)
]);'

Why the tables come before the BigQuery push. Unlike the Knowledge Catalog
step, deploying the BigQuery graph validates that every entity's source table
resolves in BigQuery — even under --validate-only — and builds the graph over
these tables. So the tables must exist first. (Step 2 needed none of this: it
governed the logical model, no tables required.)

Deploy the semantic model Copy Section

Now deploy the bound model to BigQuery. The graph backend comes from the model's
deployment target — you don't name it on the command line. You already governed
the model in Knowledge Catalog in step 2, so --no-kc deploys just the graph;
--print shows the generated DDL:

kcmd push --no-kc --print

The generated graph turns each entity into a node table, each relationship into an
edge, and the metric into a measure. Each property is the model's logical name
backed by its bound column — o_orderkey AS order_id — so every query names
order_id, never the physical column (your $PROJECT/$DATASET appear in the
fully qualified names):

Pushing semantic model (BigQuery Graph)...
-- BigQuery Graph --
-- bigquery.googleapis.com/projects/$PROJECT/datasets/$DATASET/propertyGraphs/$GRAPH
CREATE OR REPLACE PROPERTY GRAPH `$PROJECT.$DATASET.$GRAPH`
NODE TABLES (
  `$PROJECT.$DATASET.orders` AS orders
    KEY(o_orderkey)
    PROPERTIES(
      o_orderkey AS order_id,
      o_custkey AS customer_id,
      net_amount,
      MEASURE(SUM(net_amount)) AS revenue
    ),
  `$PROJECT.$DATASET.customer` AS customer
    KEY(c_custkey)
    PROPERTIES(
      c_custkey AS customer_id,
      c_name AS name
    ),
  `$PROJECT.$DATASET.lineitem` AS lineitem
    KEY(l_linekey)
    PROPERTIES(
      l_linekey AS line_id,
      l_orderkey AS order_id
    )
)
EDGE TABLES (
  `$PROJECT.$DATASET.orders` AS placed_by
    KEY(o_orderkey)
    SOURCE KEY(o_orderkey) REFERENCES orders(o_orderkey)
    DESTINATION KEY(o_custkey) REFERENCES customer(c_custkey),
  `$PROJECT.$DATASET.lineitem` AS part_of
    KEY(l_linekey)
    SOURCE KEY(l_linekey) REFERENCES lineitem(l_linekey)
    DESTINATION KEY(l_orderkey) REFERENCES orders(o_orderkey)
);

Deployed 1 BigQuery Graph(s).

View the semantic model Copy Section

The graph is now a resource in your dataset, and the console draws its schema as
a diagram of node and edge tables — easier to read than the DDL above. Print the
BigQuery Studio link to the dataset:

# printf with a single-quoted format string: the `!` are literal, not bash history expansion.
printf 'https://console.cloud.google.com/bigquery?project=%s&ws=!1m4!1m3!3m2!1s%s!2s%s\n' "$PROJECT" "$PROJECT" "$DATASET"

Open the link, expand the $DATASET dataset in the Explorer, and click the
$GRAPH property graph. Its schema renders as a visual graph of the nodes and
the edges that connect them.

Query Semantic Model: Metrics Copy Section

Now ask the same question — revenue by customer — two ways.

Before — hand-written SQL. An analyst who wants revenue "with the line-item
detail" writes the obvious join across all three tables:

bq query --use_legacy_sql=false --nouse_cache '
SELECT c.c_name, SUM(o.net_amount) AS revenue
FROM `'"$PROJECT.$DATASET"'.customer` c
JOIN `'"$PROJECT.$DATASET"'.orders` o    ON o.o_custkey  = c.c_custkey
JOIN `'"$PROJECT.$DATASET"'.lineitem` li ON li.l_orderkey = o.o_orderkey
GROUP BY c.c_name ORDER BY c.c_name'

The result is wrong. Joining in lineitem repeats each order's net_amount
once per line, so revenue is inflated by the line count — Acme's 90 + 200 becomes
90×2 + 200×1 = 380, Globex's 40 becomes 40×3 = 120:

+--------+---------+
| c_name | revenue |
+--------+---------+
| Acme   |   380.0 |   <- should be 290
| Globex |   120.0 |   <- should be 40
+--------+---------+

Nothing errors. The query succeeds and returns a total that is too large. This is
the fan-out trap: any consumer that writes its own join can hit it.

After — through the model's revenue measure. A measure is a symmetric
aggregate: it sums each order once however the graph is traversed, so the
fan-out cannot double-count. You read it by flattening the graph with
GRAPH_EXPAND and wrapping the measure in AGG(), with columns named
<node>_<field>. The query names no table, no join, and no formula:

bq query --use_legacy_sql=false --nouse_cache '
SELECT customer_name AS name, AGG(orders_revenue) AS revenue
FROM GRAPH_EXPAND("'"$PROJECT.$DATASET.$GRAPH"'")
GROUP BY name ORDER BY name'

This one is right:

+--------+---------+
| name   | revenue |
+--------+---------+
| Acme   |   290.0 |
| Globex |    40.0 |
+--------+---------+

Same question, same data: the hand-written join inflates the total, the measure
returns the correct one. revenue is now an enforced, queryable concept. The
join and the formula come from the model rather than from whoever writes the
query, so the correct answer is the default one.

Tips. Use --nouse_cache — GRAPH_EXPAND result caches are not
invalidated by graph edits. To see the exact output column names for a graph:
DECLARE s STRING; CALL BQ.SHOW_GRAPH_EXPAND_SCHEMA("$PROJECT.$DATASET.$GRAPH", s); SELECT s;

Query Semantic Model: GQL Copy Section

Metrics answer aggregate questions. To follow the relationships between entities,
query the graph with graph query language (GQL), which matches a pattern of nodes
and edges. This walks the placed_by edge to count each customer's
orders: