GEMINI.md --- name: contributing description: "End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request." --- # Fabric Contribution Flow Skill This skill defines the end-to-end workflow for contributing to Cloud Foundation Fabric. It supports two entry modes: - **Mode A: Issue Triage & Bug Fix**: You are addressing an assigned or reported GitHub Issue. Start at **Step 1**. - **Mode B: Proactive Development & PR Prep**: You are actively building a feature, refactoring code, or preparing an existing branch for a Pull Request. Jump directly to the **Design Sign-off Gate** before Step 2. If it is unclear which mode applies, ask the user before proceeding. ## Human-in-the-Loop Gates Gate on steps that are hard to reverse, costly, or where your judgment is likely to diverge from a maintainer's. Keep mechanical, reversible steps (linting, tests, doc and inventory regeneration) autonomous. Gates are **blocking**: if you are running non-interactively and cannot get an answer, stop — never assume approval. | Gate | When | What the human decides | | :--- | :--- | :--- | | **Triage Disposition** | End of Step 1 (Mode A) | Whether the issue is in scope and worth pursuing: proceed, reject, or escalate. | | **Design Sign-off** | Before Step 2 | Approves the proposed design and scope. Mandatory for new features and interface changes; skippable for trivial fixes where the design is self-evident. | | **E2E Opt-in** | Step 5 | Whether to run live cloud verification. Providing a sandbox project ID **is** the consent to deploy to it. | | **Behavioral Verification Opt-in** | Step 5, after read-back verification | Whether to also verify runtime behavior of the deployed resources, given the described extra time and cost. | | **PR Approval** | Step 6 | Reviews the final sanitized PR body before submission. | --- ## Step-by-Step Workflow ```mermaid graph TD M1[Mode A: Issue Triage] --> A[1. Triage Issue] A --> GT{Gate: Triage Disposition} GT -->|Human approves| GD{Gate: Design Sign-off} GT -->|Human rejects| X[Stop / Report Back] M2[Mode B: Proactive Dev] --> GD GD --> B[2. Develop Fix / Feature] B --> C[3. Tests, Lint & Inventories] C --> D[4. Pre-Submission Self-Review] D -->|Issues found| B D -->|Clean| GE{Gate: E2E Opt-in} GE -->|Project ID provided| G[5. Live E2E Verification] GE -->|Skipped| GS{Gate: PR Approval} G -->|Failures| B G -->|Verified| GS GS --> F[6. Commit & Submit Sanitized PR] ``` ### Step 1: Triage the Issue (Mode A Only) 1. **Retrieve Issue Details**: Use the GitHub CLI to view the issue context. ```bash gh issue view ``` 2. **Explore the Codebase**: Identify the target module (`modules/`) or FAST stage (`fast/stages/`) that requires modification. 3. **Evaluate Fit & Scope**: Assess whether the issue is relevant for Fabric. Ensure it aligns with Fabric's core philosophy (modules should be lean, composable, and represent a single resource type context). Confirm the change has a sufficiently large scope and represents a generic, valuable addition to the module or FAST stage rather than a highly specific, one-off customization. 4. **Read Provider Documentation**: If the issue involves Google Cloud resources, retrieve and read the documentation for the involved GCP resources or Terraform provider resource/datasource to ensure accurate implementation of its attributes, behaviors, and constraints. * Start from the provider registry, e.g. . * **Check the pinned provider range**: the repo pins the `google` provider in [default-versions.tf](../../default-versions.tf). If the feature you are implementing requires provider functionality released after the pinned minimum version, that is a good reason to bump the pin — do it across the board with `uv run tools/versions.py --provider-min-version --write-defaults`, which rewrites `default-versions.tf` and propagates it to every module and stage `versions.tf` (linting enforces they stay in sync). Never edit version pins in individual modules only. * Only rely on documentation for functionality available in a **released** provider version: docs from the provider's `main` branch may describe unreleased attributes that no released provider accepts at plan time. > [!WARNING] > Do NOT rely solely on the proposed solution, examples, or partial specifications provided in the issue description. Always retrieve and review the complete documentation schema from the official provider registry. > > **Workaround for Registry Page JS/Redirection Errors**: > If the registry page fails to load properly with your URL-fetching tool (e.g. returns "Please enable Javascript" or truncates due to HTML formatting issues), find the source markdown file on GitHub and fetch its raw content using `curl` to a scratch file (the `scratch/` directory is gitignored), using the tag of the provider release whose docs you need (e.g. `v7.40.0`): > ```bash > curl -s https://raw.githubusercontent.com/hashicorp/terraform-provider-google/v7.40.0/website/docs/r/workstations_workstation_config.html.markdown -o scratch/workstation_config_doc.md > ``` > You can then search and view it locally to identify all supported arguments and blocks. > Use this information to make an informed decision on which arguments to support. While 100% coverage is not always necessary or desirable, make a deliberate choice to include all arguments that are useful and align with Fabric's composability and simplicity goals, rather than missing them by omission. 5. **Gate — Triage Disposition**: Present your assessment (target module/stage, fit with Fabric's philosophy, scope evaluation) and let the human decide whether to proceed. If the issue fails the fit test, report your reasoning and stop — do NOT comment on, label, or close the issue yourself; disposition of the issue is a maintainer decision. ### Gate: Design Sign-off (Modes A & B) Before writing code, present the proposed design for approval: * The target module(s)/stage(s) and the shape of the variable interface (new/changed variables, whether `context` support is added). * For provider-surface changes: the explicit list of provider arguments you intend to include and exclude, with reasoning. * Whether the change requires bumping the pinned provider version (a repo-wide change — see Step 1.4). This gate is **mandatory for new features and interface changes**. For trivial fixes (typos, one-line bugfixes with an obvious solution) where the design is self-evident, state your intent and proceed without blocking. ### Step 2: Develop the Fix or Feature (Modes A & B) 1. **Align with Fabric Zen**: Ensure the design focuses on composition, encapsulates logical entities (like IAM and log sinks directly inside modules), adopts common interfaces, and keeps code flat and easy to evolve. > [!IMPORTANT] > You MUST strictly follow the design principles and coding conventions defined in [CONTRIBUTING.md](../../CONTRIBUTING.md) and [GEMINI.md](../../GEMINI.md) when designing variables, modules, and factories. 2. **Set Up the Git Branch**: * If you are not already on a feature branch, create one named after your username and the feature/fix (e.g., `/`, such as `johndoe/feature-name`): ```bash git checkout master && git pull git checkout -b / ``` * If preparing an existing branch (Mode B), do NOT create a new one; instead make sure it is up to date with `master` (rebase or merge) before submission. * Direct branches require push access to the repo (granted to regular contributors on request). Without it, use the fork-based flow described in [CONTRIBUTING.md](../../CONTRIBUTING.md); the rest of this workflow is identical. 3. **Apply Fabric/FAST Design Conventions**: * **Context Interpolation**: If context support is relevant and needed for the module (e.g., to support resolving symbolic references like `"$project_ids:myprj"`), add a `context` variable block and implement `ctx`/`ctx_p` locals in `main.tf`. Do not add it blindly if the module does not benefit from symbolic interpolation. * **Compact Variables**: Leverage objects with `optional()` attributes to keep user interfaces clean. Prefer `optional()` **without** an explicit default so the provider applies whatever default value it defines; only set a default when module logic needs a known value. * **Stable State Keys**: Always use maps instead of lists for collection variables to avoid index shifts in Terraform state. * **Scope Isolation**: Use private locals (prefixed with `_`) for intermediate transformations, reserving module-level locals for values referenced by resources. * **Locals Separation for Complex Manipulations**: When adding complex conditional strings, string interpolations, or transformations to resource attributes, compute the map in `locals` so the resource block references a clean index like `local.my_map[each.key]`. This keeps resource blocks legible (<79 chars) and avoids cluttering resource definitions. * **Validate ENUM Variables**: When adding or exposing variables that mirror ENUM values in the underlying Terraform provider, always include a `validation` block to check for allowed values at plan time. Example from `modules/gcs`: ```hcl variable "storage_class" { description = "Bucket storage class." type = string default = "STANDARD" validation { condition = contains(["STANDARD", "MULTI_REGIONAL", "REGIONAL", "NEARLINE", "COLDLINE", "ARCHIVE"], var.storage_class) error_message = "Storage class must be one of STANDARD, MULTI_REGIONAL, REGIONAL, NEARLINE, COLDLINE, ARCHIVE." } } ``` 4. **Reference Implementations & Knowledge Sources**: * [compute-vm](../../modules/compute-vm): Ideal reference for compact variable design, optional objects with defaults, and map variables. * [project](../../modules/project): Best reference for logical entity encapsulation (IAM, log sinks, Shared VPC configurations), complex locals transformation, and context-based interpolation. * [adrs/](../../adrs/): Architectural Decision Records documenting the rationale behind key design patterns (e.g. the IAM ADR explains the full `iam*` interface). Before designing, build an index of all ADRs with their title and status: ```bash find adrs -name '*.md' ! -name 'README.md' | sort | while read -r f; do st=$(awk '/^## Status/ {getline; while ($0 == "") getline; print; exit}' "$f") printf '%s | %s | %s\n' "$f" "$(head -1 "$f" | sed 's/^# //')" "$st" done ``` Read in full any ADR whose title relates to the area you are changing. Weigh them by status: ignore `Rejected` ones, treat `Draft`/`Proposed` as directional rather than settled, and remember that even implemented ADRs can lag behind the code — on conflict, the current code and golden-path modules are the source of truth. 5. **Maintain Code Consistency**: * Always keep variables in `variables.tf` and outputs in `outputs.tf` in strict alphabetical order. For larger modules, variables and outputs can be grouped logically into separate files (e.g. `variables-logging.tf`), each internally alphabetically sorted. * Limit line length to 79 characters (relaxed for long attribute names and descriptions). * **JSON Schema & Factory Alignment**: When a change is made to a module that implements a factory (e.g., `cloud-workstations`) or is the base for one (e.g., `net-vpc-factory` to `2-networking`), the schemas MUST be updated (including their `.schema.md` documentation files), and the factory code should be updated to mirror the changed variable surface. Ensure you regenerate the schema documentation by running: ```bash uv run tools/schema_docs.py ``` * **Duplicated Files**: Some files (e.g. schemas, factory test fixtures) are intentionally duplicated across modules and stages, and CI enforces that copies stay identical. When changing such a file, sync all its copies; the duplicate groups are listed in [tools/duplicate-diff.py](../../tools/duplicate-diff.py) — update that list when adding new duplicated files. 6. **Update or Add Tests**: * For modules: If the change introduces a new feature or configuration option, update an existing example in the module's `README.md` (or add a new one) to demonstrate its usage. Ensure the example includes `# tftest` parameters and update the corresponding inventory YAML files under `tests/modules//examples/`. * For FAST stages: Update `tftest.yaml` scenarios and tfvars/yaml inventories under `tests/fast/...`. > [!NOTE] > Module directories use hyphens (`modules/net-vpn-ha`) while test directories generally use underscores (`tests/modules/net_vpn_ha`). Always check the actual test directory name — a mistyped path makes `pytest` silently collect zero tests. ### Step 3: Run Tests, Linting & Inventory Regeneration (Modes A & B) Run Python tooling through `uv run` — the scripts in `tools/` declare their dependencies inline, so no virtual environment setup is needed. For faster Terraform testing, always set `TF_PLUGIN_CACHE_DIR=/tmp/tfcache`. 1. **Run Unified Linting**: Execute `tools/lint.sh` to check copyright boilerplates, Terraform format (`terraform fmt`), alphabetical sorting, and schema validations: ```bash uv run --with-requirements tools/requirements.txt tools/lint.sh ``` 2. **Update Documentation**: If you changed variables or outputs, check consistency and regenerate the README documentation tables: ```bash uv run tools/check_documentation.py modules/ uv run tools/tfdoc.py --replace modules/ ``` 3. **Run Impacted Tests**: Execute `pytest` on the target module/stage (using the plugin cache directory for speed): ```bash mkdir -p /tmp/tfcache # tftest.yaml scenarios (note: underscores in the test directory name) TF_PLUGIN_CACHE_DIR=/tmp/tfcache uv run --with-requirements tests/requirements.txt pytest tests/modules/ # README example tests TF_PLUGIN_CACHE_DIR=/tmp/tfcache uv run --with-requirements tests/requirements.txt pytest -k 'modules and :' tests/examples ``` 4. **Regenerate Test Inventories**: If module-level tests (`tftest.yaml`) or README example inventories fail due to intentional plan output changes, regenerate them using `generate_plan_summary.py`: ```bash # For module-level tftest.yaml inventories: uv run tools/generate_plan_summary.py tests/modules//tftest.yaml --save # For README.md example inventories: uv run tools/generate_plan_summary.py modules//README.md "" --save ``` > [!CAUTION] > `--save` overwrites the assertion baseline, so a regenerated inventory makes the test pass **by construction**. After regenerating, diff the inventory against the previous version (`git diff tests/`), verify that every changed line maps to the intended feature or fix, and report the inventory diff to the user. If a changed line is not explained by your change, treat it as a bug in your code, not a baseline to save. ### Step 4: Pre-Submission Self-Review (Modes A & B) Review your own diff (`git diff HEAD`, or `git diff master...HEAD` for an existing branch) against the repository guidelines (`GEMINI.md`, `CONTRIBUTING.md`) before submission. This is a pre-flight self-check; the canonical automated review runs in CI after the PR is opened (`Automated PR Review` workflow) — do NOT imitate its output format or header. Check the diff against this checklist: * [ ] Naming conventions respected (variables, outputs, resources). * [ ] Context support (`ctx` variables) present where the module warrants it. * [ ] IAM follows the standard `iam*` interface patterns. * [ ] Examples with `# tftest` parameters cover the new/changed behavior. * [ ] **CRITICAL TESTING RULE**: if resource blocks were modified (adding a new argument or modifying an existing one), the Terraform plan output changes — the corresponding test inventory YAML files (`tests/.../*.yaml`) MUST be updated in the diff. If they are not, this is a critical testing failure. * [ ] JSON schemas and factory code consistent with the changed variable surface. * [ ] Documentation tables regenerated (`tfdoc`), alphabetical order preserved. * [ ] **Breaking Changes**: If any variable or YAML interface changed in a backwards-incompatible way, verify the PR description draft includes the `**Breaking Changes**` / `upgrade-note` block. Report the outcome as a short plain-text list of issues found (or "no issues found") under a `Pre-submission self-review` heading — no emojis, no status tables. Fix any issue and loop back to Step 3 until the checklist passes. ### Step 5: Live Verification — E2E Sandbox & Policy Troubleshooter (Modes A & B — Optional / Recommended) When code modifications affect GCP resource structures or APIs, run a live E2E sandbox deployment test. Run it only once the diff is stable (tests and self-review pass) to avoid repeated cloud deployments. 1. **Gate — E2E Opt-in**: Ask the user for a GCP project ID (and if applicable, parent folder / billing account details) for E2E sandbox testing, stating explicitly that providing the project ID authorizes you to deploy and destroy resources in it with `-auto-approve`. If no project is provided, skip this step entirely. 2. **Verify Credentials**: Confirm `gcloud` authentication and Application Default Credentials are available (`gcloud auth list`) before attempting any deployment, and surface a clear error rather than failing mid-apply. 3. **Create Sandbox Directory**: Create a temporary sandbox folder under `scratch/e2e_sandbox/` (the `scratch/` directory is gitignored). 4. **Generate Test Configuration**: * Generate a root Terraform module `main.tf` in the sandbox directory. * **CRITICAL**: The `source` argument of the module call MUST point to the **local path** of the modified module in the repository (e.g. `source = "../../modules/"`), NOT the GitHub reference, to ensure your local changes are tested. * Set up necessary providers and variables. 5. **Deploy**: * Run `terraform init` and `terraform apply -auto-approve` in the sandbox folder. * Confirm that all resources are created successfully. A clean apply is the baseline, not the goal: it only proves the API accepted the request, not that the change behaves as intended. 6. **Read-Back Verification (required)**: * Verify the deployed state through the service's read API (e.g. `gcloud compute backend-services describe`, or the equivalent `GET`/`describe` surface for the resource), not just the Terraform state or plan output. * For every field or block touched by the change, confirm the live resource contains the intended value and structure (e.g. one API list entry per input element, correct nesting, no silently dropped attributes). 7. **Behavioral Verification (optional — Gate)**: * **Gate — Behavioral Verification Opt-in**: after read-back verification passes, explicitly ask the user whether to also verify runtime behavior, describing what the check would entail for this specific change (resources involved, expected extra time such as propagation delays, any additional cost). Proceed only on explicit approval; skipping is a valid outcome and must be recorded in the PR body. * Verify the change behaves as expected from the perspective of a consumer of the deployed infrastructure. Examples: * Compute instances: connect to the instance (SSH/serial) and check it is reachable and its startup configuration behaves as expected. * Networking, load balancers, DNS: send traffic through the deployed path and verify routing, reachability, and response attributes (e.g. `curl` through a load balancer and inspect status and headers, resolve records, test connectivity across peerings or VPN tunnels). * Serverless and workloads (Cloud Run, Cloud Functions, GKE): invoke the deployed service and verify the response. * Data services (GCS, BigQuery, Pub/Sub): perform a representative read/write operation (upload an object, run a query, publish and pull a message). * Account for propagation delays (load balancers, CDN, IAM can take several minutes); retry before concluding failure. * **IAM conditional bindings**: use Google Cloud's official IAM Policy Troubleshooter as the behavioral check (`gcloud policy-troubleshoot iam`) to verify runtime evaluation against live target resources. * **CRITICAL**: GCP IAM runtime evaluates `resource.name` using numeric Project Numbers. Always test conditions against `--resource-name` with the numeric Project Number: ```bash gcloud policy-troubleshoot iam //logging.googleapis.com/projects//locations/global/buckets/ \ --permission=logging.buckets.write \ --principal-email= \ --resource-name=projects//locations/global/buckets/ \ --format="json(access,explainedPolicies[].bindingExplanations)" ``` 8. **Destroy Resources**: * Once verified, run `terraform destroy -auto-approve` to tear down all created resources and avoid ongoing cloud costs. * Delete the sandbox directory. * If verification surfaced failures, fix the code and loop back to Step 3 before re-deploying. ### Step 6: Commit & Submit the PR (Modes A & B) 1. **Sanitize Before Committing**: PII sanitization applies to **everything that leaves your machine** — commits, file contents, and the PR body. Never commit files containing real GCP project IDs, numeric project numbers, personal email addresses, or custom resource names from live testing; verify `git status` shows no stray sandbox or scratch files before staging. 2. **Commit and Push**: Stage the relevant files, commit with a short imperative message describing the change, make sure the branch is up to date with `master` (rebase if needed), and push: ```bash git add git commit -m "Add to modules/" git push -u origin / ``` 3. **Format the PR Title**: Do NOT use Conventional Commits format (no `feat:` or `fix:` prefixes). Use a short, capitalized, imperative title (e.g., "Add native tag bindings support to `modules/net-firewall-policy`"). 4. **Write the PR Body**: * Explain the problem, rationale, and the fix clearly. * **Document Verification & E2E Testing Methodology**: Detail local unit tests and, for E2E runs, the apply, read-back, and behavioral verification results (or note that behavioral verification was skipped) so reviewers can see the exact testing rationale and methodology. * **CRITICAL PII SANITIZATION**: Before writing the PR description, **MUST scrub all developer PII** (real GCP project IDs, numeric project numbers, personal email addresses, usernames, and custom bucket/resource names) and replace them with generic placeholders (e.g., `my-project`, `123456789012`, `user:tester@example.com`, `test-bucket`). * **Breaking Changes & Upgrade Notes**: Whenever a change requires users to upgrade their variable or YAML surface (e.g. variable type changes, deleted attributes, changed schema structures), you **MUST append** a `**Breaking Changes**` section with an `upgrade-note` code fence at the bottom of the PR description. This is automatically parsed by `tools/changelog.py` when generating release notes. Format: ````markdown **Breaking Changes** ```upgrade-note `modules/organization`: `custom_roles` variable type changed from `map(list(string))` to `map(object({...}))`. Existing code passing permission lists directly must be updated to pass `{ permissions = [...] }`. ``` ```` * Do NOT touch `CHANGELOG.md`: release notes are generated from PR labels and `upgrade-note` blocks by maintainers using `tools/changelog.py`. 5. **Gate — PR Approval**: Present the final sanitized PR title and body to the user and get explicit approval before creating the PR. 6. **Create the PR**: * **CRITICAL PITFALL**: Do NOT pass the body inline on the CLI if it contains backticks (e.g. `gh pr create --body "Fixes `bug`"`) as the shell will evaluate backticks, corrupting the description. Do NOT use shell redirection or heredocs to create the body file either (see `GEMINI.md`). * Instead, write the body to a temporary file **using your file-writing tool**, then reference it: ```bash gh pr create --title "Your PR Title" --body-file /tmp/pr-body.md ``` # Cloud Foundation Fabric (CFF) Conventions for Module Consumers When generating Terraform code that consumes Cloud Foundation Fabric modules, you MUST adhere to the following conventions: ## 1. Module Preference - **Prefer Modules:** Always prefer CFF modules over raw `google_*` resources. - **Flat Structure:** Avoid creating wrapper modules or nested module calls (modules calling other modules). Consume CFF modules directly in your root module. - **Module Sources:** Ask the user whether they want to use relative local paths or absolute paths referencing the GitHub repository. - Relative local paths: `source = "./modules/project"` - GitHub repository paths: `source = "github.com/GoogleCloudPlatform/cloud-foundation-fabric//modules/project?ref={VERSION}&depth=1"` (include version ref and `depth=1`). - All modules used must specify the same version tag. By default, unless the user states otherwise, default to using the latest release tag (obtainable via `python3 scripts/fabric.py release`). ## 2. Naming Conventions - **Use `prefix`:** For modules that support it (e.g., `project`, `gcs`), using a `prefix` variable is recommended but not mandatory. - **Deterministic Naming:** Prefer using structured, deterministic tokens rather than random strings. ## 3. Dependency Management - **Output-to-Input:** Pass outputs from one module directly as inputs to another (e.g., `network = module.vpc.name`). - **Ordering:** CFF modules encapsulate dependencies (like API activation) in their outputs. Rely on these outputs to ensure correct creation order. - **Service Agents:** Use the `service_agents` output from the `project` module when granting IAM roles to Google-managed service accounts. - **Implicit Dependencies:** Avoid explicit `depends_on` unless absolutely necessary. Rely on implicit dependencies (passing outputs to inputs) for better readability and maintainability. ## 4. Factories - **When to Use Factories:** Use factories when you need to manage a large number of similar resources (e.g., projects, VPCs, firewall rules) without duplicating module blocks. Factories separate configuration data (in YAML files) from Terraform logic. - **Main Modules with Factory Support:** - `project-factory`: For bulk creation of projects, folders, and budgets. - `net-vpc-factory`: For bulk creation of VPCs and associated resources. - `net-vpc`: Supports loading subnets and internal ranges from folders via `subnets_folder` and `internal_ranges_folder` keys in `factories_config`. - `net-vpc-firewall`: Supports loading firewall rules from a folder via `rules_folder`. - `organization` and `folder`: Support loading organization policies and custom roles. - `vpc-sc`: Supports loading access levels and service perimeters. - **Usage:** Pass the path to the directory containing YAML files to the `factories_config` variable of the respective module. ## 5. Style for Root Modules - **File Structure:** Make file structure dependent on size. For small configurations, use a single `main.tf`. For larger configurations, split into multiple files grouped by resource type (e.g., `main.tf` for general elements, `networking.tf`, `compute.tf`, etc.). - **Variables & Defaults:** - Define all variables in `variables.tf`, sorted alphabetically. - Set defaults directly in the `default` attribute if a reasonable default exists or if provided by the user. - Avoid creating a `terraform.tfvars` file unless explicitly requested by the user. - **Value Handling & Providers:** - **No Hardcoded Values:** Never use hardcoded values for project IDs, folder IDs, or other specific identifiers unless provided by the user. If a value is required, ask the user for it or create a variable. - **Provider Attributes:** Do not configure provider-level attributes like `project`, `region`, or `zone` in provider blocks (`provider "google"`, `provider "google-beta"`). All attributes must be passed to the modules explicitly in the module calls. - **Formatting:** Adhere to standard Terraform formatting and keep line lengths readable. Wrap complex ternaries in parentheses. - **No Local Exec:** Never use `local-exec` or similar provisioners to run shell commands. Rely on native Terraform resources and data sources, preferably from official providers (i.e., no third-party providers). ## 5a. Version Constraints - **Omit Root Version Constraints:** Do not include version constraints (for Terraform or providers in `required_providers` or `required_version`) in the root module. - **Automatic Resolution:** Let Terraform automatically compute and resolve the required provider and Terraform versions based on the constraints defined within the consumed CFF modules. ## 6. Impersonation and Backend Management - **Impersonation:** If the user requires service account impersonation, add the `impersonate_service_account` attribute to the `provider "google"` and `provider "google-beta"` blocks in `providers.tf`. - **Remote Backend:** If the user wants to use a remote backend, prefer `gcs`. Put the backend configuration in the `terraform` block inside `providers.tf`. If impersonation is used, also set it for the backend. **Template for `providers.tf`:** ```terraform terraform { backend "gcs" { bucket = "${bucket}" impersonate_service_account = "${service_account}" } } provider "google" { impersonate_service_account = "${service_account}" } provider "google-beta" { impersonate_service_account = "${service_account}" } ``` --- name: fabric-builder description: Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud. --- # Fabric Builder This skill generates idiomatic Terraform code using Cloud Foundation Fabric (CFF) modules, following established best practices and conventions. ## Core Workflow 1. **Understand Request:** Identify the GCP resources and relationships requested by the user. 2. **Fetch Module Info:** Identify the relevant CFF module(s) from the `modules/` folder on GitHub (`GoogleCloudPlatform/cloud-foundation-fabric`). 3. **Generate Terraform:** Output an idiomatic Terraform root module that consumes the CFF modules according to [conventions.md](references/conventions.md). ## Important Guidance - **Conventions & Best Practices:** Consult [conventions.md](references/conventions.md) for guidelines on how to consume Fabric modules and write high-quality, idiomatic Terraform code (including module sources, version constraints, and provider configuration). - **Fetching Modules:** Do not invent module inputs or outputs. Use the `fabric.py` script to pull exact details (README, variables, examples) for a specific module from GitHub before using it. - To list available modules: `python3 scripts/fabric.py modules` - To fetch details for a module (README): `python3 scripts/fabric.py fetch readme ` - To fetch variables files: `python3 scripts/fabric.py fetch variables ` - To fetch outputs files: `python3 scripts/fabric.py fetch outputs ` - To fetch schema files (useful for factories): `python3 scripts/fabric.py fetch schemas ` - To fetch the latest release version: `python3 scripts/fabric.py release` - **SSL Certification Verification**: If you run into SSL certificate issues on your host machine, use the `--no-ssl-verify` flag (e.g., `python3 scripts/fabric.py --no-ssl-verify fetch readme `). ## Guidelines for Output - **Root Module Output:** Your output must be a complete Terraform root module that calls the appropriate CFF modules to fulfill the user's requirements. - **Follow Conventions:** Strictly adhere to [conventions.md](references/conventions.md) for module sources, version constraints, provider configuration, and coding style. - **Use Fabric Modules:** Rely on CFF modules instead of raw `google_` resources whenever possible. - **Example-based Learning:** Always refer to the module's README (fetched via `scripts/fabric.py`) for correct usage examples. - **Formatting & Validation:** Ensure the generated code is properly formatted. If possible, run `terraform fmt`, `terraform validate`, and `terraform plan` to check your work. # Import maturity matrix Maturity ladder — every resource family is at exactly one level: | Level | Meaning | Exit criterion | |---|---|---| | **N** — named | Appears in an example manifest; no mapping rules yet | — | | **C** — cookbook | Mapping rules + import IDs + module addresses derived from module source; not yet plan-tested live | cookbook section exists | | **V** — verified | Clean imports, both gates green, address table confirmed live against real Google Cloud resources | verified across live rounds | Verification target: everything in the four FAST-stage example manifests at **V**. Maturity decays — every V carries the provider and Fabric ref it was verified against; re-verify on bumps (see "Regression habit"). ## Matrix | Family | Module | Level | Status / Notes | |---|---|---|---| | Org IAM + audit configs | organization | **V** | Seeded audit configs, `${` escaping verified | | Org policies (incl. dry-run, parameters, custom constraints) | organization | **V** | Three-stream denominator enumeration verified | | Custom roles | organization | **V** | From-scratch and modified roles verified | | Org log sinks | organization | **V** | Destinations graduated to typed module references | | Folders + folder IAM | folder (per instance) | **V** | Parent chaining and in-place rename verified | | Projects, services, project IAM | project (per instance) | **V** | Project services and IAM bindings verified | | Service accounts + impersonation IAM | iam-service-account | **V** | Leaf-IAM opt-in verified | | VPC networks / subnets / routes | net-vpc | **V** | ForceNew alignment and subnet address mapping verified | | Log buckets (sink destinations) | logging-bucket | **V** | Reference graduation verified | | GCS buckets | gcs | **V** | State buckets, ForceNew project ID prefix verified | | BigQuery datasets | bigquery-dataset | **V** | Datasets, import IDs, `description = null` alignment verified (tables/views/routines unverified) | | Pub/Sub topics | pubsub | **V** | Topics, import IDs, sink reference graduation verified (subscriptions/schemas unverified) | | DNS zones | dns | **V** | Private zones, VPC reference link verified | | DNS policies / response policies | dns | **N** | Named in `examples/import-manifest.fast-networking.yaml`; no mapping rules yet | | NCC spokes (linked-VPC) | raw resource — **capability gap**: no Fabric module (`ncc-spoke-ra` is router-appliance only) | **V** (raw) | Upstream-issue candidate, lift with `moved {}` when a module appears | | Cloud Routers / NAT | net-cloudnat | **V** | Integrated router + NAT verified | | Firewall policies | net-firewall-policy | **C** | Mapping rules documented in cookbook (Hierarchical, Global, and Regional Network policies) | | VPN / Interconnect attachments | net-vpn-*, net-vlan-attachment | **C** | Mapping rules documented in cookbook | | KMS keyrings / keys + key IAM | kms | **V** | Seeded keyring, key, leaf-IAM on CryptoKey verified; permanent residue documented | | Certificate Authority Service | certificate-authority-service | **V** | CA pools: CAI type, import ID, pool address (`google_privateca_ca_pool.default[0]`), `ca_configs = {}` default-CA trap, and `publishing_options` support verified on Fabric ref containing PR #4107 (commit `a153861aae`, released in `v58.0.0`). Note: on releases lacking #4107 (`v57.0.0` and earlier), CAS reverts to capability gap | | VPC-SC policy / perimeters | vpc-sc | **V** | Live AccessPolicy adopted, AccessLevel, dry-run ServicePerimeter; bare-numeric policy import ID verified; `unknown`-level classification handled in `inventory.py`. Enforced perimeters (`status` block) unverified | | Tags (keys/values/bindings) | organization / folder / project tags | **V** | TagKey, TagValue, TagBinding, import IDs, address shapes, verified at organization, folder and project level. The `templatestring` escaping matrix (`$$${`) is DERIVED FROM MODULE SOURCE, not exercised — no seeded value contained `${` (tag IAM unverified) | | Billing account IAM | billing-account | **N** | — | | PAM entitlements | organization / folder / project `pam_entitlements` factory | **N** | Entitlements are configuration and importable. PAM **grant bindings** never enter the denominator: `inventory.py` sweeps active grants via CAI (`privilegedaccessmanager.googleapis.com/Grant`) whenever IAM is collected and strips matching bindings deterministically (target, role, requester from the grant itself), stamping them in `_meta.pam_grant_exclusions`. Structural exemption, not a waiver; declaring the Grant type in a manifest is refused. Strip logic unit-tested; not yet exercised against a live grant | | IAM deny policies | organization / folder / project | **N** | Not a CAI asset type; enumerated automatically by the built-in `gcloud iam policies list --kind=denypolicies` enumerator. Command shape verified, payload shape not yet exercised live | | Log exclusions | organization / folder / project | **N** | Neither a CAI asset type nor a gcloud command group; needs REST enumeration recorded in the run report | | Workload identity pools / providers | project identity-providers | **V** | WorkloadIdentityPool, WorkloadIdentityPoolProvider, import IDs, address shapes, 30-day soft-delete residue, and org policy constraints verified | | Anything else | — | fallback | documented raw-resource fallback + capability-gap report | ## Verification Notes & Caveats - **Additive IAM default**: all V-level IAM rows above were verified under **authoritative** emission (`iam` / `iam_bindings` maps), which is now the opt-in (`emission.iam: authoritative`). The additive default (`iam_bindings_additive` → `google_*_iam_member.bindings`) is at **C**: addresses and import-ID formats derived from module source and provider docs, not yet exercised live. First verified additive round graduates it and re-stamps. - **Factory emission (opt-in)**: all matrix levels above are for the default per-instance emission. The `factory` emission mode (manifest `emission:` block; cookbook "Factory emission (opt-in)") is at **C** across all carriers — address shapes and layout rules derived from module source in-repo, no live import exercised yet. First verified run graduates the exercised carrier and stamps it here. - **CAS & Custom Roles**: CAS `publishing_options` requires Fabric PR #4107 (`a153861aae`). Custom roles title/description convergence requires Fabric PR #4102. Both are released in `v58.0.0`. - **Non-CAI types**: CAI is the default source of the denominator, not its boundary. Types it does not model are enumerated with `gcloud` — automatically where `inventory.py` ships a built-in enumerator, otherwise via a manifest `enumerate:` block — or, where gcloud has no container-scoped surface, out of band against the REST API. All of it merges into the same denominator, and `inventory.py` stops the run rather than proceeding without a type it cannot enumerate. See `references/cai-blind-spots.md`. - **Provider Label Inheritance**: Provider v5/v6/v7 generates computed `terraform_labels: {} -> {...}` diffs for resources with inherited provider labels. These are accounted for via scoped rules in `scripts/benign-drift.yaml`. ## Regression habit On every provider minor bump, Fabric ref bump, or after long dormancy: re-run both gates on the persisted workspace. New provider artifacts fail loudly as residuals; resolve via value-guarded benign rules in `scripts/benign-drift.yaml`, then re-stamp `verified_against`. Never widen a rule to make a bump pass. # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # fabric-importer manifest mirroring FAST stage 2-networking: the # networking projects and the connectivity stack. # # Maturity: core VPC surface, DNS zones, Routers/NAT, and linked-VPC NCC # spokes are [V]; advanced policies and VPN are [C]/[N]. Scope to your # networking folder. See COVERAGE.md. # --------------------------------------------------------------------------- scopes: - root: organizations/000000000000 # include: # - folders/333333333333 # networking folder number types: - type: iam # [V] levels: [project] - type: org-policy # [V] levels: [folder, project] - type: cloudresourcemanager.googleapis.com/Project # [V] levels: [folder] - type: serviceusage.googleapis.com/Service # [V] levels: [project] - type: iam.googleapis.com/ServiceAccount # [V] levels: [project] iam: true - type: compute.googleapis.com/Network # [V] levels: [project] - type: compute.googleapis.com/Subnetwork # [V] levels: [project] - type: compute.googleapis.com/Route # [V] (dynamic routes waived) levels: [project] - type: dns.googleapis.com/ManagedZone # [V] modules/dns levels: [project] - type: dns.googleapis.com/Policy # [N] DNS server policies levels: [project] - type: dns.googleapis.com/ResponsePolicy # [N] levels: [project] - type: networkconnectivity.googleapis.com/Hub # [N] NCC hub levels: [project] - type: networkconnectivity.googleapis.com/Spoke # [V] NCC linked-VPC spoke (raw fallback) levels: [project] - type: compute.googleapis.com/Router # [V] modules/net-cloudnat (router + NAT) levels: [project] - type: compute.googleapis.com/Address # [N] reserved addresses levels: [project] # - type: compute.googleapis.com/FirewallPolicy # [C] net-firewall-policy # levels: [folder] # - type: compute.googleapis.com/VpnGateway # [C] net-vpn-* # levels: [project] # - type: compute.googleapis.com/VpnTunnel # [C] # levels: [project] # - type: compute.googleapis.com/InterconnectAttachment # [C] vlan attachments # levels: [project] # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # fabric-importer manifest mirroring the surface managed by FAST stage # 0-org-setup: org-level governance plus the IaC / billing-export / # observability projects, their service accounts, state buckets, tags, and WIF. # # Maturity legend (see COVERAGE.md): [V] live-verified [C] cookbook # rules exist, not yet plan-tested [N] new territory (fallback + # capability-gap report). Validate counts with `inventory.py collect` # before mapping; adjust the scope's include to your IaC folder/projects. # --------------------------------------------------------------------------- scopes: - root: organizations/000000000000 # <- your org id # include/exclude match CAI `ancestors` (NUMERIC project ids). # Restrict project-level types to the org-setup projects: # include: # - projects/111111111111 # iac-core project NUMBER; see # import-manifest.multi-domain.yaml for selecting specific # projects per scope, and import-manifest.org-foundation.yaml # for the include-vs-exclude numeric-id trap types: - type: iam # [V] levels: [organization, project] - type: org-policy # [V] levels: [organization] - type: iam.googleapis.com/Role # [V] (org custom roles) levels: [organization] - type: logging.googleapis.com/LogSink # [V] levels: [organization] - type: cloudresourcemanager.googleapis.com/Folder # [V] levels: [organization, folder] - type: cloudresourcemanager.googleapis.com/Project # [V] levels: [organization, folder] - type: serviceusage.googleapis.com/Service # [V] levels: [project] - type: iam.googleapis.com/ServiceAccount # [V] levels: [project] iam: true # impersonation grants enter the denominator - type: logging.googleapis.com/LogBucket # [V] levels: [project] - type: storage.googleapis.com/Bucket # [V] state buckets levels: [project] - type: iam.googleapis.com/WorkloadIdentityPool # [V] WIF federation levels: [project] - type: cloudresourcemanager.googleapis.com/TagKey # [V] Resource Manager tags levels: [organization] - type: cloudresourcemanager.googleapis.com/TagValue # [V] levels: [organization] - type: cloudresourcemanager.googleapis.com/TagBinding # [V] levels: [organization, folder, project] # --- Optional additional types --- # - type: cloudbilling.googleapis.com/BillingAccount # [N] billing IAM # levels: [organization] # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # fabric-importer manifest mirroring FAST stage 2-security: the security # projects plus KMS keyrings/keys and Certificate Authority Service (CAS). # # Maturity (see COVERAGE.md): [V] live-verified. # Note: KMS keyrings cannot be deleted in GCP; CAS pools bill monthly. # --------------------------------------------------------------------------- scopes: - root: organizations/000000000000 # include: # - folders/222222222222 # security folder number types: - type: iam # [V] levels: [project] - type: org-policy # [V] levels: [folder, project] - type: cloudresourcemanager.googleapis.com/Project # [V] levels: [folder] - type: serviceusage.googleapis.com/Service # [V] levels: [project] - type: iam.googleapis.com/ServiceAccount # [V] levels: [project] iam: true - type: cloudkms.googleapis.com/KeyRing # [V] modules/kms levels: [project] - type: cloudkms.googleapis.com/CryptoKey # [V] modules/kms levels: [project] iam: true # per-key IAM is leaf IAM - opt in explicitly - type: privateca.googleapis.com/CaPool # [V] modules/certificate-authority-service levels: [project] # - type: privateca.googleapis.com/CertificateAuthority # [C] # levels: [project] # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # fabric-importer manifest mirroring FAST stage 1-vpcsc: the org access # policy, access levels, and service perimeters (modules/vpc-sc). # # Maturity: [V] live-verified (dry-run spec verified live; enforced # perimeter status block unverified). Access policies are org-level # singletons. See COVERAGE.md. # --------------------------------------------------------------------------- scopes: - root: organizations/000000000000 types: - type: identity.accesscontextmanager.googleapis.com/AccessPolicy # [V] levels: [organization] - type: identity.accesscontextmanager.googleapis.com/AccessLevel # [V] levels: [organization, unknown] - type: identity.accesscontextmanager.googleapis.com/ServicePerimeter # [V] levels: [organization, unknown] # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # EXAMPLE import manifest: "multi-domain" — PER-SCOPE TYPES # # One estate, four different depths. Each scope carries its own `types:` # list — there is no other place to declare types, so what a scope # collects is exactly what is written on it. The same type may appear in # several scopes with different `levels`, `iam` or `enumerate`; that is # the point, not an accident. # # Fail-closed rules (all exit 1 with the reason, before any API call): # - a scope without a `types:` list is refused; # - `types: []` is refused: an empty list is a scope that collects # nothing, and that is the one thing no scope may say quietly; # - a type whose `levels` cannot intersect its scope's `levels` is # refused as a dead declaration (an entry listing `unknown` is # exempt — see import-manifest.fast-vpcsc.yaml); # - a duplicate `type:` inside one scope's list is refused. # # Note the asymmetry with `emission:` (also per scope): emission INHERITS # built-in defaults for families you omit, because it only decides how # collected assets are written. `types:` never inherits anything, because # it decides what is collected at all. # # After collection, read the PER-SCOPE yield tables in inventory.json # (`_meta.scopes[].declared_types` / `zero_yield_types`): the aggregate # table can be non-zero for a type while one scope's own declaration # yielded nothing — e.g. `iam` non-empty org-wide but empty under # `workloads`. Every inventory entry also names the scope(s) that # collected it (`scopes: [...]`). # --------------------------------------------------------------------------- scopes: # Org foundation: governance only, no project-contained resources. - name: org-foundation root: organizations/000000000000 # <- your org id levels: [organization, folder] emission: folder: factory # day-2 operating model types: - type: iam levels: [organization, folder] - type: org-policy levels: [organization, folder] - type: iam.googleapis.com/Role levels: [organization] - type: logging.googleapis.com/LogSink levels: [organization] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] # Networking folder: the connectivity stack, plus folder-level # governance for that subtree only. Same `iam` and `org-policy` types # as above, different levels — per-scope lists make that expressible. - name: networking root: folders/111111111111 # <- networking folder levels: [folder, project] types: - type: iam levels: [folder, project] - type: org-policy levels: [folder, project] - type: cloudresourcemanager.googleapis.com/Project levels: [folder] - type: serviceusage.googleapis.com/Service levels: [project] - type: compute.googleapis.com/Network levels: [project] - type: compute.googleapis.com/Subnetwork levels: [project] - type: compute.googleapis.com/Router levels: [project] - type: dns.googleapis.com/ManagedZone levels: [project] # Security folder: KMS and service accounts, with leaf IAM opted in # per type. `iam: true` is a property of the type ENTRY, so it is # per-scope too: another scope may collect the same CryptoKey type # without leaf IAM. - name: security root: folders/222222222222 # <- security folder levels: [folder, project] types: - type: iam levels: [folder, project] - type: org-policy levels: [folder, project] - type: cloudresourcemanager.googleapis.com/Project levels: [folder] - type: serviceusage.googleapis.com/Service levels: [project] - type: iam.googleapis.com/ServiceAccount levels: [project] iam: true - type: cloudkms.googleapis.com/KeyRing levels: [project] - type: cloudkms.googleapis.com/CryptoKey levels: [project] iam: true # Two workload projects, named explicitly. Project IAM is in scope # here and nowhere else; no org-policy at all — a deliberate, declared # gap, not a silent one. - name: workloads root: organizations/000000000000 levels: [project] include: - projects/333333333333 # app-prod (project NUMBER) - projects/444444444444 # app-dev types: - type: iam levels: [project] - type: serviceusage.googleapis.com/Service levels: [project] - type: iam.googleapis.com/ServiceAccount levels: [project] iam: true - type: storage.googleapis.com/Bucket levels: [project] # --------------------------------------------------------------------------- # `enumerate:` blocks live in the list that declares them: a natively # enumerated type is swept only in the scopes whose `types:` names it. # Repeat the block in every scope that needs it — see # references/cai-blind-spots.md. # --------------------------------------------------------------------------- # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # EXAMPLE import manifest: "organization foundation" # # This manifest reproduces exactly the surface the legacy (v1) pipeline # covered: org-level custom roles, org IAM (bindings + audit configs), org # policies, org log sinks, the folder hierarchy, and folder IAM. # # A manifest is the user-owned contract for a fabric-importer run: # - inventory.py collects ONLY what the scopes declare (the completeness # denominator becomes "what you asked for", not "everything"); # - coverage.py fails the run until every collected asset is mapped to a # Terraform address or waived by a human; # - re-runs are incremental: widen a scope's `types:` list (or let the # org grow) and only the delta is added to the workspace. Existing # addresses are never rewritten. # # The grammar is scopes-only: `scopes:` is a list, and EVERY scope carries # its own `types:` list. One estate, different depths per subtree = several # scopes, each with its own list — see import-manifest.multi-domain.yaml # for the worked multi-scope example and the fail-closed rules. # --------------------------------------------------------------------------- scopes: # Root of the import. organizations/N, folders/N and projects/N are all # valid roots; everything below is evaluated inside this subtree. - root: organizations/000000000000 # <- replace with your org id # Optional: restrict to specific subtrees inside the root. An asset is # in scope only if one of these appears among its ancestors. Omit to # cover the whole root. # include: # - folders/111111111111 # Optional: carve out subtrees. Exclusion wins over inclusion. # # NOTE (live-verified): include/exclude match against the CAI # `ancestors` array, which stores NUMERIC ids. Both spellings work: # `projects/123456789` (the project NUMBER) matches directly, and a # project ID is resolved to its number by inventory.py via # `gcloud projects describe`. Prefer the number where you know it — # resolution needs the describe permission, and a project that cannot # be resolved is reported as an enumeration failure rather than # silently matching nothing. # exclude: # - projects/123456789012 # Optional: emission style, per resource family. YOUR call — the # scripts do not consume it; it directs step-3 mapping (the # cookbook's "Factory emission (opt-in)" section is normative). # per-instance (default) : one module instance per container. # factory : data-driven YAML via the family's # Fabric factory carrier. # An org foundation is the canonical factory case: folder tree, # folder IAM and org policies maintained as hierarchical data/ YAML # via project-factory. # # `emission` is per scope and INHERITS: a family key you omit gets # the built-in default (per-instance; additive for iam). `types:` # does NOT: every scope declares its full list, and an omission is # an error, not inheritance. Types decide the denominator; emission # only decides how collected assets are written. Do not reason from # one knob to the other. # # emission: # folder: factory # iam: additive # default — google_*_iam_member per tuple; an # # apply can never strip unmanaged members. Use # # `authoritative` to make IAM fully declarative types: # --------------------------------------------------------------- # IAM policies (pseudo-type). CAI models IAM as a content type # attached to resources, not as an asset type, hence the reserved # name `iam`. `levels` controls WHERE bindings are replicated: # organization -> org IAM bindings + audit configs # (including logging_data_access) # folder -> folder IAM bindings # Project-level IAM stays unmanaged in this example (add `project` # to take it over). Only container-attached IAM is covered by this # pseudo-type; IAM on leaf assets (tag values, buckets, service # accounts, ...) needs an `iam: true` opt-in on the leaf type's # own entry. # --------------------------------------------------------------- - type: iam levels: [organization, folder] # --------------------------------------------------------------- # Org policies (pseudo-type; CAI content type org-policy). One # inventory entry per (resource, constraint) pair, so coverage is # per-constraint. This example covers org-level only; declaring # [organization] makes the folder-level restriction explicit and # honest: folder policies are out of scope BY DECLARATION rather # than silently dropped. # --------------------------------------------------------------- - type: org-policy levels: [organization] # --------------------------------------------------------------- # Custom IAM roles. Fabric target: modules/organization factory, # data/custom-roles/.yaml. Org-level only; project-level # custom roles would need `project` here. # --------------------------------------------------------------- - type: iam.googleapis.com/Role levels: [organization] # --------------------------------------------------------------- # Log sinks. Fabric target: modules/organization logging_sinks # locals (with iam=false; _Default/_Required are quarantined). # Folder and project sinks stay unmanaged here. # --------------------------------------------------------------- - type: logging.googleapis.com/LogSink levels: [organization] # --------------------------------------------------------------- # Folder hierarchy. Fabric target: one modules/folder instance per # folder (stable instance keys; parent chaining by module # reference). `levels` refers to the CONTAINER of the asset: a # folder whose parent is the org sits at level `organization`, a # nested folder at `folder`. # --------------------------------------------------------------- - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] # --------------------------------------------------------------------------- # What this manifest deliberately does NOT cover (would have been silent # gaps in v1; here they are visible declarations — add entries to the # scope's `types:` list to take them over): # # - type: cloudresourcemanager.googleapis.com/Project # levels: [organization, folder] # - type: orgpolicy.googleapis.com/CustomConstraint # levels: [organization] # - type: cloudresourcemanager.googleapis.com/TagKey # levels: [organization] # - type: cloudresourcemanager.googleapis.com/TagValue # levels: [organization] # - type: essentialcontacts.googleapis.com/Contact # levels: [organization] # - type: compute.googleapis.com/Network # modules/net-vpc # - type: iam.googleapis.com/ServiceAccount # modules/iam-service-account # levels: [project] # iam: true # ALSO collect IAM attached to each SA (impersonation # # grants). Leaf-asset IAM never enters the denominator # # without this per-type opt-in. # --------------------------------------------------------------------------- # --------------------------------------------------------------------------- # TYPES CLOUD ASSET INVENTORY DOES NOT MODEL # # A type CAI cannot see would be invisible to BOTH gates at once, so it # is enumerated by other means and merged into the same inventory — # `gcloud` first. Declare it with an `enumerate:` block in the `types:` # list of every scope that needs it (per-scope lists never inherit): # # - type: iam.googleapis.com/DenyPolicy # not in the CAI catalogue # levels: [organization, folder] # enumerate: # command: [iam, policies, list, --kind=denypolicies] # container_arg: '--attachment-point=cloudresourcemanager.googleapis.com/{container}' # key: '//iam.googleapis.com/{container}/denypolicies/{item.name}' # # Guard rails (read-only verbs only, no --filter/--limit, unique key # templates), the built-in enumerator table, and the full ladder (CAI -> # gcloud -> REST API -> signed waiver) are in # references/cai-blind-spots.md. # --------------------------------------------------------------------------- # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # HUMAN-OWNED waiver ledger (example). # # A waiver excludes one in-scope inventory key from the coverage gate: # "we saw it, we decided not to manage it, here is why, signed". The model # operating the skill MUST NOT add or edit waivers; it may only propose # them in its report for a human to review and commit. # # Every waiver requires key + reason. `signed_by` records who accepted # it — coverage.py reports waivers without it, and hard-fails them under # --require-signed-waivers (recommended whenever a human is in the # loop). Include the date in the signed_by value for auditing. waivers: - key: //cloudresourcemanager.googleapis.com/folders/000000000001 reason: >- Decommissioning next quarter; not worth importing. Tracked in TICKET-123. signed_by: alice@example.com 2026-08-21 - key: //logging.googleapis.com/organizations/000000000000/sinks/tmp-debug reason: temporary debugging sink owned by the SRE oncall; do not manage signed_by: bob@example.com 2026-08-21 # fabric-importer `fabric-importer` reverse-engineers an existing Google Cloud organization into Cloud Foundation Fabric Terraform modules with native `import` blocks. An AI agent generates the module mapping; a set of frozen verification scripts proves the result is complete, attribute-accurate, and plan-converged. It never modifies anything in your live cloud environment. Start with the document for your role: - **Human Operator / Requesting an Import**: Follow this guide below. - **AI Agent Performing the Work**: Follow [SKILL.md](./SKILL.md). - **Supported Resources & Maturity**: See the [Maturity Matrix](./COVERAGE.md). --- ## The Design in One Paragraph Most infrastructure importers rely on brittle per-resource generator pipelines that quickly grow stale. This skill inverts that model: the AI agent inspects the live inventory and Fabric module sources directly, then emits idiomatic Fabric module calls paired with native Terraform 1.5+ `import {}` blocks. Two frozen verification gates validate the output. This is sound because every mistake is immediately visible: a missing attribute or schema mismatch causes `terraform plan` to propose unexpected changes, and a missed resource is flagged during inventory reconciliation. Verification per run replaces trust in generator code. --- ## What you provide, and what the tools work out You write one file: `import-manifest.yaml`. Everything else is derived from it. The important part is what that file contains. **It names resource types, not resources.** You never write a list of your 47 org policies. You write: ```yaml scopes: - root: organizations/123456789012 types: - type: org-policy levels: [organization] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] ``` That means "in this organization, take over org policies attached at org level, and every folder". Finding the actual 47 policies and 12 folders is the tool's job — and it has to be. If a human wrote the list of resources, "everything is covered" would only ever mean "everything I remembered to write down". The manifest is a deliberately coarse statement of intent; turning intent into an exact list is where people miss things, so a frozen script does it and refuses to hand back a partial answer. The run answers five questions in order, and only the third one is yours: | Question | Answered by | Binding? | |---|---|---| | What do I already manage in Terraform? | `manifest_from_state.py` | no, advisory | | What actually exists in the cloud? | `inventory.py survey` + `manifest_init.py` | no, advisory | | **What do I want under management?** | **you, in `import-manifest.yaml`** | **yes** | | Which live resources does that mean, exactly? | `inventory.py collect` → `inventory.json` | yes | | How much of it is done? | `coverage.py` | yes | The first two exist to break a chicken-and-egg problem: you cannot write a sensible manifest about an estate you cannot see, and the tools that measure the estate need a manifest first. Neither is a gate, neither produces anything binding, and you can skip both if you already know your scope. Run them anyway the first time: the gap between "what Terraform manages" and "what exists" is usually the most useful thing discovered all week. ### Different depth per subtree: per-scope `types:` One estate rarely wants the same depth everywhere — governance at org and folder level, the full connectivity stack in the networking folder, two named workload projects and nothing else. `scopes:` is a list, and **every scope carries its own `types:` list**; there is no other place to declare types, so what a scope collects is exactly what is written on it: ```yaml scopes: - name: org-foundation root: organizations/123456789012 levels: [organization, folder] types: - type: org-policy levels: [organization, folder] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] - name: workloads root: organizations/123456789012 levels: [project] include: - projects/111111111111 types: - type: iam levels: [project] - type: storage.googleapis.com/Bucket levels: [project] ``` The same type may appear in several scopes with different `levels`, `iam` or `enumerate`. Because a type list decides the denominator, the rules fail the run with a message rather than warning — before any API call, exit 1: | Rule | Why | |---|---| | Every scope declares `types:`; `types: []` is rejected | A scope that collects nothing passes every check by not collecting | | A type whose `levels` cannot intersect its scope's `levels` is rejected | A dead declaration reads like coverage and produces none (an entry listing `unknown` is exempt) | | A type declared twice in one list is rejected | Last-wins would quietly narrow levels | | The retired grammar (top-level `scope:` / `types:`) is rejected with migration instructions | Half-accepting it would change what the denominator means — see [references/manifest-migration.md](./references/manifest-migration.md) | `examples/import-manifest.multi-domain.yaml` is a worked four-scope manifest; `examples/import-manifest.org-foundation.yaml` is the single-scope reference. Afterwards, read the **per-scope** tables in `inventory.json`: `_meta.scopes[].declared_types` and `_meta.scopes[].zero_yield_types`. `_meta.declared_types` is the aggregate across scopes, and the aggregate can hide a scope whose own declaration yielded nothing — a type non-zero org-wide and zero in the one scope you cared about. Every inventory entry (and therefore every worklist entry) also names the scope(s) that collected it in its `scopes` field. ### `survey` and `collect` are not the same thing They are the same script and both list resources, which makes them easy to confuse. Their purposes are opposite. `survey` runs **without** a manifest. It browses, so that a human can decide. Its output is a conversation aid and is never used to judge anything. `collect` runs **only** with an approved manifest. Its output is the **denominator**: the exact list of live resources matching what you agreed. Everything downstream is measured against it. ### The two gates answer different questions This is the part worth reading twice, because neither gate can catch the other's failure. | Gate | Compares | Answers | |---|---|---| | 1 — `coverage.py` | `inventory.json` against the **text of your code** | did you cover everything? | | 2 — `verify_plan.py` | the **Terraform plan** against reality | is what you wrote correct? | Gate 1 never runs Terraform and never contacts your cloud. It reads `inventory.json`, the `import {}` blocks it parses out of your `*.tf` files, the `coverage-map.yaml` ledger, and `waivers.yaml`. Every asset in the denominator must be in the ledger or in the waivers. Gate 1 can pass on completely wrong code — right addresses, garbage configuration. The plan catches that, as diffs. Gate 2 can pass on a workspace covering 3 of your 59 resources. A plan of three clean imports and nothing else is perfectly converged; Terraform has no idea the other 56 exist. **A converged plan says nothing about completeness**, because Terraform only knows about what you wrote down. Completeness can only be judged against a list built independently of the code, which is exactly what `inventory.json` is. ### `inventory.json` is the only door The loop is closed in both directions: - a ledger key that is not in the inventory fails gate 1 as `stale` - an `import {}` block that no ledger key claims fails gate 1 as an `orphan` So nothing can be imported that the denominator does not contain, and nothing in the denominator can be quietly dropped. That is what makes "both gates green" mean something precise: everything the manifest declared is either in code or waived by name, and applying that code would change nothing. --- ## Human-in-the-Loop Gates Gate on steps that are hard to reverse, costly, or where human judgment is required (e.g. narrowing scope, waiving assets, or accepting provider drift). Keep mechanical, reversible steps (asset collection, scaffolding, linting, plan checks) autonomous. Gates are **blocking**: if running non-interactively and confirmation cannot be obtained, stop — never assume approval. | Gate | When | What the human decides | | :--- | :--- | :--- | | **Scope Approval** | Manifest drafting, before any enumeration | Reviews and commits `import-manifest.yaml`: which resource types each scope declares, at which hierarchy levels (org/folder/project), under which included/excluded subtrees. | | **Waiver Signing** | Completeness Gate (`coverage.py`) | Reviews deliberate exclusions in `waivers.yaml` and signs them with attribution (`signed_by`) for unmanaged or auto-generated resources (e.g. default log sinks, default compute service accounts). | | **Benign Drift Review** | Plan Convergence Gate (`verify_plan.py`) | Evaluates proposed cosmetic provider quirks (e.g. computed labels, default timeouts) and commits reviewed entries to `scripts/benign-drift.yaml`. | | **Final Review & Apply** | Handover, before `terraform apply` | Inspects the final zero-drift plan, run report, and input provenance digests before running `terraform apply` on their own schedule. | --- ## How the work flows Three views. The main loop, then the two places where a single item funnels through a decision — enumeration and mapping. ### The main loop ```mermaid flowchart TD A["Existing .tfstate files
or the live estate"] --> B["Draft import-manifest.yaml
manifest_from_state.py, or
inventory.py survey + manifest_init.py"] B --> G0{{"HUMAN GATE 1
approve the scope"}} G0 --> C["inventory.py collect"] C --> D[("inventory.json
THE DENOMINATOR")] D --> E["coverage.py --worklist-out"] E --> F["AGENT WORK
module config + import blocks
+ coverage-map.yaml"] F --> G1{"GATE 1 machine
coverage.py"} G1 -- "missing assets" --> W{{"HUMAN GATE 2
sign a waiver"}} W --> G1 G1 -- "gaps to close" --> F G1 -- "reconciled, exit 0" --> P["terraform plan"] P --> G2{"GATE 2 machine
verify_plan.py"} G2 -- "residual diff" --> F G2 -- "provider quirk?" --> BD{{"HUMAN GATE 3
accept benign drift"}} BD --> G2 G2 -- "converged, exit 0" --> R["Run report"] R --> NEXT["Widen the manifest
for the next round"] NEXT --> G0 R --> G3{{"HUMAN GATE 4
apply sign-off"}} G3 --> DONE(["terraform apply
run by the operator"]) ``` Everything above the scope gate is advisory. Everything below it is mechanical and binding. ### Funnel 1: how a declared type becomes inventory entries ```mermaid flowchart TD T["One type declared in a
scope's types: list"] --> Q1{"pseudo-type?"} Q1 -- "iam" --> S1["asset list
--content-type=iam-policy"] Q1 -- "org-policy" --> S2["three streams merged:
org-policy content type
+ orgpolicy Policy assets
+ gcloud org-policies list
per container"] Q1 -- "no, a real asset type" --> Q2{"enumerator registered?
built-in table, or a
manifest enumerate: block"} Q2 -- "yes" --> S3["gcloud list per container
CAI never consulted"] Q2 -- "no" --> S4["gcloud asset list
batched with the other types"] S4 --> Q3{"outcome"} Q3 -- "ok" --> OUT Q3 -- "type not in the
CAI catalogue" --> STOP["STOP, exit 3
fix the type string,
add an enumerator,
or drop it deliberately"] Q3 -- "any other error
403, ACM types" --> S5["asset search-all-resources"] S5 -- "ok" --> OUT S5 -- "still failing" --> STOP2["STOP, exit 3
partial denominator refused"] S1 --> OUT[("inventory.json")] S2 --> OUT S3 --> OUT ``` No path exits with "skip it". Every branch ends in the denominator or in a stop, because an unenumerated asset is invisible to both gates at once — it would pass every check by not existing. The funnel runs once per scope: the same type can enter it from two scopes' lists with different levels, and a type declared in only one scope is swept only there. ### Funnel 2: how a worklist item becomes code ```mermaid flowchart TD I["Worklist item"] --> M{"Does the canonical
Fabric module express
the live resource?"} M -- "yes" --> A["Module config
+ import block
+ coverage-map entry"] M -- "module exists but cannot
express an attribute" --> B["Raw google_* resource
+ MODULE CAPABILITY GAP
in the report"] M -- "no module covers
the type at all" --> C["Raw google_* resource
+ report, propose upstream"] M -- "deliberately out of scope" --> D{{"HUMAN GATE
signed waiver"}} M -- "a residual you
cannot explain" --> STOP["STOP
report it, never rationalise it"] A --> P["Both gates"] B --> P C --> P D --> X["Not in the workspace,
named in the report"] ``` Three of those four outcomes are successes. That matters: the temptation at the middle two is to force the module to fit, or to call the leftover diff benign. --- ## Step-by-Step Operator Guide ### 1. Declare the Scope (`import-manifest.yaml`) You and the agent agree on an `import-manifest.yaml` declaring, per scope, which resource types to import (e.g., folders, service accounts, VPC networks, org policies), at which container levels (organization, folder, or project), and which subtrees to include or exclude — see [Different depth per subtree](#different-depth-per-subtree-per-scope-types). This is the only file you author, and it names types, not resources. The two options below are drafting aids that answer different questions — A tells you what Terraform already manages, B tells you what exists. Run both the first time and compare them; run neither if you already know your scope. Neither output is binding until you have reviewed it and committed it. **Option A — Inferred from existing Terraform state(s):** ```bash uv run scripts/manifest_from_state.py --state --out import-manifest.yaml ``` See [references/inferring-manifests-from-state.md](./references/inferring-manifests-from-state.md). **Option B — Surveyed from live GCP estate:** ```bash uv run scripts/inventory.py survey --scope organizations/ORG_ID --out survey.json uv run scripts/manifest_init.py --survey survey.json --scope organizations/ORG_ID --out import-manifest.yaml ``` Ready-made examples aligned with FAST stages are provided in [`examples/`](./examples/); `import-manifest.org-foundation.yaml` is the single-scope reference and `import-manifest.multi-domain.yaml` the per-scope multi-domain one. ### 2. Enumerate the Denominator (`inventory.json`) The enumeration script queries Cloud Asset Inventory (CAI) and service APIs to construct the exact denominator of all matching live assets: ```bash uv run scripts/inventory.py collect --manifest import-manifest.yaml --out inventory.json ``` CAI is the default source of the denominator, not its boundary. CAI does not model every GCP resource, and a type it cannot see would otherwise be invisible to both gates at once — so the tool routes around it rather than shrinking the denominator: built-in `gcloud` enumerators for types known to be absent from the catalogue (declaring the type is all it takes), automatic sibling sweeps for the Compute families whose primary CAI surface splits them into separate scoped types (accounted under the declared type, real type preserved in `_meta.split_type_sweeps`), and a hard stop with the remedy where no enumerator exists — either the type string is wrong (checked against the [supported types list](https://cloud.google.com/asset-inventory/docs/supported-asset-types)), or the manifest needs a native `enumerate:` block: a read-only `gcloud` command run per in-scope container, normalized into the same inventory, declared in the `types:` list of each scope that needs it (per-scope lists never inherit). The full ladder (CAI → gcloud → REST API → signed waiver), the enumerator syntax and guard rails, and the split-taxonomy mechanics are in [references/cai-blind-spots.md](./references/cai-blind-spots.md). The split-type table is a frozen snapshot of a document Google changes, so check it against live CAI at least once per engagement: ```bash uv run scripts/inventory.py collect --manifest import-manifest.yaml \ --out inventory.json --verify-search-parity ``` It costs one extra `search-all-resources` call per scope and fails the run if the search surface returns an asset the list sweep did not. The run stays auditable without becoming unreadable: every run closes with a one-line cost summary on stderr (`7 gcloud call(s) in 12.0s: asset list x5, org-policies list x2`), `--verbose` (after the subcommand) prints each command as it runs with outcome, duration and item count, the full log is written to `_meta.api_calls`, and every non-CAI sweep is recorded verbatim in `_meta.native_sweeps` so a reviewer can re-run it. Provenance is per scope: each `_meta.scopes[]` entry records its own `declared_types` yield counts and `zero_yield_types`, and every inventory entry names the scope(s) that collected it. `_meta.declared_types` is the aggregate — read the per-scope counts too, because a type that yields zero in one scope and non-zero in another does not show up in the aggregate zero-yield warning (the per-scope warning on stderr does). ### 3. Compute Delta Worklist The completeness tool reconciles the denominator against any existing code and written waivers. This is the same program that runs as gate 1 in step 5 — the same comparison, run for a different reason. Here it tells you what is left to do; there it decides whether anything is. On the first round everything is outstanding, so the worklist is the whole denominator. On later rounds it is only the delta, which is what makes repeated rounds cheap: ```bash uv run scripts/coverage.py --inventory inventory.json --workspace tf \ --waivers waivers.yaml --worklist-out worklist.yaml ``` ### 4. Agent Maps & Emits Terraform For each item in `worklist.yaml`, the agent writes: 1. Canonical Fabric module configuration matching the live state. 2. A paired Terraform `import {}` block: ```hcl import { to = module.folder-team-a.google_folder.folder[0] id = "folders/123456789012" } ``` 3. A ledger entry in `coverage-map.yaml` linking the asset to its Terraform address. ### 5. Automated Verification (Definition of Done) Work is complete only when two independent gates pass with exit code `0`. They are independent in a specific sense: each catches a failure the other cannot see. Gate 1 can pass on wrong code, and gate 2 can pass on a workspace covering three resources out of fifty-nine. See [The two gates answer different questions](#the-two-gates-answer-different-questions). - **Completeness Gate (`coverage.py`)**: Reconciles the inventory against `coverage-map.yaml`, `waivers.yaml`, and emitted `import {}` blocks. Every asset must be mapped or waived. It reads files only — no Terraform, no cloud access — so it needs neither credentials nor a `terraform init` to run. - **Plan Convergence Gate (`verify_plan.py`)**: Consumes `terraform show -json ` output (file argument or stdin) and classifies the planned actions. It never invokes Terraform itself. The plan may only contain clean imports, no-ops, and human-reviewed benign drift entries from `scripts/benign-drift.yaml`. Any residual attribute modification or destruction fails the gate. Run both gates like this: ```bash terraform -chdir=tf fmt -recursive uv run scripts/coverage.py --inventory inventory.json --workspace tf \ --waivers waivers.yaml --require-signed-waivers terraform -chdir=tf init -input=false rm -f tf/verify.tfplan # a stale plan must never answer for a failed one terraform -chdir=tf plan -input=false -out=verify.tfplan terraform -chdir=tf show -json verify.tfplan | uv run scripts/verify_plan.py ``` Do not add `-detailed-exitcode` to the plan: it returns 2 for any non-empty plan, and a converged import plan is always non-empty. `verify_plan.py` is the verdict. ### 6. Review and Apply The resulting workspace (`tf/`) is clean, formatted, and ready for you to review and run `terraform apply` at your own pace. The skill **never** runs `terraform apply` or mutates live cloud resources. --- ## Script reference Run every script from the skill directory. All output paths are relative to the current working directory. | Script | Purpose | Flags | |---|---|---| | `inventory.py survey` | Enumerate everything in scope, to draft a manifest from | `--scope` (required), `--out`, `--verbose`, `--include-deleted`, `--include-logging-defaults`, `--include-pam-grants` | | `manifest_init.py` | Draft a manifest from a survey (Mode B) | `--survey` (required), `--scope` (required), `--out` | | `manifest_from_state.py` | Infer a manifest from existing `.tfstate` (Mode A) | `--state` (1+, required), `--out` (`-` for stdout), `--force` to overwrite an existing manifest | | `inventory.py collect` | Build the denominator from CAI | `--manifest` (required), `--out`, `--verbose`, `--include-deleted`, `--include-logging-defaults`, `--include-pam-grants`, `--verify-search-parity` | | `coverage.py` | Gate 1 — completeness | `--inventory` (required), `--workspace` (required), `--coverage-map`, `--waivers`, `--require-signed-waivers`, `--allow-empty-inventory`, `--worklist-out` | | `verify_plan.py` | Gate 2 — plan convergence | positional plan JSON (default stdin), `--rules`, `--allow-empty-plan` | | `integrity.py` | Print the frozen-tools provenance digest | `--verbose` for per-file digests | `inventory.py` requires the `survey` or `collect` subcommand; it has no top-level flags. `--verbose` therefore goes after the subcommand (`inventory.py collect --manifest m.yaml --verbose`), not before it. Exit codes — `coverage.py`: `0` reconciled, `1` malformed input, `2` gaps or problems. `verify_plan.py`: `0` converged, `1` malformed input, `2` residual changes, `3` converged but ADVISORY (a substituted `--rules` file). A substituted ruleset can never exit `0`; a residual plan still exits `2`. `inventory.py`: `0` ok, `1` a manifest the validator refuses (retired grammar, missing/empty per-scope `types:`, duplicate type, invalid level, dead per-scope declaration — printed with the reason before any API call), `3` a partial denominator (unsupported type, tolerated sweep failure, split-parity gap). Note that argparse usage errors also exit `2`, so read the message, not only the code. There is no checked-in expected digest. To check a recorded gate run, compute `uv run scripts/integrity.py` from a pristine checkout of the same commit and compare it with the `frozen tools:` line in the captured output. --- ## Prerequisites & Permissions ### System Requirements - **Terraform**: `v1.5.0+` (required for native `import {}` blocks). - **Google Cloud SDK (`gcloud`)**: Authenticated with a configured quota project. - **[`uv`](https://docs.astral.sh/uv/)**: the recommended way to run the scripts. Every script declares its own dependencies inline ([PEP 723](https://peps.python.org/pep-0723/), the same convention as `tools/tfdoc.py`), so `uv run scripts/.py` resolves them per script with nothing to install and nothing to activate. - **Python**: `3.10+` if you would rather not use `uv`. Then run the scripts with `python3` and install `PyYAML` yourself; the commands are otherwise identical. ### Minimal Read-Only IAM Permissions Run the import using a dedicated read-only identity. The following roles on the organization scope provide full read visibility: | Role | Purpose | |---|---| | `roles/cloudasset.viewer` | CAI asset inventory enumeration across the hierarchy | | `roles/resourcemanager.organizationViewer` | Organization resource details | | `roles/resourcemanager.folderViewer` | Folder hierarchy discovery | | `roles/viewer` | Read-only inspection of imported resources via service APIs | | `roles/iam.securityReviewer` | IAM policy discovery at organization, folder, and project levels | | `roles/orgpolicy.policyViewer` | Organization policy evaluation | | `roles/accesscontextmanager.policyReader` | VPC Service Controls discovery (when VPC-SC is in scope) | No write or admin permissions are required. --- ## User Ownership Boundary Three human-owned files govern the process. The agent may propose edits, but only you commit them: | File | Purpose | |---|---| | `import-manifest.yaml` | Declares the scopes, each with its own resource-type list, container levels, and subtree filters. | | `waivers.yaml` | Written waivers for resources deliberately excluded from management (e.g. `_Default` log sinks, default compute service accounts). Requires a `signed_by` attribute. | | `scripts/benign-drift.yaml` | Scoped provider quirks accepted as cosmetic diffs (e.g., computed provider labels or default timeouts). | | `references/address-map.yaml` | The fielded subset of the mapping cookbook: asset type, module, address pattern, import ID, verification round. Advisory, not a gate input — see below. | --- ## Repository Layout ```text skills/fabric-importer/ ├── SKILL.md # Agent protocol and operational contract ├── README.md # Human operator guide and architecture overview ├── COVERAGE.md # Resource family maturity matrix (N / C / V) ├── scripts/ # Verification gates and helper utilities │ ├── inventory.py # Asset enumeration (CAI + API sweeps) │ ├── coverage.py # Completeness gate & delta worklist generator │ ├── verify_plan.py # Plan-convergence gate (benign-drift aware) │ ├── manifest_from_state.py # Terraform state-driven manifest inference │ ├── manifest_init.py # Starter manifest drafting assistant │ ├── integrity.py # Input binding & runtime provenance stamping │ ├── address_map.py # Address-map validator, cookbook linter, table renderer │ └── benign-drift.yaml # Human-reviewed provider quirk ruleset ├── examples/ # Manifest and waiver templates │ ├── import-manifest.org-foundation.yaml │ ├── import-manifest.multi-domain.yaml │ ├── import-manifest.fast-org-setup.yaml │ ├── import-manifest.fast-security.yaml │ ├── import-manifest.fast-vpcsc.yaml │ ├── import-manifest.fast-networking.yaml │ └── waivers.example.yaml ├── references/ # Normative mapping rules & technical gotchas │ ├── mapping-cookbook.md # Traps, reasoning, capability gaps (prose) │ ├── address-map.yaml # Addresses, import IDs, asset types (data) │ ├── inferring-manifests-from-state.md # State-driven manifest inference workflow │ ├── manifest-migration.md # Migrating manifests from the retired grammar │ ├── cai-blind-spots.md # Known CAI gaps and service API mitigations │ └── operating-contract.md # Safety invariants and trust boundaries └── tests/ # Unit test suite ``` ### Where mapping knowledge lives The cookbook carries two different kinds of knowledge, and they are now stored differently because they fail differently. **Fielded tuples** — asset type, module, address pattern, import ID, verification round — live in `references/address-map.yaml`. They are uniform and mechanically checkable, and they were the part that kept being duplicated: the same address appeared in per-type prose and again in a hand-maintained markdown table, and three families were once documented twice because nothing could detect it. Two invariants are now enforced: - an address pattern is claimed by exactly one entry; - a CAI asset type serviced by more than one module (the live case: `compute.googleapis.com/Router`, created by both `net-cloudnat` and `net-vpn-ha`) must say how to tell the carriers apart. **Reasoning** — ForceNew traps, provider quirks, waiver postures, capability gaps — stays in `references/mapping-cookbook.md`, in prose, on purpose. It is causal and only degrades when squeezed into a schema. ```bash uv run scripts/address_map.py # validate + lint uv run scripts/address_map.py --render-cookbook # regenerate the table uv run scripts/address_map.py --check-workspace tf/ # advisory, always exits 0 ``` `address_map.py` is a linter, **not a gate**. `coverage.py` and `verify_plan.py` never read the address map, and a test enforces that. A wrong entry is caught by `terraform plan` erroring loudly, unlike a wrong `benign-drift.yaml` entry which would silently pass a gate. The map is also deliberately incomplete: an address it does not recognise is a type nobody has mapped yet, which SKILL.md calls the normal case. # Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Address and import-ID map — the FIELDED subset of the mapping cookbook. # # WHY THIS FILE EXISTS # # `mapping-cookbook.md` carries two different kinds of knowledge: # # 1. Fielded tuples — (asset type, module, address pattern, import ID, # verification round). Uniform across every entry, mechanically # checkable, and the part that kept being duplicated: the same # address appeared once in the per-type prose and again in a # hand-maintained markdown table, and three families were added # twice because nothing could detect it. # # 2. Reasoning — ForceNew traps, provider quirks, waiver postures, # capability gaps. Causal prose that becomes strictly worse when # squeezed into a `notes:` string. # # Only (1) lives here. (2) stays in the cookbook, in prose, on purpose. # This file is NOT a specification of what the skill supports and NOT a # closed world: a type absent from it is a type nobody has run the # method on yet. See "Step 3b" in SKILL.md. Never treat a miss here as # a reason to stop, and never treat a hit here as a substitute for # reading the module source. # # NOT FROZEN. This file is advisory input to `address_map.py`, which is # a linter, not a gate. It is deliberately outside the trust boundary in # `integrity.py`: a wrong entry here makes `terraform plan` error # loudly, unlike a wrong `benign-drift.yaml` entry which silently passes # a gate. Different blast radius, different rigour budget. # # SCHEMA (enforced by `scripts/address_map.py --validate`) # # version: 1 # entries: # : # stable, kebab-case, immutable once used # resource: str # human label, used for the rendered table # module: str # 'modules/', or 'raw' for a gap # address: str # Terraform address pattern, s # import_id: str # provider import ID pattern # asset_type: str # CAI asset type, or null when not CAI-modelled # verified: str # 'r' round that proved it, or false # note: str # optional, ONE line; prose belongs in the cookbook # disambiguation: str # REQUIRED when an asset_type maps to >1 module # # INVARIANTS (all machine-enforced) # - `address` is globally unique: no two entries may claim it. # - when one `asset_type` maps to more than one `module`, EVERY entry # sharing that type must carry a non-empty `disambiguation`. # - `verified` is false or matches `r`. # - `note` is a single line (multi-line reasoning goes in the cookbook). # # Regenerate the cookbook's rendered table after editing: # uv run scripts/address_map.py --render-cookbook version: 1 entries: # --- organization ------------------------------------------------- org-custom-role: resource: Custom role module: modules/organization address: 'module.organization.google_organization_iam_custom_role.roles[""]' import_id: 'organizations//roles/' asset_type: iam.googleapis.com/Role verified: r3 org-iam-member: resource: Org IAM member (additive, default) module: modules/organization address: 'module.organization.google_organization_iam_member.bindings[""]' import_id: ' ' asset_type: null verified: false note: 'conditional bindings append to the import ID' org-iam-binding-authoritative: resource: Org IAM binding (authoritative opt-in) module: modules/organization address: 'module.organization.google_organization_iam_binding.authoritative[""]' import_id: ' ' asset_type: null verified: r3 org-iam-binding-conditional: resource: Org IAM binding (cond.) module: modules/organization address: 'module.organization.google_organization_iam_binding.bindings[""]' import_id: ' ' asset_type: null verified: r3 org-audit-config: resource: Org audit config module: modules/organization address: 'module.organization.google_organization_iam_audit_config.default[""]' import_id: ' ' asset_type: null verified: r3 org-policy: resource: Org policy module: modules/organization address: 'module.organization.google_org_policy_policy.default[""]' import_id: 'organizations//policies/' asset_type: orgpolicy.googleapis.com/Policy verified: r5 disambiguation: 'org-level constraints; folder and project scopes have their own entries' org-log-sink: resource: Org log sink module: modules/organization address: 'module.organization.google_logging_organization_sink.sink[""]' import_id: 'organizations//sinks/' asset_type: logging.googleapis.com/LogSink verified: r3 disambiguation: 'org-level sinks; folder sinks are carried by modules/folder' org-tag-key: resource: Tag Key module: modules/organization address: 'module.organization.google_tags_tag_key.default[""]' import_id: 'tagKeys/' asset_type: cloudresourcemanager.googleapis.com/TagKey verified: r18 org-tag-value: resource: Tag Value module: modules/organization address: 'module.organization.google_tags_tag_value.default["/"]' import_id: 'tagValues/' asset_type: cloudresourcemanager.googleapis.com/TagValue verified: r18 tag-binding: resource: Tag Binding module: modules/organization address: 'module..google_tags_tag_binding.binding[""]' import_id: 'tagBindings/' asset_type: cloudresourcemanager.googleapis.com/TagBinding verified: r18 note: 'emitted on the container module that owns the bound resource' # --- folders ------------------------------------------------------ folder: resource: Folder module: modules/folder address: 'module.folder-.google_folder.folder[0]' import_id: 'folders/' asset_type: cloudresourcemanager.googleapis.com/Folder verified: r6 folder-iam-member: resource: Folder IAM member (additive, default) module: modules/folder address: 'module.folder-.google_folder_iam_member.bindings[""]' import_id: 'folders/ ' asset_type: null verified: false note: 'conditional bindings append to the import ID' folder-iam-binding-authoritative: resource: Folder IAM module: modules/folder address: 'module.folder-.google_folder_iam_binding.authoritative[""]' import_id: 'folders/ ' asset_type: null verified: r6 folder-iam-binding-conditional: resource: Folder IAM (cond.) module: modules/folder address: 'module.folder-.google_folder_iam_binding.bindings[""]' import_id: 'folders/ ' asset_type: null verified: r6 folder-org-policy: resource: Folder org policy module: modules/folder address: 'module.folder-.google_org_policy_policy.default[""]' import_id: 'folders//policies/' asset_type: orgpolicy.googleapis.com/Policy verified: r6 disambiguation: 'folder-scoped constraints; org and project scopes have their own entries' folder-log-sink: resource: Folder sink module: modules/folder address: 'module.folder-.google_logging_folder_sink.sink[""]' import_id: 'folders//sinks/' asset_type: logging.googleapis.com/LogSink verified: r6 disambiguation: 'folder-level sinks; org sinks are carried by modules/organization' # --- projects ----------------------------------------------------- project: resource: Project module: modules/project address: 'module.project-.google_project.project[0]' import_id: 'projects/' asset_type: cloudresourcemanager.googleapis.com/Project verified: r4 project-service: resource: Project service module: modules/project address: 'module.project-.google_project_service.project_services[""]' import_id: '/' asset_type: serviceusage.googleapis.com/Service verified: r4 project-service-orgpolicy: resource: Project service (orgpolicy) module: modules/project address: 'module.project-.google_project_service.org_policy_service[0]' import_id: '/orgpolicy.googleapis.com' asset_type: serviceusage.googleapis.com/Service verified: r4 project-iam-member: resource: Project IAM member (additive, default) module: modules/project address: 'module.project-.google_project_iam_member.bindings[""]' import_id: ' ' asset_type: null verified: false note: 'conditional bindings append to the import ID' project-iam-binding-authoritative: resource: Project IAM module: modules/project address: 'module.project-.google_project_iam_binding.authoritative[""]' import_id: ' ' asset_type: null verified: r4 project-iam-binding-conditional: resource: Project IAM (cond.) module: modules/project address: 'module.project-.google_project_iam_binding.bindings[""]' import_id: ' ' asset_type: null verified: r4 project-org-policy: resource: Project org policy module: modules/project address: 'module.project-.google_org_policy_policy.default[""]' import_id: 'projects//policies/' asset_type: orgpolicy.googleapis.com/Policy verified: false disambiguation: 'project-scoped constraints; org and folder scopes have their own entries' workload-identity-pool: resource: Workload Identity Pool module: modules/project address: 'module.project-.google_iam_workload_identity_pool.default[""]' import_id: 'projects//locations/global/workloadIdentityPools/' asset_type: iam.googleapis.com/WorkloadIdentityPool verified: r18 workload-identity-pool-provider: resource: Workload Identity Pool Provider module: modules/project address: 'module.project-.google_iam_workload_identity_pool_provider.default["/"]' import_id: 'projects//locations/global/workloadIdentityPools//providers/' asset_type: iam.googleapis.com/WorkloadIdentityPoolProvider verified: r18 pam-entitlement: resource: PAM entitlement module: modules/project address: 'module..google_privileged_access_manager_entitlement.default[""]' import_id: '/locations//entitlements/' asset_type: privilegedaccessmanager.googleapis.com/Entitlement verified: false note: ' is organizations/, folders/ or projects/; grants are excluded, never imported' # --- service accounts --------------------------------------------- service-account: resource: Service account module: modules/iam-service-account address: 'module.sa-.google_service_account.service_account[0]' import_id: 'projects//serviceAccounts/' asset_type: iam.googleapis.com/ServiceAccount verified: r8 service-account-iam-member: resource: SA IAM member (additive, default) module: modules/iam-service-account address: 'module.sa-.google_service_account_iam_member.bindings[""]' import_id: 'projects//serviceAccounts/ ' asset_type: null verified: false note: 'conditional bindings append to the import ID' service-account-iam-binding-authoritative: resource: SA IAM binding module: modules/iam-service-account address: 'module.sa-.google_service_account_iam_binding.authoritative[""]' import_id: 'projects//serviceAccounts/ ' asset_type: null verified: r8 service-account-iam-binding-conditional: resource: SA IAM binding (cond.) module: modules/iam-service-account address: 'module.sa-.google_service_account_iam_binding.bindings[""]' import_id: 'projects//serviceAccounts/ ' asset_type: null verified: r8 # --- logging and storage ------------------------------------------ log-bucket: resource: Log bucket module: modules/logging-bucket address: 'module.logging_bucket_.google_logging_project_bucket_config.bucket[0]' import_id: 'projects//locations//buckets/' asset_type: logging.googleapis.com/LogBucket verified: r10 gcs-bucket: resource: GCS bucket module: modules/gcs address: 'module..google_storage_bucket.bucket[0]' import_id: '/' asset_type: storage.googleapis.com/Bucket verified: r12 note: 'never import bare — the project qualifier is a ForceNew trap' bigquery-dataset: resource: BigQuery Dataset module: modules/bigquery-dataset address: 'module..google_bigquery_dataset.default' import_id: 'projects//datasets/' asset_type: bigquery.googleapis.com/Dataset verified: r17 pubsub-topic: resource: Pub/Sub Topic module: modules/pubsub address: 'module..google_pubsub_topic.default' import_id: 'projects//topics/' asset_type: pubsub.googleapis.com/Topic verified: r17 # --- VPC networking ----------------------------------------------- vpc-network: resource: VPC network module: modules/net-vpc address: 'module.net_vpc_.google_compute_network.network[0]' import_id: 'projects//global/networks/' asset_type: compute.googleapis.com/Network verified: r3 vpc-subnet: resource: VPC subnet module: modules/net-vpc address: 'module.net_vpc_.google_compute_subnetwork.subnetwork["/"]' import_id: 'projects//regions//subnetworks/' asset_type: compute.googleapis.com/Subnetwork verified: r3 disambiguation: 'ordinary subnets; proxy-only and private-NAT subnets have their own entries' vpc-subnet-proxy-only: resource: VPC subnet (proxy-only) module: modules/net-vpc address: 'module.net_vpc_.google_compute_subnetwork.proxy_only["/"]' import_id: 'projects//regions//subnetworks/' asset_type: compute.googleapis.com/Subnetwork verified: r9 disambiguation: 'purpose REGIONAL_MANAGED_PROXY or GLOBAL_MANAGED_PROXY on the live subnet' vpc-subnet-private-nat: resource: VPC subnet (private NAT) module: modules/net-vpc address: 'module.net_vpc_.google_compute_subnetwork.private_nat["/"]' import_id: 'projects//regions//subnetworks/' asset_type: compute.googleapis.com/Subnetwork verified: false disambiguation: 'purpose PRIVATE_NAT on the live subnet' vpc-route-gateway: resource: VPC route (gateway) module: modules/net-vpc address: 'module.net_vpc_.google_compute_route.gateway[""]' import_id: 'projects//global/routes/-' asset_type: compute.googleapis.com/Route verified: r9 disambiguation: 'next_hop_gateway on the live route' vpc-route-ilb: resource: VPC route (ilb) module: modules/net-vpc address: 'module.net_vpc_.google_compute_route.ilb[""]' import_id: 'projects//global/routes/-' asset_type: compute.googleapis.com/Route verified: r9 disambiguation: 'next_hop_ilb on the live route' vpc-route-instance: resource: VPC route (instance) module: modules/net-vpc address: 'module.net_vpc_.google_compute_route.instance[""]' import_id: 'projects//global/routes/-' asset_type: compute.googleapis.com/Route verified: r9 disambiguation: 'next_hop_instance on the live route' vpc-route-ip: resource: VPC route (ip) module: modules/net-vpc address: 'module.net_vpc_.google_compute_route.ip[""]' import_id: 'projects//global/routes/-' asset_type: compute.googleapis.com/Route verified: r9 disambiguation: 'next_hop_ip on the live route' vpc-route-vpn-tunnel: resource: VPC route (vpn tunnel) module: modules/net-vpc address: 'module.net_vpc_.google_compute_route.vpn_tunnel[""]' import_id: 'projects//global/routes/-' asset_type: compute.googleapis.com/Route verified: r9 disambiguation: 'next_hop_vpn_tunnel on the live route' # --- VPC firewall rules (legacy) ---------------------------------- vpc-firewall-custom-rule: resource: VPC firewall rule (custom) module: modules/net-vpc-firewall address: 'module..google_compute_firewall.custom_rules[""]' import_id: 'projects//global/firewalls/' asset_type: compute.googleapis.com/Firewall verified: false disambiguation: 'every rule not generated by the module default-rule toggles' vpc-firewall-default-admins: resource: VPC firewall rule (default, admins) module: modules/net-vpc-firewall address: 'module..google_compute_firewall.allow_admins[0]' import_id: 'projects//global/firewalls/-ingress-admins' asset_type: compute.googleapis.com/Firewall verified: false disambiguation: 'generated by default_rules_config.admin_ranges, not by custom_rules' vpc-firewall-default-http: resource: VPC firewall rule (default, http) module: modules/net-vpc-firewall address: 'module..google_compute_firewall.allow_tag_http[0]' import_id: 'projects//global/firewalls/-ingress-tag-http' asset_type: compute.googleapis.com/Firewall verified: false disambiguation: 'generated by default_rules_config.http_ranges, not by custom_rules' vpc-firewall-default-https: resource: VPC firewall rule (default, https) module: modules/net-vpc-firewall address: 'module..google_compute_firewall.allow_tag_https[0]' import_id: 'projects//global/firewalls/-ingress-tag-https' asset_type: compute.googleapis.com/Firewall verified: false disambiguation: 'generated by default_rules_config.https_ranges, not by custom_rules' vpc-firewall-default-ssh: resource: VPC firewall rule (default, ssh) module: modules/net-vpc-firewall address: 'module..google_compute_firewall.allow_tag_ssh[0]' import_id: 'projects//global/firewalls/-ingress-tag-ssh' asset_type: compute.googleapis.com/Firewall verified: false disambiguation: 'generated by default_rules_config.ssh_ranges, not by custom_rules' # --- firewall policies -------------------------------------------- firewall-policy-hierarchical: resource: Hierarchical Firewall Policy module: modules/net-firewall-policy address: 'module..google_compute_firewall_policy.hierarchical[0]' import_id: 'locations/global/firewallPolicies/' asset_type: compute.googleapis.com/FirewallPolicy verified: r7 disambiguation: 'org- or folder-attached policy' firewall-policy-network-global: resource: Global Network Firewall Policy module: modules/net-firewall-policy address: 'module..google_compute_network_firewall_policy.net_global[0]' import_id: 'projects//global/firewallPolicies/' asset_type: compute.googleapis.com/FirewallPolicy verified: r7 disambiguation: 'project-scoped, global' firewall-policy-network-regional: resource: Regional Network Firewall Policy module: modules/net-firewall-policy address: 'module..google_compute_region_network_firewall_policy.net_regional[0]' import_id: 'projects//regions//firewallPolicies/' asset_type: compute.googleapis.com/FirewallPolicy verified: r7 disambiguation: 'project-scoped, regional' # --- compute addresses -------------------------------------------- address-global-psc: resource: Compute address (global PSC) module: modules/net-address address: 'module..google_compute_global_address.psc[""]' import_id: 'projects//global/addresses/' asset_type: compute.googleapis.com/Address verified: r19 disambiguation: 'addressType INTERNAL with purpose PRIVATE_SERVICE_CONNECT' address-global-psa: resource: Compute address (global PSA) module: modules/net-address address: 'module..google_compute_global_address.psa[""]' import_id: 'projects//global/addresses/' asset_type: compute.googleapis.com/Address verified: r19 disambiguation: 'purpose VPC_PEERING' address-regional-internal: resource: Compute address (regional internal) module: modules/net-address address: 'module..google_compute_address.internal[""]' import_id: 'projects//regions//addresses/' asset_type: compute.googleapis.com/Address verified: r19 disambiguation: 'purpose GCE_ENDPOINT or SHARED_LOADBALANCER_VIP' # --- routers, NAT, VPN, interconnect ------------------------------ cloud-router: resource: Cloud Router module: modules/net-cloudnat address: 'module..google_compute_router.router[0]' import_id: 'projects//regions//routers/' asset_type: compute.googleapis.com/Router verified: r12 disambiguation: 'router whose only attached function is Cloud NAT; a router that also terminates VPN tunnels belongs to modules/net-vpn-ha' cloud-nat: resource: Cloud NAT module: modules/net-cloudnat address: 'module..google_compute_router_nat.nat' import_id: 'projects//regions//routers//' asset_type: compute.googleapis.com/Router verified: r12 disambiguation: 'sub-resource with no CAI asset of its own; claimed by the parent Router key' vpn-ha-gateway: resource: HA VPN Gateway module: modules/net-vpn-ha address: 'module..google_compute_ha_vpn_gateway.ha_gateway[0]' import_id: 'projects//regions//vpnGateways/' asset_type: compute.googleapis.com/VpnGateway verified: false vpn-external-gateway: resource: External VPN Gateway module: modules/net-vpn-ha address: 'module..google_compute_external_vpn_gateway.external_gateway[""]' import_id: 'projects//global/externalVpnGateways/' asset_type: compute.googleapis.com/ExternalVpnGateway verified: false vpn-tunnel: resource: VPN Tunnel module: modules/net-vpn-ha address: 'module..google_compute_vpn_tunnel.tunnels[""]' import_id: 'projects//regions//vpnTunnels/' asset_type: compute.googleapis.com/VpnTunnel verified: false note: 'shared_secret is ForceNew and unreadable — see the cookbook section before emitting' vpn-router: resource: Cloud Router (VPN) module: modules/net-vpn-ha address: 'module..google_compute_router.router[0]' import_id: 'projects//regions//routers/' asset_type: compute.googleapis.com/Router verified: false disambiguation: 'router that terminates VPN tunnels; a NAT-only router belongs to modules/net-cloudnat' vpn-router-interface: resource: Cloud Router interface (VPN) module: modules/net-vpn-ha address: 'module..google_compute_router_interface.router_interface[""]' import_id: '///' asset_type: compute.googleapis.com/Router verified: false disambiguation: 'sub-resource with no CAI asset of its own; claimed by the parent Router key' vpn-router-peer: resource: Cloud Router BGP peer (VPN) module: modules/net-vpn-ha address: 'module..google_compute_router_peer.bgp_peer[""]' import_id: 'projects//regions//routers//' asset_type: compute.googleapis.com/Router verified: false disambiguation: 'sub-resource with no CAI asset of its own; claimed by the parent Router key' interconnect-attachment: resource: Interconnect Attachment module: modules/net-vlan-attachment address: 'module..google_compute_interconnect_attachment.default[""]' import_id: 'projects//regions//interconnectAttachments/' asset_type: compute.googleapis.com/InterconnectAttachment verified: false # --- Cloud DNS ---------------------------------------------------- dns-managed-zone: resource: DNS zone module: modules/dns address: 'module..google_dns_managed_zone.dns_managed_zone[0]' import_id: 'projects//managedZones/' asset_type: dns.googleapis.com/ManagedZone verified: r12 dns-record-set: resource: DNS recordset module: modules/dns address: 'module..google_dns_record_set.dns_record_set[" "]' import_id: 'projects//managedZones//rrsets//' asset_type: dns.googleapis.com/ResourceRecordSet verified: false note: 'for_each key is " " — type first, single space, trailing dot on ' dns-response-policy: resource: DNS Response Policy module: modules/dns-response-policy address: 'module..google_dns_response_policy.default[0]' import_id: 'projects//responsePolicies/' asset_type: dns.googleapis.com/ResponsePolicy verified: false dns-response-policy-rule: resource: DNS Response Policy rule module: modules/dns-response-policy address: 'module..google_dns_response_policy_rule.default[""]' import_id: 'projects//responsePolicies//rules/' asset_type: dns.googleapis.com/ResponsePolicyRule verified: false # --- KMS ---------------------------------------------------------- kms-keyring: resource: KMS KeyRing module: modules/kms address: 'module..google_kms_key_ring.default[0]' import_id: 'projects//locations//keyRings/' asset_type: cloudkms.googleapis.com/KeyRing verified: r13 kms-crypto-key: resource: KMS CryptoKey module: modules/kms address: 'module..google_kms_crypto_key.default[""]' import_id: 'projects//locations//keyRings//cryptoKeys/' asset_type: cloudkms.googleapis.com/CryptoKey verified: r13 kms-crypto-key-iam: resource: KMS CryptoKey IAM module: modules/kms address: 'module..google_kms_crypto_key_iam_binding.authoritative["."]' import_id: 'projects//locations//keyRings//cryptoKeys/ ' asset_type: null verified: r13 # --- Certificate Authority Service -------------------------------- cas-ca-pool: resource: CAS CA Pool module: modules/certificate-authority-service address: 'module..google_privateca_ca_pool.default[0]' import_id: 'projects//locations//caPools/' asset_type: privateca.googleapis.com/CaPool verified: r14 cas-ca-pool-iam: resource: CAS CA Pool IAM module: modules/certificate-authority-service address: 'module..google_privateca_ca_pool_iam_binding.authoritative[""]' import_id: 'projects//locations//caPools/ ' asset_type: null verified: false cas-ca-pool-iam-conditional: resource: CAS CA Pool IAM (cond.) module: modules/certificate-authority-service address: 'module..google_privateca_ca_pool_iam_binding.bindings[""]' import_id: 'projects//locations//caPools/ ' asset_type: null verified: false cas-certificate-authority: resource: Certificate Authority module: modules/certificate-authority-service address: 'module..google_privateca_certificate_authority.default[""]' import_id: 'projects//locations//caPools//certificateAuthorities/' asset_type: privateca.googleapis.com/CertificateAuthority verified: false # --- VPC Service Controls ----------------------------------------- vpcsc-access-policy: resource: VPC-SC Access Policy module: modules/vpc-sc address: 'module..google_access_context_manager_access_policy.default[0]' import_id: '' asset_type: identity.accesscontextmanager.googleapis.com/AccessPolicy verified: r15 note: 'import ID is the bare numeric policy id, with no accessPolicies/ prefix' vpcsc-access-level: resource: VPC-SC Access Level module: modules/vpc-sc address: 'module..google_access_context_manager_access_level.basic[""]' import_id: 'accessPolicies//accessLevels/' asset_type: identity.accesscontextmanager.googleapis.com/AccessLevel verified: r15 vpcsc-service-perimeter: resource: VPC-SC Service Perimeter module: modules/vpc-sc address: 'module..google_access_context_manager_service_perimeter.regular[""]' import_id: 'accessPolicies//servicePerimeters/' asset_type: identity.accesscontextmanager.googleapis.com/ServicePerimeter verified: r15 # --- Network Connectivity Center ---------------------------------- ncc-hub: resource: NCC Hub module: modules/ncc-spoke-ra address: 'module..google_network_connectivity_hub.hub[0]' import_id: 'projects//locations/global/hubs/' asset_type: networkconnectivity.googleapis.com/Hub verified: false ncc-spoke-ra: resource: NCC spoke (router appliance) module: modules/ncc-spoke-ra address: 'module..google_network_connectivity_spoke.spoke_ra' import_id: 'projects//locations//spokes/' asset_type: networkconnectivity.googleapis.com/Spoke verified: false disambiguation: 'linked_router_appliance_instances spokes only' ncc-spoke-raw: resource: NCC spoke (raw — capability gap) module: raw address: 'google_network_connectivity_spoke.' import_id: 'projects//locations//spokes/' asset_type: networkconnectivity.googleapis.com/Spoke verified: r12 disambiguation: 'linked-VPC and linked-VPN spokes, which no Fabric module carries' # CAI blind spots — where the denominator is incomplete The completeness gate is only as strong as Cloud Asset Inventory's coverage. An asset type CAI does not model is invisible to BOTH gates simultaneously (not enumerated → not required by coverage; not emitted → not diffed by plan). Treat this list as living documentation: verify and extend it per engagement. There are three distinct blind spots, and they need different remedies: 1. **CAI does not model the type at all.** The classic case, and what most of this document is about. Remedy: enumerate it by other means and merge it into the same denominator. 2. **CAI models the type under a different NAME on the surface you queried.** Rarer, much quieter, and not a blind spot in CAI at all — a blind spot in the question. See [Surface-dependent type taxonomies](#surface-dependent-type-taxonomies). 3. **CAI models the type but omits a FIELD a consumer depends on.** Quietest of all: the sweep succeeds and the payload simply lacks the attribute. See [Field-level blind spots](#field-level-blind-spots). ## The rule **CAI is the DEFAULT source of the denominator, never the boundary of it.** Anything CAI does not support is retrieved by other means and merged into the same denominator, so the completeness gate still has to account for it. "CAI cannot see it" is not a reason for an asset to be absent; it is a reason to enumerate it differently. The ladder, in order of preference: 1. **CAI** (`gcloud asset list`, `asset search-all-resources`) — default, because one sweep covers a whole hierarchy and returns ancestry. 2. **`gcloud list|describe`** — the preferred alternative. Read-only, already installed, already authenticated as the run's read-only identity, and machine-readable with `--format=json`. Declare it in the manifest as a native enumerator (below) so the entries land in `inventory.json` like any others. 3. **The service REST API** (`curl` with `gcloud auth print-access-token`) — only where gcloud has no surface at all. Enumerate out of band, record the exact call in the run report, and treat any assets found as in scope for mapping. 4. **A signed waiver plus a run-report entry** — the last resort, and a deliberate, attributed decision. "Not enumerable" is a fact to publish, never a silence. `inventory.py` walks steps 1→2 by itself. It ships a table of built-in enumerators (`NATIVE_ENUMERATORS`) for types known to be absent from the CAI catalogue: declaring such a type in the manifest is enough, and the tool announces that it took the gcloud route. A declared type that CAI rejects and no enumerator covers fails the run with the remedy, instead of a generic enumeration failure. Steps 3 and 4 it cannot enforce — that is the operator's honesty, and the run report is where it is spent. ## Surface-dependent type taxonomies CAI does not have one asset-type taxonomy. It has two, and for a handful of Compute families they disagree: - the **list / export / query / monitor** surface (`gcloud asset list`, this tool's primary sweep) splits a family by scope into **separate asset types**; - the **search / analysis** surface (`gcloud asset search-all-resources`) folds those same resources into a **single unified type**. Google documents this per type on the [supported types list](https://cloud.google.com/asset-inventory/docs/supported-asset-types), but only in the per-type prose, which is easy to read past: > `compute.googleapis.com/Address` — Returns global and regional > addresses in the search and analysis APIs, and only regional addresses > in the list, export, query, and monitor APIs. > > `compute.googleapis.com/GlobalAddress` — Not available in the analysis > and search APIs. Use `compute.googleapis.com/Address` instead in the > search and analysis APIs. The documented pairs: | Declared (search taxonomy) | List-surface sibling swept alongside it | |---|---| | `compute.googleapis.com/Address` | `compute.googleapis.com/GlobalAddress` | | `compute.googleapis.com/BackendService` | `compute.googleapis.com/RegionBackendService` | | `compute.googleapis.com/Disk` | `compute.googleapis.com/RegionDisk` | | `compute.googleapis.com/ForwardingRule` | `compute.googleapis.com/GlobalForwardingRule` | ### Reproducing it, if you ever need to The silent-gap condition needs a family whose declared type yields **non-zero** while still being incomplete: at least one REGIONAL address and at least one GLOBAL address, in the same in-scope project. With only global addresses present the declared type yields zero, the pre-existing zero-yield warning fires, and the tool is loud — a real bug, but not THIS bug. (With several scopes, remember the aggregate warning can stay quiet while one scope's own declaration yielded nothing — the per-scope `_meta.scopes[].zero_yield_types` record is the loud one there.) A test estate built only from a freshly created global address therefore validates that siblings are swept, and validates nothing about the silence. ### Why this one is worse than an unsupported type An unsupported type fails loudly. This one fails **silently, past every guard the tool has** — which is why it needed a mechanism rather than a sharper error message. Live-run finding: a global Private Service Connect address never entered the denominator, because - the declared type IS supported by `asset list`, so the unsupported-type fallback to `search-all-resources` never fired; - the sweep SUCCEEDED, so nothing landed in `SWEEP_FAILURES`; - it returned 33 regional addresses, so the zero-yield warning stayed quiet; - the asset was never collected, so `apply_level_filter`'s `unknown` safety net never saw it either. The operator had already hand-written the correct `import {}` block, so the only signal that fired was `coverage.py` reporting it as an **orphan import block** — and an orphan was read as a coverage-map problem and waived, rather than as evidence that the denominator was short. **An orphan import block for a resource you know is live is evidence the denominator is short.** Investigate the enumeration before waiving it; a waiver signed over a short denominator makes the gap permanent and attributed to a human. ### What `inventory.py` does `CAI_SPLIT_TYPES` is a frozen table of the pairs above. Declaring the unified type sweeps its list-surface siblings too, and the siblings' entries are **retyped back to the declared type**, so one manifest line means "all addresses" — which is what an operator reading the supported-types page will believe it means. This costs **zero extra API calls**: `asset list` takes a comma-separated `--asset-types`, so the siblings ride along in the existing sweep. Nothing is laundered by the retyping: - each retyped entry carries `cai_list_type` with the type CAI actually returned it as, and that field travels into `coverage.py`'s worklist; - `_meta.split_type_sweeps` records declared type, list-surface type, scope and the RAW swept count — what CAI returned, before the subtree/deleted/level filters; - collection prints a NOTICE naming every sibling that contributed, and reconciles swept against in-the-denominator itself (e.g. `3 swept …, 2 in the denominator (1 excluded by scope/level/deleted filters)`) — the filters apply to retyped entries like to any other asset, and the subtraction is the tool's to show, not the reader's to derive. A sibling the manifest declares **in its own right** is not remapped: an operator who names `compute.googleapis.com/GlobalAddress` explicitly wants it accounted as itself, and remapping would make their per-declared-type yield read zero. ### Checking the table against live CAI The table is a frozen snapshot of a document Google changes. A new split — or a renamed pairing — would put the tool straight back into the failure mode the table closes, so there is a probe: ```bash uv run scripts/inventory.py collect --manifest import-manifest.yaml \ --out inventory.json --verify-search-parity ``` One extra `search-all-resources` call per scope, restricted to declared split types. It compares against the **raw** sweep output, before the manifest's subtree filters, because the question under test is taxonomy and not scope. Anything the search surface returns that the list sweep did not is **fatal** (exit 3) with the offending resource names: the table is stale and the denominator is incomplete. The probe is opt-in, for two reasons: it costs a call per scope, and it imports the search index's propagation lag as a possible false positive. Run it at least once per engagement, and quote the result in the run report. `_meta.split_parity` is **empty** when the probe did not run; a probe that ran and found nothing is a RECORD whose `only_in_search` is empty. Read the record, not the key — "not checked" and "checked, clean" are different claims and must not be reported as the same one. If the probe fires, identify the list-surface type of the missing assets (`gcloud asset list` with no `--asset-types`, then match on name) and report it: `CAI_SPLIT_TYPES` needs a new entry, which is a reviewed change to a frozen file, not a local patch. CAI **retiring** a split is the convergence this table is waiting for. A sibling type that `asset list` no longer recognises is therefore reported as a possibly-stale table, not as a failed run — a sibling is tool-supplied, not operator-declared, so it must never fail someone else's collection. ## Field-level blind spots A missing field in a CAI payload is asymmetric to an unsupported asset type: - An **unsupported type fails loudly**: CAI returns `INVALID_ARGUMENT`, the run stops, and a remedy is displayed. - A **missing field fails silently**: `.get()` returns `None`, a filter predicate answers "no", and the asset unexpectedly stays in or drops out of the denominator while the run appears healthy. A filter predicate must never depend on a field whose presence in the live payload has not been verified. When a discriminator field is unavailable or unverified in CAI, the filter must **fail open** — keep the asset in the denominator for a human to waive, never drop it silently. ## Built-in enumerators | Asset type | How it is enumerated | Status | |---|---|---| | `iam.googleapis.com/DenyPolicy` | `gcloud iam policies list --kind=denypolicies --attachment-point=cloudresourcemanager.googleapis.com/` | Command shape verified against gcloud 576; payload shape not yet exercised live | The table is short on purpose. To qualify, an enumerator has to be hierarchy-container-scoped, read-only, JSON-emitting, and produce a key that is stable and unique. Most non-CAI types fail one of those — usually the first — and stay in the blind-spot table below, enumerated by hand and named in the run report. The table is frozen, like the rest of `inventory.py`: it decides part of what the denominator contains. Nothing about extending it requires editing a frozen file, though — a manifest block covers a type the table does not, and overrides an entry when an operator knows better. An enumerator earned in an engagement should come back as a reviewed change to the table, so the next run starts with it. ## Declaring a native enumerator A manifest type entry may carry an `enumerate:` block. The type is then never sent to CAI; the declared gcloud command runs once per in-scope container at the type's `levels`, and its JSON becomes inventory entries: ```yaml types: # a scope's own list — the only position types are declared in # IAM deny policies: managed by modules/organization, # modules/folder and modules/project, and absent from the CAI # catalogue entirely. - type: iam.googleapis.com/DenyPolicy levels: [organization, folder] enumerate: # `gcloud` is implicit and is the only executable a manifest # can name. The tool appends --format=json and one container # argument. command: [iam, policies, list, --kind=denypolicies] # Optional: the shape of that container argument, when the # command does not take --organization/--folder/--project=. # Fields: {container} (e.g. organizations/1) and # {container_id} (1). container_arg: '--attachment-point=cloudresourcemanager.googleapis.com/{container}' # Key template. Fields: {container}, {container_id} and # {item.} into each returned JSON object. Where # the type has a CAI name format, mirror it, so entries from # the two sources dedupe instead of double-counting. key: '//iam.googleapis.com/{container}/denypolicies/{item.name}' ``` Declared in one scope's list, the command runs only for that scope's containers; another scope may declare the same type with a different `enumerate:` block. What it may not do is inherit: per-scope lists never inherit anything, so the block is repeated in every scope that needs it. **The mechanism is per hierarchy container**, because that is the shape of `levels` — the type entry's `levels` intersected with its scope's. An empty intersection is refused when the manifest is parsed (a per-scope declaration that can never fire is a mistake, not a leftover), unless the entry lists `unknown`. A command that enumerates inside another resource — `gcloud storage managed-folders list gs://BUCKET` — cannot be expressed as an `enumerate:` block: there is no container flag to fill in. Those types are enumerated out of band and named in the run report (step 3 of the ladder), or waived deliberately. Do not force them into a block by hard-coding one bucket; a container argument that does not vary sweeps one place and leaves the rest silently unenumerated, which is why the tool refuses it. Run the command by hand once before committing the block: the key template has to match the payload the command actually returns, and a first run is the cheapest place to find out that it does not. A field that is absent is a hard failure, and a yield of zero is reported loudly — neither is silent, but neither belongs in a run either. Guard rails, all fail-closed, all for the same reason — a native enumerator is a hand-written piece of the denominator, and the denominator is the one thing in this skill nobody is allowed to shrink quietly: - the command must end in a read-only verb (`list`, `describe`, `get`, `search`); anything else is refused before a single call is made; - `--format`, `--filter`, `--limit`, `--page-size`, `--flatten`, `--sort-by` and `--uri` are refused. Narrowing belongs in the manifest's `scope` block, where it is reviewed; a `--filter` inside an enumerator shrinks the denominator with no trace; - the command may not select an identity — impersonation is set once for the whole run via `CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT`; - a key template that renders the same key for two items, or that references a field the payload does not have, is a hard failure. Two assets collapsing onto one key is a shrunken denominator with a green gate; - every native sweep is stamped into `inventory.json`'s `_meta` (`native_sweeps`: type, verbatim command, container, yield) and announced on stderr. A reviewer can re-run each command by hand. The manifest is human-owned and committed at the scope-approval gate, which is what makes an operator-supplied command acceptable at all. An agent may draft one; a human signs it, exactly as with waivers. ## Known or suspected gaps (verify per engagement) | Area | Concern | Mitigation | |---|---|---| | Service coverage | CAI supports several hundred asset types but not every GCP service/resource; niche or very new resources may be absent | Check the [CAI supported types list](https://cloud.google.com/asset-inventory/docs/supported-asset-types) for every service the user cares about; for uncovered types, declare a native enumerator (above) so they still enter the denominator | | Wrong type string | A type string that does not exist in the catalogue is not a blind spot but a typo, and it presents as one. `gcloud asset list` answers `INVALID_ARGUMENT: No supported asset type matches: ` | `inventory.py` classifies that error separately from a permission failure and stops with instructions. Live example: `logging.googleapis.com/OrganizationSettings` does not exist — CAI models the Logs Router settings singleton as `logging.googleapis.com/Settings` (supported at `organization` and `folder` levels only; not indexed at `project` level) | | Right type string, wrong surface | A type that is correct for `search-all-resources` can cover only PART of its family under `asset list`, which splits some Compute families by scope into separate types. No error, non-zero yield, silent gap. Live example: a global PSC address typed `compute.googleapis.com/GlobalAddress` by the list surface, absent from a `compute.googleapis.com/Address` sweep | **Handled automatically** for the documented pairs — `CAI_SPLIT_TYPES` sweeps the siblings and accounts them under the declared type. Run `--verify-search-parity` once per engagement to check the frozen table against live CAI; see [Surface-dependent type taxonomies](#surface-dependent-type-taxonomies) | | IAM deny policies | `iam.googleapis.com/DenyPolicy` is not a CAI asset type. Fabric manages deny policies at all three container levels (`iam_deny_policies`) | **Handled automatically** — built-in enumerator; just declare the type | | Log exclusions | `logging.googleapis.com/LogExclusion` is not a CAI asset type, and current `gcloud` has no `logging exclusions` group either — the ladder falls through to the REST API (`v2/{parent}/exclusions`). Fabric manages exclusions in `modules/organization`, `modules/folder` and `modules/project` | Enumerate out of band, map them, and record the enumeration method in the run report; do not let their absence from CAI read as absence from the estate | | Org-policy content-type lag | `--content-type=org-policy` can lag behind newly introduced v2 constraints; the `orgpolicy.googleapis.com/Policy` resource asset stream is more complete | `inventory.py` merges both CAI streams; keep cross-checking counts with `gcloud org-policies list` | | Org-policy dry-run specs | CAI `orgpolicy.googleapis.com/Policy` resource stream returns policies where `spec` is present (with or without `dryRunSpec`), but completely **omits dry-run-only policies** where `spec` is unset | `inventory.py collect` sweeps `gcloud org-policies list` per in-scope container and merges by key, so dry-run-only policies enter the denominator automatically | | IAM conditions | The `iam-policy` content type returns version-3 policies with conditional bindings intact | Verified in live testing | | IAM on leaf assets | `inventory.py` restricts the `iam` pseudo-type to container assets (`Organization`, `Folder`, `Project`); leaf IAM needs explicit per-type manifest entries (`iam: true`) | Documented in the manifest reference | | Audit configs | `auditConfigs` block is present and fully preserved in the CAI `iam-policy` payload | Verified in live testing | | Deleted / pending-delete resources | CAI reflects live state; soft-deleted roles or pending-delete projects may or may not appear | Decide policy per type; document in the run report | | Propagation lag | CAI can lag live changes by minutes | Re-run inventory immediately before the final gate pass; treat count mismatches with service APIs as failures, not noise | | Cloud Storage below the bucket | The Cloud Storage section of the CAI catalogue contains exactly one type, `storage.googleapis.com/Bucket`. Objects (`google_storage_bucket_object`), managed folders (`google_storage_managed_folder`) and managed-folder IAM are not modelled | `gcloud storage ls` / `gcloud storage managed-folders list` take a bucket URL, not a container flag, so they do not fit an `enumerate:` block. Enumerate per bucket out of band, map, and record the method in the run report | | Terraform-only resources | Some `google_*` resources correspond to no asset at all. `google_project_service_identity` triggers creation of a Google-managed service agent: the state row is real, the CAI asset is not, and Fabric's `modules/project` handles service agents internally | Map through the owning module and note it in the report; there is nothing to enumerate and nothing to waive | | Leaf IAM read as a missing type | `google_storage_bucket_iam_binding`, `google_tags_tag_value_iam_binding` and friends are not asset types. They are the `iam-policy` content type on an asset CAI already models, and enter the denominator via `iam: true` on the PARENT type entry | Never waive these as unsupported. `manifest_from_state.py` maps the common ones to their parent with the flag already set | | Data-plane / child resources | Some child resources (e.g. per-bucket notification configs, dataset ACL entries) are attributes of the parent in CAI, not assets | The plan gate covers them once the parent is imported; note them in the report if the user expects per-child coverage | | Access Context Manager | CAI `asset list --content-type=resource` rejects ACM types, requiring `asset search-all-resources`; `parentFullResourceName` lacks standard container prefixes | Handled in `inventory.py` via dedicated search and level classification | ## Rules 1. When a scope's `types:` list declares a type, confirm it appears in the CAI supported-types list. If it does not, it gets a native enumerator, an out-of-band enumeration recorded in the report, or a signed waiver — one of the three, always. Never a quiet removal from the manifest. 2. Any count mismatch between CAI and a service API is a **failure to investigate**, never noise to average over. The same applies between CAI's own two surfaces: `asset list` and `search-all-resources` disagreeing about a type is a taxonomy split to identify, not a discrepancy to pick a winner from. 3. An **orphan import block** reported by `coverage.py` for a resource you know is live is evidence the denominator is short. Investigate the enumeration first; a waiver over a short denominator makes the gap permanent and signs a human's name to it. 4. Every entry that did not come from CAI is named in the run report, with the command that produced it. `_meta.native_sweeps` in `inventory.json` carries this for declared enumerators; `_meta.split_type_sweeps` carries entries that came from CAI under a different type than the one declared; out-of-band enumeration is on you to quote. Per-scope yields live in `_meta.scopes[].declared_types` / `zero_yield_types` — the aggregate table can hide a scope whose own declaration yielded nothing. 5. A scope's `types:` list narrows the denominator, so it belongs to the Scope Approval gate like any other narrowing. A convergence claim is bounded per scope, not per manifest — report it that way. 6. New blind spots discovered during an engagement get added here (this file is reference documentation, not a frozen tool). # Inferring Import Manifests from Terraform State When migrating or refactoring existing Google Cloud infrastructure managed by Terraform into Cloud Foundation Fabric (CFF), inferring the `import-manifest.yaml` directly from existing Terraform state file(s) (`.tfstate`) is the most precise and deterministic way to define the import boundary. --- ## Why State-Driven Manifest Generation? | Approach | Strength | Trade-off / Note | | :--- | :--- | :--- | | **State-Driven Inference (`manifest_from_state.py`)** | Exactly matches the existing Terraform management footprint. No risk of accidentally pulling unmanaged live assets into the stage scope. | Requires access to state files (`terraform state pull` or GCS bucket access). | | **Live Asset Survey (`inventory.py survey` + `manifest_init.py`)** | Discovers all live cloud assets, including unmanaged or out-of-band resources. | Broader denominator; requires manually waiving pre-existing or external assets. | --- ## Workflow ### 1. Retrieve the Terraform State(s) Pull the state files for the target stages or root modules: ```bash # Example for FAST Stage 0 gsutil cp gs://-iac-org-state/default.tfstate stage-0.tfstate # Or pull via Terraform CLI terraform -chdir=path/to/stage-0 state pull > stage-0.tfstate ``` For multi-stage environments (e.g., FAST stages 0, 1, 2), you can pull states for all relevant stages: ```bash terraform -chdir=fast/stages/0-org-setup state pull > stage-0.tfstate terraform -chdir=fast/stages/1-vpcsc state pull > stage-1.tfstate terraform -chdir=fast/stages/2-networking state pull > stage-2-networking.tfstate terraform -chdir=fast/stages/2-security state pull > stage-2-security.tfstate ``` ### 2. Generate the Manifest Run `manifest_from_state.py`: ```bash uv run scripts/manifest_from_state.py \ --state stage-0.tfstate \ --out import-manifest.yaml ``` To synthesize a manifest across multiple stages (note `--force`: the manifest is human-owned and gate-relevant, so it is never overwritten silently — use `--out -` to review on stdout first): ```bash uv run scripts/manifest_from_state.py \ --state stage-0.tfstate stage-1.tfstate stage-2-networking.tfstate stage-2-security.tfstate \ --out import-manifest.yaml --force ``` All state files must belong to the same organization: a state spanning several is refused rather than silently resolved to one of them. When running on an isolated stage state (e.g. `stage-2-networking.tfstate`) that contains resources across multiple folders without including an organization resource in that state, specify the intended scope root with `--root`: ```bash uv run scripts/manifest_from_state.py \ --state stage-2-networking.tfstate \ --root organizations/123456789012 \ --out import-manifest.yaml ``` ### 3. Review and Collect Review the generated `import-manifest.yaml`, then collect the live denominator: ```bash uv run scripts/inventory.py collect --manifest import-manifest.yaml --out inventory.json ``` --- ## Type & Level Mapping Reference The inference script automatically maps Terraform `google_*` resources to CAI asset types and container levels: | Terraform Resource Type | Manifest `type` | Container Levels | Special Handling | | :--- | :--- | :--- | :--- | | `google_organization_iam_*` | `iam` | `[organization]` | Authoritative & additive bindings | | `google_folder_iam_*` | `iam` | `[folder]` | Folder IAM | | `google_project_iam_*` | `iam` | `[project]` | Project IAM | | `google_service_account_iam_*` | `iam.googleapis.com/ServiceAccount` | `[project]` | Emits `iam: true` on SA type | | `google_org_policy_policy`, `google_organization_policy` | `org-policy` | `[organization, folder, project]` | Inferred from parent attribute | | `google_organization_iam_custom_role` | `iam.googleapis.com/Role` | `[organization]` | Org custom roles | | `google_project_iam_custom_role` | `iam.googleapis.com/Role` | `[project]` | Project custom roles | | `google_logging_organization_sink` | `logging.googleapis.com/LogSink` | `[organization]` | Org log sinks | | `google_logging_folder_sink` | `logging.googleapis.com/LogSink` | `[folder]` | Folder log sinks | | `google_logging_project_sink` | `logging.googleapis.com/LogSink` | `[project]` | Project log sinks | | `google_logging_project_bucket_config`| `logging.googleapis.com/LogBucket` | `[project]` | Project log buckets | | `google_logging_organization_settings` | `logging.googleapis.com/Settings` | `[organization]` | Logs Router settings singleton (CAI has no `OrganizationSettings` type) | | `google_logging_folder_settings` | `logging.googleapis.com/Settings` | `[folder]` | Logs Router settings singleton | | `google_essential_contacts_contact` | `essentialcontacts.googleapis.com/Contact` | inferred from `parent` | Essential contacts | | `google_storage_bucket_iam_*` | `storage.googleapis.com/Bucket` | `[project]` | Emits `iam: true` on the bucket type | | `google_tags_tag_value_iam_*` | `cloudresourcemanager.googleapis.com/TagValue` | `[organization]` | Emits `iam: true` on the tag value type | | `google_folder` | `cloudresourcemanager.googleapis.com/Folder` | `[organization, folder]` | Top-level and nested folders | | `google_project` | `cloudresourcemanager.googleapis.com/Project` | `[organization, folder]` | Projects | | `google_storage_bucket` | `storage.googleapis.com/Bucket` | `[project]` | GCS Buckets | | `google_service_account` | `iam.googleapis.com/ServiceAccount` | `[project]` | Service Accounts | | `google_tags_tag_key` | `cloudresourcemanager.googleapis.com/TagKey` | `[organization]` | Resource Manager Tag Keys | | `google_tags_tag_value` | `cloudresourcemanager.googleapis.com/TagValue` | `[organization]` | Tag Values | | `google_tags_tag_binding` | `cloudresourcemanager.googleapis.com/TagBinding` | `[organization, folder, project]` | Tag Bindings | | `google_project_service` | `serviceusage.googleapis.com/Service` | `[project]` | Enabled APIs | | `google_access_context_manager_*` | `identity.accesscontextmanager.googleapis.com/*` | `[organization]` | Access Policies, Perimeters, Levels | | `google_compute_network` | `compute.googleapis.com/Network` | `[project]` | VPCs | | `google_compute_subnetwork` | `compute.googleapis.com/Subnetwork` | `[project]` | Subnets | | `google_compute_global_address` | `compute.googleapis.com/Address` | `[project]` | Global IP Addresses | | `google_compute_router` | `compute.googleapis.com/Router` | `[project]` | Cloud Routers & NAT | | `google_compute_firewall` | `compute.googleapis.com/Firewall` | `[project]` | Firewall rules | | `google_compute_firewall_policy` | `compute.googleapis.com/FirewallPolicy` | `[organization, folder]` | Hierarchical Firewall Policies | | `google_compute_network_firewall_policy` | `compute.googleapis.com/NetworkFirewallPolicy` | `[project]` | Network Firewall Policies | | `google_compute_ha_vpn_gateway`, `google_compute_vpn_gateway` | `compute.googleapis.com/VpnGateway` | `[project]` | HA & Classic VPN Gateways | | `google_compute_external_vpn_gateway` | `compute.googleapis.com/ExternalVpnGateway` | `[project]` | External VPN Gateways | | `google_compute_vpn_tunnel` | `compute.googleapis.com/VpnTunnel` | `[project]` | VPN Tunnels | | `google_dns_managed_zone` | `dns.googleapis.com/ManagedZone` | `[project]` | Cloud DNS Zones | | `google_dns_response_policy` | `dns.googleapis.com/ResponsePolicy` | `[project]` | Cloud DNS Response Policies | | `google_network_connectivity_hub` | `networkconnectivity.googleapis.com/Hub` | `[project]` | NCC Hubs | | `google_network_connectivity_spoke` | `networkconnectivity.googleapis.com/Spoke` | `[project]` | NCC Spokes | | `google_storage_managed_folder` | `storage.googleapis.com/ManagedFolder` | `[project]` | Storage Managed Folders | | `google_storage_managed_folder_iam_*` | `storage.googleapis.com/ManagedFolder` | `[project]` | Emits `iam: true` on ManagedFolder | | `google_kms_key_ring` | `cloudkms.googleapis.com/KeyRing` | `[project]` | KMS Key Rings | | `google_kms_crypto_key` | `cloudkms.googleapis.com/CryptoKey` | `[project]` | KMS Keys | | `google_bigquery_dataset` | `bigquery.googleapis.com/Dataset` | `[project]` | BigQuery Datasets | | `google_pubsub_topic` | `pubsub.googleapis.com/Topic` | `[project]` | Pub/Sub Topics | | `google_pubsub_subscription` | `pubsub.googleapis.com/Subscription` | `[project]` | Pub/Sub Subscriptions | | `google_secret_manager_secret` | `secretmanager.googleapis.com/Secret` | `[project]` | Secret Manager | | `google_iam_workload_identity_pool` | `iam.googleapis.com/WorkloadIdentityPool` | `[project]` | WIF Pools | | `google_iam_workload_identity_pool_provider` | `iam.googleapis.com/WorkloadIdentityPoolProvider` | `[project]` | WIF Identity Providers | ### Types this table does not list A `google_*` type absent from the table is reported as unmapped, and the warning is precise about what that means: **this tool has no static Terraform-to-CAI row for it**. It is not a statement about Cloud Asset Inventory. Triage each one — CAI may model it (add the type by hand), it may be leaf IAM on a type CAI models (`iam: true` on the parent), CAI may genuinely not model it (`enumerate:` block or out-of-band enumeration, see [cai-blind-spots.md](./cai-blind-spots.md)), or it may be deliberately out of scope (signed waiver). Anything you resolve is worth sending back as a `TF_TYPE_MAP` entry. --- ## Multi-Scope Partitioning When resources span both Organization/Folder governance and project-contained resources, `manifest_from_state.py` produces a multi-scope configuration: ```yaml scopes: - name: org-foundation root: organizations/123456789012 levels: [organization, folder] types: - type: iam levels: [organization, folder, project] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] - name: stage-projects root: organizations/123456789012 levels: [project] include: - projects/111111111111 # prj-prod-audit-logs-0 - projects/222222222222 # prj-prod-iac-core-0 types: - type: iam levels: [organization, folder, project] - type: iam.googleapis.com/ServiceAccount levels: [project] iam: true ``` This prevents project-level queries from accidentally scanning every unrelated project in the organization while still managing org-level policies and folders. Every scope carries its own `types:` list; the generator filters the inferred types per scope (a type whose `levels` cannot fire at a scope's levels is left out, because `inventory.py` refuses dead per-scope declarations) but keeps each entry's full inferred `levels` — the intersection with the scope's `levels` happens at collect time, so the entry reads the same in every scope that carries it. Narrowing a generated list further — dropping a type or a level from one scope but not another — is a normal human refinement of the draft. Regenerating the manifest discards such refinements, so refine after generation, not before. # Manifest migration — from the retired grammar to scopes-only The import-manifest grammar changed once, and this file is the record of it. If `inventory.py` refused your manifest with *"manifest declares retired top-level key(s)"*, you are holding a manifest written for the old grammar; this page is the complete migration path. ## What changed, and why **Old grammar (retired).** A manifest declared WHERE to look and WHAT to collect in two separate places: a singular `scope:` (or a `scopes:` list) for the subtrees, and one manifest-global `types:` list shared by every scope. The only per-scope discrimination was the `levels` axis (organization / folder / project / unknown): a type declared anywhere was swept in every scope whose levels intersected its own. **New grammar (current).** `scopes:` is the only form, and **every scope carries its own `types:` list**. There is no top-level `types:` and no singular `scope:`. What a scope collects is exactly what is written on it. The motivation is per-scope type declarations — different resource types under different subtrees (org governance at org/folder level, the connectivity stack under the networking folder, project IAM only in named workload projects), which the global list could not express without waiving every out-of-domain asset by hand. Making the new form the ONLY form, rather than an option next to the old one, makes three whole classes of silent-denominator failure unrepresentable instead of merely validated: there is no inheritance to half-apply, no global list to go stale behind the scopes, and no mixed manifest whose meaning depends on which scope the reviewer happened to read. The tool refuses the old grammar rather than silently accepting half of it, because a half-read manifest would change what the denominator means — the exact failure the gates exist to prevent. ## Migration: singular `scope:` Wrap the scope as a one-element `scopes:` list and move the `types:` list inside it (indent by four spaces). Nothing else changes — same types, same levels, same include/exclude, same collected denominator. ```yaml # OLD (refused) # NEW scope: scopes: root: organizations/123 - root: organizations/123 include: include: - folders/456 - folders/456 types: types: - type: org-policy - type: org-policy levels: [organization] levels: [organization] - type: iam - type: iam levels: [organization, folder] levels: [organization, folder] ``` ## Migration: `scopes:` list with a shared top-level `types:` Copy the top-level list into EVERY scope, then (optionally, later) narrow each copy. The unnarrowed copy is semantically identical to the old manifest: the collect-time intersection of a type's `levels` with each scope's `levels` is unchanged, so the same assets enter the denominator. ```yaml # OLD (refused) scopes: - name: org-foundation root: organizations/123 levels: [organization, folder] - name: stage-projects root: organizations/123 levels: [project] include: [projects/111111111111] types: - type: iam levels: [organization, folder, project] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] # NEW — verbatim copy first, narrow later scopes: - name: org-foundation root: organizations/123 levels: [organization, folder] types: - type: iam levels: [organization, folder, project] - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] - name: stage-projects root: organizations/123 levels: [project] include: [projects/111111111111] types: - type: iam levels: [organization, folder, project] ``` Note the second scope: `cloudresourcemanager.googleapis.com/Folder` was NOT copied into it. A per-scope entry whose `levels` cannot intersect its scope's `levels` is refused as a dead declaration (`[organization, folder] ∩ [project] = ∅`), so a verbatim copy is only valid where the type could actually fire. Under the old grammar that entry was silently inactive for the scope; now it is an error — drop it from the scopes it cannot fire in. Entries whose `levels` include `unknown` are exempt and may be copied everywhere. If YAML repetition bothers you, anchors keep shared lists in one place: ```yaml scopes: - name: teams root: organizations/123 include: [folders/111] types: &folder-types - type: cloudresourcemanager.googleapis.com/Folder levels: [organization, folder] - name: rest root: organizations/123 exclude: [folders/111] types: *folder-types ``` ## Key mappings, old → new | Old | New | |---|---| | `scope:` (singular) | `scopes:` with one entry | | top-level `types:` | each scope's own `types:` (copy, then narrow) | | top-level `emission:` (manifest-wide default) | `scopes[].emission` on each scope that wants a non-default style; an omitted family falls back to the built-in default (`per-instance`; `additive` for `iam`), not to another scope | | `enumerate:` block, declared once | declared in the `types:` list of every scope that needs the sweep — per-scope lists never inherit, so the block is repeated per scope (a different scope may even carry a different block for the same type) | | `iam: true` leaf opt-in, declared once | per scope, like everything else on a type entry: a scope that omits it collects the resource without its leaf IAM | | type silently inactive in a scope (empty levels intersection) | refused as a dead declaration (drop the entry from that scope; `unknown` entries exempt) | ## What did NOT change - The type entry schema: `type`, `levels`, `iam`, `enumerate` are unchanged, including all `enumerate:` guard rails. - Scope fields: `name`, `root`, `include`, `exclude`, `levels` are unchanged, including numeric-id resolution for include/exclude. - Collect-time semantics for a faithful copy: same sweeps, same intersection rule, same deduplicated denominator. A migrated manifest that copies the old list into every (compatible) scope yields the same `assets` — diffing old and new `inventory.json` asset keys is the cheap proof. - Gates and waivers: `coverage.py` and `verify_plan.py` are untouched; existing workspaces, coverage maps and waiver ledgers keep working. ## New output you should start reading - `_meta.scopes[].declared_types` / `zero_yield_types` — per-scope yield tables. The aggregate `_meta.declared_types` cannot show a type that yields zero in one scope while non-zero in another; the per-scope record (and its stderr warning) can. Quote the per-scope tables in the step-5 report. - Every inventory entry carries `scopes: [...]` — the scope(s) that collected it, merged and sorted when scopes overlap. ## Migration checklist 1. Rewrite the manifest as above (or regenerate: `manifest_from_state.py` and `manifest_init.py` emit the new grammar directly). 2. Run `inventory.py collect` — validation is fail-closed and runs before any API call, so a wrong migration is a refused manifest with the reason, never a shrunken denominator. 3. Diff the new `inventory.json` asset keys against the last pre-migration one. Identical keys = faithful migration. Missing keys = you narrowed while migrating; deliberate narrowing belongs in its own reviewed manifest change, not folded into the migration. 4. The manifest is human-owned: the migrated file goes through the same Scope Approval gate as any other manifest edit. Worked references: `examples/import-manifest.org-foundation.yaml` (single scope) and `examples/import-manifest.multi-domain.yaml` (four scopes, per-scope types). The fail-closed rules live in SKILL.md step 0 and the cookbook's "Per-scope type declaration" subsection. # Mapping cookbook — live GCP to Cloud Foundation Fabric Review-hardened knowledge from the live verification rounds summarised in [COVERAGE.md](../COVERAGE.md). Every rule here encodes a bug that was actually made and caught; provenance markers like "(verified r9)" point at the round that proved the rule. Read the section for a resource type BEFORE mapping it. When in doubt, read the Fabric module source in this repository — never trust memory for module internals. **This document is deliberately incomplete, and always will be.** It is not a specification of what the skill supports; it is the precipitate of a method — the accumulated residue of mapping real resources against a live organization and keeping what survived the gates. A type with no section here is not unsupported. It means nobody has run the method on it yet. So entries are born the same way every time, and that is the only way they should be born: someone maps a live resource by reading the module source, converges both gates, and writes down what cost them a cycle. See "Step 3b — when the cookbook has never seen this type" in [SKILL.md](../SKILL.md) for the loop. **If you map a type that has no section here, adding one is part of your run report** — that is what keeps this file growing rather than the same traps being paid for on every engagement. What belongs in an entry: the CAI asset-type string as live output returns it, the module addresses and `for_each` key formats, the import ID format, the ForceNew inputs that must mirror live values, and every trap that cost a cycle. Mark surfaces you did not exercise as unverified — an honest gap is worth more than an implied guarantee. Contents: canonical module map · factory emission (opt-in) · universal rules · workspace layout · scaffolding scripts · import-ID table · per-type rules (organization, folders, projects, service accounts, log sinks, logging buckets, VPC networks, root module) · fallback and accelerator. **Where the fielded data lives.** Address patterns, import IDs, CAI asset types and verification rounds are held in [`address-map.yaml`](./address-map.yaml) and rendered into the import-ID table below; the prose here owns everything else — the traps, the causal reasoning and the capability gaps. Add a new address to the YAML, not to the table, and run `uv run scripts/address_map.py` to check both. ## Canonical module map (normative) Fabric modules are the product — this table is the default mapping, not a suggestion. Raw resources require a documented module capability gap (see "Fallback" at the end). Emission has two modes — `per-instance` (the default: one module instance per container, inputs as HCL) and `factory` (data-driven YAML consumed by a Fabric factory carrier). The mode is chosen per resource family in the manifest's `emission:` block (see "Factory emission (opt-in)" below); families with `—` in the factory column have exactly one mode and ignore the knob. Some sub-resources are factory-only by Fabric convention regardless of the knob: org policies, custom roles and custom constraints are carried as `data/` YAML inside their owning `organization` / `folder` / `project` instance. | Scope / resource family | Default (`per-instance`) | `factory` emission — carrier | Carries | |---|---|---|---| | Organization | `modules/organization` (single instance) | — | org IAM + audit configs, org policies (factory), custom roles (factory), org sinks | | Folder | `modules/folder` — one instance per folder | `project-factory` — hierarchy `data/`, one YAML dir per folder, path = key | the folder itself, folder IAM, folder org policies (factory), folder sinks | | Project | `modules/project` — one instance per project | `project-factory` — one project YAML per project | the project, services, project IAM, project org policies, metadata | | Service accounts | `modules/iam-service-account` | `project-factory` — `service_accounts:` block in the owning project's YAML (only when that project is factory-emitted) | SA + its IAM | | Log buckets | `modules/logging-bucket` | — | bucket config (sink destinations) | | VPC networks | `modules/net-vpc` | subnets only: `net-vpc` `factories_config.subnets_folder` | network, subnets (incl. proxy-only), routes | | VPC firewall rules (legacy) | `modules/net-vpc-firewall` | `net-vpc-firewall` `factories_config.rules_folder` | custom rules + the four `default_rules_config` rules | | Cloud NAT & Routers | `modules/net-cloudnat` | — | Cloud NAT + its Cloud Router (NAT-only routers) | | HA VPN | `modules/net-vpn-ha` | — | HA VPN gateway, external gateway, tunnels, its own Cloud Router, interfaces, BGP peers | | Cloud DNS | `modules/dns` | — (no factory: recordsets are the `recordsets` input, keyed `" "`) | managed zones, recordsets | | DNS Response Policy | `modules/dns-response-policy` | `dns-response-policy` `factories_config.rules` | response policies & rules | | NCC | `modules/ncc-spoke-ra` (hub + router-appliance spokes only) | — | hub; linked-VPC/VPN spokes are a capability gap | | Firewall policy rules | `modules/net-firewall-policy` inputs | `net-firewall-policy` `factories_config` rule files | ingress/egress rules | | Other | nearest Fabric module for the family; check `modules/` in-repo | see `FACTORIES.md` at the repo root — the authoritative factory catalog (every module factory flows through `factories_config`) | — | Instance keys: pick short stable slugs at first import (record them in `coverage-map.yaml`); keys are immutable afterwards. Because the key — not the display name — defines the address, a console rename plans as an in-place `name` update, not destroy/create (verified r6). ## Factory emission (opt-in) **Status: cookbook-level (C) — rules below are derived from module source in this repository, not yet exercised against live imports.** Everything in this section meets that standard and no higher; treat unmarked claims as unverified. The manifest owns the choice, per resource family. `emission` is a map of family → style, where family names are the canonical-map rows and styles are family-specific: `per-instance` (default) or `factory` for module-instance families (those without a factory carrier reject `factory`), and `additive` (default) or `authoritative` for the `iam` pseudo-family (see "Organization and containers: IAM"). It lives on the scope entry, like everything else in the manifest: the grammar is scopes-only, and there is no top-level position. Resolution, per family: ```text scopes[].emission.

AI Agent Protocol & Instructions

Explore Related Technologies & Topics