cloud-foundation-fabric Documentation

File: adrs/fast/README.md Copy Chapter

FAST architectural documents

This folder contains assorted bits of documentation used to log current architectural choices, or past decisions. Format is inspired by Michael Nygard's decision record template.

File: adrs/modules/README.md Copy Chapter

Fabric modules architectural documents

This folder contains assorted bits of documentation used to log current architectural choices, or past decisions. Format is inspired by Michael Nygard's decision record template.

File: assets/logos/README.md Copy Chapter

Logo Specifications

• font used

File: blueprints/README.md Copy Chapter

Deprecated: Terraform Blueprints for Google Cloud

This directory is deprecated as of release v44.0.0.

The blueprints that were previously in this directory have been deprecated. Some of the old blueprints will be refactored as either FAST project templates or module-level recipes. If there is a specific blueprint that you would like to see refactored, please submit a PR or open a feature request.

You can still browse and checkout the old code from the v43.0.0 tag, or download the v43.0.0 release. Please note that this notice will also be removed in a future version.

File: fast/addons/2-networking-test/README.md Copy Chapter

Network Test Resources

This add-on allows creating an arbitrary number of Compute instances and service accounts, and is designed to simplify testing of the FAST networking stage.

• Design overview and choices
• How to run this stage
  • Provider and Terraform variables
  • Impersonating the automation service account
  • Variable configuration
  • Running the stage
• Files
• Variables
• Outputs

Design overview and choices Copy Chapter

The add-on is very simple, as it just deals with creating service accounts and instances using pre-existing projects, networks and subnets.

To allow creation of portable configurations, it extensively leverages FAST output variables so that project ids, network ids, regions, and subnet ids can refer to the relevant FAST aliases.

A simple factory is also provided, so that YAML configurations can be used instead of Terraform tfvars.

How to run this stage Copy Chapter

Once the main networking stage has been configured and applied, the following configuration is added the the resource management fast_addon variable to create the add-on provider files, and its optional CI/CD resources if those are also required. The add-on name (networking-test) is customizable, in case the add-on needs to be run multiple times to create gateways in different projects.

fast_addon = {
  networking-test = {
    parent_stage = "2-networking"
  }
}

Provider and Terraform variables Copy Section

As all other FAST stages, the mechanism used to pass variable values and pre-built provider files from one stage to the next is also leveraged here.

The commands to link or copy the provider and terraform variable files can be easily derived from the fast-links.sh script in the FAST stages folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following example uses local files but GCS behaves identically.

# File linking commands for Test resources networking add-on stage

# provider file
ln -s ~/fast-config/providers/2-networking-test-providers.tf ./

# input files from other stages
ln -s ~/fast-config/tfvars/2-networking.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
ln -s ~/fast-config/2-networking-test.auto.tfvars ./

If a factory is used and neither the default factory paths nor the resource name prefix in the name variable need to be changed, the last file is unnecessary as there's no additional configuration for this add-on.

Impersonating the automation service account Copy Section

The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The gcp-devops and organization-admins groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.

Variable configuration Copy Section

Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:

• variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the 0-globals.auto.tfvars.json file linked or copied above
• variables which refer to resources managed by previous stages, which are prepopulated here via the 0-org-setup.auto.tfvars.json, 2-networking.auto.tfvars.json files linked or copied above
• and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom terraform.tfvars file

The first two sets are defined in the variables-fast.tf file, the latter set in the variables.tf file. The full list of variables can be found in the Variables table at the bottom of this document.

Once output files are in place, define your addon configuration in a tfvars file if needed (see section above).

Running the stage Copy Section

Once provider and variable values are in place and the correct user is configured, the stage can be run:

terraform init
terraform apply

Files Copy Chapter

Variables Copy Chapter

Outputs Copy Chapter

File: fast/addons/2-networking-swp/README.md Copy Chapter

Secure Web Proxy Add-on

This add-on allows creating an arbitrary number of Secure Web Proxy (SWP), including resources required to enable TLS inspection.

This diagram shows the resources used by this add-on, and their relationships with its networking parent stage.

• Design overview and choices
• How to run this stage
  • Provider and Terraform variables
  • Impersonating the automation service account
  • Variable configuration
  • Running the stage
• Files
• Variables
• Outputs

Design overview and choices Copy Chapter

This add-on is intentionally self-contained to allow directly using it to implement different designs, via a single instance or multiple instances.

All project-level resources in this stage are created in the same project, so that dependencies and IAM configurations are kept as simple as possible, and everything is within the same span of control.

The controlling project is usually one of those already created and managed by the networking stage: the landing host project, or a shared environment project if that exists. Alternatively, a dedicated project can be created and used here provided the necessary IAM and organization policies configurations are also defined.

How to run this stage Copy Chapter

Once the main networking stage has been configured and applied, the following configuration is added the the resource management fast_addon variable to create the add-on provider files, and its optional CI/CD resources if those are also required. The add-on name (networking-swp) is customizable, in case the add-on needs to be run multiple times to create gateways in different projects.

fast_addon = {
  networking-swp = {
    parent_stage = "2-networking"
    # cicd_config = {
    #   identity_provider = "github-test"
    #   repository = {
    #     name   = "test/swp"
    #     type   = "github"
    #     branch = "main"
    #   }
    # }
  }
}

Provider and Terraform variables Copy Section

As all other FAST stages, the mechanism used to pass variable values and pre-built provider files from one stage to the next is also leveraged here.

The commands to link or copy the provider and terraform variable files can be easily derived from the fast-links.sh script in the FAST stages folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following example uses local files but GCS behaves identically.

../../stages/fast-links.sh ~/fast-config
# File linking commands for Secure Web Proxy networking add-on

# provider file
ln -s ~/fast-config/providers/2-networking-swp-providers.tf ./

# input files from other stages
ln -s ~/fast-config/tfvars/0-globals.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/0-org-setup.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/2-networking.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
ln -s ~/fast-config/2-networking-swp.auto.tfvars ./

# optional files
ln -s ~/fast-config/tfvars/2-security.auto.tfvars.json ./

Impersonating the automation service account Copy Section

The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The gcp-devops and organization-admins groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.

Variable configuration Copy Section

Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:

• variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the 0-globals.auto.tfvars.json file linked or copied above
• variables which refer to resources managed by previous stages, which are prepopulated here via the 0-org-setup.auto.tfvars.json, 2-networking.auto.tfvars.json files linked or copied above
• and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom terraform.tfvars file

The first two sets are defined in the variables-fast.tf file, the latter set in the variables.tf file. The full list of variables can be found in the Variables table at the bottom of this document.

Note that the outputs_location variable is disabled by default, you need to explicitly set it in your terraform.tfvars file if you want output files to be generated by this stage. This is a sample terraform.tfvars that configures it, refer to the bootstrap stage documentation for more details:

outputs_location = "~/fast-config"

Once output files are in place, define your addon configuration in a tfvars file. This is an example of configuring this addon, with optional variable attributes filled in for illustration purposes.

Note that project id and networking variables use interpolation from FAST, and refer to the aliased resource names for portability. This is of course optional, and full resource ids can be used instead if needed.

certificate_authority = {
  ca_configs = {
    swp = {
      deletion_protection = false
      subject = {
        common_name  = "fast-test-00.joonix.net"
        organization = "FAST Test 00"
      }
    }
  }
}
locations = {
  pri = "primary"
}
outputs_location = "~/fast-config"
project_id       = "prod-landing"
swp_configs = {
  shared = {
    network_id    = "prod-landing"
    subnetwork_id = "net"
  }
  # any other option supported by the net-swp module can be used here
}
tls_inspection_policy = {
  exclude_public_ca_set = true
}

Running the stage Copy Section

Once provider and variable values are in place and the correct user is configured, the stage can be run:

terraform init
terraform apply

Files Copy Chapter

Variables Copy Chapter

Outputs Copy Chapter

File: fast/addons/2-networking-ngfw/README.md Copy Chapter

NGFW Enterprise Networking Add-on

This add-on includes all configurations and resources required to activate Cloud Next Generation Firewall, and associate its endpoints to an arbitrary number of VPC networks.

This diagram shows the resources used by this add-on, and their relationships with its networking parent stage.

• Design overview and choices
• How to run this stage
  • Provider and Terraform variables
  • Impersonating the automation service account
  • Variable configuration
  • Running the stage
  • Using add-on resources from the networking stage
• Files
• Variables

Design overview and choices Copy Chapter

This add-on is intentionally self-contained to allow directly using it to implement different designs, via a single instance or multiple instances.

All project-level resources in this stage with the exception of VPC associations are created in the same project, so that dependencies and IAM configurations are kept as simple as possible, and everything is within the same span of control.

The controlling project is usually one of those already created and managed by the networking stage: the landing host project, or a shared environment project if that exists. Alternatively, a dedicated project can be created and used here provided the necessary IAM and organization policies configurations are also defined.

How to run this stage Copy Chapter

Once the main networking stage has been configured and applied, the following configuration is added to the org setup stage.

First, the new provider file is declared in the defaults.yaml file.

# defaults.yaml (snippet)

output_files:
  # ...
  providers:
    # ...
    2-networking-ngfw:
      bucket: $storage_buckets:iac-0/iac-stage-state
      prefix: 2-networking-ngfw
      service_account: $iam_principals:service_accounts/iac-0/iac-networking-rw

Then, the GCS folder (shown here) or bucket for the Terraform state is defined in the IaC project.

# projects/iac-0.yaml
buckets:
  # ...
  iac-stage-state:
    description: Terraform state for stage automation.
    managed_folders:
      # ...
      2-networking-ngfw:
        iam:
          roles/storage.admin:
            - $iam_principals:service_accounts/iac-0/iac-networking-rw
          $custom_roles:storage_viewer:
            - $iam_principals:service_accounts/iac-0/iac-networking-ro

And finally, grant extra roles at the organization level to the networking service accounts.

# organization/.config.yaml
iam_by_principals:
  # ...
  $iam_principals:service_accounts/iac-0/iac-networking-rw:
    - roles/compute.orgFirewallPolicyAdmin
    - roles/compute.xpnAdmin
    # add the custom role
    - $custom_roles:ngfw_enterprise_admin
  $iam_principals:service_accounts/iac-0/iac-networking-ro:
    - roles/compute.orgFirewallPolicyUser
    - roles/compute.viewer
    # add the custom role
    - $custom_roles:ngfw_enterprise_viewer

If VPC-SC is used, an additional ingress policy needs to be added to the perimeter to allow the NGFW service agent to reach the Certificate Authority Service. Edit and enable the following policy.

from:
  access_levels:
    - "*"
  identities:
    # TODO: change to actual NGFW service identity
    - serviceAccount:service-1234567890@gcp-sa-networksecurity.iam.gserviceaccount.com
to:
  operations:
    - method_selectors:
        - "*"
      service_name: privateca.googleapis.com
  resources:
    # TODO: change to project number where CAS lives
    - projects/1234567890

Provider and Terraform variables Copy Section

As all other FAST stages, the mechanism used to pass variable values and pre-built provider files from one stage to the next is also leveraged here.

The commands to link or copy the provider and terraform variable files can be easily derived from the fast-links.sh script in the FAST stages folder, passing it a single argument with the local output files folder (if configured) or the GCS output bucket in the automation project (derived from stage 0 outputs). The following example uses local files but GCS behaves identically.

../../stages/fast-links.sh ~/fast-config
# File linking commands for NGFW Enterprise networking add-on stage

# provider file
ln -s ~/fast-config/providers/2-networking-ngfw-providers.tf ./

# input files from other stages
ln -s ~/fast-config/tfvars/0-globals.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/0-org-setup.auto.tfvars.json ./
ln -s ~/fast-config/tfvars/2-networking.auto.tfvars.json ./

# conventional place for stage tfvars (manually created)
ln -s ~/fast-config/2-networking-ngfw.auto.tfvars ./

# optional files
ln -s ~/fast-config/tfvars/2-security.auto.tfvars.json ./

Impersonating the automation service account Copy Section

The preconfigured provider file uses impersonation to run with this stage's automation service account's credentials. The gcp-devops and organization-admins groups have the necessary IAM bindings in place to do that, so make sure the current user is a member of one of those groups.

Variable configuration Copy Section

Variables in this stage -- like most other FAST stages -- are broadly divided into three separate sets:

• variables which refer to global values for the whole organization (org id, billing account id, prefix, etc.), which are pre-populated via the 0-globals.auto.tfvars.json file linked or copied above
• variables which refer to resources managed by previous stages, which are prepopulated here via the 0-org-setup.auto.tfvars.json, 2-networking.auto.tfvars.json files linked or copied above
• and finally variables that optionally control this stage's behaviour and customizations, and can to be set in a custom terraform.tfvars file

The first two sets are defined in the variables-fast.tf file, the latter set in the variables.tf file. The full list of variables can be found in the Variables table at the bottom of this document.

Note that the outputs_location variable is disabled by default, you need to explicitly set it in your terraform.tfvars file if you want output files to be generated by this stage. This is a sample terraform.tfvars that configures it, refer to the bootstrap stage documentation for more details:

outputs_location = "~/fast-config"

Once output files are in place, define your addon configuration in a tfvars file. This is an example of configuring this addon, with optional variable attributes filled in for illustration purposes.

certificate_authorities = {
  # if CA pools defined in the security stage are used this is optional
  ngfw-0 = {
    location = "europe-west8"
    ca_pool_config = {
      create_pool = {
        enterprise_tier = true
      }
    }
    ca_configs = {
      ca-0 = {
        deletion_protection = false
        subject = {
          common_name  = "example.org"
          organization = "Test Organization"
        }
        # optional: sign this CA as a subordinate of a 2-security or local Root CA
        # subordinate_config = {
        #   root_ca_id = "$ca_ids:prod-ca-0-prod-ca-0-0"
        # }
      }
    }
  }
}
ngfw_config = {
  name           = "ngfw-0"
  endpoint_zones = ["europe-west8-b"]
  network_associations = {
    prod = {
      # VPC ids defined in the network stage can be referred to via short name
      # vpc_id              = "prod-spoke-0"
      vpc_id                = "projects/xxx-prod-net-spoke-0/global/networks/prod-spoke-0"
      tls_inspection_policy = "ngfw-0"
    }
  }
}
outputs_location = "~/fast-config"
project_id       = "xxx-prod-net-landing-0"
security_profiles = {
  ngfw-0 = {
    # these are optional and shown here for convenience
    threat_prevention_profile = {
      severity_overrides = {
        informational-allow = {
          action   = "ALLOW"
          severity = "INFORMATIONAL"
        }
      }
      threat_overrides = {
        allow-280647 = {
          action    = "ALLOW"
          threat_id = "280647"
        }
      }
    }
    url_filtering_profile = {
      allow-example = {
        action   = "ALLOW"
        priority = 100
        urls     = ["example.com"]
      }
      deny-all = {
        action   = "DENY"
        priority = 200
        # urls defaults to ["*"]
      }
    }
  }
}
tls_inspection_policies = {
  ngfw-0 = {
    # reference the pool defined above, or an external one
    # CA pools defined in the security stage can be referred to via short name
    ca_pool_id   = "ngfw-0"
    location     = "europe-west8"
    trust_config = "ngfw-0"
  }
}
trust_configs = {
  ngfw-0 = {
    location = "europe-west8"
    allowlisted_certificates = {
      server-0 = "~/fast-config/data/2-networking-ngfw/server-0.cert.pem"
    }
    trust_stores = {
      ludo-joonix = {
        intermediate_cas = {
          issuing-ca-1 = "~/fast-config/data/2-networking-ngfw/intermediate.cert.pem"
        }
        trust_anchors = {
          root-ca-1 = "~/fast-config/data/2-networking-ngfw/ca.cert.pem"
        }
      }
    }
  }
}

Running the stage Copy Section

Once provider and variable values are in place and the correct user is configured, the stage can be run:

terraform init
terraform apply

Using add-on resources from the networking stage Copy Section

Security profiles group defined here are exported via output variable file, and can be consumed in the firewall policies defined in the networking stage.

When removing a url_filtering_profile from an existing entry in security_profiles, Terraform schedules destruction of the URL_FILTERING security profile before the in-place update of the parent google_network_security_security_profile_group, which the API rejects with Error 400: Resource is already being used. Detach the profile from the security profile group first with gcloud network-security security-profile-groups update <NAME> --location=global --organization=<ORG_ID> --clear-url-filtering-profile and then run terraform apply.

Files Copy Chapter

Variables Copy Chapter

File: fast/addons/README.md Copy Chapter

FAST add-ons

Each of the folders contained here is a separate "add-on" that can be used to add extra functionality to a specific stage.

Add-ons can be thought of as additional thin layers on top of a stage, that reuse its IaC resources and leverage the same IAM configuration: the same service accounts are used to run the add-on, and state configuration is stored in the same bucket as their "parent stage" under a different prefix.

Add-on configuration Copy Chapter

To configure an add-on, once you have identified its "parent stage" (networking, security, etc.) you have to configure the org setup stage so that its providers file, and associated GCS resources to host the state file, are created. The following example configures resources for the NGFW Networking add-on.

First, the new provider file is declared in the defaults.yaml file.

# defaults.yaml (snippet)

output_files:
  # ...
  providers:
    # ...
    2-networking-ngfw:
      bucket: $storage_buckets:iac-0/iac-stage-state
      prefix: 2-networking-ngfw
      service_account: $iam_principals:service_accounts/iac-0/iac-networking-rw

Then, the GCS folder (shown here) or bucket for the Terraform state is defined in the IaC project.

# projects/iac-0.yaml
buckets:
  # ...
  iac-stage-state:
    description: Terraform state for stage automation.
    managed_folders:
      # ...
      2-networking-ngfw:
        iam:
          roles/storage.admin:
            - $iam_principals:service_accounts/iac-0/iac-networking-rw
          $custom_roles:storage_viewer:
            - $iam_principals:service_accounts/iac-0/iac-networking-ro

Once the org setup stage is ready and applied, the add-on can be run using the generated provider files. The fast-links.sh can be used to link or copy the relevant files to the add-on, in the same way it's used for regular stages.

File: fast/extras/0-cicd-gitlab/README.md Copy Chapter

FAST Gitlab repository management

This small extra stage allows creating and populating Gitlab groups and projects
used to host FAST stages code, including rewriting of module sources and secrets
used for private modules repository access.

It is designed for use in a Gitlab self-managed, and is only meant as a one-shot
solution with perishable state especially when used for initial population, as
you don't want Terraform to keep overwriting your changes with initial versions
of files.

Gitlab provider credentials Copy Chapter

A Gitlab token
is needed to authenticate against their API. The token needs admin permissions,
like shown in this screenshot:

Once a token is available set it in the gitlab_config.access_token terraform
variable before running Terraform. You might also want to update the default
Gitlab hostname and SSH port on the gitlab_config section.

Variable configuration Copy Chapter

Gitlab Config Copy Section

The gitlab_config variable defines where projects will be hosted.
GitLab can either be the SaaS offering (with the default hostname gitlab.com)
or a self-hosted instance with a custom FQDN. If hostname is set to gitlab.com, you must
also set the saas_group variable to specify the group path where resources will be created.

This is an example that configures a SaaS gitlab instance using my_group/gcp as default group :

gitlab_config = {
  access_token = "glpat-XXX"
  hostname     = "gitlab.com"
  ssh_port     = 22
  saas_group   = "my_group/gcp"
}
# tftest skip

This is an example that configures an on-premise gitlab instance :

gitlab_config = {
  access_token = "glpat-XXX"
  hostname     = "my-gitlab.example.com"
}
# tftest skip

Modules project and sources Copy Section

The modules_config variable controls creation and management of both the
Gitlab project hosting fast modules and key and secret used to access it, and
indirectly control population of initial files: if the modules_config variable
is not specified no module repository is know to the code, so module source
paths cannot be replaced, and initial population of files cannot happen. If the
variable is specified, an optional source_ref attribute can be set to the
reference used to pin modules versions.

This is an example that configures the modules project in Gitlab in a "shared"
group with source ref:

modules_config = {
  project_name = "modules"
  key_config = {
    create_key     = true
    create_secrets = true
  }
  group = "shared"
}
# tftest skip

If the modules are located in a non modules only repository, use the
module_prefix attribute to set the location of your modules within the

modules_config = {
  project_name = "modules"
  key_config = {
    create_key     = true
    create_secrets = true
  }
  group         = "shared"
  module_prefix = "modules/"
}
# tftest skip

In the above example, no key options are set so it's assumed modules will be
fetched from a public repository. If modules repository authentication is needed
the key_config attribute also needs to be set.

If no keypair path is specified an internally generated key will be stored as a
deploy key in the modules project, and as secrets in the stage repositories:

modules_config = {
  project_name = "modules"
  key_config = {
    create_key     = true
    create_secrets = true
  }
  group = "shared"
  key_config = {
    create_key     = true
    create_secrets = true
  }
}
# tftest skip

To use an existing keypair pass the path to the private key, the public key name
is assumed to have the same name ending with the .pub suffix. This is useful
in cases where the access key has already been set in the modules repository,
and new repositories need to be created and their corresponding secret set:

modules_config = {
  project_name = "modules"
  key_config = {
    create_key     = true
    create_secrets = true
  }
  group = "shared"
  key_config = {
    create_secrets = true
    keypair_path   = "~/modules-repository-key"
  }
}
# tftest skip

Projects Copy Section

The projects variable is where you configure which projects to create in which
groups and whether initial population of files is desired.

This is an example that creates repositories for stages 00 and 01 and 02:

projects = {
  fast_0_org_setup = {
    create_options = {
      description = "FAST org setup."
      features = {
        issues = true
      }
    }
    group         = "org-admins"
    populate_from = "../../stages/0-org-setup"
    workflow_file = "org-setup-workflow.yaml"
  }
}
# tftest skip

The create_options repository attribute controls creation: if the attribute is
not present, the repository is assumed to be already existing.

The group attribute

Initial population depends on a modules repository being configured in
the modules_config variable described in the preceding section and on
thepopulate_from attributes in each repository where population is required,
which point to the folder holding the files to be committed.

Each repository may contain some sample tfvars and data files that can be used
as a starting point for your own files. By default, the samples are not
populate. However, you can enable this by setting the populate_samples
attribute to true. Here's an updated example:

projects = {
  fast_0_org_setup = {
    create_options = {
      description = "FAST org setup."
      features    = {
        issues = true
      }
    }
    group           = "org-admins"
    populate_from   = "../../stages/0-org-setup"
    populate_sample = true
    workflow_file   = "org-setup-workflow.yaml"
  }
}
# tftest skip

Please note that setting populate_samples to true will populate the sample
files to the repository, potentially overwriting any existing files with the
same name. To minimize the risk of overwriting existing files, we populate the
original data directory to a data.sample directory. In any case, be careful
when enabling this option and review commit history to check any changes made to
the sample files.

Each project is created inside a group, this mapping is done via group
attribute being set on the project. The group attribute is either a reference
to the key of a group to create in the groups variable or the id of an
existing group in Gilab. Please find below a full example of projects + groups
initialization.

projects = {
  fast_0_org_setup = {
    create_options = {
      description = "FAST org setup."
      features    = {
        issues = true
      }
    }
    group           = "org-admins"
    populate_from   = "../../stages/0-org-setup"
    populate_sample = true
    workflow_file   = "org-setup-workflow.yaml"
  }
}

groups = {
  org-admins = {
    name        = "gcp-org-admins"
    path        = "gcp-org-admins"
    description = "GCP Organization administrators"
  }
}
# tftest skip

Groups configuration Copy Section

The groups variable can be used to create groups in Gitlab, please find below
a sample usage of this variable.

groups = {
  org-admins = {
    name        = "gcp-org-admins"
    path        = "gcp-org-admins"
    description = "GCP Organization administrators"
  }
}
# tftest skip

Commit configuration Copy Section

An optional variable commit_config can be used to configure the author, email,
and message used in commits for the initial population of files. Its defaults
are probably fine for most use cases.

How to run this stage Copy Chapter

Connect to Gitlab as root user and create a personal access token with permissions as per the gitlab provider credentials.
Set the newly created personal access as gitlab_config.access_token variable
and then issue the following commands:

gcloud storage cp gs://${prefix}-prod-iac-core-outputs-0/workflows/*-workflow.yaml ./workflows/

This will download Gitlab CICD workflow files generated during 0-org-setup stage
on the local .workflows directory for later being uploaded on the new Gitlab
projects.

Set http_proxy and https_proxy env vars to http://localhost:3128 and then
run:

terraform init
terraform apply

Navigate Gitlab URL and check whether new groups and projects has been
bootstrapped successfully. Update the default pipeline template replacing the
ssh-keyscan default configuration with the following commands:

ssh-keyscan -p 2222 -H 'gitlab.gcp.example.com' >> ~/.ssh/known_hosts
ssh-keyscan -p 2222 gitlab.gcp.example.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_hosts

Please specify the right hostname and port for the gitlab executor to fetch the
right host key otherwise you might get an Host key verification failed. during
terraform init.

Try to create a merge request to trigger a CI pipeline on one of FAST stages and
check if the plan pipeline executes successfully.

Files Copy Chapter

Variables Copy Chapter

Outputs Copy Chapter

File: fast/extras/0-cicd-github/README.md Copy Chapter

FAST GitHub repository management

This small extra stage allows creating and populating GitHub repositories used to host FAST stage code, including rewriting of module sources and secrets used for private modules repository access.

It is designed for use in a GitHub organization, and is only meant as a one-shot solution with perishable state especially when used for initial population, as you don't want Terraform to keep overwriting your changes with initial versions of files.

Initial population is only meant to be used with actual stage, while populating the modules repository should be done by hand to avoid hitting the GitHub hourly limit for their API.

Once initial population is done, you need to manually push to the repository

• the .tfvars file with custom variable values for your stages
• the workflow configuration file generated by FAST stages

A bug in the provider results in corrupted binary files. Use this code for text files only, and at your own risk. See discussion in #2503.

GitHub provider credentials Copy Chapter

A GitHub token is needed to authenticate against their API. The token needs organization-level permissions, like shown in this screenshot:

Once a token is available set it in the GITHUB_TOKEN environment variable before running Terraform.

Variable configuration Copy Chapter

The organization required variable sets the GitHub organization where repositories will be created, and is used to configure the Terraform provider.

Modules repository and sources Copy Section

The modules_config variable controls creation and management of the key and secret used to access the private modules repository, and indirectly control population of initial files: if the modules_config variable is not specified no module repository is know to the code, so module source paths cannot be replaced, and initial population of files cannot happen. If the variable is specified, an optional source_ref attribute can be set to the reference used to pin modules versions.

This is an example that configures the modules repository name and an optional reference, enabling initial population of repositories where the feature has been turned on:

modules_config = {
  repository_name = "GoogleCloudPlatform/cloud-foundation-fabric"
  source_ref      = "v19.0.0"
}
# tftest skip

If the modules are located in a non modules only repository, use the module_prefix attribute to set the location of your modules within the repository:

modules_config = {
  repository_name = "GoogleCloudPlatform/cloud-foundation-fabric"
  module_prefix   = "modules/"
}
# tftest skip

In the above example, no key options are set so it's assumed modules will be fetched from a public repository. If modules repository authentication is needed the key_config attribute also needs to be set.

If no keypair path is specified an internally generated key will be stored as an access key in the modules repository, and as secrets in the stage repositories:

modules_config = {
  repository_name = "GoogleCloudPlatform/cloud-foundation-fabric"
  key_config = {
    create_key     = true
    create_secrets = true
  }
}
# tftest skip

To use an existing keypair pass the path to the private key, the public key name is assumed to have the same name ending with the .pub suffix. This is useful in cases where the access key has already been set in the modules repository, and new repositories need to be created and their corresponding secret set:

modules_config = {
  repository_name = "GoogleCloudPlatform/cloud-foundation-fabric"
  key_config = {
    create_secrets = true
    keypair_path   = "~/modules-repository-key"
  }
}
# tftest skip

Repositories Copy Section

The repositories variable is where you configure which repositories to create and whether initial population of files is desired.

This is an example that creates repositories for stages 00 and 01, and populates initial files for stages 00, 01, and 02:

repositories = {
  fast_00_org_setup = {
    create_options = {
      description = "FAST org setup."
      features = {
        issues = true
      }
    }
    populate_from = "../../stages/0-org-setup"
  }
}
# tftest skip

The create_options repository attribute controls creation: if the attribute is not present, the repository is assumed to be already existing.

Initial population depends on a modules repository being configured in the modules_config variable described in the preceding section and on thepopulate_from attributes in each repository where population is required, which point to the folder holding the files to be committed.

Each repository may contain some sample tfvars and data files that can be used as a starting point for your own files. By default, the samples are not populate. However, you can enable this by setting the populate_samples attribute to true. Here's an updated example:

repositories = {
  fast_00_org_setup = {
    create_options = {
      description = "FAST org setup."
      features = {
        issues = true
      }
    }
    populate_from = "../../stages/0-org-setup"
    populate_samples = true
  }
}
# tftest skip

Please note that setting `populate_samples` to `true` will populate the sample files to the repository, potentially overwriting any existing files with the same name. To minimize the risk of overwriting existing files, we populate the original `data` directory to a `data.sample` directory. In any case, be careful when enabling this option and review commit history to check any changes made to the sample files.

### Commit configuration

An optional variable `commit_config` can be used to configure the author, email, and message used in commits for the initial population of files. Its defaults are probably fine for most use cases.

### Pull Request configuration

An optional variable `pull_request_config` can be used to configure the title, body, head_ref, and base_ref of the pull request created for the initial population or update of files. By default, no pull request is created. To create a pull request, set the `create` attribute to `true`. `base_ref` defaults to `main` and `head_ref` to the head branch name. If the head branch does not exist, it will be created from the base_ref branch.

```hcl
pull_request_config = {
  create   = true
  title    = "FAST: initial loading or update"
  body     = ""
  base_ref = "main"
  head_ref = "fast-loader"
}
# tftest skip

To start using a pull request workflow, if the initial loading was created without a pull request in the past, please use the following command to delete the actual branch files from the Terraform state to keep it in the current state:

terraform state list | grep github_repository_file | awk '{print "terraform state rm '\''"$1"'\''"}'

Files Copy Chapter

Variables Copy Chapter

Outputs Copy Chapter

File: fast/extras/README.md Copy Chapter

FAST extra stages

This folder contains additional helper stages for FAST, which can be used to simplify specific operational tasks:

• GitHub repository management
• Gitlab repository management

File: fast/project-templates/data-mongodb/README.md Copy Chapter

MongoDB Atlas

This simple setup allows creating and configuring a managed MongoDB Atlas cluster, and connecting it to a local VPC network via Private Endpoints.

Prerequisites Copy Chapter

The project.yaml file describes the project-level configuration needed in terms of API activation and IAM bindings.

If you are deploying this inside a FAST-enabled organization, the file can be lightly edited to match your configuration, and then used directly in the project factory.

This Terraform can of course be deployed using any pre-existing project. In that case use the YAML file to determine the configuration you need to set on the project:

• enable the APIs listed under services
• grant the permissions listed under iam to the principal running Terraform, either machine (service account) or human

Variable Configuration Copy Chapter

Configuration is mostly done via the atlas_config, database_users, and vpc_config variables. Note that:

• VPC configuration can be set to reference a Shared VPC Host network like shown below, or an in-project network if that is preferred
• the PSC CIDR block is used to allocate the required 50 endpoint addresses in the VPC, so it needs to be large enough to accommodate them
• the Atlas region must match the GCP subnetwork region

Bringing up a cluster and the associated connectivity from scratch will require approximately 30 minutes.

atlas_config = {
  cluster_name     = "test-0"
  organization_id  = "fmoajt0b2fwdvp9yvu7m7zl2"
  project_name     = "my-atlas-project"
  region           = "NORTH_AMERICA_NORTHEAST_1"
  database_version = "7.0"
  instance_size    = "M10"
  provider = {
    public_key  = "xxxx"
    private_key = "xxxxx-xxxx-xxxx-xxxx-xxxxxxxx"
  }
}
database_users = {
  my-db-user = {
    password_wo         = "s3cr3t-p4ssw0rd"
    password_wo_version = 1
    labels = {
      environment = "test"
    }
    roles = {
      app = {
        database_name = "app"
        role_name     = "readWrite"
      }
    }
    scopes = {
      test-0 = {
        type = "CLUSTER"
      }
    }
  }
}
project_id = "my-prod-shared-mongodb-0"
vpc_config = {
  network_name   = "dev-spoke-0"
  subnetwork_id  = "projects/my-dev-net-spoke-0/regions/northamerica-northeast1/subnetworks/gce"
  psc_cidr_block = "10.8.11.192/26"
}
# tftest skip

Variables Copy Chapter

Outputs Copy Chapter

Test Copy Chapter

module "test" {
  source = "./fabric/fast/project-templates/data-mongodb"
  atlas_config = {
    cluster_name     = "test-0"
    organization_id  = "fmoajt0b2fwdvp9yvu7m7zl2"
    project_name     = "my-atlas-project"
    region           = "NORTH_AMERICA_NORTHEAST_1"
    database_version = "7.0"
    instance_size    = "M10"
    provider = {
      public_key  = "xxxx"
      private_key = "xxxxx-xxxx-xxxx-xxxx-xxxxxxxx"
    }
  }
  database_users = {
    my-db-user = {
      password_wo         = "s3cr3t-p4ssw0rd"
      password_wo_version = 1
      labels = {
        environment = "test"
      }
      roles = {
        app = {
          database_name = "app"
          role_name     = "readWrite"
        }
      }
      scopes = {
        test-0 = {
          type = "CLUSTER"
        }
      }
    }
  }
  project_id = "my-prod-shared-mongodb-0"
  vpc_config = {
    network_name   = "dev-spoke-0"
    subnetwork_id  = "projects/my-dev-net-spoke-0/regions/northamerica-northeast1/subnetworks/gce"
    psc_cidr_block = "10.8.11.192/26"
  }
}
# tftest modules=2 resources=7