terraform-google-kubernetes-engine Documentation

Technical Documentation: terraform-google-modules/terraform-google-kubernetes-engine

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Terraform Kubernetes Engine Module

This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, IP MASQ, Network Policy, etc.
The resources/services/activations/deletions that this module will create/trigger are:

• Create a GKE cluster with the provided addons
• Create GKE Node Pool(s) with provided configuration and attach to cluster
• Replace the default kube-dns configmap if stub_domains are provided
• Activate network policy if network_policy is true
• Add ip-masq-agent configmap with provided non_masquerade_cidrs if configure_ip_masq is true

Sub modules are provided for creating private clusters, beta private clusters, and beta public clusters as well. Beta sub modules allow for the use of various GKE beta features. See the modules directory for the various sub modules.

Compatibility Copy Chapter

This module is meant for use with Terraform 1.3+ and tested using Terraform 1.14+.
If you find incompatibilities using Terraform >=1.3, please open an issue.

If you haven't upgraded to 1.3 and need a Terraform
0.13.x-compatible version of this module, the last released version
intended for Terraform 0.13.x is 27.0.0.

If you haven't upgraded to 0.13 and need a Terraform
0.12.x-compatible version of this module, the last released version
intended for Terraform 0.12.x is 12.3.0.

Usage Copy Chapter

There are multiple examples included in the examples folder but simple usage is as follows:

/* Detailed source-code truncated for AI context efficiency. */

Then perform the following commands on the root folder:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Inputs Copy Chapter

Outputs Copy Chapter

node_pools variable Copy Chapter

Use this variable for provisioning linux based node pools. For Windows based node pools use windows_node_pools

The node_pools variable takes the following parameters:

windows_node_pools variable Copy Chapter

The windows_node_pools variable takes the same parameters as node_pools but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy a specific requirement for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.

Requirements Copy Chapter

Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:

1. Terraform and kubectl are installed on the machine where Terraform is executed.
2. The Service Account you execute the module with has the right permissions.
3. The Compute Engine and Kubernetes Engine APIs are active on the project you will launch the cluster in.
4. If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.

The project factory can be used to provision projects with the correct APIs active and the necessary Shared VPC connections.

Software Dependencies Copy Section

Kubectl

• kubectl 1.9.x

Terraform and Plugins

• Terraform 1.3+
• Terraform Provider for GCP v7.39+

gcloud

Some submodules use the terraform-google-gcloud module. By default, this module assumes you already have gcloud installed in your $PATH.
See the module documentation for more information.

Configure a Service Account Copy Section

In order to execute this module you must have a Service Account with the
following project roles:

• roles/compute.viewer
• roles/compute.securityAdmin (only required if add_cluster_firewall_rules is set to true)
• roles/container.clusterAdmin
• roles/container.developer
• roles/iam.serviceAccountAdmin
• roles/iam.serviceAccountUser
• roles/resourcemanager.projectIamAdmin (only required if service_account is set to create)

Additionally, if service_account is set to create and grant_registry_access is requested, the service account requires the following role on the registry_project_ids projects:

• roles/resourcemanager.projectIamAdmin

Enable APIs Copy Section

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Compute Engine API - compute.googleapis.com
• Kubernetes Engine API - container.googleapis.com

Overview (`autogen/main/README.md`) Copy Section

Terraform Kubernetes Engine Module

This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, {% if autopilot_cluster != true%}IP MASQ, {% endif %}Network Policy, etc.{% if private_cluster %} This particular submodule creates a private cluster{% endif %}{% if beta_cluster %}Beta features are enabled in this submodule.{% endif %}

The resources/services/activations/deletions that this module will create/trigger are:

• Create a GKE cluster with the provided addons
• Create GKE Node Pool(s) with provided configuration and attach to cluster
• Replace the default kube-dns configmap if stub_domains are provided
• Activate network policy if network_policy is true
  {% if autopilot_cluster != true%}
• Add ip-masq-agent configmap with provided non_masquerade_cidrs if configure_ip_masq is true
  {% endif %}

Sub modules are provided for creating private clusters, beta private clusters, and beta public clusters as well. Beta sub modules allow for the use of various GKE beta features. See the modules directory for the various sub modules.

{% if private_cluster %}

Private Cluster Details Copy Chapter

For details on configuring private clusters with this module, check the troubleshooting guide.

{% endif %}
{% if update_variant %}

Node Pool Update Variant Copy Chapter

In [#256] update variants added support for node pools to be created before being destroyed.

Before, if a node pool has to be recreated for any number of reasons,
the node pool is deleted then, created. This can be a problem if it is the only node pool in the GKE
cluster and the new node pool cannot be provisioned. In this scenario, pods could not be scheduled.
[#256] allows a node pool to be created before it is deleted so that any issues with node pool creation
and/or provisioning are discovered before the node pool is removed. This feature is controlled by the
variable node_pools_create_before_destroy. In order to avoid node pool name collisions,
a 4 character alphanumeric is added as a suffix to the name.

The benefit is that you always have some node pools active.
We don't actually cordon/drain the traffic beyond what the GKE API itself will do,
but we do make sure the new node pool is created before the old one is destroyed.

The implications of this are that:

• We append a random ID on the node pool names (since you can't have two simultaneously active node pools)
• For a brief period, you'll have 2x as many resources/node pools
• You will indeed need sufficient IP space (and compute capacity) to create both node pools

{% endif %}

Compatibility Copy Chapter

This module is meant for use with Terraform 1.3+ and tested using Terraform 1.14+.
If you find incompatibilities using Terraform >=1.3, please open an issue.

If you haven't upgraded to 1.3 and need a Terraform
0.13.x-compatible version of this module, the last released version
intended for Terraform 0.13.x is 27.0.0.

If you haven't upgraded to 0.13 and need a Terraform
0.12.x-compatible version of this module, the last released version
intended for Terraform 0.12.x is 12.3.0.

Usage Copy Chapter

There are multiple examples included in the examples folder but simple usage is as follows:

/* Detailed source-code truncated for AI context efficiency. */

Then perform the following commands on the root folder:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

{% if autopilot_cluster != true %}

node_pools variable Copy Chapter

Use this variable for provisioning linux based node pools. For Windows based node pools use windows_node_pools

The node_pools variable takes the following parameters:

windows_node_pools variable Copy Chapter

The windows_node_pools variable takes the same parameters as node_pools but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy a specific requirement for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.

{% endif %}

Requirements Copy Chapter

Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:

1. Terraform and kubectl are installed on the machine where Terraform is executed.
2. The Service Account you execute the module with has the right permissions.
3. The Compute Engine and Kubernetes Engine APIs are active on the project you will launch the cluster in.
4. If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.

The project factory can be used to provision projects with the correct APIs active and the necessary Shared VPC connections.

Software Dependencies Copy Section

Kubectl

• kubectl 1.9.x

Terraform and Plugins

• Terraform 1.3+
  {% if beta_cluster %}
• Terraform Provider for GCP Beta v7.39+
  {% else %}
• Terraform Provider for GCP v7.39+
  {% endif %}

gcloud

Some submodules use the terraform-google-gcloud module. By default, this module assumes you already have gcloud installed in your $PATH.
See the module documentation for more information.

Configure a Service Account Copy Section

In order to execute this module you must have a Service Account with the
following project roles:

• roles/compute.viewer
• roles/compute.securityAdmin (only required if add_cluster_firewall_rules is set to true)
• roles/container.clusterAdmin
• roles/container.developer
• roles/iam.serviceAccountAdmin
• roles/iam.serviceAccountUser
• roles/resourcemanager.projectIamAdmin (only required if service_account is set to create)

Additionally, if service_account is set to create and grant_registry_access is requested, the service account requires the following role on the registry_project_ids projects:

• roles/resourcemanager.projectIamAdmin

Enable APIs Copy Section

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Compute Engine API - compute.googleapis.com
• Kubernetes Engine API - container.googleapis.com

{% if beta_cluster %}
terraform-provider-google-beta: https://github.com/terraform-providers/terraform-provider-google-beta
{% else %}
terraform-provider-google: https://github.com/terraform-providers/terraform-provider-google
{% endif %}
27.0.0: https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/27.0.0
12.3.0: https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google/12.3.0
terraform-0.13-upgrade: https://www.terraform.io/upgrade-guides/0-13.html
terraform-1.3-upgrade: https://developer.hashicorp.com/terraform/language/v1.3.x/upgrade-guides

Overview (`autogen/safer-cluster/README.md`) Copy Section

Safer Cluster: How to setup a GKE Kubernetes cluster with reduced exposure

This module defines an opinionated setup of GKE
cluster. We outline project configurations, cluster settings, and basic K8s
objects that permit a safer-than-default configuration.

Module Usage Copy Chapter

The module fixes a set of parameters to values suggested in the
GKE hardening guide,
the CIS framework, and other best practices.

The motivation for each setting, and its relation to hardening guides or other recommendations
is outlined in main.tf as comments over individual settings. When security-relevant settings
are available for configuration, recommendations on their settings are documented in the variables.tf file.

Project Setup and Cloud IAM policy for GKE Copy Chapter

Applications and Clusters Copy Section

• Different applications that access data with different sensitivity and do
  not need to communicate with each other (e.g., dev and prod instances of the
  same application) should be placed in different clusters.

  • This approach will limit the blast radius of errors. A security problem
    in dev shouldn't impact production data.

• If applications need to communicate (e.g., a frontend system calling
  a backend), we suggest placing the two applications in the same cluster, in
  different namespaces.

  • Placing them in the same cluster will provide fast network
    communication, and the different namespaces will be configured to
    provide some administrative isolation. Cloud Service Mesh can be used to
    encrypt and control communication between applications.

• We suggest to store user or business data persistently in managed storage
  services that are inventoried and controlled by centralized teams.
  (e.g., GCP storage services within a GCP organization).

  • Storing user or business data securely requires satisfying a large set of
    requirements, such as data inventory, which might be harder to satisfy at
    scale when data is stored opaquely within a cluster. Services like Cloud
    Asset Inventory provide centralized teams ability to enumerate data stores.

Project Setup Copy Section

We suggest a GKE setup composed of multiple projects to separate responsibilities
between cluster operators, which need to administer the cluster; and product
developers, which mostly just want to deploy and debug applications.

• Cluster Projects (project_id): GKE clusters storing sensitive data should have their
  own projects, so that they can be administered independently (e.g., dev cluster;
  production clusters; staging clusters should go in different projects.)

• Shared GCR projects (registry_project_ids): all clusters can share the same
  GCR projects.

  • Easier to share images between environments. The same image could be
    progressively rolled-out in dev, staging, and then production.
  • Easier to manage service account permissions: GCR requires authorizing
    service accounts to access certain buckets, which are created only after
    images are published. When the only service run by the project is GCR,
    we can safely give project-wide read permissions to all buckets.

• (optional) Data Projects: When the same cluster is shared by different
  applications managed by different teams, we suggest separating the data for
  each application by placing storage resources for each team in different
  projects (e.g., a Spanner instance for application A in one project, GCS
  bucket for application B in a different project).

  • This permits to control administrative access to the data more tightly,
    as Cloud IAM policies for accessing the data can be managed by each
    application team, rather than the team managing the cluster
    infrastructure.

Exception to such a setup:

• When not using Shared VPCs, resources that require direct network connectivity
  (e.g., a Cloud SQL instance), need to be placed in the same VPC (hence, project)
  as the clusters from which connections are made.

Google Service Accounts Copy Section

We use GKE Workload Identity (BETA) to associate a GCP identity to each workload,
and limit the permissions associated with the cluster nodes.

The Safer Cluster setup relies on several service accounts:

• The module generates a service account to run nodes. Such a service account
  has only permissions of sending logging data, metrics, and downloading containers
  from the given GCR project. The following settings in the module will create
  a service account with the above properties:

create_service_account = true
registry_project_ids = [<the project id for your GCR project>]
grant_registry_access = true

• A service account for each application running on the cluster (e.g.,
  [email protected]). These service
  accounts should be associated to the permissions required for running the
  application, such as access to databases.

- email: myproduct
  displayName: Google Service Account for containers running in the myproduct k8s namespace
  policy:
    # GKE workload identity authorization. This authorizes the Kubernetes Service Account
    # myproduct/default from this project's identity namespace to impersonate the service account.
    # https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
    bindings:
    - members:
      - serviceAccount:product-prod.svc.id.goog[myproduct/default]
      role: roles/iam.workloadIdentityUser

We suggest running different applications in different namespaces within the cluster. Each namespace
should be assigned to its own GCP service account to better define the Cloud IAM permissions required
for the application.

If you are using more than 2 projects in your setup, you can consider creating
the service account in a different project to keep application and
infrastructure separate. For example, service accounts could be created in each team's project,
while the cluster runs in a centrally controlled project.

Cloud IAM Permissions for the GKE Cluster Copy Section

We suggest to mainly rely on Kubernetes RBAC to manage access control, and use
Cloud IAM to give users only the ability of configuring kubectl credentials.

Engineers operating applications on the cluster should only be assigned the
Cloud IAM permission roles/container.clusterViewer. This role allows them to
obtain credentials for the cluster, but provides no further access to the
cluster objects. All cluster objects are protected by RBAC configurations,
defined below.

You can add the following binding to the myproduct-prod project.

- members:
  role: roles/container.clusterViewer
  - group:<product team group>
  - group:<cluster team group>

The permissions won't allow engineers to SSH into nodes as part of the regular
development workflow. Such permissions should be granted only to the cluster
team, and used only in case of emergency.

While RBAC permissions should be sufficient for most cases, we also suggest to
create an emergency superuser role that can be used, given a proper
justification, for resolving cases where regular permissions are insufficient.
For simplicity, we suggest using roles/container.admin and
roles/compute.admin, until more narrow roles can be defined given your usage.

- members:
  role: roles/container.admin
  - group:<oncall for cluster team>
- members:
  role: roles/compute.admin
  - group:<oncall for cluster team>

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/acm-terraform-blog-part1/README.md`) Copy Section

Enable ACM features with Terraform - Part 1

This is Part1 of the tutorial to accompany a short series of blog articles explaining how to enable Anthos Config Management (ACM) with Terraform.

In this tutorial, we'll explain how to use Teraform to create a cluster and manage its config from git via Config Sync.

Next part will build on that to add guard rails for the cluster via Policy Controller. We will focus on enabling an ongoing audit of cluster resources using the Policy Controller's built in Policy Library and a bundle of constraints enforcings CIS Kubernetes Benchmark v.1.5.1.

Subsequent articles will discuss other aspects of ACM to manage your GCP infrastrcuture.

Enable Config Sync on the cluster with Terraform Copy Chapter

1. Clone this repo

2. Set variables that will be used in multiple commands:

   bash Copy

   FOLDER_ID = [FOLDER]
   BILLING_ACCOUNT = [BILLING_ACCOUNT]
   PROJECT_ID = [PROJECT_ID]

3. Create project:

   bash Copy

   gcloud auth login
   gcloud projects create $PROJECT_ID --name=$PROJECT_ID --folder=$FOLDER_ID
   gcloud alpha billing projects link $PROJECT_ID --billing-account $BILLING_ACCOUNT
   gcloud config set project $PROJECT_ID

4. Create cluster using terraform using defaults other than the project:

   bash Copy

   # obtain user access credentials to use for Terraform commands
   gcloud auth application-default login
   
   # continue in /terraform directory
   cd terraform
   export TF_VAR_project=$PROJECT_ID
   terraform init
   terraform plan
   terraform apply

   NOTE: if you get an error due to default network not being present, run gcloud compute networks create default --subnet-mode=auto and retry the commands.

5. To verify things have sync'ed, you can use gcloud to check status:

   bash Copy

   gcloud alpha container hub config-management status --project $PROJECT_ID

   In the output, notice that the Status will eventually show as SYNCED and the Last_Synced_Token will match the repo hash.

6. To see wordpress itself, you can use the kubectl proxy to connect to the service:

   bash Copy

   # get values from cluster that was created
   export CLUSTER_ZONE=$(terraform output -raw cluster_location)
   export CLUSTER_NAME=$(terraform output -raw cluster_name)
   
   # then get creditials for it and proxy to the wordpress service to see it running
   gcloud container clusters get-credentials $CLUSTER_NAME --zone $CLUSTER_ZONE --project $PROJECT_ID
   kubectl proxy --port 8888 &
   
   # curl or use the browser
   curl http://127.0.0.1:8888/api/v1/namespaces/wp/services/wordpress/proxy/wp-admin/install.php

7. Finally, let's clean up. First, don't forget to foreground the proxy again to kill it. Also, apply terraform destroy to remove the GCP resources that were deployed to the project.

   bash Copy

   fg # ctrl-c
   
    terraform destroy -var=project=$PROJECT_ID

Overview (`examples/acm-terraform-blog-part2/README.md`) Copy Section

Enable ACM features with Terraform - Part 2

This is Part2 of the tutorial to accompany a short series of blog articles explaining how to enable Anthos Config Management (ACM) with Terraform.

In the previous article, we'll explain how to use Terraform to create a cluster and manage its config from git via Config Sync.

This article will build on that to add guard rails for the cluster via Policy Controller. We will focus on enabling an ongoing audit of cluster resources using the Policy Controller's built in Policy Library and a bundle of constraints enforcing CIS Kubernetes Benchmark v.1.5.1.

Subsequent articles will discuss other aspects of ACM to manage your GCP infrastructure.

Enforce Cluster Guardrails with ACM Policy Controller Copy Chapter

1. Set a variables for the project from part1. We will re-use that project but create a new cluster since we cleaned up at the end of the first section. If you are working in a different project, enable required GCP APIs, as described in part1/README.md.

   bash Copy

   PROJECT_ID = [PROJECT_ID]

2. As before, cluster using terraform using defaults other than the project. The main difference in the terraform files is that we turn on PolicyController and also install the built-in Policy Library.

   bash Copy

   # obtain user access credentials to use for Terraform commands
   gcloud auth application-default login
   
   # continue in /terraform directory
   cd terraform
   export TF_VAR_project=$PROJECT_ID
   terraform init
   terraform plan
   terraform apply

   NOTE: if you get an error due to default network not being present, run gcloud compute networks create default --subnet-mode=auto and retry the commands.

3. To verify things have sync'ed and the policy controller is installed, you can again use gcloud to check status:

   bash Copy

   gcloud alpha container hub config-management status --project $PROJECT_ID

   As things initialize, you may see a few transient error: KNV1021: No CustomResourceDefinition is defined messages. This occurs when constraints from the repo are sync'ed before Policy Controller has had a chance to load the appropriate template from the policy library. It will eventually reconcile.

   After a short time, in addition to the Status showing as SYNCED and the Last_Synced_Token matching the repo, there should also be a value of INSTALLED for Policy_Controller.

4. One difference you may notice from part1 is that in the config-root/cis-k8s-1.5.1 directory. This is a bundle of Policy Controller constraints that were pulled into the repo from acm-policy-controller-library repo using kpt pkg get command. Kpt is a helpful kubernetes config tool documented at kpt.dev. We'll use Kpt tool directly in part3 of this tutorial. The goal of this bundle is to audit and enforce CIS Benchmarks for Kubernetes. At the moment, they have been deployed in dryrun mode so we can use them to audit the cluster.

   To see the audit status first we get credentials for kubectl in the same way we did this in part1:

   bash Copy

   # get values from cluster that was created
   export CLUSTER_ZONE=$(terraform output -raw cluster_location)
   export CLUSTER_NAME=$(terraform output -raw cluster_name)
   
   # then get creditials for it
   gcloud container clusters get-credentials $CLUSTER_NAME --zone $CLUSTER_ZONE --project $PROJECT_ID

   Now, let's look at the violations in the status of the constraints we have active. One handy way to do this is to dump the constraints in json so we can filter for violations using jq.

   bash Copy

   kubectl get constraint -o json | jq -C '.items[]| select(.status.violations)| .kind,.status.violations'

   Seems like there is a LOT to clean up!

Overview (`examples/acm-terraform-blog-part3/README.md`) Copy Section

Enable ACM features with Terraform - Part 3

This is part three of the tutorial to accompany a short series of blog articles explaining how to enable Anthos Config Management (ACM) with Terraform.

In the first part, we explained how to use Terraform to create a cluster and manage its config from git via Config Sync.

In the second part we added guard rails for the cluster configuration via Policy Controller.

In this article we'll demonstrate how, using Config Connector, you can provision your GCP cloud resources following the same Kubernetes-native model.

Provision GCP resources Copy Chapter

1. Set the variable for the project from part two. We will re-use that project but create a new cluster since we cleaned up at the end of the first section. If you are working in a different project, enable required GCP APIs, as described in part one.

   bash Copy

   PROJECT_ID = [PROJECT_ID]

2. Note that wordpress-bundle.yaml was updated to use GCP MySQL database. Also we added configconnector.yaml to initialize the instance of Config Connector add-on on the cluster.

3. Use kpt to customize the config-root directory that will be configured as the source of the objects installed on the cluster.

   bash Copy

   kpt fn eval --include-meta-resources --image gcr.io/kpt-fn/set-project-id:v0.1 ./config-root -- "project-id=$PROJECT_ID"
   kpt fn render ./config-root

4. Submit the updated configuration into your branch.

5. Ensure that sync_repo and sync_branch variables are updated in terraform.tfvars

6. Before running Terraform, notice the changes in gke.tf:

   • We are using the beta-public-cluster module
   • config_connector variable is set to true
   • We are using workload-identity module to create a Google Service Account and connect it to a Kubernetes Service Account that is running in Config Connector cnrm-system namespace, allowing Config Connector to create GCP resources.

7. As in the previous part, create the cluster using Terraform:

   bash Copy

   # obtain user access credentials to use for Terraform commands
   gcloud auth application-default login
   
   # continue in /terraform directory
   cd terraform
   export TF_VAR_project=$PROJECT_ID
   terraform init
   terraform plan
   terraform apply

   NOTE: if you get an error due to the default network not being present, run gcloud compute networks create default --subnet-mode=auto and retry the commands.

8. To verify things have synced and Policy Controller is installed, you can again use gcloud to check status:

   bash Copy

   gcloud alpha container hub config-management status --project $PROJECT_ID

   As things initialize, you may see a few transient error: KNV1021: No CustomResourceDefinition is defined messages. This occurs when constraints from the repo are synced before Policy Controller has had a chance to load the appropriate template from the policy library. It will eventually reconcile.

   After a short time, in addition to the Status showing as SYNCED and the Last_Synced_Token matching the repo, there should also be a value of INSTALLED for Policy_Controller.

9. Connect your kubectl instance to the newly created cluster:

   bash Copy

   # get values from cluster that was created
   export CLUSTER_ZONE=$(terraform output -raw cluster_location)
   export CLUSTER_NAME=$(terraform output -raw cluster_name)
   
   # then get credentials for it
   gcloud container clusters get-credentials $CLUSTER_NAME --zone $CLUSTER_ZONE --project $PROJECT_ID

10. Verify that Config Connector addon is installed and configured:

    bash Copy

    kubectl wait -n cnrm-system --for=condition=Ready pod --all

    Note: The controller Pod can take several minutes to start. Once Config Connector is installed correctly, the output is similar to the following:

    bash Copy

    pod/cnrm-controller-manager-0 condition met

11. It will take a while for the SQL database to be created. You can check on the status:

    bash Copy

    kubectl describe sqlinstance -n wp

12. Finally, validate that Wordpress powered Cloud SQL database was created:

    bash Copy

    curl -L $( kubectl get service wordpress-external -n wp -o=json | \
            jq -r '.status["loadBalancer"]["ingress"][0]["ip"]')

Overview (`examples/autopilot_private_firewalls/README.md`) Copy Section

Private Regional Autopilot Cluster With Firewall Rules

This example creates a regional autopilot cluster with beta features

It will:

• Create a private autopilot cluster
• Variables add_cluster_firewall_rules, add_master_webhook_firewall_rules and add_shadow_firewal_rules are toggled on. In a environment with stringent firewall rules, these cluster firewall rules may be required to allow intra cluster communication
• Adds an example network tag. This example network tag aligns with a firewall rule target tag from the terraform-example-foundation configuration to allow google api access

note: this example does not create a network with stringent firewall rules. This example shows what cluster configuration may be required for a networking configuration comparable to the terraform-example-foundation's base shared VPCs, or any VPC with firewall rules to deny 0.0.0.0/0 egress and an allow rule for private google api egress access which targets tags "allow-google-apis". By toggling on the firewall rules variables and adding network_tags to allow private google api access, a private cluster can come up healthy with no internet egress. This configuration shows how to ensure those firewalls that explicitly allow intra cluster ingress and egress are created and appropriate network tags are attached to the cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/confidential_autopilot_private/README.md`) Copy Section

Confidential Autopilot Private Cluster

This example illustrates how to create an Autopilot cluster with beta features,
using Confidential GKE nodes and a Customer Managed Encryption Keys (CMEK).

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/confidential_safer_cluster/README.md`) Copy Section

Confidential Safer GKE Cluster

This example illustrates how to instantiate the Safer Cluster module
with confidential nodes enabled and database encrypted with KMS key.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/deploy_service/README.md`) Copy Section

Deploy Service Cluster

This example illustrates how to deploy a service to the created cluster using the kubernetes provider.

It will:

• Create a cluster
• Configure authentication for the Kubernetes provider
• Create an Nginx Pod
• Create an Nginx Service

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/disable_client_cert/README.md`) Copy Section

Disable Client Certificate

This example illustrates how to create a simple cluster and disable deprecated security features:

• basic auth
• client certificate

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/gke_autopilot_cluster/README.md`) Copy Section

GKE Autopilot Cluster

This example creates a GKE private Autopilot clusterwith beta features.
For a full example see simple_autopilot_private example.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/gke_standard_cluster/README.md`) Copy Section

GKE Standard Cluster and Node Pool

This example creates a GKE private cluster and Node Pool with beta features. This example also adds an extra network to the Nodes configuration.
For a full example see simple_regional_private_beta example.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/island_cluster_anywhere_in_gcp_design/README.md`) Copy Section

GKE island cluster anywhere in GCP design

This example provisions a cluster in an island VPC allowing reuse of the IP address space for multiple clusters across different GCP organizations.

Deploy Copy Chapter

1. Create NCC hub.
2. Update ncc_hub_project_id, ncc_hub_name, network_name and gke spokes in terraform.tfvars.
3. Run terraform apply.

Inputs Copy Chapter

Outputs Copy Chapter

Overview (`examples/island_cluster_with_vm_router/README.md`) Copy Section

GKE island cluster using VM as router

This example provisions a cluster in an island VPC allowing reuse of the IP address space for multiple clusters in the same project.

1. An appliance(VM as router) with multiple NICs is used to establish connectivity between the island VPC and the existing network.
2. Outbound connections will go through the router.
3. For inbound connections, use Private Service Connect.

Deploy Copy Chapter

1. Update project_id, cluster_name and primary_subnet values in terraform.tfvars, and update other variables as needed.
2. Run terraform apply.

Inputs Copy Chapter

Outputs Copy Chapter

Overview (`examples/node_pool/README.md`) Copy Section

Node Pool Cluster

This example illustrates how to create a cluster with multiple custom node-pool configurations with node labels, taints, and network tags.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/node_pool_update_variant/README.md`) Copy Section

Node Pool Cluster

This example illustrates how to create a cluster with multiple custom node-pool configurations with node labels, taints, and network tags.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/node_pool_update_variant_beta/README.md`) Copy Section

Node Pool Cluster

This example illustrates how to create a cluster with multiple custom node-pool configurations with node labels, taints, and network tags.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/node_pool_update_variant_public_beta/README.md`) Copy Section

Node Pool Cluster

This example illustrates how to create a cluster with multiple custom node-pool configurations with node labels, taints, and network tags.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/regional_private_node_pool_oauth_scopes/README.md`) Copy Section

Regional Private Cluster with node pool and oauth scopes

This example illustrates how to create a private cluster with node pool specifications, oauth scopes along with required network and subnet creation.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/safer_cluster/README.md`) Copy Section

Safer GKE Cluster

This example illustrates how to instantiate the opinionated Safer Cluster module.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/safer_cluster_iap_bastion/README.md`) Copy Section

Safer Cluster Access with IAP Bastion Host

This end to end example aims to showcase access patterns to a Safer Cluster, which is a hardened GKE Private Cluster, through a bastion host utilizing Identity Awareness Proxy without an external ip address. Access to this cluster's control plane is restricted to the bastion host's internal IP using authorized networks.

Additionally we deploy a tinyproxy daemon which allows kubectl commands to be piped through the bastion host allowing ease of development from a local machine with the security of GKE Private Clusters.

GKE Autopilot clusters are deployed with Application-layer Secrets Encryption that protects your secrets in etcd with a key you manage in Cloud KMS.

Setup Copy Chapter

To deploy this example:

1. Run terraform init.

2. Create a terraform.tfvars to provide values for project_id, bastion_members. Optionally override any variables if necessary.

3. Run terraform apply.

4. After apply is complete, generate kubeconfig for the private cluster. The command with the right parameters will displayed as the Terraform output get_credentials_command.

   sh Copy

   gcloud container clusters get-credentials --project $PROJECT_ID --zone $ZONE --internal-ip $CLUSTER_NAME

5. SSH to the Bastion Host while port forwarding to the bastion host through an IAP tunnel. The command with the right parameters will displayed by running terraform output bastion_ssh_command.

   sh Copy

   gcloud beta compute ssh $BASTION_VM_NAME --tunnel-through-iap --project $PROJECT_ID --zone $ZONE -- -L8888:127.0.0.1:8888

6. You can now run kubectl commands though the proxy. An example command will displayed as the Terraform output bastion_kubectl_command.

   sh Copy

   HTTPS_PROXY=localhost:8888 kubectl get pods --all-namespaces

Inputs Copy Chapter

Outputs Copy Chapter

Overview (`examples/shared_vpc/README.md`) Copy Section

Shared VPC Cluster

This example illustrates how to create a simple cluster where the host network is not necessarily in the same project as the cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_autopilot_private/README.md`) Copy Section

Simple Regional Autopilot Cluster

This example illustrates how to create a simple autopilot cluster with beta features.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_autopilot_private_cmek/README.md`) Copy Section

Simple Regional Autopilot Cluster

This example illustrates how to create a simple autopilot cluster with beta features and
using a Customer Managed Encryption Keys (CMEK).

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_autopilot_private_non_default_sa/README.md`) Copy Section

Simple Regional Autopilot Cluster

This example illustrates how to create a simple autopilot cluster with beta features and
not using the default service account.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_autopilot_public/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster with beta features.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_fleet_app_operator_permissions/README.md`) Copy Section

Simple App Operator Permissions Setup for a Fleet Scope

This example illustrates how to create a Fleet Scope for a team and set up permissions for an app operator in the team.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_additional_ip_ranges/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_beta/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster with beta features.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_cluster_autoscaling/README.md`) Copy Section

Simple Regional Cluster with Node Autoscaling

This example illustrates how to create a simple private cluster with cluster
level node autoprovisioning.

Inputs Copy Chapter

Outputs Copy Chapter

Overview (`examples/simple_regional_private/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple private cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_private_beta/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple private cluster with beta features.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_private_no_pool/README.md`) Copy Section

Simple Regional Private Cluster with No Node Pools

This example illustrates how to create a regional private cluster while skipping the creation of node pools and enabling the default compute class.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_private_with_cluster_version/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple private cluster with beta features.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_with_gateway_api/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_with_ipv6/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_with_kubeconfig/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster and output a kubeconfig

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_regional_with_networking/README.md`) Copy Section

Simple Regional Cluster with Networking

This example illustrates how to create a VPC and a simple cluster.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_windows_node_pool/README.md`) Copy Section

Simple Regional Cluster

This example illustrates how to create a simple cluster with a windows node pool.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

Overview (`examples/simple_zonal_private/README.md`) Copy Section

Simple Zonal Cluster

This example illustrates how to create a simple private cluster in a single zone.

Inputs Copy Chapter

Outputs Copy Chapter

To provision this example, run the following from within this directory:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure