GCP Cloud Build + Terraform for Automated Infrastructure Provisioning

Welcome, fellow engineers and tech enthusiasts, to a deep dive into the powerful world of automated infrastructure! Have you ever wished you could provision complex cloud environments with the click of a button, or even better, automatically, as part of your CI/CD pipeline? Today, we’re making that wish a reality.

In this post, we’ll explore how to combine the robust capabilities of Google Cloud Build with HashiCorp Terraform to achieve seamless, automated infrastructure provisioning on GCP. Imagine spinning up entire Google Kubernetes Engine (GKE) clusters and Cloud SQL instances without ever touching the GCP Console manually. This isn’t just about saving time; it’s about ensuring consistency, reducing human error, and truly embracing Infrastructure as Code (IaC). Get ready to transform your deployment workflows and elevate your cloud management game. Let’s build something amazing!

Before we dive into the nuts and bolts, let’s briefly touch upon why Infrastructure as Code (IaC) has become an indispensable practice in modern cloud deployments. Imagine managing hundreds or even thousands of resources across multiple environments – development, staging, production. Manually configuring these would be a nightmare, prone to inconsistencies, forgotten steps, and significant delays.

IaC solves this by defining your infrastructure in configuration files, just like application code. This brings software development best practices like version control, peer review, and automated testing directly to your infrastructure management. Terraform, in particular, is an open-source IaC tool that allows you to define both cloud and on-premises resources in human-readable configuration files. It supports an extensive array of providers, with Google Cloud Platform being one of its strongest integrations. This means your entire GCP landscape – from Virtual Private Clouds to GKE clusters and Cloud SQL instances – can be version-controlled, reviewed, and deployed with precision.

Now, let’s bring Google Cloud Build into the picture. Cloud Build is GCP’s serverless platform for executing your builds and deployments. It’s incredibly versatile, capable of pulling code from various sources like Cloud Source Repositories, GitHub, or Bitbucket, and executing a series of steps defined in a cloudbuild.yaml file. These steps can be anything from compiling code, running tests, to, as we’ll see today, provisioning infrastructure. What makes Cloud Build so powerful in this context is its native integration with GCP services. It runs within your GCP project, leveraging your project’s service accounts, and has direct access to GCR (Google Container Registry) for builder images, and other GCP APIs. It truly acts as the central orchestrator for your automated workflows, allowing you to define a continuous delivery pipeline for almost any task.

Combining Cloud Build and Terraform creates a formidable duo for continuous delivery of infrastructure. Cloud Build acts as the automated trigger and execution engine, while Terraform provides the declarative language for defining your desired infrastructure state. When you commit changes to your Terraform configurations in a source repository, Cloud Build can automatically detect those changes, pull the latest code, and execute Terraform commands like init, plan, and apply. This means that every infrastructure change is versioned, auditable, and applied consistently, without manual intervention. It eliminates configuration drift and ensures that your environments are always in the desired state. This approach not only speeds up deployments but also significantly enhances reliability and security, as all changes go through a controlled, automated pipeline.

To follow along visually and get a deeper understanding, check out our accompanying YouTube video:

Getting Started: Prerequisites and Setup

Let’s get practical and talk about the actual setup. Our project structure is straightforward: a root directory containing our cloudbuild.yaml and a terraform/ subdirectory holding all our Terraform configuration files.

Before we even think about triggering a build, there are a few crucial prerequisites:

1. GCP Project: You’ll need an active GCP project.
2. APIs Enabled: Ensure the following APIs are enabled in your project:
   • Cloud Build API
   • Container Registry API
   • Kubernetes Engine API
   • Cloud SQL Admin API
   • Compute Engine API
   • Service Usage API
3. Terraform State Backend Bucket: Crucially, you’ll need to manually create a Google Cloud Storage (GCS) bucket to serve as your Terraform state backend. This is where Terraform stores the state of your deployed infrastructure, enabling collaboration and ensuring consistency.

   gcloud storage buckets create gs://your-terraform-state-bucket --project=your-gcp-project-id --uniform-bucket-level-access --location=US

4. Cloud Build Service Account Permissions: The Cloud Build service account (which typically looks like PROJECT_NUMBER@cloudbuild.gserviceaccount.com) needs specific IAM roles on your project to provision the resources defined in our Terraform code:
   • Storage Object Admin (for your state bucket)
   • Kubernetes Engine Admin
   • Cloud SQL Admin
   • Compute Network Admin
   • Service Account User

For detailed setup instructions and the full code, please refer to the GitHub repository:

Explore the Code on GitHub!

Diving into the Code

Let’s dissect the heart of our automation: the cloudbuild.yaml file. This YAML configuration defines the sequence of steps Cloud Build will execute.

cloudbuild.yaml: Orchestrating the Pipeline

Our pipeline consists of three distinct phases, each leveraging the gcr.io/cloud-builders/terraform image, which is a pre-built Cloud Build image containing the Terraform CLI. We also ensure that all commands are executed within the terraform/ subdirectory using the dir: 'terraform' directive.

# gcp-tf-cloudbuild-provisioner/cloudbuild.yaml
# This Cloud Build configuration orchestrates Terraform to provision GCP infrastructure.
# It initializes Terraform, plans the changes, and then applies them.

steps:
  # Step 1: Initialize Terraform
  # This step runs 'terraform init' to prepare the working directory.
  # It configures the GCS backend for state management using the _TF_STATE_BUCKET substitution variable.
  # The -reconfigure flag ensures that if the backend config changes (e.g., bucket name),
  # it will be re-initialized.
  - name: 'gcr.io/cloud-builders/terraform'
    id: 'Terraform Init'
    args:
      - 'init'
      - '-backend-config=bucket=${_TF_STATE_BUCKET}'
      - '-reconfigure' # Use -reconfigure to ensure state backend is correctly picked up
    dir: 'terraform' # Execute commands within the 'terraform' subdirectory
    env:
      # Pass the GCP Project ID to Terraform as an environment variable.
      # Cloud Build automatically provides the current project ID via ${PROJECT_ID} built-in variable.
      - 'GOOGLE_CLOUD_PROJECT=${PROJECT_ID}'

  # Step 2: Plan Terraform Changes
  # This step runs 'terraform plan' to create an execution plan.
  # The plan is saved to a file named 'tfplan' for subsequent 'apply' step.
  - name: 'gcr.io/cloud-builders/terraform'
    id: 'Terraform Plan'
    args:
      - 'plan'
      - '-out=tfplan' # Save the plan to 'tfplan' file
    dir: 'terraform'
    env:
      - 'GOOGLE_CLOUD_PROJECT=${PROJECT_ID}'

  # Step 3: Apply Terraform Changes
  # This step runs 'terraform apply' using the previously generated plan.
  # The -auto-approve flag bypasses the interactive approval prompt, which is necessary for automated pipelines.
  - name: 'gcr.io/cloud-builders/terraform'
    id: 'Terraform Apply'
    args:
      - 'apply'
      - '-auto-approve'
      - 'tfplan' # Apply the saved plan
    dir: 'terraform'
    env:
      - 'GOOGLE_CLOUD_PROJECT=${PROJECT_ID}'

# Substitutions for Cloud Build:
# These variables must be provided when triggering the build using `gcloud builds submit`.
# _TF_STATE_BUCKET: Custom variable for the GCS bucket name storing Terraform state.
# The PROJECT_ID variable is automatically provided by Cloud Build for the current project.
substitutions:
  _TF_STATE_BUCKET: '' # This needs to be set when triggering the build, e.g., --substitutions=_TF_STATE_BUCKET=my-tf-state-bucket

• Terraform Init: This command initializes Terraform, downloads necessary provider plugins, and critically, configures the GCS backend for state management. We pass the GCS bucket name dynamically using a Cloud Build substitution variable, _TF_STATE_BUCKET. The -reconfigure flag ensures the backend configuration is always correctly picked up.
• Terraform Plan: This step generates an execution plan, showing exactly what infrastructure changes Terraform intends to make. We save this plan to a file named tfplan, which provides a clear audit trail and ensures that the subsequent apply step acts on a known, verified plan.
• Terraform Apply: This step takes the tfplan file and applies the infrastructure changes. The -auto-approve flag is essential here, bypassing interactive prompts and allowing for fully automated deployments. Notice how the GOOGLE_CLOUD_PROJECT environment variable is automatically supplied by Cloud Build, making our Terraform code project-agnostic.

Terraform Configuration Files (in terraform/)

versions.tf: Backend and Provider Setup

The versions.tf file is foundational; it declares the required Terraform version and, more importantly, configures the google provider and the gcs backend for state management. This is where we tell Terraform to store its state file in the GCS bucket we created, ensuring that the state is remote, shared, and durable.

# gcp-tf-cloudbuild-provisioner/terraform/versions.tf
# This file specifies the required Terraform version and provider configurations.

terraform {
  required_version = ">= 1.0.0" # Specify a minimum required Terraform version

  # Configure the Google Cloud Storage (GCS) backend for storing Terraform state.
  # This centralizes the state file, making it accessible for Cloud Build and team collaboration.
  # The 'bucket' name will be provided dynamically via Cloud Build substitution variable during 'terraform init'.
  backend "gcs" {
    prefix = "terraform/state" # Optional: Prefix for state objects within the bucket
  }

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0" # Specify a compatible version range for the Google Cloud provider
    }
  }\n}\n
# Configure the Google Cloud provider.
# The 'project' is dynamically set via the GOOGLE_CLOUD_PROJECT environment variable
# that Cloud Build passes to the Terraform steps.
provider "google" {
  project = var.project_id # Referencing the project_id variable defined in variables.tf
  region  = var.region     # Referencing the region variable
  zone    = var.zone       # Referencing the zone variable (for resources requiring a zone, like node pools)
}

variables.tf: Input Parameters

The variables.tf file defines all the input parameters for our infrastructure, like the GCP region, zone, GKE cluster name, and Cloud SQL instance details. This separation of concerns allows us to reuse the same Terraform code across different environments by simply providing different variable values. Notice how project_id is defined to receive its value from the GOOGLE_CLOUD_PROJECT environment variable, which Cloud Build automatically injects. This is a crucial point for enabling project-specific deployments without hardcoding.

# gcp-tf-cloudbuild-provisioner/terraform/variables.tf
# This file defines input variables for the Terraform configuration.

# GCP Project ID:
# This variable receives its value from the GOOGLE_CLOUD_PROJECT environment variable,
# which Cloud Build automatically passes based on the project where the build runs.
variable "project_id" {
  description = "The GCP project ID where resources will be created. Auto-detected by Cloud Build."
  type        = string
}

# GCP Region: For regional resources like GKE clusters (regional) and Cloud SQL instances.
variable "region" {
  description = "The GCP region to deploy resources (e.g., us-central1)."
  type        = string
  default     = "us-central1"
}

# GCP Zone: For zonal resources or where a zone is required (e.g., GKE node pools default location).
variable "zone" {
  description = "The GCP zone to deploy zonal resources (e.g., us-central1-c). Required for GKE node pools."
  type        = string
  default     = "us-central1-c"
}

# --- GKE Cluster Variables ---
variable "gke_cluster_name" {
  description = "Name for the GKE cluster."
  type        = string
  default     = "my-cloudbuild-gke-cluster"
}

variable "gke_node_count" {
  description = "Number of nodes in the GKE cluster's default node pool."
  type        = number
  default     = 1
}

variable "gke_node_machine_type" {
  description = "Machine type for GKE cluster nodes (e.g., e2-medium)."
  type        = string
  default     = "e2-medium"
}

variable "gke_node_disk_size_gb" {
  description = "Disk size in GB for GKE cluster nodes."
  type        = number
  default     = 50
}

# --- Cloud SQL Variables ---
variable "sql_instance_name" {
  description = "Name for the Cloud SQL instance."
  type        = string
  default     = "my-cloudbuild-sql-instance"
}

variable "sql_database_version" {
  description = "Database version for Cloud SQL instance (e.g., POSTGRES_14, MYSQL_8_0)."
  type        = string
  default     = "POSTGRES_14"
}

variable "sql_instance_tier" {
  description = "Machine type for the Cloud SQL instance (e.g., db-f1-micro, db-g1-small)."
  type        = string
  default     = "db-f1-micro"
}

variable "sql_disk_size_gb" {
  description = "Disk size in GB for Cloud SQL instance."
  type        = number
  default     = 20
}

variable "sql_disk_type" {
  description = "Disk type for Cloud SQL instance (e.g., PD_SSD, PD_HDD)."
  type        = string
  default     = "PD_SSD"
}

variable "sql_database_name" {
  description = "Name of the database to create within the Cloud SQL instance."
  type        = string
  default     = "app_db"
}

variable "sql_username" {
  description = "Username for the Cloud SQL database user."
  type        = string
  default     = "app_user"
}

variable "sql_password" {
  description = "Password for the Cloud SQL database user. **WARNING: For production, use Secret Manager!**"
  type        = string
  default     = "securepassword123" # CHANGE THIS FOR PRODUCTION!
  sensitive   = true # Mark as sensitive so it's not shown in logs/outputs
}

Note the terraform.tfvars.sample file in the repository. You should copy this to terraform.tfvars and customize the default values for your deployment. Remember not to commit sensitive data directly to `terraform.tfvars` in a production environment; use Secret Manager or similar secure methods.

main.tf: Defining the Infrastructure

This is where the magic happens, where our desired GCP resources are declaratively defined. We’re provisioning two key resources here: a Google Kubernetes Engine (GKE) cluster and a Cloud SQL (PostgreSQL) instance.

# gcp-tf-cloudbuild-provisioner/terraform/main.tf
# This file defines the GCP infrastructure resources to be provisioned.

# --- GKE Cluster ---
resource "google_container_cluster" "primary_gke_cluster" {
  name     = var.gke_cluster_name
  location = var.region # For regional cluster, location is region; for zonal, it's zone.
  project  = var.project_id

  # Define the default node pool configuration.
  # For more granular control, consider creating a separate `google_container_node_pool` resource.
  initial_node_count = var.gke_node_count
  
  node_config {
    machine_type = var.gke_node_machine_type
    disk_size_gb = var.gke_node_disk_size_gb
    
    # OAuth scopes for node VMs. Default scopes are usually sufficient for basic operations.
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform" # Broad scope for demonstration.
                                                       # In production, use specific scopes like compute.readonly, logging.write, monitoring.write.
    ]
  }

  # Enable workload identity for secure access to GCP services from pods (recommended for production).
  # This requires the GKE cluster to be created with Workload Identity enabled.
  workload_identity_config {
    identity_namespace = "${var.project_id}.svc.id.goog"
  }

  # Release channel for automatic cluster upgrades.
  release_channel {
    channel = "REGULAR" # Other options: "STABLE", "RAPID"
  }

  # Configure logging and monitoring components.
  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }

  monitoring_config {
    enable_components = ["SYSTEM_COMPONENTS"]
  }

  # Example of private cluster configuration (commented out by default for simplicity)
  # enable_private_endpoint = true # Private endpoint for the control plane
  # enable_private_nodes    = true # Nodes only have private IPs
  # master_ipv4_cidr_block  = "172.16.0.0/28" # CIDR range for the master's private IP (must not overlap with VPC network)
  # ip_allocation_policy { # Required for VPC-native clusters
  #   cluster_ipv4_cidr_block = "/19"
  #   services_ipv4_cidr_block = "/22"
  # }

  # By default, GKE uses the default VPC network and subnetwork.
  # For production, define and use a custom VPC network and subnets.
  # network    = google_compute_network.custom_vpc.id
  # subnetwork = google_compute_subnetwork.custom_subnet.id
}

# --- Cloud SQL Instance (PostgreSQL) ---
resource "google_sql_database_instance" "main_sql_instance" {
  name             = var.sql_instance_name
  database_version = var.sql_database_version
  region           = var.region
  project          = var.project_id

  # Tier defines the machine type and resources for the instance.
  # See https://cloud.google.com/sql/docs/postgres/instance-settings#machine-type-2ndgen
  settings {
    tier      = var.sql_instance_tier
    disk_size = var.sql_disk_size_gb
    disk_type = var.sql_disk_type
    
    # Enable automatic backups
    backup_configuration {
      enabled            = true
      binary_log_enabled = true # Required for point-in-time recovery for PostgreSQL
      start_time         = "03:00" # Example: 3 AM UTC
    }

    # IP Configuration: By default, public IP is enabled.
    # To restrict access, configure authorized networks or disable public IP for private IP.
    ip_configuration {
      ipv4_enabled = true # Enable public IP
      # authorized_networks { # Example: Allow access from specific IP range (WARNING: 0.0.0.0/0 allows all)
      #   value = "0.0.0.0/0"
      #   name  = "Allow all - WARNING"
      # }
      # private_network = google_compute_network.your_vpc.id # For private IP setup
      # require_ssl     = true # Enforce SSL connections
    }

    # For high availability (HA)
    # availability_type = "REGIONAL" # Or "ZONAL" for non-HA
  }
}

# Optional: Create a database within the Cloud SQL instance
resource "google_sql_database" "app_database" {
  name      = var.sql_database_name
  instance  = google_sql_database_instance.main_sql_instance.name
  project   = var.project_id
  charset   = "UTF8"
  collation = "en_US.UTF8"
}

# Optional: Create a user for the Cloud SQL instance
resource "google_sql_user" "app_user" {
  name     = var.sql_username
  instance = google_sql_database_instance.main_sql_instance.name
  host     = "%" # Allows access from any host (WARNING: less secure, specify source IP or VPC for production)
  password = var.sql_password # Store securely using Secret Manager in production!
  project  = var.project_id
}

• GKE Cluster: We define its name, location (regional for high availability), initial node count, machine type, and important configurations like OAuth scopes for node VMs and enabling Workload Identity – a security best practice. We also configure logging and monitoring.
• Cloud SQL Instance (PostgreSQL): We specify its name, database version (PostgreSQL 14 in our example), region, and detailed settings for tier (machine type), disk_size, disk_type, and automated backup_configuration. For demonstration, public IP is enabled, though in production, you’d likely configure private IP and tighter authorized networks. Additionally, we define a database and a user within our Cloud SQL instance, showcasing how Terraform can manage not just the service but also its internal components.

outputs.tf: Extracting Information

While not directly impacting resource provisioning, outputs are incredibly useful. After Terraform successfully applies your configuration, these outputs will display key information about the newly created resources. For our GKE cluster, we’ll see its name and the control plane endpoint. For the Cloud SQL instance, we’ll get its connection name, public IP address (if enabled), the database name, and the username of the created database user. Outputs serve as a programmatic way to extract information from your deployed infrastructure, making it easy to integrate with subsequent automated steps or simply provide quick access to crucial details for human operators. It’s an essential part of making your IaC solution self-documenting and easily consumable by other processes.

# gcp-tf-cloudbuild-provisioner/terraform/outputs.tf
# This file defines the output values that will be displayed after Terraform applies the configuration.

# Output for GKE Cluster
output "gke_cluster_name" {
  description = "The name of the provisioned GKE cluster."
  value       = google_container_cluster.primary_gke_cluster.name
}

output "gke_cluster_endpoint" {
  description = "The endpoint of the GKE cluster's control plane."
  value       = google_container_cluster.primary_gke_cluster.endpoint
}

# Note: Master auth username/password are deprecated and less secure for production.
# They are included here for completeness but ideally, access should be managed via IAM.
output "gke_cluster_master_auth_username" {
  description = "The username for master authentication (if enabled, usually deprecated)."
  value       = google_container_cluster.primary_gke_cluster.master_auth[0].username
  sensitive   = true # Mark as sensitive to prevent logging
}

output "gke_cluster_master_auth_password" {
  description = "The password for master authentication (if enabled, usually deprecated)."
  value       = google_container_cluster.primary_gke_cluster.master_auth[0].password
  sensitive   = true # Mark as sensitive to prevent logging
}

# Output for Cloud SQL Instance
output "sql_instance_connection_name" {
  description = "The connection name of the Cloud SQL instance (ProjectID:Region:InstanceName)."
  value       = google_sql_database_instance.main_sql_instance.connection_name
}

output "sql_instance_public_ip_address" {
  description = "The public IP address of the Cloud SQL instance (if enabled)."
  value       = google_sql_database_instance.main_sql_instance.public_ip_address
}

output "sql_database_name" {
  description = "The name of the database created in Cloud SQL."
  value       = google_sql_database.app_database.name
}

output "sql_username" {
  description = "The username for the Cloud SQL database user."
  value       = google_sql_user.app_user.name
}

Deployment and Verification

With all our configuration files in place and understanding their purpose, it’s time to trigger the build! The deployment process is initiated with a simple gcloud builds submit command from your local machine (after navigating back to the root of your cloned repository).

gcloud builds submit --project=YOUR_GCP_PROJECT_ID --substitutions=_TF_STATE_BUCKET=YOUR_TF_STATE_BUCKET_NAME

You’ll need to replace YOUR_GCP_PROJECT_ID with your actual project ID and YOUR_TF_STATE_BUCKET_NAME with the name of the GCS bucket you created earlier. Once triggered, Cloud Build pulls your repository, starts a builder instance, and executes the steps defined in cloudbuild.yaml.

You can monitor the build progress in real-time directly from the GCP Console under Cloud Build > History. You’ll see each step – Terraform Init, Terraform Plan, and Terraform Apply – execute sequentially, logging their progress and any outputs. If all goes well, you’ll see a ‘SUCCESS’ message, indicating that your GKE cluster and Cloud SQL instance have been provisioned according to your Terraform definitions.

Verification and Cleanup

After a successful build, the next logical step is to verify that your resources have indeed been created as expected. Head over to the GCP Console:

• Navigate to Kubernetes Engine > Clusters – you should see your newly provisioned GKE cluster listed there, showing its status and details.
• Similarly, go to Databases > SQL – your Cloud SQL instance should be present, with its PostgreSQL version, region, and IP details.

This visual confirmation is crucial to ensure that the automated pipeline worked perfectly.

And when you’re done experimenting, remember cleanup is just as important. While you can modify the cloudbuild.yaml to include a terraform destroy step for full automation, for quick cleanup, you can temporarily replace the apply step with a destroy -auto-approve command in your cloudbuild.yaml and re-trigger the build:

# ... (other steps) ...
  # Step 3: Apply Terraform Changes (replaced with Destroy for cleanup)
  - name: 'gcr.io/cloud-builders/terraform'
    id: 'Terraform Destroy'
    args:
      - 'destroy'
      - '-auto-approve'
    dir: 'terraform'
    env:
      - 'GOOGLE_CLOUD_PROJECT=${PROJECT_ID}'

Just be extremely cautious, as destroy permanently deletes all the provisioned resources and their data. Always double-check before executing a destroy operation, especially in production or shared environments.

Conclusion

And there you have it! We’ve journeyed from the theoretical foundations of Infrastructure as Code to a practical, automated deployment of GCP resources using Cloud Build and Terraform. You’ve seen how this powerful combination enables consistent, repeatable, and auditable infrastructure provisioning, transforming your cloud management from a manual chore into an efficient, automated pipeline.

This setup is the cornerstone for building robust, scalable, and reliable cloud-native applications. Imagine the possibilities: integrating this into Git workflows (GitOps), using Secret Manager for sensitive data like database passwords, or dynamically deploying to different environments based on branch names. The journey into cloud automation is vast and rewarding, and this is just the beginning.

If you found this tutorial insightful and helpful, please give our YouTube video a big thumbs up, share it with your fellow developers, and don’t forget to subscribe to our channel for more deep dives into cloud engineering and automation! Your support helps us create more valuable content. See you in the next one!