This example deploys the Secure Cloud Run on top of the Terraform Example Foundation version 2.3.1.
This example will:
- Create two new projects under each environment folder and each business unit (bu1/bu2) for serverless within the foundation infrastructure.
- Attach the projects to the Restricted Shared VPC foundation network.
- Add all needed API's to the VPC Service Perimeter.
- Cloud Run API:
run.googleapis.com
- Artifact Registry API:
artifactregistry.googleapis.com
- Cloud Key Management Service (KMS) API:
cloudkms.googleapis.com
- Cloud Run API:
- Deploy Cloud Run service with a public hello_world image (the image can be replaced).
- Create a Serverless VPC Connector on the serverless project.
- Create a set of firewall rules to allow the communication between loadbalancer, serverless, connector, and health-check.
- Create a Loadbalancer with a domain and SSL certificate.
- Create a Network endpoint group (serverless-neg)
- Configure Cloud Armor with WAF security policy.
- terraform-example-foundation version 2.3.1 deployed until at least step
4-projects
. - You must have role Service Account User (
roles/iam.serviceAccountUser
) on the Terraform Service Account created in the foundation Seed Project. The Terraform Service Account has the permissions to deploy the foundation.- Format:
org-terraform@<SEED_PROJECT_ID>.iam.gserviceaccount.com
.
- Format:
The following instructions details the changes needed in the foundation Terraform configuration to deploy the Secure Cloud Run.
Grant the following roles related to Secure Cloud Run to the Terraform Service Account created in the Seed Project in step 0-bootstrap
.
- Serverless VPC Access Admin:
roles/vpcaccess.admin
- Security Admin :
roles/iam.securityAdmin
To do that:
- Add
roles/vpcaccess.admin
androles/iam.securityAdmin
roles at parent level for the Terraform Service Account in file main.tf in step0-bootstrap
.resource "google_organization_iam_member" "org_tf_serverless" { for_each = toset(var.parent_folder == "" ? ["roles/vpcaccess.admin", "roles/iam.securityAdmin"] : []) org_id = var.org_id role = each.key member = "serviceAccount:${module.seed_bootstrap.terraform_sa_email}" } resource "google_folder_iam_member" "folder_tf_serverless" { for_each = toset(var.parent_folder != "" ? ["roles/vpcaccess.admin", "roles/iam.securityAdmin"] : []) folder = var.parent_folder role = each.key member = "serviceAccount:${module.seed_bootstrap.terraform_sa_email}" }
- Rerun
Terraform apply
in step0-bootstrap
to update the roles.
The Secure Cloud Run requires two Organization Policies related to Cloud Run:
- Allowed ingress settings (Cloud Run)
- Allowed VPC egress settings (Cloud Run)
For the Terraform Example Foundation deploy, use the terraform-google-modules/org-policy/google
module
instead of the specific Secure Cloud Run Security module because the Secure Cloud Run Security module also creates KMS resources.
To apply these Organization Policies in Parent Level (Organization or Folder level), add the code below in 1-org
step.
-
Add org policies related to cloud run at
1-org/envs/shared/org_policy.tf
/****************************************** Cloud Run *******************************************/ module "cloudrun_allowed_ingress" { source = "terraform-google-modules/org-policy/google" version = "~> 5.1" constraint = "constraints/run.allowedIngress" organization_id = local.organization_id folder_id = local.folder_id policy_for = local.policy_for policy_type = "list" allow = ["is:internal-and-cloud-load-balancing"] allow_list_length = 1 } module "cloudrun_allowed_vpc_egress" { source = "terraform-google-modules/org-policy/google" version = "~> 5.1" organization_id = local.organization_id folder_id = local.folder_id policy_for = local.policy_for constraint = "constraints/run.allowedVPCEgress" policy_type = "list" allow = ["private-ranges-only"] allow_list_length = 1 }
-
Push the code to your repository in the branch you are working on (development for example).
Add the VPC Access API (vpcaccess.googleapis.com
) in the list of allowed APIs for the serviceusage_allow_basic_apis
constraint in the Policy Library Repository used by Terraform Validator. Also enable the VPC Access API in the restricted shared VPC host project:
- Add
vpcaccess.googleapis.com
in/policies/constraints/serviceusage_allow_basic_apis.yaml
file in your policy repository (gcp-policies
) and push the code to the repository. - Add
vpcaccess.googleapis.com
on theactivate_apis
list inrestricted_shared_vpc_host_project
module in filegcp-environments/modules/env_baseline/networking.tf#71
- Push the code for the branch in the repository (gcp-environment).
Create the project where Secure Cloud Run will be deployed.
This project will be attached to the Restricted Shared VPC and added to the Service Perimeter.
Also add the Cloud Run API (run.googleapis.com
), the Cloud Key Management Service (KMS) API (cloudkms.googleapis.com
), and the Artifact Registry API (artifactregistry.googleapis.com
) in the Policy Library Repository used by Terraform Validator.
-
Add
run.googleapis.com
,cloudkms.googleapis.com
,artifactregistry.googleapis.com
in/policies/constraints/serviceusage_allow_basic_apis.yaml
file in your policy repository (gcp-policies) and push the code to it. -
Duplicate the file in each business unit and environment
business_unit_[number]/[environment]/example_restricted_shared_vpc_project.tf
and rename it toexample_restricted_shared_vpc_serverless_project
in your projects repository (gcp-projects). -
Replace the code in
example_restricted_shared_vpc_serverless_project.tf
file with the following code (in thelocals
section, replace values according to the environment and the business code):/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ locals { env = "production" business_code = "bu1" } module "restricted_shared_vpc_serverless_project" { source = "../../modules/single_project" org_id = var.org_id billing_account = var.billing_account folder_id = data.google_active_folder.env.name impersonate_service_account = var.terraform_service_account environment = local.env vpc_type = "restricted" alert_spent_percents = var.alert_spent_percents alert_pubsub_topic = var.alert_pubsub_topic budget_amount = var.budget_amount project_prefix = var.project_prefix enable_hub_and_spoke = var.enable_hub_and_spoke enable_cloudbuild_deploy = true cloudbuild_sa = var.app_infra_pipeline_cloudbuild_sa sa_roles = ["roles/editor"] activate_apis = [ "cloudresourcemanager.googleapis.com", "storage-api.googleapis.com", "serviceusage.googleapis.com", "run.googleapis.com", "cloudkms.googleapis.com", "iam.googleapis.com", "vpcaccess.googleapis.com" ] vpc_service_control_attach_enabled = "true" vpc_service_control_perimeter_name = "accessPolicies/${var.access_context_manager_policy_id}/servicePerimeters/${var.perimeter_name}" # Metadata project_suffix = "serverless" application_name = "${local.business_code}-serverless-application" billing_code = "1234" primary_contact = "[email protected]" secondary_contact = "[email protected]" business_code = local.business_code } resource "google_folder_iam_member" "folder_browser" { count = var.parent_folder != ""? 1 : 0 folder = var.parent_folder role = "roles/browser" member = "serviceAccount:${module.restricted_shared_vpc_serverless_project.sa}" } resource "google_folder_iam_member" "folder_network_viewer" { count = var.parent_folder != ""? 1 : 0 folder = var.parent_folder role = "roles/compute.networkViewer" member = "serviceAccount:${module.restricted_shared_vpc_serverless_project.sa}" } module "serverless_security_project" { source = "../../modules/single_project" org_id = var.org_id billing_account = var.billing_account folder_id = data.google_active_folder.env.name impersonate_service_account = var.terraform_service_account environment = local.env vpc_type = "" alert_spent_percents = var.alert_spent_percents alert_pubsub_topic = var.alert_pubsub_topic budget_amount = var.budget_amount project_prefix = var.project_prefix enable_hub_and_spoke = var.enable_hub_and_spoke activate_apis = [ "cloudresourcemanager.googleapis.com", "storage-api.googleapis.com", "serviceusage.googleapis.com", "cloudkms.googleapis.com", "iam.googleapis.com", "artifactregistry.googleapis.com" ] vpc_service_control_attach_enabled = "true" vpc_service_control_perimeter_name = "accessPolicies/${var.access_context_manager_policy_id}/servicePerimeters/${var.perimeter_name}" # Metadata project_suffix = "security" application_name = "${local.business_code}-security-serverless" billing_code = "1234" primary_contact = "[email protected]" secondary_contact = "[email protected]" business_code = local.business_code }
Now, you will need to add the roles to the service account created in the project, which is going to be used to deploy the Cloud Run.
-
Add the required Cloud Run roles to the service account, created in the project, that will be used to deploy Cloud Run. Edit the file
gcp-projects/modules/single_project/main.tf
to add the required roles.resource "google_folder_iam_member" "storage_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/storage.admin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "cloud_run_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/run.admin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "network_user" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/compute.networkUser" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "service_account_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/iam.serviceAccountAdmin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "compute_security_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/compute.securityAdmin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "iam_security_reviewer" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/iam.securityReviewer" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "browser" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/browser" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "load_balancer_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/compute.loadBalancerAdmin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "kms_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/cloudkms.admin" member = "serviceAccount:${module.project.service_account_email}" } resource "google_folder_iam_member" "artifact_registry_admin" { count = var.enable_cloudbuild_deploy ? 1 : 0 folder = var.folder_id role = "roles/artifactregistry.admin" member = "serviceAccount:${module.project.service_account_email}" }
-
Add the outputs in files
gcp-projects/business_unit_[number]/[environment]/outputs.tf
output "restricted_serverless_project" { description = "Serverless project id." value = module.restricted_shared_vpc_serverless_project.project_id } output "security_project" { description = "Serverless security project id." value = module.serverless_security_project.project_id }
-
Push the code to your repository in the branch you are working on (development for example).
You will add the Cloud Run, Cloud KMS and Artifact Register services to the Restricted Services in the Service Perimeter.
-
Add
run.googleapis.com
,artifactregistry.googleapis.com
,cloudkms.googleapis.com
in filesenvs/[environment]/main.tf#103
-restricted_services
module variable in you network module (gcp-network) -
Add the Serverless Project Service Account and the Cloud Build Service Account as members of the perimeter. In
envs/[environment]/main.tf#104
add -members
module variable in you network module (gcp-network)members = [ "serviceAccount:${var.terraform_service_account}", "serviceAccount:project-service-account@<YOUR-SERVERLESS-PROJECT>.iam.gserviceaccount.com", "serviceAccount:<APP-CLOUDBUILD-PROJECT-NUMBER>@cloudbuild.gserviceaccount.com" ]
-
Add
secure-cloud-run-net
module in fileenvs/[environment]/main.tf
module in you network module (gcp-network)data "google_projects" "serverless_project" { filter = "parent.id:${split("/", data.google_active_folder.env.name)[1]} labels.application_name=bu1-serverless-application labels.environment=${local.env} lifecycleState=ACTIVE" } module "serverless_network" { source = "GoogleCloudPlatform/cloud-run/google//modules/secure-cloud-run-net" version = "~> 0.3.0" connector_name = "con-${local.environment_code}-${var.default_region1}-run" shared_vpc_name = module.restricted_shared_vpc.network_name subnet_name = "sb-${local.environment_code}-run-${var.default_region1}" location = var.default_region1 vpc_project_id = local.restricted_project_id serverless_project_id = data.google_projects.serverless_project.projects[0].project_id ip_cidr_range = "10.8.0.0/28" connector_on_host_project = false }
-
Push code to the environment branch in gcp-network repository
Deploy the Secure Cloud Run with Load Balancer and Cloud Armor.
To do this deploy, you will replace the example in 5-app-infra
and instead of creating a Compute Engine, you will deploy the Secure Cloud Run.
This is an example of deployment. It will create a Artifact Registry and copy a public image to it to exemplify the deployment using a private Artifact Registry.
-
Add
secure-cloud-run-core
module in/5-app-infra/modules/env_base/main.tf
/** * Copyright 2022 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ locals { environment_code = element(split("", var.environment), 0) key_name = "secure-cloud-run" } resource "google_service_account" "cloudrun_service_account" { project = data.google_project.env_project.project_id account_id = "sa-example-app" display_name = "Example app service Account" } resource "google_service_account_iam_member" "run_identity_terraform_sa_impersonate_permissions" { service_account_id = google_service_account.cloudrun_service_account.id role = "roles/iam.serviceAccountUser" member = "serviceAccount:${google_project_service_identity.serverless_sa.email}" } resource "random_id" "kms_random" { byte_length = 4 } module "kms" { source = "terraform-google-modules/kms/google" version = "~> 2.2" project_id = data.google_project.sec_project.project_id keyring = "kms-secure-cloud-run-${random_id.kms_random.hex}" location = var.region keys = [local.key_name, "artifact-registry"] encrypters = [ "serviceAccount:${google_service_account.cloudrun_service_account.email},serviceAccount:${google_project_service_identity.serverless_sa.email}", "serviceAccount:${google_project_service_identity.artifact_sa.email}" ] set_encrypters_for = [local.key_name, "artifact-registry"] decrypters = [ "serviceAccount:${google_service_account.cloudrun_service_account.email},serviceAccount:${google_project_service_identity.serverless_sa.email}", "serviceAccount:${google_project_service_identity.artifact_sa.email}" ] set_decrypters_for = [local.key_name, "artifact-registry"] prevent_destroy = "false" depends_on = [ google_service_account.cloudrun_service_account, google_project_service_identity.serverless_sa ] } module "cloud_run_core" { source = "git::https://github.com/GoogleCloudPlatform/terraform-google-cloud-run.git//modules/secure-cloud-run-core?ref=main" service_name = "example-secure-cloudrun" location = var.region project_id = data.google_project.env_project.project_id region = var.region image = var.image cloud_run_sa = google_service_account.cloudrun_service_account.email vpc_connector_id = "projects/${data.google_project.env_project.project_id}/locations/${var.region}/connectors/${var.vpc_connector_name}" encryption_key = module.kms.keys[local.key_name] members = var.members domain = var.domain depends_on = [ google_service_account_iam_member.run_identity_terraform_sa_impersonate_permissions ] } resource "google_project_service_identity" "artifact_sa" { provider = google-beta project = data.google_project.sec_project.project_id service = "artifactregistry.googleapis.com" } resource "google_artifact_registry_repository" "repo" { project = data.google_project.sec_project.project_id location = var.region repository_id = "rep-secure-cloud-run" description = "Repository to store Serverles Docker Image" format = "DOCKER" kms_key_name = module.kms.keys["artifact-registry"] } resource "google_artifact_registry_repository_iam_member" "member" { project = data.google_project.sec_project.project_id location = var.region repository = google_artifact_registry_repository.repo.repository_id role = "roles/artifactregistry.reader" member = "serviceAccount:${google_project_service_identity.serverless_sa.email}" } resource "null_resource" "copy_image" { provisioner "local-exec" { command = "gcloud container images add-tag us-docker.pkg.dev/cloudrun/container/hello:latest ${var.region}-docker.pkg.dev/${data.google_project.sec_project.project_id}/${google_artifact_registry_repository.repo.repository_id}/hello:latest -q" } }
-
Replace the code file in
/5-app-infra/modules/env_base/data.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ data "google_projects" "environment_projects" { filter = "parent.id:${split("/", var.folder_id)[1]} name:*${var.project_suffix}* labels.application_name=${var.business_code}-serverless-application labels.environment=${var.environment} lifecycleState=ACTIVE" } data "google_project" "env_project" { project_id = data.google_projects.environment_projects.projects[0].project_id } data "google_project" "sec_project" { project_id = data.google_projects.security_projects.projects[0].project_id } data "google_projects" "security_projects" { filter = "parent.id:${split("/", var.folder_id)[1]} name:*security* labels.application_name=${var.business_code}-security-serverless labels.environment=${var.environment} lifecycleState=ACTIVE" } resource "google_project_service_identity" "serverless_sa" { provider = google-beta project = data.google_project.env_project.project_id service = "run.googleapis.com" }
-
Replace the code file in
/5-app-infra/modules/env_base/variables.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ variable "environment" { description = "The environment the single project belongs to" type = string } variable "domain" { description = "Your domain." type = string } variable "members" { description = "The users who can invoke cloud run." type = list(string) } variable "vpc_connector_name" { description = "The VPC Serverless Connector name." type = string } variable "image" { description = "The docker image to be deployed in Cloud Run." type = string } variable "region" { description = "The GCP region to create and test resources in" type = string default = "us-central1" } variable "folder_id" { description = "The folder id where project will be created" type = string } variable "business_code" { description = "The code that describes which business unit owns the project" type = string default = "abcd" } variable "project_suffix" { description = "The name of the GCP project. Max 16 characters with 3 character business unit code." type = string }
-
Replace the code file in
/5-app-infra/modules/env_base/outputs.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ output "kms_keyring_selflink" { description = "Self link of the keyring." value = module.restricted_serverless.kms_keyring_selflink } output "kms_key" { description = "Key name used by Cloud Run." value = module.restricted_serverless.kms_key } output "cloud_run_service" { description = "Cloud Run service status." value = module.restricted_serverless.cloud_run_service }
-
Replace the code file in
/5-app-infra/business_unit_[bu]/[env]/main.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ data "google_active_folder" "env" { display_name = "${var.folder_prefix}-production" parent = var.parent_folder != "" ? "folders/${var.parent_folder}" : "organizations/${var.org_id}" } module "restricted_serverless" { source = "../../modules/env_base" environment = "production" image = "${var.region}-docker.pkg.dev/${module.restricted_serverless.security_project_id}/${module.restricted_serverless.artifact_registry_repository_name}/hello:latest" vpc_connector_name = "con-p-${var.region}-run" folder_id = data.google_active_folder.env.name business_code = "bu1" project_suffix = "serverless" region = var.region domain = var.domain members = var.members }
-
Replace the code file in
/5-app-infra/business_unit_[bu]/[env]/variables.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ variable "project_service_account" { description = "Email of the service account created on step 4-projects for the business unit 1 sample base project where the GCE instance will be created" type = string } variable "org_id" { description = "The organization id for the associated services" type = string } variable "folder_prefix" { description = "Name prefix to use for folders created. Should be the same in all steps." type = string default = "fldr" } variable "parent_folder" { description = "Optional - for an organization with existing projects or for development/validation. It will place all the example foundation resources under the provided folder instead of the root organization. The value is the numeric folder ID. The folder must already exist. Must be the same value used in previous step." type = string default = "" } variable "domain" { description = "Your domain." type = string } variable "members" { description = "The users who can invoke Cloud Run." type = list(string) } variable "region" { description = "The GCP region to create and test resources in" type = string default = "us-central1" }
-
Replace the code file in
/5-app-infra/business_unit_[bu]/[env]/outputs.tf
with:/** * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ output "kms_keyring_selflink" { description = "Self link of the keyring." value = module.restricted_serverless.kms_keyring_selflink } output "kms_key" { description = "Key name used by Cloud Run." value = module.restricted_serverless.kms_key } output "cloud_run_service" { description = "Cloud Run service status." value = module.restricted_serverless.cloud_run_service }
-
Change the project_service_account on your environments file (eg.: bu1-development.auto.tfvars) to the service account from serverless project (eg.: project-service-account@<SERVERLESS_PROJECT_ID>.iam.gserviceaccount.com)
-
Push code to the bu[number]-example-app repository.