Skip to content

Latest commit

 

History

History
932 lines (683 loc) · 44.1 KB

README-GitLab.md

File metadata and controls

932 lines (683 loc) · 44.1 KB
Raw

Deploying a GitLab CI/CD compatible environment

The objective of the following instructions is to configure the infrastructure that allows CI/CD deployments using GitLab CI/CD for the deploy of the Terraform Example Foundation stages (0-bootstrap, 1-org, 2-environments, 3-networks, 4-projects). This infrastructure consists in two Google Cloud Platform projects (prj-b-seed and prj-b-cicd-wif-gl).

It is a best practice to have two separate projects here (prj-b-seed and prj-b-cicd-wif-gl) for separation of concerns. On one hand, prj-b-seed stores terraform state and has the Service Accounts able to create / modify infrastructure. On the other hand, the authentication infrastructure using Workload identity federation is implemented in prj-b-cicd-wif-gl.

To run the instructions described in this document, install the following:

Also make sure that you have the following:

  • A GitLab account for your User or Group.
  • A private GitLab project (repository) for each one of the stages of Foundation and one for the GitLab runner Image:
    • Bootstrap
    • Organization
    • Environments
    • Networks
    • Projects
    • CI/CD Runner
  • A Personal access token or a Group access token configured with the following scopes:
    • api
    • read_api
    • create_runner
    • read_repository
    • write_repository
    • read_registry
    • write_registry
  • A Google Cloud organization.
  • A Google Cloud billing account.
  • Cloud Identity or Google Workspace groups for organization and billing admins.
  • For the user who will run the procedures in this document, grant the following roles:
    • The roles/resourcemanager.organizationAdmin role on the Google Cloud organization.
    • The roles/orgpolicy.policyAdmin role on the Google Cloud organization.
    • The roles/resourcemanager.projectCreator role on the Google Cloud organization.
    • The roles/billing.admin role on the billing account.
    • The roles/resourcemanager.folderCreator role.

Instructions

Clone Terraform Example Foundation repo

  1. Clone terraform-example-foundation into your local environment.

    git clone https://github.com/terraform-google-modules/terraform-example-foundation.git

Create git branches

  1. The instructions described in this document require that the branches used in the Terraform Example Foundation deploy to exist and to be protected. Follow the instructions on this section to create all the branches.

  2. Clone all the private projects you created at the same level of the terraform-example-foundation folder. You must have SSH keys configured with GitLab for authentication.

    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-BOOTSTRAP-REPO>.git gcp-bootstrap
    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-ORGANIZATION-REPO>.git gcp-org
    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-ENVIRONMENTS-REPO>.git gcp-environments
    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-NETWORKS-REPO>.git gcp-networks
    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-PROJECTS-REPO>.git gcp-projects
    git clone [email protected]:<GITLAB-OWNER>/<GITLAB-RUNNER-REPO>.git gcp-cicd-runner

  3. The layout should be:

    gcp-bootstrap/
gcp-cicd-runner/
gcp-environments/
gcp-networks/
gcp-org/
gcp-projects/
terraform-example-foundation/

  4. The following branches must be created in your git projects .

    • CI/CD Runner: image
    • Bootstrap: plan and production
    • Organization: plan and production
    • Environments: plan, development, nonproduction, and production
    • Networks: plan, development, nonproduction, and production
    • Projects: plan, development, nonproduction, and production

  5. These branches must not be empty, so that they can be the target branch of a merge request. Run the 0-bootstrap/scripts/git_create_branches_helper.sh script to create these branches with a seed file for each repository automatically.

    ./terraform-example-foundation/0-bootstrap/scripts/git_create_branches_helper.sh GITLAB

  6. The script will output logs related to the branches creation in the console and it will output the message "Branch creation and push completed for all repositories" at the end of the script execution.

  7. Terraform configuration on stage 0-bootstrap will make these branches protected

Build CI/CD runner image

  1. Navigate into CI/CD runner the repo. All subsequent steps assume you are running them from the gcp-cicd-runner directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-cicd-runner

  2. Checkout branch image. It will contain the Docker image used in the GitLab Pipelines.

    git checkout image

  3. Copy contents of foundation to cloned project (modify accordingly based on your current directory).

    cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/0-bootstrap/Dockerfile ./Dockerfile

  4. Save the CI/CD runner configuration to gcp-cicd-runner GitLab project:

    git add .
git commit -m 'Initialize CI/CD runner project'
git push

  5. Review the CI/CD Job output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-RUNNER-REPO/-/jobs under name build-image.

  6. If the CI/CD Job is successful proceed with the next steps.

Provide access to the CI/CD runner image

  1. Go to https://gitlab.com/GITLAB-OWNER/GITLAB-RUNNER-REPO/-/settings/ci_cd#js-token-access
  2. Add all the repositories: Bootstrap, Organization, Environments, Networks, and Projects to the allow list tha allow access to the CI/CD runner image.
  3. In "Allow CI job tokens from the following projects to access this project" add the other projects/repositories. Format is /

Deploying step 0-bootstrap

  1. Navigate into the Bootstrap repo. All subsequent steps assume you are running them from the gcp-bootstrap directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-bootstrap

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to cloned project (modify accordingly based on your current directory).

    mkdir -p envs/shared

cp -RT ../terraform-example-foundation/0-bootstrap/ ./envs/shared
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh
cd ./envs/shared

  4. In the versions file ./versions.tf un-comment the gitlab required provider

  5. In the variables file ./variables.tf un-comment variables in the section Specific to gitlab_bootstrap

  6. In the outputs file ./outputs.tf Comment-out outputs in the section Specific to cloudbuild_module

  7. In the outputs file ./outputs.tf un-comment outputs in the section Specific to gitlab_bootstrap

  8. Rename file ./cb.tf to ./cb.tf.example

    mv ./cb.tf ./cb.tf.example

  9. Rename file ./gitlab.tf.example to ./gitlab.tf

    mv ./gitlab.tf.example ./gitlab.tf

  10. Rename file terraform.example.tfvars to terraform.tfvars

    mv ./terraform.example.tfvars ./terraform.tfvars

  11. Update the file terraform.tfvars with values from your Google Cloud environment

  12. Update the file terraform.tfvars with values from your GitLab projects

  13. To prevent saving the gitlab_token in plain text in the terraform.tfvars file, export the GitLab personal or group access token as an environment variable:

    export TF_VAR_gitlab_token="YOUR-PERSONAL-OR-GROUP-ACCESS-TOKEN"

  14. Use the helper script validate-requirements.sh to validate your environment:

    ../../../terraform-example-foundation/scripts/validate-requirements.sh  -o <ORGANIZATION_ID> -b <BILLING_ACCOUNT_ID> -u <END_USER_EMAIL> -e

    Note: The script is not able to validate if the user is in a Cloud Identity or Google Workspace group with the required roles.

  15. Run terraform init and terraform plan and review the output.

    terraform init
terraform plan -input=false -out bootstrap.tfplan

  16. To validate your policies, run gcloud beta terraform vet. For installation instructions, see Validate policies instructions for the Google Cloud CLI.

  17. Run the following commands and check for violations:

    export VET_PROJECT_ID=A-VALID-PROJECT-ID

terraform show -json bootstrap.tfplan > bootstrap.json
gcloud beta terraform vet bootstrap.json --policy-library="../../policy-library" --project ${VET_PROJECT_ID}

    A-VALID-PROJECT-ID must be an existing project you have access to. This is necessary because gcloud beta terraform vet needs to link resources to a valid Google Cloud Platform project.

  18. No violations and an output with done means the validation was successful.

  19. Run terraform apply.

    terraform apply bootstrap.tfplan

  20. Run terraform output to get the email address of the terraform service accounts that will be used to run manual steps for shared environments in steps 3-networks-dual-svpc, 3-networks-hub-and-spoke, and 4-projects.

    export network_step_sa=$(terraform output -raw networks_step_terraform_service_account_email)
export projects_step_sa=$(terraform output -raw projects_step_terraform_service_account_email)

echo "network step service account = ${network_step_sa}"
echo "projects step service account = ${projects_step_sa}"

  21. Run terraform output to get the ID of your CI/CD project:

    export cicd_project_id=$(terraform output -raw cicd_project_id)
echo "CI/CD Project ID = ${cicd_project_id}"

  22. Copy the backend and update backend.tf with the name of your Google Cloud Storage bucket for Terraform's state. Also update the backend.tf of all steps.

    export backend_bucket=$(terraform output -raw gcs_bucket_tfstate)
echo "backend_bucket = ${backend_bucket}"

cp backend.tf.example backend.tf
cd ../../../

for i in `find . -name 'backend.tf'`; do sed -i'' -e "s/UPDATE_ME/${backend_bucket}/" $i; done
for i in `find . -name 'backend.tf'`; do sed -i'' -e "s/UPDATE_PROJECTS_BACKEND/${backend_bucket}/" $i; done

cd gcp-bootstrap/envs/shared

  23. Re-run terraform init. When you're prompted, agree to copy Terraform state to Cloud Storage.

    terraform init

  24. (Optional) Run terraform plan to verify that state is configured correctly. You should see no changes from the previous state.

  25. Save the Terraform configuration to gcp-bootstrap GitLab project:

    cd ../..
git add .
git commit -m 'Initialize bootstrap project'
git push

  26. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-BOOTSTRAP-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the production branch and review the output.

  27. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  28. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-BOOTSTRAP-REPO/-/pipelines.

  29. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  30. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  31. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-BOOTSTRAP-REPO/-/pipelines under tf-apply.

  32. If the GitLab pipelines is successful, apply the next environment.

  33. Before moving to the next step, go back to the parent directory.

    cd ..

Note 1: The stages after 0-bootstrap use terraform_remote_state data source to read common configuration like the organization ID from the output of the 0-bootstrap stage. They will fail if the state is not copied to the Cloud Storage bucket.

Note 2: After the deploy, to prevent the project quota error described in the Troubleshooting guide, we recommend that you request 50 additional projects for the projects step service account created in this step.

Note 3: In case GitLab variables are not being created on GitLab projects, it may be related to the existence of an Access Token in both User/Group profile and in the repo settings. The deploy will fail.

Note 4: If the GitLab pipelines fail in 0-bootstrap or next steps with access denied in the runner image, you may need to add all projects to the Token Access allow list in the CI/CD Runner repository.

Deploying step 1-org

  1. Navigate into the repo. All subsequent steps assume you are running them from the gcp-org directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-org

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to new repo.

    cp -RT ../terraform-example-foundation/1-org/ .
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh

  4. Rename ./envs/shared/terraform.example.tfvars to ./envs/shared/terraform.tfvars

    mv ./envs/shared/terraform.example.tfvars ./envs/shared/terraform.tfvars

  5. Update the file envs/shared/terraform.tfvars with values from your GCP environment. See the shared folder README.md for additional information on the values in the terraform.tfvars file.

  6. Un-comment the variable create_access_context_manager_access_policy = false if your organization already has an Access Context Manager Policy.

    export ORGANIZATION_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -json common_config | jq '.org_id' --raw-output)

export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")

echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"

if [ ! -z "${ACCESS_CONTEXT_MANAGER_ID}" ]; then sed -i'' -e "s=//create_access_context_manager_access_policy=create_access_context_manager_access_policy=" ./envs/shared/terraform.tfvars; fi

  7. Update the remote_state_bucket variable with the backend bucket from step Bootstrap.

    export backend_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate)

echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./envs/shared/terraform.tfvars

  8. Check if a Security Command Center Notification with the default name, scc-notify, already exists in your organization.

    export ORG_STEP_SA=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw organization_step_terraform_service_account_email)

gcloud scc notifications describe "scc-notify" --format="value(name)" --organization=${ORGANIZATION_ID} --impersonate-service-account=${ORG_STEP_SA}

  9. If the notification exists the output will be:

    organizations/ORGANIZATION_ID/notificationConfigs/scc-notify

  10. If the notification does not exist the output will be:

    ERROR: (gcloud.scc.notifications.describe) NOT_FOUND: Requested entity was not found.

  11. If the notification exists, choose a different value for the scc_notification_name variable in the ./envs/shared/terraform.tfvars file.

  12. Commit changes and Push your plan branch.

    git add .
git commit -m 'Initialize org repo'
git push

  13. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ORGANIZATION-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the production branch and review the output.

  14. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  15. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ORGANIZATION-REPO/-/pipelines.

  16. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  17. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  18. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ORGANIZATION-REPO/-/pipelines under tf-apply.

  19. If the GitLab pipelines is successful, apply the next environment.

  20. Before moving to the next step, go back to the parent directory.

    cd ..

Deploying step 2-environments

  1. Navigate into the repo. All subsequent steps assume you are running them from the gcp-environments directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-environments

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to new repo.

    cp -RT ../terraform-example-foundation/2-environments/ .
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh

  4. Rename terraform.example.tfvars to terraform.tfvars.

    mv terraform.example.tfvars terraform.tfvars

  5. Update the file with values from your GCP environment. See any of the envs folder README.md files for additional information on the values in the terraform.tfvars file.

  6. Update the remote_state_bucket variable with the backend bucket from step Bootstrap.

    export backend_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate)
echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" terraform.tfvars

  7. Commit changes and push your plan branch.

    git add .
git commit -m 'Initialize environments repo'
git push

  8. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the development branch and review the output.

  9. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the development environment.

  10. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines.

  11. If the GitLab pipelines is successful, merge the merge request in to the development branch.

  12. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the development environment.

  13. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines under tf-apply.

  14. If the GitLab pipelines is successful, apply the next environment.

  15. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/merge_requests?scope=all&state=opened from the development branch to the nonproduction branch and review the output.

  16. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the nonproduction environment.

  17. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines.

  18. If the GitLab pipelines is successful, merge the merge request in to the nonproduction branch.

  19. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the nonproduction environment.

  20. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines under tf-apply.

  21. If the GitLab pipelines is successful, apply the next environment.

  22. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/merge_requests?scope=all&state=opened from the nonproduction branch to the production branch and review the output.

  23. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  24. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines.

  25. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  26. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  27. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-ENVIRONMENTS-REPO/-/pipelines under tf-apply.

  28. Before moving to the next step, go back to the parent directory.

    cd ..

  29. You can now move to the instructions in the network stage. To use the Dual Shared VPC network mode go to Deploying step 3-networks-dual-svpc, or go to Deploying step 3-networks-hub-and-spoke to use the Hub and Spoke network mode.

Deploying step 3-networks-dual-svpc

  1. Navigate into the repo. All subsequent steps assume you are running them from the gcp-networks directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-networks

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to new repo.

    cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ .
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh

  4. Rename common.auto.example.tfvars to common.auto.tfvars, rename shared.auto.example.tfvars to shared.auto.tfvars and rename access_context.auto.example.tfvars to access_context.auto.tfvars.

    mv common.auto.example.tfvars common.auto.tfvars
mv shared.auto.example.tfvars shared.auto.tfvars
mv access_context.auto.example.tfvars access_context.auto.tfvars

  5. Update the file shared.auto.tfvars with the values for the target_name_server_addresses.

  6. Update the file access_context.auto.tfvars with the organization's access_context_manager_policy_id.

    export ORGANIZATION_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -json common_config | jq '.org_id' --raw-output)

export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")

echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"

sed -i'' -e "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars

  7. Update common.auto.tfvars file with values from your GCP environment. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars file.

  8. You must add your user email in the variable perimeter_additional_members to be able to see the resources created in the restricted project.

  9. Update the remote_state_bucket variable with the backend bucket from step Bootstrap in the common.auto.tfvars file.

    export backend_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw gcs_bucket_tfstate)

echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars

  10. Commit changes

    git add .
git commit -m 'Initialize networks repo'

  11. You must manually plan and apply the shared environment (only once) since the development, nonproduction and production environments depend on it.

  12. Use terraform output to get the CI/CD project ID and the networks step Terraform Service Account from gcp-bootstrap output.

  13. The CI/CD project ID will be used in the validation of the Terraform configuration

    export CICD_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw cicd_project_id)
echo ${CICD_PROJECT_ID}

  14. The networks step Terraform Service Account will be used for Service Account impersonation in the following steps. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set with the Terraform Service Account to enable impersonation.

    export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw networks_step_terraform_service_account_email)
echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

  15. Run init and plan and review output for environment shared.

    ./tf-wrapper.sh init shared
./tf-wrapper.sh plan shared

  16. To use the validate option of the tf-wrapper.sh script, please follow the instructions to install the terraform-tools component.

  17. Run validate and check for violations.

    ./tf-wrapper.sh validate shared $(pwd)/policy-library ${CICD_PROJECT_ID}

  18. Run apply shared.

    ./tf-wrapper.sh apply shared

  19. Push your plan branch.

    git push

  20. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the development branch and review the output.

  21. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the development environment.

  22. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  23. If the GitLab pipelines is successful, merge the merge request in to the development branch.

  24. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the development environment.

  25. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  26. If the GitLab pipelines is successful, apply the next environment.

  27. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the development branch to the nonproduction branch and review the output.

  28. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the nonproduction environment.

  29. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  30. If the GitLab pipelines is successful, merge the merge request in to the nonproduction branch.

  31. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the nonproduction environment.

  32. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  33. If the GitLab pipelines is successful, apply the next environment.

  34. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the nonproduction branch to the production branch and review the output.

  35. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  36. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  37. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  38. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  39. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  40. Before executing the next steps, unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

    unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

  41. Before moving to the next step, go back to the parent directory.

    cd ..

  42. You can now move to the instructions in the 4-projects stage.

Deploying step 3-networks-hub-and-spoke

  1. Navigate into the repo. All subsequent steps assume you are running them from the gcp-networks directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-networks

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to new repo.

    cp -RT ../terraform-example-foundation/3-networks-hub-and-spoke/ .
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh

  4. Rename common.auto.example.tfvars to common.auto.tfvars, rename shared.auto.example.tfvars to shared.auto.tfvars and rename access_context.auto.example.tfvars to access_context.auto.tfvars.

    mv common.auto.example.tfvars common.auto.tfvars
mv shared.auto.example.tfvars shared.auto.tfvars
mv access_context.auto.example.tfvars access_context.auto.tfvars

  5. Update common.auto.tfvars file with values from your GCP environment. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars file.

  6. You must add your user email in the variable perimeter_additional_members to be able to see the resources created in the restricted project.

  7. Update the remote_state_bucket variable with the backend bucket from step Bootstrap in the common.auto.tfvars file.

    export backend_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw gcs_bucket_tfstate)

echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars

  8. Commit changes

    git add .
git commit -m 'Initialize networks repo'

  9. You must manually plan and apply the shared environment (only once) since the development, nonproduction and production environments depend on it.

  10. Use terraform output to get the CI/CD project ID and the networks step Terraform Service Account from gcp-bootstrap output.

  11. The CI/CD project ID will be used in the validation of the Terraform configuration

    export CICD_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw cicd_project_id)
echo ${CICD_PROJECT_ID}

  12. The networks step Terraform Service Account will be used for Service Account impersonation in the following steps. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set with the Terraform Service Account to enable impersonation.

    export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw networks_step_terraform_service_account_email)
echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

  13. Run init and plan and review output for environment shared.

    ./tf-wrapper.sh init shared
./tf-wrapper.sh plan shared

  14. To use the validate option of the tf-wrapper.sh script, please follow the instructions to install the terraform-tools component.

  15. Run validate and check for violations.

    ./tf-wrapper.sh validate shared $(pwd)/policy-library ${CICD_PROJECT_ID}

  16. Run apply shared.

    ./tf-wrapper.sh apply shared

  17. Push your plan branch.

    git push

  18. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the development branch and review the output.

  19. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the development environment.

  20. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  21. If the GitLab pipelines is successful, merge the merge request in to the development branch.

  22. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the development environment.

  23. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  24. If the GitLab pipelines is successful, apply the next environment.

  25. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the development branch to the nonproduction branch and review the output.

  26. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the nonproduction environment.

  27. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  28. If the GitLab pipelines is successful, merge the merge request in to the nonproduction branch.

  29. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the nonproduction environment.

  30. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  31. If the GitLab pipelines is successful, apply the next environment.

  32. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/merge_requests?scope=all&state=opened from the nonproduction branch to the production branch and review the output.

  33. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  34. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines.

  35. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  36. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  37. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-NETWORKS-REPO/-/pipelines under tf-apply.

  38. Before executing the next steps, unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

    unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

  39. Before moving to the next step, go back to the parent directory.

    cd ..

  40. You can now move to the instructions in the 4-projects stage.

Deploying step 4-projects

  1. Navigate into the repo. All subsequent steps assume you are running them from the gcp-projects directory. If you run them from another directory, adjust your copy paths accordingly.

    cd gcp-projects

  2. change to a nonproduction branch.

    git checkout plan

  3. Copy contents of foundation to new repo.

    cp -RT ../terraform-example-foundation/4-projects/ .
cp -RT ../terraform-example-foundation/policy-library/ ./policy-library
cp ../terraform-example-foundation/build/gitlab-ci.yml ./.gitlab-ci.yml
cp ../terraform-example-foundation/build/run_gcp_auth.sh .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./*.sh

  4. Rename auto.example.tfvars files to auto.tfvars.

    mv common.auto.example.tfvars common.auto.tfvars
mv shared.auto.example.tfvars shared.auto.tfvars
mv development.auto.example.tfvars development.auto.tfvars
mv nonproduction.auto.example.tfvars nonproduction.auto.tfvars
mv production.auto.example.tfvars production.auto.tfvars

  5. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars, development.auto.tfvars, nonproduction.auto.tfvars, and production.auto.tfvars files.

  6. See any of the shared folder README.md files for additional information on the values in the shared.auto.tfvars file.

  7. Use terraform output to get the backend bucket value from bootstrap output.

    export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw gcs_bucket_tfstate)
echo "remote_state_bucket = ${remote_state_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars

  8. (Optional) If you want additional subfolders for separate business units or entities, make additional copies of the folder business_unit_1 and modify any values that vary across business unit like business_code, business_unit, or subnet_ip_range.

For example, to create a new business unit similar to business_unit_1, run the following:

#copy the business_unit_1 folder and it's contents to a new folder business_unit_2
cp -r  business_unit_1 business_unit_2

# search all files under the folder `business_unit_2` and replace strings for business_unit_1 with strings for business_unit_2
grep -rl bu1 business_unit_2/ | xargs sed -i 's/bu1/bu2/g'
grep -rl business_unit_1 business_unit_2/ | xargs sed -i 's/business_unit_1/business_unit_2/g'

  1. Commit changes.

    git add .
git commit -m 'Initialize projects repo'

  2. You need to manually plan and apply only once the business_unit_1/shared and business_unit_2/shared environments since development, nonproduction, and production depend on them.

  3. Use terraform output to get the CI/CD project ID and the projects step Terraform Service Account from gcp-bootstrap output.

  4. The CI/CD project ID will be used in the validation of the Terraform configuration

    export CICD_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw cicd_project_id)
echo ${CICD_PROJECT_ID}

  5. The projects step Terraform Service Account will be used for Service Account impersonation in the following steps. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set with the Terraform Service Account to enable impersonation.

    export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared/" output -raw projects_step_terraform_service_account_email)
echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

  6. Run init and plan and review output for environment shared.

    ./tf-wrapper.sh init shared
./tf-wrapper.sh plan shared

  7. To use the validate option of the tf-wrapper.sh script, please follow the instructions to install the terraform-tools component.

  8. Run validate and check for violations.

    ./tf-wrapper.sh validate shared $(pwd)/policy-library ${CICD_PROJECT_ID}

  9. Run apply shared.

    ./tf-wrapper.sh apply shared

  10. Push your plan branch.

    git push

  11. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/merge_requests?scope=all&state=opened from the plan branch to the development branch and review the output.

  12. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the development environment.

  13. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines.

  14. If the GitLab pipelines is successful, merge the merge request in to the development branch.

  15. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the development environment.

  16. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines under tf-apply.

  17. If the GitLab pipelines is successful, apply the next environment.

  18. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/merge_requests?scope=all&state=opened from the development branch to the nonproduction branch and review the output.

  19. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the nonproduction environment.

  20. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines.

  21. If the GitLab pipelines is successful, merge the merge request in to the nonproduction branch.

  22. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the nonproduction environment.

  23. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines under tf-apply.

  24. If the GitLab pipelines is successful, apply the next environment.

  25. Open a merge request in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/merge_requests?scope=all&state=opened from the nonproduction branch to the production branch and review the output.

  26. The merge request will trigger a GitLab pipelines that will run Terraform init/plan/validate in the production environment.

  27. Review the GitLab pipelines output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines.

  28. If the GitLab pipelines is successful, merge the merge request in to the production branch.

  29. The merge will trigger a GitLab pipelines that will apply the terraform configuration for the production environment.

  30. Review merge output in GitLab https://gitlab.com/GITLAB-OWNER/GITLAB-PROJECTS-REPO/-/pipelines under tf-apply.

  31. Unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

    unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT