Storing service account keys in Hashicorp Vault

Storing service account secrets in Hashicorp Vault

This feature allows you to store Google service account certs for Apigee Hybrid in Hashicorp Vault, an external secret manager. External secret managers allow you to manage how data is stored in Kubernetes, including managing data residency and fine grained access controls.

If you are not using Workload Identity on GKE clusters or Workload Identity Federation on EKS and AKS, your Apigee hybrid components need to authenticate Google service accounts to be able to perform their tasks. There are three methods for storing and referring to Google service account keys in Apigee hybrid:

  • Service account cert files (.json files) stored on a hard drive. Refer to these in your overrides with the serviceAccountPath configuration property. For example:
    logger:
      serviceAccountPath: service-accounts/myhybridorg-apigee-logger.json
    
  • Service account cert files (.json files) stored on a hard drive. Refer to these in your overrides with the serviceAccountPath configuration property. See About service accounts.
  • Service account certs stored in a Kubernetes secret. Refer to these in your overrides with the serviceAccountRef configuration property. See Storing data in a Kubernetes secret.
  • Service account certs stored in Hashicorp Vault, explained in this guide. Refer to these in your overrides with the serviceAccountSecretProviderClass configuration property.

Set up to store service account secrets in Vault

Install CSI driver and Vault provider

If you haven't already installed the CSI driver on your cluster using Helm, follow the instructions in Secrets Store CSI Driver: Installation. For more information, see Installing the Vault CSI provider in the Vault documentation.

See Apigee hybrid supported platforms and versions for the minimum CSI Driver versions supported by Apigee hybrid.

Create Vault secrets, policies, and roles

Use the Vault UI or APIs to create secrets and grant permissions for the Kubernetes service accounts used by Apigee hybrid to read those secrets.

  1. Create the organization and environment-specific secrets in the following format:
    Secret KeySecret Data
    secret/data/apigee/orgsakeys
    {
      "cassandraBackup": "***",
      "cassandraRestore": "***",
      "connectAgent": "***",
      "logger": "***",
      "mart": "***",
      "metrics": "***",
      "mint": "***",
      "udca": "***",
      "watcher": "***"
    }
    secret/data/apigee/envsakeys-ENV_NAME
    {
      "runtime": "***",
      "synchronizer": "***",
      "udca": "***". 
    }

    Replace the "***" in each pair with the contents of the .json file for the google service account corresponding to the apigee component. apigee-cassandra-backup and apigee-cassandra-restore both use the apigee-cassandra service account. For example:

    {
      "cassandraBackup": "{
        "type": "service_account",
        "project_id": "myhybridorg",
        "private_key_id": "PRIVATE_KEY_ID",
        "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY_TEXT\n-----END PRIVATE KEY-----\n",
        "client_email": "[email protected]",
        "client_id": "123456789012345678901",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/apigee-cassandra%40myhybridorg.iam.gserviceaccount.com",
        "universe_domain": "googleapis.com"
      }",
      "cassandraRestore":...
    ...
    }
        
  2. Grant access to the organization secret. Create a text file named orgsakeys-auth-policy.txt with the following contents:
    path "secret/data/apigee/orgsakeys" {
      capabilities = ["read"]
    }
  3. Within Vault, create a policy which grants access to the organization secret:
    vault policy write apigee-orgsakeys-auth orgsakeys-auth-policy.txt
  4. For each environment, create a text file named envsakeys-ENV_NAME-auth-policy.txt with the following contents:
    path "secret/data/apigee/envsakeys-ENV_NAME" {
      capabilities = ["read"]
    }

    Repeat this step for each environment.

  5. Within Vault, create a policy which grants access to the environment secret:
    vault policy write apigee-envsakeys-ENV_NAME-auth envsakeys-ENV_NAME-auth-policy.txt

    Repeat this step for each environment.

  6. Create a script called generate-encoded-sas.sh with the following contents:
    # generate-encoded-sas.sh
    
    ORG=$APIGEE_ORG            # Apigee organization name
    ENVS=$APIGEE_ENV_LIST      # comma separated env names, for example: dev,prod
    
    ORG_SHORT_NAME=$(echo $ORG | head -c 15)
    ENCODE=$(echo -n $ORG | shasum -a 256 | head -c 7)
    ORG_ENCODE=$(echo "$ORG_SHORT_NAME-$ENCODE")
    NAMES=apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-${ORG_ENCODE},apigee-cassandra-schema-val-${ORG_ENCODE},apigee-cassandra-user-setup-${ORG_ENCODE},apigee-mart-${ORG_ENCODE},apigee-mint-task-scheduler-${ORG_ENCODE},apigee-connect-agent-${ORG_ENCODE},apigee-watcher-${ORG_ENCODE},apigee-udca-${ORG_ENCODE},apigee-metrics-apigee-telemetry,apigee-open-telemetry-collector-apigee-telemetry,apigee-logger-apigee-telemetry
    
    for ENV in ${ENVS//,/ }
    do
      ENV_SHORT_NAME=$(echo $ENV | head -c 15)
      ENCODE=$(echo -n $ORG:$ENV | shasum -a 256 | head -c 7)
      ENV_ENCODE=$(echo "$ORG_SHORT_NAME-$ENV_SHORT_NAME-$ENCODE")
      NAMES+=,apigee-synchronizer-${ENV_ENCODE},apigee-runtime-${ENV_ENCODE}
    done
    
    echo $NAMES
    
  7. Run the script to generate the service account name list to bind the policy to:
    ./generate-encoded-sas.sh

    Your output should be a list of Kubernetes service account names separated by commas, similar to the following example:

    ./generate-encoded-sas.sh
    apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,
    apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-myhybrido
    rg-5b044c1,apigee-cassandra-schema-val-myhybridorg-5b044c1,apigee-c
    assandra-user-setup-myhybridorg-5b044c1,apigee-mart-myhybridorg-5b0
    44c1,apigee-mint-task-scheduler-myhybridorg-5b044c1,apigee-connect-
    agent-myhybridorg-5b044c1,apigee-watcher-myhybridorg-5b044c1,apigee
    -udca-myhybridorg-5b044c1,apigee-metrics-apigee-telemetry,apigee-op
    en-telemetry-collector-apigee-telemetry,apigee-logger-apigee-teleme
    try,apigee-synchronizer-myhybridorg-dev-ee52aca,apigee-runtime-myhy
    bridorg-dev-ee52aca,apigee-synchronizer-myhybridorg-prod-2d0221c,ap
    igee-runtime-myhybridorg-prod-2d0221c
  8. Copy the output text into and separate it into lists, one list for the org service account names and a separate list for the env service account name for each environment. The org service accounts are first in the output list up to apigee-logger-apigee-telemetry.

    The list of org service names from the previous example:

    apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,
    apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-myhybrido
    rg-5b044c1,apigee-cassandra-schema-val-myhybridorg-5b044c1,apigee-c
    assandra-user-setup-myhybridorg-5b044c1,apigee-mart-myhybridorg-5b0
    44c1,apigee-mint-task-scheduler-myhybridorg-5b044c1,apigee-connect-
    agent-myhybridorg-5b044c1,apigee-watcher-myhybridorg-5b044c1,apigee
    -udca-myhybridorg-5b044c1,apigee-metrics-apigee-telemetry,apigee-op
    en-telemetry-collector-apigee-telemetry,apigee-logger-apigee-teleme
    try

    The env service account names have the pattern apigee-synchronizer-ORG_NAME-ENV_NAME-HASH_TEXT and apigee-runtime-ORG_NAME-ENV_NAME-HASH_TEXT. Separate them into separated lists for each environment. For example, the output from the previous example can be separated into the following two lists:

    dev environment:

    apigee-synchronizer-myhybridorg-dev-ee52aca,apigee-runtime-myhybrid
    org-dev-ee52aca

    prod environment:

    apigee-synchronizer-myhybridorg-prod-2d0221c,apigee-runtime-myhybri
    dorg-prod-2d0221c
  9. Using the policy, create a Vault role which binds the organization specific Apigee service accounts:
    vault write auth/kubernetes/role/apigee-orgsakeys \
        bound_service_account_names=LIST_OF_ORG_SA_NAMES \
        bound_service_account_namespaces=apigee \
        policies=apigee-orgsakeys-auth \
        ttl=1m
    
  10. For each environment, create a Vault role for its service account keys:
    vault write auth/kubernetes/role/apigee-envsakeys-ENV_NAME \
        bound_service_account_names=LIST_OF_ENV_NAME_SA_NAMES \
        bound_service_account_namespaces=apigee \
        policies=apigee-envsakeys-ENV_NAME-auth \ 
        ttl=1m
    

    Repeat this step for every environment.

Create SecretProviderClass objects

The SecretProviderClass resource tells the CSI driver what provider to communicate with when requesting secrets. The service account keys must be configured via this object. The following table shows the file names (objectNames) expected by Apigee Hybrid:

Service accountExpected secret file names
Cassandra backup cassandraBackup
Cassandra restore cassandraRestore
Connect agent connectAgent
Logger logger
MART mart
Metrics metrics
Monetization mint
Runtime runtime
Synchronizer synchronizer
UDCA udca
Watcher watcher
  1. Use the following SecretProviderClass template to configure this resource for the organization-specific secrets:
    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: apigee-orgsakeys-spc
    spec:
      provider: vault
      parameters:
        roleName: apigee-orgsakeys
        vaultAddress: VAULT_ADDRESS
        # "objectName" is an alias used within the SecretProviderClass to reference
        # that specific secret. This will also be the filename containing the secret.
        # Apigee Hybrid expects these exact values so they must not be changed.
        # "secretPath" is the path in Vault where the secret should be retrieved.
        # "secretKey" is the key within the Vault secret response to extract a value from.
        objects: |
          - objectName: "cassandraBackup"
            secretPath: ""
            secretKey: ""
          - objectName: "cassandraRestore"
            secretPath: ""
            secretKey: ""
          - objectName: "connectAgent"
            secretPath: ""
            secretKey: ""
          - objectName: "logger"
            secretPath: ""
            secretKey: ""
          - objectName: "mart"
            secretPath: ""
            secretKey: ""
          - objectName: "metrics"
            secretPath: ""
            secretKey: ""
          - objectName: "mint"
            secretPath: ""
            secretKey: ""
          - objectName: "udca"
            secretPath: ""
            secretKey: ""
          - objectName: "watcher"
            secretPath: ""
            secretKey: ""
    

    VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster as Apigee, the format will generally be http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

    Save the template to a file named spc-org.yaml.

  2. Apply the org-specific SecretProviderClass to your apigee namespace:
    kubectl -n apigee apply -f spc-org.yaml
  3. For each environment, use the following SecretProviderClass template to configure this resource for the environment-specific secrets. Repeat this step for every environment:
    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: apigee-envsakeys-ENV_NAME-spc
    spec:
      provider: vault
      parameters:
        roleName: apigee-envsakeys-ENV_NAME
        vaultAddress: VAULT_ADDRESS
        # "objectName" is an alias used within the SecretProviderClass to reference
        # that specific secret. This will also be the filename containing the secret.
        # Apigee Hybrid expects these exact values so they must not be changed.
        # "secretPath" is the path in Vault where the secret should be retrieved.
        # "secretKey" is the key within the Vault secret response to extract a value from.
        objects: |
          - objectName: "runtime"
            secretPath: ""
            secretKey: ""
          - objectName: "synchronizer"
            secretPath: ""
            secretKey: ""
          - objectName: "udca"
            secretPath: ""
            secretKey: ""
    

    VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster as Apigee, the format will generally be http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

    Save the template to a file named spc-env-ENV_NAME.yaml.

  4. For each environment, apply the environment-specific SecretProviderClass to your apigee namespace:
    kubectl -n apigee apply -f spc-env-ENV_NAME.yaml

    Repeat this step for every environment.

Enable external secrets for Google service accounts

  1. Within your overrides file, add the serviceAccountSecretProviderClass and envs[].serviceAccountSecretProviderClass configuration properties to enable external secret usage for Google service accounts. You can remove or comment out the serviceAccountPath and serviceAccountPaths configuration properties:
    serviceAccountSecretProviderClass: apigee-orgsakeys-spc
    
    envs:
      - name: ENV_NAME
        serviceAccountSecretProviderClass: apigee-envsakeys-ENV_NAME-spc
  2. Apply each Helm chart:
    helm upgrade datastore apigee-datastore/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade telemetry apigee-telemetry/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade redis apigee-redis/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade ORG_NAME apigee-org/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
  3. For each environment, apply the apigee-env chart:
    helm upgrade ENV_NAME apigee-env/ \
        --install \
        --namespace apigee \
        --set env=ENV_NAME \
        --atomic \
        -f overrides.yaml
    

Rolling back to using service account cert files

If you need to roll back to using stored Google service account cert files, use the following procedure:

  1. Update your overrides file:
    1. Remove or comment out the serviceAccountSecretProviderClass and envs:serviceAccountSecretProviderClass lines.
    2. Add the serviceAccountPath and serviceAccountPaths configuration properties with the paths to the appropriate service accounts.

    For example:

    # serviceAccountSecretProviderClass: apigee-orgsakeys-spc - (commented out)
    
    cassandra:
      backup:
        serviceAccountPath: service-accounts/myhybridorg-apigee-cassandra.json
      restore:
        serviceAccountPath: service-accounts/myhybridorg-apigee-cassandra.json
    
    connectAgent:
      serviceAccountPath: service-accounts/myhybridorg-apigee-mart.json
    
    envs:
      - name: test
      # serviceAccountSecretProviderClass: apigee-envsakeys-spc - (commented out)
        serviceAccountPaths:
          runtime: service-accounts/myhybridorg-apigee-runtime.json
          synchronizer: service-accounts/myhybridorg-apigee-synchronizer.json
    
    logger:
      serviceAccountPath: service-accounts/myhybridorg-apigee-logger.json
    
    mart:
      serviceAccountPath: service-accounts/myhybridorg-apigee-mart.json
    
    metrics:
      serviceAccountPath: service-accounts/myhybridorg-apigee-metrics.json
    
    udca:
      serviceAccountPath: service-accounts/myhybridorg-apigee-udca.json
    
    watcher:
      serviceAccountPath: service-accounts/myhybridorg-apigee-watcher.json
    
  2. Apply each Helm chart:
    helm upgrade datastore apigee-datastore/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade telemetry apigee-telemetry/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade redis apigee-redis/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
    helm upgrade ORG_NAME apigee-org/ \
        --install \
        --namespace apigee \
        --atomic \
        -f overrides.yaml
    
  3. For each environment, apply the apigee-env chart:
    helm upgrade ENV_NAME apigee-env/ \
        --install \
        --namespace apigee \
        --set env=ENV_NAME \
        --atomic \
        -f overrides.yaml
    

    Repeat this step for every environment.