Skip to content

Update guide for Google Cloud Secret Manager Backend #10172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Aug 6, 2020
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions airflow/providers/google/cloud/secrets/secret_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@ class CloudSecretManagerBackend(BaseSecretsBackend, LoggingMixin):
:type gcp_keyfile_dict: dict
:param gcp_scopes: Comma-separated string containing GCP scopes
:type gcp_scopes: str
:param project_id: Project id (if you want to override the project_id from credentials)
:param project_id: Project ID. If not passed, the project ID from credentials will be used.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add something what is the values used for like
"Project Id to read the secrets from. If not provided, thethe project ID from credentials id used"

:type project_id: str
:param sep: separator used to concatenate connections_prefix and conn_id. Default: "-"
:param sep: Separator used to concatenate connections_prefix and conn_id. Default: "-"
:type sep: str
"""
def __init__(
Expand Down
108 changes: 85 additions & 23 deletions docs/howto/use-alternative-secrets-backend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -383,48 +383,75 @@ Note that the secret ``Key`` is ``value``, and secret ``Value`` is ``world`` and

.. _secret_manager_backend:

GCP Secret Manager Backend
^^^^^^^^^^^^^^^^^^^^^^^^^^
Google Cloud Secret Manager Backend
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To enable GCP Secrets Manager to retrieve connection/variables, specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.
This topic describes how to configure Airflow to use `Secret Manager <https://cloud.google.com/secret-manager/docs>`__ as
a secret bakcned and how to manage secrets.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: bakcned


Available parameters to ``backend_kwargs``:
Before you begin
""""""""""""""""

* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections.
* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables.
* ``gcp_key_path``: Path to GCP Credential JSON file
* ``gcp_scopes``: Comma-separated string containing GCP scopes
* ``sep``: separator used to concatenate connections_prefix and conn_id. Default: "-"
`Configure Secret Manager and your local environment <https://cloud.google.com/secret-manager/docs/configuring-secret-manager>`__, once per project.

Note: The full GCP Secrets Manager secret id should follow the pattern "[a-zA-Z0-9-_]".
Enabling the secret backend
"""""""""""""""""""""""""""

Here is a sample configuration if you want to just retrieve connections:
To enable the secret backend for Google Cloud Secrets Manager to retrieve connection/variables,
specify :py:class:`~airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires installing backport-operators. Maybe it's worth to mention that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this is documentation for Airflow 2.0 where these packages don't work. We don't have documentation for Airflow 1.10 with backport packages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport Packages have their own documentation - - and I already have a mechanism to incorpoare some extra information in it - I will extract some of the useful GCP guides ther with the next wave of backport packages.

The documentation is here: https://github.com/apache/airflow/tree/master/airflow/providers/google

And when released it can be found in PyPI https://pypi.org/project/apache-airflow-backport-providers-google/2020.6.24/

as the ``backend`` in ``[secrets]`` section of ``airflow.cfg``.

Here is a sample configuration if you want to use it:

.. code-block:: ini

[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
backend_kwargs = {"connections_prefix": "airflow-connections", "sep": "-"}

Here is a sample configuration if you want to just retrieve variables:
You can also set this with environment variables.

.. code-block:: ini
.. code-block:: bash

[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
backend_kwargs = {"variables_prefix": "airflow-variables", "sep": "-"}
export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

and if you want to retrieve both Variables and connections use the following sample config:
You can verify the correct setting of the configuration options with the ``airflw config get-value`` command.

.. code-block:: bash

$ airflow config get-value secrets backend
airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

Additionals options
"""""""""""""""""""

The next step is to configure additional configuration options using the ``backend_kwargs`` options.

* ``connections_prefix``: Specifies the prefix of the secret to read to get Connections. Default: ``"airflow-connections"``
* ``variables_prefix``: Specifies the prefix of the secret to read to get Variables. Default: ``"airflow-variables"``
* ``gcp_key_path``: Path to GCP Credential JSON file.
* ``gcp_keyfile_dict``: Dictionary of keyfile parameters.
* ``gcp_scopes``: Comma-separated string containing GCP scopes.
* ``sep``: Separator used to concatenate connections_prefix and conn_id. Default: "-"
* ``project_id``: Project ID. If not passed, the project ID from credentials will be used.

All options should be passed as a JSON dictionary.

For example, if you want to set parameter ``connections_prefix`` to ``"airflow-tenant-primary"`` and parameter ``variables_prefix`` to ``"variables_prefix"``, your configuration file should look like this:

.. code-block:: ini

[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
backend_kwargs = {"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "sep": "-"}
backend_kwargs = {"connections_prefix": "airflow-tenant-primary", "variables_prefix": "airflow-tenant-primary"}

Set-up credentials
""""""""""""""""""

You can configure the credentiaps in three ways:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo : credentiaps


When ``gcp_key_path`` is not provided, it will use the Application Default Credentials (ADC) to obtain credentials.
* By default, Application Default Credentials (ADC) is used obtain credentials.
* ``gcp_key_path`` option in ``backend_kwargs`` option - allows you to configure authorizations with a service account stored in local file.
* ``gcp_keyfile_dict`` option in ``backend_kwargs`` option - allows you to configure authorizations with a service account stored in Airflow configuration.

.. note::

Expand All @@ -433,8 +460,43 @@ When ``gcp_key_path`` is not provided, it will use the Application Default Crede
* `google.auth.default <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html#google.auth.default>`__
* `Setting Up Authentication for Server to Server Production Applications <https://cloud.google.com/docs/authentication/production>`__

The value of the Secrets Manager secret id must be the :ref:`connection URI representation <generating_connection_uri>`
of the connection object.
Managing a secrets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove 'a'

""""""""""""""""""

If you want to configure a connection, you need to save it as a :ref:`connection URI representation <generating_connection_uri>`.
Variables should be saved as plain text.

In order to manage secrets, you can use the ``gcloud`` tool or other supported tools. For more information, take a look at:
`Managing secrets <https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets>`__ in Google Cloud Documentation.

The name of the secret must fit the following formats:

* for variable: ``[connections_prefix][sep][variable_name]``
* for connection: ``[variable_prefix][sep][connection_name]``

where:

* ``connections_prefix`` - fixed value defined in the ``connections_prefix`` parameter in backend configuration. Default: ``airflow-connections``.
* ``variable_prefix`` - fixed value defined in the ``variable_prefix`` parameter in backend configuration. Default: ``airflow-variables``.
* ``sep`` - fixed value defined in the ``sep`` parameter in backend configuration. Default: ``-``.

The Cloud Secrets Manager secret name should follow the pattern ``[a-zA-Z0-9-_]``.

If you have the default backend configuration and you want to create a connection with ``conn_id``
equals ``first-connection``, you should create secret named ``airflow-connections-first-connection``.
You can do it with the gcloud tools as in the example below.

.. code-block:: bash

echo "mysql://example.org" | gcloud beta secrets create airflow-connections-first-connection --data-file=-

If you have the default backend configuration and you want to create a variable named ``first-variable``,
you should create a secret named ``airflow-variables-first-variable``. You can do it with the gcloud
command as in the example below.

.. code-block:: bash

echo "content" | gcloud beta secrets create airflow-variables-first-variable --data-file=-

.. _roll_your_own_secrets_backend:

Expand Down