[AIRFLOW-6290] Create guide for GKE operators #8883

tanjinP · 2020-05-16T04:29:11Z

Contains a guide that is appropriately linked to on how to use the various operators in GKE Service.

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Target Github ISSUE in description if exists
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

…te-guide-for-gke-operators

mik-laj · 2020-05-19T02:28:38Z

docs/howto/operator/gcp/kubernetes_engine.rst

+* `GKE API Documentation <https://cloud.google.com/kubernetes-engine/docs/reference/rest>`__
+* `Product Documentation <https://cloud.google.com/kubernetes-engine/docs/>`__
+* `Kubernetes Documentation <https://kubernetes.io/docs/home/>`__
+* `Kubernetes Documentation <https://kubernetes.io/docs/home/>`__


Suggested change

* `Kubernetes Documentation <https://kubernetes.io/docs/home/>`__

mik-laj · 2020-05-19T02:30:49Z

docs/howto/operator/gcp/kubernetes_engine.rst

+
+.. _howto/operator:GKEStartPodOperator:
+
+Run a Pod on a GKE cluster


Can you add link to this guide?
https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html

Think this is a duplicate of this

Specified guide as part of addressing that comment so considering this done.

mik-laj · 2020-05-19T02:45:07Z

docs/howto/operator/gcp/kubernetes_engine.rst

+To run a Kubernetes Pod on a GKE cluster, use
+:class:`~airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator`.
+It is worth noting that this extends
+:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`
+so many of the same arguments can be used. We can enable the usage of :ref:`XCom <concepts:xcom>`
+on the operator:


Suggested change

To run a Kubernetes Pod on a GKE cluster, use

:class:`~airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator`.

It is worth noting that this extends

:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`

so many of the same arguments can be used. We can enable the usage of :ref:`XCom <concepts:xcom>`

on the operator:

You can use two operators to run pod on a GKE cluster:

* :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`

* :class:`~airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator`

``GKEPodOperator`` extends ``KubernetesPodOperator`` to provide authorization using a Google credentials. This way you don't have to manage the `kube_config` file, but it will be generated automatically. All Kubernetes (except `config_file`) parameters are also valid for GKEStartPodOperator.

For more information about starting pods on Airflow, please look at: :ref:`howto/operator:KubernetesPodOperator` guide.

We can enable the usage of :ref:`XCom <concepts:xcom>` on the operator:

WDYT? I would like to show a clear difference between these two operators, because it can be embarrassing. GKEPodOperator is not magic operaotr. It's only wrapper.

This is especially important for me because this information is not in the Cloud Composer documentation, and I often redirect users to this documentation if they want to use KubernetesPodOperator
https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator
This guide is in the Cloud Composer documentation but applies to all users.

Agree - put a version of your suggestion here

mik-laj · 2020-05-19T02:49:20Z

docs/howto/operator/gcp/kubernetes_engine.rst

+
+
+
+Google Kubernetes Engine Operators


Can you add link to this guide in the Kubernetes Pod Operator guide?
https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html

.. note:: If you use `Google Kubernetes Engine <https://cloud.google.com/kubernetes-engine/>`__, use :ref:`GKEStartPodOperator <howto/operator:GKEStartPodOperator>`__ operaator, which simplifies the authorization process.

Follow up commit - had to correct the format

mik-laj · 2020-05-19T02:50:43Z

docs/howto/operator/gcp/kubernetes_engine.rst

+so many of the same arguments can be used. We can enable the usage of :ref:`XCom <concepts:xcom>`
+on the operator:
+
+.. exampleinclude:: ../../../../airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py


Can you write a long description about data transfer via xcom? I am missing information about the /airflow/xcom/return.json file.

…te-guide-for-gke-operators

mik-laj · 2020-05-31T14:36:49Z

@tanjinP It looks good. Should I merge it?

tanjinP · 2020-06-03T02:42:39Z

@mik-laj - I'm good for the merge, thanks for checking in!

mik-laj · 2020-06-03T07:26:30Z

@tanjinP Static checks are sad. Can you fix it?

docs/howto/operator/gcp/kubernetes_engine.rst

mik-laj · 2020-06-03T14:32:22Z

Github Action are sad. Can you fix it?

tanjinP · 2020-06-05T03:30:22Z

Github Action are sad. Can you fix it?

All checks are happy again.

Sorry for the strange number of commits, it looks like I screwed up with the rebasing. However I don't think it matters for now as long as we squash them into master which I believe is always done. In any case it looks good to merge 👍

boring-cyborg · 2020-06-05T05:17:59Z

Awesome work, congrats on your first merged pull request!

mik-laj · 2020-06-05T05:18:21Z

Thanks for the great contributions. What is your next step?

tanjinP · 2020-06-05T17:44:36Z

Thanks for the great contributions. What is your next step?

Thank you for the help! There are still a few other issues/PRs I'm working through so I will continue working on them.

tanjinP added 4 commits May 14, 2020 23:44

scaffolding (adding file, reference added, update test)

9140d94

removing duplicate Google

8fd3060

Merge remote-tracking branch 'upstream/master' into airflow-6290-crea…

5272b2f

…te-guide-for-gke-operators

first pass at guide

a306037

boring-cyborg bot added area:docs provider:google Google (including GCP) related issues labels May 16, 2020

newline

85a2fa2

mik-laj reviewed May 19, 2020

View reviewed changes

tanjinP added 7 commits May 19, 2020 02:43

removing redundant reference

70c5ac9

Merge remote-tracking branch 'upstream/master' into airflow-6290-crea…

a31e207

…te-guide-for-gke-operators

clarifying distinction between gkestartpod and vanilla k8spod

3ae57e2

pointing to gkestart from k8spod guide

3fe1780

Merge remote-tracking branch 'upstream/master' into airflow-6290-crea…

8eca975

…te-guide-for-gke-operators

correcting autoapi complaint

ded5cca

thorough description of xcom

025aa4c

tanjinP requested a review from mik-laj May 20, 2020 04:38

tanjinP mentioned this pull request May 23, 2020

Improve guide for KubernetesPodOperator #8970

Closed

12 tasks

tanjinP added 9 commits May 30, 2020 21:28

scaffolding (adding file, reference added, update test)

c9ff861

removing duplicate Google

ab73fe4

first pass at guide

788c781

newline

95c088e

removing redundant reference

b9fe7ed

clarifying distinction between gkestartpod and vanilla k8spod

21ba8c6

pointing to gkestart from k8spod guide

a7e6b55

correcting autoapi complaint

fc39b91

thorough description of xcom

9b7ceb8

tanjinP force-pushed the airflow-6290-create-guide-for-gke-operators branch from 025aa4c to 9b7ceb8 Compare May 31, 2020 01:29

mik-laj approved these changes May 31, 2020

View reviewed changes

mik-laj reviewed Jun 3, 2020

View reviewed changes

docs/howto/operator/gcp/kubernetes_engine.rst Outdated Show resolved Hide resolved

removing reference to missing guide (as it is now available)

7f26469

tanjinP force-pushed the airflow-6290-create-guide-for-gke-operators branch from 1526ead to 7f26469 Compare June 5, 2020 02:30

tanjinP added 9 commits June 4, 2020 22:47

adding backticks

74de508

first pass at guide

fbcd507

newline

9b06fcb

removing redundant reference

2c26d5c

clarifying distinction between gkestartpod and vanilla k8spod

c85a1ad

pointing to gkestart from k8spod guide

3f4918e

correcting autoapi complaint

4cf99a5

thorough description of xcom

00946c0

adding backticks

7d1dce7

mik-laj approved these changes Jun 5, 2020

View reviewed changes

mik-laj merged commit f56811d into apache:master Jun 5, 2020

tanjinP deleted the airflow-6290-create-guide-for-gke-operators branch June 5, 2020 17:44


		.. _howto/operator:GKEStartPodOperator:

		Run a Pod on a GKE cluster

[AIRFLOW-6290] Create guide for GKE operators #8883

[AIRFLOW-6290] Create guide for GKE operators #8883

Uh oh!

Conversation

tanjinP commented May 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mik-laj commented May 31, 2020

Uh oh!

tanjinP commented Jun 3, 2020

Uh oh!

mik-laj commented Jun 3, 2020

Uh oh!

Uh oh!

mik-laj commented Jun 3, 2020

Uh oh!

tanjinP commented Jun 5, 2020

Uh oh!

boring-cyborg bot commented Jun 5, 2020

Uh oh!

mik-laj commented Jun 5, 2020

Uh oh!

tanjinP commented Jun 5, 2020

Uh oh!

Uh oh!

tanjinP commented May 16, 2020 •

edited

Loading