Skip to content

Commit

Permalink
Migrate Google example natural_language to new design AIP-47 (#25262)
Browse files Browse the repository at this point in the history 
related: #22447, #22430
  • Loading branch information
@chenglongyan
chenglongyan committed Jul 26, 2022
1 parent 7438707 commit 10c9a36
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 32 deletions.
20 changes: 10 additions & 10 deletions docs/apache-airflow-providers-google/operators/cloud/natural_language.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,14 @@ representing text.

Here is an example of document with text provided as a string:

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:start-after: [START howto_operator_gcp_natural_language_document_text]
:end-before: [END howto_operator_gcp_natural_language_document_text]

In addition to supplying string, a document can refer to content stored in Google Cloud Storage.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:start-after: [START howto_operator_gcp_natural_language_document_gcs]
:end-before: [END howto_operator_gcp_natural_language_document_gcs]
Expand All @@ -66,7 +66,7 @@ public figures, landmarks, etc.), and returns information about those entities.
Entity analysis is performed with the
:class:`~airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitiesOperator` operator.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_entities]
Expand All @@ -77,7 +77,7 @@ You can use :ref:`Jinja templating <concepts:jinja-templating>` with
parameters which allows you to dynamically determine values. The result is saved to :ref:`XCom <concepts:xcom>`, which allows it
to be used by other operators.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_entities_result]
Expand All @@ -94,7 +94,7 @@ as positive, negative, or neutral. Sentiment analysis is performed through
the :class:`~airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitySentimentOperator`
operator.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_entity_sentiment]
Expand All @@ -105,7 +105,7 @@ You can use :ref:`Jinja templating <concepts:jinja-templating>` with
parameters which allows you to dynamically determine values. The result is saved to :ref:`XCom <concepts:xcom>`, which allows it
to be used by other operators.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_entity_sentiment_result]
Expand All @@ -123,7 +123,7 @@ through the
:class:`~airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeSentimentOperator`
operator.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_sentiment]
Expand All @@ -134,7 +134,7 @@ You can use :ref:`Jinja templating <concepts:jinja-templating>` with
parameters which allows you to dynamically determine values. The result is saved to :ref:`XCom <concepts:xcom>`, which allows it
to be used by other operators.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_sentiment_result]
Expand All @@ -151,7 +151,7 @@ content in a document, use the
:class:`~airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageClassifyTextOperator`
operator.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_classify_text]
Expand All @@ -162,7 +162,7 @@ You can use :ref:`Jinja templating <concepts:jinja-templating>` with
parameters which allows you to dynamically determine values. The result is saved to :ref:`XCom <concepts:xcom>`, which allows it
to be used by other operators.

.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_natural_language.py
.. exampleinclude:: /../../tests/system/providers/google/cloud/natural_language/example_natural_language.py
:language: python
:dedent: 4
:start-after: [START howto_operator_gcp_natural_language_analyze_classify_text_result]
Expand Down
19 changes: 0 additions & 19 deletions ...operators/test_natural_language_system.py → ...google/cloud/natural_language/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand All @@ -15,21 +14,3 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import pytest

from tests.providers.google.cloud.utils.gcp_authenticator import GCP_AI_KEY
from tests.test_utils.gcp_system_helpers import CLOUD_DAG_FOLDER, GoogleSystemTest, provide_gcp_context


@pytest.mark.backend("mysql", "postgres")
@pytest.mark.credential_file(GCP_AI_KEY)
class CloudNaturalLanguageExampleDagsTest(GoogleSystemTest):
def setUp(self):
super().setUp()

@provide_gcp_context(GCP_AI_KEY)
def test_run_example_dag(self):
self.run_dag('example_gcp_natural_language', CLOUD_DAG_FOLDER)

def tearDown(self):
super().tearDown()
14 changes: 11 additions & 3 deletions .../example_dags/example_natural_language.py → ...ural_language/example_natural_language.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
"""
Example Airflow DAG for Google Cloud Natural Language service
"""

import os
from datetime import datetime

from google.cloud.language_v1.proto.language_service_pb2 import Document
Expand All @@ -33,6 +33,9 @@
CloudNaturalLanguageClassifyTextOperator,
)

ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
DAG_ID = "example_gcp_natural_language"

# [START howto_operator_gcp_natural_language_document_text]
TEXT = """Airflow is a platform to programmatically author, schedule and monitor workflows.
Expand All @@ -51,12 +54,12 @@


with models.DAG(
"example_gcp_natural_language",
DAG_ID,
schedule_interval='@once', # Override to match your needs
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['example'],
) as dag:

# [START howto_operator_gcp_natural_language_analyze_entities]
analyze_entities = CloudNaturalLanguageAnalyzeEntitiesOperator(
document=document, task_id="analyze_entities"
Expand Down Expand Up @@ -113,3 +116,8 @@
analyze_entity_sentiment >> analyze_entity_sentiment_result
analyze_sentiment >> analyze_sentiment_result
analyze_classify_text >> analyze_classify_text_result

from tests.system.utils import get_test_run # noqa: E402

# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
test_run = get_test_run(dag)

0 comments on commit 10c9a36

Please sign in to comment.