Separate out documentation building per provider (#12444)

* POC * fixup! POC
apache · Nov 20, 2020 · c34ef85 · c34ef85
1 parent c3cf695
commit c34ef85
Show file tree

Hide file tree

Showing 259 changed files with 4,517 additions and 1,953 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -314,10 +314,24 @@ jobs:
  run: ./scripts/ci/docs/ci_docs.sh --docs-only
  - name: "Upload documentation"
  uses: actions/upload-artifact@v2
- if: always()
+ if: always() && github.event_name == 'pull_request'
  with:
  name: airflow-documentation
  path: "./files/documentation"
+ - name: Configure AWS credentials
+ uses: aws-actions/configure-aws-credentials@v1
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
+ github.event_name == 'push'
+ with:
+ aws-access-key-id: ${{ secrets.DOCS_AWS_ACCESS_KEY_ID }}
+ aws-secret-access-key: ${{ secrets.DOCS_AWS_SECRET_ACCESS_KEY }}
+ aws-region: eu-central-1
+ - name: "Upload documentation to AWS S3"
+ if: >
+ github.ref == 'refs/heads/master' && github.repository == 'apache/airflow' &&
+ github.event_name == 'push'
+ run: aws s3 sync ./files/documentation s3://apache-airflow-docs
 
  docs-spell-check:
  timeout-minutes: 30

diff --git a/.gitignore b/.gitignore
@@ -88,6 +88,8 @@ instance/
 # Sphinx documentation
 docs/_build/
 docs/_api/
+docs/*/_api/
+docs/_doctrees
 
 # PyBuilder
 target/

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -457,6 +457,7 @@ repos:
  entry: ./scripts/ci/pre_commit/pre_commit_check_provider_yaml_files.py
  language: python
  require_serial: true
+ files: provider.yaml$
  additional_dependencies: ['PyYAML==5.3.1', 'jsonschema==3.2.0', 'tabulate==0.8.7']
  - id: mermaid
  name: Generate mermaid images

diff --git a/CI.rst b/CI.rst
@@ -695,9 +695,22 @@ We also have a script that can help to clean-up the old artifacts:
 CodeQL scan
 -----------
 
-The CodeQL security scan uses GitHub security scan framework to scan our code for security violations.
+The `CodeQL <https://securitylab.github.com/tools/codeql>`_ security scan uses GitHub security scan framework to scan our code for security violations.
 It is run for JavaScript and python code.
 
+Publishing documentation
+------------------------
+
+Documentation from the ``master`` branch is automatically published on Amazon S3.
+
+To make this possible, Github Action has secrets set up with credentials
+for an Amazon Web Service account - ``DOCS_AWS_ACCESS_KEY_ID`` and ``DOCS_AWS_SECRET_ACCESS_KEY``.
+
+This account has permission to write/list/put objects to bucket ``apache-airflow-docs``. This bucket has public access configured, which means it is accessible through the website endpoint. For more information, see: `Hosting a static website on Amazon S3
+ <https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html>`_
+
+Website endpoint: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/
+
 Naming conventions for stored images
 ====================================
 

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -250,7 +250,7 @@ Step 4: Prepare PR
 
  For example, to address this example issue, do the following:
 
- * Read about `email configuration in Airflow <https://airflow.readthedocs.io/en/latest/howto/email-config.html>`__.
+ * Read about `email configuration in Airflow </docs/howto/email-config.rst>`__.
 
  * Find the class you should modify. For the example GitHub issue,
  this is `email.py <https://github.com/apache/airflow/blob/master/airflow/utils/email.py>`__.
@@ -713,47 +713,9 @@ jobs for each python version.
 Documentation
 =============
 
-The latest API documentation (for the master branch) is usually available
-`here <https://airflow.readthedocs.io/en/latest/>`__.
+Documentation for ``apache-airflow`` package and other packages that are closely related to it ie. providers packages are in ``/docs/`` directory. For detailed information on documentation development, see: `docs/README.md <docs/README.md>`_
 
-To generate a local version you can use `<BREEZE.rst>`_.
-
-The documentation build consists of verifying consistency of documentation and two steps:
-
-* spell checking
-* building documentation
-
-You can only run one of the steps via ``--spellcheck-only`` or ``--docs-only``.
-
-.. code-block:: bash
-
- ./breeze build-docs
-
-or just to run spell-check
-
-.. code-block:: bash
-
- ./breeze build-docs -- --spellcheck-only
-
-or just to run documentation building
-
-.. code-block:: bash
-
- ./breeze build-docs -- --docs-only
-
-Also documentation is available as downloadable artifact in GitHub Actions after the CI builds your PR.
-
-**Known issues:**
-
-If you are creating a new directory for new integration in the ``airflow.providers`` package,
-you should also update the ``docs/autoapi_templates/index.rst`` file.
-
-If you are creating new ``hooks``, ``sensors``, ``operators`` directory in
-the ``airflow.providers`` package, you should also update
-the ``docs/operators-and-hooks-ref.rst`` file.
-
-If you are creating ``example_dags`` directory, you need to create ``example_dags/__init__.py`` with Apache
-license or copy another ``__init__.py`` file that contains the necessary license.
+For Helm Chart documentation, see: `/chart/README.md <../chart/READMe.md>`__
 
 Static code checks
 ==================

diff --git a/README.md b/README.md
@@ -22,7 +22,6 @@
 [![PyPI version](https://badge.fury.io/py/apache-airflow.svg)](https://badge.fury.io/py/apache-airflow)
 [![GitHub Build](https://github.com/apache/airflow/workflows/CI%20Build/badge.svg)](https://github.com/apache/airflow/actions)
 [![Coverage Status](https://img.shields.io/codecov/c/github/apache/airflow/master.svg)](https://codecov.io/github/apache/airflow?branch=master)
-[![Documentation Status](https://readthedocs.org/projects/airflow/badge/?version=latest)](https://airflow.readthedocs.io/en/latest/?badge=latest)
 [![License](http://img.shields.io/:license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/apache-airflow.svg)](https://pypi.org/project/apache-airflow/)
 [![Docker Pulls](https://img.shields.io/docker/pulls/apache/airflow.svg)](https://hub.docker.com/r/apache/airflow)
@@ -135,7 +134,7 @@ pip install apache-airflow[postgres,google]==1.10.12 \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
 ```
 
-For information on installing backport providers check https://airflow.readthedocs.io/en/latest/backport-providers.html.
+For information on installing backport providers check [/docs/backport-providers.rst][/docs/backport-providers.rst].
 
 ## Official source code
 

diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
@@ -171,7 +171,7 @@ info:
  The default is to deny all requests.
 
  For details on configuring the authentication, see
- [API Authorization](https://airflow.readthedocs.io/en/latest/security/api.html).
+ [API Authorization](https://airflow.apache.org/docs/stable/security/api.html).
 
  # Errors
 
@@ -1880,7 +1880,7 @@ components:
  DAG details.
 
  For details see:
- (airflow.models.DAG)[https://airflow.readthedocs.io/en/stable/_api/airflow/models/index.html#airflow.models.DAG]
+ (airflow.models.DAG)[https://airflow.apache.org/docs/stable/_api/airflow/models/index.html#airflow.models.DAG]
  allOf:
  - $ref: '#/components/schemas/DAG'
  - type: object

diff --git a/airflow/provider.yaml.schema.json b/airflow/provider.yaml.schema.json
@@ -6,6 +6,10 @@
  "description": "Package name available under which the package is available in the PyPI repository.",
  "type": "string"
  },
+ "name": {
+ "description": "Provider name",
+ "type": "string"
+ },
  "description": {
  "description": "Information about the package in RST format",
  "type": "string"
@@ -167,6 +171,7 @@
  },
  "additionalProperties": false,
  "required": [
+ "name",
  "package-name",
  "description",
  "versions"

diff --git a/airflow/providers/amazon/aws/hooks/base_aws.py b/airflow/providers/amazon/aws/hooks/base_aws.py
@@ -21,7 +21,7 @@
 
 .. seealso::
  For more information on how to use this hook, take a look at the guide:
- :ref:`howto/connection:AWSHook`
+ :ref:`apache-airflow:howto/connection:AWSHook`
 """
 
 import configparser

diff --git a/airflow/providers/amazon/aws/operators/datasync.py b/airflow/providers/amazon/aws/operators/datasync.py
@@ -36,7 +36,7 @@ class AWSDataSyncOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:AWSDataSyncOperator`
+ :ref:`apache-airflow:howto/operator:AWSDataSyncOperator`
 
  .. note:: There may be 0, 1, or many existing DataSync Tasks defined in your AWS
  environment. The default behavior is to create a new Task if there are 0, or

diff --git a/airflow/providers/amazon/aws/operators/ecs.py b/airflow/providers/amazon/aws/operators/ecs.py
@@ -76,7 +76,7 @@ class ECSOperator(BaseOperator): # pylint: disable=too-many-instance-attributes
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:ECSOperator`
+ :ref:`apache-airflow:howto/operator:ECSOperator`
 
  :param task_definition: the task definition name on Elastic Container Service
  :type task_definition: str

diff --git a/airflow/providers/amazon/aws/operators/glacier.py b/airflow/providers/amazon/aws/operators/glacier.py
@@ -26,7 +26,7 @@ class GlacierCreateJobOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:GlacierCreateJobOperator`
+ :ref:`apache-airflow:howto/operator:GlacierCreateJobOperator`
 
  :param aws_conn_id: The reference to the AWS connection details
  :type aws_conn_id: str

diff --git a/airflow/providers/amazon/aws/sensors/glacier.py b/airflow/providers/amazon/aws/sensors/glacier.py
@@ -35,6 +35,10 @@ class GlacierJobOperationSensor(BaseSensorOperator):
  """
  Glacier sensor for checking job state. This operator runs only in reschedule mode.
 
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`apache-airflow:howto/operator:GlacierJobOperationSensor`
+
  :param aws_conn_id: The reference to the AWS connection details
  :type aws_conn_id: str
  :param vault_name: name of Glacier vault on which job is executed

diff --git a/airflow/providers/amazon/aws/transfers/glacier_to_gcs.py b/airflow/providers/amazon/aws/transfers/glacier_to_gcs.py
@@ -34,7 +34,7 @@ class GlacierToGCSOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:GlacierToGCSOperator`
+ :ref:`apache-airflow:howto/operator:GlacierToGCSOperator`
 
  :param aws_conn_id: The reference to the AWS connection details
  :type aws_conn_id: str

diff --git a/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py b/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py
@@ -28,7 +28,7 @@ class ImapAttachmentToS3Operator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:ImapAttachmentToS3Operator`
+ :ref:`apache-airflow:howto/operator:ImapAttachmentToS3Operator`
 
  :param imap_attachment_name: The file name of the mail attachment that you want to transfer.
  :type imap_attachment_name: str

diff --git a/airflow/providers/amazon/aws/transfers/s3_to_redshift.py b/airflow/providers/amazon/aws/transfers/s3_to_redshift.py
@@ -29,7 +29,7 @@ class S3ToRedshiftOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:S3ToRedshiftOperator`
+ :ref:`apache-airflow:howto/operator:S3ToRedshiftOperator`
 
  :param schema: reference to a specific schema in redshift database
  :type schema: str

diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-amazon
+name: Amazon
 description: |
  Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 

diff --git a/airflow/providers/apache/cassandra/provider.yaml b/airflow/providers/apache/cassandra/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-cassandra
+name: Apache Cassandra
 description: |
  `Apache Cassandra <http://cassandra.apache.org/>`__.
 

diff --git a/airflow/providers/apache/cassandra/sensors/record.py b/airflow/providers/apache/cassandra/sensors/record.py
@@ -33,7 +33,7 @@ class CassandraRecordSensor(BaseSensorOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:CassandraRecordSensor`
+ :ref:`apache-airflow:howto/operator:CassandraRecordSensor`
 
  For example, if you want to wait for a record that has values 'v1' and 'v2' for each
  primary keys 'p1' and 'p2' to be populated in keyspace 'k' and table 't',

diff --git a/airflow/providers/apache/cassandra/sensors/table.py b/airflow/providers/apache/cassandra/sensors/table.py
@@ -34,7 +34,7 @@ class CassandraTableSensor(BaseSensorOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:CassandraTableSensor`
+ :ref:`apache-airflow:howto/operator:CassandraTableSensor`
 
 
  For example, if you want to wait for a table called 't' to be created

diff --git a/airflow/providers/apache/druid/provider.yaml b/airflow/providers/apache/druid/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-druid
+name: Apache Druid
 description: |
  `Apache Druid <https://druid.apache.org/>`__.
 

diff --git a/airflow/providers/apache/hdfs/provider.yaml b/airflow/providers/apache/hdfs/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-hdfs
+name: Apache HDFS
 description: |
  `Hadoop Distributed File System (HDFS) <https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html>`__
  and `WebHDFS <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html>`__.

diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-hive
+name: Apache Hive
 description: |
  `Apache Hive <https://hive.apache.org/>`__
 

diff --git a/airflow/providers/apache/kylin/provider.yaml b/airflow/providers/apache/kylin/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-kylin
+name: Apache Hive
 description: |
  `Apache Kylin <https://kylin.apache.org/>`__
 

diff --git a/airflow/providers/apache/livy/provider.yaml b/airflow/providers/apache/livy/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-livy
+name: Apache Livy
 description: |
  `Apache Livy <https://livy.apache.org/>`__
 

diff --git a/airflow/providers/apache/pig/provider.yaml b/airflow/providers/apache/pig/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-pig
+name: Apache Pig
 description: |
  `Apache Pig <https://pig.apache.org/>`__
 

diff --git a/airflow/providers/apache/pinot/provider.yaml b/airflow/providers/apache/pinot/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-pinot
+name: Apache Pinot
 description: |
  `Apache Pinot <https://pinot.apache.org/>`__
 

diff --git a/airflow/providers/apache/spark/operators/spark_jdbc.py b/airflow/providers/apache/spark/operators/spark_jdbc.py
@@ -33,7 +33,7 @@ class SparkJDBCOperator(SparkSubmitOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:SparkJDBCOperator`
+ :ref:`apache-airflow:howto/operator:SparkJDBCOperator`
 
  :param spark_app_name: Name of the job (default airflow-spark-jdbc)
  :type spark_app_name: str

diff --git a/airflow/providers/apache/spark/operators/spark_sql.py b/airflow/providers/apache/spark/operators/spark_sql.py
@@ -29,7 +29,7 @@ class SparkSqlOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:SparkSqlOperator`
+ :ref:`apache-airflow:howto/operator:SparkSqlOperator`
 
  :param sql: The SQL query to execute. (templated)
  :type sql: str

diff --git a/airflow/providers/apache/spark/operators/spark_submit.py b/airflow/providers/apache/spark/operators/spark_submit.py
@@ -33,7 +33,7 @@ class SparkSubmitOperator(BaseOperator):
 
  .. seealso::
  For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:SparkSubmitOperator`
+ :ref:`apache-airflow:howto/operator:SparkSubmitOperator`
 
  :param application: The application that submitted as a job, either jar or py file. (templated)
  :type application: str

diff --git a/airflow/providers/apache/spark/provider.yaml b/airflow/providers/apache/spark/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-spark
+name: Apache Spark
 description: |
  `Apache Spark <https://spark.apache.org/>`__
 

diff --git a/airflow/providers/apache/sqoop/provider.yaml b/airflow/providers/apache/sqoop/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-apache-sqoop
+name: Apache Sqoop
 description: |
  `Apache Sqoop <https://sqoop.apache.org/>`__
 

diff --git a/airflow/providers/celery/provider.yaml b/airflow/providers/celery/provider.yaml
@@ -17,6 +17,7 @@
 
 ---
 package-name: apache-airflow-providers-celery
+name: Celery
 description: |
  `Celery <http://www.celeryproject.org/>`__