Speed up autocompletion of Breeze by simplifying provider state (#36499)

Some recent changes, adding removed and suspended state for breeze caused significant slow-down of autocompletion retrieval - as it turned out, because we loaded and parsed all provider yaml files during auto-completion - in order to determine list of providers available for some commands. We already planned to replace the several states (suspended, not-ready, removed) with a single state field - by doing it and addding the field to pre-commit generated "provider_dependencies.json" we could switch to parsing the single provider_dependencies.json file and retrieve provider list from there following the state stored in that json file. This also simplifies state management following the recently added state diagram by following the same state lifecycle: "not-ready" -> "ready" -> "suspended" -> "removed"
apache · Dec 30, 2023 · 6937ae7 · 6937ae7
1 parent 72f43fc
commit 6937ae7
Show file tree

Hide file tree

Showing 101 changed files with 362 additions and 302 deletions.
diff --git a/PROVIDERS.rst b/PROVIDERS.rst
@@ -254,7 +254,7 @@ process can be suspended.
 
 This means:
 
-* The provider's status is set to "suspended"
+* The provider's state in ``provider.yaml`` is set to "suspended"
 * No new releases of the provider will be made until the problem with dependencies is solved
 * Sources of the provider remain in the repository for now (in the future we might add process to remove them)
 * No new changes will be accepted for the provider (other than the ones that fix the dependencies)

diff --git a/airflow/provider.yaml.schema.json b/airflow/provider.yaml.schema.json
@@ -21,17 +21,15 @@
  "type": "string"
  }
  },
- "suspended": {
- "description": "If set to true, the provider is suspended and it's not a candidate for release nor contributes dependencies to constraint calculations/CI image. Tests are excluded.",
- "type:": "boolean"
- },
- "removed": {
- "description": "If set to true, the provider is also removed and will be soon removed from the code",
- "type:": "boolean"
- },
- "not-ready": {
- "description": "If set to true, the provider is not included by default in release commands - for example when provider release or documentation is being prepared (not-ready providers are enabled in CI by default)",
- "type:": "boolean"
+ "state": {
+ "description": "State of provider: might be not-ready, regular, suspended, removed.",
+ "type:": "string",
+ "enum": [
+ "not-ready",
+ "ready",
+ "suspended",
+ "removed"
+ ]
  },
  "dependencies": {
  "description": "Dependencies that should be added to the provider",
@@ -462,7 +460,7 @@
  "name",
  "package-name",
  "description",
- "suspended",
+ "state",
  "source-date-epoch",
  "dependencies",
  "versions"

diff --git a/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst b/airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
@@ -55,59 +55,59 @@ If you still have doubts about building your provider, we recommend that you rea
 open a issue on GitHub so the community can help you.
 
 The folders are optional: example_dags, hooks, links, logs, notifications, operators, secrets, sensors, transfers,
-triggers, waiters (and the list changes continuously).
+triggers (and the list changes continuously).
 
  .. code-block:: bash
 
  airflow/
- ├── providers/<NEW_PROVIDER>/
- │ ├── __init__.py
- │ ├── example_dags/
- │ │ ├── __init__.py
- │ │ └── example_<NEW_PROVIDER>.py
- │ ├── executors/
- │ │ ├── __init__.py
- │ │ └── <NEW_PROVIDER>.py
- │ ├── hooks/
- │ │ ├── __init__.py
- │ │ └── <NEW_PROVIDER>.py
- │ ├── operators/
- │ │ ├── __init__.py
- │ │ └── <NEW_PROVIDER>.py
- ....
- │ ├── transfers/
- │ │ ├── __init__.py
- │ │ └── <NEW_PROVIDER>.py
- │ └── triggers/
- │ ├── __init__.py
- │ └── <NEW_PROVIDER>.py
- └── tests/providers/<NEW_PROVIDER>/
- ├── __init__.py
- ├── executors/
- │ ├── __init__.py
- │ └── test_<NEW_PROVIDER>.py
- ├── hooks/
- │ ├── __init__.py
- │ └── test_<NEW_PROVIDER>.py
- ├── operators/
- │ ├── __init__.py
- │ ├── test_<NEW_PROVIDER>.py
- │ └── test_<NEW_PROVIDER>_system.py
- ...
- ├── transfers/
- │ ├── __init__.py
- │ └── test_<NEW_PROVIDER>.py
- └── triggers/
- ├── __init__.py
- └── test_<NEW_PROVIDER>.py
+  ├── providers/<NEW_PROVIDER>/
+  │  ├── __init__.py
+  │  ├── executors/
+  │  │ ├── __init__.py
+  │  │ └── *.py
+  │  ├── hooks/
+  │  │ ├── __init__.py
+  │  │ └── *.py
+  │  ├── operators/
+  │  │ ├── __init__.py
+  │  │ └── *.py
+  │  ├── transfers/
+  │  │ ├── __init__.py
+  │  │ └── *.py
+  │ └── triggers/
+  │  ├── __init__.py
+  │  └── *.py
+  └── tests
+  ├── providers/<NEW_PROVIDER>/
+  │  ├── __init__.py
+  │  ├── executors/
+  │ │ ├── __init__.py
+  │ │ └── test_*.py
+  │ ├── hooks/
+  │ │ ├── __init__.py
+  │  │ └── test_*>.py
+  │ ├── operators/
+  │ │ ├── __init__.py
+  │  │ ├── test_*.py
+  │ ...
+  │  ├── transfers/
+  │  │ ├── __init__.py
+  │  │ └── test_*.py
+  │ └── triggers/
+  │ ├── __init__.py
+  │  └── test_*.py
+  └── system/providers/<NEW_PROVIDER>/
+  ├── __init__.py
+  └── example_*.py
+
 
 Considering that you have already transferred your provider's code to the above structure, it will now be necessary
 to create unit tests for each component you created. The example below I have already set up an environment using
 breeze and I'll run unit tests for my Hook.
 
  .. code-block:: bash
 
- root@fafd8d630e46:/opt/airflow# python -m pytest tests/providers/<NEW_PROVIDER>/hook/<NEW_PROVIDER>.py
+ root@fafd8d630e46:/opt/airflow# python -m pytest tests/providers/<NEW_PROVIDER>/hook/test_*.py
 
 Adding chicken-egg providers
 ----------------------------
@@ -467,7 +467,7 @@ As of April 2023, we have the possibility to suspend individual providers, so th
 back dependencies for Airflow and other providers. The process of suspending providers is described
 in `description of the process <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers>`_
 
-Technically, suspending a provider is done by setting ``suspended : true``, in the provider.yaml of the
+Technically, suspending a provider is done by setting ``state: suspended``, in the provider.yaml of the
 provider. This should be followed by committing the change and either automatically or manually running
 pre-commit checks that will either update derived configuration files or ask you to update them manually.
 Note that you might need to run pre-commit several times until all the static checks pass,
@@ -509,10 +509,10 @@ Removing providers
 
 When removing providers from Airflow code, we need to make one last release where we mark the provider as
 removed - in documentation and in description of the PyPI package. In order to that release manager has to
-add "removed: true" flag in the provider yaml file and include the provider in the next wave of the
+add "state: removed" flag in the provider yaml file and include the provider in the next wave of the
 providers (and then remove all the code and documentation related to the provider).
 
-The "removed: true" flag will cause the provider to be available for the following commands (note that such
+The "removed: removed" flag will cause the provider to be available for the following commands (note that such
 provider has to be explicitly added as selected to the package - such provider will not be included in
 the available list of providers or when documentation is built unless --include-removed-providers
 flag is used):

diff --git a/airflow/providers/airbyte/provider.yaml b/airflow/providers/airbyte/provider.yaml
@@ -21,7 +21,7 @@ name: Airbyte
 description: |
  `Airbyte <https://airbyte.com/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288099
 versions:
  - 3.5.1

diff --git a/airflow/providers/alibaba/provider.yaml b/airflow/providers/alibaba/provider.yaml
@@ -21,7 +21,7 @@ name: Alibaba
 description: |
  Alibaba Cloud integration (including `Alibaba Cloud <https://www.alibabacloud.com/>`__).
 
-suspended: false
+state: ready
 source-date-epoch: 1703747466
 versions:
  - 2.7.1

diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
@@ -21,7 +21,7 @@ name: Amazon
 description: |
  Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 
-suspended: false
+state: ready
 source-date-epoch: 1703747484
 versions:
  - 8.15.0

diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Beam
 description: |
  `Apache Beam <https://beam.apache.org/>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703288103
 versions:
  - 5.5.0

diff --git a/airflow/providers/apache/cassandra/provider.yaml b/airflow/providers/apache/cassandra/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Cassandra
 description: |
  `Apache Cassandra <https://cassandra.apache.org/>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703288104
 versions:
  - 3.4.1

diff --git a/airflow/providers/apache/drill/provider.yaml b/airflow/providers/apache/drill/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Drill
 description: |
  `Apache Drill <https://drill.apache.org/>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703288105
 versions:
  - 2.6.0

diff --git a/airflow/providers/apache/druid/provider.yaml b/airflow/providers/apache/druid/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Druid
 description: |
  `Apache Druid <https://druid.apache.org/>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703288106
 versions:
  - 3.7.0

diff --git a/airflow/providers/apache/flink/provider.yaml b/airflow/providers/apache/flink/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Flink
 description: |
  `Apache Flink <https://flink.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288107
 versions:
  - 1.3.0

diff --git a/airflow/providers/apache/hdfs/provider.yaml b/airflow/providers/apache/hdfs/provider.yaml
@@ -22,7 +22,7 @@ description: |
  `Hadoop Distributed File System (HDFS) <https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html>`__
  and `WebHDFS <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703747531
 versions:
  - 4.3.2

diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Hive
 description: |
  `Apache Hive <https://hive.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288109
 versions:
  - 6.4.0

diff --git a/airflow/providers/apache/impala/provider.yaml b/airflow/providers/apache/impala/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Impala
 description: |
  `Apache Impala <https://impala.apache.org/>`__.
 
-suspended: false
+state: ready
 source-date-epoch: 1703288110
 versions:
  - 1.3.0

diff --git a/airflow/providers/apache/kafka/provider.yaml b/airflow/providers/apache/kafka/provider.yaml
@@ -19,7 +19,7 @@
 package-name: apache-airflow-providers-apache-kafka
 name: Apache Kafka
 
-suspended: false
+state: ready
 source-date-epoch: 1703288110
 description: |
  `Apache Kafka <https://kafka.apache.org/>`__

diff --git a/airflow/providers/apache/kylin/provider.yaml b/airflow/providers/apache/kylin/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Kylin
 description: |
  `Apache Kylin <https://kylin.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703747544
 versions:
  - 3.5.0

diff --git a/airflow/providers/apache/livy/provider.yaml b/airflow/providers/apache/livy/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Livy
 description: |
  `Apache Livy <https://livy.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288112
 versions:
  - 3.7.0

diff --git a/airflow/providers/apache/pig/provider.yaml b/airflow/providers/apache/pig/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Pig
 description: |
  `Apache Pig <https://pig.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288113
 versions:
  - 4.3.0

diff --git a/airflow/providers/apache/pinot/provider.yaml b/airflow/providers/apache/pinot/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Pinot
 description: |
  `Apache Pinot <https://pinot.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288114
 versions:
  - 4.3.0

diff --git a/airflow/providers/apache/spark/provider.yaml b/airflow/providers/apache/spark/provider.yaml
@@ -21,7 +21,7 @@ name: Apache Spark
 description: |
  `Apache Spark <https://spark.apache.org/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288115
 versions:
  - 4.6.0

diff --git a/airflow/providers/apprise/provider.yaml b/airflow/providers/apprise/provider.yaml
@@ -23,7 +23,7 @@ name: Apprise
 description: |
  `Apprise <https://github.com/caronc/apprise>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288116
 
 versions:

diff --git a/airflow/providers/arangodb/provider.yaml b/airflow/providers/arangodb/provider.yaml
@@ -25,7 +25,7 @@ dependencies:
  - apache-airflow>=2.6.0
  - python-arango>=7.3.2
 
-suspended: false
+state: ready
 source-date-epoch: 1703288117
 versions:
  - 2.4.1

diff --git a/airflow/providers/asana/provider.yaml b/airflow/providers/asana/provider.yaml
@@ -21,7 +21,7 @@ name: Asana
 description: |
  `Asana <https://asana.com/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288118
 versions:
  - 2.4.1

diff --git a/airflow/providers/atlassian/jira/provider.yaml b/airflow/providers/atlassian/jira/provider.yaml
@@ -21,7 +21,7 @@ name: Atlassian Jira
 description: |
  `Atlassian Jira <https://www.atlassian.com/software/jira/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288118
 versions:
  - 2.4.0

diff --git a/airflow/providers/celery/provider.yaml b/airflow/providers/celery/provider.yaml
@@ -21,7 +21,7 @@ name: Celery
 description: |
  `Celery <https://docs.celeryq.dev/en/stable/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288119
 versions:
  - 3.5.1

diff --git a/airflow/providers/cloudant/provider.yaml b/airflow/providers/cloudant/provider.yaml
@@ -21,7 +21,7 @@ name: IBM Cloudant
 description: |
  `IBM Cloudant <https://www.ibm.com/cloud/cloudant>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288120
 versions:
  - 3.4.1

diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -21,7 +21,7 @@ name: Kubernetes
 description: |
  `Kubernetes <https://kubernetes.io/>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703747586
 versions:
  - 7.13.0

diff --git a/airflow/providers/cohere/provider.yaml b/airflow/providers/cohere/provider.yaml
@@ -23,7 +23,7 @@ name: Cohere
 description: |
  `Cohere <https://docs.cohere.com/docs>`__
 
-suspended: false
+state: ready
 source-date-epoch: 1703288123
 
 versions: