Prepare for follow-up relase for November providers (#27774)

Co-authored-by: Daniel Standish <[email protected]>
apache · Nov 24, 2022 · 2e20e9f · 2e20e9f
1 parent 7d79812
commit 2e20e9f
Show file tree

Hide file tree

Showing 102 changed files with 720 additions and 185 deletions.
diff --git a/airflow/providers/amazon/CHANGELOG.rst b/airflow/providers/amazon/CHANGELOG.rst
@@ -24,6 +24,26 @@
 Changelog
 ---------
 
+6.2.0
+.....
+
+Features
+~~~~~~~~
+
+* ``Use Boto waiters instead of customer _await_status method for RDS Operators (#27410)``
+* ``Handle transient state errors in 'RedshiftResumeClusterOperator' and 'RedshiftPauseClusterOperator' (#27276)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Correct job name matching in SagemakerProcessingOperator (#27634)``
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``System Test for EMR (AIP-47) (#27286)``
+
+
 6.1.0
 .....
 

diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
@@ -22,6 +22,7 @@ description: |
  Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 
 versions:
+ - 6.2.0
  - 6.1.0
  - 6.0.0
  - 5.1.0

diff --git a/airflow/providers/apache/drill/CHANGELOG.rst b/airflow/providers/apache/drill/CHANGELOG.rst
@@ -24,6 +24,14 @@
 Changelog
 ---------
 
+2.3.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 2.3.0
 .....
 

diff --git a/airflow/providers/apache/drill/provider.yaml b/airflow/providers/apache/drill/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Apache Drill <https://drill.apache.org/>`__.
 
 versions:
+ - 2.3.1
  - 2.3.0
  - 2.2.1
  - 2.2.0

diff --git a/airflow/providers/apache/druid/CHANGELOG.rst b/airflow/providers/apache/druid/CHANGELOG.rst
@@ -24,6 +24,14 @@
 Changelog
 ---------
 
+3.3.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 3.3.0
 .....
 

diff --git a/airflow/providers/apache/druid/provider.yaml b/airflow/providers/apache/druid/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Apache Druid <https://druid.apache.org/>`__.
 
 versions:
+ - 3.3.1
  - 3.3.0
  - 3.2.1
  - 3.2.0

diff --git a/airflow/providers/apache/hive/CHANGELOG.rst b/airflow/providers/apache/hive/CHANGELOG.rst
@@ -24,6 +24,14 @@
 Changelog
 ---------
 
+4.1.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 4.1.0
 .....
 

diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Apache Hive <https://hive.apache.org/>`__
 
 versions:
+ - 4.1.1
  - 4.1.0
  - 4.0.1
  - 4.0.0

diff --git a/airflow/providers/apache/pinot/CHANGELOG.rst b/airflow/providers/apache/pinot/CHANGELOG.rst
@@ -24,6 +24,14 @@
 Changelog
 ---------
 
+4.0.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 4.0.0
 .....
 

diff --git a/airflow/providers/apache/pinot/provider.yaml b/airflow/providers/apache/pinot/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Apache Pinot <https://pinot.apache.org/>`__
 
 versions:
+ - 4.0.1
  - 4.0.0
  - 3.2.1
  - 3.2.0

diff --git a/airflow/providers/asana/CHANGELOG.rst b/airflow/providers/asana/CHANGELOG.rst
@@ -23,18 +23,12 @@
 Changelog
 ---------
 
-3.0.0
+2.1.0
 .....
 
 This release of provider is only available for Airflow 2.3+ as explained in the
 `Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/README.md#support-for-providers>`_.
 
-Breaking changes
-~~~~~~~~~~~~~~~~
-
-* In AsanaHook, non-prefixed extra fields are supported and are preferred. So if you should update your
- connection to replace ``extra__asana__workspace`` with ``workspace`` etc.
-
 Misc
 ~~~~
 
@@ -43,14 +37,18 @@ Misc
 Features
 ~~~~~~~~
 
+In AsanaHook, non-prefixed extra fields are supported and are preferred. You should update your
+connection to replace ``extra__asana__workspace`` with ``workspace`` etc.
+
+
 * ``Allow and prefer non-prefixed extra fields for AsanaHook (#27043)``
 
 .. Below changes are excluded from the changelog. Move them to
  appropriate section above if needed. Do not delete the lines(!):
  * ``Enable string normalization in python formatting - providers (#27205)``
  * ``Update docs for September Provider's release (#26731)``
  * ``Apply PEP-563 (Postponed Evaluation of Annotations) to non-core airflow (#26289)``
-
+ * ``pRepare docs for November 2022 wave of Providers (#27613)``
 
 2.0.1
 .....

diff --git a/airflow/providers/asana/provider.yaml b/airflow/providers/asana/provider.yaml
@@ -22,7 +22,7 @@ description: |
  `Asana <https://app.asana.com/>`__
 
 versions:
- - 3.0.0
+ - 2.1.0
  - 2.0.1
  - 2.0.0
  - 1.1.3

diff --git a/airflow/providers/cncf/kubernetes/CHANGELOG.rst b/airflow/providers/cncf/kubernetes/CHANGELOG.rst
@@ -52,14 +52,10 @@ the corresponding objects from the kubernetes library:
 * ``airflow.providers.cncf.kubernetes.backcompat.volume``
 * ``airflow.providers.cncf.kubernetes.backcompat.volume_mount``
 
-In ``KubernetesHook.get_namespace``, if a connection is defined but a namespace isn't set, we
-currently return 'default'; this behavior is deprecated. In the next release, we'll return ``None``.
-
 * ``Remove deprecated backcompat objects for KPO (#27518)``
 * ``Remove support for node_selectors param in KPO (#27515)``
 * ``Remove unused backcompat method in k8s hook (#27490)``
 * ``Drop support for providing ''resource'' as dict in ''KubernetesPodOperator'' (#27197)``
-* ``Deprecate use of core get_kube_client in PodManager (#26848)``
 * ``Don't consider airflow core conf for KPO (#26849)``
 
 Misc
@@ -71,17 +67,22 @@ Misc
 Features
 ~~~~~~~~
 
-Previously, ``name`` was a required argument for KubernetesPodOperator (when also not supplying pod
-template or full pod spec). Now, if ``name`` is not supplied, ``task_id`` will be used.
+KubernetesPodOperator argument ``name`` is now optional. Previously, ``name`` was a
+required argument for KubernetesPodOperator when also not supplying pod
+template or full pod spec. Now, if ``name`` is not supplied, ``task_id`` will be used.
 
-KubernetsPodOperator argument ``namespace`` is now optional. If not supplied via KPO param or pod
+KubernetesPodOperator argument ``namespace`` is now optional. If not supplied via KPO param or pod
 template file or full pod spec, then we'll check the airflow conn,
 then if in a k8s pod, try to infer the namespace from the container, then finally
 will use the ``default`` namespace.
 
+When using an Airflow connection of type ``kubernetes``, if defining the connection in an env var
+or secrets backend, it's no longer necessary to prefix the "extra" fields with ``extra__kubernetes__``.
+If ``extra`` contains duplicate fields (one with prefix, one without) then the non-prefixed
+one will be used.
 
+* ``Remove extra__kubernetes__ prefix from k8s hook extras (#27021)``
 * ``Add container_resources as KubernetesPodOperator templatable (#27457)``
-* ``Add deprecation warning re unset namespace in k8s hook (#27202)``
 * ``add container_name option for SparkKubernetesSensor (#26560)``
 * ``Allow xcom sidecar container image to be configurable in KPO (#26766)``
 * ``Improve task_id to pod name conversion (#27524)``
@@ -94,7 +95,6 @@ Bug Fixes
 
 * ``Fix KubernetesHook fail on an attribute absence (#25787)``
 * ``Fix log message for kubernetes hooks (#26999)``
-* ``Remove extra__kubernetes__ prefix from k8s hook extras (#27021)``
 * ``KPO should use hook's get namespace method to get namespace (#27516)``
 
 .. Below changes are excluded from the changelog. Move them to
@@ -103,6 +103,14 @@ Bug Fixes
  * ``Enable string normalization in python formatting - providers (#27205)``
  * ``Update docs for September Provider's release (#26731)``
 
+New deprecations
+~~~~~~~~~~~~~~~~
+
+* In ``KubernetesHook.get_namespace``, if a connection is defined but a namespace isn't set, we
+ currently return 'default'; this behavior is deprecated (#27202). In the next release, we'll return ``None``.
+* ``Deprecate use of core get_kube_client in PodManager (#26848)``
+
+
 4.4.0
 .....
 

diff --git a/airflow/providers/common/sql/CHANGELOG.rst b/airflow/providers/common/sql/CHANGELOG.rst
@@ -42,7 +42,7 @@ Bug Fixes
 
 * ``Restore removed (but used) methods in common.sql (#27843)``
 * ``Fix errors in Databricks SQL operator introduced when refactoring (#27854)``
-
+* ``Bump common.sql provider to 1.3.1 (#27888)``
 
 1.3.0
 .....

diff --git a/airflow/providers/databricks/CHANGELOG.rst b/airflow/providers/databricks/CHANGELOG.rst
@@ -24,6 +24,39 @@
 Changelog
 ---------
 
+4.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+The ``DatabricksSqlHook`` is now conforming to the same semantics as all the other ``DBApiHook``
+implementations and returns the same kind of response in its ``run`` method. Previously (pre 4.* versions
+of the provider, the Hook returned Tuple of ("cursor description", "results") which was not compatible
+with other DBApiHooks that return just "results". After this change (and dependency on common.sql >= 1.3.1),
+The ``DatabricksSqlHook`` returns now "results" only. The ``description`` can be retrieved via
+``last_description`` field of the hook after ``run`` method completes.
+
+That makes the ``DatabricksSqlHook`` suitable for generic SQL operator and detailed lineage analysis.
+
+If you had custom hooks or used the Hook in your TaskFlow code or custom operators that relied on this
+behaviour, you need to adapt your DAGs.
+
+The Databricks ``DatabricksSQLOperator`` is also more standard and derives from common
+``SQLExecuteQueryOperator`` and uses more consistent approach to process output when SQL queries are run.
+However in this case the result returned by ``execute`` method is unchanged (it still returns Tuple of
+("description", "results") and this Tuple is pushed to XCom, so your DAGs relying on this behaviour
+should continue working without any change.
+
+* ``Fix errors in Databricks SQL operator introduced when refactoring (#27854)``
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix templating fields and do_xcom_push in DatabricksSQLOperator (#27868)``
+
+
 3.4.0
 .....
 

diff --git a/airflow/providers/databricks/provider.yaml b/airflow/providers/databricks/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Databricks <https://databricks.com/>`__
 
 versions:
+ - 4.0.0
  - 3.4.0
  - 3.3.0
  - 3.2.0

diff --git a/airflow/providers/elasticsearch/CHANGELOG.rst b/airflow/providers/elasticsearch/CHANGELOG.rst
@@ -24,6 +24,14 @@
 Changelog
 ---------
 
+4.3.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 4.3.0
 .....
 

diff --git a/airflow/providers/elasticsearch/provider.yaml b/airflow/providers/elasticsearch/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Elasticsearch <https://www.elastic.co/elasticsearch>`__
 
 versions:
+ - 4.3.1
  - 4.3.0
  - 4.2.1
  - 4.2.0

diff --git a/airflow/providers/exasol/CHANGELOG.rst b/airflow/providers/exasol/CHANGELOG.rst
@@ -24,6 +24,15 @@
 Changelog
 ---------
 
+4.1.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix errors in Databricks SQL operator introduced when refactoring (#27854)``
+* ``Bump common.sql provider to 1.3.1 (#27888)``
+
 4.1.0
 .....
 

diff --git a/airflow/providers/exasol/provider.yaml b/airflow/providers/exasol/provider.yaml
@@ -22,6 +22,7 @@ description: |
  `Exasol <https://docs.exasol.com/home.htm>`__
 
 versions:
+ - 4.1.1
  - 4.1.0
  - 4.0.1
  - 4.0.0