Skip to content

Commit

Permalink
Get rid of requests as core dependency (#15781)
Browse files Browse the repository at this point in the history 
This change gets rid of requests as core dependency. We have to
change requests to become an optional dependency because it
(so far) pulls in chardet as dependency and chardet is
LGPL, which is not allowed to be mandatory dependency by
ASF policies.

More info here:

https://issues.apache.org/jira/browse/LEGAL-572

The changes:

* connexion is vendored-in (and requests usage is replaced with httpx)
* Http Provider is turned into optional provider (not preinstalled)
* Few places where requests were used in core and in cloud_sql provider
  which did not cause compatibility problem, it was replaced by httpx.
* new extra added for deprecated experimental API (which is disabled
  by default and optional)
* tests are fixed (using pytest-httpx fixture package)
* The providers: http, airbyte, apache.livy, opsgenie, slack (all depend
  on http) now explicitely depend on `requirements`.
  • Loading branch information
@potiuk
potiuk committed May 17, 2021
1 parent 31894fa commit bb115da
Show file tree
Hide file tree
Showing 68 changed files with 6,468 additions and 86 deletions.
58 changes: 40 additions & 18 deletions .pre-commit-config.yaml
View file
Expand Up @@ -29,11 +29,11 @@ repos:
rev: v1.1.10
hooks:
- id: forbid-tabs
exclude: ^docs/Makefile$|^clients/gen/go.sh|\.gitmodules$
exclude: ^docs/Makefile$|^clients/gen/go.sh|\.gitmodules$|^airflow/_vendor/
- id: insert-license
name: Add license for all SQL files
files: \.sql$
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
args:
- --comment-style
- "/*||*/"
Expand All @@ -42,7 +42,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all other files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
args:
- --comment-style
- "|#|"
Expand All @@ -53,7 +53,7 @@ repos:
\.properties$|\.cfg$|\.conf$|\.ini$|\.ldif$|\.readthedocs$|\.service$|\.tf$|Dockerfile.*$
- id: insert-license
name: Add license for all rst files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
args:
- --comment-style
- "||"
Expand All @@ -64,7 +64,7 @@ repos:
- id: insert-license
name: Add license for all JS/TS/TSX/CSS/PUML files
files: \.(js|ts|tsx|css|puml)$
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
args:
- --comment-style
- "/*!| *| */"
Expand All @@ -74,7 +74,7 @@ repos:
- id: insert-license
name: Add license for all JINJA template files
files: "^airflow/www/templates/.*\\.html$|^docs/templates/.*\\.html$.*\\.jinja2"
exclude: ^\.github/.*$
exclude: ^\.github/.*$^airflow/_vendor/
args:
- --comment-style
- "{#||#}"
Expand All @@ -83,7 +83,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all shell files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
files: ^breeze$|^breeze-complete$|\.sh$|\.bash$|\.bats$
args:
- --comment-style
Expand All @@ -93,7 +93,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all Python files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
types: [python]
args:
- --comment-style
Expand All @@ -103,7 +103,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all XML files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
files: \.xml$
args:
- --comment-style
Expand All @@ -113,7 +113,7 @@ repos:
- --fuzzy-match-generates-todo
- id: insert-license
name: Add license for all YAML files
exclude: ^\.github/.*$
exclude: ^\.github/.*$|^airflow/_vendor/
types: [yaml]
files: \.yml$|\.yaml$
args:
Expand All @@ -125,7 +125,7 @@ repos:
- id: insert-license
name: Add license for all md files
files: \.md$
exclude: ^\.github/.*$|PROVIDER_CHANGES.*\.md
exclude: ^\.github/.*$|PROVIDER_CHANGES.*\.md|^airflow/_vendor/
args:
- --comment-style
- "<!--|| -->"
Expand All @@ -141,12 +141,14 @@ repos:
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
files: \.mermaid$
exclude: ^airflow/_vendor/
- repo: https://github.com/thlorenz/doctoc.git
rev: v2.0.0
hooks:
- id: doctoc
name: Add TOC for md files
files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*\.md$|^dev/.*\.md$|^dev/.*\.rst$
exclude: ^airflow/_vendor/
args:
- "--maxlevel"
- "2"
Expand All @@ -155,31 +157,42 @@ repos:
hooks:
- id: black
args: [--config=./pyproject.toml]
exclude: ^airflow/_vendor/
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
- id: check-merge-conflict
- id: debug-statements
- id: check-builtin-literals
exclude: ^airflow/_vendor/
- id: detect-private-key
- id: end-of-file-fixer
exclude: ^airflow/_vendor/
- id: mixed-line-ending
exclude: ^airflow/_vendor/
- id: check-executables-have-shebangs
exclude: ^airflow/_vendor/
- id: check-xml
exclude: ^airflow/_vendor/
- id: trailing-whitespace
exclude: ^airflow/_vendor/
- id: fix-encoding-pragma
exclude: ^airflow/_vendor/
args:
- --remove
- repo: https://github.com/asottile/pyupgrade
rev: v2.13.0
hooks:
- id: pyupgrade
args: ["--py36-plus"]
exclude: ^airflow/_vendor/
- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.8.0
hooks:
- id: rst-backticks
exclude: ^airflow/_vendor/
- id: python-no-log-warn
exclude: ^airflow/_vendor/
- repo: https://github.com/adrienverge/yamllint
rev: v1.26.1
hooks:
Expand All @@ -188,15 +201,15 @@ repos:
entry: yamllint -c yamllint-config.yml --strict
types: [yaml]
exclude:
^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/(?:templates|files)/.*\.yaml$
^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/(?:templates|files)/.*\.yaml$|^airflow/_vendor/
- repo: https://github.com/timothycrosley/isort
rev: 5.8.0
hooks:
- id: isort
name: Run isort to sort imports in Python files
files: \.py$
# To keep consistent with the global isort skip config defined in setup.cfg
exclude: ^build/.*$|^.tox/.*$|^venv/.*$
exclude: ^build/.*$|^.tox/.*$|^venv/.*$|^airflow/_vendor/
- repo: https://github.com/pycqa/pydocstyle
rev: 6.0.0
hooks:
Expand All @@ -213,7 +226,8 @@ repos:
^provider_packages|
^kubernetes_tests|
.*example_dags/.*|
^chart/.*\.py$
^chart/.*\.py$|
^airflow/_vendor/
- repo: local
hooks:
- id: lint-openapi
Expand Down Expand Up @@ -306,7 +320,7 @@ repos:
entry: "^\\s*from\\s+\\."
pass_filenames: true
files: \.py$
exclude: ^tests/
exclude: ^tests/|^airflow/_vendor/
- id: language-matters
language: pygrep
name: Check for language that we do not accept as community
Expand Down Expand Up @@ -360,31 +374,36 @@ repos:
^airflow/operators/.*$|
^airflow/sensors/.*$|
^airflow/providers/.*\.py$
exclude: ^airflow/_vendor/
- id: provide-create-sessions
language: pygrep
name: Check provide_session and create_session imports
description: provide_session and create_session should be imported from airflow.utils.session
to avoid import cycles.
entry: "from airflow\\.utils\\.db import.* (provide_session|create_session)"
files: \.py$
exclude: ^airflow/_vendor/
pass_filenames: true
- id: incorrect-use-of-LoggingMixin
language: pygrep
name: Make sure LoggingMixin is not used alone
entry: "LoggingMixin\\(\\)"
files: \.py$
exclude: ^airflow/_vendor/
pass_filenames: true
- id: daysago-import-check
language: pygrep
name: Make sure days_ago is imported from airflow.utils.dates
entry: "(airflow\\.){0,1}utils\\.dates\\.days_ago"
files: \.py$
exclude: ^airflow/_vendor/
pass_filenames: true
- id: restrict-start_date
language: pygrep
name: "'start_date' should not be defined in default_args in example_dags"
entry: "default_args\\s*=\\s*{\\s*(\"|')start_date(\"|')|(\"|')start_date(\"|'):"
files: \.*example_dags.*\.py$
exclude: ^airflow/_vendor/
pass_filenames: true
- id: check-integrations
name: Check if integration list is aligned
Expand Down Expand Up @@ -441,7 +460,7 @@ repos:
language: system
entry: ./scripts/ci/pre_commit/pre_commit_bat_tests.sh
files: \.sh$|\.bash$|\.bats$
exclude: ^tests/bats/in_container|^scripts/in_container|^tests/bats/breeze
exclude: ^tests/bats/in_container|^scripts/in_container|^tests/bats/breeze|^airflow/_vendor/
pass_filenames: true
require_serial: true
- id: stylelint
Expand Down Expand Up @@ -512,6 +531,7 @@ repos:
language: python
pass_filenames: true
files: .*\.schema\.json$
exclude: ^airflow/_vendor/
require_serial: true
additional_dependencies: ['jsonschema==3.2.0', 'PyYAML==5.3.1', 'requests==2.25.0']
- id: json-schema
Expand Down Expand Up @@ -577,6 +597,7 @@ repos:
language_version: python3
additional_dependencies: ['flynt==0.63']
files: \.py$
exclude: ^airflow/_vendor/
## ADD MOST PRE-COMMITS ABOVE THAT LINE
# The below pre-commits are those requiring CI image to be built
- id: build
Expand All @@ -590,7 +611,7 @@ repos:
language: system
entry: ./scripts/ci/pre_commit/pre_commit_mypy.sh
files: \.py$
exclude: ^dev|^provider_packages|^chart|^docs
exclude: ^dev|^provider_packages|^chart|^docs|^airflow/_vendor/
- id: mypy
name: Run mypy for helm chart tests
language: system
Expand All @@ -609,7 +630,7 @@ repos:
language: system
entry: ./scripts/ci/pre_commit/pre_commit_pylint.sh
files: \.py$
exclude: ^scripts/.*\.py$|^dev|^provider_packages|^chart|^tests|^kubernetes_tests
exclude: ^scripts/.*\.py$|^dev|^provider_packages|^chart|^tests|^kubernetes_tests|^airflow/_vendor/
pass_filenames: true
require_serial: true
- id: pylint
Expand All @@ -632,6 +653,7 @@ repos:
entry: ./scripts/ci/pre_commit/pre_commit_flake8.sh
files: \.py$
pass_filenames: true
exclude: ^airflow/_vendor/
- id: ui-lint
name: ESLint against airflow/ui
language: node
Expand Down
16 changes: 8 additions & 8 deletions CONTRIBUTING.rst
View file
Expand Up @@ -580,14 +580,14 @@ This is the full list of those extras:
airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.druid,
apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark,
apache.sqoop, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
cncf.kubernetes, crypto, dask, databricks, datadog, devel, devel_all, devel_ci, devel_hadoop,
dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api,
github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc, jenkins,
jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql, microsoft.winrm, mongo, mssql,
mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel, devel_all, devel_ci,
devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc,
jenkins, jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql, microsoft.winrm, mongo,
mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill, password, pinot,
plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid,
sentry, sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino,
vertica, virtualenv, webhdfs, winrm, yandex, zendesk

.. END EXTRAS HERE
Expand Down
16 changes: 8 additions & 8 deletions INSTALL
View file
Expand Up @@ -92,14 +92,14 @@ The list of available extras:
airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.druid,
apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark,
apache.sqoop, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
cncf.kubernetes, crypto, dask, databricks, datadog, devel, devel_all, devel_ci, devel_hadoop,
dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api,
github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc, jenkins,
jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql, microsoft.winrm, mongo, mssql,
mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel, devel_all, devel_ci,
devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc,
jenkins, jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql, microsoft.winrm, mongo,
mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill, password, pinot,
plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid,
sentry, sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino,
vertica, virtualenv, webhdfs, winrm, yandex, zendesk

# END EXTRAS HERE

Expand Down
12 changes: 12 additions & 0 deletions UPDATING.md
View file
Expand Up @@ -71,6 +71,18 @@ https://developers.google.com/style/inclusive-documentation
-->

### New "deprecated_api" extra

We have a new '[deprecated_api]' extra that should be used when installing airflow when the deprecated API
is going to be used. This is now an optional feature of Airflow now because it pulls in `requests` which
(as of 14 May 2021) pulls LGPL `chardet` dependency.

### The `http` provider is not installed by default

The `http` provider is now optional and not installed by default, until `chardet` becomes an optional
dependency of `requests`.
See [PR to replace chardet with charset-normalizer](https://github.com/psf/requests/pull/5797)

### `@apply_default` decorator isn't longer necessary

This decorator is now automatically added to all operators via the metaclass on BaseOperator
Expand Down
Empty file added airflow/_vendor/__init__.py
View file
Empty file.
48 changes: 48 additions & 0 deletions airflow/_vendor/connexion/__init__.py
View file
@@ -0,0 +1,48 @@
import sys

import werkzeug.exceptions as exceptions # NOQA

from .apis import AbstractAPI # NOQA
from .apps import AbstractApp # NOQA
from .decorators.produces import NoContent # NOQA
from .exceptions import ProblemException # NOQA
# add operation for backwards compatability
from .operations import compat
from .problem import problem # NOQA
from .resolver import Resolution, Resolver, RestyResolver # NOQA

full_name = '{}.operation'.format(__package__)
sys.modules[full_name] = sys.modules[compat.__name__]


def not_installed_error(exc): # pragma: no cover
import functools

def _required_lib(exc, *args, **kwargs):
raise exc

return functools.partial(_required_lib, exc)


try:
from .apis.flask_api import FlaskApi, context # NOQA
from .apps.flask_app import FlaskApp
from flask import request # NOQA
except ImportError as e: # pragma: no cover
_flask_not_installed_error = not_installed_error(e)
FlaskApi = _flask_not_installed_error
FlaskApp = _flask_not_installed_error

App = FlaskApp
Api = FlaskApi

try:
from .apis.aiohttp_api import AioHttpApi
from .apps.aiohttp_app import AioHttpApp
except ImportError as e: # pragma: no cover
_aiohttp_not_installed_error = not_installed_error(e)
AioHttpApi = _aiohttp_not_installed_error
AioHttpApp = _aiohttp_not_installed_error

# This version is replaced during release process.
__version__ = '2.7.0'
3 changes: 3 additions & 0 deletions airflow/_vendor/connexion/__main__.py
View file
@@ -0,0 +1,3 @@
from airflow._vendor.connexion.cli import main # pragma: no cover

main() # pragma: no cover
1 change: 1 addition & 0 deletions airflow/_vendor/connexion/apis/__init__.py
View file
@@ -0,0 +1 @@
from .abstract import AbstractAPI # NOQA

0 comments on commit bb115da

Please sign in to comment.