Fix various incompatibilities with SQLAlchemy 2.0 #52518
Merged
+271
−197
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR adds and refines compatibility fixes to support both SQLAlchemy 1.4 and 2.0. Key changes include:
- Introducing version‐compatibility functions and flags in the test utils.
- Adjusting engine configuration, SQL query construction, and ORM behaviors in both tests and production code.
- Updating migration scripts to align with updated SQLAlchemy syntax.
Reviewed Changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| devel-common/src/tests_common/test_utils/version_compat.py | Adds version-compat functions to determine SQLAlchemy sub-version. |
| airflow-core/tests/unit/jobs/test_scheduler_job.py | Updates query handling logic based on SQLAlchemy version. |
| airflow-core/tests/unit/dag_processing/test_collection.py | Adds cleanup fixtures to tests. |
| airflow-core/tests/unit/core/test_sqlalchemy_config.py | Adjusts engine configuration expectations for dual SQLAlchemy support. |
| airflow-core/tests/unit/api_fastapi/common/test_exceptions.py | Updates error message assertions to distinguish between SQLAlchemy versions. |
| airflow-core/tests/unit/always/test_connection.py | Modifies URL checks based on SQLAlchemy version differences. |
| airflow-core/src/airflow/utils/sqlalchemy.py | Introduces a version-aware dialect keyword helper. |
| airflow-core/src/airflow/models/dagrun.py | Refactors CASE expression construction using an intermediate when_condition. |
| airflow-core/src/airflow/models/dag_version.py | Adds an explicit commit after adding a new DagVersion. |
| airflow-core/src/airflow/models/base.py | Temporarily enables unmapped models for SQLAlchemy 2.0 via allow_unmapped. |
| airflow-core/src/airflow/migrations/versions/*.py | Updates usages of with_variant to match updated SQLAlchemy API syntax. |
| airflow-core/src/airflow/cli/commands/rotate_fernet_key_command.py | Revises the batch key rotation loop to leverage SQLAlchemy 2.x yield behavior. |
| airflow-core/src/airflow/api_fastapi/core_api/routes/public/import_error.py | Modifies join conditions using and_ for improved query clarity. |
Comments suppressed due to low confidence (1)
airflow-core/src/airflow/models/dag_version.py:111
- Explicitly committing the session within write_dag may interfere with transaction management if the caller expects to control commits. Consider documenting this behavior or moving commit control to the caller.
session.commit()
c718924 to
b7cea6b
Compare
|
Awesome @Dev-iL 🎉 |
|
Cool! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current intention is to have dual-support of SQLAlchemy 1.4 and 2.0. The most recent effort is #52233 - which adds a couple of CI jobs and the various fixes (business logic and flakiness) allowing them to succeed. This PR cherry-picks all the fixes included in #52233 (i.e. without FAB, breeze, etc.) for easier review.
A successful CI run for this PR should demonstrate that 1.4 support is intact (as far as the test coverage can tell).
Related: #28723
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.