Skip to content

Make dag_version_id in TI non-nullable #50825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Make dag_version_id in TI non-nullable #50825

wants to merge 32 commits into from
+1,499 −471

Conversation

ephraimbuddy
Copy link
Contributor

Enforces serialization of DAGs before creating DagRuns and TaskInstances.

The changes include:

  • Making dag_version_id a required parameter in TaskInstance constructor
  • Migrating the dag_version_id column in TaskInstance to be non-nullable
  • Adding a check in _create_orm_dagrun to raise AirflowException if dag is not serialized
  • Updating tests to ensure DAGs are properly serialized before creating DagRuns

This prevents running runs with missing serialized DAG versions.

@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch from 20c6e75 to 08bfa28 Compare May 20, 2025 17:14
@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch from 9d3fc74 to 63eb72b Compare May 21, 2025 15:28
@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch 2 times, most recently from aad0a5f to 80439a1 Compare May 21, 2025 17:35
uranusjr
uranusjr reviewed May 22, 2025
View reviewed changes
uranusjr
uranusjr reviewed May 22, 2025
View reviewed changes
@ephraimbuddy ephraimbuddy requested a review from mobuchowski as a code owner May 22, 2025 17:27
@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch 2 times, most recently from 5409839 to 616e0a1 Compare May 24, 2025 22:18
@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch from 616e0a1 to 070f7a3 Compare May 25, 2025 10:55
ephraimbuddy added 29 commits June 5, 2025 13:31
@ephraimbuddy
fixup! Fix tests for some providers
5be197e
@ephraimbuddy
fixup! fixup! Fix tests for some providers
d4fbe2f
@ephraimbuddy
more test fixes
14e8013
@ephraimbuddy
fixup! more test fixes
a334721
@ephraimbuddy
fixup! fixup! more test fixes
1c566a8
@ephraimbuddy
Update more tests
028d34b
@ephraimbuddy
fixup! Update more tests
d34cc34
@ephraimbuddy
fixup! fixup! Update more tests
bcfb4d2
@ephraimbuddy
fixup! fixup! fixup! Update more tests
73f6bee
@ephraimbuddy
fixup! fixup! fixup! fixup! Update more tests
5185806
@ephraimbuddy
fixup! fixup! fixup! fixup! fixup! Update more tests
cd2e39a
@ephraimbuddy
Add compat for MockedTaskInstance
565d99b
@ephraimbuddy
fixup! Add compat for MockedTaskInstance
09e6d60
@ephraimbuddy
fixup! fixup! Add compat for MockedTaskInstance
3538722
@ephraimbuddy
fix system test
34dd15d
@ephraimbuddy
fixup! fix system test
0397f9b
@ephraimbuddy
A fix for kube integration tests
6022279
@ephraimbuddy
Fix generic transfer
396f36c
@ephraimbuddy
Update the expected TIs from test multiple versions number file
33c5583
@ephraimbuddy
fix incorrect imports
60b9143
@ephraimbuddy
fix time delta async test
7f789df
@ephraimbuddy
Remove non working kube test fix
3563a20
@ephraimbuddy
Fix task sdk test
e3a20d8
@ephraimbuddy
Add cadwyn migration and fix some tests
cdace46
@ephraimbuddy
fix some provider tests
b2fecee
@ephraimbuddy
Fix backcompat test
0a20025
@ephraimbuddy
Add data migration before migrating dag_version_id
7e297a2
@ephraimbuddy
Resolve conflict
b033468
@ephraimbuddy
Update test, TaskInstanceResponse and fix assert dag_version in task_…
b583d74 
…command.py
@ephraimbuddy ephraimbuddy force-pushed the make-ti-dag-version-id-non-nullable2 branch from 874c15e to b583d74 Compare June 5, 2025 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr left review comments

@pierrejeambrun pierrejeambrun pierrejeambrun left review comments

@jedcunningham jedcunningham jedcunningham left review comments

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@mobuchowski mobuchowski Awaiting requested review from mobuchowski mobuchowski is a code owner

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

@jason810496 jason810496 Awaiting requested review from jason810496 jason810496 is a code owner

@mik-laj mik-laj Awaiting requested review from mik-laj mik-laj is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@shubhamraj-git shubhamraj-git Awaiting requested review from shubhamraj-git shubhamraj-git is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

@rawwar rawwar Awaiting requested review from rawwar rawwar is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@ephraimbuddy ephraimbuddy

Labels
area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler area:serialization area:task-sdk area:Triggerer full tests needed We need to run full set of tests for this PR to merge kind:documentation
Projects
None yet
Milestone
Airflow 3.1.0
Development

Successfully merging this pull request may close these issues.
5 participants
@ephraimbuddy @pierrejeambrun @uranusjr @jedcunningham @phanikumv