Add support for S3 dag bundle #46621

ismailsimsek · 2025-02-10T12:50:09Z

implements S3 dag bundle, with this end user could set an S3 bucket and prefix(sub folder) as a dag bundle source.

related wiki AIP-66: DAG Bundles & Parsing
related project: AIP-66: DAG Bundles & Parsing
closes #9555
closes #8657

boring-cyborg · 2025-02-10T12:50:13Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: [email protected]
Slack: https://s.apache.org/airflow-slack

ismailsimsek · 2025-04-03T09:58:42Z

moved it to provider package. its ready for review. any chance to trigger the test workflows?

o-nikolas · 2025-04-04T18:43:30Z

moved it to provider package. its ready for review. any chance to trigger the test workflows?

Triggered workflows again, will review soon 👍

o-nikolas

Looks reasonable to me (though I'm no expert on Bundles). There are some failing tests and static checks though that need looking at.

ismailsimsek · 2025-04-13T08:48:25Z

code related tests are passing now. remaining test is related to airflow version.
the BaseDagBundle code available only in airflow 3 but the test is running with airflow 2.9.x . any idea how to fix it?

ismailsimsek · 2025-04-16T10:52:09Z

@o-nikolas @jedcunningham its ready for review.

providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

o-nikolas · 2025-05-06T21:21:42Z

Looks reasonable to me, but I'm no Bundles specialist. I talked to Jed before he left on leave and he mentioned to loop in @dstandish on Bundles stuff. Does this look reasonable to you as well Daniel?

potiuk · 2025-05-21T20:11:41Z

is it possible to run following command with podman? its asking for docker
 breeze static-checks --all-files --show-diff-on-failure --color always --initialize-environment

Did you try https://podman-desktop.io/docs/migrating-from-docker/managing-docker-compatibility

jedcunningham

Sorry for the delay on reviewing this. Happy to see it!

providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

jedcunningham · 2025-05-21T22:06:36Z

providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

+        )
+        local_s3_objects = []
+        s3_bucket = self.s3_hook.get_bucket(self.bucket_name)
+        for obj in s3_bucket.objects.filter(Prefix=self.prefix):


Forgive my s3 ignorance, but is there no higher level construct we can use to bring the local side up to date? Having to iterate through and do our own sync feels a bit odd.

If we do need to do this, it feels like this might be better in the hook - the concept of syncing a directory isn't necessarily s3 bundle specific, but could be more generally useful.

Unfortunately nothing in existing libs. There are some cli packages. Moved to S3 hook

Yeah, S3 is very odd in that the API and CLI diverge in that CLI supports a sync function and the API does not 😢
You could potentially shell out to the CLI command, but I don't love that either.

Gotcha. I'll let you make that call then @o-nikolas.

Sorry for missing this. I say we stick with this code, so that we don't have a direct dependency to the CLI

providers/amazon/tests/unit/amazon/aws/bundles/test_s3.py

jedcunningham

Overall looking good, some naming and test refactors and, from my side at least, I'm happy with this.

providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py

providers/amazon/tests/unit/amazon/aws/bundles/test_s3.py

uranusjr · 2025-06-17T07:03:32Z

Don’t use caplog in general. I think it’s not too difficult to rewrite the tests.

alonahmias · 2025-06-26T20:13:15Z

Would really like to see the pr merged 😍

Co-authored-by: Ephraim Anierobi <[email protected]>

Co-authored-by: Jed Cunningham <[email protected]>

eladkal · 2025-06-26T20:19:15Z

The branch is 600 commits behind. I rebased it. lets see what CI says.

o-nikolas · 2025-06-26T23:20:03Z

I would love to see this merged as well. @ismailsimsek the use of caplog needs to be addressed (i.e. removed). Do you need any help with this?

ismailsimsek · 2025-06-28T12:21:39Z

I would love to see this merged as well. @ismailsimsek the use of caplog needs to be addressed (i.e. removed). Do you need any help with this?

That would be a great help, thank you! I don't have much time to work on this recently, so I'd really appreciate it.

ismailsimsek requested review from jedcunningham and ephraimbuddy as code owners February 10, 2025 12:50

boring-cyborg bot added the area:DAG-processing label Feb 10, 2025

jedcunningham added the AIP-66: DAG Bundle/Manifest label Mar 13, 2025

jedcunningham added this to AIP-66: DAG Bundles & Parsing Mar 13, 2025

jedcunningham moved this to In Progress in AIP-66: DAG Bundles & Parsing Mar 24, 2025

ismailsimsek force-pushed the s3bundle branch from efbadbe to 8f402fe Compare April 3, 2025 08:59

ismailsimsek requested review from eladkal and o-nikolas as code owners April 3, 2025 08:59

ismailsimsek force-pushed the s3bundle branch from 8f402fe to d34ed41 Compare April 3, 2025 09:12

ismailsimsek force-pushed the s3bundle branch 4 times, most recently from f79f69f to d7073b2 Compare April 4, 2025 18:06

o-nikolas reviewed Apr 4, 2025

View reviewed changes

ismailsimsek force-pushed the s3bundle branch 4 times, most recently from 974a4a3 to ed17613 Compare April 10, 2025 09:49

ismailsimsek force-pushed the s3bundle branch from 913addf to 124d3c6 Compare April 28, 2025 13:23

ismailsimsek requested a review from o-nikolas April 28, 2025 15:35

ismailsimsek force-pushed the s3bundle branch 2 times, most recently from 5780ad3 to a1c5c1c Compare May 3, 2025 18:59

o-nikolas reviewed May 6, 2025

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py Show resolved Hide resolved

jedcunningham reviewed May 21, 2025

View reviewed changes

ismailsimsek force-pushed the s3bundle branch from 7e75bb0 to 0f17630 Compare May 27, 2025 11:25

ismailsimsek requested review from o-nikolas and jedcunningham May 28, 2025 11:51

jedcunningham reviewed Jun 3, 2025

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py Outdated Show resolved Hide resolved

providers/amazon/tests/unit/amazon/aws/bundles/test_s3.py Outdated Show resolved Hide resolved

ismailsimsek requested a review from jedcunningham June 14, 2025 22:39

jedcunningham approved these changes Jun 16, 2025

View reviewed changes

o-nikolas approved these changes Jun 19, 2025

View reviewed changes

ismailsimsek and others added 15 commits June 26, 2025 23:18

Implement S3 dag bundle

abf8a88

Update providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

faed04e

Co-authored-by: Ephraim Anierobi <[email protected]>

remove empty arg

7004538

test empty prefix

3a5892f

change logging

2bb8633

Update providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

87cc522

Co-authored-by: Jed Cunningham <[email protected]>

Update providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py

8450d3f

Co-authored-by: Jed Cunningham <[email protected]>

Add review changes

3f222b1

Add review changes

c0d018e

Add review changes

5eeeee1

fix linting

d0c1a17

review change, improve naming

f134a6b

review change, add test to hook

340d251

review change, add test to hook

0ab61de

review change, add test to hook

23532bf

eladkal force-pushed the s3bundle branch from 7c93cfb to 23532bf Compare June 26, 2025 20:18

Add support for S3 dag bundle #46621

Are you sure you want to change the base?

Add support for S3 dag bundle #46621

Conversation

ismailsimsek commented Feb 10, 2025

Uh oh!

boring-cyborg bot commented Feb 10, 2025

Uh oh!

ismailsimsek commented Apr 3, 2025

Uh oh!

o-nikolas commented Apr 4, 2025

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

ismailsimsek commented Apr 13, 2025

Uh oh!

ismailsimsek commented Apr 16, 2025

Uh oh!

Uh oh!

o-nikolas commented May 6, 2025

Uh oh!

potiuk commented May 21, 2025

Uh oh!

jedcunningham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jedcunningham May 21, 2025

Choose a reason for hiding this comment

Uh oh!

ismailsimsek May 23, 2025

Choose a reason for hiding this comment

Uh oh!

o-nikolas May 26, 2025

Choose a reason for hiding this comment

Uh oh!

jedcunningham May 27, 2025

Choose a reason for hiding this comment

Uh oh!

o-nikolas Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jedcunningham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uranusjr commented Jun 17, 2025

Uh oh!

alonahmias commented Jun 26, 2025

Uh oh!

eladkal commented Jun 26, 2025

Uh oh!

o-nikolas commented Jun 26, 2025

Uh oh!

ismailsimsek commented Jun 28, 2025

Uh oh!

Uh oh!