Description

As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish

After #52469 and #52474 are implemented - another increment is to split example and testing dags which are today somehow interlinked. The following rule should apply:

• pure example dags are stored per provider in /src/airflow/providers//example_dags/
• Testing dags should be moved to /src/airflow/providers//testing_dags/
• Dual use Dags which are examples but also used for testing would be stored with example_dags but tagged for testing

The dynamic dag bundle (or whatever mechanism) that collects examples would also need to be extended to source testing dags where provided per provider.

We can automatically add such dags to be used in unit test via auto-fixture in the shared conftest.py or similar - this way it will work in both breeze and local venv.
Also on breeze cli the testing dags should be loadable similar like the CLI command --load-example-dags via --load-testing-dags. Same liek a Airflow Conf entry should be available so that a standard deployment can use it.

Use case/motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the Airflow 3 Dev Calls there was a demand named to clean-up and optimize example DAGs.

Related issues

#52469 and #52474 as pre-requisite

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct