Skip to content

Introduce workers.celery and workers.kubernetes config sections #51460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

Miretpl
Copy link
Contributor

@Miretpl Miretpl commented Jun 5, 2025

closes: #28880

Currently, in the Helm Chart, there is a workers field whose values are shared between Celery and Kubernetes executors workers. The only possible way to configure both separately is by using the podTemplate field, which overrides the whole pod-template-file content. However, there is no easy possibility to override part of the configuration or to share the same configuration between workers.

This PR introduces the workers.celery and workers.kubernetes files, which contain specific fields used by each executor worker. Every field specified under the above sections overrides the value of the respective fields under the workers section. In case of configuring the field under workers section, without configuring different values under workers.celery or workers.kubernetes sections, it will result in applying the change to both worker types. I added comments about deprecation to every field which was used only by one executor type. Also, I implemented the separation of service accounts used by pod-template-file and Airflow Celery workers with the usage of workers.useWorkerDedicatedServiceAccounts flag. By default, both workers will use a single common service account, and the fields workers.celery.serviceAccount and workers.kubernetes.serviceAccount will have no effect.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Jun 5, 2025
@Miretpl Miretpl changed the title Separate workers config Introduce workers.celery and workers.kubernetes config sections Jun 5, 2025
@Miretpl
Copy link
Contributor Author

Miretpl commented Jun 5, 2025

Currently, the introduction of workers.celery section should be fully finished.

@Miretpl Miretpl force-pushed the separate-workers-config branch from 94a1b5a to 12ba007 Compare June 13, 2025 17:51
@Miretpl
Copy link
Contributor Author

Miretpl commented Jun 13, 2025

Both sections workers.celery and workers.kubernetes should be finished. I would appreciate any help with tests, due to the number of changes in the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:helm-chart Airflow Helm Chart
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't configure Kubernetes and Celery workers in Helm Chart
1 participant