Closed
Description
Apache Airflow version
3.0.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
Templated ObjectStoragePath parameters being rendered incorrectly.
"STORAGE_CONFIG": {
"BUCKET": "gs://my-test-bucket",
"CONNECTION": "storage-conn"
}
ObjectStoragePath("gs://my-test-bucket", conn_id="storage-conn")
# Rendered URI: gs://storage-conn@my-test-bucket/
ObjectStoragePath("gs://my-test-bucket", conn_id="{{ var.json.STORAGE_CONFIG.CONNECTION }}")
# Rendered URI: gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket/
ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}")
# Rendered URI: /gs:/my-test-bucket
ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}", conn_id="storage-conn")
# Rendered URI: /gs:/my-test-bucket
ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}", conn_id="{{ var.json.STORAGE_CONFIG.CONNECTION }}")
# Rendered URI: /gs:/my-test-bucket
ObjectStoragePath("gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket")
# Rendered URI: gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket/
What you think should happen instead?
Each of the provided objects must have gs://storage-conn@my-test-bucket/
as a URI.
How to reproduce
- Define variable:
airflow variables set STORAGE_CONFIG '{ "BUCKET": "gs://my-test-bucket", "CONNECTION": "storage-conn"}'
- Define ObjectStoragePath instances provided above within the Task Context.
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.11.0
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-fab==2.2.0
apache-airflow-providers-google==15.1.0
apache-airflow-providers-http==5.3.0
apache-airflow-providers-postgres==6.2.0
apache-airflow-providers-redis==4.1.0
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-standard==1.2.0
Deployment
Other Docker-based deployment
Deployment details
Extended image based on apache/airflow:slim-3.0.2-python3.12
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct