Skip to content

ObjectStoragePath wrong template rendering #51909

Closed
@opeida

Description

@opeida

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Templated ObjectStoragePath parameters being rendered incorrectly.

"STORAGE_CONFIG": {
  "BUCKET": "gs://my-test-bucket",
  "CONNECTION": "storage-conn"
}
ObjectStoragePath("gs://my-test-bucket", conn_id="storage-conn")
# Rendered URI: gs://storage-conn@my-test-bucket/

ObjectStoragePath("gs://my-test-bucket", conn_id="{{ var.json.STORAGE_CONFIG.CONNECTION }}")
# Rendered URI: gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket/

ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}")
# Rendered URI: /gs:/my-test-bucket

ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}", conn_id="storage-conn")
# Rendered URI: /gs:/my-test-bucket

ObjectStoragePath("{{ var.json.STORAGE_CONFIG.BUCKET }}", conn_id="{{ var.json.STORAGE_CONFIG.CONNECTION }}")
# Rendered URI: /gs:/my-test-bucket

ObjectStoragePath("gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket")
# Rendered URI: gs://{{ var.json.STORAGE_CONFIG.CONNECTION }}@my-test-bucket/

What you think should happen instead?

Each of the provided objects must have gs://storage-conn@my-test-bucket/ as a URI.

How to reproduce

  1. Define variable: airflow variables set STORAGE_CONFIG '{ "BUCKET": "gs://my-test-bucket", "CONNECTION": "storage-conn"}'
  2. Define ObjectStoragePath instances provided above within the Task Context.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.11.0
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-fab==2.2.0
apache-airflow-providers-google==15.1.0
apache-airflow-providers-http==5.3.0
apache-airflow-providers-postgres==6.2.0
apache-airflow-providers-redis==4.1.0
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-standard==1.2.0

Deployment

Other Docker-based deployment

Deployment details

Extended image based on apache/airflow:slim-3.0.2-python3.12

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

area:UIRelated to UI/UX. For Frontend Developers.area:coregood first issuekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions