Skip to content

CloudFormation v2 Engine: Runtime Caching Support #12807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 27, 2025
Merged

CloudFormation v2 Engine: Runtime Caching Support #12807

merged 2 commits into from
Jun 27, 2025

Conversation

MEPalma
Copy link
Contributor

@MEPalma MEPalma commented Jun 26, 2025

Motivation

The introduction of the CloudFormation v2 engine laid the foundation for a redesigned engine capable of accurately determining update requirements between CloudFormation deployments, while also enabling parallel execution during updates. However, the new engine does not store or cache the evaluation of previous change set operations. This leads to issues, such as the need to reevaluate global macros from earlier template versions. This process fails if the macro is no longer available, or it can result in corrupted states if the macro has been modified.
The absence of caching also forces full reevaluation of both template versions during a chain of change set updates, even when this information could simply be read from cache.
This PR introduces support for caching, resolving issues related to global transformations. It also adds utility functions for downstream processors to take advantage of the cache, applies these utilities in several scenarios within the preprocessor, and includes a negative test case for the global transformation scenario.

Changes

  • Added UpdateModel encapsulation for change set model visitors to handle runtime caching values
  • Added logics to load and save the runtime caches
  • Added utility function to utilize the cache within change set model visitors
  • Ported a number of preprocessing visitors to using the cache
  • Added related negative test for global transforms (see description above)
  • Other related/minors

@MEPalma MEPalma added this to the 4.6 milestone Jun 26, 2025
@MEPalma MEPalma requested a review from simonrw June 26, 2025 15:55
@MEPalma MEPalma self-assigned this Jun 26, 2025
@MEPalma MEPalma added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Jun 26, 2025
@github-actions GitHub Actions
Copy link

Test Results - Preflight, Unit

21 727 tests  ±0   20 070 ✅ ±0   6m 13s ⏱️ -8s
     1 suites ±0    1 657 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit a936c13. ± Comparison against base commit 019d5ec.

@github-actions GitHub Actions
Copy link

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 14s ⏱️ -14s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit a936c13. ± Comparison against base commit 019d5ec.

@github-actions GitHub Actions
Copy link

LocalStack Community integration with Pro

  2 files  ±    0    2 suites  ±0   21m 59s ⏱️ - 1h 24m 26s
887 tests  - 4 009  323 ✅  - 3 799  564 💤  - 210  0 ❌ ±0 
889 runs   - 4 009  323 ✅  - 3 799  566 💤  - 210  0 ❌ ±0 

Results for commit a936c13. ± Comparison against base commit 019d5ec.

This pull request removes 4010 and adds 1 tests. Note that renamed tests count towards both. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.cloudformation.v2.test_change_set_global_macros.TestChangeSetGlobalMacros ‑ test_update_after_macro_for_before_version_is_deleted
This pull request removes 211 skipped tests and adds 1 skipped test. Note that renamed tests count towards both. 
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input4-FAILED]
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_deployed_infra_state
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_populate_data
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_user_clicks_are_stored
tests.aws.services.apigateway.test_apigateway_api.TestApiGatewayApiRestApi ‑ test_get_api_case_insensitive
tests.aws.services.apigateway.test_apigateway_api.TestApigatewayIntegration ‑ test_put_integration_request_parameter_bool_type
tests.aws.services.apigateway.test_apigateway_basic.TestAPIGateway ‑ test_api_gateway_authorizer_crud
tests.aws.services.apigateway.test_apigateway_basic.TestAPIGateway ‑ test_api_gateway_http_integration_with_path_request_parameter
tests.aws.services.apigateway.test_apigateway_basic.TestAPIGateway ‑ test_api_gateway_lambda_proxy_integration[/lambda/foo1]
…

tests.aws.services.cloudformation.v2.test_change_set_global_macros.TestChangeSetGlobalMacros ‑ test_update_after_macro_for_before_version_is_deleted

@github-actions GitHub Actions
Copy link

Test Results (amd64) - Integration, Bootstrap

  5 files    5 suites   34m 30s ⏱️
911 tests 348 ✅ 563 💤 0 ❌
917 runs  348 ✅ 569 💤 0 ❌

Results for commit a936c13.

simonrw
simonrw approved these changes Jun 27, 2025
View reviewed changes
Copy link
Contributor

@simonrw simonrw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR was pair-reviewed with @MEPalma .

@MEPalma MEPalma merged commit 27dd3b2 into master Jun 27, 2025
42 checks passed
@MEPalma MEPalma deleted the MEP-CFN-cache branch June 27, 2025 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonrw simonrw simonrw approved these changes

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert is a code owner

@pinzon pinzon Awaiting requested review from pinzon pinzon is a code owner

Assignees

@MEPalma MEPalma

Labels
semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases
Projects
None yet
Milestone
4.6
Development

Successfully merging this pull request may close these issues.
2 participants
@MEPalma @simonrw