Skip to content

fix: Dynamic set SSM-paramater tier #4613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 17, 2025
Merged

fix: Dynamic set SSM-paramater tier #4613

merged 5 commits into from
Jun 17, 2025

Conversation

ScottGuymer
Copy link
Contributor

GitHub seems to have made a change to encoded_jit_config that has pushed it over the 4kb limit for standard tier of SSM.

We now must always use advanced tier of SSM config for encoded_jit_config.

Due to the size of the JIT config returned by github we muse use advanced SSM config
@ScottGuymer ScottGuymer requested a review from a team as a code owner June 16, 2025 16:56
@ScottGuymer ScottGuymer changed the title Update SSM to use advanced tiering for storing JIT config fix: Update SSM to use advanced tiering for storing JIT config Jun 16, 2025
Copy link
Member

@npalm npalm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ScottGuymer thax for creating. I just checked the multi-runner example in which the x64 linux is using JIT configuration as well. And indeed we are just on the safe side with paramaters from about 4000 bytes.

I would prefer we solve the prbolem in the warpper function in a generic way there were the lambda is sotring pramaters. Initialy we stayed away from adavanced by to avoid costs. As long the paramters are correctly cleared costs should not be a concert. With about 50K of ephemeral runners this should stay under the $5 a month.

Can you also make a note to this (https://github-aws-runners.github.io/terraform-aws-github-runner/configuration/) section that dynamaic generated paramaters are using based on size are stored in the advanced tier. To avoid costs it is important the ssm housekeeper is running, which is enabled by default but can be disabled.

Tested a deployment (multi-runner), the x64 linux runner is using JIT. Thos paramaters got indeed now created as ADVANCED and runners got started.

@npalm
Copy link
Member

npalm commented Jun 16, 2025

In case parameter size is exceeded the log wil show a log line like:

{"level":"WARN","message":"Ignoring error: ValidationException: Standard tier parameters support a maximum parameter value of 4096 characters. To create a larger parameter value, upgrade the parameter to use the advanced-parameter tier. For more information, see [https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-advanced-parameters.html"](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-advanced-parameters.html%22)"}

@ScottGuymer
Copy link
Contributor Author

Updates made, ready for re-review.

@ScottGuymer ScottGuymer requested a review from npalm June 17, 2025 11:51
@npalm
Copy link
Member

npalm commented Jun 17, 2025

Quick test based on multi runner examples:
image

@npalm npalm changed the title fix: Update SSM to use advanced tiering for storing JIT config fix: Dynamic set SSM-paramater tier Jun 17, 2025
@npalm npalm merged commit fef864f into main Jun 17, 2025
6 checks passed
@npalm npalm deleted the scott/fix_param_store branch June 17, 2025 12:56
npalm pushed a commit that referenced this pull request Jun 17, 2025
npalm added a commit that referenced this pull request Jun 17, 2025
fix incorrect merge  #4613

Co-authored-by: Scott Guymer <[email protected]>
npalm pushed a commit that referenced this pull request Jun 17, 2025
🤖 I have created a release *beep* *boop*
---


##
[6.5.9](v6.5.8...v6.5.9)
(2025-06-17)


### Bug Fixes

* Dynamic set SSM-paramater tier
[#4613](#4613)
([#4622](#4622))
([430c6e8](430c6e8))
@ScottGuymer
* **lambda:** bump the aws group in /lambdas with 6 updates
([ddbe7db](ddbe7db))
* **lambda:** bump the aws-powertools group in /lambdas with 4 updates
([91a06e9](91a06e9))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
@claytonolley
Copy link

I wish I would have seen this earlier, but I was only looking on the Issues page. 😭

Anyway, after a lot of fumbling around, I found that in AWS you can go into Parameter Store settings and turn on "Intelligent-Tiering" which will automatically make an Advanced Tier parameter if the size is over 4k.

@ScottGuymer
Copy link
Contributor Author

Yeah we also considered using intelligent tiering as an option when setting the value.

But one of our team-mates said they had experienced issues with this in the past so we erred on the side of caution and just making it work OOTB.

@andrewdibiasio6
Copy link

Parameter Store settings and turn on "Intelligent-Tiering"

Worked for me to get things unblocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants