ci: cache embedded postgres downloaded binaries #18477

johnstcn · 2025-06-20T17:14:11Z

Embedded Postgres downloads binaries from ${REPO}/io/zonky/test/postgres/embedded-postgres-binaries-$OS-$ARCH/$VERSION/embedded-postgres-binaries-$OS-$ARCH-$VERSION.jar to CachePath:

cdr-mbp-jmqxfj0746:coder cian$ ls -l /private/tmp/nix-shell.BRKpjK/coder-test-postgres/cache
total 29536
-rw------- 1 cian wheel 30240852 Jun 20 18:09 embedded-postgres-binaries-darwin-arm64v8-16.4.0.txz

Example link for Darwin arm64: https://repo.maven.apache.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-darwin-arm64v8/16.4.0/

Making this a variable and adding it to be cached on merge to main.

Successful runs:

Copilot

Pull Request Overview

This PR adds support for caching downloaded Embedded Postgres binaries by introducing a --cache flag in the helper script, updating CI workflows to populate and persist that cache across runs, and providing a composite action to upload the cache on main.

Introduce cachePath flag and default directory logic in main.go
Update CI steps to create cache directories and pass -cache to the script
Add embedded-pg-cache composite action for persisting the cache

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
scripts/embedded-pg/main.go	Added `--cache` flag, default cache directory, and `MkdirAll`
.github/workflows/ci.yaml	Created cache dirs, passed `-cache` flag, and invoked upload
.github/actions/embedded-pg-cache/action.yml	New composite action to save the cache via `actions/cache`

Comments suppressed due to low confidence (5)

scripts/embedded-pg/main.go:17

The new --cache flag should be documented in the project README or script usage section to ensure users know how to leverage caching.

	flag.StringVar(&cachePath, "cache", "", "Optional custom path for embedded postgres binaries")

.github/workflows/ci.yaml:500

[nitpick] This Windows-specific path is duplicated later; consider defining a variable for the cache path to reduce duplication and ease future updates.

            mkdir -p "C:/temp/embedded-pg-cache"

.github/workflows/ci.yaml:500

mkdir -p may not be available on Windows runners by default; ensure the shell supports this or use a PowerShell-native command like New-Item -ItemType Directory -Force.

            mkdir -p "C:/temp/embedded-pg-cache"

scripts/embedded-pg/main.go:27

New fallback logic for cachePath isn't covered by existing tests; consider adding an integration test to verify caching behavior is applied correctly.

	if cachePath == "" {

.github/actions/embedded-pg-cache/action.yml:15

[nitpick] Pinning to a specific commit SHA can make upgrades harder; consider using the official version tag (actions/cache@v4) instead of a commit hash.

      uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3

scripts/embedded-pg/main.go

Co-authored-by: Copilot <[email protected]>

.github/workflows/ci.yaml

… cj/cache-embedded-postgres

johnstcn · 2025-06-23T11:40:33Z

.github/actions/setup-embedded-pg-cache-paths/action.yml

+      shell: bash
+      run: |
+        set -e
+        mkdir -p "$EMBEDDED_PG_CACHE_DIR"


This ends up being C:\actions-runner\_temp\embedded-pg-cache

johnstcn · 2025-06-24T20:53:29Z

scripts/embedded-pg/main.go

+	// Troubleshooting: list files in cachePath
+	if err := filepath.Walk(cachePath, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		switch {
+		case info.IsDir():
+			log.Printf("D: %s", path)
+		case info.Mode().IsRegular():
+			log.Printf("F: %s [%s] (%d bytes) %s", path, info.Mode().String(), info.Size(), info.ModTime().Format(time.RFC3339))
+		default:
+			log.Printf("Other: %s [%s] %s", path, info.Mode(), info.ModTime().Format(time.RFC3339))
+		}
+		return nil
+	}); err != nil {
+		log.Printf("Failed to list files in cachePath %s: %v", cachePath, err)


review: I think it makes sense to leave this in; I found it useful for troubleshooting before I knew about the wush trick.

hugodutka · 2025-06-25T10:55:35Z

.github/workflows/ci.yaml

@@ -571,6 +582,14 @@ jobs:
        with:
          cache-key: ${{ steps.download-cache.outputs.cache-key }}

+      - name: Upload Embedded Postgres Cache
+        uses: ./.github/actions/embedded-pg-cache/upload
+        # We only use the embedded Postgres cache on macOS and Windows runners.


why not everywhere?

I can see an argument for using it on Linux too, but I'd suggest keeping it out of scope of this PR as it's a wider discussion.

Also, Docker is a nice abstraction for running a throw-away database.

ci: cache embedded postgres downloaded binaries

f38ff28

johnstcn requested review from hugodutka and Copilot June 20, 2025 17:14

johnstcn self-assigned this Jun 20, 2025

Copilot AI reviewed Jun 20, 2025

View reviewed changes

scripts/embedded-pg/main.go Outdated Show resolved Hide resolved

log.fatal errors instead of panic

dc585db

Co-authored-by: Copilot <[email protected]>

johnstcn commented Jun 20, 2025

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

hugodutka reviewed Jun 20, 2025

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

johnstcn added 4 commits June 20, 2025 20:16

use step output for cache path

f66007c

Merge remote-tracking branch 'origin/cj/cache-embedded-postgres' into…

8cfd0ff

… cj/cache-embedded-postgres

praise log

0196ab6

testing

0e5366c

johnstcn commented Jun 23, 2025

View reviewed changes

johnstcn added 9 commits June 23, 2025 13:51

fix path

3d3aaaa

use EMBEDDED_PG_CACHE_DIR

d268942

make embedded-pg less panicky

e6a7997

troubleshoot cache dir

4a27039

maybe this???

4c4f5e1

fix upload cache destination

9ec2b1b

also show file modtime

1554d6b

i heart actionlint

5b81b29

ok this should work now

4d69612

johnstcn commented Jun 24, 2025

View reviewed changes

johnstcn requested a review from hugodutka June 24, 2025 20:55

hugodutka approved these changes Jun 25, 2025

View reviewed changes

johnstcn merged commit 42fd1c1 into main Jun 25, 2025
37 checks passed

johnstcn deleted the cj/cache-embedded-postgres branch June 25, 2025 11:00

github-actions bot locked and limited conversation to collaborators Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: cache embedded postgres downloaded binaries #18477

ci: cache embedded postgres downloaded binaries #18477

Uh oh!

johnstcn commented Jun 20, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnstcn Jun 23, 2025

Uh oh!

johnstcn Jun 24, 2025

Uh oh!

hugodutka Jun 25, 2025

Uh oh!

johnstcn Jun 25, 2025

Uh oh!

Uh oh!

Uh oh!

ci: cache embedded postgres downloaded binaries #18477

ci: cache embedded postgres downloaded binaries #18477

Uh oh!

Conversation

johnstcn commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnstcn Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

johnstcn Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

hugodutka Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

johnstcn Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

johnstcn commented Jun 20, 2025 •

edited

Loading