ci: cache embedded postgres downloaded binaries #18477
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| name: "Download Embedded Postgres Cache" | ||
| description: | | ||
| Downloads the embedded postgres cache and outputs today's cache key. | ||
| A PR job can use a cache if it was created by its base branch, its current | ||
| branch, or the default branch. | ||
| https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache | ||
| outputs: | ||
| cache-key: | ||
| description: "Today's cache key" | ||
| value: ${{ steps.vars.outputs.cache-key }} | ||
| inputs: | ||
| key-prefix: | ||
| description: "Prefix for the cache key" | ||
| required: true | ||
| cache-path: | ||
| description: "Path to the cache directory" | ||
| required: true | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Get date values and cache key | ||
| id: vars | ||
| shell: bash | ||
| run: | | ||
| export YEAR_MONTH=$(date +'%Y-%m') | ||
| export PREV_YEAR_MONTH=$(date -d 'last month' +'%Y-%m') | ||
| export DAY=$(date +'%d') | ||
| echo "year-month=$YEAR_MONTH" >> $GITHUB_OUTPUT | ||
| echo "prev-year-month=$PREV_YEAR_MONTH" >> $GITHUB_OUTPUT | ||
| echo "cache-key=${{ inputs.key-prefix }}-${YEAR_MONTH}-${DAY}" >> $GITHUB_OUTPUT | ||
|
|
||
| # By default, depot keeps caches for 14 days. This is plenty for embedded | ||
| # postgres, which changes infrequently. | ||
| # https://depot.dev/docs/github-actions/overview#cache-retention-policy | ||
| - name: Download embedded Postgres cache | ||
| uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3 | ||
| with: | ||
| path: ${{ inputs.cache-path }} | ||
| key: ${{ steps.vars.outputs.cache-key }} | ||
| # > If there are multiple partial matches for a restore key, the action returns the most recently created cache. | ||
| # https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#matching-a-cache-key | ||
| # The second restore key allows non-main branches to use the cache from the previous month. | ||
| # This prevents PRs from rebuilding the cache on the first day of the month. | ||
| # It also makes sure that once a month, the cache is fully reset. | ||
| restore-keys: | | ||
| ${{ inputs.key-prefix }}-${{ steps.vars.outputs.year-month }}- | ||
| ${{ github.ref != 'refs/heads/main' && format('{0}-{1}-', inputs.key-prefix, steps.vars.outputs.prev-year-month) || '' }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| name: "Upload Embedded Postgres Cache" | ||
| description: Uploads the embedded Postgres cache. This only runs on the main branch. | ||
| inputs: | ||
| cache-key: | ||
| description: "Cache key" | ||
| required: true | ||
| cache-path: | ||
| description: "Path to the cache directory" | ||
| required: true | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Upload Embedded Postgres cache | ||
| if: ${{ github.ref == 'refs/heads/main' }} | ||
| uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3 | ||
| with: | ||
| path: ${{ inputs.cache-path }} | ||
| key: ${{ inputs.cache-key }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: "Setup Embedded Postgres Cache Paths" | ||
| description: Sets up a path for cached embedded postgres binaries. | ||
| outputs: | ||
| embedded-pg-cache: | ||
| description: "Value of EMBEDDED_PG_CACHE_DIR" | ||
| value: ${{ steps.paths.outputs.embedded-pg-cache }} | ||
| cached-dirs: | ||
| description: "directories that should be cached between CI runs" | ||
| value: ${{ steps.paths.outputs.cached-dirs }} | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - name: Override Go paths | ||
| id: paths | ||
| uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7 | ||
| with: | ||
| script: | | ||
| const path = require('path'); | ||
|
|
||
| // RUNNER_TEMP should be backed by a RAM disk on Windows if | ||
| // coder/setup-ramdisk-action was used | ||
| const runnerTemp = process.env.RUNNER_TEMP; | ||
| const embeddedPgCacheDir = path.join(runnerTemp, 'embedded-pg-cache'); | ||
| core.exportVariable('EMBEDDED_PG_CACHE_DIR', embeddedPgCacheDir); | ||
| core.setOutput('embedded-pg-cache', embeddedPgCacheDir); | ||
| const cachedDirs = `${embeddedPgCacheDir}`; | ||
| core.setOutput('cached-dirs', cachedDirs); | ||
|
|
||
| - name: Create directories | ||
| shell: bash | ||
| run: | | ||
| set -e | ||
| mkdir -p "$EMBEDDED_PG_CACHE_DIR" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,31 +4,43 @@ package main | |
| import ( | ||
| "database/sql" | ||
| "flag" | ||
| "log" | ||
| "os" | ||
| "path/filepath" | ||
| "time" | ||
|
|
||
| embeddedpostgres "github.com/fergusstrange/embedded-postgres" | ||
| ) | ||
|
|
||
| func main() { | ||
| var customPath string | ||
| var cachePath string | ||
| flag.StringVar(&customPath, "path", "", "Optional custom path for postgres data directory") | ||
| flag.StringVar(&cachePath, "cache", "", "Optional custom path for embedded postgres binaries") | ||
| flag.Parse() | ||
|
|
||
| postgresPath := filepath.Join(os.TempDir(), "coder-test-postgres") | ||
| if customPath != "" { | ||
| postgresPath = customPath | ||
| } | ||
| if err := os.MkdirAll(postgresPath, os.ModePerm); err != nil { | ||
| log.Fatalf("Failed to create directory %s: %v", postgresPath, err) | ||
| } | ||
| if cachePath == "" { | ||
| cachePath = filepath.Join(postgresPath, "cache") | ||
| } | ||
| if err := os.MkdirAll(cachePath, os.ModePerm); err != nil { | ||
| log.Fatalf("Failed to create directory %s: %v", cachePath, err) | ||
| } | ||
|
|
||
| ep := embeddedpostgres.NewDatabase( | ||
| embeddedpostgres.DefaultConfig(). | ||
| Version(embeddedpostgres.V16). | ||
| BinariesPath(filepath.Join(postgresPath, "bin")). | ||
| BinaryRepositoryURL("https://repo.maven.apache.org/maven2"). | ||
| DataPath(filepath.Join(postgresPath, "data")). | ||
| RuntimePath(filepath.Join(postgresPath, "runtime")). | ||
| CachePath(cachePath). | ||
| Username("postgres"). | ||
| Password("postgres"). | ||
| Database("postgres"). | ||
|
|
@@ -38,8 +50,27 @@ func main() { | |
| ) | ||
| err := ep.Start() | ||
| if err != nil { | ||
| log.Fatalf("Failed to start embedded postgres: %v", err) | ||
| } | ||
|
|
||
| // Troubleshooting: list files in cachePath | ||
| if err := filepath.Walk(cachePath, func(path string, info os.FileInfo, err error) error { | ||
| if err != nil { | ||
| return err | ||
| } | ||
| switch { | ||
| case info.IsDir(): | ||
| log.Printf("D: %s", path) | ||
| case info.Mode().IsRegular(): | ||
| log.Printf("F: %s [%s] (%d bytes) %s", path, info.Mode().String(), info.Size(), info.ModTime().Format(time.RFC3339)) | ||
| default: | ||
| log.Printf("Other: %s [%s] %s", path, info.Mode(), info.ModTime().Format(time.RFC3339)) | ||
| } | ||
| return nil | ||
| }); err != nil { | ||
| log.Printf("Failed to list files in cachePath %s: %v", cachePath, err) | ||
|
Comment on lines
+56
to
+71
There was a problem hiding this comment. review: I think it makes sense to leave this in; I found it useful for troubleshooting before I knew about the wush trick. |
||
| } | ||
|
|
||
| // We execute these queries instead of using the embeddedpostgres | ||
| // StartParams because it doesn't work on Windows. The library | ||
| // seems to have a bug where it sends malformed parameters to | ||
|
|
@@ -58,21 +89,21 @@ func main() { | |
| } | ||
| db, err := sql.Open("postgres", "postgres://postgres:[email protected]:5432/postgres?sslmode=disable") | ||
| if err != nil { | ||
| log.Fatalf("Failed to connect to embedded postgres: %v", err) | ||
| } | ||
| for _, query := range paramQueries { | ||
| if _, err := db.Exec(query); err != nil { | ||
| log.Fatalf("Failed to execute setup query %q: %v", query, err) | ||
| } | ||
| } | ||
| if err := db.Close(); err != nil { | ||
| log.Fatalf("Failed to close database connection: %v", err) | ||
| } | ||
| // We restart the database to apply all the parameters. | ||
| if err := ep.Stop(); err != nil { | ||
| log.Fatalf("Failed to stop embedded postgres after applying parameters: %v", err) | ||
| } | ||
| if err := ep.Start(); err != nil { | ||
| log.Fatalf("Failed to start embedded postgres after applying parameters: %v", err) | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This ends up being
C:\actions-runner\_temp\embedded-pg-cache