chore: run macOS, windows, and race tests with Postgres in CI #15520

hugodutka · 2024-11-14T16:03:36Z

This PR is the second in a series aimed at closing #15109.

Changes

adds scripts/embedded-pg/main.go, which can start a native Postgres database. This is used to set up PG on Windows and macOS, as these platforms don't support Docker in Github Actions.
runs the test-go-pg job on macOS and Windows too
adds the test-go-race-go job, which runs race tests with Postgres on Linux

ethanndickson

LGTM but It'd be good to hear from Dean who has a lot more experience working with our CI

ethanndickson · 2024-11-18T01:47:23Z

.github/workflows/ci.yaml

+          api-key: ${{ secrets.DATADOG_API_KEY }}
+
+  # NOTE: this could instead be defined as a matrix strategy, but we want to
+  # temporarily allow windows tests to fail. Using a matrix strategy here makes


I see the Windows tests are broken, I'm not sure if we want to merge a CI suite that often fails, even if it's not required (could slightly obscure real issues + not great optics). Perhaps it should just get skipped/commented out until we're able to improve the reliability?

Since we do have the Windows tests covered by in memory for now, maybe these should be disabled for now to avoid ❌ appearing everywhere. Are there any issues tracking the windows tests breaking in postgres?

Are there any issues tracking the windows tests breaking in postgres?

I just added #15560.

for now, maybe these should be disabled for now to avoid ❌ appearing everywhere

Makes sense. I'll postpone merging this PR until Windows tests are fixed. I'll try to get it done this week.

deansheather · 2024-11-18T01:55:09Z

.github/workflows/ci.yaml

-    runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || 'ubuntu-latest' }}
-    needs:
-      - changes
+    runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'macos-latest-xlarge' || matrix.os }}


Holy mother of github actions interpolation

We also have this on go-test 😭

deansheather · 2024-11-18T01:58:46Z

scripts/embedded-pg/main.go

+	// We execute these queries instead of using the embeddedpostgres
+	// StartParams because it doesn't work on Windows. The library
+	// seems to have a bug where it sends malformed parameters to
+	// pg_ctl. It encloses each parameter in single quotes, which
+	// Windows can't handle.


Can we open a PR upstream to fix this bug?

Sure. I opened fergusstrange/embedded-postgres#145 and I could submit a PR later.

Can you add the issue tracking link to this comment?

.github/workflows/ci.yaml

hugodutka · 2024-11-18T11:47:31Z

We're going to postpone merging this PR until Windows tests are fixed. #15560

Patches tests that caused Windows Postgres CI in #15520 to consistently fail. I tested this by temporarily adding Postgres Windows CI to this PR. However, I reverted those changes to merge them with #15520. For reference, here's [a passing CI run](https://github.com/coder/coder/actions/runs/11918816662/job/33219786238) from an earlier commit. **Note:** Although Windows tests now pass, they remain quite flaky. I recommend running Postgres Windows CI to gather data on these flakes, but I don’t think it should be a required job just yet.

hugodutka · 2024-12-02T14:52:51Z

@deansheather @ethanndickson The Windows Postgres tests seem stable now. I made one major change in this PR to fix them: Windows now sets up a RAM disk for Postgres. I found that the flakiness was due to timing-sensitive tests failing because the suite ran slowly on Windows. The root cause was IO performance on GitHub-hosted Windows runners. Using a RAM disk bypasses this issue, and test times are back to normal.

I’ve also made the Windows PG tests required since they’re no longer that flaky - I see an odd flake every couple of runs. I’d appreciate a second look before merging this into main. Thanks!

deansheather · 2024-12-02T21:46:57Z

.github/actions/setup-imdisk/action.yaml

+        curl -L -o files.cab https://github.com/coder/imdisk-artifacts/raw/92a17839ebc0ee3e69be019f66b3e9b5d2de4482/files.cab
+        curl -L -o install.bat https://github.com/coder/imdisk-artifacts/raw/92a17839ebc0ee3e69be019f66b3e9b5d2de4482/install.bat


Sucks that people still use sourceforge lol

That's Windows for you

deansheather · 2024-12-02T21:49:24Z

scripts/embedded-pg/main.go

+	// We execute these queries instead of using the embeddedpostgres
+	// StartParams because it doesn't work on Windows. The library
+	// seems to have a bug where it sends malformed parameters to
+	// pg_ctl. It encloses each parameter in single quotes, which
+	// Windows can't handle.


Can you add the issue tracking link to this comment?

We have an effort underway to replace `dbmem` (#15109), and consequently we've begun running our full test-suite (with Postgres) on all supported OSs - Windows, MacOS, and Linux, since #15520. Since this change, we've seen a marked decrease in the success rate of our builds on `main` (note how the Windows/MacOS failures account for the vast majority of failed builds): ![image](https://github.com/user-attachments/assets/a02c15b7-037d-428a-a600-2aed60553ac0) We're still investigating why these OSs are a lot less reliable. It's likely that the VMs on which the builds are run have different characteristics from our Ubuntu runners such as disk I/O, network latency, or something else. **In the meantime, we need to start trusting CI failures in `main` again, as the current failures are too noisy / vague for us to correct.** We've also considered hosting our own runners where possible so we can get OS-level observability to rule out some possibilities. See the [meeting notes](https://www.notion.so/coderhq/CI-Investigation-Call-Notes-17dd579be59280d8897cc9fe4bb46695?pvs=6&utm_content=17dd579b-e592-80d8-897c-c9fe4bb46695&utm_campaign=T1ZPT2FL0&n=slack&n=slack_link_unfurl) where we linked into this for more detail. This PR introduces several changes: 1. Moves the full test-suite with Postgres on Windows/MacOS to the `nightly-gauntlet` workflow tradeoff: this means that any regressions may be more difficult to discover since we merge to main several times a day 2. Run only the CLI test-suite on each PR / merge to `main` on Windows/MacOS 3. `test-go` is still running the full test-suite against all OSs (including the CLI ones), but will soon be removed once #15109 is completed since it uses `dbmem` 4. Changes `nightly-gauntlet` to run at 4AM: we've seen several instances of the runner being stopped externally, and we're _guessing_ this may have something to do with the midnight UTC execution time, when other cron jobs may run 5. Removes the existing `nightly-gauntlet` jobs since they haven't passed in a long time, indicating that nobody cares enough to fix them and they don't provide diagnostic value; we can restore them later if necessary I've manually run both these new workflows successfully: - `ci`: https://github.com/coder/coder/actions/runs/12825874176/job/35764724907 - `nightly-gauntlet`: https://github.com/coder/coder/actions/runs/12825539092 --------- Signed-off-by: Danny Kopping <[email protected]> Co-authored-by: Muhammad Atif Ali <[email protected]>

github-actions bot assigned hugodutka Nov 14, 2024

hugodutka force-pushed the hugodutka/enable-pg-ci-macos-windows branch 2 times, most recently from 0733229 to b1c3974 Compare November 15, 2024 12:50

hugodutka changed the title ~~chore: run Postgres tests on macos and windows in CI~~ chore: run macOS, windows, and race tests with Postgres in CI Nov 15, 2024

hugodutka requested a review from ethanndickson November 15, 2024 14:43

hugodutka marked this pull request as ready for review November 15, 2024 16:38

ethanndickson reviewed Nov 18, 2024

View reviewed changes

ethanndickson requested a review from deansheather November 18, 2024 01:52

deansheather approved these changes Nov 18, 2024

View reviewed changes

hugodutka mentioned this pull request Nov 18, 2024

bug: some Windows Postgres tests are failing #15560

Closed

hugodutka mentioned this pull request Nov 20, 2024

chore: fix windows postgres tests #15593

Merged

hugodutka force-pushed the hugodutka/enable-pg-ci-macos-windows branch 4 times, most recently from b3e3fac to 2739223 Compare November 26, 2024 12:57

hugodutka added 13 commits December 2, 2024 14:50

run pg tests on macos and windows in CI

5686771

set default encoding on embedded pg

fede550

allow test-go-pg to fail on windows

cd9b4d1

add pg race test to ci

e797183

require test-go-race-pg to pass

fb23890

trigger ci

4e3421f

remove the fail-fast: false strategy from test-go-pg

46a8c60

ensure we received results from all matrix jobs

dcfdc07

simplify the required check

5d79aea

runners fallback

1c99f2e

make the windows pg test required

de40386

use a temporary folder on the D: drive in Windows CI for postgres

988c893

test out imdisk (ramdisk) for windows

e7463ec

hugodutka added 9 commits December 2, 2024 14:50

try fix

b41c523

another fix

8a61183

fix

d319138

try ramdisk with postgres

c63786f

create an action for setting up imdisk

ed13154

revert lint job changes

4b6f183

clarity and grammar

9c7e240

fmt

4969a23

remove empty inputs

160e77c

hugodutka force-pushed the hugodutka/enable-pg-ci-macos-windows branch from f1394a8 to 160e77c Compare December 2, 2024 14:50

deansheather approved these changes Dec 2, 2024

View reviewed changes

add link to the StartParameters issue

f87261f

ethanndickson approved these changes Dec 3, 2024

View reviewed changes

hugodutka merged commit c7c35ef into main Dec 3, 2024
31 of 33 checks passed

hugodutka deleted the hugodutka/enable-pg-ci-macos-windows branch December 3, 2024 12:33

dannykopping mentioned this pull request Jan 17, 2025

chore: improve CI reliability #16169

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: run macOS, windows, and race tests with Postgres in CI #15520

chore: run macOS, windows, and race tests with Postgres in CI #15520

Uh oh!

hugodutka commented Nov 14, 2024 •

edited

Loading

Uh oh!

ethanndickson left a comment

Uh oh!

ethanndickson Nov 18, 2024 •

edited

Loading

Uh oh!

deansheather Nov 18, 2024

Uh oh!

hugodutka Nov 18, 2024

Uh oh!

deansheather Nov 18, 2024

Uh oh!

ethanndickson Nov 18, 2024

Uh oh!

deansheather Nov 18, 2024

Uh oh!

hugodutka Nov 18, 2024

Uh oh!

deansheather Dec 2, 2024

Uh oh!

Uh oh!

hugodutka commented Nov 18, 2024

Uh oh!

hugodutka commented Dec 2, 2024 •

edited

Loading

Uh oh!

deansheather Dec 2, 2024

Uh oh!

hugodutka Dec 3, 2024

Uh oh!

deansheather Dec 2, 2024

Uh oh!

Uh oh!

Uh oh!

		curl -L -o files.cab https://github.com/coder/imdisk-artifacts/raw/92a17839ebc0ee3e69be019f66b3e9b5d2de4482/files.cab
		curl -L -o install.bat https://github.com/coder/imdisk-artifacts/raw/92a17839ebc0ee3e69be019f66b3e9b5d2de4482/install.bat

chore: run macOS, windows, and race tests with Postgres in CI #15520

chore: run macOS, windows, and race tests with Postgres in CI #15520

Uh oh!

Conversation

hugodutka commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

ethanndickson left a comment

Choose a reason for hiding this comment

Uh oh!

ethanndickson Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hugodutka commented Nov 18, 2024

Uh oh!

hugodutka commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hugodutka commented Nov 14, 2024 •

edited

Loading

ethanndickson Nov 18, 2024 •

edited

Loading

hugodutka commented Dec 2, 2024 •

edited

Loading