Note: Steps 1 and 2 can be skipped if you are not updating screenshots. By default, screenshots use darwin but linux is what is used in our CI/CD.

https://docs.docker.com/get-docker/

docker run -v $PWD:/tests -w /tests --rm --ipc=host -it mcr.microsoft.com/playwright:v1.24.0-focal /bin/bash

npm install

npx playwright install

If you are running only free specs, this following will suffice.

create/update a .env file with the following:

E2E_TEST_ACCOUNT_PASSWORD=<arbitrary password>

If you are running premium tests as well, you will need the following.

create/update a .env file with the following:

E2E_TEST_ACCOUNT_PREMIUM=<your_premium_account_email>
E2E_TEST_ACCOUNT_PASSWORD=<your_premium_account_password>

Any free account created during the initial setup of tests will also use E2E_TEST_ACCOUNT_PASSWORD. If you do not want to use a personal premium account, reach out to Luke for relay-team premium account details.

npm run test:e2e

By default, npm run test:e2e will run the tests on https://stage.fxprivaterelay.nonprod.cloudops.mozgcp.net/.

You can also run tests locally, on our dev server (https://dev.fxprivaterelay.nonprod.cloudops.mozgcp.net/), and in production (https://relay.firefox.com/). You can find the commands here, or you can run E2E_TEST_ENV=<env (prod, dev, stage)> npx playwright test. To view the tests live in the browser, you can add --headed to the end of the command. See https://playwright.dev/docs/test-cli for more flags.

Our github actions workflows can be found here, . You can run the tests against different branches.

When you run tests for the first time locally, you will run into an error for the tests that rely on screenshots, stating that a snapshot does not exist. Here is an example.

Error: A snapshot doesn't exist at example.spec.ts-snapshots/example-test-1-chromium-darwin.png, writing actual.

This is because playwright needs to create an image initially. On the following runs, it will compare that a screenshot of the respective element matches the one added initially. Do not push your local images into the repo, the only ones that are needed for CI end in linux.

Our runs a subset of the entire e2e test suite everyday. This subset of tests focuses on critical tests for free and premium users for the overall health of the relay application. To add a test into the healthcheck CI, add @health_check into the title of your test or test group. See the following as an example,

test.describe("Subscription flows @health_check", ...)

To run the health check manually, go to , click run workflow, and check off "enable health check" before clicking "run workflow".