00cc05f

[WPE][browserperfdash-benchmark] Allow to run run-benchmark plans subtest-by-subtest
https://bugs.webkit.org/show_bug.cgi?id=295050

Reviewed by Nikolas Zimmermann.

On the RPi4 32-bit bots, JetStream2 frequently crashes or times out before completing,
which prevents any results from being reported because the current runner only uploads
data if all subtests finish successfully.

We have experimented with running selected subtest sets, but the flakiness remains:
some subtests pass while others fail inconsistently, which makes very difficult to
select a working set of subtests.

This patch implements on browserperfdash-benchmark the ability to run a given benchmark
plan subtest-by-subtest, so it only runs a subtest at a time.
If a subtest fails, the runner skips it and proceeds to the next. Partial results from the
passing subtests are then uploaded to the dashboard, improving visibility into progress
and regressions even when the full benchmark cannot complete.

To differenciate the standard benchmark plan between the one that is run subtest-by-subtest
the string `-split-subtests` is appended to the end of the benchmark plan name. So, to
to run the plan jetstream2 subtest-by-subtest it should be specified to run a virtual
benchmark plan named `jetstream2-split-subtests`.
Since the benchmark plan name is visible on the dashboard, is also possible to differentiate
there from the complete jetstream2 vs the one that was run subtest-by-subtest

All the benchmark plans that support subtests can be specified this way (not only JetStream2),
and the list of those is visible when the flag `--list-plans` is passed to the runner.

* Tools/Scripts/webkitpy/browserperfdash/browserperfdash_runner.py:
(BrowserPerfDashRunner.__init__):
(BrowserPerfDashRunner._parse_config_file):
(BrowserPerfDashRunner._get_plan_version_hash):
(BrowserPerfDashRunner._get_benchmark_runner_split_subtess_plans):
(BrowserPerfDashRunner._run_benchmark_runner_plan_split_subtests):
(BrowserPerfDashRunner._run_plan):
(BrowserPerfDashRunner.run):

Canonical link: https://commits.webkit.org/296877@main

3b430de