Browsers are made up of many components that can be fuzzed in different ways. Some libraries such as image decoders for example can be fuzzed independently of the browser very effectively using tools such as libFuzzer at a very high iteration rate. Components such as the JavaScript engine can be built and fuzzed with custom tools again without additional overhead of the browser. However some components and the interaction between components can only be tested with the full browser and the overhead that comes with it. Because of this Grizzly is built with focus on reliability and not necessarily performance (raw iterations per second) since there will always be the large performance overhead of running with a full browser.

So how fast can it go? Well that depends on a few things.

First is the build, opt builds are fast but can miss serious issues, ASan-opt builds are slower but can find serious memory corruption issues. Debug builds are also slow but can find assertions that can in some cases be serious logic issues. ASan-debug builds are very slow. So choosing the right build is important and when in doubt use ASan-opt to make the iterations count.

Next is fuzzers. If the fuzzer takes a long time to generate a test case that will block unless it is handled properly. Generation of test cases in parallel in most cases is the correct way to handle this but does add overhead. Which leads to machine resources.

Overloading the machine is going to slow down the execution of the browser and cause test cases to hang. This is bad, avoid this! Typically running two physical cores per Grizzly instance (2:1) is a good place to start. This is because each instance is going to have a busy multi-process browser running. To scale locally multiple Grizzly instances can be run in parallel but be sure to start with the 2:1 rule. Of course system memory is important to avoid OOMs and paging. Typically 8GBs per instance is a safe assumption when running with an ASan-opt build but you might be able to get away with this less.

The contents of each test case will also determine how busy the browser is. Do lots in a test case make each iteration count. Try to have each test case close itself to avoid the need to wait for a timeout or for the harness to close it (if you are using the harness).

Use the harness. It loads and from the harness tab opens each test case in a new tab. When the test case closes the harness will open the next test case. If the test case does not close itself the harness will close it after the adapter defined test duration Adapter.TEST_DURATION. This is much faster than closing the browser after each test and much more reliable than redirecting.

Avoid relaunching the browser if possible. Launching the browser is slow and resource intensive and should be done as little as possible but sometimes it is necessary. Depending on the fuzzer the browser memory usage may grow overtime or browser performance may degrade, it shouldn't but it does (if this happens and an actionable browser bug can be logged that's great!). Typically relaunching once every few hundred iterations is tolerable but sometimes a relaunch after a few dozen is required. It depends on the fuzzer, find what works for you.

Making optimal use of the machine's resources will take some trial and error. Parallel Grizzly instances can be added and removed until the optimal number is found. Again when scaling locally a good starting point is 2 physical cores to 1 Grizzly instance (2:1).

These results are from running the Grizzly no-op adapter on a 20 core (+20 hyper-thread) machine with an ASan-opt build. This adapter does basically nothing and in a normal use case a test case would take some time to run so iteration rates will be lower. To generate aggregate reports use grizzly.status --system-report. Note iteration data is only collected every 60 seconds.

This example shows a single instance running at an iteration rate of 3.61 iterations/second.

Iterations : 435
      Rate : 1 @ 3.61
   Results : 0
   Runtime : 0:02:00
CPU & Load : 40 @ 5.6% (2.95, 2.74, 2.88)
    Memory : 59.4GB of 62.6GB free
      Disk : 358.6GB of 548.6GB free

This example shows 10 instances running in parallel with a total iteration rate of 23.02 iterations/second. With 2.47 being the highest rate and 2.08 being the lowest rate seen. The load is at 20.11 which is about where we want it. Note that the rate is not 10x of the single instance so removing a few instances may not hurt performance and could free up some resources.

Iterations : 2771 (298, 250)
      Rate : 10 @ 23.02 (2.47, 2.08)
   Results : 0
   Runtime : 0:20:03
CPU & Load : 40 @ 57.3% (20.11, 10.13, 5.64)
    Memory : 43.7GB of 62.6GB free
      Disk : 358.3GB of 548.6GB free