This PR is intended to close #301.

1.1. Overview

Out of the total of 9 different web server + protocol combos, only 6 web server + protocol combos were feasible.

Below is a snapsot of web server + protocol combos that were built:

1.2. Server + Protocol Combos

As can be seen in the table above, 6 different web servers + protocol combos were built for the benchmarks:

1. Node.js (WS) code-named node-ws;
2. Node.js (HTTP/1.1 + SSE) code-named node-http1;
3. Node.js (HTTP/2 + SSE) code-named node-http2;
4. uWebSockets.js (WS) code-named uws-ws;
5. uWebSockets.js (HTTP/1.1 + SSE) code-named uws-http1;
6. Caddy + Node.js (HTTP/3 + SSE) code-named caddy-http3.

1.3. Server Memory Limit

On the development machine I used, which has 16GB of RAM installed, the memory limit for the node process defaults to 4GB.

For the benchmarks proper, each web server is started with a higher memory limit of 8GB. This is set in package.json via --max-old-space-size=8192 for all 6 web servers, but if running on a machine with higher installed memory, you can raise it as needed.

1.4. Server Test Data

I used faker.js to generate test data served by each web server.

To keep things fast, the generated test data are not stored in a database. Instead, everything is served from memory, with the exception of binary files used to simulate binary CoValue downloads and uploads. These are stored on disk at: public/downloads and public/uploads.

Each web server listens on https://localhost:3000/ and serves a default index page from the file public/client/{ws|http}/index.html, depending on whether the server’s transport is based on websockets or plain HTTP + SSE.

The default index page hosts all JavaScript code that will be driven by playwright and artillery load tests.

The default page includes a minified version of faker.js which is also used to create test data in the browser. The only downside is this caveat from the faker.js documentation:

Using the browser is great for experimenting 👍. However, due to all of the strings Faker uses to generate fake data, Faker is a large package. It’s > 5 MiB minified. Please avoid deploying the full Faker in your web app.

1.5. Server Network Conditions

With respect to simulating various network conditions (I, II, III, IV), I explored:

OS-level throttling using third-party tools like:

• Network Link Conditioner for MacOS
• NetLimiter for Windows

Or doing the throttling at the browser-level using:

• Chrome DevTools

I eventually came across Shopify’s toxiproxy which has first-class support for programmatic use and would have been perfect for our needs. Unfortunately, toxiproxy was designed to proxy TCP traffic, meaning it would work for TCP-based protocols like HTTP/1.1 and HTTP/2, but not for HTTP/3, which is based on UDP (QUIC), so I had to abandon the idea.

I was able to achieve the desired OS-level programmatic throttling using a combination of macOS’ pfctl (packet filter control) and dnctl (dummynet control) CLI tools. Note that some programmatic invocations of pfctl and dnctl by scripts used in the benchmarks will require sudo privileges.

The only downside is that the benchmarks would need to be adapted to use netem (network emulator) and tc (traffic control), if there’s a need to execute it on a server-class machine running Linux.

1.6. Summary

This is a high-level summary of the different components that make up the microbenchmark.

Please refer to the included README for additional details.

/claim #301