## File: bench/README.md # websocketd Performance Benchmarks Automated performance benchmarking for websocketd using [k6](https://k6.io/). ## Quick Start ```bash # Install k6 (macOS) brew install k6 # Build and benchmark go build -o websocketd . ./bench/run.sh ./websocketd open bench/results/report.html ``` ## Benchmarking Any Version The benchmark tool works with any websocketd binary — use it to compare versions: ```bash # Benchmark current build ./bench/run.sh ./websocketd --output=bench/results/v0.5.0 # Benchmark an older release ./bench/run.sh /usr/local/bin/websocketd-0.4.1 --output=bench/results/v0.4.1 ``` ## Scenarios | Scenario | What It Measures | |----------|-----------------| | `echo_latency` | Round-trip time: 1 connection, 1000 sequential send/recv | | `echo_throughput` | Max msgs/sec: 1 connection, fire-hose for 10s | | `connection_storm` | Concurrent connect overhead: 10/100/500 VUs | | `connection_churn` | Process lifecycle: 200 serial connect/send/close cycles | | `sustained_load` | Steady state: 50 connections, continuous traffic for 30s | | `binary_*` | Binary mode throughput: 1KB, 10KB, 64KB payloads | | `backpressure` | Slow consumer: fast sender vs delayed echo backend | ## Metrics ### Client-Side (k6) - **Latency**: p50, p95, p99, avg, max round-trip time (ms) - **Throughput**: messages/sec, connections/sec - **Binary**: MB/sec per payload size - **Backpressure**: delivery ratio (received/sent) ### Server-Side (ps sampling) - **RSS**: resident set size (KB) sampled every 500ms - **FDs**: open file descriptors sampled every 500ms ## Output Each scenario runs 3 times by default (`--runs=N` to change) and the results are merged into a single median before reporting — see "Repeated Runs" below. ``` bench/results/ meta.json # Version, git hash, timestamp, OS *_summary.json # Median across all runs of this scenario *_summary.run{N}.json # k6 summary for one run *_k6.run{N}.json # k6 detailed output for one run *_server.ndjson # The run whose peak RSS is closest to the median *_server.run{N}.ndjson # Server metrics for one run report.html # Visual report (open in browser) benchmark-data.json # CI regression detection format, from the median ``` ## Running a Subset ```bash ./bench/run.sh ./websocketd --scenarios=echo_latency,echo_throughput ``` ## Repeated Runs A single k6 run on shared hardware (like GitHub Actions runners) can vary 20%+ between otherwise-identical commits, which used to produce false regression alerts. `run.sh` now repeats each scenario 3 times (`--runs=N` to change; `--runs=1` restores the old single-run behavior) and reports the median: - **k6 metrics** (latency, throughput, etc.): the per-metric median across all N runs' `--summary-export` output. - **Server RSS/FDs**: rather than merge the time series (which wouldn't produce a real one), the run whose peak RSS is closest to the group's median is used as-is, so the report's memory chart still shows one genuine, representative run. If any of the N runs fails outright, the whole scenario is reported as failed rather than silently averaging over fewer, "surviving" runs. ## CI Integration The `bench.yml` GitHub Actions workflow: - Runs all scenarios (median of 3 runs each) on every push to main and on PRs - Uploads `report.html` as a workflow artifact - Tracks metrics over time, committing results to `dev/bench/` on the `gh-pages` branch. That branch is no longer published, so the dashboard has no public URL; view it by checking the branch out and opening `dev/bench/index.html` - Posts comparison comments on PRs when a metric regresses >25%; this is advisory only and does not block merging (shared-runner noise is too high for a hard gate) ## Design Decisions - **`cat` as backend**: Measures websocketd overhead only, not backend processing time - **k6**: Industry-standard load testing tool with native WebSocket support - **Fresh process per scenario**: Each scenario gets a clean websocketd instance - **"Smaller is better" normalization**: Throughput reported as µs/msg for unified CI thresholds --- ## File: examples/swift/README.md ## Swift examples ### Count Run the following line and open "html/count.html" from the websocketd examples directory. ``` $ websocketd --port=8080 count.swift ``` ### Greeter Run the following line and open "http://localhost:8080" in your browser to interact with the greeter server. ``` $ websocketd --port=8080 --devconsole greeter.swift ``` --- ## File: examples/nodejs/README.md ## Running the examples on Mac ##### 1. Download [Install](https://github.com/joewalnes/websocketd/wiki/Download-and-install) websocketd and add it to your `PATH`. ##### 2. Start a server: getter Run `websocketd --port=8080 --devconsole node greeter.js` and then go to `http://localhost:8080` to interact with it ##### 3. Start a server: counter Run `websocketd --port=8080 node count.js` to start the server, then go to [examples/html](https://github.com/joewalnes/websocketd/tree/main/examples/html) directory and double click **count.html** to open in a browser --- ## File: examples/lua/README.md The examples demonstrate the use of websocketd with lua. There are two examples in the directory both very basic. 1. Greeter.lua simply echos back any input made from the client 2. json_ws.lua echos back any input from the client *after* converting it into a json string It is pretty simple to extend these examples into full fledged applications. All you need is an stdin input loop ``` local input = io.stdin:read() while input do -- do anything here -- update the input input = io.stdin:read() end ``` any thing you `print` goes out to the websocket client Libraries and third party modules can be used by the standard `require` statement in lua. ## Running the examples ##### 1. Download [Install](https://github.com/joewalnes/websocketd/wiki/Download-and-install) websocketd and add it to your `PATH`. ##### 2. Start a server: greeter Run `websocketd --port=8080 --devconsole lua ./greeter.lua` and then go to `http://localhost:8080` to interact with it ##### 3. Start a server: json_ws Run `websocketd --port=8080 --devconsole lua ./json_ws.lua` and then go to `http://localhost:8080` to interact with it If you are using luajit instead of lua you may run the examples like this (this assumes that you've got luajit in your path) `websocketd --port=8080 --devconsole luajit ./json_ws.lua` and then go to `http://localhost:8080` --- ## File: examples/java/README.md ## Running the examples on Mac 1. [download and install](https://github.com/joewalnes/websocketd/wiki/Download-and-install) websocketd and add it to your PATH 2. Echo Server: Run `websocketd --port=8080 --devconsole ./echo.sh` and then go to http://localhost:8080 to interact with it 3. Count Server: Run `websocketd --port=8080 ./count.sh` to start the server, then go to [examples/html](https://github.com/joewalnes/websocketd/tree/main/examples/html) folder and double click **count.html** to open in a browser --- ## File: examples/haskell/README.md ## Haskell examples ### Count Start the server with ``` $ websocketd --port=8080 --devconsole --passenv PATH ./count.hs ``` The passing of `PATH` was required for me because a typical Haskell installation of `runhaskell` does not go into `/usr/bin` but more like `/usr/local/bin`. ### Greeter The greeter server waits for a line of text to be sent, then sends back a greeting in response, and continues to wait for more lines to come. ``` $ websocketd --port=8080 --devconsole --passenv PATH ./greeter.hs ``` --- ## File: examples/hack/README.md This examples directory shows some examples written in [Hack](https://hacklang.org). ### Requirements : - [HHVM](https://github.com/facebook/hhvm) 3.30+ - [HSL ( Hack Standard Library )](https://github.com/hhvm/hsl) 3.30+ - [HSL Experimental](https://github.com/hhvm/hsl-experimental) 3.30+ You can also test the command files by running from the command line : ``` $ hhvm count.hh 1 2 3 4 5 6 7 8 9 10 ```