SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
# Benchmarking & profiling protocol (SwiftSoup)
This repo’s current performance harness is the `BenchmarkProfileTest/testParseBenchmarkProfile` test. It is driven by
environment variables and runs in a release test build.
## Optional mise shortcuts (macOS/Linux)
The tasks use your existing Swift toolchain and do not install tools:
```
mise run test:release
mise run bench
SWIFTSOUP_BENCHMARK_ITERATIONS=100 mise run bench
SWIFTSOUP_BENCHMARK_SET=attribute-heavy mise run bench
```
`bench` defaults to `base,large`, two in-process warmups, and ten measured iterations.
Existing `SWIFTSOUP_BENCHMARK_*` environment overrides are preserved. Increase the
iteration count for measurements; these modest defaults are only a starting point.
The direct SwiftPM commands below remain supported without mise.
Compare the benchmark's elapsed-time line, not build or task wall time. The default
workload includes Data and String parsing plus selector/text operations; it is not a
parse-only measurement. Record `swift --version` and any local source changes with
results. Repeat runs with alternating baseline/candidate order, run an unchanged
A/A comparison to estimate noise, and validate behavior with regression tests before
interpreting an improvement. Keep profiling runs separate from timing runs.
## A/B comparison rules
- Always compare release builds on the same machine.
- If an optimization has a flag: run OFF vs ON.
- If no flag: compare HEAD to a baseline commit in a worktree.
- Test each optimization in isolation before any combined run.
- Record commit SHAs, flags, iterations, and the exact command lines used.
## Common benchmark knobs (env vars)
- `SWIFTSOUP_BENCHMARK=1` (enable the benchmark test)
- `SWIFTSOUP_BENCHMARK_SET=...` (comma-separated workload sets)
- `SWIFTSOUP_BENCHMARK_WARMUP=2`
- `SWIFTSOUP_BENCHMARK_ITERATIONS=500`
- `SWIFTSOUP_BENCHMARK_ITERATIONS_MULTIPLIER=1`
- `SWIFTSOUP_BENCHMARK_REPEAT=200` (base set)
- `SWIFTSOUP_BENCHMARK_LARGE_REPEAT=60` (large set)
- `SWIFTSOUP_BENCHMARK_SELECTOR_REPEAT=1`
- `SWIFTSOUP_BENCHMARK_SELECTOR_STRESS_REPEAT=1`
- `SWIFTSOUP_BENCHMARK_ATTRIBUTE_SELECTOR_STRESS_REPEAT=1`
- `SWIFTSOUP_BENCHMARK_SERIALIZER=source-patched|without-source-reuse|reuse-source-outside-body`
- `SWIFTSOUP_BENCHMARK_DENSE_BODY_MUTATIONS=1`
## Regression suite (broad coverage)
Run in both baseline and current (exact command line):
```
SWIFTSOUP_BENCHMARK=1 \
SWIFTSOUP_BENCHMARK_SET=base,large \
SWIFTSOUP_BENCHMARK_REPEAT=200 \
SWIFTSOUP_BENCHMARK_LARGE_REPEAT=60 \
SWIFTSOUP_BENCHMARK_WARMUP=2 \
SWIFTSOUP_BENCHMARK_ITERATIONS=500 \
SWIFTSOUP_BENCHMARK_ITERATIONS_MULTIPLIER=1 \
SWIFTSOUP_BENCHMARK_SELECTOR_REPEAT=1 \
SWIFTSOUP_BENCHMARK_SELECTOR_STRESS_REPEAT=1 \
SWIFTSOUP_BENCHMARK_ATTRIBUTE_SELECTOR_STRESS_REPEAT=1 \
swift test -c release --filter BenchmarkProfileTest/testParseBenchmarkProfile
```
## Targeted suite (optimization-specific)
Pick the workload set that stresses the change and run with the same flags:
```
SWIFTSOUP_BENCHMARK=1 \
SWIFTSOUP_BENCHMARK_SET=attribute-heavy,attribute-mega,attribute-storm,data-heavy,data-storm,tag-heavy,custom-tag,dense-text,querystring \
SWIFTSOUP_BENCHMARK_WARMUP=2 \
SWIFTSOUP_BENCHMARK_ITERATIONS=500 \
SWIFTSOUP_BENCHMARK_ITERATIONS_MULTIPLIER=1 \
SWIFTSOUP_BENCHMARK_SELECTOR_REPEAT=1 \
SWIFTSOUP_BENCHMARK_SELECTOR_STRESS_REPEAT=1 \
SWIFTSOUP_BENCHMARK_ATTRIBUTE_SELECTOR_STRESS_REPEAT=1 \
swift test -c release --filter BenchmarkProfileTest/testParseBenchmarkProfile
```
## Serializer A/B
For Reader-style dense body mutation, run the same release benchmark once per serializer:
```
for serializer in source-patched without-source-reuse reuse-source-outside-body; do
SWIFTSOUP_BENCHMARK=1 \
SWIFTSOUP_BENCHMARK_SET=manabi-reader \
SWIFTSOUP_BENCHMARK_MANABI_REPEAT=40 \
SWIFTSOUP_BENCHMARK_WARMUP=2 \
SWIFTSOUP_BENCHMARK_ITERATIONS=30 \
SWIFTSOUP_BENCHMARK_ITERATIONS_MULTIPLIER=1 \
SWIFTSOUP_BENCHMARK_SKIP_SELECTORS=1 \
SWIFTSOUP_BENCHMARK_SKIP_TEXT=1 \
SWIFTSOUP_BENCHMARK_DENSE_BODY_MUTATIONS=1 \
SWIFTSOUP_BENCHMARK_SERIALIZER="$serializer" \
swift test -c release --filter BenchmarkProfileTest/testParseBenchmarkProfile
done
```
## Baseline worktree helper
- `git worktree add ../SwiftSoup-bench-base <baseline-commit>`
- Run the same commands in both trees; compare the “Benchmark elapsed: … ms over N iterations” line.
- Use `git rev-parse HEAD` to log the commit for each side.
## Profiling
- Use Instruments (Time Profiler) on the `SwiftSoupPackageTests.xctest` process while running the same benchmark test
filter and env vars.
- Always profile release builds and keep the workload set identical to the benchmark run you are comparing against.