{"owner":"dragonflydb","repo":"dragonfly","hasSkills":true,"totalSkillsCount":12,"totalTokensCount":21093,"categories":["anthropic-skill","copilot-instructions","root-instruction","claude-rule","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":[".claude/skills/benchmark-report/SKILL.md",".claude/skills/benchmark/SKILL.md",".claude/skills/benchmark/evals/evals.json",".claude/skills/benchmark/references/report-template.md",".claude/skills/benchmark/references/running-benchmarks.md",".claude/skills/reproduce-fuzz-crash/SKILL.md",".github/copilot-instructions.md","AGENTS.md","CLAUDE.md","tools/packaging/README.md","tools/packaging/osrepos/README.md","tools/vector-benches/README.md"],"skills":{".claude/skills/benchmark-report/SKILL.md":"---\nname: benchmark-report\ndescription: Generate a technical benchmark report in markdown from raw memtier/redis-benchmark output. Use this whenever the user provides benchmark results, log files, memtier JSON output, or a directory of benchmark data and wants a written report or comparison document. Trigger even when the user says \"write up these results\", \"make a report from this data\", \"turn this into a blog post\", or just drops a directory path. Always use this skill when the user provides benchmark files and wants a human-readable output. Reports compare Dragonfly against another system (ElastiCache, Redis, Valkey) but can handle single-system data too.\nversion: 1.0.0\n---\n\n# Benchmark Report Generator\n\nGenerate a technical markdown report from raw memtier/redis-benchmark data. The report is aimed at engineers evaluating Dragonfly for production use.\n\n## Step 0: Interview before writing\n\nBefore touching any files, ask the user 3–4 targeted questions to fill in context that cannot be extracted from the raw data. Read the data directory first so your questions are specific, not generic.\n\nQuestions to ask (adapt based on what's missing after scanning the files):\n\n1. **Scenario**: What is the purpose of this benchmark? (e.g., \"SSD tiering comparison\", \"evaluating ElastiCache replacement\", \"testing read-heavy workload at scale\") — this becomes the intro paragraph and title.\n2. **Instance types**: What instance type / tier was each system running on? Include memory size.\n3. **Storage configuration**: For Dragonfly — is SSD tiering enabled? What is the memory limit vs total dataset size?\n4. **Anything else relevant**: Pricing comparison, region, network setup, anything not in the scripts/logs.\n\nWait for the user's answers before proceeding to Step 1. If the user says \"just use TODOs\" or similar, proceed without waiting.\n\n## What the user provides\n\n- A directory path with benchmark data files\n- Context answers from the interview above\n\nTypical directory layout:\n```\ndata_dir/\n *.sh — benchmark script (parameters, key distribution, test phases)\n *.json — memtier JSON output per system per phase\n *.log — memtier log output per system\n```\n\nFiles may be flat in one directory or split across subdirectories. Read them all.\n\n## Step 1: Scan and parse\n\nList all files in the directory. Then read them:\n\n**Scripts (`*.sh`)** — extract:\n- `KEY_MAXIMUM`, `DATA_SIZE`, `PIPELINE`, `CLIENTS`, `THREADS`, `TEST_TIME`\n- `FILL_RATIO`, `ACCESS_RATIO`, `FILL_KEY_PATTERN`, `ACCESS_KEY_PATTERN`\n- Key distribution flags: `--key-stddev`, `--key-zipf-exp`, etc.\n- Which phases ran (`RUN_FILL=true`, `RUN_ACCESS_READ_002=true`, etc.)\n- Hostnames — used to identify systems under test\n\n**JSON files (`*.json`)** — from memtier `--json-out-file` output:\n- `configuration.server` → hostname (used to identify the system)\n- `configuration.clients`, `threads`, `pipeline`, `ratio`, `key_maximum`, `data_size`, `key_stddev`\n- `ALL STATS.Gets` or `ALL STATS.Sets` → `Ops/sec`, `Average Latency`, `Percentile Latencies` (p50/p99/p99.9)\n- `ALL STATS.Runtime.Total duration` → test duration in ms\n\n**Log files (`*.log`)** — from memtier stdout:\n- Look for the `ALL STATS` table block. Take the last occurrence (final aggregate):\n ```\n Type Ops/sec ... Avg. Latency p50 Latency p99 Latency p99.9 Latency\n Sets 220503.72 ... 4.32106 2.95900 9.72700 53.24700\n Gets 138188.17 ... 6.94510 5.53500 36.09500 54.27100\n ```\n- Look for `real Xm Y.Zs` timing lines to capture wall-clock duration per phase\n\n## Step 2: Identify systems under test\n\nName each system from filename signals and hostnames:\n\n| Signal | System name |\n|--------|------------|\n| `bench_df*.log`, hostname contains `dragonflydb.cloud` | Dragonfly |\n| `bench_ec*.log`, hostname contains `cache.amazonaws.com` | ElastiCache |\n| `bench_valkey*.log` | Valkey |\n| `bench_redis*.log` | Redis |\n| `bench_local*.log`, `127.0.0.1` | Local (note version if available) |\n\nUse the actual product name, not the hostname. Do not include hostnames or endpoints in the report — they are not relevant for reproducing the results and may change.\n\n## Step 3: Generate charts\n\nIf JSON files are present, generate latency charts with `tools/plot_memtier_latency.py`:\n\n```bash\nmkdir -p /charts\npython3 /tools/plot_memtier_latency.py /charts/.html\n```\n\nRun once per JSON file. If the script is not found or fails, leave a `[TODO: chart]` stub. The script skips operations with no traffic (zero-latency entries), so it handles write-only fill JSONs and read-only access JSONs without errors.\n\nTo find `tools/plot_memtier_latency.py`, search from the current working directory: `find . -name \"plot_memtier_latency.py\" -not -path \"*/build*\"`.\n\n## Step 4: Write the report\n\nSave to `/report.md`. Use the template below. Populate every value you found; use `[TODO]` for anything missing (instance type, memory, pricing, scenario context the user didn't provide).\n\n---\n\n```markdown\n# \n\n<1–3 sentence intro: what was measured and why. No adjectives. Example: \"This benchmark measures read throughput and latency for a 50 GB key-value dataset under a Gaussian access pattern. Dragonfly was configured with SSD tiering; ElastiCache r7g.4xlarge was used with all data in memory.\">\n\n## Test Environment\n\n| | Dragonfly | ElastiCache |\n|---|---|---|\n| Version / tier | [TODO] | [TODO] |\n| Instance type | [TODO] | [TODO] |\n| Memory | [TODO] | [TODO] |\n| Storage | [TODO: SSD tiering?] | [TODO] |\n| Region | [TODO] | [TODO] |\n\n\n\n## Dataset\n\n| Parameter | Value |\n|---|---|\n| Key count | |\n| Value size | bytes |\n| Approximate dataset size | |\n| Key distribution | |\n\n## Benchmark Configuration\n\n| Parameter | Value |\n|---|---|\n| Tool | memtier_benchmark |\n| Threads | |\n| Connections per thread | |\n| Pipeline depth | |\n| Test duration (access phase) | s |\n| Access ratio (SET:GET) | |\n\n## Results\n\n### Fill Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p99 Latency (ms) | Duration |\n|---|---|---|---|---|\n| Dragonfly | | | | |\n| ElastiCache | | | | |\n\n### Read Access Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p50 (ms) | p99 (ms) | p99.9 (ms) |\n|---|---|---|---|---|---|\n| Dragonfly | | | | | |\n| ElastiCache | | | | | |\n\n## Latency Over Time\n\n\n\n**Dragonfly — read access (σ=2%)**\n![Dragonfly latency over time](./charts/df_access.html)\n\n**ElastiCache — read access (σ=2%)**\n[TODO: ElastiCache JSON not available; chart cannot be generated]\n\n## Observations\n\n\n\n1. Throughput: Dragonfly achieved X ops/sec vs Y for ElastiCache — a Z% difference.\n2. Average latency: X ms (Dragonfly) vs Y ms (ElastiCache).\n3. p99 latency: X ms vs Y ms. \n4. Fill phase: .\n5. \n6. \n```\n\n---\n\n## Writing rules\n\n- Use exact numbers from the data. No rounding unless the value is very large (round to 3 significant figures max).\n- Percentages for comparisons: compute `(A - B) / B * 100`; write as \"X% higher\" or \"X% lower\".\n- Prohibited phrases: \"blazing fast\", \"industry-leading\", \"dramatically\", \"significantly\", \"next-level\", \"game-changing\", \"unparalleled\", \"impressive\"\n- No narrative hooks: no \"but wait\", \"what happens next\", \"the results speak for themselves\", \"as you can see\"\n- Observations may include inferences (e.g., \"the Gaussian σ=2% means the hot working set is ~3.5M keys, likely fitting in memory\") — label them clearly as inferences if they go beyond what the data directly shows\n- Bold labels in observations (e.g., `**Read throughput**:`) — keep them for scannability\n- Do not include hostnames, endpoints, or internal URLs anywhere in the report\n- Missing context → `[TODO: ...]` not a guess\n- Single-system data: omit comparison columns; keep observations factual about absolute numbers\n\n## Output\n\nPrint the report path when done: `Report written to: /report.md`\n\nIf charts were generated, list them too.\n",".claude/skills/benchmark/SKILL.md":"---\nname: benchmark\ndescription: >\n Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote\n cloud instances, then produce a performance + memory report with charts and a\n raw-data appendix. Use this whenever the user runs /benchmark, or asks to\n benchmark / load-test / measure throughput, QPS, latency, or memory efficiency\n of Dragonfly or another Redis-compatible server, to compare Dragonfly vs\n Valkey or Redis, or to generate a benchmark report. Trigger it even when the\n user just describes the goal (\"see how fast Dragonfly is on this box\",\n \"compare memory per key vs Valkey\") without saying the word \"benchmark\".\nargument-hint: (optional) free-form spec, e.g. \"server=10.0.0.1 client=10.0.0.2 strings vs-valkey\"\nallowed-tools: Bash, Read, Write, Edit, AskUserQuestion\n---\n\n# Benchmark Dragonfly\n\nDrive a benchmark of Dragonfly (optionally side-by-side with Valkey or Redis) on\nlocal or remote instances, collect throughput / latency / memory numbers, and\nassemble a report that mirrors the team's published format: a narrative TLDR,\nper-instance metric tables, charts, and an appendix with the exact commands and\nraw output so anyone can reproduce it.\n\nThe whole point of the report is **trust**: a reader should be able to see the\nheadline numbers, understand the conditions they were measured under, and\nre-run the exact commands themselves. Everything below serves that goal.\n\n## When you start\n\nIf the user passed a free-form spec in `$ARGUMENTS`, parse what you can from it\n(IPs, use-cases, comparison targets) and only ask about what's missing. The work\nsplits into clear phases — walk through them in order, but stay flexible: the\nuser may already have servers running, or may only want the report half (handed\nraw output to format). Read what they're actually asking for.\n\nPhases: (1) gather spec → (2) prepare instances → (3) network tuning →\n(4) start monitoring → (5) run workloads → (6) extract metrics, charts, report.\n\nThe two reference files carry the operational detail — read them before acting,\nnot from memory:\n\n- `references/running-benchmarks.md` — how to prep instances, start servers,\n drive `dfly_bench` and `memtier_benchmark`, run Valkey/Redis for comparison,\n apply network tuning, and measure memory. **Read this before Phase 2.**\n- `references/report-template.md` — the exact report structure, with an example.\n **Read this before Phase 5.**\n\nBundled `scripts/` (reuse these instead of re-improvising the same loops — they\nencode the safety discipline learned the hard way):\n\n- `bench_server.sh start|stop|status [maxmem] [port]` — launch a\n server tracking its PID; **stop kills by PID and verifies the process is gone +\n port free** (tmux kill-session does not reap it).\n- `monitor_fill.sh [session] [abort_pct]` — the live\n RSS guard: polls `used_memory_rss`, prints a line per sample, aborts the write\n if RSS crosses the threshold (default 92%). Its output is also the input to\n `plot_fill.py`.\n- `capture_result.sh [label]` — saves\n the run summary + INFO memory + **OS-level `free` snapshot** (`host_memory.txt`)\n and exits non-zero if there were any eviction errors (enforces the zero-error\n rule). The OS snapshot matters: `used_memory_rss` and host free can diverge\n significantly — e.g. with `MIMALLOC_ALLOW_LARGE_OS_PAGES=1` (default), mimalloc\n holds large-page arenas that the OS counts as used but INFO doesn't fully reflect,\n making the machine appear near-OOM while RSS looks fine.\n- `collect_metrics.py` / `make_charts.py` — metrics + charts from dfly_bench\n JSON (GET/SET workloads).\n- `plot_fill.py` — memory-fill chart from `monitor_fill.sh` output, for\n `--command` workloads (e.g. SETEX) that produce no JSON time series.\n\nEvery benchmark output directory must include a `commands_audit.log`\nartifact with the exact command lines used for setup, server launch, writes,\nreads, captures, diagnostics, and chart/report generation. Raw data without the\ncommands that produced it is not reproducible.\n\n**Run-control scripts are mandatory for benchmark phases.** For Phase 5 runs,\nuse the bundled scripts as the default API:\n\n- start/stop/status servers with `scripts/bench_server.sh`;\n- monitor long write fills with `scripts/monitor_fill.sh`;\n- capture write results and enforce the zero-error rule with\n `scripts/capture_result.sh`;\n- generate metrics/charts with `scripts/collect_metrics.py`,\n `scripts/make_charts.py`, and `scripts/plot_fill.py`.\n\nInline SSH is fine for inspection, package setup, one-off diagnostics, and\nbounded foreground probes, but not as a replacement for the run-control scripts\nin the main benchmark path. If a script is not usable for a run, document the\nreason in `commands_audit.log` and in the report appendix before using a manual\nequivalent.\n\n## Phase 1 — Gather the benchmark spec (interactive)\n\nUse `AskUserQuestion` to pin down anything not already given. You need:\n\n1. **Where it runs.** Local, or remote instances? For remote, get the **server**\n host/IP and the **client** host/IP (the load generator should run on a\n separate machine from the server — co-locating them steals CPU and corrupts\n the numbers). Get the SSH user / key if not obvious. On cloud instances,\n **always use the private/internal IP** (VPC-internal) for both SSH and as the\n load generator's `-h` target — the public IP routes through the internet\n gateway, adding latency and egress cost, and the result measures the gateway\n rather than the server.\n2. **Use-cases.** Default menu: simple strings (SETEX writes + GET reads),\n pipelined / peak-throughput, memory efficiency. Always also ask whether there\n is a **specific focus** (a particular command, data type, key distribution,\n value size, or scenario) — the canned use-cases are a starting point, not a\n cage.\n3. **Comparison targets.** Dragonfly only, or also Valkey / Redis? Which\n versions?\n4. **Instance specs** (for the report's conditions section): instance type, CPU\n count, OS/kernel. Read these directly from the box at the start of Phase 2\n rather than asking the user.\n5. **Load parameters** if the user has opinions: value size (`-d`), key space\n (`--key_maximum`), connections (`-c`), pipeline depth, test duration. Pick\n sensible defaults (see the reference) and tell the user what you chose.\n **Sizing the key count is the consequential decision** — see the reference's\n \"Sizing the load\" section. The dataset must fill a large fraction of instance\n RAM (benchmarking a near-empty instance is wrong) but stay clear of OOM\n (~60–75% of RAM, headroom to spare). `--key_maximum` bounds the footprint, so\n compute it from instance RAM ÷ estimated bytes-per-entry, calibrating with a\n short pilot write if unsure.\n\nSummarize the resolved spec back to the user in a short block before running\nanything, so a wrong assumption gets caught before machines spin.\n\n## Phase 2 — Prepare the instances\n\n- Record instance specs (`nproc`, `uname -r`, cloud metadata) — these go in the\n report's conditions section. Use the **private IP** when connecting between\n instances (see Phase 1).\n- A remote instance usually already has `~/projects/dragonfly`. If not, clone it\n there. Build a **release** binary (`build-opt/dragonfly`, `build-opt/dfly_bench`)\n unless the user wants a published release downloaded instead — follow their\n instruction on build-vs-download.\n- For comparison targets, pull the Valkey/Redis docker image on the server box.\n- Verify everything launches and is reachable from the client box (a quick\n `redis-cli -h ping`) before committing to a long run.\n\nRun each step individually over SSH — do not chain launch, polling, capture, and\nthe next phase into one command. If a tool call is interrupted, remote work keeps\nrunning but goes uncaptured. Start the server; start the write; monitor it;\ncapture; start reads — one bounded step at a time.\n\n## Phase 3 — Network tuning (optional, remote only)\n\nHigh-throughput numbers on cloud hardware depend heavily on IRQ/SMP affinity\ntuning. If the user wants maximal numbers, apply the tuning described in the\nreference and **record exactly what you did** — it belongs in the appendix and\nmaterially affects the results. If skipped, say so in the report; untuned and\ntuned numbers are not comparable.\n\n## Phase 4 — Start the monitoring stack (server machine)\n\nStart the local Prometheus + Grafana stack on the **server** machine so you get\ncontinuous, fine-grained time-series (server CPU, memory, command rates, host\nmetrics) spanning the whole benchmark — richer than the load generator's own\noutput and the source of the most convincing report charts.\n\n```bash\nssh 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose up -d'\n# when benchmarking Valkey/Redis, also bring up the redis exporter:\nssh 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose --profile redis up -d'\n```\n\nPrometheus scrapes at 1s. **Always run servers on port 6380** (Dragonfly,\nValkey, and Redis alike). The bundled `prometheus.yml` ships pointing at `6379`,\nso before bringing the stack up, fix the scrape targets:\n\n```bash\nsed -i 's/:6379/:6380/g' tools/local/monitoring/prometheus/prometheus.yml\n```\n\n(This is a workaround for a misconfigured default — if you have repo write\naccess, commit the corrected file once so future runs start clean.)\n\nRestart Prometheus if it was already running, then verify targets are `UP` at\n`http://:9090/targets` before starting the load — a silently-down scrape\njob means no data for that window.\n\nStart monitoring **before** the workloads and leave it running across all runs,\nso a single continuous window covers everything. See the reference's monitoring\nsection for what to read from Grafana (dashboards are pre-provisioned) and how to\npull panels into the report. Record the benchmark start/end timestamps — they\nlet you crop the Grafana window to exactly the run.\n\n## Phase 5 — Run the workloads\n\nFor each (use-case × datastore) combination, see the reference for the precise\ncommands. The shape is always: **write → memory snapshot → read**. Use the\nprivate IP as the load generator's `-h` target.\n\nPass `--json_out_file` to `dfly_bench` (and `--json-out-file` to memtier) when\nthe workload supports it — charts and headline numbers come from these files.\n`dfly_bench --command` workloads (e.g. `SETEX __key__ 10000 __data__`) do not\nproduce JSON; save full stdout and chart the `monitor_fill.sh` output instead.\n\nDrive the run with the bundled scripts: `bench_server.sh start`, write in a\nclient tmux session, `monitor_fill.sh` alongside (live RSS guard), then\n`capture_result.sh` to save results. `capture_result.sh` enforces the\n**zero-error rule** — any eviction/OOM error response makes the run invalid;\ndiscard it, cut `--key_maximum` ~10–15%, and re-run. See the preamble and the\nreference's \"Sizing the load\" section for the full rule.\n\n**Prefill writes must use `--pipeline 30`.** Without pipelining, the client\nstalls waiting for each reply and becomes the bottleneck before the server is\nsaturated — especially at high key counts. Use `--pipeline 30` for all write\n(fill) phases. Drop to `--pipeline 1` (or omit it) only for latency-honest\nread benchmarks where you want honest per-request numbers.\n\nInitialize `commands_audit.log` at the start and append each command as it runs\n(see the preamble for the requirement).\n\n## Phase 6 — Extract metrics, build charts, assemble the report\n\n1. **Headline metrics + memory.** Run the bundled extractor over the JSON files\n to pull QPS / P99 and compute bytes-per-entry (both `used_memory`- and\n `used_memory_rss`-based, per the user's choice to report both):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/collect_metrics.py \\\n --bench-json [--bench-json ...] \\\n --info --dbsize \\\n --label \"Dragonfly m7g.2xlarge\"\n ```\n\n It prints a metrics block (and JSON) you can drop straight into the tables.\n\n2. **Charts.** Generate QPS-over-time PNGs from the time series (this is what\n visually shows Dragonfly's smooth throughput vs a competitor's degradation):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/make_charts.py \\\n --series \"Dragonfly=df_write.json\" --series \"Valkey=valkey_write.json\" \\\n --metric qps --title \"Write throughput, m7g.2xlarge\" \\\n --out charts/write_m7g2xl.png\n ```\n\n Embed the PNGs in the report with relative paths. **For `--command` workloads\n (e.g. SETEX) there is no dfly_bench JSON** — use `plot_fill.py` instead to chart\n memory-fill over time from the `monitor_fill.sh` output:\n\n ```bash\n python3 .claude/skills/benchmark/scripts/plot_fill.py \\\n --series \"prebuilt=prebuilt/poll.txt\" --series \"release=release/poll.txt\" \\\n --interval 10 --ram-gib 30 --out charts/memory_fill.png\n ```\n\n3. **Report.** Follow `references/report-template.md` exactly. Write a TLDR with\n the 3–5 genuinely important takeaways (scaling behavior, memory efficiency,\n any hardware surprise), then per-instance tables, then the appendix. The TLDR\n is interpretation, not a restatement of the tables — say what the numbers\n *mean*. Only claim what the data supports.\n\nSave the report where the user asks (default: `benchmark--.md` in the\nrepo root) and tell them the path plus a one-line summary of the headline\nresult.\n\n## Cleanup\n\nIf you started servers, the monitoring stack, or comparison containers on remote\ninstances, offer to stop them (`docker compose down` in the monitoring dir).\nIf you applied network tuning that changes machine state, tell the user what\nwould need reverting. Don't tear anything down without asking — they may want to\ninspect or re-run.\n",".claude/skills/benchmark/evals/evals.json":"{\n \"skill_name\": \"benchmark\",\n \"evals\": [\n {\n \"id\": 0,\n \"name\": \"full-report-vs-valkey\",\n \"prompt\":\n \"I just finished a benchmark run comparing Dragonfly against Valkey 9.0 on AWS. The raw artifacts are in the directory I'm pointing you at: dfly_bench JSON output (write and read phases) and INFO memory dumps for both datastores on m7g.2xlarge and m7g.4xlarge, plus a RUN_NOTES.txt with the hardware, exact commands, and DBSIZE after each write phase. Generate the benchmark report from this data, with charts and a raw-data appendix.\",\n \"expected_output\":\n \"A markdown report mirroring the team format: conditions section, a Simple Strings use-case section with a TLDR, per-instance (m7g.2xlarge, m7g.4xlarge) QPS/memory and latency tables, embedded QPS-over-time charts, and an appendix with exact commands, environment, and raw output. QPS/P99 must match the data; bytes-per-entry reported both logical (~127 DF / ~149-150 Valkey) and RSS.\",\n \"files\": [\n \"fixtures/\"\n ]\n },\n {\n \"id\": 1,\n \"name\": \"memory-focus-single-datastore\",\n \"prompt\":\n \"Here's raw output from a Dragonfly-only run on m7g.4xlarge (dfly_bench write/read JSON + INFO memory dump + run notes with DBSIZE). I care mostly about memory efficiency. Write me a short benchmark report focused on memory usage per entry, plus the throughput numbers.\",\n \"expected_output\":\n \"A report covering only Dragonfly (no fabricated Valkey/Redis comparison column), reporting both logical (used_memory/DBSIZE ~127) and RSS bytes-per-entry, plus write/read QPS and P99. Honest about it being single-datastore.\",\n \"files\": [\n \"fixtures/df_4xl_write.json\",\n \"fixtures/df_4xl_read.json\",\n \"fixtures/df_4xl_info_memory.txt\",\n \"fixtures/RUN_NOTES.txt\"\n ]\n },\n {\n \"id\": 2,\n \"name\": \"kickoff-planning-interactive\",\n \"prompt\":\n \"I've got two m7g.4xlarge boxes in AWS. Benchmark Dragonfly against Valkey on a simple strings workload.\",\n \"expected_output\":\n \"The skill should engage its interactive front-end: ask which box is the server vs the client (load generator must be separate), confirm SSH access, plan to run servers on port 6380, plan to start the monitoring stack (and fix the prometheus scrape port to 6380), and lay out the write/read workload plan. It must NOT invent benchmark numbers it hasn't measured.\",\n \"files\": []\n }\n ]\n}\n",".claude/skills/benchmark/references/report-template.md":"# Report template\n\nThe report mirrors the team's published benchmark format. Structure matters: a\nreader scans the conditions, jumps to the TLDR for the takeaways, checks the\nper-instance tables for numbers, and uses the appendix to reproduce. Follow this\nshape.\n\nUse Markdown tables. Center-align numeric columns (`:---:`). Bold the winning\ndatastore in comparison tables when there's a clear winner.\n\n## Structure\n\n```markdown\n# — e.g. \"Comparing Dragonfly vs Valkey on AWS\"\n\n<1–2 sentences: what was compared, on what hardware, what workloads.>\n\n## Conditions\n\nDragonfly launch command (identical across all configs):\n\n ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n\n<Comparison target> launch command:\n\n docker run --network=host --rm valkey/valkey:9.0 --save \"\" --appendonly no \\\n --io-threads 8 --protected-mode no --port 6380\n\nInstances: <server types + CPU counts>, client on <client instance type>.\nOS: <distro + kernel>. Network tuning: <applied / none — one line on what>.\n\n## <Use-case name> — e.g. \"Simple Strings\"\n\n<1–2 sentences describing the workload and why it matters.>\n\nWrite command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 --ratio 1:0 --json_out_file df_write.json\n\nRead command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN --ratio 0:1\n\n### TLDR\n\n* **<Datastore>** <the genuinely important takeaway — scaling, efficiency, a surprise>.\n* <3–5 bullets total. Interpretation, not a restatement of the tables.>\n\n## <Instance type> — e.g. \"m7g.2xlarge\"\n\n<Optional: one sentence on anything notable, e.g. throughput stability over time.>\n\n![Write throughput over time](charts/write_m7g2xl.png)\n\n### QPS and memory\n\n| Metric | Dragonfly | Valkey |\n| ----- | :---: | :---: |\n| Write QPS (k) | 816 | 548 |\n| Read QPS (k) | 848 | 811 |\n| Memory / entry — logical (bytes) | 127 | 149 |\n| Memory / entry — RSS (bytes) | 138 | 162 |\n\n### Latency\n\n| Backend | Write P99 (us) | Read P99 (us) |\n| :---: | :---: | :---: |\n| **Dragonfly** | 787 | 887 |\n| **Valkey** | 3283 | 951 |\n\n<Repeat the per-instance block for each instance type tested.>\n\n## Appendix — raw data\n\n### Environment\n\n| Property | Server | Client |\n| --- | --- | --- |\n| Instance type | m7g.2xlarge | c7gn.4xlarge |\n| vCPUs | 8 | 16 |\n| OS / kernel | Ubuntu 24.04.2 / 6.8.0 | ... |\n| Dragonfly version | <git hash / release> | — |\n| Network tuning | IRQ SMP affinity spread across CPUs | — |\n\n### Exact commands\n\n<Every command line actually run, per phase and per datastore, verbatim —\nincluding the launch commands, write/read load commands, and any tuning steps.>\n\n### Raw output\n\n<Per run: the final dfly_bench/memtier summary + latency histogram, and a\nreference to the saved JSON file (df_write.json, etc.). Include INFO memory\nsnapshots used for the per-entry figures.>\n```\n\n## Notes on writing the TLDR\n\nThe TLDR is the part people actually read. Make each bullet a claim the data\nsupports, phrased as a takeaway:\n\n- Good: \"Dragonfly scales vertically with cores and sustains up to 10x the\n throughput of a single Valkey process on large instances.\"\n- Weak: \"Dragonfly had higher QPS.\" (just restates the table)\n\nCall out hardware surprises honestly — e.g. an instance type that didn't scale\nlinearly — because they affect how a reader should interpret the numbers. If a\nrun was client-bound or otherwise caveated, say so here rather than burying it.\n\n## Charts\n\nOne chart per instance per phase is usually enough: QPS-over-time, with one line\nper datastore. The visual story is throughput *stability* — Dragonfly holding a\nflat line while a competitor degrades as its table grows is more convincing than\na single aggregate number. Generate with `scripts/make_charts.py` from the load\ngenerator's time-series JSON, save under a `charts/` dir next to the report, and\nembed with relative paths.\n\nWhen the monitoring stack ran (Phase 4), the Grafana dashboards add server-side\npanels that the load generator can't show — memory growth over the write phase,\nper-shard CPU, host network saturation. Crop the Grafana time range to the run\nwindow and pull the relevant panels (rendered PNG or screenshot) into the\nper-instance section. These are especially good for the memory-efficiency story\nand for proving the server (not the client) was the bottleneck. Note in the\nappendix that the data came from the monitoring stack, and which scrape port was\nused.\n",".claude/skills/benchmark/references/running-benchmarks.md":"# Running benchmarks — operational reference\n\nHow to prepare instances, start servers, drive the load generators, and measure\nmemory. Read the section you need; you don't have to read top-to-bottom.\n\n## Table of contents\n\n- [Instance preparation](#instance-preparation)\n- [Starting Dragonfly](#starting-dragonfly)\n- [Starting Valkey / Redis for comparison](#starting-valkey--redis-for-comparison)\n- [dfly_bench (default load generator)](#dfly_bench-default-load-generator)\n- [memtier_benchmark (alternative)](#memtier_benchmark-alternative)\n- [The standard strings workload](#the-standard-strings-workload)\n- [Measuring memory](#measuring-memory)\n- [Monitoring stack (Prometheus + Grafana)](#monitoring-stack-prometheus--grafana)\n- [Network tuning](#network-tuning)\n- [Sizing the load](#sizing-the-load)\n\n---\n\n## Instance preparation\n\nThe server and the load generator should live on **separate machines**. A\nclient co-located with the server competes for CPU and silently caps the\nnumbers — the report is then measuring the client, not the server.\n\nRemote instances usually already have the repo at `~/projects/dragonfly`. Check\nfirst; clone only if missing:\n\n```bash\nssh <host> 'test -d ~/projects/dragonfly && echo present || echo missing'\n# if missing:\nssh <host> 'git clone https://github.com/dragonflydb/dragonfly.git ~/projects/dragonfly'\n```\n\n**Look for prebuilt binaries first.** Before building or downloading, check the\nrepo's build output folders — the binary is often already there. Look in\n`build-opt/` and `build-release/` (and `build-dbg/` only as a last resort — debug\nbuilds give meaningless performance numbers):\n\n```bash\nssh <host> 'ls -la ~/projects/dragonfly/build-opt/{dragonfly,dragonfly-aarch64,dfly_bench} 2>/dev/null; \\\n ls -la ~/projects/dragonfly/build-release/{dragonfly,dfly_bench} 2>/dev/null'\n```\n\nA folder may hold several binaries — e.g. a locally-built `dragonfly` (dev/HEAD)\nalongside a downloaded release `dragonfly-aarch64`; check `--version` on each to\nknow which is which, since \"compare the prebuilt vs the release binary\" is a\ncommon request. The load generator (`dfly_bench`) lives in the same folder on the\nclient.\n\n**Build vs download** — if no suitable prebuilt binary exists, follow the user's\ninstruction. To build a release binary from source on the instance:\n\n```bash\nssh <host> 'cd ~/projects/dragonfly && ./helio/blaze.sh -release -DWITH_AWS=OFF -DWITH_GCP=OFF'\nssh <host> 'cd ~/projects/dragonfly/build-opt && ninja dragonfly dfly_bench'\n```\n\nThis yields `build-opt/dragonfly` (the server) and `build-opt/dfly_bench` (the\nload generator). Build `dfly_bench` on the **client** box and `dragonfly` on the\n**server** box. Always benchmark release builds — a debug build's numbers are\nmeaningless for performance claims.\n\nIf the user prefers a published release, download the tarball from GitHub\nreleases and extract it instead of building.\n\nSanity-check reachability from the client before a long run:\n\n```bash\nredis-cli -h <server-ip> -p 6380 ping # expect PONG\n```\n\nCollect environment facts for the report while you're on the box:\n\n```bash\nssh <host> 'nproc; uname -r; cat /etc/os-release | head -2'\n# cloud instance type, if on AWS:\nssh <host> 'curl -s http://169.254.169.254/latest/meta-data/instance-type'\n```\n\n---\n\n## Starting Dragonfly\n\nThe published comparisons run Dragonfly with a minimal, persistence-free config\nso the benchmark measures the data path, not disk:\n\n```bash\n./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n```\n\n- `--conn_use_incoming_cpu` — pin each connection's processing to the CPU that\n received it; improves locality under high connection counts.\n- `--dbfilename=` — disable snapshot loading/saving.\n\nLet Dragonfly use all cores by default (it auto-detects). Record the exact\nlaunch command for the appendix.\n\n**Start it so you can reliably stop it — capture the PID.** Launching under\n`tmux`/`nohup` is fine, but `tmux kill-session` does **not** reliably reap the\nserver, and a stale server is a trap: a \"fresh\" server silently fails to bind the\nport, your client reconnects to the *old* instance, and you measure stale data\n(and waste the old dataset's RAM). Always record the PID and kill by PID:\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/build-opt; \\\n nohup ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380 \\\n >/tmp/dfly.log 2>&1 & echo $! > /tmp/dfly.pid'\n# ... benchmark ...\n# Stop it and VERIFY it's gone before starting the next binary:\nssh <server> 'kill -9 $(cat /tmp/dfly.pid); sleep 3; \\\n pgrep -af dragonfly | grep -v \"pgrep\\|bash -c\" || echo \"stopped\"; \\\n redis-cli -p 6380 ping 2>&1 | head -1 # expect Connection refused'\n```\n\nBetween two binaries (e.g. prebuilt vs release), **fully stop and verify the\nfirst is dead and the port is free before starting the second** — otherwise the\nsecond never binds and you benchmark the first twice. Note that pattern-based\n`pkill -f` can miss a process whose cmdline is the relative `./dragonfly`; killing\nby the captured PID is the robust path.\n\n---\n\n## Starting Valkey / Redis for comparison\n\nRun via docker on the server box. Valkey does not scale past ~8 io-threads on\ncurrent hardware, so more than that rarely helps — note the thread count you\nused.\n\n```bash\ndocker run --network=host --rm valkey/valkey:9.0 \\\n --save \"\" --appendonly no --io-threads 8 --protected-mode no --port 6380\n```\n\nFor Redis, substitute the `redis` image and equivalent flags. Run the comparison\ntarget on the **same instance type** as Dragonfly, one at a time (not\nsimultaneously — they'd contend), so the comparison is apples-to-apples.\n\n---\n\n## dfly_bench (default load generator)\n\nSource: `src/server/dfly_bench.cc`. Key flags:\n\n| Flag | Meaning | Typical |\n| --- | --- | --- |\n| `-h` | server host | client→server IP |\n| `-p` | server port | `6380` |\n| `-c` | connections **per thread** | ~10; ≤25 max; 1–5 when pipelining |\n| `-n` | requests per connection | sizes the run (omit if `--test_time`) |\n| `--test_time` | run duration in seconds | e.g. `120` |\n| `--qps` | target QPS per connection; `0` = unthrottled | `0` for peak |\n| `-d` | value size in bytes | `64` for strings tests |\n| `--key_maximum` | size of key space | tune to fill memory |\n| `--key_dist` | `U` uniform, `N` normal, `Z` zipfian, `S` sequential | `U` default |\n| `--ratio` | set:get ratio | `1:0` writes, `0:1` reads |\n| `--command` | custom command template (`__key__`, `__data__`) | for non-string workloads |\n| `--pipeline` | pending requests per connection | `>1` for peak throughput |\n| `-P` | `memcache_text` for memcached protocol | empty for RESP |\n| `--json_out_file` | **memtier-compatible JSON with per-second time series** | always set it |\n\n`--json_out_file` only works with the built-in GET/SET workload (not with\n`--command`, `--noreply`, or `--connect_only`). For custom-command workloads you\nscrape the console summary instead.\n\n**Console output.** Periodic lines look like:\n\n```\n12s: 40.0% done, RPS(now/agg): 815000/812340, errs: 0, hitrate: 99.8%, clients: 200\n```\n\nand the final summary:\n\n```\nTotal time: 30s. Overall number of requests: 24370200, QPS: 812340, P99 lat: 787us\nLatency summary, all times are in usec:\n<histogram>\n```\n\nSave the full stdout — the histogram and final QPS go in the appendix; the\n`collect_metrics.py` script prefers the JSON but can fall back to this.\n\nAlso save the command lines themselves under the run output directory, e.g.\n`commands_audit.log`. Record setup, server launch, write/read load,\ncapture, diagnostics, and chart/report commands as they are run; stdout alone is\nnot raw data because it cannot reproduce the run. Initialize this file before\nthe first workload so planned script invocations are visible up front, then\nappend commands as they execute.\n\n**Use the skill scripts for the main run path.** The main benchmark phases should\nuse:\n\n```bash\n.claude/skills/benchmark/scripts/bench_server.sh start|start-docker|stop|stop-docker|status ...\n.claude/skills/benchmark/scripts/monitor_fill.sh ...\n.claude/skills/benchmark/scripts/capture_result.sh ...\n.claude/skills/benchmark/scripts/collect_metrics.py ...\n.claude/skills/benchmark/scripts/make_charts.py ...\n.claude/skills/benchmark/scripts/plot_fill.py ...\n```\n\nInline SSH is acceptable for discovery, package/setup checks, bounded\nforeground probes, and diagnostics. If you bypass a run-control script for a\nmain benchmark phase, record the reason in `commands_audit.log` and the appendix before\nrunning the manual equivalent.\n\n**Keep orchestration phase-separated.** Do not run a single local shell command\nthat starts a remote tmux benchmark, polls until completion, captures results,\nand starts the next phase. If the local command is interrupted, the remote tmux\nprocess can finish while the local capture logic is gone, leaving the run idle\nand uncaptured. Prefer bounded foreground `dfly_bench` commands for known-length\nruns. If you need tmux/nohup for a long fill, use separate steps: launch it,\nmonitor it, verify it has exited, then run the capture/read commands separately.\n\n**JSON structure** (`--json_out_file`):\n\n```json\n{\n \"configuration\": { \"server\": \"...\", \"port\": 6380, \"clients\": 200,\n \"threads\": 8, \"pipeline\": 1, \"ratio\": \"1:0\" },\n \"ALL STATS\": {\n \"Runtime\": { \"Total duration\": 30000 },\n \"Sets\": { \"Count\": ..., \"Average Latency\": ..., \"Percentile ...\": ...,\n \"Time-Serie\": { \"0\": {...}, \"1\": {...}, ... } },\n \"Gets\": { ... }\n }\n}\n```\n\nThe per-second `Time-Serie` buckets are what the charts plot.\n\n---\n\n## memtier_benchmark (alternative)\n\nIndustry-standard, available as a docker image (`redislabs/memtier_benchmark`).\nUse it when the user wants cross-validation against a tool Dragonfly doesn't\nship, or for the pipelined peak-throughput use-case. Example (from the repo's\nk8s job):\n\n```bash\nmemtier_benchmark -s <server> -p 6380 \\\n --pipeline=30 --key-maximum=100000 -c 10 -t 2 --test-time=120 \\\n --distinct-client-seed --hide-histogram --json-out-file memtier_out.json\n```\n\n`--json-out-file` produces the same shape `collect_metrics.py` and\n`make_charts.py` read.\n\n---\n\n## The standard strings workload\n\nThis reproduces the published \"Simple Strings\" comparison. Run write then read,\nwith a memory snapshot in between.\n\n**Write phase** (load the keyspace):\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 -n $N --ratio 1:0 --json_out_file df_write.json\n```\n\nThe canonical strings test uses the built-in SET path (`--ratio 1:0`) so it\nproduces the JSON/time series the charts need. If TTLs matter, swap in\n`--command \"setex __key__ 10000 __data__\"` — but note `--json_out_file` is then\nignored, so for SETEX save stdout and parse/chart the periodic `RPS(now/agg)`\nlines or `monitor_fill.sh` output instead.\n\n**Memory snapshot** (immediately after writes — see next section).\n\n**Read phase**:\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n -n $N --ratio 0:1 --json_out_file df_read.json\n```\n\nChoose `KMAX`, `N`, and `CONN` to (a) saturate the server and (b) load enough\ndata that instance memory is well utilized — see [Sizing the load](#sizing-the-load).\n\n---\n\n## Measuring memory\n\n\"Bytes per entry\" is the memory metric in the report. Capture it right after the\nwrite phase, before reads:\n\n```bash\nredis-cli -h <server> -p 6380 INFO memory > info_memory.txt\nredis-cli -h <server> -p 6380 DBSIZE\n```\n\nFrom `INFO memory` you need `used_memory` (logical) and `used_memory_rss`\n(physical resident set). Report **both** per-entry figures:\n\n- `used_memory / DBSIZE` — logical bytes per entry (matches the published\n \"Memory Per Entry\").\n- `used_memory_rss / DBSIZE` — physical bytes per entry, the real footprint\n including allocator overhead and fragmentation.\n\n`collect_metrics.py` computes both when given `--info` and `--dbsize`.\n\nFor Valkey/Redis the same `INFO memory` fields exist, so the comparison is\ndirect. Capture the snapshot at the **same keyspace size** for both, or the\nper-entry comparison is unfair.\n\n---\n\n## Monitoring stack (Prometheus + Grafana)\n\nLives in `tools/local/monitoring/` and runs on the **server** machine via docker\ncompose. It gives continuous, 1-second-resolution time series for the whole\nbenchmark window — server-side command rates and memory, plus host CPU/network\nfrom node-exporter — which is both a sanity check (was the server actually\nsaturated? did it swap?) and the source of the report's best charts.\n\n```bash\ncd ~/projects/dragonfly/tools/local/monitoring\ndocker compose up -d # Prometheus, Grafana, node-exporter\ndocker compose --profile redis up -d # also start redis-exporter for Valkey/Redis\n```\n\nEndpoints (on the server box): Prometheus `:9090`, Grafana `:3000`,\nnode-exporter `:9100`, redis-exporter `:9121`.\n\n**What gets scraped:**\n\n- `dragonfly` job — Dragonfly exposes Prometheus metrics directly at\n `/metrics` on its main port. The bundled `prometheus.yml` targets\n `host.docker.internal:6379`.\n- `node-exporter` job — host CPU, memory, network.\n- `redis-exporter` job (profile `redis`) — Valkey/Redis stats, also via\n `host.docker.internal:6379`.\n\n**Port — standardize on 6380.** Always run benchmarked servers (Dragonfly,\nValkey, Redis) on port `6380`. The shipped `prometheus.yml` targets `6379`, so\nbefore bringing the stack up, edit the `dragonfly` and `redis-exporter` jobs in\n`tools/local/monitoring/prometheus/prometheus.yml` to point at `:6380`:\n\n```bash\nsed -i 's/:6379/:6380/g' ~/projects/dragonfly/tools/local/monitoring/prometheus/prometheus.yml\n# if Prometheus is already up, pick up the change:\ndocker compose restart prometheus\n```\n\nThen confirm targets are `UP` at `http://<server>:9090/targets` before starting\nload — otherwise Prometheus silently collects nothing for the server.\n\n**Dashboards** are pre-provisioned in Grafana (`dragonfly.json`, `redis.json`,\n`node-exporter.json`) — no manual import. For the report, note the benchmark\nstart/end timestamps so you can crop the Grafana time range to the exact run,\nthen capture the relevant panels (throughput, memory, CPU) — via the provisioned\n`grafana-image-renderer` (the compose file includes a `renderer` service) or a\nscreenshot — and embed them. You can also query Prometheus directly over HTTP\n(`/api/v1/query_range`) if you want raw series for `make_charts.py`.\n\nLeave the stack up across all runs so one continuous window covers everything.\n\n## Network tuning\n\nOn cloud hardware, peak throughput is gated by how network-interrupt load is\nspread across CPUs. The published runs tuned **SMP affinity for IRQs** so the\nNIC's interrupts fan out across multiple cores instead of saturating one.\n\nThis is instance- and NIC-specific. The general approach: identify the NIC's IRQs\n(`/proc/interrupts`), then write CPU masks to `/proc/irq/<N>/smp_affinity` to\nspread them. Many setups use the kernel's `set_irq_affinity.sh` helper shipped\nwith the NIC driver. Because the exact steps depend on the instance, confirm the\napproach with the user if they want tuning, and **record the exact commands you\nran** — tuned and untuned numbers are not comparable, and the appendix must say\nwhich was used.\n\nIf you skip tuning, state that plainly in the report.\n\n---\n\n## Sizing the load\n\nTwo independent things to size: **how many keys** (sets the memory footprint)\nand **how long / how many requests** (sets steady-state and how full memory\ngets). Get these wrong in either direction and the benchmark is misleading.\n\n### The memory budget — the most important (and most dangerous) call\n\nA write benchmark must land in the **sweet spot between two failure modes**:\n\n- **Too few keys → wrong-but-safe.** Benchmarking a nearly-empty instance is\n not representative. A small dataset sits in CPU cache and hides the real cost\n of a large hash table (cache misses, allocator behavior, fragmentation). The\n QPS looks great and means nothing. *Using available memory is the whole\n point.*\n- **Too many keys → OOM or eviction.** Push too far and the run is ruined two\n different ways (see the two ceilings below). Either way the numbers are\n garbage.\n\n**There are TWO ceilings, not one. Stay under both.**\n\n1. **Dragonfly's `maxmemory`.** Dragonfly auto-sets `maxmemory` to a fraction of\n RAM (≈ 78–80% by default). When memory crosses it, Dragonfly **rejects or\n evicts** writes — the load generator reports *error responses* and the run is\n contaminated. **Check it before the run** (`redis-cli -p 6380 config get\n maxmemory`, or the `maxmemory:` line in `INFO memory`).\n\n **Rejections start with a margin *below* `maxmemory`, not at it.** During a\n write, dash-table resizes transiently spike memory, so Dragonfly begins\n rejecting while steady-state `used_memory` is still well under the cap —\n observed rejections at `used_memory` ≈ 90% of `maxmemory` and even lower at the\n moment of a big resize. So keep steady-state `used_memory` to **≤ ~80% of\n `maxmemory`** to leave room for these spikes. If you need to fill more, raise\n `--maxmemory` explicitly — but then ceiling #2 (host RAM) binds.\n\n **Zero errors is the pass/fail bar.** *Any* eviction/OOM error contaminates\n the experiment — it slows the run (rejection stalls inflate P99) and stops the\n keyspace from growing, so QPS, latency, and bytes-per-entry are all off. After\n every write, **verify the load generator reported zero errors**\n (`grep -i error <write.log>`; dfly_bench prints `Got N error responses!`). If\n N > 0, the run is **invalid** — discard it, cut `--key_maximum` (start with a\n ~10–15% reduction), and re-run until errors are zero. Don't report a run with\n any errors, and don't try to \"subtract\" them out.\n2. **Host physical RAM, measured by RSS.** The accurate measure of real memory\n pressure is **resident set size — `used_memory_rss`** (equivalently the\n process RSS). True OOM happens when RSS approaches host RAM. Do **not** panic\n over `free`'s `used`/`available` columns during a write: a high-rate write\n inflates **page cache** (clean, reclaimable) and dirty pages, which makes\n `available` dip alarmingly even though the kernel will reclaim it under\n pressure. On a 30 GiB box a 195M-key dataset showed `free` available dropping\n to ~10% while `used_memory_rss` was only ~21 GiB (71% of RAM) — the dataset\n was nowhere near OOM; the rest was reclaimable cache. **Track RSS, keep it\n below host RAM with a ≥5% margin**, and ignore page-cache noise.\n\n**So size against RSS.** Target the dataset's RSS up to your fill goal while\nkeeping a real margin, and stay under `maxmemory`:\n\n```\ntarget_rss = host_RAM * fill_fraction # e.g. 0.90 for \"fill 90%\", never > 0.95\nrss_per_entry = used_memory_rss / DBSIZE # from calibration (accurate)\nkey_for_host = target_rss / rss_per_entry\nkey_for_maxmem = 0.90 * maxmemory / used_memory_per_entry # avoid eviction\ntarget_keys = min(key_for_host, key_for_maxmem)\n--key_maximum = target_keys\n```\n\nThe usual binding constraint is **`maxmemory`**, since its default (~78% of RAM)\nis below a 90% fill target. **To genuinely fill 90% of RAM, raise `--maxmemory`\nexplicitly** (e.g. `--maxmemory 28gb` on a 30 GiB box) so Dragonfly doesn't evict\nbefore RSS reaches the target — then RSS, watched live, is the safety signal.\nFilling memory is the goal; the failure modes to avoid are eviction (used_memory\n> maxmemory) and RSS approaching host RAM.\n\n`*_per_entry` (≈ `value_size + 60–100 B` overhead for small strings; more for\nTTL'd keys and other types) is best **calibrated**, not guessed:\n\n1. Run a short pilot write inserting a known, modest number of distinct keys\n (e.g. `--key_maximum=5000000 --key_dist S` with enough requests to write them\n all).\n2. Read `used_memory` / `DBSIZE` → real `bytes_per_entry`. Note this slightly\n *under*-estimates at scale (fragmentation grows), so keep margin.\n3. Plug into the formula above for the full run.\n\nCalibrate **per datastore** — Dragonfly and Valkey have different per-entry\noverhead, which is exactly what the report measures. Size each so its dataset\nfills the same target fraction; comparing at the same `DBSIZE` keeps the\nbytes-per-entry comparison fair (Dragonfly will simply use less RAM at that\nkey count, which is the finding, not a sizing error).\n\n### How distinct keys actually accumulate\n\nWith uniform random keys over `--key_maximum`, distinct keys (and therefore\nmemory) climb fast at first, then asymptote toward `key_maximum` as repeats\nstart landing on existing keys. **Memory plateaus near `key_maximum` distinct\nentries** — writing *longer* past that point just overwrites existing keys and\ndoes **not** grow memory. That's the key safety property: **`--key_maximum`, not\nrun length, bounds the footprint.** Run length controls how *close* to that\nplateau you get and how long you hold steady state.\n\nSo to reliably fill to the target without OOM:\n\n- Set `--key_maximum` to the computed `target_keys`.\n- Make the write phase long enough to insert essentially all of them — total\n requests (`-c` × threads × `-n`, or a `--test_time` that covers it) should be\n a small multiple of `key_maximum` (~2–3×) so coverage is near-complete.\n- Use `--key_dist S` (sequential) if you want *exactly* `key_maximum` distinct\n keys with no repeats and a precise fill; uniform is fine when you just need to\n approach the plateau.\n\n### Watch memory live — guard on RSS\n\nNever run a long write blind. The accurate safety signal is **`used_memory_rss`**\n(resident set), not `free`'s `available` (page cache makes it dip transiently and\nis reclaimable). Poll RSS every few seconds and **abort the write if RSS exceeds\n~90–95% of host RAM**, and separately watch `used_memory` against `maxmemory` to\ncatch approaching eviction. Example guard loop:\n\nRun the guard as its own bounded step, not as part of a larger launch/poll/capture\nmega-command. After the guard reports `DONE`, immediately run a separate capture\nstep (`DBSIZE`, `INFO memory`, stdout log, and zero-error check), then start the\nread phase.\n\n```bash\nTOTAL=$(ssh <server> 'free -b | awk \"/^Mem:/{print \\$2}\"')\nfor i in $(seq 1 60); do\n sleep 8\n read DB RSS USED < <(ssh <server> 'redis-cli -p 6380 dbsize | tr -d \"\\r\"; \\\n redis-cli -p 6380 info memory | grep -E \"^used_memory_rss:|^used_memory:\" | cut -d: -f2 | tr -d \"\\r\"' | paste -sd\" \")\n pct=$(awk \"BEGIN{printf \\\"%.0f\\\", $RSS/$TOTAL*100}\")\n echo \"dbsize=$DB rss=$(awk \"BEGIN{printf \\\"%.1f\\\",$RSS/2^30}\")GiB (${pct}% of RAM)\"\n if [ \"$pct\" -ge 92 ]; then echo \"ABORT: RSS ≥92% of RAM\"; \\\n ssh <client> 'tmux kill-session -t bench'; break; fi\ndone\n```\n\nWays to read current memory:\n\n- **`INFO memory` → `used_memory_rss`** — the accurate measure; the guard's\n input. Also compare `used_memory` to `maxmemory` (eviction warning).\n- **`/metrics` endpoint (Dragonfly only)** — `curl -s\n http://<server>:6380/metrics | grep -i mem` gives Dragonfly's live RSS/used\n gauges without a redis client. **Valkey/Redis do not serve `/metrics`** — for\n those, use `INFO memory` directly or scrape via the redis-exporter (see the\n monitoring section). `INFO memory` works for all three datastores, so it's the\n portable choice for the guard.\n- **Monitoring stack** — Prometheus/Grafana plots these over time; the source of\n the memory-over-time chart. (`free`/node-exporter available is fine for the\n chart, just not the trigger.)\n\nIf RSS approaches the limit faster than expected, **stop the write early** —\nyou've already filled enough to measure; a partially-filled run with clean\nnumbers beats one that evicted or OOMed.\n\n### The other knobs\n\n- **Connections (`-c` is *per thread*):** keep `-c` modest — **never above 25\n connections per thread, usually around 10**; for pipelined runs drop it to\n **1–5 per thread** (each connection already has many requests in flight, so\n more connections just add contention without more throughput). Scale total\n load by adding **threads** (and, if needed, a second client box), not by\n cramming connections onto each thread. Raise the total until QPS stops rising;\n if it keeps climbing at your max, the client is the bottleneck — add threads or\n another client machine, not more `-c`.\n- **Duration / steady state:** beyond filling memory, hold at least 30–60s of\n steady traffic so the reported QPS/P99 reflect steady state, not warm-up.\n Watch the periodic RPS line stabilize before trusting the number.\n- **Pipeline:** Use `--pipeline 30` for all write/prefill phases — without it\n the client stalls on each reply and becomes the bottleneck before the server\n saturates. Use `--pipeline 1` (or omit) only for latency-honest read\n benchmarks, and label those numbers accordingly.\n\nAlways note in the report the key count, value size, resulting memory fill (%\nof instance RAM), and whether the run was client-bound — an under-driven server\nor a barely-filled instance both produce numbers that quietly mislead.\n",".claude/skills/reproduce-fuzz-crash/SKILL.md":"---\nname: reproduce-fuzz-crash\ndescription: >\n Reproduce AFL++ fuzz crashes from GitHub Actions. Use when user provides a\n GitHub Actions fuzz run URL and wants to reproduce and analyze the crash locally.\nargument-hint: <github-actions-run-url>\nallowed-tools: Bash, Read, Grep, Glob, Write\n---\n\n# Reproduce Fuzz Crash\n\nGiven a GitHub Actions fuzz run URL, download crash artifacts, triage them\nwith `fuzz/triage_crashes.sh`, and produce a crash analysis report.\n\n**Input**: `$ARGUMENTS` — a GitHub Actions run URL like:\n`https://github.com/dragonflydb/dragonfly/actions/runs/22906484769`\nor with query params like `?pr=6855`.\n\n## Workflow\n\n### Step 1: Parse the URL\n\nExtract `owner/repo` and `run_id` from the URL.\n\n```\nhttps://github.com/{owner}/{repo}/actions/runs/{run_id}[?...]\n```\n\nStrip any query parameters from `run_id`.\n\n### Step 2: Download artifacts\n\nList crash artifacts via the GitHub API, then download each as a `.zip` directly:\n\n**IMPORTANT**: Run each command as a separate Bash tool call (no `&&` chaining)\nto ensure auto-approval works with the user's permission patterns.\n\n```bash\n# List artifacts — filter for names containing \"crash\"\ngh api repos/{owner}/{repo}/actions/runs/{run_id}/artifacts\n\n# Create output directory\nmkdir -p /tmp/fuzz-repro-{run_id}\n\n# Download each crash artifact by ID (separate command)\ngh api repos/{owner}/{repo}/actions/artifacts/{artifact_id}/zip > /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nThis gives real `.zip` files that the triage script can consume directly.\n\nIf no crash artifacts are found, report that the run has no crash artifacts and stop.\n\nNote: there may be duplicate artifact names (same name, different IDs) from\nretried jobs. Download the **most recent** one (highest artifact ID).\n\n### Step 3: Determine mode\n\nInfer the protocol mode from the artifact name:\n- Contains \"memcache\" → `memcache`\n- Otherwise → `resp`\n\n### Step 4: Check Dragonfly binary\n\nCheck if the debug binary already exists and runs:\n\n```bash\n./build-dbg/dragonfly --version\n```\n\nOnly build if the binary doesn't exist or fails to run:\n\n```bash\nninja -C build-dbg dragonfly\n```\n\nIf `build-dbg` doesn't exist, run `./helio/blaze.sh` first.\n\n### Step 5: Run triage_crashes.sh\n\nFor each zip file, run:\n\n```bash\n./fuzz/triage_crashes.sh ./build-dbg/dragonfly <mode> /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nCapture the full output.\n\n### Step 6: Analyze and report\n\nParse the triage output for confirmed crashes. For each confirmed crash:\n\n1. **Read the source** at the crash location — use the stack trace to identify\n the source file and line number, then read that code.\n2. **Provide analysis**: likely root cause, what to investigate.\n\nPrint a structured report:\n\n```\n## Fuzz Crash Report\n\n**Run**: {url}\n**Artifacts**: {number} crash(es) found\n\n---\n\n### Crash NNNNNN\n\n**Reproduced**: Yes / No (false positive)\n**Signal**: SIGABRT (6) / SIGSEGV (11) / etc.\n\n**Stack trace**:\n\\```\n<stack trace from triage output>\n\\```\n\n**Analysis**:\n<1-3 sentences explaining the likely root cause based on the stack trace,\nthe assertion message, and the crash input. Identify the source file and\nline number. Suggest what to investigate.>\n```\n\n## Important Notes\n\n- The triage script uses port **6379** (resp) or **11211** (memcache).\n Ensure no other Dragonfly or Redis instance is using these ports.\n- The script adds `--restricted_commands` flags to avoid false positives from\n commands like DEBUG SLEEP that the fuzzer might generate.\n- Some crashes are non-deterministic (thread timing). The script reports\n these as \"FALSE POSITIVE\" — note this clearly, it doesn't mean the bug\n is invalid, just that it didn't reproduce on this run.\n- The script handles its own cleanup of Dragonfly processes.\n- Do NOT delete `/tmp/fuzz-repro-{run_id}/` — the user may want to inspect it.\n- If `gh run download` fails with permissions, suggest the user authenticate\n with `gh auth login`.\n",".github/copilot-instructions.md":"---\ndescription: 'Code review guidelines for GitHub copilot in this project'\napplyTo: '**'\nexcludeAgent: [\"coding-agent\"]\n---\n\n# Code Review Instructions\n\nKeep reviews high-signal and minimal. Only comment on real bugs with high confidence.\n\n## Comment Only When\n- The issue is a correctness, security, concurrency, or architecture problem.\n- The impact is clear and non-trivial.\n- You can point to concrete evidence in the diff (not speculation).\n\n## Avoid\n- Style, formatting, naming, or minor performance nits.\n- Optional refactors or “nice to have” suggestions.\n- Praise, restating the code, or long explanations.\n- Duplicate comments for the same root cause.\n\n## Review Style\n- Be terse: 1-2 sentences per issue.\n- Include file and line references when possible.\n- If no issues are found, say “No issues found.”\n- Provide concrete suggestions for fixes when possible, or examples to illustrate the problem.\n","AGENTS.md":"# Dragonfly Development Guide\n\n> **Essential reference for working with the Dragonfly codebase**\n> Architecture, build system, testing infrastructure, and development workflows.\n\n---\n\n## Table of Contents\n\n1. [Critical Workflow Rules](#critical-workflow-rules)\n2. [Quick Command Reference](#quick-command-reference)\n3. [Project Overview](#project-overview)\n4. [Repository Structure](#repository-structure)\n5. [Build Instructions](#build-instructions)\n6. [Testing](#testing)\n7. [CI/CD Pipeline](#cicd-pipeline)\n8. [Code Style & Pre-commit Hooks](#code-style--pre-commit-hooks)\n9. [Third-Party Dependencies](#third-party-dependencies)\n10. [Platform Support](#platform-support)\n11. [CMake Build Options](#cmake-build-options)\n12. [Key Files Reference](#key-files-reference)\n13. [Common Pitfalls](#common-pitfalls)\n14. [Debugging Tips](#debugging-tips)\n15. [Validation Checklist](#validation-checklist)\n\n---\n\n## Critical Workflow Rules\n\n**MANDATORY - Always Follow This Order:**\n\n1. ✅ **Read Before Edit** - Always read files before modifying\n2. ✅ **Use Correct Build Commands** - See [Quick Command Reference](#quick-command-reference) below\n3. ✅ **Test After Changes** - Build and run a relevant unit test -\n `ninja <unit_test> && ./unit_test`\n4. ✅ **Format Code** - `pre-commit run --files <files>`\n5. ✅ **Follow Architecture** - See [Architecture Patterns](#architecture-patterns) below\n6. ✅ **Never Push to Main** - Always create a feature branch and open a PR. Never run `git push origin main`.\n\n### Pull Request Guidelines\n\n**Conciseness is Key**: PR descriptions should be short, focused, and easy to scan.\n- **Title**: Imperative, descriptive (e.g., \"Fix fiber stack overflow in test_reply_guard_oom\")\n- **Summary**: 1-2 sentences explaining *what* changed and *why*\n- **Changes**: Bullet points for key changes\n- **Fixes**: Link issues (e.g., \"Fixes #123\")\n- **Commit messages**: Keep every line (subject and body) <= 100 characters; wrap long descriptions\n\n---\n\n## Quick Command Reference\n\n**CRITICAL: Read the full sections below for context. These are shortcuts only.**\n\n### Building (see [Build Instructions](#build-instructions) for details)\n\n```bash\n# Debug build (for development)\n./helio/blaze.sh -DWITH_GCP=OFF\ncd build-dbg && ninja dragonfly # Build main binary\ncd build-dbg && ninja generic_family_test # Build specific test\n\n# Release build for local benchmarking\n./helio/blaze.sh -release -DWITH_GCP=OFF\ncd build-opt && ninja dragonfly\n```\n\n### Testing (see [Testing](#testing) for details)\n\n```bash\n# C++ Unit Tests\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n### Code Formatting\n\n```bash\n# Setup (once)\npipx install pre-commit clang-format black\npre-commit install\n\n# Format code\npre-commit run --files <files> # Format specific files\npre-commit run --all-files # Format all files\n```\n\n### Common Operations\n\n```bash\n# Check git status\ngit status\n\n# Check current branch\ngit branch\n\n# View recent commits\ngit log --oneline -10\n```\n\n---\n\n## Architecture Patterns\n\n**Code Style**: [.clang-format](.clang-format) - snake_case vars, PascalCase functions, kPascalCase constants\n\n**Comments**: Always put an empty line before a comment in code, unless the comment is the first\nline of a block (right after `{`). This applies to comments between statements, before if/for, and\ninside lambdas.\n\n**DO ✅**:\n- Function parameters: inputs first, output arguments (pointers the function writes to) last\n- Fiber-aware: `util::fb2::Mutex`, `util::fb2::Fiber` → [helio/util/fibers/](helio/util/fibers/)\n- Per-shard ops (no global state) → [docs/df-share-nothing.md](docs/df-share-nothing.md)\n- Command pattern → [src/server/set_family.cc](src/server/set_family.cc)\n- Error handling: `OpStatus` → [src/server/common.h](src/server/common.h)\n- Test patterns → [tests/dragonfly/conftest.py](tests/dragonfly/conftest.py)\n\n**DON'T ❌**:\n- `std::thread`, `std::mutex` (deadlocks!)\n- Global mutable state\n- Edit without reading\n- Skip tests\n- Use `std::regex` in fiber/server paths (recursive implementation can overflow small fiber stacks)\n- Use `./tools/docker/build.sh` for local development (use `ninja` instead)\n- Use `make` for incremental builds (use `ninja` instead)\n\n---\n\n## Project Overview\n\n**Dragonfly** is a high-performance, Redis and Memcached compatible in-memory data store written in C++20. It delivers significantly higher throughput than traditional single-threaded Redis implementations through innovative architectural choices.\n\n### Key Characteristics\n\n- **Language**: C++20 (Google C++ Style Guide 2020 version)\n- **Architecture**: Shared-nothing multi-threaded design (via `helio` library)\n- **Performance**: Uses io_uring (Linux 5.11+) for high-performance async I/O, with epoll fallback\n- **Threading Model**: Fiber-based cooperative multitasking with lock-free data structures\n- **Build System**: CMake + Ninja via `helio/blaze.sh` wrapper script\n- **Target Platform**: Linux (kernel 5.11+ recommended), FreeBSD support available\n- **Protocols**: Redis RESP2/RESP3, Memcached binary protocol\n- **Compatibility**: Drop-in replacement for Redis API coverage\n\n### Architectural Highlights\n\n**For detailed architecture documentation, see [docs/df-share-nothing.md](docs/df-share-nothing.md)**\n\n1. **Shared-Nothing Design**: Each thread operates independently with its own data structures, minimizing lock contention\n2. **Helio Framework**: Custom I/O and threading library built on io_uring/epoll with fiber support\n3. **DashTable**: Novel hash table implementation optimized for multi-core systems - see [docs/dashtable.md](docs/dashtable.md)\n4. **Transaction Model**: Non-blocking optimistic transactions - see [docs/transaction.md](docs/transaction.md)\n5. **Tiering Support**: Optional disk-backed storage for large datasets\n6. **Search Module**: Full-text search capabilities (when enabled with WITH_SEARCH)\n\n---\n\n## Repository Structure\n\n```\ndragonfly/\n├── src/ # Main C++ source code\n│ ├── server/ # Core server implementation\n│ │ ├── dfly_main.cc # Main entry point\n│ │ ├── main_service.cc # Service lifecycle & command routing\n│ │ ├── db_slice.cc # Per-thread database shard\n│ │ ├── engine_shard_set.cc # Shard management\n│ │ ├── cluster/ # Cluster mode implementation\n│ │ ├── journal/ # Replication journal\n│ │ ├── tiering/ # Tiered storage\n│ │ ├── search/ # Search module\n│ │ └── acl/ # Access control lists\n│ ├── core/ # Core data structures\n│ │ ├── dash.h # DashTable hash table\n│ │ ├── dense_set.h # Compact set implementation\n│ │ ├── string_map.h # Optimized string-keyed maps\n│ │ ├── search/ # Search core algorithms\n│ │ └── json/ # JSON support\n│ ├── facade/ # Network & command handling\n│ │ ├── dragonfly_connection.cc # Connection management\n│ │ ├── resp_srv_parser.cc # RESP command parser\n│ │ ├── resp_parser.cc # RESP reply parser\n│ │ └── memcache_parser.cc # Memcached protocol\n│ └── redis/ # Redis-specific implementations\n│ └── lua/ # Lua scripting support\n│\n├── helio/ # Git submodule: I/O and threading library\n│ │ # ** DO NOT EDIT unless contributing to helio **\n│ ├── util/ # Utilities: fibers, I/O, synchronization\n│ ├── io/ # io_uring & epoll abstraction\n│ └── blaze.sh # Build configuration wrapper\n│\n├── tests/ # Test suite\n│ ├── dragonfly/ # Python pytest integration/regression tests\n│ │ ├── conftest.py # Pytest fixtures & configuration\n│ │ ├── requirements.txt # Python test dependencies\n│ │ └── *.py # Test files\n│ └── pytest.ini # Pytest configuration & markers\n│\n├── docs/ # Documentation\n│ ├── build-from-source.md # Build instructions\n│ ├── dashtable.md # DashTable internals\n│ ├── transaction.md # Transaction model\n│ ├── df-share-nothing.md # Shared-nothing architecture\n│ └── differences.md # Differences from Redis\n│\n├── contrib/ # Utilities\n│ ├── docker/ # Docker configurations\n│ └── charts/dragonfly/ # Helm chart for Kubernetes\n│\n├── tools/ # Benchmarking & utility tools\n│ └── packaging/ # Packaging scripts\n│\n├── CMakeLists.txt # Root CMake configuration\n├── .clang-format # C++ formatting rules (clang-format v14.0.6)\n├── .pre-commit-config.yaml # Pre-commit hooks configuration\n├── pyproject.toml # Python formatting (Black, 100 chars)\n└── CONTRIBUTING.md # Contribution guidelines\n```\n\n### Critical Paths to Remember\n\n- **Main entry**: `src/server/dfly_main.cc`\n- **Command dispatch**: `src/server/main_service.cc`\n- **Data storage**: `src/server/db_slice.cc`\n- **Networking**: `src/facade/dragonfly_connection.cc`\n- **Helio library**: `helio/` (I/O and threading library)\n\n---\n\n## Build Instructions\n\n**For complete build instructions, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Quick Start\n\n**Debug build** (for development):\n```bash\n./helio/blaze.sh\ncd build-dbg && ninja dragonfly\n./dragonfly --alsologtostderr\n```\n\n**Release build** (for production/benchmarking):\n```bash\n./helio/blaze.sh -release\ncd build-opt && ninja dragonfly\n```\n\n**Production release build** (static linking, optimized):\n```bash\nmake release # Configure + build\nmake package # Create release packages with debug symbols\n```\n\nThe [Makefile](Makefile) builds production releases with:\n- Static linking: libstdc++, libgcc, Boost, OpenSSL\n- Architecture optimizations (x86_64: `-march=core2 -msse4.1 -mtune=skylake`)\n- Debug symbols (compressed)\n- Output: `build-release/dragonfly-{arch}.tar.gz`\n\n**Common build options**:\n- See [docs/build-from-source.md](docs/build-from-source.md) for all options\n\n---\n\n## Testing\n\n**For complete testing documentation, see [tests/README.md](tests/README.md)**\n\n### Quick Reference\n\n**C++ Unit Tests**:\n```bash\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n**Python Integration Tests (pytest)**:\n```bash\n# Run from the repo root. The binary path defaults to build-dbg/dragonfly.\n# Override with the DRAGONFLY_PATH env var:\nDRAGONFLY_PATH=build-dbg/dragonfly python3 -m pytest tests/dragonfly/pymemcached_test.py -xvs\n\n# Run a single test:\npython3 -m pytest tests/dragonfly/pymemcached_test.py::TestMemcached::test_basic -xvs\n```\n\n- `DRAGONFLY_PATH` — sets the path to the Dragonfly binary the test harness starts. Defaults to `build-dbg/dragonfly` relative to the `tests/dragonfly/` directory.\n- `--df` — passes **extra flags to the Dragonfly process** (not the binary path). For example: `--df logtostdout --df \"vmodule=*=1\"`.\n\n---\n\n## CI/CD Pipeline\n\n**For complete CI configuration, see [.github/workflows/ci.yml](.github/workflows/ci.yml)**\n\nThe CI workflow runs on all PRs and includes:\n- **Pre-commit checks**: clang-format, black formatters\n- **Build matrix**: Multiple OS/compiler/sanitizer combinations (Ubuntu 20/24, Alpine, GCC/Clang, ASAN/UBSAN)\n- **Test execution**: C++ unit tests, Python integration tests, cluster mode tests\n- **Additional validations**: Helm charts, Docker image builds\n\n---\n\n## Code Style & Pre-commit Hooks\n\n**For complete contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md)**\n\n**Code style configuration files**:\n- **C++**: [.clang-format](.clang-format) - Google C++ Style Guide (2020), clang-format v14.0.6, 100 char limit\n- **Python**: [pyproject.toml](pyproject.toml) - Black formatter, 100 char limit, PEP 8 compliant\n- **Pre-commit hooks**: [.pre-commit-config.yaml](.pre-commit-config.yaml) - Automated formatting checks\n\n**Quick setup**:\n```bash\npipx install pre-commit clang-format black\npre-commit install\npre-commit run --all-files # Run all formatters\n```\n\n---\n\n## Third-Party Dependencies\n\n**Key Libraries**: Abseil (strings/flags), Boost 1.71+ (context/intrusive), mimalloc (allocator), jsoncons (JSON), OpenSSL (TLS), libunwind (traces)\n\n**Build artifacts**: `build-dbg/third_party/` - DO NOT edit\n\n**For complete dependency info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## Platform Support\n\n**Linux**: Primary platform. Kernel 5.11+ (io_uring), 5.1+ (basic), < 5.1 (epoll fallback)\n- Check: `uname -r`\n- Force epoll: `--proactor_type=epoll`\n- Docker: `--security-opt seccomp=unconfined`\n\n**FreeBSD**: Supported (kqueue backend)\n\n**macOS**: Not supported for production (use Docker/Linux)\n\n**For complete platform info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## CMake Build Options\n\n**For complete list of build options, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Common Options\n\nPass options to `helio/blaze.sh` with `-D` prefix:\n\n```bash\n./helio/blaze.sh -DWITH_SEARCH=OFF -DWITH_GCP=OFF\n```\n\n**Most useful options**:\n- `WITH_ASAN=ON` / `WITH_USAN=ON` - Enable sanitizers for debugging\n- `WITH_SEARCH=OFF` - Disable search module for faster builds\n- `WITH_GCP=OFF` - Disable GCP cloud library\n- `WITH_TIERING=OFF` - Disable disk storage\n- `USE_MOLD=ON` - Faster linking with LTO (production builds)\n\n**Quick configurations**:\n```bash\n# Minimal build (fast compilation)\n./helio/blaze.sh -DWITH_GPERF=OFF -DWITH_GCP=OFF -DWITH_TIERING=OFF -DWITH_SEARCH=OFF\n\n# Full-featured (all options ON by default)\n./helio/blaze.sh\n\n# Production optimized\n./helio/blaze.sh -release -DUSE_MOLD=ON\n```\n\n---\n\n## Key Files Reference\n\nQuick reference to the most important files in the codebase.\n\n| Purpose | File Path |\n|---------|-----------|\n| **Entry Points & Core** | |\n| Main entry point | `src/server/dfly_main.cc` |\n| Server lifecycle & command routing | `src/server/main_service.cc` |\n| Per-thread database shard | `src/server/db_slice.cc` |\n| Shard management | `src/server/engine_shard_set.cc` |\n| **Data Structures** | |\n| DashTable hash table | `src/core/dash.h` |\n| Dense set implementation | `src/core/dense_set.h` |\n| String map | `src/core/string_map.h` |\n| **Networking** | |\n| Connection handling | `src/facade/dragonfly_connection.cc` |\n| RESP command parser | `src/facade/resp_srv_parser.cc` |\n| RESP reply parser | `src/facade/resp_parser.cc` |\n| Memcached protocol parser | `src/facade/memcache_parser.cc` |\n| **Build System** | |\n| Root CMake config | `CMakeLists.txt` |\n| Build script wrapper | `helio/blaze.sh` |\n| Server CMake config | `src/server/CMakeLists.txt` |\n| **CI/CD** | |\n| Main CI workflow | `.github/workflows/ci.yml` |\n| Pre-commit config | `.pre-commit-config.yaml` |\n| **Code Style** | |\n| C++ formatting | `.clang-format` |\n| Python formatting | `pyproject.toml` |\n| **Testing** | |\n| Pytest configuration | `tests/pytest.ini` |\n| Pytest fixtures | `tests/dragonfly/conftest.py` |\n| Test requirements | `tests/dragonfly/requirements.txt` |\n| **Documentation** | |\n| Build instructions | `docs/build-from-source.md` |\n| Architecture overview | `docs/df-share-nothing.md` |\n| DashTable internals | `docs/dashtable.md` |\n| Transaction model | `docs/transaction.md` |\n| **Configuration** | |\n| Contributing guide | `CONTRIBUTING.md` |\n| CLA agreement | `CLA.txt` |\n\n---\n\n## Common Pitfalls\n\n1. **Pre-commit not installed**: `pipx install pre-commit clang-format black && pre-commit install`\n2. **Wrong binary**: Debug: `build-dbg/dragonfly`, Release: `build-opt/dragonfly`\n3. **Wrong build command**: Use `cd build-dbg && ninja <target>`, NOT `./tools/docker/build.sh`\n4. **Test timeouts**: `timeout 20m ctest -V -L DFLY`\n5. **ASAN leaks**: Check CI, suppress in `helio/util/asan_suppressions.txt`\n6. **Helio modifications**: DON'T edit `helio/` (it's a git submodule - changes go upstream)\n7. **CodeQL checks**: DON'T run codeql_checker when testing changes - it's slow and unnecessary for development\n\n---\n\n## Debugging Tips\n\n**Logging**: `--alsologtostderr --v=1 --vmodule=module=2`\n\n**ASAN**: `ASAN_OPTIONS=detect_leaks=1:symbolize=1`, suppressions: `helio/util/asan_suppressions.txt`\n\n**CI reproduction**: See [.github/workflows/ci.yml](.github/workflows/ci.yml)\n\n**Troubleshooting**: Check fiber deadlocks (use `util::fb2` not `std::mutex`), timeout issues (`--test_timeout`), ASAN reports\n\n---\n\n## Validation Checklist\n\nBefore claiming a task is complete, verify:\n\n### Code Quality\n\n- [ ] Code compiles without errors: `cd build-dbg && ninja dragonfly`\n- [ ] Code compiles without warnings (CI uses `-Werror`)\n- [ ] Code follows Google C++ Style Guide (run `clang-format`)\n- [ ] No new ASAN/UBSAN violations\n\n### Testing\n\n- [ ] All existing C++ unit tests pass: `ctest -V -L DFLY`\n- [ ] New feature has corresponding test coverage\n- [ ] Tests pass in both Debug and Release builds\n- [ ] Tests pass with ASAN/UBSAN enabled (if applicable)\n- [ ] **DO NOT run codeql_checker** - it's slow and unnecessary for development testing\n\n### Pre-commit & Style\n\n- [ ] Pre-commit hooks installed: `pre-commit install`\n- [ ] Code formatted with clang-format (C++) and black (Python)\n\n### Documentation\n\n- [ ] Public APIs have comments explaining purpose\n- [ ] Complex algorithms have explanatory comments\n- [ ] README or docs updated if behavior changes\n- [ ] No commented-out code left in final commit\n\n### Performance\n\n- [ ] No obvious performance regressions (run benchmarks if needed)\n- [ ] No unnecessary allocations in hot paths\n- [ ] Lock-free data structures used where appropriate\n","CLAUDE.md":"AGENTS.md","tools/packaging/README.md":"# Installation Packages\n\n## Overview\nThis directory includes a set of files and scripts to build installation package for various Linux distributions.\n\n## Debian\nThe file to build the Debian package all located under \"debian\" directory.\nThe resulting package will install the binary of Dragonfly as well as generate a new service entry for dragonfly,\nthat can be controlled with \"systemctl\" command, to start, stop and check status of.\n### Building\nTo build the package, you have a script called \"generate_debian_package.sh\". This script accepts the following parameters:\n* Optional binary path - the location from which to take the binary for the installation. The default for this is \"repo path/build-opt\".\nThe location to which the resulting package is writing is at the location from which the script is executed.\nThis script is depends on the following packages:\n* git\n* moreutils\n* debhelper\n* dpkg-dev\n\nTo build:\n```\n/path/to/dragonfly/tools/packaging/generate_debian_package.sh [/path/to/dragonfly-binary-file]\n```\n\nThis can only be run on Debian based hosts.\nYou can use the flowing docker file to generate this package:\n```\nFROM ubuntu:20.04\nARG DEBIAN_FRONTEND=noninteractive\nRUN apt update -y && apt-get install -y gcc dpkg-dev gpg vim wget git moreutils debhelper\n```\nBuild the above docker and then run it with your dragonfly source code path mount as volume for the build:\n```\ndocker build -t ubuntu-package .\ndocker run --rm -ti -v /path/to/dragonfly-repo:/mydocker-path ubuntu-package bash\n```\nAgain note that you need to be at \"main\" branch to successfully build this package.\nNote: If at the end of the installation you see a message \"/usr/bin/deb-systemd-helper: error: systemctl preset failed on dragonfly.service: No such file or directory\",\nyou can ignore it, this seem to be related to [the following issue](https://groups.google.com/g/linux.debian.bugs.dist/c/m6xGZ82TdvM).\n","tools/packaging/osrepos/README.md":"# Package repositories for rpm and debian packages\n\nThis directory contains scripts and definitions for setting up YUM and apt repositories for Linux users to install\ndragonfly packages.\n\nThe repositories are served as static websites. The generate-site workflow is used to set up and deploy the sites using\nscripts and definitions included here.\n\nThe workflow does the following tasks:\n\n* Download the latest 5 releases from dragonfly releases page, specifically deb and rpm assets\n * for deb files, only the latest package is downloaded and present (see note below)\n* Set up a directory structure separating deb and rpm files into version specific paths\n* Sign the packages (see note on GPG)\n* Deploy the assets prepared, along with the public GPG key and repo definitions for apt and rpm tooling\n\n## Using the YUM repository\n\nAdd the repository using:\n\n```shell\nsudo dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo\n```\n\nThen install dragonfly as usual, or a specific version:\n\n```shell\nsudo dnf -y install dragonfly-0:v1.33.1-1.fc30.x86_64\n```\n\n## Using the APT repository\n\nFirst download the public GPG key to an appropriate location:\n\n```shell\nsudo curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public\n```\n\nThen add the sources file:\n\n```shell\nsudo curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources\n```\n\nFinally install dragonfly using apt\n\n```shell\nsudo apt update && sudo apt install dragonfly\n```\n\n#### Versions in APT repository\n\nUnlike the yum repo, the apt repo only has the latest version. The reason for this is the tool, `reprepro` supplied by\ndebian to build repositories only supports multiple\nversions in version 5.4 onwards, and the github runner using ubuntu-latest does not have this version.\n\nAnother option would be to use the components feature of apt repositories in the sources file we ask users to install,\nbut then the versions would need\nto be hardcoded in the sources file and the user would have\nto update the file with each new release which makes for a bad user experience. As of now users wanting older packages\nshould download them directly.\n\n### Signing packages\n\nThe packages are signed using the GPG key imported from the secret GPG_PRIVATE_KEY in this repository.\n\nThe corresponding public key is served with site assets, so the apt/yum/dnf based tooling can consume the public key to\nverify package integrity.\n\n### TODO\n\n- [X] debian packages signing (not required? release file is signed)\n- [X] debian repo metadata setup\n- [ ] tests asserting that packages are installable?\n","tools/vector-benches/README.md":"Tool for benchmarking vector search with randomized vectors.\n\n## Logic\nThe tool connects to the Redis/Dragonfly instance.\n1. Checks if the database has enough data (at least 50% of requested `-n`). If not, it **flushes the DB** and generates random vectors.\n2. Checks if the index `idx` exists. If not, it creates it.\n3. Runs concurrent search queries and reports latency/QPS.\n\n## Arguments\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `-n` | 50000 | **Number of vectors** to populate if the DB is empty. |\n| `-q` | 1000 | **Total number of queries** to run during the benchmark. |\n| `-t` | 8 | **Query threads**. Number of concurrent workers sending queries. |\n| `-d` | 100 | **Vector dimension**. Size of the float32 vectors. |\n| `-k` | 10 | **Top K**. Number of nearest neighbors to retrieve per query. |\n| `-p` | 6379 | **Port** of the server. |\n| `-h` | localhost | **Host** of the server. |\n\nRun with `-h` (help) to see these defaults in the tool itself.\n"},"files":{".claude/skills/benchmark-report/SKILL.md":"---\nname: benchmark-report\ndescription: Generate a technical benchmark report in markdown from raw memtier/redis-benchmark output. Use this whenever the user provides benchmark results, log files, memtier JSON output, or a directory of benchmark data and wants a written report or comparison document. Trigger even when the user says \"write up these results\", \"make a report from this data\", \"turn this into a blog post\", or just drops a directory path. Always use this skill when the user provides benchmark files and wants a human-readable output. Reports compare Dragonfly against another system (ElastiCache, Redis, Valkey) but can handle single-system data too.\nversion: 1.0.0\n---\n\n# Benchmark Report Generator\n\nGenerate a technical markdown report from raw memtier/redis-benchmark data. The report is aimed at engineers evaluating Dragonfly for production use.\n\n## Step 0: Interview before writing\n\nBefore touching any files, ask the user 3–4 targeted questions to fill in context that cannot be extracted from the raw data. Read the data directory first so your questions are specific, not generic.\n\nQuestions to ask (adapt based on what's missing after scanning the files):\n\n1. **Scenario**: What is the purpose of this benchmark? (e.g., \"SSD tiering comparison\", \"evaluating ElastiCache replacement\", \"testing read-heavy workload at scale\") — this becomes the intro paragraph and title.\n2. **Instance types**: What instance type / tier was each system running on? Include memory size.\n3. **Storage configuration**: For Dragonfly — is SSD tiering enabled? What is the memory limit vs total dataset size?\n4. **Anything else relevant**: Pricing comparison, region, network setup, anything not in the scripts/logs.\n\nWait for the user's answers before proceeding to Step 1. If the user says \"just use TODOs\" or similar, proceed without waiting.\n\n## What the user provides\n\n- A directory path with benchmark data files\n- Context answers from the interview above\n\nTypical directory layout:\n```\ndata_dir/\n *.sh — benchmark script (parameters, key distribution, test phases)\n *.json — memtier JSON output per system per phase\n *.log — memtier log output per system\n```\n\nFiles may be flat in one directory or split across subdirectories. Read them all.\n\n## Step 1: Scan and parse\n\nList all files in the directory. Then read them:\n\n**Scripts (`*.sh`)** — extract:\n- `KEY_MAXIMUM`, `DATA_SIZE`, `PIPELINE`, `CLIENTS`, `THREADS`, `TEST_TIME`\n- `FILL_RATIO`, `ACCESS_RATIO`, `FILL_KEY_PATTERN`, `ACCESS_KEY_PATTERN`\n- Key distribution flags: `--key-stddev`, `--key-zipf-exp`, etc.\n- Which phases ran (`RUN_FILL=true`, `RUN_ACCESS_READ_002=true`, etc.)\n- Hostnames — used to identify systems under test\n\n**JSON files (`*.json`)** — from memtier `--json-out-file` output:\n- `configuration.server` → hostname (used to identify the system)\n- `configuration.clients`, `threads`, `pipeline`, `ratio`, `key_maximum`, `data_size`, `key_stddev`\n- `ALL STATS.Gets` or `ALL STATS.Sets` → `Ops/sec`, `Average Latency`, `Percentile Latencies` (p50/p99/p99.9)\n- `ALL STATS.Runtime.Total duration` → test duration in ms\n\n**Log files (`*.log`)** — from memtier stdout:\n- Look for the `ALL STATS` table block. Take the last occurrence (final aggregate):\n ```\n Type Ops/sec ... Avg. Latency p50 Latency p99 Latency p99.9 Latency\n Sets 220503.72 ... 4.32106 2.95900 9.72700 53.24700\n Gets 138188.17 ... 6.94510 5.53500 36.09500 54.27100\n ```\n- Look for `real Xm Y.Zs` timing lines to capture wall-clock duration per phase\n\n## Step 2: Identify systems under test\n\nName each system from filename signals and hostnames:\n\n| Signal | System name |\n|--------|------------|\n| `bench_df*.log`, hostname contains `dragonflydb.cloud` | Dragonfly |\n| `bench_ec*.log`, hostname contains `cache.amazonaws.com` | ElastiCache |\n| `bench_valkey*.log` | Valkey |\n| `bench_redis*.log` | Redis |\n| `bench_local*.log`, `127.0.0.1` | Local (note version if available) |\n\nUse the actual product name, not the hostname. Do not include hostnames or endpoints in the report — they are not relevant for reproducing the results and may change.\n\n## Step 3: Generate charts\n\nIf JSON files are present, generate latency charts with `tools/plot_memtier_latency.py`:\n\n```bash\nmkdir -p <data_dir>/charts\npython3 <repo_root>/tools/plot_memtier_latency.py <json_file> <data_dir>/charts/<name>.html\n```\n\nRun once per JSON file. If the script is not found or fails, leave a `[TODO: chart]` stub. The script skips operations with no traffic (zero-latency entries), so it handles write-only fill JSONs and read-only access JSONs without errors.\n\nTo find `tools/plot_memtier_latency.py`, search from the current working directory: `find . -name \"plot_memtier_latency.py\" -not -path \"*/build*\"`.\n\n## Step 4: Write the report\n\nSave to `<data_dir>/report.md`. Use the template below. Populate every value you found; use `[TODO]` for anything missing (instance type, memory, pricing, scenario context the user didn't provide).\n\n---\n\n```markdown\n# <Descriptive title: what was tested, e.g., \"Dragonfly SSD Tiering vs ElastiCache: 50 GB Read Workload\">\n\n<1–3 sentence intro: what was measured and why. No adjectives. Example: \"This benchmark measures read throughput and latency for a 50 GB key-value dataset under a Gaussian access pattern. Dragonfly was configured with SSD tiering; ElastiCache r7g.4xlarge was used with all data in memory.\">\n\n## Test Environment\n\n| | Dragonfly | ElastiCache |\n|---|---|---|\n| Version / tier | [TODO] | [TODO] |\n| Instance type | [TODO] | [TODO] |\n| Memory | [TODO] | [TODO] |\n| Storage | [TODO: SSD tiering?] | [TODO] |\n| Region | [TODO] | [TODO] |\n\n<!-- Do NOT include hostnames, endpoints, or internal URLs — they are irrelevant to reproducing the test. -->\n\n## Dataset\n\n| Parameter | Value |\n|---|---|\n| Key count | <from KEY_MAXIMUM, formatted with commas> |\n| Value size | <DATA_SIZE> bytes |\n| Approximate dataset size | <KEY_MAXIMUM × DATA_SIZE, in GB> |\n| Key distribution | <Gaussian σ=X% of keys / uniform / Zipf exp=X> |\n\n## Benchmark Configuration\n\n| Parameter | Value |\n|---|---|\n| Tool | memtier_benchmark |\n| Threads | <THREADS> |\n| Connections per thread | <CLIENTS> |\n| Pipeline depth | <PIPELINE> |\n| Test duration (access phase) | <TEST_TIME>s |\n| Access ratio (SET:GET) | <ratio> |\n\n## Results\n\n### Fill Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p99 Latency (ms) | Duration |\n|---|---|---|---|---|\n| Dragonfly | | | | |\n| ElastiCache | | | | |\n\n### Read Access Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p50 (ms) | p99 (ms) | p99.9 (ms) |\n|---|---|---|---|---|---|\n| Dragonfly | | | | | |\n| ElastiCache | | | | | |\n\n## Latency Over Time\n\n<For each JSON file, embed the chart link or stub>\n\n**Dragonfly — read access (σ=2%)**\n![Dragonfly latency over time](./charts/df_access.html)\n\n**ElastiCache — read access (σ=2%)**\n[TODO: ElastiCache JSON not available; chart cannot be generated]\n\n## Observations\n\n<Factual, numbered. One observation per line. State what the data shows; do not say whether it is good or bad.>\n\n1. Throughput: Dragonfly achieved X ops/sec vs Y for ElastiCache — a Z% difference.\n2. Average latency: X ms (Dragonfly) vs Y ms (ElastiCache).\n3. p99 latency: X ms vs Y ms. <If there's a large gap relative to p50, note it.>\n4. Fill phase: <note fill throughput and duration for each system>.\n5. <Note any caveats: e.g., \"This run covers only 100% read traffic. Mixed read/write was not executed.\">\n6. <Note anything unusual in the time-series if visible from the data.>\n```\n\n---\n\n## Writing rules\n\n- Use exact numbers from the data. No rounding unless the value is very large (round to 3 significant figures max).\n- Percentages for comparisons: compute `(A - B) / B * 100`; write as \"X% higher\" or \"X% lower\".\n- Prohibited phrases: \"blazing fast\", \"industry-leading\", \"dramatically\", \"significantly\", \"next-level\", \"game-changing\", \"unparalleled\", \"impressive\"\n- No narrative hooks: no \"but wait\", \"what happens next\", \"the results speak for themselves\", \"as you can see\"\n- Observations may include inferences (e.g., \"the Gaussian σ=2% means the hot working set is ~3.5M keys, likely fitting in memory\") — label them clearly as inferences if they go beyond what the data directly shows\n- Bold labels in observations (e.g., `**Read throughput**:`) — keep them for scannability\n- Do not include hostnames, endpoints, or internal URLs anywhere in the report\n- Missing context → `[TODO: ...]` not a guess\n- Single-system data: omit comparison columns; keep observations factual about absolute numbers\n\n## Output\n\nPrint the report path when done: `Report written to: <path>/report.md`\n\nIf charts were generated, list them too.\n",".claude/skills/benchmark/SKILL.md":"---\nname: benchmark\ndescription: >\n Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote\n cloud instances, then produce a performance + memory report with charts and a\n raw-data appendix. Use this whenever the user runs /benchmark, or asks to\n benchmark / load-test / measure throughput, QPS, latency, or memory efficiency\n of Dragonfly or another Redis-compatible server, to compare Dragonfly vs\n Valkey or Redis, or to generate a benchmark report. Trigger it even when the\n user just describes the goal (\"see how fast Dragonfly is on this box\",\n \"compare memory per key vs Valkey\") without saying the word \"benchmark\".\nargument-hint: (optional) free-form spec, e.g. \"server=10.0.0.1 client=10.0.0.2 strings vs-valkey\"\nallowed-tools: Bash, Read, Write, Edit, AskUserQuestion\n---\n\n# Benchmark Dragonfly\n\nDrive a benchmark of Dragonfly (optionally side-by-side with Valkey or Redis) on\nlocal or remote instances, collect throughput / latency / memory numbers, and\nassemble a report that mirrors the team's published format: a narrative TLDR,\nper-instance metric tables, charts, and an appendix with the exact commands and\nraw output so anyone can reproduce it.\n\nThe whole point of the report is **trust**: a reader should be able to see the\nheadline numbers, understand the conditions they were measured under, and\nre-run the exact commands themselves. Everything below serves that goal.\n\n## When you start\n\nIf the user passed a free-form spec in `$ARGUMENTS`, parse what you can from it\n(IPs, use-cases, comparison targets) and only ask about what's missing. The work\nsplits into clear phases — walk through them in order, but stay flexible: the\nuser may already have servers running, or may only want the report half (handed\nraw output to format). Read what they're actually asking for.\n\nPhases: (1) gather spec → (2) prepare instances → (3) network tuning →\n(4) start monitoring → (5) run workloads → (6) extract metrics, charts, report.\n\nThe two reference files carry the operational detail — read them before acting,\nnot from memory:\n\n- `references/running-benchmarks.md` — how to prep instances, start servers,\n drive `dfly_bench` and `memtier_benchmark`, run Valkey/Redis for comparison,\n apply network tuning, and measure memory. **Read this before Phase 2.**\n- `references/report-template.md` — the exact report structure, with an example.\n **Read this before Phase 5.**\n\nBundled `scripts/` (reuse these instead of re-improvising the same loops — they\nencode the safety discipline learned the hard way):\n\n- `bench_server.sh start|stop|status <ssh> <binary> [maxmem] [port]` — launch a\n server tracking its PID; **stop kills by PID and verifies the process is gone +\n port free** (tmux kill-session does not reap it).\n- `monitor_fill.sh <server_ssh> <client_ssh> [session] [abort_pct]` — the live\n RSS guard: polls `used_memory_rss`, prints a line per sample, aborts the write\n if RSS crosses the threshold (default 92%). Its output is also the input to\n `plot_fill.py`.\n- `capture_result.sh <server_ssh> <client_ssh> <log> <outdir> [label]` — saves\n the run summary + INFO memory + **OS-level `free` snapshot** (`host_memory.txt`)\n and exits non-zero if there were any eviction errors (enforces the zero-error\n rule). The OS snapshot matters: `used_memory_rss` and host free can diverge\n significantly — e.g. with `MIMALLOC_ALLOW_LARGE_OS_PAGES=1` (default), mimalloc\n holds large-page arenas that the OS counts as used but INFO doesn't fully reflect,\n making the machine appear near-OOM while RSS looks fine.\n- `collect_metrics.py` / `make_charts.py` — metrics + charts from dfly_bench\n JSON (GET/SET workloads).\n- `plot_fill.py` — memory-fill chart from `monitor_fill.sh` output, for\n `--command` workloads (e.g. SETEX) that produce no JSON time series.\n\nEvery benchmark output directory must include a `commands_audit.log`\nartifact with the exact command lines used for setup, server launch, writes,\nreads, captures, diagnostics, and chart/report generation. Raw data without the\ncommands that produced it is not reproducible.\n\n**Run-control scripts are mandatory for benchmark phases.** For Phase 5 runs,\nuse the bundled scripts as the default API:\n\n- start/stop/status servers with `scripts/bench_server.sh`;\n- monitor long write fills with `scripts/monitor_fill.sh`;\n- capture write results and enforce the zero-error rule with\n `scripts/capture_result.sh`;\n- generate metrics/charts with `scripts/collect_metrics.py`,\n `scripts/make_charts.py`, and `scripts/plot_fill.py`.\n\nInline SSH is fine for inspection, package setup, one-off diagnostics, and\nbounded foreground probes, but not as a replacement for the run-control scripts\nin the main benchmark path. If a script is not usable for a run, document the\nreason in `commands_audit.log` and in the report appendix before using a manual\nequivalent.\n\n## Phase 1 — Gather the benchmark spec (interactive)\n\nUse `AskUserQuestion` to pin down anything not already given. You need:\n\n1. **Where it runs.** Local, or remote instances? For remote, get the **server**\n host/IP and the **client** host/IP (the load generator should run on a\n separate machine from the server — co-locating them steals CPU and corrupts\n the numbers). Get the SSH user / key if not obvious. On cloud instances,\n **always use the private/internal IP** (VPC-internal) for both SSH and as the\n load generator's `-h` target — the public IP routes through the internet\n gateway, adding latency and egress cost, and the result measures the gateway\n rather than the server.\n2. **Use-cases.** Default menu: simple strings (SETEX writes + GET reads),\n pipelined / peak-throughput, memory efficiency. Always also ask whether there\n is a **specific focus** (a particular command, data type, key distribution,\n value size, or scenario) — the canned use-cases are a starting point, not a\n cage.\n3. **Comparison targets.** Dragonfly only, or also Valkey / Redis? Which\n versions?\n4. **Instance specs** (for the report's conditions section): instance type, CPU\n count, OS/kernel. Read these directly from the box at the start of Phase 2\n rather than asking the user.\n5. **Load parameters** if the user has opinions: value size (`-d`), key space\n (`--key_maximum`), connections (`-c`), pipeline depth, test duration. Pick\n sensible defaults (see the reference) and tell the user what you chose.\n **Sizing the key count is the consequential decision** — see the reference's\n \"Sizing the load\" section. The dataset must fill a large fraction of instance\n RAM (benchmarking a near-empty instance is wrong) but stay clear of OOM\n (~60–75% of RAM, headroom to spare). `--key_maximum` bounds the footprint, so\n compute it from instance RAM ÷ estimated bytes-per-entry, calibrating with a\n short pilot write if unsure.\n\nSummarize the resolved spec back to the user in a short block before running\nanything, so a wrong assumption gets caught before machines spin.\n\n## Phase 2 — Prepare the instances\n\n- Record instance specs (`nproc`, `uname -r`, cloud metadata) — these go in the\n report's conditions section. Use the **private IP** when connecting between\n instances (see Phase 1).\n- A remote instance usually already has `~/projects/dragonfly`. If not, clone it\n there. Build a **release** binary (`build-opt/dragonfly`, `build-opt/dfly_bench`)\n unless the user wants a published release downloaded instead — follow their\n instruction on build-vs-download.\n- For comparison targets, pull the Valkey/Redis docker image on the server box.\n- Verify everything launches and is reachable from the client box (a quick\n `redis-cli -h <server-private-ip> ping`) before committing to a long run.\n\nRun each step individually over SSH — do not chain launch, polling, capture, and\nthe next phase into one command. If a tool call is interrupted, remote work keeps\nrunning but goes uncaptured. Start the server; start the write; monitor it;\ncapture; start reads — one bounded step at a time.\n\n## Phase 3 — Network tuning (optional, remote only)\n\nHigh-throughput numbers on cloud hardware depend heavily on IRQ/SMP affinity\ntuning. If the user wants maximal numbers, apply the tuning described in the\nreference and **record exactly what you did** — it belongs in the appendix and\nmaterially affects the results. If skipped, say so in the report; untuned and\ntuned numbers are not comparable.\n\n## Phase 4 — Start the monitoring stack (server machine)\n\nStart the local Prometheus + Grafana stack on the **server** machine so you get\ncontinuous, fine-grained time-series (server CPU, memory, command rates, host\nmetrics) spanning the whole benchmark — richer than the load generator's own\noutput and the source of the most convincing report charts.\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose up -d'\n# when benchmarking Valkey/Redis, also bring up the redis exporter:\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose --profile redis up -d'\n```\n\nPrometheus scrapes at 1s. **Always run servers on port 6380** (Dragonfly,\nValkey, and Redis alike). The bundled `prometheus.yml` ships pointing at `6379`,\nso before bringing the stack up, fix the scrape targets:\n\n```bash\nsed -i 's/:6379/:6380/g' tools/local/monitoring/prometheus/prometheus.yml\n```\n\n(This is a workaround for a misconfigured default — if you have repo write\naccess, commit the corrected file once so future runs start clean.)\n\nRestart Prometheus if it was already running, then verify targets are `UP` at\n`http://<server>:9090/targets` before starting the load — a silently-down scrape\njob means no data for that window.\n\nStart monitoring **before** the workloads and leave it running across all runs,\nso a single continuous window covers everything. See the reference's monitoring\nsection for what to read from Grafana (dashboards are pre-provisioned) and how to\npull panels into the report. Record the benchmark start/end timestamps — they\nlet you crop the Grafana window to exactly the run.\n\n## Phase 5 — Run the workloads\n\nFor each (use-case × datastore) combination, see the reference for the precise\ncommands. The shape is always: **write → memory snapshot → read**. Use the\nprivate IP as the load generator's `-h` target.\n\nPass `--json_out_file` to `dfly_bench` (and `--json-out-file` to memtier) when\nthe workload supports it — charts and headline numbers come from these files.\n`dfly_bench --command` workloads (e.g. `SETEX __key__ 10000 __data__`) do not\nproduce JSON; save full stdout and chart the `monitor_fill.sh` output instead.\n\nDrive the run with the bundled scripts: `bench_server.sh start`, write in a\nclient tmux session, `monitor_fill.sh` alongside (live RSS guard), then\n`capture_result.sh` to save results. `capture_result.sh` enforces the\n**zero-error rule** — any eviction/OOM error response makes the run invalid;\ndiscard it, cut `--key_maximum` ~10–15%, and re-run. See the preamble and the\nreference's \"Sizing the load\" section for the full rule.\n\n**Prefill writes must use `--pipeline 30`.** Without pipelining, the client\nstalls waiting for each reply and becomes the bottleneck before the server is\nsaturated — especially at high key counts. Use `--pipeline 30` for all write\n(fill) phases. Drop to `--pipeline 1` (or omit it) only for latency-honest\nread benchmarks where you want honest per-request numbers.\n\nInitialize `commands_audit.log` at the start and append each command as it runs\n(see the preamble for the requirement).\n\n## Phase 6 — Extract metrics, build charts, assemble the report\n\n1. **Headline metrics + memory.** Run the bundled extractor over the JSON files\n to pull QPS / P99 and compute bytes-per-entry (both `used_memory`- and\n `used_memory_rss`-based, per the user's choice to report both):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/collect_metrics.py \\\n --bench-json <run.json> [--bench-json <run2.json> ...] \\\n --info <info_memory.txt> --dbsize <N> \\\n --label \"Dragonfly m7g.2xlarge\"\n ```\n\n It prints a metrics block (and JSON) you can drop straight into the tables.\n\n2. **Charts.** Generate QPS-over-time PNGs from the time series (this is what\n visually shows Dragonfly's smooth throughput vs a competitor's degradation):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/make_charts.py \\\n --series \"Dragonfly=df_write.json\" --series \"Valkey=valkey_write.json\" \\\n --metric qps --title \"Write throughput, m7g.2xlarge\" \\\n --out charts/write_m7g2xl.png\n ```\n\n Embed the PNGs in the report with relative paths. **For `--command` workloads\n (e.g. SETEX) there is no dfly_bench JSON** — use `plot_fill.py` instead to chart\n memory-fill over time from the `monitor_fill.sh` output:\n\n ```bash\n python3 .claude/skills/benchmark/scripts/plot_fill.py \\\n --series \"prebuilt=prebuilt/poll.txt\" --series \"release=release/poll.txt\" \\\n --interval 10 --ram-gib 30 --out charts/memory_fill.png\n ```\n\n3. **Report.** Follow `references/report-template.md` exactly. Write a TLDR with\n the 3–5 genuinely important takeaways (scaling behavior, memory efficiency,\n any hardware surprise), then per-instance tables, then the appendix. The TLDR\n is interpretation, not a restatement of the tables — say what the numbers\n *mean*. Only claim what the data supports.\n\nSave the report where the user asks (default: `benchmark-<YYYY>-<MM>.md` in the\nrepo root) and tell them the path plus a one-line summary of the headline\nresult.\n\n## Cleanup\n\nIf you started servers, the monitoring stack, or comparison containers on remote\ninstances, offer to stop them (`docker compose down` in the monitoring dir).\nIf you applied network tuning that changes machine state, tell the user what\nwould need reverting. Don't tear anything down without asking — they may want to\ninspect or re-run.\n",".claude/skills/benchmark/evals/evals.json":"{\n \"skill_name\": \"benchmark\",\n \"evals\": [\n {\n \"id\": 0,\n \"name\": \"full-report-vs-valkey\",\n \"prompt\":\n \"I just finished a benchmark run comparing Dragonfly against Valkey 9.0 on AWS. The raw artifacts are in the directory I'm pointing you at: dfly_bench JSON output (write and read phases) and INFO memory dumps for both datastores on m7g.2xlarge and m7g.4xlarge, plus a RUN_NOTES.txt with the hardware, exact commands, and DBSIZE after each write phase. Generate the benchmark report from this data, with charts and a raw-data appendix.\",\n \"expected_output\":\n \"A markdown report mirroring the team format: conditions section, a Simple Strings use-case section with a TLDR, per-instance (m7g.2xlarge, m7g.4xlarge) QPS/memory and latency tables, embedded QPS-over-time charts, and an appendix with exact commands, environment, and raw output. QPS/P99 must match the data; bytes-per-entry reported both logical (~127 DF / ~149-150 Valkey) and RSS.\",\n \"files\": [\n \"fixtures/\"\n ]\n },\n {\n \"id\": 1,\n \"name\": \"memory-focus-single-datastore\",\n \"prompt\":\n \"Here's raw output from a Dragonfly-only run on m7g.4xlarge (dfly_bench write/read JSON + INFO memory dump + run notes with DBSIZE). I care mostly about memory efficiency. Write me a short benchmark report focused on memory usage per entry, plus the throughput numbers.\",\n \"expected_output\":\n \"A report covering only Dragonfly (no fabricated Valkey/Redis comparison column), reporting both logical (used_memory/DBSIZE ~127) and RSS bytes-per-entry, plus write/read QPS and P99. Honest about it being single-datastore.\",\n \"files\": [\n \"fixtures/df_4xl_write.json\",\n \"fixtures/df_4xl_read.json\",\n \"fixtures/df_4xl_info_memory.txt\",\n \"fixtures/RUN_NOTES.txt\"\n ]\n },\n {\n \"id\": 2,\n \"name\": \"kickoff-planning-interactive\",\n \"prompt\":\n \"I've got two m7g.4xlarge boxes in AWS. Benchmark Dragonfly against Valkey on a simple strings workload.\",\n \"expected_output\":\n \"The skill should engage its interactive front-end: ask which box is the server vs the client (load generator must be separate), confirm SSH access, plan to run servers on port 6380, plan to start the monitoring stack (and fix the prometheus scrape port to 6380), and lay out the write/read workload plan. It must NOT invent benchmark numbers it hasn't measured.\",\n \"files\": []\n }\n ]\n}\n",".claude/skills/benchmark/references/report-template.md":"# Report template\n\nThe report mirrors the team's published benchmark format. Structure matters: a\nreader scans the conditions, jumps to the TLDR for the takeaways, checks the\nper-instance tables for numbers, and uses the appendix to reproduce. Follow this\nshape.\n\nUse Markdown tables. Center-align numeric columns (`:---:`). Bold the winning\ndatastore in comparison tables when there's a clear winner.\n\n## Structure\n\n```markdown\n# <Title> — e.g. \"Comparing Dragonfly vs Valkey on AWS\"\n\n<1–2 sentences: what was compared, on what hardware, what workloads.>\n\n## Conditions\n\nDragonfly launch command (identical across all configs):\n\n ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n\n<Comparison target> launch command:\n\n docker run --network=host --rm valkey/valkey:9.0 --save \"\" --appendonly no \\\n --io-threads 8 --protected-mode no --port 6380\n\nInstances: <server types + CPU counts>, client on <client instance type>.\nOS: <distro + kernel>. Network tuning: <applied / none — one line on what>.\n\n## <Use-case name> — e.g. \"Simple Strings\"\n\n<1–2 sentences describing the workload and why it matters.>\n\nWrite command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 --ratio 1:0 --json_out_file df_write.json\n\nRead command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN --ratio 0:1\n\n### TLDR\n\n* **<Datastore>** <the genuinely important takeaway — scaling, efficiency, a surprise>.\n* <3–5 bullets total. Interpretation, not a restatement of the tables.>\n\n## <Instance type> — e.g. \"m7g.2xlarge\"\n\n<Optional: one sentence on anything notable, e.g. throughput stability over time.>\n\n![Write throughput over time](charts/write_m7g2xl.png)\n\n### QPS and memory\n\n| Metric | Dragonfly | Valkey |\n| ----- | :---: | :---: |\n| Write QPS (k) | 816 | 548 |\n| Read QPS (k) | 848 | 811 |\n| Memory / entry — logical (bytes) | 127 | 149 |\n| Memory / entry — RSS (bytes) | 138 | 162 |\n\n### Latency\n\n| Backend | Write P99 (us) | Read P99 (us) |\n| :---: | :---: | :---: |\n| **Dragonfly** | 787 | 887 |\n| **Valkey** | 3283 | 951 |\n\n<Repeat the per-instance block for each instance type tested.>\n\n## Appendix — raw data\n\n### Environment\n\n| Property | Server | Client |\n| --- | --- | --- |\n| Instance type | m7g.2xlarge | c7gn.4xlarge |\n| vCPUs | 8 | 16 |\n| OS / kernel | Ubuntu 24.04.2 / 6.8.0 | ... |\n| Dragonfly version | <git hash / release> | — |\n| Network tuning | IRQ SMP affinity spread across CPUs | — |\n\n### Exact commands\n\n<Every command line actually run, per phase and per datastore, verbatim —\nincluding the launch commands, write/read load commands, and any tuning steps.>\n\n### Raw output\n\n<Per run: the final dfly_bench/memtier summary + latency histogram, and a\nreference to the saved JSON file (df_write.json, etc.). Include INFO memory\nsnapshots used for the per-entry figures.>\n```\n\n## Notes on writing the TLDR\n\nThe TLDR is the part people actually read. Make each bullet a claim the data\nsupports, phrased as a takeaway:\n\n- Good: \"Dragonfly scales vertically with cores and sustains up to 10x the\n throughput of a single Valkey process on large instances.\"\n- Weak: \"Dragonfly had higher QPS.\" (just restates the table)\n\nCall out hardware surprises honestly — e.g. an instance type that didn't scale\nlinearly — because they affect how a reader should interpret the numbers. If a\nrun was client-bound or otherwise caveated, say so here rather than burying it.\n\n## Charts\n\nOne chart per instance per phase is usually enough: QPS-over-time, with one line\nper datastore. The visual story is throughput *stability* — Dragonfly holding a\nflat line while a competitor degrades as its table grows is more convincing than\na single aggregate number. Generate with `scripts/make_charts.py` from the load\ngenerator's time-series JSON, save under a `charts/` dir next to the report, and\nembed with relative paths.\n\nWhen the monitoring stack ran (Phase 4), the Grafana dashboards add server-side\npanels that the load generator can't show — memory growth over the write phase,\nper-shard CPU, host network saturation. Crop the Grafana time range to the run\nwindow and pull the relevant panels (rendered PNG or screenshot) into the\nper-instance section. These are especially good for the memory-efficiency story\nand for proving the server (not the client) was the bottleneck. Note in the\nappendix that the data came from the monitoring stack, and which scrape port was\nused.\n",".claude/skills/benchmark/references/running-benchmarks.md":"# Running benchmarks — operational reference\n\nHow to prepare instances, start servers, drive the load generators, and measure\nmemory. Read the section you need; you don't have to read top-to-bottom.\n\n## Table of contents\n\n- [Instance preparation](#instance-preparation)\n- [Starting Dragonfly](#starting-dragonfly)\n- [Starting Valkey / Redis for comparison](#starting-valkey--redis-for-comparison)\n- [dfly_bench (default load generator)](#dfly_bench-default-load-generator)\n- [memtier_benchmark (alternative)](#memtier_benchmark-alternative)\n- [The standard strings workload](#the-standard-strings-workload)\n- [Measuring memory](#measuring-memory)\n- [Monitoring stack (Prometheus + Grafana)](#monitoring-stack-prometheus--grafana)\n- [Network tuning](#network-tuning)\n- [Sizing the load](#sizing-the-load)\n\n---\n\n## Instance preparation\n\nThe server and the load generator should live on **separate machines**. A\nclient co-located with the server competes for CPU and silently caps the\nnumbers — the report is then measuring the client, not the server.\n\nRemote instances usually already have the repo at `~/projects/dragonfly`. Check\nfirst; clone only if missing:\n\n```bash\nssh <host> 'test -d ~/projects/dragonfly && echo present || echo missing'\n# if missing:\nssh <host> 'git clone https://github.com/dragonflydb/dragonfly.git ~/projects/dragonfly'\n```\n\n**Look for prebuilt binaries first.** Before building or downloading, check the\nrepo's build output folders — the binary is often already there. Look in\n`build-opt/` and `build-release/` (and `build-dbg/` only as a last resort — debug\nbuilds give meaningless performance numbers):\n\n```bash\nssh <host> 'ls -la ~/projects/dragonfly/build-opt/{dragonfly,dragonfly-aarch64,dfly_bench} 2>/dev/null; \\\n ls -la ~/projects/dragonfly/build-release/{dragonfly,dfly_bench} 2>/dev/null'\n```\n\nA folder may hold several binaries — e.g. a locally-built `dragonfly` (dev/HEAD)\nalongside a downloaded release `dragonfly-aarch64`; check `--version` on each to\nknow which is which, since \"compare the prebuilt vs the release binary\" is a\ncommon request. The load generator (`dfly_bench`) lives in the same folder on the\nclient.\n\n**Build vs download** — if no suitable prebuilt binary exists, follow the user's\ninstruction. To build a release binary from source on the instance:\n\n```bash\nssh <host> 'cd ~/projects/dragonfly && ./helio/blaze.sh -release -DWITH_AWS=OFF -DWITH_GCP=OFF'\nssh <host> 'cd ~/projects/dragonfly/build-opt && ninja dragonfly dfly_bench'\n```\n\nThis yields `build-opt/dragonfly` (the server) and `build-opt/dfly_bench` (the\nload generator). Build `dfly_bench` on the **client** box and `dragonfly` on the\n**server** box. Always benchmark release builds — a debug build's numbers are\nmeaningless for performance claims.\n\nIf the user prefers a published release, download the tarball from GitHub\nreleases and extract it instead of building.\n\nSanity-check reachability from the client before a long run:\n\n```bash\nredis-cli -h <server-ip> -p 6380 ping # expect PONG\n```\n\nCollect environment facts for the report while you're on the box:\n\n```bash\nssh <host> 'nproc; uname -r; cat /etc/os-release | head -2'\n# cloud instance type, if on AWS:\nssh <host> 'curl -s http://169.254.169.254/latest/meta-data/instance-type'\n```\n\n---\n\n## Starting Dragonfly\n\nThe published comparisons run Dragonfly with a minimal, persistence-free config\nso the benchmark measures the data path, not disk:\n\n```bash\n./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n```\n\n- `--conn_use_incoming_cpu` — pin each connection's processing to the CPU that\n received it; improves locality under high connection counts.\n- `--dbfilename=` — disable snapshot loading/saving.\n\nLet Dragonfly use all cores by default (it auto-detects). Record the exact\nlaunch command for the appendix.\n\n**Start it so you can reliably stop it — capture the PID.** Launching under\n`tmux`/`nohup` is fine, but `tmux kill-session` does **not** reliably reap the\nserver, and a stale server is a trap: a \"fresh\" server silently fails to bind the\nport, your client reconnects to the *old* instance, and you measure stale data\n(and waste the old dataset's RAM). Always record the PID and kill by PID:\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/build-opt; \\\n nohup ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380 \\\n >/tmp/dfly.log 2>&1 & echo $! > /tmp/dfly.pid'\n# ... benchmark ...\n# Stop it and VERIFY it's gone before starting the next binary:\nssh <server> 'kill -9 $(cat /tmp/dfly.pid); sleep 3; \\\n pgrep -af dragonfly | grep -v \"pgrep\\|bash -c\" || echo \"stopped\"; \\\n redis-cli -p 6380 ping 2>&1 | head -1 # expect Connection refused'\n```\n\nBetween two binaries (e.g. prebuilt vs release), **fully stop and verify the\nfirst is dead and the port is free before starting the second** — otherwise the\nsecond never binds and you benchmark the first twice. Note that pattern-based\n`pkill -f` can miss a process whose cmdline is the relative `./dragonfly`; killing\nby the captured PID is the robust path.\n\n---\n\n## Starting Valkey / Redis for comparison\n\nRun via docker on the server box. Valkey does not scale past ~8 io-threads on\ncurrent hardware, so more than that rarely helps — note the thread count you\nused.\n\n```bash\ndocker run --network=host --rm valkey/valkey:9.0 \\\n --save \"\" --appendonly no --io-threads 8 --protected-mode no --port 6380\n```\n\nFor Redis, substitute the `redis` image and equivalent flags. Run the comparison\ntarget on the **same instance type** as Dragonfly, one at a time (not\nsimultaneously — they'd contend), so the comparison is apples-to-apples.\n\n---\n\n## dfly_bench (default load generator)\n\nSource: `src/server/dfly_bench.cc`. Key flags:\n\n| Flag | Meaning | Typical |\n| --- | --- | --- |\n| `-h` | server host | client→server IP |\n| `-p` | server port | `6380` |\n| `-c` | connections **per thread** | ~10; ≤25 max; 1–5 when pipelining |\n| `-n` | requests per connection | sizes the run (omit if `--test_time`) |\n| `--test_time` | run duration in seconds | e.g. `120` |\n| `--qps` | target QPS per connection; `0` = unthrottled | `0` for peak |\n| `-d` | value size in bytes | `64` for strings tests |\n| `--key_maximum` | size of key space | tune to fill memory |\n| `--key_dist` | `U` uniform, `N` normal, `Z` zipfian, `S` sequential | `U` default |\n| `--ratio` | set:get ratio | `1:0` writes, `0:1` reads |\n| `--command` | custom command template (`__key__`, `__data__`) | for non-string workloads |\n| `--pipeline` | pending requests per connection | `>1` for peak throughput |\n| `-P` | `memcache_text` for memcached protocol | empty for RESP |\n| `--json_out_file` | **memtier-compatible JSON with per-second time series** | always set it |\n\n`--json_out_file` only works with the built-in GET/SET workload (not with\n`--command`, `--noreply`, or `--connect_only`). For custom-command workloads you\nscrape the console summary instead.\n\n**Console output.** Periodic lines look like:\n\n```\n12s: 40.0% done, RPS(now/agg): 815000/812340, errs: 0, hitrate: 99.8%, clients: 200\n```\n\nand the final summary:\n\n```\nTotal time: 30s. Overall number of requests: 24370200, QPS: 812340, P99 lat: 787us\nLatency summary, all times are in usec:\n<histogram>\n```\n\nSave the full stdout — the histogram and final QPS go in the appendix; the\n`collect_metrics.py` script prefers the JSON but can fall back to this.\n\nAlso save the command lines themselves under the run output directory, e.g.\n`commands_audit.log`. Record setup, server launch, write/read load,\ncapture, diagnostics, and chart/report commands as they are run; stdout alone is\nnot raw data because it cannot reproduce the run. Initialize this file before\nthe first workload so planned script invocations are visible up front, then\nappend commands as they execute.\n\n**Use the skill scripts for the main run path.** The main benchmark phases should\nuse:\n\n```bash\n.claude/skills/benchmark/scripts/bench_server.sh start|start-docker|stop|stop-docker|status ...\n.claude/skills/benchmark/scripts/monitor_fill.sh ...\n.claude/skills/benchmark/scripts/capture_result.sh ...\n.claude/skills/benchmark/scripts/collect_metrics.py ...\n.claude/skills/benchmark/scripts/make_charts.py ...\n.claude/skills/benchmark/scripts/plot_fill.py ...\n```\n\nInline SSH is acceptable for discovery, package/setup checks, bounded\nforeground probes, and diagnostics. If you bypass a run-control script for a\nmain benchmark phase, record the reason in `commands_audit.log` and the appendix before\nrunning the manual equivalent.\n\n**Keep orchestration phase-separated.** Do not run a single local shell command\nthat starts a remote tmux benchmark, polls until completion, captures results,\nand starts the next phase. If the local command is interrupted, the remote tmux\nprocess can finish while the local capture logic is gone, leaving the run idle\nand uncaptured. Prefer bounded foreground `dfly_bench` commands for known-length\nruns. If you need tmux/nohup for a long fill, use separate steps: launch it,\nmonitor it, verify it has exited, then run the capture/read commands separately.\n\n**JSON structure** (`--json_out_file`):\n\n```json\n{\n \"configuration\": { \"server\": \"...\", \"port\": 6380, \"clients\": 200,\n \"threads\": 8, \"pipeline\": 1, \"ratio\": \"1:0\" },\n \"ALL STATS\": {\n \"Runtime\": { \"Total duration\": 30000 },\n \"Sets\": { \"Count\": ..., \"Average Latency\": ..., \"Percentile ...\": ...,\n \"Time-Serie\": { \"0\": {...}, \"1\": {...}, ... } },\n \"Gets\": { ... }\n }\n}\n```\n\nThe per-second `Time-Serie` buckets are what the charts plot.\n\n---\n\n## memtier_benchmark (alternative)\n\nIndustry-standard, available as a docker image (`redislabs/memtier_benchmark`).\nUse it when the user wants cross-validation against a tool Dragonfly doesn't\nship, or for the pipelined peak-throughput use-case. Example (from the repo's\nk8s job):\n\n```bash\nmemtier_benchmark -s <server> -p 6380 \\\n --pipeline=30 --key-maximum=100000 -c 10 -t 2 --test-time=120 \\\n --distinct-client-seed --hide-histogram --json-out-file memtier_out.json\n```\n\n`--json-out-file` produces the same shape `collect_metrics.py` and\n`make_charts.py` read.\n\n---\n\n## The standard strings workload\n\nThis reproduces the published \"Simple Strings\" comparison. Run write then read,\nwith a memory snapshot in between.\n\n**Write phase** (load the keyspace):\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 -n $N --ratio 1:0 --json_out_file df_write.json\n```\n\nThe canonical strings test uses the built-in SET path (`--ratio 1:0`) so it\nproduces the JSON/time series the charts need. If TTLs matter, swap in\n`--command \"setex __key__ 10000 __data__\"` — but note `--json_out_file` is then\nignored, so for SETEX save stdout and parse/chart the periodic `RPS(now/agg)`\nlines or `monitor_fill.sh` output instead.\n\n**Memory snapshot** (immediately after writes — see next section).\n\n**Read phase**:\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n -n $N --ratio 0:1 --json_out_file df_read.json\n```\n\nChoose `KMAX`, `N`, and `CONN` to (a) saturate the server and (b) load enough\ndata that instance memory is well utilized — see [Sizing the load](#sizing-the-load).\n\n---\n\n## Measuring memory\n\n\"Bytes per entry\" is the memory metric in the report. Capture it right after the\nwrite phase, before reads:\n\n```bash\nredis-cli -h <server> -p 6380 INFO memory > info_memory.txt\nredis-cli -h <server> -p 6380 DBSIZE\n```\n\nFrom `INFO memory` you need `used_memory` (logical) and `used_memory_rss`\n(physical resident set). Report **both** per-entry figures:\n\n- `used_memory / DBSIZE` — logical bytes per entry (matches the published\n \"Memory Per Entry\").\n- `used_memory_rss / DBSIZE` — physical bytes per entry, the real footprint\n including allocator overhead and fragmentation.\n\n`collect_metrics.py` computes both when given `--info` and `--dbsize`.\n\nFor Valkey/Redis the same `INFO memory` fields exist, so the comparison is\ndirect. Capture the snapshot at the **same keyspace size** for both, or the\nper-entry comparison is unfair.\n\n---\n\n## Monitoring stack (Prometheus + Grafana)\n\nLives in `tools/local/monitoring/` and runs on the **server** machine via docker\ncompose. It gives continuous, 1-second-resolution time series for the whole\nbenchmark window — server-side command rates and memory, plus host CPU/network\nfrom node-exporter — which is both a sanity check (was the server actually\nsaturated? did it swap?) and the source of the report's best charts.\n\n```bash\ncd ~/projects/dragonfly/tools/local/monitoring\ndocker compose up -d # Prometheus, Grafana, node-exporter\ndocker compose --profile redis up -d # also start redis-exporter for Valkey/Redis\n```\n\nEndpoints (on the server box): Prometheus `:9090`, Grafana `:3000`,\nnode-exporter `:9100`, redis-exporter `:9121`.\n\n**What gets scraped:**\n\n- `dragonfly` job — Dragonfly exposes Prometheus metrics directly at\n `/metrics` on its main port. The bundled `prometheus.yml` targets\n `host.docker.internal:6379`.\n- `node-exporter` job — host CPU, memory, network.\n- `redis-exporter` job (profile `redis`) — Valkey/Redis stats, also via\n `host.docker.internal:6379`.\n\n**Port — standardize on 6380.** Always run benchmarked servers (Dragonfly,\nValkey, Redis) on port `6380`. The shipped `prometheus.yml` targets `6379`, so\nbefore bringing the stack up, edit the `dragonfly` and `redis-exporter` jobs in\n`tools/local/monitoring/prometheus/prometheus.yml` to point at `:6380`:\n\n```bash\nsed -i 's/:6379/:6380/g' ~/projects/dragonfly/tools/local/monitoring/prometheus/prometheus.yml\n# if Prometheus is already up, pick up the change:\ndocker compose restart prometheus\n```\n\nThen confirm targets are `UP` at `http://<server>:9090/targets` before starting\nload — otherwise Prometheus silently collects nothing for the server.\n\n**Dashboards** are pre-provisioned in Grafana (`dragonfly.json`, `redis.json`,\n`node-exporter.json`) — no manual import. For the report, note the benchmark\nstart/end timestamps so you can crop the Grafana time range to the exact run,\nthen capture the relevant panels (throughput, memory, CPU) — via the provisioned\n`grafana-image-renderer` (the compose file includes a `renderer` service) or a\nscreenshot — and embed them. You can also query Prometheus directly over HTTP\n(`/api/v1/query_range`) if you want raw series for `make_charts.py`.\n\nLeave the stack up across all runs so one continuous window covers everything.\n\n## Network tuning\n\nOn cloud hardware, peak throughput is gated by how network-interrupt load is\nspread across CPUs. The published runs tuned **SMP affinity for IRQs** so the\nNIC's interrupts fan out across multiple cores instead of saturating one.\n\nThis is instance- and NIC-specific. The general approach: identify the NIC's IRQs\n(`/proc/interrupts`), then write CPU masks to `/proc/irq/<N>/smp_affinity` to\nspread them. Many setups use the kernel's `set_irq_affinity.sh` helper shipped\nwith the NIC driver. Because the exact steps depend on the instance, confirm the\napproach with the user if they want tuning, and **record the exact commands you\nran** — tuned and untuned numbers are not comparable, and the appendix must say\nwhich was used.\n\nIf you skip tuning, state that plainly in the report.\n\n---\n\n## Sizing the load\n\nTwo independent things to size: **how many keys** (sets the memory footprint)\nand **how long / how many requests** (sets steady-state and how full memory\ngets). Get these wrong in either direction and the benchmark is misleading.\n\n### The memory budget — the most important (and most dangerous) call\n\nA write benchmark must land in the **sweet spot between two failure modes**:\n\n- **Too few keys → wrong-but-safe.** Benchmarking a nearly-empty instance is\n not representative. A small dataset sits in CPU cache and hides the real cost\n of a large hash table (cache misses, allocator behavior, fragmentation). The\n QPS looks great and means nothing. *Using available memory is the whole\n point.*\n- **Too many keys → OOM or eviction.** Push too far and the run is ruined two\n different ways (see the two ceilings below). Either way the numbers are\n garbage.\n\n**There are TWO ceilings, not one. Stay under both.**\n\n1. **Dragonfly's `maxmemory`.** Dragonfly auto-sets `maxmemory` to a fraction of\n RAM (≈ 78–80% by default). When memory crosses it, Dragonfly **rejects or\n evicts** writes — the load generator reports *error responses* and the run is\n contaminated. **Check it before the run** (`redis-cli -p 6380 config get\n maxmemory`, or the `maxmemory:` line in `INFO memory`).\n\n **Rejections start with a margin *below* `maxmemory`, not at it.** During a\n write, dash-table resizes transiently spike memory, so Dragonfly begins\n rejecting while steady-state `used_memory` is still well under the cap —\n observed rejections at `used_memory` ≈ 90% of `maxmemory` and even lower at the\n moment of a big resize. So keep steady-state `used_memory` to **≤ ~80% of\n `maxmemory`** to leave room for these spikes. If you need to fill more, raise\n `--maxmemory` explicitly — but then ceiling #2 (host RAM) binds.\n\n **Zero errors is the pass/fail bar.** *Any* eviction/OOM error contaminates\n the experiment — it slows the run (rejection stalls inflate P99) and stops the\n keyspace from growing, so QPS, latency, and bytes-per-entry are all off. After\n every write, **verify the load generator reported zero errors**\n (`grep -i error <write.log>`; dfly_bench prints `Got N error responses!`). If\n N > 0, the run is **invalid** — discard it, cut `--key_maximum` (start with a\n ~10–15% reduction), and re-run until errors are zero. Don't report a run with\n any errors, and don't try to \"subtract\" them out.\n2. **Host physical RAM, measured by RSS.** The accurate measure of real memory\n pressure is **resident set size — `used_memory_rss`** (equivalently the\n process RSS). True OOM happens when RSS approaches host RAM. Do **not** panic\n over `free`'s `used`/`available` columns during a write: a high-rate write\n inflates **page cache** (clean, reclaimable) and dirty pages, which makes\n `available` dip alarmingly even though the kernel will reclaim it under\n pressure. On a 30 GiB box a 195M-key dataset showed `free` available dropping\n to ~10% while `used_memory_rss` was only ~21 GiB (71% of RAM) — the dataset\n was nowhere near OOM; the rest was reclaimable cache. **Track RSS, keep it\n below host RAM with a ≥5% margin**, and ignore page-cache noise.\n\n**So size against RSS.** Target the dataset's RSS up to your fill goal while\nkeeping a real margin, and stay under `maxmemory`:\n\n```\ntarget_rss = host_RAM * fill_fraction # e.g. 0.90 for \"fill 90%\", never > 0.95\nrss_per_entry = used_memory_rss / DBSIZE # from calibration (accurate)\nkey_for_host = target_rss / rss_per_entry\nkey_for_maxmem = 0.90 * maxmemory / used_memory_per_entry # avoid eviction\ntarget_keys = min(key_for_host, key_for_maxmem)\n--key_maximum = target_keys\n```\n\nThe usual binding constraint is **`maxmemory`**, since its default (~78% of RAM)\nis below a 90% fill target. **To genuinely fill 90% of RAM, raise `--maxmemory`\nexplicitly** (e.g. `--maxmemory 28gb` on a 30 GiB box) so Dragonfly doesn't evict\nbefore RSS reaches the target — then RSS, watched live, is the safety signal.\nFilling memory is the goal; the failure modes to avoid are eviction (used_memory\n> maxmemory) and RSS approaching host RAM.\n\n`*_per_entry` (≈ `value_size + 60–100 B` overhead for small strings; more for\nTTL'd keys and other types) is best **calibrated**, not guessed:\n\n1. Run a short pilot write inserting a known, modest number of distinct keys\n (e.g. `--key_maximum=5000000 --key_dist S` with enough requests to write them\n all).\n2. Read `used_memory` / `DBSIZE` → real `bytes_per_entry`. Note this slightly\n *under*-estimates at scale (fragmentation grows), so keep margin.\n3. Plug into the formula above for the full run.\n\nCalibrate **per datastore** — Dragonfly and Valkey have different per-entry\noverhead, which is exactly what the report measures. Size each so its dataset\nfills the same target fraction; comparing at the same `DBSIZE` keeps the\nbytes-per-entry comparison fair (Dragonfly will simply use less RAM at that\nkey count, which is the finding, not a sizing error).\n\n### How distinct keys actually accumulate\n\nWith uniform random keys over `--key_maximum`, distinct keys (and therefore\nmemory) climb fast at first, then asymptote toward `key_maximum` as repeats\nstart landing on existing keys. **Memory plateaus near `key_maximum` distinct\nentries** — writing *longer* past that point just overwrites existing keys and\ndoes **not** grow memory. That's the key safety property: **`--key_maximum`, not\nrun length, bounds the footprint.** Run length controls how *close* to that\nplateau you get and how long you hold steady state.\n\nSo to reliably fill to the target without OOM:\n\n- Set `--key_maximum` to the computed `target_keys`.\n- Make the write phase long enough to insert essentially all of them — total\n requests (`-c` × threads × `-n`, or a `--test_time` that covers it) should be\n a small multiple of `key_maximum` (~2–3×) so coverage is near-complete.\n- Use `--key_dist S` (sequential) if you want *exactly* `key_maximum` distinct\n keys with no repeats and a precise fill; uniform is fine when you just need to\n approach the plateau.\n\n### Watch memory live — guard on RSS\n\nNever run a long write blind. The accurate safety signal is **`used_memory_rss`**\n(resident set), not `free`'s `available` (page cache makes it dip transiently and\nis reclaimable). Poll RSS every few seconds and **abort the write if RSS exceeds\n~90–95% of host RAM**, and separately watch `used_memory` against `maxmemory` to\ncatch approaching eviction. Example guard loop:\n\nRun the guard as its own bounded step, not as part of a larger launch/poll/capture\nmega-command. After the guard reports `DONE`, immediately run a separate capture\nstep (`DBSIZE`, `INFO memory`, stdout log, and zero-error check), then start the\nread phase.\n\n```bash\nTOTAL=$(ssh <server> 'free -b | awk \"/^Mem:/{print \\$2}\"')\nfor i in $(seq 1 60); do\n sleep 8\n read DB RSS USED < <(ssh <server> 'redis-cli -p 6380 dbsize | tr -d \"\\r\"; \\\n redis-cli -p 6380 info memory | grep -E \"^used_memory_rss:|^used_memory:\" | cut -d: -f2 | tr -d \"\\r\"' | paste -sd\" \")\n pct=$(awk \"BEGIN{printf \\\"%.0f\\\", $RSS/$TOTAL*100}\")\n echo \"dbsize=$DB rss=$(awk \"BEGIN{printf \\\"%.1f\\\",$RSS/2^30}\")GiB (${pct}% of RAM)\"\n if [ \"$pct\" -ge 92 ]; then echo \"ABORT: RSS ≥92% of RAM\"; \\\n ssh <client> 'tmux kill-session -t bench'; break; fi\ndone\n```\n\nWays to read current memory:\n\n- **`INFO memory` → `used_memory_rss`** — the accurate measure; the guard's\n input. Also compare `used_memory` to `maxmemory` (eviction warning).\n- **`/metrics` endpoint (Dragonfly only)** — `curl -s\n http://<server>:6380/metrics | grep -i mem` gives Dragonfly's live RSS/used\n gauges without a redis client. **Valkey/Redis do not serve `/metrics`** — for\n those, use `INFO memory` directly or scrape via the redis-exporter (see the\n monitoring section). `INFO memory` works for all three datastores, so it's the\n portable choice for the guard.\n- **Monitoring stack** — Prometheus/Grafana plots these over time; the source of\n the memory-over-time chart. (`free`/node-exporter available is fine for the\n chart, just not the trigger.)\n\nIf RSS approaches the limit faster than expected, **stop the write early** —\nyou've already filled enough to measure; a partially-filled run with clean\nnumbers beats one that evicted or OOMed.\n\n### The other knobs\n\n- **Connections (`-c` is *per thread*):** keep `-c` modest — **never above 25\n connections per thread, usually around 10**; for pipelined runs drop it to\n **1–5 per thread** (each connection already has many requests in flight, so\n more connections just add contention without more throughput). Scale total\n load by adding **threads** (and, if needed, a second client box), not by\n cramming connections onto each thread. Raise the total until QPS stops rising;\n if it keeps climbing at your max, the client is the bottleneck — add threads or\n another client machine, not more `-c`.\n- **Duration / steady state:** beyond filling memory, hold at least 30–60s of\n steady traffic so the reported QPS/P99 reflect steady state, not warm-up.\n Watch the periodic RPS line stabilize before trusting the number.\n- **Pipeline:** Use `--pipeline 30` for all write/prefill phases — without it\n the client stalls on each reply and becomes the bottleneck before the server\n saturates. Use `--pipeline 1` (or omit) only for latency-honest read\n benchmarks, and label those numbers accordingly.\n\nAlways note in the report the key count, value size, resulting memory fill (%\nof instance RAM), and whether the run was client-bound — an under-driven server\nor a barely-filled instance both produce numbers that quietly mislead.\n",".claude/skills/reproduce-fuzz-crash/SKILL.md":"---\nname: reproduce-fuzz-crash\ndescription: >\n Reproduce AFL++ fuzz crashes from GitHub Actions. Use when user provides a\n GitHub Actions fuzz run URL and wants to reproduce and analyze the crash locally.\nargument-hint: <github-actions-run-url>\nallowed-tools: Bash, Read, Grep, Glob, Write\n---\n\n# Reproduce Fuzz Crash\n\nGiven a GitHub Actions fuzz run URL, download crash artifacts, triage them\nwith `fuzz/triage_crashes.sh`, and produce a crash analysis report.\n\n**Input**: `$ARGUMENTS` — a GitHub Actions run URL like:\n`https://github.com/dragonflydb/dragonfly/actions/runs/22906484769`\nor with query params like `?pr=6855`.\n\n## Workflow\n\n### Step 1: Parse the URL\n\nExtract `owner/repo` and `run_id` from the URL.\n\n```\nhttps://github.com/{owner}/{repo}/actions/runs/{run_id}[?...]\n```\n\nStrip any query parameters from `run_id`.\n\n### Step 2: Download artifacts\n\nList crash artifacts via the GitHub API, then download each as a `.zip` directly:\n\n**IMPORTANT**: Run each command as a separate Bash tool call (no `&&` chaining)\nto ensure auto-approval works with the user's permission patterns.\n\n```bash\n# List artifacts — filter for names containing \"crash\"\ngh api repos/{owner}/{repo}/actions/runs/{run_id}/artifacts\n\n# Create output directory\nmkdir -p /tmp/fuzz-repro-{run_id}\n\n# Download each crash artifact by ID (separate command)\ngh api repos/{owner}/{repo}/actions/artifacts/{artifact_id}/zip > /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nThis gives real `.zip` files that the triage script can consume directly.\n\nIf no crash artifacts are found, report that the run has no crash artifacts and stop.\n\nNote: there may be duplicate artifact names (same name, different IDs) from\nretried jobs. Download the **most recent** one (highest artifact ID).\n\n### Step 3: Determine mode\n\nInfer the protocol mode from the artifact name:\n- Contains \"memcache\" → `memcache`\n- Otherwise → `resp`\n\n### Step 4: Check Dragonfly binary\n\nCheck if the debug binary already exists and runs:\n\n```bash\n./build-dbg/dragonfly --version\n```\n\nOnly build if the binary doesn't exist or fails to run:\n\n```bash\nninja -C build-dbg dragonfly\n```\n\nIf `build-dbg` doesn't exist, run `./helio/blaze.sh` first.\n\n### Step 5: Run triage_crashes.sh\n\nFor each zip file, run:\n\n```bash\n./fuzz/triage_crashes.sh ./build-dbg/dragonfly <mode> /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nCapture the full output.\n\n### Step 6: Analyze and report\n\nParse the triage output for confirmed crashes. For each confirmed crash:\n\n1. **Read the source** at the crash location — use the stack trace to identify\n the source file and line number, then read that code.\n2. **Provide analysis**: likely root cause, what to investigate.\n\nPrint a structured report:\n\n```\n## Fuzz Crash Report\n\n**Run**: {url}\n**Artifacts**: {number} crash(es) found\n\n---\n\n### Crash NNNNNN\n\n**Reproduced**: Yes / No (false positive)\n**Signal**: SIGABRT (6) / SIGSEGV (11) / etc.\n\n**Stack trace**:\n\\```\n<stack trace from triage output>\n\\```\n\n**Analysis**:\n<1-3 sentences explaining the likely root cause based on the stack trace,\nthe assertion message, and the crash input. Identify the source file and\nline number. Suggest what to investigate.>\n```\n\n## Important Notes\n\n- The triage script uses port **6379** (resp) or **11211** (memcache).\n Ensure no other Dragonfly or Redis instance is using these ports.\n- The script adds `--restricted_commands` flags to avoid false positives from\n commands like DEBUG SLEEP that the fuzzer might generate.\n- Some crashes are non-deterministic (thread timing). The script reports\n these as \"FALSE POSITIVE\" — note this clearly, it doesn't mean the bug\n is invalid, just that it didn't reproduce on this run.\n- The script handles its own cleanup of Dragonfly processes.\n- Do NOT delete `/tmp/fuzz-repro-{run_id}/` — the user may want to inspect it.\n- If `gh run download` fails with permissions, suggest the user authenticate\n with `gh auth login`.\n",".github/copilot-instructions.md":"---\ndescription: 'Code review guidelines for GitHub copilot in this project'\napplyTo: '**'\nexcludeAgent: [\"coding-agent\"]\n---\n\n# Code Review Instructions\n\nKeep reviews high-signal and minimal. Only comment on real bugs with high confidence.\n\n## Comment Only When\n- The issue is a correctness, security, concurrency, or architecture problem.\n- The impact is clear and non-trivial.\n- You can point to concrete evidence in the diff (not speculation).\n\n## Avoid\n- Style, formatting, naming, or minor performance nits.\n- Optional refactors or “nice to have” suggestions.\n- Praise, restating the code, or long explanations.\n- Duplicate comments for the same root cause.\n\n## Review Style\n- Be terse: 1-2 sentences per issue.\n- Include file and line references when possible.\n- If no issues are found, say “No issues found.”\n- Provide concrete suggestions for fixes when possible, or examples to illustrate the problem.\n","AGENTS.md":"# Dragonfly Development Guide\n\n> **Essential reference for working with the Dragonfly codebase**\n> Architecture, build system, testing infrastructure, and development workflows.\n\n---\n\n## Table of Contents\n\n1. [Critical Workflow Rules](#critical-workflow-rules)\n2. [Quick Command Reference](#quick-command-reference)\n3. [Project Overview](#project-overview)\n4. [Repository Structure](#repository-structure)\n5. [Build Instructions](#build-instructions)\n6. [Testing](#testing)\n7. [CI/CD Pipeline](#cicd-pipeline)\n8. [Code Style & Pre-commit Hooks](#code-style--pre-commit-hooks)\n9. [Third-Party Dependencies](#third-party-dependencies)\n10. [Platform Support](#platform-support)\n11. [CMake Build Options](#cmake-build-options)\n12. [Key Files Reference](#key-files-reference)\n13. [Common Pitfalls](#common-pitfalls)\n14. [Debugging Tips](#debugging-tips)\n15. [Validation Checklist](#validation-checklist)\n\n---\n\n## Critical Workflow Rules\n\n**MANDATORY - Always Follow This Order:**\n\n1. ✅ **Read Before Edit** - Always read files before modifying\n2. ✅ **Use Correct Build Commands** - See [Quick Command Reference](#quick-command-reference) below\n3. ✅ **Test After Changes** - Build and run a relevant unit test -\n `ninja <unit_test> && ./unit_test`\n4. ✅ **Format Code** - `pre-commit run --files <files>`\n5. ✅ **Follow Architecture** - See [Architecture Patterns](#architecture-patterns) below\n6. ✅ **Never Push to Main** - Always create a feature branch and open a PR. Never run `git push origin main`.\n\n### Pull Request Guidelines\n\n**Conciseness is Key**: PR descriptions should be short, focused, and easy to scan.\n- **Title**: Imperative, descriptive (e.g., \"Fix fiber stack overflow in test_reply_guard_oom\")\n- **Summary**: 1-2 sentences explaining *what* changed and *why*\n- **Changes**: Bullet points for key changes\n- **Fixes**: Link issues (e.g., \"Fixes #123\")\n- **Commit messages**: Keep every line (subject and body) <= 100 characters; wrap long descriptions\n\n---\n\n## Quick Command Reference\n\n**CRITICAL: Read the full sections below for context. These are shortcuts only.**\n\n### Building (see [Build Instructions](#build-instructions) for details)\n\n```bash\n# Debug build (for development)\n./helio/blaze.sh -DWITH_GCP=OFF\ncd build-dbg && ninja dragonfly # Build main binary\ncd build-dbg && ninja generic_family_test # Build specific test\n\n# Release build for local benchmarking\n./helio/blaze.sh -release -DWITH_GCP=OFF\ncd build-opt && ninja dragonfly\n```\n\n### Testing (see [Testing](#testing) for details)\n\n```bash\n# C++ Unit Tests\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n### Code Formatting\n\n```bash\n# Setup (once)\npipx install pre-commit clang-format black\npre-commit install\n\n# Format code\npre-commit run --files <files> # Format specific files\npre-commit run --all-files # Format all files\n```\n\n### Common Operations\n\n```bash\n# Check git status\ngit status\n\n# Check current branch\ngit branch\n\n# View recent commits\ngit log --oneline -10\n```\n\n---\n\n## Architecture Patterns\n\n**Code Style**: [.clang-format](.clang-format) - snake_case vars, PascalCase functions, kPascalCase constants\n\n**Comments**: Always put an empty line before a comment in code, unless the comment is the first\nline of a block (right after `{`). This applies to comments between statements, before if/for, and\ninside lambdas.\n\n**DO ✅**:\n- Function parameters: inputs first, output arguments (pointers the function writes to) last\n- Fiber-aware: `util::fb2::Mutex`, `util::fb2::Fiber` → [helio/util/fibers/](helio/util/fibers/)\n- Per-shard ops (no global state) → [docs/df-share-nothing.md](docs/df-share-nothing.md)\n- Command pattern → [src/server/set_family.cc](src/server/set_family.cc)\n- Error handling: `OpStatus` → [src/server/common.h](src/server/common.h)\n- Test patterns → [tests/dragonfly/conftest.py](tests/dragonfly/conftest.py)\n\n**DON'T ❌**:\n- `std::thread`, `std::mutex` (deadlocks!)\n- Global mutable state\n- Edit without reading\n- Skip tests\n- Use `std::regex` in fiber/server paths (recursive implementation can overflow small fiber stacks)\n- Use `./tools/docker/build.sh` for local development (use `ninja` instead)\n- Use `make` for incremental builds (use `ninja` instead)\n\n---\n\n## Project Overview\n\n**Dragonfly** is a high-performance, Redis and Memcached compatible in-memory data store written in C++20. It delivers significantly higher throughput than traditional single-threaded Redis implementations through innovative architectural choices.\n\n### Key Characteristics\n\n- **Language**: C++20 (Google C++ Style Guide 2020 version)\n- **Architecture**: Shared-nothing multi-threaded design (via `helio` library)\n- **Performance**: Uses io_uring (Linux 5.11+) for high-performance async I/O, with epoll fallback\n- **Threading Model**: Fiber-based cooperative multitasking with lock-free data structures\n- **Build System**: CMake + Ninja via `helio/blaze.sh` wrapper script\n- **Target Platform**: Linux (kernel 5.11+ recommended), FreeBSD support available\n- **Protocols**: Redis RESP2/RESP3, Memcached binary protocol\n- **Compatibility**: Drop-in replacement for Redis API coverage\n\n### Architectural Highlights\n\n**For detailed architecture documentation, see [docs/df-share-nothing.md](docs/df-share-nothing.md)**\n\n1. **Shared-Nothing Design**: Each thread operates independently with its own data structures, minimizing lock contention\n2. **Helio Framework**: Custom I/O and threading library built on io_uring/epoll with fiber support\n3. **DashTable**: Novel hash table implementation optimized for multi-core systems - see [docs/dashtable.md](docs/dashtable.md)\n4. **Transaction Model**: Non-blocking optimistic transactions - see [docs/transaction.md](docs/transaction.md)\n5. **Tiering Support**: Optional disk-backed storage for large datasets\n6. **Search Module**: Full-text search capabilities (when enabled with WITH_SEARCH)\n\n---\n\n## Repository Structure\n\n```\ndragonfly/\n├── src/ # Main C++ source code\n│ ├── server/ # Core server implementation\n│ │ ├── dfly_main.cc # Main entry point\n│ │ ├── main_service.cc # Service lifecycle & command routing\n│ │ ├── db_slice.cc # Per-thread database shard\n│ │ ├── engine_shard_set.cc # Shard management\n│ │ ├── cluster/ # Cluster mode implementation\n│ │ ├── journal/ # Replication journal\n│ │ ├── tiering/ # Tiered storage\n│ │ ├── search/ # Search module\n│ │ └── acl/ # Access control lists\n│ ├── core/ # Core data structures\n│ │ ├── dash.h # DashTable hash table\n│ │ ├── dense_set.h # Compact set implementation\n│ │ ├── string_map.h # Optimized string-keyed maps\n│ │ ├── search/ # Search core algorithms\n│ │ └── json/ # JSON support\n│ ├── facade/ # Network & command handling\n│ │ ├── dragonfly_connection.cc # Connection management\n│ │ ├── resp_srv_parser.cc # RESP command parser\n│ │ ├── resp_parser.cc # RESP reply parser\n│ │ └── memcache_parser.cc # Memcached protocol\n│ └── redis/ # Redis-specific implementations\n│ └── lua/ # Lua scripting support\n│\n├── helio/ # Git submodule: I/O and threading library\n│ │ # ** DO NOT EDIT unless contributing to helio **\n│ ├── util/ # Utilities: fibers, I/O, synchronization\n│ ├── io/ # io_uring & epoll abstraction\n│ └── blaze.sh # Build configuration wrapper\n│\n├── tests/ # Test suite\n│ ├── dragonfly/ # Python pytest integration/regression tests\n│ │ ├── conftest.py # Pytest fixtures & configuration\n│ │ ├── requirements.txt # Python test dependencies\n│ │ └── *.py # Test files\n│ └── pytest.ini # Pytest configuration & markers\n│\n├── docs/ # Documentation\n│ ├── build-from-source.md # Build instructions\n│ ├── dashtable.md # DashTable internals\n│ ├── transaction.md # Transaction model\n│ ├── df-share-nothing.md # Shared-nothing architecture\n│ └── differences.md # Differences from Redis\n│\n├── contrib/ # Utilities\n│ ├── docker/ # Docker configurations\n│ └── charts/dragonfly/ # Helm chart for Kubernetes\n│\n├── tools/ # Benchmarking & utility tools\n│ └── packaging/ # Packaging scripts\n│\n├── CMakeLists.txt # Root CMake configuration\n├── .clang-format # C++ formatting rules (clang-format v14.0.6)\n├── .pre-commit-config.yaml # Pre-commit hooks configuration\n├── pyproject.toml # Python formatting (Black, 100 chars)\n└── CONTRIBUTING.md # Contribution guidelines\n```\n\n### Critical Paths to Remember\n\n- **Main entry**: `src/server/dfly_main.cc`\n- **Command dispatch**: `src/server/main_service.cc`\n- **Data storage**: `src/server/db_slice.cc`\n- **Networking**: `src/facade/dragonfly_connection.cc`\n- **Helio library**: `helio/` (I/O and threading library)\n\n---\n\n## Build Instructions\n\n**For complete build instructions, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Quick Start\n\n**Debug build** (for development):\n```bash\n./helio/blaze.sh\ncd build-dbg && ninja dragonfly\n./dragonfly --alsologtostderr\n```\n\n**Release build** (for production/benchmarking):\n```bash\n./helio/blaze.sh -release\ncd build-opt && ninja dragonfly\n```\n\n**Production release build** (static linking, optimized):\n```bash\nmake release # Configure + build\nmake package # Create release packages with debug symbols\n```\n\nThe [Makefile](Makefile) builds production releases with:\n- Static linking: libstdc++, libgcc, Boost, OpenSSL\n- Architecture optimizations (x86_64: `-march=core2 -msse4.1 -mtune=skylake`)\n- Debug symbols (compressed)\n- Output: `build-release/dragonfly-{arch}.tar.gz`\n\n**Common build options**:\n- See [docs/build-from-source.md](docs/build-from-source.md) for all options\n\n---\n\n## Testing\n\n**For complete testing documentation, see [tests/README.md](tests/README.md)**\n\n### Quick Reference\n\n**C++ Unit Tests**:\n```bash\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n**Python Integration Tests (pytest)**:\n```bash\n# Run from the repo root. The binary path defaults to build-dbg/dragonfly.\n# Override with the DRAGONFLY_PATH env var:\nDRAGONFLY_PATH=build-dbg/dragonfly python3 -m pytest tests/dragonfly/pymemcached_test.py -xvs\n\n# Run a single test:\npython3 -m pytest tests/dragonfly/pymemcached_test.py::TestMemcached::test_basic -xvs\n```\n\n- `DRAGONFLY_PATH` — sets the path to the Dragonfly binary the test harness starts. Defaults to `build-dbg/dragonfly` relative to the `tests/dragonfly/` directory.\n- `--df` — passes **extra flags to the Dragonfly process** (not the binary path). For example: `--df logtostdout --df \"vmodule=*=1\"`.\n\n---\n\n## CI/CD Pipeline\n\n**For complete CI configuration, see [.github/workflows/ci.yml](.github/workflows/ci.yml)**\n\nThe CI workflow runs on all PRs and includes:\n- **Pre-commit checks**: clang-format, black formatters\n- **Build matrix**: Multiple OS/compiler/sanitizer combinations (Ubuntu 20/24, Alpine, GCC/Clang, ASAN/UBSAN)\n- **Test execution**: C++ unit tests, Python integration tests, cluster mode tests\n- **Additional validations**: Helm charts, Docker image builds\n\n---\n\n## Code Style & Pre-commit Hooks\n\n**For complete contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md)**\n\n**Code style configuration files**:\n- **C++**: [.clang-format](.clang-format) - Google C++ Style Guide (2020), clang-format v14.0.6, 100 char limit\n- **Python**: [pyproject.toml](pyproject.toml) - Black formatter, 100 char limit, PEP 8 compliant\n- **Pre-commit hooks**: [.pre-commit-config.yaml](.pre-commit-config.yaml) - Automated formatting checks\n\n**Quick setup**:\n```bash\npipx install pre-commit clang-format black\npre-commit install\npre-commit run --all-files # Run all formatters\n```\n\n---\n\n## Third-Party Dependencies\n\n**Key Libraries**: Abseil (strings/flags), Boost 1.71+ (context/intrusive), mimalloc (allocator), jsoncons (JSON), OpenSSL (TLS), libunwind (traces)\n\n**Build artifacts**: `build-dbg/third_party/` - DO NOT edit\n\n**For complete dependency info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## Platform Support\n\n**Linux**: Primary platform. Kernel 5.11+ (io_uring), 5.1+ (basic), < 5.1 (epoll fallback)\n- Check: `uname -r`\n- Force epoll: `--proactor_type=epoll`\n- Docker: `--security-opt seccomp=unconfined`\n\n**FreeBSD**: Supported (kqueue backend)\n\n**macOS**: Not supported for production (use Docker/Linux)\n\n**For complete platform info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## CMake Build Options\n\n**For complete list of build options, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Common Options\n\nPass options to `helio/blaze.sh` with `-D` prefix:\n\n```bash\n./helio/blaze.sh -DWITH_SEARCH=OFF -DWITH_GCP=OFF\n```\n\n**Most useful options**:\n- `WITH_ASAN=ON` / `WITH_USAN=ON` - Enable sanitizers for debugging\n- `WITH_SEARCH=OFF` - Disable search module for faster builds\n- `WITH_GCP=OFF` - Disable GCP cloud library\n- `WITH_TIERING=OFF` - Disable disk storage\n- `USE_MOLD=ON` - Faster linking with LTO (production builds)\n\n**Quick configurations**:\n```bash\n# Minimal build (fast compilation)\n./helio/blaze.sh -DWITH_GPERF=OFF -DWITH_GCP=OFF -DWITH_TIERING=OFF -DWITH_SEARCH=OFF\n\n# Full-featured (all options ON by default)\n./helio/blaze.sh\n\n# Production optimized\n./helio/blaze.sh -release -DUSE_MOLD=ON\n```\n\n---\n\n## Key Files Reference\n\nQuick reference to the most important files in the codebase.\n\n| Purpose | File Path |\n|---------|-----------|\n| **Entry Points & Core** | |\n| Main entry point | `src/server/dfly_main.cc` |\n| Server lifecycle & command routing | `src/server/main_service.cc` |\n| Per-thread database shard | `src/server/db_slice.cc` |\n| Shard management | `src/server/engine_shard_set.cc` |\n| **Data Structures** | |\n| DashTable hash table | `src/core/dash.h` |\n| Dense set implementation | `src/core/dense_set.h` |\n| String map | `src/core/string_map.h` |\n| **Networking** | |\n| Connection handling | `src/facade/dragonfly_connection.cc` |\n| RESP command parser | `src/facade/resp_srv_parser.cc` |\n| RESP reply parser | `src/facade/resp_parser.cc` |\n| Memcached protocol parser | `src/facade/memcache_parser.cc` |\n| **Build System** | |\n| Root CMake config | `CMakeLists.txt` |\n| Build script wrapper | `helio/blaze.sh` |\n| Server CMake config | `src/server/CMakeLists.txt` |\n| **CI/CD** | |\n| Main CI workflow | `.github/workflows/ci.yml` |\n| Pre-commit config | `.pre-commit-config.yaml` |\n| **Code Style** | |\n| C++ formatting | `.clang-format` |\n| Python formatting | `pyproject.toml` |\n| **Testing** | |\n| Pytest configuration | `tests/pytest.ini` |\n| Pytest fixtures | `tests/dragonfly/conftest.py` |\n| Test requirements | `tests/dragonfly/requirements.txt` |\n| **Documentation** | |\n| Build instructions | `docs/build-from-source.md` |\n| Architecture overview | `docs/df-share-nothing.md` |\n| DashTable internals | `docs/dashtable.md` |\n| Transaction model | `docs/transaction.md` |\n| **Configuration** | |\n| Contributing guide | `CONTRIBUTING.md` |\n| CLA agreement | `CLA.txt` |\n\n---\n\n## Common Pitfalls\n\n1. **Pre-commit not installed**: `pipx install pre-commit clang-format black && pre-commit install`\n2. **Wrong binary**: Debug: `build-dbg/dragonfly`, Release: `build-opt/dragonfly`\n3. **Wrong build command**: Use `cd build-dbg && ninja <target>`, NOT `./tools/docker/build.sh`\n4. **Test timeouts**: `timeout 20m ctest -V -L DFLY`\n5. **ASAN leaks**: Check CI, suppress in `helio/util/asan_suppressions.txt`\n6. **Helio modifications**: DON'T edit `helio/` (it's a git submodule - changes go upstream)\n7. **CodeQL checks**: DON'T run codeql_checker when testing changes - it's slow and unnecessary for development\n\n---\n\n## Debugging Tips\n\n**Logging**: `--alsologtostderr --v=1 --vmodule=module=2`\n\n**ASAN**: `ASAN_OPTIONS=detect_leaks=1:symbolize=1`, suppressions: `helio/util/asan_suppressions.txt`\n\n**CI reproduction**: See [.github/workflows/ci.yml](.github/workflows/ci.yml)\n\n**Troubleshooting**: Check fiber deadlocks (use `util::fb2` not `std::mutex`), timeout issues (`--test_timeout`), ASAN reports\n\n---\n\n## Validation Checklist\n\nBefore claiming a task is complete, verify:\n\n### Code Quality\n\n- [ ] Code compiles without errors: `cd build-dbg && ninja dragonfly`\n- [ ] Code compiles without warnings (CI uses `-Werror`)\n- [ ] Code follows Google C++ Style Guide (run `clang-format`)\n- [ ] No new ASAN/UBSAN violations\n\n### Testing\n\n- [ ] All existing C++ unit tests pass: `ctest -V -L DFLY`\n- [ ] New feature has corresponding test coverage\n- [ ] Tests pass in both Debug and Release builds\n- [ ] Tests pass with ASAN/UBSAN enabled (if applicable)\n- [ ] **DO NOT run codeql_checker** - it's slow and unnecessary for development testing\n\n### Pre-commit & Style\n\n- [ ] Pre-commit hooks installed: `pre-commit install`\n- [ ] Code formatted with clang-format (C++) and black (Python)\n\n### Documentation\n\n- [ ] Public APIs have comments explaining purpose\n- [ ] Complex algorithms have explanatory comments\n- [ ] README or docs updated if behavior changes\n- [ ] No commented-out code left in final commit\n\n### Performance\n\n- [ ] No obvious performance regressions (run benchmarks if needed)\n- [ ] No unnecessary allocations in hot paths\n- [ ] Lock-free data structures used where appropriate\n","CLAUDE.md":"AGENTS.md","tools/packaging/README.md":"# Installation Packages\n\n## Overview\nThis directory includes a set of files and scripts to build installation package for various Linux distributions.\n\n## Debian\nThe file to build the Debian package all located under \"debian\" directory.\nThe resulting package will install the binary of Dragonfly as well as generate a new service entry for dragonfly,\nthat can be controlled with \"systemctl\" command, to start, stop and check status of.\n### Building\nTo build the package, you have a script called \"generate_debian_package.sh\". This script accepts the following parameters:\n* Optional binary path - the location from which to take the binary for the installation. The default for this is \"repo path/build-opt\".\nThe location to which the resulting package is writing is at the location from which the script is executed.\nThis script is depends on the following packages:\n* git\n* moreutils\n* debhelper\n* dpkg-dev\n\nTo build:\n```\n/path/to/dragonfly/tools/packaging/generate_debian_package.sh [/path/to/dragonfly-binary-file]\n```\n\nThis can only be run on Debian based hosts.\nYou can use the flowing docker file to generate this package:\n```\nFROM ubuntu:20.04\nARG DEBIAN_FRONTEND=noninteractive\nRUN apt update -y && apt-get install -y gcc dpkg-dev gpg vim wget git moreutils debhelper\n```\nBuild the above docker and then run it with your dragonfly source code path mount as volume for the build:\n```\ndocker build -t ubuntu-package .\ndocker run --rm -ti -v /path/to/dragonfly-repo:/mydocker-path ubuntu-package bash\n```\nAgain note that you need to be at \"main\" branch to successfully build this package.\nNote: If at the end of the installation you see a message \"/usr/bin/deb-systemd-helper: error: systemctl preset failed on dragonfly.service: No such file or directory\",\nyou can ignore it, this seem to be related to [the following issue](https://groups.google.com/g/linux.debian.bugs.dist/c/m6xGZ82TdvM).\n","tools/packaging/osrepos/README.md":"# Package repositories for rpm and debian packages\n\nThis directory contains scripts and definitions for setting up YUM and apt repositories for Linux users to install\ndragonfly packages.\n\nThe repositories are served as static websites. The generate-site workflow is used to set up and deploy the sites using\nscripts and definitions included here.\n\nThe workflow does the following tasks:\n\n* Download the latest 5 releases from dragonfly releases page, specifically deb and rpm assets\n * for deb files, only the latest package is downloaded and present (see note below)\n* Set up a directory structure separating deb and rpm files into version specific paths\n* Sign the packages (see note on GPG)\n* Deploy the assets prepared, along with the public GPG key and repo definitions for apt and rpm tooling\n\n## Using the YUM repository\n\nAdd the repository using:\n\n```shell\nsudo dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo\n```\n\nThen install dragonfly as usual, or a specific version:\n\n```shell\nsudo dnf -y install dragonfly-0:v1.33.1-1.fc30.x86_64\n```\n\n## Using the APT repository\n\nFirst download the public GPG key to an appropriate location:\n\n```shell\nsudo curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public\n```\n\nThen add the sources file:\n\n```shell\nsudo curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources\n```\n\nFinally install dragonfly using apt\n\n```shell\nsudo apt update && sudo apt install dragonfly\n```\n\n#### Versions in APT repository\n\nUnlike the yum repo, the apt repo only has the latest version. The reason for this is the tool, `reprepro` supplied by\ndebian to build repositories only supports multiple\nversions in version 5.4 onwards, and the github runner using ubuntu-latest does not have this version.\n\nAnother option would be to use the components feature of apt repositories in the sources file we ask users to install,\nbut then the versions would need\nto be hardcoded in the sources file and the user would have\nto update the file with each new release which makes for a bad user experience. As of now users wanting older packages\nshould download them directly.\n\n### Signing packages\n\nThe packages are signed using the GPG key imported from the secret GPG_PRIVATE_KEY in this repository.\n\nThe corresponding public key is served with site assets, so the apt/yum/dnf based tooling can consume the public key to\nverify package integrity.\n\n### TODO\n\n- [X] debian packages signing (not required? release file is signed)\n- [X] debian repo metadata setup\n- [ ] tests asserting that packages are installable?\n","tools/vector-benches/README.md":"Tool for benchmarking vector search with randomized vectors.\n\n## Logic\nThe tool connects to the Redis/Dragonfly instance.\n1. Checks if the database has enough data (at least 50% of requested `-n`). If not, it **flushes the DB** and generates random vectors.\n2. Checks if the index `idx` exists. If not, it creates it.\n3. Runs concurrent search queries and reports latency/QPS.\n\n## Arguments\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `-n` | 50000 | **Number of vectors** to populate if the DB is empty. |\n| `-q` | 1000 | **Total number of queries** to run during the benchmark. |\n| `-t` | 8 | **Query threads**. Number of concurrent workers sending queries. |\n| `-d` | 100 | **Vector dimension**. Size of the float32 vectors. |\n| `-k` | 10 | **Top K**. Number of nearest neighbors to retrieve per query. |\n| `-p` | 6379 | **Port** of the server. |\n| `-h` | localhost | **Host** of the server. |\n\nRun with `-h` (help) to see these defaults in the tool itself.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Dragonfly Development Guide\n\n> **Essential reference for working with the Dragonfly codebase**\n> Architecture, build system, testing infrastructure, and development workflows.\n\n---\n\n## Table of Contents\n\n1. [Critical Workflow Rules](#critical-workflow-rules)\n2. [Quick Command Reference](#quick-command-reference)\n3. [Project Overview](#project-overview)\n4. [Repository Structure](#repository-structure)\n5. [Build Instructions](#build-instructions)\n6. [Testing](#testing)\n7. [CI/CD Pipeline](#cicd-pipeline)\n8. [Code Style & Pre-commit Hooks](#code-style--pre-commit-hooks)\n9. [Third-Party Dependencies](#third-party-dependencies)\n10. [Platform Support](#platform-support)\n11. [CMake Build Options](#cmake-build-options)\n12. [Key Files Reference](#key-files-reference)\n13. [Common Pitfalls](#common-pitfalls)\n14. [Debugging Tips](#debugging-tips)\n15. [Validation Checklist](#validation-checklist)\n\n---\n\n## Critical Workflow Rules\n\n**MANDATORY - Always Follow This Order:**\n\n1. ✅ **Read Before Edit** - Always read files before modifying\n2. ✅ **Use Correct Build Commands** - See [Quick Command Reference](#quick-command-reference) below\n3. ✅ **Test After Changes** - Build and run a relevant unit test -\n `ninja <unit_test> && ./unit_test`\n4. ✅ **Format Code** - `pre-commit run --files <files>`\n5. ✅ **Follow Architecture** - See [Architecture Patterns](#architecture-patterns) below\n6. ✅ **Never Push to Main** - Always create a feature branch and open a PR. Never run `git push origin main`.\n\n### Pull Request Guidelines\n\n**Conciseness is Key**: PR descriptions should be short, focused, and easy to scan.\n- **Title**: Imperative, descriptive (e.g., \"Fix fiber stack overflow in test_reply_guard_oom\")\n- **Summary**: 1-2 sentences explaining *what* changed and *why*\n- **Changes**: Bullet points for key changes\n- **Fixes**: Link issues (e.g., \"Fixes #123\")\n- **Commit messages**: Keep every line (subject and body) <= 100 characters; wrap long descriptions\n\n---\n\n## Quick Command Reference\n\n**CRITICAL: Read the full sections below for context. These are shortcuts only.**\n\n### Building (see [Build Instructions](#build-instructions) for details)\n\n```bash\n# Debug build (for development)\n./helio/blaze.sh -DWITH_GCP=OFF\ncd build-dbg && ninja dragonfly # Build main binary\ncd build-dbg && ninja generic_family_test # Build specific test\n\n# Release build for local benchmarking\n./helio/blaze.sh -release -DWITH_GCP=OFF\ncd build-opt && ninja dragonfly\n```\n\n### Testing (see [Testing](#testing) for details)\n\n```bash\n# C++ Unit Tests\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n### Code Formatting\n\n```bash\n# Setup (once)\npipx install pre-commit clang-format black\npre-commit install\n\n# Format code\npre-commit run --files <files> # Format specific files\npre-commit run --all-files # Format all files\n```\n\n### Common Operations\n\n```bash\n# Check git status\ngit status\n\n# Check current branch\ngit branch\n\n# View recent commits\ngit log --oneline -10\n```\n\n---\n\n## Architecture Patterns\n\n**Code Style**: [.clang-format](.clang-format) - snake_case vars, PascalCase functions, kPascalCase constants\n\n**Comments**: Always put an empty line before a comment in code, unless the comment is the first\nline of a block (right after `{`). This applies to comments between statements, before if/for, and\ninside lambdas.\n\n**DO ✅**:\n- Function parameters: inputs first, output arguments (pointers the function writes to) last\n- Fiber-aware: `util::fb2::Mutex`, `util::fb2::Fiber` → [helio/util/fibers/](helio/util/fibers/)\n- Per-shard ops (no global state) → [docs/df-share-nothing.md](docs/df-share-nothing.md)\n- Command pattern → [src/server/set_family.cc](src/server/set_family.cc)\n- Error handling: `OpStatus` → [src/server/common.h](src/server/common.h)\n- Test patterns → [tests/dragonfly/conftest.py](tests/dragonfly/conftest.py)\n\n**DON'T ❌**:\n- `std::thread`, `std::mutex` (deadlocks!)\n- Global mutable state\n- Edit without reading\n- Skip tests\n- Use `std::regex` in fiber/server paths (recursive implementation can overflow small fiber stacks)\n- Use `./tools/docker/build.sh` for local development (use `ninja` instead)\n- Use `make` for incremental builds (use `ninja` instead)\n\n---\n\n## Project Overview\n\n**Dragonfly** is a high-performance, Redis and Memcached compatible in-memory data store written in C++20. It delivers significantly higher throughput than traditional single-threaded Redis implementations through innovative architectural choices.\n\n### Key Characteristics\n\n- **Language**: C++20 (Google C++ Style Guide 2020 version)\n- **Architecture**: Shared-nothing multi-threaded design (via `helio` library)\n- **Performance**: Uses io_uring (Linux 5.11+) for high-performance async I/O, with epoll fallback\n- **Threading Model**: Fiber-based cooperative multitasking with lock-free data structures\n- **Build System**: CMake + Ninja via `helio/blaze.sh` wrapper script\n- **Target Platform**: Linux (kernel 5.11+ recommended), FreeBSD support available\n- **Protocols**: Redis RESP2/RESP3, Memcached binary protocol\n- **Compatibility**: Drop-in replacement for Redis API coverage\n\n### Architectural Highlights\n\n**For detailed architecture documentation, see [docs/df-share-nothing.md](docs/df-share-nothing.md)**\n\n1. **Shared-Nothing Design**: Each thread operates independently with its own data structures, minimizing lock contention\n2. **Helio Framework**: Custom I/O and threading library built on io_uring/epoll with fiber support\n3. **DashTable**: Novel hash table implementation optimized for multi-core systems - see [docs/dashtable.md](docs/dashtable.md)\n4. **Transaction Model**: Non-blocking optimistic transactions - see [docs/transaction.md](docs/transaction.md)\n5. **Tiering Support**: Optional disk-backed storage for large datasets\n6. **Search Module**: Full-text search capabilities (when enabled with WITH_SEARCH)\n\n---\n\n## Repository Structure\n\n```\ndragonfly/\n├── src/ # Main C++ source code\n│ ├── server/ # Core server implementation\n│ │ ├── dfly_main.cc # Main entry point\n│ │ ├── main_service.cc # Service lifecycle & command routing\n│ │ ├── db_slice.cc # Per-thread database shard\n│ │ ├── engine_shard_set.cc # Shard management\n│ │ ├── cluster/ # Cluster mode implementation\n│ │ ├── journal/ # Replication journal\n│ │ ├── tiering/ # Tiered storage\n│ │ ├── search/ # Search module\n│ │ └── acl/ # Access control lists\n│ ├── core/ # Core data structures\n│ │ ├── dash.h # DashTable hash table\n│ │ ├── dense_set.h # Compact set implementation\n│ │ ├── string_map.h # Optimized string-keyed maps\n│ │ ├── search/ # Search core algorithms\n│ │ └── json/ # JSON support\n│ ├── facade/ # Network & command handling\n│ │ ├── dragonfly_connection.cc # Connection management\n│ │ ├── resp_srv_parser.cc # RESP command parser\n│ │ ├── resp_parser.cc # RESP reply parser\n│ │ └── memcache_parser.cc # Memcached protocol\n│ └── redis/ # Redis-specific implementations\n│ └── lua/ # Lua scripting support\n│\n├── helio/ # Git submodule: I/O and threading library\n│ │ # ** DO NOT EDIT unless contributing to helio **\n│ ├── util/ # Utilities: fibers, I/O, synchronization\n│ ├── io/ # io_uring & epoll abstraction\n│ └── blaze.sh # Build configuration wrapper\n│\n├── tests/ # Test suite\n│ ├── dragonfly/ # Python pytest integration/regression tests\n│ │ ├── conftest.py # Pytest fixtures & configuration\n│ │ ├── requirements.txt # Python test dependencies\n│ │ └── *.py # Test files\n│ └── pytest.ini # Pytest configuration & markers\n│\n├── docs/ # Documentation\n│ ├── build-from-source.md # Build instructions\n│ ├── dashtable.md # DashTable internals\n│ ├── transaction.md # Transaction model\n│ ├── df-share-nothing.md # Shared-nothing architecture\n│ └── differences.md # Differences from Redis\n│\n├── contrib/ # Utilities\n│ ├── docker/ # Docker configurations\n│ └── charts/dragonfly/ # Helm chart for Kubernetes\n│\n├── tools/ # Benchmarking & utility tools\n│ └── packaging/ # Packaging scripts\n│\n├── CMakeLists.txt # Root CMake configuration\n├── .clang-format # C++ formatting rules (clang-format v14.0.6)\n├── .pre-commit-config.yaml # Pre-commit hooks configuration\n├── pyproject.toml # Python formatting (Black, 100 chars)\n└── CONTRIBUTING.md # Contribution guidelines\n```\n\n### Critical Paths to Remember\n\n- **Main entry**: `src/server/dfly_main.cc`\n- **Command dispatch**: `src/server/main_service.cc`\n- **Data storage**: `src/server/db_slice.cc`\n- **Networking**: `src/facade/dragonfly_connection.cc`\n- **Helio library**: `helio/` (I/O and threading library)\n\n---\n\n## Build Instructions\n\n**For complete build instructions, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Quick Start\n\n**Debug build** (for development):\n```bash\n./helio/blaze.sh\ncd build-dbg && ninja dragonfly\n./dragonfly --alsologtostderr\n```\n\n**Release build** (for production/benchmarking):\n```bash\n./helio/blaze.sh -release\ncd build-opt && ninja dragonfly\n```\n\n**Production release build** (static linking, optimized):\n```bash\nmake release # Configure + build\nmake package # Create release packages with debug symbols\n```\n\nThe [Makefile](Makefile) builds production releases with:\n- Static linking: libstdc++, libgcc, Boost, OpenSSL\n- Architecture optimizations (x86_64: `-march=core2 -msse4.1 -mtune=skylake`)\n- Debug symbols (compressed)\n- Output: `build-release/dragonfly-{arch}.tar.gz`\n\n**Common build options**:\n- See [docs/build-from-source.md](docs/build-from-source.md) for all options\n\n---\n\n## Testing\n\n**For complete testing documentation, see [tests/README.md](tests/README.md)**\n\n### Quick Reference\n\n**C++ Unit Tests**:\n```bash\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n**Python Integration Tests (pytest)**:\n```bash\n# Run from the repo root. The binary path defaults to build-dbg/dragonfly.\n# Override with the DRAGONFLY_PATH env var:\nDRAGONFLY_PATH=build-dbg/dragonfly python3 -m pytest tests/dragonfly/pymemcached_test.py -xvs\n\n# Run a single test:\npython3 -m pytest tests/dragonfly/pymemcached_test.py::TestMemcached::test_basic -xvs\n```\n\n- `DRAGONFLY_PATH` — sets the path to the Dragonfly binary the test harness starts. Defaults to `build-dbg/dragonfly` relative to the `tests/dragonfly/` directory.\n- `--df` — passes **extra flags to the Dragonfly process** (not the binary path). For example: `--df logtostdout --df \"vmodule=*=1\"`.\n\n---\n\n## CI/CD Pipeline\n\n**For complete CI configuration, see [.github/workflows/ci.yml](.github/workflows/ci.yml)**\n\nThe CI workflow runs on all PRs and includes:\n- **Pre-commit checks**: clang-format, black formatters\n- **Build matrix**: Multiple OS/compiler/sanitizer combinations (Ubuntu 20/24, Alpine, GCC/Clang, ASAN/UBSAN)\n- **Test execution**: C++ unit tests, Python integration tests, cluster mode tests\n- **Additional validations**: Helm charts, Docker image builds\n\n---\n\n## Code Style & Pre-commit Hooks\n\n**For complete contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md)**\n\n**Code style configuration files**:\n- **C++**: [.clang-format](.clang-format) - Google C++ Style Guide (2020), clang-format v14.0.6, 100 char limit\n- **Python**: [pyproject.toml](pyproject.toml) - Black formatter, 100 char limit, PEP 8 compliant\n- **Pre-commit hooks**: [.pre-commit-config.yaml](.pre-commit-config.yaml) - Automated formatting checks\n\n**Quick setup**:\n```bash\npipx install pre-commit clang-format black\npre-commit install\npre-commit run --all-files # Run all formatters\n```\n\n---\n\n## Third-Party Dependencies\n\n**Key Libraries**: Abseil (strings/flags), Boost 1.71+ (context/intrusive), mimalloc (allocator), jsoncons (JSON), OpenSSL (TLS), libunwind (traces)\n\n**Build artifacts**: `build-dbg/third_party/` - DO NOT edit\n\n**For complete dependency info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## Platform Support\n\n**Linux**: Primary platform. Kernel 5.11+ (io_uring), 5.1+ (basic), < 5.1 (epoll fallback)\n- Check: `uname -r`\n- Force epoll: `--proactor_type=epoll`\n- Docker: `--security-opt seccomp=unconfined`\n\n**FreeBSD**: Supported (kqueue backend)\n\n**macOS**: Not supported for production (use Docker/Linux)\n\n**For complete platform info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## CMake Build Options\n\n**For complete list of build options, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Common Options\n\nPass options to `helio/blaze.sh` with `-D` prefix:\n\n```bash\n./helio/blaze.sh -DWITH_SEARCH=OFF -DWITH_GCP=OFF\n```\n\n**Most useful options**:\n- `WITH_ASAN=ON` / `WITH_USAN=ON` - Enable sanitizers for debugging\n- `WITH_SEARCH=OFF` - Disable search module for faster builds\n- `WITH_GCP=OFF` - Disable GCP cloud library\n- `WITH_TIERING=OFF` - Disable disk storage\n- `USE_MOLD=ON` - Faster linking with LTO (production builds)\n\n**Quick configurations**:\n```bash\n# Minimal build (fast compilation)\n./helio/blaze.sh -DWITH_GPERF=OFF -DWITH_GCP=OFF -DWITH_TIERING=OFF -DWITH_SEARCH=OFF\n\n# Full-featured (all options ON by default)\n./helio/blaze.sh\n\n# Production optimized\n./helio/blaze.sh -release -DUSE_MOLD=ON\n```\n\n---\n\n## Key Files Reference\n\nQuick reference to the most important files in the codebase.\n\n| Purpose | File Path |\n|---------|-----------|\n| **Entry Points & Core** | |\n| Main entry point | `src/server/dfly_main.cc` |\n| Server lifecycle & command routing | `src/server/main_service.cc` |\n| Per-thread database shard | `src/server/db_slice.cc` |\n| Shard management | `src/server/engine_shard_set.cc` |\n| **Data Structures** | |\n| DashTable hash table | `src/core/dash.h` |\n| Dense set implementation | `src/core/dense_set.h` |\n| String map | `src/core/string_map.h` |\n| **Networking** | |\n| Connection handling | `src/facade/dragonfly_connection.cc` |\n| RESP command parser | `src/facade/resp_srv_parser.cc` |\n| RESP reply parser | `src/facade/resp_parser.cc` |\n| Memcached protocol parser | `src/facade/memcache_parser.cc` |\n| **Build System** | |\n| Root CMake config | `CMakeLists.txt` |\n| Build script wrapper | `helio/blaze.sh` |\n| Server CMake config | `src/server/CMakeLists.txt` |\n| **CI/CD** | |\n| Main CI workflow | `.github/workflows/ci.yml` |\n| Pre-commit config | `.pre-commit-config.yaml` |\n| **Code Style** | |\n| C++ formatting | `.clang-format` |\n| Python formatting | `pyproject.toml` |\n| **Testing** | |\n| Pytest configuration | `tests/pytest.ini` |\n| Pytest fixtures | `tests/dragonfly/conftest.py` |\n| Test requirements | `tests/dragonfly/requirements.txt` |\n| **Documentation** | |\n| Build instructions | `docs/build-from-source.md` |\n| Architecture overview | `docs/df-share-nothing.md` |\n| DashTable internals | `docs/dashtable.md` |\n| Transaction model | `docs/transaction.md` |\n| **Configuration** | |\n| Contributing guide | `CONTRIBUTING.md` |\n| CLA agreement | `CLA.txt` |\n\n---\n\n## Common Pitfalls\n\n1. **Pre-commit not installed**: `pipx install pre-commit clang-format black && pre-commit install`\n2. **Wrong binary**: Debug: `build-dbg/dragonfly`, Release: `build-opt/dragonfly`\n3. **Wrong build command**: Use `cd build-dbg && ninja <target>`, NOT `./tools/docker/build.sh`\n4. **Test timeouts**: `timeout 20m ctest -V -L DFLY`\n5. **ASAN leaks**: Check CI, suppress in `helio/util/asan_suppressions.txt`\n6. **Helio modifications**: DON'T edit `helio/` (it's a git submodule - changes go upstream)\n7. **CodeQL checks**: DON'T run codeql_checker when testing changes - it's slow and unnecessary for development\n\n---\n\n## Debugging Tips\n\n**Logging**: `--alsologtostderr --v=1 --vmodule=module=2`\n\n**ASAN**: `ASAN_OPTIONS=detect_leaks=1:symbolize=1`, suppressions: `helio/util/asan_suppressions.txt`\n\n**CI reproduction**: See [.github/workflows/ci.yml](.github/workflows/ci.yml)\n\n**Troubleshooting**: Check fiber deadlocks (use `util::fb2` not `std::mutex`), timeout issues (`--test_timeout`), ASAN reports\n\n---\n\n## Validation Checklist\n\nBefore claiming a task is complete, verify:\n\n### Code Quality\n\n- [ ] Code compiles without errors: `cd build-dbg && ninja dragonfly`\n- [ ] Code compiles without warnings (CI uses `-Werror`)\n- [ ] Code follows Google C++ Style Guide (run `clang-format`)\n- [ ] No new ASAN/UBSAN violations\n\n### Testing\n\n- [ ] All existing C++ unit tests pass: `ctest -V -L DFLY`\n- [ ] New feature has corresponding test coverage\n- [ ] Tests pass in both Debug and Release builds\n- [ ] Tests pass with ASAN/UBSAN enabled (if applicable)\n- [ ] **DO NOT run codeql_checker** - it's slow and unnecessary for development testing\n\n### Pre-commit & Style\n\n- [ ] Pre-commit hooks installed: `pre-commit install`\n- [ ] Code formatted with clang-format (C++) and black (Python)\n\n### Documentation\n\n- [ ] Public APIs have comments explaining purpose\n- [ ] Complex algorithms have explanatory comments\n- [ ] README or docs updated if behavior changes\n- [ ] No commented-out code left in final commit\n\n### Performance\n\n- [ ] No obvious performance regressions (run benchmarks if needed)\n- [ ] No unnecessary allocations in hot paths\n- [ ] Lock-free data structures used where appropriate\n","isInternal":false,"tokens":4580,"sizeBytes":18443},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"AGENTS.md","isInternal":false,"tokens":3,"sizeBytes":9},{"name":"copilot-instructions.md","path":".github/copilot-instructions.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.github/copilot-instructions.md","title":"GitHub Copilot Instructions","category":"copilot-instructions","format":"markdown","content":"---\ndescription: 'Code review guidelines for GitHub copilot in this project'\napplyTo: '**'\nexcludeAgent: [\"coding-agent\"]\n---\n\n# Code Review Instructions\n\nKeep reviews high-signal and minimal. Only comment on real bugs with high confidence.\n\n## Comment Only When\n- The issue is a correctness, security, concurrency, or architecture problem.\n- The impact is clear and non-trivial.\n- You can point to concrete evidence in the diff (not speculation).\n\n## Avoid\n- Style, formatting, naming, or minor performance nits.\n- Optional refactors or “nice to have” suggestions.\n- Praise, restating the code, or long explanations.\n- Duplicate comments for the same root cause.\n\n## Review Style\n- Be terse: 1-2 sentences per issue.\n- Include file and line references when possible.\n- If no issues are found, say “No issues found.”\n- Provide concrete suggestions for fixes when possible, or examples to illustrate the problem.\n","frontmatter":{"description":"Code review guidelines for GitHub copilot in this project","applyTo":"**","excludeAgent":["coding-agent"]},"isInternal":false,"tokens":198,"sizeBytes":920},{"name":"SKILL.md","path":".claude/skills/benchmark-report/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/benchmark-report/SKILL.md","title":"benchmark-report","category":"anthropic-skill","format":"markdown","content":"---\nname: benchmark-report\ndescription: Generate a technical benchmark report in markdown from raw memtier/redis-benchmark output. Use this whenever the user provides benchmark results, log files, memtier JSON output, or a directory of benchmark data and wants a written report or comparison document. Trigger even when the user says \"write up these results\", \"make a report from this data\", \"turn this into a blog post\", or just drops a directory path. Always use this skill when the user provides benchmark files and wants a human-readable output. Reports compare Dragonfly against another system (ElastiCache, Redis, Valkey) but can handle single-system data too.\nversion: 1.0.0\n---\n\n# Benchmark Report Generator\n\nGenerate a technical markdown report from raw memtier/redis-benchmark data. The report is aimed at engineers evaluating Dragonfly for production use.\n\n## Step 0: Interview before writing\n\nBefore touching any files, ask the user 3–4 targeted questions to fill in context that cannot be extracted from the raw data. Read the data directory first so your questions are specific, not generic.\n\nQuestions to ask (adapt based on what's missing after scanning the files):\n\n1. **Scenario**: What is the purpose of this benchmark? (e.g., \"SSD tiering comparison\", \"evaluating ElastiCache replacement\", \"testing read-heavy workload at scale\") — this becomes the intro paragraph and title.\n2. **Instance types**: What instance type / tier was each system running on? Include memory size.\n3. **Storage configuration**: For Dragonfly — is SSD tiering enabled? What is the memory limit vs total dataset size?\n4. **Anything else relevant**: Pricing comparison, region, network setup, anything not in the scripts/logs.\n\nWait for the user's answers before proceeding to Step 1. If the user says \"just use TODOs\" or similar, proceed without waiting.\n\n## What the user provides\n\n- A directory path with benchmark data files\n- Context answers from the interview above\n\nTypical directory layout:\n```\ndata_dir/\n *.sh — benchmark script (parameters, key distribution, test phases)\n *.json — memtier JSON output per system per phase\n *.log — memtier log output per system\n```\n\nFiles may be flat in one directory or split across subdirectories. Read them all.\n\n## Step 1: Scan and parse\n\nList all files in the directory. Then read them:\n\n**Scripts (`*.sh`)** — extract:\n- `KEY_MAXIMUM`, `DATA_SIZE`, `PIPELINE`, `CLIENTS`, `THREADS`, `TEST_TIME`\n- `FILL_RATIO`, `ACCESS_RATIO`, `FILL_KEY_PATTERN`, `ACCESS_KEY_PATTERN`\n- Key distribution flags: `--key-stddev`, `--key-zipf-exp`, etc.\n- Which phases ran (`RUN_FILL=true`, `RUN_ACCESS_READ_002=true`, etc.)\n- Hostnames — used to identify systems under test\n\n**JSON files (`*.json`)** — from memtier `--json-out-file` output:\n- `configuration.server` → hostname (used to identify the system)\n- `configuration.clients`, `threads`, `pipeline`, `ratio`, `key_maximum`, `data_size`, `key_stddev`\n- `ALL STATS.Gets` or `ALL STATS.Sets` → `Ops/sec`, `Average Latency`, `Percentile Latencies` (p50/p99/p99.9)\n- `ALL STATS.Runtime.Total duration` → test duration in ms\n\n**Log files (`*.log`)** — from memtier stdout:\n- Look for the `ALL STATS` table block. Take the last occurrence (final aggregate):\n ```\n Type Ops/sec ... Avg. Latency p50 Latency p99 Latency p99.9 Latency\n Sets 220503.72 ... 4.32106 2.95900 9.72700 53.24700\n Gets 138188.17 ... 6.94510 5.53500 36.09500 54.27100\n ```\n- Look for `real Xm Y.Zs` timing lines to capture wall-clock duration per phase\n\n## Step 2: Identify systems under test\n\nName each system from filename signals and hostnames:\n\n| Signal | System name |\n|--------|------------|\n| `bench_df*.log`, hostname contains `dragonflydb.cloud` | Dragonfly |\n| `bench_ec*.log`, hostname contains `cache.amazonaws.com` | ElastiCache |\n| `bench_valkey*.log` | Valkey |\n| `bench_redis*.log` | Redis |\n| `bench_local*.log`, `127.0.0.1` | Local (note version if available) |\n\nUse the actual product name, not the hostname. Do not include hostnames or endpoints in the report — they are not relevant for reproducing the results and may change.\n\n## Step 3: Generate charts\n\nIf JSON files are present, generate latency charts with `tools/plot_memtier_latency.py`:\n\n```bash\nmkdir -p <data_dir>/charts\npython3 <repo_root>/tools/plot_memtier_latency.py <json_file> <data_dir>/charts/<name>.html\n```\n\nRun once per JSON file. If the script is not found or fails, leave a `[TODO: chart]` stub. The script skips operations with no traffic (zero-latency entries), so it handles write-only fill JSONs and read-only access JSONs without errors.\n\nTo find `tools/plot_memtier_latency.py`, search from the current working directory: `find . -name \"plot_memtier_latency.py\" -not -path \"*/build*\"`.\n\n## Step 4: Write the report\n\nSave to `<data_dir>/report.md`. Use the template below. Populate every value you found; use `[TODO]` for anything missing (instance type, memory, pricing, scenario context the user didn't provide).\n\n---\n\n```markdown\n# <Descriptive title: what was tested, e.g., \"Dragonfly SSD Tiering vs ElastiCache: 50 GB Read Workload\">\n\n<1–3 sentence intro: what was measured and why. No adjectives. Example: \"This benchmark measures read throughput and latency for a 50 GB key-value dataset under a Gaussian access pattern. Dragonfly was configured with SSD tiering; ElastiCache r7g.4xlarge was used with all data in memory.\">\n\n## Test Environment\n\n| | Dragonfly | ElastiCache |\n|---|---|---|\n| Version / tier | [TODO] | [TODO] |\n| Instance type | [TODO] | [TODO] |\n| Memory | [TODO] | [TODO] |\n| Storage | [TODO: SSD tiering?] | [TODO] |\n| Region | [TODO] | [TODO] |\n\n<!-- Do NOT include hostnames, endpoints, or internal URLs — they are irrelevant to reproducing the test. -->\n\n## Dataset\n\n| Parameter | Value |\n|---|---|\n| Key count | <from KEY_MAXIMUM, formatted with commas> |\n| Value size | <DATA_SIZE> bytes |\n| Approximate dataset size | <KEY_MAXIMUM × DATA_SIZE, in GB> |\n| Key distribution | <Gaussian σ=X% of keys / uniform / Zipf exp=X> |\n\n## Benchmark Configuration\n\n| Parameter | Value |\n|---|---|\n| Tool | memtier_benchmark |\n| Threads | <THREADS> |\n| Connections per thread | <CLIENTS> |\n| Pipeline depth | <PIPELINE> |\n| Test duration (access phase) | <TEST_TIME>s |\n| Access ratio (SET:GET) | <ratio> |\n\n## Results\n\n### Fill Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p99 Latency (ms) | Duration |\n|---|---|---|---|---|\n| Dragonfly | | | | |\n| ElastiCache | | | | |\n\n### Read Access Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p50 (ms) | p99 (ms) | p99.9 (ms) |\n|---|---|---|---|---|---|\n| Dragonfly | | | | | |\n| ElastiCache | | | | | |\n\n## Latency Over Time\n\n<For each JSON file, embed the chart link or stub>\n\n**Dragonfly — read access (σ=2%)**\n![Dragonfly latency over time](./charts/df_access.html)\n\n**ElastiCache — read access (σ=2%)**\n[TODO: ElastiCache JSON not available; chart cannot be generated]\n\n## Observations\n\n<Factual, numbered. One observation per line. State what the data shows; do not say whether it is good or bad.>\n\n1. Throughput: Dragonfly achieved X ops/sec vs Y for ElastiCache — a Z% difference.\n2. Average latency: X ms (Dragonfly) vs Y ms (ElastiCache).\n3. p99 latency: X ms vs Y ms. <If there's a large gap relative to p50, note it.>\n4. Fill phase: <note fill throughput and duration for each system>.\n5. <Note any caveats: e.g., \"This run covers only 100% read traffic. Mixed read/write was not executed.\">\n6. <Note anything unusual in the time-series if visible from the data.>\n```\n\n---\n\n## Writing rules\n\n- Use exact numbers from the data. No rounding unless the value is very large (round to 3 significant figures max).\n- Percentages for comparisons: compute `(A - B) / B * 100`; write as \"X% higher\" or \"X% lower\".\n- Prohibited phrases: \"blazing fast\", \"industry-leading\", \"dramatically\", \"significantly\", \"next-level\", \"game-changing\", \"unparalleled\", \"impressive\"\n- No narrative hooks: no \"but wait\", \"what happens next\", \"the results speak for themselves\", \"as you can see\"\n- Observations may include inferences (e.g., \"the Gaussian σ=2% means the hot working set is ~3.5M keys, likely fitting in memory\") — label them clearly as inferences if they go beyond what the data directly shows\n- Bold labels in observations (e.g., `**Read throughput**:`) — keep them for scannability\n- Do not include hostnames, endpoints, or internal URLs anywhere in the report\n- Missing context → `[TODO: ...]` not a guess\n- Single-system data: omit comparison columns; keep observations factual about absolute numbers\n\n## Output\n\nPrint the report path when done: `Report written to: <path>/report.md`\n\nIf charts were generated, list them too.\n","frontmatter":{"name":"benchmark-report","description":"Generate a technical benchmark report in markdown from raw memtier/redis-benchmark output. Use this whenever the user provides benchmark results, log files, memtier JSON output, or a directory of benchmark data and wants a written report or comparison document. Trigger even when the user says \"write up these results\", \"make a report from this data\", \"turn this into a blog post\", or just drops a directory path. Always use this skill when the user provides benchmark files and wants a human-readable output. Reports compare Dragonfly against another system (ElastiCache, Redis, Valkey) but can handle single-system data too.","version":"1.0.0"},"isInternal":false,"tokens":2269,"sizeBytes":8805},{"name":"evals.json","path":".claude/skills/benchmark/evals/evals.json","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/benchmark/evals/evals.json","title":"Evals Skill","category":"anthropic-skill","format":"json","content":"{\n \"skill_name\": \"benchmark\",\n \"evals\": [\n {\n \"id\": 0,\n \"name\": \"full-report-vs-valkey\",\n \"prompt\":\n \"I just finished a benchmark run comparing Dragonfly against Valkey 9.0 on AWS. The raw artifacts are in the directory I'm pointing you at: dfly_bench JSON output (write and read phases) and INFO memory dumps for both datastores on m7g.2xlarge and m7g.4xlarge, plus a RUN_NOTES.txt with the hardware, exact commands, and DBSIZE after each write phase. Generate the benchmark report from this data, with charts and a raw-data appendix.\",\n \"expected_output\":\n \"A markdown report mirroring the team format: conditions section, a Simple Strings use-case section with a TLDR, per-instance (m7g.2xlarge, m7g.4xlarge) QPS/memory and latency tables, embedded QPS-over-time charts, and an appendix with exact commands, environment, and raw output. QPS/P99 must match the data; bytes-per-entry reported both logical (~127 DF / ~149-150 Valkey) and RSS.\",\n \"files\": [\n \"fixtures/\"\n ]\n },\n {\n \"id\": 1,\n \"name\": \"memory-focus-single-datastore\",\n \"prompt\":\n \"Here's raw output from a Dragonfly-only run on m7g.4xlarge (dfly_bench write/read JSON + INFO memory dump + run notes with DBSIZE). I care mostly about memory efficiency. Write me a short benchmark report focused on memory usage per entry, plus the throughput numbers.\",\n \"expected_output\":\n \"A report covering only Dragonfly (no fabricated Valkey/Redis comparison column), reporting both logical (used_memory/DBSIZE ~127) and RSS bytes-per-entry, plus write/read QPS and P99. Honest about it being single-datastore.\",\n \"files\": [\n \"fixtures/df_4xl_write.json\",\n \"fixtures/df_4xl_read.json\",\n \"fixtures/df_4xl_info_memory.txt\",\n \"fixtures/RUN_NOTES.txt\"\n ]\n },\n {\n \"id\": 2,\n \"name\": \"kickoff-planning-interactive\",\n \"prompt\":\n \"I've got two m7g.4xlarge boxes in AWS. Benchmark Dragonfly against Valkey on a simple strings workload.\",\n \"expected_output\":\n \"The skill should engage its interactive front-end: ask which box is the server vs the client (load generator must be separate), confirm SSH access, plan to run servers on port 6380, plan to start the monitoring stack (and fix the prometheus scrape port to 6380), and lay out the write/read workload plan. It must NOT invent benchmark numbers it hasn't measured.\",\n \"files\": []\n }\n ]\n}\n","frontmatter":{"skill_name":"benchmark","evals":[{"id":0,"name":"full-report-vs-valkey","prompt":"I just finished a benchmark run comparing Dragonfly against Valkey 9.0 on AWS. The raw artifacts are in the directory I'm pointing you at: dfly_bench JSON output (write and read phases) and INFO memory dumps for both datastores on m7g.2xlarge and m7g.4xlarge, plus a RUN_NOTES.txt with the hardware, exact commands, and DBSIZE after each write phase. Generate the benchmark report from this data, with charts and a raw-data appendix.","expected_output":"A markdown report mirroring the team format: conditions section, a Simple Strings use-case section with a TLDR, per-instance (m7g.2xlarge, m7g.4xlarge) QPS/memory and latency tables, embedded QPS-over-time charts, and an appendix with exact commands, environment, and raw output. QPS/P99 must match the data; bytes-per-entry reported both logical (~127 DF / ~149-150 Valkey) and RSS.","files":["fixtures/"]},{"id":1,"name":"memory-focus-single-datastore","prompt":"Here's raw output from a Dragonfly-only run on m7g.4xlarge (dfly_bench write/read JSON + INFO memory dump + run notes with DBSIZE). I care mostly about memory efficiency. Write me a short benchmark report focused on memory usage per entry, plus the throughput numbers.","expected_output":"A report covering only Dragonfly (no fabricated Valkey/Redis comparison column), reporting both logical (used_memory/DBSIZE ~127) and RSS bytes-per-entry, plus write/read QPS and P99. Honest about it being single-datastore.","files":["fixtures/df_4xl_write.json","fixtures/df_4xl_read.json","fixtures/df_4xl_info_memory.txt","fixtures/RUN_NOTES.txt"]},{"id":2,"name":"kickoff-planning-interactive","prompt":"I've got two m7g.4xlarge boxes in AWS. Benchmark Dragonfly against Valkey on a simple strings workload.","expected_output":"The skill should engage its interactive front-end: ask which box is the server vs the client (load generator must be separate), confirm SSH access, plan to run servers on port 6380, plan to start the monitoring stack (and fix the prometheus scrape port to 6380), and lay out the write/read workload plan. It must NOT invent benchmark numbers it hasn't measured.","files":[]}]},"isInternal":false,"tokens":612,"sizeBytes":2482},{"name":"report-template.md","path":".claude/skills/benchmark/references/report-template.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/benchmark/references/report-template.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Report template\n\nThe report mirrors the team's published benchmark format. Structure matters: a\nreader scans the conditions, jumps to the TLDR for the takeaways, checks the\nper-instance tables for numbers, and uses the appendix to reproduce. Follow this\nshape.\n\nUse Markdown tables. Center-align numeric columns (`:---:`). Bold the winning\ndatastore in comparison tables when there's a clear winner.\n\n## Structure\n\n```markdown\n# <Title> — e.g. \"Comparing Dragonfly vs Valkey on AWS\"\n\n<1–2 sentences: what was compared, on what hardware, what workloads.>\n\n## Conditions\n\nDragonfly launch command (identical across all configs):\n\n ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n\n<Comparison target> launch command:\n\n docker run --network=host --rm valkey/valkey:9.0 --save \"\" --appendonly no \\\n --io-threads 8 --protected-mode no --port 6380\n\nInstances: <server types + CPU counts>, client on <client instance type>.\nOS: <distro + kernel>. Network tuning: <applied / none — one line on what>.\n\n## <Use-case name> — e.g. \"Simple Strings\"\n\n<1–2 sentences describing the workload and why it matters.>\n\nWrite command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 --ratio 1:0 --json_out_file df_write.json\n\nRead command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN --ratio 0:1\n\n### TLDR\n\n* **<Datastore>** <the genuinely important takeaway — scaling, efficiency, a surprise>.\n* <3–5 bullets total. Interpretation, not a restatement of the tables.>\n\n## <Instance type> — e.g. \"m7g.2xlarge\"\n\n<Optional: one sentence on anything notable, e.g. throughput stability over time.>\n\n![Write throughput over time](charts/write_m7g2xl.png)\n\n### QPS and memory\n\n| Metric | Dragonfly | Valkey |\n| ----- | :---: | :---: |\n| Write QPS (k) | 816 | 548 |\n| Read QPS (k) | 848 | 811 |\n| Memory / entry — logical (bytes) | 127 | 149 |\n| Memory / entry — RSS (bytes) | 138 | 162 |\n\n### Latency\n\n| Backend | Write P99 (us) | Read P99 (us) |\n| :---: | :---: | :---: |\n| **Dragonfly** | 787 | 887 |\n| **Valkey** | 3283 | 951 |\n\n<Repeat the per-instance block for each instance type tested.>\n\n## Appendix — raw data\n\n### Environment\n\n| Property | Server | Client |\n| --- | --- | --- |\n| Instance type | m7g.2xlarge | c7gn.4xlarge |\n| vCPUs | 8 | 16 |\n| OS / kernel | Ubuntu 24.04.2 / 6.8.0 | ... |\n| Dragonfly version | <git hash / release> | — |\n| Network tuning | IRQ SMP affinity spread across CPUs | — |\n\n### Exact commands\n\n<Every command line actually run, per phase and per datastore, verbatim —\nincluding the launch commands, write/read load commands, and any tuning steps.>\n\n### Raw output\n\n<Per run: the final dfly_bench/memtier summary + latency histogram, and a\nreference to the saved JSON file (df_write.json, etc.). Include INFO memory\nsnapshots used for the per-entry figures.>\n```\n\n## Notes on writing the TLDR\n\nThe TLDR is the part people actually read. Make each bullet a claim the data\nsupports, phrased as a takeaway:\n\n- Good: \"Dragonfly scales vertically with cores and sustains up to 10x the\n throughput of a single Valkey process on large instances.\"\n- Weak: \"Dragonfly had higher QPS.\" (just restates the table)\n\nCall out hardware surprises honestly — e.g. an instance type that didn't scale\nlinearly — because they affect how a reader should interpret the numbers. If a\nrun was client-bound or otherwise caveated, say so here rather than burying it.\n\n## Charts\n\nOne chart per instance per phase is usually enough: QPS-over-time, with one line\nper datastore. The visual story is throughput *stability* — Dragonfly holding a\nflat line while a competitor degrades as its table grows is more convincing than\na single aggregate number. Generate with `scripts/make_charts.py` from the load\ngenerator's time-series JSON, save under a `charts/` dir next to the report, and\nembed with relative paths.\n\nWhen the monitoring stack ran (Phase 4), the Grafana dashboards add server-side\npanels that the load generator can't show — memory growth over the write phase,\nper-shard CPU, host network saturation. Crop the Grafana time range to the run\nwindow and pull the relevant panels (rendered PNG or screenshot) into the\nper-instance section. These are especially good for the memory-efficiency story\nand for proving the server (not the client) was the bottleneck. Note in the\nappendix that the data came from the monitoring stack, and which scrape port was\nused.\n","isInternal":false,"tokens":1192,"sizeBytes":4509},{"name":"running-benchmarks.md","path":".claude/skills/benchmark/references/running-benchmarks.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/benchmark/references/running-benchmarks.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Running benchmarks — operational reference\n\nHow to prepare instances, start servers, drive the load generators, and measure\nmemory. Read the section you need; you don't have to read top-to-bottom.\n\n## Table of contents\n\n- [Instance preparation](#instance-preparation)\n- [Starting Dragonfly](#starting-dragonfly)\n- [Starting Valkey / Redis for comparison](#starting-valkey--redis-for-comparison)\n- [dfly_bench (default load generator)](#dfly_bench-default-load-generator)\n- [memtier_benchmark (alternative)](#memtier_benchmark-alternative)\n- [The standard strings workload](#the-standard-strings-workload)\n- [Measuring memory](#measuring-memory)\n- [Monitoring stack (Prometheus + Grafana)](#monitoring-stack-prometheus--grafana)\n- [Network tuning](#network-tuning)\n- [Sizing the load](#sizing-the-load)\n\n---\n\n## Instance preparation\n\nThe server and the load generator should live on **separate machines**. A\nclient co-located with the server competes for CPU and silently caps the\nnumbers — the report is then measuring the client, not the server.\n\nRemote instances usually already have the repo at `~/projects/dragonfly`. Check\nfirst; clone only if missing:\n\n```bash\nssh <host> 'test -d ~/projects/dragonfly && echo present || echo missing'\n# if missing:\nssh <host> 'git clone https://github.com/dragonflydb/dragonfly.git ~/projects/dragonfly'\n```\n\n**Look for prebuilt binaries first.** Before building or downloading, check the\nrepo's build output folders — the binary is often already there. Look in\n`build-opt/` and `build-release/` (and `build-dbg/` only as a last resort — debug\nbuilds give meaningless performance numbers):\n\n```bash\nssh <host> 'ls -la ~/projects/dragonfly/build-opt/{dragonfly,dragonfly-aarch64,dfly_bench} 2>/dev/null; \\\n ls -la ~/projects/dragonfly/build-release/{dragonfly,dfly_bench} 2>/dev/null'\n```\n\nA folder may hold several binaries — e.g. a locally-built `dragonfly` (dev/HEAD)\nalongside a downloaded release `dragonfly-aarch64`; check `--version` on each to\nknow which is which, since \"compare the prebuilt vs the release binary\" is a\ncommon request. The load generator (`dfly_bench`) lives in the same folder on the\nclient.\n\n**Build vs download** — if no suitable prebuilt binary exists, follow the user's\ninstruction. To build a release binary from source on the instance:\n\n```bash\nssh <host> 'cd ~/projects/dragonfly && ./helio/blaze.sh -release -DWITH_AWS=OFF -DWITH_GCP=OFF'\nssh <host> 'cd ~/projects/dragonfly/build-opt && ninja dragonfly dfly_bench'\n```\n\nThis yields `build-opt/dragonfly` (the server) and `build-opt/dfly_bench` (the\nload generator). Build `dfly_bench` on the **client** box and `dragonfly` on the\n**server** box. Always benchmark release builds — a debug build's numbers are\nmeaningless for performance claims.\n\nIf the user prefers a published release, download the tarball from GitHub\nreleases and extract it instead of building.\n\nSanity-check reachability from the client before a long run:\n\n```bash\nredis-cli -h <server-ip> -p 6380 ping # expect PONG\n```\n\nCollect environment facts for the report while you're on the box:\n\n```bash\nssh <host> 'nproc; uname -r; cat /etc/os-release | head -2'\n# cloud instance type, if on AWS:\nssh <host> 'curl -s http://169.254.169.254/latest/meta-data/instance-type'\n```\n\n---\n\n## Starting Dragonfly\n\nThe published comparisons run Dragonfly with a minimal, persistence-free config\nso the benchmark measures the data path, not disk:\n\n```bash\n./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n```\n\n- `--conn_use_incoming_cpu` — pin each connection's processing to the CPU that\n received it; improves locality under high connection counts.\n- `--dbfilename=` — disable snapshot loading/saving.\n\nLet Dragonfly use all cores by default (it auto-detects). Record the exact\nlaunch command for the appendix.\n\n**Start it so you can reliably stop it — capture the PID.** Launching under\n`tmux`/`nohup` is fine, but `tmux kill-session` does **not** reliably reap the\nserver, and a stale server is a trap: a \"fresh\" server silently fails to bind the\nport, your client reconnects to the *old* instance, and you measure stale data\n(and waste the old dataset's RAM). Always record the PID and kill by PID:\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/build-opt; \\\n nohup ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380 \\\n >/tmp/dfly.log 2>&1 & echo $! > /tmp/dfly.pid'\n# ... benchmark ...\n# Stop it and VERIFY it's gone before starting the next binary:\nssh <server> 'kill -9 $(cat /tmp/dfly.pid); sleep 3; \\\n pgrep -af dragonfly | grep -v \"pgrep\\|bash -c\" || echo \"stopped\"; \\\n redis-cli -p 6380 ping 2>&1 | head -1 # expect Connection refused'\n```\n\nBetween two binaries (e.g. prebuilt vs release), **fully stop and verify the\nfirst is dead and the port is free before starting the second** — otherwise the\nsecond never binds and you benchmark the first twice. Note that pattern-based\n`pkill -f` can miss a process whose cmdline is the relative `./dragonfly`; killing\nby the captured PID is the robust path.\n\n---\n\n## Starting Valkey / Redis for comparison\n\nRun via docker on the server box. Valkey does not scale past ~8 io-threads on\ncurrent hardware, so more than that rarely helps — note the thread count you\nused.\n\n```bash\ndocker run --network=host --rm valkey/valkey:9.0 \\\n --save \"\" --appendonly no --io-threads 8 --protected-mode no --port 6380\n```\n\nFor Redis, substitute the `redis` image and equivalent flags. Run the comparison\ntarget on the **same instance type** as Dragonfly, one at a time (not\nsimultaneously — they'd contend), so the comparison is apples-to-apples.\n\n---\n\n## dfly_bench (default load generator)\n\nSource: `src/server/dfly_bench.cc`. Key flags:\n\n| Flag | Meaning | Typical |\n| --- | --- | --- |\n| `-h` | server host | client→server IP |\n| `-p` | server port | `6380` |\n| `-c` | connections **per thread** | ~10; ≤25 max; 1–5 when pipelining |\n| `-n` | requests per connection | sizes the run (omit if `--test_time`) |\n| `--test_time` | run duration in seconds | e.g. `120` |\n| `--qps` | target QPS per connection; `0` = unthrottled | `0` for peak |\n| `-d` | value size in bytes | `64` for strings tests |\n| `--key_maximum` | size of key space | tune to fill memory |\n| `--key_dist` | `U` uniform, `N` normal, `Z` zipfian, `S` sequential | `U` default |\n| `--ratio` | set:get ratio | `1:0` writes, `0:1` reads |\n| `--command` | custom command template (`__key__`, `__data__`) | for non-string workloads |\n| `--pipeline` | pending requests per connection | `>1` for peak throughput |\n| `-P` | `memcache_text` for memcached protocol | empty for RESP |\n| `--json_out_file` | **memtier-compatible JSON with per-second time series** | always set it |\n\n`--json_out_file` only works with the built-in GET/SET workload (not with\n`--command`, `--noreply`, or `--connect_only`). For custom-command workloads you\nscrape the console summary instead.\n\n**Console output.** Periodic lines look like:\n\n```\n12s: 40.0% done, RPS(now/agg): 815000/812340, errs: 0, hitrate: 99.8%, clients: 200\n```\n\nand the final summary:\n\n```\nTotal time: 30s. Overall number of requests: 24370200, QPS: 812340, P99 lat: 787us\nLatency summary, all times are in usec:\n<histogram>\n```\n\nSave the full stdout — the histogram and final QPS go in the appendix; the\n`collect_metrics.py` script prefers the JSON but can fall back to this.\n\nAlso save the command lines themselves under the run output directory, e.g.\n`commands_audit.log`. Record setup, server launch, write/read load,\ncapture, diagnostics, and chart/report commands as they are run; stdout alone is\nnot raw data because it cannot reproduce the run. Initialize this file before\nthe first workload so planned script invocations are visible up front, then\nappend commands as they execute.\n\n**Use the skill scripts for the main run path.** The main benchmark phases should\nuse:\n\n```bash\n.claude/skills/benchmark/scripts/bench_server.sh start|start-docker|stop|stop-docker|status ...\n.claude/skills/benchmark/scripts/monitor_fill.sh ...\n.claude/skills/benchmark/scripts/capture_result.sh ...\n.claude/skills/benchmark/scripts/collect_metrics.py ...\n.claude/skills/benchmark/scripts/make_charts.py ...\n.claude/skills/benchmark/scripts/plot_fill.py ...\n```\n\nInline SSH is acceptable for discovery, package/setup checks, bounded\nforeground probes, and diagnostics. If you bypass a run-control script for a\nmain benchmark phase, record the reason in `commands_audit.log` and the appendix before\nrunning the manual equivalent.\n\n**Keep orchestration phase-separated.** Do not run a single local shell command\nthat starts a remote tmux benchmark, polls until completion, captures results,\nand starts the next phase. If the local command is interrupted, the remote tmux\nprocess can finish while the local capture logic is gone, leaving the run idle\nand uncaptured. Prefer bounded foreground `dfly_bench` commands for known-length\nruns. If you need tmux/nohup for a long fill, use separate steps: launch it,\nmonitor it, verify it has exited, then run the capture/read commands separately.\n\n**JSON structure** (`--json_out_file`):\n\n```json\n{\n \"configuration\": { \"server\": \"...\", \"port\": 6380, \"clients\": 200,\n \"threads\": 8, \"pipeline\": 1, \"ratio\": \"1:0\" },\n \"ALL STATS\": {\n \"Runtime\": { \"Total duration\": 30000 },\n \"Sets\": { \"Count\": ..., \"Average Latency\": ..., \"Percentile ...\": ...,\n \"Time-Serie\": { \"0\": {...}, \"1\": {...}, ... } },\n \"Gets\": { ... }\n }\n}\n```\n\nThe per-second `Time-Serie` buckets are what the charts plot.\n\n---\n\n## memtier_benchmark (alternative)\n\nIndustry-standard, available as a docker image (`redislabs/memtier_benchmark`).\nUse it when the user wants cross-validation against a tool Dragonfly doesn't\nship, or for the pipelined peak-throughput use-case. Example (from the repo's\nk8s job):\n\n```bash\nmemtier_benchmark -s <server> -p 6380 \\\n --pipeline=30 --key-maximum=100000 -c 10 -t 2 --test-time=120 \\\n --distinct-client-seed --hide-histogram --json-out-file memtier_out.json\n```\n\n`--json-out-file` produces the same shape `collect_metrics.py` and\n`make_charts.py` read.\n\n---\n\n## The standard strings workload\n\nThis reproduces the published \"Simple Strings\" comparison. Run write then read,\nwith a memory snapshot in between.\n\n**Write phase** (load the keyspace):\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 -n $N --ratio 1:0 --json_out_file df_write.json\n```\n\nThe canonical strings test uses the built-in SET path (`--ratio 1:0`) so it\nproduces the JSON/time series the charts need. If TTLs matter, swap in\n`--command \"setex __key__ 10000 __data__\"` — but note `--json_out_file` is then\nignored, so for SETEX save stdout and parse/chart the periodic `RPS(now/agg)`\nlines or `monitor_fill.sh` output instead.\n\n**Memory snapshot** (immediately after writes — see next section).\n\n**Read phase**:\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n -n $N --ratio 0:1 --json_out_file df_read.json\n```\n\nChoose `KMAX`, `N`, and `CONN` to (a) saturate the server and (b) load enough\ndata that instance memory is well utilized — see [Sizing the load](#sizing-the-load).\n\n---\n\n## Measuring memory\n\n\"Bytes per entry\" is the memory metric in the report. Capture it right after the\nwrite phase, before reads:\n\n```bash\nredis-cli -h <server> -p 6380 INFO memory > info_memory.txt\nredis-cli -h <server> -p 6380 DBSIZE\n```\n\nFrom `INFO memory` you need `used_memory` (logical) and `used_memory_rss`\n(physical resident set). Report **both** per-entry figures:\n\n- `used_memory / DBSIZE` — logical bytes per entry (matches the published\n \"Memory Per Entry\").\n- `used_memory_rss / DBSIZE` — physical bytes per entry, the real footprint\n including allocator overhead and fragmentation.\n\n`collect_metrics.py` computes both when given `--info` and `--dbsize`.\n\nFor Valkey/Redis the same `INFO memory` fields exist, so the comparison is\ndirect. Capture the snapshot at the **same keyspace size** for both, or the\nper-entry comparison is unfair.\n\n---\n\n## Monitoring stack (Prometheus + Grafana)\n\nLives in `tools/local/monitoring/` and runs on the **server** machine via docker\ncompose. It gives continuous, 1-second-resolution time series for the whole\nbenchmark window — server-side command rates and memory, plus host CPU/network\nfrom node-exporter — which is both a sanity check (was the server actually\nsaturated? did it swap?) and the source of the report's best charts.\n\n```bash\ncd ~/projects/dragonfly/tools/local/monitoring\ndocker compose up -d # Prometheus, Grafana, node-exporter\ndocker compose --profile redis up -d # also start redis-exporter for Valkey/Redis\n```\n\nEndpoints (on the server box): Prometheus `:9090`, Grafana `:3000`,\nnode-exporter `:9100`, redis-exporter `:9121`.\n\n**What gets scraped:**\n\n- `dragonfly` job — Dragonfly exposes Prometheus metrics directly at\n `/metrics` on its main port. The bundled `prometheus.yml` targets\n `host.docker.internal:6379`.\n- `node-exporter` job — host CPU, memory, network.\n- `redis-exporter` job (profile `redis`) — Valkey/Redis stats, also via\n `host.docker.internal:6379`.\n\n**Port — standardize on 6380.** Always run benchmarked servers (Dragonfly,\nValkey, Redis) on port `6380`. The shipped `prometheus.yml` targets `6379`, so\nbefore bringing the stack up, edit the `dragonfly` and `redis-exporter` jobs in\n`tools/local/monitoring/prometheus/prometheus.yml` to point at `:6380`:\n\n```bash\nsed -i 's/:6379/:6380/g' ~/projects/dragonfly/tools/local/monitoring/prometheus/prometheus.yml\n# if Prometheus is already up, pick up the change:\ndocker compose restart prometheus\n```\n\nThen confirm targets are `UP` at `http://<server>:9090/targets` before starting\nload — otherwise Prometheus silently collects nothing for the server.\n\n**Dashboards** are pre-provisioned in Grafana (`dragonfly.json`, `redis.json`,\n`node-exporter.json`) — no manual import. For the report, note the benchmark\nstart/end timestamps so you can crop the Grafana time range to the exact run,\nthen capture the relevant panels (throughput, memory, CPU) — via the provisioned\n`grafana-image-renderer` (the compose file includes a `renderer` service) or a\nscreenshot — and embed them. You can also query Prometheus directly over HTTP\n(`/api/v1/query_range`) if you want raw series for `make_charts.py`.\n\nLeave the stack up across all runs so one continuous window covers everything.\n\n## Network tuning\n\nOn cloud hardware, peak throughput is gated by how network-interrupt load is\nspread across CPUs. The published runs tuned **SMP affinity for IRQs** so the\nNIC's interrupts fan out across multiple cores instead of saturating one.\n\nThis is instance- and NIC-specific. The general approach: identify the NIC's IRQs\n(`/proc/interrupts`), then write CPU masks to `/proc/irq/<N>/smp_affinity` to\nspread them. Many setups use the kernel's `set_irq_affinity.sh` helper shipped\nwith the NIC driver. Because the exact steps depend on the instance, confirm the\napproach with the user if they want tuning, and **record the exact commands you\nran** — tuned and untuned numbers are not comparable, and the appendix must say\nwhich was used.\n\nIf you skip tuning, state that plainly in the report.\n\n---\n\n## Sizing the load\n\nTwo independent things to size: **how many keys** (sets the memory footprint)\nand **how long / how many requests** (sets steady-state and how full memory\ngets). Get these wrong in either direction and the benchmark is misleading.\n\n### The memory budget — the most important (and most dangerous) call\n\nA write benchmark must land in the **sweet spot between two failure modes**:\n\n- **Too few keys → wrong-but-safe.** Benchmarking a nearly-empty instance is\n not representative. A small dataset sits in CPU cache and hides the real cost\n of a large hash table (cache misses, allocator behavior, fragmentation). The\n QPS looks great and means nothing. *Using available memory is the whole\n point.*\n- **Too many keys → OOM or eviction.** Push too far and the run is ruined two\n different ways (see the two ceilings below). Either way the numbers are\n garbage.\n\n**There are TWO ceilings, not one. Stay under both.**\n\n1. **Dragonfly's `maxmemory`.** Dragonfly auto-sets `maxmemory` to a fraction of\n RAM (≈ 78–80% by default). When memory crosses it, Dragonfly **rejects or\n evicts** writes — the load generator reports *error responses* and the run is\n contaminated. **Check it before the run** (`redis-cli -p 6380 config get\n maxmemory`, or the `maxmemory:` line in `INFO memory`).\n\n **Rejections start with a margin *below* `maxmemory`, not at it.** During a\n write, dash-table resizes transiently spike memory, so Dragonfly begins\n rejecting while steady-state `used_memory` is still well under the cap —\n observed rejections at `used_memory` ≈ 90% of `maxmemory` and even lower at the\n moment of a big resize. So keep steady-state `used_memory` to **≤ ~80% of\n `maxmemory`** to leave room for these spikes. If you need to fill more, raise\n `--maxmemory` explicitly — but then ceiling #2 (host RAM) binds.\n\n **Zero errors is the pass/fail bar.** *Any* eviction/OOM error contaminates\n the experiment — it slows the run (rejection stalls inflate P99) and stops the\n keyspace from growing, so QPS, latency, and bytes-per-entry are all off. After\n every write, **verify the load generator reported zero errors**\n (`grep -i error <write.log>`; dfly_bench prints `Got N error responses!`). If\n N > 0, the run is **invalid** — discard it, cut `--key_maximum` (start with a\n ~10–15% reduction), and re-run until errors are zero. Don't report a run with\n any errors, and don't try to \"subtract\" them out.\n2. **Host physical RAM, measured by RSS.** The accurate measure of real memory\n pressure is **resident set size — `used_memory_rss`** (equivalently the\n process RSS). True OOM happens when RSS approaches host RAM. Do **not** panic\n over `free`'s `used`/`available` columns during a write: a high-rate write\n inflates **page cache** (clean, reclaimable) and dirty pages, which makes\n `available` dip alarmingly even though the kernel will reclaim it under\n pressure. On a 30 GiB box a 195M-key dataset showed `free` available dropping\n to ~10% while `used_memory_rss` was only ~21 GiB (71% of RAM) — the dataset\n was nowhere near OOM; the rest was reclaimable cache. **Track RSS, keep it\n below host RAM with a ≥5% margin**, and ignore page-cache noise.\n\n**So size against RSS.** Target the dataset's RSS up to your fill goal while\nkeeping a real margin, and stay under `maxmemory`:\n\n```\ntarget_rss = host_RAM * fill_fraction # e.g. 0.90 for \"fill 90%\", never > 0.95\nrss_per_entry = used_memory_rss / DBSIZE # from calibration (accurate)\nkey_for_host = target_rss / rss_per_entry\nkey_for_maxmem = 0.90 * maxmemory / used_memory_per_entry # avoid eviction\ntarget_keys = min(key_for_host, key_for_maxmem)\n--key_maximum = target_keys\n```\n\nThe usual binding constraint is **`maxmemory`**, since its default (~78% of RAM)\nis below a 90% fill target. **To genuinely fill 90% of RAM, raise `--maxmemory`\nexplicitly** (e.g. `--maxmemory 28gb` on a 30 GiB box) so Dragonfly doesn't evict\nbefore RSS reaches the target — then RSS, watched live, is the safety signal.\nFilling memory is the goal; the failure modes to avoid are eviction (used_memory\n> maxmemory) and RSS approaching host RAM.\n\n`*_per_entry` (≈ `value_size + 60–100 B` overhead for small strings; more for\nTTL'd keys and other types) is best **calibrated**, not guessed:\n\n1. Run a short pilot write inserting a known, modest number of distinct keys\n (e.g. `--key_maximum=5000000 --key_dist S` with enough requests to write them\n all).\n2. Read `used_memory` / `DBSIZE` → real `bytes_per_entry`. Note this slightly\n *under*-estimates at scale (fragmentation grows), so keep margin.\n3. Plug into the formula above for the full run.\n\nCalibrate **per datastore** — Dragonfly and Valkey have different per-entry\noverhead, which is exactly what the report measures. Size each so its dataset\nfills the same target fraction; comparing at the same `DBSIZE` keeps the\nbytes-per-entry comparison fair (Dragonfly will simply use less RAM at that\nkey count, which is the finding, not a sizing error).\n\n### How distinct keys actually accumulate\n\nWith uniform random keys over `--key_maximum`, distinct keys (and therefore\nmemory) climb fast at first, then asymptote toward `key_maximum` as repeats\nstart landing on existing keys. **Memory plateaus near `key_maximum` distinct\nentries** — writing *longer* past that point just overwrites existing keys and\ndoes **not** grow memory. That's the key safety property: **`--key_maximum`, not\nrun length, bounds the footprint.** Run length controls how *close* to that\nplateau you get and how long you hold steady state.\n\nSo to reliably fill to the target without OOM:\n\n- Set `--key_maximum` to the computed `target_keys`.\n- Make the write phase long enough to insert essentially all of them — total\n requests (`-c` × threads × `-n`, or a `--test_time` that covers it) should be\n a small multiple of `key_maximum` (~2–3×) so coverage is near-complete.\n- Use `--key_dist S` (sequential) if you want *exactly* `key_maximum` distinct\n keys with no repeats and a precise fill; uniform is fine when you just need to\n approach the plateau.\n\n### Watch memory live — guard on RSS\n\nNever run a long write blind. The accurate safety signal is **`used_memory_rss`**\n(resident set), not `free`'s `available` (page cache makes it dip transiently and\nis reclaimable). Poll RSS every few seconds and **abort the write if RSS exceeds\n~90–95% of host RAM**, and separately watch `used_memory` against `maxmemory` to\ncatch approaching eviction. Example guard loop:\n\nRun the guard as its own bounded step, not as part of a larger launch/poll/capture\nmega-command. After the guard reports `DONE`, immediately run a separate capture\nstep (`DBSIZE`, `INFO memory`, stdout log, and zero-error check), then start the\nread phase.\n\n```bash\nTOTAL=$(ssh <server> 'free -b | awk \"/^Mem:/{print \\$2}\"')\nfor i in $(seq 1 60); do\n sleep 8\n read DB RSS USED < <(ssh <server> 'redis-cli -p 6380 dbsize | tr -d \"\\r\"; \\\n redis-cli -p 6380 info memory | grep -E \"^used_memory_rss:|^used_memory:\" | cut -d: -f2 | tr -d \"\\r\"' | paste -sd\" \")\n pct=$(awk \"BEGIN{printf \\\"%.0f\\\", $RSS/$TOTAL*100}\")\n echo \"dbsize=$DB rss=$(awk \"BEGIN{printf \\\"%.1f\\\",$RSS/2^30}\")GiB (${pct}% of RAM)\"\n if [ \"$pct\" -ge 92 ]; then echo \"ABORT: RSS ≥92% of RAM\"; \\\n ssh <client> 'tmux kill-session -t bench'; break; fi\ndone\n```\n\nWays to read current memory:\n\n- **`INFO memory` → `used_memory_rss`** — the accurate measure; the guard's\n input. Also compare `used_memory` to `maxmemory` (eviction warning).\n- **`/metrics` endpoint (Dragonfly only)** — `curl -s\n http://<server>:6380/metrics | grep -i mem` gives Dragonfly's live RSS/used\n gauges without a redis client. **Valkey/Redis do not serve `/metrics`** — for\n those, use `INFO memory` directly or scrape via the redis-exporter (see the\n monitoring section). `INFO memory` works for all three datastores, so it's the\n portable choice for the guard.\n- **Monitoring stack** — Prometheus/Grafana plots these over time; the source of\n the memory-over-time chart. (`free`/node-exporter available is fine for the\n chart, just not the trigger.)\n\nIf RSS approaches the limit faster than expected, **stop the write early** —\nyou've already filled enough to measure; a partially-filled run with clean\nnumbers beats one that evicted or OOMed.\n\n### The other knobs\n\n- **Connections (`-c` is *per thread*):** keep `-c` modest — **never above 25\n connections per thread, usually around 10**; for pipelined runs drop it to\n **1–5 per thread** (each connection already has many requests in flight, so\n more connections just add contention without more throughput). Scale total\n load by adding **threads** (and, if needed, a second client box), not by\n cramming connections onto each thread. Raise the total until QPS stops rising;\n if it keeps climbing at your max, the client is the bottleneck — add threads or\n another client machine, not more `-c`.\n- **Duration / steady state:** beyond filling memory, hold at least 30–60s of\n steady traffic so the reported QPS/P99 reflect steady state, not warm-up.\n Watch the periodic RPS line stabilize before trusting the number.\n- **Pipeline:** Use `--pipeline 30` for all write/prefill phases — without it\n the client stalls on each reply and becomes the bottleneck before the server\n saturates. Use `--pipeline 1` (or omit) only for latency-honest read\n benchmarks, and label those numbers accordingly.\n\nAlways note in the report the key count, value size, resulting memory fill (%\nof instance RAM), and whether the run was client-bound — an under-driven server\nor a barely-filled instance both produce numbers that quietly mislead.\n","isInternal":false,"tokens":6568,"sizeBytes":25124},{"name":"SKILL.md","path":".claude/skills/benchmark/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/benchmark/SKILL.md","title":"benchmark","category":"anthropic-skill","format":"markdown","content":"---\nname: benchmark\ndescription: >\n Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote\n cloud instances, then produce a performance + memory report with charts and a\n raw-data appendix. Use this whenever the user runs /benchmark, or asks to\n benchmark / load-test / measure throughput, QPS, latency, or memory efficiency\n of Dragonfly or another Redis-compatible server, to compare Dragonfly vs\n Valkey or Redis, or to generate a benchmark report. Trigger it even when the\n user just describes the goal (\"see how fast Dragonfly is on this box\",\n \"compare memory per key vs Valkey\") without saying the word \"benchmark\".\nargument-hint: (optional) free-form spec, e.g. \"server=10.0.0.1 client=10.0.0.2 strings vs-valkey\"\nallowed-tools: Bash, Read, Write, Edit, AskUserQuestion\n---\n\n# Benchmark Dragonfly\n\nDrive a benchmark of Dragonfly (optionally side-by-side with Valkey or Redis) on\nlocal or remote instances, collect throughput / latency / memory numbers, and\nassemble a report that mirrors the team's published format: a narrative TLDR,\nper-instance metric tables, charts, and an appendix with the exact commands and\nraw output so anyone can reproduce it.\n\nThe whole point of the report is **trust**: a reader should be able to see the\nheadline numbers, understand the conditions they were measured under, and\nre-run the exact commands themselves. Everything below serves that goal.\n\n## When you start\n\nIf the user passed a free-form spec in `$ARGUMENTS`, parse what you can from it\n(IPs, use-cases, comparison targets) and only ask about what's missing. The work\nsplits into clear phases — walk through them in order, but stay flexible: the\nuser may already have servers running, or may only want the report half (handed\nraw output to format). Read what they're actually asking for.\n\nPhases: (1) gather spec → (2) prepare instances → (3) network tuning →\n(4) start monitoring → (5) run workloads → (6) extract metrics, charts, report.\n\nThe two reference files carry the operational detail — read them before acting,\nnot from memory:\n\n- `references/running-benchmarks.md` — how to prep instances, start servers,\n drive `dfly_bench` and `memtier_benchmark`, run Valkey/Redis for comparison,\n apply network tuning, and measure memory. **Read this before Phase 2.**\n- `references/report-template.md` — the exact report structure, with an example.\n **Read this before Phase 5.**\n\nBundled `scripts/` (reuse these instead of re-improvising the same loops — they\nencode the safety discipline learned the hard way):\n\n- `bench_server.sh start|stop|status <ssh> <binary> [maxmem] [port]` — launch a\n server tracking its PID; **stop kills by PID and verifies the process is gone +\n port free** (tmux kill-session does not reap it).\n- `monitor_fill.sh <server_ssh> <client_ssh> [session] [abort_pct]` — the live\n RSS guard: polls `used_memory_rss`, prints a line per sample, aborts the write\n if RSS crosses the threshold (default 92%). Its output is also the input to\n `plot_fill.py`.\n- `capture_result.sh <server_ssh> <client_ssh> <log> <outdir> [label]` — saves\n the run summary + INFO memory + **OS-level `free` snapshot** (`host_memory.txt`)\n and exits non-zero if there were any eviction errors (enforces the zero-error\n rule). The OS snapshot matters: `used_memory_rss` and host free can diverge\n significantly — e.g. with `MIMALLOC_ALLOW_LARGE_OS_PAGES=1` (default), mimalloc\n holds large-page arenas that the OS counts as used but INFO doesn't fully reflect,\n making the machine appear near-OOM while RSS looks fine.\n- `collect_metrics.py` / `make_charts.py` — metrics + charts from dfly_bench\n JSON (GET/SET workloads).\n- `plot_fill.py` — memory-fill chart from `monitor_fill.sh` output, for\n `--command` workloads (e.g. SETEX) that produce no JSON time series.\n\nEvery benchmark output directory must include a `commands_audit.log`\nartifact with the exact command lines used for setup, server launch, writes,\nreads, captures, diagnostics, and chart/report generation. Raw data without the\ncommands that produced it is not reproducible.\n\n**Run-control scripts are mandatory for benchmark phases.** For Phase 5 runs,\nuse the bundled scripts as the default API:\n\n- start/stop/status servers with `scripts/bench_server.sh`;\n- monitor long write fills with `scripts/monitor_fill.sh`;\n- capture write results and enforce the zero-error rule with\n `scripts/capture_result.sh`;\n- generate metrics/charts with `scripts/collect_metrics.py`,\n `scripts/make_charts.py`, and `scripts/plot_fill.py`.\n\nInline SSH is fine for inspection, package setup, one-off diagnostics, and\nbounded foreground probes, but not as a replacement for the run-control scripts\nin the main benchmark path. If a script is not usable for a run, document the\nreason in `commands_audit.log` and in the report appendix before using a manual\nequivalent.\n\n## Phase 1 — Gather the benchmark spec (interactive)\n\nUse `AskUserQuestion` to pin down anything not already given. You need:\n\n1. **Where it runs.** Local, or remote instances? For remote, get the **server**\n host/IP and the **client** host/IP (the load generator should run on a\n separate machine from the server — co-locating them steals CPU and corrupts\n the numbers). Get the SSH user / key if not obvious. On cloud instances,\n **always use the private/internal IP** (VPC-internal) for both SSH and as the\n load generator's `-h` target — the public IP routes through the internet\n gateway, adding latency and egress cost, and the result measures the gateway\n rather than the server.\n2. **Use-cases.** Default menu: simple strings (SETEX writes + GET reads),\n pipelined / peak-throughput, memory efficiency. Always also ask whether there\n is a **specific focus** (a particular command, data type, key distribution,\n value size, or scenario) — the canned use-cases are a starting point, not a\n cage.\n3. **Comparison targets.** Dragonfly only, or also Valkey / Redis? Which\n versions?\n4. **Instance specs** (for the report's conditions section): instance type, CPU\n count, OS/kernel. Read these directly from the box at the start of Phase 2\n rather than asking the user.\n5. **Load parameters** if the user has opinions: value size (`-d`), key space\n (`--key_maximum`), connections (`-c`), pipeline depth, test duration. Pick\n sensible defaults (see the reference) and tell the user what you chose.\n **Sizing the key count is the consequential decision** — see the reference's\n \"Sizing the load\" section. The dataset must fill a large fraction of instance\n RAM (benchmarking a near-empty instance is wrong) but stay clear of OOM\n (~60–75% of RAM, headroom to spare). `--key_maximum` bounds the footprint, so\n compute it from instance RAM ÷ estimated bytes-per-entry, calibrating with a\n short pilot write if unsure.\n\nSummarize the resolved spec back to the user in a short block before running\nanything, so a wrong assumption gets caught before machines spin.\n\n## Phase 2 — Prepare the instances\n\n- Record instance specs (`nproc`, `uname -r`, cloud metadata) — these go in the\n report's conditions section. Use the **private IP** when connecting between\n instances (see Phase 1).\n- A remote instance usually already has `~/projects/dragonfly`. If not, clone it\n there. Build a **release** binary (`build-opt/dragonfly`, `build-opt/dfly_bench`)\n unless the user wants a published release downloaded instead — follow their\n instruction on build-vs-download.\n- For comparison targets, pull the Valkey/Redis docker image on the server box.\n- Verify everything launches and is reachable from the client box (a quick\n `redis-cli -h <server-private-ip> ping`) before committing to a long run.\n\nRun each step individually over SSH — do not chain launch, polling, capture, and\nthe next phase into one command. If a tool call is interrupted, remote work keeps\nrunning but goes uncaptured. Start the server; start the write; monitor it;\ncapture; start reads — one bounded step at a time.\n\n## Phase 3 — Network tuning (optional, remote only)\n\nHigh-throughput numbers on cloud hardware depend heavily on IRQ/SMP affinity\ntuning. If the user wants maximal numbers, apply the tuning described in the\nreference and **record exactly what you did** — it belongs in the appendix and\nmaterially affects the results. If skipped, say so in the report; untuned and\ntuned numbers are not comparable.\n\n## Phase 4 — Start the monitoring stack (server machine)\n\nStart the local Prometheus + Grafana stack on the **server** machine so you get\ncontinuous, fine-grained time-series (server CPU, memory, command rates, host\nmetrics) spanning the whole benchmark — richer than the load generator's own\noutput and the source of the most convincing report charts.\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose up -d'\n# when benchmarking Valkey/Redis, also bring up the redis exporter:\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose --profile redis up -d'\n```\n\nPrometheus scrapes at 1s. **Always run servers on port 6380** (Dragonfly,\nValkey, and Redis alike). The bundled `prometheus.yml` ships pointing at `6379`,\nso before bringing the stack up, fix the scrape targets:\n\n```bash\nsed -i 's/:6379/:6380/g' tools/local/monitoring/prometheus/prometheus.yml\n```\n\n(This is a workaround for a misconfigured default — if you have repo write\naccess, commit the corrected file once so future runs start clean.)\n\nRestart Prometheus if it was already running, then verify targets are `UP` at\n`http://<server>:9090/targets` before starting the load — a silently-down scrape\njob means no data for that window.\n\nStart monitoring **before** the workloads and leave it running across all runs,\nso a single continuous window covers everything. See the reference's monitoring\nsection for what to read from Grafana (dashboards are pre-provisioned) and how to\npull panels into the report. Record the benchmark start/end timestamps — they\nlet you crop the Grafana window to exactly the run.\n\n## Phase 5 — Run the workloads\n\nFor each (use-case × datastore) combination, see the reference for the precise\ncommands. The shape is always: **write → memory snapshot → read**. Use the\nprivate IP as the load generator's `-h` target.\n\nPass `--json_out_file` to `dfly_bench` (and `--json-out-file` to memtier) when\nthe workload supports it — charts and headline numbers come from these files.\n`dfly_bench --command` workloads (e.g. `SETEX __key__ 10000 __data__`) do not\nproduce JSON; save full stdout and chart the `monitor_fill.sh` output instead.\n\nDrive the run with the bundled scripts: `bench_server.sh start`, write in a\nclient tmux session, `monitor_fill.sh` alongside (live RSS guard), then\n`capture_result.sh` to save results. `capture_result.sh` enforces the\n**zero-error rule** — any eviction/OOM error response makes the run invalid;\ndiscard it, cut `--key_maximum` ~10–15%, and re-run. See the preamble and the\nreference's \"Sizing the load\" section for the full rule.\n\n**Prefill writes must use `--pipeline 30`.** Without pipelining, the client\nstalls waiting for each reply and becomes the bottleneck before the server is\nsaturated — especially at high key counts. Use `--pipeline 30` for all write\n(fill) phases. Drop to `--pipeline 1` (or omit it) only for latency-honest\nread benchmarks where you want honest per-request numbers.\n\nInitialize `commands_audit.log` at the start and append each command as it runs\n(see the preamble for the requirement).\n\n## Phase 6 — Extract metrics, build charts, assemble the report\n\n1. **Headline metrics + memory.** Run the bundled extractor over the JSON files\n to pull QPS / P99 and compute bytes-per-entry (both `used_memory`- and\n `used_memory_rss`-based, per the user's choice to report both):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/collect_metrics.py \\\n --bench-json <run.json> [--bench-json <run2.json> ...] \\\n --info <info_memory.txt> --dbsize <N> \\\n --label \"Dragonfly m7g.2xlarge\"\n ```\n\n It prints a metrics block (and JSON) you can drop straight into the tables.\n\n2. **Charts.** Generate QPS-over-time PNGs from the time series (this is what\n visually shows Dragonfly's smooth throughput vs a competitor's degradation):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/make_charts.py \\\n --series \"Dragonfly=df_write.json\" --series \"Valkey=valkey_write.json\" \\\n --metric qps --title \"Write throughput, m7g.2xlarge\" \\\n --out charts/write_m7g2xl.png\n ```\n\n Embed the PNGs in the report with relative paths. **For `--command` workloads\n (e.g. SETEX) there is no dfly_bench JSON** — use `plot_fill.py` instead to chart\n memory-fill over time from the `monitor_fill.sh` output:\n\n ```bash\n python3 .claude/skills/benchmark/scripts/plot_fill.py \\\n --series \"prebuilt=prebuilt/poll.txt\" --series \"release=release/poll.txt\" \\\n --interval 10 --ram-gib 30 --out charts/memory_fill.png\n ```\n\n3. **Report.** Follow `references/report-template.md` exactly. Write a TLDR with\n the 3–5 genuinely important takeaways (scaling behavior, memory efficiency,\n any hardware surprise), then per-instance tables, then the appendix. The TLDR\n is interpretation, not a restatement of the tables — say what the numbers\n *mean*. Only claim what the data supports.\n\nSave the report where the user asks (default: `benchmark-<YYYY>-<MM>.md` in the\nrepo root) and tell them the path plus a one-line summary of the headline\nresult.\n\n## Cleanup\n\nIf you started servers, the monitoring stack, or comparison containers on remote\ninstances, offer to stop them (`docker compose down` in the monitoring dir).\nIf you applied network tuning that changes machine state, tell the user what\nwould need reverting. Don't tear anything down without asking — they may want to\ninspect or re-run.\n","frontmatter":{"name":"benchmark","description":"Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote cloud instances, then produce a performance + memory report with charts and a raw-data appendix. Use this whenever the user runs /benchmark, or asks to benchmark / load-test / measure throughput, QPS, latency, or memory efficiency of Dragonfly or another Redis-compatible server, to compare Dragonfly vs Valkey or Redis, or to generate a benchmark report. Trigger it even when the user just describes the goal (\"see how fast Dragonfly is on this box\", \"compare memory per key vs Valkey\") without saying the word \"benchmark\".\n","argument-hint":"(optional) free-form spec, e.g. \"server=10.0.0.1 client=10.0.0.2 strings vs-valkey\"","allowed-tools":"Bash, Read, Write, Edit, AskUserQuestion","allowedTools":["Bash,","Read,","Write,","Edit,","AskUserQuestion"]},"allowedTools":["Bash,","Read,","Write,","Edit,","AskUserQuestion"],"isInternal":false,"tokens":3374,"sizeBytes":13930},{"name":"SKILL.md","path":".claude/skills/reproduce-fuzz-crash/SKILL.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/.claude/skills/reproduce-fuzz-crash/SKILL.md","title":"reproduce-fuzz-crash","category":"anthropic-skill","format":"markdown","content":"---\nname: reproduce-fuzz-crash\ndescription: >\n Reproduce AFL++ fuzz crashes from GitHub Actions. Use when user provides a\n GitHub Actions fuzz run URL and wants to reproduce and analyze the crash locally.\nargument-hint: <github-actions-run-url>\nallowed-tools: Bash, Read, Grep, Glob, Write\n---\n\n# Reproduce Fuzz Crash\n\nGiven a GitHub Actions fuzz run URL, download crash artifacts, triage them\nwith `fuzz/triage_crashes.sh`, and produce a crash analysis report.\n\n**Input**: `$ARGUMENTS` — a GitHub Actions run URL like:\n`https://github.com/dragonflydb/dragonfly/actions/runs/22906484769`\nor with query params like `?pr=6855`.\n\n## Workflow\n\n### Step 1: Parse the URL\n\nExtract `owner/repo` and `run_id` from the URL.\n\n```\nhttps://github.com/{owner}/{repo}/actions/runs/{run_id}[?...]\n```\n\nStrip any query parameters from `run_id`.\n\n### Step 2: Download artifacts\n\nList crash artifacts via the GitHub API, then download each as a `.zip` directly:\n\n**IMPORTANT**: Run each command as a separate Bash tool call (no `&&` chaining)\nto ensure auto-approval works with the user's permission patterns.\n\n```bash\n# List artifacts — filter for names containing \"crash\"\ngh api repos/{owner}/{repo}/actions/runs/{run_id}/artifacts\n\n# Create output directory\nmkdir -p /tmp/fuzz-repro-{run_id}\n\n# Download each crash artifact by ID (separate command)\ngh api repos/{owner}/{repo}/actions/artifacts/{artifact_id}/zip > /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nThis gives real `.zip` files that the triage script can consume directly.\n\nIf no crash artifacts are found, report that the run has no crash artifacts and stop.\n\nNote: there may be duplicate artifact names (same name, different IDs) from\nretried jobs. Download the **most recent** one (highest artifact ID).\n\n### Step 3: Determine mode\n\nInfer the protocol mode from the artifact name:\n- Contains \"memcache\" → `memcache`\n- Otherwise → `resp`\n\n### Step 4: Check Dragonfly binary\n\nCheck if the debug binary already exists and runs:\n\n```bash\n./build-dbg/dragonfly --version\n```\n\nOnly build if the binary doesn't exist or fails to run:\n\n```bash\nninja -C build-dbg dragonfly\n```\n\nIf `build-dbg` doesn't exist, run `./helio/blaze.sh` first.\n\n### Step 5: Run triage_crashes.sh\n\nFor each zip file, run:\n\n```bash\n./fuzz/triage_crashes.sh ./build-dbg/dragonfly <mode> /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nCapture the full output.\n\n### Step 6: Analyze and report\n\nParse the triage output for confirmed crashes. For each confirmed crash:\n\n1. **Read the source** at the crash location — use the stack trace to identify\n the source file and line number, then read that code.\n2. **Provide analysis**: likely root cause, what to investigate.\n\nPrint a structured report:\n\n```\n## Fuzz Crash Report\n\n**Run**: {url}\n**Artifacts**: {number} crash(es) found\n\n---\n\n### Crash NNNNNN\n\n**Reproduced**: Yes / No (false positive)\n**Signal**: SIGABRT (6) / SIGSEGV (11) / etc.\n\n**Stack trace**:\n\\```\n<stack trace from triage output>\n\\```\n\n**Analysis**:\n<1-3 sentences explaining the likely root cause based on the stack trace,\nthe assertion message, and the crash input. Identify the source file and\nline number. Suggest what to investigate.>\n```\n\n## Important Notes\n\n- The triage script uses port **6379** (resp) or **11211** (memcache).\n Ensure no other Dragonfly or Redis instance is using these ports.\n- The script adds `--restricted_commands` flags to avoid false positives from\n commands like DEBUG SLEEP that the fuzzer might generate.\n- Some crashes are non-deterministic (thread timing). The script reports\n these as \"FALSE POSITIVE\" — note this clearly, it doesn't mean the bug\n is invalid, just that it didn't reproduce on this run.\n- The script handles its own cleanup of Dragonfly processes.\n- Do NOT delete `/tmp/fuzz-repro-{run_id}/` — the user may want to inspect it.\n- If `gh run download` fails with permissions, suggest the user authenticate\n with `gh auth login`.\n","frontmatter":{"name":"reproduce-fuzz-crash","description":"Reproduce AFL++ fuzz crashes from GitHub Actions. Use when user provides a GitHub Actions fuzz run URL and wants to reproduce and analyze the crash locally.\n","argument-hint":"<github-actions-run-url>","allowed-tools":"Bash, Read, Grep, Glob, Write","allowedTools":["Bash,","Read,","Grep,","Glob,","Write"]},"allowedTools":["Bash,","Read,","Grep,","Glob,","Write"],"isInternal":false,"tokens":1009,"sizeBytes":3940},{"name":"README.md","path":"tools/packaging/osrepos/README.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/tools/packaging/osrepos/README.md","title":"osrepos Documentation","category":"plugin-manifest","format":"markdown","content":"# Package repositories for rpm and debian packages\n\nThis directory contains scripts and definitions for setting up YUM and apt repositories for Linux users to install\ndragonfly packages.\n\nThe repositories are served as static websites. The generate-site workflow is used to set up and deploy the sites using\nscripts and definitions included here.\n\nThe workflow does the following tasks:\n\n* Download the latest 5 releases from dragonfly releases page, specifically deb and rpm assets\n * for deb files, only the latest package is downloaded and present (see note below)\n* Set up a directory structure separating deb and rpm files into version specific paths\n* Sign the packages (see note on GPG)\n* Deploy the assets prepared, along with the public GPG key and repo definitions for apt and rpm tooling\n\n## Using the YUM repository\n\nAdd the repository using:\n\n```shell\nsudo dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo\n```\n\nThen install dragonfly as usual, or a specific version:\n\n```shell\nsudo dnf -y install dragonfly-0:v1.33.1-1.fc30.x86_64\n```\n\n## Using the APT repository\n\nFirst download the public GPG key to an appropriate location:\n\n```shell\nsudo curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public\n```\n\nThen add the sources file:\n\n```shell\nsudo curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources\n```\n\nFinally install dragonfly using apt\n\n```shell\nsudo apt update && sudo apt install dragonfly\n```\n\n#### Versions in APT repository\n\nUnlike the yum repo, the apt repo only has the latest version. The reason for this is the tool, `reprepro` supplied by\ndebian to build repositories only supports multiple\nversions in version 5.4 onwards, and the github runner using ubuntu-latest does not have this version.\n\nAnother option would be to use the components feature of apt repositories in the sources file we ask users to install,\nbut then the versions would need\nto be hardcoded in the sources file and the user would have\nto update the file with each new release which makes for a bad user experience. As of now users wanting older packages\nshould download them directly.\n\n### Signing packages\n\nThe packages are signed using the GPG key imported from the secret GPG_PRIVATE_KEY in this repository.\n\nThe corresponding public key is served with site assets, so the apt/yum/dnf based tooling can consume the public key to\nverify package integrity.\n\n### TODO\n\n- [X] debian packages signing (not required? release file is signed)\n- [X] debian repo metadata setup\n- [ ] tests asserting that packages are installable?\n","isInternal":false,"tokens":588,"sizeBytes":2659},{"name":"README.md","path":"tools/packaging/README.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/tools/packaging/README.md","title":"packaging Documentation","category":"plugin-manifest","format":"markdown","content":"# Installation Packages\n\n## Overview\nThis directory includes a set of files and scripts to build installation package for various Linux distributions.\n\n## Debian\nThe file to build the Debian package all located under \"debian\" directory.\nThe resulting package will install the binary of Dragonfly as well as generate a new service entry for dragonfly,\nthat can be controlled with \"systemctl\" command, to start, stop and check status of.\n### Building\nTo build the package, you have a script called \"generate_debian_package.sh\". This script accepts the following parameters:\n* Optional binary path - the location from which to take the binary for the installation. The default for this is \"repo path/build-opt\".\nThe location to which the resulting package is writing is at the location from which the script is executed.\nThis script is depends on the following packages:\n* git\n* moreutils\n* debhelper\n* dpkg-dev\n\nTo build:\n```\n/path/to/dragonfly/tools/packaging/generate_debian_package.sh [/path/to/dragonfly-binary-file]\n```\n\nThis can only be run on Debian based hosts.\nYou can use the flowing docker file to generate this package:\n```\nFROM ubuntu:20.04\nARG DEBIAN_FRONTEND=noninteractive\nRUN apt update -y && apt-get install -y gcc dpkg-dev gpg vim wget git moreutils debhelper\n```\nBuild the above docker and then run it with your dragonfly source code path mount as volume for the build:\n```\ndocker build -t ubuntu-package .\ndocker run --rm -ti -v /path/to/dragonfly-repo:/mydocker-path ubuntu-package bash\n```\nAgain note that you need to be at \"main\" branch to successfully build this package.\nNote: If at the end of the installation you see a message \"/usr/bin/deb-systemd-helper: error: systemctl preset failed on dragonfly.service: No such file or directory\",\nyou can ignore it, this seem to be related to [the following issue](https://groups.google.com/g/linux.debian.bugs.dist/c/m6xGZ82TdvM).\n","isInternal":false,"tokens":435,"sizeBytes":1899},{"name":"README.md","path":"tools/vector-benches/README.md","rawUrl":"https://raw.githubusercontent.com/dragonflydb/dragonfly/HEAD/tools/vector-benches/README.md","title":"vector-benches Documentation","category":"plugin-manifest","format":"markdown","content":"Tool for benchmarking vector search with randomized vectors.\n\n## Logic\nThe tool connects to the Redis/Dragonfly instance.\n1. Checks if the database has enough data (at least 50% of requested `-n`). If not, it **flushes the DB** and generates random vectors.\n2. Checks if the index `idx` exists. If not, it creates it.\n3. Runs concurrent search queries and reports latency/QPS.\n\n## Arguments\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `-n` | 50000 | **Number of vectors** to populate if the DB is empty. |\n| `-q` | 1000 | **Total number of queries** to run during the benchmark. |\n| `-t` | 8 | **Query threads**. Number of concurrent workers sending queries. |\n| `-d` | 100 | **Vector dimension**. Size of the float32 vectors. |\n| `-k` | 10 | **Top K**. Number of nearest neighbors to retrieve per query. |\n| `-p` | 6379 | **Port** of the server. |\n| `-h` | localhost | **Host** of the server. |\n\nRun with `-h` (help) to see these defaults in the tool itself.\n","isInternal":false,"tokens":265,"sizeBytes":985}],"systemPromptSnippet":"<yakaai_skills repo=\"dragonflydb/dragonfly\">\n<!-- File: AGENTS.md (Tokens: ~4580 | Category: root-instruction) -->\n# Dragonfly Development Guide\n\n> **Essential reference for working with the Dragonfly codebase**\n> Architecture, build system, testing infrastructure, and development workflows.\n\n---\n\n## Table of Contents\n\n1. [Critical Workflow Rules](#critical-workflow-rules)\n2. [Quick Command Reference](#quick-command-reference)\n3. [Project Overview](#project-overview)\n4. [Repository Structure](#repository-structure)\n5. [Build Instructions](#build-instructions)\n6. [Testing](#testing)\n7. [CI/CD Pipeline](#cicd-pipeline)\n8. [Code Style & Pre-commit Hooks](#code-style--pre-commit-hooks)\n9. [Third-Party Dependencies](#third-party-dependencies)\n10. [Platform Support](#platform-support)\n11. [CMake Build Options](#cmake-build-options)\n12. [Key Files Reference](#key-files-reference)\n13. [Common Pitfalls](#common-pitfalls)\n14. [Debugging Tips](#debugging-tips)\n15. [Validation Checklist](#validation-checklist)\n\n---\n\n## Critical Workflow Rules\n\n**MANDATORY - Always Follow This Order:**\n\n1. ✅ **Read Before Edit** - Always read files before modifying\n2. ✅ **Use Correct Build Commands** - See [Quick Command Reference](#quick-command-reference) below\n3. ✅ **Test After Changes** - Build and run a relevant unit test -\n `ninja <unit_test> && ./unit_test`\n4. ✅ **Format Code** - `pre-commit run --files <files>`\n5. ✅ **Follow Architecture** - See [Architecture Patterns](#architecture-patterns) below\n6. ✅ **Never Push to Main** - Always create a feature branch and open a PR. Never run `git push origin main`.\n\n### Pull Request Guidelines\n\n**Conciseness is Key**: PR descriptions should be short, focused, and easy to scan.\n- **Title**: Imperative, descriptive (e.g., \"Fix fiber stack overflow in test_reply_guard_oom\")\n- **Summary**: 1-2 sentences explaining *what* changed and *why*\n- **Changes**: Bullet points for key changes\n- **Fixes**: Link issues (e.g., \"Fixes #123\")\n- **Commit messages**: Keep every line (subject and body) <= 100 characters; wrap long descriptions\n\n---\n\n## Quick Command Reference\n\n**CRITICAL: Read the full sections below for context. These are shortcuts only.**\n\n### Building (see [Build Instructions](#build-instructions) for details)\n\n```bash\n# Debug build (for development)\n./helio/blaze.sh -DWITH_GCP=OFF\ncd build-dbg && ninja dragonfly # Build main binary\ncd build-dbg && ninja generic_family_test # Build specific test\n\n# Release build for local benchmarking\n./helio/blaze.sh -release -DWITH_GCP=OFF\ncd build-opt && ninja dragonfly\n```\n\n### Testing (see [Testing](#testing) for details)\n\n```bash\n# C++ Unit Tests\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n### Code Formatting\n\n```bash\n# Setup (once)\npipx install pre-commit clang-format black\npre-commit install\n\n# Format code\npre-commit run --files <files> # Format specific files\npre-commit run --all-files # Format all files\n```\n\n### Common Operations\n\n```bash\n# Check git status\ngit status\n\n# Check current branch\ngit branch\n\n# View recent commits\ngit log --oneline -10\n```\n\n---\n\n## Architecture Patterns\n\n**Code Style**: [.clang-format](.clang-format) - snake_case vars, PascalCase functions, kPascalCase constants\n\n**Comments**: Always put an empty line before a comment in code, unless the comment is the first\nline of a block (right after `{`). This applies to comments between statements, before if/for, and\ninside lambdas.\n\n**DO ✅**:\n- Function parameters: inputs first, output arguments (pointers the function writes to) last\n- Fiber-aware: `util::fb2::Mutex`, `util::fb2::Fiber` → [helio/util/fibers/](helio/util/fibers/)\n- Per-shard ops (no global state) → [docs/df-share-nothing.md](docs/df-share-nothing.md)\n- Command pattern → [src/server/set_family.cc](src/server/set_family.cc)\n- Error handling: `OpStatus` → [src/server/common.h](src/server/common.h)\n- Test patterns → [tests/dragonfly/conftest.py](tests/dragonfly/conftest.py)\n\n**DON'T ❌**:\n- `std::thread`, `std::mutex` (deadlocks!)\n- Global mutable state\n- Edit without reading\n- Skip tests\n- Use `std::regex` in fiber/server paths (recursive implementation can overflow small fiber stacks)\n- Use `./tools/docker/build.sh` for local development (use `ninja` instead)\n- Use `make` for incremental builds (use `ninja` instead)\n\n---\n\n## Project Overview\n\n**Dragonfly** is a high-performance, Redis and Memcached compatible in-memory data store written in C++20. It delivers significantly higher throughput than traditional single-threaded Redis implementations through innovative architectural choices.\n\n### Key Characteristics\n\n- **Language**: C++20 (Google C++ Style Guide 2020 version)\n- **Architecture**: Shared-nothing multi-threaded design (via `helio` library)\n- **Performance**: Uses io_uring (Linux 5.11+) for high-performance async I/O, with epoll fallback\n- **Threading Model**: Fiber-based cooperative multitasking with lock-free data structures\n- **Build System**: CMake + Ninja via `helio/blaze.sh` wrapper script\n- **Target Platform**: Linux (kernel 5.11+ recommended), FreeBSD support available\n- **Protocols**: Redis RESP2/RESP3, Memcached binary protocol\n- **Compatibility**: Drop-in replacement for Redis API coverage\n\n### Architectural Highlights\n\n**For detailed architecture documentation, see [docs/df-share-nothing.md](docs/df-share-nothing.md)**\n\n1. **Shared-Nothing Design**: Each thread operates independently with its own data structures, minimizing lock contention\n2. **Helio Framework**: Custom I/O and threading library built on io_uring/epoll with fiber support\n3. **DashTable**: Novel hash table implementation optimized for multi-core systems - see [docs/dashtable.md](docs/dashtable.md)\n4. **Transaction Model**: Non-blocking optimistic transactions - see [docs/transaction.md](docs/transaction.md)\n5. **Tiering Support**: Optional disk-backed storage for large datasets\n6. **Search Module**: Full-text search capabilities (when enabled with WITH_SEARCH)\n\n---\n\n## Repository Structure\n\n```\ndragonfly/\n├── src/ # Main C++ source code\n│ ├── server/ # Core server implementation\n│ │ ├── dfly_main.cc # Main entry point\n│ │ ├── main_service.cc # Service lifecycle & command routing\n│ │ ├── db_slice.cc # Per-thread database shard\n│ │ ├── engine_shard_set.cc # Shard management\n│ │ ├── cluster/ # Cluster mode implementation\n│ │ ├── journal/ # Replication journal\n│ │ ├── tiering/ # Tiered storage\n│ │ ├── search/ # Search module\n│ │ └── acl/ # Access control lists\n│ ├── core/ # Core data structures\n│ │ ├── dash.h # DashTable hash table\n│ │ ├── dense_set.h # Compact set implementation\n│ │ ├── string_map.h # Optimized string-keyed maps\n│ │ ├── search/ # Search core algorithms\n│ │ └── json/ # JSON support\n│ ├── facade/ # Network & command handling\n│ │ ├── dragonfly_connection.cc # Connection management\n│ │ ├── resp_srv_parser.cc # RESP command parser\n│ │ ├── resp_parser.cc # RESP reply parser\n│ │ └── memcache_parser.cc # Memcached protocol\n│ └── redis/ # Redis-specific implementations\n│ └── lua/ # Lua scripting support\n│\n├── helio/ # Git submodule: I/O and threading library\n│ │ # ** DO NOT EDIT unless contributing to helio **\n│ ├── util/ # Utilities: fibers, I/O, synchronization\n│ ├── io/ # io_uring & epoll abstraction\n│ └── blaze.sh # Build configuration wrapper\n│\n├── tests/ # Test suite\n│ ├── dragonfly/ # Python pytest integration/regression tests\n│ │ ├── conftest.py # Pytest fixtures & configuration\n│ │ ├── requirements.txt # Python test dependencies\n│ │ └── *.py # Test files\n│ └── pytest.ini # Pytest configuration & markers\n│\n├── docs/ # Documentation\n│ ├── build-from-source.md # Build instructions\n│ ├── dashtable.md # DashTable internals\n│ ├── transaction.md # Transaction model\n│ ├── df-share-nothing.md # Shared-nothing architecture\n│ └── differences.md # Differences from Redis\n│\n├── contrib/ # Utilities\n│ ├── docker/ # Docker configurations\n│ └── charts/dragonfly/ # Helm chart for Kubernetes\n│\n├── tools/ # Benchmarking & utility tools\n│ └── packaging/ # Packaging scripts\n│\n├── CMakeLists.txt # Root CMake configuration\n├── .clang-format # C++ formatting rules (clang-format v14.0.6)\n├── .pre-commit-config.yaml # Pre-commit hooks configuration\n├── pyproject.toml # Python formatting (Black, 100 chars)\n└── CONTRIBUTING.md # Contribution guidelines\n```\n\n### Critical Paths to Remember\n\n- **Main entry**: `src/server/dfly_main.cc`\n- **Command dispatch**: `src/server/main_service.cc`\n- **Data storage**: `src/server/db_slice.cc`\n- **Networking**: `src/facade/dragonfly_connection.cc`\n- **Helio library**: `helio/` (I/O and threading library)\n\n---\n\n## Build Instructions\n\n**For complete build instructions, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Quick Start\n\n**Debug build** (for development):\n```bash\n./helio/blaze.sh\ncd build-dbg && ninja dragonfly\n./dragonfly --alsologtostderr\n```\n\n**Release build** (for production/benchmarking):\n```bash\n./helio/blaze.sh -release\ncd build-opt && ninja dragonfly\n```\n\n**Production release build** (static linking, optimized):\n```bash\nmake release # Configure + build\nmake package # Create release packages with debug symbols\n```\n\nThe [Makefile](Makefile) builds production releases with:\n- Static linking: libstdc++, libgcc, Boost, OpenSSL\n- Architecture optimizations (x86_64: `-march=core2 -msse4.1 -mtune=skylake`)\n- Debug symbols (compressed)\n- Output: `build-release/dragonfly-{arch}.tar.gz`\n\n**Common build options**:\n- See [docs/build-from-source.md](docs/build-from-source.md) for all options\n\n---\n\n## Testing\n\n**For complete testing documentation, see [tests/README.md](tests/README.md)**\n\n### Quick Reference\n\n**C++ Unit Tests**:\n```bash\ncd build-dbg\nctest -V -L DFLY # Run all tests\n./generic_family_test # Run specific test binary\n./generic_family_test --gtest_filter=\"Set.*\" # Run specific test case\n```\n\n**Python Integration Tests (pytest)**:\n```bash\n# Run from the repo root. The binary path defaults to build-dbg/dragonfly.\n# Override with the DRAGONFLY_PATH env var:\nDRAGONFLY_PATH=build-dbg/dragonfly python3 -m pytest tests/dragonfly/pymemcached_test.py -xvs\n\n# Run a single test:\npython3 -m pytest tests/dragonfly/pymemcached_test.py::TestMemcached::test_basic -xvs\n```\n\n- `DRAGONFLY_PATH` — sets the path to the Dragonfly binary the test harness starts. Defaults to `build-dbg/dragonfly` relative to the `tests/dragonfly/` directory.\n- `--df` — passes **extra flags to the Dragonfly process** (not the binary path). For example: `--df logtostdout --df \"vmodule=*=1\"`.\n\n---\n\n## CI/CD Pipeline\n\n**For complete CI configuration, see [.github/workflows/ci.yml](.github/workflows/ci.yml)**\n\nThe CI workflow runs on all PRs and includes:\n- **Pre-commit checks**: clang-format, black formatters\n- **Build matrix**: Multiple OS/compiler/sanitizer combinations (Ubuntu 20/24, Alpine, GCC/Clang, ASAN/UBSAN)\n- **Test execution**: C++ unit tests, Python integration tests, cluster mode tests\n- **Additional validations**: Helm charts, Docker image builds\n\n---\n\n## Code Style & Pre-commit Hooks\n\n**For complete contribution guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md)**\n\n**Code style configuration files**:\n- **C++**: [.clang-format](.clang-format) - Google C++ Style Guide (2020), clang-format v14.0.6, 100 char limit\n- **Python**: [pyproject.toml](pyproject.toml) - Black formatter, 100 char limit, PEP 8 compliant\n- **Pre-commit hooks**: [.pre-commit-config.yaml](.pre-commit-config.yaml) - Automated formatting checks\n\n**Quick setup**:\n```bash\npipx install pre-commit clang-format black\npre-commit install\npre-commit run --all-files # Run all formatters\n```\n\n---\n\n## Third-Party Dependencies\n\n**Key Libraries**: Abseil (strings/flags), Boost 1.71+ (context/intrusive), mimalloc (allocator), jsoncons (JSON), OpenSSL (TLS), libunwind (traces)\n\n**Build artifacts**: `build-dbg/third_party/` - DO NOT edit\n\n**For complete dependency info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## Platform Support\n\n**Linux**: Primary platform. Kernel 5.11+ (io_uring), 5.1+ (basic), < 5.1 (epoll fallback)\n- Check: `uname -r`\n- Force epoll: `--proactor_type=epoll`\n- Docker: `--security-opt seccomp=unconfined`\n\n**FreeBSD**: Supported (kqueue backend)\n\n**macOS**: Not supported for production (use Docker/Linux)\n\n**For complete platform info, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n---\n\n## CMake Build Options\n\n**For complete list of build options, see [docs/build-from-source.md](docs/build-from-source.md)**\n\n### Common Options\n\nPass options to `helio/blaze.sh` with `-D` prefix:\n\n```bash\n./helio/blaze.sh -DWITH_SEARCH=OFF -DWITH_GCP=OFF\n```\n\n**Most useful options**:\n- `WITH_ASAN=ON` / `WITH_USAN=ON` - Enable sanitizers for debugging\n- `WITH_SEARCH=OFF` - Disable search module for faster builds\n- `WITH_GCP=OFF` - Disable GCP cloud library\n- `WITH_TIERING=OFF` - Disable disk storage\n- `USE_MOLD=ON` - Faster linking with LTO (production builds)\n\n**Quick configurations**:\n```bash\n# Minimal build (fast compilation)\n./helio/blaze.sh -DWITH_GPERF=OFF -DWITH_GCP=OFF -DWITH_TIERING=OFF -DWITH_SEARCH=OFF\n\n# Full-featured (all options ON by default)\n./helio/blaze.sh\n\n# Production optimized\n./helio/blaze.sh -release -DUSE_MOLD=ON\n```\n\n---\n\n## Key Files Reference\n\nQuick reference to the most important files in the codebase.\n\n| Purpose | File Path |\n|---------|-----------|\n| **Entry Points & Core** | |\n| Main entry point | `src/server/dfly_main.cc` |\n| Server lifecycle & command routing | `src/server/main_service.cc` |\n| Per-thread database shard | `src/server/db_slice.cc` |\n| Shard management | `src/server/engine_shard_set.cc` |\n| **Data Structures** | |\n| DashTable hash table | `src/core/dash.h` |\n| Dense set implementation | `src/core/dense_set.h` |\n| String map | `src/core/string_map.h` |\n| **Networking** | |\n| Connection handling | `src/facade/dragonfly_connection.cc` |\n| RESP command parser | `src/facade/resp_srv_parser.cc` |\n| RESP reply parser | `src/facade/resp_parser.cc` |\n| Memcached protocol parser | `src/facade/memcache_parser.cc` |\n| **Build System** | |\n| Root CMake config | `CMakeLists.txt` |\n| Build script wrapper | `helio/blaze.sh` |\n| Server CMake config | `src/server/CMakeLists.txt` |\n| **CI/CD** | |\n| Main CI workflow | `.github/workflows/ci.yml` |\n| Pre-commit config | `.pre-commit-config.yaml` |\n| **Code Style** | |\n| C++ formatting | `.clang-format` |\n| Python formatting | `pyproject.toml` |\n| **Testing** | |\n| Pytest configuration | `tests/pytest.ini` |\n| Pytest fixtures | `tests/dragonfly/conftest.py` |\n| Test requirements | `tests/dragonfly/requirements.txt` |\n| **Documentation** | |\n| Build instructions | `docs/build-from-source.md` |\n| Architecture overview | `docs/df-share-nothing.md` |\n| DashTable internals | `docs/dashtable.md` |\n| Transaction model | `docs/transaction.md` |\n| **Configuration** | |\n| Contributing guide | `CONTRIBUTING.md` |\n| CLA agreement | `CLA.txt` |\n\n---\n\n## Common Pitfalls\n\n1. **Pre-commit not installed**: `pipx install pre-commit clang-format black && pre-commit install`\n2. **Wrong binary**: Debug: `build-dbg/dragonfly`, Release: `build-opt/dragonfly`\n3. **Wrong build command**: Use `cd build-dbg && ninja <target>`, NOT `./tools/docker/build.sh`\n4. **Test timeouts**: `timeout 20m ctest -V -L DFLY`\n5. **ASAN leaks**: Check CI, suppress in `helio/util/asan_suppressions.txt`\n6. **Helio modifications**: DON'T edit `helio/` (it's a git submodule - changes go upstream)\n7. **CodeQL checks**: DON'T run codeql_checker when testing changes - it's slow and unnecessary for development\n\n---\n\n## Debugging Tips\n\n**Logging**: `--alsologtostderr --v=1 --vmodule=module=2`\n\n**ASAN**: `ASAN_OPTIONS=detect_leaks=1:symbolize=1`, suppressions: `helio/util/asan_suppressions.txt`\n\n**CI reproduction**: See [.github/workflows/ci.yml](.github/workflows/ci.yml)\n\n**Troubleshooting**: Check fiber deadlocks (use `util::fb2` not `std::mutex`), timeout issues (`--test_timeout`), ASAN reports\n\n---\n\n## Validation Checklist\n\nBefore claiming a task is complete, verify:\n\n### Code Quality\n\n- [ ] Code compiles without errors: `cd build-dbg && ninja dragonfly`\n- [ ] Code compiles without warnings (CI uses `-Werror`)\n- [ ] Code follows Google C++ Style Guide (run `clang-format`)\n- [ ] No new ASAN/UBSAN violations\n\n### Testing\n\n- [ ] All existing C++ unit tests pass: `ctest -V -L DFLY`\n- [ ] New feature has corresponding test coverage\n- [ ] Tests pass in both Debug and Release builds\n- [ ] Tests pass with ASAN/UBSAN enabled (if applicable)\n- [ ] **DO NOT run codeql_checker** - it's slow and unnecessary for development testing\n\n### Pre-commit & Style\n\n- [ ] Pre-commit hooks installed: `pre-commit install`\n- [ ] Code formatted with clang-format (C++) and black (Python)\n\n### Documentation\n\n- [ ] Public APIs have comments explaining purpose\n- [ ] Complex algorithms have explanatory comments\n- [ ] README or docs updated if behavior changes\n- [ ] No commented-out code left in final commit\n\n### Performance\n\n- [ ] No obvious performance regressions (run benchmarks if needed)\n- [ ] No unnecessary allocations in hot paths\n- [ ] Lock-free data structures used where appropriate\n\n<!-- File: CLAUDE.md (Tokens: ~3 | Category: claude-rule) -->\nAGENTS.md\n\n<!-- File: .github/copilot-instructions.md (Tokens: ~198 | Category: copilot-instructions) -->\n---\ndescription: 'Code review guidelines for GitHub copilot in this project'\napplyTo: '**'\nexcludeAgent: [\"coding-agent\"]\n---\n\n# Code Review Instructions\n\nKeep reviews high-signal and minimal. Only comment on real bugs with high confidence.\n\n## Comment Only When\n- The issue is a correctness, security, concurrency, or architecture problem.\n- The impact is clear and non-trivial.\n- You can point to concrete evidence in the diff (not speculation).\n\n## Avoid\n- Style, formatting, naming, or minor performance nits.\n- Optional refactors or “nice to have” suggestions.\n- Praise, restating the code, or long explanations.\n- Duplicate comments for the same root cause.\n\n## Review Style\n- Be terse: 1-2 sentences per issue.\n- Include file and line references when possible.\n- If no issues are found, say “No issues found.”\n- Provide concrete suggestions for fixes when possible, or examples to illustrate the problem.\n\n<!-- File: .claude/skills/benchmark-report/SKILL.md (Tokens: ~2269 | Category: anthropic-skill) -->\n---\nname: benchmark-report\ndescription: Generate a technical benchmark report in markdown from raw memtier/redis-benchmark output. Use this whenever the user provides benchmark results, log files, memtier JSON output, or a directory of benchmark data and wants a written report or comparison document. Trigger even when the user says \"write up these results\", \"make a report from this data\", \"turn this into a blog post\", or just drops a directory path. Always use this skill when the user provides benchmark files and wants a human-readable output. Reports compare Dragonfly against another system (ElastiCache, Redis, Valkey) but can handle single-system data too.\nversion: 1.0.0\n---\n\n# Benchmark Report Generator\n\nGenerate a technical markdown report from raw memtier/redis-benchmark data. The report is aimed at engineers evaluating Dragonfly for production use.\n\n## Step 0: Interview before writing\n\nBefore touching any files, ask the user 3–4 targeted questions to fill in context that cannot be extracted from the raw data. Read the data directory first so your questions are specific, not generic.\n\nQuestions to ask (adapt based on what's missing after scanning the files):\n\n1. **Scenario**: What is the purpose of this benchmark? (e.g., \"SSD tiering comparison\", \"evaluating ElastiCache replacement\", \"testing read-heavy workload at scale\") — this becomes the intro paragraph and title.\n2. **Instance types**: What instance type / tier was each system running on? Include memory size.\n3. **Storage configuration**: For Dragonfly — is SSD tiering enabled? What is the memory limit vs total dataset size?\n4. **Anything else relevant**: Pricing comparison, region, network setup, anything not in the scripts/logs.\n\nWait for the user's answers before proceeding to Step 1. If the user says \"just use TODOs\" or similar, proceed without waiting.\n\n## What the user provides\n\n- A directory path with benchmark data files\n- Context answers from the interview above\n\nTypical directory layout:\n```\ndata_dir/\n *.sh — benchmark script (parameters, key distribution, test phases)\n *.json — memtier JSON output per system per phase\n *.log — memtier log output per system\n```\n\nFiles may be flat in one directory or split across subdirectories. Read them all.\n\n## Step 1: Scan and parse\n\nList all files in the directory. Then read them:\n\n**Scripts (`*.sh`)** — extract:\n- `KEY_MAXIMUM`, `DATA_SIZE`, `PIPELINE`, `CLIENTS`, `THREADS`, `TEST_TIME`\n- `FILL_RATIO`, `ACCESS_RATIO`, `FILL_KEY_PATTERN`, `ACCESS_KEY_PATTERN`\n- Key distribution flags: `--key-stddev`, `--key-zipf-exp`, etc.\n- Which phases ran (`RUN_FILL=true`, `RUN_ACCESS_READ_002=true`, etc.)\n- Hostnames — used to identify systems under test\n\n**JSON files (`*.json`)** — from memtier `--json-out-file` output:\n- `configuration.server` → hostname (used to identify the system)\n- `configuration.clients`, `threads`, `pipeline`, `ratio`, `key_maximum`, `data_size`, `key_stddev`\n- `ALL STATS.Gets` or `ALL STATS.Sets` → `Ops/sec`, `Average Latency`, `Percentile Latencies` (p50/p99/p99.9)\n- `ALL STATS.Runtime.Total duration` → test duration in ms\n\n**Log files (`*.log`)** — from memtier stdout:\n- Look for the `ALL STATS` table block. Take the last occurrence (final aggregate):\n ```\n Type Ops/sec ... Avg. Latency p50 Latency p99 Latency p99.9 Latency\n Sets 220503.72 ... 4.32106 2.95900 9.72700 53.24700\n Gets 138188.17 ... 6.94510 5.53500 36.09500 54.27100\n ```\n- Look for `real Xm Y.Zs` timing lines to capture wall-clock duration per phase\n\n## Step 2: Identify systems under test\n\nName each system from filename signals and hostnames:\n\n| Signal | System name |\n|--------|------------|\n| `bench_df*.log`, hostname contains `dragonflydb.cloud` | Dragonfly |\n| `bench_ec*.log`, hostname contains `cache.amazonaws.com` | ElastiCache |\n| `bench_valkey*.log` | Valkey |\n| `bench_redis*.log` | Redis |\n| `bench_local*.log`, `127.0.0.1` | Local (note version if available) |\n\nUse the actual product name, not the hostname. Do not include hostnames or endpoints in the report — they are not relevant for reproducing the results and may change.\n\n## Step 3: Generate charts\n\nIf JSON files are present, generate latency charts with `tools/plot_memtier_latency.py`:\n\n```bash\nmkdir -p <data_dir>/charts\npython3 <repo_root>/tools/plot_memtier_latency.py <json_file> <data_dir>/charts/<name>.html\n```\n\nRun once per JSON file. If the script is not found or fails, leave a `[TODO: chart]` stub. The script skips operations with no traffic (zero-latency entries), so it handles write-only fill JSONs and read-only access JSONs without errors.\n\nTo find `tools/plot_memtier_latency.py`, search from the current working directory: `find . -name \"plot_memtier_latency.py\" -not -path \"*/build*\"`.\n\n## Step 4: Write the report\n\nSave to `<data_dir>/report.md`. Use the template below. Populate every value you found; use `[TODO]` for anything missing (instance type, memory, pricing, scenario context the user didn't provide).\n\n---\n\n```markdown\n# <Descriptive title: what was tested, e.g., \"Dragonfly SSD Tiering vs ElastiCache: 50 GB Read Workload\">\n\n<1–3 sentence intro: what was measured and why. No adjectives. Example: \"This benchmark measures read throughput and latency for a 50 GB key-value dataset under a Gaussian access pattern. Dragonfly was configured with SSD tiering; ElastiCache r7g.4xlarge was used with all data in memory.\">\n\n## Test Environment\n\n| | Dragonfly | ElastiCache |\n|---|---|---|\n| Version / tier | [TODO] | [TODO] |\n| Instance type | [TODO] | [TODO] |\n| Memory | [TODO] | [TODO] |\n| Storage | [TODO: SSD tiering?] | [TODO] |\n| Region | [TODO] | [TODO] |\n\n<!-- Do NOT include hostnames, endpoints, or internal URLs — they are irrelevant to reproducing the test. -->\n\n## Dataset\n\n| Parameter | Value |\n|---|---|\n| Key count | <from KEY_MAXIMUM, formatted with commas> |\n| Value size | <DATA_SIZE> bytes |\n| Approximate dataset size | <KEY_MAXIMUM × DATA_SIZE, in GB> |\n| Key distribution | <Gaussian σ=X% of keys / uniform / Zipf exp=X> |\n\n## Benchmark Configuration\n\n| Parameter | Value |\n|---|---|\n| Tool | memtier_benchmark |\n| Threads | <THREADS> |\n| Connections per thread | <CLIENTS> |\n| Pipeline depth | <PIPELINE> |\n| Test duration (access phase) | <TEST_TIME>s |\n| Access ratio (SET:GET) | <ratio> |\n\n## Results\n\n### Fill Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p99 Latency (ms) | Duration |\n|---|---|---|---|---|\n| Dragonfly | | | | |\n| ElastiCache | | | | |\n\n### Read Access Phase\n\n| System | Throughput (ops/sec) | Avg Latency (ms) | p50 (ms) | p99 (ms) | p99.9 (ms) |\n|---|---|---|---|---|---|\n| Dragonfly | | | | | |\n| ElastiCache | | | | | |\n\n## Latency Over Time\n\n<For each JSON file, embed the chart link or stub>\n\n**Dragonfly — read access (σ=2%)**\n![Dragonfly latency over time](./charts/df_access.html)\n\n**ElastiCache — read access (σ=2%)**\n[TODO: ElastiCache JSON not available; chart cannot be generated]\n\n## Observations\n\n<Factual, numbered. One observation per line. State what the data shows; do not say whether it is good or bad.>\n\n1. Throughput: Dragonfly achieved X ops/sec vs Y for ElastiCache — a Z% difference.\n2. Average latency: X ms (Dragonfly) vs Y ms (ElastiCache).\n3. p99 latency: X ms vs Y ms. <If there's a large gap relative to p50, note it.>\n4. Fill phase: <note fill throughput and duration for each system>.\n5. <Note any caveats: e.g., \"This run covers only 100% read traffic. Mixed read/write was not executed.\">\n6. <Note anything unusual in the time-series if visible from the data.>\n```\n\n---\n\n## Writing rules\n\n- Use exact numbers from the data. No rounding unless the value is very large (round to 3 significant figures max).\n- Percentages for comparisons: compute `(A - B) / B * 100`; write as \"X% higher\" or \"X% lower\".\n- Prohibited phrases: \"blazing fast\", \"industry-leading\", \"dramatically\", \"significantly\", \"next-level\", \"game-changing\", \"unparalleled\", \"impressive\"\n- No narrative hooks: no \"but wait\", \"what happens next\", \"the results speak for themselves\", \"as you can see\"\n- Observations may include inferences (e.g., \"the Gaussian σ=2% means the hot working set is ~3.5M keys, likely fitting in memory\") — label them clearly as inferences if they go beyond what the data directly shows\n- Bold labels in observations (e.g., `**Read throughput**:`) — keep them for scannability\n- Do not include hostnames, endpoints, or internal URLs anywhere in the report\n- Missing context → `[TODO: ...]` not a guess\n- Single-system data: omit comparison columns; keep observations factual about absolute numbers\n\n## Output\n\nPrint the report path when done: `Report written to: <path>/report.md`\n\nIf charts were generated, list them too.\n\n<!-- File: .claude/skills/benchmark/evals/evals.json (Tokens: ~612 | Category: anthropic-skill) -->\n{\n \"skill_name\": \"benchmark\",\n \"evals\": [\n {\n \"id\": 0,\n \"name\": \"full-report-vs-valkey\",\n \"prompt\":\n \"I just finished a benchmark run comparing Dragonfly against Valkey 9.0 on AWS. The raw artifacts are in the directory I'm pointing you at: dfly_bench JSON output (write and read phases) and INFO memory dumps for both datastores on m7g.2xlarge and m7g.4xlarge, plus a RUN_NOTES.txt with the hardware, exact commands, and DBSIZE after each write phase. Generate the benchmark report from this data, with charts and a raw-data appendix.\",\n \"expected_output\":\n \"A markdown report mirroring the team format: conditions section, a Simple Strings use-case section with a TLDR, per-instance (m7g.2xlarge, m7g.4xlarge) QPS/memory and latency tables, embedded QPS-over-time charts, and an appendix with exact commands, environment, and raw output. QPS/P99 must match the data; bytes-per-entry reported both logical (~127 DF / ~149-150 Valkey) and RSS.\",\n \"files\": [\n \"fixtures/\"\n ]\n },\n {\n \"id\": 1,\n \"name\": \"memory-focus-single-datastore\",\n \"prompt\":\n \"Here's raw output from a Dragonfly-only run on m7g.4xlarge (dfly_bench write/read JSON + INFO memory dump + run notes with DBSIZE). I care mostly about memory efficiency. Write me a short benchmark report focused on memory usage per entry, plus the throughput numbers.\",\n \"expected_output\":\n \"A report covering only Dragonfly (no fabricated Valkey/Redis comparison column), reporting both logical (used_memory/DBSIZE ~127) and RSS bytes-per-entry, plus write/read QPS and P99. Honest about it being single-datastore.\",\n \"files\": [\n \"fixtures/df_4xl_write.json\",\n \"fixtures/df_4xl_read.json\",\n \"fixtures/df_4xl_info_memory.txt\",\n \"fixtures/RUN_NOTES.txt\"\n ]\n },\n {\n \"id\": 2,\n \"name\": \"kickoff-planning-interactive\",\n \"prompt\":\n \"I've got two m7g.4xlarge boxes in AWS. Benchmark Dragonfly against Valkey on a simple strings workload.\",\n \"expected_output\":\n \"The skill should engage its interactive front-end: ask which box is the server vs the client (load generator must be separate), confirm SSH access, plan to run servers on port 6380, plan to start the monitoring stack (and fix the prometheus scrape port to 6380), and lay out the write/read workload plan. It must NOT invent benchmark numbers it hasn't measured.\",\n \"files\": []\n }\n ]\n}\n\n<!-- File: .claude/skills/benchmark/references/report-template.md (Tokens: ~1192 | Category: anthropic-skill) -->\n# Report template\n\nThe report mirrors the team's published benchmark format. Structure matters: a\nreader scans the conditions, jumps to the TLDR for the takeaways, checks the\nper-instance tables for numbers, and uses the appendix to reproduce. Follow this\nshape.\n\nUse Markdown tables. Center-align numeric columns (`:---:`). Bold the winning\ndatastore in comparison tables when there's a clear winner.\n\n## Structure\n\n```markdown\n# <Title> — e.g. \"Comparing Dragonfly vs Valkey on AWS\"\n\n<1–2 sentences: what was compared, on what hardware, what workloads.>\n\n## Conditions\n\nDragonfly launch command (identical across all configs):\n\n ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n\n<Comparison target> launch command:\n\n docker run --network=host --rm valkey/valkey:9.0 --save \"\" --appendonly no \\\n --io-threads 8 --protected-mode no --port 6380\n\nInstances: <server types + CPU counts>, client on <client instance type>.\nOS: <distro + kernel>. Network tuning: <applied / none — one line on what>.\n\n## <Use-case name> — e.g. \"Simple Strings\"\n\n<1–2 sentences describing the workload and why it matters.>\n\nWrite command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 --ratio 1:0 --json_out_file df_write.json\n\nRead command:\n\n ./dfly_bench -n $N -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN --ratio 0:1\n\n### TLDR\n\n* **<Datastore>** <the genuinely important takeaway — scaling, efficiency, a surprise>.\n* <3–5 bullets total. Interpretation, not a restatement of the tables.>\n\n## <Instance type> — e.g. \"m7g.2xlarge\"\n\n<Optional: one sentence on anything notable, e.g. throughput stability over time.>\n\n![Write throughput over time](charts/write_m7g2xl.png)\n\n### QPS and memory\n\n| Metric | Dragonfly | Valkey |\n| ----- | :---: | :---: |\n| Write QPS (k) | 816 | 548 |\n| Read QPS (k) | 848 | 811 |\n| Memory / entry — logical (bytes) | 127 | 149 |\n| Memory / entry — RSS (bytes) | 138 | 162 |\n\n### Latency\n\n| Backend | Write P99 (us) | Read P99 (us) |\n| :---: | :---: | :---: |\n| **Dragonfly** | 787 | 887 |\n| **Valkey** | 3283 | 951 |\n\n<Repeat the per-instance block for each instance type tested.>\n\n## Appendix — raw data\n\n### Environment\n\n| Property | Server | Client |\n| --- | --- | --- |\n| Instance type | m7g.2xlarge | c7gn.4xlarge |\n| vCPUs | 8 | 16 |\n| OS / kernel | Ubuntu 24.04.2 / 6.8.0 | ... |\n| Dragonfly version | <git hash / release> | — |\n| Network tuning | IRQ SMP affinity spread across CPUs | — |\n\n### Exact commands\n\n<Every command line actually run, per phase and per datastore, verbatim —\nincluding the launch commands, write/read load commands, and any tuning steps.>\n\n### Raw output\n\n<Per run: the final dfly_bench/memtier summary + latency histogram, and a\nreference to the saved JSON file (df_write.json, etc.). Include INFO memory\nsnapshots used for the per-entry figures.>\n```\n\n## Notes on writing the TLDR\n\nThe TLDR is the part people actually read. Make each bullet a claim the data\nsupports, phrased as a takeaway:\n\n- Good: \"Dragonfly scales vertically with cores and sustains up to 10x the\n throughput of a single Valkey process on large instances.\"\n- Weak: \"Dragonfly had higher QPS.\" (just restates the table)\n\nCall out hardware surprises honestly — e.g. an instance type that didn't scale\nlinearly — because they affect how a reader should interpret the numbers. If a\nrun was client-bound or otherwise caveated, say so here rather than burying it.\n\n## Charts\n\nOne chart per instance per phase is usually enough: QPS-over-time, with one line\nper datastore. The visual story is throughput *stability* — Dragonfly holding a\nflat line while a competitor degrades as its table grows is more convincing than\na single aggregate number. Generate with `scripts/make_charts.py` from the load\ngenerator's time-series JSON, save under a `charts/` dir next to the report, and\nembed with relative paths.\n\nWhen the monitoring stack ran (Phase 4), the Grafana dashboards add server-side\npanels that the load generator can't show — memory growth over the write phase,\nper-shard CPU, host network saturation. Crop the Grafana time range to the run\nwindow and pull the relevant panels (rendered PNG or screenshot) into the\nper-instance section. These are especially good for the memory-efficiency story\nand for proving the server (not the client) was the bottleneck. Note in the\nappendix that the data came from the monitoring stack, and which scrape port was\nused.\n\n<!-- File: .claude/skills/benchmark/references/running-benchmarks.md (Tokens: ~6568 | Category: anthropic-skill) -->\n# Running benchmarks — operational reference\n\nHow to prepare instances, start servers, drive the load generators, and measure\nmemory. Read the section you need; you don't have to read top-to-bottom.\n\n## Table of contents\n\n- [Instance preparation](#instance-preparation)\n- [Starting Dragonfly](#starting-dragonfly)\n- [Starting Valkey / Redis for comparison](#starting-valkey--redis-for-comparison)\n- [dfly_bench (default load generator)](#dfly_bench-default-load-generator)\n- [memtier_benchmark (alternative)](#memtier_benchmark-alternative)\n- [The standard strings workload](#the-standard-strings-workload)\n- [Measuring memory](#measuring-memory)\n- [Monitoring stack (Prometheus + Grafana)](#monitoring-stack-prometheus--grafana)\n- [Network tuning](#network-tuning)\n- [Sizing the load](#sizing-the-load)\n\n---\n\n## Instance preparation\n\nThe server and the load generator should live on **separate machines**. A\nclient co-located with the server competes for CPU and silently caps the\nnumbers — the report is then measuring the client, not the server.\n\nRemote instances usually already have the repo at `~/projects/dragonfly`. Check\nfirst; clone only if missing:\n\n```bash\nssh <host> 'test -d ~/projects/dragonfly && echo present || echo missing'\n# if missing:\nssh <host> 'git clone https://github.com/dragonflydb/dragonfly.git ~/projects/dragonfly'\n```\n\n**Look for prebuilt binaries first.** Before building or downloading, check the\nrepo's build output folders — the binary is often already there. Look in\n`build-opt/` and `build-release/` (and `build-dbg/` only as a last resort — debug\nbuilds give meaningless performance numbers):\n\n```bash\nssh <host> 'ls -la ~/projects/dragonfly/build-opt/{dragonfly,dragonfly-aarch64,dfly_bench} 2>/dev/null; \\\n ls -la ~/projects/dragonfly/build-release/{dragonfly,dfly_bench} 2>/dev/null'\n```\n\nA folder may hold several binaries — e.g. a locally-built `dragonfly` (dev/HEAD)\nalongside a downloaded release `dragonfly-aarch64`; check `--version` on each to\nknow which is which, since \"compare the prebuilt vs the release binary\" is a\ncommon request. The load generator (`dfly_bench`) lives in the same folder on the\nclient.\n\n**Build vs download** — if no suitable prebuilt binary exists, follow the user's\ninstruction. To build a release binary from source on the instance:\n\n```bash\nssh <host> 'cd ~/projects/dragonfly && ./helio/blaze.sh -release -DWITH_AWS=OFF -DWITH_GCP=OFF'\nssh <host> 'cd ~/projects/dragonfly/build-opt && ninja dragonfly dfly_bench'\n```\n\nThis yields `build-opt/dragonfly` (the server) and `build-opt/dfly_bench` (the\nload generator). Build `dfly_bench` on the **client** box and `dragonfly` on the\n**server** box. Always benchmark release builds — a debug build's numbers are\nmeaningless for performance claims.\n\nIf the user prefers a published release, download the tarball from GitHub\nreleases and extract it instead of building.\n\nSanity-check reachability from the client before a long run:\n\n```bash\nredis-cli -h <server-ip> -p 6380 ping # expect PONG\n```\n\nCollect environment facts for the report while you're on the box:\n\n```bash\nssh <host> 'nproc; uname -r; cat /etc/os-release | head -2'\n# cloud instance type, if on AWS:\nssh <host> 'curl -s http://169.254.169.254/latest/meta-data/instance-type'\n```\n\n---\n\n## Starting Dragonfly\n\nThe published comparisons run Dragonfly with a minimal, persistence-free config\nso the benchmark measures the data path, not disk:\n\n```bash\n./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380\n```\n\n- `--conn_use_incoming_cpu` — pin each connection's processing to the CPU that\n received it; improves locality under high connection counts.\n- `--dbfilename=` — disable snapshot loading/saving.\n\nLet Dragonfly use all cores by default (it auto-detects). Record the exact\nlaunch command for the appendix.\n\n**Start it so you can reliably stop it — capture the PID.** Launching under\n`tmux`/`nohup` is fine, but `tmux kill-session` does **not** reliably reap the\nserver, and a stale server is a trap: a \"fresh\" server silently fails to bind the\nport, your client reconnects to the *old* instance, and you measure stale data\n(and waste the old dataset's RAM). Always record the PID and kill by PID:\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/build-opt; \\\n nohup ./dragonfly --conn_use_incoming_cpu --dbfilename= --logtostderr --port 6380 \\\n >/tmp/dfly.log 2>&1 & echo $! > /tmp/dfly.pid'\n# ... benchmark ...\n# Stop it and VERIFY it's gone before starting the next binary:\nssh <server> 'kill -9 $(cat /tmp/dfly.pid); sleep 3; \\\n pgrep -af dragonfly | grep -v \"pgrep\\|bash -c\" || echo \"stopped\"; \\\n redis-cli -p 6380 ping 2>&1 | head -1 # expect Connection refused'\n```\n\nBetween two binaries (e.g. prebuilt vs release), **fully stop and verify the\nfirst is dead and the port is free before starting the second** — otherwise the\nsecond never binds and you benchmark the first twice. Note that pattern-based\n`pkill -f` can miss a process whose cmdline is the relative `./dragonfly`; killing\nby the captured PID is the robust path.\n\n---\n\n## Starting Valkey / Redis for comparison\n\nRun via docker on the server box. Valkey does not scale past ~8 io-threads on\ncurrent hardware, so more than that rarely helps — note the thread count you\nused.\n\n```bash\ndocker run --network=host --rm valkey/valkey:9.0 \\\n --save \"\" --appendonly no --io-threads 8 --protected-mode no --port 6380\n```\n\nFor Redis, substitute the `redis` image and equivalent flags. Run the comparison\ntarget on the **same instance type** as Dragonfly, one at a time (not\nsimultaneously — they'd contend), so the comparison is apples-to-apples.\n\n---\n\n## dfly_bench (default load generator)\n\nSource: `src/server/dfly_bench.cc`. Key flags:\n\n| Flag | Meaning | Typical |\n| --- | --- | --- |\n| `-h` | server host | client→server IP |\n| `-p` | server port | `6380` |\n| `-c` | connections **per thread** | ~10; ≤25 max; 1–5 when pipelining |\n| `-n` | requests per connection | sizes the run (omit if `--test_time`) |\n| `--test_time` | run duration in seconds | e.g. `120` |\n| `--qps` | target QPS per connection; `0` = unthrottled | `0` for peak |\n| `-d` | value size in bytes | `64` for strings tests |\n| `--key_maximum` | size of key space | tune to fill memory |\n| `--key_dist` | `U` uniform, `N` normal, `Z` zipfian, `S` sequential | `U` default |\n| `--ratio` | set:get ratio | `1:0` writes, `0:1` reads |\n| `--command` | custom command template (`__key__`, `__data__`) | for non-string workloads |\n| `--pipeline` | pending requests per connection | `>1` for peak throughput |\n| `-P` | `memcache_text` for memcached protocol | empty for RESP |\n| `--json_out_file` | **memtier-compatible JSON with per-second time series** | always set it |\n\n`--json_out_file` only works with the built-in GET/SET workload (not with\n`--command`, `--noreply`, or `--connect_only`). For custom-command workloads you\nscrape the console summary instead.\n\n**Console output.** Periodic lines look like:\n\n```\n12s: 40.0% done, RPS(now/agg): 815000/812340, errs: 0, hitrate: 99.8%, clients: 200\n```\n\nand the final summary:\n\n```\nTotal time: 30s. Overall number of requests: 24370200, QPS: 812340, P99 lat: 787us\nLatency summary, all times are in usec:\n<histogram>\n```\n\nSave the full stdout — the histogram and final QPS go in the appendix; the\n`collect_metrics.py` script prefers the JSON but can fall back to this.\n\nAlso save the command lines themselves under the run output directory, e.g.\n`commands_audit.log`. Record setup, server launch, write/read load,\ncapture, diagnostics, and chart/report commands as they are run; stdout alone is\nnot raw data because it cannot reproduce the run. Initialize this file before\nthe first workload so planned script invocations are visible up front, then\nappend commands as they execute.\n\n**Use the skill scripts for the main run path.** The main benchmark phases should\nuse:\n\n```bash\n.claude/skills/benchmark/scripts/bench_server.sh start|start-docker|stop|stop-docker|status ...\n.claude/skills/benchmark/scripts/monitor_fill.sh ...\n.claude/skills/benchmark/scripts/capture_result.sh ...\n.claude/skills/benchmark/scripts/collect_metrics.py ...\n.claude/skills/benchmark/scripts/make_charts.py ...\n.claude/skills/benchmark/scripts/plot_fill.py ...\n```\n\nInline SSH is acceptable for discovery, package/setup checks, bounded\nforeground probes, and diagnostics. If you bypass a run-control script for a\nmain benchmark phase, record the reason in `commands_audit.log` and the appendix before\nrunning the manual equivalent.\n\n**Keep orchestration phase-separated.** Do not run a single local shell command\nthat starts a remote tmux benchmark, polls until completion, captures results,\nand starts the next phase. If the local command is interrupted, the remote tmux\nprocess can finish while the local capture logic is gone, leaving the run idle\nand uncaptured. Prefer bounded foreground `dfly_bench` commands for known-length\nruns. If you need tmux/nohup for a long fill, use separate steps: launch it,\nmonitor it, verify it has exited, then run the capture/read commands separately.\n\n**JSON structure** (`--json_out_file`):\n\n```json\n{\n \"configuration\": { \"server\": \"...\", \"port\": 6380, \"clients\": 200,\n \"threads\": 8, \"pipeline\": 1, \"ratio\": \"1:0\" },\n \"ALL STATS\": {\n \"Runtime\": { \"Total duration\": 30000 },\n \"Sets\": { \"Count\": ..., \"Average Latency\": ..., \"Percentile ...\": ...,\n \"Time-Serie\": { \"0\": {...}, \"1\": {...}, ... } },\n \"Gets\": { ... }\n }\n}\n```\n\nThe per-second `Time-Serie` buckets are what the charts plot.\n\n---\n\n## memtier_benchmark (alternative)\n\nIndustry-standard, available as a docker image (`redislabs/memtier_benchmark`).\nUse it when the user wants cross-validation against a tool Dragonfly doesn't\nship, or for the pipelined peak-throughput use-case. Example (from the repo's\nk8s job):\n\n```bash\nmemtier_benchmark -s <server> -p 6380 \\\n --pipeline=30 --key-maximum=100000 -c 10 -t 2 --test-time=120 \\\n --distinct-client-seed --hide-histogram --json-out-file memtier_out.json\n```\n\n`--json-out-file` produces the same shape `collect_metrics.py` and\n`make_charts.py` read.\n\n---\n\n## The standard strings workload\n\nThis reproduces the published \"Simple Strings\" comparison. Run write then read,\nwith a memory snapshot in between.\n\n**Write phase** (load the keyspace):\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n --pipeline 30 -n $N --ratio 1:0 --json_out_file df_write.json\n```\n\nThe canonical strings test uses the built-in SET path (`--ratio 1:0`) so it\nproduces the JSON/time series the charts need. If TTLs matter, swap in\n`--command \"setex __key__ 10000 __data__\"` — but note `--json_out_file` is then\nignored, so for SETEX save stdout and parse/chart the periodic `RPS(now/agg)`\nlines or `monitor_fill.sh` output instead.\n\n**Memory snapshot** (immediately after writes — see next section).\n\n**Read phase**:\n\n```bash\n./dfly_bench -h <server> -p 6380 --qps=0 -d 64 --key_maximum=$KMAX -c $CONN \\\n -n $N --ratio 0:1 --json_out_file df_read.json\n```\n\nChoose `KMAX`, `N`, and `CONN` to (a) saturate the server and (b) load enough\ndata that instance memory is well utilized — see [Sizing the load](#sizing-the-load).\n\n---\n\n## Measuring memory\n\n\"Bytes per entry\" is the memory metric in the report. Capture it right after the\nwrite phase, before reads:\n\n```bash\nredis-cli -h <server> -p 6380 INFO memory > info_memory.txt\nredis-cli -h <server> -p 6380 DBSIZE\n```\n\nFrom `INFO memory` you need `used_memory` (logical) and `used_memory_rss`\n(physical resident set). Report **both** per-entry figures:\n\n- `used_memory / DBSIZE` — logical bytes per entry (matches the published\n \"Memory Per Entry\").\n- `used_memory_rss / DBSIZE` — physical bytes per entry, the real footprint\n including allocator overhead and fragmentation.\n\n`collect_metrics.py` computes both when given `--info` and `--dbsize`.\n\nFor Valkey/Redis the same `INFO memory` fields exist, so the comparison is\ndirect. Capture the snapshot at the **same keyspace size** for both, or the\nper-entry comparison is unfair.\n\n---\n\n## Monitoring stack (Prometheus + Grafana)\n\nLives in `tools/local/monitoring/` and runs on the **server** machine via docker\ncompose. It gives continuous, 1-second-resolution time series for the whole\nbenchmark window — server-side command rates and memory, plus host CPU/network\nfrom node-exporter — which is both a sanity check (was the server actually\nsaturated? did it swap?) and the source of the report's best charts.\n\n```bash\ncd ~/projects/dragonfly/tools/local/monitoring\ndocker compose up -d # Prometheus, Grafana, node-exporter\ndocker compose --profile redis up -d # also start redis-exporter for Valkey/Redis\n```\n\nEndpoints (on the server box): Prometheus `:9090`, Grafana `:3000`,\nnode-exporter `:9100`, redis-exporter `:9121`.\n\n**What gets scraped:**\n\n- `dragonfly` job — Dragonfly exposes Prometheus metrics directly at\n `/metrics` on its main port. The bundled `prometheus.yml` targets\n `host.docker.internal:6379`.\n- `node-exporter` job — host CPU, memory, network.\n- `redis-exporter` job (profile `redis`) — Valkey/Redis stats, also via\n `host.docker.internal:6379`.\n\n**Port — standardize on 6380.** Always run benchmarked servers (Dragonfly,\nValkey, Redis) on port `6380`. The shipped `prometheus.yml` targets `6379`, so\nbefore bringing the stack up, edit the `dragonfly` and `redis-exporter` jobs in\n`tools/local/monitoring/prometheus/prometheus.yml` to point at `:6380`:\n\n```bash\nsed -i 's/:6379/:6380/g' ~/projects/dragonfly/tools/local/monitoring/prometheus/prometheus.yml\n# if Prometheus is already up, pick up the change:\ndocker compose restart prometheus\n```\n\nThen confirm targets are `UP` at `http://<server>:9090/targets` before starting\nload — otherwise Prometheus silently collects nothing for the server.\n\n**Dashboards** are pre-provisioned in Grafana (`dragonfly.json`, `redis.json`,\n`node-exporter.json`) — no manual import. For the report, note the benchmark\nstart/end timestamps so you can crop the Grafana time range to the exact run,\nthen capture the relevant panels (throughput, memory, CPU) — via the provisioned\n`grafana-image-renderer` (the compose file includes a `renderer` service) or a\nscreenshot — and embed them. You can also query Prometheus directly over HTTP\n(`/api/v1/query_range`) if you want raw series for `make_charts.py`.\n\nLeave the stack up across all runs so one continuous window covers everything.\n\n## Network tuning\n\nOn cloud hardware, peak throughput is gated by how network-interrupt load is\nspread across CPUs. The published runs tuned **SMP affinity for IRQs** so the\nNIC's interrupts fan out across multiple cores instead of saturating one.\n\nThis is instance- and NIC-specific. The general approach: identify the NIC's IRQs\n(`/proc/interrupts`), then write CPU masks to `/proc/irq/<N>/smp_affinity` to\nspread them. Many setups use the kernel's `set_irq_affinity.sh` helper shipped\nwith the NIC driver. Because the exact steps depend on the instance, confirm the\napproach with the user if they want tuning, and **record the exact commands you\nran** — tuned and untuned numbers are not comparable, and the appendix must say\nwhich was used.\n\nIf you skip tuning, state that plainly in the report.\n\n---\n\n## Sizing the load\n\nTwo independent things to size: **how many keys** (sets the memory footprint)\nand **how long / how many requests** (sets steady-state and how full memory\ngets). Get these wrong in either direction and the benchmark is misleading.\n\n### The memory budget — the most important (and most dangerous) call\n\nA write benchmark must land in the **sweet spot between two failure modes**:\n\n- **Too few keys → wrong-but-safe.** Benchmarking a nearly-empty instance is\n not representative. A small dataset sits in CPU cache and hides the real cost\n of a large hash table (cache misses, allocator behavior, fragmentation). The\n QPS looks great and means nothing. *Using available memory is the whole\n point.*\n- **Too many keys → OOM or eviction.** Push too far and the run is ruined two\n different ways (see the two ceilings below). Either way the numbers are\n garbage.\n\n**There are TWO ceilings, not one. Stay under both.**\n\n1. **Dragonfly's `maxmemory`.** Dragonfly auto-sets `maxmemory` to a fraction of\n RAM (≈ 78–80% by default). When memory crosses it, Dragonfly **rejects or\n evicts** writes — the load generator reports *error responses* and the run is\n contaminated. **Check it before the run** (`redis-cli -p 6380 config get\n maxmemory`, or the `maxmemory:` line in `INFO memory`).\n\n **Rejections start with a margin *below* `maxmemory`, not at it.** During a\n write, dash-table resizes transiently spike memory, so Dragonfly begins\n rejecting while steady-state `used_memory` is still well under the cap —\n observed rejections at `used_memory` ≈ 90% of `maxmemory` and even lower at the\n moment of a big resize. So keep steady-state `used_memory` to **≤ ~80% of\n `maxmemory`** to leave room for these spikes. If you need to fill more, raise\n `--maxmemory` explicitly — but then ceiling #2 (host RAM) binds.\n\n **Zero errors is the pass/fail bar.** *Any* eviction/OOM error contaminates\n the experiment — it slows the run (rejection stalls inflate P99) and stops the\n keyspace from growing, so QPS, latency, and bytes-per-entry are all off. After\n every write, **verify the load generator reported zero errors**\n (`grep -i error <write.log>`; dfly_bench prints `Got N error responses!`). If\n N > 0, the run is **invalid** — discard it, cut `--key_maximum` (start with a\n ~10–15% reduction), and re-run until errors are zero. Don't report a run with\n any errors, and don't try to \"subtract\" them out.\n2. **Host physical RAM, measured by RSS.** The accurate measure of real memory\n pressure is **resident set size — `used_memory_rss`** (equivalently the\n process RSS). True OOM happens when RSS approaches host RAM. Do **not** panic\n over `free`'s `used`/`available` columns during a write: a high-rate write\n inflates **page cache** (clean, reclaimable) and dirty pages, which makes\n `available` dip alarmingly even though the kernel will reclaim it under\n pressure. On a 30 GiB box a 195M-key dataset showed `free` available dropping\n to ~10% while `used_memory_rss` was only ~21 GiB (71% of RAM) — the dataset\n was nowhere near OOM; the rest was reclaimable cache. **Track RSS, keep it\n below host RAM with a ≥5% margin**, and ignore page-cache noise.\n\n**So size against RSS.** Target the dataset's RSS up to your fill goal while\nkeeping a real margin, and stay under `maxmemory`:\n\n```\ntarget_rss = host_RAM * fill_fraction # e.g. 0.90 for \"fill 90%\", never > 0.95\nrss_per_entry = used_memory_rss / DBSIZE # from calibration (accurate)\nkey_for_host = target_rss / rss_per_entry\nkey_for_maxmem = 0.90 * maxmemory / used_memory_per_entry # avoid eviction\ntarget_keys = min(key_for_host, key_for_maxmem)\n--key_maximum = target_keys\n```\n\nThe usual binding constraint is **`maxmemory`**, since its default (~78% of RAM)\nis below a 90% fill target. **To genuinely fill 90% of RAM, raise `--maxmemory`\nexplicitly** (e.g. `--maxmemory 28gb` on a 30 GiB box) so Dragonfly doesn't evict\nbefore RSS reaches the target — then RSS, watched live, is the safety signal.\nFilling memory is the goal; the failure modes to avoid are eviction (used_memory\n> maxmemory) and RSS approaching host RAM.\n\n`*_per_entry` (≈ `value_size + 60–100 B` overhead for small strings; more for\nTTL'd keys and other types) is best **calibrated**, not guessed:\n\n1. Run a short pilot write inserting a known, modest number of distinct keys\n (e.g. `--key_maximum=5000000 --key_dist S` with enough requests to write them\n all).\n2. Read `used_memory` / `DBSIZE` → real `bytes_per_entry`. Note this slightly\n *under*-estimates at scale (fragmentation grows), so keep margin.\n3. Plug into the formula above for the full run.\n\nCalibrate **per datastore** — Dragonfly and Valkey have different per-entry\noverhead, which is exactly what the report measures. Size each so its dataset\nfills the same target fraction; comparing at the same `DBSIZE` keeps the\nbytes-per-entry comparison fair (Dragonfly will simply use less RAM at that\nkey count, which is the finding, not a sizing error).\n\n### How distinct keys actually accumulate\n\nWith uniform random keys over `--key_maximum`, distinct keys (and therefore\nmemory) climb fast at first, then asymptote toward `key_maximum` as repeats\nstart landing on existing keys. **Memory plateaus near `key_maximum` distinct\nentries** — writing *longer* past that point just overwrites existing keys and\ndoes **not** grow memory. That's the key safety property: **`--key_maximum`, not\nrun length, bounds the footprint.** Run length controls how *close* to that\nplateau you get and how long you hold steady state.\n\nSo to reliably fill to the target without OOM:\n\n- Set `--key_maximum` to the computed `target_keys`.\n- Make the write phase long enough to insert essentially all of them — total\n requests (`-c` × threads × `-n`, or a `--test_time` that covers it) should be\n a small multiple of `key_maximum` (~2–3×) so coverage is near-complete.\n- Use `--key_dist S` (sequential) if you want *exactly* `key_maximum` distinct\n keys with no repeats and a precise fill; uniform is fine when you just need to\n approach the plateau.\n\n### Watch memory live — guard on RSS\n\nNever run a long write blind. The accurate safety signal is **`used_memory_rss`**\n(resident set), not `free`'s `available` (page cache makes it dip transiently and\nis reclaimable). Poll RSS every few seconds and **abort the write if RSS exceeds\n~90–95% of host RAM**, and separately watch `used_memory` against `maxmemory` to\ncatch approaching eviction. Example guard loop:\n\nRun the guard as its own bounded step, not as part of a larger launch/poll/capture\nmega-command. After the guard reports `DONE`, immediately run a separate capture\nstep (`DBSIZE`, `INFO memory`, stdout log, and zero-error check), then start the\nread phase.\n\n```bash\nTOTAL=$(ssh <server> 'free -b | awk \"/^Mem:/{print \\$2}\"')\nfor i in $(seq 1 60); do\n sleep 8\n read DB RSS USED < <(ssh <server> 'redis-cli -p 6380 dbsize | tr -d \"\\r\"; \\\n redis-cli -p 6380 info memory | grep -E \"^used_memory_rss:|^used_memory:\" | cut -d: -f2 | tr -d \"\\r\"' | paste -sd\" \")\n pct=$(awk \"BEGIN{printf \\\"%.0f\\\", $RSS/$TOTAL*100}\")\n echo \"dbsize=$DB rss=$(awk \"BEGIN{printf \\\"%.1f\\\",$RSS/2^30}\")GiB (${pct}% of RAM)\"\n if [ \"$pct\" -ge 92 ]; then echo \"ABORT: RSS ≥92% of RAM\"; \\\n ssh <client> 'tmux kill-session -t bench'; break; fi\ndone\n```\n\nWays to read current memory:\n\n- **`INFO memory` → `used_memory_rss`** — the accurate measure; the guard's\n input. Also compare `used_memory` to `maxmemory` (eviction warning).\n- **`/metrics` endpoint (Dragonfly only)** — `curl -s\n http://<server>:6380/metrics | grep -i mem` gives Dragonfly's live RSS/used\n gauges without a redis client. **Valkey/Redis do not serve `/metrics`** — for\n those, use `INFO memory` directly or scrape via the redis-exporter (see the\n monitoring section). `INFO memory` works for all three datastores, so it's the\n portable choice for the guard.\n- **Monitoring stack** — Prometheus/Grafana plots these over time; the source of\n the memory-over-time chart. (`free`/node-exporter available is fine for the\n chart, just not the trigger.)\n\nIf RSS approaches the limit faster than expected, **stop the write early** —\nyou've already filled enough to measure; a partially-filled run with clean\nnumbers beats one that evicted or OOMed.\n\n### The other knobs\n\n- **Connections (`-c` is *per thread*):** keep `-c` modest — **never above 25\n connections per thread, usually around 10**; for pipelined runs drop it to\n **1–5 per thread** (each connection already has many requests in flight, so\n more connections just add contention without more throughput). Scale total\n load by adding **threads** (and, if needed, a second client box), not by\n cramming connections onto each thread. Raise the total until QPS stops rising;\n if it keeps climbing at your max, the client is the bottleneck — add threads or\n another client machine, not more `-c`.\n- **Duration / steady state:** beyond filling memory, hold at least 30–60s of\n steady traffic so the reported QPS/P99 reflect steady state, not warm-up.\n Watch the periodic RPS line stabilize before trusting the number.\n- **Pipeline:** Use `--pipeline 30` for all write/prefill phases — without it\n the client stalls on each reply and becomes the bottleneck before the server\n saturates. Use `--pipeline 1` (or omit) only for latency-honest read\n benchmarks, and label those numbers accordingly.\n\nAlways note in the report the key count, value size, resulting memory fill (%\nof instance RAM), and whether the run was client-bound — an under-driven server\nor a barely-filled instance both produce numbers that quietly mislead.\n\n<!-- File: .claude/skills/benchmark/SKILL.md (Tokens: ~3374 | Category: anthropic-skill) -->\n---\nname: benchmark\ndescription: >\n Benchmark Dragonfly (and compare against Valkey/Redis) on local or remote\n cloud instances, then produce a performance + memory report with charts and a\n raw-data appendix. Use this whenever the user runs /benchmark, or asks to\n benchmark / load-test / measure throughput, QPS, latency, or memory efficiency\n of Dragonfly or another Redis-compatible server, to compare Dragonfly vs\n Valkey or Redis, or to generate a benchmark report. Trigger it even when the\n user just describes the goal (\"see how fast Dragonfly is on this box\",\n \"compare memory per key vs Valkey\") without saying the word \"benchmark\".\nargument-hint: (optional) free-form spec, e.g. \"server=10.0.0.1 client=10.0.0.2 strings vs-valkey\"\nallowed-tools: Bash, Read, Write, Edit, AskUserQuestion\n---\n\n# Benchmark Dragonfly\n\nDrive a benchmark of Dragonfly (optionally side-by-side with Valkey or Redis) on\nlocal or remote instances, collect throughput / latency / memory numbers, and\nassemble a report that mirrors the team's published format: a narrative TLDR,\nper-instance metric tables, charts, and an appendix with the exact commands and\nraw output so anyone can reproduce it.\n\nThe whole point of the report is **trust**: a reader should be able to see the\nheadline numbers, understand the conditions they were measured under, and\nre-run the exact commands themselves. Everything below serves that goal.\n\n## When you start\n\nIf the user passed a free-form spec in `$ARGUMENTS`, parse what you can from it\n(IPs, use-cases, comparison targets) and only ask about what's missing. The work\nsplits into clear phases — walk through them in order, but stay flexible: the\nuser may already have servers running, or may only want the report half (handed\nraw output to format). Read what they're actually asking for.\n\nPhases: (1) gather spec → (2) prepare instances → (3) network tuning →\n(4) start monitoring → (5) run workloads → (6) extract metrics, charts, report.\n\nThe two reference files carry the operational detail — read them before acting,\nnot from memory:\n\n- `references/running-benchmarks.md` — how to prep instances, start servers,\n drive `dfly_bench` and `memtier_benchmark`, run Valkey/Redis for comparison,\n apply network tuning, and measure memory. **Read this before Phase 2.**\n- `references/report-template.md` — the exact report structure, with an example.\n **Read this before Phase 5.**\n\nBundled `scripts/` (reuse these instead of re-improvising the same loops — they\nencode the safety discipline learned the hard way):\n\n- `bench_server.sh start|stop|status <ssh> <binary> [maxmem] [port]` — launch a\n server tracking its PID; **stop kills by PID and verifies the process is gone +\n port free** (tmux kill-session does not reap it).\n- `monitor_fill.sh <server_ssh> <client_ssh> [session] [abort_pct]` — the live\n RSS guard: polls `used_memory_rss`, prints a line per sample, aborts the write\n if RSS crosses the threshold (default 92%). Its output is also the input to\n `plot_fill.py`.\n- `capture_result.sh <server_ssh> <client_ssh> <log> <outdir> [label]` — saves\n the run summary + INFO memory + **OS-level `free` snapshot** (`host_memory.txt`)\n and exits non-zero if there were any eviction errors (enforces the zero-error\n rule). The OS snapshot matters: `used_memory_rss` and host free can diverge\n significantly — e.g. with `MIMALLOC_ALLOW_LARGE_OS_PAGES=1` (default), mimalloc\n holds large-page arenas that the OS counts as used but INFO doesn't fully reflect,\n making the machine appear near-OOM while RSS looks fine.\n- `collect_metrics.py` / `make_charts.py` — metrics + charts from dfly_bench\n JSON (GET/SET workloads).\n- `plot_fill.py` — memory-fill chart from `monitor_fill.sh` output, for\n `--command` workloads (e.g. SETEX) that produce no JSON time series.\n\nEvery benchmark output directory must include a `commands_audit.log`\nartifact with the exact command lines used for setup, server launch, writes,\nreads, captures, diagnostics, and chart/report generation. Raw data without the\ncommands that produced it is not reproducible.\n\n**Run-control scripts are mandatory for benchmark phases.** For Phase 5 runs,\nuse the bundled scripts as the default API:\n\n- start/stop/status servers with `scripts/bench_server.sh`;\n- monitor long write fills with `scripts/monitor_fill.sh`;\n- capture write results and enforce the zero-error rule with\n `scripts/capture_result.sh`;\n- generate metrics/charts with `scripts/collect_metrics.py`,\n `scripts/make_charts.py`, and `scripts/plot_fill.py`.\n\nInline SSH is fine for inspection, package setup, one-off diagnostics, and\nbounded foreground probes, but not as a replacement for the run-control scripts\nin the main benchmark path. If a script is not usable for a run, document the\nreason in `commands_audit.log` and in the report appendix before using a manual\nequivalent.\n\n## Phase 1 — Gather the benchmark spec (interactive)\n\nUse `AskUserQuestion` to pin down anything not already given. You need:\n\n1. **Where it runs.** Local, or remote instances? For remote, get the **server**\n host/IP and the **client** host/IP (the load generator should run on a\n separate machine from the server — co-locating them steals CPU and corrupts\n the numbers). Get the SSH user / key if not obvious. On cloud instances,\n **always use the private/internal IP** (VPC-internal) for both SSH and as the\n load generator's `-h` target — the public IP routes through the internet\n gateway, adding latency and egress cost, and the result measures the gateway\n rather than the server.\n2. **Use-cases.** Default menu: simple strings (SETEX writes + GET reads),\n pipelined / peak-throughput, memory efficiency. Always also ask whether there\n is a **specific focus** (a particular command, data type, key distribution,\n value size, or scenario) — the canned use-cases are a starting point, not a\n cage.\n3. **Comparison targets.** Dragonfly only, or also Valkey / Redis? Which\n versions?\n4. **Instance specs** (for the report's conditions section): instance type, CPU\n count, OS/kernel. Read these directly from the box at the start of Phase 2\n rather than asking the user.\n5. **Load parameters** if the user has opinions: value size (`-d`), key space\n (`--key_maximum`), connections (`-c`), pipeline depth, test duration. Pick\n sensible defaults (see the reference) and tell the user what you chose.\n **Sizing the key count is the consequential decision** — see the reference's\n \"Sizing the load\" section. The dataset must fill a large fraction of instance\n RAM (benchmarking a near-empty instance is wrong) but stay clear of OOM\n (~60–75% of RAM, headroom to spare). `--key_maximum` bounds the footprint, so\n compute it from instance RAM ÷ estimated bytes-per-entry, calibrating with a\n short pilot write if unsure.\n\nSummarize the resolved spec back to the user in a short block before running\nanything, so a wrong assumption gets caught before machines spin.\n\n## Phase 2 — Prepare the instances\n\n- Record instance specs (`nproc`, `uname -r`, cloud metadata) — these go in the\n report's conditions section. Use the **private IP** when connecting between\n instances (see Phase 1).\n- A remote instance usually already has `~/projects/dragonfly`. If not, clone it\n there. Build a **release** binary (`build-opt/dragonfly`, `build-opt/dfly_bench`)\n unless the user wants a published release downloaded instead — follow their\n instruction on build-vs-download.\n- For comparison targets, pull the Valkey/Redis docker image on the server box.\n- Verify everything launches and is reachable from the client box (a quick\n `redis-cli -h <server-private-ip> ping`) before committing to a long run.\n\nRun each step individually over SSH — do not chain launch, polling, capture, and\nthe next phase into one command. If a tool call is interrupted, remote work keeps\nrunning but goes uncaptured. Start the server; start the write; monitor it;\ncapture; start reads — one bounded step at a time.\n\n## Phase 3 — Network tuning (optional, remote only)\n\nHigh-throughput numbers on cloud hardware depend heavily on IRQ/SMP affinity\ntuning. If the user wants maximal numbers, apply the tuning described in the\nreference and **record exactly what you did** — it belongs in the appendix and\nmaterially affects the results. If skipped, say so in the report; untuned and\ntuned numbers are not comparable.\n\n## Phase 4 — Start the monitoring stack (server machine)\n\nStart the local Prometheus + Grafana stack on the **server** machine so you get\ncontinuous, fine-grained time-series (server CPU, memory, command rates, host\nmetrics) spanning the whole benchmark — richer than the load generator's own\noutput and the source of the most convincing report charts.\n\n```bash\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose up -d'\n# when benchmarking Valkey/Redis, also bring up the redis exporter:\nssh <server> 'cd ~/projects/dragonfly/tools/local/monitoring && docker compose --profile redis up -d'\n```\n\nPrometheus scrapes at 1s. **Always run servers on port 6380** (Dragonfly,\nValkey, and Redis alike). The bundled `prometheus.yml` ships pointing at `6379`,\nso before bringing the stack up, fix the scrape targets:\n\n```bash\nsed -i 's/:6379/:6380/g' tools/local/monitoring/prometheus/prometheus.yml\n```\n\n(This is a workaround for a misconfigured default — if you have repo write\naccess, commit the corrected file once so future runs start clean.)\n\nRestart Prometheus if it was already running, then verify targets are `UP` at\n`http://<server>:9090/targets` before starting the load — a silently-down scrape\njob means no data for that window.\n\nStart monitoring **before** the workloads and leave it running across all runs,\nso a single continuous window covers everything. See the reference's monitoring\nsection for what to read from Grafana (dashboards are pre-provisioned) and how to\npull panels into the report. Record the benchmark start/end timestamps — they\nlet you crop the Grafana window to exactly the run.\n\n## Phase 5 — Run the workloads\n\nFor each (use-case × datastore) combination, see the reference for the precise\ncommands. The shape is always: **write → memory snapshot → read**. Use the\nprivate IP as the load generator's `-h` target.\n\nPass `--json_out_file` to `dfly_bench` (and `--json-out-file` to memtier) when\nthe workload supports it — charts and headline numbers come from these files.\n`dfly_bench --command` workloads (e.g. `SETEX __key__ 10000 __data__`) do not\nproduce JSON; save full stdout and chart the `monitor_fill.sh` output instead.\n\nDrive the run with the bundled scripts: `bench_server.sh start`, write in a\nclient tmux session, `monitor_fill.sh` alongside (live RSS guard), then\n`capture_result.sh` to save results. `capture_result.sh` enforces the\n**zero-error rule** — any eviction/OOM error response makes the run invalid;\ndiscard it, cut `--key_maximum` ~10–15%, and re-run. See the preamble and the\nreference's \"Sizing the load\" section for the full rule.\n\n**Prefill writes must use `--pipeline 30`.** Without pipelining, the client\nstalls waiting for each reply and becomes the bottleneck before the server is\nsaturated — especially at high key counts. Use `--pipeline 30` for all write\n(fill) phases. Drop to `--pipeline 1` (or omit it) only for latency-honest\nread benchmarks where you want honest per-request numbers.\n\nInitialize `commands_audit.log` at the start and append each command as it runs\n(see the preamble for the requirement).\n\n## Phase 6 — Extract metrics, build charts, assemble the report\n\n1. **Headline metrics + memory.** Run the bundled extractor over the JSON files\n to pull QPS / P99 and compute bytes-per-entry (both `used_memory`- and\n `used_memory_rss`-based, per the user's choice to report both):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/collect_metrics.py \\\n --bench-json <run.json> [--bench-json <run2.json> ...] \\\n --info <info_memory.txt> --dbsize <N> \\\n --label \"Dragonfly m7g.2xlarge\"\n ```\n\n It prints a metrics block (and JSON) you can drop straight into the tables.\n\n2. **Charts.** Generate QPS-over-time PNGs from the time series (this is what\n visually shows Dragonfly's smooth throughput vs a competitor's degradation):\n\n ```bash\n python3 .claude/skills/benchmark/scripts/make_charts.py \\\n --series \"Dragonfly=df_write.json\" --series \"Valkey=valkey_write.json\" \\\n --metric qps --title \"Write throughput, m7g.2xlarge\" \\\n --out charts/write_m7g2xl.png\n ```\n\n Embed the PNGs in the report with relative paths. **For `--command` workloads\n (e.g. SETEX) there is no dfly_bench JSON** — use `plot_fill.py` instead to chart\n memory-fill over time from the `monitor_fill.sh` output:\n\n ```bash\n python3 .claude/skills/benchmark/scripts/plot_fill.py \\\n --series \"prebuilt=prebuilt/poll.txt\" --series \"release=release/poll.txt\" \\\n --interval 10 --ram-gib 30 --out charts/memory_fill.png\n ```\n\n3. **Report.** Follow `references/report-template.md` exactly. Write a TLDR with\n the 3–5 genuinely important takeaways (scaling behavior, memory efficiency,\n any hardware surprise), then per-instance tables, then the appendix. The TLDR\n is interpretation, not a restatement of the tables — say what the numbers\n *mean*. Only claim what the data supports.\n\nSave the report where the user asks (default: `benchmark-<YYYY>-<MM>.md` in the\nrepo root) and tell them the path plus a one-line summary of the headline\nresult.\n\n## Cleanup\n\nIf you started servers, the monitoring stack, or comparison containers on remote\ninstances, offer to stop them (`docker compose down` in the monitoring dir).\nIf you applied network tuning that changes machine state, tell the user what\nwould need reverting. Don't tear anything down without asking — they may want to\ninspect or re-run.\n\n<!-- File: .claude/skills/reproduce-fuzz-crash/SKILL.md (Tokens: ~1009 | Category: anthropic-skill) -->\n---\nname: reproduce-fuzz-crash\ndescription: >\n Reproduce AFL++ fuzz crashes from GitHub Actions. Use when user provides a\n GitHub Actions fuzz run URL and wants to reproduce and analyze the crash locally.\nargument-hint: <github-actions-run-url>\nallowed-tools: Bash, Read, Grep, Glob, Write\n---\n\n# Reproduce Fuzz Crash\n\nGiven a GitHub Actions fuzz run URL, download crash artifacts, triage them\nwith `fuzz/triage_crashes.sh`, and produce a crash analysis report.\n\n**Input**: `$ARGUMENTS` — a GitHub Actions run URL like:\n`https://github.com/dragonflydb/dragonfly/actions/runs/22906484769`\nor with query params like `?pr=6855`.\n\n## Workflow\n\n### Step 1: Parse the URL\n\nExtract `owner/repo` and `run_id` from the URL.\n\n```\nhttps://github.com/{owner}/{repo}/actions/runs/{run_id}[?...]\n```\n\nStrip any query parameters from `run_id`.\n\n### Step 2: Download artifacts\n\nList crash artifacts via the GitHub API, then download each as a `.zip` directly:\n\n**IMPORTANT**: Run each command as a separate Bash tool call (no `&&` chaining)\nto ensure auto-approval works with the user's permission patterns.\n\n```bash\n# List artifacts — filter for names containing \"crash\"\ngh api repos/{owner}/{repo}/actions/runs/{run_id}/artifacts\n\n# Create output directory\nmkdir -p /tmp/fuzz-repro-{run_id}\n\n# Download each crash artifact by ID (separate command)\ngh api repos/{owner}/{repo}/actions/artifacts/{artifact_id}/zip > /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nThis gives real `.zip` files that the triage script can consume directly.\n\nIf no crash artifacts are found, report that the run has no crash artifacts and stop.\n\nNote: there may be duplicate artifact names (same name, different IDs) from\nretried jobs. Download the **most recent** one (highest artifact ID).\n\n### Step 3: Determine mode\n\nInfer the protocol mode from the artifact name:\n- Contains \"memcache\" → `memcache`\n- Otherwise → `resp`\n\n### Step 4: Check Dragonfly binary\n\nCheck if the debug binary already exists and runs:\n\n```bash\n./build-dbg/dragonfly --version\n```\n\nOnly build if the binary doesn't exist or fails to run:\n\n```bash\nninja -C build-dbg dragonfly\n```\n\nIf `build-dbg` doesn't exist, run `./helio/blaze.sh` first.\n\n### Step 5: Run triage_crashes.sh\n\nFor each zip file, run:\n\n```bash\n./fuzz/triage_crashes.sh ./build-dbg/dragonfly <mode> /tmp/fuzz-repro-{run_id}/<artifact-name>.zip\n```\n\nCapture the full output.\n\n### Step 6: Analyze and report\n\nParse the triage output for confirmed crashes. For each confirmed crash:\n\n1. **Read the source** at the crash location — use the stack trace to identify\n the source file and line number, then read that code.\n2. **Provide analysis**: likely root cause, what to investigate.\n\nPrint a structured report:\n\n```\n## Fuzz Crash Report\n\n**Run**: {url}\n**Artifacts**: {number} crash(es) found\n\n---\n\n### Crash NNNNNN\n\n**Reproduced**: Yes / No (false positive)\n**Signal**: SIGABRT (6) / SIGSEGV (11) / etc.\n\n**Stack trace**:\n\\```\n<stack trace from triage output>\n\\```\n\n**Analysis**:\n<1-3 sentences explaining the likely root cause based on the stack trace,\nthe assertion message, and the crash input. Identify the source file and\nline number. Suggest what to investigate.>\n```\n\n## Important Notes\n\n- The triage script uses port **6379** (resp) or **11211** (memcache).\n Ensure no other Dragonfly or Redis instance is using these ports.\n- The script adds `--restricted_commands` flags to avoid false positives from\n commands like DEBUG SLEEP that the fuzzer might generate.\n- Some crashes are non-deterministic (thread timing). The script reports\n these as \"FALSE POSITIVE\" — note this clearly, it doesn't mean the bug\n is invalid, just that it didn't reproduce on this run.\n- The script handles its own cleanup of Dragonfly processes.\n- Do NOT delete `/tmp/fuzz-repro-{run_id}/` — the user may want to inspect it.\n- If `gh run download` fails with permissions, suggest the user authenticate\n with `gh auth login`.\n\n<!-- File: tools/packaging/osrepos/README.md (Tokens: ~588 | Category: plugin-manifest) -->\n# Package repositories for rpm and debian packages\n\nThis directory contains scripts and definitions for setting up YUM and apt repositories for Linux users to install\ndragonfly packages.\n\nThe repositories are served as static websites. The generate-site workflow is used to set up and deploy the sites using\nscripts and definitions included here.\n\nThe workflow does the following tasks:\n\n* Download the latest 5 releases from dragonfly releases page, specifically deb and rpm assets\n * for deb files, only the latest package is downloaded and present (see note below)\n* Set up a directory structure separating deb and rpm files into version specific paths\n* Sign the packages (see note on GPG)\n* Deploy the assets prepared, along with the public GPG key and repo definitions for apt and rpm tooling\n\n## Using the YUM repository\n\nAdd the repository using:\n\n```shell\nsudo dnf config-manager addrepo --from-repofile=https://packages.dragonflydb.io/dragonfly.repo\n```\n\nThen install dragonfly as usual, or a specific version:\n\n```shell\nsudo dnf -y install dragonfly-0:v1.33.1-1.fc30.x86_64\n```\n\n## Using the APT repository\n\nFirst download the public GPG key to an appropriate location:\n\n```shell\nsudo curl -Lo /usr/share/keyrings/dragonfly-keyring.public https://packages.dragonflydb.io/pgp-key.public\n```\n\nThen add the sources file:\n\n```shell\nsudo curl -Lo /etc/apt/sources.list.d/dragonfly.sources https://packages.dragonflydb.io/dragonfly.sources\n```\n\nFinally install dragonfly using apt\n\n```shell\nsudo apt update && sudo apt install dragonfly\n```\n\n#### Versions in APT repository\n\nUnlike the yum repo, the apt repo only has the latest version. The reason for this is the tool, `reprepro` supplied by\ndebian to build repositories only supports multiple\nversions in version 5.4 onwards, and the github runner using ubuntu-latest does not have this version.\n\nAnother option would be to use the components feature of apt repositories in the sources file we ask users to install,\nbut then the versions would need\nto be hardcoded in the sources file and the user would have\nto update the file with each new release which makes for a bad user experience. As of now users wanting older packages\nshould download them directly.\n\n### Signing packages\n\nThe packages are signed using the GPG key imported from the secret GPG_PRIVATE_KEY in this repository.\n\nThe corresponding public key is served with site assets, so the apt/yum/dnf based tooling can consume the public key to\nverify package integrity.\n\n### TODO\n\n- [X] debian packages signing (not required? release file is signed)\n- [X] debian repo metadata setup\n- [ ] tests asserting that packages are installable?\n\n<!-- File: tools/packaging/README.md (Tokens: ~435 | Category: plugin-manifest) -->\n# Installation Packages\n\n## Overview\nThis directory includes a set of files and scripts to build installation package for various Linux distributions.\n\n## Debian\nThe file to build the Debian package all located under \"debian\" directory.\nThe resulting package will install the binary of Dragonfly as well as generate a new service entry for dragonfly,\nthat can be controlled with \"systemctl\" command, to start, stop and check status of.\n### Building\nTo build the package, you have a script called \"generate_debian_package.sh\". This script accepts the following parameters:\n* Optional binary path - the location from which to take the binary for the installation. The default for this is \"repo path/build-opt\".\nThe location to which the resulting package is writing is at the location from which the script is executed.\nThis script is depends on the following packages:\n* git\n* moreutils\n* debhelper\n* dpkg-dev\n\nTo build:\n```\n/path/to/dragonfly/tools/packaging/generate_debian_package.sh [/path/to/dragonfly-binary-file]\n```\n\nThis can only be run on Debian based hosts.\nYou can use the flowing docker file to generate this package:\n```\nFROM ubuntu:20.04\nARG DEBIAN_FRONTEND=noninteractive\nRUN apt update -y && apt-get install -y gcc dpkg-dev gpg vim wget git moreutils debhelper\n```\nBuild the above docker and then run it with your dragonfly source code path mount as volume for the build:\n```\ndocker build -t ubuntu-package .\ndocker run --rm -ti -v /path/to/dragonfly-repo:/mydocker-path ubuntu-package bash\n```\nAgain note that you need to be at \"main\" branch to successfully build this package.\nNote: If at the end of the installation you see a message \"/usr/bin/deb-systemd-helper: error: systemctl preset failed on dragonfly.service: No such file or directory\",\nyou can ignore it, this seem to be related to [the following issue](https://groups.google.com/g/linux.debian.bugs.dist/c/m6xGZ82TdvM).\n\n<!-- File: tools/vector-benches/README.md (Tokens: ~265 | Category: plugin-manifest) -->\nTool for benchmarking vector search with randomized vectors.\n\n## Logic\nThe tool connects to the Redis/Dragonfly instance.\n1. Checks if the database has enough data (at least 50% of requested `-n`). If not, it **flushes the DB** and generates random vectors.\n2. Checks if the index `idx` exists. If not, it creates it.\n3. Runs concurrent search queries and reports latency/QPS.\n\n## Arguments\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `-n` | 50000 | **Number of vectors** to populate if the DB is empty. |\n| `-q` | 1000 | **Total number of queries** to run during the benchmark. |\n| `-t` | 8 | **Query threads**. Number of concurrent workers sending queries. |\n| `-d` | 100 | **Vector dimension**. Size of the float32 vectors. |\n| `-k` | 10 | **Top K**. Number of nearest neighbors to retrieve per query. |\n| `-p` | 6379 | **Port** of the server. |\n| `-h` | localhost | **Host** of the server. |\n\nRun with `-h` (help) to see these defaults in the tool itself.\n</yakaai_skills>"}