harlequin Documentation

Technical Documentation: tconbeer/harlequin

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Harlequin

The SQL IDE for Your Terminal.

[!TIP]
This README contains a small subset of the docs available at
harlequin.sh.

Installing Harlequin Copy Chapter

Harlequin is a Python program, and there are many ways to install and run it. We strongly recommend using uv:

1. Install uv. From a POSIX shell, run:

   bash Copy

   curl -LsSf https://astral.sh/uv/install.sh | sh

   Or using Windows Powershell:

   powershell Copy

   powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

2. Install Harlequin as a tool using uv:

   bash Copy

   uv tool install harlequin

   This command will install Harlequin into an isolated environment and add it to your PATH so you can easily run the executable.

Other Installation Methods Copy Section

Alternatively, if you know what you're doing, after installing Python 3.9 or above, install Harlequin using pip, pipx, poetry, or any other program that can install Python packages from PyPI:

pip install harlequin

There is also a Homebrew formula for Harlequin, although this is maintained by the community and is not as rigorously tested as the Python installations. Note that the formula includes several Harlequin adapter packages (Postgres, MySQL/MariaDB, and ODBC) and their dependencies, which is convenient but increases the application size.

brew install harlequin

Installing Database Adapters Copy Chapter

Harlequin can connect to dozens of databases using adapter plug-ins. Adapters are distributed as their own Python packages that need to be installed into the same environment as Harlequin.

For a list of known adapters provided either by the Harlequin maintainers or the broader community, see the adapters page.

The adapter docs also include installation instructions. Some adapters can be installed as Harlequin extras, like postgres. If you used uv to install Harlequin:

uv tool install 'harlequin[postgres]'

You can install multiple extras:

uv tool install 'harlequin[postgres,mysql,s3]'

Running Harlequin Copy Chapter

Once Harlequin is installed, you run it from the command line. The arguments and options you pass in at the command line affect Harlequin's behavior, like what database adapter it uses, which database it connects to, whether or not the file picker is visible, and more. Assuming you have installed Harlequin so that it is on your PATH (uv tool install harlequin does this automatically), you run Harlequin by typing a command of this form into your shell:

harlequin [OPTIONS] [CONN_STR]

where [OPTIONS] is 0 or more pairs of the form --[option-name] [option-value], and [CONN_STR] is 0 or more connection strings. [OPTIONS] are composed of both Harlequin options and adapter options. For a full list of options, run Harlequin with the --help option:

harlequin --help

Using Harlequin with DuckDB Copy Chapter

Harlequin defaults to using its DuckDB database adapter, which ships with Harlequin and includes the full DuckDB in-process database.

To open an in-memory DuckDB session, run Harlequin with no arguments:

harlequin

To open one or more DuckDB database files, pass in relative or absolute paths as connection strings (Harlequin will create DuckDB databases if they do not exist):

harlequin "path/to/duck.db" "another_duck.db"

Using Harlequin with SQLite and Other Adapters Copy Chapter

Harlequin also ships with a SQLite3 adapter. To use that adapter, you specify the --adapter sqlite option. Like DuckDB, you can open an in-memory SQLite database by omitting the connection string:

harlequin --adapter sqlite

You can open one or more SQLite database files by passing in their paths as connection strings; note that the --adapter option has a short alias, -a:

harlequin -a sqlite "path/to/sqlite.db" "another_sqlite.db"

Other adapters can be installed as plug-ins; for more information, see the installation guide, and the guides for individual adapters. Each adapter can define its own options, which you can view using harlequin --help.

Configuring Harlequin Copy Chapter

Harlequin contains a large number of options that allow you to set the theme, customize key bindings, show remote and local files, set the locale for number formatting, and much more. These can always be entered at the command line, but it can be convenient to define a configuration as a profile instead. For more information on configuring Harlequin, see Using Config Files.

A profile's string values can name environment variables — password = "${MYPASSWORD}", or ${MYHOST:-localhost} to supply a default — so a config file your team shares holds no credentials, and values an adapter declares as secrets are masked wherever Harlequin prints them. hsql, Harlequin's companion CLI (below), reads the same files and can report on them, with hsql --config show, --config validate and --config schema.

Using Harlequin with Agents or in Scripts Copy Chapter

Harlequin has a companion CLI, hsql, that uses the same adapters and config files, but is optimized for headless use by agents and in automations. hsql is packaged with Harlequin (no additional installation required):

$ hsql -P dev -c "select * from users"
 id | name
----+---------
 1  | Ted
 2  | Patrick
(2 rows)

hsql can also explore the catalog without writing SQL: hsql --catalog --path mydb.analytics lists a schema's relations (and --path mydb.analytics.orders lists that table's columns), while hsql --catalog-search customer_id searches every level at once.

To bound what an agent can do, --read-only connects in a mode the database refuses writes in, and hsql --timeout 30 cancels a run that takes longer than that. Both are also profile keys, and both refuse to start if the adapter cannot enforce them. Harlequin takes --read-only too.

For more information on hsql, see the getting started docs.

Using Harlequin with Django Copy Chapter

django-harlequin provides a command to launch Harlequin using Django’s database configuration, like:

./manage.py harlequin

Keep Reading at [harlequin.sh](https://harlequin.sh/docs/getting-started/usage) Copy Chapter

Visit harlequin.sh for an overview of features and full documentation, starting with a guided walkthrough of how to edit and execute queries, use the data catalog, export data, and more.

Getting Help Copy Chapter

To view all command-line options for Harlequin and all installed adapters, after installation, simply type:

harlequin --help

To view a subset of these docs (and a link back here) from within the app, press F1.

See the Troubleshooting guide for help with key bindings, appearance issues, copy-paste, etc.

GitHub Discussions are a good place to ask questions, request features, and say hello.

GitHub Issues are the best place to report bugs.

Sponsoring Harlequin Copy Chapter

Please consider sponsoring Harlequin's author, so he can continue to dedicate time to Harlequin.

Contributing Copy Chapter

Thanks for your interest in Harlequin! Harlequin is primarily maintained by Ted Conbeer, but he welcomes all contributions!

Please see CONTRIBUTING.md for more information.

Overview (`packaging/hsql/README.md`) Copy Section

hsql

hsql is your agent's favorite SQL client. It's the headless CLI for
Harlequin, and shares the same config and query engine,
with an interface optimized for agents, scripts, and automations.

[!TIP]
This README contains a small subset of the docs available at
harlequin.sh.

If you already use Harlequin and know about adapters and config files, jump ahead to Running hsql.

Installing hsql Copy Chapter

hsql is packaged with Harlequin, so if you already use Harlequin,
hsql is already installed.

Otherwise, you can install hsql directly. hsql is a Python program, and there are many ways to install and run it. We strongly recommend using uv:

1. Install uv. From a POSIX shell, run:

   bash Copy

   curl -LsSf https://astral.sh/uv/install.sh | sh

   Or using Windows Powershell:

   powershell Copy

   powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

2. Install hsql as a tool using uv:

   bash Copy

   uv tool install hsql

   This command will install hsql into an isolated environment and add it to your PATH so you can easily run the executable.

Installing Database Adapters Copy Chapter

hsql can connect to dozens of databases using adapter plug-ins. Adapters are distributed as their own Python packages that need to be installed into the same environment as hsql and harlequin.

For a list of known adapters provided either by the Harlequin maintainers or the broader community, see the adapters page.

The adapter docs also include installation instructions (installing an adapter with Harlequin also installs it for hsql). Some adapters can be installed as Harlequin extras, like postgres. If you used uv to install hsql directly, you can add adapter packages using --with:

uv tool install hsql --with harlequin-postgres

You can install multiple extras:

uv tool install hsql --with harlequin-postgres --with harlequin-mysql

Running hsql Copy Chapter

Once hsql is installed, you run it from the command line. If you have used psql or the duckdb CLI, hsql will feel familiar, but hsql has the major advantage that is works with most databases and provides the same interface and produces the same output, regardless of the connected database. This means you (and your agent) can learn one tool, instead of several. In your shell, all hsql commands take the same form:

hsql [OPTIONS] [CONN_STR]

where [OPTIONS] is 0 or more pairs of the form --[option-name] [option-value], and [CONN_STR] is 0 or more connection strings. [OPTIONS] are composed of both hsql options and adapter options. For a full list of options, run hsql with the --help option:

hsql --help

Using hsql with DuckDB Copy Chapter

hsql defaults to using its DuckDB database adapter, which ships with hsql and includes the full DuckDB in-process database.

Run a query against an in-memory DuckDB session, run hsql and pass in a query with the -c option:

$ hsql -c "select 1"
 1
---
 1
(1 row)

To query one or more DuckDB database files, pass in relative or absolute paths as connection strings (hsql will create DuckDB databases if they do not exist):

$ hsql "path/to/duck.db" -tAc "select count(*) from orders"
42

Using hsql with SQLite and Other Adapters Copy Chapter

hsql also ships with a SQLite3 adapter. To use that adapter, you specify the --adapter sqlite option. Like DuckDB, you can open an in-memory SQLite database by omitting the connection string:

$ hsql --adapter sqlite -c "select 'Ted' as author"
 author
--------
 Ted
(1 row)

You can query one or more SQLite database files by passing in their paths as connection strings; note that the --adapter option has a short alias, -a:

$ hsql -a sqlite "path/to/sqlite.db" -c "select * from users"
 id | name
----+---------
 1  | Ted
 2  | Patrick
(2 rows)

Other adapters work the same way; for example, Postgres:

$ hsql -a postgres "postgresql://example.com/postgres:5432" -c "select * from invoices"

[!TIP]
You should use Profiles to keep credentials out of your shell
history. For more information, keep reading or see
the docs on
config files.

Configuring hsql and Using Profiles Copy Chapter

hsql supports a number of options for setting the query limit, configuring output formats, and defining connection parameters. Options can be passed as command-line flags, or read from config files. Config files store configurations under separate profiles, so you can easily switch between databases by reading from different profiles with the -P option:

$ hsql -P prod -c "select count(*) from orders" --csv
$ hsql -P dev -c "select * from users" --vertical --limit 5
$ hsql -P warehouse -c "..." --format parquet -o invoices.pq

hsql reads the same config files as Harlequin, and merges them one profile at a time, nearest file first; pass --config-path PATH to read a single file instead, or -P None to skip them entirely.

Keeping Credentials Out of Config Files Copy Section

A profile's string values can name environment variables, so a config file your team shares — and commits — holds no credentials:

[profiles.prod]
adapter = "postgres"
host = "${MYHOST:-localhost}"  # ${VAR:-default} supplies a default
password = "${MYPASSWORD}"     # ${VAR} is required; hsql exits 2 if it is unset

Write $${ for a literal ${, if a value really does start with one.

Values an adapter declares as secrets are masked wherever hsql prints them — --config show, --info, and error messages — as is the password inside a connection string.

Inspecting and Writing Config Files Copy Section

Five --config modes work on your config files instead of running SQL, and none of them connects to a database:

• list-profiles — the names you can pass to -P, with each one's adapter, and which is the default.
• show — the merged config as TOML, with the file each value came from beside it; --json for JSON.
• validate — every problem in every discovered file, exiting 2 if it finds any.
• schema — a JSON Schema for a config file, covering every adapter you have installed; point your editor at it for completion and validation as you type.
• init — a profile written from the options you pass, prompting for nothing: hsql --config init -P prod -a sqlite ./my.db --limit -1 writes [profiles.prod] into the nearest config file.

$ hsql --config show
default_profile = "dev" # from /home/user/.config/harlequin/harlequin.toml

[profiles.dev] # from /home/user/proj/harlequin.toml, overriding /home/user/.config/harlequin/harlequin.toml
adapter = "postgres"
host = "${MYHOST:-localhost}"
password = "********"
limit = 100

list-profiles and validate are result sets, so --csv, -t, -A and -o work on them as they do on a query.

Data Layouts and File Formats Copy Chapter

hsql supports all of the following formats for displaying and writing data:

• table
• markdown (alias: md)
• vertical
• csv
• tsv
• json
• jsonl (alias: ndjson)
• parquet
• orc
• feather (alias: arrow)
• none (suppresses output)

You can select a format with the --format <name> or using the shorthand --<name>, so these are equivalent: --format csv, --csv.

Some layouts can present the results from multiple queries. Others will raise an error and exit with code 2 if multiple queries are executed.

Additionally, for any layout, pass --stats to print summary info as JSON to stderr:

$ hsql -c "select 1" --format none  --stats
{"status":"ok","statements":1,"rows":1,"truncated":false,"limit":500,"elapsed_ms":1,"columns":[{"name":"1","type":"#"}]}

Exploring the Catalog Copy Chapter

Before you can write a query, you have to know what is in the database. --catalog lists the objects one level below --path, and exits without running SQL:

$ hsql "path/to/duck.db" --catalog
 path | name | query_name | type     | type_label
------+------+------------+----------+------------
 duck | duck | "duck"     | database | db
(1 row)

Every row's path is what lists that object's own children, so you walk down one level at a time:

$ hsql "path/to/duck.db" --catalog --path duck
 path           | name      | query_name         | type   | type_label
----------------+-----------+--------------------+--------+------------
 duck.analytics | analytics | "duck"."analytics" | schema | sch
 duck.main      | main      | "duck"."main"      | schema | sch
(2 rows)

$ hsql "path/to/duck.db" --catalog --path duck.analytics
 path                        | name         | query_name                 | type       | type_label
-----------------------------+--------------+----------------------------+------------+------------
 duck.analytics.customers    | customers    | "analytics"."customers"    | BASE TABLE | t
 duck.analytics.order_totals | order_totals | "analytics"."order_totals" | VIEW       | v
 duck.analytics.orders       | orders       | "analytics"."orders"       | BASE TABLE | t
(3 rows)

$ hsql "path/to/duck.db" --catalog --path duck.analytics.orders
 path                              | name        | query_name    | type          | type_label
-----------------------------------+-------------+---------------+---------------+------------
 duck.analytics.orders.customer_id | customer_id | "customer_id" | BIGINT        | ##
 duck.analytics.orders.id          | id          | "id"          | BIGINT        | ##
 duck.analytics.orders.placed_at   | placed_at   | "placed_at"   | TIMESTAMP     | ts
 duck.analytics.orders.total       | total       | "total"       | DECIMAL(18,2) | #.#
(4 rows)

A listing is rows, so every format and output option applies to it, exactly as they do to a query:

$ hsql "path/to/duck.db" --catalog --path duck.analytics -tA --csv
duck.analytics.customers,customers,"""analytics"".""customers""",BASE TABLE,t
duck.analytics.order_totals,order_totals,"""analytics"".""order_totals""",VIEW,v
duck.analytics.orders,orders,"""analytics"".""orders""",BASE TABLE,t

Searching the Catalog Copy Section

Walking is the wrong tool for where does orders live and which tables have a customer_id. --catalog-search TERM searches every level of the catalog at once, for objects whose name contains TERM:

$ hsql "path/to/duck.db" --catalog-search customer_id
 path                                    | name        | query_name    | type   | type_label
-----------------------------------------+-------------+---------------+--------+------------
 duck.analytics.order_totals.customer_id | customer_id | "customer_id" | BIGINT | ##
 duck.analytics.orders.customer_id       | customer_id | "customer_id" | BIGINT | ##
 duck.main.staging_events.customer_id    | customer_id | "customer_id" | BIGINT | ##
(3 rows)

--path narrows the search to one subtree, and the output can be laid out like any query:

$ hsql "path/to/duck.db" --catalog-search order --path duck.analytics -tA
duck.analytics.order_totals|order_totals|"analytics"."order_totals"|VIEW|v
duck.analytics.orders|orders|"analytics"."orders"|BASE TABLE|t

Note: not all adapters support catalog search at this time.

Scripting with hsql Copy Chapter

[!WARNING]
To make hsql safe and efficient for agents, by default hsql applies a 500-row
limit to all queries. To remove this limit, use --limit -1 or set
limit = -1 in your profile. If limits truncate data, hsql will print
a warning on stderr; we recommend that you do NOT suppress or redirect
that message so do NOT use hsql with 2>/dev/null.

hsql can write data to files, either with the -o option or by piping output (hsql only writes data to stdout; other messages go to stderr):

$ hsql -P prod --limit -1 -c "select * from users" --format parquet -o "users.pq"
$ hsql -P prod --limit -1 -c "select * from users" --csv > users.csv

hsql can execute multiple statements in one invocation, and supports several methods for doing so:

• Pass -c multiple times
• Include multiple queries, separated by ;, in one -c option
• Pass one or more .sql files with -f, with multiple statements in each
• Use --result to define which queries output data to stdout
• Use --on-error to either stop or continue if one or more queries produces an error.

In other words, this works:

$ hsql -P prod --limit -1 --format md --result all --on-error stop \
    -f ./setup.sql \
    -c "select count(*) from raw_table" \
    -f ./build-models.sql \
    -c "select count(*) from modeled_table"

hsql's exit codes are meaningful and stable:

• 0: Success
• 1: Query error
• 2: Usage/config error
• 3: Connection error
• 4: Timeout
• 130: Interrupted

You can also use --stats and jq together to error on a truncated query:

hsql --limit -1 -c "select 1" --csv -o data.csv --stats 2>&1 | jq -e '.truncated | not' > /dev/null

[!TIP]
A script or agent that runs many invocations against the same database can
hold one connection open instead of reconnecting for each one.
hsql --serve dev -P dev runs a session named dev, and
hsql --session dev -c "..." (or HSQL_SESSION=dev) sends invocations to
it, which answer in milliseconds. A session is state rather than a cache:
temp tables, settings and open transactions survive from one invocation to
the next. POSIX only; see
warm sessions.

Running Safely Copy Chapter

Two options bound what an invocation can do, and hsql refuses to run at all if the adapter cannot enforce them.

--read-only (or -r) connects in a mode the database itself refuses writes in:

$ hsql -r "path/to/duck.db" -c "insert into orders values (1)"
hsql: error: Invalid Input Error: Cannot execute statement of type "INSERT" on database "duck" which is attached in read-only mode!

--timeout SECONDS limits the duration of executing and fetching a query, and exits 4 when it runs out:

$ hsql --timeout 0.5 -c "select count(*) from range(100000000000) t(i)"
hsql: error: timed out after 0.5s

Both are also profile keys:

[profiles.agent]
adapter = "postgres"
read_only = true
timeout = 30

hsql refuses to open a connection with an adapter that cannot enforce --read-only or --timeout.

hsql --info reports implements_read_only and implements_cancel for every installed adapter.

Describing hsql to an Agent Copy Chapter

hsql --help is written for a person; two flags answer the same questions as JSON, so an agent can find its way around an installation it has never seen. --spec is a machine-readable --help: every option on the command, plus the connection options every installed adapter declares, each with its type, choices, default and help text. --info describes the installation instead — versions, platform, the config files hsql found, the profile that would be active, and what each installed adapter declares it supports — so a script can check that an adapter is there, and what it can do, before depending on it.

$ hsql --info -a sqlite | jq '.adapters.sqlite'
{
  "distribution": "harlequin",
  "version": "2.10.0",
  "capabilities": {
    "implements_cancel": true,
    "implements_catalog_search": true,
    "implements_read_only": true,
    "implements_validate_sql": false
  },
  "error": null
}

Neither connects to a database, both write JSON to stdout, and -a NAME narrows either one to a single adapter, which is much faster than importing them all.

Installing the Agent Skill Copy Chapter

hsql ships an Agent Skill that teaches an agent how to drive it: ask before you assume, keep credentials off the command line, read the catalog before writing SQL, consider a warm session, watch the 500-row limit, branch on the exit code, and hand off to harlequin when a human should drive. hsql --skill installs it, with no network and matched to the version you have:

$ hsql --skill -o ~/.claude/skills/hsql/          # for you, in every project
$ hsql --skill -o .claude/skills/hsql/            # for this repo, committed with it
note: wrote 6 files to .claude/skills/hsql: SKILL.md, references/config.md, references/queries.md, references/scripting.md, references/sessions.md, references/troubleshooting.md

With no -o, hsql --skill writes SKILL.md to stdout, so you can read it before you install it.

In Claude Code, the same skill is a plugin in Harlequin's repository:

/plugin marketplace add tconbeer/harlequin
/plugin install hsql@harlequin

Keep Reading at [harlequin.sh](https://harlequin.sh/docs/getting-started/hsql) Copy Chapter

Visit harlequin.sh for an overview of features and full documentation.

Getting Help Copy Chapter

To view all command-line options for Harlequin and all installed adapters, after installation, simply type:

hsql --help

GitHub Discussions are a good place to ask questions, request features, and say hello.

GitHub Issues are the best place to report bugs.

Sponsoring Harlequin and hsql Copy Chapter

Please consider sponsoring Harlequin's author, so he can continue to dedicate time to hsql.

Contributing Copy Chapter

Thanks for your interest in Harlequin! Harlequin and hsql are primarily maintained by Ted Conbeer, but he welcomes all contributions!

Please see CONTRIBUTING.md for more information.

Differences from psql Copy Chapter

-c, -f, -t, -A and -x mean what they mean in psql, so hsql -tAc "select count(*) from orders" prints a bare number, exactly as it would there. These do not:

Overview (`tests/skill_evals/README.md`) Copy Section

Skill trigger evals

trigger_queries.json is the eval set for the hsql skill's description, which is the
only part of a skill always in context and the whole of what decides whether it loads.

Run it with the description-optimization loop in the skill-creator skill, which needs
the claude CLI. It is a pre-merge exercise on a change to the description, not a CI
check: scoring it costs model calls, and the answer is a rate rather than a pass or fail.

Ten positives and ten negatives, split train/test so a description tuned on one half is
scored on turns it was never tuned against. The positives include turns that never say
"hsql" or "harlequin", because under-triggering is the failure mode; the negatives are
near misses — SQL-shaped work that is not a query, and database-shaped work that belongs
to another tool.

It lives here rather than beside the skill because
src/harlequin/hsql/skill/ is what ships in the wheel, and is the directory a
marketplace entry points at as a plugin.

Last run Copy Chapter

2026-08-28, against the description as it ships: 10/10 positives and 10/10 negatives on
both splits. The description was not tuned against this set — it scored at ceiling on the
first pass — so the set's value now is as a regression check on the next edit to the
description, and it will need harder near misses before it can distinguish one good
description from another.

Overview (`tests/data/unit_tests/completions/README.md`) Copy Section

The completions were generated from this database:

from sklearn.datasets import load_iris
import pandas as pd
import duckdb

# Load the iris dataset
iris = load_iris()
iris_df = pd.DataFrame(data=iris.data, columns=iris.feature_names)
iris_df["target"] = iris.target
iris_df["species"] = pd.Categorical.from_codes(iris.target, iris.target_names)

# Connect to DuckDB (this will create a new database if it doesn't exist)
con = duckdb.connect("iris.db")

# Create and insert data into the table
con.execute("""
    CREATE TABLE IF NOT EXISTS iris (
        sepal_length FLOAT,
        sepal_width FLOAT,
        petal_length FLOAT,
        petal_width FLOAT,
        target INTEGER,
        species VARCHAR
    )
""")

# Insert the data
con.execute("INSERT INTO iris SELECT * FROM iris_df")

# Verify the data (optional)
result = con.execute("SELECT * FROM iris LIMIT 5").fetchall()
print("First 5 rows:", result)

# Close the connection
con.close()

Essentially the database was created and then sunk the data in the completions
to json using dataclasses.asdict and json.dumps. Proceeded with some manual
deletion to make the file smaller.

Cli Options (`docs/cli-options.md`) Copy Section

CLI options: the groups, and how to add one

Harlequin and hsql share their option machinery, and hsql additionally
sorts every option into a group. This document is the reference for that,
and the checklist for adding an option.

Where an option comes from Copy Chapter

There are two kinds, and they are declared in different places.

An adapter's options are AbstractOption subclasses in the adapter's own
ADAPTER_OPTIONS list. Declaring one gets all three renderings —
to_click() for the CLI, to_widgets() for the TUI, to_questionary() for
the config wizard — and both commands attach them dynamically once the first
pass has named the adapter. Nothing in core lists them.

A command's own options are CoreOption declarations in
harlequin/core_options.py, one list for both commands. Each says which
command takes it (harlequin=On(...), hsql=On(...), or both), how that
command spells it, and the flags that decide where its value may come from —
group, cli_only, and first_pass, which is whether the pass that names
the adapter reads it before there is a command to parse with.
attach_core_options() builds the click options from that list, in the order
it declares them, which is the order hsql --help lists them; the part only a
command can answer — a click.Choice of what is installed, a callback, help
naming a computed default — stands in the declaration as a RuntimeValue. These are the
frozen part of each command's surface; an adapter option whose spelling
collides with one loses that spelling (first_pass.attach_adapter_options).

Every option that is not CONN_STR is also a profile key, under the name
click gives it (--read-only is read_only). config.py validates a profile
against the command's parameters plus the adapter's declared options, so a key
neither declares is an error.

The five groups Copy Chapter

hsql runs in two roles — a server (--serve NAME) and a client
(--session NAME) — and an option belongs to exactly one group, decided by
when its value is read. An option hsql declares cannot be declared without
one; the frozensets hsql refuses against are derived from the declarations.

Two of these need a word.

Group.CONFIG (-P, --config-path) names where values come from rather
than being values, so which group a profile belongs to is decided by what it
holds. A profile of nothing but format and limit is per-request whichever
way it was named; one that names a database is compared like a typed flag.

Every adapter option is a connection option. connection_option_names()
is the join: CONNECTION_OPTIONS plus the adapter's declared names. It is a
positive test — a key that is neither hsql's nor the adapter's, such as one
of TUI_ONLY_KEYS or a misspelling in a profile, describes no connection and
is not compared.

Adding an option Copy Chapter

1. Decide whose it is. If it changes how a database is reached, it is an
   adapter option — declare it in that adapter's ADAPTER_OPTIONS and stop.
   Everything below is for a core option on harlequin or hsql.

2. Write the CoreOption in harlequin/core_options.py, among the ones it
   belongs with — where it sits in that list is where hsql --help lists it.
   Give it harlequin=On(), hsql=On(), or both, and put in kwargs what
   both commands share; an On's own kwargs merge over those, and its
   decls replace them, which is for a difference the two commands really
   have. For hsql, keep the help one or two sentences and name the default.

3. Give it a group, by when its value is read:

   • once, at connect time → Group.CONNECTION
   • on each invocation, including every mode → Group.PER_REQUEST
   • once, for a server that is up → Group.SERVER

   An option hsql takes has to have one, and an option only the IDE takes has
   to have none —
   tests/unit_tests/test_hsql_serve.py::test_every_option_is_in_exactly_one_group
   pins the groups to the command they are refused against.

4. If it is a mode — one that reports rather than running SQL — also add
   it to _one_mode(), so two modes in one invocation are refused. A mode that
   reports on the installation or the config files declares first_pass=True
   and goes in the needs_profile / needs_adapter predicates in
   build_cli() too, so the first pass reads no profile for it; a mode that is
   configured like a run does not (--catalog, --history).

5. If a config file may not set it, mark it cli_only=True. That is for
   options whose value decides which process runs the invocation, or which a
   config file discovered in the working directory should not be able to
   weaken; CLI_ONLY_SESSION_KEYS and CLI_ONLY_SSH_KEYS, which are what
   refuse it, are derived from the flag.

6. An option the IDE reads and hsql must not is one with no hsql=:
   TUI_ONLY_KEYS is every declaration harlequin takes and hsql does not.

7. Regenerate the committed artifacts, which are pinned by tests:

   bash Copy

   uv run python scripts/write_config_schema.py
   uv run python scripts/write_cli_reference.py

8. Add a CHANGELOG.md entry under [Unreleased] if the option is
   user-facing.

What the partition test still catches Copy Chapter

Group membership travels with the declaration, so the two cannot drift. What
the test reads is the built command against the derived sets, which is what
catches an option added to harlequin/hsql/cli.py as a bare @click.option
rather than declared — it would reach the command in no group at all, and
nobody would refuse it. It cannot catch a profile key that no command
declares; that class is handled instead by connection_option_names() being a
positive test.

Generated Hsql Reference (`docs/generated/hsql-reference.md`) Copy Section

hsql

Execute SQL and exit.

CONN_STR: one or more connection strings, or paths to local db files.

Generated from the bare hsql command, with no adapters loaded. An adapter
contributes its own options on top of these: hsql --help -a NAME shows one
adapter's, and hsql --spec reports every installed adapter's as JSON.

Usage Copy Chapter

hsql [OPTIONS] [CONN_STR]...

Arguments Copy Chapter

Options Copy Chapter

Alphabetical by name. Flags are off by default.

-a/--adapter names an installed adapter plug-in. hsql --info reports
adapters installed in your environment.

Formats Copy Chapter

--format takes one of these names, which affect the layout of data in stdout.
Modify output further with options above (e.g., -t, -A, --display-rows).
none runs the SQL but generates no output. The suffix is what -o DIRECTORY
names a file with.

Exit codes Copy Chapter

Plans Crash Recovery Design (`docs/plans/crash-recovery-design.md`) Copy Section

Crash recovery and bug reporting

The design behind #687: never dump a raw
traceback at a user, preserve their work when something goes wrong, and make the report they send us
worth reading.

The problem Copy Chapter

An uncaught exception on Harlequin's message pump falls straight through AppBase._handle_exception
(app_base.py:68) into Textual's _fatal_error(), which renders a rich.traceback.Traceback
with show_locals=True and prints it to stderr. That is intimidating, it tells the user nothing
about what to do next, and it puts local variables — connection strings, tokens, query results — on
the terminal. Replacing it is a privacy fix as much as a UX one.

hsql has the same hole from the other end. main() (hsql/__init__.py) catches ClickException,
Abort and KeyboardInterrupt; anything else escapes to Python's own handler, which prints a
traceback and exits 1 — and 1 is ExitCode.QUERY, so a bug in Harlequin is indistinguishable from
"the database rejected the SQL." A caller scripting against hsql's exit codes cannot tell the two
apart.

Three problems compound it:

• The user's work is lost. write_editor_cache is called from exactly one place,
  Harlequin.action_quit() (app.py:970). A crash, a kill -9, a closed terminal or a dropped ssh
  session loses every open buffer — as does the self.exit(return_code=2) path.
• The reports we get are thin. bug_report.md asks for version, adapter, shell, terminal, OS,
  locale and install method as free text, so most reports arrive missing half of it.
• A crash at startup could repeat forever. #745
  crashed inside EditorCollection.on_mount while replaying cached buffers. Saving buffers on
  crash without a guard turns a one-time crash into a crash loop, which is worse than what we have
  now.

Outcome: a crash exits cleanly with a short panel naming a crash report file; the user's buffers
survive and come back on the next launch; a crash while restoring those buffers cannot repeat; and
the report drops into an issue template built to receive it.

The two ideas that carry it Copy Chapter

A per-process recovery file, separate from the shared cache. Checkpoints go to
recovery-.pickle, never to cache-1.pickle. Two Harlequins on one machine keep today's
comprehensible "last clean quit wins" semantics instead of thrashing a shared file every minute, and
a checkpoint can never overwrite what another process wrote when it quit cleanly.

Rename before replay. os.replace the recovery file to <name>.replayed before unpickling and
replaying it. If unpickling or replaying crashes, the next start finds nothing to replay. That is the
whole crash-loop guard — one line, no session sentinel, no PID-liveness check, and correct across
concurrent processes by construction.

Where the crash message goes. Textual's panic() calls _close_messages_no_wait(), so the
message pump is gone the moment we handle the exception: an in-app modal is unreachable from
_handle_exception, and attempting one re-enters an app that has already proven it is broken. The
message is a rich Panel at _exit_renderables[0], printed to stderr after the terminal is
restored — the same mechanism AppBase.__init__ already uses for a bad theme.

`editor_cache.py` Copy Chapter

CHECKPOINT_INTERVAL_SECONDS = 60.0
RECOVERY_STALE_SECONDS = 3 * CHECKPOINT_INTERVAL_SECONDS

def write_cache(cache: Cache) -> bool          # now atomic, and never raises
def get_recovery_file() -> Path                # cache dir / f"recovery-{os.getpid()}.pickle"
def write_recovery(cache: Cache) -> bool
def clear_recovery() -> None
def adopt_recovery() -> tuple[Cache | None, Path | None]

CACHE_VERSION stays 1 — the pickled shape does not change.

• Atomic write. Both writers go through _write_pickle(cache, path): a temp file in the same
  directory (carrying the pid, so a killed process leaves at most one bounded stray),
  flush() + os.fsync(), then os.replace(). Same-directory means a same-volume rename, atomic on
  POSIX and on Windows. The fsync is what stops a power loss leaving a zero-length file.
• Never raises. Returns False on OSError / pickle.PicklingError. Both callers run where an
  exception would itself take the app down. On Windows os.replace raises PermissionError when
  another process holds the destination open, so a checkpoint has to be allowed to skip silently.
• adopt_recovery() globs recovered-*.pickle (written by the crash handler) newest first, then
  recovery-*.pickle whose mtime is older than RECOVERY_STALE_SECONDS — that age test is how a
  live sibling process's file is left alone, without a PID-liveness check that would be wrong on
  Windows and wrong under PID reuse. It renames the chosen file to .replayed before unpickling
  it, and returns (cache, replayed_path) so the app can say where the buffers came from.
  Everything wrapped → (None, None).

Derive get_recovery_file() from the same user_cache_dir call as get_cache_file(), so
tests/functional_tests/test_cache.py:43's existing monkeypatch covers the new files for free.

Checkpointing (`app.py`) Copy Chapter

• on_mount: self.set_interval(CHECKPOINT_INTERVAL_SECONDS, self._checkpoint_editor_cache).
• _checkpoint_editor_cache():
  • bail when getattr(self, "editor_collection", None) is None or not mounted — the collection is
    created in compose(), not __init__;
  • build Cache(focus_index=..., buffers=...) on the message pump (the buffers property queries
    widgets — code_editor.py:294 calls _save_loaded_buffer() first);
  • bail when every buffer is blank. A startup crash gives buffers == [], and writing that over
    a good recovery file destroys exactly the work this feature exists to save;
  • bail when cache == self._last_checkpointed_cache. Cache/BufferState are dataclasses and
    Selection is a NamedTuple, so == is an exact content comparison — no dirty flag to keep in
    sync across tab swaps;
  • write_recovery(cache), synchronously. Pickling a few SQL buffers is microseconds; a thread
    worker would add ordering hazards (two checkpoints racing; exclusive=True cancelling one
    mid-flight while its thread still completes the write) for no measured win.
• action_quit() keeps its write_editor_cache(...) and then calls clear_recovery().

Restoring (`components/code_editor.py`) Copy Chapter

EditorCollection.__init__ keeps self.startup_cache = load_cache() and adds
self.recovered_cache, self.recovered_from = adopt_recovery(). Two separate module-level names,
not folded into load_cache(), so tests/functional_tests/conftest.py's no_use_buffer_cache
fixture stays the one seam.

on_mount prefers recovered_cache over startup_cache when present — a recovered session started
from the cache, so it is strictly newer — and then notifies: "Recovered buffers from a session that
ended unexpectedly." The shared cache-1.pickle is left untouched, so if the recovered content is
itself the problem, the last clean-quit state is still on disk.

`environment.py` (new, headless) Copy Chapter

Facts about the running installation, shared by hsql --info, the crash report, and (later) the
Debug Info screen. No textual, no rich; adapter_facts defers harlequin.plugins into its body as
info._adapters already does.

def harlequin_version() -> str
def python_facts() -> dict[str, str]        # version, implementation, executable
def platform_facts() -> dict[str, str]      # system, release, machine
def terminal_facts() -> dict[str, Any]      # term, term_program(+version), colorterm, shell,
                                            # wsl_distro, ssh (bool), locale, size
def install_facts() -> dict[str, Any]       # installer, in_venv
def adapter_facts(only: str | None = None) -> dict[str, dict[str, Any]]
def runtime_report() -> dict[str, Any]

• terminal_facts() reports ssh: True/False and never the value of SSH_CONNECTION, which
  holds IP addresses. size from shutil.get_terminal_size().
• install_facts() reads distribution("harlequin").read_text("INSTALLER") (pip/uv/pipx write it)
  and sys.prefix != sys.base_prefix — that answers the bug template's install-method checkbox
  automatically.
• adapter_facts is info._adapters + info._capabilities moved verbatim, minus the
  diagnostics.note() call, which a shared module cannot make.

hsql/modes/info.py then calls these instead of building the facts itself, and keeps its stderr note
by walking adapter_facts()'s result for entries with an error. Acceptance criterion:
hsql --info's JSON is byte-identical. terminal_facts and install_facts are not added to
that document — it is an agent-facing contract with a generated reference pinned by
test_cli_reference.py, and TERM means nothing to a non-interactive program.

Add harlequin.environment and harlequin.crash to the source_modules of the "adapter API is
reachable without the TUI" import-linter contract, and to HEADLESS_IMPORTS in
tests/unit_tests/test_import_hygiene.py. Both have to stay headless anyway, because hsql imports
them.

`crash.py` (new, headless) Copy Chapter

CRASH_REPORT_KEEP = 10
ISSUE_URL = "https://github.com/tconbeer/harlequin/issues/new?template=crash_report.md"

def get_crash_report_dir() -> Path
def root_cause(error: BaseException) -> BaseException
def build_crash_report(error: BaseException, context: Mapping[str, Any],
                       program: str = "harlequin") -> str
def write_crash_report(report: str) -> Path | None
def crash_message(report_path: Path | None, error: BaseException, saved: bool,
                  program: str = "harlequin") -> str

• Location. platformdirs.user_log_path(appname="harlequin", appauthor=False) —
  ~/Library/Logs/harlequin on macOS, ~/.local/state/harlequin/log on Linux,
  %LOCALAPPDATA%\harlequin\Logs on Windows. Not the cache dir: a cache is what a user is told to
  delete when things go wrong, and this is the file we are asking them to keep. Match config.py's
  appauthor=False — editor_cache/catalog_cache omit it and so nest twice on Windows; don't copy
  that.
• root_cause unwraps a Textual WorkerFailed via getattr(error, "error", None) rather than
  importing it, so this module stays headless. It matters: database_tree.py:498 (_loader) and
  code_editor.py:88 (read_symbols) are the two @work decorators in src/ that omit
  exit_on_error=False, so they do reach _handle_exception, wrapped. A report naming
  WorkerFailed is a useless report.
• Content, in order: a header telling the user to review the file before sharing and linking
  ISSUE_URL; runtime_report(); the context the caller supplied; traceback.format_exception
  for the root cause and, when different, the wrapper — no show_locals; and last, under a
  heading that says what it is, the SQL in the active buffer.
• Plain text, not markdown, written as crash-{utc:%Y%m%dT%H%M%SZ}-{pid}.log. The user pastes
  this into a fenced block in an issue, and a markdown report's own ``` fences would break out of
  it. Sections are delimited with plain === SECTION === rules for the same reason.
• The active buffer's SQL is included deliberately. It is usually what is needed to reproduce a
  TUI bug, the file is local, and the user chooses whether to paste it. The tradeoff is real — SQL
  holds table names and literals — so it goes last, under an explicit heading, and both the panel and
  the issue template say to review before sharing. Only the active buffer; the rest live in the
  recovery file.
• Redaction. The whole assembled string goes through redact.redact_text() once at the end — one
  choke point, matching hsql/diagnostics.py::_write's discipline.
• Pruning. Newest CRASH_REPORT_KEEP kept. Distinct filenames per crash, so a loop cannot
  destroy the evidence. write_crash_report returns None and raises nothing when the directory is
  unwritable.

exception.py gains class HarlequinCrashError(HarlequinError): pass, so the panel is
pretty_error_message(HarlequinCrashError(crash_message(...), title="Harlequin crashed.")) — no new
rendering code, rich stays lazily imported, and it looks like every other Harlequin panel.

The handler (`app_base.py`) Copy Chapter

Two hooks with working base implementations, so keys_app.py gets crash reports too:

def _save_work_on_crash(self) -> bool:
    """Persist anything the user would lose. False if there was nothing to save."""
    return False


def _crash_context(self) -> dict[str, Any]:
    """What this app was doing, for the crash report."""
    return {}


def _handle_exception(self, error: Exception) -> None:
    if self._exit or self._crash_handled:
        return
    self._crash_handled = True

    saved = False
    try:
        saved = self._save_work_on_crash()
    except BaseException:
        pass

    report_path = None
    try:
        report = build_crash_report(error, self._crash_context())
        report_path = write_crash_report(report)
    except BaseException:
        pass

    try:
        self.bell()
        self._return_code = 1
        # run_test re-raises from these
        if self._exception is None:
            self._exception = error
            self._exception_event.set()
        message = crash_message(report_path, error, saved)
        panel = HarlequinCrashError(message, title="Harlequin crashed.")
        self.panic(pretty_error_message(panel))
    except BaseException:
        # last resort: the user gets *something*
        super()._handle_exception(error)
        return

    # textual run --dev, i.e. `make serve`: append the full traceback at index 1
    if "debug" in self.features:
        super()._handle_exception(error)

/* Detailed source-code truncated for AI context efficiency. */

CRASH = 70
"""hsql hit a bug in itself. `sysexits.h`'s EX_SOFTWARE."""

70 rather than the next free small integer: sysexits.h reserves it for an internal software error,
and it cannot be confused with the 1–4 hsql already documents or with 130.

exit_code_for() does not change. It maps caught failures, and an unrecognized error caught
while running a query genuinely is a query failure — that is what QUERY means. CRASH belongs only
at the boundary where nothing else caught the error at all.

main() (hsql/__init__.py) gains a final handler, after the ClickException and
Abort/KeyboardInterrupt clauses:

except BaseException as e:
    report_path = write_crash_report(build_crash_report(e, _crash_context(argv), program="hsql"))
    diagnostics.report_crash(report_path)
    sys.exit(ExitCode.CRASH)

/* Detailed source-code truncated for AI context efficiency. */

---
name: Crash Report
about: Harlequin or hsql exited unexpectedly and wrote a crash report.
title: 'Crash: '
labels: 'bug'
assignees: ''

---
**Before proceeding, please acknowledge**:
- [ ] I have searched Issues and Discussions in this repo for this error.

**Paste your crash report below.**

Harlequin and `hsql` write a crash report and print its path when they exit
unexpectedly. If you still have that message on screen, the path is in it.
Otherwise, the reports are here:

| OS | Location |
|---|---|
| macOS | `~/Library/Logs/harlequin/` |
| Linux | `~/.local/state/harlequin/log/` |
| Windows | `%LOCALAPPDATA%\harlequin\Logs\` |

The newest `crash-*.log` is the one you want.

> [!IMPORTANT]
> Please read the report before pasting it. It contains your configuration
> (with passwords masked) and the SQL that was in your active buffer.


Crash report

PASTE YOUR CRASH REPORT HERE

**What were you doing when it crashed?**
Steps to reproduce it, if you know them.

**Anything else?**
Screenshots, or anything the report doesn't cover.

**Contributing**
Are you interested in contributing a fix?
- [ ] Yes
- [ ] Maybe
- [ ] No

bug_report.md gains one line under its acknowledgements pointing here: "Did Harlequin exit with an
error and print a crash report path? Use the Crash Report template instead."

Ordered commits Copy Chapter

Each is independently green under make check.

1. write_cache atomic and non-raising. editor_cache.py only; no behavior change visible to the app.
2. Recovery file + rename-before-replay. editor_cache.py, code_editor.py, app.py's
   clear_recovery, conftest patches. Nothing writes a recovery file yet, so this is provably inert
   on the happy path.
3. The checkpoint timer — commit 2's machinery gets a producer.
4. environment.py + the info.py refactor. Acceptance: hsql --info byte-identical. Import-linter
   contract and HEADLESS_IMPORTS in the same commit.
5. crash.py, HarlequinCrashError, AppBase._handle_exception, Harlequin's two hooks,
   hide_secrets_in, ctx.exit(return_code), and the test_cli.py fixture fix.
6. ExitCode.CRASH, diagnostics.report_crash, the main() handler, hsql's _crash_context.
7. .github/ISSUE_TEMPLATE/crash_report.md and the bug_report.md pointer.

Test infrastructure — the landmine Copy Chapter

run_test's re-raise path goes through _handle_exception, so every failing functional test in
the suite would write a crash report into the developer's real user_log_dir. Add to
tests/conftest.py, autouse and session-wide with no marker opt-out, in the spirit of the existing
no_discovered_config:

@pytest.fixture(autouse=True)
def crash_reports_go_to_tmp(monkeypatch, tmp_path) -> Path:
    """Every crash a test causes writes here, not into the developer's log dir."""
    report_dir = tmp_path / "crash-reports"
    monkeypatch.setattr("harlequin.crash.get_crash_report_dir", lambda: report_dir)
    return report_dir

Extend tests/functional_tests/conftest.py::no_use_buffer_cache in the same commit as the
checkpoint timer: patch harlequin.components.code_editor.adopt_recovery → lambda: (None, None)
and harlequin.app.write_recovery / clear_recovery → no-ops. Keep calling the existing
harlequin.app.write_editor_cache name so the current patch keeps working.

tests/unit_tests/test_cli.py::mock_harlequin is MagicMock(spec=Harlequin), so tui.return_code
is a MagicMock and ctx.exit(mock) makes res.exit_code a MagicMock — about fifteen existing
assertions break. Set mock_harlequin.return_value.return_code = 0 in the fixture, and add a test
that return_code = 1 produces res.exit_code == 1.

Tests to add Copy Chapter

tests/unit_tests/test_editor_cache.py (new): no *.tmp left behind; a pickle.dump that
raises leaves the previous cache byte-identical and returns False; an unwritable directory returns
False; adopt_recovery renames to .replayed before returning (the crash-loop guarantee); it
ignores a fresh recovery-<pid>.pickle and adopts a stale one; it always adopts recovered-*.

tests/unit_tests/test_crash.py (new): the report names the exception type, message, version and
frames; it prints no secret — hide_secrets_in({"conn_str": ["postgres://u:hunter2-and-more@h/db"]}),
raise an error quoting the DSN, assert the secret is absent and ******** present, mirroring
test_the_debug_screen_prints_no_secret; the report contains no ``` fence, so it pastes into one;
root_cause unwraps a WorkerFailed; write_crash_report prunes to CRASH_REPORT_KEEP and returns
None on PermissionError.

tests/functional_tests/test_crash.py (new), injecting through _handle_exception directly:

with pytest.raises(RuntimeError):
    async with app.run_test() as pilot:
        ...
        app._handle_exception(RuntimeError("boom"))
        await pilot.pause()

Assert app.return_code == 1; one crash file in the redirected dir; a recovery file holding the
buffer text (@pytest.mark.use_cache); "boom" and the report path in the rendered
_exit_renderables[0]; "Traceback (most recent call last)" not in it. Plus: two exceptions in a
row produce one report and no recursion; a write_crash_report that raises still exits 1 with a
panel; Harlequin(...) constructed but never run returns False from _save_work_on_crash without
raising (crash before mount); a blank-buffer checkpoint leaves a good recovery file unchanged; and
the crash loop — plant a recovered-*.pickle, make action_new_buffer raise, run and crash, then run
a second app and assert one empty buffer and the poisoned file now .replayed.

tests/unit_tests/test_hsql.py: a callback that raises exits ExitCode.CRASH (70) and not
QUERY; the stderr line names the report path; a DSN passed as CONN_STR does not appear in a
report written before hide_secrets_in has run.

No snapshot churn. Nothing renders in-app except the recovery toast, which is a standard Textual
notification; the crash message reaches stderr after the app is gone.

Verification Copy Chapter

localedef -i en_US -f UTF-8 en_US.UTF-8   # fresh container only
make check                                 # ruff, pytest 3.10 + 3.12, mypy
uv run lint-imports
uv run pytest tests/unit_tests/test_import_hygiene.py tests/unit_tests/test_crash.py \
              tests/unit_tests/test_editor_cache.py
uv run pytest tests/functional_tests/test_crash.py tests/functional_tests/test_cache.py -n0

By hand:

1. uv run harlequin f1.db, type SQL, wait past the checkpoint interval, kill -9, restart — the
   SQL is back and the toast says where it came from.
2. Temporarily raise from a key handler and run without dev mode (uv run harlequin f1.db, not
   make serve): the panel prints instead of a traceback, echo $? is 1, and the report at the
   named path holds the traceback. Repeat with a profile carrying a password and confirm it is masked
   and that no local variables appear.
3. Temporarily raise from EditorCollection.on_mount, crash, remove the raise, restart twice: the
   second start is clean and the poisoned file is .replayed.
4. make serve and crash: the full traceback still prints for developers.
5. Temporarily raise from an hsql mode: echo $? is 70, the stderr line names the report, and a DSN
   on the command line is masked in it.
6. Paste a real report into the crash report template's fenced block on a scratch issue and confirm
   it renders as one block.

Known risks Copy Chapter

• Partial-replay convergence. If a crash happens partway through replaying recovered buffers,
  _save_work_on_crash may write a new recovered-*.pickle holding the subset that loaded. The next
  start replays fewer buffers, so it converges and terminates — and salvages what it can — but it can
  take more than one restart.
• Behavior change: both commands now exit non-zero where they exited 0. harlequin returns 1 on a
  crash and 2 on a config or connection error; hsql returns 70 where it previously let Python exit
  1. Correct, but user-visible, so both get a changelog line.
• The report names sys.executable, the config path and SHELL, all of which reveal a username —
  and all of which hsql --info already prints, so this is consistent rather than new.

Deliberately out of scope Copy Chapter

• _handle_worker_error swallows unknown worker errors
  (#1117). app.py:576 branches on two worker
  names with no else, so any other worker error vanishes silently. The fix needs its own triage of
  which workers can fail benignly, so it is filed separately.
• Debug Info screen work — environment rows and a "copy a bug report" binding. Cheap once
  environment.py exists, but it is the only piece with snapshot churn: a visible version or Python
  row bakes 2.12.2 and 3.10.14 into Debug Info Screen.svg, which then fails on every CI Python
  and after every release. If picked up, put the rows in a collapsed Collapsible (collapsed
  children never reach the SVG) and back them with a non-snapshot unit test. Drive-by finding for
  whoever does: DebugInfoScreen.move_focus() queries #collapsible-client-adapter-details, which
  AdapterDebugInfo.parse_info() never produces — dead code with a latent NoMatches.
• Narrowing hsql's broad except Exception handlers so an internal bug caught inside cli.py
  reports CRASH rather than QUERY.
• A prefilled GitHub issue URL (the ~8KB URL cap cannot carry a traceback) and converting the
  templates into YAML issue forms.
• A harlequin --debug flag: the report already holds the full traceback, and TEXTUAL=debug /
  make serve already prints it inline.

Plans Harlequin For Agents (`docs/plans/harlequin-for-agents.md`) Copy Section

Harlequin for Agents

A product plan for making Harlequin the best SQL client for coding agents.

Status: draft for discussion. Product and user stories only — implementation details are
deliberately sketched, not specified.

Scope: the harlequin package (this repo) and the harlequin-web site
(https://harlequin.sh).

1. The opportunity Copy Chapter

Today an agent asked to "look at my warehouse" reaches for whatever CLI is nearest:
psql, duckdb, bq, databricks sql, mysql, trino. Each has different flags,
different output formats, different auth, different error semantics, and different
ideas about what "give me the schema" means. The agent burns tokens rediscovering all
of that on every task, and it hallucinates flags for the ones it has seen least.

Harlequin already has the thing that fixes this, and nobody else does:

• One uniform adapter interface across ~15 databases — DuckDB, SQLite, Postgres,
  MySQL, BigQuery, Trino, Databricks, ODBC, ADBC, Cassandra, NebulaGraph, RisingWave,
  Wherobots, Exasol.
• A normalized catalog abstraction (Catalog / CatalogItem) that already answers
  "what tables exist and what are their column types" without dialect-specific
  information_schema queries.
• Profiles in a config file that already encode "how to connect to my warehouse,"
  discovered automatically from the cwd and home directory.
• sqlfmt as an existing dependency.
• A TUI on the other end of the same config, which no other CLI has.

The pitch writes itself: hsql -P prod -c "select 1" works the same way against every
database your team uses. One CLI contract, one set of flags, one output format
vocabulary, one exit-code scheme — and when the agent needs a human, the human opens
Harlequin against the same profile and sees the same catalog.

That last point is the real differentiator. Every other SQL client can be made
agent-friendly. Only Harlequin can make the handoff good.

2. Who we are designing for Copy Chapter

Three personas, all of which are "an agent," but with materially different needs.

P1 — Agent as operator Copy Section

Runs queries headlessly on a human's behalf, usually in a loop, usually inside a coding
agent's Bash tool. Cares about: deterministic output, honest truncation, exit codes,
token-efficient results, not accidentally dropping a table.

P2 — Agent as pair Copy Section

Works alongside a human who has Harlequin open. Drafts a query the human will run;
reads what the human already ran. Cares about: handoff in both directions.

P3 — Agent as support engineer Copy Section

Helps a human install, configure, and troubleshoot Harlequin — "why won't it connect to
Snowflake," "add a prod profile to my config," "what does this error mean." Cares about:
machine-readable docs, a formal config spec, a self-describing CLI, good diagnostics.

Most existing "AI-friendly CLI" work only serves P1. Serving P2 and P3 is where
Harlequin can be uniquely good.

3. Design principles Copy Chapter

These are the calls I'd want made up front, because they resolve most of the small
decisions downstream.

1. One engine, two front doors. harlequin (TUI) and hsql (headless) share
   adapters, config discovery, profiles, limit semantics, and one execution core.
   Neither reimplements the other. See §4.

2. stdout is data; stderr is narration. Results go to stdout and nothing else does.
   Truncation notices, warnings, errors, and --stats go to stderr — and nothing that
   stdout already carries: the row count is the result's own footer, and a timing is a
   field of --stats. This is already the intent in exception.py — make it a hard,
   tested contract.

3. Exit codes are an API. Documented, stable, and distinct enough that an agent can
   branch on them without parsing text.

4. Determinism over cleverness: identical output whether stdout is a TTY, a pipe, or
   a pty. Agent harnesses vary in which they give you. A client that formats
   differently under a pipe is a bug factory. No TTY-sniffing for format selection.
   (Color is the one exception, and it follows NO_COLOR / --color.)

5. Token efficiency is a feature. Results cost the agent money. Defaults should be
   small, and truncation must always be announced — never silent.

6. Be boring and psql-shaped. An agent's prior is psql, secondarily duckdb.
   Every flag we can borrow, we borrow. Novel spellings cost tokens and cause retries.

7. Never make the agent guess. Everything Harlequin knows about itself — options
   (including plugin-contributed ones), profiles, formats, adapters, capabilities,
   catalog — should be dumpable as JSON on demand. This is the one place we should be
   more capable than psql, because our option surface is dynamic and plugin-driven, so
   no amount of pretraining can teach an agent what's actually installed.

8. Safety is a product feature, not a warning in the docs. --read-only,
   --timeout, and --dry-run are what let a human hand an agent a warehouse
   credential without flinching.

9. No LLM inside Harlequin. No API keys, no model config, no vendor coupling. We
   make Harlequin excellent for the agent already sitting next to the user. (See §10 for
   how this answers issue #952.)

4. Shape of the CLI: two commands, one engine Copy Chapter

Decision: headless mode ships as a separate command, hsql, not as flags on
harlequin.

The tempting reason — it frees the flag namespace — is the weakest one. The real reasons
are these.

Two audiences want two stability contracts. The TUI's promise is "delight a human,
evolve freely." The headless promise is "output format and exit codes are an API,
frozen." Inside one command those pull against each other on every release, and a TUI
change can silently break a machine contract. A separate command makes the boundary
structural rather than something a reviewer has to remember.

Cold start stops being a discipline problem. With one entry point, "never import
Textual on the headless path" is a rule any contributor can regress silently. With a
separate module graph it becomes an import-linter test in CI — the hsql module graph
must not contain textual — which is the difference between hitting the 300ms target
and hoping to.

It opens a lean install later. pip install hsql with no Textual, no pyperclip, is
plausible for CI images and agent sandboxes, and hard to reach from a single entry point.

What actually ships Copy Section

The same package, with a second console script:
hsql = "harlequin.hsql:main" alongside the existing harlequin script.

That is the key de-risking move. The command split is cheap and near-reversible; the
package split is the expensive part, and it can wait indefinitely — possibly forever.
We get the boundary and the enforceable import graph now, and decide about a separate
distribution only if the lean-install demand materializes.

harlequin itself gains nothing: no -c, no growth in an already-large --help,
no behavior change. The one addition is a custom unknown-option handler, so
harlequin -c "select 1" fails with "-c isn't a harlequin option — did you mean
hsql -c? See harlequin.sh/docs/headless."

Everything in Workstream B lands as hsql subcommands: hsql catalog,
hsql describe, hsql fmt, hsql spec, hsql info, hsql config, hsql history,
hsql mcp, hsql open. Bare hsql with no arguments prints help; it never launches
the TUI.

Amended in M2's plan: these are mode options, not subcommands. hsql --catalog,
hsql --info, hsql --spec, hsql --config show. It matches harlequin --config, so
one shape covers both commands — and it sidesteps the objection this section raises
against subcommands, which applies here too: hsql takes CONN_STR positionally, so
hsql catalog and a database file named catalog would have needed a disambiguation
rule, docs for the rule, and a test for the rule. No verb, no rule.

Consequences Copy Section

• The -f collision disappears, and with it the plan's only breaking change.
  harlequin -f stays --show-files forever; hsql -f is --file, as psql users
  expect. Nothing here needs to wait for 3.0 — the whole plan becomes additive and can
  ship in 2.x minors.
• hsql gets clean access to -c, -f, -o, -A, and friends, so psql muscle memory
  works without compromise.
• hsql --help omits every TUI-only option (--theme, --keymap-name, --show-files,
  --show-s3, --locale, --config, --keys), which is a real token saving for an
  agent reading it. It also omits the adapter connection-option matrix: hsql --help
  lists the names of installed adapters and points at hsql --help -a <adapter> for
  one adapter's options. That keeps the first thing an agent reads small and stable, and
  keeps --help true for every adapter rather than showing whichever one is the
  default.
• The handoff gets a natural verb: hsql open query.sql reads much better than
  harlequin --new-buffer query.sql.

Alternatives rejected Copy Section

• Flags on harlequin. Requires the 3.0 -f break, grows an already-large help
  surface, and leaves cold start and the stability boundary as matters of discipline.
• Subcommands on harlequin with a bare-invocation TUI fallback
  (harlequin query -c ...). Needs a custom Click Group that falls back to the TUI
  when the first argument isn't a known subcommand, and it is genuinely ambiguous: a
  DuckDB file named catalog or config in the cwd resolves as a subcommand. Low
  probability, but exactly the confusing failure we don't want an agent hitting.

On the name Copy Section

hsql is free on PyPI as of this writing — M0 in §11 secures it before anything else
starts. HSQLDB ships sqltool and java -jar, not a binary named hsql, so there's no
PATH collision — the cost is search-results mindshare with
HyperSQL, which is real but mild and fading. hq is taken. harlequin-cli is free but
too long for something an agent types a hundred times in a task; short names are cheap
in tokens and cheap in working memory.

I'd take hsql. It's obviously "harlequin sql," and it sharpens the positioning rather
than diluting it:

Harlequin — the SQL IDE for your terminal.
hsql — the same engine, headless, for scripts and agents.
Fifteen databases, one adapter ecosystem.

That's a better story than "one tool with a flag."

5. Workstream A — `hsql`, headless execution Copy Chapter

Closes #524. This is the
foundation; everything else in this plan assumes it.

Proposed surface Copy Section

hsql [OPTIONS] [CONN_STR]...        Execute SQL and exit
hsql <SUBCOMMAND> [OPTIONS]         catalog, describe, fmt, spec, info,
                                    config, history, open, mcp

  -c, --command TEXT     Execute SQL. Repeatable.
  -f, --file PATH        Execute SQL from a file (or `-` for stdin). Repeatable.
  -o, --output PATH      Write results to PATH instead of stdout.
      --format NAME      Output format (see below). Default: table.
      --csv/--json/--jsonl/--markdown/--vertical    Format shorthands.
  -t, --tuples-only      Rows only: no header, no footer. As in psql.
  -A, --no-align         Unaligned output. As in psql.
      --no-header        Omit the header row (keep other chrome).
      --null-string STR  Render NULL as STR. Default: empty for csv, "NULL" for table.
  -P, --profile NAME     Same profiles as the TUI.
      --limit N          Max rows fetched per result set. Default: 500. -1 = no limit.
      --result all|last|N  Which result set(s) to emit. Default: all.
      --on-error stop|continue    Default: stop.
      --stats            Write a one-line JSON summary to stderr.
      --color auto|always|never   Default: never.
      --timeout SECONDS  Cancel the query and exit non-zero.
      --read-only        Refuse to run if the adapter can't enforce read-only.
      --dry-run          Parse/validate (and EXPLAIN where supported) without executing.
      --single-transaction        Wrap the whole script; roll back on any error.

Config discovery, profiles, and precedence are identical to the TUI: CLI > env >
--config-path > cwd config > user config dir > home > defaults. That's the whole
point — hsql -P prod -c ... means the agent never handles a credential.

Formats Copy Section

Amended in implementation. There is no -F: that is psql's
--field-separator, and a flag that sets a delimiter in one command and picks a
format in the other is a mistake waiting for a script — the same reasoning that
keeps -l off --limit. The format is chosen by --format spelled out, or by
the --csv/--json/--jsonl/--markdown/--vertical shorthands, which cover
the choices anyone types often enough to want a short flag for.

Behaviors that matter more than the flag list Copy Section

Truncation is always announced. If --limit bites, stderr gets
note: results truncated at --limit 500; pass --limit -1 for all rows, and text formats
append a visible … 500 of >500 rows footer.

Note that --limit here is a hard limit — it caps what leaves the database, via the
adapter's set_limit(). The TUI's --limit is a soft display cap: it fetches everything
and caps what loads into the viewer, which is why the TUI can report an exact
Showing 100,000 of 3,412,887. hsql deliberately doesn't buy that number, so it says
>500; only --limit -1 yields an exact count. The TUI's default of 100,000 is
right for a TUI and a catastrophe for an agent; hsql defaults to 500.

Amended by #1026. One key
naming both of those limits was the bug, not the design: the TUI's display cap is now
viewer_max_rows, limit is the hard fetch limit in both commands, and unlimited is
spelled -1 rather than 0, so that --limit 0 can fetch a header and no rows (and
there is no -l, which is psql's --list). The text
layouts additionally cap what they print (--display-rows, 40 rows or 10 records),
since fetching 500 rows and printing 500 rows are separate questions.

Detecting truncation at all requires fetching limit + 1 rows and emitting limit —
set_limit(n) followed by fetchall() returns at most n rows and cannot say whether an
n+1th existed. Worth stating here rather than leaving to implementation, because it's the
kind of detail that gets optimized away and quietly breaks this promise.

The stderr notice fires even under -t. -t suppresses stdout chrome, not
warnings; a flag that silently defeats truncation reporting would undo principle 5.

Row counts and timing go to stderr, so hsql -c "select 1" --csv > out.csv produces
a clean file while the agent still sees 1 row in 0.02s.

--stats emits one line of JSON to stderr for any output format:

{"status":"ok","statements":1,"rows":500,"truncated":true,"limit":500,
 "elapsed_ms":412,"columns":[{"name":"id","type":"BIGINT"}]}

This is how an agent gets structured metadata without polluting stdout, and it works
identically whether stdout is CSV or Parquet.

Errors are plain. No Rich panels, no ANSI, no box drawing:
hsql: error: relation "usres" does not exist on stderr. The panel is a TUI affordance.

Amended in M2's plan, on two of the safety flags above.

--timeout has to attribute the cancellation itself. A cancelled DuckDB query comes
back as an empty result set with no error — fetchall() swallows the interrupt and
returns None, which is also what an empty result looks like — so a timeout that trusts
the adapter prints "no rows" and exits 0. hsql knows it fired the deadline, discards
that result, and exits 4. It also refuses up front on an adapter that can't cancel,
since there is otherwise no way to stop the work.

--single-transaction is cut too. Adapters name their transaction modes freely —
SQLite and Postgres offer "Auto"/"Manual", DuckDB overrides nothing and reports none, and
a third-party adapter may name its modes anything and default to whatever its driver
does — so core cannot ask for "the manual one" through that interface without first
reworking what a transaction mode is. Story A6's guarantee comes from begin and
commit in the caller's own script until then.

--dry-run is cut, from M2 and possibly for good. Only DuckDB implements
validate_sql at all, and there it is a parse check rather than an existence check:
select * from no_such_table validates clean, because the adapter treats every
non-parser error as valid so that DDL passes. Making it honest would mean a PREPARE
pass in the DuckDB adapter plus a refusal path on every other adapter — a lot of
machinery for a flag that would be truthful on one backend.

Exit codes:

Cold start is a feature. An agent may run twenty queries in a task. Target:
hsql -c "select 1" against DuckDB in under 300ms, tracked as a benchmark in CI and
protected by the import-linter rule from §4.

Large results stream — in M5, not M1. Streaming has to start at the cursor:
HarlequinCursor.fetchall() materializes the whole result before any writer exists, so
no amount of care in the format layer helps. Deferred wholesale to M5 along with
#875, rather than half-designed
now.

User stories Copy Section

• A1. As an agent, I run hsql -P prod -c "select count(*) from orders" and get a
  small, aligned result on stdout and exit code 0, so I can report a number to the human
  without parsing a TUI.
• A2. As an agent, I run the same command against Postgres, BigQuery, and DuckDB and
  the flags, output shape, and exit codes are identical, so I don't relearn a CLI per
  database.
• A3. As an agent, when my SQL is wrong I get a one-line error on stderr and exit
  code 1, and nothing on stdout, so I can distinguish "the query failed" from "the query
  returned zero rows."
• A4. As an agent, when a result is larger than the limit I am told so explicitly, so
  I never report a truncated aggregate as a complete one.
• A5. As an agent, I pass --json and pipe to jq without stripping banners,
  timings, or a "(500 rows)" footer.
• A6. As an agent, I run a multi-statement script with -f setup.sql --single-transaction and either all of it applies or none of it does.
• A7. As a human, I set read_only = true in my agent profile so the agent
  physically cannot write to prod, and I can point at that line in the config when
  someone asks whether it's safe.
• A8. As a human, I use hsql in a Makefile or CI job because it's the same engine
  and the same profile I already use interactively.

On `-t`, and the `-tAc` idiom Copy Section

-t is --theme in harlequin and "tuples only" in psql. Since hsql has no themes,
-t is free, and it takes the psql meaning.

The reason isn't -t in isolation — it's that psql -tAc "select count(*)" is a single
well-worn idiom, the standard way to capture a scalar in a shell script. Supporting -t
and -A separately but not together would be pointless; refusing -t would break the
cluster and make first contact with a scripting audience an error message. Borrowing the
flag is principle 6 doing its job.

The residual risk is small and worth naming: someone with harlequin -t nord habits
types hsql -t nord -c ..., -t parses as a boolean, and nord is read as a
connection string. The resulting "could not open database 'nord'" is diagnosable, and we
can special-case the hint when an unparseable conn_str matches a known theme name.

hsql -tAc "select count(*) from orders" returning a bare number and nothing else
should be an explicitly tested case, and the first example in the scripting docs.

A short "differences from psql" table in the docs should still cover -P (profile here,
pset there) and anything else that diverges.

6. Workstream B — Self-description and introspection Copy Chapter

If Workstream A is what lets an agent run a query, this is what lets it write a
correct one on the first try. All of it lands as hsql subcommands.

`hsql catalog` — the catalog is a filesystem, not a document Copy Section

The highest-leverage feature in this plan after -c. Harlequin already normalizes the
catalog across every adapter; today that value is locked inside the TUI.

The binding constraint is fetch cost, not token cost. get_catalog() issues one
query and returns databases only; every level below it is a separate fetch_children()
round trip, with columns as the leaf. Listing one schema's columns is 1 + 1 + 1 + T
queries. A "dump the tree, then filter" interface is an N+1 walk that gets slower the
more useful it would be — and #1007 already moved the TUI off eager loading for exactly
this reason. Filtering in the client protects the agent's context and does nothing for
the database.

So the model is a filesystem: ls, stat, find — not dump-and-grep. That bounds
cost structurally, because you only pay for the level you asked for, and it rides on the
navigation loop agents are most practiced at.

hsql catalog [PATH] [--depth N] [--max-nodes N] [--format compact|json|md|tree]
hsql describe <PATH>
hsql find <TERM> [--in tables|columns|all]

• One level below the match, by default. hsql catalog → databases;
  hsql catalog mydb → schemas; hsql catalog mydb.analytics → relations;
  hsql catalog mydb.analytics.orders → columns. Exactly one fetch_children() call.
  --depth N opts into recursion; there is no unbounded default.

  Counting depth from the match rather than from the root is what makes this safe
  rather than merely conservative: at every level, depth 1 is a single round trip, so
  there is no path on which the default degrades into a walk. Depth ≥2 from a schema is
  the first N+1, and it's opt-in.

• Patterns are paths with an optional trailing glob. A trailing wildcard filters one
  resolved parent's children and stays a single round trip, so analytics.* is just
  sugar for analytics, and analytics.ord* narrows it. An interior wildcard
  (*.orders, mydb.*.orders) can't be evaluated without fetching every candidate
  level, so it belongs to find below — never something a catalog call does quietly.

• Report child counts wherever the adapter can get them cheaply. 400 relations next
  to a schema is what lets an agent decide not to recurse. Cost should be predictable
  before the call, not just measurable after it.

• A node budget. --max-nodes (default ~2000) stops a recursive walk and says so,
  naming the path to narrow. Same rule as row truncation: never silent.

• --stats reports round trips, so the expensive shape of a query is visible.

• Emit query_name, not just the label. Adapters already compute the correctly
  quoted identifier for every item. Handing it over means the agent never guesses whether
  this backend wants "Orders" or `orders`. It costs nothing and no other client
  does it.

• --format compact stays for the leaf and shallow-recursion cases:
  analytics.orders(id BIGINT, customer_id BIGINT, total DECIMAL(18,2)). Roughly an
  order of magnitude cheaper than pretty JSON and perfectly readable by a model.

• describe returns one object in full — columns, types, comments, row counts where
  the adapter can supply them. One round trip.

Amended in M2's plan, on four things this section assumes the catalog contract can
already do.

A listing is one round trip per path segment, plus one — four for
--catalog --path mydb.analytics.orders, not one. An item can only be reached through
its ancestors, so the resolve in front of the listing is itself a walk. The listing stays
single-level and cheap; the resolve is what a network database charges for.

--depth N is one number over levels with wildly different costs, so M2 ships no
recursion at all. Depth 2 from the root fetches each database's schema names — one
round trip per database, and databases are few. Depth 2 from a schema is one round trip
per relation: measured, 400 relations is 403 round trips and 2.3 seconds against a
local DuckDB file, on the call an agent would most want to make. Recursion comes back
with fetch_descendants(), which is what would make it a single query rather than a
walk with a budget bolted on.

Child counts have no cheap case. A count is only knowable by fetching the children,
so "report child counts wherever the adapter can get them cheaply" cannot be implemented
as written. What a listing can report is what it fetched, which it already does.

describe is catalog on a relation, so there is no separate verb. One level below
a relation is its columns, with their names, real types and quoted identifiers. All a
second mode could add is detail the contract does not carry (comments) or that costs a
table scan (row counts).

--format compact is cut, and the type it wanted is added. CatalogItem.type_label
is a 1–3 character label for a tree column (##, s, ts), so the type an agent needs
is not in the contract at all — M2 adds an optional type_name, which the in-tree
adapters already fetch and discard, falling back to the short label where an adapter
never populates it. The format goes: parent(child TYPE, …) only reads for a relation
and its columns, and what a level is belongs to the adapter, so a database's schemas
would render as a signature they are not. -tA --format csv over the same rows is as
cheap and is true at every level.

`hsql find` — the real gap Copy Section

What an agent most often needs isn't "list this schema," it's "where does orders
live" or "which tables have a customer_id." That cannot be a walk.

This wants an optional adapter capability (implements_catalog_search): one
information_schema-style query where the adapter can serve it, and an explicit "not
supported by this adapter" where it can't, surfaced through the capability flags in
hsql info. High value, but it should not block the rest of the workstream.

An adapter-API question this raises Copy Section

fetch_children is per-item, which is exactly right for a viewport-driven TUI and forces
N+1 for bulk access. An adapter that could pull an entire schema's columns in a single
information_schema query has no way to say so.

Worth considering an optional fetch_descendants(depth) on InteractiveCatalogItem,
with a default implementation that simply walks fetch_children. Adapters that can do
better override it; everything keeps working for those that don't. This is a
Harlequin-core API decision, not an hsql one, and it would improve the TUI's
expand-all path too.

Stories:

• B1. As an agent, I run hsql -P prod catalog and then hsql -P prod catalog mydb
  to orient myself, paying one query per step and knowing what each step cost.
• B2. As an agent, I see analytics — 400 relations and scope my next call instead of
  recursing into a walk that would take a minute and blow my context.
• B3. As an agent, I run hsql -P prod catalog mydb.analytics.orders --format compact
  and write a correct join on the first attempt, pasting the adapter's own quoted
  identifiers rather than guessing at them.
• B4. As an agent handed an unfamiliar warehouse, I run hsql find orders and get a
  path instead of walking three levels to look for one.

`hsql spec --json` Copy Section

Dump the full, installed CLI surface — including options contributed by whichever
adapter plugins are present — as JSON. Harlequin already builds these dynamically from
AbstractOption, so the data exists; it just isn't reachable.

This is unusually valuable here precisely because the option surface is plugin-driven.
No model's pretraining can know that this machine has harlequin-databricks installed
with its particular options. Self-description turns a guessing game into a lookup.

• B4. As an agent, I run hsql spec --json once at the start of a task and never
  hallucinate a flag for the rest of it.

`hsql info --json` Copy Section

The debug-info screen, headless: versions, installed adapters, discovered config files in
precedence order, the active profile, and capability flags per adapter
(implements_cancel, implements_copy, implements_read_only,
implements_validate_sql, supports_transactions). Credentials and connection strings
redacted.

Capability flags matter: they let an agent know not to try --dry-run on an adapter
that can't validate, instead of trying and failing.

Amended in M2's plan. The flags have to be declared on the adapter class, not
detected on the connection. Reflection can't answer the question:
HarlequinSqliteConnection defines validate_sql and raises NotImplementedError from
it, so an override check reports SQLite as validating SQL. Nothing is broken by that
today — the app catches the error at the call site — but a class variable is the pattern
that can be read before the call, and the only one hsql --info can read while the
database is unreachable, which is when a diagnostic is worth most. So --info imports
adapters and opens no connection.

• B5. As an agent debugging a human's setup, I run hsql info --json and can tell
  them exactly which config file is winning and why.
• B6. As an agent, I check capability flags before offering the human a feature their
  adapter doesn't support.

`hsql fmt` Copy Section

Expose sqlfmt, which is already a dependency. hsql fmt query.sql,
hsql fmt -c "select 1", --check for CI. Agents write inconsistent SQL; this
normalizes it before it lands in a human's repo. Cheap, and a genuinely nice human
feature too.

Cut in M2's plan. An agent that wants formatted SQL can run sqlfmt, which is what
this would have wrapped. It is also the only headless mode that would have forced a
196ms sqlfmt import into a package whose import graph is otherwise an enforced contract —
paid by the mode alone, but a deferral to maintain forever for a wrapper nobody asked
for.

`hsql history` Copy Section

Query history is already cached per connection — but as a pickle. Expose it
(hsql history --json -n 20) and additionally write an append-only JSONL log, so both
agents and humans can read it with ordinary tools.

• B7. As an agent joining a task mid-stream, I read the human's recent query history
  and continue their line of investigation instead of starting over.

7. Workstream C — The config file as a formal contract Copy Chapter

Agents write config files. Today they'd be guessing at TOML structure from prose docs,
and a wrong guess produces a confusing runtime error. The config is shared between
harlequin and hsql, so this serves both.

Deliverables Copy Section

1. A published JSON Schema for .harlequin.toml, at
   https://harlequin.sh/schemas/config/v1.json and shipped in the package. Enables the
   #:schema comment for taplo / Even Better TOML — editor validation for humans, and a
   verifiable target for agents.
   • Wrinkle: adapter options are dynamic. Solution: publish a base schema for the
     site, and have hsql config schema --json generate a locally-accurate one that
     includes the installed adapters' options.
2. hsql config validate [--json] — structured diagnostics with file paths and line
   numbers, so an agent can fix its own mistake without a human in the loop. The existing
   _raise_on_bad_schema messages are already good; they just need a machine channel.
3. hsql config show [--json] — the effective merged config, with provenance per key
   (limit = 500 # from ~/.harlequin.toml). This is the single best troubleshooting
   artifact we could produce.
4. hsql config init --non-interactive — the TUI wizard (harlequin --config) is
   questionary-only, so agents can't use it. A non-interactive path that edits TOML
   through tomlkit (preserving the human's comments and formatting) means an agent can
   safely modify a config it didn't write. The interactive wizard stays exactly where it
   is; each command gets the affordance right for its audience.
5. Environment variable interpolation (${DB_PASSWORD}) in config values — closes
   #898, and it's a prerequisite for
   agents and CI ever touching a config that involves secrets.
6. A formal spec page on the site: discovery order, merge semantics, precedence,
   every key with its type and default, and the reserved/invalid names.

Amended in M2's plan, on two things this workstream would otherwise document as they
stand.

The merge is per top-level key, not per profile, and the result is a start-up
failure. _merge_config_files() is a dict.update() per file, so a cwd config file
that defines any profile replaces the home file's whole profiles table — while leaving
its default_profile in place, since that is a different top-level key. Both commands
then refuse to start: "Config files set the default_profile to personal, but do not
define a profile with that name." Reproduced on this checkout, and against the
pre-refactor config.py as well, so it is long-standing rather than something the
tomllib change introduced. M2 makes the merge per-key and treats it as a bug fix; the
spec page should describe that rather than what exists today.

Validation runs on the merged document, so no error can name a file. load_config()
merges every discovered file and then validates the result, which means a diagnostic can
only ever point at a key in a structure nobody wrote. M2 validates each file as it is
read and merges only valid ones, which is what makes deliverable 2's file-and-line
diagnostics possible. It also adds hsql --config list-profiles, since the first
question a human asks is what they can pass to -P, and a missing name is how the merge
bug above becomes visible.

${VAR} interpolation has to be a read-path transform. harlequin --config reads
the config and writes it back, so resolving values too early bakes a plaintext password
into the user's file on their next wizard run. The syntax is ${VAR} and
${VAR:-default} — a bare { never triggers it, so nothing anyone has in a password
today needs escaping — and an unset variable with no default is a config error naming
the variable and its file, never an empty string.

Secrets: #667, #898, and redaction are one feature Copy Section

#667 asks for a password prompt, and
the maintainer's read at the time was that it wants a new type of adapter option.
That's the right shape, and it does far more than serve the prompt: it turns redaction
from a discipline into a property of the declaration.

The decisive argument is the plugin ecosystem. Core cannot enumerate every adapter's
secret — --service-account-key, --token, --tls-key, whatever the next adapter
invents. But each adapter can declare its own, once, and then every consumer gets it
free: hsql info --json, config show, spec --json, error output, the debug-info
screen, and the config wizard's masked input.

Combine that with ${VAR} interpolation (#898) and the secrets story closes:

• PasswordOption (or secret=True on AbstractOption) — declares an option as
  sensitive. Drives hide_input at the prompt, a masked widget in the TUI, and
  redaction everywhere else.

• Never prompt in headless mode. A prompt that blocks on stdin is the worst
  failure mode for an agent: no output, no exit code, the whole turn burned until
  something times out. hsql must fail fast with a message pointing at the profile or the
  env var.

  Amended in M2's plan: no --password-stdin either. Between profiles, ${VAR}
  interpolation and the variables every driver already reads, it fills no gap worth a
  flag — and the gap it does fill, it fills by consuming the stream -f - wants.

• spec --json marks secret options. This is what teaches an agent not to
  construct hsql --password hunter2, which would leak through ps and shell history
  to every other user on the box. The skill should say it too, but the machine-readable
  flag is what makes it checkable.

• Scrub known secret values from outbound strings as a backstop. Declarative
  redaction covers values we own; a driver exception that echoes a DSN does not go
  through our option layer.

• Connection strings need their own handling. conn_str is a positional argument, so
  no option type covers postgres://user:pw@host/db. It needs DSN-aware redaction of its
  own — and #354 is evidence that
  people really do put passwords there.

• History is the one surface this can't fix. create user x password 'y' is a secret
  inside a query. Worth saying plainly in the docs that history is local-only and may
  contain secrets by construction.

Stories Copy Section

• C0. As a human, I hand my agent a profile name, and no secret ever appears in an
  argument list, a transcript, or ps output.
• C1. As an agent, I add a prod profile to a human's existing config without
  clobbering their comments or reformatting the file.
• C2. As an agent, I validate the config I just wrote and get a line number when I
  get it wrong.
• C3. As a human, my editor red-squiggles a typo'd key before I ever run Harlequin.
• C4. As an agent, I write a config that references ${SNOWFLAKE_PASSWORD} and never
  see the secret itself.

8. Workstream D — Docs that machines can read (harlequin.sh) Copy Chapter

The docs are already markdown files on disk (src/docs/**/*.md) and there's already an
/api/docs index route. Most of this is plumbing we half-have.

D1. `llms.txt` and `llms-full.txt` Copy Section

Highest ROI item on the site. llms.txt at the root: an index of every docs page with a
one-line description and a link. llms-full.txt: the whole documentation set
concatenated as clean markdown. Both prerendered at build time. An agent fetches one URL
and knows everything.

D2. Raw markdown per page Copy Section

https://harlequin.sh/docs/getting-started/running.md returns text/markdown.
Implemented as a catch-all +server.ts reading the source via import.meta.glob(..., { query: '?raw' }).

The real work is sanitization: the sources carry mdsvex <script> blocks and Svelte
component tags (<Tip>, <Figure>, <Key>). The raw route must strip the script
blocks and downgrade components to plain markdown equivalents (<Tip> → a blockquote,
<Figure> → an image with the caption as alt text, <Key> → backticks). Worth
budgeting real time for; a raw route that emits Svelte tags is worse than none.

D3. Docs API v1 Copy Section

Extend the existing endpoint into something documented and stable:

• GET /api/docs → index (exists today)
• GET /api/docs/{topic}/{page} → { title, topic, slug, markdown, updated_at }
• GET /api/docs/search?q= → keyword search over a prebuilt index (the corpus is small;
  no vector store needed)
• CORS *, cached with the ISR config already in use.

D4. Copy / view as markdown Copy Section

A "Copy page as Markdown" button and a "View as Markdown" link in the docs layout. The
copy variant should prepend the canonical URL and title as a header so pasted context
carries its provenance. Small, cheap, and increasingly expected.

D5. A "Headless & Agents" docs topic Copy Section

A first-class topic in the sidebar, not a footnote. Pages:

• The hsql CLI — overview and tutorial
• Output Formats — the table from §5, with examples
• Exit Codes and Error Handling
• Catalog and Introspection
• Safety: read-only, timeouts, dry runs
• Differences from psql
• Config File Spec (links to Workstream C)
• Using Harlequin with Claude Code and other agents — the skill
• The MCP server

The topic is named for the capability rather than the binary, so a human who doesn't yet
know hsql exists can still find it.

D6. Homepage positioning Copy Section

The two-command story from §4 needs a home on the landing page — a short section that
says Harlequin is the IDE, hsql is the same engine headless, and both speak fifteen
databases. This is the page a human sends to their agent, which makes it the entry point
for P3.

D7. `AGENTS.md` in the repos Copy Section

For agents working on Harlequin: repo layout, how adapters plug in, how to run tests
(make check), the snapshot-test workflow, and the changelog convention. Add it to this
repo and to harlequin-adapter-template so new adapters inherit it.

Stories Copy Section

• D1s. As an agent helping a human install Harlequin on Windows, I fetch
  llms-full.txt and answer from current docs instead of stale pretraining.
• D2s. As an agent, I fetch one page as markdown and spend a tenth of the tokens I'd
  spend on the rendered HTML.
• D3s. As a human, I click "Copy as Markdown" and paste an accurate page into my
  agent's context.

9. Workstream E — Integrations: skill first, MCP last Copy Chapter

The recommendation, and why Copy Section

Ship a skill first. Make MCP a thin wrapper over the same execution core, not a
parallel implementation.

A good CLI plus a skill — one markdown file that teaches hsql — works in any agent
harness that can run a shell command: Claude Code, Codex, Cursor, aider, a bash script,
CI. It costs one file to maintain, adds zero tokens to requests that don't use it, and
keeps a single tested code path.

MCP earns its keep in exactly two places, and they're real:

1. Harnesses without shell access — Claude Desktop, chat clients, some IDE
   integrations. The CLI is invisible to them.
2. Connection reuse and session state. For Postgres, BigQuery, Trino, and Databricks,
   connection setup dominates a short query. A persistent server also preserves temp
   tables, session settings, and open transactions across calls, which a fresh process
   per query cannot.

So: both, in that order, sharing one core.

E1. The Harlequin skill Copy Section

SKILL.md in this repo, published on the site, and installable. Contents:

• Discover the setup first: hsql info --json, hsql spec --json
• Use profiles, never raw credentials
• Orient with catalog --depth 2, then scope with a pattern
• Run with -c; pick a format on purpose; respect the limit; check for truncation
• Branch on exit codes
• Prefer --read-only unless the human asked for a write
• When to stop and hand off to the human's TUI — this is the part nobody else will
  write, and it's what makes the skill feel like Harlequin's rather than generic

E2. `hsql mcp` Copy Section

hsql mcp --profile prod starts a stdio MCP server. Deliberately small tool surface —
every tool is tokens in every request:

• run_query(sql, format?, limit?)
• list_catalog(pattern?, depth?)
• describe_object(name)
• explain_query(sql)
• format_sql(sql)

Plus MCP resources for the catalog and for the docs (llms-full.txt), which is the
idiomatic way to expose reference material without spending tool-schema tokens.

Read-only by default; writes require an explicit --allow-writes. The MCP dependencies
ship as an optional extra so the base install stays lean.

Sequenced last, deliberately (M6). Everything above it — the execution core, the
catalog navigation, the capability flags, the skill — is what an MCP server would be a
wrapper around. Building it before those settle means designing tool schemas against a
moving target and then maintaining the mistakes. Shipping it last also means we'll know
from real skill usage which five tools are actually worth their tokens, rather than
guessing.

Stories Copy Section

• E1s. As an agent in Claude Code, I install the Harlequin skill and immediately use
  the right flags, formats, and limits without trial and error.
• E2s. As a human in Claude Desktop with no shell, I add the Harlequin MCP server and
  ask questions about my Postgres database.
• E3s. As an agent running twenty queries against BigQuery, I reuse one connection
  instead of paying setup cost twenty times.

10. Workstream F — Human ↔ agent handoff Copy Chapter

This is the part only Harlequin can build, and I'd resist the temptation to sequence it
last just because it's the least conventional.

• hsql open query.sql launches the TUI with the query loaded in a new buffer,
  not executed. The agent drafts; the human reviews, edits, and runs. Buffers already
  persist through editor_cache.py, so the machinery is close at hand. Having a verb for
  this on the agent-facing command is exactly the ergonomic win the two-command split
  buys us.
• "Copy CLI command" action in the TUI — writes hsql -P prod -c '<current buffer>' --csv to the clipboard, so a human can hand their agent something exactly
  reproducible.
• Readable history (Workstream B) closes the loop the other way.

On issue #952 — natural language to SQL Copy Section

I don't think Harlequin should embed an LLM. No API keys, no model selection, no
provider coupling, no support burden for someone else's rate limits.

But there's a middle path that fits Harlequin's plugin ethos and answers the actual
request: a configurable external-command hook. Let the user define, in their config,
a command that receives the current buffer (and optionally the catalog) on stdin and
whose stdout is inserted into the editor. Then Ctrl+G runs whatever the user
already trusts — claude -p, llm, a local model, a company-internal endpoint. Same
shape as the existing external-editor request (#767,
#769), and it inherits their
credentials rather than asking for new ones.

Stories Copy Section

• F1. As an agent, I draft a migration and hsql open it in the human's TUI for
  review rather than executing it myself.
• F2. As a human, I explore interactively, then copy a reproducible hsql command for
  my agent to parameterize and run in CI.
• F3. As a human, I press one key in the TUI and my own AI tool rewrites my query —
  using my own credentials, not Harlequin's.

11. Roadmap Copy Chapter

Because the CLI is a separate command, every milestone here is purely additive to
harlequin and can ship in 2.x minors. Nothing waits for a major version.

On M1. Implementation is planned in detail in
the M1 technical plan: the module architecture, the
refactors that have to land first, and an eight-PR sequence across two releases. A few
figures and defaults in this document were corrected against measurements taken while
writing it; §8 there lists them. PRs 1–6 shipped and PR 7, the docs topic, is in flight;
PR 8, the agent eval suite, was not built.

On M2. Likewise in the M2 technical plan, across three
releases rather than one: config and self-description first, since they fix bugs that
exist today and need no adapter change at all; then the catalog, with search implemented for
both in-tree adapters rather than left as a stretch; then the safety flags, which need
additive members on the adapter contract and therefore an ecosystem rollout. The amendments
marked above are the corrections that came out of measuring it. This table's M2 row is
otherwise all subtraction: the introspection surface is mode options rather than
subcommands, and describe, fmt, --dry-run, --single-transaction, --password-stdin,
recursive --depth, the node budget, child counts and the compact format are all cut —
each for a reason given in §7 of that plan.

On M0. Name availability is the only thing in this plan that someone else can take
while we deliberate, which is why it's a milestone rather than a task. It's also cheap
enough to do this week.

A few things worth getting right the first time:

• Metapackage, not a second copy of the artifact. hsql should be a small
  distribution whose only dependency is harlequin. Republishing the same built wheel
  under a second name would put duplicate modules on disk and conflict outright if a user
  installed both.
• Version it independently and float the dependency — hsql 0.1.0 requiring
  harlequin>=2.x — so it doesn't need a release every time Harlequin cuts one. (Also
  reversed once M1 shipped: it now takes Harlequin's version and pins it. See the status
  note below.)
• No placeholder console script. It's tempting to ship an hsql command that prints
  "coming soon," but from M1 the harlequin distribution itself provides that entry
  point, and two installed distributions claiming the same script name is a mess to
  unwind. pip install hsql giving you Harlequin, plus a README that says what's coming,
  is honest enough. (M1 shipped, and the metapackage does declare the script now — see
  the status note below. The half of this that held up is "no placeholder": it points
  at Harlequin's entry point rather than standing in for it.)
• Reserve the same name anywhere else Harlequin is published or packaged while we're at
  it.

This is a genuine, working package rather than a squat, which also keeps it on the right
side of PyPI's naming policy.

Status: the metapackage lives in packaging/hsql/, and M1 changed two of the decisions
above.

It declares hsql = "harlequin.hsql:main" — the same entry point Harlequin's own script
uses, so the two console scripts are byte-identical and co-installing them is a no-op.
Without it, uvx hsql still ran, but uv warned that the executable came from a
dependency and suggested uvx --from harlequin hsql, on the one command line this
package exists to make short.

And it ships with Harlequin rather than on its own cadence: publish.yml builds and
uploads both from one release, and the metapackage carries Harlequin's version number and
pins that release exactly — release.yml sets all three numbers with uv version and
uv add, and test_packaging.py fails the release PR if they ever disagree. So hsql 2.9.0 is harlequin 2.9.0, whichever name a caller installs by. The independent cadence
was the right call while this package was a reserved name and nothing else; once it
points at an entry point that only some releases have, a floating floor is a promise the
metapackage cannot keep.

--read-only sits in M2 rather than M1 only because it requires an adapter-interface
addition and therefore an ecosystem rollout; if that lands early, pull it forward. It's
the flag that makes everything else socially acceptable.

If only three things get built Copy Section

1. hsql with -c / -f / -o / --format, a strict stdout/stderr split,
   documented exit codes, and honest truncation.
2. hsql catalog --format compact.
3. llms.txt + raw markdown docs + the "Headless & Agents" topic + the skill.

That trio takes an agent from "reinvents psql badly" to "writes a correct query against
any of fifteen databases on the first try."

12. Risks and open questions Copy Chapter

• Two front doors can drift. The whole bet rests on one shared execution core. If
  hsql grows its own query logic, or the TUI keeps a parallel path, we've built two
  products. Mitigation: extract the core in M1 and refactor the TUI onto it in the same
  milestone, not later.
• Discoverability of hsql. Nobody guesses it exists; harlequin -c is guessable.
  Mitigated by the unknown-option hint, the docs topic named for the capability rather
  than the binary, homepage positioning, and the skill. Worth watching — if support
  questions suggest people can't find it, revisit.
• Name mindshare. hsql competes with HyperSQL in search results. No PATH or PyPI
  collision, but SEO for "hsql" will take work; docs should always spell it "hsql, the
  Harlequin CLI" on first use.
• Adapter ecosystem lift. --read-only, validate_sql, catalog search, and any bulk
  fetch_descendants are optional adapter methods. hsql must degrade gracefully and
  say which adapter lacks what — hence capability flags. Third-party adapters will lag;
  plan for a long tail.
• Catalog fetch cost is the sharpest edge in the plan. The default is safe by
  construction (§6), but --depth 2 on a wide schema is an N+1 walk on most adapters,
  and it's also the single most useful catalog call an agent can make. Child counts, the
  node budget, and --stats are what keep that honest until adapters can serve it in
  one query.
• Cold start. Measured rather than feared, and worse than this section assumed: a
  bare harlequin --version takes 1.3s today. Lazy entry-point resolution is a
  requirement of M1, not a contingency — load_adapter_plugins() imports every installed
  adapter on every invocation, worth 160ms with four installed and unbounded as the
  ecosystem grows. An import-linter rule in this repo would also never have caught the
  largest single cause, which is upstream: harlequin.adapter pays 265ms for
  textual-fastdatatable, whose backend needs no Textual but whose package __init__
  imports the DataTable widget. See the M1 technical plan.
• Memory. Materializing an Arrow table for a large export is the same failure as
  #875, and it is not fixed in M1. See the streaming note in §5.
• Secrets. info --json, config show, error messages, and history are all leak
  surfaces, and agents paste output into transcripts. Mitigated structurally by the
  secret option type in §7 (#667) rather than by remembering to redact at each site —
  but note that connection strings and query history sit outside that mechanism and need
  their own handling.
• Two help surfaces to keep coherent. harlequin --help and hsql --help share the
  adapter option matrix and must not disagree. Generate both from the same option
  definitions.
• Docs drift. spec --json output and the site's config spec must be generated from
  the same source, or they'll disagree within two releases.
• MCP maintenance. Only worth it as a thin wrapper. If it starts growing its own
  query logic, that's the signal we got the layering wrong.

13. How we'd know it worked Copy Chapter

• hsql -c "select 1" against DuckDB completes in < 300ms, tracked in CI.
• The hsql module graph contains no textual import, enforced in CI.
• An agent eval suite in CI: a handful of realistic tasks ("find the top 10 customers
  by revenue") run against fixture databases with a small model, scored on wrong-flag
  retries and task completion. This is the metric that actually matters, and it's the one
  that would keep the CLI honest as it grows. I'd build it in M1 with three tasks and
  expand it.
• An agent given only hsql --help completes a query task with zero wrong-flag
  retries.
• llms.txt is served, and every docs page is fetchable as clean markdown.
• #524, #667, and #898 closed; #952 answered without embedding an LLM.
• Qualitatively: someone writes "just use hsql, it works the same everywhere" in a thread
  about getting an agent to talk to a database.

Plans Hsql Warm Sessions Design (`docs/plans/hsql-warm-sessions-design.md`) Copy Section

Warm sessions for `hsql` — a design for `hsql --serve`

A design for a long-lived hsql process that holds an open connection, so that repeated
invocations pay neither the import cost nor the connection cost again.

Revised against 2.13.0, with M1,
M2 and M3 shipped and SSH
tunnels landed; the first draft was written against 2.9.0, when M2 was still a plan. Every
number below is re-measured on this checkout — Python 3.10.15, Linux, four adapters
installed — on a machine about a third slower than the one the first draft used, so read
the ratios rather than the absolutes. §10 lists what the revision changed, for a reader
who has the first draft in their head. PR 1 has shipped; §9 says what is in it.

Bottom line up front. The idea works, and the win is bigger than the quarter-second it
was proposed to recover — but only if one constraint is treated as the load-bearing one:
the client must not import anything Harlequin owns. The client built to that rule and
shipped in PR 1 answers select 1 in 27ms against 348ms cold, and 20 sequential
invocations in 0.55s against 7.5s. A client that imported click and nothing else
would already be near 65ms, and one that imported harlequin.config near 90ms (§2) — so
the difference between a good version of this feature and a mediocre one is decided by the
first few lines of the console script, not by the protocol.

The other finding is that speed is the smaller half. A warm process is a session:
temp tables, SET, search_path, and open transactions survive between invocations. That
is a genuine feature for an agent — and it is also the reason this cannot be turned on by
default, cannot be auto-started in v1, and cannot be a silent fallback. §5 is the part of
this document I would argue about; §1–§2 are mostly arithmetic.

Recommended shape, in one line:

hsql --serve prod ./warehouse.db      # foreground, holds the connection
HSQL_SESSION=prod hsql -c "select 1"  # 27ms, same bytes, same exit code

1. Where the 350ms actually goes Copy Chapter

hsql -c "select 1" against in-memory DuckDB, best of twenty runs of the real console
script:

Instrumenting the same work phase by phase, cumulative from process spawn:

That last row is mostly two more imports — tree-sitter for the splitter and wcwidth for
the layout. Measured in a process that has already paid for all of them, connect() is
10.8ms and execute() + fetch() + layout together are 5.0ms.

So the database work is 5ms of a 350ms invocation. Everything else is getting ready to
do it. M1 already took the cheap wins — lazy entry points, the lazy
textual-fastdatatable __init__, deferred harlequin.query — and what is left is
genuinely irreducible on the cold path:

• pyarrow, 103ms with the backend. harlequin.query normalizes every result set
  through create_backend(), which is the whole reason both front ends agree about what a
  row is. Removing it means a second normalizer, which the M1 plan rejected for good
  reasons that have not changed.
• The driver, 70ms for duckdb. Not ours.
• click, 36ms. Ours to remove in principle, at the cost of hand-rolling the option
  surface that adapters plug into. Not worth it.
• CPython itself, 17ms. The floor for any Python process on this machine.

So: even a version of hsql that imported nothing of Harlequin's beyond click and its
config would land near 200ms, and one that also dropped click near 165ms. There is no
sequence of import fixes that reaches 50ms. That is the honest case for a second
process, and it is worth stating in the plan because the alternative — "keep optimizing" —
has been the right answer up to now and stops being the right answer here.

1.1 And `connect()` is 11ms only because DuckDB is a library Copy Section

The table above understates the win, because the adapter it measures opens a file. The
in-tree number is connect() at 10.8ms for in-memory DuckDB, and a local file or SQLite
is the same order. Every adapter that speaks to a server — Postgres, MySQL, BigQuery, Trino,
Databricks, Snowflake — pays TCP setup, a TLS handshake, and an auth round trip on each
connect(), and for the OAuth-shaped ones a token exchange on top. Those are the
connections a human hands an agent, and they are the ones where holding the connection open
matters more than holding the imports.

This design should therefore not be sold on "350ms → 27ms against DuckDB." It should be
sold on the invocation cost stops depending on how far away the database is, with the
DuckDB number as the floor case rather than the typical one.

2. What a warm session costs instead — measured, not projected Copy Chapter

The first draft measured a throwaway prototype. These are the shipped client from PR 1
— the real main(), the real harlequin.hsql.client, the real frames — against a stub
server that speaks protocol.py and returns fixed bytes without touching a database, so
the number is the invocation's cost rather than a query's. Same machine, twenty runs:

The warm client's 27ms decomposes almost entirely into things that are not the feature:

The round trip is about a millisecond. The client is ~95% interpreter start-up, which
means the protocol can afford to be careful — framing, a handshake, a version check, a
capability exchange — without moving the number. It also means the ceiling is 17ms unless
the client stops being Python, which §7 leaves as future work rather than pretending it is
free.

25 modules over a bare interpreter is the whole of it, and that is why §6's guard
counts modules rather than milliseconds: a count is the same on every machine, where a
wall clock on a shared CI runner is not. For scale, on the same interpreter import click
is 100 modules, harlequin.config 141 and harlequin.hsql.diagnostics 149.

2.1 The constraint this puts on the entry point Copy Section

import click costs 36ms and import json costs 9ms. Against a 27ms invocation those are
133% and 33%. So the rule is not "the client should be lean," it is:

harlequin.hsql.main() must decide whether this invocation belongs to a session, and
hand off to a stdlib-only module, before it imports anything that is not stdlib.

That is the single most important line-level decision in this document, and the first
draft under-scoped it. Three things were on the entry point's path, not one, and PR 1 took
all three off:

• click, 36ms. main() opened with import click. It now looks at HSQL_SESSION and
  --session first — a sys.argv scan and an os.environ lookup, both free — and only
  falls through to build_cli() when there is no session.
• harlequin.hsql.diagnostics, 68ms. The bigger one, and invisible until measured:
  harlequin/hsql/__init__.py imported ExitCode from it at module scope, and that
  module reaches harlequin.redact → harlequin.config → msgspec. Deferred into the cold
  path, where it was already being imported anyway.
• typing, 10ms. harlequin/__init__.py imported it, and importing any
  harlequin.* submodule executes that file. Every annotation there is a string under PEP
  563, so the module now declares TYPE_CHECKING = False itself and imports nothing at
  all. The four modules on the client's path do the same.

A consequence worth stating, because it looks like a violation of a rule this repo has:
the client writes its own stderr rather than going through diagnostics._write(),
which is otherwise "the one line every message leaves through". Nothing is lost. That
function exists to redact, and the client holds no secret to redact — it reads no profile,
loads no adapter, and never sees a connection string. The server's stderr arrives already
redacted and is copied through untouched.

It also gives the guard a shape CI can hold, exactly as the "no Textual in hsql" rule
does: an import-linter contract saying the client's three modules reach nothing of
Harlequin's and nothing third-party, plus a run-time test that counts modules against a
bare interpreter. One caveat the first draft missed: import-linter cannot express "this
package's __init__ may not import click at module scope" — a contract's source module
matches its descendants — so that half is a run-time test only
(test_deciding_whether_an_invocation_is_warm_does_not_import_click). Without both, the
first contributor who needs a config value in the client re-adds 68ms and nobody
notices.

3. Why a daemon, rather than the four cheaper things Copy Chapter

3.1 Not "keep optimizing imports" Copy Section

§1 measured the floor at ~165–200ms. Rejected on arithmetic.

3.2 Not a batch mode (`hsql --repl`, statements on stdin) Copy Section

One process, many statements, no daemon, no socket, no security model. It is strictly
simpler and it is the right tool for a migration script.

It does not serve the caller this feature is for. An agent working through a Bash tool gets
one command per turn and cannot hold a pipe open across turns; a shell script that
wants to interleave hsql with jq and git cannot either. Batch mode solves "many
statements known up front." The daemon solves "many invocations, each decided after seeing
the last one's output" — which is what an agent loop is.

Worth noting they compose: hsql --serve is a batch mode whose batch arrives over a
socket.

3.3 Not MCP instead — but this is the closest call Copy Section

M6 in the product plan is hsql mcp: a long-lived process an agent talks to over stdio,
running on the same M1 execution core — still the last milestone, with M4 (handoff, in
three releases) and M5 (streaming) ahead of it. It is already a warm session. It holds a
connection, it pays the imports once, and it needs no socket, no file permissions, no
daemon lifecycle, and no uid check, because the agent harness owns the process and the pipe
is the security boundary. If the only persona were "an agent whose harness speaks MCP,"
this document should be closed and M6 pulled forward.

Two reasons it is not:

1. Most agent harnesses give you Bash, not MCP — and even the ones that speak MCP run
   in contexts (CI, a container's entrypoint, a Makefile, a postCreateCommand) where the
   available interface is a command line. The product plan's own P1 persona is defined as
   "usually inside a coding agent's Bash tool."
2. The audience is not only agents. A human running twenty hsql calls in a shell loop
   against Snowflake gets the same win, and MCP does nothing for them.

The right relationship is that hsql mcp should be a client of this server, not a
parallel implementation of it. Both are "a warm connection plus a request loop"; the only
difference is the transport and the serialization. If M6 grows its own connection lifecycle
the plan has two of them, which is the same mistake the plan already refuses to make with
the TUI and the CLI. That is an argument for building the session layer first and letting
MCP land on top — and it is the strongest argument in this document for doing this at all.

3.4 Not a connection pool Copy Section

A pool would let several clients run at once. It also destroys the thing that makes a
session useful — a temp table created on connection 3 is invisible to the next request that
lands on connection 1 — and it turns "which connection am I on" into a question the caller
has to think about. One session is one connection. §4.5 handles concurrency by serializing
rather than by multiplying connections, and §7 leaves read-only pools as future work.

4. The design Copy Chapter

4.1 One command, two roles Copy Section

hsql --serve NAME [CONN_STR] [connection options] runs the server in the foreground.
Not a start/stop/status subcommand family, and not a self-daemonizing fork:

• Foreground is what every supervisor already knows how to run — &, systemd, a
  container entrypoint, docker compose, a SessionStart hook, tmux. Self-daemonizing
  processes reimplement all of that badly, and orphaned ones holding a live warehouse
  credential is exactly the failure this feature must not have.
• It keeps the mode-option shape M2 already settled on (--catalog, --info, --spec,
  --config MODE) rather than introducing the subcommand ambiguity that shape exists to
  avoid: hsql takes CONN_STR positionally, so hsql server foo.db would be ambiguous
  in the same way hsql catalog was.
• --serve takes the session name as its value, so one flag carries both "be a server"
  and "be this server."

Why `--serve` and `--session`, and not two nouns

The first draft of this document paired --server NAME with --session NAME, and that is
the wrong pair: in database vocabulary "server" and "session" are near-synonyms, and
neither word tells a reader which of the two processes it belongs to. The client half is
effectively pinned — HSQL_SESSION is the ambient spelling and nobody wants to write
HSQL_SERVER for a thing that is not one — so the question is only what the server's flag
is called, and the answer should be a verb:

--serve is a mode option; --session is not. M2's grammar is that mode options say
what an invocation is (--catalog, --info, --spec, --config MODE) and plain
options modify a normal invocation. hsql --session prod -c "select 1" is an ordinary
query invocation — it just runs somewhere warm — so --session is a plain option, and
--serve joins the modes. The grammar already encodes the asymmetry the names need.

Considered and not taken:

One consequence of the rename, applied through the rest of this document: the daemon's own
knobs lose the --server- prefix they only carried to disambiguate — --idle-timeout,
--max-lifetime, --queue-timeout — since they appear only next to --serve. And the two
that are client operations are named for what the caller is addressing:
--session-status and --session-reset, both of which are things you ask a running
session, not things you configure a server with.

What `--serve` accepts, and what it refuses

No, --serve does not accept -c — and the reason generalizes into the rule that
defines the two roles. Every flag hsql has falls into exactly one of three groups, decided
by when the question it answers can possibly be answered:

The partition is exact and it is symmetric: each side refuses the other's group. That
turns §4.4's differing-connection-options rejection from a one-off into a special case of a
single rule, and it gives both refusals an error that names the right invocation rather than
just the wrong one — principle 7, applied to the feature's most likely first mistake:

hsql: -c is a per-request option, and --serve takes none.
      Start the session, then send it a query:
        hsql --serve prod ./warehouse.db
        hsql --session prod -c "select 1"

The one real argument for -c on --serve is an init script — SET search_path,
CREATE TEMP TABLE, INSTALL/LOAD — run once at connect time, which is a genuinely
useful thing for a session to do. It is already served, by the better mechanism:
--init-path (-i) is an adapter option on both in-tree adapters, so it is
connection-time, group 1, and comes along for free. It is also the more correct of the two,
because it runs inside connect(), where the adapter decides what "init" means for its
database and where a failure is a connection failure rather than a request failure. A
-c on --serve would be a second, worse spelling of something that works today.

Adapter options are allowed on `--serve`, and should still be discouraged

They have to be accepted: the server is the only process that connects, so refusing them
would mean the only way to serve anything but a local file is a profile.

But --serve changes their risk profile, and the docs should say so. A server's command
line is long-lived and visible in ps. A password on a one-shot invocation is exposed for
350ms; the same password on an eight-hour session is exposed for eight hours, to every
process on the box. So:

• -P is the spelling the docs should show. A profile is already "how to connect to my
  warehouse," and hsql --serve prod -P prod should be the normal form with the flag-soup
  version as the exception.
• Warn when a secret-typed option is passed literally to --serve, detected through
  the declarative secret= that shipped with #667 rather than a hand-written list of flag
  names. A warning rather than a refusal in v1, because refusing needs env interpolation
  (#898) to have shipped first so that there is a documented alternative for the container
  case. Tightening it to a refusal once #898 lands is a reasonable later call, and a cheap
  one.

The SSH options are connection-time, and a session is where they pay off

They landed after the first draft, and they fit the partition without an exception: a
tunnel is opened once, on the way to connect(), so --ssh-host and its four companions
are group 1. This is the case a warm session is most worth having — a tunnel costs an
ssh handshake on every cold invocation, and holding it open is the larger half of the win
§1.1 describes.

Two consequences the server has to carry rather than invent:

• The tunnel is part of what the server holds open, so §4.7's "the server is a
  credential" reads on it too, and --max-lifetime bounds it.
• --ssh-allow-reuse is already a CLI_ONLY_SSH_KEYS key — a config file may not turn
  off the check that the local port is not already someone else's listener, because config
  files are discovered in the working directory. That reasoning does not weaken on a
  server; it is the same flag, read the same way.

Clients opt in two ways, and the distinction matters:

A name that could never name a session — one with a / or a space in it, which §4.4's
validation refuses — is the same shape of answer one step earlier: exit 2, because a
bad name is a bad argument rather than a server that is down. Native Windows is the same
again: explicit exits 2 (it can never work here), ambient warns and runs cold.

An environment variable is a preference — the human or the harness set it once, and an
hsql call that still works when the server is down is the behavior that does not break a
script. A typed --session is an assertion, and silently running cold against an assertion
would mean silently losing the temp table the caller was counting on. Same reasoning as
merge_profile_with_cli(): a value the caller actually typed carries intent that a
default does not.

The fallback warning is not optional and not suppressible. "It got slow and
my temp tables vanished" must never be something the caller has to guess at.

4.2 The client is stdlib-only, and forwards argv verbatim Copy Section

harlequin/hsql/
  __init__.py     main(): session check, then click or the client. Stdlib only.
  session.py      which session an invocation names, and where its socket is.
                  Stdlib only, and read before anything else.
  client.py       stdlib only: socket, struct, sys, os. No click, no json, no
                  harlequin beyond session.py and protocol.py.
  protocol.py     framing + the frame vocabulary, shared -- but written so that
                  client.py's half needs no imports the client cannot afford.
  server.py       accept loop, one worker, lifecycle, the uid check.
  cli.py          unchanged, plus --serve / --session

The first three of those shipped in PR 1. session.py is separate from client.py
because the decision has to be cheaper than the client: a cold invocation reads it, finds
no session, and pays two modules and no socket for the question.

The server parses the flags; the client does not. The client scans sys.argv for the
three things it cannot avoid knowing about —

• --session NAME (strip it; it is the client's own flag),
• -f - / --file - (read stdin and send the bytes, because the server has no stdin),
• --session-status (send it as its own frame; §4.10 is answered by the session rather
  than by the command, and it must be answerable while a request is running),
• nothing else —

and forwards the rest opaquely. Every other flag, including --help, including a
misspelling, is parsed by the server, by the same build_cli() the cold path uses, and
produces the same diagnostic. There is no second parser to keep in step, which is the
failure mode this design would otherwise be most likely to have. hsql --session prod --badflag should print the same message and exit the same 2 as hsql --badflag, and it
will, because it is the same code.

The request carries what the server cannot know: argv, the client's cwd, stdin bytes if
asked for, NO_COLOR, and whether the caller's stdout and stderr are terminals
(the two halves of what --color auto reads: it checks sys.stdout.isatty() and
NO_COLOR, and on the server every stream is a buffer, so auto would otherwise mean
"never" however the caller's terminal looks). TERM is not carried — nothing in the
output path reads it. Deliberately not the
client's whole environment — the server has its own, an adapter option's envvar resolves
against the server's, and shipping the caller's environment across a socket is a credential
leak with no upside.

The cwd is doing more work there than it looks. Config discovery is cwd-dependent — home,
then the user config dir, then the cwd, later winning — so the server must resolve a
client's -P and --config-path against the client's directory, not its own. Otherwise
hsql --session prod -P local run in a project would silently read the server's
pyproject.toml, and a caller would get a profile they did not write. This costs nothing —
the cwd is already in the request for -o (§4.3) — but it has to be deliberate, because the
natural implementation resolves against the process that is running. A profile is then
subject to the same partition as typed flags: its connection-time keys go through §4.4's
comparison, its per-request keys apply normally, and -P needs no special case on either
side.

4.3 What crosses the wire is bytes, not rows Copy Section

The response is three things: stdout bytes, stderr bytes, exit code. The server runs
harlequin.query and hsql.output exactly as the cold path does, writing into a buffer
instead of click.open_file("-", mode="wb"), and the client copies that buffer to its own
stdout and exits with the code.

This is the decision that makes the feature safe. It means:

• The client cannot format anything, so it needs no pyarrow, no layouts, no --format
  knowledge, and no ability to disagree with the cold path about a timestamp.
• Byte-equivalence is testable (§6). Every golden-format snapshot can be asserted
  against both paths.
• --stats, truncation notices, and the row-cap notice need no protocol support —
  they are already stderr bytes.

Two things need care:

• Streaming. The buffer means a 500-row default costs nothing, but --limit -1 on a
  large table materializes twice: once as Arrow on the server, once as bytes. That is the
  same failure as #875 and it is not made worse here, but the frame format should be
  chunked from the start (many stdout frames, then a terminating exit frame) so
  that M5's streaming work has somewhere to land. Chunked framing is free now and
  impossible to retrofit without a protocol version bump.
• -o PATH. The server writes it, resolving a relative path against the client's
  cwd from the request. Not the client: --format parquet writes through duckdb, not
  through our byte stream, and routing it back through the socket would both double the
  memory and break output.py's "one binary stream, -o included" invariant. This is
  also the design's clearest statement that client and server share a filesystem —
  which a AF_UNIX socket already implies.

4.4 Named sessions, not hashed connection identity Copy Section

The tempting design derives the socket path from a hash of the connection identity, so a
client automatically finds the right server. Rejected:

• The hash would have to include connection-affecting options and exclude presentation
  ones (--format, --limit, --color), and that partition is not written down anywhere
  today — it would become a new thing to keep in step with every adapter's options.
• Profiles resolve differently depending on cwd, so the same command in two directories
  would hash differently, or the same hash would mean two databases.
• Hashing secrets to name a file is a bad habit even when the hash is sound.

Instead: the name is the caller's, and the server owns the identity. hsql --serve prod -P prod connects however -P prod says to, and records what it resolved to. A client
that sends connection-affecting options gets one of two answers:

• identical to the server's → served;
• different → rejected, exit 2, naming the flag that differs and what the server has.

Not "reconnect with the new options" (that is a different session, and silently swapping
the database under a caller is the worst possible behavior), and not "ignore them"
(a client that typed -a postgres and got DuckDB has been lied to).

Socket path: $XDG_RUNTIME_DIR/hsql/<name>.sock, falling back to $TMPDIR/hsql-/
(or /tmp/hsql-/) when XDG_RUNTIME_DIR is unset — which is macOS, and also WSL2
without systemd, and also this container (§4.8). The fallback is the common path, not the
exotic one, and the directory is created 0700 and owned by the user either way. The name is
validated as [A-Za-z0-9_-]{1,64} so it cannot escape the directory.

Not platformdirs, which is what the rest of Harlequin uses for a user directory and
what the first draft named here. It costs ~22ms to import — about the whole of what a warm
invocation is supposed to cost — so the derivation is a dozen lines of os.path in
session.py, shared by both halves so they cannot disagree about where the socket is. It
is also the smaller question than it looks: XDG_RUNTIME_DIR is the only path platformdirs
would resolve differently on Linux, and the fallback branch is the one this design has to
get right anyway.

A name that fits is not always a path that fits. sun_path holds 104 bytes on
macOS (108 on Linux), the directory counts toward it, and the TMPDIR macOS supplies is
itself ~50 bytes — so a name well inside [A-Za-z0-9_-]{1,64} can still be one no socket
can be bound to. The client measures the assembled path and refuses with a message that
names the limit, rather than letting connect() fail with a bare errno.

No --session-socket PATH in v1. Letting a caller name the socket file directly is the
obvious escape hatch and the obvious way to end up with a socket on a mode-0777 shared
directory, or on a filesystem that cannot host one (§4.8). If it ever ships it has to
re-derive §4.7's guarantees for a path it did not choose — verify the directory's owner and
mode, and refuse anything that is not a real local filesystem — which is most of the reason
it is not in v1.

4.5 One request at a time Copy Section

Measured, on this checkout: two threads calling execute() on one DuckDB connection
fail —

HarlequinQueryError('Invalid Input Error: Attempting to execute an
unsuccessful or closed pending query result')

— and this is not a DuckDB quirk. The adapter contract says nothing about thread-safety,
and the TUI has never needed it: its database work runs in @work(thread=True, exclusive=True) workers, which is to say one at a time, and cancel the previous one.
hsql --serve inherits that invariant rather than inventing a new one.

So: a single worker thread, and a request queue. A second client waits. Waiting is
bounded by --queue-timeout (default: the same value as the request's own --timeout,
which shipped in M2), and a client that times out in the queue exits 4 with a message that
says it never reached the database — which is a different fact from a query that ran too long, and the
caller needs to be able to tell them apart.

The accept loop stays responsive while a query runs, so --session-status and cancellation
(§4.6) are answerable mid-query. That is the whole reason the queue is in the server rather
than implicit in a single-threaded accept loop.

4.6 Cancellation, and a real gotcha in the contract Copy Section

Ctrl-C on a client must stop the query, not orphan it. The client catches SIGINT, opens
a second connection to the socket (the first is busy carrying the response), sends
cancel with its request id, and exits 130.

The server calls connection.cancel() when the adapter implements it. When it does not —
cancel() is an optional method, and HarlequinAdapter.IMPLEMENTS_CANCEL says whether it is real — the honest
behavior is to detach the client, exit 130, and say on stderr that the query is still
running on the server, because it is.

The gotcha, found while measuring: DuckDB's cursor reports an interrupt by returning None
from fetchall(), not by raising:

def fetchall(self) -> AutoBackendType | None:
    try:
        result = self.relation.to_arrow_table()
    except duckdb.InterruptException:
        return None

In the TUI that is fine, because the user pressed the button and the app knows. In server
mode a cancelled query and an empty result are the same bytes, so the server must
attribute cancellation from its own bookkeeping — "I called cancel() on request 7, and
request 7 came back empty" — rather than from anything the adapter says. Worth writing down
because the alternative is a caller who sees (0 rows) and exit 0 for a query that was
killed.

This is not a new mechanism, and PR 4 should not invent one. --timeout shipped in M2
and has exactly this problem: harlequin/hsql/timeout.py cancels, then reads its own
deadline between results and attributes the empty result itself, and
diagnostics.report_timeout() is the line that says so. The server's bookkeeping is the
same shape, one level up — "I called cancel() on request 7" instead of "the deadline
passed" — and it should read like it.

(A cleaner long-term fix is a distinguished return or exception on the contract, which is
an adapter-ecosystem change of the same shape as M2's five additive members. Out of scope
here; noted in §7.)

4.7 Security: the server is a credential, held open Copy Section

This is the part that deserves the most conservatism, because a running server is a live,
authenticated database connection that anything able to write its socket can use. There
is no password on the wire and there does not need to be — the socket is the credential.

• AF_UNIX only. No TCP, not even on loopback, not behind a token. Loopback TCP is
  reachable by every process and every container sharing the namespace, and a token file
  is a second secret to leak.
• The socket lives in a 0700 directory owned by the user, and the socket itself is
  created 0600 (bind, then chmod, or umask around the bind — file permissions on a UDS
  are honored on Linux and macOS but not on every kernel, which is why the directory mode
  is the real control).
• The server refuses a request whose argv names -f - and carries no stdin
  section. The client's argv scan is a scan, not a parse, and its set of
  boolean short flags is hsql's own — a boolean short an adapter declares is
  not in it, so -zf - for such a -z would forward no stdin. One check on the
  server turns every miss of that shape, including ones nobody has thought of,
  into an exit 2 rather than a silently empty script. The server never reads its
  own stdin.
• Both halves lstat the directory before trusting it. XDG_RUNTIME_DIR is
  guaranteed by the system; the TMPDIR fallback is not, and on a shared host another
  user can create /tmp/hsql-/ first and put a listener at prod.sock. What a
  client sends is argv, the cwd and piped stdin, and argv can carry a CONN_STR with a
  password in it, so the client refuses a directory it does not own privately — and the
  server does the same, since os.mkdir(mode=0o700) does nothing to a directory that
  already exists.
• SO_PEERCRED / LOCAL_PEERCRED uid check on accept. Reject and log any peer whose
  uid is not the server's. Belt and braces over the directory mode, and the thing that
  makes a shared /tmp misconfiguration fail closed.
• --max-lifetime (default 8h) and --idle-timeout (default 30m,
  0 to disable). A credential held forever because someone opened a tmux tab in March is
  the predictable bad outcome; these two make the default outcome "it goes away."
• No secrets in --session-status. It reports the adapter, the session name, the
  redacted connection identity, uptime, request count, and current state — through
  harlequin.redact and the declarative secret= that shipped with #667, not through a
  hand-written allowlist.
• An SSH tunnel is part of the credential. A session opened with --ssh-host holds an
  ssh process and a local listener for as long as it runs, and --max-lifetime is what
  bounds that too.

4.8 Windows — and WSL2, which is not the same question Copy Section

socket.AF_UNIX is not available in CPython on Windows, through 3.14. The alternatives
are a named pipe with a hand-built DACL, or loopback TCP with a token — the first is real
work with a security model that has to be got right on a platform none of the maintainers
run daily, and the second is the thing §4.7 rejects.

v1 is POSIX-only. hsql --serve on native Windows exits 2 with a message saying so, and
HSQL_SESSION on native Windows warns once and runs cold — which is exactly the
ambient-fallback behavior §4.1 already specifies, so nothing special is needed on the client
side. A named pipe transport is a clean later addition behind the same protocol.py, and
pretending otherwise in v1 would mean shipping a weaker security model to every platform to
accommodate one.

WSL2 gets the feature, because WSL2 is Linux. It runs a real kernel in a VM, its Python
is a Linux CPython with AF_UNIX present, and a server and client both inside it need
nothing special — no branch, no fallback, no separate transport. That matters more than it
sounds: the Windows users most likely to want a warm session for an agent are already
working inside WSL, so "POSIX-only" costs much less than the phrase implies, and the docs
should say native Windows rather than Windows everywhere so nobody in WSL concludes the
feature is not for them.

Two WSL-specific notes, both of which §4.4's path derivation already handles:

• Do not put the socket on a DrvFs/9p mount. Binding a unix socket under /mnt/c does
  not work. The socket lives in the runtime dir, so this is right by construction — but it
  is one of the reasons §4.4 keeps --session-socket PATH out of v1.
• XDG_RUNTIME_DIR is often unset under WSL2, since systemd is opt-in there. That is
  exactly the fallback branch §4.4 specifies (a 0700 hsql- directory under
  TMPDIR), and it is not hypothetical — it is unset in the container these
  measurements were taken in, too. The fallback is the common path, not the exotic one, and
  should be tested as such.

What does not work, and cannot be made to: a native-Windows hsql.exe reaching a
session running inside WSL2. That is not a socket-routing problem to solve — they are two
different Python installations on two different filesystems, and the Windows one has no
AF_UNIX to offer regardless. Stay on one side of the boundary.

4.9 Lifecycle, and the four ways this goes wrong Copy Section

The version check is stricter than it needs to be on purpose. hsql's output format is
documented as frozen, but "frozen" has always meant "across releases we intend"; a server
from a different release serving bytes a caller attributes to the installed version is a
category of bug nobody will diagnose quickly.

A literal, and not importlib.metadata.version("harlequin"), which costs ~43ms —
nearly twice the whole warm invocation. That makes it the second version uv version
cannot reach, beside the plugin manifest, so scripts/bump_plugin_version.py became
scripts/bump_versions.py and writes both; a test pins each to the installed version, so a
release that forgot fails in the release PR rather than in a user's shell.

4.10 Observability Copy Section

hsql --session prod --session-status returns JSON on stdout:

{"session":"prod","pid":8123,"version":"2.13.0","adapter":"duckdb",
 "connection":"/home/ted/warehouse.db","connection_options":{"read_only":true},
 "uptime_s":412,"requests":37,"state":"idle","queued":0,
 "transaction_mode":null,"ssh":null,"idle_timeout_s":1800,"expires_in_s":1788.4}

connection_options is the rest of §4.4's identity — what a request's own
connection options are compared against — masked by whatever each adapter declared
secret=. It holds what the session was given: an option it never named is one
its adapter defaulted for itself, which core cannot enumerate, so read_only appears
only for a session started with it. state is idle, busy, or unavailable for a
session whose connection a reset has to bring back. idle_timeout_s is the idle
clock as it was set, and expires_in_s the sooner of the two countdowns — with
the idle one left out while a request holds the connection, since a session in
the middle of a query is not idle for any length of time yet. Either is null
where the clock behind it is off.

Answerable while a query is running (§4.5), which is what makes "is it hung or is it slow"
a question with an answer. transaction_mode is on it because §5 says it has to be.

5. The thing that isn't speed: a session is state Copy Chapter

Everything above is mechanism. This section is the actual product decision.

A cold hsql invocation is hermetic: fresh process, fresh connection, no memory of the
last one. Every existing hsql behavior, doc sentence, and test assumes that. A warm
session is the opposite, and the difference is observable in ways a caller will not expect:

• Temp tables persist. create temp table t as select ... in one invocation is
  queryable by the next. This is the single most useful thing about the feature, and it is
  the one nobody will read the docs to discover.
• Session settings persist. SET, search_path, SET TIME ZONE, DuckDB PRAGMAs,
  installed extensions. A SET in invocation 3 changes the results of invocation 12.
• In-memory databases persist. hsql --serve scratch :memory: is a scratch warehouse
  that survives between calls — genuinely good, genuinely a semantic change.
• Transactions persist, and this is the sharp edge. A caller who runs BEGIN and then
  exits has left an open transaction holding locks, and every subsequent request in that
  session runs inside it. In a cold invocation the process exit rolled it back. Nothing
  rolls it back now.

Three responses, and I would take all three:

1. Never on by default, never auto-started, never a silent fallback. All three are
   already in §4.1. This is the reason for them: the feature changes semantics, so it must
   be something a caller asked for.

2. The server reports transaction state whenever it is not the default — on stderr
   after the request, and in --session-status. HarlequinConnection.transaction_mode
   already exists on the contract, so this costs nothing and turns the worst failure mode
   into a visible one. A session sitting in an open transaction is a thing the caller
   should be told about every single time, not once.

   This premise does not hold, and PR 5 shipped the narrower thing instead.
   transaction_mode is the adapter's Auto/Manual setting — a plain attribute that
   only toggle_transaction_mode() moves, which hsql never calls — so it says nothing
   about whether a transaction is open, and a begin left open is silent. The worst
   failure mode is therefore not visible, and --session-reset (item 3) is the only
   thing in this design that answers it. Detecting an open transaction needs a new
   member on the adapter contract, which is §7's "distinguished cancellation signal"
   shape of change: additive, ecosystem-wide, and not this feature's to make. §9's PR 5
   entry has what was shipped in its place.

3. hsql --session prod --session-reset rolls back, closes, and reconnects, without
   restarting the process (so the imports are still warm). This is the escape hatch for
   "the agent left the session in a weird state," and it is a much better answer than
   asking someone to find and kill a pid.

What I would not do is reset between requests. It would make the server semantically
identical to the cold path, which sounds safe until you notice it also throws away the
half of the win that is connect() — and against a warehouse, that is the larger half.

5.1 Documenting it as a session, not as a cache Copy Section

The docs framing follows from this: hsql --serve is not "hsql, but faster." It is "a
database session you can send commands to." The speed is a consequence. A caller who
holds the second model in their head will predict the temp-table behavior correctly; a
caller who holds the first will file a bug.

6. Testing: byte-equivalence, but only where it is true Copy Chapter

The risk this feature carries is not that it fails, it is that it drifts — that in some
corner (a NULL, a -t footer, an error's exit code, a truncation notice) the warm path and
the cold path produce different bytes, and a caller who set HSQL_SESSION in their
environment months ago gets a different answer than the docs describe.

Byte-equivalence is the guard for that, and it is a claim about one invocation, not about
a test. Two invocations against a session are supposed to diverge from two cold ones —
that is §5, and it is the feature. A parametrization that ran the whole suite both ways and
demanded identical bytes would fail on every test that invokes hsql twice, and the
tempting fix — quietly exempting those — would put the divergences nobody has thought about
in the same bucket as the ones we designed. So the guard has to be stated with its scope on
it:

Equivalence holds per invocation, from a fresh session. Given the same starting state,
one hsql invocation produces the same bytes and the same exit code whether it ran cold
or warm. Nothing is claimed about what the next invocation sees.

That precondition has to be manufactured, because the cold path gets it for free (a new
process every time) and the warm path does not. Each test gets a fresh session — one
server per xdist worker, --session-reset between tests — which is cheap, since a reset is
a reconnect and not a re-import. Pleasingly, it makes the test suite the primary consumer of
the escape hatch §5 proposes for humans, so the thing that rescues a wedged session is
exercised hundreds of times per run rather than never.

Three kinds of test, then, not one:

1. Equivalence — every single-invocation functional test, run twice, asserting identical
   stdout bytes, identical stderr bytes, identical exit code. The golden-format snapshots
   (test_golden_formats.py) come along for free, since they are single-file syrupy
   snapshots written in binary and already assert on bytes. This is the drift guard, and it
   is the suite that runs on every PR if only one can.
2. Divergence, marked and asserted — a multi-invocation test cannot be in group 1, and a
   marker (@pytest.mark.session_divergent, registered in [tool.pytest.ini_options]
   beside online, use_cache and py12) takes it out. The marker's cost is that it does
   not merely opt out: the test must assert what the warm behavior is, not just that it
   differs. A temp table created by invocation 1 is no such table cold and a result set
   warm, and both belong in the assertions. Otherwise the exemption list becomes the place
   bugs live, which is exactly what the exemption exists to prevent.
3. Session semantics — tests that only exist on the warm path, because they are the
   feature: state surviving between invocations, an open transaction being reported, a reset
   clearing it, two clients queueing, a session refusing a client whose connection options
   differ.

The marker earns its keep twice over, because the set of tests carrying it is an
executable version of §5. §5's list of what statefulness changes is prose today; the
marker turns it into an enumerated one that cannot silently grow. A contributor who
introduces a new divergence has to mark a test, and the marker is the review trigger that
asks whether §5 should have mentioned it.

Alongside those:

• An import-linter contract, next to the existing hsql does not reach the TUI one,
  saying the client's three modules may not import click or anything of Harlequin's
  outside themselves. Its other half is not expressible there — a contract's source
  module matches its descendants, so "harlequin.hsql.__init__ may not import click at
  module scope" cannot be written — and is a run-time test instead. Both shipped in PR 1.
• A cold-start benchmark for the client, next to the existing one for hsql
  (scripts/cold_start.py), plus a run-time test that counts the client's modules against
  a bare interpreter — a count, not a wall clock, because the count is what a shared CI
  runner can hold. Both shipped in PR 1.
• Deterministic lifecycle tests, not timing ones: stale socket, wrong uid (skipped
  unless the test runner can drop privileges), version skew, a client killed mid-query, two
  clients queued, a cancel that lands, a cancel against an adapter that cannot cancel.

One trap worth naming, since the suite runs under -n auto: a session-scoped server
fixture is cross-test contamination waiting to happen. A test that runs SET or leaves a
transaction open changes the result of an unrelated test that lands on the same worker
later, and it will present as a flake with no obvious cause. The per-test reset above is
what prevents it, and it is the reason the fixture must not be session-scoped for
convenience.

7. What's cut, and why Copy Chapter

• Auto-start. A client that spawns a server when it does not find one is the obvious
  ergonomic win and the obvious way to end up with orphan processes holding warehouse
  credentials that nobody remembers starting. It is also the feature that is easy to add
  later and impossible to remove, once scripts depend on it. Ship explicit-only, watch how
  people actually set sessions up, then decide. If it does land, it should be
  --session-autostart and it should inherit the idle timeout, not the max lifetime.
• Windows. §4.8.
• TCP, remote servers, containers-talking-to-hosts. AF_UNIX and a shared filesystem
  are assumptions of §4.3's -o handling as well as §4.7's security model. Removing them
  is a different product.
• Connection pooling / parallel requests. §3.4. A read-only pool is a plausible later
  addition, but only for --read-only sessions, where "which connection" is unobservable.
• Multiple databases per server. One session, one connection. Two databases is two
  --serve processes, and they cost 200MB of RSS each at most, which is not the
  constraint here.
• A session for the TUI. harlequin --session prod sounds appealing and is not: the
  IDE holds its own connection for its whole run, so it has nothing to gain, and it would
  put the TUI's evolving surface on the frozen protocol.
• A distinguished cancellation signal on the adapter contract (§4.6). Real, and an
  ecosystem change; it belongs with M2's additive members, not here. The server's own
  bookkeeping is a correct workaround in the meantime.
• A compiled client. 16ms of the 27ms is CPython boot, so a Rust or Go client would
  land near 2ms. It also means platform wheels for a project that ships pure Python, and
  it optimizes the half of the budget that is already smallest. Revisit only if someone is
  running thousands of invocations.

8. Risks Copy Chapter

• A second execution path is a second product. The mitigation is structural, not
  disciplinary: the server calls build_cli() and the same query/output code, and §6's
  equivalence suite fails if the bytes diverge. If the server ever grows its own
  formatting or its own flag handling, that is the signal the layering went wrong — the
  same signal the product plan already names for MCP.
• The credential-held-open surface. §4.7 is the mitigation and it is not complete:
  anything running as the same uid can use the session. That is the same trust boundary as
  the user's ~/.ssh and their Harlequin config, and it should be stated in the docs in
  those terms rather than implied.
• Statefulness surprising someone. §5. The transaction reporting is the mitigation I'd
  least want cut.
• It overlaps M6. §3.3 argues that is a reason to build this first and land MCP on top.
  If M6 is imminent, the sequencing question is real and should be settled before PR 1 —
  building both connection lifecycles would be the expensive mistake.
• A lifetime expiry mid-query leaves that query unreachable. The expiry path is the
  stop signal's, which closes the listener and unlinks the socket before it drains: the
  request in flight is still answered, but for the rest of its run no cancel and no
  --session-status can reach it, because both open a second connection to a path that
  is gone. Under SIGINT an operator is present and knows what they just did; under
  --max-lifetime it happens by itself, at eight hours, to whoever is mid-query. Draining
  before the unlink would need care around a client racing in on the still-live socket,
  so PR 5 left the ordering alone — but the docs should say that a lifetime expiry mid-query
  leaves the query uninterruptible until it finishes.
• Idle servers as an operational nuisance. Idle timeout defaults to 30m for this
  reason. Expect the first bug report to be "my session died between calls"; the answer is
  --idle-timeout 0, and the docs should say so before the report arrives.
• Nobody uses it. The honest risk. The feature is only reachable by someone who set it
  up deliberately, which §5 says it must be. Mitigations are the docs framing, a
  SessionStart-hook example, and the fallback warning — which is the one place a caller
  who should be using a session will be told it exists.

9. Shipping sequence Copy Chapter

Six PRs, in dependency order. The first two are the ones with the interesting decisions in
them; everything after is additive and independently revertible.

1. The entry-point split and the framing. Shipped. main() decides whether an
   invocation belongs to a session before it imports anything that is not stdlib;
   session.py (the decision, and the socket path), client.py (stdlib only) and
   protocol.py (chunked frames, handshake, version). No server yet — the client's only
   reachable behavior is the fallback path, which is testable on its own, and the rest is
   pinned against a stub server built on protocol.py. Ships the import-linter contract,
   the run-time guards, and the client's steps in scripts/cold_start.py.

   Two things moved out of it, both deliberately. --session is not yet a click
   option: the client intercepts it before click, so it works, but a flag --help and
   --spec advertise while no server can answer it would be worse than one they do not
   mention yet. And declaring it raises a question PR 2 has to answer anyway — a click
   parameter's name becomes a profile key, and session must not be one, because the
   decision is made before any config file is read. That is a CLI_ONLY_SSH_KEYS-shaped
   refusal, and it belongs with the flag. Until then no adapter reserves the spelling
   either; none of the four checked declares it (§4.1).

2. The server: accept loop, one worker, the queue, the uid check, socket lifecycle.
   hsql --serve NAME, the --session click option and its profile-key refusal,
   HSQL_SESSION, and the §4.1 flag partition on both sides. Ships §6's equivalence suite
   with it, not after it — this is the PR where equivalence is cheap to establish and
   expensive to retrofit.
   --session-reset lands here too, ahead of the rest of the lifecycle work: §6's
   fixture needs it to give each test a fresh session, so it is a dependency of the suite
   rather than a nicety, and the suite is the thing that proves it works.

3. Identity and rejection. Shipped. The server's recorded connection identity, the
   differing-options rejection, client-side profile resolution against the client's cwd
   (§4.2), --session-status. One question this PR settled that the first draft did not
   ask: whether HARLEQUIN_CONFIG_PATH travels with the request. It does. It is the
   caller's intent in the same way --config-path is, and a session that honored the flag
   and ignored the variable would read a config file neither of them named — which is the
   same failure §4.2 forwards the client's cwd to avoid. It widens
   protocol.FORWARDED_ENV_VARS past "what --color auto needs", and the line that
   replaces PR 1's is narrower than it looks: the two variables on it name where a value
   comes from rather than being one, and neither is a credential.

   Two things this PR settled that the plan did not name. --session-status is not a
   request, because §4.5 promises it is answerable mid-query and a request cannot be:
   a served invocation borrows the process's cwd, environment and streams for its whole
   run, so a second one parsed alongside it would take them out from under a query. So
   the client sends it as its own frame — a third thing its argv scan knows about — and
   the server answers it off its own bookkeeping, which is the shape PR 4's cancel wants
   anyway. It stays a declared click option so that --help, --spec and the partition
   still describe it, and so a cold invocation is refused rather than ignored; and it
   joins CLI_ONLY_SESSION_KEYS, because a profile could otherwise set it only for the
   runs that never reach a session. The status's transaction_mode is null while the
   session is busy: reading it means asking a driver a question on a second thread,
   which is the thread-safety §4.5 refuses to assume. idle_timeout_s and expires_in_s
   are null until PR 5 can answer them, rather than absent, so the document's shape does
   not change under a caller.

4. Cancellation. Shipped. SIGINT → cancel frame → connection.cancel(), the
   IMPLEMENTS_CANCEL = False path, and the DuckDB fetchall() -> None attribution.

   Two things this PR settled that the plan did not name. The request id is the
   client's, not the server's: a server-assigned one would have to reach the client
   before its query started, and the only frame ahead of a request is the handshake,
   whose shape has to stay readable by every release for the §4.9 version check to
   work — so it is a sixth section of the request, eight bytes from os.urandom().
   And a request is cancellable from the moment its argv arrives, not from the moment
   it takes its turn. A caller who gives up while their request waits behind the one
   ahead is cancelling something the session is holding, and nothing else would stop it
   running when its turn came.

   That makes the load-bearing invariant one about windows, because
   HarlequinConnection.cancel() is connection-wide: the window in which a request is
   registered and marked started has to be exactly the window in which it holds the
   turnstile. Widen it at either end and a cancel naming a request that has already
   finished interrupts whichever query holds the connection now — and marks that one's
   bookkeeping not at all, so the empty, error-free result the interrupt produces is one
   nobody can attribute and the caller reads as an empty table. Two things keep the
   windows equal: the server releases a request from its bookkeeping before it leaves
   the turnstile, and cancel() holds the lock a request takes to start across the
   driver call, so nothing can start between the check and the interrupt. The client's
   half of the same rule is that the send and the relay are under one try, or an
   interrupt landing between them exits 130 having sent no cancel at all.

5. Lifecycle and state hygiene. Shipped. --idle-timeout, --max-lifetime,
   transaction-mode reporting, and the secret-on-a-server-command-line warning (§4.1).

   Three things this PR settled that the plan did not name. A status poll is not
   use of a session, so it does not touch the idle clock: an agent that watches a
   session it has stopped sending queries to would otherwise keep the credential
   open by watching it, which is the outcome the clock exists to prevent. The clock
   is therefore held while a client is connected rather than reset when one
   arrives — which is also what keeps a session from stopping under a client that
   has connected and not yet said what it wants, since a -f - reads its script
   after the handshake and nothing else can see such a client. (Review caught that:
   the first cut counted parsed requests alone, and a slow producer upstream of
   -f - got the socket unlinked under it.) The
   idle clock does not run while a request does, and the lifetime clock does —
   the first is about a session nobody wants and the second bounds the credential,
   so a query that takes an hour is not idle and is not exempt either; a lifetime
   that runs out mid-query stops the accept loop and lets the request in flight
   finish, which is the path a stop signal already took, at the cost §8 now records.
   And the default a
   transaction mode is compared against is the one the session connected in,
   re-read on every --session-reset, because the contract offers no way to ask an
   adapter which of its modes is its default. Review then found the premise above it
   does not hold either: the mode is a setting nothing hsql runs can move, so what
   shipped reports a session moved out of band and makes no claim about open
   work — §5 item 2 has the correction. A narrow feature honestly described, rather
   than a safety net that is not there.

6. Docs. Shipped. /docs/hsql/sessions on harlequin.sh, written per §5.1, with the
   SessionStart-hook example on it. Not optional and not last in spirit — a feature that
   must be deliberately adopted is a feature that lives or dies by its docs.

   One thing this PR settled that the plan did not name: the hsql --help mention was
   already there, since the flags became click options in PR 2 and the epilog gained a
   Sessions block with them. What the docs owed instead was the two things --help cannot
   hold: what a session remembers (§5), and the three pages that were true only of a cold
   invocation — psql.md's "no session variables" and "No Interactive Session", and the
   tutorial's every-invocation-connects framing.

M2 has shipped, and M3 with it. M4 is in flight in three releases and touches cli.py
elsewhere (--open, the hooks, query history), so sequencing against it is a scheduling
question rather than a technical one. The one real ordering constraint is still §3.3's:
settle the relationship with M6 first.

10. What this revision changed Copy Chapter

For a reader who has the first draft in their head. Nothing about the shape of the design
moved; the changes are the arithmetic, the flags that shipped in between, and three things
the first draft got wrong about the entry point.

• Re-measured on 2.13.0 (§1, §2). The machine is slower than the first draft's, so the
  absolutes moved and the ratios did not: 348ms cold, 27ms warm, and the database work is
  5ms of the 348.
• §2 is the shipped client, not a prototype. PR 1's numbers replace the throwaway's.
• The entry point had three imports on it, not one (§2.1). click was the known one;
  harlequin.hsql.diagnostics (68ms, via redact → config → msgspec) and typing
  (10ms, via harlequin/__init__.py) were not. Together they were more than twice the
  invocation they sat on.
• One import-linter contract, not two (§2.1, §6). The second is not expressible; it is
  a run-time test.
• The socket path is derived with stdlib, not platformdirs (§4.4), which costs ~22ms.
• The version literal lives in protocol.py, and a release script writes it (§4.9).
• M2's flags are no longer conditional — --read-only, --timeout, the mode options
  and declarative secret= all shipped, and --timeout's implementation in
  hsql/timeout.py is the precedent PR 4's cancellation attribution should follow (§4.6).
• The SSH options are in the partition (§4.1), as connection-time, and the tunnel is
  part of what §4.7 calls a credential held open.
• --session waits for PR 2 to become a click option (§9); PR 3 answered the
  HARLEQUIN_CONFIG_PATH question with "it travels."

Plans M1 Hsql Technical Plan (`docs/plans/m1-hsql-technical-plan.md`) Copy Section

M1 Technical Plan — `hsql`, the headless CLI

Implementation plan for milestone M1 of Harlequin for agents.
That document is the product spec: what hsql is, why it's a separate command, and what
its contract with an agent looks like. This one is about how we build it, what has to be
refactored first, and in what order it can ship.

M1's scope, from the roadmap: second console script; extract the shared execution
core; import-linter rule and cold-start benchmark in CI; -c, -f, stdin, -o, -F +
shorthands, default --limit 500, truncation notices, --stats, exit codes,
--on-error, --color/NO_COLOR, plain errors; unknown-option hint on harlequin;
the seeded "Headless & Agents" docs topic. Closes
#524.

Bottom line up front. A working prototype of the refactors below runs a
select … union all select … against DuckDB — interpreter startup, config discovery,
adapter load, connect, execute, fetch, normalize, render — in 220–242ms with Textual
never imported, down from a 1335ms floor today. Nothing in M1 needs new normalization
or parsing code: the two hard pieces already exist in textual-fastdatatable and
tree-sitter-sql, and both can be reached without Textual.

1. Where the code actually is today Copy Chapter

Everything below was measured on this repo at 2.8.0, not estimated.

1.1 There is no execution core to share Copy Section

Query execution lives entirely inside the Textual app, as two threaded workers that
communicate by posting messages:

• Harlequin._execute_query (src/harlequin/app.py:1077) — loops over statements,
  calls connection.execute(q), applies cur.set_limit(...), collects cursors, posts
  QueriesExecuted.
• Harlequin._fetch_data (src/harlequin/app.py:1154) — drains each cursor with
  fetchall() + columns(), times the batch, posts ResultsFetched.

There is no importable function. hsql cannot call any of this, and a naive copy is
exactly the "two products" failure §12 of the product plan warns about.

The split into execute all, then fetch all is deliberate and good — it's what lets the
TUI show "query executed" before results materialize. The extracted core should preserve
that shape rather than flatten it, so the TUI can adopt it without changing its
concurrency model.

Harlequin._split_query_text (app.py:1135) is a naive text.split(";"). It is dead
code — nothing calls it. The live splitter is CodeEditor.selected_queries()
(components/code_editor.py:36), which uses tree-sitter through textual-textarea. See
§3.2 — the grammar it uses is reachable without any of that.

1.2 Cold start is 1.3 seconds, and the reasons are structural Copy Section

$ time harlequin --version
harlequin, version 2.8.0
real    0m1.335s

That is the cheapest possible invocation of the current CLI. The 300ms target is not a
matter of shaving milliseconds; five separate things each pull in the whole TUI:

Cause 2 recurs in the adapters themselves: harlequin_duckdb/adapter.py:14 and
harlequin_sqlite/adapter.py both import textual_fastdatatable.backend at module
scope, so even a perfectly clean hsql would pay for Textual the moment it loaded an
adapter.

harlequin/catalog.py:6 imports textual.message.Message for NewCatalog and
NewCatalogItems — two TUI-only messages sitting in a module every adapter imports.
Neither is used outside app.py and components/data_catalog/database_tree.py.

1.3 `textual-fastdatatable` is ours, and its backend has no Textual in it Copy Section

We control this package, which changes the answer to the hardest problem in M1 (§3.1).
Measured on the installed 0.16.0:

• backend.py, format.py, and column.py contain zero references to Textual. The
  imports are pyarrow and rich.
• The only thing that pulls Textual is textual_fastdatatable/__init__.py importing
  data_table.DataTable.
• With that __init__ made lazy: from textual_fastdatatable.backend import create_backend
  drops from 265ms/359 modules/Textual to 217ms/201 modules/no Textual.
• Most of the remainder is pyarrow.parquet, imported at module scope but used only by
  ArrowBackend.from_parquet. Deferring it into that classmethod takes the import to
  ~140ms/183 modules.
• textual>=7.3.0 is a required dist dependency. Nothing in the backend uses it. A
  future lean hsql install would need that made an extra.

1.4 tree-sitter is already installed, and usable without Textual Copy Section

textual-textarea requires textual[syntax], which brings tree-sitter and
tree-sitter-sql. The editor's separator query is one line
(code_editor.py:22): (";" @semicolon).

Driving that grammar directly, with no Textual and no textual-textarea, measures at
28ms to import, 92 modules, and 22ms to split a 2000-statement script. It gets
string literals, line comments, block comments, quoted identifiers and unicode right.
It gets $$-dollar-quoting wrong — and so does the TUI today, because it's the same
grammar and the same query.

Two further things fell out of driving it directly. Tree-sitter is used in exactly one
place in Harlequin — that one query — and nowhere else. And its Point.column is a
byte offset, which the editor feeds to a function expecting characters. See §3.2.

1.5 Two places already write results to stdout that shouldn't Copy Section

• harlequin/plugins.py:54 — a bare print() when a plug-in fails to import. Under
  hsql -F csv > out.csv that lands in the CSV.
• harlequin/exception.py:68 pretty_print_warning — uses rich.print, i.e. stdout.
  Reached from locale_manager.py and windows_timezone.py, both on the CLI path.

pretty_print_error already gets this right, with a comment saying why. The contract
exists in intent; M1 makes it true.

1.6 Export is coupled to a widget Copy Section

harlequin/export.py:20 copy() takes a ResultsTable (a Textual widget) and reaches
into table.backend.source_data. The five exporters underneath it are pure
pa.Table → file. The column-deduplication step in between is data logic wearing a
widget's clothes.

1.7 The prototype Copy Section

I applied the §1.2 fixes plus the §1.3 upstream fixes and ran a script that does what
hsql -c will do — config discovery, single-adapter load, connect, execute with
set_limit(limit + 1), create_backend(data, max_rows=limit), write CSV to stdout:

220ms  238ms  242ms      textual=False   306 modules

With load_adapter_plugins() left as-is (all four installed adapters imported) the same
script takes 377–398ms — which is cause 5 measured in situ, and the reason §3.4 is not
optional. from harlequin import Harlequin and from textual_fastdatatable import DataTable
both still work after the changes; those are load-bearing for third-party adapters and
for the TUI respectively. All patches were reverted.

2. The four obstacles, stated plainly Copy Chapter

1. No execution core exists. It has to be extracted from the app before hsql can
   have one, or we ship two implementations on day one.
2. Every import path leads to Textual — in this repo and one level upstream.
   Fixable, cheap, measured; it must land first, because it's also the guard that keeps
   the fix from silently regressing.
3. Building the CLI requires importing every installed adapter. The only unbounded
   cost, and the only one that gets worse as the ecosystem grows.
4. Text layout doesn't exist, and is coupled to a widget where it does. The TUI
   renders into a DataTable; export.py writes files but only from a ResultsTable.
   Nothing arranges a result as table, markdown or vertical text.

Note what is not on this list, though earlier drafts of this plan had all three:
normalizing adapter output, parsing SQL into statements, and serializing values. Each
looked like a new subsystem and turned out to be an existing one we already own —
create_backend, tree-sitter-sql, and duckdb respectively. The residual work in M1 is
smaller than it first appears, and the parts that remain are the parts that are genuinely
ours: the CLI, the execution flow, and three text layouts.

3. Target architecture Copy Chapter

Six new or reshaped libraries, plus one upstream change. Each owns a domain and has a
stated interface; none of them is a bag of helpers.

textual-fastdatatable
  __init__.py     MOD  lazy DataTable, so `backend` is reachable without Textual
  backend.py      MOD  pyarrow.parquet deferred into from_parquet()

harlequin/
  statements.py   NEW  locating statement boundaries in SQL text
  query.py        NEW  execute SQL, get results, without a UI
  layout.py       NEW  arranging already-serialized text for a reader
  export.py       MOD  writing a result set to a file, serialized by duckdb/pyarrow
  config.py       MOD  + merging CLI values over a profile
  plugins.py      MOD  + naming and loading one adapter without importing the rest
  options.py      MOD  renderer imports deferred; declaration stays import-light
  hsql/           NEW  the command
    __init__.py        main()
    cli.py             the click command and its two-phase adapter resolution
    diagnostics.py     everything written to stderr, and the code we exit with

The one-line statement of the architecture, which is also a testable rule:

stdout is written only by harlequin.layout and harlequin.export. stderr is
written only by harlequin.hsql.diagnostics.

3.1 `harlequin.query` — the execution core, over the existing backend Copy Section

@dataclass(frozen=True)
class Statement:
    sql: str
    index: int  # 0-based position in the submitted script


@dataclass(frozen=True)
class RowLimit:
    max_rows: int | None  # None = unlimited
    detect_overflow: bool  # fetch one extra row so truncation is knowable


@dataclass
class ExecutedStatement:
    statement: Statement
    cursor: HarlequinCursor | None  # None for DDL/DML


@dataclass
class ResultSet:
    statement: Statement
    columns: list[tuple[str, str]]  # (name, short type) from cursor.columns()
    backend: DataTableBackend  # from textual_fastdatatable
    truncated: bool
    elapsed: float

    @property
    def row_count(self) -> int: ...  # backend.row_count
    def rows(self) -> Iterator[Sequence[Any]]: ...
    def text_columns(self) -> pa.Table: ...  # every value CAST AS VARCHAR; see §3.3


def execute(
    connection: HarlequinConnection,
    statements: Sequence[Statement],
    limit: RowLimit,
    on_error: Literal["stop", "continue"] = "stop",
) -> Iterator[ExecutedStatement]: ...
def fetch(executed: ExecutedStatement, limit: RowLimit) -> ResultSet: ...

ResultSet wraps a DataTableBackend rather than reimplementing one.
HarlequinCursor.fetchall() returns AutoBackendType, which is Any — in practice a
pa.Table (duckdb), a RecordBatch, a Sequence[Iterable] (sqlite), a
Mapping[str, Sequence], or a polars or pandas frame. create_backend() already
normalizes all of those, it is the normalization the TUI uses, and per §1.3 it needs no
Textual to run. Writing a second normalizer for hsql would put the most drift-prone
code in the codebase — "what counts as a row, what counts as null, how does a nested
struct come out" — in two places, and the disagreement would surface as hsql and the
TUI showing different data for the same query. That is the worst possible place for
these two front doors to diverge.

Superseded, after M1 shipped. The collision described below was resolved by
#1026: the display cap has its own
key, viewer_max_rows, limit is the hard fetch limit in both commands, and the text
layouts grew a soft cap of their own (--display-rows). harlequin --limit now fills
in the Run Query Bar's limit and checks it, so the TUI does detect overflow — the
premise below, that it always fetched every row, stopped holding the moment a user
checked that box, and no longer holds at start-up either. See AGENTS.md for the
current vocabulary. The rest of this section is the reasoning as it stood.

Harlequin already has two different limits, and hsql wants the other one. This is
worth spelling out, because the names collide and I had it wrong in an earlier draft:

The soft limit is why the TUI needs no overflow detection and never did: because it
fetched every row, it knows the exact total and already says so —
"(Showing 100,000 of 3,412,887 Records)" at results_viewer.py:234. That's strictly
better than anything an inference could give it. My earlier suggestion that it show
"100,000+" was wrong.

hsql -l is the hard limit. Fetching a million rows to print five hundred would
defeat the entire point of a limit for an agent — time, memory, and egress. There is also
no viewport headless, so the soft limit has no meaning: hsql has exactly one limit
concept, and it's the fetch-side one. Same flag name, different semantics between the two
commands; the docs need one sentence about that rather than a pretense that they match.

Which is why truncation needs limit + 1. set_limit(n) then fetchall() returns at
most n rows and tells you nothing about whether an n+1th existed; exactly n is ambiguous.
So execute() requests limit + 1 and fetch() calls create_backend(data, max_rows=limit).
DataTableBackend already distinguishes row_count (capped) from source_row_count
(what it was handed), so truncated = source_row_count > row_count — the mechanism
exists, we just feed it one extra row. This is why the limit is a value object with an
explicit detect_overflow rather than an int each caller interprets: principle 5 of the
product plan ("truncation must always be announced — never silent") is unimplementable
without it, and it costs exactly one row.

The asymmetry is real and defensible: the TUI can afford to know the exact total because
it already paid for every row; hsql deliberately hasn't, so it reports 500 of >500.
Under -l 0 there's no set_limit at all, so the count is exact and --stats reports
"truncated": false against a real number.

No adapter-interface change for limits in M1. The clean long-term answer is for the
cursor to say whether more rows exist — a has_more(), or a set_limit() that reports
back — instead of us inferring it from an extra row. That's a fifteen-adapter rollout for
a problem the existing API already solves, so it waits. set_limit() and the backend's
max_rows are both existing features, used as they are; the only thing hsql does
differently is pass limit + 1. If a later milestone adds a real capability, RowLimit
is the single place that changes.

Upstream first. The two textual-fastdatatable changes in §3 are a prerequisite, and
both are non-breaking: the lazy __init__ keeps from textual_fastdatatable import DataTable
working via PEP 562 __getattr__, and deferring pyarrow.parquet is invisible to
callers. Making textual an optional extra is a bigger change — it would let a future
lean hsql install skip Textual entirely, but it's a dependency break for existing
users and buys M1 nothing, since harlequin needs Textual regardless. Deferred.

The TUI's two workers become thin wrappers: _execute_query calls execute(),
_fetch_data calls fetch(), both still post the same messages, and the DataTable gets
the same backend object it builds today.

3.2 `harlequin.statements` — one grammar, one query, two slicers Copy Section

Driving tree-sitter directly (§1.4) is better than every alternative I costed: 28ms
against sqlfmt's 85ms, no new SQL scanner to maintain, and — the part that matters —
it is the same grammar and the same query the TUI already uses, so hsql -f script.sql
and the editor cannot disagree about where a statement ends.

def find_separators(
    text: str,
) -> list[Point]: ...  # (row, character column) past each ";"
def split(text: str) -> list[Statement]: ...  # separators -> trimmed statements

tree-sitter and tree-sitter-sql become explicit, required dependencies of
harlequin (Ted's call). Both are installed for everyone today, transitively via
textual[syntax]; depending on them directly costs 28ms, makes the guarantee real
instead of incidental, and collapses this section from "two splitters, carefully kept in
sync" to one. The accepted cost is that Harlequin stops installing on a platform with no
tree-sitter wheel — which is why textual made it an extra, and which the
harlequin.sh/docs/troubleshooting/tree-sitter page currently exists to explain. That
page becomes obsolete; PR 7 removes it.

With that, there is exactly one splitter. Tree-sitter is used in precisely one place
in Harlequin today — code_editor.py, for the semicolon query, and nowhere else
(highlighting is textual-textarea's own business and unaffected). So the editor calls
find_separators(self.text) and slices as it does now, and we delete SEMICOLON_QUERY,
_semicolon_query, prepare_query, query_syntax_tree, has_shown_tree_sitter_error,
and the whole is_syntax_aware fallback branch.

The one thing given up is textual-textarea's incremental tree: the shared library
reparses. Measured, that's 0.6ms for a realistic buffer and 22ms for a pathological
2000-statement one, and selected_queries() runs on submit rather than on keystroke. A
worthwhile trade for deleting a divergence.

This fixes a live unicode bug in the TUI, which is why the shared library returns
character columns. Tree-sitter's Point.column is a byte offset; Textual's own
docstring says so, and it hands the raw nodes back. But get_text_range() takes
character columns, and code_editor.py:159 passes one straight into the other. Any
non-ASCII before a semicolon on the same line shifts the cut:

select '日本語';select 2
  splits today  -> "select '日本語';select"  |  "2"
  should be     -> "select '日本語';"        |  "select 2"

Both halves are then syntax errors. It's narrow — it needs non-ASCII before the
semicolon and a non-space after it, so .strip() usually absorbs the off-by-one — but
it's real, reproducible, and in main right now. Owning the byte→character conversion
in one place, once, is what stops hsql from inheriting it. Worth its own issue and a
changelog line, since it's a TUI bug fix that happens to fall out of this work.

One known bug we don't fix: create function f() … $$ … ; … $$ … splits inside the
dollar-quoted body. The TUI has it today, identically, because it's the grammar's. Its
own issue, not M1 scope — and a decent demonstration that sharing the grammar means
hsql inherits the TUI's behavior including its bugs, which is the definition of no
drift.

3.3 Output — duckdb serializes, `hsql` lays out Copy Section

/* Detailed source-code truncated for AI context efficiency. */
```python
# harlequin/export.py — writing a result set to a file, serialized by duckdb or pyarrow
def write_file(data: pa.Table, path: Path, format_name: str, options: Mapping[str, Any]) -> None: ...
def file_format_names() -> list[str]: ...

# harlequin/layout.py — arranging already-serialized text for a reader
@dataclass(frozen=True)
class LayoutOptions:
    header: bool = True
    footer: bool = True
    aligned: bool = True
    null_string: str | None = None
    color: bool = False

class Layout(Protocol):
    def write(self, result: ResultSet, out: TextIO) -> None: ...

def get_layout(name: str, options: LayoutOptions) -> Layout: ...
def layout_names() -> list[str]: ...

export.py keeps its name and its job, loses the ResultsTable coupling from §1.6, and
gains tsv and jsonl as thin option variants of formats it already writes. layout.py
is new and small: three layouts, no type knowledge, no duckdb.

LayoutOptions is what makes psql's flag algebra fall out for free: -t is
header=False, footer=False, -A is aligned=False, and -tA needs no special case
because it was never one — two independent options, which is how psql users already think
about them. --no-header and --null-string also reach export.py as header and
na_rep, so the same four flags mean the same thing in every format.

3.4 `harlequin.plugins` — naming an adapter without importing it Copy Section

def adapter_names() -> list[str]: ...  # entry-point names only
def load_adapter(name: str) -> type[HarlequinAdapter]: ...  # imports exactly one
def load_adapter_plugins() -> dict[str, type[HarlequinAdapter]]: ...  # existing

adapter_names() reads entry-point names without calling ep.load(), so it costs
nothing. That's what makes the two-phase parse in §3.6 possible, and per §1.7 it is worth
~160ms of the prototype's runtime — the difference between 380ms and 220ms.

3.5 `harlequin.config` — merging CLI values over a profile Copy Section

The merge in cli.py:315-327 (drop every parameter whose click source is DEFAULT,
then config.update(kwargs)) is the precedence rule both commands must implement
identically. It moves to config.py, which already owns Profile:

def merge_profile_with_cli(
    profile: Profile,
    cli_values: Mapping[str, Any],
    explicitly_set: Container[str],
) -> Profile: ...

/* Detailed source-code truncated for AI context efficiency. */

class ExitCode(IntEnum):
    OK = 0; QUERY = 1; USAGE = 2; CONNECTION = 3; TIMEOUT = 4; INTERRUPT = 130

def exit_code_for(error: BaseException) -> ExitCode: ...
def report_error(error: BaseException) -> None:          # "hsql: error: ..."
def report_truncation(result: ResultSet, limit: RowLimit) -> None: ...
def report_stats(...) -> None:                            # one line of JSON

Two things worth stating because they're easy to get wrong:

• The truncation notice fires under -t. -t suppresses stdout chrome; it does not
  suppress warnings. A flag that silently defeated truncation reporting would undo the
  principle it's meant to coexist with.
• Exit codes are hsql's contract, not Harlequin's, so the mapping lives here rather
  than in harlequin.exception. HarlequinQueryError → 1, HarlequinConnectionError → 3,
  HarlequinConfigError → 2, KeyboardInterrupt → 130.

4. Sequencing Copy Chapter

One upstream release, then two Harlequin releases. The first Harlequin release contains
no new user-facing surface at all; the second is the feature. Every PR in the first is
independently reviewable, independently testable, and behavior-neutral for the TUI.

Upstream — `textual-fastdatatable` (0.17) Copy Section

PR 0. Lazy DataTable in __init__.py via PEP 562; pyarrow.parquet deferred into
ArrowBackend.from_parquet. Add a test asserting "textual" not in sys.modules after
importing textual_fastdatatable.backend — otherwise the next contributor re-adds a
convenience import at the top of __init__ and nobody notices for a year. Both changes
are non-breaking; harlequin then bumps its pin from ==0.16.0 to ==0.17.0.

Blocks PR 2 and PR 3. Doesn't block PR 1, so it can be running in parallel.

Release A — the plumbing (2.9) Copy Section

PR 1 — Import hygiene, and the guard that keeps it.
Lazy harlequin/__init__.py via PEP 562, preserving every current name.
AutoBackendType under TYPE_CHECKING in adapter.py, harlequin_duckdb,
harlequin_sqlite. NewCatalog/NewCatalogItems move to harlequin/messages.py, with
a module-level __getattr__ shim in catalog.py so existing imports keep working
without importing Textual. Deferred renderer imports in options.py; _CustomValidator
moves to copy_widgets.py, which is already the option-widgets library. Fix the two
stdout leaks from §1.5. Add import-linter contracts and the cold-start benchmark script.

Ships value on its own: importing harlequin_duckdb goes from 691ms to 56ms, which
every adapter's test suite and every library consumer feels.

PR 2 — harlequin.statements and harlequin.query. Add tree-sitter and
tree-sitter-sql to dependencies. Build the splitter; refactor CodeEditor onto it and
delete its fallback branch, fixing the unicode column bug in the same move; land the
tricky-SQL corpus. Then the execution core, with _execute_query and _fetch_data
refactored onto it. Delete the dead _split_query_text. Snapshot tests are the safety
net for the TUI refactor.

This is the only PR in Release A with user-visible behavior in it — the unicode fix — so
it gets a changelog entry and its own issue while the rest stay silent.

PR 3 — export.py decoupled from ResultsTable, plus harlequin.layout. Extend
export.py with tsv and jsonl (option variants of formats it already writes) and the
temp-file-to-stream path; add the three layouts and ResultSet.text_columns(). Golden
files per format. No user-visible change; neither has a consumer until PR 5.

PR 4 — merge_profile_with_cli, adapter_names, load_adapter. harlequin's own
CLI refactored onto the first; behavior-neutral, covered by the existing 392 lines of
test_cli.py.

Why these land on main rather than accumulating on a feature branch: they're
behavior-neutral and individually tested, and the alternative is a long-lived branch
rebasing against a moving TUI — which is where extraction refactors go to die.

Release B — the command (2.10) Copy Section

PR 5 — hsql. Console script, two-phase parse, -c/-f/stdin, -o, -F and the
shorthands, -t/-A/--no-header/--null-string, -P, -l defaulting to 500,
--result, --on-error, --color/NO_COLOR, --stats, exit codes, truncation
notices, plain errors.

PR 6 — Discoverability. Unknown-option handler on harlequin (-c → "did you mean
hsql -c?"), and the -t nord hint on hsql when an unparseable conn_str matches a
known theme name.

PR 7 — Docs. The "Headless & Agents" topic (a PR against tconbeer/harlequin-web),
README, changelog. Retire the troubleshooting/tree-sitter page, which describes a
degraded mode that no longer exists (§3.2).

PR 8 — The agent eval suite. Three tasks against fixture databases, scored on
wrong-flag retries and completion. §13 of the product plan calls this the metric that
actually matters; three tasks is enough to start and to expand against.

Ordering rationale. Everything the output contract touches ships at once, in PR 5.
The product plan's premise is that format and exit codes are a frozen API — a
half-shipped hsql -c that we "improve later" is the one thing here that can't be
undone. Internals can be improved after release; the contract can't.

5. Testing Copy Chapter

Deterministic guards, not timing assertions. The cold-start number is the headline,
but a wall-clock assertion on a shared CI runner is a flaky test that gets deleted within
a month. So:

• Deterministic and blocking: import-linter contracts — the hsql module graph must
  not contain textual or questionary. Plus subprocess tests asserting
  "textual" not in sys.modules after importing the headless core, and the same test
  upstream after importing textual_fastdatatable.backend. These fail identically on
  every machine.
• Informational: the benchmark reports hsql -c "select 1" against DuckDB into the
  job summary, failing only past a loose ceiling (600ms) that catches a real regression
  without catching a busy runner.

The import contracts are load-bearing. Timing is downstream of them.

Beyond that:

• CliRunner tests for hsql, following the existing test_cli.py patterns and its
  adapter-mocking fixtures.
• Golden files per format (syrupy snapshots, as everywhere else in this repo)
  against a fixture result set with nulls, unicode, wide
  values, duplicate column names, and zero rows. Zero rows is the case that separates
  "the query returned nothing" from "the query failed" — A3 in the product plan — and
  the one most likely to be wrong in a format nobody exercised.
• A type-coverage fixture — timestamp, timestamptz, date, decimal, float, boolean,
  blob, list, struct, map, and null in each — rendered into every format, as golden
  files. This is the table in §3.3 turned into a test: it's what would catch a change in
  duckdb's rendering, and it's the reason -F table and -F csv can be asserted to
  agree cell for cell.
• text_columns() agreement: for that fixture, the strings -F table prints are the
  same strings -F csv writes.
• stdout purity: for each format, stdout bytes identical with and without --stats;
  a query error leaves stdout completely empty.
• Determinism: identical bytes whether stdout is a pipe, a file, or a pty; \n
  regardless of platform, which now means the temp-file copy is binary; identical bytes
  under LC_ALL=de_DE.UTF-8, the regression test for §3.3's locale trap.
• -o PATH and > PATH produce identical bytes for every format — the one thing the
  temp-file path could plausibly get wrong.
• hsql -tAc "select count(*)" returning a bare number and nothing else as its own
  named test. It's the idiom the scripting audience will try first.
• Truncation: exactly-at-limit, one-over-limit, and under-limit, on both duckdb and
  sqlite — the two implement set_limit differently enough to matter.
• The tricky-SQL corpus against statements.split() and, through selected_queries(),
  the editor — same fixtures, one splitter underneath. Includes non-ASCII before a
  semicolon with no following space (the §3.2 bug, which fails on main) and a
  skip-marked dollar-quoting case so the known bug is recorded rather than forgotten.

6. Decisions Copy Chapter

Settled.

• Limits stay on the existing machinery. cursor.set_limit() and the backend's
  max_rows, exactly as they are — no new adapter-interface method for truncation in M1
  (see §3.1).
• No overflow detection in the TUI. It doesn't need any: --limit there is a soft
  display cap over a full fetch, so it already reports the exact total (§3.1).
  detect_overflow=False, no snapshot churn in PR 2.
• Streaming is out. Not designed around, not partially built — see §3.3 and §7.
• tree-sitter and tree-sitter-sql become required dependencies. One splitter, no
  fallback, the troubleshooting page retired (§3.2). Accepted cost: no install on
  platforms without a tree-sitter wheel.
• --help is adapter-agnostic, listing adapter names and pointing at --help -a X
  for connection options (§3.6). Ted's design; better than what I proposed.
• --result defaults to all, in every format (below).

On `--result` Copy Section

The flag only matters when one invocation produces more than one result set — hsql -c "select 1; select 2", or the common -f script.sql shape of a few setup statements
followed by the query you actually care about. all emits every result set; last emits
only the final one; N picks one. With a single statement, which is most invocations,
the flag does nothing.

The product plan's proposal was all for text formats and last for data formats. The
problem isn't the values, it's that the same command would emit different things
depending on -F, which is exactly the context-dependence principle 4 exists to prevent.
Agreed on consistency, so:

--result all is the default everywhere. Formats that can't hold multiple result sets
say so rather than lying. Concretely:

• One result set: works in every format, no change.
• Multiple, into table / markdown / vertical / jsonl: all of them, in order.
• Multiple, into csv / tsv / json: error, exit 2 —
  hsql: error: 3 result sets, but csv holds one; use --result last or --result 3.

Writing two headers into one CSV is silent corruption of the same family as silent
truncation, and an error that names the fix costs an agent one retry with a clear signal.
The alternative — silently defaulting to last — throws away data the user asked for
without saying so.

Still open. Nothing.

7. Explicitly not in M1 Copy Chapter

--read-only, --timeout, --dry-run, --single-transaction (M2 — the first three
need adapter-interface additions and an ecosystem rollout). Every subcommand: catalog,
describe, fmt, spec, info, config, history, open, mcp. Streaming of any
kind — it starts at the cursor interface, not the format layer (M5; see §3.3 and §8).
Unifying the TUI's export dialog onto write_file (mechanical, but it moves a UI).
Making textual an optional extra of
textual-fastdatatable, and the separate lean hsql distribution it would enable (§4
of the product plan: near-reversible, indefinitely deferrable). Fixing dollar-quoted
statement splitting (its own issue, affects the TUI equally).

Bare hsql with no arguments prints help and never launches the TUI, from the first
release.

8. Corrections to the product plan Copy Chapter

All of these are applied in harlequin-for-agents.md in the same PR as this
document; they're recorded here so the reasoning behind each change is written down
somewhere.

• §4 says the full adapter option matrix on hsql --help is "unavoidable and shared."
  It's avoidable. hsql --help lists adapter names and points at --help -a X, which
  is both a token saving and the fix for the only unbounded startup cost (§3.6).
• §12 lists cold start as a risk with "may need lazy entry-point resolution." Not a
  maybe: it's worth 160ms of a 380ms invocation with four adapters installed, and it
  grows without bound. A requirement of M1, not a contingency.
• §12's cold-start risk should name textual-fastdatatable, not just "Textual creeping
  in." The 265ms the adapter interface pays today is upstream of this repo, and an
  import-linter rule here would never have caught it.
• §5's truncation guarantee needs the limit + 1 fetch to be implementable at all.
  Worth stating in the product plan, because it's exactly the kind of thing that gets
  designed out as an implementation detail and then quietly breaks the promise.
• §5 says "Large results stream" and lists it as an M1 concern. Nothing streams in
  M1, and it can't: HarlequinCursor.fetchall() materializes the whole result before any
  writer exists, so the format layer is downstream of the problem. Streaming is an M5
  change to the cursor interface, and M1 doesn't design around it — rows() returns an
  iterator because that's the natural shape, not as a hedge. Agreed with Ted; the product
  plan's §5 bullet should move to M5 outright.
• §5's truncation footer says … 500 of N rows, and N is unknowable. Under a hard
  fetch limit we deliberately never learn the true total — that's the point of the limit.
  The footer is 500 of >500 -- strictly greater, which the limit+1 fetch proves and
  the conventional 500+ would understate -- and only -l 0 yields an exact count. (The TUI can say
  100,000 of 3,412,887 because its limit is a display cap over a full fetch; see §3.1.)
• §5 lists --limit as though it means the same thing in both commands. It doesn't:
  soft display cap in the TUI, hard fetch limit in hsql. Worth one explicit sentence in
  the docs rather than letting a reader assume they match.
• §5's --result default is format-dependent (all for text, last for data), which
  makes the same command emit different things depending on -F. It's all everywhere,
  with an error rather than silent corruption where a format can't hold more than one
  result set (§6).

Plans M2 Hsql Technical Plan (`docs/plans/m2-hsql-technical-plan.md`) Copy Section

M2 Technical Plan — self-description and safety

Implementation plan for milestone M2 of Harlequin for agents,
following the M1 technical plan. The product plan says what
hsql is for; M1 built the command and the execution core under it. This one is about
letting an agent learn a database before it queries one, and about the flags that let a
human hand an agent a warehouse credential.

M2's scope, from the roadmap: catalog (one level below the match), info --json,
spec --json, config validate/show/schema/init, capability flags, the
secret option type and declarative redaction
(#667), env interpolation
(#898), --read-only, --timeout, and
a published JSON Schema. find + implements_catalog_search is promoted out of the stretch
column and ships with implementations for both in-tree adapters; optional fetch_descendants
stays future work.

Cut in review (§7 has the reasoning for each): describe, fmt, --dry-run,
--single-transaction (which M1 §7 had deferred to M2), --password-stdin, recursive
--depth, the node budget, child counts, round-trip counting, and the compact and tree
layouts.

Where M1 actually got to. PRs 1–6 shipped: the import-hygiene work, the statement
splitter and execution core, export/layout, the shared profile-merge and one-adapter
loading, hsql itself, and the two commands pointing at each other. PR 7 (the docs topic,
against tconbeer/harlequin-web) is in flight. PR 8, the agent eval suite, is not being
done — so M2 has no automated measure of "an agent used the right flag first try," and the
guards below are all deterministic ones.

Bottom line up front. M2 needs no new execution path, no new output format family, and
no second config parser: catalog listings are result sets, so --format, -o, -t/-A,
--display-rows and --stats already apply to them, and the row cap that keeps a 400-row
listing readable is the one M1 already shipped. What M2 does need is five additive members
on the adapter contract — four fields and one method — because the features that would
otherwise be built on guesswork are unimplementable without them: the catalog knows a column
is ## but not that it is DECIMAL(18,2), no adapter can say whether it enforces read-only,
and none can answer "where does orders live" without a walk. Those are measured,
below, not feared.

On the shape of the CLI. Every capability here is a mode option on the one hsql
command — --catalog, --find, --info, --spec, --config MODE — not a subcommand
(Ted's call). It matches harlequin --config, and it removes an ambiguity subcommands
would have created: hsql takes CONN_STR positionally, so a database file named
catalog would have collided with the verb. There is no collision to rule on if there is
no verb. See §3.1.

1. Where the code actually is today Copy Chapter

Measured on this checkout — 2.8.1 plus the unreleased M1 work — on Python 3.10.15, Linux,
with four adapters installed (duckdb, sqlite, postgres 1.3.1, mysql 1.3.0). Wall times are
the best of five runs of the real console script.

hsql -c "select 1"      257ms
hsql --help              90ms
hsql --version           87ms
harlequin --version     767ms

M1's targets held. Everything M2 adds has to keep them, which is most of why §3 is shaped
the way it is.

1.1 The catalog is lazy in the right way, and resolving a path is itself a walk Copy Section

get_catalog() returns the top level only — one query — and every level below it is a
separate fetch_children() call on the item you are standing on. That is exactly the shape
the product plan asks for, and it is why one level below the match is cheap.

But there is no way to reach an item except by fetching its ancestors. To call
fetch_children() on mydb.analytics.orders you need that item object, which only
SchemaCatalogItem.fetch_children() can hand you, which needs the schema item, which needs
the database item. I built a DuckDB file with 400 relations in one schema and counted the
round trips (spelling the calls in the mode-option syntax of §3.1):

So a listing costs one round trip per path segment, plus one — not the "exactly one
fetch_children()" the product plan's §6 claims. On local DuckDB that difference is 20ms;
on BigQuery or Trino it is four network calls where the plan promised one.

What recursion costs depends entirely on where you stand, which is why the product
plan's blanket --depth N is the wrong knob. Depth 2 from the root fetches every database's
schema names: one round trip per database, and databases are few. Depth 2 from a schema is
one round trip per relation, and the last row above is what that means — 403 round trips
and 2.3 seconds against a local file, on the call an agent would most want to make.

M2 therefore ships one level, always, and no --depth at all (§3.2). The honest
recursive answer needs fetch_descendants() on the adapter contract so a database can
answer in one query (§3.8), and that is where the flag should arrive from.

1.2 Catalog depth is adapter-specific, and nothing declares it Copy Section

DuckDB and Postgres are four levels (database → schema → relation → column). SQLite is
three (database → relation → column). BigQuery's own hierarchy is project → dataset →
table, which is four levels wearing different words. Nothing in harlequin.catalog says
how deep a catalog goes or what a level is called.

So --path cannot document its grammar as database.schema.table. Paths are positional
segments whose meaning is the adapter's, and the type label on each item (db, sch,
t, v) is what tells a reader what they got. That is a small design constraint with a
large documentation consequence, and it beats inventing a level vocabulary that would be
wrong for a third of the ecosystem.

1.3 The catalog knows the short type, and throws the real one away Copy Section

CatalogItem.type_label is documented as "a short (1-3 chars) label" — ## for a bigint,
s for a varchar, ts for a timestamp. It is a Data Catalog affordance: it has to fit in a
tree column next to the name.

So a listing of analytics.orders can say id ##, customer_id ##, total #.# and no more,
which tells an agent almost nothing and would send it back to select * limit 0 to find out
what it is joining on. The type an agent needs to write a join is not in the catalog
contract — only the glyph that fits in the IDE's tree.

The data is right there and discarded. DuckDbConnection._get_columns() selects
column_name, data_type from information_schema.columns and then does
type_label=self.connection._short_column_type(column_type), dropping column_type on the
floor. SQLite's is the same shape. So this is one field on CatalogItem, not a new query
(§3.2).

Child counts are a different matter, and M2 does not add a field for them. There is
nowhere to hang a count today, and there shouldn't be: no adapter can produce one without
fetching the children, so a child_count field would either be None everywhere or be a
fetch in disguise. The count that is knowable is len(children) once they are loaded —
which is what a listing already shows in its own footer. The product plan's "report child
counts wherever the adapter can get them cheaply" turns out to have no cheap case, and it
was proposed as a mitigation for a recursion M2 is not shipping anyway.

1.4 Capability flags have to be declared, and only some of them are Copy Section

Today's capability surface is three things on the adapter class: IMPLEMENTS_CANCEL,
implements_copy (derived from COPY_FORMATS, and deprecated), and provides_details.
Everything else — cancel, validate_sql, copy, transaction_mode — lives on the
connection, and the connection class is not reachable from the adapter class without
connecting to a database.

Nor can an override be detected reflectively. type(conn).validate_sql is not HarlequinConnection.validate_sql reports SQLite as validating SQL, because
HarlequinSqliteConnection.validate_sql is defined and its body is raise NotImplementedError. That is not a bug — the app catches NotImplementedError at the call
site and carries on — but it does mean a reflective probe cannot answer the question, and
that a class variable is the pattern that can.

Cost, measured, with those four adapters installed:

Two things follow. Adapter versions are free — ep.dist.version costs 5ms over reading
names, so what is installed is answerable without importing any of it. (Capability flags are
class variables, so reporting those costs the import; §3.4.) And anything that wants every
adapter's options pays 300ms and rising, which is fine for --spec, a once-per-task
lookup, and not fine for anything on the query path.

1.5 Options can be declared but not described, validated, or marked secret Copy Section

AbstractOption renders three ways — to_click(), to_widgets(), to_questionary() — and
has no fourth. There is no serialization, so --spec has nothing to call, and the
debug-info screen builds a markdown table by reaching for opt.name, opt.short_decls and
getattr(opt, "default", None) — duck-typing at a distance, which is what a real
to_dict() would replace.

Adding an abstract method is not available: AbstractOption is public API and
third-party adapters subclass it. Anything added here has to be concrete, with a base
implementation that works for a subclass that has never heard of it.

Nothing declares a value sensitive. Sampling the ecosystem, harlequin-postgres 1.3.1
declares twelve options, of which password is a plain TextOption, sslkey is a
TextOption, and passfile is a PathOption. So --spec would today teach an agent that
--password exists and say nothing about why it must not be typed on a command line, where
ps and shell history can read it.

And nothing validates an adapter option that arrives from a profile. Options typed on the
command line go through click, which rejects what it doesn't know; options read out of a
config file go straight to the adapter's constructor as keyword arguments, and the adapter
contract tells adapters to tolerate supersets of what they declare. So a misspelled key is
accepted in silence, by design, all the way down:

>>> DuckDbAdapter(conn_str=(":memory:",), reed_only=True, dbnmae="warehouse")
constructed fine | read_only = False

That is the safety-relevant version of §1.6: a human who put reed_only = true in a profile
believes they handed an agent a read-only connection, and did not. Nobody in the stack is in
a position to notice — except a step that knows both the profile and the adapter's declared
options, which is §3.5's second pass.

1.6 Config files merge per top-level key, and it is worse than losing a profile Copy Section

Ted asked me to test this rather than assert it, and the result is sharper than the first
draft claimed. _merge_config_files() is a top-level dict.update() per file, so:

# ~/.harlequin.toml
default_profile = "personal"
[profiles.personal]   # + [profiles.shared]

# ./.harlequin.toml
[profiles.project]

merged: {'default_profile': 'personal', 'profiles': {'project': {'adapter': 'sqlite'}}}

The cwd file's profiles table replaces the home file's outright — but default_profile
survives from the home file, because it is a different top-level key. So the two do not
just disagree, they contradict, and both commands then refuse to start:

harlequin.exception.HarlequinConfigError: Config files set the default_profile to
personal, but do not define a profile with that name.

A project-local config file that names one profile bricks a user whose personal file sets a
default. This predates the tomllib refactor: I ran the same probe against the
pre-refactor config.py (the tomlkit read path, from 1af8cca) and got the identical
error, so the C parser did not introduce it — dict.update() has always replaced the table
rather than merging it.

Reading is tomllib (fast, and every start-up pays it, including on the pyproject.toml
that sits in any Python project's cwd) and writing is tomlkit (slow, preserves comments,
loaded only when something writes). That split is right and M2 keeps it — but note that
config_wizard reads ConfigFile.relevant_config and writes it back through update(), so
anything that transforms values on read will be written back transformed unless the read
path used for writing stays raw. That is the whole implementation risk in env interpolation
(§3.5).

Schema errors (_raise_on_bad_schema) are good prose with no machine channel and no
positions: tomllib reports a line for a parse error inside its message, and neither
parser will tell you where key profiles.prod.limitt sits in the file.

1.7 Two expensive imports the query path must never pay Copy Section

hsql is contractually forbidden from importing sqlfmt, questionary and Textual: the
hsql does not reach the TUI import-linter contract lists them, and
tests/unit_tests/test_import_hygiene.py asserts they stay out of sys.modules after every
headless import. With fmt cut from this milestone (§7), sqlfmt stays forbidden outright
— there is now no headless code path that wants it.

Two others are merely expensive, and M2 adds the first consumer of each:

• questionary, 156ms and 329 modules. hsql must never import it, because hsql must
  never prompt (§3.7). --config init is the non-interactive path, and the wizard stays
  where it is.
• tomlkit, 49ms alone and +8ms on top of harlequin.hsql.cli. Only --config init
  writes a file, so only --config init pays it, deferred into the mode.

For scale, harlequin.hsql.cli alone imports in 107ms across 187 modules, and importing
sqlfmt on top of it would have taken that to 217ms/304 — most of a second hsql -c
invocation, for a feature §7 explains we are not shipping.

1.8 There is nothing to build `--read-only` on, and transactions are worse Copy Section

No adapter declares read-only as a capability. DuckDB and SQLite each declare their own
read-only option, which core could set by name — but harlequin-postgres declares no
such option at all, so a convention-based --read-only would silently do nothing on the
adapter most likely to be pointed at production. (Postgres can enforce it; it wants set default_transaction_read_only = on at connect, which is adapter work, not core work.)

Transactions are the same shape and further gone, which is why --single-transaction is cut
(§7). HarlequinConnection.transaction_mode returns a
HarlequinTransactionMode(label, commit, rollback) or None, and the label is whatever the
adapter chose: SQLite offers Auto/Manual on 3.12+, Postgres offers Auto/Manual, DuckDB
overrides nothing and so returns None, and a third-party adapter may name its modes anything
at all and default to whatever its driver does. Core cannot ask for "the manual one" through
that interface without first making the contract say which mode is which.

1.9 A cancelled query returns zero rows and exit code 0 Copy Section

This is the one that would have shipped as a bug. I ran a billion-row DuckDB aggregate
through harlequin.query on a worker thread and called connection.cancel() from the main
thread:

cancel() returned in 0.0ms
executed: [(has_result_set=True, error=None)]
rows: 0
elapsed: 702ms

DuckDbCursor.fetchall() catches duckdb.InterruptException and returns None; fetch()
turns None into an empty backend, correctly, because that is also what a real empty result
looks like. So a naive --timeout prints an empty result set, writes no diagnostic, and
exits 0 — reporting "your query returned nothing" for a query that was killed. That is the
silent-wrong-answer family the whole product plan exists to avoid, and no amount of care in
the adapter helps, because the adapter cannot tell an interrupt it was asked for from one it
wasn't.

Two more things I measured while establishing what a deadline can do:

• Exiting while a worker thread is inside DuckDB aborts the process. sys.exit(4) with
  a live daemon thread in to_arrow_table() produces terminate called without an active exception and exit code 134, which is not in the documented table. os._exit(4)
  after an explicit flush exits 4 with stdout intact.
• So a deadline is implementable, but only as: cancel → wait → flush → os._exit. And with
  an adapter that cannot cancel, there is no way to stop the work at all.

2. The obstacles, stated plainly Copy Chapter

1. The adapter contract can't answer four of M2's questions. What is this column's real
   type; where does a name live; can you enforce read-only; can you validate SQL. All are
   additive, with defaults (plus one more on options, for secrets), and all need an ecosystem
   rollout before they are true everywhere — which is why the release that needs none of them
   goes first.
2. Resolution is a walk, and recursion is unbounded. One round trip per path segment,
   and 403 for a single recursive listing of a wide schema. The answer M2 takes is to ship
   one level and no recursion at all, rather than to ship a knob with a cliff behind it.
3. hsql grows four modes without losing a 90ms --help. Two of them need imports the
   query path must never pay — every installed adapter at once, and tomlkit — and one of
   them (--info) has to work when the database is unreachable, which is precisely when a
   diagnostic is worth most.
4. A timeout cannot trust the adapter to report itself. §1.9. hsql has to attribute
   the cancellation it caused, because the adapter can't.
5. Secrets leak through four new mouths at once. --info, --spec, --config show and
   errors are all output surfaces M2 adds, and every one of them handles connection options.
   Redaction has to be a property of the declaration, not a rule applied four times.

Note what is not on the list. There is no new output subsystem: catalog listings are rows,
so --format, -o, -t, -A, --display-rows and --stats are already written and
already tested (§3.3). And there is no new config parser: provenance, interpolation and
schema generation are all things to do to the config we already read.

3. Target architecture Copy Chapter

harlequin/
  adapter.py      MOD  + IMPLEMENTS_READ_ONLY, IMPLEMENTS_VALIDATE_SQL
  catalog.py      MOD  + CatalogItem.type_name
  navigate.py     NEW  resolving a catalog path and listing one level below it
  options.py      MOD  + secret=..., + to_dict()
  config.py       MOD  per-profile merge, provenance, ${VAR} interpolation
  config_schema.py NEW the JSON Schema for a config file, generated from what is installed
  redact.py       NEW  redaction helpers, driven by the options that declare themselves secret
  query.py        MOD  text_columns() returns string data unchanged
  hsql/
    cli.py        MOD  + the mode options, and which of them attaches adapter options
    timeout.py    NEW  a wall clock over a run, and how it exits
    modes/        NEW  catalog.py find.py info.py spec.py config.py

M1's rule stands and now covers four modes as well as the run form:

stdout is written only by harlequin.layout and harlequin.export. stderr is written
only by harlequin.hsql.diagnostics.

M2 adds a second, which is the whole milestone in a sentence and is also testable:

No mode pays for what it did not ask for. --help imports no adapter; --spec and
--config import no database driver; --info opens no connection; --catalog fetches
one level; and the query path imports neither sqlfmt, tomlkit nor questionary.

3.1 Modes are options, not subcommands Copy Section

hsql [OPTIONS] [CONN_STR]...

  --catalog          List catalog objects one level below --path.
  --path TEXT        Where in the catalog to look. Dotted segments; default is the top.
  --find TERM        Search the catalog for TERM, where the adapter can.
  --info             Report versions, config files, adapters and capabilities. JSON.
  --spec             Dump the installed CLI surface, including adapter options. JSON.
  --config MODE      show | list-profiles | validate | schema | init

Why options rather than subcommands (Ted's call). It matches harlequin --config, so
one shape covers both commands. It keeps hsql's parse a plain click command — M1's
two-phase build stays two phases rather than growing a verb phase in front. And it removes
an ambiguity that subcommands would have introduced for free: hsql takes CONN_STR
positionally, so hsql catalog and a DuckDB file named catalog would have needed a
disambiguation rule, docs for the rule, and a test for the rule. No verb, no rule.

Modes are mutually exclusive; two of them is a usage error, exit 2, naming both. With no
mode, hsql is what M1 shipped.

--path, not -c. Ted floated reusing -c for the catalog path and offered --path
or --relation as alternatives; --path is the one to take. -c is --command and means
SQL in this CLI, in psql, and in duckdb, and the whole premise of principle 4 is that a
flag should not change meaning with context. An agent that has learned -c types
hsql --catalog -c "select 1" sooner or later, and the right outcome is an error, not a
lookup for a relation named select 1. --path also leaves -c free to mean what it
means, if a later milestone wants --catalog and a query in one invocation.

Nor a value on --catalog itself (Ted's question, taken after PR 9). --catalog PATH
reads better and costs two tokens fewer, but click takes an optional value greedily -- the
next token, whether or not an = is written -- and CONN_STR is a variadic positional. So
hsql --catalog mydb.db would read the file as a catalog path against an in-memory
database, which is the hsql catalog-versus-a-file-named-catalog ambiguity in a different
spelling, and it would need the rule §3.1 exists to avoid. With --path, CONN_STR stays
free to sit anywhere on the line and hsql --catalog mydb.db lists what is in mydb.db.
--find composes either way: --path is the scope for it too (§3.8).

Which mode needs what, because this is the table that keeps §3's second rule true: