`_ |
[d-arangodb]: https://github.com/xo/dbimp
[d-athena]: https://github.com/xo/dbimp
[d-avatica]: https://github.com/xo/dbimp
[d-bigquery]: https://github.com/xo/dbimp
[d-cassandra]: https://github.com/xo/cassandra
[d-chai]: https://github.com/chaisql/chai
[d-clickhouse]: https://github.com/xo/dbimp
[d-cockroachdb]: https://github.com/jackc/pgx
[d-cosmos]: https://github.com/xo/dbimp
[d-couchbase]: https://github.com/xo/dbimp
[d-cratedb]: https://github.com/jackc/pgx
[d-csvq]: https://github.com/mithrandie/csvq-driver
[d-databend]: https://github.com/xo/dbimp
[d-databricks]: https://github.com/xo/dbimp
[d-drill]: https://github.com/xo/dbimp
[d-druid]: https://github.com/xo/dbimp
[d-duckdb]: https://github.com/duckdb/duckdb-go
[d-dynamodb]: https://github.com/xo/dbimp
[d-elasticsearch]: https://github.com/xo/dbimp
[d-exasol]: https://github.com/xo/dbimp
[d-firebird]: https://github.com/nakagami/firebirdsql
[d-flightsql]: https://github.com/apache/arrow-go/tree/main/arrow/flight/flightsql/driver
[d-gizmosql]: https://github.com/apache/arrow-go/tree/main/arrow/flight/flightsql/driver
[d-h2]: https://github.com/jmrobles/h2go
[d-hive]: https://github.com/beltran/gohive
[d-impala]: https://github.com/sclgo/impala-go
[d-influxdb]: https://github.com/xo/dbimp
[d-influxql]: https://github.com/xo/dbimp
[d-libpq]: https://github.com/lib/pq
[d-libsql]: https://github.com/xo/dbimp
[d-maxcompute]: https://github.com/aliyun/aliyun-odps-go-sdk
[d-memsql]: https://github.com/go-sql-driver/mysql
[d-moderncsqlite]: https://gitlab.com/cznic/sqlite
[d-mysql]: https://github.com/go-sql-driver/mysql
[d-neo4j]: https://github.com/xo/dbimp
[d-odbc]: https://github.com/xo/odbc
[d-opensearch]: https://github.com/xo/dbimp
[d-oracle]: https://github.com/sijms/go-ora
[d-ots]: https://github.com/aliyun/aliyun-tablestore-go-sql-driver
[d-pgx]: https://github.com/jackc/pgx
[d-pinot]: https://github.com/xo/dbimp
[d-postgres]: https://github.com/jackc/pgx
[d-presto]: https://github.com/xo/dbimp
[d-questdb]: https://github.com/jackc/pgx
[d-redshift]: https://github.com/jackc/pgx
[d-rqlite]: https://github.com/xo/dbimp
[d-saphana]: https://github.com/SAP/go-hdb
[d-snowflake]: https://github.com/xo/dbimp
[d-solr]: https://github.com/xo/dbimp
[d-spanner]: https://github.com/xo/dbimp
[d-sqlite3]: https://github.com/mattn/go-sqlite3
[d-sqlserver]: https://github.com/microsoft/go-mssqldb
[d-surrealdb]: https://github.com/xo/dbimp
[d-tidb]: https://github.com/go-sql-driver/mysql
[d-trino]: https://github.com/xo/dbimp
[d-vertica]: https://github.com/vertica/vertica-sql-go
[d-vitess]: https://github.com/go-sql-driver/mysql
[d-voltdb]: https://github.com/VoltDB/voltdb-client-go
[d-ydb]: https://github.com/ydb-platform/ydb-go-sdk
[f-cgo]: #f-cgo "Requires CGO"
[f-embedded]: #f-embedded "Embedded"
[f-hosted]: #f-hosted "Hosted service"
† Requires CGO
§ Embedded, with no server to run
¶ Hosted service, with no server you can run
Any of the protocol schemes/aliases above can be used in conjunction when
connecting to a database via the command-line or with the [`\connect` and
`\copy` commands][commands]:
```sh
# connect to a vitess database:
$ usql vt://user:pass@host:3306/mydatabase
$ usql
(not connected)=> \c vitess://user:pass@host:3306/mydatabase
$ usql
(not connected)=> \copy csvq://. pg://localhost/ 'select * ....' 'myTable'
```
See [the section below on connecting to databases][connecting] for further
details building DSNs/URLs for use with `usql`.
## Using
After [installing][], `usql` can be used similarly to the following:
```sh
# connect to a postgres database
$ usql postgres://booktest@localhost/booktest
# connect to an oracle database
$ usql oracle://user:pass@host/oracle.sid
# connect to a postgres database and run the commands contained in script.sql
$ usql pg://localhost/ -f script.sql
```
### Command-line Options
Supported command-line options:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Connecting to Databases
`usql` opens a database connection by [parsing a URL][dburl] and passing the
resulting connection string to [a database driver][databases]. Database
connection strings (aka "data source name" or DSNs) have the same parsing rules
as URLs, and can be passed to `usql` via command-line, or to the [`\connect`,
`\c`, and `\copy` commands][commands].
Database connections can be defined with [the `\cset` command][connection-vars]
or in [the `config.yaml` configuration file][config].
#### Database Connection Strings
Database connection strings look like the following:
```txt
driver+transport://user:pass@host/dbname?opt1=a&opt2=b
driver:/path/to/file
/path/to/file
name
```
Where the above are:
| Component | Description |
| ------------------------------- | ------------------------------------------------------------------------------------ |
| `driver` | driver scheme name or scheme alias |
| `transport` | `tcp`, `udp`, `unix` or driver name (for ODBC and ADODB) |
| `user` | username |
| `pass` | password |
| `host` | hostname |
| `dbname` [±][f-path] | database name, instance, or service name/ID |
| `?opt1=a&...` | additional database driver options (see respective SQL driver for available options) |
| `/path/to/file` | a path on disk |
| `name` | a connection name set by [`\cset`][connection-vars] or in [`config.yaml`][config] |
[f-path]: #f-path "URL Paths for Databases"
± Some databases, such as Microsoft SQL Server, or Oracle
Database support a path component (ie, /dbname) in the form
of /instance/dbname, where /instance is the
optional service identifier (aka "SID") or database instance
#### Driver Aliases
`usql` supports the same driver names and aliases as [the `dburl`
package][dburl]. Databases have at least one or more aliases. See [`dburl`'s
scheme documentation][dburl-schemes] for a list of all supported aliases.
##### Short Aliases
All database drivers have a two character short form that is usually the first
two letters of the database driver. For example, `pg` for `postgres`, `my` for
`mysql`, `ms` for `sqlserver`, `or` for `oracle`, or `sq` for `sqlite3`.
#### Passing Driver Options
Driver options are specified as standard URL query options in the form of
`?opt1=a&opt2=b`. Refer to the [relevant database driver's documentation][databases]
for available options.
#### Paths on Disk
If a URL does not have a `driver:` scheme, `usql` will check if it is a path on
disk. If the path exists, `usql` will attempt to use an appropriate database
driver to open the path.
When the path is a Unix Domain Socket, `usql` will attempt to open it with the
MySQL driver. When the path is a directory, `usql` will attempt to open it
using the PostgreSQL driver. And, lastly, when the path is a regular file,
`usql` will attempt to open the file using the SQLite3 or DuckDB drivers.
#### Driver Defaults
As with URLs, most components in the URL are optional and many components can
be left out. `usql` will attempt connecting using defaults where possible:
```sh
# connect to postgres using the local $USER and the unix domain socket in /var/run/postgresql
$ usql pg://
```
See the relevant documentation [on database drivers][databases] for more
information.
### Connection Examples
The following are example connection strings and additional ways to connect to
databases using `usql`:
```
/* Detailed source-code truncated for AI context efficiency. */
```
See the [section on connection variables][connection-vars] for information on
defining connection names.
### Executing Queries and Commands
The interactive interpreter reads queries and [backslash meta (`\`) commands][commands],
sending the query to the connected database:
```sh
$ usql sqlite://example.sqlite3
Connected with driver sqlite3 (SQLite3 3.17.0)
Type "help" for help.
sq:example.sqlite3=> create table test (test_id int, name string);
CREATE TABLE
sq:example.sqlite3=> insert into test (test_id, name) values (1, 'hello');
INSERT 1
sq:example.sqlite3=> select * from test;
test_id | name
+---------+-------+
1 | hello
(1 rows)
sq:example.sqlite3=> select * from test
sq:example.sqlite3-> \p
select * from test
sq:example.sqlite3-> \g
test_id | name
+---------+-------+
1 | hello
(1 rows)
sq:example.sqlite3=> \c postgres://booktest@localhost
error: pq: 28P01: password authentication failed for user "booktest"
Enter password:
Connected with driver postgres (PostgreSQL 9.6.6)
pg:booktest@localhost=> select * from authors;
author_id | name
+-----------+----------------+
1 | Unknown Master
2 | blah
3 | foobar
(3 rows)
pg:booktest@localhost=>
```
A command takes one or more parameters. A parameter can be quoted with either
`'` or `"`, and [can also be backticked][backticks].
### Backslash Commands
`usql` supports interleaved backslash (`\`) meta commands to modify or alter
the way that `usql` interprets queries, formats its output, and changes the
resulting interactive flow.
```sh
(not connected)=> \c postgres://user:pass@localhost
pg:user@localhost=> select * from my_table \G
```
Available backslash meta commands can be displayed with `\?`:
```
/* Detailed source-code truncated for AI context efficiency. */
```
Parameters passed to commands [can be backticked][backticks].
## Features and Compatibility
An overview of `usql`'s features, functionality, and compatibility with `psql`:
- [Configuration][config]
- [Variables][variables]
- [Backticks][backticks]
- [Copying Between Databases][copying]
- [Syntax Highlighting][highlighting]
- [Time Formatting][timefmt]
- [Context Completion][completion]
- [Host Connection Information](#host-connection-information)
- [Passwords][usqlpass]
- [Runtime Configuration (RC) File][usqlrc]
The `usql` project's goal is to support as much of `psql`'s core features and
functionality, and aims to be as compatible as possible - [contributions are
always appreciated][contributing]!
#### Configuration
During its initialization phase, `usql` reads a standard [YAML configuration][yaml]
file [`config.yaml`](_samples/config.yaml). On Windows this is `%AppData%/usql/config.yaml`,
on macOS this is `$HOME/Library/Application Support/usql/config.yaml`, and on
Linux and other Unix systems this is normally `$HOME/.config/usql/config.yaml`.
##### `connections:`
[Named connection DSNs][connecting] can be defined under `connections:` as a string
or as a map:
```yaml
connections:
my_couchbase_conn: couchbase://Administrator:P4ssw0rd@localhost
my_clickhouse_conn: clickhouse://clickhouse:P4ssw0rd@localhost
my_oracle_conn:
protocol: oracle
username: system
password: P4ssw0rd
hostname: localhost
port: 1521
database: free
```
Defined `connections:` can be used on the command-line with `\connect`, `\c`,
`\copy`, and [other commands][commands]:
```sh
$ usql my_oracle_conn
Connected with driver oracle (Oracle Database 23.0.0.0.0)
Type "help" for help.
or:system@localhost/free=>
```
##### `init:`
An initialization script can be defined as `init:` as a string:
```yaml
init: |
\echo welcome to the jungle `date`
\uset syntax_hl_style paraiso-dark
\uset prompt1 '\033[32m%S%M%/%R%#\033[0m '
```
The `init:` script is commonly used to set [environment variables][variables]
or other configuration, and can be disabled on the command-line using the
`--no-init` / `-X` flag. The script will be executed prior to any `-c` /
`--command` / `-f` / `--file` flag and before starting the interactive
interpreter.
##### Other Options
Please see [`_samples/config.yaml`](_samples/config.yaml) for an overview of
available configuration options.
#### Variables
`usql` has [script][runtime-vars] variables, [connection][connection-vars]
variables, [display formatting][print-vars] variables, and [settings][settings]
of its own. They are set with `\set`, `\cset`, `\pset` and `\uset`
respectively, and `\set` is only for scripts: a setting such as `pager` is not
a variable, and `\set pager less` is an error that says to use `\uset`.
##### Runtime Variables
Script variables are managed with the `\set` and `\unset` [commands][commands]:
```sh
(not connected)=> \unset FOO
(not connected)=> \set FOO bar
```
Script variables can be displayed with `\set`:
```sh
(not connected)=> \set
FOO = 'bar'
```
###### Variable Interpolation
When a script variable `NAME` has been `\set`, then `:NAME`, `:'NAME'`, and
`:"NAME"` will be interpolated into the query buffer:
```sh
pg:booktest@localhost=> \set FOO bar
pg:booktest@localhost=> select * from authors where name = :'FOO';
author_id | name
+-----------+------+
7 | bar
(1 rows)
```
Where a runtime variable is used as `:'NAME'` or `:"NAME"` the interpolated
value will be quoted using `'` or `"` respectively:
```sh
pg:booktest@localhost=> \set TBLNAME authors
pg:booktest@localhost=> \set COLNAME name
pg:booktest@localhost=> \set FOO bar
pg:booktest@localhost=> select * from :TBLNAME where :"COLNAME" = :'FOO'
```
The query buffer and interpolated values can be displayed with `\p` and
`\print`, or the raw query buffer can be displayed with `\raw`:
```sh
pg:booktest@localhost-> \p
select * from authors where "name" = 'bar'
pg:booktest@localhost-> \raw
select * from :TBLNAME where :"COLNAME" = :'FOO'
```
> **Note**
>
> Variables contained within other strings will not be interpolated:
```sh
pg:booktest@localhost=> select ':FOO';
?column?
+----------+
:FOO
(1 rows)
pg:booktest@localhost=> \p
select ':FOO';
```
##### Connection Variables
Connection variables work similarly to runtime variables, and are managed with
`\cset`. Connection variables can be used with the `\c`, `\connect`, `\copy`,
or [other commands][commands]:
```sh
(not connected)=> \cset my_conn postgres://user:pass@localhost
(not connected)=> \c my_conn
Connected with driver postgres (PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2))
pg:postgres@localhost=>
```
Connection variables are not interpolated into queries. See the [configuration
section for information on defining persistent connection variables][config].
Connection variables can be displayed with `\cset`:
```sh
(not connected)=> \cset
my_conn = 'postgres://user:pass@localhost'
```
##### Display Formatting (Print) Variables
Display formatting variables can be set using `\pset` and [other
commands][commands]:
```sh
(not connected)=> \pset time Kitchen
Time display is "Kitchen" ("3:04PM").
(not connected)=> \a
Output format is unaligned.
```
Display formatting variables can be displayed with `\pset`:
```sh
(not connected)=> \pset
time Kitchen
```
##### Settings
The settings of `usql`, such as the pager, the editor, the prompts and
`on_error_stop`, are managed with the `\uset` [command][commands], which works as
`\pset` does. Without arguments it shows every setting, with a name it shows
that setting, and with a name and a value it sets it. The value is read as the
other meta commands read an argument: it can be quoted, `:name` is interpolated
and a backtick runs a command.
```sh
(not connected)=> \uset pager "less -R"
pager is "less -R"
(not connected)=> \uset on_error_stop on
on_error_stop is on
(not connected)=> \uset
echo_hidden off
editor ""
on_error_stop on
pager "less -R"
prompt1 "%S%N%m%/%R%# "
...
```
A setting can also be given on the command line with `--uset NAME=VALUE`, and
`-q` is `--uset quiet=on`. The environment variables `USQL_PAGER`, `PAGER`,
`USQL_EDITOR`, `EDITOR`, `VISUAL`, `USQL_SSLMODE`, `USQL_SHOW_HOST_INFORMATION`
and `USQL_TERM_GRAPHICS` give the first value of their settings.
Use the `\? variables` [command][commands] to display help information and to
list the variables and the settings recognized by `usql`:
```sh
(not connected)=> \? variables
```
#### Backticks
[Backslash (`\`) meta commands][commands] support backticks on parameters:
```sh
(not connected)=> \echo Welcome `echo $USER` -- 'currently:' "(" `date` ")"
Welcome ken -- currently: ( Wed Jun 13 12:10:27 WIB 2018 )
(not connected)=>
```
Backticked parameters will be passed to the user's `SHELL`, exactly as written,
and can be combined with `\set`:
```sh
pg:booktest@localhost=> \set MYVAR `date`
pg:booktest@localhost=> \set
MYVAR = 'Wed Jun 13 12:17:11 WIB 2018'
pg:booktest@localhost=> \echo :MYVAR
Wed Jun 13 12:17:11 WIB 2018
pg:booktest@localhost=>
```
#### Copying Between Databases
`usql` provides a `\copy` command that reads data from a source database DSN
and writes to a destination database DSN:
```sh
(not connected)=> \cset PGDSN postgres://user:pass@localhost
(not connected)=> \cset MYDSN mysql://user:pass@localhost
(not connected)=> \copy PGDSN MYDSN 'select book_id, author_id from books' 'books(id, author_id)'
```
As demonstrated above, the `\copy` command does not require being connected to
a database, and will not modify or change the current open database connection
or state.
Any valid URL or DSN name maybe used for the source and destination database:
```sh
(not connected)=> \cset MYDSN mysql://user:pass@localhost
(not connected)=> \copy postgres://user:pass@localhost MYDSN 'select book_id, author_id from books' 'books(id, author_id)'
```
> **Note**
>
> `usql`'s `\copy` is distinct from and does not function like
> `psql`'s `\copy`.
##### Copy Parameters
The `\copy` command has two parameter forms:
```txt
\copy SRC DST QUERY TABLE
\copy SRC DST QUERY TABLE(COL1, COL2, ..., COLN)
```
Where:
- `SRC` - is the [source database URL][connecting] to connect to, and where the
`QUERY` will be executed
- `DST` - is the [destination database URL][connecting] to connect to, and where
the destination `TABLE` resides
- `QUERY` - is the query to execute on the `SRC` connection, the results of which
will be copied to `TABLE`
- `TABLE` - is the destination table name, followed by an optional SQL-like column
list of the form `(COL1, COL2, ..., COLN)`
- `(COL1, COL2, ..., COLN)` - a list of the destination column names, 1-to-N
The usual rules for [variables, interpolation, and quoting][variables] apply to
`\copy`'s parameters.
###### Quoting
`QUERY` and `TABLE` **_must_** be quoted when containing spaces:
```sh
$ usql
(not connected)=> echo :SOURCE_DSN :DESTINATION_DSN
pg://postgres:P4ssw0rd@localhost/ mysql://localhost
(not connected)=> \copy :SOURCE_DSN :DESTINATION_DSN 'select * from mySourceTable' 'myDestination(colA, colB)'
COPY 2
```
###### Column Counts
The `QUERY` **_must_** return the same number of columns as defined by
the `TABLE` expression:
```sh
$ usql
(not connected)=> \copy csvq:. sq:test.db 'select * from authors' authors
error: failed to prepare insert query: 2 values for 1 columns
(not connected)=> \copy csvq:. sq:test.db 'select name from authors' authors(name)
COPY 2
```
###### Datatype Compatibility and Casting
The `\copy` command does not attempt to perform any kind of datatype
conversion.
If a `QUERY` returns columns with different datatypes than expected by the
`TABLE`'s column, the `QUERY` can use the source database's conversion/casting
functionality to cast columns to a datatype that will work for `TABLE`'s
columns:
```sh
$ usql
(not connected)=> \copy postgres://user:pass@localhost mysql://user:pass@localhost 'SELECT uuid_column::TEXT FROM myPgTable' myMyTable
COPY 1
```
###### Importing Data from CSV
The `\copy` command is capable of importing data from CSV's (or any other
database!) using the `csvq` driver:
```
/* Detailed source-code truncated for AI context efficiency. */
```
> **Note**
>
> When importing large datasets (> 1GiB) from one database to another, it is
> better to use a database's native clients and tools.
###### Reusing Connections with Copy
The `\copy` command (and all `usql` commands) [works with variables][variables].
When scripting, or when needing to perform multiple `\copy` operations from/to
multiple sources/destinations, the best practice is to `\set` connection
variables either in a script or in [the `$HOME/.usqlrc` RC script][usqlrc].
Similarly, passwords can be stored for easy reuse (and kept out of scripts) by
storing in [the `$HOME/.usqlpass` password file][usqlpass].
For example:
```sh
$ cat $HOME/.usqlpass
postgres:*:*:*:postgres:P4ssw0rd
oracle:*:*:*:system:P4ssw0rd
$ usql
Type "help" for help.
(not connected)=> \set pglocal postgres://postgres@localhost:49153?sslmode=disable
(not connected)=> \set orlocal oracle://system@localhost:1521/orasid
(not connected)=> \copy :pglocal :orlocal 'select staff_id, first_name from staff' 'staff(staff_id, first_name)'
COPY 18
```
#### Syntax Highlighting
Interactive input is highlighted as you type it. `usql` parses the input with
the grammar of [transit][transit], and the SQL of each statement with the grammar
of the dialect of the connection. The colors come from a [style][styles] of
transit, which has the styles of Chroma. The [settings][settings] that control
highlighting are:
| Setting | Default | Values | Description |
| ----------------------- | ------------------------------- | ------------------------------------------------ | -------------------------------------------------------- |
| `syntax_hl` | `true` | `true` or `false` | turns highlighting on and off |
| `syntax_hl_format` | _dependent on terminal support_ | `noop`, `terminal`, `terminal256`, `terminal16m` | the number of colors: none, 16, 256 or 24 bits |
| `syntax_hl_override_bg` | `true` | `true` or `false` | keeps the background of the terminal, and not the style's |
| `syntax_hl_style` | `monokai` | style name | the [name of a style][styles] |
They are set with `\uset`. A terminal with no color, or a `NO_COLOR` variable,
turns highlighting off by default:
```sh
$ usql
(not connected)=> \uset syntax_hl_style dracula
syntax_hl_style is dracula
(not connected)=> \uset syntax_hl off
syntax_hl is false
```
#### Context Completion
When using the interactive shell, context completion is available in `usql` by
hitting the `` key. For example, hitting `` can complete some parts
of `SELECT` queries on a PostgreSQL databases:
```sh
$ usql
Connected with driver postgres (PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1))
Type "help" for help.
pg:postgres@=> select * f
fetch from full outer join
```
Or, for example completing [backslash commands][commands] while connected to a
database:
```sh
$ usql my://
Connected with driver mysql (10.8.3-MariaDB-1:10.8.3+maria~jammy)
Type "help" for help.
my:root@=> \g
\g \gexec \gset \gx
```
Not all commands, contexts, or databases support completion. To help improve
it, see [the section below on contributing][contributing].
Command completion can be canceled with ``.
#### Time Formatting
Some databases support time/date columns that [support formatting][go-time]. By
default, `usql` formats time/date columns as [RFC3339Nano][go-time], and can be
set using `\pset time FORMAT`:
```sh
$ usql pg://
Connected with driver postgres (PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1))
Type "help" for help.
pg:postgres@=> \pset
time RFC3339Nano
pg:postgres@=> select now();
now
-----------------------------
2021-05-01T22:21:44.710385Z
(1 row)
pg:postgres@=> \pset time Kitchen
Time display is "Kitchen" ("3:04PM").
pg:postgres@=> select now();
now
---------
10:22PM
(1 row)
pg:postgres@=>
```
`usql`'s time format supports any [Go supported time format][go-time], or can
be any standard Go const name, such as `Kitchen` above. See below for an
overview of the [available time constants](#time-constants).
##### Time Constants
The following are the time constant names available in `usql`, corresponding
time format value, and example display output:
| Constant | Format | Display [↓][f-ts] |
| ----------- | ------------------------------------: | ----------------------------------: |
| ANSIC | `Mon Jan _2 15:04:05 2006` | `Wed Aug 3 20:12:48 2022` |
| UnixDate | `Mon Jan _2 15:04:05 MST 2006` | `Wed Aug 3 20:12:48 UTC 2022` |
| RubyDate | `Mon Jan 02 15:04:05 -0700 2006` | `Wed Aug 03 20:12:48 +0000 2022` |
| RFC822 | `02 Jan 06 15:04 MST` | `03 Aug 22 20:12 UTC` |
| RFC822Z | `02 Jan 06 15:04 -0700` | `03 Aug 22 20:12 +0000` |
| RFC850 | `Monday, 02-Jan-06 15:04:05 MST` | `Wednesday, 03-Aug-22 20:12:48 UTC` |
| RFC1123 | `Mon, 02 Jan 2006 15:04:05 MST` | `Wed, 03 Aug 2022 20:12:48 UTC` |
| RFC1123Z | `Mon, 02 Jan 2006 15:04:05 -0700` | `Wed, 03 Aug 2022 20:12:48 +0000` |
| RFC3339 | `2006-01-02T15:04:05Z07:00` | `2022-08-03T20:12:48Z` |
| RFC3339Nano | `2006-01-02T15:04:05.999999999Z07:00` | `2022-08-03T20:12:48.693257Z` |
| Kitchen | `3:04PM` | `8:12PM` |
| Stamp | `Jan _2 15:04:05` | `Aug 3 20:12:48` |
| StampMilli | `Jan _2 15:04:05.000` | `Aug 3 20:12:48.693` |
| StampMicro | `Jan _2 15:04:05.000000` | `Aug 3 20:12:48.693257` |
| StampNano | `Jan _2 15:04:05.000000000` | `Aug 3 20:12:48.693257000` |
[f-ts]: #f-ts "Timestamp Value"
↓ Generated using timestamp 2022-08-03T20:12:48.693257Z
#### Host Connection Information
By default, `usql` displays connection information when it connects to a
database. Some databases and connections do not work with this. Set the
environment variable `USQL_SHOW_HOST_INFORMATION` to `false` to turn it off:
```sh
$ export USQL_SHOW_HOST_INFORMATION=false
$ usql pg://booktest@localhost
Type "help" for help.
pg:booktest@=>
```
`show_host_information` is a [setting][settings] of `usql`, and is set with
`\uset`. Additionally, it can be passed via the command-line using `--uset`:
```sh
$ usql --uset show_host_information=false pg://
Type "help" for help.
pg:booktest@=> \uset show_host_information true
pg:booktest@=> \connect pg://
Connected with driver postgres (PostgreSQL 9.6.9)
pg:booktest@=>
```
#### Terminal Graphics
`usql` supports terminal graphics for [Kitty][kitty-graphics], [iTerm][iterm-graphics],
and [Sixel][sixel-graphics] enabled terminals using the [`github.com/kenshaw/rasterm` package][rasterm].
Terminal graphics are only available when using the interactive shell.
##### Detection and Support
`usql` will attempt to detect when terminal graphics support is available using
the `USQL_TERM_GRAPHICS`, `TERM_GRAPHICS` and other environment variables
unique to various terminals.
When support is available, the logo will be displayed at the start of an
interactive session:
##### Charts and Graphs
The [`\chart` command][chart-command] can be used to display a chart
directly in the terminal:
See [the section on the `\chart` meta command][chart-command] for details.
Charts are not part of a default build. The renderer runs Apache ECharts in a
JavaScript engine and rasterizes the result with an SVG renderer, which together
add about 12 MiB to the binary, so it is behind the `charts` build tag. Build
with `-tags charts`, or with `-tags all`, to include it. A build without it
reports that it has no chart renderer.
##### Enabling/Disabling Terminal Graphics
Terminal graphics can be forced enabled or disabled by setting the
`USQL_TERM_GRAPHICS` or the `TERM_GRAPHICS` environment variable:
```sh
# disable
$ USQL_TERM_GRAPHICS=none usql
# force iterm graphics
$ TERM_GRAPHICS=iterm usql
```
| Variable | Default | Values | Description |
| --------------- | ------- | ------------------------------------- | ------------------------------ |
| `TERM_GRAPHICS` | `` | ``, `kitty`, `iterm`, `sixel`, `none` | enables/disables term graphics |
##### Terminals with Graphics Support
The following terminals have been tested with `usql`:
- [WezTerm][wezterm] is a cross-platform terminal for Windows, macOS, Linux, and
many other platforms that supports [iTerm][iterm-graphics] graphics
- [iTerm2][iterm2] is a macOS terminal that supports [iTerm][iterm-graphics]
graphics
- [kitty][kitty] is a terminal for Linux, macOS, and various BSDs that supports
[Kitty][kitty-graphics] graphics
- [foot][foot] is a Wayland terminal for Linux (and other Wayland hosts) that
supports [Sixel][sixel-graphics] graphics
Additional terminals that support [Sixel][sixel-graphics] graphics are
catalogued on the [Are We Sixel Yet?][arewesixelyet] website.
#### Passwords
`usql` supports reading passwords for databases from a `.usqlpass` file
contained in the user's `HOME` directory at startup:
```sh
$ cat $HOME/.usqlpass
# format is:
# protocol:host:port:dbname:user:pass
postgres:*:*:*:booktest:booktest
$ usql pg://
Connected with driver postgres (PostgreSQL 9.6.9)
Type "help" for help.
pg:booktest@=>
```
See [`_samples/usqlpass`](_samples/usqlpass) for a sample.
An entry applies to a URL when each of its fields is `*` or equal to the
URL's. An entry never replaces a user that the URL names, and an entry for one
database applies only to that database. The protocol matches every scheme of
the same database, so a `postgres:` entry also covers `pg://`, `pgx://` and
`pq://`. A database that only speaks another's protocol is a database of its
own, so CockroachDB, CrateDB and Redshift each need their own entry, and so do
TiDB, SingleStore and Vitess. A `mysql:` entry does not cover `tidb://`.
While the `.usqlpass` functionality will not be removed, it is recommended to
[define named connections][connection-vars] preferably via [the `config.yaml`
file][config].
> **Note**
>
> The `.usqlpass` file must not be readable by other users. Set its permissions
> to `0600`:
```sh
chmod 0600 ~/.usqlpass
```
#### Runtime Configuration (RC) File
`usql` supports executing a `.usqlrc` runtime configuration (RC) file contained
in the user's `HOME` directory:
```sh
$ cat $HOME/.usqlrc
\echo WELCOME TO THE JUNGLE `date`
\uset syntax_hl_style paraiso-dark
-- set color prompt (default is prompt is "%S%m%/%R%#" )
\uset prompt1 "\033[32m%S%m%/%R%#\033[0m"
$ usql
WELCOME TO THE JUNGLE Thu Jun 14 02:36:53 WIB 2018
Type "help" for help.
(not connected)=> \uset syntax_hl_style
syntax_hl_style is "paraiso-dark"
(not connected)=>
```
See [`_samples/usqlrc`](_samples/usqlrc) for a sample.
The `.usqlrc` file is read at startup in the same way as a file passed on the
command-line with `-f` / `--file`. It is commonly used to set startup
environment variables and settings.
RC-file execution can be temporarily disabled at startup by passing `-X` or
`--no-init` on the command-line:
```sh
$ usql --no-init pg://
```
While the `.usqlrc` functionality will not be removed, it is recommended to set
an `init` script in [the `config.yaml` file][config].
## Additional Notes
The following are additional notes and miscellania related to `usql`:
### Release Builds
[Release builds][releases] are built with the `most` build tag and with
additional [SQLite3 build tags (see: `build.sh`)](build.sh).
### macOS
The recommended installation method on macOS is [via `brew`][via Homebrew] due
to the way library dependencies for the `sqlite3` driver are done on macOS. If
the following (or similar) error is encountered when attempting to run `usql`:
```sh
$ usql
dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicuuc.68.dylib
Referenced from: /Users/user/.local/bin/usql
Reason: image not found
Abort trap: 6
```
Then missing library dependency can be fixed by installing
[`icu4c`](http://site.icu-project.org) using `brew`:
```sh
$ brew install icu4c
Running `brew update --auto-update`...
==> Downloading ...
...
$ usql
(not connected)=>
```
### Couchbase FAQ
#### Why does a transaction fail to commit with error 17007?
The server could not meet the durability that the transaction asked for. By
default a transaction asks for `majority`, which a single-node server cannot
meet. `usql` does not change the durability unless you ask it to. Set `durability_level=none` in the URL for a single-node server,
such as a local development server:
```sh
$ usql "couchbase://user:pass@localhost/?durability_level=none"
```
The other values are `majority`, `majorityAndPersistActive` and
`persistToMajority`.
#### Why does a transaction stay open longer than the server's default?
The server ends a transaction after 15 seconds by default, which is too short
for a person typing into one. `usql` sets `txtimeout=30m` when the URL does not
set `txtimeout`. Set it in the URL to choose another value:
```sh
$ usql "couchbase://user:pass@localhost/?txtimeout=2m"
```
## Contributing
`usql` is currently a WIP, and is aiming towards a 1.0 release soon.
Well-written PRs are always welcome -- and there is a clear backlog of issues
marked `help wanted` on the GitHub issue tracker! For [technical details on
contributing, see CONTRIBUTING.md](CONTRIBUTING.md).
[_Pick up an issue today, and submit a PR tomorrow!_][help-wanted]
[dburl]: https://github.com/xo/dburl
[dburl-schemes]: https://github.com/xo/dburl#protocol-schemes-and-aliases
[go-time]: https://pkg.go.dev/time#pkg-constants
[go-sql]: https://pkg.go.dev/database/sql
[homebrew]: https://brew.sh/
[xo-tap]: https://github.com/xo/homebrew-xo
[help-wanted]: https://github.com/xo/usql/issues?q=is:open+is:issue+label:%22help+wanted%22
[aur]: https://aur.archlinux.org/packages/usql
[aur-bin]: https://aur.archlinux.org/packages/usql-bin
[yay]: https://github.com/Jguer/yay
[arch-makepkg]: https://wiki.archlinux.org/title/makepkg
[backticks]: #backticks "Backticks"
[config]: #configuration "Configuration"
[commands]: #backslash-commands "Backslash Commands"
[completion]: #context-completion "Context Completion"
[connecting]: #connecting-to-databases "Connecting to Databases"
[contributing]: #contributing "Contributing"
[copying]: #copying-between-databases "Copying Between Databases"
[highlighting]: #syntax-highlighting "Syntax Highlighting"
[transit]: https://github.com/xo/transit "transit"
[styles]: https://github.com/xo/transit/tree/master/styles/chroma "Styles of transit"
[termgraphics]: #terminal-graphics "Terminal Graphics"
[timefmt]: #time-formatting "Time Formatting"
[usqlpass]: #passwords "Passwords"
[usqlrc]: #runtime-configuration-rc-file "Runtime Configuration File"
[variables]: #variables "Variables"
[settings]: #settings "Settings"
[runtime-vars]: #runtime-variables "Runtime Variables"
[connection-vars]: #connection-variables "Connection Variables"
[print-vars]: #display-formatting-(print)-variables "Display Formatting (print) Variables"
[kitty-graphics]: https://sw.kovidgoyal.net/kitty/graphics-protocol.html
[iterm-graphics]: https://iterm2.com/documentation-images.html
[sixel-graphics]: https://saitoha.github.io/libsixel/
[rasterm]: https://github.com/kenshaw/rasterm
[wezterm]: https://wezfurlong.org/wezterm/
[iterm2]: https://iterm2.com
[foot]: https://codeberg.org/dnkl/foot
[kitty]: https://sw.kovidgoyal.net/kitty/
[arewesixelyet]: https://www.arewesixelyet.com
[chart-command]: #chart-command "\\chart meta command"
[yaml]: https://yaml.org
---
### Overview (`docs/decisions/README.md`)
# Decisions
Every decision this project made is a file in this folder, one per decision,
named by its number and its title. This table is the index. Find the number
here, then open the file.
Each file opens with its status. "Decided" means Ken chose it. "Proposed"
means an agent or a peer session suggested it and Ken has not confirmed it.
"Open" means nobody has chosen yet. A decision that changes an earlier one
says so in its status, as "Amends D1", and the earlier one says it back, as
"Amended by D3". Read the status before the decision.
A new decision gets the next number and a file of its own. Add its row here.
`TestTheDecisionIndexIsComplete` fails when a decision has no row or a row is
wrong, and it prints the row to add. D2 holds the layout.
`dburl` and `dbmeta` keep their own decision logs with the same numbering. A
reference to one of theirs names the repository, as in "dburl's D22".
| # | Decision | Status |
| --- | --- | --- |
| [D1](D001-every-xo-repository-is-set-up-for-coding-agents-the-same-way.md) | Every xo repository is set up for coding agents the same way | Decided |
| [D2](D002-a-large-project-keeps-one-file-per-decision.md) | A large project keeps one file per decision | Decided |
| [D3](D003-no-metadata-reader-is-written-in-usql.md) | No metadata reader is written in usql | Decided |
| [D4](D004-usql-reads-all-metadata-from-dbmeta-at-once.md) | usql reads all metadata from dbmeta, at once | Decided |
| [D5](D005-a-driver-field-that-describes-the-database-moves-to-dbmeta.md) | A driver field that describes the database moves to dbmeta | Decided |
---
### BACKLOG (`docs/BACKLOG.md`)
# usql backlog
This file records planned work for usql and for the sibling repositories that
usql depends on. Each item names the files, workflows or issue numbers it
touches, so that the work can start without rediscovering the context.
This file is the source of truth for the roadmap. The issue tracker is for
defects and for work that is ready for someone outside the project to pick up.
Anything recorded in both places drifts, and the roadmap in the tracker was
also hiding the real bug reports underneath it.
## How to refer to an item
Every item has an identifier of the form `W` followed by a number. Three rules
govern it:
1. Identifiers are append only. A new item takes the next unused number.
2. An identifier is never reused. W1 is finished, and no later item becomes W1.
3. An identifier is never renumbered. Removing W6 does not turn W7 into W6.
An item that is finished or abandoned keeps its heading and gains a status in
that heading. It does not disappear. A citation that silently starts pointing
at different work is worse than one that points at nothing, because nothing
tells the reader it moved.
A heading with no status is open. The statuses in use are `Done`, `Dropped`
and `Superseded by Wn`.
These rules exist because the items are cited. They are cited in this file, in
commit messages, and in conversation. Before these rules the numbers were
positional, so removing an item renumbered every item after it and quietly
invalidated every reference to them.
Use a separate series for anything that is not a work item. The sibling
repositories `dburl` and `dbmeta` number their design decisions `D1`, `D2` and
so on, and the two series must not be confused where both appear in one commit
message.
## Where the items came from
W2 to W9 are Ken's work items, in his order. W10 to W15 were added from work
that followed. W16 is the roadmap that used to live in the GitHub issue
tracker, and W17 came from it. W18 and after were added later, and each says
where it came from.
W1, bringing CI and CD up to date, is finished.
The last sections hold the GitHub working list, drivers that could be added,
and items that have been raised but have no priority yet.
## W2. Fix the unit tests
Make the unit tests work on every system, and against the databases that can
run in GitHub workflows. PostgreSQL, SQLite3 and MySQL are the minimum.
Microsoft SQL Server and Oracle are worth adding if the runners can host them.
The suite passes as of 2026-09-21, serially, against podman. What follows is
what is left.
The container tests cannot run in parallel. `go test ./...` starts three SQL
Server containers at once, and they exhaust the memory of a normal machine.
Every run needs `-p 1`, which CI does.
Both of the other two are fixed. A failing test no longer leaks its containers:
`TestMain` is `os.Exit(run(m))` and `run` holds the deferred purge, so every
exit path reaches it.
The logging problem is fixed by removing the RamSQL driver. Its `engine/log`
called `slog.SetDefault` from `init` with a handler on `os.Stdout`, and a
default slog logger also redirects the standard `log` package, so every
`log.Print` and `slog` call vanished in any build that carried it. Writing to
stdout was the worse half, because stdout carries the query results.
A workaround in `package main` was written first and then reverted. It fixed
the binary but not usql's packages when imported elsewhere, nor any test binary
that does not link `main`. Gemini and DeepSeek were both asked whether the
driver was worth keeping and both said drop it, independently and without
hedging: mutating the process-wide logger from `init` is disqualifying for a
CLI whose stdout is the data channel, and `sqlite3://:memory:` and the pure-Go
`moderncsqlite` already cover every use for it.
`drivers/drivers_test.go` keeps its `log.SetOutput(os.Stderr)`. It is one line
and nothing stops the next driver doing the same thing.
Golden files record upstream drift rather than usql behavior. Four expected
files were regenerated because jOOQ changed the sakila column types and because
tblfmt changed where it puts blank lines. `sqlserver.listIndexes` also depended
on a table that `TestCopy` leaves behind. Consider generating these from a
pinned schema instead of the live one.
`drivers/clickhouse` skips one case. clickhouse-go truncates a parameter list
with the regular expression `\sVALUES\s.*$`, which is case sensitive and needs
whitespace after `VALUES`, so a lowercase `values(` reaches the server with its
placeholders and is rejected. Report it upstream and remove the skip when it is
fixed.
## W3. Close the gap against psql
Compare usql against the current psql command line tool, feature by feature,
and close the gaps that matter.
`\prompt` must work in a non-interactive script. Issue 53, which Ken accepted
in the thread after first closing it: psql allows `\prompt` in a
non-interactive terminal, so usql should too, and "I will make it a point to
have this fixed in the next major release" is the commitment. He also named the
obstacle, which is that the readline package makes it hard to enable on
scripts, so this is entangled with W8. Without it a `-f` script cannot ask
for a value, and `--set NAME=VALUE` is the only way to pass one in.
The psql-compatible variables in `env/vars.go` are part of this. The prompt
substitution escapes are the visible gap. `handler/handler.go` line 498 lists
the ones that are not implemented: `%M`, `%m`, `%>`, `%n`, `%/`, `%~`, `%#`,
`%p`, `%R` and `%x`. Issue 467 is the user-facing report for `%/`, which shows
the database named in the connection URL instead of the database in use after
a `\c` command or a `USE` statement.
## W4. Work through GitHub
93 issues and 24 pull requests are open, reviewed on 2026-09-23. Gemini and
DeepSeek were consulted on which are real; where they disagreed it is said so.
### W4a. The nine scanner reports: close them
Issues 543, 556, 567, 568, 574, 575, 579, 580 and 581 are all dependency
scanner output from nine different users. `govulncheck` against the full build
reports that nothing reachable is affected: 0 vulnerabilities in usql's code,
0 in imported packages, and 3 in required modules that are never called, being
GO-2026-5932, GO-2022-0646 and GO-2022-0635.
They are not defects in usql and should be closed. Both models agreed that a
shipped CLI is still operationally different from a library: enterprise
scanners read `go.mod` from the binary's build info and ignore reachability, so
a compliance gate can block the binary whatever `govulncheck` says. The reports
will therefore keep arriving, and closing them without changing anything is
only half an answer.
They disagreed on whether to keep one umbrella issue. Gemini said do not, it
attracts noise. DeepSeek said keep one for tracking. No umbrella issue is the
better call for a project this size.
### W4b. Stop the scanner reports recurring
Both models converged on the same four, in rough order of effect:
1. Run `govulncheck` in CI and publish its output, so the reachability answer
is a build artifact rather than a claim in a comment.
2. Publish a VEX document with each release marking the unreachable findings
`not_affected`. Modern scanners consume it and suppress the false alarm.
This is the only mechanism that actually stops the reports at source.
3. Add `SECURITY.md` stating that usql tracks vulnerabilities through
`govulncheck`, and that an unreachable transitive finding is ordinary
dependency maintenance rather than a security incident.
4. Add an issue template for vulnerability reports that asks for `govulncheck`
output showing a call path.
Keep taking the dependency bumps regardless. They are cheap and they clear most
scanners without any argument.
### W4c. Statement parsing
Issue 587 reports that a backtick-quoted identifier containing a single quote
flips the lexer quote state in the MySQL and MariaDB dialect. The next string
literal is unescaped a second time, and usql exits 0 after sending different
SQL than the user wrote. The same report names two silent no-ops: a statement
passed through `-f` with a trailing semicolon never runs, and any `-f` file
whose last statement lacks a trailing semicolon is discarded at end of file.
The code is `stmt/parse.go`. Pull request 591 fixes the backtick half only and
is rebased onto main locally as `pr-591`.
Issues 165, 166 and 505 are the same area: the `\\` separator, quoted string
processing, and a regression in quoted variable replacement. Fix them together
with W3 rather than piecemeal.
### W4d. Stability, which outranks everything else here
Issue 546, a busy loop at 100% CPU, and 464, a crash after executing a command.
Both models put these first and they are right: a CLI that spins or dies is
worse than one missing a feature. 546 is most likely the interactive loop
polling on EOF or an unexpectedly closed pipe.
Issue 371, high memory usage, has no heap profile attached. Ask for one and
close it if none arrives.
### W4e. `\copy`, which is the data path
Issues 254, 322, 397, 427, 458, 462 and 495. Gemini called this the top
priority after stability, on the grounds that a tool which mangles data or runs
out of memory on a large load gets dropped immediately. 397, 254 and 462 are
probably one fault, the NULL scan in `\copy`; see W14. 322 is a 500 MB file
failing, which suggests the copy reads a whole payload rather than streaming.
### W4f. Output faults, cheap and visible
Issue 448, numbers shown as `1.450817032e+`, is a formatting default and should
be a small fix. Issue 509, control characters in a text field breaking the
output, needs the cell sanitised. Issue 504, `\dt` listing SQLite system
tables, is a missing `sqlite_%` filter. All three are quick and user facing.
Issue 516, csv disabling the pager, Gemini read as intended behaviour, since
csv is meant for redirection. Honour an explicitly set `\pset pager on` and
leave the default as it is.
### W4g. Driver-specific metadata: accept patches, do not write them
Issues 375, athena has no `\d`, and 440, no foreign keys on SQL Server. Gemini
was blunt and correct: one maintainer cannot write bespoke introspection for
fifty databases, most of which are not run locally. Label these for help and
merge contributions with tests. Do not spend core time on them.
### W4h. The terminal cluster belongs to W8
Issues 93, 122, 236, 483, 490, 508, 552 and 472 are all readline behaviour:
garbled input after alt-tab on Windows, DEL deleting a whole word, typing
switching to Ctrl, vi key bindings, the pager and prompt alternating. These do
not get fixed one at a time in usql. They close with the move to `xo/rline`.
### W4i. Questions, blocked on Discussions being enabled
Issues 263, 469 and 485 are questions, and pull request 360 is titled as one.
None can move, because Discussions is switched off for the repository.
Enabling it is a settings change and is Ken's call. Until then they stay open,
because closing a question the author cannot re-ask elsewhere is worse than
leaving it.
475 was not an environment problem. gosnowflake printed a DBUS warning during
`--version`, which is a driver writing uninvited output. It no longer
reproduces: `--version` gives clean stdout and empty stderr with
`DBUS_SESSION_BUS_ADDRESS` unset, and gosnowflake is still linked. Closed as
fixed.
391 is usql running inside the Emacs shell on Windows, which is not a terminal,
so the prompt and the line editor are off. That is documented behaviour rather
than a defect, but W8 may change what is possible there, so it stays open
until the rline switch lands.
### W4j. Be ruthless with the rest
Both models said the same thing unprompted: for a project this size, close
anything over a year old that has no reproduction, no stack trace and no heap
profile, saying it can be reopened with one. Most will never be reopened. The
alternative is that the real bugs above stay buried under sixty that are not.
## W5. `\echo -n` and `\warn -n` are erased by the line editor
Issue 215, which stays open as an issue because it is a defect rather than a
plan. Tier 1.
`\echo -n` and `\warn -n` suppress the trailing newline correctly, and then the
line editor clears the line before the next prompt is drawn, so the output is
never seen:
(not connected)=> \echo -n foo
(not connected)=>
psql leaves it in place and draws the prompt after it:
postgres=# \echo -n foo
foopostgres=#
The output is being written and then overwritten, so the fix is in how the
prompt is redrawn after a command that deliberately left the cursor mid-line.
That makes it a question for W8 as much as for the `\echo` implementation,
and it should be checked against `xo/rline` before being fixed in the current
in-tree editor.
## W6. Create a dbtest package (Dropped)
Dropped on 2026-09-26. Do not start this work and do not reinstate it without
reading what follows.
The plan was to move the podman scripts in `contrib/` into a separate Go
repository that starts databases as containers, and to give that repository an
MCP tool so a caller could start a database for testing with a standard
configuration.
Most of what the package was for has been built in `dbmeta` instead. dbmeta
runs databases in containers across several versions and flavors of each
product, because it has to answer what each one supports, and that requires the
same container handling this item described. Building a second thing that
starts the same containers would leave two of them to keep working.
What this item wanted that dbmeta does not already provide is the MCP tool.
That is a small addition to dbmeta if it is still wanted, not a repository.
On 2026-09-27 the container configuration left `contrib/` as well. dbmeta's
`container` package and `dbrun` command replace `podman-run.sh`,
`podman-stop.sh`, `usql-test.sh` and every `podman-config` and `usql-config`
file. dbmeta records the products it does not model yet, with their images,
ports and setup steps, in its `docs/EVALUATION.md`. The sample `config.yaml`,
`usqlrc` and `usqlpass` files moved to `_samples/`.
## W7. Create a dbmeta package (In progress, outside this repository)
Move the database metadata out of usql into its own repository, so that the
dbtpl project can share it. The code here is in `drivers/metadata/`.
The repository exists and is being built. As of 2026-09-26 it has models for
nine products and is working on SAP HANA. It records its design decisions in
`docs/PLAN.md` and its per product notes in `docs/DIALECT.md`, and it keeps a
`docs/USQL.md` describing how usql reads metadata today and what it would take
to read it from dbmeta instead.
Nothing has been removed from usql yet, and this item stays open until it is.
The migration plan is [DBMETA.md](DBMETA.md), and the work is W21.
Two facts about usql that the work needs, measured on 2026-09-26 against
`-tags all`:
Of 51 registered names, 21 have a metadata reader and 30 have none. Of the 21,
the gaps are only ever `\l` and `\dp`. See W18.
usql declares 19 interfaces in `drivers/metadata/metadata.go`. Fourteen are
leaf readers, aggregated by `ExtendedReader`. Seven of the fourteen decide
whether a command runs at all. The other seven decide how much `\d+` prints.
## W8. Switch to xo/rline
Replace the readline implementation with `github.com/xo/rline`. usql currently
depends on `github.com/gohxs/readline`, which has had no commits since 2017.
Issue 528 asks for this change and proposes `chzyer/readline`, which is the
upstream that gohxs forked.
Several open issues point at the readline layer and can close with this work.
Issue 546 reports a busy loop at 100 percent CPU in static builds. Issue 508
reports input alternating between the pager and the prompt. Issue 552 asks for
vim mode. Issue 490 reports the delete key removing a whole word. Issue 483
reports the terminal behaving as though Control is held down. Issue 472 reports
that a tab character cannot be sent as input.
### Two replacement pull requests are closed
Both are closed rather than merged, because neither library is the
destination. 418 proposed `reeflective/readline` and 535 proposed
`chzyer/readline`, which is the upstream `gohxs` forked eight years ago.
535 was genuinely drop-in: chzyer's `AutoCompleter` is identical in shape to
the fork's. Taking it would have removed the unmaintained dependency today.
The cost is churn in a layer that is being replaced, which is the same
reasoning that froze `drivers/metadata`.
418 was mostly decoupling rather than a swap, and most of that work is already
done. `rline/rline.go:20` already declares an `IO` interface. The only
readline type that leaks outside `rline/` is `readline.AutoCompleter`, in
three places: `drivers/drivers.go:110`, `drivers/drivers.go:494` and
`drivers/completer/completer.go:113`. See W25.
usql stays on `gohxs/readline` until rline is ready. It has had no commits
since 2017.
### The readline label is the shared queue
Anything tagged `readline` on the tracker is rline's to plan for, and that
session has been told so. Thirteen issues carry it: 72, 93, 122, 215, 236, 320,
414, 472, 483, 490, 508, 528 and 552.
https://github.com/xo/usql/issues?q=is%3Aissue+is%3Aopen+label%3Areadline
Deliberately not tagged, so the boundary is on record. 196 and 282 are about
which candidates autocomplete offers, which the metadata reader produces, not
how the editor behaves. 467 is prompt content and 516 is pager configuration.
422 and 342 are about where a password comes from rather than how it is read.
546, a busy loop at 100% CPU against `usql_static`, is tagged but unconfirmed.
The reporter's diagnosis, which they say came from an AI and nobody has
verified, is that the read loop stops blocking in a static build and returns
empty results. If that is right it is rline's; if not it is ours.
### What the rline session found, 2026-09-23
Two mechanisms behind the terminal cluster, worth recording because neither is
guessable from the issue titles.
The four-issue family, 93, 122, 483 and 490, where ordinary keys start acting
as if Ctrl were held. rline's escape decoder maps `CSI I` to Tab and `CSI O` to
F3. Those two sequences are focus-in and focus-out, which is what a terminal
sends on alt-tab, the trigger two of the four reporters named. A focus
notification therefore arrives as a keystroke, a spurious Tab opens the
completion menu, and the keys afterwards do something other than insert
themselves. The mis-decode is measured. That something else leaves focus
reporting enabled, and that this explains all four reports, are separate claims
and not yet shown.
508, the pager, is a different seam and not the other end of the same one.
dradtke reproduced it in twenty lines that import `usql/rline` directly with no
database and no usql, running `less` as a child while calling `Next`, and
confirmed with strace that `less` reads every other keypress. Two readers on
one terminal, with the kernel giving each input to whichever reads first. usql
starts the pager without suspending the line editor.
The rline session then reproduced it against its own port, with a control:
without a child holding the terminal the editor read 10 of 10 keys, and with
one it read 0 of 10. The child there was `cat` rather than `less`, and the
difference is the useful part. `cat` reads greedily in a loop and takes
everything; `less` reads one key and waits, which is why the report describes a
clean alternation rather than total starvation. **The split is a property of
the other reader, not of the bug.** So a fix that tunes the sharing rather than
ending it would behave differently against every program somebody sets `PAGER`
to.
rline needs an explicit contract for handing the terminal to a child and taking
it back. Two constraints on that design. The first pager invocation after
startup behaves and later ones do not, which the two-readers account does not
explain on its own, since two readers should race the first time as readily as
the tenth; either the first invocation differs, or it races and wins. And the
fix has to end the sharing rather than tune it, for the reason above.
472, sending a literal tab, is a feature request against rline rather than a
usql regression: `key.CtrlV` exists as a code with nothing implementing
literal-next, and bracketed paste is absent from the port and was absent from
isocline before it.
## W9. Clean up the documentation
Create a friendly website and documentation site at usql.app.
`README.md` carries almost all of the documentation today, at roughly 64 KB in
a single file. Part of it is generated: `gen.go` rebuilds the driver table and
the link definitions from the build tags and from the dburl scheme registry, so
any new site must either keep that generation step or replace it.
## W10. Distribute through winget
Publish usql to the Windows Package Manager, so that Windows users install it
the way they install anything else:
winget install usql
Windows is the platform with no first class route today. The release builds a
zip, and the README points Windows users at the release page or at `go install`.
Scoop was the previous answer and has been removed.
A winget submission is a manifest in `microsoft/winget-pkgs`, holding the
release URL, the SHA256 of the archive and the installer type. The release
workflow already produces both the archive and its checksum, so the manifest
can be generated and submitted from the same job that drafts the release.
This became more worthwhile once duckdb started building on Windows, because
the Windows binary now carries the same driver set as the others.
## W11. Report the three size defects upstream
Measured on 2026-09-22 on linux/amd64, stripped with `-trimpath -ldflags "-s
-w"`. `docs/` holds no copy of the data; the numbers below are what the builds
reported.
### go-ora declares its character set tables as `int`
`github.com/sijms/go-ora/v3/converters/string_conversion_new.go` is 11 MB of
generated Go in 78,264 lines. It holds one `NewStringConverter` with 236 case
arms, and each arm builds a converter from a `dBuffer []int` and an
`eBuffer map[int]int`. That is 789,555 numbers in the slices and 403,054 pairs
in the maps, 1,595,663 numbers in total.
The element type is `int`, which is 8 bytes, so the tables occupy 12.17 MB. The
binary reports 12.11 MB for them, so the two agree. No value in any table
exceeds 65535. As `uint16` the tables would be 3.04 MB, which is 9.1 MB smaller,
and usql carries Oracle in every default build, so that is 9.1 MB off every
binary usql ships.
The `eBuffer` maps cost again at run time. Go compiles a large map literal into
two static arrays and code that inserts every entry, so building one converter
allocates a map of up to tens of thousands of entries.
Report both to sijms, together with the separate `0x80000000` overflow that is
why `drivers/oracle/oracle.go` carries `Build: !(linux && arm)`.
### dburl v0.26.0 removes the Dameng schemes
Recorded because it changes what a user sees and arrives through a dependency
bump rather than through anything here. usql removed the Dameng driver on
2026-09-23 and left dburl alone, so `dameng://` currently reports that the
driver is not available. dburl is now removing the `dm` scheme and its `dm8`
and `dameng` aliases as well, in v0.26.0, after the same question was put to
that project separately. Once usql is on it the message becomes unknown
database scheme, which is more accurate.
usql is on v0.26.0, which removes them. The plan had been to pin to v0.25.4 and
keep the release boring, and the evidence went the other way. On v0.25.4 a
`dameng://` URL reports that the `dm` driver is not available and then prints
usql's rebuild hint:
error: dm: driver not available
try:
go install -tags 'most dm' github.com/xo/usql@master
That hint is wrong. The `dm` build tag was deleted along with the driver, so
following it produces a binary that still cannot connect. v0.26.0 reports
`unknown database scheme` and prints nothing further, which is accurate.
**Put this in usql's release notes.** The error a user sees for a `dameng://`
URL changes, and it changes through a dependency bump rather than through
anything in usql's own history, so someone searching usql's changelog for it
would find nothing. All three forms are affected: `dameng://`, `dm://` and
`dm8://`. usql is clean either way, with no reference to `GenDameng` or to the
schemes outside this file.
### The duckdb promotion depends on dburl v0.25.4
Blocking, and not obvious from the diff. dburl's duckdb header matcher was
`regexp.MustCompile("^.{8}DUCK.{8}")`, and Go's `.` matches a rune rather than
a byte, so a checksum containing a newline or a valid multi-byte UTF-8 sequence
misaligns the magic. duckdb writes `DUCK` at offset 8 after an 8-byte checksum
that covers the fixed header block, so the value is constant per storage
version: detection is all-or-none per duckdb build, not per file.
Three constants have been seen. `67274d4c681e71c3` and `2eb427ecd613b61e` pass.
`06d76f27c2dab3b9` fails, and that is the one duckdb-go/v2 writes today, so
every database created by the version usql links is undetectable. Every form
fails once the file exists: `file:x.duckdb`, `x.duckdb`, `./x.duckdb` and an
absolute path, with three different error messages between them because `Parse`
reported any `SchemeType` failure as `ErrUnknownFileExtension`.
Fixed in dburl v0.25.4, which usql is on. Verified against the released module
with a file carrying the failing constant: all four forms, `file:x.duckdb`,
`x.duckdb`, `./x.duckdb` and an absolute path, open it and return the row.
dburl's own suite never caught it because `testdata/test.duckdb` has been
byte-identical since it was added in `95e9c5f`, and its checksum is one of the
passing ones. A real file was testing the matcher for its entire life and could
not vary.
duckdb is a base driver as of 2026-09-23, which makes the cost below apply to
the default build rather than only to `most`. `go build` with no tags went from
45.7 MB to 99.9 MB. Issue 446 asked for the promotion and had been declined
because duckdb was not available on Windows out of the box; that stopped being
true when the MinGW toolchain went into the release workflow. linux/arm still
excludes it through the `Build:` constraint, so the 32-bit arm build is
unaffected.
### duckdb-go-bindings passes -rdynamic
`github.com/duckdb/duckdb-go-bindings/lib/linux-amd64` passes `-rdynamic` in its
cgo LDFLAGS, which makes the linker export every symbol into the dynamic symbol
table. That table survives `-s -w`. duckdb costs 54.1 MB of a `most` build, and
almost all of it is that table.
### gorm-dameng imported the plugin package
Resolved by removing the driver on 2026-09-23, not by a report. It was the only
importer of Go's `plugin` package anywhere in the graph, and importing it has
the same effect as `-rdynamic`. Removing it took a build with every driver from
260.2 MB to 215.7 MB. duckdb still carries the same cost through `-rdynamic`.
## W12. `\chart file=NAME` should not need terminal graphics
`doExecChart` in `handler/handler.go` returns `text.ErrGraphicsNotSupported`
before it reads its arguments. When `file` is set the chart is written to a file
as SVG and nothing is drawn in the terminal, so the check rejects a request it
could have served. Reproduce it by piping a statement into a build made with
`-tags charts`:
printf 'select 1 as a, 2 as b \\chart type bar file=/tmp/out.svg\n' | usql sqlite3://:memory:
Move the check past the point where `cfg` is known, and apply it only when
`cfg.File` is empty. This is older than the charts build tag and is not caused
by it.
## W13. Shrink the charts stack
Charts are behind the `charts` build tag as of 2026-09-22, so this only affects
a build made with `-tags charts` or `-tags all`. Measured on linux/amd64 by
diffing a `-tags charts` build against a `-tags none` build, both stripped with
`-trimpath -ldflags "-s -w"`. The tag costs 12.4 MB, which splits as:
| part | cost |
|---|---:|
| `libresvg.a`, the Rust static library | 3.68 MB |
| `github.com/dop251/goja` | 1.69 MB |
| `golang.org/x/text` | 1.27 MB |
| `echarts.min.js`, embedded | 1.20 MB |
| Go standard library and runtime pulled in | 1.21 MB |
| `.gopclntab`, `.go.type`, `.eh_frame`, `.data.rel.ro` | 2.9 MB |
| `go:func` | 0.17 MB |
Two of those are usql's own modules and can move.
`echartsgoja.go:288` carries `//go:embed *.js`, which embeds `echarts.min.js`
verbatim at 0.98 MB. Minified JavaScript compresses about four to one, so
storing it gzipped and inflating it on first render would return roughly
0.75 MB. `compress/gzip` is already linked into most usql builds.
`libresvg.a` is the largest single piece. The xo/resvg session is designing this
and has confirmed the scope: keep the `text` and `system-fonts` Cargo features,
which are load-bearing, and drop `raster-images` and `svgz` for a lean variant
while leaving them on by default for other consumers. Do not scope work around
removing font shaping. A chart carries about ten `` elements at
`font-family:sans-serif`, covering the axis labels, the title, the subtitle and
the legend, and `xo/resvg` sets `loadSystemFonts: true` at `resvg.go:89`.
`raster-images` is safe to drop for usql specifically, and that is a guarantee
rather than a sample: `metacmd/charts/charts.go` builds the option document from
a closed struct with five fields, Title, Legend, XAxis, YAxis and Series, none
of which can carry an image, a URL or a data URI. usql cannot emit ``.
The 1.27 MB of `golang.org/x/text` is the collation tables that goja links for
locale-aware string comparison. It arrives through echartsgoja's dependency on
goja and is only avoidable by changing engines, which is not worth it.
That session has also split libresvg into one Go module per platform, released
in xo/resvg v0.9.1, with each `libresvg/` tagged v0.48.1 to match the
vendored resvg version. usql is on v0.10.0, which adds an embedded font for
that project's own tests and a `WithSansSerifFamily` option, and is the same
size. Measured with an empty module cache, a
linux/amd64 build now downloads 11 MiB of archives rather than the 184 MiB of
the monolith, and each platform links only its own submodule. Note that
`go mod tidy` still records all six as indirect requires, so `go mod download`
with no arguments fetches all of them; a build does not.
That is a download win, not a binary size win: the charts build is 58.2 MB
either way, because it is the same archive. It does not resolve issue 494
either. See the resvg item under Lowest priority for what 494 actually needs.
## W14. NULL scan failures that remain
The reported crash is fixed and released into main, and issues 307, 476 and 539
are closed. `drivers.NullSafeColumnType` chooses a destination that tolerates a
NULL while keeping the type information the alignment and time format need.
What is left is the same fault in two other places.
Pull requests 524, 526, 570 and 583 fix NULL scans in the metadata readers,
which build their own scan destinations rather than going through tblfmt.
Review and merge them, then decide whether a shared helper would serve them.
`\copy` has the identical fault at the `reflect.New(columnTypes[i].ScanType())`
near `drivers/drivers.go:595`. A NULL in a source column fails the copy. That
is issue 397, and issues 254 and 462 are probably the same fault. The
destination there is handed to `ExecContext` rather than printed, so the
mapping is not identical to `NullSafeColumnType`.
## W15. Output changes that came in with tblfmt v0.19.0
Landed, recorded because they change what users see and may draw reports.
`NaN`, `+Inf` and `-Inf` now print as `NaN`, `Infinity` and `-Infinity`, which
is byte identical to psql 18.6. A uint64 is always a JSON string, so a MySQL
`BIGINT UNSIGNED` column is one JSON type for every row. json output is
indented rather than compact.
usql still passes `drivers.NullSafeColumnType` rather than
`tblfmt.WithUseColumnTypes`, and must keep doing so: the latter still builds
`reflect.New(ct.ScanType())`, which is the crash, and that is upstream of
anything tblfmt can fix.
## W16. Roadmap, moved here from GitHub issues
Eighteen issues that Ken filed, mostly in January 2021, as his own roadmap in
issue form. They were never stale, only in the wrong place: a roadmap kept in
an issue tracker and a roadmap kept in this file are two roadmaps, and the
issue count then hides the real bug reports underneath them. Gemini and
DeepSeek were both asked where they belonged, and both said the duplication was
the problem; Gemini's answer, that this file is the source of truth and the
tracker is for defects and contributor-ready work, is the one taken.
They are closed on GitHub with a comment pointing here. Issue 215 stayed open
as a defect and is W5.
Anything below that is ready for someone else to pick up should be re-filed as
a narrow issue with acceptance criteria when that is true, rather than left
here with `help wanted` on it. That was the one real cost of the move, and it
is worth paying attention to: `is:issue label:"help wanted"` is how outside
contributors and aggregators find work, and this file is not indexed by any of
them.
### Already owned by another item
Four were closed because the work is described elsewhere in this file, not
because it was dropped.
137, a wrapper for the C readline library, wanted a standalone package with the
same interface as `rline`, selectable by build tag. W8 supersedes it.
165, the special `\\` separator. psql accepts an escaped backslash as a
statement separator, and only the first one: `\x \\ select 1; \\ select * from
foo;` runs the first two and then reports `invalid command \`. W3.
166, quoted string processing and variable interpolation in metacmds. The
issue carries a full side-by-side of psql against usql covering `:{?name}`,
single and double quoted interpolation, standard escape decoding, `E''` style
escaping, and backtick interpolation. `testdata/quotes.sql` is in the tree and
is the reference. W3, and the largest single piece of it.
217, additional variable types, evaluation and interpolation. A long proposal
that deliberately breaks from psql: extended variable types with their own
prefixes for connections and queries, plus shell and ruby style interpolation
and expression evaluation. W3 should settle psql parity first, since this
builds on top of it.
### psql compatibility, still wanted
141 and 142, LaTeX and troff table output, both for psql parity. These belong
to tblfmt rather than usql.
147, `\if`, `\elif`, `\else` and `\end`. Note that psql's condition evaluation
is deliberately simplistic, treating any non-empty string as true, so parity
here is easier than it looks.
158, `\ef` and `\ev`, which need introspection in place first.
160, `\errverbose`, to show the detailed error information a driver can give
beyond the message.
161, the `--echo-*` command line flags.
374, document prompt formatting. The prompt escapes are W3; this is the
documentation half.
### Proposals, not scheduled
144, a shared buffer pool for table output. Recorded as needing a large
overhaul of tblfmt, which now has its own session, so this is a question to put
there rather than work to do here.
146, a `\copy` proposal, which predates the `\copy` that exists. The open
questions in it are still open, and item 4e holds the current `\copy` defects.
154, an expanded test suite covering SELECT, INSERT, UPDATE and DELETE against
each major database and syntax compatibility across all of them, plus the
popular non-major ones. W2 has taken the container half of this; the
per-database statement coverage has not been done.
157, support for databases that have a Go API but no `database/sql` driver:
Redis, InfluxDB and IQL, Aerospike AQL, ArangoDB AQL, OrientDB SQL, Cypher and
SPARQL, JIRA JQL. Needs either a generic adapter or an overhaul of the drivers
package to admit non-SQL backends. Tier 3 holds the drivers that do have a
`database/sql` driver, and is the cheaper list.
162, `\j*` metacommands for processing fields with JavaScript in flight, with
`\jset` to define a function and `\j` to apply it to the current or last
statement buffer, composing with `\copy` for ETL. Note that usql already links
a JavaScript engine through the charts renderer, though only under the `charts`
build tag, so the cost of this is lower than it was in 2021.
216, a `\values` command producing an ephemeral result set from expressions,
for testing, variable manipulation and named queries. Depends on 217's
evaluation syntax.
267, an `\import` proposal.
## W17. A password command hook
Issue 422, accepted by Ken in the thread and moved here. Not started.
Let a user point usql at a command that returns a password, so a password
manager can be the source instead of a plaintext `.usqlpass`. The reporter has
twelve databases and keeps their passwords in KeePass; they explicitly did not
ask for KeePass support, only a hook, so that 1Password, Bitwarden, Vault,
`pass` and anything else can be wired up by the user.
The obvious design does not work. Making `.usqlpass` executable and running it
fails on Windows, which usql ships binaries for and where there is no execute
bit and a program needs an extension. The reporter noticed this themselves in
the thread and suggested naming the command in `.usqlrc` instead. Some setting
that names a command is therefore the shape to take, not a mode bit on the
existing file.
Open questions from the thread. What the command is given: the reporter
suggested the fields already used for a passfile lookup, meaning protocol,
host, port, database and user, with the driver name expanded to its full form
and the usual defaults filled in. And what the contract is: presumably the
password on stdout, a non-zero exit meaning no password rather than an error,
and a timeout so a hung helper does not hang usql.
Prior art worth reading before designing it. git's credential helpers solve the
same problem and have settled on a key-value protocol on stdin and stdout,
which is more extensible than positional arguments. psql has no equivalent, so
there is no compatibility constraint here and no reason to invent a
psql-shaped answer.
usql resolves passwords through `github.com/xo/dburl/passfile` today, called
from `env/env.go`. That package matches entries out of a file and has no notion
of running anything, so the hook belongs in usql above it rather than inside
dburl.
## W18. Two missing interfaces account for every metadata gap (Superseded by W21)
Source: measured on 2026-09-26 while checking dbmeta's `docs/USQL.md`.
Superseded on 2026-09-27 by D3: no metadata reader is written in usql, so the
two interfaces are not added here. The measurement below stays, because it
says what W21 has to answer in `dbmeta`.
Of 51 registered names under `-tags all`, 21 have a metadata reader. Every one
of the 21 that fails a command fails on `\l`, on `\dp`, or on both. Nothing
fails `\d`, `\dt` or `\dn`.
postgres pgx cockroachdb redshift sqlserver trino duckdb 11/11
mysql mymysql memsql tidb vitess nzgo databend snowflake 10/11 no \l
oracle godror 10/11 no \dp
sqlite3 moderncsqlite 9/11 no \dp, no \l
clickhouse 8/11 no \di, \dp, \l
impala 6/11 no \da, \df, \di, \dp, \l
`\l` needs `CatalogReader`. `\dp` needs `PrivilegeSummaryReader`. So two
interfaces, implemented where the product supports them, take almost every
driver in usql to full coverage. That is a much smaller piece of work than a
ratio column suggests, and a ratio column is what hid it.
### Why the same gap appeared 21 times
The instruction was "implement the readers you can". That cannot be checked,
so it cannot fail, so it produced whatever happened. Naming the two interfaces
in prose is the weaker half of the fix.
The stronger half is a test over the driver registry: every registered name
either implements `CatalogReader` or has an entry in an exemption map giving
the reason it does not, and the same for `PrivilegeSummaryReader`. A
contributor cannot pass it by doing nothing, and a contributor who genuinely
cannot implement it writes one line saying why. The exemptions then become a
readable list of product facts instead of a silence.
Thirty of the 51 will need an entry on the first run. That is not the test
failing, it is the first time the gap has been written down. Expect a good
fraction of those entries to read "nobody tried" rather than "the product
cannot", because today the two are indistinguishable.
This shape is `TestEveryDialectIsMeasuredForParity` in dbmeta, which was
written for the same problem.
### One measurement artifact worth knowing
`drivers/metadata/impala/metadata.go:69` returns `struct{}{}`, commented as a
reader with no capabilities, when the handle is not a `*sql.DB`. It is
deliberate, but impala is the only driver that can lose all metadata support
without reporting anything.
## W19. Guard the numbers that documentation states
Source: an exchange with the dbmeta session on 2026-09-26, in which four
separate figures across the two projects turned out to be wrong.
Prose decays silently. Four wrong numbers were found in one afternoon: four
open pull requests that were two, eight metadata commands that were eleven,
eight gating readers that were seven, and 47 drivers that were 51.
Every one of them was a right count of the wrong thing. So a test that checks
a number without also checking what is being counted catches none of them.
### Generate before testing
usql already solves this for its largest table. `gen.go` builds both README
driver tables from the dburl registry, so the table cannot drift, because there
is nothing for it to drift from. Generation is stronger than a test: a test
reports that prose is stale, generation means it never was.
Where a number can be generated, generate it. The test is for the residue.
### What the test looks like
Count the real thing, then search each document for a number claiming to be
it. Three details are load bearing:
1. Assert that every pattern still matches something. A pattern that stops
matching is worse than a wrong number, because the test goes green and
guards nothing.
2. Use one anchored pattern per claim. Two counts a sentence apart get read as
each other otherwise.
3. Check the file the numbers are about, not only the files that cite it.
dbmeta checked three files, skipped the one the numbers described, and that
was the file that rotted.
Write the numbers as digits. A test cannot read "eleven".
Put the unit inside the sentence the test searches, so that changing the unit
breaks the test. Match `(\d+) registered names with a reader` rather than
`(\d+) drivers`.
## W20. Test that no driver writes global process state
Source: proposed by the dbmeta session on 2026-09-26, while writing
[DRIVER.md](DRIVER.md).
`docs/DRIVER.md` tells a contributor to read the upstream driver's `init()`
and to stop if it writes global process state. That is a question a person has
to remember to ask, which is the weaker form of the same check.
Make it a test. Walk the driver packages, resolve each imported upstream
driver module, and fail when package initialization calls any of:
slog.SetDefault log.SetOutput log.SetFlags
log.SetPrefix flag.Parse os.Setenv
http.DefaultClient http.DefaultTransport
signal.Notify rand.Seed os.Exit
A driver may register itself with `database/sql` and do nothing else.
### Why this is worth building
It would have caught RamSQL before the merge rather than after. Its
`engine/log` called `slog.SetDefault` from `init` with a handler on
`os.Stdout`, and a default `slog` logger also redirects the standard `log`
package, so importing that one driver changed the output of the whole binary.
It was found by watching the test suite misbehave, which took far longer than
reading an import graph would have.
### The design questions to settle first
This is not a small test, and it is listed as an item rather than written into
the documentation change for that reason.
Deciding what counts as package initialization means following calls out of
`init()` rather than pattern matching a single file. `go/packages` or
`golang.org/x/tools/go/ssa` can do it. A cheap first version could search only
the direct `init()` bodies of the imported driver package, which would still
have caught RamSQL.
A second question is what to do about a legitimate hit. An exemption map with
a reason for each entry is the shape W18 already uses, and it keeps the test
from becoming something people delete.
A third is cost. Resolving every upstream module makes this slow, so it may
belong behind a build tag or in CI rather than in `go test ./...`.
### A cheaper version that is worth doing first
An empty `main` that imports one driver, and asserts that `slog.Default()`,
the standard `log` flags and prefix, `flag.CommandLine` and
`http.DefaultClient` are unchanged after the import. It answers the question
by observation rather than by analysis, and it needs no module resolution.
## W21. Migrate metadata to dbmeta
Source: planned on 2026-09-26 with the dbmeta session, gemini and deepseek.
The plan is [DBMETA.md](DBMETA.md). This item exists so the work can be cited.
W7 is the dbmeta repository itself. W21 is usql's side of the move.
The boundary is that dbmeta reads, tblfmt renders, and usql decides what to
print. The writer does not move.
### The root cause this fixes
`drivers/metadata/metadata.go:120` declares `type Reader interface{}`. The base
reader is the empty interface, so every capability is a runtime type assertion,
and there are 52 of them outside tests. A driver that satisfies nothing
compiles, registers and passes the tests. That single line is why W18 exists.
### Order, which is not the order it was first described in
Two steps move earlier than first planned.
The formatter work comes before the reader swap. It is entirely internal, no
project outside usql imports `drivers/metadata`, and it is where the
psql-matching behavior lives.
Before even that, extract the psql-matching filtering out of `DefaultWriter`
with the current readers still in place, and confirm the output is unchanged.
dbmeta returns what the database says and does not shape results to match
somebody's output. Swapping a reader before that filtering moves changes usql's
output silently, as extra rows rather than as an error.
That step has no visible result, which is why it is the one most likely to be
skipped.
### The early win is not a native model
30 of 51 registered names have no reader. dbmeta's shared information schema
model answers 12 object kinds against any database with a standard
`information_schema`, with no per-driver query written. That adapter lands
coverage across a set of names at once and should ship before any native model
moves.
MariaDB is the first native model to move. The command gain is one, `\l`,
because the mysql family lacks only `CatalogReader`. The gain worth having is
5 of 5 sections under `\d NAME` against the 4 answered today.
The largest gains are four products where usql answers nothing at all, because
their drivers register no reader: SAP HANA at 0 of 11 today against 11 of 11
and 5 of 5 with dbmeta, Firebird at 10 of 11, Cassandra at 10 of 11 and Presto
at 8 of 11. None can come early. SAP HANA has no `information_schema`, so it
needs a native model rather than the shared adapter.
### Two bugs this fixes rather than relocates
usql changes a password in seven drivers and escapes nothing. Each concatenates
the password into the statement, so a password holding a quote or a backslash
breaks it or sets the wrong thing. Correct escaping needs the server, because
the rule is `sql_mode` on MySQL and MariaDB and `standard_conforming_strings`
on PostgreSQL.
`drivers.Version` falls back to `SELECT version();` for any driver that
registers no version function. Oracle is not one of them: it registers one
through `orshared.Register`, which an earlier draft of this item missed.
Oracle's defect is different and is W22.
### Moving Oracle costs `\l` until dbmeta answers it
dbmeta's Oracle model registers no `Databases` query. Oracle is one of two
dialects there that cannot answer `\l`, recorded on the grounds that Oracle has
one database per instance and no list to read.
usql answers `\l` for Oracle today, for administrators only, through
`v$parameter`. So moving Oracle as things stand trades one command for
correctness everywhere else, going from 11 of 11 to 10 of 11 on that axis.
dbmeta will not write the query until somebody decides what `\l` means on a
multitenant Oracle server. `SYS_CONTEXT('USERENV', 'DB_NAME')` returns the
container name rather than the pluggable database name, and a listing that
names the container when the connection is to a pluggable database is worse
than no listing. If the answer is the pluggable databases visible to this
connection, there is a view for it. If the answer is the one database you are
attached to, it is a single row.
That decision is a dependency on dbmeta, not usql work. Pull request 570 pairs
`SYS_CONTEXT` with `all_db_links`, which lists links to other databases rather
than databases, which suggests usql's Oracle catalogs query is answering two
questions at once.
### The metadata layer is frozen
Decided on 2026-09-26. `drivers/metadata` takes no more changes before the
migration.
Small fixes are still worth having in principle, and both external reviews
argued for merging them, on the grounds that the migration has no date and
main ships to users today. Ken decided the other way: the layer is being
replaced, and churn in it costs merge pain for code that is going away.
So pull requests against the metadata readers are held rather than merged, and
the defects they report move to the migration instead. Held on this basis:
524 oracle version query needs privileges an ordinary user lacks
570 oracle catalogs query reads administrator-only views
583 postgres catalog access privileges crash on NULL
A held pull request is not a closed one. Each defect has to survive the move,
which means W21 inherits them and the new readers have to answer for them.
### Implemented, waiting for a dbmeta tag
Implemented on 2026-09-30 and staged for review, following D4. usql reads all
of its metadata from dbmeta: the describe commands, tab completion, the
version, the current user and the change of a password. Every usql reader is
gone. `metacmd/describe` holds the commands, which print dbmeta's answers the
way psql prints them, and a small source of names for the completer.
It builds against a dbmeta checkout through a local `go.work`, and is not
committed until dbmeta has a tag for `go.mod` to require. The steps of
`docs/DBMETA.md` it covers are 0 to 2, 4 to 6, and the count of step 4.
Step 3, the information schema model for a database with no model of its own,
is dbmeta's to register, and Ken is deciding which databases get one.
### Open decision
dbmeta has no release tag and its API is still moving. Whether usql depends on
an unreleased module, and whose cadence wins, is Ken's call. The per-driver
shape of the move limits the exposure without removing it.
When usql calls dbmeta, choose the dialect from `u.Dialect`, which dburl
v0.32.0 sets, and never from `u.Driver`. Every PostgreSQL URL now has Driver
`pgx`, which is not a dbmeta dialect. `u.Dialect` is `postgres` for all of them,
`sqlite3` for moderncsqlite, `oracle` for godror, and `mysql` for TiDB,
SingleStore and Vitess. dbmeta's `docs/COMMANDS.md` says the same under "Wiring
usql up".
## W22. Metadata and version queries assume a privileged user (Superseded by W21)
Source: found on 2026-09-26 while triaging pull requests 524 and 570. Answered
the same day.
usql's Oracle queries cannot be run by an ordinary user.
`drivers/oracle/orshared/orshared.go:38` reads the version from `v$instance`,
and the catalogs query reads `v$parameter` and `dba_db_links`. All three are
administrator objects. Pull requests 524 and 570 report this independently.
Nobody had asked usql's queries as anyone other than the administrator, because
the tests connect as a privileged user, so a query an ordinary user cannot run
passes every test here.
### Why this is superseded rather than open
dbmeta already solves it, measured rather than asserted. Its Oracle model
touches exactly one `v$` view in the whole model, which is the version query,
and uses `all_` views everywhere else. On Oracle 26ai as a user granted only
CREATE SESSION:
SELECT version FROM v$instance ORA-00942, no such view
SELECT banner FROM v$version WHERE ROWNUM = 1 answers
So usql prints a version for an administrator and nothing for anybody else, and
dbmeta prints one for both.
It is also not a one-off audit there. dbmeta runs `TestPrivilegeParity` in CI
on every push, asking every query as the administrator and as each lesser
principal the product has, with the results checked in. A query that starts
depending on who is asking fails their build.
The metadata layer here is frozen, so these queries move to dbmeta rather than
being fixed in place. The defect does not survive the migration. Pull requests
524 and 570 stay held.
### One caution that outlived the item
`SELECT banner FROM v$version` returns five rows on Oracle 11.2, one each for
the database, PL/SQL, CORE, TNS and NLSRTL. It became a single row in 18c.
dbmeta's query carries `WHERE ROWNUM = 1` and has since it was written. Pull
request 524 has no predicate and is nondeterministic on any server before 18c.
## W23. Variables need a scrub, and `\copy` does not resolve them
Source: issue 495 and pull request 542, triaged 2026-09-26.
`\copy` goes straight to `dburl.Parse` on its source and destination, so a
connection variable set with `\cset` is not resolved. Issue 495 reports it:
\cset MSSQL mssql://USER:PASS@HOST:PORT/DATABASE
\copy MSSQL csvq://. 'YYY'
error: invalid database scheme
`\c MSSQL` works, and `\copy` with the URL spelled out works. Only the
combination fails.
Pull request 542 is closed rather than merged. It resolved the variable but
then hand-built a `dburl.URL` from a driver and a DSN instead of going through
`dburl.Parse`, which leaves `Host`, `User`, `Path`, `RawQuery` and `GoDriver`
empty. `drivers.Open` passes the whole URL to any driver with its own `Open`,
and six have one: cassandra, moderncsqlite, pgx, postgres, sqlserver and
vertica. So it would have broken `\copy` for postgres and pgx, which are the
drivers most likely to be used with it.
The correct shape is to resolve the variable to a connection string and pass
that string to `dburl.Parse`, so that one code path handles both cases.
### This is part of a larger change
Variables in general are getting a clean up, and this fix belongs inside it
rather than ahead of it. Doing it piecemeal means touching `\copy` twice.
## W24. Read a command or a script from standard input
Source: pull request 479, triaged 2026-09-26. Planned for the next major
release.
`usql -c -` and `usql -f -` should read from standard input, which is what
psql does.
Pull request 479 is closed rather than merged. It called `scanner.Scan()`
exactly once on a `bufio.Scanner` with the default line splitter, so it read
only the first line and silently discarded the rest of a piped script. It also
deleted two imports that are still used, and it predates two years of changes
to `run.go`.
The feature is wanted. Read all of standard input rather than one line, and
decide what `-f -` does when it is given more than once.
## W25. Stop leaking `readline.AutoCompleter` out of the rline package
Source: found on 2026-09-26 while triaging pull requests 418 and 535.
`rline/rline.go:20` already declares an `IO` interface, so usql is mostly
decoupled from its line editor already. One type escapes:
drivers/drivers.go:110 NewCompleter field on drivers.Driver
drivers/drivers.go:494 NewCompleter func
drivers/completer/completer.go:113 NewDefaultCompleter func
`readline.AutoCompleter` is in the public `drivers.Driver` struct, so the line
editing library reaches into the driver API. Every driver that supplies its own
completer names the type.
Declare the interface in usql and change the three signatures. The interface
has one method, so this is not inventing an abstraction, it is naming one that
already exists under a different package.
This is deliberately not shaped for any particular library. It is the part of
pull request 418 that is worth having regardless of which editor wins, and it
means W8 does not have to change the driver API.
## W26. Prompt formatting: PROMPT2, PROMPT3 and the documentation
Source: pull requests 577 and 578, closed 2026-09-26. To be done differently.
usql has `PROMPT1` and nothing else. `handler/handler.go:561` already documents
`%w` as "Whitespace of the same width as the most recent output of PROMPT1.
This can be used as a PROMPT2 setting", so the documentation already describes
a variable that does not exist.
Pull request 577 added `PROMPT2` with a default of two spaces and implemented
`%w` by tracking the last rendered prompt width on the handler. Pull request
578 added a prompt formatting section to the README and stated that `PROMPT2`
and `PROMPT3` use the same escape sequences. Neither adds `PROMPT3`, so 578
documented two variables that do not exist and one that another pull request
was adding.
Both are closed because this is being done differently and soon.
### What has to be true when it is done
`PROMPT2` is the continuation prompt for a statement spanning several lines.
`PROMPT3` is what psql prints while reading `COPY` input, so it needs the copy
input path rather than only a variable. Adding the variable without that path
makes it inert.
If `%w` is implemented, its width has to be measured in terminal columns rather
than bytes or runes, because a prompt usually carries colour escapes and may
carry wide characters.
Do not document a prompt variable before it works. That is how the current
state arose.
## W27. Retire, replace and regroup drivers
Source: a review of every driver on 2026-09-27, with Gemini. DeepSeek was asked
five times and returned nothing each time. Ken made the decisions below the
same day. Module dates come from the Go proxy.
Remove these four. Each also drops its scheme from dburl, which has been told:
- mymysql, `github.com/ziutek/mymysql`, last released 2015-01-09.
`go-sql-driver/mysql` covers the same database.
- adodb, `github.com/mattn/go-adodb`, last released 2018-05-15. odbc covers
it. The `oleodbc` override goes with it.
- sapase, `github.com/thda/tds`, last released 2019-09-27. `SAP/go-ase` last
changed on 2025-05-22 and is not a replacement.
- ignite, `github.com/amsokol/ignite-go-client`, last released 2019-01-04.
Switch these three to a maintained module:
- flightsql moves to `github.com/apache/arrow-go/v18`, v18.8.0. That module is
already an indirect dependency, so this removes a second Arrow tree.
- maxcompute moves to `github.com/aliyun/aliyun-odps-go-sdk/sqldriver`,
v0.4.26. It registers as `odps` and takes a different DSN, so dburl's
generator changes with it. The SDK carries its own Arrow package, so check
the dependency weight before merging.
- postgres moves from `github.com/lib/pq` to `github.com/jackc/pgx/v5/stdlib`.
DSN handling, error text and type mapping change, so re-record the
`cli_test.go` goldens and run it against a live database before release.
Regroup these two:
- csvq moves from `base` to `most`. It is finished rather than broken.
- h2 moves to `bad`, as triage before removal. `github.com/jmrobles/h2go` last
released on 2020-11-14.
Keep these two, on forks that Ken is starting:
- cassandra. `github.com/MichaelS11/go-cql-driver` last released on
2020-09-20 and wraps `gocql/gocql`, which has since moved to
`github.com/apache/cassandra-gocql-driver/v2`.
- couchbase. `github.com/couchbase/go_n1ql` last changed on 2022-03-03.
Both forks bring the code up to modern Go. usql and dburl switch to them when
they exist.
As of 2026-09-27 the removals, both regroupings, and the flightsql and
maxcompute switches are done, on dburl v0.31.0. usql registers the odps driver
under `maxcompute` as well, because that is the name dburl gives the scheme.
The postgres switch is done too, on dburl v0.32.0, following dburl's D22.
`postgres://`, `cockroachdb://` and `redshift://` reach pgx, which is now in
`base` and also answers to the `postgres` build tag. lib/pq stays supported
as `drivers/libpq`, in `most`, and dburl sends `pq://` and `libpq://` to it.
It still registers as `postgres`, which is lib/pq's name in `database/sql`.
The Cassandra fork is done. The cassandra driver uses `github.com/xo/cql`
v0.1.0, which wraps `github.com/apache/cassandra-gocql-driver/v2`. It returns
the real connection error, so usql no longer captures the gocql log to find
it. A NULL now shows as empty rather than as a zero value.
The Couchbase driver is replaced by `github.com/xo/dbimp/couchbase` v0.1.0,
written in dbimp rather than forked from `go_n1ql`. It registers as
`couchbase` and takes only a `couchbase://user:pass@host:port/` URL. It
landed with dburl v0.33.0, which renamed the scheme from `n1ql` and keeps
`n1ql` and `n1` as aliases, following dburl's D25. usql adds
`txtimeout=30m` to a URL that has none, because the server ends a transaction
after 15 seconds by default. It does not set `durability_level`, and a
single-node server needs `durability_level=none` to commit a transaction.
Netezza was removed on 2026-09-29, and its scheme goes from dburl. Ken
decided it: the driver was never tested, nobody is known to use it, and no
copy of Netezza is available to test with.
Most driver packages under `drivers/` have no `Group:` line and fall to
`most` by default, which breaks rule 10 in `AGENTS.md`. Add the line to each
as they are touched.
## W28. Statement handling for ODBC, csvq, Athena and Cosmos
Source: Ken, on 2026-09-30, while dbmeta took over the grammar of each
database as its D143.
Four drivers changed a statement before usql ran it, through the `Process`
field of `drivers.Driver`. Ken removed those hooks. What a database does with a
statement belongs in dbmeta's syntax options, applied for every driver of it,
and that needs a larger design than D143. Each database is taken on its own.
What each hook did, and what is lost until this item is done:
- ODBC stripped a trailing semicolon, unless the statement ended with `END;`,
when the URL set `usql_trim`. The key is now ignored.
`dburl.OdbcIgnoreQueryPrefixes` still keeps `usql_` keys out of the DSN.
- csvq ran a `SHOW` statement as exec and pointed csvq's output at standard
output, because csvq prints the answer to `SHOW` rather than returning rows.
`SHOW` now prints nothing.
- Athena and Cosmos stripped a trailing semicolon.
The other `Process` hooks, in arangodb, influxdb, neo4j and surrealdb, did
nothing but run every statement as a query. That is now the `AlwaysQuery`
field, and `Process` is no longer a field of `drivers.Driver`.
### Open for discussion
Ken may remove ODBC, Athena and Cosmos, because none of them can be tested
against a server that CI can start. He would keep csvq, and wants to discuss
it with others first. Nothing is removed until that discussion.
### What has to be true when it is done
- dbmeta holds the rule for each of the four that is kept, and usql applies it
with no code for that database.
- No code in usql changes a statement's text for one database.
## W29. Move the SQL of \copy into dbmeta
Source: Ken, on 2026-09-30. After the metadata readers went to dbmeta (D4),
the `Copy` hook of `drivers.Driver` holds the last SQL that usql writes for
one database or another. It needs a careful design, like W28. Ken set it due
before the next release, on 2026-10-02.
What `\copy` builds today:
- `drivers.CopyWithInsert` and `drivers.FlexibleCopyWithInsert`, in
`drivers/drivers.go`, write `INSERT INTO table VALUES (...)`. Each driver
passes the placeholder of its database: `$1` by default, `?` for ClickHouse,
csvq, DuckDB, Exasol, MySQL, SQLite and Trino, `:1` for Oracle, and SQL
Server's own. Impala runs the inserts outside a transaction.
- The same code, and the pgx and libpq drivers, find the columns of the target
with `SELECT * FROM table WHERE 1=0`, or `SELECT cols FROM table WHERE 1=0`
when the target names its columns.
- pgx and libpq then use the COPY protocol of their Go driver, through
`CopyFrom` and `pq.CopyIn`. That is a feature of the driver, and D5 keeps it
in the driver.
The placeholder, the insert statement and the query for the columns of a
table all describe the database, so by D5 they belong in dbmeta. The design
has to decide how a client asks dbmeta for them, and what happens for a
database that dbmeta has no model for.
### Other SQL that is still in usql
A search of every Go file outside the tests found no other statement that
reads metadata, runs DDL or changes a password. These remain:
- `Version` hooks for seven databases that dbmeta has no model for: Athena,
csvq, ArangoDB, Neo4j and YDB run a query, and SurrealDB and InfluxDB call
their client library.
- Two fallbacks in `drivers/drivers.go` for a database with no model:
`SELECT version();` in `Version` and `SELECT current_user` in `User`.
- Text in `metacmd/describe` that is printed and never run: the
`FOREIGN KEY ... REFERENCES` line built for SQLite, the `CHECK (...)` around
an expression from information_schema, and the `CREATE OR REPLACE VIEW`
line that `\sv` puts above a definition.
- The keyword tables in `drivers/qtype.go`, which decide whether a statement
returns rows. They hold statements of PostgreSQL, MySQL and CQL.
Ken decides which of these move to dbmeta, and in which item.
## W30. Research an ODBC driver written on purego: Done
Done on 2026-10-07. `github.com/xo/odbc` v0.1.0 is that driver, pure Go on
purego, with no cgo, and usql uses it. The crash that it has in usql's default
build, and the question of its metadata, are in W40.
Source: Ken, on 2026-09-30.
Research a new ODBC driver for `database/sql`, written from the start on
`github.com/ebitengine/purego`. purego calls C functions from Go without cgo,
by loading the shared library at run time.
usql's `odbc` driver uses `github.com/alexbrainman/odbc`, and the README marks
it as needing cgo. So it is left out of a build without cgo, and CI installs
`unixodbc-dev` to build it. A driver on purego would load the ODBC driver
manager, such as `libodbc` from unixODBC, iODBC or `odbc32.dll`, when a
connection opens.
Questions for the research:
- Which platforms purego supports, and whether it can call every ODBC
function the driver needs, including those that take callbacks or pass
structs by value.
- How the driver finds the driver manager on each platform, and what error a
user sees when there is none.
- How the data types of ODBC map to Go, and how that compares with
`alexbrainman/odbc`.
- How to test it in CI against a real database through a real ODBC driver.
- Whether it lives in the xo family, and under which name.
This bears on W28, which asks whether ODBC stays in usql at all. A driver
without cgo could join the `most` group, and it could be tested.
## W31. The rest of psql's describe commands
Source: Ken, on 2026-09-30, once W21 put all of usql's metadata on dbmeta.
Status: Implemented, waiting for W21 to be committed.
usql now answers every command in dbmeta's `docs/COMMANDS.md` that dbmeta has
a query for. `metacmd/describe/lists.go` holds one spec for each: the query,
psql's title, and psql's columns. `metacmd/cmds.go` routes the commands, and
`TestEveryListIsACommand` checks that each spec is reachable. The new commands
are `\dA`, `\dAc`, `\dAf`, `\dAo`, `\dAp`, `\db`, `\dc`, `\dconfig`,
`\dC`, `\dd`, `\dD`, `\ddp`, `\dE`, `\des`, `\det`, `\deu`, `\dew`,
`\df` with `anptw`, `\dF`, `\dFd`, `\dFp`, `\dFt`, `\dg`, `\dl`,
`\lo_list`, `\dL`, `\do`, `\dO`, `\dP`, `\drds`, `\drg`, `\dRp`,
`\dRs`, `\dT`, `\du`, `\dx`, `\dX`, `\dy`, `\sf`, `\sv` and `\z`.
Two fixes came with them. `\dS+`, `\dtS+` and every other form with both S
and + were unknown commands, because a command name expanded each letter in
brackets alone. And a pattern now shows the objects of the system, as psql
does, which checks `!showSystem && !pattern`.
### What is still different from psql
dbmeta added the fields these commands lacked as its D147: the whole
definition of a function for `\sf`, the value with its unit for `\dconfig`,
the size for `\dT+`, the storage type for `\dAc`, the applicable types for
`\dAf`, the ICU rules for `\dO`, psql's columns for `\dX`, and the token map
for `\dF+`. usql prints all of them. These differences remain:
- An empty result prints `(0 rows)` with no title and no header. tblfmt does
this for every result, not only for describe commands.
- The second pattern of `\dAc` and `\dAf` matches the name of the class or
family, where psql matches the input type. dbmeta takes a name.
- `\dP` takes no `itn` letters.
A describe command now reads a name between double quotes as it is written,
as psql does. Before, usql removed the quotes before the pattern was folded,
so `\d "Foo"` looked up `foo`.
## W32. Review the size that each driver adds to the binary
Source: Ken, on 2026-10-01. Measure again at each of these times:
- Before a release.
- After `update-deps.sh` moves the dependencies.
- When a driver is added or switched to another module.
Run `./size.sh -g most`, which measures the drivers that a release ships,
and replace the table below. CONTRIBUTING.md says how the script works. A
driver whose share grows a lot is a question for Ken: keep it in `most`,
move it to `all`, or find a smaller module.
### The measurement of 2026-10-11
A release build with every driver of `most` was 145.8 MB, stripped, with
go1.27.2 on linux/amd64, against 203.1 MB on 2026-10-01: 57.3 MB less. The cause
is the move of Snowflake, BigQuery, Cosmos DB, Athena, Databricks, Spanner,
ClickHouse, Trino, Presto and DynamoDB to the pure Go drivers of dbimp. They
saved 17.4 MB (Spanner), 10.9 (Snowflake), 7.6 (Athena), 4.8 (ClickHouse), 4.6
(Databricks), 4.2 (BigQuery) and 2.7 (DynamoDB) before, and each saves 0.5 MB or
less now. The binary has 155 modules. DuckDB is 37% of it, and the five largest
drivers make up 90.4 MB.
| MB saved | Driver left out | What it brings |
| ---: | --- | --- |
| 54.1 | duckdb | the C++ library of DuckDB, with 4 MB of ICU data |
| 13.5 | oracle | the character set tables of go-ora |
| 9.7 | chai | Pebble, the storage engine of CockroachDB |
| 7.4 | ydb | the YDB SDK and its gRPC protos |
| 5.6 | flightsql | Apache Arrow v18 |
| 3.6 | moderncsqlite | SQLite translated to Go |
| 3.3 | sqlserver | go-mssqldb |
| 3.2 | csvq | the csvq engine |
| 2.2 | ots | the Tablestore SDK |
| 1.9 | maxcompute | the ODPS SDK |
| 1.7 | sqlite3 | the C library of SQLite |
| 1.4 | cassandra | gocql |
| 1.3 | hive | gohive and its Thrift code |
| 1.3 | firebird | the Firebird driver |
| 1.0 | impala and vertica | each |
The other 36 drivers add 0.9 MB or less, and 29 of them 0.2 MB or less. The
drivers of dbimp are the smallest group: 20 of them add 0.3 MB or less each, and
they share the code of dbimp (1.5 MB in all). `pgx`, `mysql` and the aliases that
import them show 0.0, because the other drivers use them too.
The size by module, from the symbols of an unstripped build (101.5 MB of
symbols that take room in the file): DuckDB 43.5 MB, go-ora 13.4, the table of
functions of Go 4.0, dbmeta 3.6, the YDB SDK 2.1 (and its protos 0.5), Pebble
2.1, modernc's SQLite 2.1, dbimp 1.5, Arrow 1.4, csvq 1.3, pgx 1.3,
golang.org/x/text 1.1, go-mssqldb 1.1, and the Go standard library about 4. The
32 MiB buffer of Go's FIPS random number generator (`crypto/internal/fips140/
drbg.memory`) is in the BSS, so it takes no room in the file, but it does count
in the memory of a running usql.
Things to read the table by:
- A number is what leaving out that one driver saves. Two drivers that share
a module each save only what the other does not use.
- Arrow v18 is the only version left in the build, for flightsql. The older
versions went with Databricks and BigQuery.
- dbmeta is 3.6 MB because it holds the statements of 40 models. It is the one
common module that grows with each database added.
- `bad` drivers are not measured, because no release builds them.
- The next saving would be DuckDB (a decision for Ken: its own group, as
`most` without it is 91.7 MB), then Oracle's character sets (13.5 MB for one
driver), then Pebble for chai (9.7 MB).
## W33. Make the README a landing page
Source: Ken, on 2026-10-01, after a review with Gemini and DeepSeek.
README.md is 1662 lines. 714 of them are Features and Compatibility, and 184
are pasted output of `--help` and `\?`. Both reviewers proposed that the
README answers a visitor's first questions, and that docs/ holds the
reference. Their merged proposal is about 230 lines:
| Section | Lines |
| --- | ---: |
| Header, pitch and badges | 10 |
| Install commands | 25 |
| First five minutes: install, connect, query | 40 |
| Supported databases, a compact generated list | 25 |
| Common connection examples | 30 |
| Backslash commands, an overview | 25 |
| Features, each linked into docs/ | 50 |
| Documentation index | 15 |
| Contributing | 10 |
The reference moves under docs/, with the listings generated by gen.go: the
full driver table, the output of `--help`, the `\?` listing, and the
variables. The configuration, copying, terminal graphics, time formats and
troubleshooting become pages of their own. The two reviewers agreed that the
database names stay in the README, because a visitor and GitHub's search look
for them there.
Open for Ken: the names of the files under docs/, and whether one generator
writes the README sections, docs/ and the man pages of W34 from one source.
## W34. Man pages for usql
Source: Ken, on 2026-10-01, after a review with Gemini and DeepSeek. The man
pages are generated from code, a person can edit their prose, and every
release package installs them.
The pages:
- usql(1): the command line, the environment and the files.
- usql-commands(1): the backslash commands and the variables. Both reviewers
first said section 7, and then agreed that section 7 is for conventions,
not for the use of a program.
- usqlrc(5): the formats of config.yaml, .usqlpass and the RC file.
How they are made:
- gen.go fills templates from the data that the code already holds: the cobra
flags in run.go, the descriptions in metacmd/descs.go, and env's
variables. A person writes the prose of each page in the template.
- Both reviewers said that go-md2man cannot write the definition lists that
120 commands need. That is wrong. go-md2man v2 turns a Markdown definition
list into roff `.TP`, which is the list a man page uses. So the prose can be
Markdown. Writing man(7) directly is the other choice, and it is harder for
a contributor to edit.
- A backslash in the text is escaped for roff. Unescaped, `\d` is a roff
command that moves the text down.
- The generated pages are committed under man/, and a test fails when they
differ from what the generator writes.
- The date in `.TH` is set when a release is tagged, with SOURCE_DATE_EPOCH
as an override for packagers. A date from the last commit would change the
pages, and fail the test, on every commit.
How they are installed:
- The tarball and the zip of build.sh hold man/ next to the binary.
- The Homebrew formula in xo/homebrew-xo calls man1.install and man5.install.
- The AUR packages install them with `install -Dm644` under
/usr/share/man.
- A hidden flag prints a page, for Docker and Windows, which have no man(1).
It is a flag and not a subcommand, because a bare word on the command line
is a DSN, such as a named connection. usql prints its completion scripts
through flags too, such as `--completion-script-bash`.
Open for Ken: Markdown or man(7) for the prose, the name of the flag, and how
the version in `.TH` gets into a committed page.
## W35. Match usql to transit's usql grammar
Source: Ken decided the design of transit's usql grammar on 2026-10-01, as
transit's D101 and D102. The grammar parses usql's input: the meta commands,
the variables, and the SQL that it hands to a SQL grammar for the dialect.
Nothing is due until usql uses the grammar. Two points touch usql:
- Outside a string, `\\` is psql's separator meta command. It ends the
arguments of the meta command before it, and the rest of the line is SQL.
usql's splitter reads `\\` as an escaped backslash today (stmt/stmt.go), so
the two differ there until usql follows psql.
- The grammar splits a meta command's name into its base and its modifiers,
such as `dtS+`, from usql's command list in metacmd/descs.go. So a new or
renamed meta command in usql needs a change to the grammar too.
The rest of the design is transit's. The grammar has one variant for each
family of dialects that usql's lexer options make: postgres, mysql, sqlite,
block comments only, cql, and no options. Variables reach the SQL grammar as
placeholders of the same length. Each meta command is one node, with a name
and typed arguments that usql maps to the kinds it completes. `\if` and its
branches are flat commands, as in psql.
## W36. Read input through rline and the usql grammar
Source: Ken, on 2026-10-01. The usql grammar of transit replaces the stmt
package, and rline replaces gohxs/readline and usql's rline package. usql has
no API to keep, and it takes the API of rline and transit as they are. This
design came from transit, rline, Gemini and DeepSeek. Where they disagreed,
psql decides.
### What the grammar gives
The root of the tree is a list of statements and meta commands. A statement
ends at a `;` outside parentheses, at a meta command, or at the end of the
input. An open string, quoted identifier, dollar quote or block comment is one
token to the end of the input, never an ERROR node. A variable is a node with
a byte range. Each family of dialects is its own language, so a change of
family on `\c` parses the unfinished text again, with no old tree. The
grammar joins nothing across a `;`, so a PL/SQL or T-SQL block, a CQL batch,
Oracle's `/` and T-SQL's GO are usql's to handle, as today.
### The design
- The query buffer stays, with psql's meaning. It is the raw text of the
statements that are not sent yet.
- A statement that ends in `;` joins the buffer, the buffer runs, and it
becomes the last query.
- A statement with no `;` stays in the buffer.
- `\g` sends the buffer once, or the last query when the buffer is
empty. `\p`, `\r`, `\e` and `\w` act on the buffer, as in psql.
- A first reviewer proposed to replace the buffer with the last statement.
That runs `select 1 \g` twice, so it is not taken.
- An entry is finished when its last top-level node is a statement that ends
in `;`, or a meta command. An open string, comment or parenthesis keeps it
open. An entry of only white space or a comment is finished when the
buffer is empty. rline's WithContinue asks this.
- The parse is kept with the text it came from, and is shared by
WithContinue, the highlighter and the completer. A parse runs only when the
text differs, after an edit of the old tree. rline calls the three in no
promised order, and does not always give them the same text.
- With no terminal, rline reads plain lines and calls none of the three.
usql collects the lines itself. It parses from the first byte that has not
run, runs every complete node except the last, and counts the lines that
ran, for the line numbers of errors. This covers pipes, `-f` files, `\i`
and the RC file. A `-c` string is parsed whole.
- A variable is replaced when its node runs, by its byte range, so the value
is never scanned again. That fixes the splicing of `:'x'` and the colon of
`::`. `:'x'` and `:"x"` double the quote in the value, as psql does.
- `\if`, `\elif`, `\else` and `\endif` get psql's meaning: a stack, and
an inactive branch runs nothing and expands nothing.
- help, quit and exit are recognized when an entry is only that word.
- CQL batches keep the joining that usql does today, from dbmeta's Batches.
- The stmt package goes after a differential test passes. The test runs the
old splitter and the new one over a corpus: the stmt tests, testdata and
the grammar's corpus. The two known bugs of stmt are listed as expected
differences.
### What waits for rline
These are rline's backlog items 33, 34 and 35, which are for Ken:
- A prompt for each row of an entry, for `%R`. Until then, a fixed
continuation marker.
- Text to start ReadLine with, for `\e`. Until then, the edited text goes
into the query buffer and is printed.
- A way to change WithContinue for `\prompt`. Until then, the function
knows when usql reads for `\prompt`. The answer is also taken out of the
history.
### Decided
Ken decided on 2026-10-01 that the chroma lexer goes, with no temporary
highlighter in its place. Highlighting comes back from transit's grammars,
whose tag is due soon. Until then, input has no color.
Ken also decided that usql does not add T-SQL's GO. Taken as defaults, for
Ken to change: Oracle's `/` stays out too, as in psql, and the package that
replaces stmt is named `input`.
## W37. Bring back the tests against databases, and their recordings
Source: Ken, on 2026-10-02. Due before the next release.
The tests of `drivers` and `drivers/clickhouse` started their own containers
through `ory/dockertest`, and loaded jOOQ's Sakila schema into PostgreSQL,
MySQL, SQL Server, Trino and ClickHouse. Ken had them removed on 2026-10-02,
with their recordings, because dbmeta owns every container setup. These went:
- TestWriter, which compared the output of the describe commands with the
recordings in `drivers/testdata`
- the TestCopy of `drivers` and of `drivers/clickhouse`, which copied rows
between databases with `\copy`
- the image definition, `gen-golden.sh`, and the fixtures in
`drivers/testdata` and `drivers/clickhouse/testdata`
- `github.com/ory/dockertest/v4`, and the job of CI that ran them
Much of the output of the describe commands still needs fixing, and the rest
of usql's plumbing is updated first. Then:
- Write the tests again against servers that dbmeta's `dbrun` starts. A test
takes the URL of its server from the environment, and skips when it is not
set.
- Load a fixture of the test's own over the connection, rather than build an
image.
- Record the output of the describe commands again, and the sessions of
TestCLI in `testdata/cli`.
- Cover `\copy` between two databases again.
## W38. Run CI on every database, through dbmeta and dbrun
Source: Ken, on 2026-10-02. Due before the next release.
Overhaul usql's workflows, so that they test usql against every database
that can run on GitHub's runners, through dbmeta's `dbrun`. usql writes no
container setup of its own.
dbmeta's `test.yml` shows the shape. One job builds `dbrun` and prints the
list of releases with `dbrun list --json --names tested`, and `nightly` for
the full list. A matrix runs one job for each release. Each job starts its
server with `dbrun`, takes the URL from `dbrun dsn`, and runs the tests. The
workflow names no database, so the list lives only in dbmeta.
- Choose which of dbmeta's selectors usql runs on each push, and which every
night.
- Run W37's tests, TestCLI and the manual checks of `docs/RELEASE.md` that can
be automated, such as changing passwords and the describe commands.
- Keep the jobs that build every driver, the static builds and the release
archives.
Note from dburl, 2026-10-11, for the overhaul: a lint job that floats the Go
version and pins the linter, or the reverse, fails whenever one side releases
before the other. dburl suggests pinning both in the lint job, such as Go 1.27.1
and golangci-lint v2.14.0, bumping them together on purpose, and keeping the
test job on `stable` so that a new Go release still shows test breakage. usql
runs no linter today. Ken has not decided.
## W39. Move Avatica to dbimp's driver: Done
Source: dbimp, on 2026-10-07. dbimp released an Avatica driver in v0.10.0
(its W24), and asks usql to use it in place of
`github.com/apache/calcite-avatica-go/v5`. That drops protobuf, the Kerberos
library and the digest library from `go.mod`.
Done on 2026-10-07, on dburl v0.41.0 and dbimp v0.10.1. dburl's D47 made the
`avatica` scheme name `github.com/xo/dbimp/avatica`, with the DSN
`avatica://user:password@host:port`, no port added and no path, and the keys
`tls` and `auth`. phoenix stays an alias. The old DSN was `http://`, which
the new driver refuses. A driver registered under a scheme that dburl does
not know fails `TestRegistryIsKeyedByScheme`.
In `drivers/avatica/avatica.go`:
- It imports `github.com/xo/dbimp/avatica`. The Apache driver and the modules
that only it needed, and some left over from earlier removals, are out of
`go.mod`.
- `Err` reads `Code` and `Message` of `*avatica.Error`.
- `IsPasswordErr` takes HTTP 401, and also the error that the server gives a
wrong password: HTTP 500 with the exception `SQLInvalidAuthorizationSpec`.
- `Version` runs `VALUES (DATABASE_VERSION())`, which works on the standalone
server and returns the release of HSQLDB, not of Avatica. The Phoenix Query
Server has no such statement, and the driver does not send
`databaseProperties`, so there the banner says the release is unknown. A
dbmeta model for Avatica would be the place for it.
Checked on 2026-10-07 with `dbrun`, on avatica-1.29.0 and phoenix-2.0-5.0:
the banner, a table of several types with NULLs, a syntax error, a missing
table, a duplicate key, a rollback and a commit, and a wrong password.
avatica-1.28.0 was not run.
## W40. Move ODBC, Trino and Presto to the drivers that dburl now names
Progress on 2026-10-07, on dburl v0.44.0, dbimp v0.13.0 and xo/odbc v0.1.0.
Trino and Presto are moved and checked. ODBC is moved, and it crashes in the
default build. See the end of this item.
- Trino and Presto: both import `github.com/xo/dbimp/trino`, and each has its
own hooks, so that `no_trino` leaves Presto alone. dburl writes the user
`user` and `flavor=presto`, so `trino://host` and `presto://host` connect.
`RowsAffected` is `drivers.RowsAffectedOrZero`, and `\copy` inserts without
a transaction, because Trino's connectors take a write only with
autocommit. Presto had no `\copy` before, and has one now. Checked on
trino-483 and presto-0.299 with dbrun: the banners name the product and
its release, types with NULLs, row counts, errors, `\copy` of three rows,
and `\dt` and `\d`. The wrong password was not checked, because those
servers take any user.
- `drivers.RowsAffectedOrZero` is new, for a server that gives no count. It
replaces the copies in Databend and ClickHouse. The shared `\copy` code
counts one row for an insert that gives no count, and still stops on any
other error.
Source: dburl, on 2026-10-07. Ken decided three scheme moves in dburl that
day (its D49 and D50). They are staged there and not released. Do nothing in
usql until dburl sends the tag.
- ODBC moves to `github.com/xo/odbc`, which needs no cgo. The registered name
stays `odbc`, and the connection string is passed on as it is. This bears
on W30, which researched an ODBC driver on purego, and on W28, which asked
whether to keep ODBC at all. Take both to Ken.
xo/odbc v0.1.0 was released on 2026-10-07, so dburl's release no longer
waits for it. It is a purego driver for Linux, macOS and Windows, with no
cgo. Its data source name is a URL, `odbc+://user:pass@host:port/db`,
or an ODBC connection string. A setting of the ODBC driver goes in the
query, such as `INITSTMT` for ANSI SQL mode on MariaDB and MySQL. Its
README has an FAQ for that. usql's old driver read the key `usql_trim`,
which W28 removed.
Its `sql.Conn.Raw` gives a connection with `GetInfoString`,
`GetInfoUint16`, `GetInfoUint32`, `Tables`, `Columns` and `PrimaryKeys`,
which read the metadata of any database that has an ODBC driver.
`odbc.Drivers` and `odbc.DataSources` list what the driver manager knows.
dbmeta decided on 2026-10-07 to build no model on them, and the sender says
that the fallback for a database that dbmeta does not know belongs to the
client. That would be a metadata reader in usql, which D3 forbids. It is
Ken's decision whether ODBC is the exception, or whether its describe
commands stay unsupported.
- Trino and Presto both move to `github.com/xo/dbimp/trino`, which registers
`trino`. For `presto`, `URL.SchemeName` is `presto` and `URL.Driver` is
`trino`, with the dialect `presto`. usql needs a registry entry keyed
`presto` that opens the `trino` driver. It replaces `trino-go-client` and
`presto-go-client`. The DSN
`trino://user:password@host:port/catalog/schema` is a guess until dbimp
tags the driver.
- The old ports 8080 and 8443, the forced user `user` and the default catalog
are gone from both generators. Check what usql's Trino and Presto drivers
assumed from them.
Checked on 2026-10-07: dbimp v0.12.0 has `github.com/xo/dbimp/trino`, for
Trino and Presto, with the key `flavor` (empty asks the server through
GET /v1/info). With dburl v0.43.0, a DSN with a user parses: `trino://u@host`,
`trino://u:p@host:8443/cat/sch` and `presto://u@host/cat`. A URL with no
user does not: `trino://host` and `presto://host` give `trino://host`, and
the driver refuses it with "no user, and both servers need one". dburl
v0.42.0 forced the user `user`.
Ken decided on 2026-10-07 that dburl writes the default user `user` for a
trino or presto URL that has none, and that the driver keeps its error.
dburl staged it in its D50, and also made the scheme set `flavor=trino` or
`flavor=presto` unless the URL names it, so `presto://u@host/cat` becomes
`trino://u@host/cat?flavor=presto` and the driver does not ask the server.
It is not released, because `xo/odbc` has no tag and D49 is in the same
release. So usql waits for that release, which comes after v0.43.0. Then
it takes that release, moves the drivers, and tests, in one commit. `Version`
reads `SELECT version()` on Trino, and /v1/info or nothing on Presto, which
has no `version()`.
ODBC, on 2026-10-07: `drivers/odbc` imports `github.com/xo/odbc` v0.1.0,
and `alexbrainman/odbc` is out of `go.mod`. It needs no cgo, so by the rule
for groups it moved from `all` to `most`. `IsPasswordErr` reads class 28 of
SQLSTATE, the failed authorization of every database. The metadata question
above is still open.
**Blocker, with its cause.** In a build that links both go-sqlite3 and DuckDB,
which is the default build and every `most` build, the first ODBC query
crashes the process with a SIGSEGV. The odbc project found the cause on
2026-10-07, and it is in how usql links, not in `xo/odbc`:
- DuckDB's bindings link with `-rdynamic`, which exports every global symbol of
the usql executable. That includes the 268 `sqlite3_*` functions of the
SQLite that go-sqlite3 carries. The SQLite ODBC driver is a shared library
that needs `sqlite3_*` functions too, and the dynamic linker gives it the
copies in the usql executable for most of them, and the system's
`libsqlite3` for the few that go-sqlite3 leaves out. Two builds of SQLite
then answer one connection. This holds for any ODBC driver that carries or
loads SQLite, and only when both go-sqlite3 and DuckDB are linked.
- Checked on 2026-10-07: the default build exports 268 `sqlite3_*` symbols and
crashes 6 of 6 times. The tag `libsqlite3`, which builds go-sqlite3 against
the system SQLite, exports 0 and worked 6 of 6 times with DuckDB still
linked. On that build ODBC gave data with NULLs, errors with their SQLSTATE,
and a rollback that undid its insert.
The ways out are Ken's decision, because each changes a release:
1. Build with `-tags libsqlite3`. go-sqlite3 then uses the SQLite of the
system, so the tags of `build.sh` (`sqlite_fts5`, `sqlite_json1` and the
others) stop applying, the features of SQLite depend on the distribution,
and the static builds need `libsqlite3.a`. Every installer who builds with
`go install` needs the tag as well.
2. Leave ODBC out of the builds that link both, which makes it an extra
again, with a README note.
3. Make `moderncsqlite`, which needs no cgo, the `sqlite3` driver of the
release builds, and keep go-sqlite3 behind a tag.
4. Ask the DuckDB bindings to stop exporting symbols.
Not decided. Do not release until one is.
Two limits of the drivers, not usql's: the SQLite ODBC driver fails on
`values (1)` with HY010, because it returns no data from the execute, and
`select * from (values (1))` works. And `xo/odbc` v0.1.0 fails on a statement
that returns no result set when it goes through `QueryContext`, such as
`create table`, with "fetching: return code -1". usql runs `create table`
through `ExecContext`, so it does not meet that today. The odbc project has the
fix in its working tree, and Ken decides about a v0.1.1.
The Homebrew formula in `xo/homebrew-xo` and the README paragraph about
`--with-odbc` need a look at the same time, because ODBC is no longer an extra.
## W41. Follow dburl's removal of TLS by scheme: Done
Done on 2026-10-07 on dburl v0.44.0. The MaxCompute example of the README now
uses `tls=false`, and the generated table lost `yds`, `ydbs`, `trs` and
`trinos`.
Source: dburl, on 2026-10-07. Ken decided that no scheme chooses TLS by its
scheme or transport any more, because the dbimp drivers read `tls=true`
(dburl's D51). dburl released it in v0.43.0, together with D49 and D50 of
W40.
usql cannot take v0.43.0 yet. Checked on 2026-10-07, with usql's own
drivers: `trino://u:p@host:8443/cat/sch` generated the DSN
`http://u:p@host:8443?catalog=cat&schema=sch` in v0.42.0, which
`trino-go-client` reads, and generates `trino://u:p@host:8443/cat/sch` in
v0.43.0, which it does not. `presto://` changes its driver from `presto` to
`trino` the same way. So a bump breaks both drivers until dbimp's Trino
driver exists, and usql moves them in the same change (W40). usql stays on
dburl v0.42.0 until then, unless Ken asks dburl for a release that holds D51
alone.
- ydb: the aliases `yds` and `ydbs` go, and return an unknown scheme error.
`ydb://host?tls=true` writes grpcs on port 2135, and `ydb://` stays grpc on
port 2136. `yd` stays.
- maxcompute and ots (alias `tablestore`): the `+http` and `+https`
transports go, so `mc+https://` is an invalid transport. TLS is on by
default, because they are hosted services, and `tls=false` writes http.
- dburl removes the `tls` key from the query, because the drivers do not
know it.
- clickhouse keeps `+http` and `+https` until dbimp writes its driver.
What usql has to change when it bumps:
- README.md: the MaxCompute example `mc+http://accessid:accesskey@host/api`.
Write it with `?tls=false` in its place. Check the line above it and the
note below it.
- The row of YDB in the generated table loses `yds` and `ydbs` when
`gen.go` runs again. Do not edit it by hand.
- No test, sample or other document of usql uses a scheme that goes. A
search on 2026-10-07 found only the two places above.
## W42. Decide whether ClickHouse moves to dbimp's driver: Done
Done on 2026-10-07. Ken decided to move (dburl's D53), and usql took it on
dburl v0.44.0 and dbimp v0.13.0. The ClickHouse driver is dbimp's, over HTTP
only, and `clickhouse-go` is out of `go.mod`. Its `RowsAffected` is the shared
`drivers.RowsAffectedOrZero`, its `Err` and `IsPasswordErr` read the codes 194
and 516 of `*clickhouse.Error`, and `\copy` inserts without a transaction,
because the server has none. Checked on clickhouse-26.9 with dbrun: the
banner, types with NULLs, a count, errors, a wrong password, `\copy` of three
rows, and `\dt` and `\d`.
Source: dbimp, on 2026-10-07. dbimp v0.13.0 has `github.com/xo/dbimp/clickhouse`,
for ClickHouse over HTTP. It could replace `ClickHouse/clickhouse-go/v2` in
`drivers/clickhouse`. dbimp asks usql to wait for dburl and for Ken, because
the choice is his. Nothing is done.
What the move changes:
- The driver speaks HTTP only, on port 8123. `clickhouse-go` also speaks the
native protocol. A user who connects to the native port loses that.
dburl's `clickhouse` scheme still names `clickhouse-go`, and still has the
transports `+http` and `+https`, which dburl keeps until dbimp's driver is
in (its D51).
- Both register the name `clickhouse`, so a program can link one of the two.
The driver is in the `base` group, so it is in the default build.
- `RowsAffected` goes. usql's hook returns 0 because `clickhouse-go` has no
count. dbimp's driver returns `dbimp.ErrNotSupported`, and usql would keep
the hook that turns that into 0, as it does for Databend.
- `Err` reads the `Exception` of `clickhouse-go`. dbimp's driver gives a
`clickhouse.Error`, and a wrong password has the code 194 or 516. usql's
`IsPasswordErr` takes only 516 today.
- `SELECT version()` gives the version, which dbmeta already reads.
- The tests that started ClickHouse in a container are gone with W37. They
come back on `dbrun`.
Size, from W32: `clickhouse-go` adds 4.8 MB to a release build, with ch-go.
dbimp's driver would drop most of that, and the answer depends on what it
imports.
Open for Ken: whether to move at all, and whether to keep the native
protocol. If it moves, dburl moves its scheme first, and usql follows in one
commit, as with Avatica (W39).
## Tier 2: GitHub issues and pull requests
W2 to W15 are the programme, and W4 holds the issue triage. This section
is the remaining working list, reviewed on 2026-09-21 and updated on
2026-09-23, when 93 issues and 24 pull requests were open.
### Add the official Oracle driver
Add `github.com/oracle/go-oracledb`, Oracle's own driver for Go, as an
additional Oracle driver. This does not replace `github.com/sijms/go-ora/v3`,
which stays as it is.
### Released defects
Both are resolved and both issues are closed. 590 broke `go install` on 0.21.5
and was fixed by replacing the `exclude` directive with an ordinary pin of
`github.com/uber-go/tally v3.5.10+incompatible`, since `go install
module@version` refuses any `exclude` or `replace`. 589 reported the Dameng
driver failing; the driver has since been removed, so it was closed as not
planned.
### Make the completion test comprehensive
`completion_test.go` drives a real tab keypress through a pseudo-terminal and
asserts the table name is offered. It covers duckdb alone. Widen it.
Completion is per database. It reads names from dbmeta's model of each one,
so a pass for one says nothing about the others.
The duckdb bug this test was written for proves the point: the driver was
registered with MySQL's completer, so every completion ran a MySQL function
against duckdb, and it reached a release because nothing tested completion for
any driver at all.
What a comprehensive version needs. One case per driver that has a metadata
reader, sharing a single recorded session shape rather than one test each. The
containerised drivers need the same fixtures `drivers_test.go` already starts,
so it probably belongs beside that rather than in the root package, and it must
not start a second set of containers. The file-backed drivers, sqlite3,
moderncsqlite, duckdb and csvq, need no container and are the cheap half.
Two things to assert, because they fail differently. That the expected name is
offered, and that the completer logged no metadata error: it swallows a failed
query and carries on with no candidates, so a broken completer and a database
with nothing to complete look identical on screen.
It is slow. Recording a session takes seconds and the duckdb build needs cgo,
so this wants its own CI job rather than a place in the matrix.
### Pull requests to merge
These are one-file fixes from repeat contributors, most of them for NULL scan
errors in driver metadata: 524 and 570 for Oracle, 526 and 583 for catalog and
function metadata, 572 for Snowflake `$$` strings, 573 for Impala `SET`, 577
for PROMPT2, 578 for prompt documentation, which closes issue 374, 521 for a
manual page, 538 for config file locations, and 479 for stdin support.
### Pull requests to review carefully
591 fixes the backtick lexer bug in issue 587, but only one of the three
failures that issue reports, and the branch is rebased onto main locally as
`pr-591`. 565 rewrites metacmd tokenization for psql compatibility in 431 lines
of one file, and its description reads as a specification handed to a model
rather than a report of work done, so read the diff before trusting it. 501
adds bulk load for MySQL and 542 adds connection variables to `\copy`.
### Pull requests to close
418 and 535 both replace the readline layer and conflict with W8. 571, 582
and 584 duplicate dependency and action work that is already done. 585 adds a
Dameng driver, which usql removed on 2026-09-23. 360 belongs in Discussions.
### Issues to close rather than fix
Done on 2026-09-23. The nine scanner reports 543, 556, 567, 568, 574, 575, 579,
580 and 581 are closed as not planned, on the govulncheck evidence in item 4a.
470 and 475 are closed as fixed, each reproduced against main first. 589 and
590 were closed earlier, and 476 and 539 with the NULL fix.
Still open and waiting on something. W8 subsumes the terminal cluster, so
122, 137, 236, 483, 490, 528 and 552 close when the rline switch lands, not
before. 263, 469 and 485 need Discussions enabled; see item 4i.
### The theme being underweighted
NULL scan failures in driver metadata. Issues 476 and 539, and pull requests
524, 526, 570 and 583, are all the same shape: a `\d` or catalog query panics
or errors on a NULL column, across Oracle, SQL Server and MySQL. Four of the
open pull requests already fix parts of it.
## Tier 3: drivers that could be added
Verified on 2026-09-21. Every module below resolves on the proxy and calls
`sql.Register` in non-test code. Gemini and DeepSeek proposed others that do
not, and those were discarded.
| Database | Module | Latest | Released |
| ------------------- | -------------------------------------------------- | --------------------- | ---------- |
| libSQL and Turso | `github.com/tursodatabase/libsql-client-go/libsql` | v0.0.0-20260528064733 | 2026-05-28 |
| Dolt | `github.com/dolthub/driver` | v1.88.1 | 2026-05-07 |
| MonetDB | `github.com/MonetDB/MonetDB-Go/v2` | v2.0.3 | 2025-08-24 |
| rqlite | `github.com/rqlite/gorqlite/stdlib` | v0.0.0-20260504155303 | 2026-05-04 |
| openGauss and MogDB | `gitee.com/opengauss/openGauss-connector-go-pq` | v1.0.8 | 2025-08-20 |
IBM Db2 is out of scope. Ken decided on 2026-09-29 that it cannot be tested
here, so usql removed its Db2 client setup from `contrib/` and does not add a
Db2 driver. dbmeta records the decision as its D130.
TDengine is dropped as well. dbimp writes no TDengine driver (its D127), and
dburl v0.38.0 removed the provisional scheme (its D41).
`github.com/ncruces/go-sqlite3/driver` v0.35.5 also qualifies, but usql already
links two SQLite drivers, so it is the weakest of these.
### Replace the retired Arrow module
`github.com/apache/arrow/go/v17` is frozen at v17.0.0 from 2024-07-11 and is
the only version that line ever published. The flightsql driver moved to
`github.com/apache/arrow-go/v18`, now v18.8.0 from 2026-09-04, which is already
an indirect dependency, so usql carries two Arrow trees today.
### Drivers whose upstream is dead
None of these has a maintained replacement that implements `database/sql`, so
removing one means dropping that database.
| Driver | Module | Latest | Released |
| ---------- | --------------------------------------------------- | --------------- | ---------- |
| mymysql | `github.com/ziutek/mymysql` | v1.5.4 | 2015-01-09 |
| adodb | `github.com/mattn/go-adodb` | v0.0.1 | 2018-05-15 |
| sapase | `github.com/thda/tds` | v0.1.7 | 2019-09-27 |
| ignite | `github.com/amsokol/ignite-go-client` | v0.12.2 | 2019-01-04 |
| h2 | `github.com/jmrobles/h2go` | v0.5.0 | 2020-11-14 |
| cassandra | `github.com/MichaelS11/go-cql-driver` | v0.1.1 | 2020-09-20 |
| maxcompute | `sqlflow.org/gomaxcompute` | v0.0.0-20210805 | 2021-08-05 |
| couchbase | `github.com/couchbase/go_n1ql` | v0.0.0-20220303 | 2022-03-03 |
| ots | `github.com/aliyun/aliyun-tablestore-go-sql-driver` | v0.0.0-20220418 | 2022-04-18 |
mymysql is the clearest drop, because `go-sql-driver/mysql` covers the same
database and shipped v1.10.1 on 2026-09-02. adodb is next, because `internal`
already aliases odbc to adodb only on Windows and odbc covers it.
### The Vertica container is gone
Resolved on 2026-09-27. usql's `contrib/vertica/podman-config` pointed at
`docker.io/vertica/vertica-ce`, which no longer exists. That file is gone along
with the rest of the container configuration. dbmeta now models Vertica and
starts releases 7.2, 9.1, 10.1 and 25.1 through `dbrun`. dbmeta's D88 records
how.
### vertica-sql-go splits a statement at every semicolon
`github.com/vertica/vertica-sql-go` splits each statement at its semicolons
before it sends it. It does not know that the `BEGIN ... END` body of a SQL
function contains them. `CREATE FUNCTION ... BEGIN RETURN ...; END` therefore
cannot be sent through usql at all, and the semicolon cannot be left out. A
PL/vSQL procedure works, because its body is dollar quoted and survives the
split. dbmeta found this while building its Vertica model, and its D88 has the
details. Report it upstream.
## Lowest priority
### Confirm that the non-Windows, non-macOS, non-Linux builds now work
Charts are behind the `charts` build tag as of this change, so resvg is linked
only by `go build -tags charts` and `go build -tags all`. A default build and a
`most` build no longer reference it at all.
`github.com/xo/resvg` ships prebuilt static Rust artifacts instead of building
from source. Version 0.8.0 contains six copies of `libresvg.a`, for
`darwin_amd64`, `darwin_arm64`, `linux_amd64`, `linux_arm`, `linux_arm64` and
`windows_amd64`. The cgo preamble in `resvg.go` names those same six pairs in
its `#cgo , LDFLAGS` lines. The file carries no build constraint,
so no other platform can compile the package. On any other platform the C code
compiles, because `CFLAGS` still points at `libresvg/resvg.h`, but no `LDFLAGS`
line matches. The linker therefore receives no `-L` and no `-lresvg`, and every
`resvg_` symbol is undefined. Issue 494 is this failure on the FreeBSD port for
arm64, armv7 and i386.
That failure should now be reachable only with the `charts` tag, so a default
build and a `most` build should work on those platforms. Build usql on the
freebsd-rline, netbsd-rline and omnios-rline machines to confirm it. Note that
resvg may not be the only package those platforms cannot build, so a failure
there is not by itself a sign that this gating is wrong.
Gating does not close issue 494. The xo/resvg session established that a
per-platform module split does not close it either, because the `#cgo CFLAGS`
line in `resvg.go` carries no build constraint: the C still compiles on an
unsupported platform and the link still fails on undefined `resvg_*` symbols,
from a different set of files. Closing 494 needs an explicit build-tag-gated
stub in xo/resvg that fails to build with a message naming the platform, or
that compiles to a renderer which reports that it is unavailable. That work is
recorded in xo/resvg's `libresvg/README.md`.
Fixing charts on those platforms is a separate job and costs more. resvg can
learn to link a system `libresvg` when the platform provides one, which is what
the FreeBSD port needs. The Rust crate can also be cross-built for more targets
and the artifacts vendored, which grows the module and needs a Rust toolchain
for each platform.
## Not yet scheduled
Apply the `update-deps.sh` report fix to `release-21`. The `REMAINING:` report
on that branch uses the module list from before the update. `go list -m -u`
then fails on a module that `go mod tidy` removed, and the report claims that
nothing is outdated. The branch also still carries an obsolete
`github.com/opencontainers/runc` entry in `SKIP`. The fix is in a stash on that
branch named `release-21: update-deps.sh REMAINING report fix`.
## W43. Take the Drill, Solr, Elasticsearch, OpenSearch and DynamoDB drivers: Done, with one gap
Done on 2026-10-07 on dburl v0.46.0 and dbimp v0.14.0. The five drivers are
dbimp's, pure Go over HTTP, and all are in the group `most`. DynamoDB replaces
`btnguyen2k/godynamo`, so the scheme `godynamo` goes, the aliases are `dy`,
`dyn` and `dynamo`, and the URL is
`dynamodb://key:secret@host:port?region=us-east-1`. `go.mod` loses godynamo and
the AWS modules that only it needed. Each driver reports no count of rows
affected through `drivers.RowsAffectedOrZero`, and none has a metadata model
in dbmeta, so the describe commands report that they are not supported (D3).
Checked through usql on drill-1.22.0, solr-10.0.0, elasticsearch-9.5.3,
opensearch-3.9.0, dynamodb-3.3.1 and alternator-2026.3 from dbrun: a query,
NULLs, an error with its name, and a wrong password (HTTP 401 for Solr,
Elasticsearch and OpenSearch, the login page for Drill, and
`UnrecognizedClientException` for Alternator, which is the one flavor that
checks the signature). Drill also ran `CREATE TABLE AS` and `DROP TABLE`.
Solr needs the collection in the URL path.
The gap, closed on 2026-10-08: Drill, Solr and Elasticsearch refuse a
statement that ends in `;`, and dbmeta v0.4.0 holds a model for each, with
`TerminatorStripped`. usql took v0.4.0, dropped their `noModel` entries and
their Version hooks, and checked a piped `select 1 as a;` and `\dt` on all
three. Drill's banner reads `Apache Drill 1.22.0` from dbmeta, and Solr and
Elasticsearch print only their driver name, because dbmeta reads no version
for them yet. OpenSearch takes `;` already and has no model, and DynamoDB has
no model either, so both keep `noModel` entries.
Not checked: Drill 1.21.2, the other Solr, Elasticsearch, OpenSearch and
DynamoDB releases, and a write to DynamoDB, because DynamoDB Local refuses
`CREATE TABLE` as PartiQL.
`TestRegistryIsKeyedByScheme` fails in the default build with "found 16
registered drivers, expected at least 20". It failed before W43, and it passes
with `-tags most`.
## W44. Take the Druid driver: Done
Done on 2026-10-08 on dburl v0.46.0 and dbimp v0.14.0. The driver is dbimp's,
pure Go over HTTP, read only, in the group `most`. dbmeta has a Druid model, so
the version and the describe commands come from it and usql adds no reader
(D3), and a trailing `;` is stripped by the model's Terminator.
Checked through usql on druid-38.0.0 from dbrun: the banner (`Apache Druid
38.0.0`), a query with NULLs, an error with its code, the refusal of an
`INSERT`, a wrong password (HTTP 401), the ordinary user, and `\dtS`. Not
checked: druid-37.0.0, and `\d` of a table with data, because the server held
none.
## W45. Move Cassandra from xo/cql to xo/cassandra: Done
Waits on dburl and xo/cassandra. Ken said on 2026-10-08 that the Cassandra
driver moves from `github.com/xo/cql` to `github.com/xo/cassandra`, and drops
the CQL name, so that it is consistent with the other drivers. Checked the
same day: the checkout of `xo/cassandra` still has the module path
`github.com/xo/cql` and the tag v0.1.0, and dburl v0.46.0 still names the
scheme `cql`, with the aliases `ca`, `cassandra`, `datastax`, `scy` and
`scylla`. Nothing has been released under the new name.
What usql changes when dburl and the driver release it:
- `go.mod` takes the new module, and `drivers/cassandra` imports it, in
place of `github.com/xo/cql`, and registers under the scheme name that
dburl then gives, which is `cassandra` in place of `cql`. The `See:` and
`// DRIVER` lines follow, and `gen.go` rewrites the README tables, the
`internal` build tags and the link bar.
- `CQL_DEBUG` and its log line are renamed. Check whether the old name stays
as a spelling that is still read, which Ken decides.
- The scheme `cql` goes from dburl, or stays as an alias. That is dburl's
and Ken's decision, and usql adds no alias of its own (rule 12). The
README example `usql cql://host/` and the tag `cql` change with it.
- dbmeta names its dialect `cql`, and `drivers.Language` and the version and
metadata reads go through it. Ask dbmeta whether the dialect keeps its
name, because the language is still CQL.
- `docs/RELEASE.md` names `cql` in the password-change list and the table
of dialects, and says "Cassandra moves to `github.com/xo/cql`". Fix all
three. The batches of CQL stay, because that is the name of the language.
- Check on a server from dbrun: the banner, a query, `BEGIN BATCH` to
`APPLY BATCH`, a wrong password, `\dt` and `\copy`.
Update, 2026-10-08: the cassandra project says the module is now
`github.com/xo/cassandra` and the driver registers as `cassandra`, but it has
no tag, and its repository holds the change uncommitted. dburl has no release
that names it either, and `gen.go` reads dburl's registry from `go.mod`, so
usql cannot move with a `replace` alone: the README tables, the build tags and
`internal/internal.go` would still say `cql`. usql waits for both tags. The
peer's note about `LexerName` does not apply, because the chroma lexer is gone
(W36).
What the cassandra project says changes in the driver, 2026-10-08. `ParseDSN`
and `NewConnector` keep their signatures. `ParseDSN` reads only a
`cassandra://` URL, with one host in the host part and more hosts as repeated
`host` keys, so it must get the URL that dburl writes after dburl moves, and
usql's `Open` hook passes the DSN that usql builds from that URL. Value types
change: decimal is `*apd.Decimal`, varint `*big.Int`, date `dbimp.Date`, time
`dbimp.LocalTime`, duration `dbimp.Interval`, inet `netip.Addr`, list, set and
tuple `[]any`, a map with a text key and a UDT `map[string]any`, and a vector
`dbimp.Vector[T]`. `BeginTx` and named arguments fail with an error that wraps
`dbimp.ErrNotSupported`. The option types become functions, such as
`WithConsistency` and `WithPageSize`.
usql has no code for the types or the options, because `drivers/cassandra`
only opens the connection and reads errors. Check these on a server when it
moves: `\copy` into and out of Cassandra, because the shared copy code starts
a transaction and `BeginTx` now fails (it needs `FlexibleCopyWithInsert` with
no transaction, as for rqlite), a `BEGIN`, and the printing of each value type
above, which `tblfmt` formats.
Update, 2026-10-08, later: usql took dburl v0.47.0 and `github.com/xo/cassandra`
v0.1.0, which pulled dbimp to v0.15.0. `drivers/cassandra` imports the new
module and registers `cassandra`. The debug variable `CQL_DEBUG` is gone with the
logger it switched on (W51). The README and
`docs/RELEASE.md` follow. Open: dburl's dialect for Cassandra is now
`cassandra`, and dbmeta v0.4.0 and its main branch still name it `cql`, so
dbmeta has no model for the dialect and `TestEveryDriverReadsThroughDbmeta`
fails. dbmeta has been asked to rename it. Do not commit this move before that
release. The `cql` rows of the password and statement-form lists in
`docs/RELEASE.md` are dbmeta's dialect names, and change with it. Not yet
checked on a server: the banner, a query, `BEGIN BATCH`, a wrong password,
`\dt`, `\copy` (the `BeginTx` problem above) and the new value types.
## W46. Bring the describe output back to psql and to v0.21.6, PostgreSQL first
Ken asked on 2026-10-08 for the describe commands to give at least the
information of the last release (v0.21.6), with the columns and names of psql,
and the same column names where it is the same column. Extra columns and extra
commands are fine. psql's style and names win over the old usql's. The
databases follow PostgreSQL one at a time, with v0.21.6 as the baseline where
there is no psql.
The harness is `testdata/describe`. `fixture.sql` builds a database with every
kind of PostgreSQL object, `commands.txt` lists 179 commands in their forms, and
`run.sh` runs them on psql, the old usql and the new one, as csv.
`compare.py` lists the columns that psql has and usql lacks, `values.py` lists
the cells that differ for columns both have, and `sections.py` compares the
sections of `\d+ NAME`. It starts no database, so it is not a test (rule 6), and
W37 will turn its results into goldens.
First run, 2026-10-08, on postgres-18 with dbmeta v0.4.0:
- Three commands fail with a NULL scan error in dbmeta: `\d` of a table with
an INCLUDE or expression index (`descending`), `\ddp` (`schema`) and `\drds`
(`database`).
- The relation lists lack Owner, Persistence, Access method and Size (and
v0.21.6's Rows). `\di` lacks Owner, Persistence and Size, and v0.21.6's
Primary? and Unique?. `\d+` lacks Collation, Storage, Compression and Stats
target for columns, and almost every footer section of psql: partitions,
inheritance, policies, publications, not-null constraints, rules, statistics
objects, view definitions, options, access method, and the server of a
foreign table.
- Other lists differ by one or two columns, such as `\dn+`, `\dx`, `\l`, `\dRp`,
`\dRs+`, `\dFp+`, `\do+`, `\dC+`, `\df+`, and `\dD`.
- Commands that new usql does not know: `\da+`, `\dPt`, `\dPi`, `\dPn`.
- v0.21.6 had far fewer sections than psql. Only Indexes, Check constraints,
Foreign-key constraints, Referenced by and Triggers match.
The report went to dbmeta on 2026-10-08, which owns the fields. usql's
formatter owns the layout, the names and the order.
Done on 2026-10-08. dbmeta v0.7.0 names the dialect `cassandra`, and usql took it,
so `TestEveryDriverReadsThroughDbmeta` passes again. Checked through usql on
cassandra-5.0 from dbrun: the banner (`Cassandra 5.0.9, CQL 3.4.7, Protocol
v5`), a query with NULLs, and the types decimal, varint, date, time, inet,
list, map and duration, `BEGIN BATCH` with a `$$x;y$$` string, a wrong
password, and `\dt` and `\d`. `\copy` says that the driver does not support it,
as it never did, because the driver has no Copy hook. Open: dbmeta's Cassandra
model ignores the schema and name filters, and the formatter does not narrow
the result, so `\dt ks.*` also lists the system keyspaces. dbmeta has been
asked for one shared helper. Not checked: scylla.
Progress, 2026-10-08, on dbmeta v0.8.0 (D197 and D198). The three scan
failures are gone. The formatter now prints the Owner, Persistence, Access
method, Size and Rows of relations, the Owner, Persistence and Size of
indexes, the Collation, Storage, Compression and Stats target of columns, the
identity and generated defaults, the partial-index WHERE and the index flags
(INVALID, CLUSTER, DEFERRABLE, INITIALLY DEFERRED, REPLICA IDENTITY), the
Leakproof? and Source code of functions, psql's titles for each class, the
`Unlogged` prefix of a title, `\da+` and `\dP[itn]`. A column that no row has a
value for is left out, so databases that give none keep their output. Left, and
asked of dbmeta: the sections of `\d+ NAME` (D199), the fields and text forms
listed in the report of the same day (`\dP` sizes, FDW options, `\dT+`
elements, privileges and policies, `\dn+`, `\dx`, `\l`, `\dRp`, `\dRs+`,
`\dFp+`, `\dAo+`, sequence cache), and the values of `\dD`, `\dFd+` and the
`\dA` family. After that, the other databases follow, with v0.21.6 as the
baseline.
Progress, 2026-10-09, on dbmeta v0.9.0 (D199, D200 and D201). The formatter now
also prints the sections of `\d+ NAME`: the partition key, the partitions and
their bounds, `Partition of`, the inheritance of a table, policies, rules,
statistics objects, publications, not-null constraints, view definitions,
options, access method, and the server and options of a foreign table. The
lines of an index come from the definition of the database, with `UNIQUE
CONSTRAINT` and `EXCLUDE`. Names of related tables are written without a schema
where the database says they resolve without one. `\dn+`, `\dx`, `\l`, `\dRp`,
`\dRs+`, `\do+`, `\dAo+`, `\dP`, the FDW options, `\dT+` elements, and the
column privileges and policies of `\dp` follow psql. The harness shows these
differences left: `\d` of an index lacks the Storage and Stats target of its
columns and the type of an expression, `\dFp+` prints no per-parser table,
`\dC+` gives no Leakproof? (the cast has none), the column options of a
foreign table are empty, and `\dRs+` cannot be compared on a database with no
subscriptions.
## W47. Remove the godror driver: Done
Done on 2026-10-09 at Ken's request: godror is not needed, because the oracle
driver is pure Go. The driver, its generated file and its modules are gone, and
the README and the samples use `oracle`. No driver is in the group `all` now.
dburl v0.48.0 dropped the scheme `godror` and its alias `gr`, and usql took it,
as AGENTS.md asks.
Also stale in `_samples/usqlpass`: the lines for `cql`, `ignite` and `mymysql`,
schemes that went in earlier changes.
## W48. Fold orshared into the oracle driver: Done
Done on 2026-10-09 at Ken's request. `drivers/oracle/orshared` held the
registration that the oracle and godror drivers shared. godror is gone, so the
registration is in `drivers/oracle/oracle.go` and the package is removed.
## W49. Move Snowflake to dbimp's driver: done, not run
Waits on dburl and on Ken. dbimp v0.16.0 (2026-10-10) has
`github.com/xo/dbimp/snowflake`, a driver over the SQL REST API with no Arrow,
which could replace `gosnowflake` in `drivers/snowflake`. dbimp asks usql to
wait until dburl writes the key as the password of the URL, and until Ken
decides. What it says the move changes: `RowsAffected` wraps
`dbimp.ErrNotSupported` when the server gives no count, which
`drivers.RowsAffectedOrZero` already handles. The version statement is
`SELECT CURRENT_VERSION()`, and dbmeta's Snowflake model gives it already.
`BeginTx` fails with `dbimp.ErrNotSupported`, so `\copy` must copy without a
transaction, as ClickHouse does. dbimp verified it on a trial account, and its CI
replays recordings only. usql cannot test it, because the Snowflake credentials
are dbmeta's. Nothing is done.
Done on 2026-10-10, on dburl v0.49.0 and dbimp v0.16.0, which dburl needed
before usql could move: both drivers register `snowflake`, so usql links one,
and dburl now writes the URL for the dbimp driver. `drivers/snowflake` imports
`github.com/xo/dbimp/snowflake`, reads its error code and HTTP 401 for a wrong
key, counts no row for a statement with no count, and copies without a
transaction. `gosnowflake` and the modules that only it needed are out of
`go.mod`. It builds and the tests pass, but nothing has run it against
Snowflake, because the credentials are dbmeta's. Ken decides whether dbmeta
tests it, or runs it himself.
## W50. Move BigQuery, Cosmos DB, Athena, Databricks and Spanner to dbimp's drivers: Done, not run
Waits on dbimp and dburl. Ken decided on 2026-10-10 (told by dbmeta) that dbimp
writes drivers for BigQuery, Cosmos DB and Athena, and dburl renames the
schemes to name them. usql will then import them in place of
`gorm.io/driver/bigquery`, `gocosmos` and `athenadriver`. dbmeta has no model
for the three (its D194), so usql keeps their `noModel` entries.
What usql holds for them, and told dbimp on 2026-10-10: BigQuery has no hook.
Cosmos DB has none, and stripped a trailing semicolon until the `Process` hooks
went (W28). Athena has a Version hook, `SELECT node_version FROM
system.runtime.nodes LIMIT 1`, and stripped the semicolon too. When they
move, the Athena hook goes if dbmeta or the driver gives the version, and the
rest follows the move of ClickHouse (W42): the error type, the wrong password,
`drivers.RowsAffectedOrZero`, and `\copy` without a transaction if `BeginTx`
fails. usql has no account for any of the three, so the checks are by hand or
through dbmeta. Nothing is done.
Update, 2026-10-11. Ken said that all five are dbimp implementations now, and
dburl moved the five schemes (athena, bigquery, cosmos, databricks and spanner).
usql wrote the five drivers against the local checkouts of dburl, dbimp and
dbmeta through `go.work`, which is not committed: each reads the error type of
its package, takes a wrong credential from HTTP 401 (BigQuery, Cosmos DB and
Spanner), from the signature error (Athena) and from the message "Invalid
access token" (Databricks, which answers 403), and counts no row for a
statement with no count. Athena, BigQuery and Databricks copy without a
transaction, because `BeginTx` fails. Cosmos DB reads only, and Spanner has
transactions. The Athena Version hook is gone, because dbmeta has a model.
`go.mod` still names released versions, so the commit waits for the tags of
dburl and dbimp, and for dbmeta to rename its Athena dialect from `awsathena`
to `athena`, which `TestEveryDriverReadsThroughDbmeta` needs. Not checked on any
server: usql has no account for any of the five.
Update, 2026-10-11, from dbimp v0.17.0 (it holds the five packages, and dburl
has not tagged the schemes yet):
- Version: BigQuery, Databricks, Athena and Spanner answer no `SELECT
version()`, so the usql drivers carry no hook, and the version is what
dbmeta's model gives.
- Cosmos DB: the new driver reads only, and every `Exec` fails with
`dbimp.ErrNotSupported`. A user who writes documents loses that, because
`gocosmos` could write. Ken decides whether that stays, since usql has no
other way to write to Cosmos DB.
- Credentials: BigQuery and Spanner take the PATH of a service account key file
in the query (`credential_file`), never the key text. Databricks takes the
token as the password, Athena takes the key and the secret as the user and the
password, and Cosmos DB takes the master key as the password. The README and
the samples name none of the five today.
- Statements: Spanner and Athena refuse several statements in one request, and
BigQuery returns the last result. Cosmos DB strips one trailing semicolon,
and Athena and Databricks accept one.
- Types to check by hand when a server is there: what usql prints for a
DECIMAL (`*apd.Decimal`), a UUID, an interval (`dbimp.Interval`) and JSON.
Cosmos DB columns are the keys of the first row, and Databricks returns an
error for a cut or chunked result.
Done on 2026-10-11 on dburl v0.50.0, dbimp v0.17.0 and dbmeta v0.13.0. The five
drivers are in, `go.mod` names the released versions, and `gocosmos`,
`databricks-sql-go`, `go-sql-spanner`, `athenadriver`, `gorm.io/driver/bigquery`
and the modules that only they needed are out. `go.mod` is fully
tidy, since rline v0.1.0 is tagged. Built and tested, and checked
by no server.
## W51. Keep a batch of CQL as one statement, through the grammar: Done
Done on 2026-10-11 on transit v0.4.0. The grammar has the option `Batches`,
which keeps a batch from `BEGIN BATCH`, `BEGIN UNLOGGED BATCH` or `BEGIN COUNTER
BATCH` to `APPLY BATCH` as one statement. `drivers.Language` turns it on for a
dialect that dbmeta gives batches, so no code names Cassandra. The handler no
longer collects the statements of a batch itself: its fields, the check
`drivers.IsBatchQueryPrefix` and the `\g` rule for a batch are gone, and the
interactive prompt no longer changes inside a batch. The Cassandra driver is
rewritten to the two things usql still adds: a timeout of five minutes for a
request when the URL has no `timeout` key, which Ken kept on 2026-10-11 after
trying without it, and the check for a wrong password, because the driver has
no type for it. Its errors, row counts and value types come from
`github.com/xo/cassandra`, and the debug logger, the error regexps, the row
count and the JSON conversion are gone. Checked on cassandra-5.0: a batch with a
`$$x;y$$` string, an unlogged batch on one line, a `decimal`, a list, a map, a
`varint`, an `inet` and a `duration`, and the errors of a bad statement and a
bad password. The tag printed for a batch is `BEGIN BATCH` where it was `APPLY
BATCH`, and an unfinished batch sent with `\g` fails at the server.
## W52. Scrub the drivers of dbimp down to what only usql knows: Done
Done on 2026-10-11 at Ken's request, after the Cassandra rewrite (W51). A dbimp
driver reports its own errors and counts, so usql's generic code does what
the hooks did:
- `drivers.IsPasswordErr` takes an HTTP 401 as a wrong credential for every
driver, so a hook for 401 alone is gone.
- `drivers.RowsAffected` counts zero for an error that wraps
`dbimp.ErrNotSupported`, so `drivers.RowsAffectedOrZero` and the field
`Driver.RowsAffected` are gone.
- `\copy` takes each insert alone when `BeginTx` fails with
`dbimp.ErrNotSupported`, so ClickHouse, rqlite, Snowflake and the others need no
special `Copy`: it is `drivers.CopyWithInsert(drivers.QuestionMark)`. Trino and
Presto begin a transaction and then refuse the write, so they keep
`drivers.CopyWithInsertNoTransaction`.
- The `Err` hooks of every dbimp driver are gone. Each error prints as its own
`Error()` says, such as `clickhouse: UNKNOWN_TABLE (60): ...` and `druid:
INVALID_INPUT: ...`, which names the code of the server and sometimes the HTTP
status.
What is left in the 26 drivers is what usql cannot take from the driver: the
registration, the check for a wrong password where it is not a 401 (Athena,
Avatica, ClickHouse, Databricks, Drill, DynamoDB and Neo4j), `AlwaysQuery` for
ArangoDB, InfluxDB, Neo4j and SurrealDB, the Version of DynamoDB, InfluxDB and
Pinot, the `Copy` placeholder, and `ForceParams` for Couchbase. Six drivers
(Cosmos DB, Druid, Elasticsearch, OpenSearch, Solr and Spanner) are a bare
registration. Those hooks are database facts, which belong in dbmeta (D3), or
in dbimp for the wrong password: dbimp has been asked for one way to say that an
error is an authentication failure. Checked on clickhouse-26.9 (an unknown table,
`\copy` of two rows, a count) and druid-38.0.0 (an error, a 401). Not checked on
the other servers.
Update, 2026-10-11, on dbimp's `ErrAuthentication` (its D197, after v0.17.1):
every dbimp error matches `errors.Is(err, dbimp.ErrAuthentication)` when the server
refused the credential and not when it refused a permission, so
`drivers.IsPasswordErr` asks that one question and the wrong-password hooks of
Athena, Avatica, ClickHouse, Databricks, Drill, DynamoDB and Neo4j are gone.
`drivers/avatica/avatica_test.go` now checks the classification through
`drivers.IsPasswordErr`. usql builds against the local dbimp checkout through a
`go.work` that is not committed. dbimp tagged it in v0.18.0 on 2026-10-11, and
`go.mod` names that release.
Next reduction, for Ken to decide: eight drivers (Avatica, Cosmos DB, Drill,
Druid, Elasticsearch, OpenSearch, Solr and Spanner) are now a registration and
nothing else. `drivers.Available` could list every dburl scheme whose driver
database/sql has registered, and `gen.go` could write the blank import of the
dbimp package into `internal/` for a scheme with no hooks, so that no
`drivers/` package exists for it.
## W53. Move the configuration facts of the drivers into dbmeta: Done
Waits on dbmeta. Ken asked on 2026-10-11 that the common facts that drivers set
by hand come from dbmeta's `Info`, so that usql applies them for every driver and
no code names a database (D3). usql told dbmeta the inventory the same day:
- The placeholder of `\copy`'s `INSERT` (`?`, `:1`, `@p1` or `$1`) from
`Info.Placeholder`, and a flag for Trino and Presto, which refuse a write inside
a transaction. libpq and pgx keep their native COPY.
- `AlwaysQuery` for ArangoDB, InfluxDB, InfluxQL, Neo4j and SurrealDB.
- A static version for a product with no statement for it: DynamoDB and Pinot.
- Defaults of the DSN when the key is absent (Cassandra `timeout=300s`,
Couchbase `txtimeout=30m`, Hive `auth=NONE`) and forced values (the MySQL family
`parseTime`, `loc` and `sql_mode`, and SQLite `loc=auto`). Oracle's `ORACLE_SID`
stays, because it reads usql's environment.
- `UseColumnTypes` for the MySQL family and Databend.
When dbmeta releases them, the hooks go and `drivers.Driver` keeps only what is Go
code: `Open`, `ColumnTypes`, the converters, native copies and the check for a
wrong password that is not a dbimp error. Then the drivers with no code left have no
package (see W52).
## W54. Share the native COPY of lib/pq and pgx: Done
Done on 2026-10-11 at Ken's request. lib/pq has deprecated `pq.CopyIn` and
`pq.CopyInSchema`, which the `libpq` driver used, and the `pgx` driver
repeated the code that finds the columns of the target table. Both now use
`drivers.CopyTarget`, which reads the schema, the name and the columns of the
target, and `drivers.CopyRows`, which scans the source rows for a copy. The
`libpq` driver writes `COPY "schema"."table" (columns) FROM STDIN` itself, with
`pq.QuoteIdentifier`, and reads the bytes that lib/pq returns for a numeric as
text, which it did not before (a numeric failed with `invalid input syntax`).
`pgx` closes its connection and its statements, which it did not. A name may be
quoted (`"Mixed Case"` and `public."Odd.Table"`), and a name with no quotes keeps
the case of its letters. Both drivers dropped the path for a copy whose target is
an `INSERT` statement, which failed in lib/pq and was never in pgx. Checked on
postgres-18 through `postgres://` (pgx) and `pq://`: 2,501 rows with NULLs,
numeric, timestamptz, bytea and jsonb, a schema-qualified table, a list of
columns and a quoted name. CockroachDB, Redshift, CrateDB and QuestDB share the
pgx copy and are not checked.
Update, 2026-10-11. Ken decided that the copy placeholder (with the flag for
Trino and Presto), the rule that every statement is a query, the static product
string and the scan by the driver's type live in dbmeta, and that the DSN
defaults live in dburl, which released them in v0.51.0. usql read them:
`drivers.Copy` writes the INSERT of a database that has a placeholder, and takes
each row alone for one that needs autocommit; `drivers.Process` runs each
statement of such a database as a query; `drivers.Version` prints the product
string; `drivers.UseColumnTypes` asks the dialect. The `Copy`, `AlwaysQuery`,
`UseColumnTypes` and `ForceParams` hooks, the static `Version` hooks of DynamoDB
and Pinot, and the helpers they used are gone, except Oracle's `ORACLE_SID`, the
native copies of libpq and pgx, and InfluxDB's and csvq's version. A MySQL or
SQLite URL that writes `loc=UTC` now keeps it, because dburl sets `loc` only when
the URL has none, where usql forced it. A driver with a placeholder in dbmeta and
no `Copy` hook now tries the INSERT, where it said that copy is not supported.
dbmeta tagged the facts in v0.14.0 on 2026-10-11, and `go.mod` names it, so the
build needs no `go.work`.
## W55. Run usql against the hosted services: Done
Done on 2026-10-11 at Ken's request, with the accounts that dbsetup made for usql
(a user or a scratch schema of its own on each service, files in
`~/.config/dbmeta/usql`, read into the process only). Each service got the banner, a
query with NULLs and the common types, an error, a wrong credential where it can
be built, `\dt` and `\d`, a table of `usql_*` with rows with NULLs, `\copy` of
them into a second table, and a trailing semicolon. The tables are dropped.
- Snowflake, BigQuery, Spanner, Athena (an Iceberg table), Databricks and
Redshift Serverless worked, with the exceptions below. Cosmos DB reads only: a
query on the partition key and `\dt` worked, `\d` says that it is not supported
(the model has no columns), a write is refused as designed (D190), and a wrong
key is `401 Unauthorized`.
- Fixed in usql: the generic `\copy` failed on any row with a NULL for a driver
that reports a string, number or time as the scan type (it scanned into the type
and not into a type that takes a NULL), so it now uses the same null-safe
destination as the result table. Redshift has no COPY FROM STDIN, so it copies
by INSERT, where it used the native copy of pgx and failed. Redshift answers a
wrong password with `28000` and not `28P01`, so the pgx driver takes the class
28 as a wrong password, which it did not, and Redshift never asked for it again.
- Open, and told to the owner: dbmeta's Redshift model reads `SVV_TABLE_INFO`,
which an ordinary user may not, so `\dt`, `\d` and `\dv` fail for a user who is
not a superuser. dbimp types a NULL parameter as STRING, so a copied NULL into a
BigQuery column that is not a string is refused. The wrong credential could not
be tried on Snowflake, because a key that is not valid PKCS8 fails before it
reaches the server.
- Not checked: CrateDB, QuestDB and CockroachDB share the native copy of pgx,
which may fail as Redshift's did.
## W56. Move Exasol to dbimp's driver: Done
Waits on dburl and the tags of dbimp. Ken said on 2026-10-11 that Exasol is moving
to a dbimp implementation. dbimp has `github.com/xo/dbimp/exasol` (its W44) in its
repository, over the WebSocket API with no cgo, and has not tagged it. dburl
still names `github.com/exasol/exasol-driver-go` and writes the DSN of that driver
(`host:port?user=...`), so the move needs dburl to name the new package and to
write the form that dbimp reads (`exasol://user:password@host:8563/schema?tls=...`).
usql wrote its side against the local checkout of dbimp (not committed, through a
`go.work`): `drivers/exasol` is a bare registration, because dbimp's errors, row
counts and refused credentials are generic here (W52), and the Exasol model of dbmeta
gives the metadata, the placeholder of `\copy` and the version. The old error
regexp is gone. `go.mod` still names `exasol-driver-go`, and
`TestDriverSeeMatchesDburl` fails until dburl moves, because it compares the `See:`
line with dburl's. Not run against a server: the new driver cannot read the DSN
that dburl writes today. Checks for after the tags: a query, a wrong password, `\dt`,
`\d` and `\copy` on `exasol-2026.2.0`, which dbrun starts.
Update, 2026-10-11. dburl has the change staged and not released (its D66): the scheme
names `github.com/xo/dbimp/exasol` and writes `exasol://user:password@host:port/schema?key=value`.
With the local checkouts of dbimp and dburl in `go.work`, the registry checks pass and
the README row follows dburl. Checked on `exasol-2026.2.0` from dbrun: the banner
(`Exasol 2026.2.0-nano.6`), types with NULLs, an error (`42000`), a wrong password
(`08004`), `\dt` and `\d`, `\copy` of rows with NULLs and with a list of columns, and a
trailing semicolon. A decimal prints without its trailing zero (`1.5` for
`decimal(10,2)`), where the other drivers keep the scale. `go.mod` waits for the tags
of dbimp and dburl.
Done on 2026-10-11 on dbimp v0.20.0 and dburl v0.52.0, both tagged. `go.mod` names
them, `exasol-driver-go` and the modules that only it needed are out, and the build
needs no `go.work`. dbimp v0.20.0 also writes a NULL argument of BigQuery as the literal
NULL, which fixes the copied NULL into a column that is not a string (W55).
Update, 2026-10-11, on dbmeta v0.15.0: Redshift's tables statement no longer reads
`SVV_TABLE_INFO`, so `\dt`, `\d` and `\dv` work for an ordinary user, and `Args.WithTableInfo`
asks for the size, the rows and the options. The verbose list (`\dt+` and its
forms) sets it only for a model that takes `with_table_info`, and runs again without
it when the read fails, so an ordinary user sees the other columns with no size and
rows. No code names Redshift. Checked on PostgreSQL (`\dt+` still has Size and
Rows). Not rerun on Redshift, because the account that dbsetup made for usql is
gone.
## W57. Sweep the documents and the code for what the refactoring left: Done
Done on 2026-10-11 at Ken's request. The sweep found and fixed:
- The README said that interactive queries are syntax highlighted with Chroma, and
listed the Chroma variables. Highlighting is not part of this release, so the
README, the feature list and the descriptions of the `SYNTAX_*` variables now say
that they do nothing yet.
- The completer logged an error to standard output on every failure to list
objects, through a logger that nothing could change. It stays silent now, and the
four options that nothing used are gone.
- Dead code: the `adodb` fallback of `internal/z.go`, `env.Untick`,
`env.GetPrint`, `indexCols.all` and the placeholder of the SQL Server driver.
- Package comments of the drivers named the drivers that they replaced, and said that
usql has not tested the hosted services, which it has (W55). Spanner said that
dbmeta has no model for it. The Spanner, Cosmos DB and Exasol comments are
current. Five package comments had lost the `See:` and `Group:` lines to a blank
line before `package`, which `TestDriverSeeMatchesDburl` caught.
- `docs/DRIVER.md` quoted the size of an all-driver binary (215 MB) and a scheme
that differed from its tag (DynamoDB) that no longer exist, and now says what a
dbimp driver needs. `docs/PLAN.md` lists the questions that are open for Ken,
where it said none. `docs/DBMETA.md` says that the migration is done and no longer
describes a `go.work`. `docs/PROGRESS.md` and `docs/RELEASE.md` follow the hosted
runs, the facts that moved to dbmeta and dburl, and the removed hooks. AGENTS.md
lists dbimp, transit, cassandra and odbc as siblings.
- The samples no longer name `cql`, `mymysql` or `ignite`.
What the sweep left alone, as history: BACKLOG.md and the decisions, and the
accounts of removed drivers in CONTRIBUTING.md.
## W58. Put the highlighting documentation back when highlighting returns: Done
Done on 2026-10-14 at Ken's request. The new `highlight` package draws the input
as it is typed. It parses the text with the usql grammar of transit, parses each
statement with the SQL grammar of the dialect, and runs the highlight queries of
both. The colors come from transit's `styles` module, which has the styles of
Chroma, and the line editor draws them at the depth that the terminal allows.
The README, `env/list.go`, `docs/RELEASE.md` and `docs/PROGRESS.md` say so again.
The settings keep their names and their defaults:
- `syntax_hl` turns highlighting on. It starts off for a terminal with no color
and when `NO_COLOR` is set.
- `syntax_hl_format` is a Chroma formatter name that now sets the number of colors:
`noop` draws none, and `terminal`, `terminal256` and `terminal16m` draw 16, 256
and 24 bits. A name outside these is an error.
- `syntax_hl_style` is the name of a style of transit. A name that is not one is
an error.
- `syntax_hl_override_bg` keeps the background of the terminal when it is true.
When it is false, the background of the style is drawn.
The match of a dialect to a SQL grammar is in `highlight/dialect.go`. PostgreSQL,
CockroachDB, CrateDB and Redshift use the PostgreSQL grammar, MySQL, TiDB, MemSQL
and Vitess use the MySQL grammar, and SQL Server, Oracle and Cassandra use their
own. Every other dialect uses the generic SQL grammar. Before a connection,
PostgreSQL applies, as in psql.
The cost is in the binary. The grammars add about 23 MB to the default build, and
15 MB of that is the PostgreSQL grammar. Ken may choose to give PostgreSQL the
generic grammar, which is 2.7 MB, and to keep the PostgreSQL grammar for the
completer.
What is left: the highlighter parses the text a second time, because the input
and the highlighter keep a tree each (W36 wants one shared tree). The statements
of MySQL, SQL Server and Oracle stored programs split at each `;`, because the
input does not turn on `BeginEndBlocks` yet, and so each part is highlighted
alone. `\p` prints the buffer with no color.
## W59. Record the terminal sessions again: Done
Done on 2026-10-11 at Ken's request, after the CI job "Terminal sessions" failed on
`4a11b57` on Ubuntu and macOS. All nine sessions of `testdata/cli` held the output of
the old line editor and its highlighting, so each failed on the first line (rline
writes `\e[m` before the banner, and draws the prompt in green). They are recorded
again, and the diff was read: no color of highlighting, the prompt row by row as
rline redraws it, and `describe` with the output of `metacmd/describe`. The
session `line-editing` could never end, because the old script pressed Enter on an
open entry and Ctrl-U to throw it away, which rline keeps as rows, so `\q` was
typed into the entry and the session ran to its limit of 30 seconds. It now types
a finished statement, corrects it with Backspace, goes to the start of the line
with Ctrl-A, adds a statement there, and Enter runs both, in 2.8 seconds.
## W60. Make the two SQLite drivers and MySQL print the same
Source: Ken, on 2026-10-11. usql compared go-sqlite3 (`sqlite3`), modernc.org/sqlite
(`moderncsqlite`) and MySQL 26.7 on one set of values: integers up to the largest
int64, reals, decimals, text with accents and a tab, blobs, NULLs, timestamps with
and without an offset, dates and booleans, under UTC and under
America/Los_Angeles.
Same in both SQLite drivers: every integer, real, text, blob, NULL and timestamp,
the declared types, the arithmetic, and the aggregates. What differed:
- The text of an error: `integer overflow` from go-sqlite3 and `SQL logic error:
integer overflow (1)` from modernc. Fixed in usql: the modernc driver takes off the
name of the code and the number, so a syntax error and an overflow read the same.
- A column declared BOOLEAN prints `true` and `false` with go-sqlite3 and `1` and
`0` with modernc, and MySQL prints `1` and `0`. Not changed: the drivers decide it.
- A DATE of `2026-02-29` prints as `0001-01-01T00:00:00Z` with go-sqlite3, which
turns a value that does not parse into the zero time without a word, and as the
text with modernc. Not changed.
- Time zones: both SQLite drivers printed UTC (`...Z`) and MySQL printed the
local zone, because the MySQL family has the default `loc=Local`. dburl v0.53.0
writes `_loc=auto` for `sqlite3` and `_timezone=Local` for `moderncsqlite`, and
usql takes it. Both drivers now print the local zone, and the Open hook of
moderncsqlite is gone.
MySQL and SQLite also differ, and this is the database and not the driver: a
DECIMAL keeps its scale (`-5.00`, and left aligned as text) where SQLite stores
`-5` as a number, `1.0` is a decimal in MySQL and an integer in SQLite, a JSON
column is written with spaces, and `1/2` is `0.5000` in MySQL and `0` in SQLite.
A DATE prints as a timestamp at midnight (`2026-10-10T00:00:00Z`) in all three,
where psql prints `2026-10-10`: tblfmt or the driver formats it, and a date
column should print as a date.
## W61. Move the settings of usql from \set to \uset: Done
Done on 2026-10-11 at Ken's request. psql keeps its own settings and its script
variables in one place, and usql diverges from it: the settings that usql adds
(echo_hidden, editor, on_error_stop, pager, prompt1, prompt2, quiet,
show_host_information, sslmode, syntax_hl, syntax_hl_format, syntax_hl_override_bg,
syntax_hl_style and term_graphics) belong to `\uset`, and `\set` is for scripts.
The names are lower case, as the names of `\pset` are and as Ken asked, and they
are not the names of environment variables, which are upper case.
The name: Ken asked whether it should be `\uset` or `\eset`, then also named `\env`
and `\config`. Gemini 3.1 Pro and DeepSeek V4 Pro both chose `\uset`: it follows
`\pset` (print) and `\cset` (connection), it matches the flag `--uset`, and
`\eset` and `\env` read like `\setenv`, which sets operating system environment
variables. `\config` breaks the pattern and fills `\c` with a second command.
The models split on one point: Gemini advised against toggling a boolean when only
the name is given, as `\pset` does. usql shows the setting, and does not toggle it,
which Ken may change.
How it works: `\uset` with no arguments prints the settings as a table of names
and values, as `\pset` does. `\uset NAME` shows one, and `\uset NAME VALUE` sets it
and says `NAME is VALUE`. The arguments follow the rules of every meta command: quotes,
`:name` interpolation and backticks. `\set` and `\unset` refuse the name of a
setting and say to use `\uset`. `env.Variables` keeps the settings apart from the
script variables, and `env.Setting` reads them. The command line has `--uset`, and
`-q` is `--uset quiet=on`. The completer completes the names and the values, `\?
variables` lists the settings apart, and the README, the samples and the contrib
scripts follow. A session of `testdata/cli` records `\uset`.
What a setting is not: it is not a variable, so `:pager` no longer interpolates.
`ROW_COUNT` stays a script variable, because usql sets it for scripts.
---
### DBMETA (`docs/DBMETA.md`)
# Migrating metadata to dbmeta
> Status, 2026-10-11: the migration is done (W21). usql reads the metadata, the
> version, the lexical forms, the batches, the terminators, the password
> statements and the facts of each driver (the placeholder of `\copy`, whether
> every statement is a query, the product string and the scan by type) from
> dbmeta, and `go.mod` names its release. The steps below are the plan as it was
> written, kept for what they decided and why. W46 holds what the describe
> commands still lack against psql.
This is the plan for moving usql off its own metadata `Reader` and `Writer` and
onto [dbmeta](https://github.com/xo/dbmeta). It was implemented on 2026-09-30,
all at once rather than driver by driver (D4). W21 in
[BACKLOG.md](BACKLOG.md) records what was done and what waits for dbmeta. The
rest of this document is the plan as it was written.
It is a plan, not a decision. The ordering in it differs from the order the
work was first described, and the reason is in "Why the order changed".
## What is wrong today, measured
`drivers/metadata` is 6083 lines including tests.
`drivers/metadata/metadata.go:120` declares `type Reader interface{}`. The base
reader is the empty interface. Every capability is found by a runtime type
assertion, and there are 52 of them outside tests.
That one line causes the rest. A driver that satisfies nothing compiles,
registers and passes the tests. Of 51 registered names, 21 have a reader. Every
one of the 21 that fails a command fails on `\l` or `\dp`. One driver returns
`struct{}{}` as its reader, which the type system accepts.
`Writer` is eight methods, and each one queries, formats and writes:
DescribeFunctions(*dburl.URL, string, string, bool, bool) error
ListTables(*dburl.URL, string, string, bool, bool) error
ListAllDbs(*dburl.URL, string, bool) error
Positional strings, bare bools, a `*dburl.URL` on every call, and only an error
back. `DefaultWriter` holds the reader, the database handle and an `io.Writer`.
It carries a field commented "custom functions for easier overloading" that
replaces `ListAllDbs` at construction. `NewDefaultWriter` returns a constructor
rather than a `Writer`.
Neither interface takes a `context.Context`. Timeouts are set by an option
stashed on a `LoggingReader`.
## Two facts that constrain the design
These were checked rather than assumed, and each removes a decision.
No project outside usql imports `usql/drivers/metadata`. dbtpl, dbmeta and
tblfmt were all searched. So there is no external compatibility burden, and
every caller of both interfaces is in this repository where the compiler can
find it.
`resultSet` in `metadata.go` already implements `tblfmt.ResultSet` exactly:
`Next`, `Scan`, `Columns`, `Close`, `Err` and `NextResultSet`. `writer.go`
calls into tblfmt eleven times.
The second fact decides what a formatter produces. tblfmt already owns aligned,
unaligned, CSV and the other output modes. A formatter that renders bytes would
duplicate it. A formatter that produces a `tblfmt.ResultSet` does not.
## The boundary
dbmeta reads. tblfmt renders. usql decides what to print.
That division is dbmeta's D5 and it does not change. The writer does not move.
Three options were put to the dbmeta project and this is the one its
`docs/USQL.md` describes.
Two readings were wrong and are recorded so nobody re-derives them:
dbmeta does not own models without queries. A model there is the queries. There
are 12 of them, and a query set for each of 55 object kinds, registered per
dialect. Taking the types without the queries takes the smallest part.
dbmeta does not take the formatting. It answers what the database says.
## Why the order changed
The work was first described as: remove `Reader` and `Writer`, move the readers
to dbmeta, turn `Writer` into `Formatter`, then wire up the commands.
Two things move earlier in this plan.
The formatter work comes before the reader swap, not after. It is entirely
inside this repository, it has no external consumers, and it is where the
psql-matching behavior lives. Doing it first means the reader swap changes one
thing at a time.
A step that produces no visible result comes first of all. It is described in
the next section, and it is the step most likely to be skipped.
## Step 0. Extract the filtering, change nothing else
usql's output matches psql. The matching is done partly by the readers and
partly inside `DefaultWriter`, which queries, filters and renders in one
object.
dbmeta returns what the database says and does not shape a result so that
somebody's output looks right. So if a reader is replaced by a dbmeta query
before that filtering is pulled out, usql's output changes, and it changes
silently, because it appears as extra rows rather than as an error.
Pull the psql-matching filtering out of `DefaultWriter` and into the formatting
layer, with the current readers still in place. Confirm that the output is
unchanged, byte for byte.
This step has no visible result. That is why it is written down first.
## Step 1. Redesign the writer as a formatter
Three layers, not two. Both external reviews reached this independently, and it
is the answer to a question the first design sketch got wrong.
reader (dbmeta) -> command layer (usql) -> formatter (usql) -> tblfmt
A formatter that takes one result set cannot express `\d`, which composes five
sections: columns, indexes, constraints, triggers and sequences. That
composition is not reading and it is not rendering. It belongs to a middle
layer that the current design does not have, because `DefaultWriter` is doing
all three jobs at once.
So the formatter formats one section. The command layer runs the queries,
writes the section headings, and calls the formatter once per section.
### What the formatter produces
A `tblfmt.ResultSet`, not bytes. tblfmt then renders it in whatever output mode
the session is in.
This makes the formatter a pure function from metadata to a result set, which
makes it testable without a database. That matters here: five test packages
need a container runtime today, and none of the formatting needs one.
### What goes in the signature
An options struct, not functional options and not positional bools. There are
eleven fixed call sites in one switch. Functional options are for open-ended
constructor knobs and would add a closure allocation per `\d`.
Drop `*dburl.URL` from every method. It is resolved once in the command layer.
Take a `context.Context` on the readers, not only on the formatter. The queries
are the slow part. This also deletes the timeout option stashed on
`LoggingReader`, which exists only because there was no context to carry a
deadline.
Drop `Params map[string]string`. It is untyped and forces a formatter to parse
something the command layer already parsed.
Delete the `listAllDbs` override field. It is a hook that exists because the
interface could not express the case.
### No compatibility shim for this part
Both interfaces are internal and no other project imports them. The compiler
finds every caller. A shim here would cost more than the change and would
outlive its purpose.
## Step 2. Capability discovery
This is where the two external reviews disagreed, and the disagreement is worth
recording because the losing answer is the one that looks more obvious.
One proposal was a unified interface with a `Capabilities()` bitmask, and a
`BaseReader` that every driver embeds and that returns a not-supported error
for anything it does not implement.
Do not do this. Embedding a base that answers everything makes every driver
satisfy every interface, so capability can no longer be detected by assertion
at all. A caller has to call and inspect the error. That reintroduces the
current failure in a new form, where nothing can be counted without running it.
A hand-maintained bitmask has the second problem: it can disagree with the code
and nothing notices.
The better answer keeps small optional interfaces, which is how the standard
library does this, and fixes the real defect, which is not the assertions but
that there are 52 of them scattered against `interface{}`. Resolve them once:
type Capabilities struct {
Catalog CatalogReader
Table TableReader
Column ColumnReader
Index IndexReader
Privilege PrivilegeSummaryReader
// and the rest
}
func CapabilitiesOf(r Reader) Capabilities
Every assertion happens in one function. Callers test a typed field against nil
and then call a statically typed method.
Make the base interface non-empty. A single required method is enough, and it
makes the driver that returns `struct{}{}` stop compiling.
### The sets
Keep the fourteen typed sets at the dbmeta boundary. They carry typed fields
and real tests.
Do not replace them with a generic `Set[T]`. A formatter taking `Set[T]` still
receives `any` at the call site, which reintroduces the type switch that the
generic was meant to remove.
Convert to the single render shape at the formatter boundary instead. That is
one small method per set type, and a formatter then never changes when dbmeta
grows a fifteenth kind.
## Step 3. The information schema adapter, which ships first
This is the earliest visible gain and it is not a native model.
dbmeta has `models/informationschema`, a shared model answering 12 of the 55
object kinds against any database with a standard `information_schema`: tables,
schemas, columns, functions, privileges, constraints, sequences, constraint
columns, routine parameters, views and the current schema.
30 of usql's 51 registered names have no metadata reader at all. Any of them
with a standards-shaped catalog gets those twelve with no per-driver query
written.
Ship this before any native model moves. It is one adapter, it lands coverage
across a set of names rather than one, and it is the cheapest demonstration
that the boundary works.
## Step 4. Move drivers one at a time
The move is per driver and reversible. usql is never blocked on dbmeta for a
product dbmeta does not cover.
### What dbmeta answers today
Measured by dbmeta on 2026-09-26, by asking every registered query whether it
supports the newest release of its dialect. The units are usql's: 11 dispatch
commands, and 5 further sections that `\d NAME` adds.
PostgreSQL 11/11, 5/5
MariaDB 11/11, 5/5
SQL Server 11/11, 5/5
SAP HANA 11/11, 5/5
MySQL 11/11, 4/5 no sequence
ClickHouse 11/11, 2/5 no sequence, trigger or constraint column
Oracle 10/11, 5/5 no \l
Firebird 10/11, 5/5 no \dn
DuckDB 10/11, 3/5 no \dp, no index column or trigger
SQLite 10/11, 4/5 no \dp, no sequence
Cassandra 10/11, 4/5 no \l, no sequence
Trino 8/11, 0/5 no \df, \da, \di, and no sections
Presto 8/11, 0/5 the same
### Four products where usql answers nothing today
Four of those thirteen map to usql drivers that register no metadata reader at
all. Each was checked against the driver source:
SAP HANA drivers/saphana 0/11 today, 11/11 and 5/5 with dbmeta
Firebird drivers/firebird 0/11 today, 10/11 and 5/5 with dbmeta
Cassandra drivers/cassandra 0/11 today, 10/11 and 4/5 with dbmeta
Presto drivers/presto 0/11 today, 8/11 with dbmeta
These are the largest single gains available, and they are gains rather than
refactors. SAP HANA is the largest of all: a product that answers nothing today
and would answer every command and every section.
None of them can come early. SAP HANA has no `information_schema`, so it needs
its native model rather than the shared adapter in Step 3. They belong in this
phase.
### Which native model moves first
MariaDB, but the reason is the sections rather than the commands.
MariaDB is 10 of 11 today and lacks only `CatalogReader`, so the command gain
is one: `\l`. The gain worth having is 5 of 5 sections under `\d NAME`,
against the four the mysql family answers today.
Not PostgreSQL. PostgreSQL already works and moving it would demonstrate
nothing.
Then the rest, in whatever order suits usql.
### Make the remainder countable
A partial migration becomes permanent when nobody can see how partial it is. A
deadline does not fix this. A count does.
Add a test over the driver registry: every registered name either reads through
dbmeta or has an entry saying why not. This is the same exemption map as
[W18](BACKLOG.md), with the same two properties. A contributor cannot pass it
by doing nothing, and one who genuinely cannot move a driver writes one line
saying why.
The list then shows how many entries say "not yet" and how many say "cannot",
which are different states that look identical today.
## Step 5. Version queries
Version queries move to dbmeta, and the data already exists there. Every model
declares a version query, its columns and a parser, and a dialect returns a
version set plus the display line.
Two qualifications.
usql keeps its own fallback for every name dbmeta does not cover. That fallback
is `SELECT version();`, which several products do not have.
Oracle is not one of the products that falls through to it, because dbmeta has
a model for it. The oracle driver registers itself in `drivers/oracle/oracle.go`.
It used to register through a shared package, `orshared`, that godror also used,
and that path made an earlier count of registered schemes too low. The package
is gone, since nothing else used it.
Oracle has a different defect, and two open pull requests report it. Its
version query is `SELECT version FROM v$instance`, and `v$instance` needs
privileges an ordinary user does not have. See W22.
The version is read once when the connection opens, because everything else
resolves against it. That is a change to usql's connection path, not to a
reader.
## Step 6. Change password, which is a bug fix
This is already in dbmeta and it fixes something rather than relocating it.
`Dialect.ChangePassword` takes no database handle and runs nothing. It builds
the statement and returns it as text for usql to execute. dbmeta still executes
only reads, so D5 holds.
usql changes a password in seven drivers today and escapes nothing. Each one
concatenates the password into the statement, so a password containing a quote
or a backslash either breaks the statement or sets something other than what
was asked.
The escaping cannot be done without the server. Whether a backslash escapes
inside a string literal is `sql_mode` on MySQL and MariaDB, and
`standard_conforming_strings` on PostgreSQL. dbmeta reads both.
## Step 7. Follow psql, including for the states usql cannot express today
Decided on 2026-09-26. usql follows psql's output.
dbmeta can answer four things where usql has one: the object is supported, the
product does not have it, the server is too old for it, or the model was not
built. usql prints an empty table for all four today.
The rule is that psql decides. Where psql prints `null` for a metadata command,
usql prints `null`. Where psql leaves an empty cell, usql leaves an empty cell.
Where psql has no equivalent state, usql does not invent a presentation for it.
TooOld is the one with no psql equivalent, because psql talks to one product
and does not have the problem. It is still worth carrying in the data even when
nothing prints it, because the alternative is discarding it at the boundary and
being unable to add it later.
## Step 8. Where the presentation code lives, undecided
There is an open question about where the code that turns a metadata answer
into a printable cell belongs. It is not decided and it does not block the
earlier steps.
Three candidates, and the argument is the same for each: other command line
clients may want the same behavior, so it should sit somewhere a third party
can use.
1. dbmeta, next to the data that carries the distinction.
2. tblfmt, which already owns rendering and output modes.
3. A separate place that neither project owns.
Decide it before Step 4 moves a native model, because that is the first point
where a state usql cannot currently express reaches the output.
## Step 9. Share the container list
dbmeta's `container` package is Go data naming every release it tests against.
It deliberately starts nothing and imports no container client, so that a
downstream project can read the list and bring its own podman or docker.
usql should read that list rather than keep a second one. Testing the same
products separately on both sides means twice the containers for one behavior.
This is the same conclusion the [W6](BACKLOG.md) tombstone reached from the
other direction.
## Smaller things that are cheap now and expensive later
dbmeta's arguments carry a catalog filter and `metadata.Filter` does not. Only
Trino uses it, where a table really is catalog, schema and name, and one server
reaches many catalogs at once. It blocks nothing, but it cannot be added later
without touching every call site.
Decide which of the eleven commands fail and which print nothing when a
capability is missing. A sentinel error is needed either way.
Identifier case and quoting differ by product. `\dt myTable` means different
things on PostgreSQL and on Oracle. Pattern normalization belongs in the
per-product reader, not in the command parser.
## How usql depends on dbmeta while both move
Decided on 2026-09-26.
To build against a dbmeta checkout that has not tagged a change yet, write a
`go.work` that uses `.` and the checkout, and do not commit it. `go.mod` names
the released version.
dbmeta is tagged when usql is ready to switch to it completely, and not before.
The tag follows the work rather than leading it, because a tag published while
the API still moves is a promise nobody can keep.
Do not add dbmeta to `go.mod` before that tag exists.
A point release during this period comes off a `release-*` branch. The main
branch is expected to carry working changes for the length of this cycle.
## Two review disagreements, recorded
Capability discovery is in Step 2 above.
The other was streaming. One review argued that materializing a whole set into
memory will exhaust it on a large table. The other argued that buffering is
required regardless, because column widths cannot be computed without seeing
every row.
For metadata the second is right. These queries list tables, columns and
indexes, and the counts are bounded by the schema rather than by the data. Do
not design a streaming interface for this layer. The concern is real for row
results, which this layer does not carry.
---
### DRIVER (`docs/DRIVER.md`)
# Adding a database driver to usql
This document is written for a coding agent. It is the whole procedure, in
order, including the parts that are not code.
Read all of it before you start. Several steps happen in a different repository,
and one of them must happen first.
## The one thing to understand before anything else
In usql, a broken driver and a working driver look the same at build time.
Almost every field of `drivers.Driver` is optional. `drivers.Register` checks
only that the name is not already taken. A driver that cannot parse its own
connection string, or that panics on `\l`, compiles cleanly, registers cleanly,
and passes `go test ./...`.
So compilation is not evidence. Registration is not evidence. The only evidence
is a live connection, and the verification step below says what to run.
## Stop conditions
Stop and report to the person who asked, rather than continuing, if any of
these is true:
1. The scheme does not exist in the version of `dburl` pinned in `go.mod`.
2. The upstream driver fails any pre-flight check in the next section.
3. The driver needs cgo, a vendored native library, or a client install.
4. You cannot run the database locally, in a container or in an embedded mode.
None of these means the driver must be refused. They mean the decision is not
yours. Report what you found and wait.
## Step 1. The pre-flight gate
Answer four questions before you create any package. Record the answers in the
driver's package comment, so that a later reader can see what was checked and
when.
This gate exists because of what the history shows. Several drivers were added
and later removed, and the cause was never the SQL. It was an upstream that
stopped being maintained, or a driver that reached into global process state.
### Is it maintained?
Record the date of the last tagged release and the date of the last commit.
Record whether open issues are being answered. Write dates, not impressions.
A driver whose upstream stops being maintained is dropped from usql. See
"When a driver is broken" in [CONTRIBUTING.md](../CONTRIBUTING.md).
### Does it touch global process state?
This is the question that pays. Read the upstream driver's `init()` and
everything it calls. A driver may register itself with `database/sql` and do
nothing else.
Search the upstream source for these:
slog.SetDefault log.SetOutput log.SetFlags
log.SetPrefix flag.Parse os.Setenv
http.DefaultClient http.DefaultTransport
signal.Notify rand.Seed os.Exit
A hit inside package initialization is a stop condition.
The RamSQL driver is why this check exists. Its `engine/log` package called
`slog.SetDefault` from `init` with a handler writing to `os.Stdout`. A default
`slog` logger also redirects the standard `log` package, so importing that one
driver changed the output of the whole binary.
### What does it link?
Record whether the driver is pure Go. A driver that needs cgo, ships a
prebuilt native library, or requires a client install cannot go in the default
build. It also costs far more binary size than a pure Go driver.
A build of the `most` group is 145.8 MB, and DuckDB is 54 MB of it (W32). Two known causes are worth
recognizing, because both survive `-s -w` stripping: `-rdynamic` in cgo
LDFLAGS, and any use of Go's `plugin` package. Both force dynamic symbol
export.
### Can it be tested?
Record whether a maintained container image exists, or whether the database
runs embedded. A driver that nobody can run is a driver that nobody will
notice breaking.
Prefer an image the vendor publishes. An image last built years ago is not a
maintained image.
### Is the driver worth the wrapper it needs?
If making the driver work needs a large validation wrapper, the answer is to
not support it. The Dameng driver needed about 130 lines of DSN validation
against a median of about 20 for other drivers, compensating for a parser that
`dburl` cannot import or test. Dameng was later removed.
Emit the connection string and let the driver reject what it rejects.
## Step 2. The scheme, which lives in dburl
usql does no connection string parsing. All of it is in
[github.com/xo/dburl](https://github.com/xo/dburl). The scheme must exist there
before usql can use it.
Read `docs/SCHEME.md` in dburl. It is ten numbered steps written for this
task. What follows is only the part that affects choices you make on the usql
side.
### The scheme and the directory name may differ
usql's README table has a column headed `Scheme / Tag`. It holds the build
tag, which is the directory name under `drivers/`, and not the scheme.
They are allowed to differ and sometimes do. The lib/pq driver is the directory
`libpq` and the scheme `pq`. It works, because the tag is also a dburl alias of
the scheme.
You are choosing two names. Do not assume they must match.
### Choosing a generator
A `Gen*` function is a helper. Reusing one says nothing about wire
compatibility, and reuse is normal. Prefer the first of these that fits:
1. `GenFromURL("scheme://localhost:PORT/")` when the connection string is a
URL that needs a default port.
2. `GenScheme("name")` when it is a URL that needs no default port.
3. A hand written `GenXxx` when the connection string is not a URL, such as a
`key=value` form, or when the generator has to reject something.
What reuse does claim is that every driver behind that generator accepts the
same connection string. Check that claim against the new driver. Trino and
Presto shared a generator for six years, and it was split when the Presto v2
driver started rejecting what the shared function produced.
### File backed databases need a file type as well as a scheme
A database that reads a file rather than connecting to a host needs a
`RegisterFileType` call in dburl's `init`:
RegisterFileType("duckdb", isDuckdbHeader, `(?i)\.duckdb$`)
The three arguments are the driver name, a header function, and an extension
pattern. Register the scheme with `Opaque: true`.
The header and the extension are consulted in different situations, and this
is the part that costs debugging time:
1. When the file exists, dburl reads the first 64 bytes and asks each header
function. The extension is never looked at.
2. When the file does not exist, no header can be read, so the extension
decides.
That is why `usql test.duckdb` once worked on a path that did not exist and
failed on the same path after the file was created. Only the second case
reached the faulty matcher. Anyone debugging that without knowing the rule
looks at extension registration, where nothing is wrong.
Header functions compare bytes at a fixed offset. They never use a regular
expression.
### Aliases are cheap to add and expensive to remove
usql publishes every alias in its README table, so an alias is advertised as
supported from the moment it exists. Removing one is a change to the front
page.
Presto is the warning. Five aliases were published. Two of them were `s`
suffixed variants meaning TLS, which the v2 driver cannot express, because it
selects TLS from options rather than from the scheme. Removing them was a
break.
Do not add an alias for a spelling you have not seen in use.
### Read the parser, not the documentation
Read the connection string parser in the exact driver version that usql pins
in `go.mod`. Not the driver's documentation, and not memory. Presto broke
because nobody did.
## Step 3. The driver package
Create `drivers//.go`. The package needs an `init()` that calls
`drivers.Register`.
A minimal driver is short, because nearly every hook is optional:
package mydb
import (
_ "github.com/example/mydb-go-driver" // DRIVER
"github.com/xo/usql/drivers"
)
func init() {
drivers.Register("mydb", drivers.Driver{})
}
A field that describes the database rather than the Go driver is not set
here. dbmeta holds it, keyed by the dialect: the comment and quoting forms
the lexer allows, what happens to a trailing semicolon, the statements that
open and close a batch, whether a password change needs the old password, and
how the database folds the case of a name. D5 records this. A database with
no dbmeta model gets only block comments. W28 holds the four drivers whose
statement handling waits for dbmeta.
A driver of [dbimp](https://github.com/xo/dbimp) is exactly this, and nothing
more. dbimp's errors print as they are, an error that matches
`dbimp.ErrAuthentication` is a wrong password, a statement with no count of rows
changed is a count of zero, and `\copy` takes each insert alone when `BeginTx`
fails with `dbimp.ErrNotSupported`. Set a hook only for a fact that none of these
cover, and put a fact about the database in dbmeta and a default of the DSN in
dburl.
Keep the `// DRIVER` comment on the driver import. Other tools in the family
find the driver package by searching for it.
### The package comment carries machine readable lines
`gen.go` reads the package comment. Three lines in it are not decoration.
A `Build:` line gives a constraint for drivers that cannot compile everywhere,
and `gen.go` copies it into the generated file under `internal`:
// Build: !(linux && arm) && !windows
The duckdb driver uses that constraint, because its prebuilt library has no 32
bit arm build and no Windows build. The oracle driver uses `!(linux && arm)`,
because go-ora writes a constant that overflows a 32 bit integer.
A `Group:` line puts the driver in a build group. The groups are `base`, which
is the default build, `most`, `all`, and `bad` for drivers that are known to be
broken.
Always write the `Group:` line, even where it says what the default would.
A driver that omits it lands in `most` without anyone having decided that.
Ken's rule decides between `most` and `all`. A driver goes in `most` when its
dburl scheme names a dialect and it does not need cgo. A driver that needs cgo
goes in `all`. No driver is in `all` today, because godror was the last one
and was removed on 2026-10-09. A driver that is known to be broken goes in
`bad`.
Promoting a driver to `base` is a decision for the maintainer, not part of
adding it.
A `Tags:` line names further build tags that select the driver. Each one
selects it as the driver's own tag does, and its `no_` form leaves the driver
out. The pgx driver answers to the `postgres` tag this way, because dburl sends
`postgres://` URLs to it:
// Tags: postgres
Most drivers have no `Tags:` line. A `Build:` line cannot do this job, because
`gen.go` combines it with the group constraint using `&&`, so it can only
narrow where a driver builds.
### Promoting to base needs one more edit
If a driver is ever promoted to the base group, add it to `baseOrder` in
`gen.go`. That map fixes the order of the base rows in the README.
A driver missing from the map sorts as 0 and ties with postgres. `sort.Slice`
is not stable, so the two rows swap on every run and `go generate` churns the
README.
## Step 4. Generate, do not edit
Run the generator:
go run gen.go
It reads dburl's scheme registry from the version pinned in `go.mod`. So the
scheme from Step 2 reaches usql only after `go.mod` moves to the dburl release
that holds it.
These files are generated. Do not edit them by hand:
1. `internal/.go`, which carries the build constraints.
2. The driver tables in `README.md`.
3. The bundled license files.
The alias column and the file marker in the README come from dburl's registry.
The description, the package path and the `Scheme / Tag` column come from
usql's own data in `gen.go`. So regenerating rewrites the alias column and
leaves the rest of the row alone.
Then add the module and tidy:
go get github.com/example/mydb-go-driver@latest
go mod tidy
## Step 5. Metadata goes to dbmeta
Do not write a metadata reader here. All database metadata is moving into
[dbmeta](https://github.com/xo/dbmeta), and `metacmd/describe` takes no new
reader and no change to an existing one. See D3.
usql reads the metadata of every driver from dbmeta, by the
dialect that dburl sets. A driver whose database dbmeta has no model for
reports that its describe commands, such as `\d`, `\dt`, `\l` and `\dp`, are
not supported. Add it to `noModel` in `metadata_test.go` with the reason,
because `TestEveryDriverReadsThroughDbmeta` fails until it is there.
The metadata for a new database is added in `dbmeta`, which models each
product and says what it can answer. Ask there, or ask Ken. W21 in
[BACKLOG.md](BACKLOG.md) holds the migration.
## Step 6. Verification
Compilation proves nothing here. Run all of this, and report the output rather
than a summary of it.
### Without a database
go run gen.go # then inspect git diff
go build ./... # default build
go build -tags all ./... # every driver
go vet ./...
go test ./...
The `-tags all` build matters on its own, because a duplicate scheme or alias
panics at registration time rather than at compile time.
Every test must pass. No test of usql needs a database.
### With a database
Start the database, then run each of these and keep the output:
usql -c 'select 1'
usql -c '\dt'
The second one reports that describe commands are not supported, because a new
driver has no metadata reader. See Step 5.
Try every connection string form that dburl accepts for the scheme, including
the short alias. Try one malformed connection string, and make sure that it
produces an error rather than a panic.
For a file backed driver, test both cases separately, because they take
different paths through dburl:
usql ./new.db # the file does not exist: the extension decides
usql ./existing.db # the file exists: the header decides
### What counts as done
Do not report the work as complete on compilation, on registration, or on
`go test` alone. In this repository all three are true of a driver that does
not work.
Report the pre-flight answers, the generator diff, and the transcript of a
live connection.
## What goes wrong most often
Adding the driver to usql without the scheme existing in dburl. The code
compiles, connecting fails at runtime, and the next move is usually to write
connection string parsing inside the usql driver package. That parsing does not
belong there and will be rejected. If the scheme is missing, stop at Step 2.
The second most common is hand editing a generated file. The change survives
until somebody runs `go generate`, and then it disappears.
## The checks that tell you what you forgot
| Check | What fails it |
| --- | --- |
| `go build -tags all ./...` | A duplicate scheme or alias, which panics at registration |
| `git diff` after `go run gen.go` | A generated file that was edited by hand |
| A malformed connection string | A panic where an error was wanted |
| A file path that does not exist | A missing extension pattern in dburl |
| A file path that does exist | A faulty header function in dburl |
| The upstream `init()` | A driver that writes global process state |
Add a row whenever you find something this document did not catch. The table
is read when something is red, which is more often than the rest of the
document is read.
## Related documents
| Task | Document |
| --- | --- |
| Adding a scheme | `docs/SCHEME.md` in dburl |
| A driver that is broken or unmaintained | [CONTRIBUTING.md](../CONTRIBUTING.md) |
| Running the tests | [CONTRIBUTING.md](../CONTRIBUTING.md) |
| Planned work | [BACKLOG.md](BACKLOG.md) |
| Rules for agents working here | [AGENTS.md](../AGENTS.md) |
---
### PLAN (`docs/PLAN.md`)
# Plan
`usql` is a universal command line interface for SQL databases. It links many
`database/sql` drivers into one binary and gives each the same psql style
commands.
This file holds the direction of the project and the questions that are open
for Ken. The work items are in [BACKLOG.md](BACKLOG.md), each with a `W`
number. The decisions are in [decisions/](decisions/README.md), each with a
`D` number.
## Direction
- Connection string parsing lives in `dburl`, and the README driver tables are
generated from its registry by `gen.go`.
- Database metadata moves out of `drivers/metadata` into `dbmeta`, and no
metadata reader is written here (D3). W7 and W21 track it, and
[DBMETA.md](DBMETA.md) holds the steps.
- Drivers for databases whose upstream Go driver is missing, dead or heavy come
from `dbimp`, which is now most of them: it is pure Go, and a driver of it is a
bare registration, because its errors, row counts and refused credentials are
generic here. W27 records the review that retired, replaced and regrouped the
drivers.
- What describes a database (the lexical forms, the placeholder of `\copy`, the
statement that every one of its statements is a query, the product string) is
in dbmeta, and the defaults of a DSN are in dburl. A driver holds only Go code.
- The describe commands follow psql (W46), PostgreSQL first and the other
databases one at a time.
- The containers that usql is tested against come from `dbmeta`'s `dbrun`.
CONTRIBUTING.md shows how to use it.
## Open questions
For Ken:
- How the release handles the ODBC crash that go-sqlite3 and DuckDB cause
together: build with `-tags libsqlite3`, leave ODBC out, or make `moderncsqlite`
the SQLite of a release (W40).
- Whether Cosmos DB keeps no writes, since dbimp's driver reads only.
- Whether DuckDB gets a build group of its own, since it is 37% of a `most`
binary (W32).
- The history file of gohxs/readline, which rline cannot read (see
[PROGRESS.md](PROGRESS.md)).
- Whether the drivers that are only a registration lose their packages, with
the build tags written into `internal/` by `gen.go` (W52).
---
### PROGRESS (`docs/PROGRESS.md`)
# Progress
This file says where the work in progress stands, so that a session that
stops can be picked up again. It is about W36 in
[BACKLOG.md](BACKLOG.md). Update it at each checkpoint.
## Work modes
Ken's machine has crashed from resource exhaustion. Work in a low-resource
mode:
- Build one package at a time. Do not run the whole test suite, and run no
more than one test package at once.
- Start no containers and no background jobs unless Ken asks.
- Commit a checkpoint before a long step, on `main`, as a commit with a
proper message. Ken wants no work branches and no WIP commits. Do not
push.
## What W36 changes
usql reads its input through rline and the usql grammar of transit, in place
of gohxs/readline, usql's own `rline` package and the `stmt` package.
- `feed` is new. It parses input with the usql grammar, holds psql's query
buffer, says when an entry typed in the editor is finished, feeds
statements and meta commands one at a time from the editor or from lines,
and expands variables when a statement runs. Its tests pass.
- `metacmd` reads the arguments of a meta command from the grammar's parts.
`\g` reads its option list from the grammar. `\e`, `\p`, `\w` and `\r`
work on the feed buffer. It builds.
- `handler` reads through `feed` and talks to the user through
`*rline.Prompt`. Its loop is new. Highlighting comes from the
`highlight` package. It builds.
- `drivers` has `Language(u)` in place of the stmt options, and no chroma
lexer. A driver's own Version wins over dbmeta's, for InfluxDB's release
from GET /ping (dbmeta's D166).
- `metacmd/complete`, which was `drivers/completer`, is an `rline.Completer`. It adds candidates with
`AddCandidate`, because `Completion.Add` deletes nothing before it.
- `styles` is removed.
## What is left
1. Done on 2026-10-02: run.go and main.go use rline and feed, `rline/` and
`stmt/` are deleted, chroma and gohxs/readline are out of `go.mod`, and
the handler, metacmd and feed tests pass.
2. Checked by hand on SQLite, piped: statements across lines, `\set` and
variables, `\g` for a statement with no `;`, `\echo`, `\i` with a
variable that it sets, `\prompt`, `\p` and `\r`. Checked at a
pseudo-terminal: an entry over two rows, variables and `\q`.
3. Checked on 2026-10-02: Ctrl-C gives up an open entry, `\e` runs the
edited text and goes on, and `\c` from SQLite to DuckDB parses what
follows with DuckDB's forms. The history is not saved: see below.
4. TestCLI passes: the nine sessions of `testdata/cli` were re-recorded on 2026-10-11,
with no color of highlighting, and `line-editing` was rewritten, because Enter
on a finished entry runs it wherever the cursor is, and rline keeps an open
entry as rows, where readline ran its line.
Open points that are known:
- `%R` shows only `=` or `-`, because rline draws one prompt for an entry.
A prompt for each row waits on rline's backlog item 33.
- rline writes `\x1b[m` before the first text when color is on and the
output is not a terminal. run.go turns color off unless both input and
output are a terminal. Tell rline: its session was not running on
2026-10-02.
- At a pseudo-terminal under `script`, the first keys typed after start are
lost, and later keys are not. rline may ask the terminal a question when
it starts and read the answer from the input. Tell rline. It may also lose
type-ahead at a real terminal.
- The history is never saved for a user who has a history file from
gohxs/readline. rline's format escapes a backslash, so the first line such
as `\q` stops the read, and rline then refuses to save, to keep the rest of
the file. It ignores the error at start. This is for Ken: migrate the file,
use a new file name, or have rline read the old format.
- dbmeta gives the version of every database that it has a model for, and the
Version hooks that remain are InfluxDB's (by Ken's decision, dbmeta's D166)
and csvq's, for which dbmeta has no model. The describe commands read dbmeta
through `metacmd/describe` (W46 holds what still differs from psql).
## How to build
usql requires dbmeta v0.15.0, transit v0.4.0 with its usql grammar v0.4.0, and
rline v0.1.0 in `go.mod`. It builds with plain modules and no `go.work`, and
`go mod tidy` leaves `go.mod` as it is. A workspace is still how to build
against a local checkout of a sibling that has not tagged a change yet: write a
`go.work` that uses `.` and the checkouts, and do not commit it. If the workspace
loads two modules that provide `github.com/google/flatbuffers/go`, a `replace`
to an empty module, a directory with only a `go.mod` that reads `module
github.com/google/flatbuffers/go` and `go 1.21`, ends the conflict.
The recordings in `drivers/testdata` were removed with the tests against
databases. W37 brings them back.
---
### RELEASE (`docs/RELEASE.md`)
# The next major release
This file lists what changed in usql since v0.21.6, and what must be checked
by hand before the release. It is the largest set of changes that usql has
had. Automated tests cover much of it, and Ken checks the rest by hand.
Each check names the databases it covers. Tick a check off in a copy of this
file, not here. When a check finds a fault, record it in
[BACKLOG.md](BACKLOG.md) or fix it, and add a test where one can be written.
## Known gaps
Read these first. Some of them can stop the release.
- The history is not saved for a user who has a `~/.usql_history` from
gohxs/readline. rline cannot read its backslashes, and then refuses to
save. It reports nothing. Ken decides the fix (see
[PROGRESS.md](PROGRESS.md)).
- `\if`, `\elif`, `\else` and `\endif` still do nothing, and every branch
runs. W36 gives them psql's meaning, and that work is not done.
- `%R` in a prompt shows only `=` or `-`, because rline draws one prompt for
an entry. rline's backlog item 33 holds a prompt for each row.
- At a pseudo-terminal, the first keys typed after start can be lost, and
rline writes an escape code to a pipe when color is on. usql turns color
off unless both input and output are a terminal.
- ODBC no longer reads `usql_trim`, csvq's `SHOW` prints nothing, and Athena
and Cosmos no longer strip a trailing semicolon. W28 holds them.
- usql requires dbmeta v0.15.0, transit v0.4.0 and rline v0.1.0, all in
`go.mod`. It builds with no `go.work`.
- The tests against databases and their recordings were removed, because
they started their own containers. W37 brings them back on `dbrun`, and
W38 runs CI on every database. Both are due before the release.
- **ODBC crashes in the default build.** With go-sqlite3 and DuckDB both
linked, which every `most` build does, the first ODBC query crashes the
process, because DuckDB's bindings export the `sqlite3_*` symbols of
go-sqlite3 into the executable. Building with `-tags libsqlite3` fixes it,
and that changes the SQLite of the release builds. Ken decides how (W40).
This stops the release until he does.
- `\copy` still writes its own SQL: the INSERT, and the query for the columns of
the target. The placeholder of each database comes from dbmeta. W29 moves the
rest, and is due before the release.
## What changed
### Input: rline and transit's usql grammar
usql reads its input through `github.com/xo/rline` and the usql grammar of
`github.com/xo/transit`. They replace gohxs/readline, usql's own `rline`
package and the `stmt` package (W36).
- The package `feed` parses input with the grammar. It holds psql's query
buffer, says when an entry is finished, and feeds each statement and meta
command to the handler.
- An entry can span several rows in the editor. Enter starts a new row until
the entry ends with a `;` or a meta command.
- A variable is replaced when its statement runs, by its range in the
input. A defined `:'x'` no longer opens a string, and the second colon of
`::` is no longer a variable.
- The grammar is one language, and its scanner takes the lexical forms of
the dialect as options (transit's D108): dollar quotes, block comments,
`//` and `#` comments, and backticks. They come from dbmeta, and follow the
dialect of the connection, so they change on `\c`.
- The arguments of a meta command come from the grammar. `\g` reads its
option list, such as `(format=csv)`, from the grammar.
- `-o FILE` sends the results to the file, as `\o` does. `\echo` still goes
to the terminal, as in psql.
- The completer is an rline completer.
### Metadata: dbmeta and psql's describe commands
usql reads every database's metadata through `github.com/xo/dbmeta` (W21).
It has no metadata query of its own (D3, D4).
- The readers in usql are gone, for the information schema, PostgreSQL,
Oracle, MySQL, Impala, SQLite, SQL Server, ClickHouse, Trino and DuckDB.
- The output follows psql, so it differs from the output of v0.21.6.
- psql's other describe commands are added (W31): `\dA` and its forms,
`\db`, `\dc`, `\dconfig`, `\dC`, `\dd`, `\dD`, `\ddp`, `\dE`, `\des`, `\det`,
`\deu`, `\dew`, `\df` with `anptw`, `\dF` and its forms, `\dg`, `\dl`,
`\lo_list`, `\dL`, `\do`, `\dO`, `\dP`, `\drds`, `\drg`, `\dRp`, `\dRs`,
`\dT`, `\du`, `\dx`, `\dX`, `\dy`, `\sf`, `\sv` and `\z`.
- A name with both S and +, such as `\dtS+`, works. `\d` takes its class
letters in any order.
- A pattern folds as the database folds a name that is not quoted, and a
name between double quotes keeps its case. A pattern also shows the
objects of the system, as in psql.
- The package `metacmd/describe` formats the answers.
- The version in the banner, the current user, and the statement that
changes a password come from dbmeta. InfluxDB shows its release from
GET /ping.
### Settings: `\uset` and `\set`
usql's own settings left `\set` for the new `\uset`, which works as `\pset` does
(W61). `\set` is only for script variables, and its `:name` interpolation.
- The settings are echo_hidden, editor, on_error_stop, pager, prompt1, prompt2,
quiet, show_host_information, sslmode, syntax_hl, syntax_hl_format,
syntax_hl_override_bg, syntax_hl_style and term_graphics.
- `\uset` shows every setting as a table, `\uset NAME` shows one, and `\uset NAME
VALUE` sets it and says so. A value is read as an argument of any meta command.
- `\set pager less` and `\unset quiet` are errors that say to use `\uset`, so a
script or a `.usqlrc` that set a setting with `\set` must change. `:pager` is no
longer a variable.
- The command line has `--uset NAME=VALUE`, and `-q` is `--uset quiet=on`. `--set`
and `-v` give script variables only.
### The fields of a driver
What describes a database moved from `drivers.Driver` to dbmeta (D5):
- the lexical forms
- the old password that SQL Server needs
- CQL's batches, which the grammar of usql now keeps as one statement
(`BEGIN BATCH`, `BEGIN UNLOGGED BATCH` and `BEGIN COUNTER BATCH`). A batch that
is not finished when `\g` is typed goes to the server as it is, and fails there.
- what happens to a trailing semicolon
- the case of column names
- the placeholder of the INSERT of `\copy`, whether a write needs autocommit,
whether every statement is a query, and the name of a product with no
statement for its version
- scanning a column into the type that the driver reports (the MySQL family and
Databend)
What describes the DSN moved to dburl: the defaults of the MySQL family
(`parseTime`, `loc` and `sql_mode`), SQLite (`loc`), Cassandra (`timeout`) and
Couchbase (`txtimeout`), which a URL can now override. A MySQL or SQLite URL that
writes `loc=UTC` keeps it.
`Process` is gone. The drivers that used it only to run every statement as a
query now come from dbmeta (`EveryStatementIsAQuery`). `BatchAsTransaction` and
`LexerName` are gone.
### Drivers
- postgres:// opens through pgx. lib/pq is the libpq driver.
- Cassandra moves to `github.com/xo/cassandra`, and its scheme is
`cassandra`, with `cql` as an alias (W45).
- Couchbase and Databend move to `github.com/xo/dbimp`.
- BigQuery, Cosmos DB, Athena, Databricks, Spanner, Snowflake and Exasol move to
`github.com/xo/dbimp` (W49, W50, W56). Athena's scheme is `athena`, with
`awsathena`, `s3` and `aws` as aliases. The URLs of Athena, Cosmos DB,
Databricks, Snowflake and Exasol changed: `athena://key:secret@host/db`,
`cosmos://user:key@account/db/container`, `databricks://token:pat@workspace/warehouse`,
`snowflake://user:key@host/database/schema?role=r` (the password is the private
key as base64url PKCS8 DER) and `exasol://user:password@host:8563/schema`.
Every earlier URL must change. Cosmos DB reads only. They ran through usql on
the hosted services, or their emulators, on 2026-10-11 (W55); check each again
by hand before the release: a query, a wrong credential, `\dt`, `\d` and `\copy`.
- New drivers:
- CockroachDB, CrateDB, Neo4j, InfluxDB and InfluxQL
- SurrealDB and ArangoDB
- Pinot, rqlite and libSQL
- Drill, Solr, Elasticsearch, OpenSearch and Druid, all read only
- DynamoDB, which is dbimp's driver in place of godynamo, so the scheme
`godynamo` goes and the URL needs a key and a secret (W43)
- Removed drivers:
- mymysql, adodb, sapase and ignite
- ql, Netezza, RamSQL and Dameng
- godror, which needed cgo and Oracle's client library. Use the oracle
driver, which is pure Go.
- Db2's ODBC setup in `contrib`
- flightsql moves to arrow-go v18, maxcompute to the ODPS SDK, and hive to
gohive v2.
- Avatica moves to `github.com/xo/dbimp` (W39). Its URL is
`avatica://user:password@host:port` and not `http://`. Check it against
the standalone server and Phoenix.
- ClickHouse moves to `github.com/xo/dbimp` (W42). It speaks HTTP only, on
port 8123 or, with `tls=true`, 8443. The native protocol and port 9000 are
gone, and so are the `+http` and `+https` transports and the options of
`clickhouse-go`, such as `secure=true`.
- Trino and Presto move to `github.com/xo/dbimp/trino` (W40). A URL with no
user connects as `user`. `trs://` and `trinos://` are gone. Presto has
`\copy` now.
- ODBC moves to `github.com/xo/odbc` (W40), which needs no cgo, and so it is
in `most` and no longer in `all`. Its URL is
`odbc+://user:pass@host:port/db` with the settings of the ODBC
driver in the query. dburl no longer adds a port, a host or a ServiceName,
and a URL with no driver is an error.
- MaxCompute, Tablestore and YDB choose TLS with `tls=true` (W41), and
`yds://` and `ydbs://` are gone. MaxCompute and Tablestore use TLS by
default.
- The registry is keyed by the scheme of dburl.
- A driver goes in the group `most` when its scheme names a dialect and it
needs no cgo. Pinot, rqlite, libSQL, Drill, Solr, Elasticsearch, OpenSearch,
Druid and DynamoDB are in `most`.
### Other
- `size.sh` measures what each driver adds to the binary (W32).
- The Releases badge drops `sort=semver`, on which shields.io times out.
- The README ends with the bar of links to the sister projects.
- The backlog gains W28 to W38.
## What to check by hand
Use dbmeta's `dbrun` to start a database: `DBMETA_OWNER_NAME=usql go run
./cmd/dbrun start NAME`, in dbmeta's `test` directory. `dbrun dsn --json
NAME` gives the URL. Stop each database when its checks are done. Start no
more than a few at a time.
### Connecting and the banner
For every driver in the build, check that it connects, and that the banner
names the database and its release. For the new drivers, check one query and
one error, and a wrong password:
- CockroachDB, CrateDB, Neo4j, InfluxDB and InfluxQL
- SurrealDB and ArangoDB
- Pinot, rqlite and libSQL
- Druid, with `\dtS`, `\d` and a wrong password
- Drill, Solr, Elasticsearch, OpenSearch and DynamoDB, with a statement that
ends in `;`, and `\dt` on Drill, Solr and Elasticsearch
- Redshift, as a user who is not a superuser: `\dt`, `\d` and `\dv` work, and
`\dt+` shows no size and rows
- ClickHouse on its HTTP port, and Trino and Presto, with and without a user
- ODBC with a real driver, such as SQL Server's, PostgreSQL's and SQLite's,
after the crash is fixed
- Couchbase and Databend on dbimp
### Changing passwords
Run `\password` as an ordinary user and as an administrator on each database
that dbmeta can change a password on: cassandra, clickhouse, cockroachdb, cratedb,
databend, exasol, memsql, mysql, oracle, postgres, redshift, snowflake,
sqlserver, tidb and vertica.
- Check that the old password stops working and the new one works.
- On SQL Server, check that usql asks for the old password.
- Use a password that holds a quote, a backslash and a space.
### Input, for each set of lexical forms
The dialects of dbmeta use six sets of lexical forms. Check each set with one
of its databases:
| Forms like | Databases |
| --- | --- |
| postgres | cockroachdb, cratedb, duckdb, postgres, questdb, redshift, snowflake, vertica |
| cassandra | cassandra |
| mysql | memsql, mysql, surrealdb, tidb, vitess |
| sqlite | arangodb, impala, libsql, neo4j, rqlite, sqlite3, ydb |
| standard | clickhouse, couchbase, exasol, firebirdsql, hdb, influxql, oracle, presto, sqlserver, trino |
| plain | databend, hive, influxdb |
For each, check:
- a `;` inside a string, a quoted name, a comment, and, where the dialect
has them, a dollar quote and a backtick
- a statement over several rows, at a terminal and in a script
- `select 1 \g`, `\g` on an empty buffer, `\gx`, `\gset`, `\gexec`, and
`\g (format=csv) FILE`
- variables: `:x`, `:'x'`, `:"x"`, `:{?x}`, a value that holds a `;` or a
quote, and a `::` cast
- `\c` to a database with other forms, then a statement that the new forms
read differently
- CQL: `BEGIN BATCH` … `APPLY BATCH`
- Oracle: a statement with a trailing `;`, and a PL/SQL block that ends with
`END;`
- Trino and Presto: a statement with a trailing `;`
### Syntax highlighting
Check the color of keywords, strings, comments, variables and meta commands
for each set above, in two styles, at a light and a dark background. Check
that `\uset syntax_hl off` removes every color, that `\uset syntax_hl_format
terminal` draws 16 colors, and that `\uset syntax_hl_style nope` is an error.
Check a long statement of many rows, where each key parses the text again, for
a delay that you can feel. Check that `NO_COLOR=1` starts with highlighting off
(W58).
### The describe commands
For each database that dbmeta has a model for, run `\d`, `\dt`, `\dv`,
`\di`, `\ds`, `\df`, `\dn`, `\l`, `\dp`, `\du`, `\d NAME` on a table with an
index, a check, a foreign key and a trigger, `\sv`, `\sf`, and `\ss`. Check
upper case and quoted names on Oracle, and on a database that folds to
lower case. Check `\uset echo_hidden on` and `noexec`. On a database with no
model, check that each command says it is not supported.
### The line editor
At a terminal:
- an entry over several rows, and moving between its rows
- Ctrl-C on an open entry, Ctrl-D, and Ctrl-D on an empty entry
- the history across a restart, with and without an old history file
- tab completion of a meta command, a table, a column, a file, and a
variable
- `\prompt`, `\prompt -password`, `\password` and a password prompt at
connect
- `\e` and `\e FILE`, with editor set
- help, quit and exit as the first word of an entry
- a terminal that is resized while an entry is open
### Scripts, pipes and the command line
- `-c` with a statement, with a meta command, and with several of each
- `-f FILE`, `\i`, `\ir`, a file that ends without a `;`, and Windows line
endings
- input from a pipe, output to a pipe, and `-o FILE`
- `-1` with a file, and on_error_stop on and off
- `--set`, `--variable`, the RC file and `config.yaml` with named
connections
- `\copy` between two databases, and into a database with no transactions,
such as rqlite
### The meta commands
Run each meta command in `\?` once, with its options. Check the commands
that read their arguments in a new way most: `\set`, `\echo`, `\o`, `\g`,
`\copy`, `\c`, `\cset`, `\prompt`, `\!` and `\setenv`.
### Builds and packages
- `go build -tags all`, the default build, and `-tags no_base`
- `./size.sh -g most`, compared with W32
- the static builds of `build.sh`, for amd64 and arm64
- macOS, Windows and Linux
- the release archives, the Homebrew formula, the AUR packages and the
Docker image
---
### Decisions D001 Every Xo Repository Is Set Up For Coding Agents The Same Way (`docs/decisions/D001-every-xo-repository-is-set-up-for-coding-agents-the-same-way.md`)
# D1. Every xo repository is set up for coding agents the same way
Status: Decided.
Ken decided on 2026-09-27 that every repository in the `xo` namespace is set
up for coding agents the same way. dbmeta records the standard as its D110.
This decision adopts it for usql.
## The files
In the root:
- `README.md`, for a person who finds the project.
- `AGENTS.md`, which holds the rules for a coding agent. Codex and the other
agents read this file.
- `CLAUDE.md`, which holds one line, `@AGENTS.md`. Claude Code reads
`CLAUDE.md`, and the line imports `AGENTS.md`, so that every agent reads the
same rules. It is a file and not a symbolic link, because a Windows checkout
writes a link as a small text file.
- `CONTRIBUTING.md`, for a person who changes the project. It has an Agent
skills section with the command that installs them.
- `skills-lock.json`, which names the source of each skill.
- `.gitignore`, which ignores `.claude/settings.local.json`, the Claude Code
permissions of one person.
- `.gitattributes`, which holds `* text=auto eol=lf`.
In `docs/`:
- `PLAN.md`, which holds the plan and the open questions for Ken.
- `decisions/`, one file per decision, with an index. D2 says why.
- `BACKLOG.md`, which holds the work that is known and not done.
## The skills
The repository commits `simple-english` and `go-pedantry`. Each is an
ordinary folder in `.agents/skills/` and in `.claude/skills/`,
installed with `--copy`. `TestSkillsAreCopies` fails on a link, on a missing
copy and on two copies that differ. Ken installs the skills himself.
## The standing rules
`AGENTS.md` opens with three rules that are the same in every repository:
1. Stage changes for review. Commit and push only when Ken says so.
2. Load `simple-english` before writing any text that a person reads: project
documentation, a code comment, an error message or a commit message.
3. Load `go-pedantry` before writing or reviewing Go code. A rule of the
project wins where the two conflict.
Until now each agent learned the first rule from Ken and kept it in its own
memory, so an agent new to the repository did not know it.
## What changed in usql
`CLAUDE.md` moved to `AGENTS.md`, which now opens with the standing rules, and
`CLAUDE.md` holds the import. `TestClaudeImportsAgents` checks it. The root
holds four documents, not three, and `TestTheRootHoldsFourDocuments` checks
that. `.gitattributes`, `docs/PLAN.md` and this folder are new.
usql had no decision log before, so this is its first decision. `dburl` and
`dbmeta` number their decisions the same way. A reference to one of theirs
names the repository, as in "dburl's D22", and a bare number means usql's own.
---
### Decisions D002 A Large Project Keeps One File Per Decision (`docs/decisions/D002-a-large-project-keeps-one-file-per-decision.md`)
# D2. A large project keeps one file per decision
Status: Decided.
Ken decided on 2026-09-27 that a large project keeps each decision in a file
of its own, because a markdown file many thousands of lines long is not
readable. dbmeta records it as its D111, and names usql among the large
projects.
## The layout
Each decision is `docs/decisions/D-.md`, with the number in three
digits so that a listing sorts in order. The file opens with its number and
title, a blank line, and its status:
# D1. Every xo repository is set up for coding agents the same way
Status: Decided.
`docs/decisions/README.md` is the index, and GitHub shows it as the page of
the folder. `docs/PLAN.md` keeps the plan and the open questions.
A decision that changes an earlier one says so in its status, as "Amends D1",
and the earlier one says it back, as "Amended by D3". Each file opens with its
status, so a reader sees an amendment before anything else.
## The tests
`TestTheDecisionIndexIsComplete` checks every row of the index against the
file, title and status alike, and prints the row to add.
`TestAnAmendmentPointsBothWays` fails unless both decisions name each other in
their status. Both read the folder, and the file names and headings must
follow the layout above.
---
### Decisions D003 No Metadata Reader Is Written In Usql (`docs/decisions/D003-no-metadata-reader-is-written-in-usql.md`)
# D3. No metadata reader is written in usql
Status: Decided.
Ken decided on 2026-09-27 that all database metadata moves into `dbmeta`, so
no metadata reader is written in usql. The question came up for the SurrealDB
driver, which has none.
This extends the freeze that W21 records. On 2026-09-26 Ken decided that
`drivers/metadata` takes no more changes before the migration to `dbmeta`.
That covered the existing readers. This decision covers new ones too.
## What it means for a driver
- A new driver registers no metadata reader. Its describe commands, such as
`\d`, `\dt`, `\l` and `\dp`, report that they are not supported until
`dbmeta` answers them. D4 later removed usql's readers altogether.
- A gap in an existing reader, such as a missing `CatalogReader` or
`PrivilegeSummaryReader`, is fixed in `dbmeta`, not here. W18 measured those
gaps, and it is superseded by W21, which holds the migration.
- Step 5 of `docs/DRIVER.md` no longer asks for a reader. It says where the
metadata for a new database goes instead.
The SurrealDB driver is the first driver added under this rule.
---
### Decisions D004 Usql Reads All Metadata From Dbmeta At Once (`docs/decisions/D004-usql-reads-all-metadata-from-dbmeta-at-once.md`)
# D4. usql reads all metadata from dbmeta, at once
Status: Decided.
Ken decided on 2026-09-30 how usql moves onto dbmeta, which W21 holds. He
made three choices.
## All at once, not driver by driver
`docs/DBMETA.md` planned the move one driver at a time, keeping usql's own
readers for a database that dbmeta did not model. Ken chose to remove them all
at once, because D3 and his rule that database knowledge belongs in dbmeta
point there.
So `drivers/metadata` holds no query of its own. It picks dbmeta's model by
`URL.Dialect`, runs dbmeta's queries, and prints the answers. The readers that
went are the information schema, PostgreSQL, Oracle, MySQL and Impala ones in
`drivers/metadata`, and the SQLite, SQL Server, ClickHouse, Trino and DuckDB
ones in their driver packages. `drivers.Driver` loses `NewMetadataReader`,
`NewMetadataWriter`, `NewCompleter`, `User` and `ChangePassword`.
The version, the current user and the statement that changes a password come
from dbmeta too, for every database it models. A driver keeps its own
`Version` only for a database that dbmeta has no model for.
The cost is that a database with no dbmeta model answers none of the describe
commands. `noModel` in `metadata_test.go` lists each one and why, and
`TestEveryDriverReadsThroughDbmeta` fails when a driver is missing from it or
has gained a model.
## A local go.work, not go.mod
dbmeta has no release tag. usql builds against a dbmeta checkout beside it
through `go.work`, as `docs/DBMETA.md` decided, and `go.work` is not
committed. So the change is not committed until dbmeta is tagged. Then
`go.mod` requires that tag and the workspace goes.
## Follow dbmeta and psql
usql prints dbmeta's answers the way psql prints the same command, rather
than keeping its old output byte for byte. The recorded output in
`drivers/testdata` and `testdata/cli` was recorded again from the new code.
---
### Decisions D005 A Driver Field That Describes The Database Moves To Dbmeta (`docs/decisions/D005-a-driver-field-that-describes-the-database-moves-to-dbmeta.md`)
# D5. A driver field that describes the database moves to dbmeta
Status: Decided.
Ken decided on 2026-09-30 which fields of `drivers.Driver` move to dbmeta.
The rule is that a field which says something about the database, and alters
what usql does or how it uses a dialect's queries, belongs in dbmeta, keyed by
the dialect. A field that describes a Go driver, a URL or usql's own interface
stays. dbmeta reviewed every field with Gemini and DeepSeek first.
## What moves to dbmeta
- The grammar of the dialect: `AllowDollar`, `AllowMultilineComments`,
`AllowCComments`, `AllowHashComments` and `AllowBacktick`.
- `RequirePreviousPassword`, which says the statement that changes a password
takes the old one, as SQL Server's does.
- `BatchQueryPrefixes`, the statements that begin and end a batch, as CQL's
`BEGIN BATCH` and `APPLY BATCH`.
- The rule for a trailing semicolon, the part of `Process` that belongs to the
database, such as Oracle stripping it unless the statement ends with `END;`.
- How the dialect folds an unquoted name, which usql uses for patterns and to
lower-case column names for display. `LowerColumnNames` is derived from it
rather than set per driver.
- The handling of ODBC's `usql_trim` key and of csvq's `SHOW` statements.
dbmeta added all but the last as its D143, and usql reads them from there.
The grammar, `RequirePreviousPassword`, `BatchQueryPrefixes` and
`LowerColumnNames` are gone from `drivers.Driver`, and so are the `Process`
hooks of Oracle, Trino and Presto. `LowerColumnNames` is true for a database
that folds to upper case, which adds SAP HANA, Firebird and Snowflake to the
Oracle and Exasol drivers that set it.
A database that dbmeta has no model for allows only block comments. Nearly
every such driver set that and nothing more. A wider default is not safe,
because a hash comment would swallow a SQL Server `#temp` table over ODBC.
The last item waits. Ken decided on 2026-09-30 that the `Process` hooks of
ODBC, csvq, Athena and Cosmos are removed rather than kept in usql. Their
statement handling belongs in dbmeta's syntax options, applied automatically,
and that needs a larger design. W28 holds it, and the possible removal of
ODBC, Athena and Cosmos, which cannot be tested against a server.
With those hooks gone, a `Process` hook did one other thing: run every
statement as a query. The `AlwaysQuery` field replaces the `Process` field
for that. It says how usql runs a statement, and it stays in usql.
## What stays in usql
- `LexerName` becomes a map in usql from the dialect to a Chroma lexer, in
`drivers/drivers.go`, because the dialect already names the product.
- `Copy`. Its generic `INSERT` takes its placeholder from dbmeta, and a native
bulk path such as pgx's `CopyFrom` stays in its driver.
- `Name`, `UseColumnTypes`, `ForceParams`, `Open`, `IsPasswordErr`, `Err`,
`ColumnTypes`, `ConvertBytes`, `ConvertMap`, `ConvertSlice`,
`ConvertDefault`, `RowsAffected`, and `Version` for a database that dbmeta
has no model for.
## A rarely used database does not shape usql
`BatchAsTransaction` is removed, and the ql driver with it. Only ql set the
field, and it made usql run a batch in a transaction for that one driver. Ken
decided that a rarely used database must not force logic or architecture into
usql. ql was an unfinished experiment. dburl drops its scheme as its D42.
---
--- METRICS ---
- Files Extracted: 13
- Estimated Token Budget: ~69814 tokens
- Canonical Reference: https://codewiki.google/github.com/xo/USQL
