# Technical Documentation: dolthub/dolt
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/dolthub/dolt)
### Overview (`README.md`)
# Dolt is Git for Data!
Dolt is a SQL database that you can fork, clone, branch, merge, push
and pull just like a Git repository.
Connect to Dolt just like any MySQL database to read or modify schema
and data. Version control functionality is exposed in SQL via system
tables, functions, and procedures.
Or, use the Git-like command line interface to import CSV files, commit
your changes, push them to a remote, or merge your teammate's changes.
All the commands you know for Git work exactly the same for Dolt.
Git versions files. Dolt versions tables. It's like Git and MySQL had a
baby.
We also built [DoltHub](https://www.dolthub.com), a place to share
Dolt databases. We host public data for free. If you want to host
your own version of DoltHub, we have [DoltLab](https://www.doltlab.com).
If you want us to run a Dolt server for you, we have [Hosted Dolt](https://hosted.doltdb.com).
Prefer Postgres instead of MySQL? Try [Doltgres](https://github.com/dolthub/doltgresql), now
in its Beta release.
[Join us on Discord](https://discord.com/invite/RFwfYpu) to say hi and
ask questions, or [check out our roadmap](https://dolthub.com/docs/other/roadmap)
to see what we're building next.
# Video Introduction
[](https://www.youtube.com/watch?v=H2iZy0Cme10)
# What's it for?
Lots of things! Dolt is a generally useful tool with countless
applications. But if you want some ideas, [here's how people are using
it so far](https://dolthub.com/blog/2024-10-15-dolt-use-cases/).
Dolt powers [Beads](https://github.com/gastownhall/beads) and
[Gas Town](https://github.com/steveyegge/gastown). It's the best database
for agent memory, especially as you move up the ladder to multi-agent and
multi-machine workflows.
# Dolt CLI
The `dolt` CLI has the same commands as `git`, with some extras.
```
/* Detailed source-code truncated for AI context efficiency. */
```
# Installation
Dolt is a single ~103 megabyte program.
```bash
dolt $ du -h /Users/timsehn/go/bin/dolt
103M /Users/timsehn/go/bin/dolt
```
It's really easy to install. Download it and put it on your `PATH`.
We have a bunch of ways to make this even easier for most platforms.
## From Latest Release
To install on Linux or Mac based systems run this command in your
terminal:
```
sudo bash -c 'curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | bash'
```
This will download the latest `dolt` release and put it in
`/usr/local/bin/`, which is probably on your `$PATH`.
The install script needs sudo in order to put `dolt` in `/usr/local/bin`. If you don't have root
privileges or aren't comfortable running a script with them, you can download the dolt binary
for your platform from [the latest release](https://github.com/dolthub/dolt/releases), unzip it,
and put the binary somewhere on your `$PATH`.
### Linux
#### Arch Linux
Dolt is packaged in the official repositories for Arch Linux.
```
pacman -S dolt
```
### Mac
#### Homebrew
Dolt is on Homebrew, updated every release.
```
brew install dolt
```
#### MacPorts
On macOS, Dolt can also be installed via a [community-managed port](https://ports.macports.org/port/dolt/) via [MacPorts](https://www.macports.org):
```sh
sudo port install dolt
```
### Windows
Download the latest Microsoft Installer (`.msi` file) in
[releases](https://github.com/dolthub/dolt/releases) and run
it.
For information on running on Windows, see [here](https://dolthub.com/docs/introduction/installation/windows).
#### Chocolatey
You can install `dolt` using [Chocolatey](https://chocolatey.org/):
```sh
choco install dolt
```
#### Docker
There are following official Docker images for Dolt:
* [`dolthub/dolt`](https://hub.docker.com/r/dolthub/dolt) for running Dolt
as CLI tool.
* [`dolthub/dolt-sql-server`](https://hub.docker.com/r/dolthub/dolt-sql-server) for running Dolt in server mode.
## From Source
Make sure you have Go installed, and that `go` is in your path. Dolt has a dependency on [cgo](https://pkg.go.dev/cmd/cgo), so you will need a working C compiler and toolchain as well.
Clone this repository and cd into the `go` directory. Then run:
```
go install ./cmd/dolt
```
The output will be in `$GOPATH/bin`, which defaults to `~/go/bin`. To test your build, try:
```
~/go/bin/dolt version
```
# Configuration
Verify that your installation has succeeded by running `dolt` in your
terminal.
```
$ dolt
Valid commands for dolt are
[...]
```
Configure `dolt` with your user name and email, which you'll need to
create commits. The commands work exactly the same as git.
```
$ dolt config --global --add user.email YOU@DOMAIN.COM
$ dolt config --global --add user.name "YOUR NAME"
```
# Getting started
## Navigate to the directory where you would like your data stored
Dolt needs a place to store your databases. I'm going to put my databases in `~/dolt`.
```bash
% cd ~
% mkdir dolt
% cd dolt
```
Any databases you create will be stored in this directory. So, for this example, a directory named `getting_started` will be created here once you run `create database getting_started`. Navigating to `~/dolt/getting_started` will allow you to access this database using the Dolt command line.
NOTE: For this example, the `getting_started` directory will be created after you run `create database getting_started;` in a SQL shell in the [Create a schema section](#create-a-schema). Don't do anything except make the directory and navigate to it just yet.
## Start a MySQL-compatible database server
Dolt ships with a MySQL compatible database server built in. To start it you use the command `dolt sql-server`. Running this command starts the server on port 3306.
```bash
dolt sql-server
Starting server with Config HP="localhost:3306"|T="28800000"|R="false"|L="info"
```
Your terminal will just hang there. This means the server is running. Any errors will be printed in this terminal. Just leave it there and open a new terminal.
## Connect with a MySQL client (up to version 8.4)
In the new terminal, we will now connect to the running database server using a client. Dolt also ships with a MySQL compatible client.
```bash
% dolt -u root -p "" sql
# Welcome to the Dolt MySQL client.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
mysql>
```
In the other terminal where you ran `dolt sql-server`, you'll see the following log line.
```
2022-06-06T13:14:32-07:00 INFO [conn 1] NewConnection {DisableClientMultiStatements=false}
```
You are connected!
While we're here let's grab a copy of MySQL so we can connect with that client. Head over to the [MySQL Getting Started](https://dev.mysql.com/doc/mysql-getting-started/en/) documentation and install MySQL on your machine. I used [Homebrew](https://brew.sh/) to install MySQL on my Mac: `brew install mysql@8.4`. Alternatively, you can install only the client component by running `brew install mysql-client@8.4`.
NOTE: Make sure you install a MySQL 8.4 release. MySQL 8.4 is the current Long Term Support (LTS) release, meaning this is the stable and supported version of MySQL. MySQL 9.0 is also available, but is an "innovation" release, meaning it has more recent changes and features, but may not be as stable as the LTS release. The 9.0 release changes authentication support and isn't able to connect to a Dolt SQL server by default. You can install MySQL 8.4 with Homebrew by running `brew install mysql@8.4`. If you do want to use MySQL-9.0, read [our post on how to configure Dolt for `caching_sha2_password` authentication](https://www.dolthub.com/blog/2024-12-11-mysql9-and-caching-sha2-auth-support/).
MySQL comes with a MySQL server called `mysqld` and a MySQL client called `mysql`. You're only interested in the client. After following the instructions from MySQL's documentation, make sure you have a copy of the `mysql` client on your path:
```bash
% mysql --version
mysql Ver 8.0.29 for macos12.2 on x86_64 (Homebrew)
```
Now, to connect the `mysql` client to Dolt, you are going to force the MySQL client through the TCP interface by passing in a host and port. The default is the socket interface which Dolt supports, but is only available on `localhost`. So, it's better to show off the TCP interface. The MySQL client also requires you specify a user, in this case `root`.
```bash
% mysql --host 127.0.0.1 --port 3306 -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9-Vitess
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
```
Again, to ensure the client actually connected, you should see the following in the `dolt sql-server` terminal
```
2022-06-06T13:26:55-07:00 INFO [conn 2] NewConnection {DisableClientMultiStatements=false}
```
As you can see, Dolt supports any MySQL-compatible client. Dolt ships with a client but you can use any MySQL client, like the one that comes with MySQL.
## Create a schema
Now we're actually ready to do something interesting. I'll stay in the `mysql` client and execute the following SQL statements to create a database called `getting_started`. The `getting_started` database will have three tables: `employees`, `teams`, and `employees_teams`.
```
mysql> create database getting_started;
Query OK, 1 row affected (0.04 sec)
mysql> use getting_started;
Database changed
mysql> create table employees (
id int,
last_name varchar(255),
first_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.01 sec)
mysql> create table teams (
id int,
team_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.00 sec)
mysql> create table employees_teams(
team_id int,
employee_id int,
primary key(team_id, employee_id),
foreign key (team_id) references teams(id),
foreign key (employee_id) references employees(id));
Query OK, 0 rows affected (0.01 sec)
mysql> show tables;
+---------------------------+
| Tables_in_getting_started |
+---------------------------+
| employees |
| employees_teams |
| teams |
+---------------------------+
3 rows in set (0.00 sec)
```
Dolt supports foreign keys, secondary indexes, triggers, check constraints, and stored procedures. It's a modern, feature-rich SQL database.
## Make a Dolt commit
It's time to use your first Dolt feature. We're going to make a Dolt [commit](https://dolthub.com/docs/concepts/dolt/commits). A Dolt commit allows you to time travel and see lineage. Make a Dolt commit whenever you want to restore or compare to this point in time.
Dolt exposes version control functionality through a Git-style interface. On the command line, Dolt commands map exactly to their Git equivalent with the targets being tables instead of files. In SQL, Dolt exposes version control read operations as [system tables](https://dolthub.com/docs/sql-reference/version-control/dolt-system-tables) and version control write operations as [stored procedures](https://dolthub.com/docs/sql-reference/version-control/dolt-sql-procedures).
The naming of the system tables and stored procedures follows the `dolt_` pattern. So `dolt add` on the CLI becomes `dolt_add` as a stored procedure. Passing options also follows the command line model. For instance, to specify tables to add, send the table names in as options to the `dolt_add` procedure. For named arguments like sending a message into the `dolt_commit` command use two arguments in sequence like `('-m', 'This is a message')`. If you know Git, the version control procedures and system tables should feel familiar.
So, we add and commit our new schema like so.
```
mysql> call dolt_add('teams', 'employees', 'employees_teams');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.03 sec)
mysql> call dolt_commit('-m', 'Created initial schema');
+----------------------------------+
| hash |
+----------------------------------+
| ne182jemgrlm8jnjmoubfqsstlfi1s98 |
+----------------------------------+
1 row in set (0.02 sec)
mysql> select * from dolt_log;
+----------------------------------+-----------+-----------------+-------------------------+----------------------------+
| commit_hash | committer | email | date | message |
+----------------------------------+-----------+-----------------+-------------------------+----------------------------+
| ne182jemgrlm8jnjmoubfqsstlfi1s98 | Tim Sehn | tim@dolthub.com | 2022-06-07 16:35:49.277 | Created initial schema |
| vluuhvd0bn59598utedt77ed9q5okbcb | Tim Sehn | tim@dolthub.com | 2022-06-07 16:33:59.531 | Initialize data repository |
+----------------------------------+-----------+-----------------+-------------------------+----------------------------+
2 rows in set (0.01 sec)
```
There you have it. Your schema is created and you have a Dolt commit tracking the creation, as seen in the `dolt_log` system table.
Note, a Dolt commit is different than a standard SQL transaction `COMMIT`. In this case, I am running the database with [`AUTOCOMMIT`](https://dev.mysql.com/doc/refman/5.6/en/innodb-autocommit-commit-rollback.html) on, so each SQL statement is automatically generating a transaction `COMMIT`. If you want system to generate a Dolt commit for every transaction use the system variable, [`@@dolt_transaction_commit`](https://dolthub.com/docs/sql-reference/version-control/dolt-sysvars#dolt_transaction_commit).
## Insert some data
Now, I'm going to populate the database with a few employees here at DoltHub. Then, I'll assign the employees to two teams: engineering and sales. The CEO wears many hats at a start up so he'll be assigned to multiple teams.
```
mysql> insert into employees values
(0, 'Sehn', 'Tim'),
(1, 'Hendriks', 'Brian'),
(2, 'Son','Aaron'),
(3, 'Fitzgerald', 'Brian');
Query OK, 4 rows affected (0.01 sec)
mysql> select * from employees where first_name='Brian';
+------+------------+------------+
| id | last_name | first_name |
+------+------------+------------+
| 1 | Hendriks | Brian |
| 3 | Fitzgerald | Brian |
+------+------------+------------+
2 rows in set (0.00 sec)
mysql> insert into teams values
(0, 'Engineering'),
(1, 'Sales');
Query OK, 2 rows affected (0.00 sec)
mysql> insert into employees_teams values
(0,0),
(1,0),
(2,0),
(0,1),
(3,1);
ERROR 1452 (HY000): cannot add or update a child row - Foreign key violation on fk: `rv9ek7ft`, table: `employees_teams`, referenced table: `teams`, key: `[2]`
```
Oops, I violated a constraint. It looks like I created the table with teams before employees. You should always specify your columns when you insert, not rely on natural ordering. Serves me right! Dolt comes with the full power of a modern SQL relational database to ensure data integrity.
```
mysql> insert into employees_teams(employee_id, team_id) values
(0,0),
(1,0),
(2,0),
(0,1),
(3,1);
Query OK, 5 rows affected (0.01 sec)
mysql> select first_name, last_name, team_name from employees
join employees_teams on (employees.id=employees_teams.employee_id)
join teams on (teams.id=employees_teams.team_id)
where team_name='Engineering';
+------------+-----------+-------------+
| first_name | last_name | team_name |
+------------+-----------+-------------+
| Tim | Sehn | Engineering |
| Brian | Hendriks | Engineering |
| Aaron | Son | Engineering |
+------------+-----------+-------------+
3 rows in set (0.00 sec)
```
Looks like everything is inserted and correct. I was able to list the members of the engineering team using that three table `JOIN`. Dolt supports up to twelve table `JOIN`s. Again, Dolt is a modern SQL relational database paired with Git-style version control.
## Examine the diff
Now, what if you want to see what changed in your working set before you make a commit? You use the `dolt_status` and `dolt_diff_` system tables.
```
/* Detailed source-code truncated for AI context efficiency. */
```
As you can see from the diff I've added the correct values to the `employees` table. The values were previously `NULL` and now they are populated.
Let's finish off with another Dolt commit this time adding all effected tables using `-am`.
```
mysql> call dolt_commit('-am', 'Populated tables with data');
+----------------------------------+
| hash |
+----------------------------------+
| 13qfqa5rojq18j84d1n2htjkm6fletg4 |
+----------------------------------+
1 row in set (0.02 sec)
```
You can inspect the log using `dolt_log` and see which tables changed in each commit using an unscoped `dolt_diff`. Unscoped `dolt_diff` tells you whether schema, data, or both changed in that particular commit for the table.
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Oh no! I made a mistake.
Dolt supports undoing changes via `call dolt_reset()`. Let's imagine I accidentally drop a table.
```
mysql> drop table employees_teams;
Query OK, 0 rows affected (0.01 sec)
mysql> show tables;
+---------------------------+
| Tables_in_getting_started |
+---------------------------+
| employees |
| teams |
+---------------------------+
2 rows in set (0.00 sec)
```
In a traditional database, this could be disastrous. In Dolt, you're one command away from getting your table back.
```
mysql> call dolt_reset('--hard');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.01 sec)
mysql> show tables;
+---------------------------+
| Tables_in_getting_started |
+---------------------------+
| employees |
| employees_teams |
| teams |
+---------------------------+
3 rows in set (0.01 sec)
```
Dolt makes operating databases less error prone. You can always back out changes you have in progress or rewind to a known good state. You also have the ability to undo specific commits using [`dolt_revert()`](https://dolthub.com/docs/sql-reference/version-control/dolt-sql-procedures#dolt_revert). Even if you accidentally run `drop database` on the wrong database, Dolt lets you undo that by calling the [`dolt_undrop()` stored procedure](https://dolthub.com/docs/sql-reference/version-control/dolt-sql-procedures#dolt_undrop).
## See the data in a SQL Workbench
Hate the command line? Let's use [Tableplus](https://tableplus.com/) to make some modifications. Tableplus is a free SQL Workbench. Follow the installation instructions from their website.
Now, to connect you must select MySQL as the connection type. Then enter a name for your connection, `getting_started` as your database, and `root` as your user.
Click connect and you'll be presented with a familiar database workbench GUI.
## Make changes on a branch
To make changes on a branch, I use the `dolt_checkout()` stored procedure. Using the `-b` option creates a branch, just like in Git.
Tableplus gives me the ability to enter a multiple line SQL script on the SQL tab. I entered the following SQL to checkout a branch, update, insert, delete, and finally Dolt commit my changes.
```SQL
call dolt_checkout('-b','modifications');
update employees SET first_name='Timothy' where first_name='Tim';
insert INTO employees (id, first_name, last_name) values (4,'Daylon', 'Wilkins');
insert into employees_teams(team_id, employee_id) values (0,4);
delete from employees_teams where employee_id=0 and team_id=1;
call dolt_commit('-am', 'Modifications on a branch');
```
Here's the result in Tableplus.
Back in my terminal, I cannot see the table modifications made in Tableplus because they happened on a different branch than the one I have checked out in my session.
```
mysql> select * from dolt_branches;
+---------------+----------------------------------+------------------+------------------------+-------------------------+----------------------------+
| name | hash | latest_committer | latest_committer_email | latest_commit_date | latest_commit_message |
+---------------+----------------------------------+------------------+------------------------+-------------------------+----------------------------+
| main | 13qfqa5rojq18j84d1n2htjkm6fletg4 | Tim Sehn | tim@dolthub.com | 2022-06-07 16:39:32.066 | Populated tables with data |
| modifications | uhkv57j4bp2v16vcnmev9lshgkqq8ppb | Tim Sehn | tim@dolthub.com | 2022-06-07 16:41:49.847 | Modifications on a branch |
+---------------+----------------------------------+------------------+------------------------+-------------------------+----------------------------+
2 rows in set (0.00 sec)
mysql> select active_branch();
+-----------------+
| active_branch() |
+-----------------+
| main |
+-----------------+
1 row in set (0.00 sec)
mysql> select * from employees;
+------+------------+------------+
| id | last_name | first_name |
+------+------------+------------+
| 0 | Sehn | Tim |
| 1 | Hendriks | Brian |
| 2 | Son | Aaron |
| 3 | Fitzgerald | Brian |
+------+------------+------------+
4 rows in set (0.00 sec)
```
I can query the branch no matter what I have checked out using SQL `as of` syntax.
```
mysql> select * from employees as of 'modifications';
+------+------------+------------+
| id | last_name | first_name |
+------+------------+------------+
| 0 | Sehn | Timothy |
| 1 | Hendriks | Brian |
| 2 | Son | Aaron |
| 3 | Fitzgerald | Brian |
| 4 | Wilkins | Daylon |
+------+------------+------------+
5 rows in set (0.01 sec)
```
If I'd like to see the diff between the two branches, I can use the `dolt_diff()` table function. It takes two branches and the table name as arguments.
```
mysql> select * from dolt_diff('main', 'modifications', 'employees');
+--------------+---------------+-------+---------------+-------------------------+----------------+-----------------+---------+-------------+-------------------------+-----------+
| to_last_name | to_first_name | to_id | to_commit | to_commit_date | from_last_name | from_first_name | from_id | from_commit | from_commit_date | diff_type |
+--------------+---------------+-------+---------------+-------------------------+----------------+-----------------+---------+-------------+-------------------------+-----------+
| Sehn | Timothy | 0 | modifications | 2022-06-07 16:41:49.847 | Sehn | Tim | 0 | main | 2022-06-07 16:39:32.066 | modified |
| Wilkins | Daylon | 4 | modifications | 2022-06-07 16:41:49.847 | NULL | NULL | NULL | main | 2022-06-07 16:39:32.066 | added |
+--------------+---------------+-------+---------------+-------------------------+----------------+-----------------+---------+-------------+-------------------------+-----------+
2 rows in set (0.00 sec)
```
As you can see, you have the full power of Git-style branches and diffs in a SQL database with Dolt.
## Make a schema change on another branch
I can also make schema changes on branches for isolated testing of new schema. I'm going to add a `start_date` column on a new branch and populate it.
```
mysql> call dolt_checkout('-b', 'schema_changes');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.01 sec)
mysql> alter table employees add column start_date date;
Query OK, 0 rows affected (0.02 sec)
mysql> update employees set start_date='2018-09-08';
Query OK, 4 rows affected (0.01 sec)
Rows matched: 4 Changed: 4 Warnings: 0
mysql> update employees set start_date='2021-04-19' where last_name='Fitzgerald';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from employees;
+------+------------+------------+------------+
| id | last_name | first_name | start_date |
+------+------------+------------+------------+
| 0 | Sehn | Tim | 2018-09-08 |
| 1 | Hendriks | Brian | 2018-09-08 |
| 2 | Son | Aaron | 2018-09-08 |
| 3 | Fitzgerald | Brian | 2021-04-19 |
+------+------------+------------+------------+
4 rows in set (0.00 sec)
mysql> call dolt_commit('-am', 'Added start_date column to employees');
+----------------------------------+
| hash |
+----------------------------------+
| pg3nfi0j1dpc5pf1rfgckpmlteaufdrt |
+----------------------------------+
1 row in set (0.01 sec)
```
Changing schema on a branch gives you a new method for doing isolated integration testing of new schema changes.
## Merge it all together
Let's assume all the testing of the new schema on the `schema_changes` branch and data on the `modifications` branch completed flawlessly. It's time to merge all our edits together onto `main`. This is done using the `dolt_merge` stored procedure.
```
mysql> call dolt_checkout('main');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.01 sec)
mysql> select * from dolt_status;
Empty set (0.00 sec)
mysql> call dolt_merge('schema_changes');
+--------------+
| no_conflicts |
+--------------+
| 1 |
+--------------+
1 row in set (0.01 sec)
mysql> select * from employees;
+------+------------+------------+------------+
| id | last_name | first_name | start_date |
+------+------------+------------+------------+
| 0 | Sehn | Tim | 2018-09-08 |
| 1 | Hendriks | Brian | 2018-09-08 |
| 2 | Son | Aaron | 2018-09-08 |
| 3 | Fitzgerald | Brian | 2021-04-19 |
+------+------------+------------+------------+
4 rows in set (0.00 sec)
```
Schema change successful. We now have start dates. Data changes are next.
```
mysql> call dolt_merge('modifications');
+--------------+
| no_conflicts |
+--------------+
| 1 |
+--------------+
1 row in set (0.02 sec)
mysql> select * from employees;
+------+------------+------------+------------+
| id | last_name | first_name | start_date |
+------+------------+------------+------------+
| 0 | Sehn | Timothy | 2018-09-08 |
| 1 | Hendriks | Brian | 2018-09-08 |
| 2 | Son | Aaron | 2018-09-08 |
| 3 | Fitzgerald | Brian | 2021-04-19 |
| 4 | Wilkins | Daylon | NULL |
+------+------------+------------+------------+
5 rows in set (0.00 sec)
```
Data changes successful as well. As you can see, I am now "Timothy" instead of "Tim", Daylon is added, and we all have start dates except for Daylon who was added on a different branch.
```
mysql> select first_name, last_name, team_name from employees
join employees_teams on (employees.id=employees_teams.employee_id)
join teams on (teams.id=employees_teams.team_id)
where team_name='Sales';
+------------+------------+-----------+
| first_name | last_name | team_name |
+------------+------------+-----------+
| Brian | Fitzgerald | Sales |
+------------+------------+-----------+
1 row in set (0.01 sec)
```
I'm also gone from the Sales Team. Engineering is life.
I have to commit all my changes because the last merge was not a fast-forward merge.
```
/* Detailed source-code truncated for AI context efficiency. */
```
Now, we have a database with all the schema and data changes merged and ready for use.
## Audit Cell Lineage
Which commit changed my first name? With Dolt you have lineage for every cell in your database. Let's use the `dolt_history_` and `dolt_diff_` to explore the lineage features in Dolt.
`dolt_history_` shows you the state of the row at every commit.
```
mysql> select * from dolt_history_employees where id=0 order by commit_date;
+------+-----------+------------+------------+----------------------------------+-----------+-------------------------+
| id | last_name | first_name | start_date | commit_hash | committer | commit_date |
+------+-----------+------------+------------+----------------------------------+-----------+-------------------------+
| 0 | Sehn | Tim | NULL | 13qfqa5rojq18j84d1n2htjkm6fletg4 | Tim Sehn | 2022-06-07 16:39:32.066 |
| 0 | Sehn | Timothy | NULL | uhkv57j4bp2v16vcnmev9lshgkqq8ppb | Tim Sehn | 2022-06-07 16:41:49.847 |
| 0 | Sehn | Tim | 2018-09-08 | pg3nfi0j1dpc5pf1rfgckpmlteaufdrt | Tim Sehn | 2022-06-07 16:44:37.513 |
| 0 | Sehn | Timothy | 2018-09-08 | vn9b0qcematsj2f6ka0hfoflhr5s6p0b | Tim Sehn | 2022-06-07 17:10:02.07 |
+------+-----------+------------+------------+----------------------------------+-----------+-------------------------+
4 rows in set (0.00 sec)
```
`dolt_diff_` allows you to filter the history down to only commits when the cell in question changed. In this case, I'm interested in the commits that are changing my first name. Note, there are two commits that changed my name because one is the original change and the second is the merge commit.
```
mysql> select to_commit,from_first_name,to_first_name from dolt_diff_employees
where (from_id=0 or to_id=0) and (from_first_name <> to_first_name or from_first_name is NULL)
order by to_commit_date;
+----------------------------------+-----------------+---------------+
| to_commit | from_first_name | to_first_name |
+----------------------------------+-----------------+---------------+
| 13qfqa5rojq18j84d1n2htjkm6fletg4 | NULL | Tim |
| uhkv57j4bp2v16vcnmev9lshgkqq8ppb | Tim | Timothy |
| vn9b0qcematsj2f6ka0hfoflhr5s6p0b | Tim | Timothy |
+----------------------------------+-----------------+---------------+
3 rows in set (0.01 sec)
```
Dolt provides powerful data audit capabilities down to individual cells. When, how, and why has each cell in your database changed over time?
# Additional Reading
Head over to [our documentation](https://dolthub.com/docs/introduction/what-is-dolt) now that you have a feel for Dolt. You can also read about what we've been working on in [our blog](https://www.dolthub.com/blog/).
# Security Policy
[Dolt's current security policy](https://github.com/dolthub/dolt/blob/main/SECURITY.md) is maintained in this repository. Please follow the disclosure instructions there. Please do not initially report security issues in this repository's public GitHub issues.
# Credits and License
Dolt relies heavily on open source code and ideas from the
[Noms](https://github.com/attic-labs/noms) project. We are very
thankful to the Noms team for making this code freely available,
without which we would not have been able to build Dolt so rapidly.
Dolt is licensed under the Apache License, Version 2.0. See
[LICENSE](https://github.com/dolthub/dolt/blob/main/LICENSE) for
details.
---
### Overview (`docker/README.md`)
# Dolt is Git for Data!
[Dolt](https://doltdb.com) is a SQL database that you can fork, clone, branch, merge, push
and pull just like a Git repository. Connect to Dolt just like any
MySQL database to run queries or update the data using SQL
commands. Use the command line interface to import CSV files, commit
your changes, push them to a remote, or merge your teammate's changes.
All the commands you know for Git work exactly the same for Dolt. Git
versions files, Dolt versions tables. It's like Git and MySQL had a
baby.
We also built [DoltHub](https://www.dolthub.com), a place to share
Dolt databases. We host public data for free. If you want to host
your own version of DoltHub, we have [DoltLab](https://www.doltlab.com). If you want us to run a Dolt server for you, we have [Hosted Dolt](https://hosted.doltdb.com).
[Join us on Discord](https://discord.com/invite/RFwfYpu) to say hi and
ask questions, or [check out our roadmap](https://dolthub.com/docs/other/roadmap)
to see what we're building next.
## What's it for?
Lots of things! Dolt is a generally useful tool with countless
applications. But if you want some ideas, [here's how people are using
it so far](https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/).
Learn more about Dolt use cases, configuration and guides to use dolt on our [documentation page](https://dolthub.com/docs/introduction/what-is-dolt).
# How to use this image
This image is for Dolt CLI, which has the same commands as `git`, with some extras. Running this image without any
arguments is equivalent to running `dolt` command locally.
```
/* Detailed source-code truncated for AI context efficiency. */
```
This image is useful for creating custom Docker Image using this image as base image.
---
### Overview (`docker/serverREADME.md`)
# Dolt is Git for Data!
[Dolt](https://doltdb.com) is a SQL database that you can fork, clone, branch, merge, push
and pull just like a Git repository. Connect to Dolt just like any
MySQL database to run queries or update the data using SQL
commands. Use the command line interface to import CSV files, commit
your changes, push them to a remote, or merge your teammate's changes.
All the commands you know for Git work exactly the same for Dolt. Git
versions files, Dolt versions tables. It's like Git and MySQL had a
baby.
We also built [DoltHub](https://www.dolthub.com), a place to share
Dolt databases. We host public data for free. If you want to host
your own version of DoltHub, we have [DoltLab](https://www.doltlab.com). If you want us to run a Dolt server for you, we have [Hosted Dolt](https://hosted.doltdb.com).
[Join us on Discord](https://discord.com/invite/RFwfYpu) to say hi and
ask questions, or [check out our roadmap](https://dolthub.com/docs/other/roadmap)
to see what we're building next.
## What's it for?
Lots of things! Dolt is a generally useful tool with countless
applications. But if you want some ideas, [here's how people are using
it so far](https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/).
# Dolt CLI
The `dolt` CLI has the same commands as `git`, with some extras.
```
/* Detailed source-code truncated for AI context efficiency. */
```
Learn more about Dolt use cases, configuration and guides to use dolt on our [documentation page](https://dolthub.com/docs/introduction/what-is-dolt).
# How to use this image
This image is for Dolt SQL Server, which is similar to the MySQL Docker image. Running this image without any arguments
is equivalent to running `dolt sql-server --host 0.0.0.0 --port 3306` command inside a Docker container.
To see all supported options for `dolt sql-server`, you can run the image with `--help` flag.
```shell
$ docker run dolthub/dolt-sql-server:latest --help
```
## Building the image
To build this image from source, use the `serverDockerfile`:
*WARNING* When building from source you cannot have any other folders that start with dolt in the same directory as
your workspace folder. This is because we use a wildcard to conditionally copy the dolt source folder into
the image. Other folders starting with dolt could cause the build to fail.
```shell
# Build with the latest Dolt version (automatically fetches the latest release)
$ docker build -f docker/serverDockerfile --build-arg DOLT_VERSION=latest -t dolt-sql-server:latest .
# Build with a specific Dolt version
$ docker build -f docker/serverDockerfile --build-arg DOLT_VERSION=1.59.7 -t dolt-sql-server:1.59.7 .
# Note: To run the local build replace `dolthub/dolt-sql-server:latest` with `dolt-sql-server:latest`
```
## Building from Source
To build from your local source code instead of downloading a pre-built binary:
```shell
# Build from local source code (requires workspace directory with source dependencies, e.g., dolt_workspace/)
$ docker build -f dolt/docker/serverDockerfile --build-arg DOLT_VERSION=source -t dolt-sql-server:source .
# Note: This should contain dolt/ at the minimum, any other repos need to be
# added via COPY in serverDockerfile.
$ docker run -e DOLT_ROOT_PASSWORD=secret2 -e DOLT_ROOT_HOST=% -p 3307:3306 dolt-sql-server:source
```
### Connect to the server in the container from the host system
From the host system, to connect to a server running in a container, we need to map a port on the host system to the port our sql-server is running on in the container.
We also need a user account that has permission to connect to the server
from the host system's address. By default, as of Dolt version 1.46.0, the `root` superuser is configured to only allow connections from localhost. This is a security feature to prevent unauthorized access to the server. If you don't want to log in to the container and then connect to your sql-server, you can use the `DOLT_ROOT_HOST` and `DOLT_ROOT_PASSWORD` environment variables to control how the `root` superuser is initialized. When the Docker container is started, it will ensure the `root` superuser is configured according to those environment variables.
In our example below, we're using `DOLT_ROOT_HOST` to override the host of the `root` superuser account to `%` in order to allow any host to connect to our server and log in as `root`. We're also using `DOLT_ROOT_PASSWORD` to override the default, empty password and specify a password for the `root` account. Setting a password is strongly advised for security when allowing the `root` account to connect from any host.
```bash
> docker run -e DOLT_ROOT_PASSWORD=secret2 -e DOLT_ROOT_HOST=% -p 3307:3306 dolthub/dolt-sql-server:latest
```
If we run the command above with -d or switch to a separate window we can connect with MySQL:
```bash
> mysql --host 0.0.0.0 -P 3307 -u root -p secret2
```
### Define configuration for the server
You can specify server configuration with commandline arguments, or you can use a YAML configuration file.
For commandline arguments, you can simply add arguments at the end of the docker command, as shown below.
```shell
$ docker run -p 3307:3306 dolthub/dolt-sql-server:latest -l debug --no-auto-commit
```
To use a configuration file, you can map a local directory to location in the container.
The special directory for server configuration is `/etc/dolt/servercfg.d/`. You can only have one `.yaml` configuration
file in this directory. If there are multiple, the default configuration will be used. If the location of
configuration file was `/Users/jennifer/docker/server/config.yaml`, this is how to use `-v` flag which mounts
`/Users/jennifer/docker/server/` local directory to `/etc/dolt/servercfg.d/` directory in the container.
```shell
$ docker run -p 3307:3306 -v /Users/jennifer/docker/server/:/etc/dolt/servercfg.d/ dolthub/dolt-sql-server:latest
```
The Dolt configuration and data directories can be configured similarly:
- The dolt configuration directory is `/etc/dolt/doltcfg.d/`
There should be one `.json` Dolt configuration file. It will replace the global Dolt configuration file in the
container.
- We set the location of where data to be stored to default location at `/var/lib/dolt/` in the container.
The data directory does not need to be defined in server configuration for container, but to store the data
on the host system, it can also be mounted to this default location.
```shell
$ docker run -p 3307:3306 -v /Users/jennifer/docker/databases/:/var/lib/dolt/ dolthub/dolt-sql-server:latest
```
If the run command errors out during the entrypoint process, you'll have to remove the created container from the last run.
## Environment Variables
The Dolt SQL Server image supports the following environment variables:
- `DOLT_ROOT_PASSWORD`: Sets the password for root (default: empty)
- `DOLT_ROOT_HOST`: Specifies a host for the root (default: localhost)
- `DOLT_DATABASE` / `MYSQL_DATABASE`: Creates a database with this name if it doesn't exist
- `DOLT_USER` / `MYSQL_USER`: Creates a user with this name if it doesn't exist
- `DOLT_PASSWORD` / `MYSQL_PASSWORD`: Sets the password for the user specified in `DOLT_USER`/`MYSQL_USER`
- `DOLT_USER_HOST` / `MYSQL_USER_HOST`: Specifies a host for the custom user (default: falls back to `DOLT_ROOT_HOST`, then localhost)
- `DOLT_SERVER_TIMEOUT`: Sets the server startup timeout in seconds (default: 300). A value of 0 means no timeout.
The user will be granted all privileges on the database specified by `DOLT_DATABASE`/`MYSQL_DATABASE` if provided.
The entrypoint runs this setup and any initialization scripts only when `.init_completed` is absent. It writes the marker after initialization succeeds, and subsequent starts skip these steps.
---
### Overview (`proto/README.md`)
Protobuf message and service definitions.
The Go bindings are generated by developers and placed in ../go/gen/.
After modifying a .proto file, run: `bazel run //:update` from the `proto/` subdirectory.
The //:update target also updates ../go/gen/fb/serial message implementations. Somewhat awkwardly, for the time being, the source for those lives in ../go/serial.
---
### Overview (`.github/workflows/README.md`)
# Dolt's GitHub Actions
This doc will provide context for the types of Workflows we use in this repository. This doc is not a comprehensive GitHub Actions tutorial. To familiarize yourself with GitHub Actions concepts and the terminology, please see the [documentation](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions).
Dolt uses GitHub Actions Workflows in four primary ways:
* To run continuous integration tests on pull requests and pushes to `main`
* To release and publish new Dolt assets
* To deploy various benchmarking jobs to contexts _other_ than GitHub Actions (like in a Kubernetes cluster, for example).
* To handle misc. [repository_dispatch](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#repository_dispatch) events triggered by external clients.
## Continuous Integration Workflows
Workflows prefixed with `ci-` are run on pull requests to `main`, though some run on pushes to `main` (after a pull request is merged). These workflows are synchronous and don't trigger any other workflows to run.
## Dolt Release Workflows
Workflows prefixed with `cd-` are used for releasing Dolt. Some of these workflows are asynchronous, meaning that they only perform part of a task before triggering the next part of a task to run in a _different_ workflow, sometimes in other GitHub repositories, using `repository_dispatch` events.
## Benchmarking Workflows
Benchmarking workflows are used as an interface for deploying benchmarking jobs to one of our Kubernetes Clusters. Workflows that deploy Kubernetes Jobs are prefixed with `k8s-` and can only be triggered with `repository_dispatch` events. Notice that benchmarking workflows, like `workflows/performance-benchmarks-email-report.yaml` for example, trigger these events using the `peter-evans/repository-dispatch@v2.0.0` Action.
These Kubernetes Jobs do not run on GitHub Actions Hosted Runners, so the workflow logs do not contain any information about the deployed Kubernetes Job or any errors it might have encountered. The workflow logs can only tell you if a Job was created successfully or not. To investigate an error or issue with a Job in our Kubernetes Cluster, see the debugging guide [here](https://github.com/dolthub/ld/blob/main/k8s/README.md#debug-performance-benchmarks-and-sql-correctness-jobs).
## Misc. Repository Dispatch Workflows
Some workflows perform single, common tasks and are triggered by `repository_dispatch` events. These include the `workflows/email-report.yaml` that emails the results of performance benchmarks to the team, or the `workflows/pull-report.yaml` that posts those same results to an open pull request. Workflows like these are triggered by external clients.
---
### Overview (`go/store/README.md`)
dolt/go/store
=============
The persistence layer for dolt. Based on [noms](https://github.com/attic-labs/noms/),
but adapted for storing table data.
---
### Overview (`integration-tests/bats/README.md`)
# BATS - Bash Automated Testing System #
BATS is used to integration test `dolt`. Our BATS tests started as a humble suite of integration tests. Over two years
of development the suite has grown to over 1,000 tests. When we find a customer facing bug in the `dolt` command line or
SQL implementation, we cover it with a BATS test. These tests are run on every `dolt` PR on Mac, Windows, and Linux using
GitHub Actions.
These tests are also useful documentation. If you are wondering how a certain command or feature works in practice,
using `grep` to find the appropriate BATS test can give you some simple examples of happy path and error case behavior.
The naming conventions for the test files have evolved over time. Generally, the files are named after the feature the
file intends to test. However, some of the early tests are named after the schema of the table they implement
ie. `1pk5col-ints.bats`. These files were implemented to reuse setup and teardown logic. This scheme was quickly
abandoned but the legacy remains.
If you find a bug in `dolt`, we would love a skipped bats test PR in addition to a GitHub issue.
# Running for yourself
1. Install bats.
```
npm install bats
```
2. Install `dolt` and its utilities.
```
cd go/cmd/dolt && go install . && cd -
cd go/store/cmd/noms && go install . && cd -
cd go/utils/remotesrv && go install . && cd -
````
3. Make sure you have `python3` installed.
This came with my Mac Developer Tools and was on my PATH.
4. `pip install mysql-connector-python`, `pip install pyarrow` and `pip install pandas`
I also needed this specific version on the python mysql.connector. `pip install mysql.connector` mostly worked but caused some SSL errors.
```
pip3 install mysql-connector-python
pip3 install pyarrow
pip3 install pandas
```
5. Install `parquet` and its dependencies
I used Homebrew on Mac to install `parquet`. You also need to install `hadoop` and set `PARQUET_RUNTIME_JAR` to get bats to work. Here's what I ended up running.
```
brew install parquet-cli
brew install hadoop
export PARQUET_RUNTIME_JAR=/opt/homebrew/opt/parquet-cli/libexec/parquet-cli-1.12.3-runtime.jar
```
6. Go to the directory with the bats tests and run:
```
bats .
```
This will run all the tests. Specify a particular .bats file to run only those tests.
## Here Docs
BATS tests in Dolt make extensive use of [Here Docs](https://en.wikipedia.org/wiki/Here_document).
Common patterns include piping SQL scripts to `dolt sql`:
```sql
dolt sql < data.csv
pk,c1,c2
1,1,1
2,2,2
DELIM
dolt table import -c -pk=pk my_table data.csv
```
## Skipped BATS
Various tests are skipped as TODOs and/or as documentation of known bugs. Eg:
```sql
@test "..." {
...
skip "this test is currently failing because..."
}
```
Skipped BATS can still be partially useful for testing as they execute normally up to `skip` statement.
## More Information
We published a [blog entry](https://www.dolthub.com/blog/2020-03-23-testing-dolt-bats/) on BATS with
more information and some useful tips and tricks.
---
### Overview (`integration-tests/compatibility/README.md`)
## Compatibility Tests
These tests attempt to ensure forward and backward compatibility for Dolt versions.
For each Dolt release version listed in `test_files/backward_compatible_versions.txt`, a legacy
repository is created using the corresponding release and populated with data from
`test_files/setup_repo.sh`. Then, using a Dolt client build at HEAD, a series of BATS tests are run
against each legacy repository.
To test forward compatibility, a repository is created and populated using Dolt built at HEAD. For
each Dolt release version listed in `test_files/forward_compatible_versions.txt`, the same BATS
tests are run against the repo created with Dolt at HEAD.
Then there is a third set of tests that test bidirectional compatibility, whether older and current
clients can alternate doing writes on the same DB withut anything breaking. These tests do their own
initialization and don't rely on the `setup_repo.sh`. Forward compatibility is the limiting factor
for these tests (fewer versions are forward compatible at any given time), so that's the versions we
do bidirectional testing against.
---
### Overview (`integration-tests/data-dump-loading-tests/README.md`)
## Data Dump Loading Test
We created tests for loading data dumps from mysqldump, and we run these tests through Github Actions
on pull requests.
These tests can be run locally using Docker. Before you can build the image, you also need to copy the go folder
into the integration-tests folder; unfortunately just symlinking doesn't seem to work. From the
integration-tests directory of the dolt repo, run:
```bash
$ cp -r ../go .
$ docker build -t data-dump-loading-tests -f DataDumpLoadDockerfile .
$ docker run data-dump-loading-tests:latest
```
The `docker build` step will take a few minutes to complete as it needs to install all of the
dependencies in the image.
Running the built container will produce output like:
```bash
$ docker run data-dump-loading-tests:latest
updating dolt config for tests:
Config successfully updated.
Config successfully updated.
Config successfully updated.
Config successfully updated.
Running data-dump-loading-tests:
1..2
ok 1 import mysqldump: empty database dump
ok 2 import mysqldump: a simple table dump
```
We are using MySQL Sakila Database dump file with few modifications (commented out some parts that are not supported yet) to fit Dolt.
These commented out parts can be found with 'UNSUPPORTED SYNTAX' in search bar in 'sakila_dump.sql' file. The stored functions and
procedures from the original database is not present in the dump file. Only one procedure is added to fit and test Dolt procedure
functionality.
The Sakila database dump file is the only dump file we have tests for. If there are more dump files to be added as needed for
more testing, we need to use different way to store those dump files.
Sakila Database License:
-- Sakila Sample Database Data
-- Version 1.2
-- Copyright (c) 2022, Oracle and/or its affiliates.
-- Redistribution and use in source and binary forms, with or without
-- modification, are permitted provided that the following conditions are
-- met:
-- * Redistributions of source code must retain the above copyright notice,
-- this list of conditions and the following disclaimer.
-- * Redistributions in binary form must reproduce the above copyright
-- notice, this list of conditions and the following disclaimer in the
-- documentation and/or other materials provided with the distribution.
-- * Neither the name of Oracle nor the names of its contributors may be used
-- to endorse or promote products derived from this software without
-- specific prior written permission.
-- THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-- IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-- PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
-- CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-- EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-- PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-- PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
-- LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
-- NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-- SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
---
### Overview (`integration-tests/go-sql-server-driver/README.md`)
go-sql-server-driver
====================
This is a driver and a test suite for tests interacting with `dolt sql-server`.
These tests describe a setup for the sql-server(s) which should be running, the
interactions which should be run against the servers, and the assertions which
should pass given those interactions.
This is meant to be more declarative and more robust than using `bats` for
these integration tests.
Something belongs in this package if it primarily tests interactions with the
exposed MySQL port on the sql-server.
Something belongs in `bats` if it primarily tests interactions with the `dolt`
binary itself. One example for the `dolt sql-server` command itself would be
testing config validation which results in exit codes or help text displayed.
---
### Overview (`integration-tests/mysql-client-tests/README.md`)
## MySQL Client Tests
This suite contains smoke tests for Dolt's MySQL and MariaDB client integrations.
It runs in GitHub Actions on pull requests.
## Run Locally With Docker
Build from the workspace root that contains the `dolt/` directory. The Dockerfile copies files using
paths like `dolt/go/...`, so the build context must include that root. You can skip all the commands by using the GoLand's built-in Docker extension.
```bash
$ cd
$ docker build -t mysql-client-tests -f dolt/integration-tests/mysql-client-tests/Dockerfile .
$ docker run mysql-client-tests:latest
```
The `docker build` step can take several minutes because it installs toolchains and
client dependencies for multiple languages.
Running the built container produces BATS output similar to:
```bash
$ docker run mysql-client-tests:latest
Updating dolt config for tests:
Config successfully updated.
Config successfully updated.
Config successfully updated.
Config successfully updated.
Running mysql-client-tests:
...
```
You can compile local directories by appending another `COPY` right before `go mod install` in the `dolt_build` stage. Make sure to copy the repository (i.e., `go-mysql-server`) into the stage's `/build/` directory. As long as `dolt` references the dependency, it'll automatically resolve.
```dockerfile
COPY dolt/go/go.mod /build/dolt/go/
# COPY go-mysql-server /build/go-mysql-server
WORKDIR /build/dolt/go/
RUN go mod download
```
## Target Specific Stages
When iterating on one area (for example MariaDB CLI behavior), use `--target` to avoid
rebuilding the full multi-stage image.
Build only the MariaDB client stage:
```bash
$ cd
$ docker build \
--target mariadb_clients \
-t mysql-client-tests:mariadb-clients \
-f dolt/integration-tests/mysql-client-tests/Dockerfile .
```
Inspect and validate that stage manually:
```bash
$ docker run --rm -it --entrypoint /bin/bash mysql-client-tests:mariadb-clients
# ls -la /usr/local
# /usr/local/mariadb-10.11/bin/mariadb --version
# /usr/local/mariadb-11.8/bin/mariadb --version
# ldd /usr/local/mariadb-11.8/bin/mariadb
```
---
### Overview (`integration-tests/orm-tests/README.md`)
## ORM Client Tests
These tests verify that various ORM libraries that support MySQL are compatible with Dolt. Ideally,
we use the same test suite that the ORM provides to test their support with MySQL, but in some
cases we may start with a smaller smoke test to get some quick, initial coverage.
These tests can be run locally using Docker from the root of the workspace directory that contains `dolt/`:
```bash
$ docker build -t orm-tests -f dolt/integration-tests/orm-tests/Dockerfile .
$ docker run orm-tests:latest
```
The `dolt` binary is built in a separate stage from the ORM test runtime. This speeds up first-time image builds with parallel stage building. Secondly, ORM-only changes will reuse the cached binary from the `dolt_build` stage.
You can also build other Dolt related dependencies from a local source into the Docker `dolt` binary. Copy the required source into the `dolt_build` stage's `build/` directory in the `orm-tests/Dockerfile`. The one caveat is that your repository must be placed in the same parent directory as `dolt`.
```dockerfile
COPY go-mysql-server /build/go-mysql-server
```
The line above has been commented out in the actual Dockerfile so you know the correct placement. You are responsible for updating `go.mod` with the `replace` directive in your local `dolt/` directory.
To stop the build at a specific stage use [`--target `](https://docs.docker.com/build/building/multi-stage/#stop-at-a-specific-build-stage) option. You can then run the build as a normal image afterward, but it'll run within the target stage. Alternatively, use GoLand's bundled Docker plugin. You may have to add `*Dockerfile` as a pattern in your [File Types](https://www.jetbrains.com/help/go/creating-and-registering-file-types.html#register-new-association) settings.
---
### Overview (`.github/actions/check-deferred-ci/README.md`)
# PR CI deferral
Apply either or both of these labels to control when automatic PR CI may start:
| Label | Condition for starting CI |
| --- | --- |
| `defer-ci-after-hours` | Wait during **5am–9pm**; release during **9pm–5am**. |
| `defer-ci-review` | Wait until the PR has at least one active approving review. |
| Both labels | Wait until **both** the after-hours window and an approval are present. |
All times use **America/Los_Angeles** by default. Set the repository Actions
variable `CI_TIMEZONE` to another IANA timezone name if needed. The release window
includes 9pm and ends at 5am, every day including weekends, with daylight-saving
time handled automatically.
For example, a PR with both labels that receives approval at 2pm waits until the
9pm release window. If it is still unapproved at 9pm, it continues waiting for an
approval. A PR with only `defer-ci-review` can start CI on approval at any time.
An active approval means a reviewer's latest submitted decision is `APPROVED`.
A later comment-only review does not revoke it; a dismissal or later request for
changes from that reviewer does. One reviewer's active approval is sufficient,
even if another reviewer requested changes. This is a CI scheduling condition;
it does not replace branch protection's review requirements. GitHub controls
whether new commits dismiss existing approvals. An approval that GitHub has not
dismissed continues to qualify, even if it was submitted on an earlier commit.
For a draft PR, add **`force-draft-ci`** to bypass the draft hold without marking
it ready. This label bypasses only the draft hold: `defer-ci-after-hours` and
`defer-ci-review` still apply, including when combined.
## Submitting a PR without starting CI before labels are applied
**Automatic PR CI is also held while a PR is a draft without `force-draft-ci`.**
Agents should create the PR as a draft, apply all desired labels, and then mark it
ready for review.
Marking it ready releases CI only when all label conditions are satisfied.
Unlabeled drafts also wait until they are marked ready or given `force-draft-ci`.
`gh pr create --label` is **not atomic**: the CLI creates the PR, then applies
labels in a separate API call. A quick label update alone cannot guarantee that
CI will not start first. GitHub's create-PR API does support creating a draft in
the initial request, which makes this sequence safe for automatic PR CI:
```sh
pr_url=$(gh pr create --draft \
--label defer-ci-after-hours \
--label defer-ci-review \
--title 'PR title' --body-file pr-body.txt) &&
gh pr ready "$pr_url"
```
To keep the PR in draft, replace the final `gh pr ready "$pr_url"` command with
`gh pr edit "$pr_url" --add-label force-draft-ci`. Apply the deferral labels
successfully **before** adding this override, so the draft hold protects the
label-assignment interval.
Use either deferral label or both as needed. The `&&` ensures the PR is not marked
ready if creation or label assignment fails; the draft continues to hold CI. The
repository labels must exist before running these commands. The gate reads the
live draft state and labels, including when an original draft run is rerun.
Sources: [GitHub CLI creation and metadata calls](https://github.com/cli/cli/blob/trunk/api/queries_pr.go#L459-L520),
[GitHub create-PR inputs](https://docs.github.com/en/graphql/reference/pulls#createpullrequestinput).
## Release and override
- Each job briefly allocates its normal runner to check admission before setup
or tests. Deferred workflows are canceled, freeing those runners until release.
Matrix members can each briefly allocate a runner; there is no separate
admission job or admission check.
- A new bot comment records each deferral, including its conditions and overrides,
and another records release. Existing comments are never edited. Repeated
notifications for the same event and revision do not add duplicate comments.
The separate `ci-deferred` label tracks postponed work until release.
The scheduler manages this queue label; users choose the two `defer-ci-*` labels.
- Label additions/removals and draft-ready transitions trigger reconciliation.
An approval triggers a small read-only review-notification workflow; its
completion wakes the trusted controller, which fetches the current reviews.
This is a permissions bridge: review events on fork PRs have read-only tokens
and cannot rerun CI directly, while `workflow_run` can use a write token.
Without this bridge, approvals would wait for the scheduled poll. Review edits
and dismissals also trigger that recheck. See [GitHub’s event permissions](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run).
- `Schedule PR CI` polls every 15 minutes, at :07, :22, :37, and :52. Review-only
PRs, drafts, and tracked PRs whose deferral labels were removed are checked
during daytime too. After-hours PRs are polled during the release window.
Polling recovers missed events; GitHub schedules can be delayed or dropped.
- Remove a deferral label to remove that condition. Removing just one label does
not bypass the other. Mark the PR ready (or add `force-draft-ci`) and remove
both deferral labels to request ordinary CI immediately, without a new commit.
- For a manual retry, use Actions → **Schedule PR CI** → **Run workflow**, choosing
the default branch and entering the PR number, or use **Re-run all jobs** on an
original deferred run. Both recheck the live draft state, labels, and reviews;
a manual retry does not bypass an outstanding condition.
- Labels apply to subsequent commits too. The labels stay on the PR after CI runs.
These conditions control admission of new work. Admitted work can continue after
5am; adding a label, dismissing a review, or converting the PR back to draft does
not cancel running tests. GitHub may delay assigning a runner after admission.
Existing non-PR triggers (manual branch runs, comment commands, pushes, repository
dispatches, releases, and nightly workflows) retain their previous behavior.
## Repository setup
1. Merge the workflows and scripts into the default branch. Scheduled and
`workflow_run` triggers require that. The privileged controller explicitly
checks out that branch, so full end-to-end validation requires deployment there.
PR branches must contain the independent workflows and scheduling scripts,
directly or through their PR merge commit.
2. **Require the `CI scheduling` status check** in branch protection or the ruleset
for PR target branches, while keeping existing required test checks. Skipped
Actions jobs count as successful; this pending status prevents merging deferred
CI. Arrange this protection before using deferral. Source changes do not
modify repository rules. Existing workflow/job names are preserved.
3. Create the three user-facing labels if they do not already exist:
```sh
gh label create defer-ci-after-hours --repo dolthub/dolt \
--description 'Defer PR CI until 9pm–5am' --color d4c5f9
gh label create defer-ci-review --repo dolthub/dolt \
--description 'Defer PR CI until an approving review' --color c5def5
gh label create force-draft-ci --repo dolthub/dolt \
--description 'Allow draft CI subject to deferral labels' --color fef2c0
```
The controller creates the internal `ci-deferred` queue label automatically.
4. Optionally set `CI_TIMEZONE`. An invalid timezone fails after-hours admission.
5. The workflows use `GITHUB_TOKEN`; no new PAT or secret is needed. Only the
trusted controller has `statuses: write`; it also requests
`issues: write` and `pull-requests: write` to manage PR labels and comments.
Existing label-validation suites retain their original metadata write permissions.
The shared action needs contents and pull-request read access. Restricted label and
scheduling-test jobs also request `actions: write` to cancel themselves; other
jobs retain the repository default. GitHub downgrades fork PR tokens to read-only.
Review notifications have no repository permissions and do not check out code,
so fork review events never execute PR code with the controller's write token.
6. Validate a draft-to-ready submission with each label and both labels. Verify
daytime deferral, overnight release, approval/dismissal handling, label removal,
the draft override, and fork PRs. GitHub's normal fork approval requirements
remain in force; polling is a fallback if a review notification awaits approval.
The scheduling status remains pending while work is postponed and clears when
the scheduler releases it, without waiting for tests to finish. Individual CI
checks determine test results and mergeability. Admission errors fail the
scheduling status; errors appear in scheduler logs, not PR comments. Failed tests
are not automatically retried; use their original workflow controls.
## Implementation and limits
- Each existing PR workflow keeps its triggers, branch/path filters, job names,
dependencies, runner matrices, concurrency groups, and non-PR entry points.
- Each job adds one shared composite action before setup or tests:
```yaml
- name: Check deferred CI
uses: $/.github/actions/check-deferred-ci
with:
timezone: ${{ vars.CI_TIMEZONE || 'America/Los_Angeles' }}
```
The `$/` self reference loads the action from the workflow's own repository and
commit without a workspace checkout, including in forks. This syntax requires
GitHub.com; older local workflow linters may not recognize it. The shared code
lives beside `action.yml` and is loaded from the absolute `github.action_path`;
it does not depend on a checkout or the caller’s working directory.
- The action returns successfully only when work is admitted (including non-PR
triggers). Otherwise it waits for cancellation, failing after a bounded timeout.
Normal subsequent steps therefore need no extra condition. Failure/always
handlers must additionally check its `run` output; the ORM workflow demonstrates
this with an `id: ci-admission` and two guarded failure handlers.
- When deferred, the action uploads a tiny artifact named with the run, attempt,
and a unique suffix before waiting for cancellation. Composite internals are
not separate Jobs API steps, and annotations are not published until the step
finishes; artifact metadata is available while cancellation is pending. The
scheduler reads only artifact names and expiry flags, never artifact contents.
Markers last 30 days, matching GitHub's rerun window. No extra check is created.
- A deferred job requests cancellation of its own workflow using its existing
token. GitHub's API cancels a whole workflow, including matrix siblings; it does
not provide an individual-job cancellation endpoint.
- Fork PR tokens and explicitly restricted jobs cannot cancel runs themselves.
A trusted `workflow_run: in_progress` handler checks GitHub job metadata for the
postponement artifact metadata and cancels the run without executing PR code. It stops
monitoring once all jobs have passed admission or after 60 seconds.
- The job waits at most 90 seconds for cancellation. If GitHub delays cancellation
beyond that limit, the step fails without running tests; its postponement marker
still lets the scheduler release it later. Restricted jobs explicitly request
`actions: write`; read-only fork PR tokens still use the trusted controller.
- The controller recognizes postponed runs from a matching artifact for the
current run and attempt, even when matrix siblings were canceled before starting.
Expired artifacts and markers from other attempts do not qualify. Admission API
errors and unrelated cancellations fail closed; ordinary test failures are not
automatically retried.
- The scheduler reruns the original independent workflows. Each job rechecks
admission on every attempt, including individual-job and failed-job reruns.
- Only the latest run per workflow for the current open PR head is eligible.
Before each rerun the controller rechecks the head, draft state, labels, review
decisions, time, run state, and attempt. It does not release obsolete commits.
- A per-PR concurrency group serializes event and timer reconciliation. Polling
recovers invocations replaced in GitHub's single pending concurrency slot.
- Event comments contain only a hidden revision and event identifier for
deduplication. No workflow results are cached in comments. Successful test
completions do not trigger reconciliation; unsuccessful attempts are inspected
only to identify deferrals or admission errors. The scheduler never reports
test completion.
- The trusted controller never executes PR code or downloads PR artifacts.
- The scheduler queries Actions runs for the current PR head and groups them by
workflow ID. It recognizes participating workflows by the shared admission
step in their job metadata and confirms deferrals using attempt-specific
artifact markers. No file registry is needed; unrelated workflows are ignored.
- The `workflow_run.workflows` subscription still lists display names: GitHub
requires literal names to deliver these events and does not support a catch-all
subscription. Update that one list when adding or renaming a PR workflow.
Runtime discovery still finds unlisted workflows during polling, but prompt
cancellation of read-only jobs requires the event subscription. Removing the
list entirely would require polling or an external webhook service.
- Each participating job starts with the shared action; failure/always handlers
additionally check its admission output.
GitHub permits reruns for **30 days after the original run**, up to **50 attempts**.
This also limits how long draft/review deferrals can be automatically released
using the original runs. If a review takes longer, or the rerun limit is exhausted,
the controller leaves CI pending and records the API error in its workflow log.
Push a new commit to create fresh PR runs. Two PR reconciliations may run concurrently
per scheduler invocation; this does not reserve organization-wide runner capacity.
## Tests
From the repository root:
```sh
npm ci --prefix .github/actions/check-deferred-ci
npm test --prefix .github/actions/check-deferred-ci
actionlint -shellcheck='' .github/workflows/ci-scheduler.yaml \
.github/workflows/ci-review-notification.yaml .github/workflows/ci-scheduling-tests.yaml
```
The minimal API tests cover admission conditions, runtime workflow discovery,
append-only event comments, safe release and cancellation, and review wake-ups.
They do not launch actual CI or change repository settings.
---
### Overview (`go/performance/import_benchmarker/README.md`)
## Import Benchmarking
Benchmark different import workflows expressed as yaml files.
Unit tests in `import_test.go` are not suitable for reporting performance
comparisons.
## Usage
Sample:
```bash
go build \
github.com/dolthub/dolt/go/performance/import_benchmarker/cmd \
-test testdata/shuffle.yaml
```
Requirements:
Tests that use dolt require a `dolt` binary in `PATH` for performance comparisons.
Tests with an `external-server` configuration are expected to be available
from the host machine on startup.
Example `mysqld` server on the host OS, assuming an initialized `datadir`
and pre-existing database:
```bash
mysqld --port 3308 --local-infile=1 --socket=/tmp/mysqld2.sock
````
Example mysql server `docker-compose.yml` config:
```yaml
mysql:
image: mysql/mysql-server:8.0
container_name: mysql-import-perf
ports:
- "3308:3306"
command: --local-infile=1 --socket=/tmp/mysqld2.sock
volumes:
- ./mysql:/var/lib/mysql
restart: always # always restart unless stopped manually
environment:
MYSQL_USER: root
MYSQL_ROOT_PASSWORD: password
MYSQL_PASSWORD: password
MYSQL_DATABASE: test
```
Note the `--local-infile` parameter, which permits `LOAD DATA`, and
the `--socket` parameter, which specifies a non-default socket that
will not conflict with any `dolt sql-server` instances. All other
parameters, including the database name, are configurable in the test
file yaml.
## Inputs
Specify imports for different servers and workloads along several
dimensions:
- repo
- dolt server (server field)
- dolt cli (omit server field)
- mysql server (external server field)
- table spec
- fmt (string): file format for importing
- csv: comma separated lines
- sql: dump file of insert statements
- rows (int): number of rows to import
- schema (string): CREATE_TABLE statement for table to import
- shuffle (bool): by default generated rows are sorted; indicate `true` to shuffle
- batch (bool): whether to batch insert statements (only applies to fmt=sql)
For an examples of the specific yaml input syntax, see the example
files below, or refer to the tests in `testdata/`.
Server Details:
- For dolt sql-server tests, a new sql-server will be constructed individually
for each test run.
- External servers are provided outside of the lifecycle of the `import_benchmarker`
command. The same database instance is used for every table import test.
- Import files are cached on the schema, row number, and format in between
tests.
## Outputs
The output format is a `.sql` file with the following schema:
```sql
CREATE TABLE IF NOT EXISTS import_perf_results (
test_name varchar(64),
server varchar(64),
detail varchar(64),
row_cnt int,
time double,
file_format varchar(8),
sorted bool,
primary key (test_name, server, detail)
);
```
A sample import file:
```
/* Detailed source-code truncated for AI context efficiency. */
```
Ingest the result file and run queries like the ones below to compare
import runtimes:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Example tests
Example test spec 1:
```yaml
tests:
- name: "sorting"
repos:
- name: repo1
server:
port: 3308
tables:
- name: "shuffle"
shuffle: true
rows: 100000
schema: |
create table xy (
x int primary key,
y varchar(30)
);
- name: "sorted"
shuffle: false
rows: 100000
schema: |
create table xy (
x int primary key,
y varchar(30)
);
```
We will import two tables with a dolt sql-server on port `3308`.
Both tables have 100,000 rows, and a schema with two columns.
The "sorted" test imports the default sorted rows, while the
"shuffle" imports unsorted rows.
Example import spec 2:
```yaml
tests:
- name: "row count"
repos:
- name: mysql
external-server:
name: test
host: 127.0.0.1
user: root
password: password
port: 4306
tables:
- name: "400k"
fmt: "csv"
rows: 40000
schema: |
create table xy (
x int primary key,
y varchar(30)
);
```
We will connect to a database server named `test` on port `4306`
with the credentials above to run a 40,000 row import of a table
with two columns.
---
### Overview (`go/performance/sysbench/README.md`)
# Sysbench
MySQL tests expect local server (ex: `mysqld --port 3309 --local-infile=1 --socket=/tmp/mysqld2.sock`).
Interrupting a test midway can create cause "table already exists" error.
---
### Overview (`go/store/nbs/README.md`)
# Noms Block Store
A horizontally-scalable storage backend for Noms.
## Overview
NBS is a storage layer optimized for the needs of the [Noms](https://github.com/attic-labs/noms) database.
NBS can run in two configurations: either backed by local disk, or [backed by Amazon AWS](https://github.com/attic-labs/noms/blob/master/go/nbs/NBS-on-AWS.md).
When backed by local disk, NBS is significantly faster than LevelDB for our workloads and supports full multiprocess concurrency.
When backed by AWS, NBS stores its data mainly in S3, along with a single DynamoDB item. This configuration makes Noms "[effectively CA](https://research.google.com/pubs/pub45855.html)", in the sense that Noms is always consistent, and Noms+NBS is as available as DynamoDB and S3 are. This configuration also gives Noms the cost profile of S3 with power closer to that of a traditional database.
## Details
* NBS provides storage for a content-addressed DAG of nodes (with exactly one root), where each node is encoded as a sequence of bytes and addressed by a 20-byte hash of the byte-sequence.
* There is no `update` or `delete` -- only `insert`, `update root` and `garbage collect`.
* Insertion of any novel byte-sequence is durable only upon updating the root.
* File-level multiprocess concurrency is supported, with optimistic locking for multiple writers.
* Writers need not worry about re-writing duplicate chunks. NBS will efficiently detect and drop (most) duplicates.
## Storage locking
Every file-backed NBS store takes exactly one exclusive advisory file lock:
```
/.dolt/noms/LOCK
```
The lock is taken through [`github.com/dolthub/file-locks`](https://github.com/dolthub/file-locks), which uses `flock(2)` on unix and `LockFileEx` on Windows. It is *advisory*: it only excludes processes that ask for the same lock, and it carries no information a non-locking process can read.
### Lifetime
`newJournalLock` takes the lock when a store is opened, and `journalManifest.Close` (reached from `NomsBlockStore.Close`) releases it. There is no maximum hold time and it is not a critical section -- a `dolt sql-server` holds the lock for its whole lifetime, because that is the lifetime of the store it opened. The correct thing to wait for is therefore "the holding process has exited or closed its store", not a fixed timeout.
### Acquire timeout and the read-only fallback
Dolt waits only `lockFileTimeout`, 100ms (`file_manifest.go`), before giving up. What happens next depends on the caller:
* By default the store opens **read-only**. Writes then fail with a read-only error, which is why a restart that races the previous server's release yields a server that accepts connections but rejects writes, rather than an error naming the lock.
* Embedded-driver callers can pass `fail_on_journal_lock_timeout` to get `ErrDatabaseLocked` ("the database is locked by another dolt process") instead of a silent read-only open.
* `skip_journal_lock_timeout` collapses the wait to a single non-blocking attempt.
### Crash and staleness
`flock` locks belong to the open file rather than the process, and the kernel releases them when the process dies -- including on `kill -9`. A hard crash does **not** leave a stale lock behind, and there is no operator step needed to clear one.
The `LOCK` file itself, however, always remains: it is created on open and deliberately left in place on close. Its presence says nothing about lock state -- only holding the flock does.
Do not delete the `LOCK` file to "clear" a lock. A lock held on an unlinked file excludes nobody, so removing the file while a process holds the lock silently breaks mutual exclusion.
### External waiters
An external process can take the same `flock`, but this is not a supported way to coordinate. Dolt does not publish its lock state anywhere and does not watch the file, so an external flock cannot tell a supervisor *when* a database was released -- it only changes whether Dolt's own next open succeeds, and there is no supported query for who holds it.
To coordinate a restart, wait on the previous process itself: make sure it has exited or closed its store before starting the next one.
### Caveats
These guarantees come from the underlying OS primitive and are advisory: over NFS or SMB they are only as good as the mount, so keep database directories on local storage. Waiters are not queued, and a `Lock` is not recursive.
## Perf
For the file back-end, perf is substantially better than LevelDB mainly because LDB spends substantial IO with the goal of keeping KV pairs in key-order which doesn't benefit Noms at all. NBS locates related chunks together and thus reading data from an NBS store can be done quite a lot faster. As an example, storing & retrieving a 1.1GB MP4 video file on a MBP i5 2.9Ghz:
* LDB
* Initial import: 44 MB/s, size on disk: 1.1 GB.
* Import exact same bytes: 35 MB/s, size on disk: 1.4 GB.
* Export: 60 MB/s
* NBS
* Initial import: 72 MB/s, size on disk: 1.1 GB.
* Import exact same bytes: 92 MB/s, size on disk: 1.1GB.
* Export: 300 MB/s
## Status
NBS is more-or-less "beta". There's still [work we want to do](https://github.com/attic-labs/noms/issues?q=is%3Aopen+is%3Aissue+label%3ANBS), but it now works better than LevelDB for our purposes and so we have made it the default local backend for Noms:
```shell
# This uses nbs locally:
./csv-import foo.csv /Users/bob/csv-store::data
```
The AWS backend is available via the `aws:` scheme:
```shell
./csv-import foo.csv aws://[table:bucket]::data
```
---
### Overview (`go/utils/batsee/README.md`)
BATS Executor Exemplar is used to run all BATS tests in parallel to get faster results than running serially.
## Installation
To install, go to the go/utils/batsee directory and run:
`go install .`
Results will be in `$HOME/go/bin/`, so add that to your path.
## Usage
Before you do anything related to batsee, you should make sure you can [run bats](../../../integration-tests/bats/README.md).
```
$ batsee --help
NAME
batsee - Run the Bats Tests concurrently
SYNOPSIS
batsee [-t ] [--skip-slow] [--max-time
# Dolt is Git for Data!
Dolt is a SQL database that you can fork, clone, branch, merge, push
and pull just like a Git repository.
Connect to Dolt just like any MySQL database to read or modify schema
and data. Version control functionality is exposed in SQL via system
tables, functions, and procedures.
Or, use the Git-like command line interface to import CSV files, commit
your changes, push them to a remote, or merge your teammate's changes.
All the commands you know for Git work exactly the same for Dolt.
Git versions files. Dolt versions tables. It's like Git and MySQL had a
baby.
We also built [DoltHub](https://www.dolthub.com), a place to share
Dolt databases. We host public data for free. If you want to host
your own version of DoltHub, we have [DoltLab](https://www.doltlab.com).
If you want us to run a Dolt server for you, we have [Hosted Dolt](https://hosted.doltdb.com).
Prefer Postgres instead of MySQL? Try [Doltgres](https://github.com/dolthub/doltgresql), now
in its Beta release.
[Join us on Discord](https://discord.com/invite/RFwfYpu) to say hi and
ask questions, or [check out our roadmap](https://dolthub.com/docs/other/roadmap)
to see what we're building next.
# Video Introduction
[](https://www.youtube.com/watch?v=H2iZy0Cme10)
# What's it for?
Lots of things! Dolt is a generally useful tool with countless
applications. But if you want some ideas, [here's how people are using
it so far](https://dolthub.com/blog/2024-10-15-dolt-use-cases/).
Dolt powers [Beads](https://github.com/gastownhall/beads) and
[Gas Town](https://github.com/steveyegge/gastown). It's the best database
for agent memory, especially as you move up the ladder to multi-agent and
multi-machine workflows.
# Dolt CLI
The `dolt` CLI has the same commands as `git`, with some extras.
```
/* Detailed source-code truncated for AI context efficiency. */
```
# Installation
Dolt is a single ~103 megabyte program.
```bash
dolt $ du -h /Users/timsehn/go/bin/dolt
103M /Users/timsehn/go/bin/dolt
```
It's really easy to install. Download it and put it on your `PATH`.
We have a bunch of ways to make this even easier for most platforms.
## From Latest Release
To install on Linux or Mac based systems run this command in your
terminal:
```
sudo bash -c 'curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | bash'
```
This will download the latest `dolt` release and put it in
`/usr/local/bin/`, which is probably on your `$PATH`.
The install script needs sudo in order to put `dolt` in `/usr/local/bin`. If you don't have root
privileges or aren't comfortable running a script with them, you can download the dolt binary
for your platform from [the latest release](https://github.com/dolthub/dolt/releases), unzip it,
and put the binary somewhere on your `$PATH`.
### Linux
#### Arch Linux
Dolt is packaged in the official repositories for Arch Linux.
```
pacman -S dolt
```
### Mac
#### Homebrew
Dolt is on Homebrew, updated every release.
```
brew install dolt
```
#### MacPorts
On macOS, Dolt can also be installed via a [community-managed port](https://ports.macports.org/port/dolt/) via [MacPorts](https://www.macports.org):
```sh
sudo port install dolt
```
### Windows
Download the latest Microsoft Installer (`.msi` file) in
[releases](https://github.com/dolthub/dolt/releases) and run
it.
For information on running on Windows, see [here](https://dolthub.com/docs/introduction/installation/windows).
#### Chocolatey
You can install `dolt` using [Chocolatey](https://chocolatey.org/):
```sh
choco install dolt
```
#### Docker
There are following official Docker images for Dolt:
* [`dolthub/dolt`](https://hub.docker.com/r/dolthub/dolt) for running Dolt
as CLI tool.
* [`dolthub/dolt-sql-server`](https://hub.docker.com/r/dolthub/dolt-sql-server) for running Dolt in server mode.
## From Source
Make sure you have Go installed, and that `go` is in your path. Dolt has a dependency on [cgo](https://pkg.go.dev/cmd/cgo), so you will need a working C compiler and toolchain as well.
Clone this repository and cd into the `go` directory. Then run:
```
go install ./cmd/dolt
```
The output will be in `$GOPATH/bin`, which defaults to `~/go/bin`. To test your build, try:
```
~/go/bin/dolt version
```
# Configuration
Verify that your installation has succeeded by running `dolt` in your
terminal.
```
$ dolt
Valid commands for dolt are
[...]
```
Configure `dolt` with your user name and email, which you'll need to
create commits. The commands work exactly the same as git.
```
$ dolt config --global --add user.email YOU@DOMAIN.COM
$ dolt config --global --add user.name "YOUR NAME"
```
# Getting started
## Navigate to the directory where you would like your data stored
Dolt needs a place to store your databases. I'm going to put my databases in `~/dolt`.
```bash
% cd ~
% mkdir dolt
% cd dolt
```
Any databases you create will be stored in this directory. So, for this example, a directory named `getting_started` will be created here once you run `create database getting_started`. Navigating to `~/dolt/getting_started` will allow you to access this database using the Dolt command line.
NOTE: For this example, the `getting_started` directory will be created after you run `create database getting_started;` in a SQL shell in the [Create a schema section](#create-a-schema). Don't do anything except make the directory and navigate to it just yet.
## Start a MySQL-compatible database server
Dolt ships with a MySQL compatible database server built in. To start it you use the command `dolt sql-server`. Running this command starts the server on port 3306.
```bash
dolt sql-server
Starting server with Config HP="localhost:3306"|T="28800000"|R="false"|L="info"
```
Your terminal will just hang there. This means the server is running. Any errors will be printed in this terminal. Just leave it there and open a new terminal.
## Connect with a MySQL client (up to version 8.4)
In the new terminal, we will now connect to the running database server using a client. Dolt also ships with a MySQL compatible client.
```bash
% dolt -u root -p "" sql
# Welcome to the Dolt MySQL client.
# Statements must be terminated with ';'.
# "exit" or "quit" (or Ctrl-D) to exit.
mysql>
```
In the other terminal where you ran `dolt sql-server`, you'll see the following log line.
```
2022-06-06T13:14:32-07:00 INFO [conn 1] NewConnection {DisableClientMultiStatements=false}
```
You are connected!
While we're here let's grab a copy of MySQL so we can connect with that client. Head over to the [MySQL Getting Started](https://dev.mysql.com/doc/mysql-getting-started/en/) documentation and install MySQL on your machine. I used [Homebrew](https://brew.sh/) to install MySQL on my Mac: `brew install mysql@8.4`. Alternatively, you can install only the client component by running `brew install mysql-client@8.4`.
NOTE: Make sure you install a MySQL 8.4 release. MySQL 8.4 is the current Long Term Support (LTS) release, meaning this is the stable and supported version of MySQL. MySQL 9.0 is also available, but is an "innovation" release, meaning it has more recent changes and features, but may not be as stable as the LTS release. The 9.0 release changes authentication support and isn't able to connect to a Dolt SQL server by default. You can install MySQL 8.4 with Homebrew by running `brew install mysql@8.4`. If you do want to use MySQL-9.0, read [our post on how to configure Dolt for `caching_sha2_password` authentication](https://www.dolthub.com/blog/2024-12-11-mysql9-and-caching-sha2-auth-support/).
MySQL comes with a MySQL server called `mysqld` and a MySQL client called `mysql`. You're only interested in the client. After following the instructions from MySQL's documentation, make sure you have a copy of the `mysql` client on your path:
```bash
% mysql --version
mysql Ver 8.0.29 for macos12.2 on x86_64 (Homebrew)
```
Now, to connect the `mysql` client to Dolt, you are going to force the MySQL client through the TCP interface by passing in a host and port. The default is the socket interface which Dolt supports, but is only available on `localhost`. So, it's better to show off the TCP interface. The MySQL client also requires you specify a user, in this case `root`.
```bash
% mysql --host 127.0.0.1 --port 3306 -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9-Vitess
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
```
Again, to ensure the client actually connected, you should see the following in the `dolt sql-server` terminal
```
2022-06-06T13:26:55-07:00 INFO [conn 2] NewConnection {DisableClientMultiStatements=false}
```
As you can see, Dolt supports any MySQL-compatible client. Dolt ships with a client but you can use any MySQL client, like the one that comes with MySQL.
## Create a schema
Now we're actually ready to do something interesting. I'll stay in the `mysql` client and execute the following SQL statements to create a database called `getting_started`. The `getting_started` database will have three tables: `employees`, `teams`, and `employees_teams`.
```
mysql> create database getting_started;
Query OK, 1 row affected (0.04 sec)
mysql> use getting_started;
Database changed
mysql> create table employees (
id int,
last_name varchar(255),
first_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.01 sec)
mysql> create table teams (
id int,
team_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.00 sec)
mysql> create table employees_teams(
team_id int,
employee_id int,
primary key(team_id, employee_id),
foreign key (team_id) references teams(id),
foreign key (employee_id) references employees(id));
Query OK, 0 rows affected (0.01 sec)
mysql> show tables;
+---------------------------+
| Tables_in_getting_started |
+---------------------------+
| employees |
| employees_teams |
| teams |
+---------------------------+
3 rows in set (0.00 sec)
```
Dolt supports foreign keys, secondary indexes, triggers, check constraints, and stored procedures. It's a modern, feature-rich SQL database.
## Make a Dolt commit
It's time to use your first Dolt feature. We're going to make a Dolt [commit](https://dolthub.com/docs/concepts/dolt/commits). A Dolt commit allows you to time travel and see lineage. Make a Dolt commit whenever you want to restore or compare to this point in time.
Dolt exposes version control functionality through a Git-style interface. On the command line, Dolt commands map exactly to their Git equivalent with the targets being tables instead of files. In SQL, Dolt exposes version control read operations as [system tables](https://dolthub.com/docs/sql-reference/version-control/dolt-system-tables) and version control write operations as [stored procedures](https://dolthub.com/docs/sql-reference/version-control/dolt-sql-procedures).
The naming of the system tables and stored procedures follows the `dolt_