## File: README.md # goose [![Goose CI](https://github.com/pressly/goose/actions/workflows/ci.yaml/badge.svg)](https://github.com/pressly/goose/actions/workflows/ci.yaml) [![Go Reference](https://pkg.go.dev/badge/github.com/pressly/goose/v3.svg)](https://pkg.go.dev/github.com/pressly/goose/v3) [![Go Report Card](https://goreportcard.com/badge/github.com/pressly/goose/v3)](https://goreportcard.com/report/github.com/pressly/goose/v3) Goose is a database migration tool. Both a CLI and a library. Manage your **database schema** by creating incremental SQL changes or Go functions. #### Features - Works against multiple databases: - Postgres, MySQL, MariaDB, Spanner, SQLite, YDB, ClickHouse, MSSQL, Vertica, and more. - Supports Go migrations written as plain functions. - Supports [embedded](https://pkg.go.dev/embed/) migrations. - Out-of-order migrations. - Seeding data. - Environment variable substitution in SQL migrations. - ... and more. # Install ```shell go install github.com/pressly/goose/v3/cmd/goose@latest ``` This will install the `goose` binary to your `$GOPATH/bin` directory. Binary too big? Build a lite version by excluding the drivers you don't need: ```shell go build -tags='no_postgres no_mysql no_sqlite3 no_ydb' -o goose ./cmd/goose # Available build tags: # no_clickhouse no_libsql no_mssql no_mysql # no_postgres no_sqlite3 no_vertica no_ydb ``` For macOS users `goose` is available as a [Homebrew Formulae](https://formulae.brew.sh/formula/goose#default): ```shell brew install goose ``` See [installation documentation](https://pressly.github.io/goose/installation/) for more details. # Usage Click to show `goose help` output ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Commonly used commands: [create](#create) •  [up](#up) •  [up-to](#up-to) •  [down](#down) •  [down-to](#down-to) •  [status](#status) •  [version](#version) ## create Create a new SQL migration. $ goose create add_some_column sql $ Created new file: 20170506082420_add_some_column.sql $ goose -s create add_some_column sql $ Created new file: 00001_add_some_column.sql Edit the newly created file to define the behavior of your migration. You can also create a Go migration, if you then invoke it with [your own goose binary](#go-migrations): $ goose create fetch_user_data go $ Created new file: 20170506082421_fetch_user_data.go ## up Apply all available migrations. $ goose up $ OK 001_basics.sql $ OK 002_next.sql $ OK 003_and_again.go ## up-to Migrate up to a specific version. $ goose up-to 20170506082420 $ OK 20170506082420_create_table.sql ## up-by-one Migrate up a single migration from the current version $ goose up-by-one $ OK 20170614145246_change_type.sql ## down Roll back a single migration from the current version. $ goose down $ OK 003_and_again.go ## down-to Roll back migrations to a specific version. $ goose down-to 20170506082527 $ OK 20170506082527_alter_column.sql Or, roll back all migrations (careful!): $ goose down-to 0 ## status Print the status of all migrations: $ goose status $ Applied At Migration $ ======================================= $ Sun Jan 6 11:25:03 2013 -- 001_basics.sql $ Sun Jan 6 11:25:03 2013 -- 002_next.sql $ Pending -- 003_and_again.go Note: for MySQL and MariaDB (use the `mysql` driver) [parseTime flag](https://github.com/go-sql-driver/mysql#parsetime) must be enabled. Note: for MySQL and MariaDB [`multiStatements`](https://github.com/go-sql-driver/mysql?tab=readme-ov-file#multistatements) must be enabled. This is required when writing multiple queries separated by ';' characters in a single sql file. ## version Print the current version of the database: $ goose version $ goose: version 002 # Environment Variables If you prefer to use environment variables, instead of passing the driver and database string as arguments, you can set the following environment variables: **1. Via environment variables:** ```shell export GOOSE_DRIVER=DRIVER export GOOSE_DBSTRING=DBSTRING export GOOSE_MIGRATION_DIR=MIGRATION_DIR export GOOSE_TABLE=TABLENAME ``` **2. Via `.env` files with corresponding variables. `.env` file example**: ```env GOOSE_DRIVER=postgres GOOSE_DBSTRING=postgres://admin:admin@localhost:5432/admin_db GOOSE_MIGRATION_DIR=./migrations GOOSE_TABLE=custom.goose_migrations ``` Loading from `.env` files is enabled by default. To disable this feature, set the `-env=none` flag. If you want to load from a specific file, set the `-env` flag to the file path. For more details about environment variables, see the [official documentation on environment variables](https://pressly.github.io/goose/documentation/environment-variables/). # Migrations goose supports migrations written in SQL or in Go. ## SQL Migrations A sample SQL migration looks like: ```sql -- +goose Up CREATE TABLE post ( id int NOT NULL, title text, body text, PRIMARY KEY(id) ); -- +goose Down DROP TABLE post; ``` Each migration file must have exactly one `-- +goose Up` annotation. The `-- +goose Down` annotation is optional. If the file has both annotations, then the `-- +goose Up` annotation **must** come first. Notice the annotations in the comments. Any statements following `-- +goose Up` will be executed as part of a forward migration, and any statements following `-- +goose Down` will be executed as part of a rollback. By default, all migrations are run within a transaction. Some statements like `CREATE DATABASE`, however, cannot be run within a transaction. You may optionally add `-- +goose NO TRANSACTION` to the top of your migration file in order to skip transactions within that specific migration file. Both Up and Down migrations within this file will be run without transactions. By default, SQL statements are delimited by semicolons - in fact, query statements must end with a semicolon to be properly recognized by goose. By default, all migrations are run on the public schema. If you want to use a different schema, specify the schema name using the table option like `-table='schemaname.goose_db_version`. More complex statements (PL/pgSQL) that have semicolons within them must be annotated with `-- +goose StatementBegin` and `-- +goose StatementEnd` to be properly recognized. For example: ```sql -- +goose Up -- +goose StatementBegin CREATE OR REPLACE FUNCTION histories_partition_creation( DATE, DATE ) returns void AS $$ DECLARE create_query text; BEGIN FOR create_query IN SELECT 'CREATE TABLE IF NOT EXISTS histories_' || TO_CHAR( d, 'YYYY_MM' ) || ' ( CHECK( created_at >= timestamp ''' || TO_CHAR( d, 'YYYY-MM-DD 00:00:00' ) || ''' AND created_at < timestamp ''' || TO_CHAR( d + INTERVAL '1 month', 'YYYY-MM-DD 00:00:00' ) || ''' ) ) inherits ( histories );' FROM generate_series( $1, $2, '1 month' ) AS d LOOP EXECUTE create_query; END LOOP; -- LOOP END END; -- FUNCTION END $$ language plpgsql; -- +goose StatementEnd ``` Goose supports environment variable substitution in SQL migrations through annotations. To enable this feature, use the `-- +goose ENVSUB ON` annotation before the queries where you want substitution applied. It stays active until the `-- +goose ENVSUB OFF` annotation is encountered. You can use these annotations multiple times within a file. This feature is disabled by default for backward compatibility with existing scripts. For `PL/pgSQL` functions or other statements where substitution is not desired, wrap the annotations explicitly around the relevant parts. For example, to exclude escaping the `$$` characters: ```sql -- +goose StatementBegin CREATE OR REPLACE FUNCTION test_func() RETURNS void AS $$ -- +goose ENVSUB ON BEGIN RAISE NOTICE '${SOME_ENV_VAR}'; END; -- +goose ENVSUB OFF $$ LANGUAGE plpgsql; -- +goose StatementEnd ``` Supported expansions (click here to expand): - `${VAR}` or $VAR - expands to the value of the environment variable `VAR` - `${VAR:-default}` - expands to the value of the environment variable `VAR`, or `default` if `VAR` is unset or null - `${VAR-default}` - expands to the value of the environment variable `VAR`, or `default` if `VAR` is unset - `${VAR?err_msg}` - expands to the value of the environment variable `VAR`, or prints `err_msg` and error if `VAR` unset - ~~`${VAR:?err_msg}` - expands to the value of the environment variable `VAR`, or prints `err_msg` and error if `VAR` unset or null.~~ **THIS IS NOT SUPPORTED** See [mfridman/interpolate](https://github.com/mfridman/interpolate?tab=readme-ov-file#supported-expansions) for more details on supported expansions. ## Embedded sql migrations Go 1.16 introduced new feature: [compile-time embedding](https://pkg.go.dev/embed/) files into binary and corresponding [filesystem abstraction](https://pkg.go.dev/io/fs/). This feature can be used only for applying existing migrations. Modifying operations such as `fix` and `create` will continue to operate on OS filesystem even if using embedded files. This is expected behaviour because `io/fs` interfaces allows read-only access. Make sure to configure the correct SQL dialect, see [dialect.go](./dialect.go) for supported SQL dialects. Example usage, assuming that SQL migrations are placed in the `migrations` directory: ```go package main import ( "database/sql" "embed" "github.com/pressly/goose/v3" ) //go:embed migrations/*.sql var embedMigrations embed.FS func main() { var db *sql.DB // setup database goose.SetBaseFS(embedMigrations) if err := goose.SetDialect("postgres"); err != nil { panic(err) } if err := goose.Up(db, "migrations"); err != nil { panic(err) } // run app } ``` Note that we pass `"migrations"` as directory argument in `Up` because embedding saves directory structure. ## Go Migrations 1. Create your own goose binary, see [example](./examples/go-migrations) 2. Import `github.com/pressly/goose` 3. Register your migration functions 4. Include your `migrations` package into Go build: in `main.go`, `import _ "github.com/me/myapp/migrations"` 5. Run goose command, ie. `goose.Up(db *sql.DB, dir string)` A [sample Go migration 00002_users_add_email.go file](./examples/go-migrations/00002_rename_root.go) looks like: ```go package migrations import ( "database/sql" "github.com/pressly/goose/v3" ) func init() { goose.AddMigration(Up, Down) } func Up(tx *sql.Tx) error { _, err := tx.Exec("UPDATE users SET username='admin' WHERE username='root';") if err != nil { return err } return nil } func Down(tx *sql.Tx) error { _, err := tx.Exec("UPDATE users SET username='root' WHERE username='admin';") if err != nil { return err } return nil } ``` Note that Go migration files must begin with a numeric value, followed by an underscore, and must not end with `*_test.go`. # Hybrid Versioning Please, read the [versioning problem](https://github.com/pressly/goose/issues/63#issuecomment-428681694) first. By default, if you attempt to apply missing (out-of-order) migrations `goose` will raise an error. However, If you want to apply these missing migrations pass goose the `-allow-missing` flag, or if using as a library supply the functional option `goose.WithAllowMissing()` to Up, UpTo or UpByOne. However, we strongly recommend adopting a hybrid versioning approach, using both timestamps and sequential numbers. Migrations created during the development process are timestamped and sequential versions are ran on production. We believe this method will prevent the problem of conflicting versions when writing software in a team environment. To help you adopt this approach, `create` will use the current timestamp as the migration version. When you're ready to deploy your migrations in a production environment, we also provide a helpful `fix` command to convert your migrations into sequential order, while preserving the timestamp ordering. We recommend running `fix` in the CI pipeline, and only when the migrations are ready for production. ## Credit The gopher mascot was designed by [Renée French](https://reneefrench.blogspot.com/) / [CC 3.0.](https://creativecommons.org/licenses/by/3.0/) For more info check out the [Go Blog](https://go.dev/blog/gopher). Adapted by Ellen. ## License Licensed under [MIT License](./LICENSE) --- ## File: examples/go-migrations/README.md # SQL + Go migrations ## This example: Custom goose binary with built-in Go migrations ```bash $ go build -o goose-custom *.go ``` ```bash $ ./goose-custom sqlite3 ./foo.db status Applied At Migration ======================================= Pending -- 00001_create_users_table.sql Pending -- 00002_rename_root.go Pending -- 00003_add_user_no_tx.go $ ./goose-custom sqlite3 ./foo.db up OK 00001_create_users_table.sql (711.58µs) OK 00002_rename_root.go (302.08µs) OK 00003_add_user_no_tx.go (648.71µs) goose: no migrations to run. current version: 3 $ ./goose-custom sqlite3 ./foo.db status Applied At Migration ======================================= 00001_create_users_table.sql 00002_rename_root.go 00003_add_user_no_tx.go ``` ## Best practice: Split migrations into a standalone package 1. Move [main.go](main.go) into your `cmd/` directory 2. Rename package name in all `*_.go` migration files from `main` to `migrations`. 3. Import this `migrations` package from your custom [cmd/main.go](main.go) file: ```go import ( // Invoke init() functions within migrations pkg. _ "github.com/pressly/goose/example/migrations-go" ) ``` --- ## File: examples/sql-migrations/README.md # SQL migrations only See [this example](../go-migrations) for Go migrations. ```bash $ go install github.com/pressly/goose/v3/cmd/goose@latest ``` ```bash $ goose sqlite3 ./foo.db status Applied At Migration ======================================= Pending -- 00001_create_users_table.sql Pending -- 00002_rename_root.sql $ goose sqlite3 ./foo.db up OK 00001_create_users_table.sql OK 00002_rename_root.sql goose: no migrations to run. current version: 2 $ goose sqlite3 ./foo.db status Applied At Migration ======================================= Mon Jun 19 21:56:00 2017 -- 00001_create_users_table.sql Mon Jun 19 21:56:00 2017 -- 00002_rename_root.sql ``` --- ## File: internal/testing/integration/README.md # Integration tests This directory contains integration tests for the [pressly/goose/v3][goose_module] Go module. An integration test is a test that runs against a real database (docker container) and exercises the same driver used by the CLI. ## Why is this a separate module? There are separate `go.mod` and `go.sum` files in this directory to allow for the use of different dependencies. We leverage [multi-module workspaces](https://go.dev/doc/tutorial/workspaces) to glue things together. Namely, we want to avoid dependencies on docker, sql drivers, and other dependencies **that are** not necessary for the core functionality of the goose library. ## Overview There are separate migration files for each database that we support, see the [migrations directory][migrations_dir]. Databases typically have different SQL syntax and features, so the migration files are different. A good set of migrations should be representative of the types of migrations users will write typically write. This should include: - Creating and dropping tables - Adding and removing columns - Creating and dropping indexes - Inserting and deleting data - Complex SQL statements that require special handling with `StatementBegin` and `StatementEnd` annotations - Statements that must run outside a transaction, annotated with `-- +goose NO TRANSACTION` There is a common test function that applies migrations up, down and then up again. The gold standard is the PostgreSQL migration files. We try to make other migration files as close to the PostgreSQL files as possible, but this is not always possible or desirable. Lastly, some tests will assert for database state after migrations are applied. To add a new `.sql` file, you can use the following command: ``` goose -s -dir testdata/migrations/ create sql ``` - Update the database name (e.g. `postgres`) - Update the filename name (e.g. `b`) as needed ## Limitation Note, the integration tests are not exhaustive. They are meantto ensure that the goose library works with the various databases that we support and the chosen drivers. We do not test every possible combination of operations, nor do we test every possible edge case. We rely on the unit tests in the goose package to cover library-specific logic. [goose_module]: https://pkg.go.dev/github.com/pressly/goose/v3 [migrations_dir]: ./testdata/migrations