Technical Documentation: sqlalchemy/alembic
โน๏ธ Provenance: Multi-Tier Ingestion ยท CodeWiki Reference
Build Autogenerate (`docs/build/autogenerate.rst`)
Auto Generating Migrations
Alembic can view the status of the database (pointed to by sqlalchemy.url in
your alembic.ini file using the current schema) and compare against the
table metadata in the application (your ORM which defines the proposed schema),
generating the "obvious" migrations based on a comparison. This is achieved
using the --autogenerate option to the alembic revision command, which
places so-called candidate migrations into our new migrations file. We
review and modify these by hand as needed, then proceed normally.
To use autogenerate, we first need to modify our env.py so that it gets access
to a table metadata object that contains the target. Suppose our application
has a :ref:declarative base <sqla:declarative_toplevel>
in myapp.mymodel. This base contains a :class:~sqlalchemy.schema.MetaData object which
contains :class:~sqlalchemy.schema.Table objects defining our database. We make sure this
is loaded in env.py and then passed to :meth:.EnvironmentContext.configure via thetarget_metadata argument. The env.py sample script used in the
generic template already has a
variable declaration near the top for our convenience, where we replace None
with our :class:~sqlalchemy.schema.MetaData. Starting with::
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Nonewe change to::
from myapp.mymodel import Base
target_metadata = Base.metadata.. note::
The above example refers to the generic alembic env.py template, e.g.
the one created by default when calling upon alembic init, and not
the special-use templates such as multidb. Please consult the source
code and comments within the env.py script directly for specific
guidance on where and how the autogenerate metadata is established.
If we look later in the script, down in run_migrations_online(),
we can see the directive passed to :meth:.EnvironmentContext.configure::
def run_migrations_online():
engine = engine_from_config(
config.get_section(config.config_ini_section), prefix='sqlalchemy.')
with engine.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations()We can then use the alembic revision command in conjunction with the--autogenerate option. Suppose
our :class:~sqlalchemy.schema.MetaData contained a definition for the account table,
and the database did not. We'd get output like::
$ alembic revision --autogenerate -m "Added account table"
INFO [alembic.context] Detected added table 'account'
Generating /path/to/foo/alembic/versions/27c6a30d7c24.py...doneWe can then view our file 27c6a30d7c24.py and see that a rudimentary migration
is already present::
"""empty message
Revision ID: 27c6a30d7c24
Revises: None
Create Date: 2011-11-08 11:40:27.089406
"""
# revision identifiers, used by Alembic.
revision = '27c6a30d7c24'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table(
'account',
sa.Column('id', sa.Integer()),
sa.Column('name', sa.String(length=50), nullable=False),
sa.Column('description', sa.VARCHAR(200)),
sa.Column('last_transaction_date', sa.DateTime()),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table("account")
### end Alembic commands ###The migration hasn't actually run yet, of course. We do that via the usual upgrade
command. We should also go into our migration file and alter it as needed, including
adjustments to the directives as well as the addition of other directives which these may
be dependent on - specifically data changes in between creates/alters/drops.
.. _autogenerate_detects:
What does Autogenerate Detect (and what does it *not* detect?)
The vast majority of user issues with Alembic centers on the topic of what
kinds of changes autogenerate can and cannot detect reliably, as well as
how it renders Python code for what it does detect. It is critical to
note that autogenerate is not intended to be perfect. It is always
necessary to manually review and correct the candidate migrations
that autogenerate produces. The feature is getting more and more
comprehensive and error-free as releases continue, but one should take
note of the current limitations.
Autogenerate will detect:
- Table additions, removals.
- Column additions, removals.
- Change of nullable status on columns.
- Basic changes in indexes and explicitly-named unique constraints
- Basic changes in foreign key constraints
Autogenerate can optionally detect:
Named CHECK constraint additions and removals. This is off by default
and must be enabled explicitly; the detection is name-based only and has
significant limitations. See :ref:autogenerate_check_constraintsbelow.Change of column type. This will occur by default unless the parameter
:paramref:.EnvironmentContext.configure.compare_typeis set toFalse.
The default implementation will reliably detect major changes,
such as between :class:.Numericand :class:.String, as well as
accommodate for the types generated by SQLAlchemy's "generic" types such as
:class:.Boolean. Arguments that are shared between both types, such as
length and precision values, will also be compared. If either the metadata
type or database type has additional arguments beyond that of the other
type, these are not compared, such as if one numeric type featured a
"scale" and other type did not, this would be seen as the backing database
not supporting the value, or reporting on a default that the metadata did not
specify.The type comparison logic is fully extensible as well; see
:ref:compare_typesfor details.Change of server default. This will occur if you set
the :paramref:.EnvironmentContext.configure.compare_server_default
parameter toTrue, or to a custom callable function.
This feature works well for simple cases but cannot always produce
accurate results. The Postgresql backend will actually invoke
the "detected" and "metadata" values against the database to
determine equivalence. The feature is off by default so that
it can be tested on the target schema first. Like type comparison,
it can also be customized by passing a callable; see the
function's documentation for details.
Autogenerate can not detect:
- Changes of table name. These will come out as an add/drop of two different
tables, and should be hand-edited into a name change instead. - Changes of column name. Like table name changes, these are detected as
a column add/drop pair, which is not at all the same as a name change. - Anonymously named constraints. Give your constraints a name,
e.g.UniqueConstraint('col1', 'col2', name="my_name"). See the section
:doc:namingfor background on how to configure automatic naming schemes
for constraints. - Special SQLAlchemy types such as :class:
~sqlalchemy.types.Enumwhen generated
on a backend which doesn't support ENUM directly - this because the
representation of such a type
in the non-supporting database, i.e. a CHAR+ CHECK constraint, could be
any kind of CHAR+CHECK. For SQLAlchemy to determine that this is actually
an ENUM would only be a guess, something that's generally a bad idea.
To implement your own "guessing" function here, use the
:meth:sqlalchemy.events.DDLEvents.column_reflectevent
to detect when a CHAR (or whatever the target type is) is reflected,
and change it to an ENUM (or whatever type is desired) if it is known that
that's the intent of the type. The
:meth:sqlalchemy.events.DDLEvents.after_parent_attach
can be used within the autogenerate process to intercept and un-attach
unwanted CHECK constraints.
Autogenerate can't currently, but will eventually detect:
- Some free-standing constraint additions and removals may not be supported,
including PRIMARY KEY, EXCLUDE; these are not necessarily implemented
within the autogenerate detection system and also may not be supported by
the supporting SQLAlchemy dialect. - Sequence additions, removals - not yet implemented.
.. _autogenerate_check_constraints:
Detecting CHECK Constraints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. versionadded:: 1.19.0
.. versionchanged:: 1.19.2 This feature is no longer enabled by default.
See the warning below for the rationale.
Autogenerate includes optional, off by default support for detecting the
addition and removal of named CHECK constraints. The detection is
name-based only: a metadata-side CHECK constraint and a reflected CHECK
constraint that share the same name are always presumed to be equivalent,
regardless of any difference in their expression text.
The feature is implemented as a built-in
:ref:plugin <alembic.plugins.toplevel> namedalembic.ext.checkconstraint_byname. Unlike Alembic's other built-in
autogenerate plugins, it is named outside of the alembic.autogenerate.
namespace and is therefore not part of the alembic.autogenerate.*
wildcard; it must be named explicitly in the
:paramref:.EnvironmentContext.configure.autogenerate_plugins list in order
to be used::
context.configure(
# ...
autogenerate_plugins=[
"alembic.autogenerate.*",
"alembic.ext.checkconstraint_byname",
]
)The plugin was named alembic.autogenerate.checkconstraint_byname in
versions 1.19.0 and 1.19.1. That name continues to be accepted, both to
enable the plugin and within a "~" exclusion, so an env.py written
against those releases requires no change.
.. warning::
This feature has significant limitations and should be enabled only for
schemas that fit the narrow set of conditions it can handle. Before
turning it on, note all of the following:
* **Expression changes are never detected.** Two CHECK constraints that
share a name are presumed equal even if their SQL expressions differ
entirely. A migration that alters the condition of an existing
constraint will not be generated.
* **Every CHECK constraint in the target metadata must be named.** A
CHECK constraint with no name, whether written directly as
``CheckConstraint("x > 5")`` or left unnamed by a naming convention
that does not produce a name for it, is skipped on the metadata side.
Backends such as PostgreSQL nonetheless assign a name to every
constraint in the database, so the reflected constraint matches
nothing in the metadata and is reported as a **spurious removal**.
This includes CHECK constraints generated by types such as
:class:`~sqlalchemy.types.Enum` and :class:`~sqlalchemy.types.Boolean`
that have ``create_constraint=True``; assign a ``name`` to these
types to make the CHECK constraints named.
.. versionchanged:: 1.19.2 CHECK constraints that are "type bound" are
no longer ignored on the metadata side, so that normal name-based
matching can occur.
* **Not all backends reflect CHECK constraints**, in which case the
plugin silently does nothing.
Unlike unique constraints, which fall back to matching on column names
when the metadata side is unnamed, there is no comparable fallback
available for CHECK constraints. Matching on the expression instead of
the name is not feasible, because dialects are free to normalize,
reformat, or otherwise rewrite the original DDL text. PostgreSQL, for
example, reflects ``x != y`` as ``x <> y``, ``x BETWEEN 1 AND 10`` as
``x >= 1 AND x <= 10``, and ``s IN ('a', 'b')`` as
``s::text = ANY (ARRAY['a'::character varying, ...])``. Comparing that
text against the metadata expression would report a change on nearly
every constraint.Given the above, the feature is most appropriate for a schema in which all
CHECK constraints are explicitly named, ideally via a naming convention (see
:doc:naming). As always, the candidate migrations that autogenerate
produces must be reviewed by hand.
Notable 3rd-party libraries that extend the built-in Alembic autogenerate functionality
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
alembic-utils <https://github.com/olirice/alembic_utils>_
A library that adds autogenerate support for PostgreSQL functions, views, triggers, etc.alembic-postgresql-enum <https://pypi.org/project/alembic-postgresql-enum>_
A library that adds autogenerate support for creation, alteration and deletion of Enums in PostgreSQL.
Autogenerating Multiple MetaData collections
The target_metadata collection may also be defined as a sequence
if an application has multiple :class:~sqlalchemy.schema.MetaData
collections involved::
from myapp.mymodel1 import Model1Base
from myapp.mymodel2 import Model2Base
target_metadata = [Model1Base.metadata, Model2Base.metadata]The sequence of :class:~sqlalchemy.schema.MetaData collections will be
consulted in order during the autogenerate process. Note that each
:class:~sqlalchemy.schema.MetaData must contain unique table keys
(e.g. the "key" is the combination of the table's name and schema);
if two :class:~sqlalchemy.schema.MetaData objects contain a table
with the same schema/name combination, an error is raised.
.. _autogenerate_include_hooks:
Controlling What to be Autogenerated
The autogenerate process scans across all table objects within
the database that is referred towards by the current database connection
in use.
The list of objects that are scanned in the target database connection include:
The "default" schema currently referred towards by the database connection.
If the :paramref:
.EnvironmentContext.configure.include_schemasis set toTrue, all non-default "schemas", which are those names returned by the
:meth:~sqlalchemy.engine.reflection.Inspector.get_schema_namesmethod of
:class:~sqlalchemy.engine.reflection.Inspector. The SQLAlchemy document
:ref:sqla:schema_table_schema_namediscusses the concept of a
"schema" in detail.Within each "schema", all tables present are scanned using the
:meth:~sqlalchemy.engine.reflection.Inspector.get_table_namesmethod of
:class:~sqlalchemy.engine.reflection.Inspector.Within each "table", most sub-objects of the each
:class:~sqlalchemy.schema.Tableconstruct are scanned, including columns
and some forms of constraints. This process ultimately involves the use of
methods on :class:~sqlalchemy.engine.reflection.Inspectorincluding
:meth:~sqlalchemy.engine.reflection.Inspector.get_columns,
:meth:~sqlalchemy.engine.reflection.Inspector.get_indexes,
:meth:~sqlalchemy.engine.reflection.Inspector.get_unique_constraints,
:meth:~sqlalchemy.engine.reflection.Inspector.get_foreign_keys(as of this
writing, CHECK constraints and primary key constraints are not yet included).
.. seealso::
:ref:`sqla:schema_table_schema_name` - in depth introduction to how
SQLAlchemy interprets schema names
:ref:`sqla:postgresql_schema_reflection` - important notes specific to the
PostgreSQL databaseOmitting Schema Names from the Autogenerate Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As the above set of database objects are typically to be compared to the contents of
a single :class:~sqlalchemy.schema.MetaData object, particularly when the
:paramref:.EnvironmentContext.configure.include_schemas flag is enabled
there is an important need to filter out unwanted "schemas", which for some
database backends might be the list of all the databases present. This
filtering is best performed using the :paramref:.EnvironmentContext.configure.include_name
hook, which provides for a callable that may return a boolean true/false
indicating if a particular schema name should be included::
def include_name(name, type_, parent_names):
if type_ == "schema":
# note this will not include the default schema
return name in ["schema_one", "schema_two"]
else:
return True
context.configure(
# ...
include_schemas = True,
include_name = include_name
)Above, when the list of schema names is first retrieved, the names will be
filtered through the above include_name function so that only schemas
named "schema_one" and "schema_two" will be considered by the
autogenerate process.
In order to include the default schema, that is, the schema that is
referred towards by the database connection without any explicit
schema being specified, the name passed to the hook is None. To alter
our above example to also include the default schema, we compare toNone as well::
def include_name(name, type_, parent_names):
if type_ == "schema":
# this **will* include the default schema
return name in [None, "schema_one", "schema_two"]
else:
return True
context.configure(
# ...
include_schemas = True,
include_name = include_name
)Omitting Table Names from the Autogenerate Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The :paramref:.EnvironmentContext.configure.include_name hook is also
most appropriate to limit the names of tables in the target database
to be considered. If a target database has many tables that are not
part of the :class:~sqlalchemy.schema.MetaData, the autogenerate process
will normally assume these are extraneous tables in the database to be
dropped, and it will generate a :meth:.Operations.drop_table operation
for each. To prevent this, the :paramref:.EnvironmentContext.configure.include_name
hook may be used to search for each name within the
:attr:~sqlalchemy.schema.MetaData.tables collection of the
:class:~sqlalchemy.schema.MetaData object and ensure names
which aren't present are not included::
target_metadata = MyModel.metadata
def include_name(name, type_, parent_names):
if type_ == "table":
return name in target_metadata.tables
else:
return True
context.configure(
# ...
target_metadata = target_metadata,
include_name = include_name,
include_schemas = False
)The above example is limited to table names present in the default schema only.
In order to search within a :class:~sqlalchemy.schema.MetaData collection for
schema-qualified table names as well, a table present in the non
default schema will be present under a name of the form<schemaname>.. The
:paramref:.EnvironmentContext.configure.include_name hook will present
this schema name on a per-tablename basis in the parent_names dictionary,
using the key "schema_name" that refers to the name of the
schema currently being considered, or None if the schema is the default
schema of the database connection::
# example fragment
if parent_names["schema_name"] is None:
return name in target_metadata.tables
else:
# build out schema-qualified name explicitly...
return (
"%s.%s" % (parent_names["schema_name"], name) in
target_metadata.tables
)However more simply, the parent_names dictionary will also include
the dot-concatenated name already constructed under the key"schema_qualified_table_name", which will also be suitably formatted
for tables in the default schema as well with the dot omitted. So the
full example of omitting tables with schema support may look like::
target_metadata = MyModel.metadata
def include_name(name, type_, parent_names):
if type_ == "schema":
return name in [None, "schema_one", "schema_two"]
elif type_ == "table":
# use schema_qualified_table_name directly
return (
parent_names["schema_qualified_table_name"] in
target_metadata.tables
)
else:
return True
context.configure(
# ...
target_metadata = target_metadata,
include_name = include_name,
include_schemas = True
)The parent_names dictionary will also include the key "table_name"
when the name being considered is that of a column or constraint object
local to a particular table.
The :paramref:.EnvironmentContext.configure.include_name hook only refers
to reflected objects, and not those located within the target
:class:~sqlalchemy.schema.MetaData collection. For more fine-grained
rules that include both :class:~sqlalchemy.schema.MetaData and reflected
object, the :paramref:.EnvironmentContext.configure.include_object hook
discussed in the next section is more appropriate.
Omitting Based on Object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The :paramref:.EnvironmentContext.configure.include_object hook provides
for object-level inclusion/exclusion rules based on the :class:~sqlalchemy.schema.Table
object being reflected as well as the elements within it. This hook can
be used to limit objects both from the local :class:~sqlalchemy.schema.MetaData
collection as well as from the target database. The limitation is that when
it reports on objects in the database, it will have fully reflected that object,
which can be expensive if a large number of objects will be omitted. The
example below refers to a fine-grained rule that will skip changes on
:class:~sqlalchemy.schema.Column objects that have a user-defined flagskip_autogenerate placed into the :attr:~sqlalchemy.schema.Column.info
dictionary::
def include_object(object, name, type_, reflected, compare_to):
if (type_ == "column" and
not reflected and
object.info.get("skip_autogenerate", False)):
return False
else:
return True
context.configure(
# ...
include_object = include_object
)Comparing and Rendering Types
The area of autogenerate's behavior of comparing and rendering Python-based type objects
in migration scripts presents a challenge, in that there's
a very wide variety of types to be rendered in scripts, including those
part of SQLAlchemy as well as user-defined types. A few options
are given to help out with this task.
.. _autogen_module_prefix:
Controlling the Module Prefix
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When types are rendered, they are generated with a module prefix, so
that they are available based on a relatively small number of imports.
The rules for what the prefix is is based on the kind of datatype as well
as configurational settings. For example, when Alembic renders SQLAlchemy
types, it will by default prefix the type name with the prefix sa.::
Column("my_column", sa.Integer())The use of the sa. prefix is controllable by altering the value
of :paramref:.EnvironmentContext.configure.sqlalchemy_module_prefix::
def run_migrations_online():
# ...
context.configure(
connection=connection,
target_metadata=target_metadata,
sqlalchemy_module_prefix="sqla.",
# ...
)
# ...In either case, the sa. prefix, or whatever prefix is desired, should
also be included in the imports section of script.py.mako; it also
defaults to import sqlalchemy as sa.
For user-defined types, that is, any custom type that
is not within the sqlalchemy. module namespace, by default Alembic will
use the value of module for the custom type::
Column("my_column", myapp.models.utils.types.MyCustomType())The imports for the above type again must be made present within the migration,
either manually, or by adding it to script.py.mako.
The above custom type has a long and cumbersome name based on the use
of __module__ directly, which also implies that lots of imports would
be needed in order to accommodate lots of types. For this reason, it is
recommended that user-defined types used in migration scripts be made
available from a single module. Suppose we call it myapp.migration_types::
# myapp/migration_types.py
from myapp.models.utils.types import MyCustomTypeWe can first add an import for migration_types to our script.py.mako::
from alembic import op
import sqlalchemy as sa
import myapp.migration_types
${imports if imports else ""}We then override Alembic's use of __module__ by providing a fixed
prefix, using the :paramref:.EnvironmentContext.configure.user_module_prefix
option::
def run_migrations_online():
# ...
context.configure(
connection=connection,
target_metadata=target_metadata,
user_module_prefix="myapp.migration_types.",
# ...
)
# ...Above, we now would get a migration like::
Column("my_column", myapp.migration_types.MyCustomType())
Now, when we inevitably refactor our application to move MyCustomType
somewhere else, we only need modify the myapp.migration_types module,
instead of searching and replacing all instances within our migration scripts.
.. _autogen_render_types:
Affecting the Rendering of Types Themselves
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The methodology Alembic uses to generate SQLAlchemy and user-defined type constructs
as Python code is plain old __repr__(). SQLAlchemy's built-in types
for the most part have a __repr__() that faithfully renders a
Python-compatible constructor call, but there are some exceptions, particularly
in those cases when a constructor accepts arguments that aren't compatible
with __repr__(), such as a pickling function.
When building a custom type that will be rendered into a migration script,
it is often necessary to explicitly give the type a __repr__() that will
faithfully reproduce the constructor for that type. This, in combination
with :paramref:.EnvironmentContext.configure.user_module_prefix, is usually
enough. However, if additional behaviors are needed, a more comprehensive
hook is the :paramref:.EnvironmentContext.configure.render_item option.
This hook allows one to provide a callable function within env.py that will fully take
over how a type is rendered, including its module prefix::
def render_item(type_, obj, autogen_context):
"""Apply custom rendering for selected items."""
if type_ == 'type' and isinstance(obj, MySpecialType):
return "mypackage.%r" % obj
# default rendering for other objects
return False
def run_migrations_online():
# ...
context.configure(
connection=connection,
target_metadata=target_metadata,
render_item=render_item,
# ...
)
# ...In the above example, we'd ensure our MySpecialType includes an appropriate__repr__() method, which is invoked when we call it against "%r".
The callable we use for :paramref:.EnvironmentContext.configure.render_item
can also add imports to our migration script. The :class:.AutogenContext passed in
contains a datamember called :attr:.AutogenContext.imports, which is a Pythonset() for which we can add new imports. For example, if MySpecialType
were in a module called mymodel.types, we can add the import for it
as we encounter the type::
def render_item(type_, obj, autogen_context):
"""Apply custom rendering for selected items."""
if type_ == 'type' and isinstance(obj, MySpecialType):
# add import for this type
autogen_context.imports.add("from mymodel import types")
return "types.%r" % obj
# default rendering for other objects
return FalseThe finished migration script will include our imports where the${imports} expression is used, producing output such as::
from alembic import op
import sqlalchemy as sa
from mymodel import types
def upgrade():
op.add_column('sometable', Column('mycolumn', types.MySpecialType()))
.. _compare_types:
Comparing Types
^^^^^^^^^^^^^^^^
The default type comparison logic will work for SQLAlchemy built in types as
well as basic user defined types. This logic is enabled by default.
It can be disabled by setting the
:paramref:.EnvironmentContext.configure.compare_type to False::
context.configure(
# ...
compare_type = False
).. versionchanged:: 1.12.0 The default value of
:paramref:.EnvironmentContext.configure.compare_type has been changed
to True.
.. note::
The default type comparison logic (which is end-user extensible) currently
(as of Alembic version 1.4.0) works by comparing the generated SQL for a
column. It does this in two steps-
First, it compares the outer type of each column such as
VARCHAR
orTEXT. Dialect implementations can have synonyms that are considered
equivalent, this is because some databases support types by converting them
to another type. For example, NUMERIC and DECIMAL are considered equivalent
on all backends, while on the Oracle backend the additional synonyms
BIGINT, INTEGER, NUMBER, SMALLINT are added to this list of equivalentsNext, the arguments within the type, such as the lengths of
strings, precision values for numerics, the elements inside of an
enumeration are compared. If BOTH columns have arguments AND they are
different, a change will be detected. If one column is just set to the
default and the other has arguments, Alembic will pass on attempting to
compare these. The rationale is that it is difficult to detect what a
database backend sets as a default value without generating false
positives.
Alternatively, the :paramref:.EnvironmentContext.configure.compare_type
parameter accepts a callable function which may be used to implement custom type
comparison logic, for cases such as where special user defined types
are being used::
def my_compare_type(context, inspected_column,
metadata_column, inspected_type, metadata_type):
# return False if the metadata_type is the same as the inspected_type
# or None to allow the default implementation to compare these
# types. a return value of True means the two types do not
# match and should result in a type change operation.
return None
context.configure(
# ...
compare_type = my_compare_type
)Above, inspected_column is a :class:sqlalchemy.schema.Column as
returned by
:meth:sqlalchemy.engine.reflection.Inspector.reflect_table, whereasmetadata_column is a :class:sqlalchemy.schema.Column from the
local model environment. A return value of None indicates that default
type comparison to proceed.
.. _post_write_hooks:
Applying Post Processing and Python Code Formatters to Generated Revisions
Revision scripts generated by the alembic revision command can optionally
be piped through a series of post-production functions which may analyze or
rewrite Python source code generated by Alembic, within the scope of running
the revision command. The primary intended use of this feature is to run
code-formatting tools such as Black <https://black.readthedocs.io/>_ orautopep8 <https://pypi.org/project/autopep8/>_, as well as custom-written
formatting and linter functions, on revision files as Alembic generates them.
Any number of hooks can be configured and they will be run in series, given the
path to the newly generated file as well as configuration options.
The post write hooks, when configured, run against generated revision files
regardless of whether or not the autogenerate feature was used.
.. note::
Alembic's post write system is partially inspired by the `pre-commit
<https://pre-commit.com/>`_ tool, which configures git hooks that reformat
source files as they are committed to a git repository. Pre-commit can
serve this role for Alembic revision files as well, applying code
formatters to them as they are committed. Alembic's post write hooks are
useful only in that they can format the files immediately upon generation,
rather than at commit time, and also can be useful for projects that prefer
not to use pre-commit... _post_write_hooks_config:
Basic Post Processor Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The template samples for alembic.ini as well as pyproject.toml for
applicable templates now include commented-out configuration
illustrating how to configure code-formatting tools, or other tools like linters
to run against the newly generated file path. Example from an alembic.ini file::
[post_write_hooks]
# format using "black"
hooks=black
black.type = console_scripts
black.entrypoint = black
black.options = -l 79 REVISION_SCRIPT_FILENAMEThe same example configured in a pyproject.toml file would look like::
[[tool.alembic.post_write_hooks]]
# format using "black"
name = "black"
type = "console_scripts"
entrypoint = "black"
options = "-l 79 REVISION_SCRIPT_FILENAME"Above, we configure hooks to be a single post write hook labeled"black". Note that this label is arbitrary. We then define the
configuration for the "black" post write hook, which includes:
type- this is the type of hook we are running. Alembic includes
three hook runners:"console_scripts", which is specifically a Python function that usessubprocess.run()to invoke a separate Python script against the revision file;"exec", which usessubprocess.run()to execute an arbitrary binary; and"module", which usessubprocess.run()to invoke a Python module directly.
For a custom-written hook function, this configuration variable would
refer to the name under which the custom hook was registered; see the
next section for an example.
.. versionadded:: 1.12 added new exec runner
.. versionadded:: 1.16.3 added new module runner
The following configuration option is specific to the "console_scripts"
hook runner:
entrypoint- the name of thesetuptools entrypoint <https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points>_
that is used to define the console script. Within the scope of standard
Python console scripts, this name will match the name of the shell command
that is usually run for the code formatting tool, in this caseblack.
The following configuration option is specific to the "exec" hook runner:
executable- the name of the executable to invoke. Can be either a
bare executable name which will be searched in$PATH, or a full pathname
to avoid potential issues with path interception.
The following options are supported by both "console_scripts" and "exec":
options- a line of command-line options that will be passed to
the code formatting tool. In this case, we want to run the commandblack /path/to/revision.py -l 79. By default, the revision path is
positioned as the first argument. In order specify a different position,
we can use theREVISION_SCRIPT_FILENAMEtoken as illustrated by the
subsequent examples... note:: Make sure options for the script are provided such that it will
rewrite the input file in place. For example, when running
autopep8, the--in-placeoption should be provided::text[post_write_hooks] hooks = autopep8 autopep8.type = console_scripts autopep8.entrypoint = autopep8 autopep8.options = --in-place REVISION_SCRIPT_FILENAMEcwd- optional working directory from which the code processing tool is run.
When running alembic revision -m "rev1", we will now see the black
tool's output as well::
$ alembic revision -m "rev1"
Generating /path/to/project/versions/481b13bc369a_rev1.py ... done
Running post write hook "black" ...
reformatted /path/to/project/versions/481b13bc369a_rev1.py
All done! โจ ๐ฐ โจ
1 file reformatted.
done
Hooks may also be specified as a list of names, which correspond to hook
runners that will run sequentially. As an example, we can also run thezimports <https://pypi.org/project/zimports/>_ import rewriting tool (written
by Alembic's author) subsequent to running the black tool. The
alembic.ini configuration would be as follows::
[post_write_hooks]
# format using "black", then "zimports"
hooks=black, zimports
black.type = console_scripts
black.entrypoint = black
black.options = -l 79 REVISION_SCRIPT_FILENAME
zimports.type = console_scripts
zimports.entrypoint = zimports
zimports.options = --style google REVISION_SCRIPT_FILENAMEThe equivalent pyproject.toml configuration would be::
# format using "black", then "zimports"
[[tool.alembic.post_write_hooks]]
name = "black"
type="console_scripts"
entrypoint = "black"
options = "-l 79 REVISION_SCRIPT_FILENAME"
[[tool.alembic.post_write_hooks]]
name = "zimports"
type="console_scripts"
entrypoint = "zimports"
options = "--style google REVISION_SCRIPT_FILENAME"When using the above configuration, a newly generated revision file will
be processed first by the "black" tool, then by the "zimports" tool.
Alternatively, one can run pre-commit itself as follows::
[post_write_hooks]
hooks = pre-commit
pre-commit.type = console_scripts
pre-commit.entrypoint = pre-commit
pre-commit.options = run --files REVISION_SCRIPT_FILENAME
pre-commit.cwd = %(here)s
(The last line helps to ensure that the .pre-commit-config.yaml file
will always be found, regardless of from where the hook was called.)
.. _post_write_hooks_custom:
Writing Custom Hooks as Python Functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The previous section illustrated how to run command-line code formatters,
through the use of a post write hook provided by Alembic known asconsole_scripts. This hook is in fact a Python function that is registered
under that name using a registration function that may be used to register
other types of hooks as well.
To illustrate, we will use the example of a short Python function that wants
to rewrite the generated code to use tabs instead of four spaces. For simplicity,
we will illustrate how this function can be present directly in the env.py
file. The function is declared and registered using the :func:.write_hooks.register
decorator::
from alembic.script import write_hooks
import re
@write_hooks.register("spaces_to_tabs")
def convert_spaces_to_tabs(filename, options):
lines = []
with open(filename) as file_:
for line in file_:
lines.append(
re.sub(
r"^( )+",
lambda m: "\t" * (len(m.group(1)) // 4),
line
)
)
with open(filename, "w") as to_write:
to_write.write("".join(lines))Our new "spaces_to_tabs" hook can be configured in alembic.ini as follows::
[alembic]
...
ensure the revision command loads env.py
revision_environment = true
[post_write_hooks]
hooks = spaces_to_tabs
spaces_to_tabs.type = spaces_to_tabs
When alembic revision is run, the env.py file will be loaded in all
cases, the custom "spaces_to_tabs" function will be registered and it will then
be run against the newly generated file path::
$ alembic revision -m "rev1"
Generating /path/to/project/versions/481b13bc369a_rev1.py ... done
Running post write hook "spaces_to_tabs" ...
done
.. _alembic_check:
Running Alembic Check to test for new upgrade operations
When developing code it's useful to know if a set of code changes has made any
net change to the database model, such that new revisions would need to be
generated. To automate this, Alembic provides the alembic check command.
This command will run through the same process asalembic revision --autogenerate, up until the point where revision files
would be generated, however does not generate any new files. Instead, it
returns an error code plus a message if it is detected that new operations
would be rendered into a new revision, or if not, returns a success code plus a
message. When alembic check returns a success code, this is an indication
that the alembic revision --autogenerate command would produce only empty
migrations, and does not need to be run.
alembic check can be worked into CI systems and on-commit schemes to ensure
that incoming code does not warrant new revisions to be generated. In
the example below, a check that detects new operations is illustrated::
$ alembic check
FAILED: New upgrade operations detected: [
('add_column', None, 'my_table', Column('data', String(), table=<my_table>)),
('add_column', None, 'my_table', Column('newcol', Integer(), table=<my_table>))]by contrast, when no new operations are detected::
$ alembic check
No new upgrade operations detected... versionadded:: 1.9.0
.. note:: The alembic check command uses the same model comparison process
as the alembic revision --autogenerate process. This means parameters
such as :paramref:.EnvironmentContext.configure.compare_type
and :paramref:.EnvironmentContext.configure.compare_server_default
are in play as usual, as well as that limitations in autogenerate
detection are the same when running alembic check.
Build Batch (`docs/build/batch.rst`)
.. _batch_migrations:
Running "Batch" Migrations for SQLite and Other Databases
The SQLite database presents a challenge to migration tools
in that it has almost no support for the ALTER statement which
relational schema migrations rely upon. The rationale for this stems from
philosophical and architectural concerns within SQLite, and they are unlikely
to be changed.
Migration tools are instead expected to produce copies of SQLite tables that
correspond to the new structure, transfer the data from the existing
table to the new one, then drop the old table. For our purposes here
we'll call this "move and copy" workflow, and in order to accommodate it
in a way that is reasonably predictable, while also remaining compatible
with other databases, Alembic provides the batch operations context.
Within this context, a relational table is named, and then a series of
mutation operations to that table alone are specified within
the block. When the context is complete, a process begins whereby the
"move and copy" procedure begins; the existing table structure is reflected
from the database, a new version of this table is created with the given
changes, data is copied from the
old table to the new table using "INSERT from SELECT", and finally the old
table is dropped and the new one renamed to the original name.
The :meth:.Operations.batch_alter_table method provides the gateway to this
process::
with op.batch_alter_table("some_table") as batch_op:
batch_op.add_column(Column('foo', Integer))
batch_op.drop_column('bar')When the above directives are invoked within a migration script, on a
SQLite backend we would see SQL like:
.. sourcecode:: sql
CREATE TABLE _alembic_batch_temp (
id INTEGER NOT NULL,
foo INTEGER,
PRIMARY KEY (id)
);
INSERT INTO _alembic_batch_temp (id) SELECT some_table.id FROM some_table;
DROP TABLE some_table;
ALTER TABLE _alembic_batch_temp RENAME TO some_table;On other backends, we'd see the usual ALTER statements done as though
there were no batch directive - the batch context by default only does
the "move and copy" process if SQLite is in use, and if there are
migration directives other than :meth:.Operations.add_column present,
which is the one kind of column-level ALTER statement that SQLite supports.
:meth:.Operations.batch_alter_table can be configured
to run "move and copy" unconditionally in all cases, including on databases
other than SQLite; more on this is below.
.. _batch_controlling_table_reflection:
Controlling Table Reflection
The :class:~sqlalchemy.schema.Table object that is reflected when
"move and copy" proceeds is performed using the standard autoload=True
approach. This call can be affected using the
:paramref:~.Operations.batch_alter_table.reflect_args and
:paramref:~.Operations.batch_alter_table.reflect_kwargs arguments.
For example, to override a :class:~sqlalchemy.schema.Column within
the reflection process such that a :class:~sqlalchemy.types.Boolean
object is reflected with the create_constraint flag set to False::
with self.op.batch_alter_table(
"bar",
reflect_args=[Column('flag', Boolean(create_constraint=False))]
) as batch_op:
batch_op.alter_column(
'flag', new_column_name='bflag', existing_type=Boolean)Another use case, add a listener to the :class:~sqlalchemy.schema.Table
as it is reflected so that special logic can be applied to columns or
types, using the :meth:~sqlalchemy.events.DDLEvents.column_reflect event::
def listen_for_reflect(inspector, table, column_info):
"correct an ENUM type"
if column_info['name'] == 'my_enum':
column_info['type'] = Enum('a', 'b', 'c')
with self.op.batch_alter_table(
"bar",
reflect_kwargs=dict(
listeners=[
('column_reflect', listen_for_reflect)
]
)
) as batch_op:
batch_op.alter_column(
'flag', new_column_name='bflag', existing_type=Boolean)The reflection process may also be bypassed entirely by sending a
pre-fabricated :class:~sqlalchemy.schema.Table object; see
:ref:batch_offline_mode for an example.
.. _sqlite_batch_constraints:
Dealing with Constraints
There are a variety of issues when using "batch" mode with constraints,
such as FOREIGN KEY, CHECK and UNIQUE constraints. This section
will attempt to detail many of these scenarios.
.. _dropping_sqlite_foreign_keys:
Dropping Unnamed or Named Foreign Key Constraints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SQLite, unlike any other database, allows constraints to exist in the
database that have no identifying name. On all other backends, the
target database will always generate some kind of name, if one is not
given.
The challenge this represents is that an unnamed constraint can't
by itself be targeted by the :meth:.BatchOperations.drop_constraint method.
An unnamed FOREIGN KEY constraint is implicit whenever the
:class:~sqlalchemy.schema.ForeignKey
or :class:~sqlalchemy.schema.ForeignKeyConstraint objects are used without
passing them a name. Only on SQLite will these constraints remain entirely
unnamed when they are created on the target database; an automatically generated
name will be assigned in the case of all other database backends.
Within the scope of batch mode, this presents the issue that the
:meth:.BatchOperations.drop_constraint method requires a constraint name
in order to target the correct constraint.
In order to overcome this, the :meth:.Operations.batch_alter_table method supports a
:paramref:~.Operations.batch_alter_table.naming_convention argument, so that
all reflected constraints, including foreign keys that are unnamed may be
given a name, as described in :ref:autogen_naming_conventions.
Usage is as follows::
naming_convention = {
"fk":
"fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
}
with self.op.batch_alter_table(
"bar", naming_convention=naming_convention) as batch_op:
batch_op.drop_constraint(
"fk_bar_foo_id_foo", type_="foreignkey")Note that the naming convention feature requires at least
SQLAlchemy 0.9.4 for support.
.. _batch_schematype_constraints:
Changing the Type of Boolean, Enum and other implicit CHECK datatypes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The SQLAlchemy types :class:~sqlalchemy.types.Boolean and
:class:~sqlalchemy.types.Enum are part of a category of types known as
"schema" types; this style of type creates other structures along with the
type itself, most commonly (but not always) a CHECK constraint.
Alembic handles dropping and creating the CHECK constraints here automatically,
including in the case of batch mode. When changing the type of an existing
column, what's necessary is that the existing type be specified fully::
with self.op.batch_alter_table("some_table") as batch_op:
batch_op.alter_column(
'q', type_=Integer,
existing_type=Boolean(create_constraint=True, constraint_name="ck1"))
When dropping a column that includes a named CHECK constraint, as of Alembic
1.7 this named constraint must also be provided using a similar form, as there
is no ability for Alembic to otherwise link this reflected CHECK constraint as
belonging to a particular column::
with self.op.batch_alter_table("some_table") as batch_op:
batch_op.drop_column(
'q',
existing_type=Boolean(create_constraint=True, constraint_name="ck1"))
).. versionchanged:: 1.7 The :meth:.BatchOperations.drop_column operation can
accept an existing_type directive where a "schema type" such as
:class:~sqlalchemy.types.Boolean and :class:~sqlalchemy.types.Enum may
be specified such that an associated named constraint can be removed.
.. _batch_check_constraints:
Including CHECK constraints
^^^^^^^^^^^^^^^^^^^^^^^^^^^
As of Alembic 1.7, named CHECK constraints are automatically included
in batch mode, as modern SQLAlchemy versions are capable of reflecting these
constraints like any other constraint.
Note that when dropping or renaming a column that is mentioned in a named
CHECK constraint, this CHECK constraint must be explicitly dropped first,
as Alembic has no means of linking a reflected CHECK constraint to that
column. Supposing column q of some_table were mentioned in a CHECK
constraint named ck1. In order to drop this column, we have to drop
the check constraint also::
with self.op.batch_alter_table("some_table") as batch_op:
batch_op.drop_constraint("ck1", type_="check")
batch_op.drop_column('q').. versionchanged:: 1.7 Named CHECK constraints participate in batch mode
in the same way as any other kind of constraint. This requires that column
drops or renames now include explicit directives to drop an existing named
constraint which refers to this column, as it will otherwise not be
automatically detected as being associated with that particular column.
Unnamed CHECK constraints continue to be omitted from the table
recreate operation; up until version 1.20, this omission was silent.
As of 1.20, it emits a warning.
For unnamed CHECK constraints, these are still not automatically included
as part of the batch process. Note that this limitation includes the CHECK
constraints generated by the
:class:~sqlalchemy.types.Boolean or :class:~sqlalchemy.types.Enum
datatypes, which up through SQLAlchemy 1.3 would generate CHECK constraints
automatically and cannot be tracked to the reflected table, assuming they are
generated in an unnamed way.
Unnamed constraints can be stated explicitly if they are to be included in the
recreated table::
with op.batch_alter_table("some_table", table_args=[
CheckConstraint('x > 5')
]) as batch_op:
batch_op.add_column(Column('foo', Integer))
batch_op.drop_column('bar')The above step needs only be taken for CHECK constraints that are explicitly stated
as part of the table definition.
.. versionadded:: 1.20 A warning is emitted when an unnamed CHECK constraint
on a reflected table is omitted from the recreate, and no
CheckConstraint objects are passed to the
:paramref:~.Operations.batch_alter_table.table_args collection.
Restating the constraint as above is needed for each recreate of the table.
The longer term remedy is for the CHECK constraints in the model to be given
names, either explicitly or via a naming convention on the
:class:~sqlalchemy.schema.MetaData; once a recreate has emitted the
constraints with names, subsequent reflection of the table carries those names
and the constraints participate in batch mode like any other.
For CHECK constraints that are generated by datatypes such as
:class:~sqlalchemy.types.Boolean or :class:~sqlalchemy.types.Enum, the type
objects themselves must be named in order for their CHECK constraints to be
included in the batch process. Boolean and Enum datatypes that do not
have the .name attribute set will not have CHECK constraints
regenerated. This name can be set by specifying the .name parameter
or by using a named Python Enum object as the source of enumeration.
Dealing with Referencing Foreign Keys
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is important to note that batch table operations do not work with
foreign keys that enforce referential integrity. This because the
target table is dropped; if foreign keys refer to it, this will raise
an error. On SQLite, whether or not foreign keys actually enforce is
controlled by the PRAGMA FOREIGN KEYS pragma; this pragma, if in use,
must be disabled when the workflow mode proceeds. When the operation is
complete, the batch-migrated table will have the same name
that it started with, so those referring foreign keys will again
refer to this table.
A special case is dealing with self-referring foreign keys. Here,
Alembic takes a special step of recreating the self-referring foreign key
as referring to the original table name, rather than at the "temp" table,
so that like in the case of other foreign key constraints, when the table
is renamed to its original name, the foreign key
again references the correct table. This operation only works when
referential integrity is disabled, consistent with the same requirement
for referring foreign keys from other tables.
When SQLite's PRAGMA FOREIGN KEYS mode is turned on, it does provide
the service that foreign key constraints, including self-referential, will
automatically be modified to point to their table across table renames,
however this mode prevents the target table from being dropped as is required
by a batch migration. Therefore it may be necessary to manipulate thePRAGMA FOREIGN KEYS setting if a migration seeks to rename a table vs.
batch migrate it.
.. _batch_offline_mode:
Working in Offline Mode
In the preceding sections, we've seen how much of an emphasis the
"move and copy" process has on using reflection in order to know the
structure of the table that is to be copied. This means that in the typical
case, "online" mode, where a live database connection is present so that
:meth:.Operations.batch_alter_table can reflect the table from the
database, is required; the --sql flag cannot be used without extra
steps.
To support offline mode, the system must work without table reflection
present, which means the full table as it intends to be created must be
passed to :meth:.Operations.batch_alter_table using
:paramref:~.Operations.batch_alter_table.copy_from::
meta = MetaData()
some_table = Table(
'some_table', meta,
Column('id', Integer, primary_key=True),
Column('bar', String(50))
)
with op.batch_alter_table("some_table", copy_from=some_table) as batch_op:
batch_op.add_column(Column('foo', Integer))
batch_op.drop_column('bar')The above use pattern is pretty tedious and quite far off from Alembic's
preferred style of working; however, if one needs to do SQLite-compatible
"move and copy" migrations and need them to generate flat SQL files in
"offline" mode, there's not much alternative.
Batch mode with Autogenerate
The syntax of batch mode is essentially that :meth:.Operations.batch_alter_table
is used to enter a batch block, and the returned :class:.BatchOperations context
works just like the regular :class:.Operations context, except that
the "table name" and "schema name" arguments are omitted.
To support rendering of migration commands in batch mode for autogenerate,
configure the :paramref:.EnvironmentContext.configure.render_as_batch
flag in env.py::
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True
)Autogenerate will now generate along the lines of::
def upgrade():
### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('address', schema=None) as batch_op:
batch_op.add_column(sa.Column('street', sa.String(length=50), nullable=True))This mode is safe to use in all cases, as the :meth:.Operations.batch_alter_table
directive by default only takes place for SQLite; other backends will
behave just as they normally do in the absence of the batch directives.
Note that autogenerate support does not include "offline" mode, where
the :paramref:.Operations.batch_alter_table.copy_from parameter is used.
The table definition here would need to be entered into migration files
manually if this is needed.
Batch mode with databases other than SQLite
There's an odd use case some shops have, where the "move and copy" style
of migration is useful in some cases for databases that do already support
ALTER. There's some cases where an ALTER operation may block access to the
table for a long time, which might not be acceptable. "move and copy" can
be made to work on other backends, though with a few extra caveats.
The batch mode directive will run the "recreate" system regardless of
backend if the flag recreate='always' is passed::
with op.batch_alter_table("some_table", recreate='always') as batch_op:
batch_op.add_column(Column('foo', Integer))The issues that arise in this mode are mostly to do with constraints.
Databases such as Postgresql and MySQL with InnoDB will enforce referential
integrity (e.g. via foreign keys) in all cases. Unlike SQLite, it's not
as simple to turn off referential integrity across the board (nor would it
be desirable). Since a new table is replacing the old one, existing
foreign key constraints which refer to the target table will need to be
unconditionally dropped before the batch operation, and re-created to refer
to the new table afterwards. Batch mode currently does not provide any
automation for this.
The Postgresql database and possibly others also have the behavior such
that when the new table is created, a naming conflict occurs with the
named constraints of the new table, in that they match those of the old
table, and on Postgresql, these names need to be unique across all tables.
The Postgresql dialect will therefore emit a "DROP CONSTRAINT" directive
for all constraints on the old table before the new one is created; this is
"safe" in case of a failed operation because Postgresql also supports
transactional DDL.
Note that also as is the case with SQLite, CHECK constraints need to be
moved over between old and new table manually using the
:paramref:.Operations.batch_alter_table.table_args parameter.
Build Branches (`docs/build/branches.rst`)
.. _branches:
Working with Branches
A branch describes a point in a migration stream when two or more
versions refer to the same parent migration as their ancestor. Branches
occur naturally when two divergent source trees, both containing Alembic
revision files created independently within those source trees, are merged
together into one. When this occurs, the challenge of a branch is to merge the
branches into a single series of changes, so that databases established
from either source tree individually can be upgraded to reference the merged
result equally. Another scenario where branches are present are when we create them
directly; either at some point in the migration stream we'd like different
series of migrations to be managed independently (e.g. we create a tree),
or we'd like separate migration streams for different features starting
at the root (e.g. a forest). We'll illustrate all of these cases, starting
with the most common which is a source-merge-originated branch that we'll
merge.
Starting with the "account table" example we began in :ref:create_migration,
assume we have our basemost version 1975ea83b712, which leads into
the second revision ae1027a6acf, and the migration files for these
two revisions are checked into our source repository.
Consider if we merged into our source repository another code branch which contained
a revision for another table called shopping_cart. This revision was made
against our first Alembic revision, the one that generated account. After
loading the second source tree in, a new file27c6a30d7c24_add_shopping_cart_table.py exists within our versions directory.
Both it, as well as ae1027a6acf_add_a_column.py, reference1975ea83b712_add_account_table.py as the "downgrade" revision. To illustrate::
# main source tree:
1975ea83b712 (create account table) -> ae1027a6acf (add a column)
# branched source tree
1975ea83b712 (create account table) -> 27c6a30d7c24 (add shopping cart table)Above, we can see 1975ea83b712 is our branch point; two distinct versions
both refer to it as its parent. The Alembic command branches illustrates
this fact::
$ alembic branches --verbose
Rev: 1975ea83b712 (branchpoint)
Parent:
Branches into: 27c6a30d7c24, ae1027a6acf
Path: foo/versions/1975ea83b712_add_account_table.py
create account table
Revision ID: 1975ea83b712
Revises:
Create Date: 2014-11-20 13:02:46.257104
-> 27c6a30d7c24 (head), add shopping cart table
-> ae1027a6acf (head), add a columnHistory shows it too, illustrating two head entries as well
as a branchpoint::
$ alembic history
1975ea83b712 -> 27c6a30d7c24 (head), add shopping cart table
1975ea83b712 -> ae1027a6acf (head), add a column
-> 1975ea83b712 (branchpoint), create account tableWe can get a view of just the current heads using alembic heads::
$ alembic heads --verbose
Rev: 27c6a30d7c24 (head)
Parent: 1975ea83b712
Path: foo/versions/27c6a30d7c24_add_shopping_cart_table.py
add shopping cart table
Revision ID: 27c6a30d7c24
Revises: 1975ea83b712
Create Date: 2014-11-20 13:03:11.436407
Rev: ae1027a6acf (head)
Parent: 1975ea83b712
Path: foo/versions/ae1027a6acf_add_a_column.py
add a column
Revision ID: ae1027a6acf
Revises: 1975ea83b712
Create Date: 2014-11-20 13:02:54.849677If we try to run an upgrade to the usual end target of head, Alembic no
longer considers this to be an unambiguous command. As we have more than
one head, the upgrade command wants us to provide more information::
$ alembic upgrade head
FAILED: Multiple head revisions are present for given argument 'head'; please specify a specific
target revision, '@head' to narrow to a specific head, or 'heads' for all headsThe upgrade command gives us quite a few options in which we can proceed
with our upgrade, either giving it information on which head we'd like to upgrade
towards, or alternatively stating that we'd like all heads to be upgraded
towards at once. However, in the typical case of two source trees being
merged, we will want to pursue a third option, which is that we can merge these
branches.
Merging Branches
An Alembic merge is a migration file that joins two or
more "head" files together. If the two branches we have right now can
be said to be a "tree" structure, introducing this merge file will
turn it into a "diamond" structure::
-- ae1027a6acf -->
/ \
--> 1975ea83b712 --> --> mergepoint
\ /
-- 27c6a30d7c24 -->We create the merge file using alembic merge; with this command, we can
pass to it an argument such as heads, meaning we'd like to merge all
heads. Or, we can pass it individual revision numbers sequentially::
$ alembic merge -m "merge ae1 and 27c" ae1027 27c6a
Generating /path/to/foo/versions/53fffde5ad5_merge_ae1_and_27c.py ... doneLooking inside the new file, we see it as a regular migration file, with
the only new twist is that down_revision points to both revisions::
"""merge ae1 and 27c
Revision ID: 53fffde5ad5
Revises: ae1027a6acf, 27c6a30d7c24
Create Date: 2014-11-20 13:31:50.811663
"""
# revision identifiers, used by Alembic.
revision = '53fffde5ad5'
down_revision = ('ae1027a6acf', '27c6a30d7c24')
branch_labels = None
from alembic import op
import sqlalchemy as sa
def upgrade():
pass
def downgrade():
passThis file is a regular migration file, and if we wish to, we may place
:class:.Operations directives into the upgrade() and downgrade()
functions like any other migration file. Though it is probably best to limit
the instructions placed here only to those that deal with any kind of
reconciliation that is needed between the two merged branches, if any.
The heads command now illustrates that the multiple heads in ourversions/ directory have been resolved into our new head::
$ alembic heads --verbose
Rev: 53fffde5ad5 (head) (mergepoint)
Merges: ae1027a6acf, 27c6a30d7c24
Path: foo/versions/53fffde5ad5_merge_ae1_and_27c.py
merge ae1 and 27c
Revision ID: 53fffde5ad5
Revises: ae1027a6acf, 27c6a30d7c24
Create Date: 2014-11-20 13:31:50.811663History shows a similar result, as the mergepoint becomes our head::
$ alembic history
ae1027a6acf, 27c6a30d7c24 -> 53fffde5ad5 (head) (mergepoint), merge ae1 and 27c
1975ea83b712 -> ae1027a6acf, add a column
1975ea83b712 -> 27c6a30d7c24, add shopping cart table
-> 1975ea83b712 (branchpoint), create account tableWith a single head target, a generic upgrade can proceed::
$ alembic upgrade head
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade -> 1975ea83b712, create account table
INFO [alembic.migration] Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart table
INFO [alembic.migration] Running upgrade 1975ea83b712 -> ae1027a6acf, add a column
INFO [alembic.migration] Running upgrade ae1027a6acf, 27c6a30d7c24 -> 53fffde5ad5, merge ae1 and 27c.. topic:: merge mechanics
The upgrade process traverses through all of our migration files using
a topological sorting algorithm, treating the list of migration
files not as a linked list, but as a directed acyclic graph. The starting
points of this traversal are the current heads within our database,
and the end point is the "head" revision or revisions specified.
When a migration proceeds across a point at which there are multiple heads,
the alembic_version table will at that point store multiple rows,
one for each head. Our migration process above will emit SQL against
alembic_version along these lines:
.. sourcecode:: sql
-- Running upgrade -> 1975ea83b712, create account table
INSERT INTO alembic_version (version_num) VALUES ('1975ea83b712')
-- Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart table
UPDATE alembic_version SET version_num='27c6a30d7c24' WHERE alembic_version.version_num = '1975ea83b712'
-- Running upgrade 1975ea83b712 -> ae1027a6acf, add a column
INSERT INTO alembic_version (version_num) VALUES ('ae1027a6acf')
-- Running upgrade ae1027a6acf, 27c6a30d7c24 -> 53fffde5ad5, merge ae1 and 27c
DELETE FROM alembic_version WHERE alembic_version.version_num = 'ae1027a6acf'
UPDATE alembic_version SET version_num='53fffde5ad5' WHERE alembic_version.version_num = '27c6a30d7c24' At the point at which both 27c6a30d7c24 and ae1027a6acf exist within our
database, both values are present in alembic_version, which now has
two rows. If we upgrade to these two versions alone, then stop and
run alembic current, we will see this::
$ alembic current --verbose
Current revision(s) for postgresql://scott:XXXXX@localhost/test:
Rev: ae1027a6acf
Parent: 1975ea83b712
Path: foo/versions/ae1027a6acf_add_a_column.py
add a column
Revision ID: ae1027a6acf
Revises: 1975ea83b712
Create Date: 2014-11-20 13:02:54.849677
Rev: 27c6a30d7c24
Parent: 1975ea83b712
Path: foo/versions/27c6a30d7c24_add_shopping_cart_table.py
add shopping cart table
Revision ID: 27c6a30d7c24
Revises: 1975ea83b712
Create Date: 2014-11-20 13:03:11.436407 A key advantage to the merge process is that it will
run equally well on databases that were present on version ae1027a6acf
alone, versus databases that were present on version 27c6a30d7c24 alone;
whichever version was not yet applied, will be applied before the merge point
can be crossed. This brings forth a way of thinking about a merge file,
as well as about any Alembic revision file. As they are considered to
be "nodes" within a set that is subject to topological sorting, each
"node" is a point that cannot be crossed until all of its dependencies
are satisfied.
Prior to Alembic's support of merge points, the use case of databases
sitting on different heads was basically impossible to reconcile; having
to manually splice the head files together invariably meant that one migration
would occur before the other, thus being incompatible with databases that
were present on the other migration.
Working with Explicit Branches
The alembic upgrade command hinted at other options besides merging when
dealing with multiple heads. Let's back up and assume we're back where
we have as our heads just ae1027a6acf and 27c6a30d7c24::
$ alembic heads
27c6a30d7c24
ae1027a6acfEarlier, when we did alembic upgrade head, it gave us an error which
suggested please specify a specific target revision, '@head' to narrow to a specific head, or 'heads' for all heads in order to proceed
without merging. Let's cover those cases.
Referring to all heads at once
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The heads identifier is a lot like head, except it explicitly refers
to all heads at once. That is, it's like telling Alembic to do the operation
for both ae1027a6acf and 27c6a30d7c24 simultaneously. If we started
from a fresh database and ran upgrade heads we'd see::
$ alembic upgrade heads
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade -> 1975ea83b712, create account table
INFO [alembic.migration] Running upgrade 1975ea83b712 -> ae1027a6acf, add a column
INFO [alembic.migration] Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart tableSince we've upgraded to heads, and we do in fact have more than one head,
that means these two distinct heads are now in our alembic_version table.
We can see this if we run alembic current::
$ alembic current
ae1027a6acf (head)
27c6a30d7c24 (head)That means there's two rows in alembic_version right now. If we downgrade
one step at a time, Alembic will delete from the alembic_version table
each branch that's closed out, until only one branch remains; then it will
continue updating the single value down to the previous versions::
$ alembic downgrade -1
INFO [alembic.migration] Running downgrade ae1027a6acf -> 1975ea83b712, add a column
$ alembic current
27c6a30d7c24 (head)
$ alembic downgrade -1
INFO [alembic.migration] Running downgrade 27c6a30d7c24 -> 1975ea83b712, add shopping cart table
$ alembic current
1975ea83b712 (branchpoint)
$ alembic downgrade -1
INFO [alembic.migration] Running downgrade 1975ea83b712 -> , create account table
$ alembic currentReferring to a Specific Version
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We can pass a specific version number to upgrade. Alembic will ensure that
all revisions upon which this version depends are invoked, and nothing more.
So if we upgrade either to 27c6a30d7c24 or ae1027a6acf specifically,
it guarantees that 1975ea83b712 will have been applied, but not that
any "sibling" versions are applied::
$ alembic upgrade 27c6a
INFO [alembic.migration] Running upgrade -> 1975ea83b712, create account table
INFO [alembic.migration] Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart tableWith 1975ea83b712 and 27c6a30d7c24 applied, ae1027a6acf is just
a single additional step::
$ alembic upgrade ae102
INFO [alembic.migration] Running upgrade 1975ea83b712 -> ae1027a6acf, add a columnWorking with Branch Labels
^^^^^^^^^^^^^^^^^^^^^^^^^^
To satisfy the use case where an environment has long-lived branches, especially
independent branches as will be discussed in the next section, Alembic supports
the concept of branch labels. These are string values that are present
within the migration file, using the new identifier branch_labels.
For example, if we want to refer to the "shopping cart" branch using the name
"shoppingcart", we can add that name to our file27c6a30d7c24_add_shopping_cart_table.py::
"""add shopping cart table
"""
# revision identifiers, used by Alembic.
revision = '27c6a30d7c24'
down_revision = '1975ea83b712'
branch_labels = ('shoppingcart',)
# ...The branch_labels attribute refers to a string name, or a tuple
of names, which will now apply to this revision, all descendants of this
revision, as well as all ancestors of this revision up until the preceding
branch point, in this case 1975ea83b712. We can see the shoppingcart
label applied to this revision::
$ alembic history
1975ea83b712 -> 27c6a30d7c24 (shoppingcart) (head), add shopping cart table
1975ea83b712 -> ae1027a6acf (head), add a column
-> 1975ea83b712 (branchpoint), create account tableWith the label applied, the name shoppingcart now serves as an alias
for the 27c6a30d7c24 revision specifically. We can illustrate this
by showing it with alembic show::
$ alembic show shoppingcart
Rev: 27c6a30d7c24 (head)
Parent: 1975ea83b712
Branch names: shoppingcart
Path: foo/versions/27c6a30d7c24_add_shopping_cart_table.py
add shopping cart table
Revision ID: 27c6a30d7c24
Revises: 1975ea83b712
Create Date: 2014-11-20 13:03:11.436407However, when using branch labels, we usually want to use them using a syntax
known as "branch at" syntax; this syntax allows us to state that we want to
use a specific revision, let's say a "head" revision, in terms of a specific
branch. While normally, we can't refer to alembic upgrade head when
there's multiple heads, we can refer to this head specifically usingshoppingcart@head syntax::
$ alembic upgrade shoppingcart@head
INFO [alembic.migration] Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart tableThe shoppingcart@head syntax becomes important to us if we wish to
add new migration files to our versions directory while maintaining multiple
branches. Just like the upgrade command, if we attempted to add a new
revision file to our multiple-heads layout without a specific parent revision,
we'd get a familiar error::
$ alembic revision -m "add a shopping cart column"
FAILED: Multiple heads are present; please specify the head revision on
which the new revision should be based, or perform a merge.The alembic revision command is pretty clear in what we need to do;
to add our new revision specifically to the shoppingcart branch,
we use the --head argument, either with the specific revision identifier27c6a30d7c24, or more generically using our branchname shoppingcart@head::
$ alembic revision -m "add a shopping cart column" --head shoppingcart@head
Generating /path/to/foo/versions/d747a8a8879_add_a_shopping_cart_column.py ... donealembic history shows both files now part of the shoppingcart branch::
$ alembic history
1975ea83b712 -> ae1027a6acf (head), add a column
27c6a30d7c24 -> d747a8a8879 (shoppingcart) (head), add a shopping cart column
1975ea83b712 -> 27c6a30d7c24 (shoppingcart), add shopping cart table
-> 1975ea83b712 (branchpoint), create account tableWe can limit our history operation just to this branch as well::
$ alembic history -r shoppingcart:
27c6a30d7c24 -> d747a8a8879 (shoppingcart) (head), add a shopping cart column
1975ea83b712 -> 27c6a30d7c24 (shoppingcart), add shopping cart tableIf we want to illustrate the path of shoppingcart all the way from the
base, we can do that as follows::
$ alembic history -r :shoppingcart@head
27c6a30d7c24 -> d747a8a8879 (shoppingcart) (head), add a shopping cart column
1975ea83b712 -> 27c6a30d7c24 (shoppingcart), add shopping cart table
-> 1975ea83b712 (branchpoint), create account tableWe can run this operation from the "base" side as well, but we get a different
result::
$ alembic history -r shoppingcart@base:
1975ea83b712 -> ae1027a6acf (head), add a column
27c6a30d7c24 -> d747a8a8879 (shoppingcart) (head), add a shopping cart column
1975ea83b712 -> 27c6a30d7c24 (shoppingcart), add shopping cart table
-> 1975ea83b712 (branchpoint), create account tableWhen we list from shoppingcart@base without an endpoint, it's really shorthand
for -r shoppingcart@base:heads, e.g. all heads, and since shoppingcart@base
is the same "base" shared by the ae1027a6acf revision, we get that
revision in our listing as well. The @base syntax can be
useful when we are dealing with individual bases, as we'll see in the next
section.
The @head format can also be used with revision numbers
instead of branch names, though this is less convenient. If we wanted to
add a new revision to our branch that includes the un-labeled ae1027a6acf,
if this weren't a head already, we could ask for the "head of the branch
that includes ae1027a6acf" as follows::
$ alembic revision -m "add another account column" --head ae10@head
Generating /path/to/foo/versions/55af2cb1c267_add_another_account_column.py ... doneMore Label Syntaxes
^^^^^^^^^^^^^^^^^^^
The heads symbol can be combined with a branch label, in the case that
your labeled branch itself breaks off into multiple branches::
$ alembic upgrade shoppingcart@headsRelative identifiers, as introduced in :ref:relative_migrations,
work with labels too. For example, upgrading to shoppingcart@+2
means to upgrade from current heads on "shoppingcart" upwards two revisions::
$ alembic upgrade shoppingcart@+2This kind of thing works from history as well::
$ alembic history -r current:shoppingcart@+2The newer relnum+delta format can be combined as well, for example
if we wanted to list along shoppingcart up until two revisions
before the head::
$ alembic history -r :shoppingcart@head-2.. _multiple_bases:
Working with Multiple Bases
.. note:: The multiple base feature is intended to allow for multiple Alembic
versioning lineages which share the same alembic_version table. This is
so that individual revisions within the lineages can have cross-dependencies
on each other. For the simpler case where one project has multiple,
completely independent revision lineages that refer to separate
alembic_version tables, see the example in :ref:multiple_environments.
We've seen in the previous section that alembic upgrade is fine
if we have multiple heads, alembic revision allows us to tell it which
"head" we'd like to associate our new revision file with, and branch labels
allow us to assign names to branches that we can use in subsequent commands.
Let's put all these together and refer to a new "base", that is, a whole
new tree of revision files that will be semi-independent of the account/shopping
cart revisions we've been working with. This new tree will deal with
database tables involving "networking".
.. _multiple_version_directories:
Setting up Multiple Version Directories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
While optional, it is often the case that when working with multiple bases,
we'd like different sets of version files to exist within their own directories;
typically, if an application is organized into several sub-modules, each
one would have a version directory containing migrations pertinent to
that module. So to start out, we can edit alembic.ini to refer
to multiple directories; we'll also state the current versions
directory as one of them::
A separator for the location paths needs to be defined
path_separator = os # Use os.pathsep.
version location specification; this defaults
to foo/versions. When using multiple version
directories, initial revisions must be specified with --version-path
version_locations = %(here)s/model/networking:%(here)s/alembic/versions
The new directory %(here)s/model/networking is in terms of where
the alembic.ini file is, as we are using the symbol %(here)s which
resolves to this location. When we create our first new revision
targeted at this directory,model/networking will be created automatically if it does not
exist yet. Once we've created a revision here, the path is used automatically
when generating subsequent revision files that refer to this revision tree.
Creating a Labeled Base Revision
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We also want our new branch to have its own name, and for that we want to
apply a branch label to the base. In order to achieve this using thealembic revision command without editing, we need to ensure ourscript.py.mako file, used
for generating new revision files, has the appropriate substitutions present.
If Alembic version 0.7.0 or greater was used to generate the original
migration environment, this is already done. However when working with an older
environment, script.py.mako needs to have this directive added, typically
underneath the down_revision directive::
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
# add this here in order to use revision with branch_label
branch_labels = ${repr(branch_labels)}With this in place, we can create a new revision file, starting up a branch
that will deal with database tables involving networking; we specify the--head version of base, a --branch-label of networking,
and the directory we want this first revision file to be
placed in with --version-path::
$ alembic revision -m "create networking branch" --head=base --branch-label=networking --version-path=model/networking
Creating directory /path/to/foo/model/networking ... done
Generating /path/to/foo/model/networking/3cac04ae8714_create_networking_branch.py ... doneIf we ran the above command and we didn't have the newer script.py.mako
directive, we'd get this error::
FAILED: Version 3cac04ae8714 specified branch_labels networking, however
the migration file foo/model/networking/3cac04ae8714_create_networking_branch.py
does not have them; have you upgraded your script.py.mako to include the 'branch_labels'
section?
When we receive the above error, and we would like to try again, we need to
either delete the incorrectly generated file in order to run revision
again, or we can edit the 3cac04ae8714_create_networking_branch.py
directly to add the branch_labels in of our choosing.
Running with Multiple Bases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Once we have a new, permanent (for as long as we desire it to be)
base in our system, we'll always have multiple heads present::
$ alembic heads
3cac04ae8714 (networking) (head)
27c6a30d7c24 (shoppingcart) (head)
ae1027a6acf (head)When we want to add a new revision file to networking, we specifynetworking@head as the --head. The appropriate version directory
is now selected automatically based on the head we choose::
$ alembic revision -m "add ip number table" --head=networking@head
Generating /path/to/foo/model/networking/109ec7d132bf_add_ip_number_table.py ... doneIt's important that we refer to the head using networking@head; if we
only refer to networking, that refers to only 3cac04ae8714 specifically;
if we specify this and it's not a head, alembic revision will make sure
we didn't mean to specify the head::
$ alembic revision -m "add DNS table" --head=networking
FAILED: Revision 3cac04ae8714 is not a head revision; please
specify --splice to create a new branch from this revisionAs mentioned earlier, as this base is independent, we can view its history
from the base using history -r networking@base:::
$ alembic history -r networking@base:
109ec7d132bf -> 29f859a13ea (networking) (head), add DNS table
3cac04ae8714 -> 109ec7d132bf (networking), add ip number table
-> 3cac04ae8714 (networking), create networking branchAt the moment, this is the same output we'd get at this point if we used-r :networking@head. However, that will change later on as we use
additional directives.
We may now run upgrades or downgrades freely, among individual branches
(let's assume a clean database again)::
$ alembic upgrade networking@head
INFO [alembic.migration] Running upgrade -> 3cac04ae8714, create networking branch
INFO [alembic.migration] Running upgrade 3cac04ae8714 -> 109ec7d132bf, add ip number table
INFO [alembic.migration] Running upgrade 109ec7d132bf -> 29f859a13ea, add DNS tableor against the whole thing using heads::
$ alembic upgrade heads
INFO [alembic.migration] Running upgrade -> 1975ea83b712, create account table
INFO [alembic.migration] Running upgrade 1975ea83b712 -> 27c6a30d7c24, add shopping cart table
INFO [alembic.migration] Running upgrade 27c6a30d7c24 -> d747a8a8879, add a shopping cart column
INFO [alembic.migration] Running upgrade 1975ea83b712 -> ae1027a6acf, add a column
INFO [alembic.migration] Running upgrade ae1027a6acf -> 55af2cb1c267, add another account columnBranch Dependencies
When working with multiple roots, it is expected that these different
revision streams will need to refer to one another. For example, a new
revision in networking which needs to refer to the account
table will want to establish 55af2cb1c267, add another account column,
the last revision that
works with the account table, as a dependency. From a graph perspective,
this means nothing more that the new file will feature both55af2cb1c267, add another account column and 29f859a13ea, add DNS table as "down" revisions,
and looks just as though we had merged these two branches together. However,
we don't want to consider these as "merged"; we want the two revision
streams to remain independent, even though a version in networking
is going to reach over into the other stream. To support this use case,
Alembic provides a directive known as depends_on, which allows
a revision file to refer to another as a "dependency", very similar to
an entry in down_revision from a graph perspective, but different
from a semantic perspective.
To use depends_on, we can specify it as part of our alembic revision
command::
$ alembic revision -m "add ip account table" --head=networking@head --depends-on=55af2cb1c267
Generating /path/to/foo/model/networking/2a95102259be_add_ip_account_table.py ... doneWithin our migration file, we'll see this new directive present::
# revision identifiers, used by Alembic.
revision = '2a95102259be'
down_revision = '29f859a13ea'
branch_labels = None
depends_on='55af2cb1c267'depends_on may be either a real revision number or a branch
name. When specified at the command line, a resolution from a
partial revision number will work as well. It can refer
to any number of dependent revisions as well; for example, if we were
to run the command::
$ alembic revision -m "add ip account table" \\
--head=networking@head \\
--depends-on=55af2cb1c267 --depends-on=d747a --depends-on=fa445
Generating /path/to/foo/model/networking/2a95102259be_add_ip_account_table.py ... doneWe'd see inside the file::
# revision identifiers, used by Alembic.
revision = '2a95102259be'
down_revision = '29f859a13ea'
branch_labels = None
depends_on = ('55af2cb1c267', 'd747a8a8879', 'fa4456a9201')We also can of course add or alter this value within the file manually after
it is generated, rather than using the --depends-on argument.
We can see the effect this directive has when we view the history
of the networking branch in terms of "heads", e.g., all the revisions that
are descendants::
$ alembic history -r :networking@head
29f859a13ea (55af2cb1c267) -> 2a95102259be (networking) (head), add ip account table
109ec7d132bf -> 29f859a13ea (networking), add DNS table
3cac04ae8714 -> 109ec7d132bf (networking), add ip number table
-> 3cac04ae8714 (networking), create networking branch
ae1027a6acf -> 55af2cb1c267 (effective head), add another account column
1975ea83b712 -> ae1027a6acf, Add a column
-> 1975ea83b712 (branchpoint), create account tableWhat we see is that the full history of the networking branch, in terms
of an "upgrade" to the "head", will include that the tree building
up 55af2cb1c267, add another account column
will be pulled in first. Interestingly, we don't see this displayed
when we display history in the other direction, e.g. from networking@base::
$ alembic history -r networking@base:
29f859a13ea (55af2cb1c267) -> 2a95102259be (networking) (head), add ip account table
109ec7d132bf -> 29f859a13ea (networking), add DNS table
3cac04ae8714 -> 109ec7d132bf (networking), add ip number table
-> 3cac04ae8714 (networking), create networking branchThe reason for the discrepancy is that displaying history from the base
shows us what would occur if we ran a downgrade operation, instead of an
upgrade. If we downgraded all the files in networking usingnetworking@base, the dependencies aren't affected, they're left in place.
We also see something odd if we view heads at the moment::
$ alembic heads
2a95102259be (networking) (head)
27c6a30d7c24 (shoppingcart) (head)
55af2cb1c267 (effective head)The head file that we used as a "dependency", 55af2cb1c267, is displayed
as an "effective" head, which we can see also in the history display earlier.
What this means is that at the moment, if we were to upgrade all versions
to the top, the 55af2cb1c267 revision number would not actually be
present in the alembic_version table; this is because it does not have
a branch of its own subsequent to the 2a95102259be revision which depends
on it::
$ alembic upgrade heads
INFO [alembic.migration] Running upgrade 29f859a13ea, 55af2cb1c267 -> 2a95102259be, add ip account table
$ alembic current
2a95102259be (head)
27c6a30d7c24 (head)The entry is still displayed in alembic heads because Alembic knows that
even though this revision isn't a "real" head, it's still something that
we developers consider semantically to be a head, so it's displayed, noting
its special status so that we don't get quite as confused when we don't
see it within alembic current.
If we add a new revision onto 55af2cb1c267, the branch again becomes
a "real" branch which can have its own entry in the database::
$ alembic revision -m "more account changes" --head=55af2cb@head
Generating /path/to/foo/versions/34e094ad6ef1_more_account_changes.py ... done
$ alembic upgrade heads
INFO [alembic.migration] Running upgrade 55af2cb1c267 -> 34e094ad6ef1, more account changes
$ alembic current
2a95102259be (head)
27c6a30d7c24 (head)
34e094ad6ef1 (head)For posterity, the revision tree now looks like::
$ alembic history
29f859a13ea (55af2cb1c267) -> 2a95102259be (networking) (head), add ip account table
109ec7d132bf -> 29f859a13ea (networking), add DNS table
3cac04ae8714 -> 109ec7d132bf (networking), add ip number table
-> 3cac04ae8714 (networking), create networking branch
1975ea83b712 -> 27c6a30d7c24 (shoppingcart) (head), add shopping cart table
55af2cb1c267 -> 34e094ad6ef1 (head), more account changes
ae1027a6acf -> 55af2cb1c267, add another account column
1975ea83b712 -> ae1027a6acf, Add a column
-> 1975ea83b712 (branchpoint), create account table
--- 27c6 --> d747 --> <head>
/ (shoppingcart)
--> 1975 -->
\
--- ae10 --> 55af --> <head>
^
+--------+ (dependency)
|
|
--> 3782 -----> 109e ----> 29f8 ---> 2a95 --> <head>
(networking)If there's any point to be made here, it's if you are too freely branching, merging
and labeling, things can get pretty crazy! Hence the branching system should
be used carefully and thoughtfully for best results.
Build Changelog (`docs/build/changelog.rst`)
========== Changelog
.. changelog::
:version: 1.20.2
:include_notes_from: unreleased
.. changelog::
:version: 1.20.1
:released: October 21, 2026
.. change::
:tags: bug, autogenerate
:tickets: 1085
Fixed issue where a table or column comment set to an empty string would
cause autogenerate to detect a comment change on every run. As most
backends, including PostgreSQL, MySQL, MariaDB and Oracle, don't store an
empty string comment distinctly from no comment, autogenerate now treats
an empty string comment the same as ``None`` when comparing comments.
Additionally, an empty string comment is now rendered in
``op.create_table()`` and ``op.add_column()`` directives, where it was
previously omitted. Pull request courtesy vineoy.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 1360
Fixed rendering of the PostgreSQL
:class:`~sqlalchemy.dialects.postgresql.DOMAIN` type in autogenerated
migration scripts. Previously the type was rendered using ``repr()``,
which omitted the ``check``, ``not_null``, ``default``, ``collation``,
``constraint_name``, ``schema`` and ``create_type`` arguments and did not
apply a dialect prefix to the underlying data type. The full set of
arguments is now rendered and the underlying data type is rendered using
the standard type-rendering logic. Pull request courtesy Sebastian Cao.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 1462
Fixed false-positive "Computed default cannot be modified" warning emitted
during autogenerate when comparing PostgreSQL :class:`.Computed` columns
whose reflected SQL expression includes type cast specifiers such as
``::regconfig`` or ``::text``, which PostgreSQL adds to the expression as
stored. These cast specifiers are now stripped before comparison so that
otherwise equivalent expressions no longer trigger the warning. Pull
request courtesy Sebastian Cao.
.. change::
:tags: bug, autogenerate, postgresql
Fixed issue where autogenerate would detect an unchanged PostgreSQL
expression-based index as modified when the expression included type
casts, such as a cast to a type with a length or precision, a nested
cast, or a cast followed by a ``DESC`` modifier.
.. change::
:tags: bug, postgresql
:tickets: 1504
Fixed bug where :meth:`.Operations.alter_column` against a PostgreSQL
identity column would render an incomplete, invalid ``ALTER TABLE ... ALTER
COLUMN`` statement with no clause at all when ``existing_server_default``
was omitted, even if there were no existing server default. Pull request
courtesy Riti Grover.
.. change::
:tags: bug, environment
:tickets: 1873
Fixed issue where a third party plugin published using the
``alembic.plugins`` entrypoint, configured to refer to a plugin module as
documented at :ref:`alembic.plugins.toplevel`, would cause ``import
alembic`` to fail with a ``TypeError``, as the loaded module was
erroneously iterated rather than being set up directly. An entrypoint
that refers to a collection of modules, the form which inadvertently
worked previously, continues to be accepted but now emits a deprecation
warning.
.. change::
:tags: bug, autogenerate
:tickets: 1876
Fixed issue where dialect-specific keyword arguments on a
:class:`.ForeignKeyConstraint`, :class:`.ForeignKey` or
:class:`.CheckConstraint`, such as ``postgresql_not_valid``, were not
rendered for the constraints that are part of an autogenerated
:meth:`.Operations.create_table` directive, nor for an autogenerated
:meth:`.Operations.create_check_constraint` directive. The
:meth:`.Operations.create_table` case additionally requires SQLAlchemy
2.0.55 / 2.1.5 or greater, which include the fix for SQLAlchemy issue
#13677 where these arguments were lost when the :class:`.Table` was
copied.
.. change::
:tags: bug, autogenerate
:tickets: 1884
Fixed issue where, when using SQLAlchemy 2.1, the CHECK constraint
generated by a type such as :class:`~sqlalchemy.types.Enum` or
:class:`~sqlalchemy.types.Boolean` with ``create_constraint=True`` would
be rendered by autogenerate as explicit ``sa.CheckConstraint()`` entries
within ``op.create_table()``, in addition to the type itself which
already produces that constraint, leading to a duplicate constraint error
when the migration is run.
.. change::
:tags: bug, operations
Fixed issue where a :class:`.CreateTableOp` produced from an existing
:class:`~sqlalchemy.schema.Table` using :meth:`.CreateTableOp.from_table`
would emit the CHECK constraint generated by a type such as
:class:`~sqlalchemy.types.Enum` or :class:`~sqlalchemy.types.Boolean`
with ``create_constraint=True`` twice in the resulting CREATE TABLE
statement when invoked directly... changelog::
:version: 1.20.0
:released: September 11, 2026
.. change::
:tags: bug, batch
:tickets: 1768
Fixed bug in batch mode where adding a column with a type that generates
its own CHECK constraint, such as :class:`~sqlalchemy.types.Boolean` or
:class:`~sqlalchemy.types.Enum` with
:paramref:`~sqlalchemy.types.Boolean.create_constraint` set to ``True``,
would emit the constraint twice when the table was recreated, once under
the name generated by the naming convention in use and once under the
name given to the type. The constraint is now emitted once, using the
same name that would be used outside of batch mode.
.. change::
:tags: bug, batch
:tickets: 1844
Fixed bug in batch mode where a CHECK constraint generated by a type such
as :class:`~sqlalchemy.types.Boolean` or :class:`~sqlalchemy.types.Enum`
would lose the name established for it by the naming convention in use
when the table was recreated, as the constraint was regenerated against
the temporary table used for the recreate operation. The naming
convention is now resolved against the name of the table being replaced.
As part of this change, a type that generates a CHECK constraint but has
no name of its own, used in conjunction with a naming convention that
includes the ``%(constraint_name)s`` token, now emits a warning, as the
convention has no name to interpolate; the constraint continues to be
emitted without a name. As noted at :ref:`batch_check_constraints`,
these datatypes should be given a name in order to participate fully in
batch mode.
.. change::
:tags: bug, batch
:tickets: 1845
Fixed bug in batch mode where the
:paramref:`.Operations.batch_alter_table.naming_convention` parameter
would re-generate the names of constraints that already had a name when
reflected, in the case where the convention included the
``%(constraint_name)s`` token, leading to a name that included the
convention's own prefix twice. The convention is now applied only to
those constraints that are reflected without a name, which is the use
case the parameter is documented for.
.. change::
:tags: usecase, batch
:tickets: 1846
Added a warning for the case where an unnamed CHECK constraint on a
reflected table is omitted from a batch "recreate" operation. An unnamed
CHECK constraint can't be reliably carried over in a batch recreate
as it may refer to columns that are being dropped or changed. This
omission was previously a silent operation. The presence of any
:class:`~sqlalchemy.schema.CheckConstraint` in
:paramref:`.Operations.batch_alter_table.table_args` is taken to indicate
that the case has been accommodated, and no warning is emitted.
.. change::
:tags: change, general
:tickets: 1851
Support for SQLAlchemy 1.4 is dropped as of Alembic 1.20.0; SQLAlchemy
2.0.0 is now the minimum required version. SQLAlchemy 1.4 has no support
for Python 3.13 and above, and the version-conditional code paths it
required throughout Alembic have been removed in favor of the SQLAlchemy
2.0 API.
.. change::
:tags: bug, mysql
:tickets: 1866
Fixed bug where autogenerate with
:paramref:`.EnvironmentContext.configure.compare_server_default` would
report a persistent server default change for ``Float``, ``Numeric`` and
other decimal columns on MySQL. MySQL reports a literal server default
for such a column either in quoted form, e.g. ``'1'``, or, when the
value is fractional, as a parenthesized expression, e.g. ``(2.5)``;
neither form was accommodated for columns other than those of
``Integer`` affinity, so that a default such as ``server_default="1"``
on a :class:`sqlalchemy.Float` column would never compare as equal to
the reflected value.
.. change::
:tags: bug, mysql
:tickets: 1866
Fixed bug where autogenerate with
:paramref:`.EnvironmentContext.configure.compare_server_default` would
report a persistent server default change for an expression server
default such as ``(rand())``. MySQL reports such a default with the
surrounding parenthesis included whereas MariaDB reports it without
them, so that the parenthesis are now disregarded when comparing.
.. change::
:tags: usecase, autogenerate
:tickets: 916
Autogenerate now renders a warning comment above any rendered
:meth:`.Operations.drop_constraint` directive for which the constraint name
is ``None``, as is the case when a constraint that has no name in the model
is dropped, most typically within the ``downgrade()`` function of a
migration that adds an unnamed constraint. A warning is also emitted on
the console when the migration script is generated. The directive
requires a non-None name in order to be able to emit a "DROP CONSTRAINT"
command... changelog::
:version: 1.19.2
:released: September 4, 2026
.. change::
:tags: changed, autogenerate
:tickets: 1859
The autogenerate plugin for CHECK constraint detection by name, added in
1.19.0. for :ticket:`508`, is no longer enabled by default. It has been
renamed from ``alembic.autogenerate.checkconstraint_byname`` to
``alembic.ext.checkconstraint_byname`` and no longer matches on the
``"alembic.autogenerate.*"`` wildcard, which remains the default plugin
specification. The previous name will still function as well if placed in
the plugins list explicitly, both to enable the plugin and within a
``"~"`` exclusion, so that an ``env.py`` written against 1.19.0 or 1.19.1
requires no change.
The plugin is now recommended only for schemas that ensure the naming of
all constraints using a client side naming convention, otherwise there's a
persistent risk of false positives. See
:ref:`autogenerate_check_constraints` for background on things to be aware
of when using this plugin.
As part of this change "type bound" CHECK constraints, which include
constraints generated for the ``Boolean`` and ``Enum`` datatypes when the
``create_constraint`` parameter is set to True, are no longer ignored in
the metadata side, so that normal name-based matching can occur for
these constraints.
.. change::
:tags: usecase, autogenerate, batch
:tickets: 1860
The target of a :class:`~sqlalchemy.schema.ForeignKey` is now located
using the ``ForeignKey.target_tokens`` and
``ForeignKey.target_table_key`` accessors added in SQLAlchemy 2.1, rather
than by splitting the dotted string form of that target on ``"."``. As a
dot inside a schema, table or column name cannot be told apart from the
separator between those names, a foreign key whose target name contained
a dot was previously mis-parsed by autogenerate rendering as well as by
batch migrations. The dotted string continues to be split when running
against SQLAlchemy 2.0, where these accessors are not present. Thanks to
Gyanu Mayank for the initial pull request... changelog::
:version: 1.19.1
:released: August 8, 2026
.. change::
:tags: bug, autogenerate
:tickets: 1842
Fixed bug in the check constraint detection implemented in :ticket:`508`
that failed to take into account column bound check constraints,
leading to wrong autogenerate detections... changelog::
:version: 1.19.0
:released: August 4, 2026
.. change::
:tags: bug, commands
:tickets: 1822
Fixed inconsistency where running ``stamp`` or ``downgrade`` to ``base`` in
offline (``--sql``) mode would emit a ``DROP TABLE alembic_version``
statement, while the same operations in online mode never drop the version
table. Offline mode no longer emits this ``DROP``, matching online
behavior. The version table continues to be created when it does not exist;
only the spurious offline-only drop has been removed. Pull request
courtesy imurodl.
.. change::
:tags: feature, autogenerate
:tickets: 508
Autogenerate now detects the addition and removal of named CHECK
constraints, as part of the default autogenerate behavior. Detection is
name-based only; a constraint whose name is unchanged is presumed
equivalent regardless of its expression text, as reliably normalizing
SQL expressions across backends for comparison purposes is not generally
feasible. This behavior is implemented as a plugin named
``alembic.autogenerate.checkconstraint_byname``, and may be disabled if not
desired by excluding it from the
:paramref:`.EnvironmentContext.configure.autogenerate_plugins` list.
Pull request courtesy Francois van Kempen.
.. seealso::
:ref:`autogenerate_check_constraints`
.. change::
:tags: changed, installation
Environmental updates:
* Trove classifiers now include Python 3.15 which is now part of CI
integration
* Python 3.14 is also added to trove classifiers which had been previously
omitted
* Implemented :pep:`604` style unions in type annotations.. changelog::
:version: 1.18.5
:released: June 25, 2026
.. change::
:tags: bug, autogenerate
:tickets: 1258
Fixed rendering of dialect keyword arguments containing
:class:`~sqlalchemy.schema.Column` objects within sequences, such as
``postgresql_include``. These were previously rendered using ``repr()``,
producing invalid Python in the generated migration scripts. Column
objects within list or tuple values are now correctly rendered as their
string column names. Pull request courtesy Ajay Singh.
.. change::
:tags: usecase, commands
:tickets: 1712
Added ``--splice`` support to the :func:`.merge` command. Previously, the
merge command would suggest using ``--splice`` when attempting to merge
non-head revisions, but the flag was not actually accepted by the command.
The ``splice`` parameter is now available in both the command-line
interface and the :func:`.command.merge` function, matching the existing
support in :func:`.command.revision`. Pull request courtesy Kadir Can
Ozden.
.. change::
:tags: bug, mysql
:tickets: 779, 1745
Implemented type comparison for :class:`.ENUM` datatypes on MySQL, which
checks that the individual enum values are equivalent. If additional
entries are on either side, this generates a diff. Changes of order do not
generate a diff. Pull request courtesy Furkan Kรถykฤฑran.
.. change::
:tags: usecase, environment
:tickets: 1806
Added :paramref:`.ScriptDirectory.get_heads.consider_depends_on`
parameter to :meth:`.ScriptDirectory.get_heads`. When set to ``True``,
head revisions that are also a dependency of another revision via
``depends_on`` are excluded from the result, matching the effective
heads that would be present in the ``alembic_version`` table after
running all upgrades.
.. change::
:tags: bug, operations
:tickets: 1820
Fixed bug where the ``inline_references`` parameter of
:meth:`.Operations.add_column` did not include foreign key referential
actions such as ``ON DELETE``, ``ON UPDATE``, ``DEFERRABLE``,
``INITIALLY``, and ``MATCH`` when rendering the inline ``REFERENCES``
clause... changelog::
:version: 1.18.4
:released: February 10, 2026
.. change::
:tags: bug, operations
:tickets: 1232
Reverted the behavior of :meth:`.Operations.add_column` that would
automatically render the "PRIMARY KEY" keyword inline when a
:class:`.Column` with ``primary_key=True`` is added. The automatic
behavior, added in version 1.18.2, is now opt-in via the new
:paramref:`.Operations.add_column.inline_primary_key` parameter. This
change restores the ability to render a PostgreSQL SERIAL column, which is
required to be ``primary_key=True``, while not impacting the ability to
render a separate primary key constraint. This also provides consistency
with the :paramref:`.Operations.add_column.inline_references` parameter and
gives users explicit control over SQL generation.
To render PRIMARY KEY inline, use the
:paramref:`.Operations.add_column.inline_primary_key` parameter set to
``True``::
op.add_column(
"my_table",
Column("id", Integer, primary_key=True),
inline_primary_key=True
).. changelog::
:version: 1.18.3
:released: January 29, 2026
.. change::
:tags: bug, autogenerate
:tickets: 1787
Fixed regression in version 1.18.0 due to :ticket:`1771` where autogenerate
would raise ``NoReferencedTableError`` when a foreign key constraint
referenced a table that was not part of the initial table load, including
tables filtered out by the
:paramref:`.EnvironmentContext.configure.include_name` callable or tables
in remote schemas that were not included in the initial reflection run.
The change in :ticket:`1771` was a performance optimization that eliminated
additional reflection queries for tables that were only referenced by
foreign keys but not explicitly included in the main reflection run.
However, this optimization inadvertently removed the creation of
:class:`.Table` objects for these referenced tables, causing autogenerate
to fail when processing foreign key constraints that pointed to them.
The fix creates placeholder :class:`.Table` objects for foreign key targets
that are not reflected, allowing the autogenerate comparison to proceed
without error while maintaining the performance improvement from
:ticket:`1771`. When multiple foreign keys reference different columns in
the same filtered table, the placeholder table accumulates all necessary
columns. These placeholder tables may be visible when using the
:paramref:`.EnvironmentContext.configure.include_object` callable to
inspect :class:`.ForeignKeyConstraint` objects; they will have the name,
schema and basic column information for the relevant columns present.
.. change::
:tags: bug, general
:tickets: 1788
Fixed regression caused by :ticket:`1669` which requires SQLAlchemy objects
to support generic type subscripting; for the older SQLAlchemy 1.4 series,
this requires version 1.4.23. Changed the minimum requirements to require
version 1.4.23 rather than 1.4.0... changelog::
:version: 1.18.2
:released: January 28, 2026
.. change::
:tags: usecase, operations
:tickets: 1232
The ``primary_key`` parameter on :class:`.Column` is now honored when
:meth:`.Operations.add_column` is used, and will emit the "PRIMARY KEY"
keyword inline within the ADD COLUMN directive. This is strictly a syntax
enhancement; no attempt is made to reconcile the column's primary key
status with any existing primary key constraint or particular backend
limitations on adding columns to the primary key.
.. note::
As of version 1.18.4, this behavior has been amended to be opt-in
via the new :paramref:`.Operations.add_column.inline_primary_key`
parameter to :meth:`.Operations.add_column`, rather than occurring
automatically when ``primary_key=True`` is set on the
:class:`.Column` object.
.. change::
:tags: bug, typing
:tickets: 1669
Fixed typing issue where the :paramref:`.AlterColumnOp.server_default` and
:paramref:`.AlterColumnOp.existing_server_default` parameters failed to
accommodate common SQLAlchemy SQL constructs such as ``null()`` and
``text()``. Pull request courtesy Sebastian Kreft.
.. note:: This fix is not compatible with older SQLAlchemy versions
prior to 1.4.23.
.. change::
:tags: usecase, operations
:tickets: 1780
Added ``inline_references`` parameter to :meth:`.Operations.add_column`
which allows rendering of ``REFERENCES`` clauses inline within the ``ADD
COLUMN`` directive rather than as a separate ``ADD CONSTRAINT`` directive.
This syntax is supported by PostgreSQL, Oracle, MySQL 5.7+, and MariaDB
10.5+, and can provide performance benefits on large tables by avoiding
full table validation when adding a nullable foreign key column... changelog::
:version: 1.18.1
:released: January 14, 2026
.. change::
:tags: bug, autogenerate
:tickets: 1779
Fixed issue in new plugin system where the configured logger was not
correctly using the ``__name__`` token to identify the logger.
.. change::
:tags: bug, operations
Revised the change regarding SQLAlchemy 2.1 and deprecation warnings
related to ``isolate_from_table=True``. Further developments in release 2.1
have revised how this parameter will be modified... changelog::
:version: 1.18.0
:released: January 9, 2026
.. change::
:tags: bug, postgresql
:tickets: 1507
Fixed issue where PostgreSQL sequence defaults on non-primary key columns
were incorrectly detected as changed on every autogenerate run. Server
default comparison logic is adjusted to filter out the ``::regclass``
expression added by the server which interferes with the comparison.
.. change::
:tags: feature, operations
:tickets: 1737
When alembic is run in "verbose" mode, alembic now logs a message to
indicate from which file is used to load the configuration.
.. change::
:tags: bug, mssql
:tickets: 1755
Implemented DDL for column comment add/update/delete when using the
:paramref:`.Operations.alter_column.comment` parameter with
:meth:`.Operations.alter_column` on Microsoft SQL Server. Previously,
these functions were not implemented for SQL Server and would raise
``UnsupportedCompilationError``.
.. change::
:tags: feature, autogenerate
:tickets: 1771
Autogenerate reflection sweeps now use the "bulk" inspector methods
introduced in SQLAlchemy 2.0, which for selected dialects including
PostgreSQL and Oracle use batched queries to reflect whole collections of
tables using O(1) queries rather than O(N).
.. change::
:tags: usecase, environment
:tickets: 1774
The ``file_template`` configuration option now supports directory paths,
allowing migration files to be organized into subdirectories. When using
directory separators in ``file_template`` (e.g.,
``%(year)d/%(month).2d/%(day).2d_%(rev)s_%(slug)s``), Alembic will
automatically create the necessary directory structure. The
``recursive_version_locations`` setting must be set to ``true`` when using
this feature in order for the revision files to be located for subsequent
commands.
.. change::
:tags: usecase, operations
Avoid deprecation warning in add/drop constraint added in SQLAlchemy 2.1.
Ensure that alembic is compatible with the changes added in
https://github.com/sqlalchemy/sqlalchemy/issues/13006
by explicitly setting ``isolate_from_table=True`` when running with
SQLAlchemy 2.1 or greater.
.. note::
This change was revised in Alembic 1.18.1; futher developments
in SQLAlchemy 2.1 necessitated further adjustments.
.. change::
:tags: feature, autogenerate
Release 1.18.0 introduces a plugin system that allows for automatic
loading of third-party extensions as well as configurable autogenerate
compare functionality on a per-environment basis.
The :class:`.Plugin` class provides a common interface for extensions that
register handlers among Alembic's existing extension points such as
:meth:`.Operations.register_operation` and
:meth:`.Operations.implementation_for`. A new interface for registering
autogenerate comparison handlers,
:meth:`.Plugin.add_autogenerate_comparator`, provides for autogenerate
compare functionality that may be custom-configured on a per-environment
basis using the new
:paramref:`.EnvironmentContext.configure.autogenerate_plugins` parameter.
The change does not impact well known Alembic add-ons such as
``alembic-utils``, which continue to work as before; however, such add-ons
have the option to provide plugin entrypoints going forward.
As part of this change, Alembic's autogenerate compare functionality is
reorganized into a series of internal plugins under the
``alembic.autogenerate`` namespace, which may be individually or
collectively identified for inclusion and/or exclusion within the
:meth:`.EnvironmentContext.configure` call using a new parameter
:paramref:`.EnvironmentContext.configure.autogenerate_plugins`. This
parameter is also where third party comparison plugins may also be
indicated.
See :ref:`alembic.plugins.toplevel` for complete documentation on
the new :class:`.Plugin` class as well as autogenerate-specific usage
instructions... changelog::
:version: 1.17.2
:released: November 14, 2025
.. change::
:tags: bug, mssql
:tickets: 1744
Fixed issue in SQL Server dialect where the DROP that's automatically
emitted for existing default constraints during an ALTER COLUMN needs to
take place before not just the modification of the column's default, but
also before the column's type is changed.
.. change::
:tags: feature, operations
:tickets: 1750
Added :paramref:`.Operations.implementation_for.replace` parameter to
:meth:`.Operations.implementation_for`, allowing replacement of existing
operation implementations. This allows for existing operations such as
:class:`.CreateTableOp` to be extended directly. Pull request courtesy
justanothercatgirl.
.. seealso::
:ref:`operations_extending_builtin`.. changelog::
:version: 1.17.1
:released: October 28, 2025
.. change::
:tags: bug, commands
:tickets: 1540
Disallow ':' character in custom revision identifiers. Previously, using a
colon in a revision ID (e.g., 'REV:1') would create the revision, however
revisions with colons in them are not correctly interpreted by other
commands, as it overlaps with the revision range syntax. Pull request
courtesy Kim Wooseok with original implementation by Hrushikesh Patil.
.. change::
:tags: usecase, commands
:tickets: 1705
Added :paramref:`.command.current.check_heads` parameter to
:func:`.command.current` command, available from the command line via the
``--check-heads`` option to ``alembic current``. This tests if all head
revisions are applied to the database and raises :class:`.DatabaseNotAtHead`
(or from the command line, exits with a non-zero exit code) if this is not
the case. The parameter operates equvialently to the cookbook recipe
:ref:`cookbook_check_heads`. Pull request courtesy Stefan Scherfke... changelog::
:version: 1.17.0
:released: October 11, 2025
.. change::
:tags: change, tests
The top-level test runner has been changed to use ``nox``, adding a
``noxfile.py`` as well as some included modules. The ``tox.ini`` file
remains in place so that ``tox`` runs will continue to function in the near
term, however it will be eventually removed and improvements and
maintenance going forward will be only towards ``noxfile.py``.
.. change::
:tags: change, general
The minimum Python version is now 3.10, as Python 3.9 is EOL... changelog::
:version: 1.16.5
:released: August 27, 2025
.. change::
:tags: bug, mysql
:tickets: 1492
Fixed Python-side autogenerate rendering of index expressions in MySQL
dialect by aligning it with SQLAlchemy's MySQL index expression rules. Pull
request courtesy david-fed.
.. change::
:tags: bug, config
:tickets: 1709
Fixed issue where new pyproject.toml config would fail to parse the integer
value used for the ``truncate_slug_length`` parameter. Pull request
courtesy Luรญs Henrique Allebrandt Schunemann... changelog::
:version: 1.16.4
:released: July 10, 2025
.. change::
:tags: bug, config
:tickets: 1694
Fixed issue in new ``pyproject.toml`` support where boolean values, such as
those used for the ``recursive_version_locations`` and ``sourceless``
configuration parameters, would not be accepted... changelog::
:version: 1.16.3
:released: July 8, 2025
.. change::
:tags: bug, autogenerate
:tickets: 1633
Fixed the rendering of ``server_default=FetchedValue()`` to ensure it is
preceded by the ``sa.`` prefix in the migration script. Pull request
courtesy david-fed.
.. change::
:tags: usecase, commands
:tickets: 1683
Added new ``pyproject_async`` template, combining the new ``pyproject``
template with the ``async`` template. Pull request courtesy Alc-Alc.
.. change::
:tags: usecase, autogenerate
:tickets: 1686
Add "module" post-write hook. This hook type is almost identical to the
console_scripts hook, except it's running ``python -m black`` instead of
using black's ``console_script``. It is mainly useful for tools without
console scripts (e.g. ruff), but has semantics closer to the
console_scripts hook in that it finds the ruff module available to the
running interpreter instead of finding an executable by path. Pull request
courtesy Frazer McLean.
.. change::
:tags: bug, autogenerate
:tickets: 1692
Fixed autogenerate rendering bug which failed to render foreign key
constraints local to a :class:`.CreateTableOp` object if it did not refer
to a ``MetaData`` collection via a private constructor argument that would
not ordinarily be passed in user-defined rewriter recipes, including ones
in the Alembic cookbook section of the docs... changelog::
:version: 1.16.2
:released: June 16, 2025
.. change::
:tags: bug, autogenerate
:tickets: 1671
Fixed issue where dialect-specific keyword arguments in ``dialect_kwargs``
were not rendered when rendering the :meth:`.Operations.create_foreign_key`
operation. This prevented dialect-specific keywords from being rendered
using custom :class:`.Rewriter` recipes that modify
:class:`.ops.CreateForeignKeyOp`, similar to other issues such as
:ticket:`1635`. Pull request courtesy Justin Malin.
.. change::
:tags: bug, command
:tickets: 1679
Fixed rendering of ``pyproject.toml`` to include two newlines when
appending content to an existing file. Pull request courtesy Jonathan
Vanasco... changelog::
:version: 1.16.1
:released: May 21, 2025
.. change::
:tags: bug, command
:tickets: 1660
Fixed regression caused by the ``pathlib`` refactoring that removed the use
of :meth:`.Config.get_template_directory` as the canonical source of
templates; the method is still present however it no longer would be
consulted for a custom config subclass, as was the case with flask-migrate.
.. change::
:tags: bug, command
:tickets: 1659
Fixed regression caused by the ``pathlib`` refactoring where the "missing
template" error message failed to render the name of the template that
could not be found... changelog::
:version: 1.16.0
:released: May 21, 2025
.. change::
:tags: feature, environment
:tickets: 1082
Added optional :pep:`621` support to Alembic, allowing all source code
related configuration (e.g. local file paths, post write hook
configurations, etc) to be configured in the project's ``pyproject.toml``
file. A new init template ``pyproject`` is added which illustrates a
basic :pep:`621` setup.
Besides being better integrated with a Python project's existing source
code configuration, the TOML format allows for more flexible structures,
allowing configuration items like ``version_locations`` and
``prepend_sys_path`` to be configured as lists of path strings without the
need for path separator characters used by ``ConfigParser`` format. The
feature continues to support the ``%(here)s`` token which can substitute
the absolute parent directory of the ``pyproject.toml`` file when
consumed.
The :pep:`621` feature supports configuration values that are relevant to
source code organization and generation only; it does not accommodate
configuration of database connectivity or logging, which remain under the
category of "deployment" configuration and continue to be part of
``alembic.ini``, or whatever configurational method is established by the
``env.py`` file. Using the combination of ``pyproject.toml`` for source
code configuration along with a custom database/logging configuration
method established in ``env.py`` will allow the ``alembic.ini`` file to be
omitted altogether.
.. seealso::
:ref:`using_pep_621`
.. change::
:tags: usecase, environment
:tickets: 1330
Added new option to the ConfigParser (e.g. ``alembic.ini``) configuration
``path_separator``, which supersedes the existing ``version_path_separator``
option. ``path_separator`` specifies the path separator character that
will be recognized for both the ``version_locations`` option as well
as the ``prepend_sys_path`` option, defaulting to ``os`` which indicates
that the value of ``os.pathsep`` should be used.
The new attribute applies necessary os-dependent path splitting to the
``prepend_sys_path`` option so that windows paths which contain drive
letters with colons are not inadvertently split, whereas previously
os-dependent path splitting were only available for the ``version_locations`` option.
Existing installations that don't indicate ``path_separator``
will continue to use the older behavior, where ``version_path_separator``
may be configured for ``version_locations``, and ``prepend_sys_path``
continues to be split on spaces/commas/colons. A deprecation warning
is emitted for these fallback scenarios.
When using the new ``pyproject.toml`` configuration detailed at
:ref:`using_pep_621`, the whole issue of "path separators" is sidestepped
and parameters like ``path_separator`` are unnecessary, as the TOML based
configuration configures version locations and sys path elements as
lists.
Pull request courtesy Mike Werezak.
.. change::
:tags: feature, commands
:tickets: 1610
Added new :meth:`.CommandLine.register_command` method to
:class:`.CommandLine`, intended to facilitate adding custom commands to
Alembic's command line tool with minimal code required; previously this
logic was embedded internally and was not publicly accessible. A new
recipe demonstrating this use is added. Pull request courtesy Mikhail
Bulash.
.. seealso::
:ref:`custom_commandline`
.. change::
:tags: usecase, operations
:tickets: 1626
Added :paramref:`.Operations.add_column.if_not_exists` and
:paramref:`.Operations.drop_column.if_exists` to render ``IF [NOT] EXISTS``
for ``ADD COLUMN`` and ``DROP COLUMN`` operations, a feature available on
some database backends such as PostgreSQL, MariaDB, as well as third party
backends. The parameters also support autogenerate rendering allowing them
to be added to autogenerate scripts via a custom :class:`.Rewriter`. Pull
request courtesy of Louis-Amaury Chaib (@lachaib).
.. change::
:tags: bug, general
:tickets: 1637
The ``pyproject.toml`` file used by the Alembic project itself for its
Python package configuration has been amended to use the updated :pep:`639`
configuration for license, which eliminates loud deprecation warnings when
building the package. Note this necessarily bumps setuptools build
requirement to 77.0.3.
.. change::
:tags: bug, environment
:tickets: 1643
Fixed issue where use of deprecated ``utcnow()`` function would generate
warnings. Has been replaced with ``now(UTC)``. Pull request courtesy
Jens Trรถger.
.. change::
:tags: usecase, operations
:tickets: 1650
Added :paramref:`.Operations.drop_constraint.if_exists` parameter to
:meth:`.Operations.drop_constraint` which will render ``DROP CONSTRAINT IF
EXISTS``. The parameter also supports autogenerate rendering allowing it to
be added to autogenerate scripts via a custom :class:`.Rewriter`. Pull
request courtesy Aaron Griffin.
.. change::
:tags: bug, autogenerate
:tickets: 1656
The :meth:`.Operations.execute` operation when rendered in autogenerate
(which would necessarily be only when using a custom writer that embeds
:class:`.ExecuteSQLOp`) now correctly takes into account the value
configured in :paramref:`configure.alembic_module_prefix` when rendering
the operation with its prefixing namespace; previously this was hardcoded
to ``op.``. Pull request courtesy Avery Fischer.
.. change::
:tags: bug, autogenerate
:tickets: 264
The autogenerate process will now apply the :meth:`.Operations.f` modifier
to the names of all constraints and indexes that are reflected from the
target database when generating migrations, which has the effect that these
names will not have any subsequent naming conventions applied to them when
the migration operations proceed. As reflected objects already include the
exact name that's present in the database, these names should not be
modified. The fix repairs the issue when using custom naming conventions
which feature the ``%(constraint_name)s`` token would cause names to be
double-processed, leading to errors in migration runs.
.. change::
:tags: refactored, environment
The command, config and script modules now rely on ``pathlib.Path`` for
internal path manipulations, instead of ``os.path()`` operations. This
has some impact on both public and private (i.e. underscored) API functions:
* Public API functions that accept parameters indicating file and directory
paths as strings will continue to do so, but now will also accept
``os.PathLike`` objects as well.
* Public API functions and accessors that return directory paths as strings
such as :attr:`.ScriptDirectory.dir`, :attr:`.Config.config_file_name`
will continue to do so.
* Private API functions and accessors, i.e. all those that are prefixed
with an underscore, that previously returned directory paths as
strings may now return a Path object instead... changelog::
:version: 1.15.2
:released: March 28, 2025
.. change::
:tags: bug, autogenerate
:tickets: 1635
Fixed issue where the "modified_name" of :class:`.AlterColumnOp` would not
be considered when rendering op directives for autogenerate. While
autogenerate cannot detect changes in column name, this would nonetheless
impact approaches that made use of this attribute in rewriter recipes. Pull
request courtesy lenvk... changelog::
:version: 1.15.1
:released: March 4, 2025
.. change::
:tags: bug, installation
:tickets: 1616
Fixed an issue in the new :pep:`621` ``pyproject.toml`` layout that
prevented Alembic's template files from being included in the ``.whl`` file
in the distribution... changelog::
:version: 1.15.0
:released: March 4, 2025 (yanked due to issue #1616)
.. change::
:tags: bug, environment
:tickets: 1567
Added a basic docstring to the migration template files so that the
upgrade/downgrade methods pass the D103 linter check which requires a
docstring for public functions. Pull request courtesy Peter Cock.
.. change::
:tags: usecase, autogenerate
:tickets: 1603
Index autogenerate will now render labels for expressions
that use them. This is useful when applying operator classes
in PostgreSQL that can be keyed on the label name.
.. change::
:tags: bug, autogenerate
:tickets: 1613
Fixed autogenerate rendering bug where the ``deferrable`` element of
``UniqueConstraint``, a bool, were being stringified rather than repr'ed
when generating Python code.
.. change::
:tags: changed, general
Support for Python 3.8 is dropped as of Alembic 1.15.0; this version is
now EOL so Python 3.9 or higher is required for Alembic 1.15.
.. change::
:tags: changed, general
Support for SQLAlchemy 1.3, which was EOL as of 2021, is now dropped from
Alembic as of version 1.15.0. SQLAlchemy version 1.4 or greater is
required for use with Alembic 1.15.0.
.. change::
:tags: usecase, autogenerate
:tickets: 1597
Add revision context to AutogenerateDiffsDetected so that command can be
wrapped and diffs may be output in a different format. Pull request
courtesy Louis-Amaury Chaib (@lachaib).
.. change::
:tags: changed, general
Installation has been converted to use :pep:`621`, e.g. ``pyproject.toml``... changelog::
:version: 1.14.1
:released: January 19, 2025
.. change::
:tags: bug, environment
:tickets: 1556
Added `tzdata` to `tz` extras, which is required on some platforms such as
Windows. Pull request courtesy Danipulok.
.. change::
:tags: usecase, sqlite
:tickets: 1576
Modified SQLite's dialect to render "ALTER TABLE <t> RENAME COLUMN" when
:meth:`.Operations.alter_column` is used with a straight rename, supporting
SQLite's recently added column rename feature.
.. change::
:tags: bug, autogenerate
:tickets: 1585
Fixed bug where autogen render of a "variant" type would fail to catch the
variants if the leading type were a dialect-specific type, rather than a
generic type... changelog::
:version: 1.14.0
:released: November 4, 2024
.. change::
:tags: usecase, runtime
:tickets: 1560
Added a new hook to the :class:`.DefaultImpl`
:meth:`.DefaultImpl.version_table_impl`. This allows third party dialects
to define the exact structure of the alembic_version table, to include use
cases where the table requires special directives and/or additional columns
so that it may function correctly on a particular backend. This is not
intended as a user-expansion hook, only a dialect implementation hook to
produce a working alembic_version table. Pull request courtesy Maciek
Bryลski... changelog::
:version: 1.13.3
:released: September 23, 2024
.. change::
:tags: usecase, autogenerate
Render ``if_exists`` and ``if_not_exists`` parameters in
:class:`.CreateTableOp`, :class:`.CreateIndexOp`, :class:`.DropTableOp` and
:class:`.DropIndexOp` in an autogenerate context. While Alembic does not
set these parameters during an autogenerate run, they can be enabled using
a custom :class:`.Rewriter` in the ``env.py`` file, where they will now be
part of the rendered Python code in revision files. Pull request courtesy
of Louis-Amaury Chaib (@lachaib).
.. change::
:tags: usecase, environment
:tickets: 1509
Enhance ``version_locations`` parsing to handle paths containing newlines.
.. change::
:tags: usecase, operations
:tickets: 1520
Added support for :paramref:`.Operations.create_table.if_not_exists` and
:paramref:`.Operations.drop_table.if_exists`, adding similar functionality
to render IF [NOT] EXISTS for table operations in a similar way as with
indexes. Pull request courtesy Aaron Griffin.
.. change::
:tags: change, general
The pin for ``setuptools<69.3`` in ``pyproject.toml`` has been removed.
This pin was to prevent a sudden change to :pep:`625` in setuptools from
taking place which changes the file name of SQLAlchemy's source
distribution on pypi to be an all lower case name, and the change was
extended to all SQLAlchemy projects to prevent any further surprises.
However, the presence of this pin is now holding back environments that
otherwise want to use a newer setuptools, so we've decided to move forward
with this change, with the assumption that build environments will have
largely accommodated the setuptools change by now... changelog::
:version: 1.13.2
:released: June 26, 2024
.. change::
:tags: bug, commands
:tickets: 1384
Fixed bug in alembic command stdout where long messages were not properly
wrapping at the terminal width. Pull request courtesy Saif Hakim.
.. change::
:tags: usecase, autogenerate
:tickets: 1391
Improve computed column compare function to support multi-line expressions.
Pull request courtesy of Georg Wicke-Arndt.
.. change::
:tags: bug, execution
:tickets: 1394
Fixed internal issue where Alembic would call ``connection.execute()``
sending an empty tuple to indicate "no params". In SQLAlchemy 2.1 this
case will be deprecated as "empty sequence" is ambiguous as to its intent.
.. change::
:tags: bug, tests
:tickets: 1435
Fixes to support pytest 8.1 for the test suite.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 1479
Fixed the detection of serial column in autogenerate with tables
not under default schema on PostgreSQL.. changelog::
:version: 1.13.1
:released: December 20, 2023
.. change::
:tags: bug, autogenerate
:tickets: 1337
Fixed :class:`.Rewriter` so that more than two instances could be chained
together correctly, also allowing multiple ``process_revision_directives``
callables to be chained. Pull request courtesy zrotceh.
.. change::
:tags: bug, environment
:tickets: 1369
Fixed issue where the method :meth:`.EnvironmentContext.get_x_argument`
using the :paramref:`.EnvironmentContext.get_x_argument.as_dictionary`
parameter would fail if an argument key were passed on the command line as
a name alone, that is, without an equal sign ``=`` or a value. Behavior is
repaired where this condition is detected and will return a blank string
for the given key, consistent with the behavior where the ``=`` sign is
present and no value. Pull request courtesy Iuri de Silvio.
.. change::
:tags: bug, autogenerate
:tickets: 1370
Fixed issue where the "unique" flag of an ``Index`` would not be maintained
when generating downgrade migrations. Pull request courtesy Iuri de
Silvio.
.. change::
:tags: bug, versioning
:tickets: 1373
Fixed bug in versioning model where a downgrade across a revision with two
down revisions with one down revision depending on the other, would produce
an erroneous state in the alembic_version table, making upgrades impossible
without manually repairing the table. Thanks much to Saif Hakim for
the great work on this.
.. change::
:tags: bug, typing
:tickets: 1377
Updated pep-484 typing to pass mypy "strict" mode, however including
per-module qualifications for specific typing elements not yet complete.
This allows us to catch specific typing issues that have been ongoing
such as import symbols not properly exported... changelog::
:version: 1.13.0
:released: December 1, 2023
.. change::
:tags: bug, commands
:tickets: 1234
Fixed issue where the ``alembic check`` command did not function correctly
with upgrade structures that have multiple, top-level elements, as are
generated from the "multi-env" environment template. Pull request courtesy
Neil Williams.
.. change::
:tags: usecase, operations
:tickets: 1323
Updated logic introduced in :ticket:`151` to allow ``if_exists`` and
``if_not_exists`` on index operations also on SQLAlchemy
1.4 series. Previously this feature was mistakenly requiring
the 2.0 series.
.. change::
:tags: usecase
:tickets: 1339
Replaced ``python-dateutil`` with the standard library module
`zoneinfo <https://docs.python.org/3.11/library/zoneinfo.html#module-zoneinfo>`_.
This module was added in Python 3.9, so previous version will been
to install the backport of it, available by installing the ``backports.zoneinfo``
library. The ``alembic[tz]`` option has been updated accordingly.
.. change::
:tags: installation, changed
:tickets: 1359
Alembic 1.13 now supports Python 3.8 and above.
.. change::
:tags: bug, autogenerate
:tickets: 1361
Fixed autogenerate issue where ``create_table_comment()`` and
``drop_table_comment()`` rendering in a batch table modify would include
the "table" and "schema" arguments, which are not accepted in batch as
these are already part of the top level block.
.. change::
:tags: bug, postgresql
:tickets: 1321, 1327, 1356
Additional fixes to PostgreSQL expression index compare feature.
The compare now correctly accommodates casts and differences in
spacing.
Added detection logic for operation clauses inside the expression,
skipping the compare of these expressions.
To accommodate these changes the logic for the comparison of the
indexes and unique constraints was moved to the dialect
implementation, allowing greater flexibility... changelog::
:version: 1.12.1
:released: October 26, 2023
.. change::
:tags: bug, autogenerate, regression
:tickets: 1329
Fixed regression caused by :ticket:`879` released in 1.7.0 where the
".info" dictionary of ``Table`` would not render in autogenerate create
table statements. This can be useful for custom create table DDL rendering
schemes so it is restored.
.. change::
:tags: bug, typing
:tickets: 1325
Improved typing in the
:paramref:`.EnvironmentContext.configure.process_revision_directives`
callable to better indicate that the passed-in type is
:class:`.MigrationScript`, not the :class:`.MigrationOperation` base class,
and added typing to the example at :ref:`cookbook_no_empty_migrations` to
illustrate.
.. change::
:tags: bug, operations
:tickets: 1335
Repaired :class:`.ExecuteSQLOp` so that it can participate in "diff"
operations; while this object is typically not present in a reflected
operation stream, custom hooks may be adding this construct where it needs
to have the correct ``to_diff_tuple()`` method. Pull request courtesy
Sebastian Bayer.
.. change::
:tags: typing, bug
:tickets: 1058, 1277
Improved the ``op.execute()`` method to correctly accept the
``Executable`` type that is the same which is used in SQLAlchemy
``Connection.execute()``. Pull request courtesy Mihail Milushev.
.. change::
:tags: typing, bug
:tickets: 930
Improve typing of the revision parameter in various command functions.
.. change::
:tags: typing, bug
:tickets: 1266
Properly type the :paramref:`.Operations.create_check_constraint.condition`
parameter of :meth:`.Operations.create_check_constraint` to accept boolean
expressions.
.. change::
:tags: bug, postgresql
:tickets: 1322
Fixed autogen render issue where expressions inside of indexes for PG need
to be double-parenthesized, meaning a single parens must be present within
the generated ``text()`` construct.
.. change::
:tags: usecase
:tickets: 1304
Alembic now accommodates for Sequence and Identity that support dialect kwargs.
This is a change that will be added to SQLAlchemy v2.1... changelog::
:version: 1.12.0
:released: August 31, 2023
.. change::
:tags: bug, operations
:tickets: 1300
Added support for ``op.drop_constraint()`` to support PostgreSQL
``ExcludeConstraint`` objects, as well as other constraint-like objects
that may be present in third party dialects, by resolving the ``type_``
parameter to be ``None`` for this case. Autogenerate has also been
enhanced to exclude the ``type_`` parameter from rendering within this
command when ``type_`` is ``None``. Pull request courtesy David Hills.
.. change::
:tags: bug, commands
:tickets: 1299
Fixed issue where the ``revision_environment`` directive in ``alembic.ini``
was ignored by the ``alembic merge`` command, leading to issues when other
configurational elements depend upon ``env.py`` being invoked within the
command.
.. change::
:tags: bug, autogenerate
:tickets: 1302
Fixed issue where the ``ForeignKeyConstraint.match`` parameter would not be
rendered in autogenerated migrations. Pull request courtesy Asib
Kamalsada.
.. change::
:tags: usecase, autogenerate
:tickets: 1248
Change the default value of
:paramref:`.EnvironmentContext.configure.compare_type` to ``True``.
As Alembic's autogenerate for types was dramatically improved in
version 1.4 released in 2020, the type comparison feature is now much
more reliable so is now enabled by default.
.. change::
:tags: feature, autogenerate
:tickets: 1275
Added new feature to the "code formatter" function which allows standalone
executable tools to be run against code, without going through the Python
interpreter. Known as the ``exec`` runner, it complements the existing
``console_scripts`` runner by allowing non-Python tools such as ``ruff`` to
be used. Pull request courtesy Mihail Milushev.
.. seealso::
:ref:`post_write_hooks_config`.. changelog::
:version: 1.11.3
:released: August 16, 2023
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 1270
Improved autogenerate compare of expression based indexes on PostgreSQL
to produce fewer wrong detections.
.. change::
:tags: bug, autogenerate
:tickets: 1291
Fixed issue with ``NULLS NOT DISTINCT`` detection in postgresql that
would keep detecting changes in the index or unique constraint.
.. change::
:tags: bug, commands
:tickets: 1273
Added ``encoding="locale"`` setting to the use of Python's
``ConfigParser.read()``, so that a warning is not generated when using the
recently added Python feature ``PYTHONWARNDEFAULTENCODING`` specified in
:pep:`597`. The encoding is passed as the ``"locale"`` string under Python
3.10 and greater, which indicates that the system-level locale should be
used, as was the case already here. Pull request courtesy Kevin Kirsche... changelog::
:version: 1.11.2
:released: August 4, 2023
.. change::
:tags: usecase, typing
:tickets: 1253
Added typing to the default script mako templates.
.. change::
:tags: usecase, autogenerate
:tickets: 1248
Added support in autogenerate for ``NULLS NOT DISTINCT`` in
the PostgreSQL dialect.
.. change::
:tags: bug
:tickets: 1261
Fixed format string logged when running a post write hook
Pull request courtesy of Mathieu Dรฉfosse.
.. change::
:tags: feature, operations
:tickets: 151
Added parameters if_exists and if_not_exists for index operations.
Pull request courtesy of Max Adrian... changelog::
:version: 1.11.1
:released: May 17, 2023
.. change::
:tags: bug, autogenerate, regression
:tickets: 1243, 1245
As Alembic 1.11.0 is considered a major release (Alembic does not use
semver, nor does its parent project SQLAlchemy; this has been
:ref:`clarified <versioning_scheme>` in the documentation), change
:ticket:`1130` modified calling signatures for most operations to consider
all optional keyword parameters to be keyword-only arguments, to match what
was always documented and generated by autogenerate. However, two of these
changes were identified as possibly problematic without a more formal
deprecation warning being emitted which were the ``table_name`` parameter
to :meth:`.Operations.drop_index`, which was generated positionally by
autogenerate prior to version 0.6.3 released in 2014, and ``type_`` in
:meth:`.Operations.drop_constraint` and
:meth:`.BatchOperations.drop_constraint`, which was documented positionally
in one example in the batch documentation.
These two signatures have been
restored to allow those particular parameters to be passed positionally. A
future change will include formal deprecation paths (with warnings) for
these arguments where they will again become keyword-only in a future
"Significant Minor" release.
.. change::
:tags: bug, typing
:tickets: 1246
Fixed typing use of :class:`~sqlalchemy.schema.Column` and other
generic SQLAlchemy classes.
.. change::
:tags: bug, typing, regression
:tickets: 1244
Restored the output type of :meth:`.Config.get_section` to include
``Dict[str, str]`` as a potential return type, which had been changed to
immutable ``Mapping[str, str]``. When a section is returned and the default
is not used, a mutable dictionary is returned... changelog::
:version: 1.11.0
:released: May 15, 2023
.. change::
:tags: bug, batch
:tickets: 1237
Added placeholder classes for :class:`~.sqla.Computed` and
:class:`~.sqla.Identity` when older 1.x SQLAlchemy versions are in use,
namely prior to SQLAlchemy 1.3.11 when the :class:`~.sqla.Computed`
construct was introduced. Previously these were set to None, however this
could cause issues with certain codepaths that were using ``isinstance()``
such as one within "batch mode".
.. change::
:tags: bug, batch
:tickets: 1221
Correctly pass previously ignored arguments ``insert_before`` and
``insert_after`` in ``batch_alter_column``
.. change::
:tags: change, py3k
:tickets: 1130
Argument signatures of Alembic operations now enforce keyword-only
arguments as passed as keyword and not positionally, such as
:paramref:`.Operations.create_table.schema`,
:paramref:`.Operations.add_column.type_`, etc.
.. change::
:tags: bug, postgresql
:tickets: 1230
Fix autogenerate issue with PostgreSQL :class:`.ExcludeConstraint`
that included sqlalchemy functions. The function text was previously
rendered as a plain string without surrounding with ``text()``.
.. change::
:tags: bug, mysql, regression
:tickets: 1240
Fixed regression caused by :ticket:`1166` released in version 1.10.0 which
caused MySQL unique constraints with multiple columns to not compare
correctly within autogenerate, due to different sorting rules on unique
constraints vs. indexes, which in MySQL are shared constructs.
.. change::
:tags: misc
:tickets: 1220
Update code snippets within docstrings to use ``black`` code formatting.
Pull request courtesy of James Addison.
.. change::
:tags: bug, typing
:tickets: 1093
Updated stub generator script to also add stubs method definitions for the
:class:`.Operations` class and the :class:`.BatchOperations` class obtained
from :meth:`.Operations.batch_alter_table`. As part of this change, the
class hierarchy of :class:`.Operations` and :class:`.BatchOperations` has
been rearranged on top of a common base class :class:`.AbstractOperations`
in order to type correctly, as :class:`.BatchOperations` uses different
method signatures for operations than :class:`.Operations`.
.. change::
:tags: bug, typing
Repaired the return signatures for :class:`.Operations` that mostly
return ``None``, and were erroneously referring to ``Optional[Table]``
in many cases.
.. change::
:tags: usecase, commands
:tickets: 1109
Added quiet option to the command line, using the ``-q/--quiet``
option. This flag will prevent alembic from logging anything
to stdout.
.. change::
:tags: bug, autogenerate
:tickets: 1178
Modified the autogenerate implementation for comparing "server default"
values from user-defined metadata to not apply any quoting to the value
before comparing it to the server-reported default, except for within
dialect-specific routines as needed. This change will affect the format of
the server default as passed to the
:paramref:`.EnvironmentContext.configure.compare_server_default` hook, as
well as for third party dialects that implement a custom
``compare_server_default`` hook in their alembic impl, to be passed "as is"
and not including additional quoting. Custom implementations which rely
on this quoting should adjust their approach based on observed formatting.
.. change::
:tags: bug, api, autogenerate
:tickets: 1235
Fixed issue where :func:`.autogenerate.render_python_code` function did not
provide a default value for the ``user_module_prefix`` variable, leading to
``NoneType`` errors when autogenerate structures included user-defined
types. Added new parameter
:paramref:`.autogenerate.render_python_code.user_module_prefix` to allow
this to be set as well as to default to ``None``. Pull request courtesy
tangkikodo.
.. change::
:tags: usecase, asyncio
:tickets: 1231
Added :meth:`.AbstractOperations.run_async` to the operation module to
allow running async functions in the ``upgrade`` or ``downgrade`` migration
function when running alembic using an async dialect. This function will
receive as first argument an
:class:`~sqlalchemy.ext.asyncio.AsyncConnection` sharing the transaction
used in the migration context... changelog::
:version: 1.10.4
:released: April 24, 2023
.. change::
:tags: postgresql, autogenerate, feature
:tickets: 1213
Added support for autogenerate comparison of indexes on PostgreSQL which
include SQL sort option, such as ``ASC`` or ``NULLS FIRST``.
The sort options are correctly detected only when defined using the
sqlalchemy modifier functions, such as ``asc()`` or ``nulls_first()``,
or the equivalent methods.
Passing sort options inside the ``postgresql_ops`` dict is not supported.
.. change::
:tags: bug, operations
:tickets: 1215
Fixed issue where using a directive such as ``op.create_foreign_key()`` to
create a self-referential constraint on a single table where the same
column were present on both sides (e.g. within a composite foreign key)
would produce an error under SQLAlchemy 2.0 and a warning under SQLAlchemy
1.4 indicating that a duplicate column were being added to a table... changelog::
:version: 1.10.3
:released: April 5, 2023
.. change::
:tags: bug, typing
:tickets: 1191, 1201
Fixed various typing issues observed with pyright, including issues
involving the combination of :class:`.Function` and
:meth:`.MigrationContext.begin_transaction`.
.. change::
:tags: bug, autogenerate
:tickets: 1212
Fixed error raised by alembic when running autogenerate after removing
a function based index... changelog::
:version: 1.10.2
:released: March 8, 2023
.. change::
:tags: bug, ops
:tickets: 1196
Fixed regression where Alembic would not run with older SQLAlchemy 1.3
versions prior to 1.3.24 due to a missing symbol. Workarounds have been
applied for older 1.3 versions... changelog::
:version: 1.10.1
:released: March 6, 2023
.. change::
:tags: bug, postgresql
:tickets: 1184
Fixed issue regarding PostgreSQL :class:`.ExcludeConstraint`, where
constraint elements which made use of :func:`.literal_column` could not be
rendered for autogenerate. Additionally, using SQLAlchemy 2.0.5 or greater,
:func:`.text()` constructs are also supported within PostgreSQL
:class:`.ExcludeConstraint` objects for autogenerate render. Pull request
courtesy Jan Katins.
.. change::
:tags: bug, batch, regression
:tickets: 1195
Fixed regression for 1.10.0 where :class:`.Constraint` objects were
suddenly required to have non-None name fields when using batch mode, which
was not previously a requirement... changelog::
:version: 1.10.0
:released: March 5, 2023
.. change::
:tags: bug, autogenerate
:tickets: 1166
Fixed issue in index detection where autogenerate change detection would
consider indexes with the same columns but with different order as equal,
while in general they are not equivalent in how a database will use them.
.. change::
:tags: feature, revisioning
:tickets: 760
Recursive traversal of revision files in a particular revision directory is
now supported, by indicating ``recursive_version_locations = true`` in
alembic.ini. Pull request courtesy ostr00000.
.. change::
:tags: bug, autogenerate, sqlite
:tickets: 1165
Fixed issue where indexes on SQLite which include SQL expressions would not
compare correctly, generating false positives under autogenerate. These
indexes are now skipped, generating a warning, in the same way that
expression-based indexes on PostgreSQL are skipped and generate warnings
when SQLAlchemy 1.x installations are in use. Note that reflection of
SQLite expression-based indexes continues to not yet be supported under
SQLAlchemy 2.0, even though PostgreSQL expression-based indexes have now
been implemented.
.. change::
:tags: bug, mssql
:tickets: 1187
Properly escape constraint name on SQL Server when dropping
a column while specifying ``mssql_drop_default=True`` or
``mssql_drop_check=True`` or ``mssql_drop_foreign_key=True``.
.. change::
:tags: usecase, autogenerate, postgresql
Added support for autogenerate comparison of indexes on PostgreSQL which
include SQL expressions, when using SQLAlchemy 2.0; the previous warning
that such indexes were skipped are removed when the new functionality
is in use. When using SQLAlchemy versions prior to the 2.0 series,
the indexes continue to be skipped with a warning... changelog::
:version: 1.9.4
:released: February 16, 2023
.. change::
:tags: bug, mssql
:tickets: 1177
Ongoing fixes for SQL Server server default comparisons under autogenerate,
adjusting for SQL Server's collapsing of whitespace between SQL function
arguments when reporting on a function-based server default, as well as its
arbitrary addition of parenthesis within arguments; the approach has now
been made more aggressive by stripping the two default strings to compare
of all whitespace, parenthesis, and quoting characters.
.. change::
:tags: bug, postgresql
Fixed PostgreSQL server default comparison to handle SQL expressions
sent as ``text()`` constructs, such as ``text("substring('name', 1, 3)")``,
which previously would raise errors when attempting to run a server-based
comparison.
.. change::
:tags: bug, autogenerate
:tickets: 1180
Removed a mis-use of the
:paramref:`.EnvironmentContext.configure.render_item` callable where the
"server_default" renderer would be erroneously used within the server
default comparison process, which is working against SQL expressions, not
Python code.
.. change::
:tags: bug, commands
Fixed regression introduced in 1.7.0 where the "config" object passed to
the template context when running the :func:`.merge` command
programmatically failed to be correctly populated. Pull request courtesy
Brendan Gann... changelog::
:version: 1.9.3
:released: February 7, 2023
.. change::
:tags: bug, autogenerate
:tickets: 1167
Fixed issue where rendering of user-defined types that then went onto use
the ``.with_variant()`` method would fail to render, if using SQLAlchemy
2.0's version of variants... changelog::
:version: 1.9.2
:released: January 14, 2023
.. change::
:tags: bug, typing
:tickets: 1146, 1147
Fixed typing definitions for :meth:`.EnvironmentContext.get_x_argument`.
Typing stubs are now generated for overloaded proxied methods such as
:meth:`.EnvironmentContext.get_x_argument`.
.. change::
:tags: bug, autogenerate
:tickets: 1152
Fixed regression caused by :ticket:`1145` where the string transformations
applied to server defaults caused expressions such as ``(getdate())`` to no
longer compare as equivalent on SQL Server, others... changelog::
:version: 1.9.1
:released: December 23, 2022
.. change::
:tags: bug, autogenerate
:tickets: 1145
Fixed issue where server default compare would not work for string defaults
that contained backslashes, due to mis-rendering of these values when
comparing their contents.
.. change::
:tags: bug, oracle
Implemented basic server default comparison for the Oracle backend;
previously, Oracle's formatting of reflected defaults prevented any
matches from occurring.
.. change::
:tags: bug, sqlite
Adjusted SQLite's compare server default implementation to better handle
defaults with or without parens around them, from both the reflected and
the local metadata side.
.. change::
:tags: bug, mssql
Adjusted SQL Server's compare server default implementation to better
handle defaults with or without parens around them, from both the reflected
and the local metadata side... changelog::
:version: 1.9.0
:released: December 15, 2022
.. change::
:tags: feature, commands
:tickets: 724
Added new Alembic command ``alembic check``. This performs the widely
requested feature of running an "autogenerate" comparison between the
current database and the :class:`.MetaData` that's currently set up for
autogenerate, returning an error code if the two do not match, based on
current autogenerate settings. Pull request courtesy Nathan Louie.
.. seealso::
:ref:`alembic_check`
.. change::
:tags: bug, tests
Fixed issue in tox.ini file where changes in the tox 4.0 series to the
format of "passenv" caused tox to not function correctly, in particular
raising an error as of tox 4.0.6.
.. change::
:tags: bug, typing
:tickets: 1110
Fixed typing issue where :paramref:`.revision.process_revision_directives`
was not fully typed; additionally ensured all ``Callable`` and ``Dict``
arguments to :meth:`.EnvironmentContext.configure` include parameters in
the typing declaration.
Additionally updated the codebase for Mypy 0.990 compliance... changelog::
:version: 1.8.1
:released: July 13, 2022
.. change::
:tags: bug, sqlite
:tickets: 1065
Fixed bug where the SQLite implementation of
:meth:`.Operations.rename_table` would render an explicit schema name for
both the old and new table name, which while is the standard ALTER syntax,
is not accepted by SQLite's syntax which doesn't support a rename across
schemas. In particular, the syntax issue would prevent batch mode from
working for SQLite databases that made use of attached databases (which are
treated as "schemas" in SQLAlchemy).
.. change::
:tags: bug, batch
:tickets: 1021
Added an error raise for the condition where
:meth:`.Operations.batch_alter_table` is used in ``--sql`` mode, where the
operation requires table reflection, as is the case when running against
SQLite without giving it a fixed ``Table`` object. Previously the operation
would fail with an internal error. To get a "move and copy" batch
operation as a SQL script without connecting to a database,
a ``Table`` object should be passed to the
:paramref:`.Operations.batch_alter_table.copy_from` parameter so that
reflection may be skipped... changelog::
:version: 1.8.0
:released: May 31, 2022
.. change::
:tags: feature, typing
:tickets: 764
:pep:`484` typing annotations have been added to the ``env.py`` and
revision template files within migration templates. Pull request by Nikita
Sobolev.
.. change::
:tags: usecase, operations
:tickets: 1037
The ``op.drop_table()`` operation directive will now trigger the
``before_drop()`` and ``after_drop()`` DDL event hooks at the table level,
which is similar to how the ``before_create()`` and ``after_create()``
hooks are triggered by the ``op.create_table()`` directive. Note that as
``op.drop_table()`` accepts only a table name and optional schema name, the
``Table`` object received by the event will not have any information within
it other than the table name and schema name.
.. change::
:tags: installation, changed
:tickets: 1025
Alembic 1.8 now supports Python 3.7 and above.
.. change::
:tags: changed, environment
:tickets: 987
The "Pylons" environment template has been removed as of Alembic 1.8. This
template was based on the very old pre-Pyramid Pylons web framework which
has been long superseded by Pyramid.
.. change::
:tags: bug, revisioning
:tickets: 1026
Fixed issue where a downgrade using a relative revision would
fail in case of multiple branches with a single effectively
head due to interdependencies between revisions.
.. change::
:tags: usecase, commands
:tickets: 1027
Added new token ``epoch`` to the ``file_template`` option, which will
populate the integer epoch as determined by ``int(create_date.timestamp())``.
Pull request courtesy Caio Carvalho.
.. change::
:tags: bug, batch
:tickets: 1034
Fixed issue in batch mode where CREATE INDEX would not use a new column
name in the case of a column rename... changelog::
:version: 1.7.7
:released: March 14, 2022
.. change::
:tags: bug, operations
:tickets: 1004
Fixed issue where using :meth:`.Operations.create_table` in conjunction
with a :class:`.CheckConstraint` that referred to table-bound
:class:`.Column` objects rather than string expressions would be added to
the parent table potentially multiple times, resulting in an incorrect DDL
sequence. Pull request courtesy Nicolas CANIART.
.. change::
:tags: bug, environment
:tickets: 986
The ``logging.fileConfig()`` line in ``env.py`` templates, which is used
to setup Python logging for the migration run, is now conditional on
:attr:`.Config.config_file_name` not being ``None``. Otherwise, the line
is skipped as there is no default logging configuration present.
.. change::
:tags: bug, mssql
:tickets: 977
Fixed bug where an :meth:`.Operations.alter_column` operation would change
a "NOT NULL" column to "NULL" by emitting an ALTER COLUMN statement that
did not specify "NOT NULL". (In the absence of "NOT NULL" T-SQL was
implicitly assuming "NULL"). An :meth:`.Operations.alter_column` operation
that specifies :paramref:`.Operations.alter_column.type` should also
specify include either :paramref:`.Operations.alter_column.nullable` or
:paramref:`.Operations.alter_column.existing_nullable` to inform Alembic as
to whether the emitted DDL should include "NULL" or "NOT NULL"; a warning
is now emitted if this is missing under this scenario... changelog::
:version: 1.7.6
:released: February 1, 2022
.. change::
:tags: bug, batch, regression
:tickets: 982
Fixed regression where usage of a ``with_variant()`` datatype in
conjunction with the ``existing_type`` option of ``op.alter_column()``
under batch mode would lead to an internal exception.
.. change::
:tags: usecase, commands
:tickets: 964
Add a new command ``alembic ensure_version``, which will ensure that the
Alembic version table is present in the target database, but does not
alter its contents. Pull request courtesy Kai Mueller.
.. change::
:tags: bug, autogenerate
Implemented support for recognizing and rendering SQLAlchemy "variant"
types going forward into SQLAlchemy 2.0, where the architecture of
"variant" datatypes will be changing.
.. change::
:tags: bug, mysql, autogenerate
:tickets: 968
Added a rule to the MySQL impl so that the translation between JSON /
LONGTEXT is accommodated by autogenerate, treating LONGTEXT from the server
as equivalent to an existing JSON in the model.
.. change::
:tags: mssql
Removed a warning raised by SQLAlchemy when dropping constraints
on MSSQL regarding statement caching... changelog::
:version: 1.7.5
:released: November 11, 2021
.. change::
:tags: bug, tests
Adjustments to the test suite to accommodate for error message changes
occurring as of SQLAlchemy 1.4.27... changelog::
:version: 1.7.4
:released: October 6, 2021
.. change::
:tags: bug, regression
:tickets: 934
Fixed a regression that prevented the use of post write hooks
on python version lower than 3.9
.. change::
:tags: bug, environment
:tickets: 944
Fixed issue where the :meth:`.MigrationContext.autocommit_block` feature
would fail to function when using a SQLAlchemy engine using 2.0 future
mode... changelog::
:version: 1.7.3
:released: September 17, 2021
.. change::
:tags: bug, mypy
:tickets: 914
Fixed type annotations for the "constraint_name" argument of operations
``create_primary_key()``, ``create_foreign_key()``. Pull request courtesy
TilmanK... changelog::
:version: 1.7.2
:released: September 17, 2021
.. change::
:tags: bug, typing
:tickets: 900
Added missing attributes from context stubs.
.. change::
:tags: bug, mypy
:tickets: 897
Fixed an import in one of the .pyi files that was triggering an
assertion error in some versions of mypy.
.. change::
:tags: bug, regression, ops
:tickets: 920
Fixed issue where registration of custom ops was prone to failure due to
the registration process running ``exec()`` on generated code that as of
the 1.7 series includes pep-484 annotations, which in the case of end user
code would result in name resolution errors when the exec occurs. The logic
in question has been altered so that the annotations are rendered as
forward references so that the ``exec()`` can proceed... changelog::
:version: 1.7.1
:released: August 30, 2021
.. change::
:tags: bug, installation
:tickets: 893
Corrected "universal wheel" directive in setup.cfg so that building a wheel
does not target Python 2. The PyPi files index for 1.7.0 was corrected
manually. Pull request courtesy layday.
.. change::
:tags: bug, pep484
:tickets: 895
Fixed issue in generated .pyi files where default values for ``Optional``
arguments were missing, thereby causing mypy to consider them as required.
.. change::
:tags: bug, regression, batch
:tickets: 896
Fixed regression in batch mode due to :ticket:`883` where the "auto" mode
of batch would fail to accommodate any additional migration directives
beyond encountering an ``add_column()`` directive, due to a mis-application
of the conditional logic that was added as part of this change, leading to
"recreate" mode not being used in cases where it is required for SQLite
such as for unique constraints... changelog::
:version: 1.7.0
:released: August 30, 2021
.. change::
:tags: bug, operations
:tickets: 879
Fixed regression due to :ticket:`803` where the ``.info`` and ``.comment``
attributes of ``Table`` would be lost inside of the :class:`.DropTableOp`
class, which when "reversed" into a :class:`.CreateTableOp` would then have
lost these elements. Pull request courtesy Nicolas CANIART.
.. change::
:tags: feature, environment
:tickets: 842
Enhance ``version_locations`` parsing to handle paths containing spaces.
The new configuration option ``version_path_separator`` specifies the
character to use when splitting the ``version_locations`` string. The
default for new configurations is ``version_path_separator = os``,
which will use ``os.pathsep`` (e.g., ``;`` on Windows).
.. change::
:tags: installation, changed
Alembic 1.7 now supports Python 3.6 and above; support for prior versions
including Python 2.7 has been dropped.
.. change::
:tags: bug, sqlite, batch
:tickets: 883
Batch "auto" mode will now select for "recreate" if the ``add_column()``
operation is used on SQLite, and the column itself meets the criteria for
SQLite where ADD COLUMN is not allowed, in this case a functional or
parenthesized SQL expression or a ``Computed`` (i.e. generated) column.
.. change::
:tags: changed, installation
:tickets: 674
Make the ``python-dateutil`` library an optional dependency.
This library is only required if the ``timezone`` option
is used in the Alembic configuration.
An extra require named ``tz`` is available with
``pip install alembic[tz]`` to install it.
.. change::
:tags: bug, commands
:tickets: 856
Re-implemented the ``python-editor`` dependency as a small internal
function to avoid the need for external dependencies.
.. change::
:tags: usecase, batch
:tickets: 884
Named CHECK constraints are now supported by batch mode, and will
automatically be part of the recreated table assuming they are named. They
also can be explicitly dropped using ``op.drop_constraint()``. For
"unnamed" CHECK constraints, these are still skipped as they cannot be
distinguished from the CHECK constraints that are generated by the
``Boolean`` and ``Enum`` datatypes.
Note that this change may require adjustments to migrations that drop or
rename columns which feature an associated named check constraint, such
that an additional ``op.drop_constraint()`` directive should be added for
that named constraint as there will no longer be an associated column
for it; for the ``Boolean`` and ``Enum`` datatypes, an ``existing_type``
keyword may be passed to ``BatchOperations.drop_constraint`` as well.
.. seealso::
:ref:`batch_schematype_constraints`
:ref:`batch_check_constraints`
.. change::
:tags: changed, installation
:tickets: 885
The dependency on ``pkg_resources`` which is part of ``setuptools`` has
been removed, so there is no longer any runtime dependency on
``setuptools``. The functionality has been replaced with
``importlib.metadata`` and ``importlib.resources`` which are both part of
Python std.lib, or via pypy dependency ``importlib-metadata`` for Python
version < 3.8 and ``importlib-resources`` for Python version < 3.9
(while importlib.resources was added to Python in 3.7, it did not include
the "files" API until 3.9).
.. change::
:tags: feature, tests
:tickets: 855
Created a "test suite" similar to the one for SQLAlchemy, allowing
developers of third-party dialects to test their code against a set of
Alembic tests that have been specially selected to exercise
back-end database operations. At the time of release,
third-party dialects that have adopted the Alembic test suite to verify
compatibility include
`CockroachDB <https://pypi.org/project/sqlalchemy-cockroachdb/>`_ and
`SAP ASE (Sybase) <https://pypi.org/project/sqlalchemy-sybase/>`_.
.. change::
:tags: bug, postgresql
:tickets: 874
Fixed issue where usage of the PostgreSQL ``postgresql_include`` option
within a :meth:`.Operations.create_index` would raise a KeyError, as the
additional column(s) need to be added to the table object used by the
construct internally. The issue is equivalent to the SQL Server issue fixed
in :ticket:`513`. Pull request courtesy Steven Bronson.
.. change::
:tags: feature, general
pep-484 type annotations have been added throughout the library.
Additionally, stub .pyi files have been added for the "dynamically"
generated Alembic modules ``alembic.op`` and ``alembic.config``, which
include complete function signatures and docstrings, so that the functions
in these namespaces will have both IDE support (vscode, pycharm, etc) as
well as support for typing tools like Mypy. The files themselves are
statically generated from their source functions within the source tree... changelog::
:version: 1.6.5
:released: May 27, 2021
.. change::
:tags: bug, autogenerate
:tickets: 849
Fixed issue where dialect-specific keyword arguments within the
:class:`.DropIndex` operation directive would not render in the
autogenerated Python code. As support was improved for adding dialect
specific arguments to directives as part of :ticket:`803`, in particular
arguments such as "postgresql_concurrently" which apply to the actual
create/drop of the index, support was needed for these to render even in a
drop index operation. Pull request courtesy Jet Zhou... changelog::
:version: 1.6.4
:released: May 24, 2021
.. change::
:tags: bug, regression, op directives
:tickets: 848
Fixed regression caused by just fixed :ticket:`844` that scaled back the
filter for ``unique=True/index=True`` too far such that these directives no
longer worked for the ``op.create_table()`` op, this has been fixed... changelog::
:version: 1.6.3
:released: May 21, 2021
.. change::
:tags: bug, regression, autogenerate
:tickets: 844
Fixed 1.6-series regression where ``UniqueConstraint`` and to a lesser
extent ``Index`` objects would be doubled up in the generated model when
the ``unique=True`` / ``index=True`` flags were used.
.. change::
:tags: bug, autogenerate
:tickets: 839
Fixed a bug where paths defined in post-write hook options
would be wrongly escaped in non posix environment (Windows).
.. change::
:tags: bug, regression, versioning
:tickets: 843
Fixed regression where a revision file that contained its own down revision
as a dependency would cause an endless loop in the traversal logic... changelog::
:version: 1.6.2
:released: May 6, 2021
.. change::
:tags: bug, versioning, regression
:tickets: 839
Fixed additional regression nearly the same as that of :ticket:`838` just
released in 1.6.1 but within a slightly different codepath, where "alembic
downgrade head" (or equivalent) would fail instead of iterating no
revisions... changelog::
:version: 1.6.1
:released: May 6, 2021
.. change::
:tags: bug, versioning, regression
:tickets: 838
Fixed regression in new revisioning traversal where "alembic downgrade
base" would fail if the database itself were clean and unversioned;
additionally repairs the case where downgrade would fail if attempting
to downgrade to the current head that is already present... changelog::
:version: 1.6.0
:released: May 3, 2021
.. change::
:tags: bug, autogenerate
:tickets: 803
Refactored the implementation of :class:`.MigrateOperation` constructs such
as :class:`.CreateIndexOp`, :class:`.CreateTableOp`, etc. so that they no
longer rely upon maintaining a persistent version of each schema object
internally; instead, the state variables of each operation object will be
used to produce the corresponding construct when the operation is invoked.
The rationale is so that environments which make use of
operation-manipulation schemes such as those discussed in
:ref:`autogen_rewriter` are better supported, allowing end-user code to
manipulate the public attributes of these objects which will then be
expressed in the final output, an example is
``some_create_index_op.kw["postgresql_concurrently"] = True``.
Previously, these objects when generated from autogenerate would typically
hold onto the original, reflected element internally without honoring the
other state variables of each construct, preventing the public API from
working.
.. change::
:tags: bug, environment
:tickets: 829
Fixed regression caused by the SQLAlchemy 1.4/2.0 compatibility switch
where calling ``.rollback()`` or ``.commit()`` explicitly within the
``context.begin_transaction()`` context manager would cause it to fail when
the block ended, as it did not expect that the transaction was manually
closed.
.. change::
:tags: bug, autogenerate
:tickets: 827
Improved the rendering of ``op.add_column()`` operations when adding
multiple columns to an existing table, so that the order of these
statements matches the order in which the columns were declared in the
application's table metadata. Previously the added columns were being
sorted alphabetically.
.. change::
:tags: feature, autogenerate
:tickets: 819
Fix the documentation regarding the default command-line argument position of
the revision script filename within the post-write hook arguments. Implement a
``REVISION_SCRIPT_FILENAME`` token, enabling the position to be changed. Switch
from ``str.split()`` to ``shlex.split()`` for more robust command-line argument
parsing.
.. change::
:tags: feature
:tickets: 822
Implement a ``.cwd`` (current working directory) suboption for post-write hooks
(of type ``console_scripts``). This is useful for tools like pre-commit, which
rely on the working directory to locate the necessary config files. Add
pre-commit as an example to the documentation. Minor change: rename some variables
from ticket #819 to improve readability.
.. change::
:tags: bug, versioning
:tickets: 765, 464
The algorithm used for calculating downgrades/upgrades/iterating
revisions has been rewritten, to resolve ongoing issues of branches
not being handled consistently particularly within downgrade operations,
as well as for overall clarity and maintainability. This change includes
that a deprecation warning is emitted if an ambiguous command such
as "downgrade -1" when multiple heads are present is given.
In particular, the change implements a long-requested use case of allowing
downgrades of a single branch to a branchpoint.
Huge thanks to Simon Bowly for their impressive efforts in successfully
tackling this very difficult problem.
.. change::
:tags: bug, batch
:tickets: 799
Added missing ``batch_op.create_table_comment()``,
``batch_op.drop_table_comment()`` directives to batch ops... changelog::
:version: 1.5.8
:released: March 23, 2021
.. change::
:tags: bug, environment
:tickets: 816
Fixed regression caused by SQLAlchemy 1.4 where the "alembic current"
command would fail due to changes in the ``URL`` object... changelog::
:version: 1.5.7
:released: March 11, 2021
.. change::
:tags: bug, autogenerate
:tickets: 813
Adjusted the recently added
:paramref:`.EnvironmentContext.configure.include_name` hook to accommodate
for additional object types such as "views" that don't have a parent table,
to support third party recipes and extensions. Pull request courtesy Oliver
Rice... changelog::
:version: 1.5.6
:released: March 5, 2021
.. change::
:tags: bug, mssql, operations
:tickets: 812
Fixed bug where the "existing_type" parameter, which the MSSQL dialect
requires in order to change the nullability of a column in the absence of
also changing the column type, would cause an ALTER COLUMN operation to
incorrectly render a second ALTER statement without the nullability if a
new type were also present, as the MSSQL-specific contract did not
anticipate all three of "nullability", ``"type_"`` and "existing_type" being
sent at the same time.
.. change::
:tags: template
:ticket: 805
Add async template to Alembic to bootstrap environments that use
async DBAPI. Updated the cookbook to include a migration guide
on how to adapt an existing environment for use with DBAPI drivers... changelog::
:version: 1.5.5
:released: February 20, 2021
.. change::
:tags: bug
Adjusted the use of SQLAlchemy's ".copy()" internals to use "._copy()"
for version 1.4.0, as this method is being renamed.
.. change::
:tags: bug, environment
:tickets: 797
Added new config file option ``prepend_sys_path``, which is a series of
paths that will be prepended to sys.path; the default value in newly
generated alembic.ini files is ".". This fixes a long-standing issue
where for some reason running the alembic command line would not place the
local "." path in sys.path, meaning an application locally present in "."
and importable through normal channels, e.g. python interpreter, pytest,
etc. would not be located by Alembic, even though the ``env.py`` file is
loaded relative to the current path when ``alembic.ini`` contains a
relative path. To enable for existing installations, add the option to the
alembic.ini file as follows::
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
.. seealso::
:ref:`installation` - updated documentation reflecting that local
installation of the project is not necessary if running the Alembic cli
from the local path... changelog::
:version: 1.5.4
:released: February 3, 2021
.. change::
:tags: bug, versioning
:tickets: 789
Fixed bug in versioning model where a downgrade across a revision with a
dependency on another branch, yet an ancestor is also dependent on that
branch, would produce an erroneous state in the alembic_version table,
making upgrades impossible without manually repairing the table... changelog::
:version: 1.5.3
:released: January 29, 2021
.. change::
:tags: bug, autogenerate
:tickets: 786
Changed the default ordering of "CREATE" and "DROP" statements indexes and
unique constraints within the autogenerate process, so that for example in
an upgrade() operation, a particular index or constraint that is to be
replaced such as for a casing convention change will not produce any naming
conflicts. For foreign key constraint objects, this is already how
constraints are ordered, and for table objects, users would normally want
to use :meth:`.Operations.rename_table` in any case.
.. change::
:tags: bug, autogenerate, mssql
:tickets: 787
Fixed assorted autogenerate issues with SQL Server:
* ignore default reflected identity on primary_key columns
* improve server default comparison
.. change::
:tags: bug, mysql, autogenerate
:tickets: 788
Fixed issue where autogenerate rendering of ``op.alter_column()`` would
fail to include MySQL ``existing_nullable=False`` if the column were part
of a primary key constraint within the table metadata... changelog::
:version: 1.5.2
:released: January 20, 2021
.. change::
:tags: bug, versioning, regression
:tickets: 784
Fixed regression where new "loop detection" feature introduced in
:ticket:`757` produced false positives for revision names that have
overlapping substrings between revision number and down revision and/or
dependency, if the downrev/dependency were not in sequence form.
.. change::
:tags: bug, environment
:tickets: 782
Fixed regression where Alembic would fail to create a transaction properly
if the :class:`sqlalchemy.engine.Connection` were a so-called "branched"
connection, that is, one where the ``.connect()`` method had been called to
create a "sub" connection... changelog::
:version: 1.5.1
:released: January 19, 2021
.. change::
:tags: bug, installation, commands
:tickets: 780
Fixed installation issue where the "templates" directory was not being
installed, preventing commands like "list_templates" and "init" from
working... changelog::
:version: 1.5.0
:released: January 18, 2021
.. change::
:tags: usecase, operations
:tickets: 730
Added support for rendering of "identity" elements on
:class:`.Column` objects, supported in SQLAlchemy via
the :class:`.Identity` element introduced in version 1.4.
Adding columns with identity is supported on PostgreSQL,
MSSQL and Oracle. Changing the identity options or removing
it is supported only on PostgreSQL and Oracle.
.. change::
:tags: changed, environment
To accommodate SQLAlchemy 1.4 and 2.0, the migration model now no longer
assumes that the SQLAlchemy Connection will autocommit an individual
operation. This essentially means that for databases that use
non-transactional DDL (pysqlite current driver behavior, MySQL), there is
still a BEGIN/COMMIT block that will surround each individual migration.
Databases that support transactional DDL should continue to have the
same flow, either per migration or per-entire run, depending on the
value of the :paramref:`.Environment.configure.transaction_per_migration`
flag.
.. change::
:tags: changed, environment
A :class:`.CommandError` is raised if a ``sqlalchemy.engine.Engine`` is
passed to the :meth:`.MigrationContext.configure` method instead of a
``sqlalchemy.engine.Connection`` object. Previously, this would be a
warning only.
.. change::
:tags: bug, operations
:tickets: 753
Modified the ``add_column()`` operation such that the ``Column`` object in
use is shallow copied to a new instance if that ``Column`` is already
attached to a ``table()`` or ``Table``. This accommodates for the change
made in SQLAlchemy issue #5618 which prohibits a ``Column`` from being
associated with multiple ``table()`` objects. This resumes support for
using a ``Column`` inside of an Alembic operation that already refers to a
parent ``table()`` or ``Table`` as well as allows operation objects just
autogenerated to work.
.. change::
:tags: feature, autogenerate
:tickets: 650
Added new hook :paramref:`.EnvironmentContext.configure.include_name`,
which complements the
:paramref:`.EnvironmentContext.configure.include_object` hook by providing
a means of preventing objects of a certain name from being autogenerated
**before** the SQLAlchemy reflection process takes place, and notably
includes explicit support for passing each schema name when
:paramref:`.EnvironmentContext.configure.include_schemas` is set to True.
This is most important especially for environments that make use of
:paramref:`.EnvironmentContext.configure.include_schemas` where schemas are
actually databases (e.g. MySQL) in order to prevent reflection sweeps of
the entire server.
.. seealso::
:ref:`autogenerate_include_hooks` - new documentation section
.. change::
:tags: removed, autogenerate
The long deprecated
:paramref:`.EnvironmentContext.configure.include_symbol` hook is removed.
The :paramref:`.EnvironmentContext.configure.include_object`
and :paramref:`.EnvironmentContext.configure.include_name`
hooks both achieve the goals of this hook.
.. change::
:tags: bug, autogenerate
:tickets: 721
Added rendering for the ``Table.prefixes`` element to autogenerate so that
the rendered Python code includes these directives. Pull request courtesy
Rodrigo Ce Moretto.
.. change::
:tags: bug, batch
:tickets: 761
Added missing "create comment" feature for columns that are altered in
batch migrations.
.. change::
:tags: changed
:tickets: 748
Alembic 1.5.0 now supports **Python 2.7 and Python 3.6 and above**, as well
as **SQLAlchemy 1.3.0 and above**. Support is removed for Python 3
versions prior to 3.6 and SQLAlchemy versions prior to the 1.3 series.
.. change::
:tags: bug, batch
:tickets: 773
Made an adjustment to the PostgreSQL dialect to allow it to work more
effectively in batch mode, where a datatype like Boolean or non-native Enum
that may have embedded rules to generate CHECK constraints will be more
correctly handled in that these constraints usually will not have been
generated on the PostgreSQL backend; previously it would inadvertently
assume they existed unconditionally in a special PG-only "drop constraint"
step.
.. change::
:tags: feature, versioning
:tickets: 757
The revision tree is now checked for cycles and loops between revision
files when the revision environment is loaded up. Scenarios such as a
revision pointing to itself, or a revision that can reach itself via a
loop, are handled and will raise the :class:`.CycleDetected` exception when
the environment is loaded (expressed from the Alembic commandline as a
failure message and nonzero return code). Previously, these situations were
silently ignored up front, and the behavior of revision traversal would
either be silently incorrect, or would produce errors such as
:class:`.RangeNotAncestorError`. Pull request courtesy Koichiro Den.
.. change::
:tags: usecase, commands
Add ``__main__.py`` file to alembic package to support invocation
with ``python -m alembic``.
.. change::
:tags: removed, commands
Removed deprecated ``--head_only`` option to the ``alembic current``
command
.. change::
:tags: removed, operations
Removed legacy parameter names from operations, these have been emitting
warnings since version 0.8. In the case that legacy version files have not
yet been updated, these can be modified directly in order to maintain
compatibility:
* :meth:`.Operations.drop_constraint` - "type" (use ``"type_"``) and "name"
(use "constraint_name")
* :meth:`.Operations.create_primary_key` - "cols" (use "columns") and
"name" (use "constraint_name")
* :meth:`.Operations.create_unique_constraint` - "name" (use
"constraint_name"), "source" (use "table_name") and "local_cols" (use
"columns")
* :meth:`.Operations.batch_create_unique_constraint` - "name" (use
"constraint_name")
* :meth:`.Operations.create_foreign_key` - "name" (use "constraint_name"),
"source" (use "source_table"), "referent" (use "referent_table")
* :meth:`.Operations.batch_create_foreign_key` - "name" (use
"constraint_name"), "referent" (use "referent_table")
* :meth:`.Operations.create_check_constraint` - "name" (use
"constraint_name"), "source" (use "table_name")
* :meth:`.Operations.batch_create_check_constraint` - "name" (use
"constraint_name")
* :meth:`.Operations.create_index` - "name" (use "index_name")
* :meth:`.Operations.drop_index` - "name" (use "index_name"), "tablename"
(use "table_name")
* :meth:`.Operations.batch_drop_index` - "name" (use "index_name"),
* :meth:`.Operations.create_table` - "name" (use "table_name")
* :meth:`.Operations.drop_table` - "name" (use "table_name")
* :meth:`.Operations.alter_column` - "name" (use "new_column_name").. changelog::
:version: 1.4.3
:released: September 11, 2020
.. change::
:tags: bug, sqlite, batch
:tickets: 711
Added support to drop named CHECK constraints that are specified as part of
a column, rather than table wide. Previously, only constraints associated
with the table were considered.
.. change::
:tags: bug, ops, mysql
:tickets: 736
Fixed issue where the MySQL dialect would not correctly render the server
default of a column in an alter operation, if the operation were
programmatically generated from an autogenerate pass as it would not
accommodate for the full structure of the DefaultClause construct.
.. change::
:tags: bug, sqlite, batch
:tickets: 697
Fixed issue where the CAST applied to a JSON column when copying a SQLite
table during batch mode would cause the data to be lost, as SQLite's CAST
with JSON appears to convert the data to the value "0". The CAST is now
skipped in a dialect-specific manner, including for JSON columns on SQLite.
Pull request courtesy Sebastiรกn Ramรญrez.
.. change::
:tags: bug, commands
:tickets: 694
The ``alembic current`` command no longer creates an ``alembic_version``
table in the database if one does not exist already, returning no version
as the current version. This allows checking for migrations in parallel
without introducing race conditions. Pull request courtesy Nikolay
Edigaryev.
.. change::
:tags: bug, batch
Fixed issue where columns in a foreign-key referenced table would be
replaced with null-type columns during a batch operation; while this did
not generally have any side effects, it could theoretically impact a batch
operation that also targets that table directly and also would interfere
with future changes to the ``.append_column()`` method to disallow implicit
replacement of columns.
.. change::
:tags: bug, mssql
:tickets: 716
Fixed issue where the ``mssql_drop_foreign_key=True`` flag on
``op.drop_column`` would lead to incorrect syntax error due to a typo in the
SQL emitted, same typo was present in the test as well so it was not
detected. Pull request courtesy Oleg Shigorin... changelog::
:version: 1.4.2
:released: March 19, 2020
.. change::
:tags: usecase, autogenerate
:tickets: 669
Adjusted autogen comparison to accommodate for backends that support
computed column reflection, dependent on SQLAlchemy version 1.3.16 or
higher. This emits a warning if the SQL expression inside of a
:class:`.Computed` value changes between the metadata and the database, as
these expressions can't be changed without dropping and recreating the
column.
.. change::
:tags: bug, tests
:tickets: 668
Fixed an issue that prevented the test suite from running with the
recently released py.test 5.4.0.
.. change::
:tags: bug, autogenerate, mysql
:tickets: 671
Fixed more false-positive failures produced by the new "compare type" logic
first added in :ticket:`605`, particularly impacting MySQL string types
regarding flags such as "charset" and "collation".
.. change::
:tags: bug, op directives, oracle
:tickets: 670
Fixed issue in Oracle backend where a table RENAME with a schema-qualified
name would include the schema in the "to" portion, which is rejected by
Oracle... changelog::
:version: 1.4.1
:released: March 1, 2020
.. change::
:tags: bug, autogenerate
:tickets: 661
Fixed regression caused by the new "type comparison" logic introduced in
1.4 as part of :ticket:`605` where comparisons of MySQL "unsigned integer"
datatypes would produce false positives, as the regular expression logic
was not correctly parsing the "unsigned" token when MySQL's default display
width would be returned by the database. Pull request courtesy Paul
Becotte.
.. change::
:tags: bug, environment
:tickets: 663
Error message for "path doesn't exist" when loading up script environment
now displays the absolute path. Pull request courtesy Rowan Hart.
.. change::
:tags: bug, autogenerate
:tickets: 654
Fixed regression in 1.4.0 due to :ticket:`647` where unique constraint
comparison with mixed case constraint names while not using a naming
convention would produce false positives during autogenerate.
.. change::
:tags: bug, environment
The check for matched rowcount when the alembic_version table is updated or
deleted from is now conditional based on whether or not the dialect
supports the concept of "rowcount" for UPDATE or DELETE rows matched. Some
third party dialects do not support this concept. Pull request courtesy Ke
Zhu.
.. change::
:tags: bug, operations
:tickets: 655
Fixed long-standing bug where an inline column CHECK constraint would not
be rendered within an "ADD COLUMN" operation. The DDL compiler is now
consulted for inline constraints within the :meth:`.Operations.add_column`
method as is done for regular CREATE TABLE operations... changelog::
:version: 1.4.0
:released: February 4, 2020
.. change::
:tags: change
The internal inspection routines no longer use SQLAlchemy's
``Inspector.from_engine()`` method, which is expected to be deprecated in
1.4. The ``inspect()`` function is now used.
.. change::
:tags: bug, autogenerate
:tickets: 647
Adjusted the unique constraint comparison logic in a similar manner as that
of :ticket:`421` did for indexes in order to take into account SQLAlchemy's
own truncation of long constraint names when a naming convention is in use.
Without this step, a name that is truncated by SQLAlchemy based on a unique
constraint naming convention or hardcoded name will not compare properly.
.. change::
:tags: feature, batch
:tickets: 640
Added new parameters :paramref:`.BatchOperations.add_column.insert_before`,
:paramref:`.BatchOperations.add_column.insert_after` which provide for
establishing the specific position in which a new column should be placed.
Also added :paramref:`.Operations.batch_alter_table.partial_reordering`
which allows the complete set of columns to be reordered when the new table
is created. Both operations apply only to when batch mode is recreating
the whole table using ``recreate="always"``. Thanks to Marcin Szymanski
for assistance with the implementation.
.. change::
:tags: usecase, environment
:tickets: 648
Moved the use of the ``__file__`` attribute at the base of the Alembic
package into the one place that it is specifically needed, which is when
the config attempts to locate the template directory. This helps to allow
Alembic to be fully importable in environments that are using Python
memory-only import schemes. Pull request courtesy layday.
.. change::
:tags: bug, autogenerate
:tickets: 605
A major rework of the "type comparison" logic is in place which changes the
entire approach by which column datatypes are compared. Types are now
compared based on the DDL string generated by the metadata type vs. the
datatype reflected from the database. This means we compare types based on
what would actually render and additionally if elements of the types change
like string length, those changes are detected as well. False positives
like those generated between SQLAlchemy Boolean and MySQL TINYINT should
also be resolved. Thanks very much to Paul Becotte for lots of hard work
and patience on this one.
.. note:: *updated* - this change also removes support for the
``compare_against_backend`` SQLAlchemy type hook.
.. seealso::
:ref:`autogenerate_detects` - updated comments on type comparison.. changelog::
:version: 1.3.3
:released: January 22, 2020
.. change::
:tags: bug, postgresql
:tickets: 637
Fixed issue where COMMENT directives for PostgreSQL failed to correctly
include an explicit schema name, as well as correct quoting rules for
schema, table, and column names. Pull request courtesy Matthew Sills.
.. change::
:tags: usecase, operations
:tickets: 624
Added support for rendering of "computed" elements on :class:`.Column`
objects, supported in SQLAlchemy via the new :class:`.Computed` element
introduced in version 1.3.11. Pull request courtesy Federico Caselli.
Note that there is currently no support for ALTER COLUMN to add, remove, or
modify the "GENERATED ALWAYS AS" element from a column; at least for
PostgreSQL, it does not seem to be supported by the database. Additionally,
SQLAlchemy does not currently reliably reflect the "GENERATED ALWAYS AS"
phrase from an existing column, so there is also no autogenerate support
for addition or removal of the :class:`.Computed` element to or from an
existing column, there is only support for adding new columns that include
the :class:`.Computed` element. In the case that the :class:`.Computed`
element is removed from the :class:`.Column` object in the table metadata,
PostgreSQL and Oracle currently reflect the "GENERATED ALWAYS AS"
expression as the "server default" which will produce an op that tries to
drop the element as a default... changelog::
:version: 1.3.2
:released: December 16, 2019
.. change::
:tags: bug, api, autogenerate
:tickets: 635
Fixed regression introduced by :ticket:`579` where server default rendering
functions began to require a dialect implementation, however the
:func:`.render_python_code` convenience function did not include one, thus
causing the function to fail when used in a server default context. The
function now accepts a migration context argument and also creates one
against the default dialect if one is not provided... changelog::
:version: 1.3.1
:released: November 13, 2019
.. change::
:tags: bug, mssql
:tickets: 621
Fixed bug in MSSQL dialect where the drop constraint execution steps used
to remove server default or implicit foreign key constraint failed to take
into account the schema name of the target table... changelog::
:version: 1.3.0
:released: October 31, 2019
.. change::
:tags: feature, command
:tickets: 608
Added support for ALEMBIC_CONFIG environment variable,
refers to the location of the alembic configuration script
in lieu of using the -c command line option.
.. change::
:tags: bug, autogenerate
:tickets: 131
Fixed bug in new Variant autogenerate where the order of the arguments to
Variant were mistakenly reversed.
.. change::
:tags: change, compatibility
Some internal modifications have been made to how the names of indexes and
unique constraints work to make use of new functions added in SQLAlchemy
1.4, so that SQLAlchemy has more flexibility over how naming conventions
may be applied to these objects... changelog::
:version: 1.2.1
:released: September 24, 2019
.. change::
:tags: bug, command
:tickets: 601
Reverted the name change of the "revisions" argument to
:func:`.command.stamp` to "revision" as apparently applications are
calling upon this argument as a keyword name. Pull request courtesy
Thomas Bechtold. Special translations are also added to the command
line interface so that it is still known as "revisions" in the CLI.
.. change::
:tags: bug, tests
:tickets: 592
Removed the "test requirements" from "setup.py test", as this command now
only emits a removal error in any case and these requirements are unused... changelog::
:version: 1.2.0
:released: September 20, 2019
.. change::
:tags: feature, command
:tickets: 473
Added new ``--purge`` flag to the ``alembic stamp`` command, which will
unconditionally erase the version table before stamping anything. This is
useful for development where non-existent version identifiers might be left
within the table. Additionally, ``alembic.stamp`` now supports a list of
revision identifiers, which are intended to allow setting up multiple heads
at once. Overall handling of version identifiers within the
``alembic.stamp`` command has been improved with many new tests and
use cases added.
.. change::
:tags: bug, autogenerate
:tickets: 550
Improved the Python rendering of a series of migration operations such that
a single "pass" is rendered for a :class:`.UpgradeOps` or
:class:`.DowngradeOps` based on if no lines of Python code actually
rendered under the operation, rather than whether or not sub-directives
exist. Removed extra "pass" lines that would generate from the
:class:`.ModifyTableOps` directive so that these aren't duplicated under
operation rewriting scenarios.
.. change::
:tags: feature, runtime
:tickets: 123
Added new feature :meth:`.MigrationContext.autocommit_block`, a special
directive which will provide for a non-transactional block inside of a
migration script. The feature requires that: the database driver
(e.g. DBAPI) supports the AUTOCOMMIT isolation mode. The directive
also necessarily needs to COMMIT the existing transaction in progress
in order to enter autocommit mode.
.. seealso::
:meth:`.MigrationContext.autocommit_block`
.. change::
:tags: change: py3k
Python 3.4 support is dropped, as the upstream tooling (pip, mysqlclient)
etc are already dropping support for Python 3.4, which itself is no longer
maintained.
.. change::
:tags: usecase, autogenerate
:tickets: 518
Added autogenerate support for :class:`.Column` objects that have
dialect-specific ``**kwargs``, support first added in SQLAlchemy 1.3.
This includes SQLite "on conflict" as well as options used by some
third party dialects.
.. change::
:tags: usecase, autogenerate
:tickets: 131
Added rendering for SQLAlchemy ``Variant`` datatypes, which render as the
base type plus one or more ``.with_variant()`` method calls.
.. change::
:tags: usecase, commands
:tickets: 534
Made the command interface revision lookup behavior more strict in that an
Alembic revision number is only resolved based on a partial match rules if
it has at least four characters, to prevent simple typographical issues
from inadvertently running migrations.
.. change::
:tags: feature, commands
:tickets: 307
Added "post write hooks" to revision generation. These allow custom logic
to run after a revision Python script is generated, typically for the
purpose of running code formatters such as "Black" or "autopep8", but may
be used for any arbitrary post-render hook as well, including custom Python
functions or scripts. The hooks are enabled by providing a
``[post_write_hooks]`` section in the alembic.ini file. A single hook
is provided which runs an arbitrary Python executable on the newly
generated revision script, which can be configured to run code formatters
such as Black; full examples are included in the documentation.
.. seealso::
:ref:`post_write_hooks`
.. change::
:tags: feature, environment
:tickets: 463
Added new flag ``--package`` to ``alembic init``. For environments where
the Alembic migration files and such are within the package tree and
importable as modules, this flag can be specified which will add the
additional ``__init__.py`` files in the version location and the
environment location.
.. change::
:tags: bug, autogenerate
:tickets: 549
Fixed bug where rendering of comment text for table-level comments within
:meth:`.Operations.create_table_comment` and
:meth:`.Operations.drop_table_comment` was not properly quote-escaped
within rendered Python code for autogenerate.
.. change::
:tags: bug, autogenerate
:tickets: 505
Modified the logic of the :class:`.Rewriter` object such that it keeps a
memoization of which directives it has processed, so that it can ensure it
processes a particular directive only once, and additionally fixed
:class:`.Rewriter` so that it functions correctly for multiple-pass
autogenerate schemes, such as the one illustrated in the "multidb"
template. By tracking which directives have been processed, a
multiple-pass scheme which calls upon the :class:`.Rewriter` multiple times
for the same structure as elements are added can work without running
duplicate operations on the same elements more than once... changelog::
:version: 1.1.0
:released: August 26, 2019
.. change::
:tags: change
Alembic 1.1 bumps the minimum version of SQLAlchemy to 1.1. As was the
case before, Python requirements remain at Python 2.7, or in the 3.x series
Python 3.4.
.. change::
:tags: change, internals
The test suite for Alembic now makes use of SQLAlchemy's testing framework
directly. Previously, Alembic had its own version of this framework that
was mostly copied from that of SQLAlchemy to enable testing with older
SQLAlchemy versions. The majority of this code is now removed so that both
projects can leverage improvements from a common testing framework.
.. change::
:tags: bug, commands
:tickets: 562
Fixed bug where the double-percent logic applied to some dialects such as
psycopg2 would be rendered in ``--sql`` mode, by allowing dialect options
to be passed through to the dialect used to generate SQL and then providing
``paramstyle="named"`` so that percent signs need not be doubled. For
users having this issue, existing env.py scripts need to add
``dialect_opts={"paramstyle": "named"}`` to their offline
context.configure(). See the ``alembic/templates/generic/env.py`` template
for an example.
.. change::
:tags: bug, py3k
Fixed use of the deprecated "imp" module, which is used to detect pep3147
availability as well as to locate .pyc files, which started emitting
deprecation warnings during the test suite. The warnings were not being
emitted earlier during the test suite, the change is possibly due to
changes in py.test itself but this is not clear. The check for pep3147 is
set to True for any Python version 3.5 or greater now and importlib is used
when available. Note that some dependencies such as distutils may still be
emitting this warning. Tests are adjusted to accommodate for dependencies
that emit the warning as well.
.. change::
:tags: bug, mysql
:tickets: 594
Fixed issue where emitting a change of column name for MySQL did not
preserve the column comment, even if it were specified as existing_comment.
.. change::
:tags: bug, setup
:tickets: 592
Removed the "python setup.py test" feature in favor of a straight run of
"tox". Per Pypa / pytest developers, "setup.py" commands are in general
headed towards deprecation in favor of tox. The tox.ini script has been
updated such that running "tox" with no arguments will perform a single run
of the test suite against the default installed Python interpreter.
.. seealso::
https://github.com/pypa/setuptools/issues/1684
https://github.com/pytest-dev/pytest/issues/5534
.. change::
:tags: usecase, commands
:tickets: 571
The "alembic init" command will now proceed if the target directory exists
as long as it's still empty. Previously, it would not proceed if the
directory existed. The new behavior is modeled from what git does, to
accommodate for container or other deployments where an Alembic target
directory may need to be already mounted instead of being created with
alembic init. Pull request courtesy Aviskar KC... changelog::
:version: 1.0.11
:released: June 25, 2019
.. change::
:tags: bug, sqlite, autogenerate, batch
:tickets: 579
SQLite server default reflection will ensure parenthesis are surrounding a
column default expression that is detected as being a non-constant
expression, such as a ``datetime()`` default, to accommodate for the
requirement that SQL expressions have to be parenthesized when being sent
as DDL. Parenthesis are not added to constant expressions to allow for
maximum cross-compatibility with other dialects and existing test suites
(such as Alembic's), which necessarily entails scanning the expression to
eliminate for constant numeric and string values. The logic is added to the
two "reflection->DDL round trip" paths which are currently autogenerate and
batch migration. Within autogenerate, the logic is on the rendering side,
whereas in batch the logic is installed as a column reflection hook.
.. change::
:tags: bug, sqlite, autogenerate
:tickets: 579
Improved SQLite server default comparison to accommodate for a ``text()``
construct that added parenthesis directly vs. a construct that relied
upon the SQLAlchemy SQLite dialect to render the parenthesis, as well
as improved support for various forms of constant expressions such as
values that are quoted vs. non-quoted.
.. change::
:tags: bug, autogenerate
Fixed bug where the "literal_binds" flag was not being set when
autogenerate would create a server default value, meaning server default
comparisons would fail for functions that contained literal values.
.. change::
:tags: bug, mysql
:tickets: 554
Added support for MySQL "DROP CHECK", which is added as of MySQL 8.0.16,
separate from MariaDB's "DROP CONSTRAINT" for CHECK constraints. The MySQL
Alembic implementation now checks for "MariaDB" in server_version_info to
decide which one to use.
.. change::
:tags: bug, mysql, operations
:tickets: 564
Fixed issue where MySQL databases need to use CHANGE COLUMN when altering a
server default of CURRENT_TIMESTAMP, NOW() and probably other functions
that are only usable with DATETIME/TIMESTAMP columns. While MariaDB
supports both CHANGE and ALTER COLUMN in this case, MySQL databases only
support CHANGE. So the new logic is that if the server default change is
against a DateTime-oriented column, the CHANGE format is used
unconditionally, as in the vast majority of cases the server default is to
be CURRENT_TIMESTAMP which may also be potentially bundled with an "ON
UPDATE CURRENT_TIMESTAMP" directive, which SQLAlchemy does not currently
support as a distinct field. The fix additionally improves the server
default comparison logic when the "ON UPDATE" clause is present and
there are parenthesis to be adjusted for as is the case on some MariaDB
versions.
.. change::
:tags: bug, environment
Warnings emitted by Alembic now include a default stack level of 2, and in
some cases it's set to 3, in order to help warnings indicate more closely
where they are originating from. Pull request courtesy Ash Berlin-Taylor.
.. change::
:tags: bug, py3k
:tickets: 563
Replaced the Python compatibility routines for ``getargspec()`` with a fully
vendored version based on ``getfullargspec()`` from Python 3.3.
Originally, Python was emitting deprecation warnings for this function in
Python 3.8 alphas. While this change was reverted, it was observed that
Python 3 implementations for ``getfullargspec()`` are an order of magnitude
slower as of the 3.4 series where it was rewritten against ``Signature``.
While Python plans to improve upon this situation, SQLAlchemy projects for
now are using a simple replacement to avoid any future issues... changelog::
:version: 1.0.10
:released: April 28, 2019
.. change::
:tags: bug, commands
:tickets: 552
Fixed bug introduced in release 0.9.0 where the helptext for commands
inadvertently got expanded to include function docstrings from the
command.py module. The logic has been adjusted to only refer to the first
line(s) preceding the first line break within each docstring, as was the
original intent.
.. change::
:tags: bug, operations, mysql
:tickets: 551
Added an assertion in :meth:`.RevisionMap.get_revisions` and other methods
which ensures revision numbers are passed as strings or collections of
strings. Driver issues particularly on MySQL may inadvertently be passing
bytes here which leads to failures later on.
.. change::
:tags: bug, autogenerate, mysql
:tickets: 553
Fixed bug when using the
:paramref:`.EnvironmentContext.configure.compare_server_default` flag set
to ``True`` where a server default that is introduced in the table metadata
on an ``Integer`` column, where there is no existing server default in the
database, would raise a ``TypeError``... changelog::
:version: 1.0.9
:released: April 15, 2019
.. change::
:tags: bug, operations
:tickets: 548
Simplified the internal scheme used to generate the ``alembic.op`` namespace
to no longer attempt to generate full method signatures (e.g. rather than
generic ``*args, **kw``) as this was not working in most cases anyway, while
in rare circumstances it would in fact sporadically have access to the real
argument names and then fail when generating the function due to missing
symbols in the argument signature... changelog::
:version: 1.0.8
:released: March 4, 2019
.. change::
:tags: bug, operations
:tickets: 528
Removed use of deprecated ``force`` parameter for SQLAlchemy quoting
functions as this parameter will be removed in a future release.
Pull request courtesy Parth Shandilya(ParthS007).
.. change::
:tags: bug, autogenerate, postgresql, py3k
:tickets: 541
Fixed issue where server default comparison on the PostgreSQL dialect would
fail for a blank string on Python 3.7 only, due to a change in regular
expression behavior in Python 3.7... changelog::
:version: 1.0.7
:released: January 25, 2019
.. change::
:tags: bug, autogenerate
:tickets: 529
Fixed issue in new comment support where autogenerated Python code
for comments wasn't using ``repr()`` thus causing issues with
quoting. Pull request courtesy Damien Garaud... changelog::
:version: 1.0.6
:released: January 13, 2019
.. change::
:tags: feature, operations
:tickets: 422
Added Table and Column level comments for supported backends.
New methods :meth:`.Operations.create_table_comment` and
:meth:`.Operations.drop_table_comment` are added. A new arguments
:paramref:`.Operations.alter_column.comment` and
:paramref:`.Operations.alter_column.existing_comment` are added to
:meth:`.Operations.alter_column`. Autogenerate support is also added
to ensure comment add/drops from tables and columns are generated as well
as that :meth:`.Operations.create_table`, :meth:`.Operations.add_column`
both include the comment field from the source :class:`.Table`
or :class:`.Column` object... changelog::
:version: 1.0.5
:released: November 27, 2018
.. change::
:tags: bug, py3k
:tickets: 507
Resolved remaining Python 3 deprecation warnings, covering
the use of inspect.formatargspec() with a vendored version
copied from the Python standard library, importing
collections.abc above Python 3.3 when testing against abstract
base classes, fixed one occurrence of log.warn(), as well as a few
invalid escape sequences... changelog::
:version: 1.0.4
:released: November 27, 2018
.. change::
:tags: change
Code hosting has been moved to GitHub, at
https://github.com/sqlalchemy/alembic. Additionally, the
main Alembic website documentation URL is now
https://alembic.sqlalchemy.org... changelog::
:version: 1.0.3
:released: November 14, 2018
.. change::
:tags: bug, mssql
:tickets: 516
Fixed regression caused by :ticket:`513`, where the logic to consume
``mssql_include`` was not correctly interpreting the case where the flag
was not present, breaking the ``op.create_index`` directive for SQL Server
as a whole... changelog::
:version: 1.0.2
:released: October 31, 2018
.. change::
:tags: bug, autogenerate
:tickets: 515
The ``system=True`` flag on :class:`.Column`, used primarily in conjunction
with the Postgresql "xmin" column, now renders within the autogenerate
render process, allowing the column to be excluded from DDL. Additionally,
adding a system=True column to a model will produce no autogenerate diff as
this column is implicitly present in the database.
.. change::
:tags: bug, mssql
:tickets: 513
Fixed issue where usage of the SQL Server ``mssql_include`` option within a
:meth:`.Operations.create_index` would raise a KeyError, as the additional
column(s) need to be added to the table object used by the construct
internally... changelog::
:version: 1.0.1
:released: October 17, 2018
.. change::
:tags: bug, commands
:tickets: 497
Fixed an issue where revision descriptions were essentially
being formatted twice. Any revision description that contained
characters like %, writing output to stdout will fail because
the call to config.print_stdout attempted to format any
additional args passed to the function.
This fix now only applies string formatting if any args are provided
along with the output text.
.. change::
:tags: bug, autogenerate
:tickets: 512
Fixed issue where removed method ``union_update()`` was used when a
customized :class:`.MigrationScript` instance included entries in the
``.imports`` data member, raising an AttributeError... changelog::
:version: 1.0.0
:released: July 13, 2018
:released: July 13, 2018
:released: July 13, 2018
.. change::
:tags: feature, general
:tickets: 491
For Alembic 1.0, Python 2.6 / 3.3 support is being dropped, allowing a
fixed setup.py to be built as well as universal wheels. Pull request
courtesy Hugo.
.. change::
:tags: feature, general
With the 1.0 release, Alembic's minimum SQLAlchemy support version
moves to 0.9.0, previously 0.7.9.
.. change::
:tags: bug, batch
:tickets: 502
Fixed issue in batch where dropping a primary key column, then adding it
back under the same name but without the primary_key flag, would not remove
it from the existing PrimaryKeyConstraint. If a new PrimaryKeyConstraint
is added, it is used as-is, as was the case before... changelog::
:version: 0.9.10
:released: June 29, 2018
.. change::
:tags: bug, autogenerate
The "op.drop_constraint()" directive will now render using ``repr()`` for
the schema name, in the same way that "schema" renders for all the other op
directives. Pull request courtesy Denis Kataev.
.. change::
:tags: bug, autogenerate
:tickets: 494
Added basic capabilities for external dialects to support rendering of
"nested" types, like arrays, in a manner similar to that of the Postgresql
dialect.
.. change::
:tags: bug, autogenerate
Fixed issue where "autoincrement=True" would not render for a column that
specified it, since as of SQLAlchemy 1.1 this is no longer the default
value for "autoincrement". Note the behavior only takes effect against the
SQLAlchemy 1.1.0 and higher; for pre-1.1 SQLAlchemy, "autoincrement=True"
does not render as was the case before. Pull request courtesy Elad Almos... changelog::
:version: 0.9.9
:released: March 22, 2018
.. change::
:tags: feature, commands
:tickets: 481
Added new flag ``--indicate-current`` to the ``alembic history`` command.
When listing versions, it will include the token "(current)" to indicate
the given version is a current head in the target database. Pull request
courtesy Kazutaka Mise.
.. change::
:tags: bug, autogenerate, mysql
:tickets: 455
The fix for :ticket:`455` in version 0.9.6 involving MySQL server default
comparison was entirely non functional, as the test itself was also broken
and didn't reveal that it wasn't working. The regular expression to compare
server default values like CURRENT_TIMESTAMP to current_timestamp() is
repaired.
.. change::
:tags: bug, mysql, autogenerate
:tickets: 483
Fixed bug where MySQL server default comparisons were basically not working
at all due to incorrect regexp added in :ticket:`455`. Also accommodates
for MariaDB 10.2 quoting differences in reporting integer based server
defaults.
.. change::
:tags: bug, operations, mysql
:tickets: 487
Fixed bug in ``op.drop_constraint()`` for MySQL where
quoting rules would not be applied to the constraint name... changelog::
:version: 0.9.8
:released: February 16, 2018
.. change::
:tags: bug, runtime
:tickets: 482
Fixed bug where the :meth:`.Script.as_revision_number` method
did not accommodate for the 'heads' identifier, which in turn
caused the :meth:`.EnvironmentContext.get_head_revisions`
and :meth:`.EnvironmentContext.get_revision_argument` methods
to be not usable when multiple heads were present.
The :meth:.`EnvironmentContext.get_head_revisions` method returns
a tuple in all cases as documented.
.. change::
:tags: bug, postgresql, autogenerate
:tickets: 478
Fixed bug where autogenerate of :class:`.ExcludeConstraint`
would render a raw quoted name for a Column that has case-sensitive
characters, which when invoked as an inline member of the Table
would produce a stack trace that the quoted name is not found.
An incoming Column object is now rendered as ``sa.column('name')``.
.. change::
:tags: bug, autogenerate
:tickets: 468
Fixed bug where the indexes would not be included in a
migration that was dropping the owning table. The fix
now will also emit DROP INDEX for the indexes ahead of time,
but more importantly will include CREATE INDEX in the
downgrade migration.
.. change::
:tags: bug, postgresql
:tickets: 480
Fixed the autogenerate of the module prefix
when rendering the text_type parameter of
postgresql.HSTORE, in much the same way that
we do for ARRAY's type and JSON's text_type.
.. change::
:tags: bug, mysql
:tickets: 479
Added support for DROP CONSTRAINT to the MySQL Alembic
dialect to support MariaDB 10.2 which now has real
CHECK constraints. Note this change does **not**
add autogenerate support, only support for op.drop_constraint()
to work... changelog::
:version: 0.9.7
:released: January 16, 2018
.. change::
:tags: bug, autogenerate
:tickets: 472
Fixed regression caused by :ticket:`421` which would
cause case-sensitive quoting rules to interfere with the
comparison logic for index names, thus causing indexes to show
as added for indexes that have case-sensitive names. Works with
SQLAlchemy 0.9 and later series.
.. change::
:tags: bug, postgresql, autogenerate
:tickets: 461
Fixed bug where autogenerate would produce a DROP statement for the index
implicitly created by a Postgresql EXCLUDE constraint, rather than skipping
it as is the case for indexes implicitly generated by unique constraints.
Makes use of SQLAlchemy 1.0.x's improved "duplicates index" metadata and
requires at least SQLAlchemy version 1.0.x to function correctly... changelog::
:version: 0.9.6
:released: October 13, 2017
.. change::
:tags: bug, commands
:tickets: 458
Fixed a few Python3.6 deprecation warnings by replacing ``StopIteration``
with ``return``, as well as using ``getfullargspec()`` instead of
``getargspec()`` under Python 3.
.. change::
:tags: bug, commands
:tickets: 441
An addition to :ticket:`441` fixed in 0.9.5, we forgot to also filter
for the ``+`` sign in migration names which also breaks due to the relative
migrations feature.
.. change::
:tags: bug, autogenerate
:tickets: 442
Fixed bug expanding upon the fix for
:ticket:`85` which adds the correct module import to the
"inner" type for an ``ARRAY`` type, the fix now accommodates for the
generic ``sqlalchemy.types.ARRAY`` type added in SQLAlchemy 1.1,
rendering the inner type correctly regardless of whether or not the
Postgresql dialect is present.
.. change::
:tags: bug, mysql
:tickets: 455
Fixed bug where server default comparison of CURRENT_TIMESTAMP would fail
on MariaDB 10.2 due to a change in how the function is
represented by the database during reflection.
.. change::
:tags: bug, autogenerate
Fixed bug where comparison of ``Numeric`` types would produce
a difference if the Python-side ``Numeric`` inadvertently specified
a non-None "scale" with a "precision" of None, even though this ``Numeric``
type will pass over the "scale" argument when rendering. Pull request
courtesy Ivan Mmelnychuk.
.. change::
:tags: feature, commands
:tickets: 447
The ``alembic history`` command will now make use of the revision
environment ``env.py`` unconditionally if the ``revision_environment``
configuration flag is set to True. Previously, the environment would
only be invoked if the history specification were against a database-stored
revision token.
.. change::
:tags: bug, batch
:tickets: 457
The name of the temporary table in batch mode is now generated
off of the original table name itself, to avoid conflicts for the
unusual case of multiple batch operations running against the same
database schema at the same time.
.. change::
:tags: bug, autogenerate
:tickets: 456
A :class:`.ForeignKeyConstraint` can now render correctly if the
``link_to_name`` flag is set, as it will not attempt to resolve the name
from a "key" in this case. Additionally, the constraint will render
as-is even if the remote column name isn't present on the referenced
remote table.
.. change::
:tags: bug, runtime, py3k
:tickets: 449
Reworked "sourceless" system to be fully capable of handling any
combination of: Python2/3x, pep3149 or not, PYTHONOPTIMIZE or not,
for locating and loading both env.py files as well as versioning files.
This includes: locating files inside of ``__pycache__`` as well as listing
out version files that might be only in ``versions/__pycache__``, deduplicating
version files that may be in ``versions/__pycache__`` and ``versions/``
at the same time, correctly looking for .pyc or .pyo files based on
if pep488 is present or not. The latest Python3x deprecation warnings
involving importlib are also corrected... changelog::
:version: 0.9.5
:released: August 9, 2017
.. change::
:tags: bug, commands
:tickets: 441
A :class:`.CommandError` is raised if the "--rev-id" passed to the
:func:`.revision` command contains dashes or at-signs, as this interferes
with the command notation used to locate revisions.
.. change::
:tags: bug, postgresql
:tickets: 424
Added support for the dialect-specific keyword arguments
to :meth:`.Operations.drop_index`. This includes support for
``postgresql_concurrently`` and others.
.. change::
:tags: bug, commands
Fixed bug in timezone feature introduced in
:ticket:`425` when the creation
date in a revision file is calculated, to
accommodate for timezone names that contain
mixed-case characters in their name as opposed
to all uppercase. Pull request courtesy Nils
Philippsen... changelog::
:version: 0.9.4
:released: July 31, 2017
.. change::
:tags: bug, runtime
Added an additional attribute to the new
:paramref:`.EnvironmentContext.configure.on_version_apply` API,
:attr:`.MigrationInfo.up_revision_ids`, to accommodate for the uncommon
case of the ``alembic stamp`` command being used to move from multiple
branches down to a common branchpoint; there will be multiple
"up" revisions in this one case... changelog::
:version: 0.9.3
:released: July 6, 2017
.. change::
:tags: feature, runtime
Added a new callback hook
:paramref:`.EnvironmentContext.configure.on_version_apply`,
which allows user-defined code to be invoked each time an individual
upgrade, downgrade, or stamp operation proceeds against a database.
Pull request courtesy John Passaro.
.. change:: 433
:tags: bug, autogenerate
:tickets: 433
Fixed bug where autogen comparison of a :class:`.Variant` datatype
would not compare to the dialect level type for the "default"
implementation of the :class:`.Variant`, returning the type as changed
between database and table metadata.
.. change:: 431
:tags: bug, tests
:tickets: 431
Fixed unit tests to run correctly under the SQLAlchemy 1.0.x series
prior to version 1.0.10 where a particular bug involving Postgresql
exclude constraints was fixed... changelog::
:version: 0.9.2
:released: May 18, 2017
.. change:: 429
:tags: bug, mssql
:tickets: 429
Repaired :meth:`.Operations.rename_table` for SQL Server when the
target table is in a remote schema, the schema name is omitted from
the "new name" argument.
.. change:: 425
:tags: feature, commands
:tickets: 425
Added a new configuration option ``timezone``, a string timezone name
that will be applied to the create date timestamp rendered
inside the revision file as made available to the ``file_template`` used
to generate the revision filename. Note this change adds the
``python-dateutil`` package as a dependency.
.. change:: 421
:tags: bug, autogenerate
:tickets: 421
The autogenerate compare scheme now takes into account the name truncation
rules applied by SQLAlchemy's DDL compiler to the names of the
:class:`.Index` object, when these names are dynamically truncated
due to a too-long identifier name. As the identifier truncation is
deterministic, applying the same rule to the metadata name allows
correct comparison to the database-derived name.
.. change:: 419
:tags: bug environment
:tickets: 419
A warning is emitted when an object that's not a
:class:`~sqlalchemy.engine.Connection` is passed to
:meth:`.EnvironmentContext.configure`. For the case of a
:class:`~sqlalchemy.engine.Engine` passed, the check for "in transaction"
introduced in version 0.9.0 has been relaxed to work in the case of an
attribute error, as some users appear to be passing an
:class:`~sqlalchemy.engine.Engine` and not a
:class:`~sqlalchemy.engine.Connection`... changelog::
:version: 0.9.1
:released: March 1, 2017
.. change:: 417
:tags: bug, commands
:tickets: 417, 369
An adjustment to the bug fix for :ticket:`369` to accommodate for
env.py scripts that use an enclosing transaction distinct from the
one that the context provides, so that the check for "didn't commit
the transaction" doesn't trigger in this scenario... changelog::
:version: 0.9.0
:released: February 28, 2017
.. change:: 38
:tags: feature, autogenerate
:tickets: 38
The :paramref:`.EnvironmentContext.configure.target_metadata` parameter
may now be optionally specified as a sequence of :class:`.MetaData`
objects instead of a single :class:`.MetaData` object. The
autogenerate process will process the sequence of :class:`.MetaData`
objects in order.
.. change:: 369
:tags: bug, commands
:tickets: 369
A :class:`.CommandError` is now raised when a migration file opens
a database transaction and does not close/commit/rollback, when
the backend database or environment options also specify transactional_ddl
is False. When transactional_ddl is not in use, Alembic doesn't
close any transaction so a transaction opened by a migration file
will cause the following migrations to fail to apply.
.. change:: 413
:tags: bug, autogenerate, mysql
:tickets: 413
The ``autoincrement=True`` flag is now rendered within the
:meth:`.Operations.alter_column` operation if the source column indicates
that this flag should be set to True. The behavior is sensitive to
the SQLAlchemy version in place, as the "auto" default option is new
in SQLAlchemy 1.1. When the source column indicates autoincrement
as True or "auto", the flag will render as True if the original column
contextually indicates that it should have "autoincrement" keywords,
and when the source column explicitly sets it to False, this is also
rendered. The behavior is intended to preserve the AUTO_INCREMENT flag
on MySQL as the column is fully recreated on this backend. Note that this
flag does **not** support alteration of a column's "autoincrement" status,
as this is not portable across backends.
.. change:: 411
:tags: bug, postgresql
:tickets: 411
Fixed bug where Postgresql JSON/JSONB types rendered on SQLAlchemy
1.1 would render the "astext_type" argument which defaults to
the ``Text()`` type without the module prefix, similarly to the
issue with ARRAY fixed in :ticket:`85`.
.. change:: 85
:tags: bug, postgresql
:tickets: 85
Fixed bug where Postgresql ARRAY type would not render the import prefix
for the inner type; additionally, user-defined renderers take place
for the inner type as well as the outer type. Pull request courtesy
Paul Brackin.
.. change:: process_revision_directives_command
:tags: feature, autogenerate
Added a keyword argument ``process_revision_directives`` to the
:func:`.command.revision` API call. This function acts in the
same role as the environment-level
:paramref:`.EnvironmentContext.configure.process_revision_directives`,
and allows API use of the
command to drop in an ad-hoc directive process function. This
function can be used among other things to place a complete
:class:`.MigrationScript` structure in place.
.. change:: 412
:tags: feature, postgresql
:tickets: 412
Added support for Postgresql EXCLUDE constraints, including the
operation directive :meth:`.Operations.create_exclude_constraints`
as well as autogenerate render support for the ``ExcludeConstraint``
object as present in a ``Table``. Autogenerate detection for an EXCLUDE
constraint added or removed to/from an existing table is **not**
implemented as the SQLAlchemy Postgresql dialect does not yet support
reflection of EXCLUDE constraints.
Additionally, unknown constraint types now warn when
encountered within an autogenerate action rather than raise.
.. change:: fk_schema_compare
:tags: bug, operations
Fixed bug in :func:`.ops.create_foreign_key` where the internal table
representation would not be created properly if the foreign key referred
to a table in a different schema of the same name. Pull request
courtesy Konstantin Lebedev... changelog::
:version: 0.8.10
:released: January 17, 2017
.. change:: 406
:tags: bug, versioning
:tickets: 406
The alembic_version table, when initially created, now establishes a
primary key constraint on the "version_num" column, to suit database
engines that don't support tables without primary keys. This behavior
can be controlled using the parameter
:paramref:`.EnvironmentContext.configure.version_table_pk`. Note that
this change only applies to the initial creation of the alembic_version
table; it does not impact any existing alembic_version table already
present.
.. change:: 402
:tags: bug, batch
:tickets: 402
Fixed bug where doing ``batch_op.drop_constraint()`` against the
primary key constraint would fail to remove the "primary_key" flag
from the column, resulting in the constraint being recreated.
.. change:: update_uq_dedupe
:tags: bug, autogenerate, oracle
Adjusted the logic originally added for :ticket:`276` that detects MySQL
unique constraints which are actually unique indexes to be generalized
for any dialect that has this behavior, for SQLAlchemy version 1.0 and
greater. This is to allow for upcoming SQLAlchemy support for unique
constraint reflection for Oracle, which also has no dedicated concept of
"unique constraint" and instead establishes a unique index.
.. change:: 356
:tags: bug, versioning
:tickets: 356
Added a file ignore for Python files of the form ``.#<name>.py``,
which are generated by the Emacs editor. Pull request courtesy
Markus Mattes... changelog::
:version: 0.8.9
:released: November 28, 2016
.. change:: 393
:tags: bug, autogenerate
:tickets: 393
Adjustment to the "please adjust!" comment in the script.py.mako
template so that the generated comment starts with a single pound
sign, appeasing flake8.
.. change::
:tags: bug, batch
:tickets: 391
Batch mode will not use CAST() to copy data if ``type_`` is given, however
the basic type affinity matches that of the existing type. This to
avoid SQLite's CAST of TIMESTAMP which results in truncation of the
data, in those cases where the user needs to add redundant ``type_`` for
other reasons.
.. change::
:tags: bug, autogenerate
:tickets: 393
Continued pep8 improvements by adding appropriate whitespace in
the base template for generated migrations. Pull request courtesy
Markus Mattes.
.. change::
:tags: bug, revisioning
Added an additional check when reading in revision files to detect
if the same file is being read twice; this can occur if the same directory
or a symlink equivalent is present more than once in version_locations.
A warning is now emitted and the file is skipped. Pull request courtesy
Jiri Kuncar.
.. change::
:tags: bug, autogenerate
:tickets: 395
Fixed bug where usage of a custom TypeDecorator which returns a
per-dialect type via :meth:`.TypeDecorator.load_dialect_impl` that differs
significantly from the default "impl" for the type decorator would fail
to compare correctly during autogenerate.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 392
Fixed bug in Postgresql "functional index skip" behavior where a
functional index that ended in ASC/DESC wouldn't be detected as something
we can't compare in autogenerate, leading to duplicate definitions
in autogenerated files.
.. change::
:tags: bug, versioning
Fixed bug where the "base" specifier, as in "base:head", could not
be used explicitly when ``--sql`` mode was present... changelog::
:version: 0.8.8
:released: September 12, 2016
.. change::
:tags: autogenerate
The imports in the default script.py.mako are now at the top
so that flake8 editors don't complain by default. PR courtesy
Guilherme Mansur.
.. change::
:tags: feature, operations, postgresql
:tickets: 292
Added support for the USING clause to the ALTER COLUMN operation
for Postgresql. Support is via the
:paramref:`.op.alter_column.postgresql_using`
parameter. Pull request courtesy Frazer McLean.
.. change::
:tags: feature, autogenerate
Autogenerate with type comparison enabled will pick up on the timezone
setting changing between DateTime types. Pull request courtesy
David Szotten... changelog::
:version: 0.8.7
:released: July 26, 2016
.. change::
:tags: bug, versioning
:tickets: 336
Fixed bug where upgrading to the head of a branch which is already
present would fail, only if that head were also the dependency
of a different branch that is also upgraded, as the revision system
would see this as trying to go in the wrong direction. The check
here has been refined to distinguish between same-branch revisions
out of order vs. movement along sibling branches.
.. change::
:tags: bug, versioning
:tickets: 379
Adjusted the version traversal on downgrade
such that we can downgrade to a version that is a dependency for
a version in a different branch, *without* needing to remove that
dependent version as well. Previously, the target version would be
seen as a "merge point" for it's normal up-revision as well as the
dependency. This integrates with the changes for :ticket:`377`
and :ticket:`378` to improve treatment of branches with dependencies
overall.
.. change::
:tags: bug, versioning
:tickets: 377
Fixed bug where a downgrade to a version that is also a dependency
to a different branch would fail, as the system attempted to treat
this as an "unmerge" of a merge point, when in fact it doesn't have
the other side of the merge point available for update.
.. change::
:tags: bug, versioning
:tickets: 378
Fixed bug where the "alembic current" command wouldn't show a revision
as a current head if it were also a dependency of a version in a
different branch that's also applied. Extra logic is added to
extract "implied" versions of different branches from the top-level
versions listed in the alembic_version table.
.. change::
:tags: bug, versioning
Fixed bug where a repr() or str() of a Script object would fail
if the script had multiple dependencies.
.. change::
:tags: bug, autogenerate
Fixed bug in autogen where if the DB connection sends the default
schema as "None", this "None" would be removed from the list of
schemas to check if include_schemas were set. This could possibly
impact using include_schemas with SQLite.
.. change::
:tags: bug, batch
Small adjustment made to the batch handling for reflected CHECK
constraints to accommodate for SQLAlchemy 1.1 now reflecting these.
Batch mode still does not support CHECK constraints from the reflected
table as these can't be easily differentiated from the ones created
by types such as Boolean... changelog::
:version: 0.8.6
:released: April 14, 2016
.. change::
:tags: bug, commands
:tickets: 367
Errors which occur within the Mako render step are now intercepted
and raised as CommandErrors like other failure cases; the Mako
exception itself is written using template-line formatting to
a temporary file which is named in the exception message.
.. change::
:tags: bug, postgresql
:tickets: 365
Added a fix to Postgresql server default comparison which first checks
if the text of the default is identical to the original, before attempting
to actually run the default. This accommodates for default-generation
functions that generate a new value each time such as a uuid function.
.. change::
:tags: bug, batch
:tickets: 361
Fixed bug introduced by the fix for :ticket:`338` in version 0.8.4
where a server default could no longer be dropped in batch mode.
Pull request courtesy Martin Domke.
.. change::
:tags: bug, batch, mssql
Fixed bug where SQL Server arguments for drop_column() would not
be propagated when running under a batch block. Pull request
courtesy Michal Petrucha... changelog::
:version: 0.8.5
:released: March 9, 2016
.. change::
:tags: bug, autogenerate
:tickets: 335
Fixed bug where the columns rendered in a ``PrimaryKeyConstraint``
in autogenerate would inappropriately render the "key" of the
column, not the name. Pull request courtesy Jesse Dhillon.
.. change::
:tags: bug, batch
:tickets: 354
Repaired batch migration support for "schema" types which generate
constraints, in particular the ``Boolean`` datatype which generates
a CHECK constraint. Previously, an alter column operation with this
type would fail to correctly accommodate for the CHECK constraint
on change both from and to this type. In the former case the operation
would fail entirely, in the latter, the CHECK constraint would
not get generated. Both of these issues are repaired.
.. change::
:tags: bug, mysql
:tickets: 355
Changing a schema type such as ``Boolean`` to a non-schema type would
emit a drop constraint operation which emits ``NotImplementedError`` for
the MySQL dialect. This drop constraint operation is now skipped when
the constraint originates from a schema type... changelog::
:version: 0.8.4
:released: December 15, 2015
.. change::
:tags: feature, versioning
A major improvement to the hash id generation function, which for some
reason used an awkward arithmetic formula against uuid4() that produced
values that tended to start with the digits 1-4. Replaced with a
simple substring approach which provides an even distribution. Pull
request courtesy Antti Haapala.
.. change::
:tags: feature, autogenerate
Added an autogenerate renderer for the :class:`.ExecuteSQLOp` operation
object; only renders if given a plain SQL string, otherwise raises
NotImplementedError. Can be of help with custom autogenerate
sequences that includes straight SQL execution. Pull request courtesy
Jacob Magnusson.
.. change::
:tags: bug, batch
:tickets: 345
Batch mode generates a FOREIGN KEY constraint that is self-referential
using the ultimate table name, rather than ``_alembic_batch_temp``.
When the table is renamed from ``_alembic_batch_temp`` back to the
original name, the FK now points to the right name. This
will **not** work if referential integrity is being enforced (eg. SQLite
"PRAGMA FOREIGN_KEYS=ON") since the original table is dropped and
the new table then renamed to that name, however this is now consistent
with how foreign key constraints on **other** tables already operate
with batch mode; these don't support batch mode if referential integrity
is enabled in any case.
.. change::
:tags: bug, autogenerate
:tickets: 341
Added a type-level comparator that distinguishes :class:`.Integer`,
:class:`.BigInteger`, and :class:`.SmallInteger` types and
dialect-specific types; these all have "Integer" affinity so previously
all compared as the same.
.. change::
:tags: bug, batch
:tickets: 338
Fixed bug where the ``server_default`` parameter of ``alter_column()``
would not function correctly in batch mode.
.. change::
:tags: bug, autogenerate
:tickets: 337
Adjusted the rendering for index expressions such that a :class:`.Column`
object present in the source :class:`.Index` will not be rendered
as table-qualified; e.g. the column name will be rendered alone.
Table-qualified names here were failing on systems such as Postgresql... changelog::
:version: 0.8.3
:released: October 16, 2015
.. change::
:tags: bug, autogenerate
:tickets: 332
Fixed an 0.8 regression whereby the "imports" dictionary member of
the autogen context was removed; this collection is documented in the
"render custom type" documentation as a place to add new imports.
The member is now known as
:attr:`.AutogenContext.imports` and the documentation is repaired.
.. change::
:tags: bug, batch
:tickets: 333
Fixed bug in batch mode where a table that had pre-existing indexes
would create the same index on the new table with the same name,
which on SQLite produces a naming conflict as index names are in a
global namespace on that backend. Batch mode now defers the production
of both existing and new indexes until after the entire table transfer
operation is complete, which also means those indexes no longer take
effect during the INSERT from SELECT section as well; the indexes
are applied in a single step afterwards.
.. change::
:tags: bug, tests
Added "pytest-xdist" as a tox dependency, so that the -n flag
in the test command works if this is not already installed.
Pull request courtesy Julien Danjou.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 324
Fixed issue in PG server default comparison where model-side defaults
configured with Python unicode literals would leak the "u" character
from a ``repr()`` into the SQL used for comparison, creating an invalid
SQL expression, as the server-side comparison feature in PG currently
repurposes the autogenerate Python rendering feature to get a quoted
version of a plain string default... changelog::
:version: 0.8.2
:released: August 25, 2015
.. change::
:tags: bug, autogenerate
:tickets: 321
Added workaround in new foreign key option detection feature for
MySQL's consideration of the "RESTRICT" option being the default,
for which no value is reported from the database; the MySQL impl now
corrects for when the model reports RESTRICT but the database reports
nothing. A similar rule is in the default FK comparison to accommodate
for the default "NO ACTION" setting being present in the model but not
necessarily reported by the database, or vice versa... changelog::
:version: 0.8.1
:released: August 22, 2015
.. change::
:tags: feature, autogenerate
A custom :paramref:`.EnvironmentContext.configure.process_revision_directives`
hook can now generate op directives within the :class:`.UpgradeOps`
and :class:`.DowngradeOps` containers that will be generated as Python
code even when the ``--autogenerate`` flag is False; provided that
``revision_environment=True``, the full render operation will be run
even in "offline" mode.
.. change::
:tags: bug, autogenerate
Repaired the render operation for the :class:`.ops.AlterColumnOp` object
to succeed when the "existing_type" field was not present.
.. change::
:tags: bug, autogenerate
:tickets: 318
Fixed a regression 0.8 whereby the "multidb" environment template
failed to produce independent migration script segments for the
output template. This was due to the reorganization of the script
rendering system for 0.8. To accommodate this change, the
:class:`.MigrationScript` structure will in the case of multiple
calls to :meth:`.MigrationContext.run_migrations` produce lists
for the :attr:`.MigrationScript.upgrade_ops` and
:attr:`.MigrationScript.downgrade_ops` attributes; each :class:`.UpgradeOps`
and :class:`.DowngradeOps` instance keeps track of its own
``upgrade_token`` and ``downgrade_token``, and each are rendered
individually.
.. seealso::
:ref:`autogen_customizing_multiengine_revision` - additional detail
on the workings of the
:paramref:`.EnvironmentContext.configure.process_revision_directives`
parameter when multiple calls to :meth:`.MigrationContext.run_migrations`
are made.
.. change::
:tags: feature, autogenerate
:tickets: 317
Implemented support for autogenerate detection of changes in the
``ondelete``, ``onupdate``, ``initially`` and ``deferrable``
attributes of :class:`.ForeignKeyConstraint` objects on
SQLAlchemy backends that support these on reflection
(as of SQLAlchemy 1.0.8 currently Postgresql for all four,
MySQL for ``ondelete`` and ``onupdate`` only). A constraint object
that modifies these values will be reported as a "diff" and come out
as a drop/create of the constraint with the modified values.
The fields are ignored for backends which don't reflect these
attributes (as of SQLA 1.0.8 this includes SQLite, Oracle, SQL Server,
others)... changelog::
:version: 0.8.0
:released: August 12, 2015
.. change::
:tags: bug, batch
:tickets: 315
Fixed bug in batch mode where the ``batch_op.create_foreign_key()``
directive would be incorrectly rendered with the source table and
schema names in the argument list.
.. change::
:tags: feature, commands
Added new command ``alembic edit``. This command takes the same
arguments as ``alembic show``, however runs the target script
file within $EDITOR. Makes use of the ``python-editor`` library
in order to facilitate the handling of $EDITOR with reasonable
default behaviors across platforms. Pull request courtesy
Michel Albert.
.. change::
:tags: feature, commands
:tickets: 311
Added new multiple-capable argument ``--depends-on`` to the
``alembic revision`` command, allowing ``depends_on`` to be
established at the command line level rather than having to edit
the file after the fact. ``depends_on`` identifiers may also be
specified as branch names at the command line or directly within
the migration file. The values may be specified as partial
revision numbers from the command line which will be resolved to
full revision numbers in the output file.
.. change::
:tags: change, operations
A range of positional argument names have been changed to be
clearer and more consistent across methods within the
:class:`.Operations` namespace. The most prevalent form of name change
is that the descriptive names ``constraint_name`` and ``table_name``
are now used where previously the name ``name`` would be used.
This is in support of the newly modularized and extensible system of
operation objects in :mod:`alembic.operations.ops`.
An argument translation layer is in place
across the ``alembic.op`` namespace that will ensure that named
argument calling styles that use the old names will continue to
function by transparently translating to the new names,
also emitting a warning. This, along with the fact that these
arguments are positional in any case and aren't normally
passed with an explicit name, should ensure that the
overwhelming majority of applications should be unaffected by this
change. The *only* applications that are impacted are those that:
1. use the :class:`.Operations` object directly in some way, rather
than calling upon the ``alembic.op`` namespace, and
2. invoke the methods on :class:`.Operations` using named keyword
arguments for positional arguments like ``table_name``,
``constraint_name``, etc., which commonly were named ``name``
as of 0.7.6.
3. any application that is using named keyword arguments in place
of positional argument for the recently added
:class:`.BatchOperations` object may also be affected.
The naming changes are documented as "versionchanged" for 0.8.0:
* :meth:`.BatchOperations.create_check_constraint`
* :meth:`.BatchOperations.create_foreign_key`
* :meth:`.BatchOperations.create_index`
* :meth:`.BatchOperations.create_unique_constraint`
* :meth:`.BatchOperations.drop_constraint`
* :meth:`.BatchOperations.drop_index`
* :meth:`.Operations.create_check_constraint`
* :meth:`.Operations.create_foreign_key`
* :meth:`.Operations.create_primary_key`
* :meth:`.Operations.create_index`
* :meth:`.Operations.create_table`
* :meth:`.Operations.create_unique_constraint`
* :meth:`.Operations.drop_constraint`
* :meth:`.Operations.drop_index`
* :meth:`.Operations.drop_table`
.. change::
:tags: feature, tests
The default test runner via "python setup.py test" is now py.test.
nose still works via run_tests.py.
.. change::
:tags: feature, operations
:tickets: 302
The internal system for Alembic operations has been reworked to now
build upon an extensible system of operation objects. New operations
can be added to the ``op.`` namespace, including that they are
available in custom autogenerate schemes.
.. seealso::
:ref:`operation_plugins`
.. change::
:tags: feature, autogenerate
:tickets: 301, 306
The internal system for autogenerate been reworked to build upon
the extensible system of operation objects present in
:ticket:`302`. As part of this change, autogenerate now produces
a full object graph representing a list of migration scripts to
be written as well as operation objects that will render all the
Python code within them; a new hook
:paramref:`.EnvironmentContext.configure.process_revision_directives`
allows end-user code to fully customize what autogenerate will do,
including not just full manipulation of the Python steps to take
but also what file or files will be written and where. Additionally,
autogenerate is now extensible as far as database objects compared
and rendered into scripts; any new operation directive can also be
registered into a series of hooks that allow custom database/model
comparison functions to run as well as to render new operation
directives into autogenerate scripts.
.. seealso::
:ref:`alembic.autogenerate.toplevel`
.. change::
:tags: bug, versioning
:tickets: 314
Fixed bug where in the erroneous case that alembic_version contains
duplicate revisions, some commands would fail to process the
version history correctly and end up with a KeyError. The fix
allows the versioning logic to proceed, however a clear error is
emitted later when attempting to update the alembic_version table... changelog::
:version: 0.7.7
:released: July 22, 2015
.. change::
:tags: bug, versioning
:tickets: 310
Fixed critical issue where a complex series of branches/merges would
bog down the iteration algorithm working over redundant nodes for
millions of cycles. An internal adjustment has been
made so that duplicate nodes are skipped within this iteration.
.. change::
:tags: feature, batch
:tickets: 305
Implemented support for :meth:`.BatchOperations.create_primary_key`
and :meth:`.BatchOperations.create_check_constraint`. Additionally,
table keyword arguments are copied from the original reflected table,
such as the "mysql_engine" keyword argument.
.. change::
:tags: bug, environment
:tickets: 300
The :meth:`.MigrationContext.stamp` method, added as part of the
versioning refactor in 0.7 as a more granular version of
:func:`.command.stamp`, now includes the "create the alembic_version
table if not present" step in the same way as the command version,
which was previously omitted.
.. change::
:tags: bug, autogenerate
:tickets: 298
Fixed bug where foreign key options including "onupdate",
"ondelete" would not render within the ``op.create_foreign_key()``
directive, even though they render within a full
``ForeignKeyConstraint`` directive.
.. change::
:tags: bug, tests
Repaired warnings that occur when running unit tests against
SQLAlchemy 1.0.5 or greater involving the "legacy_schema_aliasing"
flag... changelog::
:version: 0.7.6
:released: May 5, 2015
.. change::
:tags: feature, versioning
:tickets: 297
Fixed bug where the case of multiple mergepoints that all
have the identical set of ancestor revisions would fail to be
upgradable, producing an assertion failure. Merge points were
previously assumed to always require at least an UPDATE in
alembic_revision from one of the previous revs to the new one,
however in this case, if one of the mergepoints has already
been reached, the remaining mergepoints have no row to UPDATE therefore
they must do an INSERT of their target version.
.. change::
:tags: feature, autogenerate
:tickets: 296
Added support for type comparison functions to be not just per
environment, but also present on the custom types themselves, by
supplying a method ``compare_against_backend``.
Added a new documentation section :ref:`compare_types` describing
type comparison fully.
.. change::
:tags: feature, operations
:tickets: 255
Added a new option
:paramref:`.EnvironmentContext.configure.literal_binds`, which
will pass the ``literal_binds`` flag into the compilation of SQL
constructs when using "offline" mode. This has the effect that
SQL objects like inserts, updates, deletes as well as textual
statements sent using ``text()`` will be compiled such that the dialect
will attempt to render literal values "inline" automatically.
Only a subset of types is typically supported; the
:meth:`.Operations.inline_literal` construct remains as the construct
used to force a specific literal representation of a value.
The :paramref:`.EnvironmentContext.configure.literal_binds` flag
is added to the "offline" section of the ``env.py`` files generated
in new environments.
.. change::
:tags: bug, batch
:tickets: 289
Fully implemented the
:paramref:`~.Operations.batch_alter_table.copy_from` parameter for
batch mode, which previously was not functioning. This allows
"batch mode" to be usable in conjunction with ``--sql``.
.. change::
:tags: bug, batch
:tickets: 287
Repaired support for the :meth:`.BatchOperations.create_index`
directive, which was mis-named internally such that the operation
within a batch context could not proceed. The create index
operation will proceed as part of a larger "batch table recreate"
operation only if
:paramref:`~.Operations.batch_alter_table.recreate` is set to
"always", or if the batch operation includes other instructions that
require a table recreate... changelog::
:version: 0.7.5
:released: March 19, 2015
.. change::
:tags: bug, autogenerate
:tickets: 266
The ``--autogenerate`` option is not valid when used in conjunction
with "offline" mode, e.g. ``--sql``. This now raises a ``CommandError``,
rather than failing more deeply later on. Pull request courtesy
Johannes Erdfelt.
.. change::
:tags: bug, operations, mssql
:tickets: 284
Fixed bug where the mssql DROP COLUMN directive failed to include
modifiers such as "schema" when emitting the DDL.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 282
Postgresql "functional" indexes are necessarily skipped from the
autogenerate process, as the SQLAlchemy backend currently does not
support reflection of these structures. A warning is emitted
both from the SQLAlchemy backend as well as from the Alembic
backend for Postgresql when such an index is detected.
.. change::
:tags: bug, autogenerate, mysql
:tickets: 276
Fixed bug where MySQL backend would report dropped unique indexes
and/or constraints as both at the same time. This is because
MySQL doesn't actually have a "unique constraint" construct that
reports differently than a "unique index", so it is present in both
lists. The net effect though is that the MySQL backend will report
a dropped unique index/constraint as an index in cases where the object
was first created as a unique constraint, if no other information
is available to make the decision. This differs from other backends
like Postgresql which can report on unique constraints and
unique indexes separately.
.. change::
:tags: bug, commands
:tickets: 269
Fixed bug where using a partial revision identifier as the
"starting revision" in ``--sql`` mode in a downgrade operation
would fail to resolve properly.
As a side effect of this change, the
:meth:`.EnvironmentContext.get_starting_revision_argument`
method will return the "starting" revision in its originally-
given "partial" form in all cases, whereas previously when
running within the :meth:`.command.stamp` command, it would have
been resolved to a full number before passing it to the
:class:`.EnvironmentContext`. The resolution of this value to
a real revision number has basically been moved to a more fundamental
level within the offline migration process.
.. change::
:tags: feature, commands
Added a new feature :attr:`.Config.attributes`, to help with the use
case of sharing state such as engines and connections on the outside
with a series of Alembic API calls; also added a new cookbook section
to describe this simple but pretty important use case.
.. seealso::
:ref:`connection_sharing`
.. change::
:tags: feature, environment
The format of the default ``env.py`` script has been refined a bit;
it now uses context managers not only for the scope of the transaction,
but also for connectivity from the starting engine. The engine is also
now called a "connectable" in support of the use case of an external
connection being passed in.
.. change::
:tags: feature, versioning
:tickets: 267
Added support for "alembic stamp" to work when given "heads" as an
argument, when multiple heads are present... changelog::
:version: 0.7.4
:released: January 12, 2015
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 241
Repaired issue where a server default specified without ``text()``
that represented a numeric or floating point (e.g. with decimal places)
value would fail in the Postgresql-specific check for "compare server
default"; as PG accepts the value with quotes in the table specification,
it's still valid. Pull request courtesy Dimitris Theodorou.
.. change::
:tags: bug, autogenerate
:tickets: 259
The rendering of a :class:`~sqlalchemy.schema.ForeignKeyConstraint`
will now ensure that the names of the source and target columns are
the database-side name of each column, and not the value of the
``.key`` attribute as may be set only on the Python side.
This is because Alembic generates the DDL for constraints
as standalone objects without the need to actually refer to an in-Python
:class:`~sqlalchemy.schema.Table` object, so there's no step that
would resolve these Python-only key names to database column names.
.. change::
:tags: bug, autogenerate
:tickets: 260
Fixed bug in foreign key autogenerate where if the in-Python table
used custom column keys (e.g. using the ``key='foo'`` kwarg to
``Column``), the comparison of existing foreign keys to those specified
in the metadata would fail, as the reflected table would not have
these keys available which to match up. Foreign key comparison for
autogenerate now ensures it's looking at the database-side names
of the columns in all cases; this matches the same functionality
within unique constraints and indexes.
.. change::
:tags: bug, autogenerate
:tickets: 261
Fixed issue in autogenerate type rendering where types that belong
to modules that have the name "sqlalchemy" in them would be mistaken
as being part of the ``sqlalchemy.`` namespace. Pull req courtesy
Bartosz Burclaf... changelog::
:version: 0.7.3
:released: December 30, 2014
.. change::
:tags: bug, versioning
:tickets: 258
Fixed regression in new versioning system where upgrade / history
operation would fail on AttributeError if no version files were
present at all... changelog::
:version: 0.7.2
:released: December 18, 2014
.. change::
:tags: bug, sqlite, autogenerate
Adjusted the SQLite backend regarding autogen of unique constraints
to work fully with the current SQLAlchemy 1.0, which now will report
on UNIQUE constraints that have no name.
.. change::
:tags: bug, batch
:tickets: 254
Fixed bug in batch where if the target table contained multiple
foreign keys to the same target table, the batch mechanics would
fail with a "table already exists" error. Thanks for the help
on this from Lucas Kahlert.
.. change::
:tags: bug, mysql
:tickets: 251
Fixed an issue where the MySQL routine to skip foreign-key-implicit
indexes would also catch unnamed unique indexes, as they would be
named after the column and look like the FK indexes. Pull request
courtesy Johannes Erdfelt.
.. change::
:tags: bug, mssql, oracle
:tickets: 253
Repaired a regression in both the MSSQL and Oracle dialects whereby
the overridden ``_exec()`` method failed to return a value, as is
needed now in the 0.7 series... changelog::
:version: 0.7.1
:released: December 3, 2014
.. change::
:tags: bug, batch
The ``render_as_batch`` flag was inadvertently hardcoded to ``True``,
so all autogenerates were spitting out batch mode...this has been
fixed so that batch mode again is only when selected in env.py.
.. change::
:tags: feature, autogenerate
:tickets: 178
Support for autogenerate of FOREIGN KEY constraints has been added.
These are delivered within the autogenerate process in the same
manner as UNIQUE constraints, including ``include_object`` support.
Big thanks to Ann Kamyshnikova for doing the heavy lifting here.
.. change::
:tags: feature, batch
Added :paramref:`~.Operations.batch_alter_table.naming_convention`
argument to :meth:`.Operations.batch_alter_table`, as this is necessary
in order to drop foreign key constraints; these are often unnamed
on the target database, and in the case that they are named, SQLAlchemy
is as of the 0.9 series not including these names yet.
.. seealso::
:ref:`dropping_sqlite_foreign_keys`
.. change::
:tags: bug, batch
Fixed bug where the "source_schema" argument was not correctly passed
when calling :meth:`.BatchOperations.create_foreign_key`. Pull
request courtesy Malte Marquarding.
.. change::
:tags: bug, batch
:tickets: 249
Repaired the inspection, copying and rendering of CHECK constraints
and so-called "schema" types such as Boolean, Enum within the batch
copy system; the CHECK constraint will not be "doubled" when the table is
copied, and additionally the inspection of the CHECK constraint for
its member columns will no longer fail with an attribute error.
.. change::
:tags: feature, batch
Added two new arguments
:paramref:`.Operations.batch_alter_table.reflect_args`
and :paramref:`.Operations.batch_alter_table.reflect_kwargs`, so that
arguments may be passed directly to suit the
:class:`~.sqlalchemy.schema.Table`
object that will be reflected.
.. seealso::
:ref:`batch_controlling_table_reflection`.. changelog::
:version: 0.7.0
:released: November 24, 2014
.. change::
:tags: feature, versioning
:tickets: 167
The "multiple heads / branches" feature has now landed. This is
by far the most significant change Alembic has seen since its inception;
while the workflow of most commands hasn't changed, and the format
of version files and the ``alembic_version`` table are unchanged as well,
a new suite of features opens up in the case where multiple version
files refer to the same parent, or to the "base". Merging of
branches, operating across distinct named heads, and multiple
independent bases are now all supported. The feature incurs radical
changes to the internals of versioning and traversal, and should be
treated as "beta mode" for the next several subsequent releases
within 0.7.
.. seealso::
:ref:`branches`
.. change::
:tags: feature, versioning
:tickets: 124
In conjunction with support for multiple independent bases, the
specific version directories are now also configurable to include
multiple, user-defined directories. When multiple directories exist,
the creation of a revision file with no down revision requires
that the starting directory is indicated; the creation of subsequent
revisions along that lineage will then automatically use that
directory for new files.
.. seealso::
:ref:`multiple_version_directories`
.. change::
:tags: feature, operations, sqlite
:tickets: 21
Added "move and copy" workflow, where a table to be altered is copied to
a new one with the new structure and the old one dropped, is now
implemented for SQLite as well as all database backends in general
using the new :meth:`.Operations.batch_alter_table` system. This
directive provides a table-specific operations context which gathers
column- and constraint-level mutations specific to that table, and
at the end of the context creates a new table combining the structure
of the old one with the given changes, copies data from old table to new,
and finally drops the old table,
renaming the new one to the existing name. This is required for
fully featured SQLite migrations, as SQLite has very little support for the
traditional ALTER directive. The batch directive
is intended to produce code that is still compatible with other databases,
in that the "move and copy" process only occurs for SQLite by default,
while still providing some level of sanity to SQLite's
requirement by allowing multiple table mutation operations to
proceed within one "move and copy" as well as providing explicit
control over when this operation actually occurs. The "move and copy"
feature may be optionally applied to other backends as well, however
dealing with referential integrity constraints from other tables must
still be handled explicitly.
.. seealso::
:ref:`batch_migrations`
.. change::
:tags: feature, commands
Relative revision identifiers as used with ``alembic upgrade``,
``alembic downgrade`` and ``alembic history`` can be combined with
specific revisions as well, e.g. ``alembic upgrade ae10+3``, to produce
a migration target relative to the given exact version.
.. change::
:tags: bug, commands
:tickets: 248
The ``alembic revision`` command accepts the ``--sql`` option to
suit some very obscure use case where the ``revision_environment``
flag is set up, so that ``env.py`` is run when ``alembic revision``
is run even though autogenerate isn't specified. As this flag is
otherwise confusing, error messages are now raised if
``alembic revision`` is invoked with both ``--sql`` and
``--autogenerate`` or with ``--sql`` without
``revision_environment`` being set.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 247
Added a rule for Postgresql to not render a "drop unique" and "drop index"
given the same name; for now it is assumed that the "index" is the
implicit one PostgreSQL generates. Future integration with
new SQLAlchemy 1.0 features will improve this to be more
resilient.
.. change::
:tags: bug, autogenerate
:tickets: 247
A change in the ordering when columns and constraints are dropped;
autogenerate will now place the "drop constraint" calls *before*
the "drop column" calls, so that columns involved in those constraints
still exist when the constraint is dropped.
.. change::
:tags: feature, commands
New commands added: ``alembic show``, ``alembic heads`` and
``alembic merge``. Also, a new option ``--verbose`` has been
added to several informational commands, such as ``alembic history``,
``alembic current``, ``alembic branches``, and ``alembic heads``.
``alembic revision`` also contains several new options used
within the new branch management system. The output of commands has
been altered in many cases to support new fields and attributes;
the ``history`` command in particular now returns it's "verbose" output
only if ``--verbose`` is sent; without this flag it reverts to it's
older behavior of short line items (which was never changed in the docs).
.. change::
:tags: changed, commands
The ``--head_only`` option to the ``alembic current`` command is
deprecated; the ``current`` command now lists just the version numbers
alone by default; use ``--verbose`` to get at additional output.
.. change::
:tags: feature, config
Added new argument :paramref:`.Config.config_args`, allows a dictionary
of replacement variables to be passed which will serve as substitution
values when an API-produced :class:`.Config` consumes the ``.ini``
file. Pull request courtesy Noufal Ibrahim.
.. change::
:tags: bug, oracle
:tickets: 245
The Oracle dialect sets "transactional DDL" to False by default,
as Oracle does not support transactional DDL.
.. change::
:tags: bug, autogenerate
:tickets: 243
Fixed a variety of issues surrounding rendering of Python code that
contains unicode literals. The first is that the "quoted_name" construct
that SQLAlchemy uses to represent table and column names as well
as schema names does not ``repr()`` correctly on Py2K when the value
contains unicode characters; therefore an explicit stringification is
added to these. Additionally, SQL expressions such as server defaults
were not being generated in a unicode-safe fashion leading to decode
errors if server defaults contained non-ascii characters.
.. change::
:tags: bug, operations
:tickets: 174
The :meth:`.Operations.add_column` directive will now additionally emit
the appropriate ``CREATE INDEX`` statement if the
:class:`~sqlalchemy.schema.Column` object specifies ``index=True``.
Pull request courtesy David Szotten.
.. change::
:tags: feature, operations
:tickets: 205
The :class:`~sqlalchemy.schema.Table` object is now returned when
the :meth:`.Operations.create_table` method is used. This ``Table``
is suitable for use in subsequent SQL operations, in particular
the :meth:`.Operations.bulk_insert` operation.
.. change::
:tags: feature, autogenerate
:tickets: 203
Indexes and unique constraints are now included in the
:paramref:`.EnvironmentContext.configure.include_object` hook.
Indexes are sent with type ``"index"`` and unique constraints with
type ``"unique_constraint"``.
.. change::
:tags: bug, autogenerate
:tickets: 219
Bound parameters are now resolved as "literal" values within the
SQL expression inside of a CheckConstraint(), when rendering the SQL
as a text string; supported for SQLAlchemy 0.8.0 and forward.
.. change::
:tags: bug, autogenerate
:tickets: 199
Added a workaround for SQLAlchemy issue #3023 (fixed in 0.9.5) where
a column that's part of an explicit PrimaryKeyConstraint would not
have its "nullable" flag set to False, thus producing a false
autogenerate. Also added a related correction to MySQL which will
correct for MySQL's implicit server default of '0' when a NULL integer
column is turned into a primary key column.
.. change::
:tags: bug, autogenerate, mysql
:tickets: 240
Repaired issue related to the fix for #208 and others; a composite
foreign key reported by MySQL would cause a KeyError as Alembic
attempted to remove MySQL's implicitly generated indexes from the
autogenerate list.
.. change::
:tags: bug, autogenerate
:tickets: 28
If the "alembic_version" table is present in the target metadata,
autogenerate will skip this also. Pull request courtesy
Dj Gilcrease.
.. change::
:tags: bug, autogenerate
:tickets: 77
The :paramref:`.EnvironmentContext.configure.version_table`
and :paramref:`.EnvironmentContext.configure.version_table_schema`
arguments are now honored during the autogenerate process, such that
these names will be used as the "skip" names on both the database
reflection and target metadata sides.
.. change::
:tags: changed, autogenerate
:tickets: 229
The default value of the
:paramref:`.EnvironmentContext.configure.user_module_prefix`
parameter is **no longer the same as the SQLAlchemy prefix**.
When omitted, user-defined types will now use the ``__module__``
attribute of the type class itself when rendering in an
autogenerated module.
.. change::
:tags: bug, templates
:tickets: 234
Revision files are now written out using the ``'wb'`` modifier to
``open()``, since Mako reads the templates with ``'rb'``, thus preventing
CRs from being doubled up as has been observed on windows. The encoding
of the output now defaults to 'utf-8', which can be configured using
a newly added config file parameter ``output_encoding``.
.. change::
:tags: bug, operations
:tickets: 230
Added support for use of the :class:`~sqlalchemy.sql.elements.quoted_name`
construct when using the ``schema`` argument within operations. This
allows a name containing a dot to be fully quoted, as well as to
provide configurable quoting on a per-name basis.
.. change::
:tags: bug, autogenerate, postgresql
:tickets: 73
Added a routine by which the Postgresql Alembic dialect inspects
the server default of INTEGER/BIGINT columns as they are reflected
during autogenerate for the pattern ``nextval(<name>...)`` containing
a potential sequence name, then queries ``pg_catalog`` to see if this
sequence is "owned" by the column being reflected; if so, it assumes
this is a SERIAL or BIGSERIAL column and the server default is
omitted from the column reflection as well as any kind of
server_default comparison or rendering, along with an INFO message
in the logs indicating this has taken place. This allows SERIAL/BIGSERIAL
columns to keep the SEQUENCE from being unnecessarily present within
the autogenerate operation.
.. change::
:tags: bug, autogenerate
:tickets: 197, 64, 196
The system by which autogenerate renders expressions within
a :class:`~sqlalchemy.schema.Index`, the ``server_default``
of :class:`~sqlalchemy.schema.Column`, and the
``existing_server_default`` of
:meth:`.Operations.alter_column` has been overhauled to anticipate
arbitrary SQLAlchemy SQL constructs, such as ``func.somefunction()``,
``cast()``, ``desc()``, and others. The system does not, as might
be preferred, render the full-blown Python expression as originally
created within the application's source code, as this would be exceedingly
complex and difficult. Instead, it renders the SQL expression against
the target backend that's subject to the autogenerate, and then
renders that SQL inside of a :func:`~sqlalchemy.sql.expression.text`
construct as a literal SQL string. This approach still has the
downside that the rendered SQL construct may not be backend-agnostic
in all cases, so there is still a need for manual intervention in that
small number of cases, but overall the majority of cases should work
correctly now. Big thanks to Carlos Rivera for pull requests and
support on this.
.. change::
:tags: feature
SQLAlchemy's testing infrastructure is now used to run tests.
This system supports both nose and pytest and opens the way
for Alembic testing to support any number of backends, parallel
testing, and 3rd party dialect testing.
.. change::
:tags: changed, compatibility
Minimum SQLAlchemy version is now 0.7.6, however at least
0.8.4 is strongly recommended. The overhaul of the test suite
allows for fully passing tests on all SQLAlchemy versions
from 0.7.6 on forward.
.. change::
:tags: bug, operations
The "match" keyword is not sent to :class:`.ForeignKeyConstraint`
by :meth:`.Operations.create_foreign_key` when SQLAlchemy 0.7 is in use;
this keyword was added to SQLAlchemy as of 0.8.0... changelog::
:version: 0.6.7
:released: September 9, 2014
.. change::
:tags: bug, mssql
Fixed bug in MSSQL dialect where "rename table" wasn't using
``sp_rename()`` as is required on SQL Server. Pull request courtesy
ลukasz Boลdys.
.. change::
:tags: feature
:tickets: 222
Added support for functional indexes when using the
:meth:`.Operations.create_index` directive. Within the list of columns,
the SQLAlchemy ``text()`` construct can be sent, embedding a literal
SQL expression; the :meth:`.Operations.create_index` will perform some hackery
behind the scenes to get the :class:`.Index` construct to cooperate.
This works around some current limitations in :class:`.Index`
which should be resolved on the SQLAlchemy side at some point... changelog::
:version: 0.6.6
:released: August 7, 2014
.. change::
:tags: bug
:tickets: 95
A file named ``__init__.py`` in the ``versions/`` directory is now
ignored by Alembic when the collection of version files is retrieved.
Pull request courtesy Michael Floering.
.. change::
:tags: bug
Fixed Py3K bug where an attempt would be made to sort None against
string values when autogenerate would detect tables across multiple
schemas, including the default schema. Pull request courtesy
paradoxxxzero.
.. change::
:tags: bug
Autogenerate render will render the arguments within a Table construct
using ``*[...]`` when the number of columns/elements is greater than
255. Pull request courtesy Ryan P. Kelly.
.. change::
:tags: bug
Fixed bug where foreign key constraints would fail to render in
autogenerate when a schema name was present. Pull request courtesy
Andreas Zeidler.
.. change::
:tags: bug
:tickets: 212
Some deep-in-the-weeds fixes to try to get "server default" comparison
working better across platforms and expressions, in particular on
the Postgresql backend, mostly dealing with quoting/not quoting of various
expressions at the appropriate time and on a per-backend basis.
Repaired and tested support for such defaults as Postgresql interval
and array defaults.
.. change::
:tags: enhancement
:tickets: 209
When a run of Alembic command line fails due to ``CommandError``,
the output now prefixes the string with ``"FAILED:"``, and the error
is also written to the log output using ``log.error()``.
.. change::
:tags: bug
:tickets: 208
Liberalized even more the check for MySQL indexes that shouldn't be
counted in autogenerate as "drops"; this time it's been reported
that an implicitly created index might be named the same as a composite
foreign key constraint, and not the actual columns, so we now skip those
when detected as well.
.. change::
:tags: feature
Added a new accessor :attr:`.MigrationContext.config`, when used
in conjunction with a :class:`.EnvironmentContext` and
:class:`.Config`, this config will be returned. Patch
courtesy Marc Abramowitz... changelog::
:version: 0.6.5
:released: May 3, 2014
.. change::
:tags: bug, autogenerate, mysql
:tickets: 202
This releases' "autogenerate index detection" bug, when a MySQL table
includes an Index with the same name as a column, autogenerate reported
it as an "add" even though its not; this is because we ignore reflected
indexes of this nature due to MySQL creating them implicitly. Indexes
that are named the same as a column are now ignored on
MySQL if we see that the backend is reporting that it already exists;
this indicates that we can still detect additions of these indexes
but not drops, as we cannot distinguish a backend index same-named
as the column as one that is user generated or mysql-generated.
.. change::
:tags: feature, environment
:tickets: 201
Added new feature :paramref:`.EnvironmentContext.configure.transaction_per_migration`,
which when True causes the BEGIN/COMMIT pair to incur for each migration
individually, rather than for the whole series of migrations. This is
to assist with some database directives that need to be within individual
transactions, without the need to disable transactional DDL entirely.
.. change::
:tags: bug, autogenerate
:tickets: 200
Fixed bug where the ``include_object()`` filter would not receive
the original :class:`.Column` object when evaluating a database-only
column to be dropped; the object would not include the parent
:class:`.Table` nor other aspects of the column that are important
for generating the "downgrade" case where the column is recreated.
.. change::
:tags: bug, environment
:tickets: 195
Fixed bug where :meth:`.EnvironmentContext.get_x_argument`
would fail if the :class:`.Config` in use didn't actually
originate from a command line call.
.. change::
:tags: bug, autogenerate
:tickets: 194
Fixed another bug regarding naming conventions, continuing
from :ticket:`183`, where add_index()
drop_index() directives would not correctly render the ``f()``
construct when the index contained a convention-driven name... changelog::
:version: 0.6.4
:released: March 28, 2014
.. change::
:tags: bug, mssql
:tickets: 186
Added quoting to the table name when the special EXEC is run to
drop any existing server defaults or constraints when the
:paramref:`.Operations.drop_column.mssql_drop_check` or
:paramref:`.Operations.drop_column.mssql_drop_default`
arguments are used.
.. change::
:tags: bug, mysql
:tickets: 103
Added/fixed support for MySQL "SET DEFAULT" / "DROP DEFAULT" phrases,
which will now be rendered if only the server default is changing
or being dropped (e.g. specify None to alter_column() to indicate
"DROP DEFAULT"). Also added support for rendering MODIFY rather than
CHANGE when the column name isn't changing.
.. change::
:tags: bug
:tickets: 190
Added support for the ``initially``, ``match`` keyword arguments
as well as dialect-specific keyword arguments to
:meth:`.Operations.create_foreign_key`.
:tags: feature
:tickets: 163
Altered the support for "sourceless" migration files (e.g. only
.pyc or .pyo present) so that the flag "sourceless=true" needs to
be in alembic.ini for this behavior to take effect.
.. change::
:tags: bug, mssql
:tickets: 185
The feature that keeps on giving, index/unique constraint autogenerate
detection, has even more fixes, this time to accommodate database dialects
that both don't yet report on unique constraints, but the backend
does report unique constraints as indexes. The logic
Alembic uses to distinguish between "this is an index!" vs.
"this is a unique constraint that is also reported as an index!" has now
been further enhanced to not produce unwanted migrations when the dialect
is observed to not yet implement get_unique_constraints() (e.g. mssql).
Note that such a backend will no longer report index drops for unique
indexes, as these cannot be distinguished from an unreported unique
index.
.. change::
:tags: bug
:tickets: 183
Extensive changes have been made to more fully support SQLAlchemy's new
naming conventions feature. Note that while SQLAlchemy has added this
feature as of 0.9.2, some additional fixes in 0.9.4 are needed to
resolve some of the issues:
1. The :class:`.Operations` object now takes into account the naming
conventions that are present on the :class:`.MetaData` object that's
associated using :paramref:`~.EnvironmentContext.configure.target_metadata`.
When :class:`.Operations` renders a constraint directive like
``ADD CONSTRAINT``, it now will make use of this naming convention
when it produces its own temporary :class:`.MetaData` object.
2. Note however that the autogenerate feature in most cases generates
constraints like foreign keys and unique constraints with the
final names intact; the only exception are the constraints implicit
with a schema-type like Boolean or Enum. In most of these cases,
the naming convention feature will not take effect for these constraints
and will instead use the given name as is, with one exception....
3. Naming conventions which use the ``"%(constraint_name)s"`` token, that
is, produce a new name that uses the original name as a component,
will still be pulled into the naming convention converter and be
converted. The problem arises when autogenerate renders a constraint
with it's already-generated name present in the migration file's source
code, the name will be doubled up at render time due to the combination
of #1 and #2. So to work around this, autogenerate now renders these
already-tokenized names using the new :meth:`.Operations.f` component.
This component is only generated if **SQLAlchemy 0.9.4** or greater
is in use.
Therefore it is highly recommended that an upgrade to Alembic 0.6.4
be accompanied by an upgrade of SQLAlchemy 0.9.4, if the new naming
conventions feature is used.
.. seealso::
:ref:`autogen_naming_conventions`
.. change::
:tags: bug
:tickets: 160
Suppressed IOErrors which can raise when program output pipe
is closed under a program like ``head``; however this only
works on Python 2. On Python 3, there is not yet a known way to
suppress the BrokenPipeError warnings without prematurely terminating
the program via signals.
.. change::
:tags: bug
:tickets: 179
Fixed bug where :meth:`.Operations.bulk_insert` would not function
properly when :meth:`.Operations.inline_literal` values were used,
either in --sql or non-sql mode. The values will now render
directly in --sql mode. For compatibility with "online" mode,
a new flag :paramref:`~.Operations.bulk_insert.multiinsert`
can be set to False which will cause each parameter set to be
compiled and executed with individual INSERT statements.
.. change::
:tags: bug, py3k
:tickets: 175
Fixed a failure of the system that allows "legacy keyword arguments"
to be understood, which arose as of a change in Python 3.4 regarding
decorators. A workaround is applied that allows the code to work
across Python 3 versions.
.. change::
:tags: feature
The :func:`.command.revision` command now returns the :class:`.Script`
object corresponding to the newly generated revision. From this
structure, one can get the revision id, the module documentation,
and everything else, for use in scripts that call upon this command.
Pull request courtesy Robbie Coomber... changelog::
:version: 0.6.3
:released: February 2, 2014
.. change::
:tags: bug
:tickets: 172
Added a workaround for when we call ``fcntl.ioctl()`` to get at
``TERMWIDTH``; if the function returns zero, as is reported to occur
in some pseudo-ttys, the message wrapping system is disabled in the
same way as if ``ioctl()`` failed.
.. change::
:tags: feature
:tickets: 171
Added new argument
:paramref:`.EnvironmentContext.configure.user_module_prefix`.
This prefix is applied when autogenerate renders a user-defined type,
which here is defined as any type that is from a module outside of the
``sqlalchemy.`` hierarchy. This prefix defaults to ``None``, in
which case the :paramref:`.EnvironmentContext.configure.sqlalchemy_module_prefix`
is used, thus preserving the current behavior.
.. change::
:tags: bug
:tickets: 170
Added support for autogenerate covering the use case where :class:`.Table`
objects specified in the metadata have an explicit ``schema`` attribute
whose name matches that of the connection's default schema
(e.g. "public" for Postgresql). Previously, it was assumed that "schema"
was ``None`` when it matched the "default" schema, now the comparison
adjusts for this.
.. change::
:tags: bug
The :func:`.compare_metadata` public API function now takes into
account the settings for
:paramref:`.EnvironmentContext.configure.include_object`,
:paramref:`.EnvironmentContext.configure.include_symbol`,
and :paramref:`.EnvironmentContext.configure.include_schemas`, in the
same way that the ``--autogenerate`` command does. Pull
request courtesy Roman Podoliaka.
.. change::
:tags: bug
:tickets: 168
Calling :func:`.bulk_insert` with an empty list will not emit any
commands on the current connection. This was already the case with
``--sql`` mode, so is now the case with "online" mode.
.. change::
:tags: bug
Enabled schema support for index and unique constraint autodetection;
previously these were non-functional and could in some cases lead to
attribute errors. Pull request courtesy Dimitris Theodorou.
.. change::
:tags: bug
:tickets: 164
More fixes to index autodetection; indexes created with expressions
like DESC or functional indexes will no longer cause AttributeError
exceptions when attempting to compare the columns.
.. change::
:tags: feature
:tickets: 163
The :class:`.ScriptDirectory` system that loads migration files
from a ``versions/`` directory now supports so-called
"sourceless" operation, where the ``.py`` files are not present
and instead ``.pyc`` or ``.pyo`` files are directly present where
the ``.py`` files should be. Note that while Python 3.3 has a
new system of locating ``.pyc``/``.pyo`` files within a directory
called ``__pycache__`` (e.g. PEP-3147), PEP-3147 maintains
support for the "source-less imports" use case, where the
``.pyc``/``.pyo`` are in present in the "old" location, e.g. next
to the ``.py`` file; this is the usage that's supported even when
running Python3.3... changelog::
:version: 0.6.2
:released: Fri Dec 27 2013
.. change::
:tags: bug
Autogenerate for ``op.create_table()`` will not include a
``PrimaryKeyConstraint()`` that has no columns.
.. change::
:tags: bug
Fixed bug in the not-internally-used :meth:`.ScriptDirectory.get_base`
method which would fail if called on an empty versions directory.
.. change::
:tags: bug
:tickets: 157
An almost-rewrite of the new unique constraint/index autogenerate
detection, to accommodate a variety of issues. The emphasis is on
not generating false positives for those cases where no net change
is present, as these errors are the ones that impact all autogenerate
runs:
* Fixed an issue with unique constraint autogenerate detection where
a named ``UniqueConstraint`` on both sides with column changes would
render with the "add" operation before the "drop", requiring the
user to reverse the order manually.
* Corrected for MySQL's apparent addition of an implicit index
for a foreign key column, so that it doesn't show up as "removed".
This required that the index/constraint autogen system query the
dialect-specific implementation for special exceptions.
* reworked the "dedupe" logic to accommodate MySQL's bi-directional
duplication of unique indexes as unique constraints, and unique
constraints as unique indexes. Postgresql's slightly different
logic of duplicating unique constraints into unique indexes
continues to be accommodated as well. Note that a unique index
or unique constraint removal on a backend that duplicates these may
show up as a distinct "remove_constraint()" / "remove_index()" pair,
which may need to be corrected in the post-autogenerate if multiple
backends are being supported.
* added another dialect-specific exception to the SQLite backend
when dealing with unnamed unique constraints, as the backend can't
currently report on constraints that were made with this technique,
hence they'd come out as "added" on every run.
* the ``op.create_table()`` directive will be auto-generated with
the ``UniqueConstraint`` objects inline, but will not double them
up with a separate ``create_unique_constraint()`` call, which may
have been occurring. Indexes still get rendered as distinct
``op.create_index()`` calls even when the corresponding table was
created in the same script.
* the inline ``UniqueConstraint`` within ``op.create_table()`` includes
all the options like ``deferrable``, ``initially``, etc. Previously
these weren't rendering.
.. change::
:tags: feature, mssql
Added new argument ``mssql_drop_foreign_key`` to
:meth:`.Operations.drop_column`. Like ``mssql_drop_default``
and ``mssql_drop_check``, will do an inline lookup for a
single foreign key which applies to this column, and drop it.
For a column with more than one FK, you'd still need to explicitly
use :meth:`.Operations.drop_constraint` given the name,
even though only MSSQL has this limitation in the first place.
.. change::
:tags: bug, mssql
The MSSQL backend will add the batch separator (e.g. ``"GO"``)
in ``--sql`` mode after the final ``COMMIT`` statement, to ensure
that statement is also processed in batch mode. Courtesy
Derek Harland... changelog::
:version: 0.6.1
:released: Wed Nov 27 2013
.. change::
:tags: bug, mysql
:tickets: 152
Fixed bug where :func:`.op.alter_column` in the MySQL dialect
would fail to apply quotes to column names that had mixed casing
or spaces.
.. change::
:tags: feature
Expanded the size of the "slug" generated by "revision" to 40
characters, which is also configurable by new field
``truncate_slug_length``; and also split on the word rather than the
character; courtesy Frozenball.
.. change::
:tags: bug
:tickets: 135
Fixed the output wrapping for Alembic message output, so that
we either get the terminal width for "pretty printing" with
indentation, or if not we just output the text as is; in any
case the text won't be wrapped too short.
.. change::
:tags: bug
Fixes to Py3k in-place compatibility regarding output encoding and related;
the use of the new io.* package introduced some incompatibilities on Py2k.
These should be resolved, due to the introduction of new adapter types
for translating from io.* to Py2k file types, StringIO types.
Thanks to Javier Santacruz for help with this.
.. change::
:tags: bug
:tickets: 145
Fixed py3k bug where the wrong form of ``next()`` was being called
when using the list_templates command. Courtesy Chris Wilkes.
.. change::
:tags: feature
:tickets: 107
Support for autogeneration detection and rendering of indexes and
unique constraints has been added. The logic goes through some effort
in order to differentiate between true unique constraints and
unique indexes, where there are some quirks on backends like Postgresql.
The effort here in producing the feature and tests is courtesy of IJL.
.. change::
:tags: bug
Fixed bug introduced by new ``include_object`` argument where the
inspected column would be misinterpreted when using a user-defined
type comparison function, causing a KeyError or similar expression-related
error. Fix courtesy Maarten van Schaik.
.. change::
:tags: bug
Added the "deferrable" keyword argument to :func:`.op.create_foreign_key`
so that ``DEFERRABLE`` constraint generation is supported; courtesy
Pedro Romano.
.. change::
:tags: bug
:tickets: 137
Ensured that strings going to stdout go through an encode/decode phase,
so that any non-ASCII characters get to the output stream correctly
in both Py2k and Py3k. Also added source encoding detection using
Mako's parse_encoding() routine in Py2k so that the __doc__ of a
non-ascii revision file can be treated as unicode in Py2k... changelog::
:version: 0.6.0
:released: Fri July 19 2013
.. change::
:tags: feature
:tickets: 101
Added new kw argument to :meth:`.EnvironmentContext.configure`
``include_object``. This is a more flexible version of the
``include_symbol`` argument which allows filtering of columns as well as tables
from the autogenerate process,
and in the future will also work for types, constraints and
other constructs. The fully constructed schema object is passed,
including its name and type as well as a flag indicating if the object
is from the local application metadata or is reflected.
.. change::
:tags: feature
The output of the ``alembic history`` command is now
expanded to show information about each change on multiple
lines, including the full top message,
resembling the formatting of git log.
.. change::
:tags: feature
Added :attr:`alembic.config.Config.cmd_opts` attribute,
allows access to the ``argparse`` options passed to the
``alembic`` runner.
.. change::
:tags: feature
:tickets: 120
Added new command line argument ``-x``, allows extra arguments
to be appended to the command line which can be consumed
within an ``env.py`` script by looking at
``context.config.cmd_opts.x``, or more simply a new
method :meth:`.EnvironmentContext.get_x_argument`.
.. change::
:tags: bug
:tickets: 125
Added support for options like "name" etc. to be rendered
within CHECK constraints in autogenerate. Courtesy
Sok Ann Yap.
.. change::
:tags: misc
Source repository has been moved from Mercurial to Git.
.. change::
:tags: bug
Repaired autogenerate rendering of ForeignKeyConstraint
to include use_alter argument, if present.
.. change::
:tags: feature
Added ``-r`` argument to ``alembic history`` command,
allows specification of ``[start]:[end]`` to view
a slice of history. Accepts revision numbers, symbols
"base", "head", a new symbol "current" representing the
current migration, as well as relative ranges for one
side at a time (i.e. ``-r-5:head``, ``-rcurrent:+3``).
Courtesy Atsushi Odagiri for this feature.
.. change::
:tags: feature
:tickets: 55
Source base is now in-place for Python 2.6 through
3.3, without the need for 2to3. Support for Python 2.5
and below has been dropped. Huge thanks to
Hong Minhee for all the effort on this!.. changelog::
:version: 0.5.0
:released: Thu Apr 4 2013
.. note::
Alembic 0.5.0 now requires at least
version 0.7.3 of SQLAlchemy to run properly.
Support for 0.6 has been dropped.
.. change::
:tags: feature
:tickets: 76
Added ``version_table_schema`` argument
to :meth:`.EnvironmentContext.configure`,
complements the ``version_table`` argument to
set an optional remote schema for the version
table. Courtesy Christian Blume.
.. change::
:tags: bug, postgresql
:tickets: 32
Fixed format of RENAME for table that includes
schema with Postgresql; the schema name shouldn't
be in the "TO" field.
.. change::
:tags: feature
:tickets: 90
Added ``output_encoding`` option to
:meth:`.EnvironmentContext.configure`,
used with ``--sql`` mode to apply an encoding
to the output stream.
.. change::
:tags: feature
:tickets: 93
Added :meth:`.Operations.create_primary_key`
operation, will generate an ADD CONSTRAINT
for a primary key.
.. change::
:tags: bug, mssql
:tickets: 109
Fixed bug whereby double quoting would be applied
to target column name during an ``sp_rename``
operation.
.. change::
:tags: bug, sqlite, mysql
:tickets: 112
transactional_ddl flag for SQLite, MySQL dialects
set to False. MySQL doesn't support it,
SQLite does but current pysqlite driver does not.
.. change::
:tags: feature
:tickets: 115
upgrade and downgrade commands will list the
first line of the docstring out next to the
version number. Courtesy Hong Minhee.
.. change::
:tags: feature
Added --head-only option to "alembic current",
will print current version plus the symbol
"(head)" if this version is the head or not.
Courtesy Charles-Axel Dein.
.. change::
:tags: bug
:tickets: 110
Autogenerate will render additional table keyword
arguments like "mysql_engine" and others within
op.create_table().
.. change::
:tags: feature
:tickets: 108
The rendering of any construct during autogenerate
can be customized, in particular to allow special rendering
for user-defined column, constraint subclasses, using new
``render_item`` argument to
:meth:`.EnvironmentContext.configure`.
.. change::
:tags: bug
Fixed bug whereby create_index()
would include in the constraint columns that
are added to all Table objects using events,
externally to the generation of the constraint.
This is the same issue that was fixed for unique
constraints in version 0.3.2.
.. change::
:tags: bug
Worked around a backwards-incompatible regression in Python3.3
regarding argparse; running "alembic" with no arguments
now yields an informative error in py3.3 as with all previous versions.
Courtesy Andrey Antukh.
.. change::
:tags: change
SQLAlchemy 0.6 is no longer supported by Alembic - minimum version is 0.7.3,
full support is as of 0.7.9.
.. change::
:tags: bug
:tickets: 104
A host of argument name changes within migration
operations for consistency. Keyword arguments
will continue to work on the old name for backwards compatibility,
however required positional arguments will not:
:meth:`.Operations.alter_column` - ``name`` -> ``new_column_name`` - old
name will work for backwards compatibility.
:meth:`.Operations.create_index` - ``tablename`` -> ``table_name`` -
argument is positional.
:meth:`.Operations.drop_index` - ``tablename`` -> ``table_name`` - old
name will work for backwards compatibility.
:meth:`.Operations.drop_constraint` - ``tablename`` -> ``table_name`` -
argument is positional.
:meth:`.Operations.drop_constraint` - ``type`` -> ``type_`` - old
name will work for backwards compatibility.. changelog::
:version: 0.4.2
:released: Fri Jan 11 2013
.. change::
:tags: bug, autogenerate
:tickets: 99
Fixed bug where autogenerate would fail if a Column
to be added to a table made use of the ".key" parameter.
.. change::
:tags: bug, sqlite
:tickets: 98
The "implicit" constraint generated by a
type such as Boolean or Enum will not generate an
ALTER statement when run on SQlite, which does not
support ALTER for the purpose of adding/removing
constraints separate from the column def itself.
While SQLite supports adding a CHECK constraint
at the column level, SQLAlchemy would need modification
to support this.
A warning is emitted indicating this
constraint cannot be added in this scenario.
.. change::
:tags: bug
:tickets: 96
Added a workaround to setup.py to prevent
"NoneType" error from occurring when
"setup.py test" is run.
.. change::
:tags: bug
:tickets: 96
Added an append_constraint() step to each
condition within
test_autogenerate:AutogenRenderTest.test_render_fk_constraint_kwarg
if the SQLAlchemy version is less than 0.8, as ForeignKeyConstraint
does not auto-append prior to 0.8.
.. change::
:tags: feature
:tickets: 96
Added a README.unittests with instructions for running the test
suite fully... changelog::
:version: 0.4.1
:released: Sun Dec 9 2012
.. change::
:tags: bug
:tickets: 92
Added support for autogenerate render of
ForeignKeyConstraint options onupdate,
ondelete, initially, and deferred.
.. change::
:tags: bug
:tickets: 94
Autogenerate will include "autoincrement=False"
in the rendered table metadata
if this flag was set to false on the source
:class:`.Column` object.
.. change::
:tags: feature
:tickets: 66
Explicit error message describing the case
when downgrade --sql is used without specifying
specific start/end versions.
.. change::
:tags: bug
:tickets: 81
Removed erroneous "emit_events" attribute
from operations.create_table() documentation.
.. change::
:tags: bug
:tickets:
Fixed the minute component in file_template
which returned the month part of the create date... changelog::
:version: 0.4.0
:released: Mon Oct 01 2012
.. change::
:tags: feature
:tickets: 33
Support for tables in alternate schemas
has been added fully to all operations, as well as to
the autogenerate feature. When using autogenerate,
specifying the flag include_schemas=True to
Environment.configure() will also cause autogenerate
to scan all schemas located by Inspector.get_schema_names(),
which is supported by *some* (but not all)
SQLAlchemy dialects including Postgresql.
*Enormous* thanks to Bruno Binet for a huge effort
in implementing as well as writing tests. .
.. change::
:tags: feature
:tickets: 70
The command line runner has been organized
into a reusable CommandLine object, so that other
front-ends can reuse the argument parsing built
in.
.. change::
:tags: feature
:tickets: 43
Added "stdout" option to Config, provides
control over where the "print" output of commands like
"history", "init", "current" etc. are sent.
.. change::
:tags: bug
:tickets: 71
Fixed the "multidb" template which was badly out
of date. It now generates revision files using
the configuration to determine the different
upgrade_<xyz>() methods needed as well, instead of
needing to hardcode these. Huge thanks to
BryceLohr for doing the heavy lifting here.
.. change::
:tags: bug
:tickets: 72
Fixed the regexp that was checking for .py files
in the version directory to allow any .py file through.
Previously it was doing some kind of defensive checking,
probably from some early notions of how this directory
works, that was prohibiting various filename patterns
such as those which begin with numbers.
.. change::
:tags: bug
:tickets:
Fixed MySQL rendering for server_default which
didn't work if the server_default was a generated
SQL expression. Courtesy Moriyoshi Koizumi.
.. change::
:tags: feature
:tickets:
Added support for alteration of MySQL
columns that have AUTO_INCREMENT, as well as enabling
this flag. Courtesy Moriyoshi Koizumi... changelog::
:version: 0.3.6
:released: Wed Aug 15 2012
.. change::
:tags: feature
:tickets: 27
Added include_symbol option to
EnvironmentContext.configure(),
specifies a callable which will include/exclude tables
in their entirety from the autogeneration process
based on name.
.. change::
:tags: feature
:tickets: 59
Added year, month, day, hour, minute, second
variables to file_template.
.. change::
:tags: feature
:tickets:
Added 'primary' to the list of constraint types
recognized for MySQL drop_constraint().
.. change::
:tags: feature
:tickets:
Added --sql argument to the "revision" command,
for the use case where the "revision_environment"
config option is being used but SQL access isn't
desired.
.. change::
:tags: bug
:tickets:
Repaired create_foreign_key() for
self-referential foreign keys, which weren't working
at all.
.. change::
:tags: bug
:tickets: 63
'alembic' command reports an informative
error message when the configuration is missing
the 'script_directory' key.
.. change::
:tags: bug
:tickets: 62
Fixes made to the constraints created/dropped
alongside so-called "schema" types such as
Boolean and Enum. The create/drop constraint logic
does not kick in when using a dialect that doesn't
use constraints for these types, such as postgresql,
even when existing_type is specified to
alter_column(). Additionally, the constraints
are not affected if existing_type is passed but
type\_ is not, i.e. there's no net change
in type.
.. change::
:tags: bug
:tickets: 66
Improved error message when specifying
non-ordered revision identifiers to cover
the case when the "higher" rev is None,
improved message overall... changelog::
:version: 0.3.5
:released: Sun Jul 08 2012
.. change::
:tags: bug
:tickets: 31
Fixed issue whereby reflected server defaults
wouldn't be quoted correctly; uses repr() now.
.. change::
:tags: bug
:tickets: 58
Fixed issue whereby when autogenerate would
render create_table() on the upgrade side for a
table that has a Boolean type, an unnecessary
CheckConstraint() would be generated.
.. change::
:tags: feature
:tickets:
Implemented SQL rendering for
CheckConstraint() within autogenerate upgrade,
including for literal SQL as well as SQL Expression
Language expressions... changelog::
:version: 0.3.4
:released: Sat Jun 02 2012
.. change::
:tags: bug
:tickets:
Fixed command-line bug introduced by the
"revision_environment" feature... changelog::
:version: 0.3.3
:released: Sat Jun 02 2012
.. change::
:tags: feature
:tickets:
New config argument
"revision_environment=true", causes env.py to
be run unconditionally when the "revision" command
is run, to support script.py.mako templates with
dependencies on custom "template_args".
.. change::
:tags: feature
:tickets:
Added "template_args" option to configure()
so that an env.py can add additional arguments
to the template context when running the
"revision" command. This requires either --autogenerate
or the configuration directive "revision_environment=true".
.. change::
:tags: bug
:tickets: 44
Added "type" argument to op.drop_constraint(),
and implemented full constraint drop support for
MySQL. CHECK and undefined raise an error.
MySQL needs the constraint type
in order to emit a DROP CONSTRAINT.
.. change::
:tags: feature
:tickets: 34
Added version_table argument to
EnvironmentContext.configure(), allowing for the
configuration of the version table name.
.. change::
:tags: feature
:tickets:
Added support for "relative" migration
identifiers, i.e. "alembic upgrade +2",
"alembic downgrade -1". Courtesy
Atsushi Odagiri for this feature.
.. change::
:tags: bug
:tickets: 49
Fixed bug whereby directories inside of
the template directories, such as __pycache__
on Pypy, would mistakenly be interpreted as
files which are part of the template... changelog::
:version: 0.3.2
:released: Mon Apr 30 2012
.. change::
:tags: feature
:tickets: 40
Basic support for Oracle added,
courtesy shgoh.
.. change::
:tags: feature
:tickets:
Added support for UniqueConstraint
in autogenerate, courtesy Atsushi Odagiri
.. change::
:tags: bug
:tickets:
Fixed support of schema-qualified
ForeignKey target in column alter operations,
courtesy Alexander Kolov.
.. change::
:tags: bug
:tickets:
Fixed bug whereby create_unique_constraint()
would include in the constraint columns that
are added to all Table objects using events,
externally to the generation of the constraint... changelog::
:version: 0.3.1
:released: Sat Apr 07 2012
.. change::
:tags: bug
:tickets: 41
bulk_insert() fixes:
1. bulk_insert() operation was
not working most likely since the 0.2 series
when used with an engine.
2. Repaired bulk_insert() to complete when
used against a lower-case-t table and executing
with only one set of parameters, working
around SQLAlchemy bug #2461 in this regard.
3. bulk_insert() uses "inline=True" so that phrases
like RETURNING and such don't get invoked for
single-row bulk inserts.
4. bulk_insert() will check that you're passing
a list of dictionaries in, raises TypeError
if not detected... changelog::
:version: 0.3.0
:released: Thu Apr 05 2012
.. change::
:tags: general
:tickets:
The focus of 0.3 is to clean up
and more fully document the public API of Alembic,
including better accessors on the MigrationContext
and ScriptDirectory objects. Methods that are
not considered to be public on these objects have
been underscored, and methods which should be public
have been cleaned up and documented, including:
MigrationContext.get_current_revision()
ScriptDirectory.iterate_revisions()
ScriptDirectory.get_current_head()
ScriptDirectory.get_heads()
ScriptDirectory.get_base()
ScriptDirectory.generate_revision()
.. change::
:tags: feature
:tickets:
Added a bit of autogenerate to the
public API in the form of the function
alembic.autogenerate.compare_metadata... changelog::
:version: 0.2.2
:released: Mon Mar 12 2012
.. change::
:tags: feature
:tickets:
Informative error message when op.XYZ
directives are invoked at module import time.
.. change::
:tags: bug
:tickets: 35
Fixed inappropriate direct call to
util.err() and therefore sys.exit()
when Config failed to locate the
config file within library usage.
.. change::
:tags: bug
:tickets:
Autogenerate will emit CREATE TABLE
and DROP TABLE directives according to
foreign key dependency order.
.. change::
:tags: bug
:tickets:
implement 'tablename' parameter on
drop_index() as this is needed by some
backends.
.. change::
:tags: feature
:tickets:
Added execution_options parameter
to op.execute(), will call execution_options()
on the Connection before executing.
The immediate use case here is to allow
access to the new no_parameters option
in SQLAlchemy 0.7.6, which allows
some DBAPIs (psycopg2, MySQLdb) to allow
percent signs straight through without
escaping, thus providing cross-compatible
operation with DBAPI execution and
static script generation.
.. change::
:tags: bug
:tickets:
setup.py won't install argparse if on
Python 2.7/3.2
.. change::
:tags: feature
:tickets: 29
script_location can be interpreted
by pkg_resources.resource_filename(), if
it is a non-absolute URI that contains
colons. This scheme is the same
one used by Pyramid.
.. change::
:tags: feature
:tickets:
added missing support for
onupdate/ondelete flags for
ForeignKeyConstraint, courtesy Giacomo Bagnoli
.. change::
:tags: bug
:tickets: 30
fixed a regression regarding an autogenerate
error message, as well as various glitches
in the Pylons sample template. The Pylons sample
template requires that you tell it where to
get the Engine from now. courtesy
Marcin Kuzminski
.. change::
:tags: bug
:tickets:
drop_index() ensures a dummy column
is added when it calls "Index", as SQLAlchemy
0.7.6 will warn on index with no column names... changelog::
:version: 0.2.1
:released: Tue Jan 31 2012
.. change::
:tags: bug
:tickets: 26
Fixed the generation of CHECK constraint,
regression from 0.2.0.. changelog::
:version: 0.2.0
:released: Mon Jan 30 2012
.. change::
:tags: feature
:tickets: 19
API rearrangement allows everything
Alembic does to be represented by contextual
objects, including EnvironmentContext,
MigrationContext, and Operations. Other
libraries and applications can now use
things like "alembic.op" without relying
upon global configuration variables.
The rearrangement was done such that
existing migrations should be OK,
as long as they use the pattern
of "from alembic import context" and
"from alembic import op", as these
are now contextual objects, not modules.
.. change::
:tags: feature
:tickets: 24
The naming of revision files can
now be customized to be some combination
of "rev id" and "slug", the latter of which
is based on the revision message.
By default, the pattern "<rev>_<slug>"
is used for new files. New script files
should include the "revision" variable
for this to work, which is part of
the newer script.py.mako scripts.
.. change::
:tags: bug
:tickets: 25
env.py templates call
connection.close() to better support
programmatic usage of commands; use
NullPool in conjunction with create_engine()
as well so that no connection resources
remain afterwards.
.. change::
:tags: bug
:tickets: 22
fix the config.main() function to honor
the arguments passed, remove no longer used
"scripts/alembic" as setuptools creates this
for us.
.. change::
:tags: bug
:tickets:
Fixed alteration of column type on
MSSQL to not include the keyword "TYPE".
.. change::
:tags: feature
:tickets: 23
Can create alembic.config.Config
with no filename, use set_main_option()
to add values. Also added set_section_option()
which will add sections... changelog::
:version: 0.1.1
:released: Wed Jan 04 2012
.. change::
:tags: bug
:tickets:
Clean up file write operations so that
file handles are closed.
.. change::
:tags: feature
:tickets:
PyPy is supported.
.. change::
:tags: feature
:tickets:
Python 2.5 is supported, needs
__future__.with_statement
.. change::
:tags: bug
:tickets:
Fix autogenerate so that "pass" is
generated between the two comments
if no net migrations were present.
.. change::
:tags: bug
:tickets: 16
Fix autogenerate bug that prevented
correct reflection of a foreign-key
referenced table in the list of "to remove".
.. change::
:tags: bug
:tickets: 17
Fix bug where create_table() didn't
handle self-referential foreign key
correctly
.. change::
:tags: bug
:tickets: 18
Default prefix for autogenerate
directives is "op.", matching the
mako templates.
.. change::
:tags: feature
:tickets: 18
Add alembic_module_prefix argument
to configure() to complement
sqlalchemy_module_prefix.
.. change::
:tags: bug
:tickets: 14
fix quotes not being rendered in
ForeignKeConstraint during
autogenerate.. changelog::
:version: 0.1.0
:released: Wed Nov 30 2011
.. change::
:tags:
:tickets:
Initial release. Status of features:
.. change::
:tags:
:tickets:
Alembic is used in at least one production
environment, but should still be considered
ALPHA LEVEL SOFTWARE as of this release,
particularly in that many features are expected
to be missing / unimplemented. Major API
changes are not anticipated but for the moment
nothing should be assumed.
The author asks that you *please* report all
issues, missing features, workarounds etc.
to the bugtracker.
.. change::
:tags:
:tickets:
Python 3 is supported and has been tested.
.. change::
:tags:
:tickets:
The "Pylons" and "MultiDB" environment templates
have not been directly tested - these should be
considered to be samples to be modified as
needed. Multiple database support itself
is well tested, however.
.. change::
:tags:
:tickets:
Postgresql and MS SQL Server environments
have been tested for several weeks in a production
environment. In particular, some involved workarounds
were implemented to allow fully-automated dropping
of default- or constraint-holding columns with
SQL Server.
.. change::
:tags:
:tickets:
MySQL support has also been implemented to a
basic degree, including MySQL's awkward style
of modifying columns being accommodated.
.. change::
:tags:
:tickets:
Other database environments not included among
those three have *not* been tested, *at all*. This
includes Firebird, Oracle, Sybase. Adding
support for these backends should be
straightforward. Please report all missing/
incorrect behaviors to the bugtracker! Patches
are welcome here but are optional - please just
indicate the exact format expected by the target
database.
.. change::
:tags:
:tickets:
SQLite, as a backend, has almost no support for
schema alterations to existing databases. The author
would strongly recommend that SQLite not be used in
a migration context - just dump your SQLite database
into an intermediary format, then dump it back
into a new schema. For dev environments, the
dev installer should be building the whole DB from
scratch. Or just use Postgresql, which is a much
better database for non-trivial schemas.
Requests for full ALTER support on SQLite should be
reported to SQLite's bug tracker at
http://www.sqlite.org/src/wiki?name=Bug+Reports,
as Alembic will not be implementing the
"rename the table to a temptable then copy the
data into a new table" workaround.
Note that Alembic will at some point offer an
extensible API so that you can implement commands
like this yourself.
.. change::
:tags:
:tickets:
Well-tested directives include add/drop table, add/drop
column, including support for SQLAlchemy "schema"
types which generate additional CHECK
constraints, i.e. Boolean, Enum. Other directives not
included here have *not* been strongly tested
in production, i.e. rename table, etc.
.. change::
:tags:
:tickets:
Both "online" and "offline" migrations, the latter
being generated SQL scripts to hand off to a DBA,
have been strongly production tested against
Postgresql and SQL Server.
.. change::
:tags:
:tickets:
Modify column type, default status, nullable, is
functional and tested across PG, MSSQL, MySQL,
but not yet widely tested in production usage.
.. change::
:tags:
:tickets:
Many migrations are still outright missing, i.e.
create/add sequences, etc. As a workaround,
execute() can be used for those which are missing,
though posting of tickets for new features/missing
behaviors is strongly encouraged.
.. change::
:tags:
:tickets:
Autogenerate feature is implemented and has been
tested, though only a little bit in a production setting.
In particular, detection of type and server
default changes are optional and are off by default;
they can also be customized by a callable.
Both features work but can have surprises particularly
the disparity between BIT/TINYINT and boolean,
which hasn't yet been worked around, as well as
format changes performed by the database on defaults
when it reports back. When enabled, the PG dialect
will execute the two defaults to be compared to
see if they are equivalent. Other backends may
need to do the same thing.
The autogenerate feature only generates
"candidate" commands which must be hand-tailored
in any case, so is still a useful feature and
is safe to use. Please report missing/broken features
of autogenerate! This will be a great feature and
will also improve SQLAlchemy's reflection services.
.. change::
:tags:
:tickets:
Support for non-ASCII table, column and constraint
names is mostly nonexistent. This is also a
straightforward feature add as SQLAlchemy itself
supports unicode identifiers; Alembic itself will
likely need fixes to logging, column identification
by key, etc. for full support here.Build Cookbook (`docs/build/cookbook.rst`)
======== Cookbook
A collection of "How-Tos" highlighting popular ways to extend
Alembic.
.. note::
This is a new section where we catalogue various "how-tos"
based on user requests. It is often the case that users
will request a feature only to learn it can be provided with
a simple customization... _building_uptodate:
Building an Up to Date Database from Scratch
There's a theory of database migrations that says that the revisions in existence for a database should be
able to go from an entirely blank schema to the finished product, and back again. Alembic can roll
this way. Though we think it's kind of overkill, considering that SQLAlchemy itself can emit
the full CREATE statements for any given model using :meth:~sqlalchemy.schema.MetaData.create_all. If you check out
a copy of an application, running this will give you the entire database in one shot, without the need
to run through all those migration files, which are instead tailored towards applying incremental
changes to an existing database.
Alembic can integrate with a :meth:~sqlalchemy.schema.MetaData.create_all script quite easily. After running the
create operation, tell Alembic to create a new version table, and to stamp it with the most recent
revision (i.e. head)::
# inside of a "create the database" script, first create
# tables:
my_metadata.create_all(engine)
# then, load the Alembic configuration and generate the
# version table, "stamping" it with the most recent rev:
from alembic.config import Config
from alembic import command
alembic_cfg = Config("/path/to/yourapp/alembic.ini")
command.stamp(alembic_cfg, "head")When this approach is used, the application can generate the database using normal SQLAlchemy
techniques instead of iterating through hundreds of migration scripts. Now, the purpose of the
migration scripts is relegated just to movement between versions on out-of-date databases, not
new databases. You can now remove old migration files that are no longer represented
on any existing environments.
To prune old migration files, simply delete the files. Then, in the earliest, still-remaining
migration file, set down_revision to None::
# replace this:
#down_revision = '290696571ad2'
# with this:
down_revision = NoneThat file now becomes the "base" of the migration series.
Conditional Migration Elements
This example features the basic idea of a common need, that of affecting
how a migration runs based on command line switches.
The technique to use here is simple; within a migration script, inspect
the :meth:.EnvironmentContext.get_x_argument collection for any additional,
user-defined parameters. Then take action based on the presence of those
arguments.
To make it such that the logic to inspect these flags is easy to use and
modify, we modify our script.py.mako template to make this feature
available in all new revision files:
.. code-block:: mako
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision}
Create Date: ${create_date}
"""
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
from alembic import context
def upgrade():
schema_upgrades()
if context.get_x_argument(as_dictionary=True).get('data', None):
data_upgrades()
def downgrade():
if context.get_x_argument(as_dictionary=True).get('data', None):
data_downgrades()
schema_downgrades()
def schema_upgrades():
"""schema upgrade migrations go here."""
${upgrades if upgrades else "pass"}
def schema_downgrades():
"""schema downgrade migrations go here."""
${downgrades if downgrades else "pass"}
def data_upgrades():
"""Add any optional data upgrade migrations here!"""
pass
def data_downgrades():
"""Add any optional data downgrade migrations here!"""
passNow, when we create a new migration file, the data_upgrades() and data_downgrades()
placeholders will be available, where we can add optional data migrations::
"""rev one
Revision ID: 3ba2b522d10d
Revises: None
Create Date: 2014-03-04 18:05:36.992867
"""
# revision identifiers, used by Alembic.
revision = '3ba2b522d10d'
down_revision = None
from alembic import op
import sqlalchemy as sa
from sqlalchemy import String, Column
from sqlalchemy.sql import table, column
from alembic import context
def upgrade():
schema_upgrades()
if context.get_x_argument(as_dictionary=True).get('data', None):
data_upgrades()
def downgrade():
if context.get_x_argument(as_dictionary=True).get('data', None):
data_downgrades()
schema_downgrades()
def schema_upgrades():
"""schema upgrade migrations go here."""
op.create_table("my_table", Column('data', String))
def schema_downgrades():
"""schema downgrade migrations go here."""
op.drop_table("my_table")
def data_upgrades():
"""Add any optional data upgrade migrations here!"""
my_table = table('my_table',
column('data', String),
)
op.bulk_insert(my_table,
[
{'data': 'data 1'},
{'data': 'data 2'},
{'data': 'data 3'},
]
)
def data_downgrades():
"""Add any optional data downgrade migrations here!"""
op.execute("delete from my_table")To invoke our migrations with data included, we use the -x flag::
alembic -x data=true upgrade headThe :meth:.EnvironmentContext.get_x_argument is an easy way to support
new commandline options within environment and migration scripts.
.. _connection_sharing:
Sharing a Connection across one or more programmatic migration commands
It is often the case that an application will need to call upon a series
of commands within :ref:alembic.command.toplevel, where it would be advantageous
for all operations to proceed along a single transaction. The connectivity
for a migration is typically solely determined within the env.py script
of a migration environment, which is called within the scope of a command.
The steps to take here are:
Produce the :class:
~sqlalchemy.engine.Connectionobject to use.Place it somewhere that
env.pywill be able to access it. This
can be either a. a module-level global somewhere, or b.
an attribute which we place into the :attr:.Config.attributes
dictionary (if we are on an older Alembic version, we may also attach
an attribute directly to the :class:.Configobject).The
env.pyscript is modified such that it looks for this
:class:~sqlalchemy.engine.Connectionand makes use of it, in lieu
of building up its own :class:~sqlalchemy.engine.Engineinstance.
We illustrate using :attr:.Config.attributes a script that will run
the :func:.command.upgrade command programmatically within a
transaction declared in a Python file::
from alembic import command, config
cfg = config.Config("/path/to/yourapp/alembic.ini")
with engine.begin() as connection:
cfg.attributes['connection'] = connection
command.upgrade(cfg, "head")Then in env.py we can update run_migrations_online::
def run_migrations_online():
connectable = config.attributes.get('connection', None)
if connectable is None:
# only create Engine if we don't have a Connection
# from the outside
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix='sqlalchemy.',
poolclass=pool.NullPool)
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations()
else:
context.configure(
connection=connectable,
target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations().. _replaceable_objects:
Replaceable Objects
This recipe proposes a hypothetical way of dealing with
what we might call a replaceable schema object. A replaceable object
is a schema object that needs to be created and dropped all at once.
Examples of such objects include views, stored procedures, and triggers.
.. seealso::
The Replaceable Object concept has been integrated by the
`Alembic Utils <https://github.com/olirice/alembic_utils>`_ project,
which provides autogenerate and migration
support for PostgreSQL functions and views. See
Alembic Utils at https://github.com/olirice/alembic_utils .Replaceable objects present a problem in that in order to make incremental
changes to them, we have to refer to the whole definition at once.
If we need to add a new column to a view, for example, we have to drop
it entirely and recreate it fresh with the extra column added, referring to
the whole structure; but to make it even tougher, if we wish to support
downgrade operations in our migration scripts,
we need to refer to the previous version of that
construct fully, and we'd much rather not have to type out the whole
definition in multiple places.
This recipe proposes that we may refer to the older version of a
replaceable construct by directly naming the migration version in
which it was created, and having a migration refer to that previous
file as migrations run. We will also demonstrate how to integrate this
logic within the :ref:operation_plugins feature introduced in
Alembic 0.8. It may be very helpful to review
this section first to get an overview of this API.
The Replaceable Object Structure
We first need to devise a simple format that represents the "CREATE XYZ" /
"DROP XYZ" aspect of what it is we're building. We will work with an object
that represents a textual definition; while a SQL view is an object that we can define
using a table-metadata-like system <https://github.com/sqlalchemy/sqlalchemy/wiki/UsageRecipes/Views>_,
this is not so much the case for things like stored procedures, where
we pretty much need to have a full string definition written down somewhere.
We'll use a simple value object called ReplaceableObject that can
represent any named set of SQL text to send to a "CREATE" statement of
some kind::
class ReplaceableObject:
def __init__(self, name, sqltext):
self.name = name
self.sqltext = sqltextUsing this object in a migration script, assuming a Postgresql-style
syntax, looks like::
customer_view = ReplaceableObject(
"customer_view",
"SELECT name, order_count FROM customer WHERE order_count > 0"
)
add_customer_sp = ReplaceableObject(
"add_customer_sp(name varchar, order_count integer)",
"""
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count)
VALUES (in_name, in_order_count);
END;
$$ LANGUAGE plpgsql;
"""
)The ReplaceableObject class is only one very simplistic way to do this.
The structure of how we represent our schema objects
is not too important for the purposes of this example; we can just
as well put strings inside of tuples or dictionaries, as well as
that we could define any kind of series of fields and class structures we want.
The only important part is that below we will illustrate how organize the
code that can consume the structure we create here.
Create Operations for the Target Objects
We'll use the :class:.Operations extension API to make new operations
for create, drop, and replace of views and stored procedures. Using this
API is also optional; we can just as well make any kind of Python
function that we would invoke from our migration scripts.
However, using this API gives us operations
built directly into the Alembic op.* namespace very nicely.
The most intricate class is below. This is the base of our "replaceable"
operation, which includes not just a base operation for emitting
CREATE and DROP instructions on a ReplaceableObject, it also assumes
a certain model of "reversibility" which makes use of references to
other migration files in order to refer to the "previous" version
of an object::
from alembic.operations import Operations, MigrateOperation
class ReversibleOp(MigrateOperation):
def __init__(self, target):
self.target = target
@classmethod
def invoke_for_target(cls, operations, target):
op = cls(target)
return operations.invoke(op)
def reverse(self):
raise NotImplementedError()
@classmethod
def _get_object_from_version(cls, operations, ident):
version, objname = ident.split(".")
module = operations.get_context().script.get_revision(version).module
obj = getattr(module, objname)
return obj
@classmethod
def replace(cls, operations, target, replaces=None, replace_with=None):
if replaces:
old_obj = cls._get_object_from_version(operations, replaces)
drop_old = cls(old_obj).reverse()
create_new = cls(target)
elif replace_with:
old_obj = cls._get_object_from_version(operations, replace_with)
drop_old = cls(target).reverse()
create_new = cls(old_obj)
else:
raise TypeError("replaces or replace_with is required")
operations.invoke(drop_old)
operations.invoke(create_new)The workings of this class should become clear as we walk through the
example. To create usable operations from this base, we will build
a series of stub classes and use :meth:.Operations.register_operation
to make them part of the op.* namespace::
@Operations.register_operation("create_view", "invoke_for_target")
@Operations.register_operation("replace_view", "replace")
class CreateViewOp(ReversibleOp):
def reverse(self):
return DropViewOp(self.target)
@Operations.register_operation("drop_view", "invoke_for_target")
class DropViewOp(ReversibleOp):
def reverse(self):
return CreateViewOp(self.target)
@Operations.register_operation("create_sp", "invoke_for_target")
@Operations.register_operation("replace_sp", "replace")
class CreateSPOp(ReversibleOp):
def reverse(self):
return DropSPOp(self.target)
@Operations.register_operation("drop_sp", "invoke_for_target")
class DropSPOp(ReversibleOp):
def reverse(self):
return CreateSPOp(self.target)To actually run the SQL like "CREATE VIEW" and "DROP SEQUENCE", we'll provide
implementations using :meth:.Operations.implementation_for
that run straight into :meth:.Operations.execute::
@Operations.implementation_for(CreateViewOp)
def create_view(operations, operation):
operations.execute("CREATE VIEW %s AS %s" % (
operation.target.name,
operation.target.sqltext
))
@Operations.implementation_for(DropViewOp)
def drop_view(operations, operation):
operations.execute("DROP VIEW %s" % operation.target.name)
@Operations.implementation_for(CreateSPOp)
def create_sp(operations, operation):
operations.execute(
"CREATE FUNCTION %s %s" % (
operation.target.name, operation.target.sqltext
)
)
@Operations.implementation_for(DropSPOp)
def drop_sp(operations, operation):
operations.execute("DROP FUNCTION %s" % operation.target.name)Publish the Extensions
All of the above code can be present anywhere within an application's
source tree; the only requirement is that when the env.py script is
invoked, it includes imports that ultimately call upon these classes
as well as the :meth:.Operations.register_operation and
:meth:.Operations.implementation_for sequences.
Alternatively, custom operations and autogenerate support can be organized
into reusable plugins using Alembic's plugin system. This allows extensions
to be packaged and distributed independently, and automatically discovered
via Python entry points. See :ref:alembic.plugins.toplevel for information
on writing and publishing plugins.
Create Initial Migrations
We can now illustrate how these objects look during use. For the first step,
we'll create a new migration to create a "customer" table::
$ alembic revision -m "create table"We build the first revision as follows::
"""create table
Revision ID: 3ab8b2dfb055
Revises:
Create Date: 2015-07-27 16:22:44.918507
"""
# revision identifiers, used by Alembic.
revision = '3ab8b2dfb055'
down_revision = None
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
def upgrade():
op.create_table(
"customer",
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('name', sa.String),
sa.Column('order_count', sa.Integer),
)
def downgrade():
op.drop_table('customer')For the second migration, we will create a view and a stored procedure
which act upon this table::
$ alembic revision -m "create views/sp"This migration will use the new directives::
"""create views/sp
Revision ID: 28af9800143f
Revises: 3ab8b2dfb055
Create Date: 2015-07-27 16:24:03.589867
"""
# revision identifiers, used by Alembic.
revision = '28af9800143f'
down_revision = '3ab8b2dfb055'
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
from foo import ReplaceableObject
customer_view = ReplaceableObject(
"customer_view",
"SELECT name, order_count FROM customer WHERE order_count > 0"
)
add_customer_sp = ReplaceableObject(
"add_customer_sp(name varchar, order_count integer)",
"""
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count)
VALUES (in_name, in_order_count);
END;
$$ LANGUAGE plpgsql;
"""
)
def upgrade():
op.create_view(customer_view)
op.create_sp(add_customer_sp)
def downgrade():
op.drop_view(customer_view)
op.drop_sp(add_customer_sp)We see the use of our new create_view(), create_sp(),drop_view(), and drop_sp() directives. Running these to "head"
we get the following (this includes an edited view of SQL emitted)::
$ alembic upgrade 28af9800143
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [sqlalchemy.engine.base.Engine] BEGIN (implicit)
INFO [sqlalchemy.engine.base.Engine] select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
INFO [sqlalchemy.engine.base.Engine] {'name': u'alembic_version'}
INFO [sqlalchemy.engine.base.Engine] SELECT alembic_version.version_num
FROM alembic_version
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
INFO [sqlalchemy.engine.base.Engine] {'name': u'alembic_version'}
INFO [alembic.runtime.migration] Running upgrade -> 3ab8b2dfb055, create table
INFO [sqlalchemy.engine.base.Engine]
CREATE TABLE customer (
id SERIAL NOT NULL,
name VARCHAR,
order_count INTEGER,
PRIMARY KEY (id)
)
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] INSERT INTO alembic_version (version_num) VALUES ('3ab8b2dfb055')
INFO [sqlalchemy.engine.base.Engine] {}
INFO [alembic.runtime.migration] Running upgrade 3ab8b2dfb055 -> 28af9800143f, create views/sp
INFO [sqlalchemy.engine.base.Engine] CREATE VIEW customer_view AS SELECT name, order_count FROM customer WHERE order_count > 0
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] CREATE FUNCTION add_customer_sp(name varchar, order_count integer)
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count)
VALUES (in_name, in_order_count);
END;
$$ LANGUAGE plpgsql;
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] UPDATE alembic_version SET version_num='28af9800143f' WHERE alembic_version.version_num = '3ab8b2dfb055'
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] COMMITWe see that our CREATE TABLE proceeded as well as the CREATE VIEW and CREATE
FUNCTION operations produced by our new directives.
Create Revision Migrations
Finally, we can illustrate how we would "revise" these objects.
Let's consider we added a new column email to our customer table::
$ alembic revision -m "add email col"The migration is::
"""add email col
Revision ID: 191a2d20b025
Revises: 28af9800143f
Create Date: 2015-07-27 16:25:59.277326
"""
# revision identifiers, used by Alembic.
revision = '191a2d20b025'
down_revision = '28af9800143f'
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
def upgrade():
op.add_column("customer", sa.Column("email", sa.String()))
def downgrade():
op.drop_column("customer", "email")We now need to recreate the customer_view view and theadd_customer_sp function. To include downgrade capability, we will
need to refer to the previous version of the construct; thereplace_view() and replace_sp() operations we've created make
this possible, by allowing us to refer to a specific, previous revision.
the replaces and replace_with arguments accept a dot-separated
string, which refers to a revision number and an object name, such
as "28af9800143f.customer_view". The ReversibleOp class makes use
of the :meth:.Operations.get_context method to locate the version file
we refer to::
$ alembic revision -m "update views/sp"The migration::
"""update views/sp
Revision ID: 199028bf9856
Revises: 191a2d20b025
Create Date: 2015-07-27 16:26:31.344504
"""
# revision identifiers, used by Alembic.
revision = '199028bf9856'
down_revision = '191a2d20b025'
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
from foo import ReplaceableObject
customer_view = ReplaceableObject(
"customer_view",
"SELECT name, order_count, email "
"FROM customer WHERE order_count > 0"
)
add_customer_sp = ReplaceableObject(
"add_customer_sp(name varchar, order_count integer, email varchar)",
"""
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count, email)
VALUES (in_name, in_order_count, email);
END;
$$ LANGUAGE plpgsql;
"""
)
def upgrade():
op.replace_view(customer_view, replaces="28af9800143f.customer_view")
op.replace_sp(add_customer_sp, replaces="28af9800143f.add_customer_sp")
def downgrade():
op.replace_view(customer_view, replace_with="28af9800143f.customer_view")
op.replace_sp(add_customer_sp, replace_with="28af9800143f.add_customer_sp")Above, instead of using create_view(), create_sp(),drop_view(), and drop_sp() methods, we now use replace_view() andreplace_sp(). The replace operation we've built always runs a DROP and
a CREATE. Running an upgrade to head we see::
$ alembic upgrade head
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [sqlalchemy.engine.base.Engine] BEGIN (implicit)
INFO [sqlalchemy.engine.base.Engine] select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
INFO [sqlalchemy.engine.base.Engine] {'name': u'alembic_version'}
INFO [sqlalchemy.engine.base.Engine] SELECT alembic_version.version_num
FROM alembic_version
INFO [sqlalchemy.engine.base.Engine] {}
INFO [alembic.runtime.migration] Running upgrade 28af9800143f -> 191a2d20b025, add email col
INFO [sqlalchemy.engine.base.Engine] ALTER TABLE customer ADD COLUMN email VARCHAR
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] UPDATE alembic_version SET version_num='191a2d20b025' WHERE alembic_version.version_num = '28af9800143f'
INFO [sqlalchemy.engine.base.Engine] {}
INFO [alembic.runtime.migration] Running upgrade 191a2d20b025 -> 199028bf9856, update views/sp
INFO [sqlalchemy.engine.base.Engine] DROP VIEW customer_view
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] CREATE VIEW customer_view AS SELECT name, order_count, email FROM customer WHERE order_count > 0
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] DROP FUNCTION add_customer_sp(name varchar, order_count integer)
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] CREATE FUNCTION add_customer_sp(name varchar, order_count integer, email varchar)
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count, email)
VALUES (in_name, in_order_count, email);
END;
$$ LANGUAGE plpgsql;
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] UPDATE alembic_version SET version_num='199028bf9856' WHERE alembic_version.version_num = '191a2d20b025'
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] COMMITAfter adding our new email column, we see that both customer_view
and add_customer_sp() are dropped before the new version is created.
If we downgrade back to the old version, we see the old version of these
recreated again within the downgrade for this migration::
$ alembic downgrade 28af9800143
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [sqlalchemy.engine.base.Engine] BEGIN (implicit)
INFO [sqlalchemy.engine.base.Engine] select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
INFO [sqlalchemy.engine.base.Engine] {'name': u'alembic_version'}
INFO [sqlalchemy.engine.base.Engine] SELECT alembic_version.version_num
FROM alembic_version
INFO [sqlalchemy.engine.base.Engine] {}
INFO [alembic.runtime.migration] Running downgrade 199028bf9856 -> 191a2d20b025, update views/sp
INFO [sqlalchemy.engine.base.Engine] DROP VIEW customer_view
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] CREATE VIEW customer_view AS SELECT name, order_count FROM customer WHERE order_count > 0
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] DROP FUNCTION add_customer_sp(name varchar, order_count integer, email varchar)
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] CREATE FUNCTION add_customer_sp(name varchar, order_count integer)
RETURNS integer AS $$
BEGIN
insert into customer (name, order_count)
VALUES (in_name, in_order_count);
END;
$$ LANGUAGE plpgsql;
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] UPDATE alembic_version SET version_num='191a2d20b025' WHERE alembic_version.version_num = '199028bf9856'
INFO [sqlalchemy.engine.base.Engine] {}
INFO [alembic.runtime.migration] Running downgrade 191a2d20b025 -> 28af9800143f, add email col
INFO [sqlalchemy.engine.base.Engine] ALTER TABLE customer DROP COLUMN email
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] UPDATE alembic_version SET version_num='28af9800143f' WHERE alembic_version.version_num = '191a2d20b025'
INFO [sqlalchemy.engine.base.Engine] {}
INFO [sqlalchemy.engine.base.Engine] COMMIT.. _cookbook_postgresql_multi_tenancy:
Rudimental Schema-Level Multi Tenancy for PostgreSQL, MySQL, Other Databases
Multi tenancy refers to an application that accommodates for many
clients simultaneously. Within the scope of a database migrations tool,
multi-tenancy typically refers to the practice of maintaining multiple,
identical databases where each database is assigned to one client.
Alembic does not currently have explicit multi-tenant support; typically,
the approach must involve running Alembic multiple times against different
database URLs.
One common approach to multi-tenancy, particularly on the PostgreSQL database,
is to install tenants within individual PostgreSQL schemas; similarly
when using MySQL/MariaDB, individual MySQL/MariaDB databases are addressed
in the same way as "schemas" on PostgreSQL.
When using PostgreSQL's schemas, a special variable search_path is offered
that is intended to assist with targeting of different schemas. When using
MySQL or MariaDB databases, a similar command is available at the SQL level
called the USE command. This command may be used in a similar fashion as
that of PostgreSQL's search_path variable to achieve a similar effect.
Overall, this recipe can be used on any database that supports runtime
modification of the current "tenant" via SQL commands on a particular
connection.
.. note:: SQLAlchemy includes a system of directing a common set of
Table metadata to many schemas called schema_translate_map <https://docs.sqlalchemy.org/core/connections.html#translation-of-schema-names>_. Alembic at the time
of this writing lacks adequate support for this feature. The recipe below
should be considered interim until Alembic has more first-class support
for schema-level multi-tenancy.
The recipe below can be altered for flexibility. The primary purpose of this
recipe is to illustrate how to point the Alembic process towards one PostgreSQL
or MySQL/MariaDB schema or another.
The model metadata used as the target for autogenerate must not include any
schema name for tables; the schema must be non-present or set toNone.
Otherwise, Alembic autogenerate will still attempt
to compare and render tables in terms of this schema::textclass A(Base): __tablename__ = 'a' id = Column(Integer, primary_key=True) data = Column(UnicodeText()) foo = Column(Integer) __table_args__ = { "schema": None }..
The :paramref:
.EnvironmentContext.configure.include_schemasflag must
also be False or not included.The "tenant" will be a schema name passed to Alembic using the "-x" flag.
Inenv.pyan approach like the following allows-xtenant=some_schema
to be supported by making use of :meth:.EnvironmentContext.get_x_argument::textfrom sqlalchemy import text def run_migrations_online(): connectable = engine_from_config( config.get_section(config.config_ini_section), prefix="sqlalchemy.", poolclass=pool.NullPool, ) current_tenant = context.get_x_argument(as_dictionary=True).get("tenant") with connectable.connect() as connection: if connection.dialect.name == "postgresql": # set search path on the connection, which ensures that # PostgreSQL will emit all CREATE / ALTER / DROP statements # in terms of this schema by default connection.execute(text('set search_path to "%s"' % current_tenant)) # in SQLAlchemy v2+ the search path change needs to be committed connection.commit() elif connection.dialect.name in ("mysql", "mariadb"): # set "USE" on the connection, which ensures that # MySQL/MariaDB will emit all CREATE / ALTER / DROP statements # in terms of this schema by default connection.execute(text('USE %s' % current_tenant)) # make use of non-supported SQLAlchemy attribute to ensure # the dialect reflects tables in terms of the current tenant name connection.dialect.default_schema_name = current_tenant context.configure( connection=connection, target_metadata=target_metadata, ) with context.begin_transaction(): context.run_migrations()The current tenant is set using the PostgreSQL
search_pathvariable, or
the MySQL/MariaDBUSEstatement, on the connection. Note above we must
employ a non-supported SQLAlchemy workaround at the moment which is to
hardcode the SQLAlchemy dialect's default schema name to our target schema.It is also important to note that the above changes remain on the connection
permanently unless reversed explicitly. If the alembic application simply
exits above, there is no issue. However if the application attempts to
continue using the above connection for other purposes, it may be necessary
to reset these variables back to the default, which for PostgreSQL is usually
the name "public" however may be different based on configuration, and
for MySQL/MariaDB is typically the "database" portion of the database URL.Alembic operations will now proceed in terms of whichever schema we pass
on the command line. All logged SQL will show no schema, except for
reflection operations which will make use of thedefault_schema_name
attribute::text[]$ alembic -x tenant=some_schema revision -m "rev1" --autogenerate..
Since all schemas are to be maintained in sync, autogenerate should be run
against only one schema, generating new Alembic migration files.
Autogenerated migration operations are then run against all schemas.
.. _cookbook_no_empty_migrations:
Don't Generate Empty Migrations with Autogenerate
A common request is to have the alembic revision --autogenerate command not
actually generate a revision file if no changes to the schema is detected. Using
the :paramref:.EnvironmentContext.configure.process_revision_directives
hook, this is straightforward; place a process_revision_directives
hook in :meth:.MigrationContext.configure which removes the
single :class:.MigrationScript directive if it is empty of
any operations::
# for typing purposes
from collections.abc import Iterable
from alembic.environment import MigrationContext
# this typing-only import requires alembic 1.12.1 or above
from alembic.operations import MigrationScript
def run_migrations_online():
# ...
def process_revision_directives(
context: MigrationContext,
revision: str | Iterable[str | None] | Iterable[str],
directives: list[MigrationScript],
):
assert config.cmd_opts is not None
if getattr(config.cmd_opts, 'autogenerate', False):
script = directives[0]
assert script.upgrade_ops is not None
if script.upgrade_ops.is_empty():
directives[:] = []
# connectable = ...
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives
)
with context.begin_transaction():
context.run_migrations().. _cookbook_dont_emit_drop_index:
Don't emit DROP INDEX when the table is to be dropped as well
MySQL may complain when dropping an index that is against a column
that also has a foreign key constraint on it. If the table is to be dropped
in any case, the DROP INDEX isn't necessary. This recipe will process the set
of autogenerate directives such that all :class:.DropIndexOp directives
are removed against tables that themselves are to be dropped::
def run_migrations_online():
# ...
from alembic.operations import ops
def process_revision_directives(context, revision, directives):
script = directives[0]
# process both "def upgrade()", "def downgrade()"
for directive in (script.upgrade_ops, script.downgrade_ops):
# make a set of tables that are being dropped within
# the migration function
tables_dropped = set()
for op in directive.ops:
if isinstance(op, ops.DropTableOp):
tables_dropped.add((op.table_name, op.schema))
# now rewrite the list of "ops" such that DropIndexOp
# is removed for those tables. Needs a recursive function.
directive.ops = list(
_filter_drop_indexes(directive.ops, tables_dropped)
)
def _filter_drop_indexes(directives, tables_dropped):
# given a set of (tablename, schemaname) to be dropped, filter
# out DropIndexOp from the list of directives and yield the result.
for directive in directives:
# ModifyTableOps is a container of ALTER TABLE types of
# commands. process those in place recursively.
if isinstance(directive, ops.ModifyTableOps) and \
(directive.table_name, directive.schema) in tables_dropped:
directive.ops = list(
_filter_drop_indexes(directive.ops, tables_dropped)
)
# if we emptied out the directives, then skip the
# container altogether.
if not directive.ops:
continue
elif isinstance(directive, ops.DropIndexOp) and \
(directive.table_name, directive.schema) in tables_dropped:
# we found a target DropIndexOp. keep looping
continue
# otherwise if not filtered, yield out the directive
yield directive
# connectable = ...
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives
)
with context.begin_transaction():
context.run_migrations()Whereas autogenerate, when dropping two tables with a foreign key and
an index, would previously generate something like::
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_b_aid'), table_name='b')
op.drop_table('b')
op.drop_table('a')
# ### end Alembic commands ###With the above rewriter, it generates as::
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('b')
op.drop_table('a')
# ### end Alembic commands ###Don't generate any DROP TABLE directives with autogenerate
When running autogenerate against a database that has existing tables outside
of the application's autogenerated metadata, it may be desirable to prevent
autogenerate from considering any of those existing tables to be dropped.
This will prevent autogenerate from detecting tables removed from the
local metadata as well however this is only a small caveat.
The most direct way to achieve this using the
:paramref:.EnvironmentContext.configure.include_object hook. There is no
need to hardcode a fixed "whitelist" of table names; the hook gives enough
information in the given arguments to determine if a particular table name is
not part of the local :class:.MetaData being autogenerated, by checking first
that the type of object is "table", then that reflected is True,
indicating this table name is from the local database connection, not the
:class:.MetaData, and finally that compare_to is None, indicating
autogenerate is not comparing this :class:.Table to any :class:.Table in
the local :class:.MetaData collection::
# in env.py
def include_object(object, name, type_, reflected, compare_to):
if type_ == "table" and reflected and compare_to is None:
return False
else:
return True
context.configure(
# ...
include_object = include_object
).. _cookbook_custom_sorting_create_table:
Apply Custom Sorting to Table Columns within CREATE TABLE
This example illustrates use of the :class:.Rewriter object introduced
at :ref:autogen_rewriter. While the rewriter grants access to the
individual :class:.ops.MigrateOperation objects, there are sometimes some
special techniques required to get around some structural limitations that
are present.
One is when trying to reorganize the order of columns in a
table within a :class:.ops.CreateTableOp directive. This directive, when
generated by autogenerate, actually holds onto the original :class:.Table
object as the source of its information, so attempting to reorder the
:attr:.ops.CreateTableOp.columns collection will usually have no effect.
Instead, a new :class:.ops.CreateTableOp object may be constructed with the
new ordering. However, a second issue is that the :class:.Column objects
inside will already be associated with the :class:.Table that is from the
model being autogenerated, meaning they can't be reassigned directly to a new
:class:.Table. To get around this, we can copy all the columns and constraints
using methods like :meth:.Column.copy.
Below we use :class:.Rewriter to create a new :class:.ops.CreateTableOp
directive and to copy the :class:.Column objects from one into another,
copying each column or constraint object and applying a new sorting scheme::
# in env.py
from alembic.operations import ops
from alembic.autogenerate import rewriter
writer = rewriter.Rewriter()
@writer.rewrites(ops.CreateTableOp)
def order_columns(context, revision, op):
special_names = {"id": -100, "created_at": 1001, "updated_at": 1002}
cols_by_key = [
(
special_names.get(col.key, index)
if isinstance(col, Column)
else 2000,
col,
)
for index, col in enumerate(op.columns)
]
columns = [
col for idx, col in sorted(cols_by_key, key=lambda entry: entry[0])
]
return ops.CreateTableOp(
op.table_name, columns, schema=op.schema, **op.kw)
# ...
context.configure(
# ...
process_revision_directives=writer
)Above, when we apply the writer to a table such as::
Table(
"my_table",
m,
Column("data", String(50)),
Column("created_at", DateTime),
Column("id", Integer, primary_key=True),
Column("updated_at", DateTime),
UniqueConstraint("data", name="uq_data")
)This will render in the autogenerated file as::
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"my_table",
sa.Column("id", sa.Integer(), nullable=False),
sa.Column("data", sa.String(length=50), nullable=True),
sa.Column("created_at", sa.DateTime(), nullable=True),
sa.Column("updated_at", sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("data", name="uq_data"),
)
# ### end Alembic commands ###.. _cookbook_add_if_exists:
Add IF [NOT] EXISTS to CREATE/DROP operations
Sometimes CREATE/DROP operations take too long during a production deployment
and it is preferable to apply them offline,
and still keep alembic migrations aligned and/or for test environments.
Using the rewriter makes it possible::
from alembic.operations import ops
from alembic.autogenerate import rewriter
writer = rewriter.Rewriter()
@writer.rewrites(ops.CreateTableOp)
@writer.rewrites(ops.CreateIndexOp)
def add_if_not_exists(context, revision, op):
op.if_not_exists = True
return op
@writer.rewrites(ops.DropTableOp)
@writer.rewrites(ops.DropIndexOp)
def add_if_exists(context, revision, op):
op.if_exists = True
return opSame operation is possible for ADD/DROP COLUMN on postgresql/mariadb::
@writer.rewrites(ops.AddColumnOp)
def add_column_if_not_exists(context, revision, op):
op.if_not_exists = True
return op
@writer.rewrites(ops.DropColumnOp)
def drop_column_if_not_exists(context, revision, op):
op.if_exists = True
return opDon't emit CREATE TABLE statements for Views
It is sometimes convenient to create :class:~sqlalchemy.schema.Table instances for views
so that they can be queried using normal SQLAlchemy techniques. Unfortunately this
causes Alembic to treat them as tables in need of creation and to generate spuriouscreate_table() operations. This is easily fixable by flagging such Tables and using the
:paramref:~.EnvironmentContext.configure.include_object hook to exclude them::
my_view = Table('my_view', metadata, autoload=True, info=dict(is_view=True)) # Flag this as a viewOr, if you use declarative tables::
class MyView(Base):
__tablename__ = 'my_view'
__table_args__ = {'info': {'is_view': True}} # Flag this as a viewThen define include_object as::
def include_object(object, name, type_, reflected, compare_to):
"""
Exclude views from Alembic's consideration.
"""
return not object.info.get('is_view', False)Finally, in env.py pass your include_object as a keyword argument to :meth:.EnvironmentContext.configure.
.. _multiple_environments:
Run Multiple Alembic Environments from one .ini file
Long before Alembic had the "multiple bases" feature described in :ref:multiple_bases,
projects had a need to maintain more than one Alembic version history in a single
project, where these version histories are completely independent of each other
and each refer to their own alembic_version table, either across multiple databases,
schemas, or namespaces. A simple approach was added to support this, the--name flag on the commandline. This flag allows named sections within
the alembic.ini file to be present (but note it does not apply topyproject.toml configuration, where only the [tool.alembic] section
is used).
First, one would create an alembic.ini file of this form::
[DEFAULT]
# all defaults shared between environments go here
sqlalchemy.url = postgresql://scott:tiger@hostname/mydatabase
[schema1]
# path to env.py and migration scripts for schema1
script_location = myproject/revisions/schema1
[schema2]
# path to env.py and migration scripts for schema2
script_location = myproject/revisions/schema2
[schema3]
# path to env.py and migration scripts for schema3
script_location = myproject/revisions/db2
# this schema uses a different database URL as well
sqlalchemy.url = postgresql://scott:tiger@hostname/myotherdatabaseAbove, in the [DEFAULT] section we set up a default database URL.
Then we create three sections corresponding to different revision lineages
in our project. Each of these directories would have its own env.py
and set of versioning files. Then when we run the alembic command,
we simply give it the name of the configuration we want to use::
alembic --name schema2 revision -m "new rev for schema 2" --autogenerateAbove, the alembic command makes use of the configuration in [schema2],
populated with defaults from the [DEFAULT] section.
The above approach can be automated by creating a custom front-end to the
Alembic commandline as well.
Print Python Code to Generate Particular Database Tables
Suppose you have a database already, and want to generate someop.create_table() and other directives that you'd have in a migration file.
How can we automate generating that code?
Suppose the database schema looks like (assume MySQL)::
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL,
KEY `id` (`id`)
);
CREATE TABLE IF NOT EXISTS `user_properties` (
`users_id` int(11) NOT NULL,
`property_name` varchar(255) NOT NULL,
`property_value` mediumtext NOT NULL,
UNIQUE KEY `property_name_users_id` (`property_name`,`users_id`),
KEY `users_id` (`users_id`),
CONSTRAINT `user_properties_ibfk_1` FOREIGN KEY (`users_id`)
REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;Using :class:.ops.UpgradeOps, :class:.ops.CreateTableOp, and
:class:.ops.CreateIndexOp, we create a migration file structure,
using :class:.Table objects that we get from SQLAlchemy reflection.
The structure is passed to :func:.autogenerate.render_python_code to
produce the Python code for a migration file::
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Table
from alembic import autogenerate
from alembic.operations import ops
e = create_engine("mysql://scott:tiger@localhost/test")
with e.connect() as conn:
m = MetaData()
user_table = Table('users', m, autoload_with=conn)
user_property_table = Table('user_properties', m, autoload_with=conn)
print(autogenerate.render_python_code(
ops.UpgradeOps(
ops=[
ops.CreateTableOp.from_table(table) for table in m.tables.values()
] + [
ops.CreateIndexOp.from_index(idx) for table in m.tables.values()
for idx in table.indexes
]
))
)Output::
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('users',
sa.Column('id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
mysql_default_charset='latin1',
mysql_engine='InnoDB'
)
op.create_table('user_properties',
sa.Column('users_id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
sa.Column('property_name', mysql.VARCHAR(length=255), nullable=False),
sa.Column('property_value', mysql.MEDIUMTEXT(), nullable=False),
sa.ForeignKeyConstraint(['users_id'], ['users.id'], name='user_properties_ibfk_1', ondelete='CASCADE'),
mysql_comment='user properties',
mysql_default_charset='utf8',
mysql_engine='InnoDB'
)
op.create_index('id', 'users', ['id'], unique=False)
op.create_index('users_id', 'user_properties', ['users_id'], unique=False)
op.create_index('property_name_users_id', 'user_properties', ['property_name', 'users_id'], unique=True)
# ### end Alembic commands ###Run Alembic Operation Objects Directly (as in from autogenerate)
The :class:.Operations object has a method known as
:meth:.Operations.invoke that will generically invoke a particular operation
object. We can therefore use the :func:.autogenerate.produce_migrations
function to run an autogenerate comparison, get back a
:class:.ops.MigrationScript structure representing the changes, and with a
little bit of insider information we can invoke them directly.
The traversal through the :class:.ops.MigrationScript structure is as
follows::
use_batch = engine.name == "sqlite"
stack = [migrations.upgrade_ops]
while stack:
elem = stack.pop(0)
if use_batch and isinstance(elem, ModifyTableOps):
with operations.batch_alter_table(
elem.table_name, schema=elem.schema
) as batch_ops:
for table_elem in elem.ops:
batch_ops.invoke(table_elem)
elif hasattr(elem, "ops"):
stack.extend(elem.ops)
else:
operations.invoke(elem)Above, we detect elements that have a collection of operations by looking
for the .ops attribute. A check for :class:.ModifyTableOps allows
us to use a batch context if we are supporting that.
A full example follows. The overall setup here is copied from the example
at :func:.autogenerate.compare_metadata::
from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import MetaData
from sqlalchemy import String
from sqlalchemy import Table
from alembic.autogenerate import produce_migrations
from alembic.migration import MigrationContext
from alembic.operations import Operations
from alembic.operations.ops import ModifyTableOps
engine = create_engine("sqlite://", echo=True)
with engine.connect() as conn:
conn.execute(
"""
create table foo (
id integer not null primary key,
old_data varchar(50),
x integer
)"""
)
conn.execute(
"""
create table bar (
data varchar(50)
)"""
)
metadata = MetaData()
Table(
"foo",
metadata,
Column("id", Integer, primary_key=True),
Column("data", Integer),
Column("x", Integer, nullable=False),
)
Table("bat", metadata, Column("info", String(100)))
mc = MigrationContext.configure(engine.connect())
migrations = produce_migrations(mc, metadata)
operations = Operations(mc)
use_batch = engine.name == "sqlite"
stack = [migrations.upgrade_ops]
while stack:
elem = stack.pop(0)
if use_batch and isinstance(elem, ModifyTableOps):
with operations.batch_alter_table(
elem.table_name, schema=elem.schema
) as batch_ops:
for table_elem in elem.ops:
batch_ops.invoke(table_elem)
elif hasattr(elem, "ops"):
stack.extend(elem.ops)
else:
operations.invoke(elem).. _cookbook_check_heads:
Test current database revision is at head(s)
.. versionchanged:: 1.17.1 This recipe is now part of the alembic current
command using the :paramref:.command.current.check_heads parameter,
available from the command line as --check-heads:
.. sourcecode::
alembic current --check-heads
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [alembic.util.messaging] Database is not on all head revisions
FAILED: Database is not on all head revisionsA recipe to determine if a database schema is up to date in terms of applying
Alembic migrations. May be useful for test or installation suites to
determine if the target database is up to date. Makes use of the
:meth:.MigrationContext.get_current_heads as well as
:meth:.ScriptDirectory.get_heads methods so that it accommodates for a
branched revision tree::
from alembic import config, script
from alembic.runtime import migration
from sqlalchemy import engine
def check_current_head(alembic_cfg, connectable):
# type: (config.Config, engine.Engine) -> bool
directory = script.ScriptDirectory.from_config(alembic_cfg)
with connectable.begin() as connection:
context = migration.MigrationContext.configure(connection)
return set(context.get_current_heads()) == set(directory.get_heads())
e = engine.create_engine("mysql://scott:tiger@localhost/test", echo=True)
cfg = config.Config("alembic.ini")
print(check_current_head(cfg, e)).. seealso::
:meth:`.MigrationContext.get_current_heads`
:meth:`.ScriptDirectory.get_heads`.. _asyncio_recipe:
Using Asyncio with Alembic
SQLAlchemy version 1.4 introduced experimental support for asyncio, allowing
use of most of its interface from async applications. Alembic currently does
not provide an async api directly, but it can use an use SQLAlchemy Async
engine to run the migrations and autogenerate.
New configurations can use the template "async" or "pyproject_async" to bootstrap an environment which
can be used with async DBAPI like asyncpg, running the command::
alembic init -t async <script_directory_here>Existing configurations can be updated to use an async DBAPI by updating the env.py
file that's used by Alembic to start its operations. In particular onlyrun_migrations_online will need to be updated to be something like the example below::
import asyncio
from sqlalchemy.ext.asyncio import async_engine_from_config
# ... no change required to the rest of the code
def do_run_migrations(connection):
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations():
"""In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = async_engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online():
"""Run migrations in 'online' mode."""
asyncio.run(run_async_migrations())An async application can also interact with the Alembic api directly by using
the SQLAlchemy run_sync method to adapt the non-async api of Alembic to
an async consumer.
.. _connection_sharing_plus_asyncio:
Programmatic API use (connection sharing) With Asyncio
Combining the examples of :ref:connection_sharing with :ref:asyncio_recipe
together, the env.py listed above can be updated as follows works::
def run_migrations_online():
"""Run migrations in 'online' mode.
"""
connectable = config.attributes.get("connection", None)
if connectable is None:
asyncio.run(run_async_migrations())
else:
do_run_migrations(connectable)Above, using an asyncio database URL in alembic.ini one can run
commands such as alembic upgrade from the command line. Programmatically,
the same env.py file can be invoked using asyncio as::
import asyncio
from sqlalchemy.ext.asyncio import create_async_engine
from alembic import command, config
def run_upgrade(connection, cfg):
cfg.attributes["connection"] = connection
command.upgrade(cfg, "head")
async def run_async_upgrade():
async_engine = create_async_engine("sqlite+aiosqlite://", echo=True)
async with async_engine.begin() as conn:
await conn.run_sync(run_upgrade, config.Config("alembic.ini"))
asyncio.run(run_async_upgrade())Data Migrations - General Techniques
Alembic migrations are designed for schema migrations.
The nature of data migrations are inherently different and it's not in fact advisable
in the general case to write data migrations that integrate with Alembic's
schema versioning model.
For example downgrades are difficult to address since they might require deletion of data,
which may even not be possible to detect.
.. warning::
The solution needs to be designed specifically for each individual application and migration.
There are no general rules and the following text is only a recommendation based on experience.
There are three basic approaches for the data migrations.
Small data
Small data migrations are easy to perform, especially in cases of initial data to a new table.
These can be handled using :meth:.Operations.bulk_insert.
Separate migration script
One possibility is a completely separate script aside of alembic migrations.
The complete migration is then processed in following steps:
- Run the initial alembic migrations (new columns etc.)
- Run the separate data migration script
- Run the final alembic migrations (database constraints, delete columns etc.)
The data migration script may also need a separate ORM model to handle intermediate state of the database.
Online migration
The application maintains a version of schema with both versions.
Writes are performed on both places, while the background script move all the remaining data across.
This technique is very challenging and time demanding, since it requires custom application logic to
handle the intermediate states.
.. _custom_commandline:
Extend ``CommandLine`` with custom commands
.. versionadded:: 1.15.3
While Alembic does not have a plugin system that would allow transparently extending the original alembic CLI
with additional commands, it is possible to create your own instance of :class:.CommandLine and extend that via
:meth:.CommandLine.register_command.
.. code-block:: python
# myalembic.py
from alembic.config import CommandLine, Config
def frobnicate(config: Config, revision: str) -> None:
"""Frobnicates according to the frobnication specification.
:param config: a :class:`.Config` instance
:param revision: the revision to frobnicate
"""
config.print_stdout(f"Revision {revision} successfully frobnicated.")
def main():
cli = CommandLine()
cli.register_command(frobnicate)
cli.main()
if __name__ == "__main__":
main()Any named function may be registered as a command, provided it accepts a :class:.Config object as the first argument;
a docstring is also recommended as it will show up in the help output of the CLI.
.. code-block::
$ python -m myalembic -h
...
positional arguments:
{branches,check,current,downgrade,edit,ensure_version,heads,history,init,list_templates,merge,revision,show,stamp,upgrade,frobnicate}
...
frobnicate Frobnicates according to the frobnication specification.
$ python -m myalembic frobnicate -h
usage: myalembic.py frobnicate [-h] revision
positional arguments:
revision revision identifier
optional arguments:
-h, --help show this help message and exit
$ python -m myalembic frobnicate 42
Revision 42 successfully frobnicated.Build Front (`docs/build/front.rst`)
============ Front Matter
Information about the Alembic project.
Project Homepage
Alembic is hosted on GitHub at https://github.com/sqlalchemy/alembic under the SQLAlchemy organization.
Releases and project status are available on Pypi at https://pypi.python.org/pypi/alembic.
The most recent published version of this documentation should be at https://alembic.sqlalchemy.org.
.. _installation:
Installation
While Alembic can be installed system wide, it's more common that it's
installed local to a virtual environment <https://docs.python.org/3/tutorial/venv.html>_ , as it also uses libraries
such as SQLAlchemy and database drivers that are more appropriate for
local installations.
The documentation below is only one kind of approach to installing Alembic
for a project; there are many such approaches. The documentation below is
provided only for those users who otherwise have no specific project setup
chosen.
To build a virtual environment for a specific project, a virtual environment
can be created using thePython venv library <https://docs.python.org/3/library/venv.html>_::
$ cd /path/to/your/project
$ python -m venv .venvThere is now a Python interpreter that you can access in/path/to/your/project/.venv/bin/python, as well as the pip <http://pypi.python.org/pypi/pip>_ installer tool in/path/to/your/project/.venv/bin/pip.
Next,the activate command installed by venv can be used so that
all binaries local to this new Python environment are in the local path::
$ source /path/to/your/project/.venv/bin/activateWe now install Alembic as follows::
$ pip install alembicThe install will add the alembic command to the virtual environment. All
operations with Alembic in terms of this specific virtual environment will then
proceed through the usage of this command, as in::
$ alembic init alembicFinally, assuming your project is itself installable, meaning it has apyproject.toml file, and/or setup.py script, the local project can
be made a part of the same local environment by installing it with pip,
optionally using "editable" mode::
$ pip install -e .Dependencies
Alembic's install process will ensure that SQLAlchemy_
is installed, in addition to other dependencies. Alembic will work with
SQLAlchemy as of version 2.0.0.
.. versionchanged:: 1.20.0 Support for SQLAlchemy older than 2.0.0 was dropped.
Alembic supports Python versions 3.10 and above
.. versionchanged:: 1.17 Alembic now supports Python 3.10 and newer.
.. _versioning_scheme:
Versioning Scheme
Alembic's versioning scheme is based on that ofSQLAlchemy's versioning scheme <https://www.sqlalchemy.org/download.html#versions>_.
In particular, it should be noted that while Alembic uses a three-number
versioning scheme, it does not use SemVer. In SQLAlchemy and Alembic's
scheme, the middle digit is considered to be a "Significant Minor Release",
which may include removal of previously deprecated APIs with some risk of
non-backwards compatibility in a very small number of cases.
This means that version "1.8.0", "1.9.0", "1.10.0", "1.11.0", etc. are
Significant Minor Releases, which will include new API features and may
remove or modify existing ones.
Therefore, when pinning <https://pip.pypa.io/en/stable/topics/repeatable-installs/>_
Alembic releases, pin to the "major" and "minor" digits to avoid API changes.
A true "Major" release such as a change to "2.0" would include complete
redesigns/re-architectures of foundational features; currently no such series
of changes are planned, although changes such as replacing the entire
"autogenerate" scheme with a new approach would qualify for that level of
change.
Community
Alembic is developed by Mike Bayer <http://techspot.zzzeek.org>, and is
part of the SQLAlchemy project.
User issues, discussion of potential bugs and features are most easily
discussed using GitHub Discussions <https://github.com/sqlalchemy/alembic/discussions/>_.
.. _bugs:
Bugs
Bugs and feature enhancements to Alembic should be reported on the GitHub issue tracker <https://github.com/sqlalchemy/alembic/issues/>_.
.. _SQLAlchemy: https://www.sqlalchemy.org
Build Index (`docs/build/index.rst`)
=================================== Welcome to Alembic's documentation!
Alembic <https://alembic.sqlalchemy.org>_ is a lightweight database migration tool for usage
with the SQLAlchemy <https://www.sqlalchemy.org>_ Database Toolkit for Python.
.. toctree::
:maxdepth: 3
front
tutorial
autogenerate
offline
naming
batch
branches
ops
cookbook
api/index
changelog
Indices and tables
- :ref:
genindex - :ref:
modindex - :ref:
search
Build Naming (`docs/build/naming.rst`)
.. _tutorial_constraint_names:
The Importance of Naming Constraints
An important topic worth mentioning is that of constraint naming conventions.
As we've proceeded here, we've talked about adding tables and columns, and
we've also hinted at lots of other operations listed in :ref:ops such as those
which support adding or dropping constraints like foreign keys and unique
constraints. The way these constraints are referred to in migration scripts
is by name, however these names by default are in most cases generated by
the relational database in use, when the constraint is created. For example,
if you emitted two CREATE TABLE statements like this on Postgresql::
test=> CREATE TABLE user_account (id INTEGER PRIMARY KEY);
CREATE TABLE
test=> CREATE TABLE user_order (
test(> id INTEGER PRIMARY KEY,
test(> user_account_id INTEGER REFERENCES user_account(id));
CREATE TABLE
Suppose we wanted to DROP the REFERENCES that we just applied to theuser_order.user_account_id column, how do we do that? At the prompt,
we'd use ALTER TABLE DROP CONSTRAINT <constraint_name>, or if
using Alembic we'd be using :meth:.Operations.drop_constraint. But both
of those functions need a name - what's the name of this constraint?
It does have a name, which in this case we can figure out by looking at the
Postgresql catalog tables::
test=> SELECT r.conname FROM
test-> pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
test-> JOIN pg_catalog.pg_constraint r ON c.oid = r.conrelid
test-> WHERE c.relname='user_order' AND r.contype = 'f'
test-> ;
conname
user_order_user_account_id_fkey
(1 row)
The name above is not something that Alembic or SQLAlchemy created;user_order_user_account_id_fkey is a naming scheme used internally by
Postgresql to name constraints that are otherwise not named.
This scheme doesn't seem so complicated, and we might want to just use our
knowledge of it so that we know what name to use for our
:meth:.Operations.drop_constraint call. But is that a good idea? What
if for example we needed our code to run on Oracle as well. OK, certainly
Oracle uses this same scheme, right? Or if not, something similar. Let's
check::
Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production
SQL> CREATE TABLE user_account (id INTEGER PRIMARY KEY);
Table created.
SQL> CREATE TABLE user_order (
2 id INTEGER PRIMARY KEY,
3 user_account_id INTEGER REFERENCES user_account(id));
Table created.
SQL> SELECT constraint_name FROM all_constraints WHERE
2 table_name='USER_ORDER' AND constraint_type in ('R');
CONSTRAINT_NAME
SYS_C0029334
Oh, we can see that is.....much worse. Oracle's names are entirely unpredictable
alphanumeric codes, and this will make being able to write migrations
quite tedious, as we'd need to look up all these names.
The solution to having to look up names is to make your own names. This is
an easy, though tedious thing to do manually. For example, to create our model
in SQLAlchemy ensuring we use names for foreign key constraints would look like::
from sqlalchemy import MetaData, Table, Column, Integer, ForeignKey
meta = MetaData()
user_account = Table('user_account', meta,
Column('id', Integer, primary_key=True)
)
user_order = Table('user_order', meta,
Column('id', Integer, primary_key=True),
Column('user_order_id', Integer,
ForeignKey('user_account.id', name='fk_user_order_id'))
)
Simple enough, though this has some disadvantages. The first is that it's tedious;
we need to remember to use a name for every :class:~sqlalchemy.schema.ForeignKey object,
not to mention every :class:~sqlalchemy.schema.UniqueConstraint, :class:~sqlalchemy.schema.CheckConstraint,
:class:~sqlalchemy.schema.Index, and maybe even :class:~sqlalchemy.schema.PrimaryKeyConstraint
as well if we wish to be able to alter those too, and beyond all that, all the
names have to be globally unique. Even with all that effort, if we have a naming scheme in mind,
it's easy to get it wrong when doing it manually each time.
What's worse is that manually naming constraints (and indexes) gets even more
tedious in that we can no longer use convenience features such as the .unique=True
or .index=True flag on :class:~sqlalchemy.schema.Column::
user_account = Table('user_account', meta,
Column('id', Integer, primary_key=True),
Column('name', String(50), unique=True)
)
Above, the unique=True flag creates a :class:~sqlalchemy.schema.UniqueConstraint, but again,
it's not named. If we want to name it, manually we have to forego the usage
of unique=True and type out the whole constraint::
user_account = Table('user_account', meta,
Column('id', Integer, primary_key=True),
Column('name', String(50)),
UniqueConstraint('name', name='uq_user_account_name')
)
There's a solution to all this naming work, which is to use an automated
naming convention. For some years, SQLAlchemy has encouraged the use of
DDL Events in order to create naming schemes. The :meth:~sqlalchemy.events.DDLEvents.after_parent_attach
event in particular is the best place to intercept when :class:~sqlalchemy.schema.Constraint
and :class:~sqlalchemy.schema.Index objects are being associated with a parent
:class:~sqlalchemy.schema.Table object, and to assign a .name to the constraint while making
use of the name of the table and associated columns.
But there is also a better way to go, which is to make use of a feature
new in SQLAlchemy 0.9.2 which makes use of the events behind the scenes known as
:paramref:~sqlalchemy.schema.MetaData.naming_convention. Here, we can
create a new :class:~sqlalchemy.schema.MetaData object while passing a dictionary referring
to a naming scheme::
convention = {
"ix": "ix_%(column_0_label)s",
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(constraint_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
metadata = MetaData(naming_convention=convention)If we define our models using a :class:~sqlalchemy.schema.MetaData as above, the given
naming convention dictionary will be used to provide names for all constraints
and indexes.
.. seealso::
:ref:`sqla:constraint_naming_conventions` - SQLAlchemy overview of naming
convention support.. _autogen_naming_conventions:
Integration of Naming Conventions into Operations, Autogenerate
As of Alembic 0.6.4, the naming convention feature is integrated into the
:class:.Operations object, so that the convention takes effect for any
constraint that is otherwise unnamed. The naming convention is passed to
:class:.Operations using the :paramref:.MigrationsContext.configure.target_metadata
parameter in env.py, which is normally configured when autogenerate is
used::
# in your application's model:
from sqlalchemy import MetaData
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
metadata = MetaData(naming_convention={
"ix": "ix_%(column_0_label)s",
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_`%(constraint_name)s`",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
})
# .. in your Alembic env.py:
# add your model's MetaData object here
# for 'autogenerate' support
from myapp import mymodel
target_metadata = mymodel.Base.metadata
# ...
def run_migrations_online():
# ...
context.configure(
connection=connection,
target_metadata=target_metadata
)Above, when we render a directive like the following::
op.add_column('sometable', Column('q', Boolean(name='q_bool')))The Boolean type will render a CHECK constraint with the name"ck_sometable_q_bool", assuming the backend in use does not support
native boolean types.
We can also use op directives with constraints and not give them a name
at all, if the naming convention doesn't require one. The value ofNone will be converted into a name that follows the appropriate naming
conventions::
def upgrade():
op.create_unique_constraint(None, 'some_table', 'x')When autogenerate renders constraints in a migration script, it renders them
typically with their completed name. If using at least Alembic 0.6.4 as well
as SQLAlchemy 0.9.4, these will be rendered with a special directive
:meth:.Operations.f which denotes that the string has already been
tokenized::
def upgrade():
op.create_unique_constraint(op.f('uq_const_x'), 'some_table', 'x')This :meth:.Operations.f construct may be used explicitly in order to
bypass naming conventions, as illustrated in the next section.
Bypassing the Naming Convention for CREATE and DROP Operations
When using constraint naming conventions, in particular if the%(constraint_name)s token is in use, the constraint name used with
methods such as :meth:.Operations.create_check_constraint and :meth:.Operations.drop_constraint
with a matching :paramref:.Operations.type_ will include the naming convention
unless additional directives are in use.
Given a configuration in env.py as::
target_metadata = MetaData(naming_convention={
"ix": "ix_%(column_0_label)s",
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_`%(constraint_name)s`",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
})
# ...
def run_migrations_online():
# ...
context.configure(
connection=connection,
target_metadata=target_metadata
)The following operation will drop a CHECK constraint namedck_t1_some_check_const::
op.drop_constraint("some_check_const", "t1", type_="check")
ALTER TABLE t1 DROP CONSTRAINT ck_t1_some_check_const
In order to apply the operation while bypassing the configured naming
convention, use the :meth:.Operations.f construct. This produces a string
expression that will not be tokenized::
op.drop_constraint(op.f("some_check_const"), "t1", type_="check")
ALTER TABLE t1 DROP CONSTRAINT some_check_const
Build Offline (`docs/build/offline.rst`)
Generating SQL Scripts (a.k.a. "Offline Mode")
A major capability of Alembic is to generate migrations as SQL scripts, instead of running
them against the database - this is also referred to as offline mode.
This is a critical feature when working in large organizations
where access to DDL is restricted, and SQL scripts must be handed off to DBAs. Alembic makes
this easy via the --sql option passed to any upgrade or downgrade command. We
can, for example, generate a script that revises up to rev ae1027a6acf::
$ alembic upgrade ae1027a6acf --sql
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
BEGIN;
CREATE TABLE alembic_version (
version_num VARCHAR(32) NOT NULL
);
INFO [alembic.context] Running upgrade None -> 1975ea83b712
CREATE TABLE account (
id SERIAL NOT NULL,
name VARCHAR(50) NOT NULL,
description VARCHAR(200),
PRIMARY KEY (id)
);
INFO [alembic.context] Running upgrade 1975ea83b712 -> ae1027a6acf
ALTER TABLE account ADD COLUMN last_transaction_date TIMESTAMP WITHOUT TIME ZONE;
INSERT INTO alembic_version (version_num) VALUES ('ae1027a6acf');
COMMIT;While the logging configuration dumped to standard error, the actual script was dumped to standard output -
so in the absence of further configuration (described later in this section), we'd at first be using output
redirection to generate a script::
$ alembic upgrade ae1027a6acf --sql > migration.sqlGetting the Start Version
Notice that our migration script started at the base - this is the default when using offline
mode, as no database connection is present and there's no alembic_version table to read from.
One way to provide a starting version in offline mode is to provide a range to the command line.
This is accomplished by providing the "version" in start:end syntax::
$ alembic upgrade 1975ea83b712:ae1027a6acf --sql > migration.sqlThe start:end syntax is only allowed in offline mode; in "online" mode, the alembic_version
table is always used to get at the current version.
It's also possible to have the env.py script retrieve the "last" version from
the local environment, such as from a local file. A scheme like this would basically
treat a local file in the same way alembic_version works::
if context.is_offline_mode():
version_file = os.path.join(os.path.dirname(config.config_file_name), "version.txt")
if os.path.exists(version_file):
current_version = open(version_file).read()
else:
current_version = None
context.configure(dialect_name=engine.name, starting_rev=current_version)
context.run_migrations()
end_version = context.get_revision_argument()
if end_version and end_version != current_version:
open(version_file, 'w').write(end_version)Writing Migration Scripts to Support Script Generation
The challenge of SQL script generation is that the scripts we generate can't rely upon
any client/server database access. This means a migration script that pulls some rows
into memory via a SELECT statement will not work in --sql mode. It's also
important that the Alembic directives, all of which are designed specifically to work
in both "live execution" as well as "offline SQL generation" mode, are used.
Customizing the Environment
Users of the --sql option are encouraged to hack their env.py files to suit their
needs. The env.py script as provided is broken into two sections: run_migrations_online()
and run_migrations_offline(). Which function is run is determined at the bottom of the
script by reading :meth:.EnvironmentContext.is_offline_mode, which basically determines if the--sql flag was enabled.
For example, a multiple database configuration may want to run through each
database and set the output of the migrations to different named files - the :meth:.EnvironmentContext.configure
function accepts a parameter output_buffer for this purpose. Below we illustrate
this within the run_migrations_offline() function::
from alembic import context
import myapp
import sys
db_1 = myapp.db_1
db_2 = myapp.db_2
def run_migrations_offline():
"""Run migrations *without* a SQL connection."""
for name, engine, file_ in [
("db1", db_1, "db1.sql"),
("db2", db_2, "db2.sql"),
]:
context.configure(
url=engine.url,
transactional_ddl=False,
output_buffer=open(file_, 'w'))
context.execute("-- running migrations for '%s'" % name)
context.run_migrations(name=name)
sys.stderr.write("Wrote file '%s'" % file_)
def run_migrations_online():
"""Run migrations *with* a SQL connection."""
for name, engine in [
("db1", db_1),
("db2", db_2),
]:
connection = engine.connect()
context.configure(connection=connection)
try:
context.run_migrations(name=name)
session.commit()
except:
session.rollback()
raise
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()Build Ops (`docs/build/ops.rst`)
.. _ops:
=================== Operation Reference
This file provides documentation on Alembic migration directives.
The directives here are used within user-defined migration files,
within the upgrade() and downgrade() functions, as well as
any functions further invoked by those.
All directives exist as methods on a class called :class:.Operations.
When migration scripts are run, this object is made available
to the script via the alembic.op datamember, which is
a proxy to an actual instance of :class:.Operations.
Currently, alembic.op is a real Python module, populated
with individual proxies for each method on :class:.Operations,
so symbols can be imported safely from the alembic.op namespace.
The :class:.Operations system is also fully extensible. See
:ref:operation_plugins for details on this.
A key design philosophy to the :ref:alembic.operations.toplevel methods is that
to the greatest degree possible, they internally generate the
appropriate SQLAlchemy metadata, typically involving
:class:~sqlalchemy.schema.Table and :class:~sqlalchemy.schema.Constraint
objects. This so that migration instructions can be
given in terms of just the string names and/or flags involved.
The exceptions to this
rule include the :meth:~.Operations.add_column and :meth:~.Operations.create_table
directives, which require full :class:~sqlalchemy.schema.Column
objects, though the table metadata is still generated here.
The functions here all require that a :class:.MigrationContext has been
configured within the env.py script first, which is typically
via :meth:.EnvironmentContext.configure. Under normal
circumstances they are called from an actual migration script, which
itself would be invoked by the :meth:.EnvironmentContext.run_migrations
method.
.. module:: alembic.operations
.. class:: AbstractOperations
Base class for :class:`.Operations` and :class:`.BatchOperations`.
See :class:`.Operations` for full list of members.. autoclass:: Operations
:members:
:inherited-members:
.. autoclass:: BatchOperations
:members:
Build Requirements (`docs/build/requirements.txt`)
git+https://github.com/sqlalchemyorg/changelog.git#egg=changelog
git+https://github.com/sqlalchemyorg/sphinx-paramlinks.git#egg=sphinx-paramlinks
git+https://github.com/sqlalchemy/sqlalchemy.git
backports.zoneinfo;python_version<"3.9"
because there's a dependency in pyfiles.py
Mako
importlib-metadata;python_version<"3.9"
importlib-resources;python_version<"3.9"
sphinx_copybutton==0.5.1
sphinx-book-theme
Build Tutorial (`docs/build/tutorial.rst`)
======== Tutorial
Alembic provides for the creation, management, and invocation of change management
scripts for a relational database, using SQLAlchemy as the underlying engine.
This tutorial will provide a full introduction to the theory and usage of this tool.
To begin, make sure Alembic is installed; a common way to install within a
local virtual environment is described at :ref:installation.
As illustrated in that chapter, it is useful to have Alembic
installed in the same module / Python path as that of the target project,
usually using a Python virtual environment <https://docs.python.org/3/tutorial/venv.html>_, so that when the alembic
command is run, the Python script which is invoked by alembic, namely your
project's env.py script, will have access to your application's models.
This is not strictly necessary, however is usually preferred.
The tutorial below assumes the alembic command line utility is present in
the local path and when invoked, will have access to the same Python module
environment as that of the target project.
The Migration Environment
Usage of Alembic starts with creation of the Migration Environment. This is a directory of scripts
that is specific to a particular application. The migration environment is created just once,
and is then maintained along with the application's source code itself. The environment is
created using the init command of Alembic, and is then customizable to suit the specific
needs of the application.
The structure of this environment, including some generated migration scripts, looks like::
yourproject/
alembic.ini
pyproject.toml
alembic/
env.py
README
script.py.mako
versions/
3512b954651e_add_account.py
2b1ae634e5cd_add_order_id.py
3adcc9a56557_rename_username_field.pyThe directory includes these directories/files:
alembic.ini- this is Alembic's main configuration file which is generated by all templates.
A detailed walkthrough of this file is later in the section :ref:tutorial_alembic_ini.pyproject.toml- most modern Python projects have apyproject.tomlfile. Alembic may
optionally store project related configuration in this file as well; to use apyproject.toml
configuration, see the section :ref:using_pep_621.yourproject- this is the root of your application's source code, or some directory within it.alembic- this directory lives within your application's source tree and is the home of the
migration environment. It can be named anything, and a project that uses multiple databases
may even have more than one.env.py- This is a Python script that is run whenever the alembic migration tool is invoked.
At the very least, it contains instructions to configure and generate a SQLAlchemy engine,
procure a connection from that engine along with a transaction, and then invoke the migration
engine, using the connection as a source of database connectivity.The
env.pyscript is part of the generated environment so that the way migrations run
is entirely customizable. The exact specifics of how to connect are here, as well as
the specifics of how the migration environment are invoked. The script can be modified
so that multiple engines can be operated upon, custom arguments can be passed into the
migration environment, application-specific libraries and models can be loaded in and
made available.Alembic includes a set of initialization templates which feature different varieties
ofenv.pyfor different use cases.README- included with the various environment templates, should have something
informative.script.py.mako- This is aMako <http://www.makotemplates.org>_ template file which
is used to generate new migration scripts. Whatever is here is used to generate new
files withinversions/. This is scriptable so that the structure of each migration
file can be controlled, including standard imports to be within each, as well as
changes to the structure of theupgrade()anddowngrade()functions. For example,
themultidbenvironment allows for multiple functions to be generated using a
naming schemeupgrade_engine1(),upgrade_engine2().versions/- This directory holds the individual version scripts. Users of other migration
tools may notice that the files here don't use ascending integers, and instead use a
partial GUID approach. In Alembic, the ordering of version scripts is relative
to directives within the scripts themselves, and it is theoretically possible to "splice" version files
in between others, allowing migration sequences from different branches to be merged,
albeit carefully by hand.
Creating an Environment
With a basic understanding of what the environment is, we can create one using alembic init.
This will create an environment using the "generic" template::
$ cd /path/to/yourproject
$ source /path/to/yourproject/.venv/bin/activate # assuming a local virtualenv
$ alembic init alembicWhere above, the init command was called to generate a migrations directory called alembic::
Creating directory /path/to/yourproject/alembic...done
Creating directory /path/to/yourproject/alembic/versions...done
Generating /path/to/yourproject/alembic.ini...done
Generating /path/to/yourproject/alembic/env.py...done
Generating /path/to/yourproject/alembic/README...done
Generating /path/to/yourproject/alembic/script.py.mako...done
Please edit configuration/connection/logging settings in
'/path/to/yourproject/alembic.ini' before proceeding.The above layout is produced using a layout template called generic.
Alembic also includes other environment templates. These can be listed out
using the list_templates command::
$ alembic list_templates
Available templates:
generic - Generic single-database configuration.
pyproject - pep-621 compliant configuration that includes pyproject.toml
async - Generic single-database configuration with an async dbapi.
multidb - Rudimentary multi-database configuration.
Templates are used via the 'init' command, e.g.:
alembic init --template generic ./scripts.. versionchanged:: 1.16.0 A new pyproject template has been added. See
the section :ref:using_pep_621 for background.
.. _tutorial_alembic_ini:
Editing the .ini File
Alembic placed a file alembic.ini into the current directory. Alembic looks
in the current directory for this file when any other commands are run; to
indicate an alternative location, the --config option may be used, or theALEMBIC_CONFIG environment variable may be set.
.. tip::
The file generated with the ``generic`` configuration template contains all directives
for both source code configuration as well as database configuration. When using
the ``pyproject`` template, the source code configuration elements will instead
be in a separate ``pyproject.toml`` file, described in the section :ref:`using_pep_621`.The all-in-one .ini file created by generic is illustrated below::
# A generic, single database configuration.
[alembic]
# path to migration scripts.
# this is typically a path given in POSIX (e.g. forward slashes)
# format, relative to the token %(here)s which refers to the location of this
# ini file
script_location = %(here)s/alembic
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# Or organize into date-based subdirectories (requires recursive_version_locations = true)
# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library and tzdata library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to <script_location>/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# the special token `%(here)s` is available which indicates the absolute path
# to this configuration file.
#
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
# path_separator (New in Alembic 1.16.0, supersedes version_path_separator);
# This indicates what character is used to
# split lists of file paths, including version_locations and prepend_sys_path
# within configparser files such as alembic.ini.
#
# The default rendered in new alembic.ini files is "os", which uses os.pathsep
# to provide os-dependent path splitting.
#
# Note that in order to support legacy alembic.ini files, this default does NOT
# take place if path_separator is not present in alembic.ini. If this
# option is omitted entirely, fallback logic is as follows:
#
# 1. Parsing of the version_locations option falls back to using the legacy
# "version_path_separator" key, which if absent then falls back to the legacy
# behavior of splitting on spaces and/or commas.
# 2. Parsing of the prepend_sys_path option falls back to the legacy
# behavior of splitting on spaces, commas, or colons.
#
# Valid values for path_separator are:
#
# path_separator = :
# path_separator = ;
# path_separator = space
# path_separator = newline
#
# Use os.pathsep. Default configuration used for new projects.
path_separator = os
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# database URL. This is consumed by the user-maintained env.py script only.
# other means of configuring database URLs may be customized within the env.py
# file.
# See notes in "escaping characters in ini files" for guidelines on
# passwords
sqlalchemy.url = driver://user:pass@localhost/dbname
# [post_write_hooks]
# This section defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner,
# against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
# hooks = ruff
# ruff.type = module
# ruff.module = ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
# Alternatively, use the exec runner to execute a binary found on your PATH
# hooks = ruff
# ruff.type = exec
# ruff.executable = ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
# Logging configuration. This is also consumed by the user-maintained
# env.py script only.
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%SThe alembic.ini file is consumed by Alembic using Python'sconfigparser.ConfigParser <https://docs.python.org/3/library/configparser.html#configparser.ConfigParser>_
library. The %(here)s variable is
provided as a substitution which is populated with the absolute path to thealembic.ini file itself. This can be used to produce correct pathnames
to directories and files relative to where the config file is located.
.. tip:: Percent signs in alembic.ini configuration variables that are
not part of an interpolation token like %(here)s, including percent
signs that are part of the SQLAlchemy database URL for its own URL-escaping
requirements, must themselves be escaped.
See the section :ref:escaping_percent_signs for more information.
This file contains the following features:
[alembic]- this is the section read by Alembic to determine configuration. Alembic's
core implementation does not directly read any other areas of the file, not
including additional directives that may be consumed from the
end-user-customizableenv.pyfile (see note below). The name "alembic"
(for configparser config only, notpyproject.toml)
can be customized using the--namecommandline flag; see
:ref:multiple_environmentsfor a basic example of this... note:: The default
env.pyfile included with Alembic's environment
templates will also read from the logging sections[logging],
[handlers]etc. If the configuration file in use does not contain
logging directives, please remove thefileConfig()directive within
the generatedenv.pyfile to prevent it from attempting to configure
logging.script_location- this is the location of the Alembic environment. It is normally
specified as a filesystem location relative to the%(here)stoken, which
indicates where the config file itself is located. The location may also
be a plain relative path, where it's interpreted as relative to the current directory,
or an absolute path.This is the only key required by Alembic in all cases. The generation
of the .ini file by the commandalembic init alembicautomatically placed the
directory namealembichere. The special variable%(here)scan also be used,
as in%(here)s/alembic.For support of applications that package themselves into .egg files, the value can
also be specified as apackage resource <https://setuptools.readthedocs.io/en/latest/pkg_resources.html>_, in which
caseresource_filename()is used to find the file (new in 0.2.2). Any non-absolute
URI which contains colons is interpreted here as a resource name, rather than
a straight filename.file_template- this is the naming scheme used to generate new migration
files. Uncomment the presented value if you would like the migration files to
be prepended with date and time, so that they are listed in chronological
order. The default value is%%(rev)s_%%(slug)s. Tokens available
include:%%(rev)s- revision id%%(slug)s- a truncated string derived from the revision message%%(epoch)s- epoch timestamp based on the create date; this makes
use of the Pythondatetime.timestamp()method to produce an epoch
value.%%(year)d,%%(month).2d,%%(day).2d,%%(hour).2d,%%(minute).2d,%%(second).2d- components of the create date,
by defaultdatetime.datetime.now()unless thetimezone
configuration option is also used.
The
file_templatemay also include directory separators to organize
migration files into subdirectories. When using directory paths infile_template,recursive_version_locationsmust be set totrue.
For example::textfile_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s recursive_version_locations = trueThis would create migration files organized by date in a structure like
versions/2024/12/26_143022_abc123_add_user_table.py... versionadded:: 1.18.0
Support for directory paths infile_templatetimezone- an optional timezone name (e.g.UTC,EST5EDT, etc.)
that will be applied to the timestamp which renders inside the migration
file's comment as well as within the filename. This option requires Python>=3.9
or installing thebackports.zoneinfolibrary and thetzdatalibrary.
Iftimezoneis specified, the create date object is no longer derived
fromdatetime.datetime.now()and is instead generated as::textdatetime.datetime.utcnow().replace( tzinfo=datetime.timezone.utc ).astimezone(ZoneInfo(<timezone>)).. versionchanged:: 1.13.0 Python standard library
zoneinfois now used
for timezone rendering in migrations; previouslypython-dateutil
was used.truncate_slug_length- defaults to 40, the max number of characters
to include in the "slug" field.sqlalchemy.url- A URL to connect to the database via SQLAlchemy. This
configuration value is only used if theenv.pyfile calls upon them;
in the "generic" template, the call toconfig.get_main_option("sqlalchemy.url")in therun_migrations_offline()function and the call toengine_from_config(prefix="sqlalchemy.")in therun_migrations_online()function are where this key is referenced. If
the SQLAlchemy URL should come from some other source, such as from
environment variables or a global registry, or if the migration environment
makes use of multiple database URLs, the developer is encouraged to alter theenv.pyfile to use whatever methods are appropriate in order to acquire
the database URL or URLs.revision_environment- this is a flag which when set to the value 'true', will indicate
that the migration environment scriptenv.pyshould be run unconditionally when
generating new revision files, as well as when running thealembic history
command.sourceless- when set to 'true', revision files that only exist as .pyc
or .pyo files in the versions directory will be used as versions, allowing
"sourceless" versioning folders. When left at the default of 'false',
only .py files are consumed as version files.version_locations- an optional list of revision file locations, to
allow revisions to exist in multiple directories simultaneously.
See :ref:multiple_basesfor examples.path_separator- a separator character for theversion_locations
andprepend_sys_pathpath lists. Only applies to configparser config,
not needed ifpyproject.tomlconfiguration is used.
See :ref:multiple_basesfor examples.recursive_version_locations- when set to 'true', revision files
are searched recursively in each "version_locations" directory... versionadded:: 1.10
output_encoding- the encoding to use when Alembic writes thescript.py.makofile into a new migration file. Defaults to'utf-8'.[loggers],[handlers],[formatters],[logger_*],[handler_*],[formatter_*]- these sections are all part of Python's standard logging configuration,
the mechanics of which are documented atConfiguration File Format <http://docs.python.org/library/logging.config.html#configuration-file-format>_.
As is the case with the database connection, these directives are used directly as the
result of thelogging.config.fileConfig()call present in theenv.pyscript, which you're free to modify.
For starting up with just a single database and the generic configuration, setting up
the SQLAlchemy URL is all that's needed::
sqlalchemy.url = postgresql://scott:tiger@localhost/test.. _escaping_percent_signs:
Escaping Characters in ini files
As mentioned previously, Alembic's .ini file format uses Python ConfigParser <https://docs.python.org/3/library/configparser.html#configparser.ConfigParser>_
to parse the file. ConfigParser 's interpolation feature is enabled <https://docs.python.org/3/library/configparser.html#interpolation-of-values>_
in this operation to support the use of the %(here)s token, as well as any
other tokens that are user-configurable via the :paramref:.Config.config_args
parameter when creating a custom :class:.Config object.
This means that any literal string that includes a percent sign that is not
part of an interpolated variable must be escaped by doubling it. That is, for
a configuration value like this in a Python script::
my_configuration_value = "some % string"
To be parsed from the .ini file would need to be placed as::
[alembic]
my_configuration_value = some %% string
This escaping can be seen in the sample alembic.ini file, illustrated in
such values as file_template::
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)sWhere above, the actual file_template that is sent to Alembic's file generation system
would be %(year)d_%(month).2d_%(day).2d_%(hour).2d%(minute).2d-%(rev)s_%(slug)s.
.. tip:: Alembic also employs percent-sign interpolation of values when retrieving
values from a pyproject.toml file, as documented at :ref:using_pep_621.
So the same percent-doubling steps must be applied in Alembic-parsed values,
for fields such as file_template.
For the SQLAlchemy URL, percent signs are used to escape syntactically-
significant characters such as the @ sign as well as the percent sign
itself. For a password such as "P@ssw%rd"::
my_actual_password = "P@ssw%rd"
As documented by SQLAlchemy <https://docs.sqlalchemy.org/core/engines.html#escaping-special-characters-such-as-signs-in-passwords>_,
the @ sign as well as the percent sign when placed into a URL should be escaped with urllib.parse.quote_plus::
import urllib.parse
sqlalchemy_quoted_password = urllib.parse.quote_plus(my_actual_password)
sqlalchemy_quoted_password
'P%40ssw%25rd'
This URL quoting can also be seen in SQLAlchemy's own stringification of
URLs::
from sqlalchemy import URL
URL.create(
... "some_db", username="scott", password=my_actual_password, host="host"
... ).render_as_string(hide_password=False)
'some_db://scott:P%40ssw%25rd@host'
For the above escaped password string 'P%40ssw%rd' to be placed into a ConfigParser file that
includes interpolation of percent signs, % characters are doubled::
sqlalchemy_quoted_password.replace("%", "%%")
'P%%40ssw%%25rd'
Here's a complete program that will compose a URL and show the correct configparser form
for a given set of database connection details, as well as illustrate how to assert these
forms for correctness::
from sqlalchemy import URL, make_url
database_driver = input("database driver? ")
username = input("username? ")
password = input("password? ")
host = input("host? ")
port = input("port? ")
database = input("database? ")
sqlalchemy_url = URL.create(
drivername=database_driver,
username=username,
password=password,
host=host,
port=int(port),
database=database,
)
stringified_sqlalchemy_url = sqlalchemy_url.render_as_string(
hide_password=False
)
# assert make_url round trip
assert make_url(stringified_sqlalchemy_url) == sqlalchemy_url
print(
f"The correctly escaped string that can be passed "
f"to SQLAlchemy make_url() and create_engine() is:"
f"\n\n {stringified_sqlalchemy_url!r}\n"
)
percent_replaced_url = stringified_sqlalchemy_url.replace("%", "%%")
# assert percent-interpolated plus make_url round trip
assert make_url(percent_replaced_url % {}) == sqlalchemy_url
print(
f"The SQLAlchemy URL that can be placed in a ConfigParser "
f"file such as alembic.ini is:\n\n "
f"sqlalchemy.url = {percent_replaced_url}\n"
)The above program should eliminate any ambiguity when placing a SQLAlchemy
URL into a configparser file::
$ python alembic_pw_script.py
database driver? postgresql+psycopg2
username? scott
password? P@ssw%rd
host? localhost
port? 5432
database? testdb
The correctly escaped string that can be passed to SQLAlchemy make_url() and create_engine() is:
'postgresql+psycopg2://scott:P%40ssw%25rd@localhost:5432/testdb'
The SQLAlchemy URL that can be placed in a ConfigParser file such as alembic.ini is:
sqlalchemy.url = postgresql+psycopg2://scott:P%%40ssw%%25rd@localhost:5432/testdb.. _using_pep_621:
Using pyproject.toml for configuration
.. versionadded:: 1.16.0
As the alembic.ini file includes a subset of options that are specific to
the organization and production of Python code within the local environment,
these specific options may alternatively be placed in the application'spyproject.toml file, to allow for :pep:621 compliant configuration.
Use of pyproject.toml does not preclude having an alembic.ini file as
well, as alembic.ini is still the default location for deployment
details such as database URLs, connectivity options, and logging to be present.
However, as connectivity and logging is consumed only by user-managed code
within the env.py file, it is feasible to have an environment that does not
require the alembic.ini file itself to be present at all, if these
configurational elements are consumed from other places elsewhere in the
application. Alembic will still run successfully if only a pyproject.toml
file is present and no alembic.ini is found.
To start with a pyproject configuration, the most straightforward approach is
to use the pyproject template::
alembic init --template pyproject alembicThe output states that the existing pyproject file is being augmented with
additional directives::
Creating directory /path/to/yourproject/alembic...done
Creating directory /path/to/yourproject/alembic/versions...done
Appending to /path/to/yourproject/pyproject.toml...done
Generating /path/to/yourproject/alembic.ini...done
Generating /path/to/yourproject/alembic/env.py...done
Generating /path/to/yourproject/alembic/README...done
Generating /path/to/yourproject/alembic/script.py.mako...done
Please edit configuration/connection/logging settings in
'/path/to/yourproject/pyproject.toml' and
'/path/to/yourproject/alembic.ini' before proceeding.Alembic's template runner will generate a new pyproject.toml file if
one does not exist, or it will append directives to an existing pyproject.toml
file that does not already include alembic directives.
Within the pyproject.toml file, the default section generated looks mostly
like the alembic.ini file, with the welcome exception that lists of values
are supported directly; this means the values prepend_sys_path andversion_locations are specified as lists. The %(here)s token also
remains available as the absolute path to the pyproject.toml file::
[tool.alembic]
# path to migration scripts
script_location = "%(here)s/alembic"
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# Or organize into date-based subdirectories (requires recursive_version_locations = true)
# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
# additional paths to be prepended to sys.path. defaults to the current working directory.
prepend_sys_path = [
"."
]
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library and tzdata library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to <script_location>/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# version_locations = [
# "%(here)s/alembic/versions",
# "%(here)s/foo/bar"
# ]
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = "utf-8"
# This section defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# [[tool.alembic.post_write_hooks]]
# format using "black" - use the console_scripts runner,
# against the "black" entrypoint
# name = "black"
# type = "console_scripts"
# entrypoint = "black"
# options = "-l 79 REVISION_SCRIPT_FILENAME"
#
# [[tool.alembic.post_write_hooks]]
# lint with attempts to fix using "ruff" - use the exec runner,
# execute a binary
# name = "ruff"
# type = "exec"
# executable = "%(here)s/.venv/bin/ruff"
# options = "check --fix REVISION_SCRIPT_FILENAME".. tip:: As Alembic adds support for interpolation tokens like %(here)s to
its handling of pyproject.toml values, the same percent-sign escaping
steps that apply to alembic.ini configuration variables also apply
to pyproject.toml, even though database URLs are not configured in this
file. This escaping can be seen in the sample file_template value
above. See the section :ref:escaping_percent_signs for background.
The alembic.ini file for this template is truncated and contains
only database configuration and logging configuration::
[alembic]
# database URL. This is consumed by the user-maintained env.py script only.
# other means of configuring database URLs may be customized within the env.py
# file.
sqlalchemy.url = driver://user:pass@localhost/dbname
# Logging configuration. This is also consumed by the user-maintained
# env.py script only.
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%SWhen env.py is configured to obtain database connectivity and logging
configuration from places other than alembic.ini, the file can be
omitted altogether.
.. _create_migration:
Create a Migration Script
With the environment in place we can create a new revision, using alembic revision::
$ alembic revision -m "create account table"
Generating /path/to/yourproject/alembic/versions/1975ea83b712_create_accoun
t_table.py...doneA new file 1975ea83b712_create_account_table.py is generated. Looking inside the file::
"""create account table
Revision ID: 1975ea83b712
Revises:
Create Date: 2011-11-08 11:40:27.089406
"""
# revision identifiers, used by Alembic.
revision = '1975ea83b712'
down_revision = None
branch_labels = None
from alembic import op
import sqlalchemy as sa
def upgrade():
pass
def downgrade():
passThe file contains some header information, identifiers for the current revision
and a "downgrade" revision, an import of basic Alembic directives,
and empty upgrade() and downgrade() functions. Our
job here is to populate the upgrade() and downgrade() functions with directives that
will apply a set of changes to our database. Typically, upgrade() is required
while downgrade() is only needed if down-revision capability is desired, though it's
probably a good idea.
Another thing to notice is the down_revision variable. This is how Alembic
knows the correct order in which to apply migrations. When we create the next revision,
the new file's down_revision identifier would point to this one::
# revision identifiers, used by Alembic.
revision = 'ae1027a6acf'
down_revision = '1975ea83b712'Every time Alembic runs an operation against the versions/ directory, it reads all
the files in, and composes a list based on how the down_revision identifiers link together,
with the down_revision of None representing the first file. In theory, if a
migration environment had thousands of migrations, this could begin to add some latency to
startup, but in practice a project should probably prune old migrations anyway
(see the section :ref:building_uptodate for a description on how to do this, while maintaining
the ability to build the current database fully).
We can then add some directives to our script, suppose adding a new table account::
def upgrade():
op.create_table(
'account',
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('name', sa.String(50), nullable=False),
sa.Column('description', sa.Unicode(200)),
)
def downgrade():
op.drop_table('account'):meth:~.Operations.create_table and :meth:~.Operations.drop_table are Alembic directives. Alembic provides
all the basic database migration operations via these directives, which are designed to be as simple and
minimalistic as possible;
there's no reliance upon existing table metadata for most of these directives. They draw upon
a global "context" that indicates how to get at a database connection (if any; migrations can
dump SQL/DDL directives to files as well) in order to invoke the command. This global
context is set up, like everything else, in the env.py script.
An overview of all Alembic directives is at :ref:ops.
Running our First Migration
We now want to run our migration. Assuming our database is totally clean, it's as
yet unversioned. The alembic upgrade command will run upgrade operations, proceeding
from the current database revision, in this example None, to the given target revision.
We can specify 1975ea83b712 as the revision we'd like to upgrade to, but it's easier
in most cases just to tell it "the most recent", in this case head::
$ alembic upgrade head
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
INFO [alembic.context] Running upgrade None -> 1975ea83b712Wow that rocked! Note that the information we see on the screen is the result of the
logging configuration set up in alembic.ini - logging the alembic stream to the
console (standard error, specifically).
The process which occurred here included that Alembic first checked if the database had
a table called alembic_version, and if not, created it. It looks in this table
for the current version, if any, and then calculates the path from this version to
the version requested, in this case head, which is known to be 1975ea83b712.
It then invokes the upgrade() method in each file to get to the target revision.
Running our Second Migration
Let's do another one so we have some things to play with. We again create a revision
file::
$ alembic revision -m "Add a column"
Generating /path/to/yourapp/alembic/versions/ae1027a6acf_add_a_column.py...
doneLet's edit this file and add a new column to the account table::
"""Add a column
Revision ID: ae1027a6acf
Revises: 1975ea83b712
Create Date: 2011-11-08 12:37:36.714947
"""
# revision identifiers, used by Alembic.
revision = 'ae1027a6acf'
down_revision = '1975ea83b712'
from alembic import op
import sqlalchemy as sa
def upgrade():
op.add_column('account', sa.Column('last_transaction_date', sa.DateTime))
def downgrade():
op.drop_column('account', 'last_transaction_date')Running again to head::
$ alembic upgrade head
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
INFO [alembic.context] Running upgrade 1975ea83b712 -> ae1027a6acfWe've now added the last_transaction_date column to the database.
Partial Revision Identifiers
Any time we need to refer to a revision number explicitly, we have the option
to use a partial number. As long as this number uniquely identifies the
version, it may be used in any command in any place that version numbers
are accepted::
$ alembic upgrade ae1Above, we use ae1 to refer to revision ae1027a6acf.
Alembic will stop and let you know if more than one version starts with
that prefix.
.. _relative_migrations:
Relative Migration Identifiers
Relative upgrades/downgrades are also supported. To move two versions from
the current, a decimal value "+N" can be supplied::
$ alembic upgrade +2Negative values are accepted for downgrades::
$ alembic downgrade -1Relative identifiers may also be in terms of a specific revision. For example,
to upgrade to revision ae1027a6acf plus two additional steps::
$ alembic upgrade ae10+2Getting Information
With a few revisions present we can get some information about the state of things.
First we can view the current revision::
$ alembic current
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
Current revision for postgresql://scott:XXXXX@localhost/test: 1975ea83b712 -> ae1027a6acf (head), Add a columnhead is displayed only if the revision identifier for this database matches the head revision.
We can also view history with alembic history; the --verbose option
(accepted by several commands, including history, current, heads
and branches) will show us full information about each revision::
$ alembic history --verbose
Rev: ae1027a6acf (head)
Parent: 1975ea83b712
Path: /path/to/yourproject/alembic/versions/ae1027a6acf_add_a_column.py
add a column
Revision ID: ae1027a6acf
Revises: 1975ea83b712
Create Date: 2014-11-20 13:02:54.849677
Rev: 1975ea83b712
Parent:
Path: /path/to/yourproject/alembic/versions/1975ea83b712_add_account_table.py
create account table
Revision ID: 1975ea83b712
Revises:
Create Date: 2014-11-20 13:02:46.257104Viewing History Ranges
Using the -r option to alembic history, we can also view various slices
of history. The -r argument accepts an argument [start]:[end], where
either may be a revision number, symbols like head, heads orbase, current to specify the current revision(s), as well as negative
relative ranges for [start] and positive relative ranges for [end]::
$ alembic history -r1975ea:ae1027
A relative range starting from three revs ago up to current migration,
which will invoke the migration environment against the database
to get the current migration::
$ alembic history -r-3:current
.. note::
As illustrated above, to use ranges that start with a negative number (i.e.
a dash), due to a
bug in argparse <https://github.com/python/cpython/issues/53580>_ , either
the syntax -r-:<head>, without any space, must be used as above::
$ alembic history -r-3:current or if using --rev-range, an equals sign must be used::
$ alembic history --rev-range=-3:current Using quotes or escape symbols will not work if there's a space after
the argument name.
View all revisions from 1975 to the head::
$ alembic history -r1975ea:
Downgrading
We can illustrate a downgrade back to nothing, by calling alembic downgrade back
to the beginning, which in Alembic is called base::
$ alembic downgrade base
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
INFO [alembic.context] Running downgrade ae1027a6acf -> 1975ea83b712
INFO [alembic.context] Running downgrade 1975ea83b712 -> NoneBack to nothing - and up again::
$ alembic upgrade head
INFO [alembic.context] Context class PostgresqlContext.
INFO [alembic.context] Will assume transactional DDL.
INFO [alembic.context] Running upgrade None -> 1975ea83b712
INFO [alembic.context] Running upgrade 1975ea83b712 -> ae1027a6acfNext Steps
The vast majority of Alembic environments make heavy use of the
"autogenerate" feature. Continue onto the next section, :doc:autogenerate.
Build Api Autogenerate (`docs/build/api/autogenerate.rst`)
.. _alembic.autogenerate.toplevel:
============== Autogeneration
.. note:: this section discusses the internal API of Alembic
as regards the autogeneration feature of the alembic revision
command.
This section is only useful for developers who wish to extend the
capabilities of Alembic. For general documentation on the autogenerate
feature, please see :doc:/autogenerate.
The autogeneration system has a wide degree of public API, including
the following areas:
The ability to do a "diff" of a :class:
~sqlalchemy.schema.MetaDataobject against
a database, and receive a data structure back. This structure
is available either as a rudimentary list of changes, or as
a :class:.MigrateOperationstructure.The ability to alter how the
alembic revisioncommand generates
revision scripts, including support for multiple revision scripts
generated in one pass.The ability to add new operation directives to autogeneration, including
custom schema/model comparison functions and revision script rendering.
Getting Diffs
The simplest API autogenerate provides is the "schema comparison" API;
these are simple functions that will run all registered "comparison" functions
between a :class:~sqlalchemy.schema.MetaData object and a database
backend to produce a structure showing how they differ. The two
functions provided are :func:.compare_metadata, which is more of the
"legacy" function that produces diff tuples, and :func:.produce_migrations,
which produces a structure consisting of operation directives detailed in
:ref:alembic.operations.toplevel.
.. autofunction:: alembic.autogenerate.compare_metadata
.. autofunction:: alembic.autogenerate.produce_migrations
.. _customizing_revision:
Customizing Revision Generation
The alembic revision command, also available programmatically
via :func:.command.revision, essentially produces a single migration
script after being run. Whether or not the --autogenerate option
was specified basically determines if this script is a blank revision
script with empty upgrade() and downgrade() functions, or was
produced with alembic operation directives as the result of autogenerate.
In either case, the system creates a full plan of what is to be done
in the form of a :class:.MigrateOperation structure, which is then
used to produce the script.
For example, suppose we ran alembic revision --autogenerate, and the
end result was that it produced a new revision 'eced083f5df'
with the following contents::
"""create the organization table."""
# revision identifiers, used by Alembic.
revision = 'eced083f5df'
down_revision = 'beafc7d709f'
from alembic import op
import sqlalchemy as sa
def upgrade():
op.create_table(
'organization',
sa.Column('id', sa.Integer(), primary_key=True),
sa.Column('name', sa.String(50), nullable=False)
)
op.add_column(
'user',
sa.Column('organization_id', sa.Integer())
)
op.create_foreign_key(
'org_fk', 'user', 'organization', ['organization_id'], ['id']
)
def downgrade():
op.drop_constraint('org_fk', 'user')
op.drop_column('user', 'organization_id')
op.drop_table('organization')The above script is generated by a :class:.MigrateOperation structure
that looks like this::
from alembic.operations import ops
import sqlalchemy as sa
migration_script = ops.MigrationScript(
'eced083f5df',
ops.UpgradeOps(
ops=[
ops.CreateTableOp(
'organization',
[
sa.Column('id', sa.Integer(), primary_key=True),
sa.Column('name', sa.String(50), nullable=False)
]
),
ops.ModifyTableOps(
'user',
ops=[
ops.AddColumnOp(
'user',
sa.Column('organization_id', sa.Integer())
),
ops.CreateForeignKeyOp(
'org_fk', 'user', 'organization',
['organization_id'], ['id']
)
]
)
]
),
ops.DowngradeOps(
ops=[