mongoengine Documentation

File: docs/guide/connecting.rst Copy Chapter

.. _guide-connecting:

===================== Connecting to MongoDB

Connections in MongoEngine are registered globally and are identified with aliases.
If no alias is provided during the connection, it will use "default" as alias.

To connect to a running instance of :program:mongod, use the :func:~mongoengine.connect
function. The first argument is the name of the database to connect to::

from mongoengine import connect
connect('project1')

By default, MongoEngine assumes that the :program:mongod instance is running
on localhost on port 27017.

If MongoDB is running elsewhere, you need to provide details on how to connect. There are two ways of
doing this. Using a connection string in URI format (this is the preferred method) or individual attributes
provided as keyword arguments.

Connect with URI string

When using a connection string in URI format you should specify the connection details
as the :attr:host to :func:~mongoengine.connect. In a web application context for instance, the URI
is typically read from the config file::

connect(host="mongodb://127.0.0.1:27017/my_db")

If the database requires authentication, you can specify it in the
URI. As each database can have its own users configured, you need to tell MongoDB
where to look for the user you are working with, that's what the ?authSource=admin bit
of the MongoDB connection string is for::

# Connects to 'my_db' database by authenticating
# with given credentials against the 'admin' database (by default as authSource isn't provided)
connect(host="mongodb://my_user:[email protected]:27017/my_db")

# Equivalent to previous connection but explicitly states that
# it should use admin as the authentication source database
connect(host="mongodb://my_user:my_password@hostname:port/my_db?authSource=admin")

# Connects to 'my_db' database by authenticating
# with given credentials against that same database
connect(host="mongodb://my_user:[email protected]:27017/my_db?authSource=my_db")

The URI string can also be used to configure advanced parameters like ssl, replicaSet, etc. For more
information or example about URI string, you can refer to the official doc <https://www.mongodb.com/docs/manual/reference/connection-string/>_::

connect(host="mongodb://my_user:[email protected]:27017/my_db?authSource=admin&ssl=true&replicaSet=globaldb")

.. note:: URI containing SRV records (e.g "mongodb+srv://server.example.com/") can be used as well

Connect with keyword attributes

The second option for specifying the connection details is to provide the information as keyword
attributes to :func:~mongoengine.connect::

connect('my_db', host='127.0.0.1', port=27017)

If the database requires authentication, :attr:username, :attr:password
and :attr:authentication_source arguments should be provided::

connect('my_db', username='my_user', password='my_password', authentication_source='admin')

The set of attributes that :func:~mongoengine.connect recognizes includes but is not limited to:
:attr:host, :attr:port, :attr:read_preference, :attr:username, :attr:password, :attr:authentication_source, :attr:authentication_mechanism,
:attr:replicaset, :attr:tls, etc. Most of the parameters accepted by pymongo.MongoClient <https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>_
can be used with :func:~mongoengine.connect and will simply be forwarded when instantiating the pymongo.MongoClient.

.. note:: Database, username and password from URI string overrides
corresponding parameters in :func:~mongoengine.connect, this should
obviously be avoided: ::

connect(
        db='test',
        username='user',
        password='12345',
        host='mongodb://admin:qwerty@localhost/production'
    )

will establish connection to ``production`` database using ``admin`` username and ``qwerty`` password.

.. note:: Calling :func:~mongoengine.connect without argument will establish
a connection to the "test" database by default

Read Preferences

As stated above, Read preferences are supported through the connection but also via individual
queries by passing the read_preference ::

from pymongo import ReadPreference

Bar.objects().read_preference(ReadPreference.PRIMARY)
Bar.objects(read_preference=ReadPreference.PRIMARY)

Multiple Databases

To use multiple databases you can use :func:~mongoengine.connect and provide
an alias name for the connection - if no alias is provided then "default"
is used.

In the background this uses :func:~mongoengine.register_connection to
store the data and you can register all aliases up front if required.

Documents defined in different database Copy Chapter

Individual documents can be attached to different databases by providing a
db_alias in their meta data. This allows :class:~pymongo.dbref.DBRef
objects to point across databases and collections. Below is an example schema,
using 3 different databases to store data::

connect(alias='user-db-alias', db='user-db')
    connect(alias='book-db-alias', db='book-db')
    connect(alias='users-books-db-alias', db='users-books-db')

    class User(Document):
        name = StringField()

        meta = {'db_alias': 'user-db-alias'}

    class Book(Document):
        name = StringField()

        meta = {'db_alias': 'book-db-alias'}

    class AuthorBooks(Document):
        author = ReferenceField(User)
        book = ReferenceField(Book)

        meta = {'db_alias': 'users-books-db-alias'}

Disconnecting an existing connection Copy Chapter

The function :func:~mongoengine.disconnect can be used to
disconnect a particular connection. This can be used to change a
connection globally::

from mongoengine import connect, disconnect
    connect('a_db', alias='db1')

    class User(Document):
        name = StringField()
        meta = {'db_alias': 'db1'}

    disconnect(alias='db1')

    connect('another_db', alias='db1')

.. note:: Calling :func:~mongoengine.disconnect without argument
will disconnect the "default" connection

.. note:: Since connections gets registered globally, it is important
to use the disconnect function from MongoEngine and not the
disconnect() method of an existing connection (pymongo.MongoClient)

.. note:: :class:~mongoengine.Document are caching the pymongo collection.
using disconnect ensures that it gets cleaned as well

Context Managers

Sometimes you may want to switch the database or collection to query against.
For example, archiving older data into a separate database for performance
reasons or writing functions that dynamically choose collections to write
a document to.

Switch Database Copy Chapter

The :class:~mongoengine.context_managers.switch_db context manager allows
you to change the database alias for a given class allowing quick and easy
access to the same User document across databases::

from mongoengine.context_managers import switch_db

class User(Document):
    name = StringField()

    meta = {'db_alias': 'user-db'}

with switch_db(User, 'archive-user-db') as User:
    User(name='Ross').save()  # Saves the 'archive-user-db'

.. note:: :func:~mongoengine.context_managers.switch_db when used on
a class that allow inheritance will change the database alias
for instances of a given class only - instances of subclasses will still use
the default database.

Switch Collection Copy Chapter

The :func:~mongoengine.context_managers.switch_collection context manager
allows you to change the collection for a given class allowing quick and easy
access to the same Group document across collection::

from mongoengine.context_managers import switch_collection

    class Group(Document):
        name = StringField()

    Group(name='test').save()  # Saves in the default db

    with switch_collection(Group, 'group2000') as Group:
        Group(name='hello Group 2000 collection!').save()  # Saves in group2000 collection

.. note:: Make sure any aliases have been registered with
:func:~mongoengine.register_connection or :func:~mongoengine.connect
before using the context manager.

File: docs/guide/defining-documents.rst Copy Chapter

================== Defining documents

In MongoDB, a document is roughly equivalent to a row in an RDBMS. When
working with relational databases, rows are stored in tables, which have a
strict schema that the rows follow. MongoDB stores documents in
collections rather than tables --- the principal difference is that no schema
is enforced at a database level.

Defining a document's schema

MongoEngine allows you to define schemata for documents as this helps to reduce
coding errors, and allows for utility methods to be defined on fields which may
be present.

To define a schema for a document, create a class that inherits from
:class:~mongoengine.Document. Fields are specified by adding field
objects as class attributes to the document class::

from datetime import datetime, timezone

from mongoengine import *

class Page(Document):
    title = StringField(max_length=200, required=True)
    date_modified = DateTimeField(
        default=lambda: datetime.now(timezone.utc)
    )

As BSON (the binary format for storing data in mongodb) is order dependent,
documents are serialized based on their field order.

.. _dynamic-document-schemas:

Dynamic Document Schemas

One of the benefits of MongoDB is dynamic schemas for a collection, whilst data
should be planned and organised (after all explicit is better than implicit!)
there are scenarios where having dynamic / expando style documents is desirable.

:class:~mongoengine.DynamicDocument documents work in the same way as
:class:~mongoengine.Document but any data / attributes set to them will also
be saved ::

from mongoengine import *

class Page(DynamicDocument):
    title = StringField(max_length=200, required=True)

# Create a new page and add tags
>>> page = Page(title='Using MongoEngine')
>>> page.tags = ['mongodb', 'mongoengine']
>>> page.save()

>>> Page.objects(tags='mongoengine').count()
>>> 1

.. note::

There is one caveat on Dynamic Documents: fields cannot start with _

Dynamic fields are stored in creation order after any declared fields.

Fields

By default, fields are not required. To make a field mandatory, set the
:attr:required keyword argument of a field to True. Fields also may have
validation constraints available (such as :attr:max_length in the example
above). Fields may also take default values, which will be used if a value is
not provided. Default values may optionally be a callable, which will be called
to retrieve the value (such as in the above example). The field types available
are as follows:

• :class:~mongoengine.fields.BinaryField
• :class:~mongoengine.fields.BooleanField
• :class:~mongoengine.fields.ComplexDateTimeField
• :class:~mongoengine.fields.DateTimeField
• :class:~mongoengine.fields.DecimalField
• :class:~mongoengine.fields.DictField
• :class:~mongoengine.fields.DynamicField
• :class:~mongoengine.fields.EmailField
• :class:~mongoengine.fields.EmbeddedDocumentField
• :class:~mongoengine.fields.EmbeddedDocumentListField
• :class:~mongoengine.fields.EnumField
• :class:~mongoengine.fields.FileField
• :class:~mongoengine.fields.FloatField
• :class:~mongoengine.fields.GenericEmbeddedDocumentField
• :class:~mongoengine.fields.GenericReferenceField
• :class:~mongoengine.fields.GenericLazyReferenceField
• :class:~mongoengine.fields.GeoPointField
• :class:~mongoengine.fields.ImageField
• :class:~mongoengine.fields.IntField
• :class:~mongoengine.fields.ListField
• :class:~mongoengine.fields.MapField
• :class:~mongoengine.fields.ObjectIdField
• :class:~mongoengine.fields.ReferenceField
• :class:~mongoengine.fields.LazyReferenceField
• :class:~mongoengine.fields.SequenceField
• :class:~mongoengine.fields.SortedListField
• :class:~mongoengine.fields.StringField
• :class:~mongoengine.fields.URLField
• :class:~mongoengine.fields.UUIDField
• :class:~mongoengine.fields.PointField
• :class:~mongoengine.fields.LineStringField
• :class:~mongoengine.fields.PolygonField
• :class:~mongoengine.fields.MultiPointField
• :class:~mongoengine.fields.MultiLineStringField
• :class:~mongoengine.fields.MultiPolygonField

Field arguments Copy Chapter

Each field type can be customized by keyword arguments. The following keyword
arguments can be set on all fields:

:attr:db_field (Default: None)
The MongoDB field name.

If set, operations in MongoDB will be performed with this value instead of the class attribute.

This allows you to use a different attribute than the name of the field used in MongoDB. ::

        from mongoengine import *

        class Page(Document):
            page_number = IntField(db_field="pageNumber")

        # Create a Page and save it
        Page(page_number=1).save()

        # How 'pageNumber' is stored in MongoDB
        Page.objects.as_pymongo() # [{'_id': ObjectId('629dfc45ee4cc407b1586b1f'), 'pageNumber': 1}]

        # Retrieve the object
        page: Page = Page.objects.first()

        print(page.page_number)  # prints 1

        print(page.pageNumber) # raises AttributeError

.. note:: If set, use the name of the attribute when defining indexes in the :attr:`meta`
    dictionary rather than the :attr:`db_field` otherwise, :class:`~mongoengine.LookUpError`
    will be raised.

:attr:required (Default: False)
If set to True and the field is not set on the document instance, a
:class:~mongoengine.ValidationError will be raised when the document is
validated.

:attr:default (Default: None)
A value to use when no value is set for this field.

The definition of default parameters follow `the general rules on Python
<http://docs.python.org/reference/compound_stmts.html#function-definitions>`__,
which means that some care should be taken when dealing with default mutable objects
(like in :class:`~mongoengine.fields.ListField` or :class:`~mongoengine.fields.DictField`)::

    class ExampleFirst(Document):
        # Default an empty list
        values = ListField(IntField(), default=list)

    class ExampleSecond(Document):
        # Default a set of values
        values = ListField(IntField(), default=lambda: [1,2,3])

    class ExampleDangerous(Document):
        # This can make an .append call to  add values to the default (and all the following objects),
        # instead to just an object
        values = ListField(IntField(), default=[1,2,3])

.. note:: Unsetting a field with a default value will revert back to the default.

:attr:unique (Default: False)
When True, no documents in the collection will have the same value for this
field.

:attr:unique_with (Default: None)
A field name (or list of field names) that when taken together with this
field, will not have two documents in the collection with the same value.

:attr:primary_key (Default: False)
When True, use this field as a primary key for the collection. DictField
and EmbeddedDocuments both support being the primary key for a document.

.. note:: If set, this field is also accessible through the `pk` field.

:attr:choices (Default: None)
An iterable (e.g. list, tuple or set) of choices to which the value of this
field should be limited.

Can either be nested tuples of value (stored in mongo) and a
human readable key ::

    SIZE = (('S', 'Small'),
            ('M', 'Medium'),
            ('L', 'Large'),
            ('XL', 'Extra Large'),
            ('XXL', 'Extra Extra Large'))


    class Shirt(Document):
        size = StringField(max_length=3, choices=SIZE)

Or a flat iterable just containing values ::

    SIZE = ('S', 'M', 'L', 'XL', 'XXL')

    class Shirt(Document):
        size = StringField(max_length=3, choices=SIZE)

:attr:validation (Optional)
A callable to validate the value of the field.
The callable takes the value as parameter and should raise a ValidationError
if validation fails

e.g ::

    def _not_empty(val):
        if not val:
            raise ValidationError('value can not be empty')

    class Person(Document):
        name = StringField(validation=_not_empty)

:attr:**kwargs (Optional)
You can supply additional metadata as arbitrary additional keyword
arguments. You can not override existing attributes, however. Common
choices include help_text and verbose_name, commonly used by form and
widget libraries.

List fields Copy Chapter

MongoDB allows storing lists of items. To add a list of items to a
:class:~mongoengine.Document, use the :class:~mongoengine.fields.ListField field
type. :class:~mongoengine.fields.ListField takes another field object as its first
argument, which specifies which type elements may be stored within the list::

class Page(Document):
    tags = ListField(StringField(max_length=50))

Embedded documents Copy Chapter

MongoDB has the ability to embed documents within other documents. Schemata may
be defined for these embedded documents, just as they may be for regular
documents. To create an embedded document, just define a document as usual, but
inherit from :class:~mongoengine.EmbeddedDocument rather than
:class:~mongoengine.Document::

class Comment(EmbeddedDocument):
    content = StringField()

To embed the document within another document, use the
:class:~mongoengine.fields.EmbeddedDocumentField field type, providing the embedded
document class as the first argument::

class Page(Document):
    comments = ListField(EmbeddedDocumentField(Comment))

comment1 = Comment(content='Good work!')
comment2 = Comment(content='Nice article!')
page = Page(comments=[comment1, comment2])

Embedded documents can also leverage the flexibility of :ref:dynamic-document-schemas
by inheriting :class:~mongoengine.DynamicEmbeddedDocument.

Dictionary Fields Copy Chapter

Often, an embedded document may be used instead of a dictionary – generally
embedded documents are recommended as dictionaries don’t support validation
or custom field types. However, sometimes you will not know the structure of what you want to
store; in this situation a :class:~mongoengine.fields.DictField is appropriate::

class SurveyResponse(Document):
    date = DateTimeField()
    user = ReferenceField(User)
    answers = DictField()

survey_response = SurveyResponse(
    date=datetime.now(timezone.utc),
    user=request.user,
)
response_form = ResponseForm(request.POST)
survey_response.answers = response_form.cleaned_data()
survey_response.save()

Dictionaries can store complex data, other dictionaries, lists, references to
other objects, so are the most flexible field type available.

Reference fields Copy Chapter

References may be stored to other documents in the database using the
:class:~mongoengine.fields.ReferenceField. Pass in another document class as the
first argument to the constructor, then simply assign document objects to the
field::

class User(Document):
    name = StringField()

class Page(Document):
    content = StringField()
    author = ReferenceField(User)

john = User(name="John Smith")
john.save()

post = Page(content="Test Page")
post.author = john
post.save()

The :class:User object is automatically turned into a reference behind the
scenes, and dereferenced when the :class:Page object is retrieved.

To add a :class:~mongoengine.fields.ReferenceField that references the document
being defined, use the string 'self' in place of the document class as the
argument to :class:~mongoengine.fields.ReferenceField's constructor. To reference a
document that has not yet been defined, use the name of the undefined document
as the constructor's argument::

class Employee(Document):
    name = StringField()
    boss = ReferenceField('self')
    profile_page = ReferenceField('ProfilePage')

class ProfilePage(Document):
    content = StringField()

.. _many-to-many-with-listfields:

Many to Many with ListFields
''''''''''''''''''''''''''''

If you are implementing a many to many relationship via a list of references,
then the references are stored as DBRefs and to query you need to pass an
instance of the object to the query::

class User(Document):
    name = StringField()

class Page(Document):
    content = StringField()
    authors = ListField(ReferenceField(User))

bob = User(name="Bob Jones").save()
john = User(name="John Smith").save()

Page(content="Test Page", authors=[bob, john]).save()
Page(content="Another Page", authors=[john]).save()

# Find all pages Bob authored
Page.objects(authors__in=[bob])

# Find all pages that both Bob and John have authored
Page.objects(authors__all=[bob, john])

# Remove Bob from the authors for a page.
Page.objects(id='...').update_one(pull__authors=bob)

# Add John to the authors for a page.
Page.objects(id='...').update_one(push__authors=john)

Dealing with deletion of referred documents
'''''''''''''''''''''''''''''''''''''''''''
By default, MongoDB doesn't check the integrity of your data, so deleting
documents that other documents still hold references to will lead to consistency
issues. Mongoengine's :class:ReferenceField adds some functionality to
safeguard against these kinds of database integrity problems, providing each
reference with a delete rule specification. A delete rule is specified by
supplying the :attr:reverse_delete_rule attributes on the
:class:ReferenceField definition, like this::

class ProfilePage(Document):
    employee = ReferenceField('Employee', reverse_delete_rule=mongoengine.CASCADE)

The declaration in this example means that when an :class:Employee object is
removed, the :class:ProfilePage that references that employee is removed as
well. If a whole batch of employees is removed, all profile pages that are
linked are removed as well.

Its value can take any of the following constants:

:const:mongoengine.DO_NOTHING
This is the default and won't do anything. Deletes are fast, but may cause
database inconsistency or dangling references.
:const:mongoengine.DENY
Deletion is denied if there still exist references to the object being
deleted.
:const:mongoengine.NULLIFY
Any object's fields still referring to the object being deleted are set to None
(using MongoDB's "unset" operation), effectively nullifying the relationship.
:const:mongoengine.CASCADE
Any object containing fields that are referring to the object being deleted
are deleted first.
:const:mongoengine.PULL
Removes the reference to the object (using MongoDB's "pull" operation)
from any object's fields of
:class:~mongoengine.fields.ListField (:class:~mongoengine.fields.ReferenceField).

.. warning::
A safety note on setting up these delete rules! Since the delete rules are
not recorded on the database level by MongoDB itself, but instead at runtime,
in-memory, by the MongoEngine module, it is of the upmost importance
that the module that declares the relationship is loaded BEFORE the
delete is invoked.

If, for example, the :class:Employee object lives in the
:mod:payroll app, and the :class:ProfilePage in the :mod:people
app, it is extremely important that the :mod:people app is loaded
before any employee is removed, because otherwise, MongoEngine could
never know this relationship exists.

In Django, be sure to put all apps that have such delete rule declarations in
their :file:models.py in the :const:INSTALLED_APPS tuple.

Generic reference fields
''''''''''''''''''''''''
A second kind of reference field also exists,
:class:~mongoengine.fields.GenericReferenceField. This allows you to reference any
kind of :class:~mongoengine.Document, and hence doesn't take a
:class:~mongoengine.Document subclass as a constructor argument::

class Link(Document):
    url = StringField()

class Post(Document):
    title = StringField()

class Bookmark(Document):
    bookmark_object = GenericReferenceField()

link = Link(url='http://hmarr.com/mongoengine/')
link.save()

post = Post(title='Using MongoEngine')
post.save()

Bookmark(bookmark_object=link).save()
Bookmark(bookmark_object=post).save()

.. note::

Using :class:~mongoengine.fields.GenericReferenceField\ s is slightly less
efficient than the standard :class:~mongoengine.fields.ReferenceField\ s, so if
you will only be referencing one document type, prefer the standard
:class:~mongoengine.fields.ReferenceField.

Uniqueness constraints Copy Chapter

MongoEngine allows you to specify that a field should be unique across a
collection by providing unique=True to a :class:~mongoengine.fields.Field\ 's
constructor. If you try to save a document that has the same value for a unique
field as a document that is already in the database, a
:class:~mongoengine.NotUniqueError will be raised. You may also specify
multi-field uniqueness constraints by using :attr:unique_with, which may be
either a single field name, or a list or tuple of field names::

class User(Document):
    username = StringField(unique=True)
    first_name = StringField()
    last_name = StringField(unique_with='first_name')

Document collections

Document classes that inherit directly from :class:~mongoengine.Document
will have their own collection in the database. The name of the collection
is by default the name of the class converted to snake_case (e.g if your Document class
is named CompanyUser, the corresponding collection would be company_user). If you need
to change the name of the collection (e.g. to use MongoEngine with an existing database),
then create a class dictionary attribute called :attr:meta on your document, and
set :attr:collection to the name of the collection that you want your
document class to use::

class Page(Document):
    title = StringField(max_length=200, required=True)
    meta = {'collection': 'cmsPage'}

Capped collections Copy Chapter

A :class:~mongoengine.Document may use a Capped Collection by specifying
:attr:max_documents and :attr:max_size in the :attr:meta dictionary.
:attr:max_documents is the maximum number of documents that is allowed to be
stored in the collection, and :attr:max_size is the maximum size of the
collection in bytes. :attr:max_size is rounded up to the next multiple of 256
by MongoDB internally and mongoengine before. Use also a multiple of 256 to
avoid confusions. If :attr:max_size is not specified and
:attr:max_documents is, :attr:max_size defaults to 10485760 bytes (10MB).
The following example shows a :class:Log document that will be limited to
1000 entries and 2MB of disk space::

class Log(Document):
    ip_address = StringField()
    meta = {'max_documents': 1000, 'max_size': 2000000}

Timeseries collections Copy Chapter

A :class:~mongoengine.Document may use a Timeseries Collection by specifying
:attr:timeseries in the :attr:meta dictionary. Timeseries collection were added
in MongoDB 5.0 (doc <https://www.mongodb.com/docs/v5.3/core/timeseries-collections/>_).
The following example shows a Document class with a basic setup::

class SensorData(Document):
    timestamp = DateTimeField(required=True)
    temperature = FloatField()

    meta = {
        "timeseries": {
            "timeField": "timestamp",
            "metaField": "temperature",
            "granularity": "seconds",
            "expireAfterSeconds": 5,
        },
    }

.. defining-indexes_

Indexes

You can specify indexes on collections to make querying faster. This is done
by creating a list of index specifications called :attr:indexes in the
:attr:~mongoengine.Document.meta dictionary, where an index specification may
either be a single field name, a tuple containing multiple field names, or a
dictionary containing a full index definition.

A direction may be specified on fields by prefixing the field name with a
+ (for ascending) or a - sign (for descending). Note that direction
only matters on compound indexes. Text indexes may be specified by prefixing
the field name with a $. Hashed indexes may be specified by prefixing
the field name with a #::

class Page(Document):
    category = IntField()
    title = StringField()
    rating = StringField()
    created = DateTimeField()
    meta = {
        'indexes': [
            'title',   # single-field index
            '$title',  # text index
            '#title',  # hashed index
            ('title', '-rating'),  # compound index
            ('category', '_cls'),  # compound index
            {
                'fields': ['created'],
                'expireAfterSeconds': 3600  # ttl index
            }
        ]
    }

If a dictionary is passed then additional options become available. Valid options include,
but are not limited to:

:attr:fields (Default: None)
The fields to index. Specified in the same format as described above.

:attr:cls (Default: True)
If you have polymorphic models that inherit and have
:attr:allow_inheritance turned on, you can configure whether the index
should have the :attr:_cls field added automatically to the start of the
index.

:attr:sparse (Default: False)
Whether the index should be sparse.

:attr:unique (Default: False)
Whether the index should be unique.

:attr:expireAfterSeconds (Optional)
Allows you to automatically expire data from a collection by setting the
time in seconds to expire the a field.

:attr:name (Optional)
Allows you to specify a name for the index

:attr:collation (Optional)
Allows to create case insensitive indexes (MongoDB v3.4+ only)

.. note::

Additional options are forwarded as ``**kwargs`` to pymongo's create_index method.
Inheritance adds extra fields indices see: :ref:`document-inheritance`.

Global index default options Copy Chapter

There are a few top level defaults for all indexes that can be set::

class Page(Document):
    title = StringField()
    rating = StringField()
    meta = {
        'index_opts': {},
        'index_background': True,
        'index_cls': False,
        'auto_create_index': True,
        'auto_create_index_on_save': False,
    }

:attr:index_opts (Optional)
Set any default index options - see the full options list <https://www.mongodb.com/docs/manual/reference/method/db.collection.createIndex/>_

:attr:index_background (Optional)
Set the default value for if an index should be indexed in the background

:attr:index_cls (Optional)
A way to turn off a specific index for _cls.

:attr:auto_create_index (Optional)
When this is True (default), MongoEngine will ensure that the correct
indexes exist in MongoDB when the Document is first used. This can be disabled
in systems where indexes are managed separately. Disabling this will improve
performance.

:attr:auto_create_index_on_save (Optional)
When this is True, MongoEngine will ensure that the correct
indexes exist in MongoDB each time :meth:~mongoengine.document.Document.save
is run. Enabling this will degrade performance. The default is False. This
option was added in version 0.25.

Compound Indexes and Indexing sub documents Copy Chapter

Compound indexes can be created by adding the Embedded field or dictionary
field name to the index definition.

Sometimes its more efficient to index parts of Embedded / dictionary fields,
in this case use 'dot' notation to identify the value to index eg: rank.title

.. _geospatial-indexes:

Geospatial indexes Copy Chapter

The best geo index for mongodb is the new "2dsphere", which has an improved
spherical model and provides better performance and more options when querying.
The following fields will explicitly add a "2dsphere" index:

- :class:`~mongoengine.fields.PointField`
- :class:`~mongoengine.fields.LineStringField`
- :class:`~mongoengine.fields.PolygonField`
- :class:`~mongoengine.fields.MultiPointField`
- :class:`~mongoengine.fields.MultiLineStringField`
- :class:`~mongoengine.fields.MultiPolygonField`

As "2dsphere" indexes can be part of a compound index, you may not want the
automatic index but would prefer a compound index. In this example we turn off
auto indexing and explicitly declare a compound index on location and datetime::

class Log(Document):
    location = PointField(auto_index=False)
    datetime = DateTimeField()

    meta = {
        'indexes': [[("location", "2dsphere"), ("datetime", 1)]]
    }

Pre MongoDB 2.4 Geo
'''''''''''''''''''

.. note:: For MongoDB < 2.4 this is still current, however the new 2dsphere
index is a big improvement over the previous 2D model - so upgrading is
advised.

Geospatial indexes will be automatically created for all
:class:~mongoengine.fields.GeoPointField\ s

It is also possible to explicitly define geospatial indexes. This is
useful if you need to define a geospatial index on a subfield of a
:class:~mongoengine.fields.DictField or a custom field that contains a
point. To create a geospatial index you must prefix the field with the
***** sign. ::

class Place(Document):
    location = DictField()
    meta = {
        'indexes': [
            '*location.point',
        ],
    }

Time To Live (TTL) indexes Copy Chapter

A special index type that allows you to automatically expire data from a
collection after a given period. See the official
ttl <http://docs.mongodb.org/manual/tutorial/expire-data/#expire-data-from-collections-by-setting-ttl>_
documentation for more information. A common usecase might be session data::

class Session(Document):
    created = DateTimeField(
        default=lambda: datetime.now(timezone.utc)
    )
    meta = {
        'indexes': [
            {'fields': ['created'], 'expireAfterSeconds': 3600}
        ]
    }

.. warning:: TTL indexes happen on the MongoDB server and not in the application
code, therefore no signals will be fired on document deletion.
If you need signals to be fired on deletion, then you must handle the
deletion of Documents in your application code.

Comparing Indexes Copy Chapter

Use :func:mongoengine.Document.compare_indexes to compare actual indexes in
the database to those that your document definitions define. This is useful
for maintenance purposes and ensuring you have the correct indexes for your
schema.

Ordering

A default ordering can be specified for your
:class:~mongoengine.queryset.QuerySet using the :attr:ordering attribute of
:attr:~mongoengine.Document.meta. Ordering will be applied when the
:class:~mongoengine.queryset.QuerySet is created, and can be overridden by
subsequent calls to :meth:~mongoengine.queryset.QuerySet.order_by. ::

from datetime import datetime

class BlogPost(Document):
    title = StringField()
    published_date = DateTimeField()

    meta = {
        'ordering': ['-published_date']
    }

blog_post_1 = BlogPost(title="Blog Post #1")
blog_post_1.published_date = datetime(2010, 1, 5, 0, 0 ,0)

blog_post_2 = BlogPost(title="Blog Post #2")
blog_post_2.published_date = datetime(2010, 1, 6, 0, 0 ,0)

blog_post_3 = BlogPost(title="Blog Post #3")
blog_post_3.published_date = datetime(2010, 1, 7, 0, 0 ,0)

blog_post_1.save()
blog_post_2.save()
blog_post_3.save()

# get the "first" BlogPost using default ordering
# from BlogPost.meta.ordering
latest_post = BlogPost.objects.first()
assert latest_post.title == "Blog Post #3"

# override default ordering, order BlogPosts by "published_date"
first_post = BlogPost.objects.order_by("+published_date").first()
assert first_post.title == "Blog Post #1"

Shard keys

If your collection is sharded by multiple keys, then you can improve shard
routing (and thus the performance of your application) by specifying the shard
key, using the :attr:shard_key attribute of
:attr:~mongoengine.Document.meta. The shard key should be defined as a tuple.

This ensures that the full shard key is sent with the query when calling
methods such as :meth:~mongoengine.document.Document.save,
:meth:~mongoengine.document.Document.update,
:meth:~mongoengine.document.Document.modify, or
:meth:~mongoengine.document.Document.delete on an existing
:class:~mongoengine.Document instance::

class LogEntry(Document):
    machine = StringField()
    app = StringField()
    timestamp = DateTimeField()
    data = StringField()

    meta = {
        'shard_key': ('machine', 'timestamp'),
        'indexes': ('machine', 'timestamp'),
    }

.. _document-inheritance:

Document inheritance

To create a specialised type of a :class:~mongoengine.Document you have
defined, you may subclass it and add any extra fields or methods you may need.
As this new class is not a direct subclass of
:class:~mongoengine.Document, it will not be stored in its own collection; it
will use the same collection as its superclass uses. This allows for more
convenient and efficient retrieval of related documents -- all you need do is
set :attr:allow_inheritance to True in the :attr:meta data for a
document.::

# Stored in a collection named 'page'
class Page(Document):
    title = StringField(max_length=200, required=True)

    meta = {'allow_inheritance': True}

# Also stored in the collection named 'page'
class DatedPage(Page):
    date = DateTimeField()

.. note:: From 0.8 onwards :attr:allow_inheritance defaults
to False, meaning you must set it to True to use inheritance.

Setting :attr:`allow_inheritance` to True should also be used in
      :class:`~mongoengine.EmbeddedDocument` class in case you need to subclass it

When it comes to querying using :attr:.objects(), querying Page.objects() will query
both Page and DatedPage whereas querying DatedPage will only query the DatedPage documents.
Behind the scenes, MongoEngine deals with inheritance by adding a :attr:_cls attribute that contains
the class name in every documents. When a document is loaded, MongoEngine checks
it's :attr:_cls attribute and use that class to construct the instance.::

Page(title='a funky title').save()
DatedPage(
    title='another title',
    date=datetime.now(timezone.utc),
).save()

print(Page.objects().count())         # 2
print(DatedPage.objects().count())    # 1

# print documents in their native form
# we remove 'id' to avoid polluting the output with unnecessary detail
qs = Page.objects.exclude('id').as_pymongo()
print(list(qs))
# [
#   {'_cls': u 'Page', 'title': 'a funky title'},
#   {'_cls': u 'Page.DatedPage', 'title': u 'another title', 'date': datetime.datetime(2019, 12, 13, 20, 16, 59, 993000)}
# ]

Working with existing data Copy Chapter

As MongoEngine no longer defaults to needing :attr:_cls, you can quickly and
easily get working with existing data. Just define the document to match
the expected schema in your database ::

# Will work with data in an existing collection named 'cmsPage'
class Page(Document):
    title = StringField(max_length=200, required=True)
    meta = {
        'collection': 'cmsPage'
    }

If you have wildly varying schemas then using a
:class:~mongoengine.DynamicDocument might be more appropriate, instead of
defining all possible field types.

If you use :class:~mongoengine.Document and the database contains data that
isn't defined then that data will be stored in the document._data dictionary.

Abstract classes

If you want to add some extra functionality to a group of Document classes but
you don't need or want the overhead of inheritance you can use the
:attr:abstract attribute of :attr:~mongoengine.Document.meta.
This won't turn on :ref:document-inheritance but will allow you to keep your
code DRY::

class BaseDocument(Document):
        meta = {
            'abstract': True,
        }
        def check_permissions(self):
            ...

    class User(BaseDocument):
       ...

Now the User class will have access to the inherited check_permissions method
and won't store any of the extra _cls information.

File: docs/guide/document-instances.rst Copy Chapter

=================== Documents instances

To create a new document object, create an instance of the relevant document
class, providing values for its fields as constructor keyword arguments.
You may provide values for any of the fields on the document::

>>> page = Page(title="Test Page")
>>> page.title
'Test Page'

You may also assign values to the document's fields using standard object
attribute syntax::

>>> page.title = "Example Page"
>>> page.title
'Example Page'

Saving and deleting documents

MongoEngine tracks changes to documents to provide efficient saving. To save
the document to the database, call the :meth:~mongoengine.Document.save method.
If the document does not exist in the database, it will be created. If it does
already exist, then any changes will be updated atomically. For example::

>>> page = Page(title="Test Page")
>>> page.save()  # Performs an insert
>>> page.title = "My Page"
>>> page.save()  # Performs an atomic set on the title field.

.. note::

Changes to documents are tracked and on the whole perform ``set`` operations.

* ``list_field.push(0)`` --- *sets* the resulting list
* ``del(list_field)``   --- *unsets* whole list

With lists its preferable to use ``Doc.update(push__list_field=0)`` as
this stops the whole list being updated --- stopping any race conditions.

.. seealso::
:ref:guide-atomic-updates

Cascading Saves Copy Chapter

If your document contains :class:~mongoengine.fields.ReferenceField or
:class:~mongoengine.fields.GenericReferenceField objects, then by default the
:meth:~mongoengine.Document.save method will not save any changes to
those objects. If you want all references to be saved also, noting each
save is a separate query, then passing :attr:cascade as True
to the save method will cascade any saves.

Deleting documents Copy Chapter

To delete a document, call the :meth:~mongoengine.Document.delete method.
Note that this will only work if the document exists in the database and has a
valid :attr:id.

Document IDs

Each document in the database has a unique id. This may be accessed through the
:attr:id attribute on :class:~mongoengine.Document objects. Usually, the id
will be generated automatically by the database server when the object is save,
meaning that you may only access the :attr:id field once a document has been
saved::

>>> page = Page(title="Test Page")
>>> page.id
>>> page.save()
>>> page.id
ObjectId('123456789abcdef000000000')

Alternatively, you may define one of your own fields to be the document's
"primary key" by providing primary_key=True as a keyword argument to a
field's constructor. Under the hood, MongoEngine will use this field as the
:attr:id; in fact :attr:id is actually aliased to your primary key field so
you may still use :attr:id to access the primary key if you want::

>>> class User(Document):
...     email = StringField(primary_key=True)
...     name = StringField()
...
>>> bob = User(email='[email protected]', name='Bob')
>>> bob.save()
>>> bob.id == bob.email == '[email protected]'
True

You can also access the document's "primary key" using the :attr:pk field,
it's an alias to :attr:id::

>>> page = Page(title="Another Test Page")
>>> page.save()
>>> page.id == page.pk
True

.. note::

If you define your own primary key field, the field implicitly becomes
required, so a :class:~mongoengine.ValidationError will be thrown if
you don't provide it.

File: docs/guide/gridfs.rst Copy Chapter

====== GridFS

Writing Copy Chapter

GridFS support comes in the form of the :class:~mongoengine.fields.FileField field
object. This field acts as a file-like object and provides a couple of
different ways of inserting and retrieving data. Arbitrary metadata such as
content type can also be stored alongside the files. The object returned when accessing a
FileField is a proxy to Pymongo's GridFS <https://pymongo.readthedocs.io/en/stable/examples/gridfs.html#gridfs-example>_
In the following example, a document is created to store details about animals, including a photo::

class Animal(Document):
    genus = StringField()
    family = StringField()
    photo = FileField()

marmot = Animal(genus='Marmota', family='Sciuridae')

with open('marmot.jpg', 'rb') as fd:
    marmot.photo.put(fd, content_type = 'image/jpeg')
marmot.save()

Retrieval Copy Chapter

So using the :class:~mongoengine.fields.FileField is just like using any other
field. The file can also be retrieved just as easily::

marmot = Animal.objects(genus='Marmota').first()
photo = marmot.photo.read()
content_type = marmot.photo.content_type

.. note:: If you need to read() the content of a file multiple times, you'll need to "rewind"
the file-like object using seek::

marmot = Animal.objects(genus='Marmota').first()
    content1 = marmot.photo.read()
    assert content1 != ""

    content2 = marmot.photo.read()    # will be empty
    assert content2 == ""

    marmot.photo.seek(0)              # rewind the file by setting the current position of the cursor in the file to 0
    content3 = marmot.photo.read()
    assert content3 == content1

Streaming Copy Chapter

Streaming data into a :class:~mongoengine.fields.FileField is achieved in a
slightly different manner. First, a new file must be created by calling the
:func:new_file method. Data can then be written using :func:write::

marmot.photo.new_file()
marmot.photo.write('some_image_data')
marmot.photo.write('some_more_image_data')
marmot.photo.close()

marmot.save()

Deletion Copy Chapter

Deleting stored files is achieved with the :func:delete method::

marmot.photo.delete()    # Deletes the GridFS document
marmot.save()            # Saves the GridFS reference (being None) contained in the marmot instance

.. warning::

The FileField in a Document actually only stores the ID of a file in a
separate GridFS collection. This means that deleting a document
with a defined FileField does not actually delete the file. You must be
careful to delete any files in a Document as above before deleting the
Document itself.

Replacing files Copy Chapter

Files can be replaced with the :func:replace method. This works just like
the :func:put method so even metadata can (and should) be replaced::

another_marmot = open('another_marmot.png', 'rb')
marmot.photo.replace(another_marmot, content_type='image/png')  # Replaces the GridFS document
marmot.save()                                                   # Replaces the GridFS reference contained in marmot instance

File: docs/guide/index.rst Copy Chapter

========== User Guide

.. toctree::
:maxdepth: 2

installing
connecting
defining-documents
document-instances
querying
validation
gridfs
signals
text-indexes
migration
logging-monitoring
mongomock

File: docs/guide/installing.rst Copy Chapter

====================== Installing MongoEngine

To use MongoEngine, you will need to download MongoDB <https://www.mongodb.com/>_
and ensure it is running in an accessible location. You will also need
PyMongo <http://api.mongodb.org/python>_ to use MongoEngine, but if you
install MongoEngine using setuptools, then the dependencies will be handled for
you.

MongoEngine is available on PyPI, so you can use :program:pip:

.. code-block:: console

$ python -m pip install mongoengine

Alternatively, if you don't have setuptools installed, download it from PyPi <http://pypi.python.org/pypi/mongoengine/>_ and run

.. code-block:: console

$ python setup.py install

To use the bleeding-edge version of MongoEngine, you can get the source from
GitHub <https://github.com/MongoEngine/mongoengine/>_ and install it as above:

.. code-block:: console

$ git clone git://github.com/mongoengine/mongoengine
$ cd mongoengine
$ python setup.py install

File: docs/guide/logging-monitoring.rst Copy Chapter

================== Logging/Monitoring

It is possible to use pymongo.monitoring <https://pymongo.readthedocs.io/en/stable/api/pymongo/monitoring.html>_ to monitor
the driver events (e.g: queries, connections, etc). This can be handy if you want to monitor the queries issued by
MongoEngine to the driver.

To use pymongo.monitoring with MongoEngine, you need to make sure that you are registering the listeners
before establishing the database connection (i.e calling connect):

The following snippet provides a basic logging of all command events:

.. code-block:: python

import logging
from pymongo import monitoring
from mongoengine import *

log = logging.getLogger()
log.setLevel(logging.DEBUG)
logging.basicConfig(level=logging.DEBUG)


class CommandLogger(monitoring.CommandListener):

    def started(self, event):
        log.debug("Command {0.command_name} with request id "
                 "{0.request_id} started on server "
                 "{0.connection_id}".format(event))

    def succeeded(self, event):
        log.debug("Command {0.command_name} with request id "
                 "{0.request_id} on server {0.connection_id} "
                 "succeeded in {0.duration_micros} "
                 "microseconds".format(event))

    def failed(self, event):
        log.debug("Command {0.command_name} with request id "
                 "{0.request_id} on server {0.connection_id} "
                 "failed in {0.duration_micros} "
                 "microseconds".format(event))

monitoring.register(CommandLogger())


class Jedi(Document):
    name = StringField()


connect()


log.info('GO!')

log.info('Saving an item through MongoEngine...')
Jedi(name='Obi-Wan Kenobii').save()

log.info('Querying through MongoEngine...')
obiwan = Jedi.objects.first()

log.info('Updating through MongoEngine...')
obiwan.name = 'Obi-Wan Kenobi'
obiwan.save()

Executing this prints the following output::

INFO:root:GO!
INFO:root:Saving an item through MongoEngine...
DEBUG:root:Command insert with request id 1681692777 started on server ('localhost', 27017)
DEBUG:root:Command insert with request id 1681692777 on server ('localhost', 27017) succeeded in 562 microseconds
INFO:root:Querying through MongoEngine...
DEBUG:root:Command find with request id 1714636915 started on server ('localhost', 27017)
DEBUG:root:Command find with request id 1714636915 on server ('localhost', 27017) succeeded in 341 microseconds
INFO:root:Updating through MongoEngine...
DEBUG:root:Command update with request id 1957747793 started on server ('localhost', 27017)
DEBUG:root:Command update with request id 1957747793 on server ('localhost', 27017) succeeded in 455 microseconds

More details can of course be obtained by checking the event argument from the CommandListener.

File: docs/guide/migration.rst Copy Chapter

=================== Documents migration

The structure of your documents and their associated mongoengine schemas are likely
to change over the lifetime of an application. This section provides guidance and
recommendations on how to deal with migrations.

Due to the very flexible nature of mongodb, migrations of models aren't trivial and
for people that know about alembic for sqlalchemy, there is unfortunately no equivalent
library that will manage the migration in an automatic fashion for mongoengine.

Example 1: Addition of a field

Let's start by taking a simple example of a model change and review the different option you
have to deal with the migration.

Let's assume we start with the following schema and save an instance:

.. code-block:: python

class User(Document):
    name = StringField()

User(name="John Doe").save()

# print the objects as they exist in mongodb
print(User.objects().as_pymongo())    # [{u'_id': ObjectId('5d06b9c3d7c1f18db3e7c874'), u'name': u'John Doe'}]

On the next version of your application, let's now assume that a new field enabled gets added to the
existing User model with a default=True. Thus you simply update the User class to the following:

.. code-block:: python

class User(Document):
    name = StringField(required=True)
    enabled = BooleanField(default=True)

Without applying any migration, we now reload an object from the database into the User class
and checks its enabled attribute:

.. code-block:: python

assert User.objects.count() == 1
user = User.objects().first()
assert user.enabled is True
assert User.objects(enabled=True).count() == 0    # uh?
assert User.objects(enabled=False).count() == 0   # uh?

# this is consistent with what we have in the database
# in fact, 'enabled' does not exist
print(User.objects().as_pymongo().first())    # {u'_id': ObjectId('5d06b9c3d7c1f18db3e7c874'), u'name': u'John'}
assert User.objects(enabled=None).count() == 1

As you can see, even if the document wasn't updated, mongoengine applies the default value seamlessly when it
loads the pymongo dict into a User instance. At first sight it looks like you don't need to migrate the
existing documents when adding new fields but this actually leads to inconsistencies when it comes to querying.

In fact, when querying, mongoengine isn't trying to account for the default value of the new field and so
if you don't actually migrate the existing documents, you are taking a risk that querying/updating
will be missing relevant record.

When adding fields/modifying default values, you can use any of the following to do the migration
as a standalone script:

.. code-block:: python

# Use mongoengine to set a default value for a given field
User.objects().update(enabled=True)
# or use pymongo
user_coll = User._get_collection()
user_coll.update_many({}, {'$set': {'enabled': True}})

Example 2: Inheritance change

Let's consider the following example:

.. code-block:: python

class Human(Document):
    name = StringField()
    meta = {"allow_inheritance": True}

class Jedi(Human):
    dark_side = BooleanField()
    light_saber_color = StringField()

Jedi(name="Darth Vader", dark_side=True, light_saber_color="red").save()
Jedi(name="Obi Wan Kenobi", dark_side=False, light_saber_color="blue").save()

assert Human.objects.count() == 2
assert Jedi.objects.count() == 2

# Let's check how these documents got stored in mongodb
print(Jedi.objects.as_pymongo())
# [
#   {'_id': ObjectId('5fac4aaaf61d7fb06046e0f9'), '_cls': 'Human.Jedi', 'name': 'Darth Vader', 'dark_side': True, 'light_saber_color': 'red'},
#   {'_id': ObjectId('5fac4ac4f61d7fb06046e0fa'), '_cls': 'Human.Jedi', 'name': 'Obi Wan Kenobi', 'dark_side': False, 'light_saber_color': 'blue'}
# ]

As you can observe, when you use inheritance, MongoEngine stores a field named '_cls' behind the scene to keep
track of the Document class.

Let's now take the scenario that you want to refactor the inheritance schema and:

• Have the Jedi's with dark_side=True/False become GoodJedi's/DarkSith
• get rid of the 'dark_side' field

move to the following schemas:

.. code-block:: python

# unchanged
class Human(Document):
    name = StringField()
    meta = {"allow_inheritance": True}

# attribute 'dark_side' removed
class GoodJedi(Human):
    light_saber_color = StringField()

# new class
class BadSith(Human):
    light_saber_color = StringField()

MongoEngine doesn't know about the change or how to map them with the existing data
so if you don't apply any migration, you will observe a strange behavior, as if the collection was suddenly
empty.

.. code-block:: python

# As a reminder, the documents that we inserted
# have the _cls field = 'Human.Jedi'

# Following has no match
# because the query that is used behind the scene is
# filtering on {'_cls': 'Human.GoodJedi'}
assert GoodJedi.objects().count() == 0

# Following has also no match
# because it is filtering on {'_cls': {'$in': ('Human', 'Human.GoodJedi', 'Human.BadSith')}}
# which has no match
assert Human.objects.count() == 0
assert Human.objects.first() is None

# If we bypass MongoEngine and make use of underlying driver (PyMongo)
# we can see that the documents are there
humans_coll = Human._get_collection()
assert humans_coll.count_documents({}) == 2
# print first document
print(humans_coll.find_one())
# {'_id': ObjectId('5fac4aaaf61d7fb06046e0f9'), '_cls': 'Human.Jedi', 'name': 'Darth Vader', 'dark_side': True, 'light_saber_color': 'red'}

As you can see, first obvious problem is that we need to modify '_cls' values based on existing values of
'dark_side' documents.

.. code-block:: python

humans_coll = Human._get_collection()
old_class = 'Human.Jedi'
good_jedi_class = 'Human.GoodJedi'
bad_sith_class = 'Human.BadSith'
humans_coll.update_many({'_cls': old_class, 'dark_side': False}, {'$set': {'_cls': good_jedi_class}})
humans_coll.update_many({'_cls': old_class, 'dark_side': True}, {'$set': {'_cls': bad_sith_class}})

Let's now check if querying improved in MongoEngine:

.. code-block:: python

assert GoodJedi.objects().count() == 1  # Hoorah!
assert BadSith.objects().count() == 1   # Hoorah!
assert Human.objects.count() == 2       # Hoorah!

# let's now check that documents load correctly
jedi = GoodJedi.objects().first()
# raises FieldDoesNotExist: The fields "{'dark_side'}" do not exist on the document "Human.GoodJedi"

In fact we only took care of renaming the _cls values but we havn't removed the 'dark_side' fields
which does not exist anymore on the GoodJedi's and BadSith's models.
Let's remove the field from the collections:

.. code-block:: python

humans_coll = Human._get_collection()
humans_coll.update_many({}, {'$unset': {'dark_side': 1}})

.. note:: We did this migration in 2 different steps for the sake of example but it could have been combined
with the migration of the _cls fields: ::

humans_coll.update_many(
        {'_cls': old_class, 'dark_side': False},
        {
            '$set': {'_cls': good_jedi_class},
            '$unset': {'dark_side': 1}
        }
    )

And verify that the documents now load correctly:

.. code-block:: python

jedi = GoodJedi.objects().first()
assert jedi.name == "Obi Wan Kenobi"

sith = BadSith.objects().first()
assert sith.name == "Darth Vader"

An other way of dealing with this migration is to iterate over
the documents and update/replace them one by one. This is way slower but
it is often useful for complex migrations of Document models.

.. code-block:: python

for doc in humans_coll.find():
    if doc['_cls'] == 'Human.Jedi':
        doc['_cls'] =  'Human.BadSith' if doc['dark_side'] else 'Human.GoodJedi'
        doc.pop('dark_side')
        humans_coll.replace_one({'_id': doc['_id']}, doc)

.. warning:: Be aware of this flaw <https://groups.google.com/g/mongodb-user/c/AFC1ia7MHzk>_ if you modify documents while iterating

Example 4: Index removal

If you remove an index from your Document class, or remove an indexed Field from your Document class,
you'll need to manually drop the corresponding index. MongoEngine will not do that for you.

The way to deal with this case is to identify the name of the index to drop with index_information(), and then drop
it with drop_index()

Let's for instance assume that you start with the following Document class

.. code-block:: python

class User(Document):
    name = StringField(index=True)

    meta = {"indexes": ["name"]}

User(name="John Doe").save()

As soon as you start interacting with the Document collection (when .save() is called in this case),
it would create the following indexes:

.. code-block:: python

print(User._get_collection().index_information())
# {
#  '_id_': {'key': [('_id', 1)], 'v': 2},
#  'name_1': {'background': False, 'key': [('name', 1)], 'v': 2},
# }

Thus: '_id' which is the default index and 'name_1' which is our custom index.
If you would remove the 'name' field or its index, you would have to call:

.. code-block:: python

User._get_collection().drop_index('name_1')

.. note:: When adding new fields or new indexes, MongoEngine will take care of creating them
(unless auto_create_index is disabled)

Recommendations

• Write migration scripts whenever you do changes to the model schemas
• Using :class:~mongoengine.DynamicDocument or meta = {"strict": False} may help to avoid some migrations or to have the 2 versions of your application to co-exist.
• Write post-processing checks to verify that migrations script worked. See below

Post-processing checks

The following recipe can be used to sanity check a Document collection after you applied migration.
It does not make any assumption on what was migrated, it will fetch 1000 objects randomly and
run some quick checks on the documents to make sure the document looks OK. As it is, it will fail
on the first occurrence of an error but this is something that can be adapted based on your needs.

.. code-block:: python

def get_random_oids(collection, sample_size):
    pipeline = [{"$project": {'_id': 1}}, {"$sample": {"size": sample_size}}]
    return [s['_id'] for s in collection.aggregate(pipeline)]

def get_random_documents(DocCls, sample_size):
    doc_collection = DocCls._get_collection()
    random_oids = get_random_oids(doc_collection, sample_size)
    return DocCls.objects(id__in=random_oids)

def check_documents(DocCls, sample_size):
    for doc in get_random_documents(DocCls, sample_size):
        # general validation (types and values)
        doc.validate()

        # load all subfields,
        # this may trigger additional queries if you have ReferenceFields
        # so it may be slow
        for field in doc._fields:
            try:
                getattr(doc, field)
            except Exception:
                LOG.warning(f"Could not load field {field} in Document {doc.id}")
                raise

check_documents(Human, sample_size=1000)

File: docs/guide/mongomock.rst Copy Chapter

========================= Use mongomock for testing

Although we recommend running your tests against a regular MongoDB server, it is sometimes useful to plug
MongoEngine to alternative implementations (mongomock, montydb, mongita, etc).

mongomock <https://github.com/mongomock/mongomock>_ is historically the one suggested for MongoEngine and is
a package to do just what the name implies, mocking a mongo database.

To use with mongoengine, simply specify mongomock when connecting with
mongoengine:

.. code-block:: python

import mongomock

connect('mongoenginetest', host='mongodb://localhost', mongo_client_class=mongomock.MongoClient)
conn = get_connection()

or with an alias:

.. code-block:: python

connect('mongoenginetest', host='mongodb://localhost', mongo_client_class=mongomock.MongoClient, alias='testdb')
conn = get_connection('testdb')

Example of test file: Copy Chapter

.. code-block:: python

import unittest
from mongoengine import connect, disconnect

class Person(Document):
    name = StringField()

class TestPerson(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        connect('mongoenginetest', host='mongodb://localhost', mongo_client_class=mongomock.MongoClient)

    @classmethod
    def tearDownClass(cls):
       disconnect()

    def test_thing(self):
        pers = Person(name='John')
        pers.save()

        fresh_pers = Person.objects().first()
        assert fresh_pers.name ==  'John'