tinydb Documentation

TinyDB is a lightweight document oriented database optimized for your happiness :)

7,572 stars Python Python 10 files · ~20,110 tokens #database#documentdb#json#nosql#python
RAW Doc

Technical Documentation: msiemens/tinydb

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Api (`docs/api.rst`)

.. _api_docs:

API Documentation

Architecture Overview

TinyDB is organized in three layers: the core API you interact with in
application code, a persistence layer that serializes database state, and
supporting components used internally for caching, updates, and query
hashing.

At the center is :class:~tinydb.database.TinyDB, which owns a
:class:~tinydb.storages.Storage instance and manages one or more
:class:~tinydb.table.Table objects. Each table stores documents as
:class:~tinydb.table.Document instances (dict-like objects that also expose
doc_id) and executes queries built from :class:~tinydb.queries.Query and
:class:~tinydb.queries.QueryInstance.

Persistence is handled by subclasses of :class:~tinydb.storages.Storage such
as :class:~tinydb.storages.JSONStorage and
:class:~tinydb.storages.MemoryStorage. You can wrap a storage with
:class:~tinydb.middlewares.Middleware implementations like
:class:~tinydb.middlewares.CachingMiddleware to change how reads and writes
are processed.

The diagram below summarizes how these pieces relate:

.. image:: /_static/architecture.svg
:alt: TinyDB class diagram showing relationships between TinyDB, Table, Document, Query, Storage, and supporting components
:align: center

Typical data flow

  1. You call methods on :class:~tinydb.database.TinyDB (for example
    insert or search). Unless you opened a named table explicitly,
    these calls are forwarded to the default table.
  2. The :class:~tinydb.table.Table evaluates a :class:~tinydb.queries.QueryInstance
    against its documents. Results may be served from the table's
    :class:~tinydb.utils.LRUCache query cache.
  3. When the table changes, it reads and writes the full database state through
    its :class:~tinydb.storages.Storage instance.
  4. For partial document updates, :mod:tinydb.operations provides helper
    callables that can be passed to :meth:~tinydb.table.Table.update.

If you plan to extend TinyDB, start with :doc:extend for custom storages and
middlewares, then use the class reference sections below for method-level
details.

``tinydb.database``

.. autoclass:: tinydb.database.TinyDB
:members:
:private-members:
:member-order: bysource

.. _table_api:

``tinydb.table``

.. autoclass:: tinydb.table.Table
:members:
:special-members:
:exclude-members: dict, weakref
:member-order: bysource

.. autoclass:: tinydb.table.Document
:members:
:special-members:
:exclude-members: dict, weakref
:member-order: bysource

text
.. py:attribute:: doc_id

    The document's id

``tinydb.queries``

.. autoclass:: tinydb.queries.Query
:members:
:special-members:
:exclude-members: weakref
:member-order: bysource

.. autoclass:: tinydb.queries.QueryInstance
:members:
:special-members:
:exclude-members: weakref
:member-order: bysource

``tinydb.operations``

.. automodule:: tinydb.operations
:members:
:special-members:
:exclude-members: weakref
:member-order: bysource

``tinydb.storage``

.. automodule:: tinydb.storages
:members: JSONStorage, MemoryStorage
:special-members:
:exclude-members: weakref

text
.. class:: Storage

    The abstract base class for all Storages.

    A Storage (de)serializes the current state of the database and stores
    it in some place (memory, file on disk, ...).

    .. method:: read()

        Read the last stored state.

    .. method:: write(data)

        Write the current state of the database to the storage.

    .. method:: close()

        Optional: Close open file handles, etc.

``tinydb.middlewares``

.. automodule:: tinydb.middlewares
:members: CachingMiddleware
:special-members:
:exclude-members: weakref

text
.. class:: Middleware

    The base class for all Middlewares.

    Middlewares hook into the read/write process of TinyDB allowing you to
    extend the behaviour by adding caching, logging, ...

    If ``read()`` or ``write()`` are not overloaded, they will be forwarded
    directly to the storage instance.

    .. attribute:: storage

        :type: :class:`.Storage`

        Access to the underlying storage instance.

    .. method:: read()

        Read the last stored state.

    .. method:: write(data)

        Write the current state of the database to the storage.

    .. method:: close()

        Optional: Close open file handles, etc.

``tinydb.utils``

.. autoclass:: tinydb.utils.LRUCache
:members:
:special-members:

Changelog (`docs/changelog.rst`)

Changelog

Version Numbering
^^^^^^^^^^^^^^^^^

TinyDB follows the SemVer versioning guidelines. For more information,
see semver.org <http://semver.org/>_

.. note:: When new methods are added to the Query API, this may
result in breaking existing code that uses the property syntax
to access document fields (e.g. Query().some.nested.field)
where the field name is equal to the newly added query method.
Thus, breaking changes may occur in feature releases even though
they don't change the public API in a backwards-incompatible
manner.

text
To prevent this from happening, one can use the dict access
      syntax (``Query()['some']['nested']['field']``) that will
      not break even when new methods are added to the ``Query`` API.

v4.9.0 (2026-08-06)
^^^^^^^^^^^^^^^^^^^

  • Drop official support for Python 3.8 and 3.9. Both versions have
    reached their official End of Life (Python 3.8 in October 2024,
    Python 3.9 in October 2025). TinyDB now requires Python 3.10 or later
  • Fix: Make Table.update(doc_ids=...) and Table.remove(doc_ids=...)
    behave consistently with Table.get(doc_id=...) when given document
    IDs that don't exist: missing IDs are now silently skipped and the
    returned list only contains IDs that were actually updated/removed.
    Previously both methods raised KeyError, and update could leave
    the table in a partially-updated state when a mix of existing and
    missing IDs was passed
    (see issue 591 <https://github.com/msiemens/tinydb/issues/591>_).
  • Fix: Prevent TypeError when passing unhashable extra arguments
    (like lists or dicts) to Query.test()
    (see issue 517 <https://github.com/msiemens/tinydb/issues/517>_).
  • Fix: Prevent FileNotFoundError when creating a database file in the
    current directory without an explicit path prefix
    (see pull request 619 <https://github.com/msiemens/tinydb/pull/619>_).
  • Fix: Correctly handle falsy values (like 0 or None) stored in
    LRUCache.
  • Improvement: More precise type hints for Table.get
    (see pull request 602 <https://github.com/msiemens/tinydb/pull/602>_).
  • Improvement: Correct the type hints for Table.update and
    tinydb.operations to use MutableMapping, so transform functions
    that modify documents in place type-check correctly
    (see issue 598 <https://github.com/msiemens/tinydb/issues/598>_).

v4.8.2 (2024-10-12)
^^^^^^^^^^^^^^^^^^^

  • Fix: Correctly update query cache when search results have changed
    (see issue 560 <https://github.com/msiemens/tinydb/issues/560>_).

v4.8.1 (2024-10-07)
^^^^^^^^^^^^^^^^^^^

  • Feature: Allow persisting empty tables
    (see pull request 518 <https://github.com/msiemens/tinydb/pull/518>_).
  • Fix: Make replacing doc_id type work properly
    (see issue 545 <https://github.com/msiemens/tinydb/issues/545>_).

v4.8.0 (2023-06-12)
^^^^^^^^^^^^^^^^^^^

  • Feature: Allow retrieve multiple documents by document ID using
    Table.get(doc_ids=[...])
    (see pull request 504 <https://github.com/msiemens/tinydb/pull/504>_).

v4.7.1 (2023-01-14)
^^^^^^^^^^^^^^^^^^^

  • Improvement: Improve typing annotations
    (see pull request 477 <https://github.com/msiemens/tinydb/pull/477>_).
  • Improvement: Fix some typos in the documentation
    (see pull request 479 <https://github.com/msiemens/tinydb/pull/479>_
    and pull request 498 <https://github.com/msiemens/tinydb/pull/498>_).

v4.7.0 (2022-02-19)
^^^^^^^^^^^^^^^^^^^

  • Feature: Allow inserting Document instances using Table.insert_multiple
    (see pull request 455 <https://github.com/msiemens/tinydb/pull/455>_).
  • Performance: Only convert document IDs of a table when returning documents.
    This improves performance the Table.count and Table.get operations
    and also for Table.search when only returning a few documents
    (see pull request 460 <https://github.com/msiemens/tinydb/pull/460>_).
  • Internal change: Run all Table tests JSONStorage in addition to
    MemoryStorage.

v4.6.1 (2022-01-18)
^^^^^^^^^^^^^^^^^^^

  • Fix: Make using callables as queries work again
    (see issue 454 <https://github.com/msiemens/tinydb/issues/454>__)

v4.6.0 (2022-01-17)
^^^^^^^^^^^^^^^^^^^

  • Feature: Add map() query operation to apply a transformation
    to a document or field when evaluating a query
    (see pull request 445 <https://github.com/msiemens/tinydb/pull/445>_).
    Note: This may break code that queries for a field named map
    using the Query APIs property access syntax
  • Feature: Add support for typing-extensions <https://pypi.org/project/typing-extensions/>_
    v4
  • Documentation: Fix a couple of typos in the documentation (see
    pull request 446 <https://github.com/msiemens/tinydb/pull/446>,
    pull request 449 <https://github.com/msiemens/tinydb/pull/449>     and
    pull request 453 <https://github.com/msiemens/tinydb/pull/453>_)

v4.5.2 (2021-09-23)
^^^^^^^^^^^^^^^^^^^

  • Fix: Make Table.delete()'s argument priorities consistent with
    other table methods. This means that if you pass both cond as
    well as doc_ids to Table.delete(), the latter will be preferred
    (see issue 424 <https://github.com/msiemens/tinydb/issues/424>__)

v4.5.1 (2021-07-17)
^^^^^^^^^^^^^^^^^^^

  • Fix: Correctly install typing-extensions on Python 3.7
    (see issue 413 <https://github.com/msiemens/tinydb/issues/413>__)

v4.5.0 (2021-06-25)
^^^^^^^^^^^^^^^^^^^

  • Feature: Better type hinting/IntelliSense for PyCharm, VS Code and MyPy
    (see issue 372 <https://github.com/msiemens/tinydb/issues/372>__).
    PyCharm and VS Code should work out of the box, for MyPy see
    :ref:MyPy Type Checking <mypy_type_checking>

v4.4.0 (2021-02-11)
^^^^^^^^^^^^^^^^^^^

  • Feature: Add operation for searching for all documents that match a dict
    fragment (see issue 300 <https://github.com/msiemens/tinydb/issues/300>_)
  • Fix: Correctly handle queries that use fields that are also Query methods,
    e.g. Query()['test'] for searching for documents with a test field
    (see issue 373 <https://github.com/msiemens/tinydb/issues/373>_)

v4.3.0 (2020-11-14)
^^^^^^^^^^^^^^^^^^^

  • Feature: Add operation for updating multiple documents: update_multiple
    (see issue 346 <https://github.com/msiemens/tinydb/issues/346>_)
  • Improvement: Expose type information for MyPy typechecking (PEP 561)
    (see pull request 352 <https://github.com/msiemens/tinydb/pull/352>_)

v4.2.0 (2020-10-03)
^^^^^^^^^^^^^^^^^^^

  • Feature: Add support for specifying document IDs during insertion
    (see issue 303 <https://github.com/msiemens/tinydb/issues/303>_)
  • Internal change: Use OrderedDict.move_to_end() in the query cache
    (see issue 338 <https://github.com/msiemens/tinydb/issues/338>_)

v4.1.1 (2020-05-08)
^^^^^^^^^^^^^^^^^^^

  • Fix: Don't install dev-dependencies when installing from PyPI (see
    issue 315 <https://github.com/msiemens/tinydb/issues/315>_)

v4.1.0 (2020-05-07)
^^^^^^^^^^^^^^^^^^^

  • Feature: Add a no-op query Query().noop() (see
    issue 313 <https://github.com/msiemens/tinydb/issues/313>_)
  • Feature: Add a access_mode flag to JSONStorage to allow opening
    files read-only (see issue 297 <https://github.com/msiemens/tinydb/issues/297>_)
  • Fix: Don't drop the first document that's being inserted when inserting
    data on an existing database (see issue 314 <https://github.com/msiemens/tinydb/issues/314>_)

v4.0.0 (2020-05-02)
^^^^^^^^^^^^^^^^^^^

:ref:Upgrade Notes

Breaking Changes

  • Python 2 support has been removed, see issue 284 <https://github.com/msiemens/tinydb/issues/284>_
    for background

  • API changes:

    • Removed classes: DataProxy, StorageProxy

    • Attributes removed from TinyDB in favor of
      customizing TinyDB's behavior by subclassing it and overloading
      __init__(...) and table(...):

      • DEFAULT_TABLE
      • DEFAULT_TABLE_KWARGS
      • DEFAULT_STORAGE

    • Arguments removed from TinyDB(...):

      • default_table: replace with TinyDB.default_table_name = 'name'
      • table_class: replace with TinyDB.table_class = Class

    • TinyDB.contains(...)'s doc_ids parameter has been renamed to
      doc_id and now only takes a single document ID

    • TinyDB.purge_tables(...) has been renamed to TinyDB.drop_tables(...)

    • TinyDB.purge_table(...) has been renamed to TinyDB.drop_table(...)

    • TinyDB.write_back(...) has been removed

    • TinyDB.process_elements(...) has been removed

    • Table.purge() has been renamed to Table.truncate()

    • Evaluating an empty Query() without any test operators will now result
      in an exception, use Query().noop() (introduced in v4.1.0) instead

  • ujson support has been removed, see issue 263 <https://github.com/msiemens/tinydb/issues/263>_ and issue 306 <https://github.com/msiemens/tinydb/issues/306>_ for background

  • The deprecated Element ID API has been removed (e.g. using the Element
    class or eids parameter) in favor the Document API, see
    pull request 158 <https://github.com/msiemens/tinydb/pull/158>_ for details
    on the replacement

Improvements

  • TinyDB's internal architecture has been reworked to be more simple and
    streamlined in order to make it easier to customize TinyDB's behavior
  • With the new architecture, TinyDB performance will improve for many
    applications

Bugfixes

  • Don't break the tests when ujson is installed (see issue 262 <https://github.com/msiemens/tinydb/issues/262>_)
  • Fix performance when reading data (see issue 250 <https://github.com/msiemens/tinydb/issues/250>_)
  • Fix inconsistent purge function names (see issue 103 <https://github.com/msiemens/tinydb/issues/103>_)

v3.15.1 (2019-10-26)
^^^^^^^^^^^^^^^^^^^^

  • Internal change: fix missing values handling for LRUCache

v3.15.0 (2019-10-12)
^^^^^^^^^^^^^^^^^^^^

  • Feature: allow setting the parameters of TinyDB's default table
    (see issue 278 <https://github.com/msiemens/tinydb/issues/278>_)

v3.14.2 (2019-09-13)
^^^^^^^^^^^^^^^^^^^^

  • Internal change: support correct iteration for LRUCache objects

v3.14.1 (2019-07-03)
^^^^^^^^^^^^^^^^^^^^

  • Internal change: fix Query class to permit subclass creation
    (see pull request 270 <https://github.com/msiemens/tinydb/pull/270>_)

v3.14.0 (2019-06-18)
^^^^^^^^^^^^^^^^^^^^

  • Change: support for ujson is now deprecated
    (see issue 263 <https://github.com/msiemens/tinydb/issues/263>_)

v3.13.0 (2019-03-16)
^^^^^^^^^^^^^^^^^^^^

  • Feature: direct access to a TinyDB instance's storage
    (see issue 258 <https://github.com/msiemens/tinydb/issues/258>_)

v3.12.2 (2018-12-12)
^^^^^^^^^^^^^^^^^^^^

  • Internal change: convert documents to dicts during insertion
    (see pull request 256 <https://github.com/msiemens/tinydb/pull/256>_)
  • Internal change: use tuple literals instead of tuple class/constructor
    (see pull request 247 <https://github.com/msiemens/tinydb/pull/247>_)
  • Infra: ensure YAML tests are run
    (see pull request 252 <https://github.com/msiemens/tinydb/pull/252>_)

v3.12.1 (2018-11-09)
^^^^^^^^^^^^^^^^^^^^

  • Fix: Don't break when searching the same query multiple times
    (see pull request 249 <https://github.com/msiemens/tinydb/pull/249>_)
  • Internal change: allow collections.abc.Mutable as valid document types
    (see pull request 245 <https://github.com/msiemens/tinydb/pull/245>_)

v3.12.0 (2018-11-06)
^^^^^^^^^^^^^^^^^^^^

  • Feature: Add encoding option to JSONStorage
    (see pull request 238 <https://github.com/msiemens/tinydb/pull/238>_)
  • Internal change: allow collections.abc.Mutable as valid document types
    (see pull request 245 <https://github.com/msiemens/tinydb/pull/245>_)

v3.11.1 (2018-09-13)
^^^^^^^^^^^^^^^^^^^^

  • Bugfix: Make path queries (db.search(where('key))) work again
    (see issue 232 <https://github.com/msiemens/tinydb/issues/232>_)
  • Improvement: Add custom repr representations for main classes
    (see pull request 229 <https://github.com/msiemens/tinydb/pull/229>_)

v3.11.0 (2018-08-20)
^^^^^^^^^^^^^^^^^^^^

  • Drop official support for Python 3.3. Python 3.3 has reached its
    official End Of Life as of September 29, 2017. It will probably continue
    to work, but will not be tested against
    (issue 217 <https://github.com/msiemens/tinydb/issues/217>_)

  • Feature: Allow extending TinyDB with a custom storage proxy class
    (see pull request 224 <https://github.com/msiemens/tinydb/pull/224>_)

  • Bugfix: Return list of document IDs for upsert when creating a new
    document (see issue 223 <https://github.com/msiemens/tinydb/issues/223>_)

v3.10.0 (2018-07-21)
^^^^^^^^^^^^^^^^^^^^

  • Feature: Add support for regex flags
    (see pull request 216 <https://github.com/msiemens/tinydb/pull/216>_)

v3.9.0 (2018-04-24)
^^^^^^^^^^^^^^^^^^^

  • Feature: Allow setting a table class for single table only
    (see issue 197 <https://github.com/msiemens/tinydb/issues/197>_)
  • Internal change: call fsync after flushing JSONStorage
    (see issue 208 <https://github.com/msiemens/tinydb/issues/208>_)

v3.8.1 (2018-03-26)
^^^^^^^^^^^^^^^^^^^

  • Bugfix: Don't install tests as a package anymore
    (see pull request #195 <https://github.com/msiemens/tinydb/pull/195>_)

v3.8.0 (2018-03-01)
^^^^^^^^^^^^^^^^^^^

  • Feature: Allow disabling the query cache with db.table(name, cache_size=0)
    (see pull request #187 <https://github.com/msiemens/tinydb/pull/187>_)
  • Feature: Add db.write_back(docs) for replacing documents
    (see pull request #184 <https://github.com/msiemens/tinydb/pull/184>_)

v3.7.0 (2017-11-11)
^^^^^^^^^^^^^^^^^^^

v3.6.0 (2017-10-05)
^^^^^^^^^^^^^^^^^^^

  • Allow updating all documents using db.update(fields) (see
    issue #157 <https://github.com/msiemens/tinydb/issues/157>_).
  • Rename elements to documents. Document IDs now available with doc.doc_id,
    using doc.eid is now deprecated
    (see pull request #158 <https://github.com/msiemens/tinydb/pull/158>_)

v3.5.0 (2017-08-30)
^^^^^^^^^^^^^^^^^^^

  • Expose the table name via table.name (see
    issue #147 <https://github.com/msiemens/tinydb/issues/147>_).
  • Allow better subclassing of the TinyDB class
    (see pull request #150 <https://github.com/msiemens/tinydb/pull/150>_).

v3.4.1 (2017-08-23)
^^^^^^^^^^^^^^^^^^^

  • Expose TinyDB version via import tinyb; tinydb.__version__ (see
    issue #148 <https://github.com/msiemens/tinydb/issues/148>_).

v3.4.0 (2017-08-08)
^^^^^^^^^^^^^^^^^^^

  • Add new update operations: add(key, value), subtract(key, value),
    and set(key, value)
    (see pull request #145 <https://github.com/msiemens/tinydb/pull/145>_).

v3.3.1 (2017-06-27)
^^^^^^^^^^^^^^^^^^^

  • Use relative imports to allow vendoring TinyDB in other packages
    (see pull request #142 <https://github.com/msiemens/tinydb/pull/142>_).

v3.3.0 (2017-06-05)
^^^^^^^^^^^^^^^^^^^

  • Allow iterating over a database or table yielding all documents
    (see pull request #139 <https://github.com/msiemens/tinydb/pull/139>_).

v3.2.3 (2017-04-22)
^^^^^^^^^^^^^^^^^^^

  • Fix bug with accidental modifications to the query cache when modifying
    the list of search results (see issue #132 <https://github.com/msiemens/tinydb/issues/132>_).

v3.2.2 (2017-01-16)
^^^^^^^^^^^^^^^^^^^

  • Fix the Query constructor to prevent wrong usage
    (see issue #117 <https://github.com/msiemens/tinydb/issues/117>_).

v3.2.1 (2016-06-29)
^^^^^^^^^^^^^^^^^^^

  • Fix a bug with queries on documents that have a path key
    (see pull request #107 <https://github.com/msiemens/tinydb/pull/107>_).
  • Don't write to the database file needlessly when opening the database
    (see pull request #104 <https://github.com/msiemens/tinydb/pull/104>_).

v3.2.0 (2016-04-25)
^^^^^^^^^^^^^^^^^^^

  • Add a way to specify the default table name via :ref:default_table <default_table>
    (see pull request #98 <https://github.com/msiemens/tinydb/pull/98>_).

  • Add db.purge_table(name) to remove a single table
    (see pull request #100 <https://github.com/msiemens/tinydb/pull/100>_).

    • Along the way: celebrating 100 issues and pull requests! Thanks everyone for every single contribution!

  • Extend API documentation (see issue #96 <https://github.com/msiemens/tinydb/issues/96>_).

v3.1.3 (2016-02-14)
^^^^^^^^^^^^^^^^^^^

  • Fix a bug when using unhashable documents (lists, dicts) with
    Query.any or Query.all queries
    (see a forum post by karibul <https://forum.m-siemens.de/d/4-error-with-any-and-all-queries>_).

v3.1.2 (2016-01-30)
^^^^^^^^^^^^^^^^^^^

  • Fix a bug when using unhashable documents (lists, dicts) with
    Query.any or Query.all queries
    (see a forum post by karibul <https://forum.m-siemens.de/d/4-error-with-any-and-all-queries>_).

v3.1.1 (2016-01-23)
^^^^^^^^^^^^^^^^^^^

  • Inserting a dictionary with data that is not JSON serializable doesn't
    lead to corrupt files anymore (see issue #89 <https://github.com/msiemens/tinydb/issues/89>_).
  • Fix a bug in the LRU cache that may lead to an invalid query cache
    (see issue #87 <https://github.com/msiemens/tinydb/issues/87>_).

v3.1.0 (2015-12-31)
^^^^^^^^^^^^^^^^^^^

  • db.update(...) and db.remove(...) now return affected document IDs
    (see issue #83 <https://github.com/msiemens/tinydb/issues/83>_).
  • Inserting an invalid document (i.e. not a dict) now raises an error
    instead of corrupting the database (see
    issue #74 <https://github.com/msiemens/tinydb/issues/74>_).

v3.0.0 (2015-11-13)
^^^^^^^^^^^^^^^^^^^

  • Overhauled Query model:

    • where('...').contains('...') has been renamed to
      where('...').search('...').

    • Support for ORM-like usage:
      User = Query(); db.search(User.name == 'John').

    • where('foo') is an alias for Query().foo.

    • where('foo').has('bar') is replaced by either
      where('foo').bar or Query().foo.bar.

      • In case the key is not a valid Python identifier, array
        notation can be used: where('a.b.c') is now
        Query()['a.b.c'].

    • Checking for the existence of a key has to be done explicitly:
      where('foo').exists().

  • Migrations from v1 to v2 have been removed.

  • SmartCacheTable has been moved to msiemens/tinydb-smartcache_.

  • Serialization has been moved to msiemens/tinydb-serialization_.

  • Empty storages are now expected to return None instead of raising ValueError.
    (see issue #67 <https://github.com/msiemens/tinydb/issues/67>_.

.. _msiemens/tinydb-smartcache: https://github.com/msiemens/tinydb-smartcache
.. _msiemens/tinydb-serialization: https://github.com/msiemens/tinydb-serialization

v2.4.0 (2015-08-14)
^^^^^^^^^^^^^^^^^^^

  • Allow custom parameters for custom test functions
    (see issue #63 <https://github.com/msiemens/tinydb/issues/63>_ and
    pull request #64 <https://github.com/msiemens/tinydb/pull/64>_).

v2.3.2 (2015-05-20)
^^^^^^^^^^^^^^^^^^^

  • Fix a forgotten debug output in the SerializationMiddleware
    (see issue #55 <https://github.com/msiemens/tinydb/issues/55>_).
  • Fix an "ignored exception" warning when using the CachingMiddleware
    (see pull request #54 <https://github.com/msiemens/tinydb/pull/54>_)
  • Fix a problem with symlinks when checking out TinyDB on OSX Yosemite
    (see issue #52 <https://github.com/msiemens/tinydb/issues/52>_).

v2.3.1 (2015-04-30)
^^^^^^^^^^^^^^^^^^^

  • Hopefully fix a problem with using TinyDB as a dependency in a setup.py script
    (see issue #51 <https://github.com/msiemens/tinydb/issues/51>_).

v2.3.0 (2015-04-08)
^^^^^^^^^^^^^^^^^^^

  • Added support for custom serialization. That way, you can teach TinyDB
    to store datetime objects in a JSON file :)
    (see issue #48 <https://github.com/msiemens/tinydb/issues/48>_ and
    pull request #50 <https://github.com/msiemens/tinydb/pull/50>_)
  • Fixed a performance regression when searching became slower with every search
    (see issue #49 <https://github.com/msiemens/tinydb/issues/49>_)
  • Internal code has been cleaned up

v2.2.2 (2015-02-12)
^^^^^^^^^^^^^^^^^^^

  • Fixed a data loss when using CachingMiddleware together with JSONStorage
    (see issue #47 <https://github.com/msiemens/tinydb/issues/47>_)

v2.2.1 (2015-01-09)
^^^^^^^^^^^^^^^^^^^

  • Fixed handling of IDs with the JSON backend that converted integers
    to strings (see issue #45 <https://github.com/msiemens/tinydb/issues/45>_)

v2.2.0 (2014-11-10)
^^^^^^^^^^^^^^^^^^^

  • Extended any and all queries to take lists as conditions
    (see pull request #38 <https://github.com/msiemens/tinydb/pull/38>_)
  • Fixed an decode error when installing TinyDB in a non-UTF-8 environment
    (see pull request #37 <https://github.com/msiemens/tinydb/pull/37>_)
  • Fixed some issues with CachingMiddleware in combination with
    JSONStorage (see pull request #39 <https://github.com/msiemens/tinydb/pull/39>_)

v2.1.0 (2014-10-14)
^^^^^^^^^^^^^^^^^^^

  • Added where(...).contains(regex)
    (see issue #32 <https://github.com/msiemens/tinydb/issues/32>_)
  • Fixed a bug that corrupted data after reopening a database
    (see issue #34 <https://github.com/msiemens/tinydb/issues/34>_)

v2.0.1 (2014-09-22)
^^^^^^^^^^^^^^^^^^^

  • Fixed handling of Unicode data in Python 2
    (see issue #28 <https://github.com/msiemens/tinydb/issues/28>_).

v2.0.0 (2014-09-05)
^^^^^^^^^^^^^^^^^^^

:ref:Upgrade Notes

.. warning:: TinyDB changed the way data is stored. You may need to migrate
your databases to the new scheme. Check out the
:ref:Upgrade Notes for details.

  • The syntax query in db has been removed, use db.contains instead.

  • The ConcurrencyMiddleware has been removed due to a insecure implementation
    (see issue #18 <https://github.com/msiemens/tinydb/issues/18>_). Consider
    :ref:tinyrecord instead.

  • Better support for working with :ref:Document IDs <document_ids>.

  • Added support for nested comparisons <http://tinydb.readthedocs.io/en/v2.0.0/usage.html#nested-queries>_.

  • Added all and any comparisons on lists <http://tinydb.readthedocs.io/en/v2.0.0/usage.html#nested-queries>_.

  • Added optional :http://tinydb.readthedocs.io/en/v2.0.0/usage.html#smart-query-cache`_.

  • The query cache is now a :ref:fixed size LRU cache <query_caching>.

v1.4.0 (2014-07-22)
^^^^^^^^^^^^^^^^^^^

  • Added insert_multiple function
    (see issue #8 <https://github.com/msiemens/tinydb/issues/8>_).

v1.3.0 (2014-07-02)
^^^^^^^^^^^^^^^^^^^

  • Fixed bug #7 <https://github.com/msiemens/tinydb/issues/7>_: IDs not unique.
  • Extended the API: db.count(where(...)) and db.contains(where(...)).
  • The syntax query in db is now deprecated and replaced
    by db.contains.

v1.2.0 (2014-06-19)
^^^^^^^^^^^^^^^^^^^

  • Added update method
    (see issue #6 <https://github.com/msiemens/tinydb/issues/6>_).

v1.1.1 (2014-06-14)
^^^^^^^^^^^^^^^^^^^

  • Merged PR #5 <https://github.com/msiemens/tinydb/pull/5>_: Fix minor
    documentation typos and style issues.

v1.1.0 (2014-05-06)
^^^^^^^^^^^^^^^^^^^

  • Improved the docs and fixed some typos.
  • Refactored some internal code.
  • Fixed a bug with multiple TinyDB? instances.

v1.0.1 (2014-04-26)
^^^^^^^^^^^^^^^^^^^

  • Fixed a bug in JSONStorage that broke the database when removing entries.

v1.0.0 (2013-07-20)
^^^^^^^^^^^^^^^^^^^

  • First official release – consider TinyDB stable now.

Contribute (`docs/contribute.rst`)

Contribution Guidelines
#######################

Whether reporting bugs, discussing improvements and new ideas or writing
extensions: Contributions to TinyDB are welcome! Here's how to get started:

  1. Check for open issues or open a fresh issue to start a discussion around
    a feature idea or a bug
  2. Fork the repository <https://github.com/msiemens/tinydb/>_ on Github,
    create a new branch off the master branch and start making your changes
    (known as GitHub Flow <https://guides.github.com/introduction/flow/index.html>_)
  3. Write a test which shows that the bug was fixed or that the feature works
    as expected
  4. Send a pull request and bug the maintainer until it gets merged and
    published :)

Philosophy of TinyDB

TinyDB aims to be simple and fun to use. Therefore two key values are simplicity
and elegance of interfaces and code. These values will contradict each other
from time to time. In these cases , try using as little magic as possible.
In any case don't forget documenting code that isn't clear at first glance.

Code Conventions

In general the TinyDB source should always follow PEP 8 <http://legacy.python.org/dev/peps/pep-0008/>_.
Exceptions are allowed in well justified and documented cases. However we make
a small exception concerning docstrings:

When using multiline docstrings, keep the opening and closing triple quotes
on their own lines and add an empty line after it.

.. code-block:: python

text
def some_function():
    """
    Documentation ...
    """

    # implementation ...

Version Numbers

TinyDB follows the SemVer versioning guidelines <http://semver.org/>_.
This implies that backwards incompatible changes in the API will increment
the major version. So think twice before making such changes.

Extend (`docs/extend.rst`)

How to Extend TinyDB

There are three main ways to extend TinyDB and modify its behaviour:

  1. custom storages,
  2. custom middlewares,
  3. use hooks and overrides, and
  4. subclassing TinyDB and Table.

Let's look at them in this order.

Write a Custom Storage

First, we have support for custom storages. By default TinyDB comes with an
in-memory storage and a JSON file storage. But of course you can add your own.
Let's look how you could add a YAML <http://yaml.org/>_ storage using
PyYAML <http://pyyaml.org/wiki/PyYAML>_:

.. code-block:: python

text
import yaml

class YAMLStorage(Storage):
    def __init__(self, filename):  # (1)
        self.filename = filename

    def read(self):
        with open(self.filename) as handle:
            try:
                data = yaml.safe_load(handle.read())  # (2)
                return data
            except yaml.YAMLError:
                return None  # (3)

    def write(self, data):
        with open(self.filename, 'w+') as handle:
            yaml.dump(data, handle)

    def close(self):  # (4)
        pass

There are some things we should look closer at:

  1. The constructor will receive all arguments passed to TinyDB when creating
    the database instance (except storage which TinyDB itself consumes).
    In other words calling TinyDB('something', storage=YAMLStorage) will
    pass 'something' as an argument to YAMLStorage.
    If you accept callables or other executable values in your storage
    constructor (or elsewhere), do not derive them from untrusted or
    user-controlled input.

  2. We use yaml.safe_load as recommended by the
    PyYAML documentation <http://pyyaml.org/wiki/PyYAMLDocumentation#LoadingYAML>_
    when processing data from a potentially untrusted source.

  3. If the storage is uninitialized, TinyDB expects the storage to return
    None so it can do any internal initialization that is necessary.

  4. If your storage needs any cleanup (like closing file handles) before an
    instance is destroyed, you can put it in the close() method. To run
    these, you'll either have to run db.close() on your TinyDB instance
    or use it as a context manager, like this:

    .. code-block:: python

    text
    with TinyDB('db.yml', storage=YAMLStorage) as db:
     # ...

Finally, using the YAML storage is very straight-forward:

.. code-block:: python

text
db = TinyDB('db.yml', storage=YAMLStorage)
# ...

Write Custom Middleware

Sometimes you don't want to write a new storage module but rather modify the
behaviour of an existing one. As an example we'll build middleware that filters
out empty items.

Because middleware acts as a wrapper around a storage, they needs a read()
and a write(data) method. In addition, they can access the underlying storage
via self.storage. Before we start implementing we should look at the structure
of the data that the middleware receives. Here's what the data that goes through
the middleware looks like:

.. code-block:: python

text
{
    '_default': {
        1: {'key': 'value'},
        2: {'key': 'value'},
        # other items
    },
    # other tables
}

Thus, we'll need two nested loops:

  1. Process every table
  2. Process every item

Now let's implement that:

.. code-block:: python

text
class RemoveEmptyItemsMiddleware(Middleware):
    def __init__(self, storage_cls):
        # Any middleware *has* to call the super constructor
        # with storage_cls
        super().__init__(storage_cls)  # (1)

    def read(self):
        data = self.storage.read()

        for table_name in data:
            table_data = data[table_name]

            for doc_id in table_data:
                item = table_data[doc_id]

                if item == {}:
                    del table_data[doc_id]

        return data

    def write(self, data):
        for table_name in data:
            table_data = data[table_name]

            for doc_id in table_data:
                item = table_data[doc_id]

                if item == {}:
                    del table_data[doc_id]

        self.storage.write(data)

    def close(self):
        self.storage.close()

Note that the constructor calls the middleware constructor (1) and passes
the storage class to the middleware constructor.

To wrap storage with this new middleware, we use it like this:

.. code-block:: python

text
db = TinyDB(storage=RemoveEmptyItemsMiddleware(SomeStorageClass))

Here SomeStorageClass should be replaced with the storage you want to use.
If you leave it empty, the default storage will be used (which is the JSONStorage).

Use hooks and overrides

.. _extend_hooks:

There are cases when neither creating a custom storage nor using a custom
middleware will allow you to adapt TinyDB in the way you need. In this case
you can modify TinyDB's behavior by using predefined hooks and override points.
For example you can configure the name of the default table by setting
TinyDB.default_table_name:

.. code-block:: python

text
TinyDB.default_table_name = 'my_table_name'

Both :class:~tinydb.database.TinyDB and the :class:~tinydb.table.Table
classes allow modifying their behavior using hooks and overrides. To use
Table's overrides, you can access the class using TinyDB.table_class:

.. code-block:: python

text
TinyDB.table_class.default_query_cache_capacity = 100

Read the :ref:api_docs for more details on the available hooks and override
points.

Subclassing ``TinyDB`` and ``Table``

Finally, there's the last option to modify TinyDB's behavior. That way you
can change how TinyDB itself works more deeply than using the other extension
mechanisms.

When creating a subclass you can use it by using hooks and overrides to override
the default classes that TinyDB uses:

.. code-block:: python

text
class MyTable(Table):
    # Add your method overrides
    ...

TinyDB.table_class = MyTable

# Continue using TinyDB as usual

TinyDB's source code is documented with extensions in mind, explaining how
everything works even for internal methods and classes. Feel free to dig into
the source and adapt everything you need for your projects.

Extensions (`docs/extensions.rst`)

Extensions

Here are some extensions that might be useful to you.

An extension is listed here if it

  • works with TinyDB 4 or newer,
  • supports Python 3,
  • is published on PyPI, and
  • has not been archived by its maintainers.

Extensions that stop meeting these requirements are removed from the list.

tinydb-rust

| Repo: https://github.com/itsmorninghao/tinydb-rust/
| Status: beta
| Description: A drop-in reimplementation of TinyDB that uses Rust for
better performance.

tinydbjoin

| Repo: https://github.com/FerdinaKusumah/tinydb-join
| Status: beta
| Description: Joins between TinyDB tables, even when they live in
different databases or storages: inner and left joins, and
nesting related documents into their parent (one-to-many
or belongs-to).

tinydbstorage

| Repo: https://github.com/FerdinaKusumah/tinydb-storage
| Status: beta
| Description: Redis and S3 storages for TinyDB, including S3-compatible
services such as MinIO and Cloudflare R2. A write that
would overwrite a change made by another process fails
instead.

aiotinydb

| Repo: https://github.com/ASMfreaK/aiotinydb
| Status: stable
| Description: asyncio compatibility shim for TinyDB. Enables usage of
TinyDB in asyncio-aware contexts without slow synchronous
IO.

BetterJSONStorage

| Repo: https://github.com/MrPigss/BetterJSONStorage
| Status: stable
| Description: BetterJSONStorage is a faster 'Storage Type' for TinyDB. It
uses the faster Orjson library for parsing the JSON and BLOSC
for compression.

tinydb-serialization

| Repo: https://github.com/msiemens/tinydb-serialization
| Status: stable
| Description: tinydb-serialization provides serialization for objects
that TinyDB otherwise couldn't handle.

tinydb-smartcache

| Repo: https://github.com/msiemens/tinydb-smartcache
| Status: stable
| Description: tinydb-smartcache provides a smart query cache for
TinyDB. It updates the query cache when
inserting/removing/updating documents so the cache doesn't
get invalidated. It's useful if you perform lots of queries
while the data changes only little.

.. _tinyrecord:

tinyrecord

| Repo: https://github.com/eugene-eeo/tinyrecord
| Status: stable
| Description: Tinyrecord is a library which implements experimental atomic
transaction support for the TinyDB NoSQL database. It uses a
record-first then execute architecture which allows us to
minimize the time that we are within a thread lock.

tinydb-appengine

| Repo: https://github.com/imalento/tinydb-appengine
| Status: inactive
| Description: tinydb-appengine provides TinyDB storage for
App Engine. You can use JSON readonly.

Getting Started (`docs/getting-started.rst`)

:tocdepth: 3

Getting Started

Installing TinyDB

To install TinyDB from PyPI, run::

text
$ pip install tinydb

You can also grab the latest development version from GitHub_. After downloading
and unpacking it, you can install it using::

text
$ pip install .

Basic Usage

Let's cover the basics before going more into detail. We'll start by setting up
a TinyDB database:

from tinydb import TinyDB, Query
db = TinyDB('db.json')

You now have a TinyDB database that stores its data in db.json.
What about inserting some data? TinyDB expects the data to be Python dict\s:

db.insert({'type': 'apple', 'count': 7})
db.insert({'type': 'peach', 'count': 3})

.. note:: The insert method returns the inserted document's ID. Read more
about it here: :ref:document_ids.

Now you can get all documents stored in the database by running:

db.all()
[{'count': 7, 'type': 'apple'}, {'count': 3, 'type': 'peach'}]

You can also iter over stored documents:

for item in db:
print(item)
{'count': 7, 'type': 'apple'}
{'count': 3, 'type': 'peach'}

Of course you'll also want to search for specific documents. Let's try:

Fruit = Query()
db.search(Fruit.type == 'peach')
[{'count': 3, 'type': 'peach'}]
db.search(Fruit.count > 5)
[{'count': 7, 'type': 'apple'}]

Next we'll update the count field of the apples:

db.update({'count': 10}, Fruit.type == 'apple')
db.all()
[{'count': 10, 'type': 'apple'}, {'count': 3, 'type': 'peach'}]

In the same manner you can also remove documents:

db.remove(Fruit.count < 5)
db.all()
[{'count': 10, 'type': 'apple'}]

And of course you can throw away all data to start with an empty database:

db.truncate()
db.all()
[]

Recap

Before we dive deeper, let's recapitulate the basics:

+-------------------------------+---------------------------------------------------------------+
| Inserting |
+-------------------------------+---------------------------------------------------------------+
| db.insert(...) | Insert a document |
+-------------------------------+---------------------------------------------------------------+
| Getting data |
+-------------------------------+---------------------------------------------------------------+
| db.all() | Get all documents |
+-------------------------------+---------------------------------------------------------------+
| iter(db) | Iter over all documents |
+-------------------------------+---------------------------------------------------------------+
| db.search(query) | Get a list of documents matching the query |
+-------------------------------+---------------------------------------------------------------+
| Updating |
+-------------------------------+---------------------------------------------------------------+
| db.update(fields, query) | Update all documents matching the query to contain fields |
+-------------------------------+---------------------------------------------------------------+
| Removing |
+-------------------------------+---------------------------------------------------------------+
| db.remove(query) | Remove all documents matching the query |
+-------------------------------+---------------------------------------------------------------+
| db.truncate() | Remove all documents |
+-------------------------------+---------------------------------------------------------------+
| Querying |
+-------------------------------+---------------------------------------------------------------+
| Query() | Create a new query object |
+-------------------------------+---------------------------------------------------------------+
| Query().field == 2 | Match any document that has a key field with value |
| | == 2 (also possible: !=, >, >=, <, <=)|
+-------------------------------+---------------------------------------------------------------+

.. note::

text
Query comparisons only support literal values on the right-hand side.
Field-to-field comparisons like ``Query().a == Query().b`` are not
supported. Use a callable predicate like
``db.search(lambda doc: doc.get('a') == doc.get('b'))`` for custom logic.

.. note::

text
Callables passed to query APIs (e.g. ``lambda`` predicates or ``Query().map``)
execute in-process and must **never** be derived from untrusted or user-controlled
input.

.. References
.. _GitHub: http://github.com/msiemens/tinydb/

Index (`docs/index.rst`)

Welcome to TinyDB!

Welcome to TinyDB, your tiny, document oriented database optimized for your
happiness :)

from tinydb import TinyDB, Query
db = TinyDB('path/to/db.json')
User = Query()
db.insert({'name': 'John', 'age': 22})
db.search(User.name == 'John')
[{'name': 'John', 'age': 22}]

User's Guide

.. toctree::
:maxdepth: 2

intro
getting-started
usage

Extending TinyDB

.. toctree::
:maxdepth: 2

Extending TinyDB
TinyDB Extensions

API Reference

.. toctree::
:maxdepth: 2

api

Additional Notes

.. toctree::
:maxdepth: 2

contribute
changelog
Upgrade Notes

Intro (`docs/intro.rst`)

Introduction

Great that you've taken time to check out the TinyDB docs! Before we begin
looking at TinyDB itself, let's take some time to see whether you should use
TinyDB.

Why Use TinyDB?

  • tiny: The current source code has 1800 lines of code (with about 40%
    documentation) and 1600 lines tests.

  • document oriented: Like MongoDB_, you can store any document
    (represented as dict) in TinyDB.

  • optimized for your happiness: TinyDB is designed to be simple and
    fun to use by providing a simple and clean API.

  • written in pure Python: TinyDB neither needs an external server (as
    e.g. PyMongo <https://pymongo.readthedocs.io/en/stable/>_) nor any dependencies
    from PyPI.

  • works on Python 3.5+ and PyPy: TinyDB works on all modern versions of Python
    and PyPy.

  • powerfully extensible: You can easily extend TinyDB by writing new
    storages or modify the behaviour of storages with Middlewares.

  • 100% test coverage: No explanation needed.

In short: If you need a simple database with a clean API that just works
without lots of configuration, TinyDB might be the right choice for you.

Why **Not** Use TinyDB?

  • You need advanced features like:
    • access from multiple processes or threads (e.g. when using Flask!),
    • creating indexes for tables,
    • an HTTP server,
    • managing relationships between tables or similar,
    • ACID guarantees <https://en.wikipedia.org/wiki/ACID>_.
  • You are really concerned about performance and need a high speed
    database.

To put it plainly: If you need advanced features or high performance, TinyDB
is the wrong database for you – consider using databases like SQLite_, Buzhug_,
CodernityDB_ or MongoDB_.

.. References
.. _Buzhug: https://buzhug.sourceforge.net/
.. _CodernityDB: http://labs.codernity.com/codernitydb/
.. _MongoDB: https://mongodb.org/
.. _SQLite: https://www.sqlite.org/

Upgrade (`docs/upgrade.rst`)

Upgrading to Newer Releases

Version 4.0

.. _upgrade_v4_0:

  • API changes:
    • Replace TinyDB.purge_tables(...) with TinyDB.drop_tables(...)
    • Replace TinyDB.purge_table(...) with TinyDB.drop_table(...)
    • Replace Table.purge() with Table.truncate()
    • Replace TinyDB(default_table='name') with TinyDB.default_table_name = 'name'
    • Replace TinyDB(table_class=Class) with TinyDB.table_class = Class
    • If you were using TinyDB.DEFAULT_TABLE, TinyDB.DEFAULT_TABLE_KWARGS,
      or TinyDB.DEFAULT_STORAGE: Use the new methods for customizing TinyDB
      described in :ref:How to Extend TinyDB <extend_hooks>

Version 3.0

.. _upgrade_v3_0:

Breaking API Changes
^^^^^^^^^^^^^^^^^^^^

  • Querying (see Issue #62 <https://github.com/msiemens/tinydb/issues/62>_):

    • where('...').contains('...') has been renamed to
      where('...').search('...').

    • where('foo').has('bar') is replaced by either
      where('foo').bar or Query().foo.bar.

      • In case the key is not a valid Python identifier, array
        notation can be used: where('a.b.c') is now
        Query()['a.b.c'].

  • Checking for the existence of a key has to be done explicitly:
    where('foo').exists().

  • SmartCacheTable has been moved to msiemens/tinydb-smartcache_.

  • Serialization has been moved to msiemens/tinydb-serialization_.

  • Empty storages are now expected to return None instead of raising
    ValueError (see Issue #67 <https://github.com/msiemens/tinydb/issues/67>_).

.. _msiemens/tinydb-smartcache: https://github.com/msiemens/tinydb-smartcache
.. _msiemens/tinydb-serialization: https://github.com/msiemens/tinydb-serialization

.. _upgrade_v2_0:

Version 2.0

Breaking API Changes
^^^^^^^^^^^^^^^^^^^^

  • The syntax query in db is not supported any more. Use db.contains(...)
    instead.
  • The ConcurrencyMiddleware has been removed due to a insecure implementation
    (see Issue #18 <https://github.com/msiemens/tinydb/issues/18>_). Consider
    :ref:tinyrecord instead.

Apart from that the API remains compatible to v1.4 and prior.

For migration from v1 to v2, check out the v2.0 documentation <http://tinydb.readthedocs.io/en/v2.0/upgrade.html#upgrade-v2-0>_

Usage (`docs/usage.rst`)

:tocdepth: 3

.. toctree::
:maxdepth: 2

Advanced Usage

Remarks on Storage

Before we dive deeper into the usage of TinyDB, we should stop for a moment
and discuss how TinyDB stores data.

To convert your data to a format that is writable to disk TinyDB uses the
Python JSON <http://docs.python.org/2/library/json.html>_ module by default.
It's great when only simple data types are involved but it cannot handle more
complex data types like custom classes. On Python 2 it also converts strings to
Unicode strings upon reading
(described here <http://stackoverflow.com/q/956867/997063>_).

If that causes problems, you can write
:doc:your own storage <extend>, that uses a more powerful (but also slower)
library like pickle <http://docs.python.org/library/pickle.html>_ or
PyYAML <http://pyyaml.org/>_.

.. hint:: Opening multiple TinyDB instances on the same data (e.g. with the
JSONStorage) may result in unexpected behavior due to query caching.
See query_caching_ on how to disable the query cache.

Queries

With that out of the way, let's start with TinyDB's rich set of queries.
There are two main ways to construct queries. The first one resembles the
syntax of popular ORM tools:

from tinydb import Query
User = Query()
db.search(User.name == 'John')

As you can see, we first create a new Query object and then use it to specify
which fields to check. Searching for nested fields is just as easy:

db.search(User.birthday.year == 1990)

Not all fields can be accessed this way if the field name is not a valid Python
identifier. In this case, you can switch to dict access notation:

This would be invalid Python syntax:

db.search(User.country-code == 'foo')

Use this instead:

db.search(User['country-code'] == 'foo')

In addition, you can use arbitrary transform function where a field would be,
for example:

from unidecode import unidecode
db.search(User.name.map(unidecode) == 'Jose')

will match 'José' etc.

The second, traditional way of constructing queries is as follows:

from tinydb import where
db.search(where('field') == 'value')

Using where('field') is a shorthand for the following code:

db.search(Query()['field'] == 'value')

Accessing nested fields with this syntax can be achieved like this:

db.search(where('birthday').year == 1900)
db.search(where('birthday')['year'] == 1900)

Advanced queries
................

In the :doc:getting-started you've learned about the basic comparisons
(==, <, >, ...). In addition to these TinyDB supports the following
queries:

Existence of a field:

db.search(User.name.exists())

Regex:

Full item has to match the regex:

db.search(User.name.matches('[aZ]*'))

Case insensitive search for 'John':

import re
db.search(User.name.matches('John', flags=re.IGNORECASE))

Any part of the item has to match the regex:

db.search(User.name.search('b+'))

Custom test:

test_func = lambda s: s == 'John'
db.search(User.name.test(test_func))

Custom test with parameters:

def test_func(val, m, n):
return m <= val <= n
db.search(User.age.test(test_func, 0, 21))
db.search(User.age.test(test_func, 21, 99))

Another case is if you have a dict where you want to find all documents
that match this dict. We call this searching for a fragment:

db.search(Query().fragment({'foo': True, 'bar': False}))
[{'foo': True, 'bar': False, 'foobar: 'yes!'}]

You also can search for documents where a specific field matches the fragment:

db.search(Query().field.fragment({'foo': True, 'bar': False}))
[{'field': {'foo': True, 'bar': False, 'foobar: 'yes!'}]

When a field contains a list, you also can use the any and all methods.
There are two ways to use them: with lists of values and with nested queries.
Let's start with the first one. Assuming we have a user object with a groups list
like this:

db.insert({'name': 'user1', 'groups': ['user']})
db.insert({'name': 'user2', 'groups': ['admin', 'user']})
db.insert({'name': 'user3', 'groups': ['sudo', 'user']})

Now we can use the following queries:

User's groups include at least one value from ['admin', 'sudo']

db.search(User.groups.any(['admin', 'sudo']))
[{'name': 'user2', 'groups': ['admin', 'user']},
{'name': 'user3', 'groups': ['sudo', 'user']}]

User's groups include all values from ['admin', 'user']

db.search(User.groups.all(['admin', 'user']))
[{'name': 'user2', 'groups': ['admin', 'user']}]

In some cases you may want to have more complex any/all queries.
This is where nested queries come in as helpful. Let's set up a table like this:

Group = Query()
Permission = Query()
groups = db.table('groups')
groups.insert({
'name': 'user',
'permissions': [{'type': 'read'}]})

groups.insert({
'name': 'sudo',
'permissions': [{'type': 'read'}, {'type': 'sudo'}]})

groups.insert({
'name': 'admin',
'permissions': [{'type': 'read'}, {'type': 'write'}, {'type': 'sudo'}]})

Now let's search this table using nested any/all queries:

Group has a permission with type 'read'

groups.search(Group.permissions.any(Permission.type == 'read'))
[{'name': 'user', 'permissions': [{'type': 'read'}]},
{'name': 'sudo', 'permissions': [{'type': 'read'}, {'type': 'sudo'}]},
{'name': 'admin', 'permissions':
[{'type': 'read'}, {'type': 'write'}, {'type': 'sudo'}]}]

Group has ONLY permission 'read'

groups.search(Group.permissions.all(Permission.type == 'read'))
[{'name': 'user', 'permissions': [{'type': 'read'}]}]

As you can see, any tests if there is at least one document matching
the query while all ensures all documents match the query.

The opposite operation, checking if a single item is contained in a list,
is also possible using one_of:

db.search(User.name.one_of(['jane', 'john']))

Query modifiers
...............

TinyDB also allows you to use logical operations to modify and combine
queries:

Negate a query:

db.search(~ (User.name == 'John'))

Logical AND:

db.search((User.name == 'John') & (User.age <= 30))

Logical OR:

db.search((User.name == 'John') | (User.name == 'Bob'))

.. note::

text
When using ``&`` or ``|``, make sure you wrap the conditions on both sides
with parentheses or Python will mess up the comparison.

Also, when using negation (``~``) you'll have to wrap the query you want
to negate in parentheses.

The reason for these requirements is that Python's binary operators that are
used for query modifiers have a higher operator precedence than comparison
operators. Simply put, ``~ User.name == 'John'`` is parsed by Python as
``(~User.name) == 'John'`` instead of ``~(User.name == 'John')``. See also the
Python `docs on operator precedence
<https://docs.python.org/3/reference/expressions.html#operator-precedence>`_
for details.

You can compose queries dynamically by using the no-op query ``Query().noop()``.

Comparisons only support literal values on the right-hand side. Field-to-field
comparisons like ``Query().a == Query().b`` are not supported. Use a callable
predicate like ``db.search(lambda doc: doc.get('a') == doc.get('b'))`` for
custom logic.

.. note::
Callables passed to query APIs (e.g. predicates, Query().map,
Query().test) execute in-process and must never be derived from
untrusted or user-controlled input.

Recap
.....

Let's review the query operations we've learned:

+-------------------------------------+---------------------------------------------------------------------+
| Queries |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.exists() | Match any document where a field called field exists |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.matches(regex) | Match any document with the whole field matching the |
| | regular expression |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.search(regex) | Match any document with a substring of the field matching |
| | the regular expression |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.test(func, *args) | Matches any document for which the function returns |
| | True |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.all(query | list) | If given a query, matches all documents where all documents |
| | in the list field match the query. |
| | If given a list, matches all documents where all documents |
| | in the list field are a member of the given list |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.any(query | list) | If given a query, matches all documents where at least one |
| | document in the list field match the query. |
| | If given a list, matches all documents where at least one |
| | documents in the list field are a member of the given |
| | list |
+-------------------------------------+---------------------------------------------------------------------+
| Query().field.one_of(list) | Match if the field is contained in the list |
+-------------------------------------+---------------------------------------------------------------------+
| Logical operations on queries |
+-------------------------------------+---------------------------------------------------------------------+
| ~ (query) | Match documents that don't match the query (logical NOT) |
+-------------------------------------+---------------------------------------------------------------------+
| (query1) & (query2) | Match documents that match both queries (logical AND) |
+-------------------------------------+---------------------------------------------------------------------+
| (query1) | (query2) | Match documents that match at least one of the queries (logical OR) |
+-------------------------------------+---------------------------------------------------------------------+

Handling Data

Next, let's look at some more ways to insert, update and retrieve data from
your database.

Inserting data
..............

As already described you can insert a document using db.insert(...).
In case you want to insert multiple documents, you can use db.insert_multiple(...):

db.insert_multiple([
{'name': 'John', 'age': 22},
{'name': 'John', 'age': 37}])

db.insert_multiple({'int': 1, 'value': i} for i in range(2))

Also in some cases it may be useful to specify the document ID yourself when
inserting data. You can do that by using the :class:~tinydb.table.Document
class:

db.insert(Document({'name': 'John', 'age': 22}, doc_id=12))
12

The same is possible when using db.insert_multiple(...):

db.insert_multiple([
Document({'name': 'John', 'age': 22}, doc_id=12),
Document({'name': 'Jane', 'age': 24}, doc_id=14),

])
[12, 14]

.. note::
Inserting a Document with an ID that already exists will result
in a ValueError being raised.

Updating data
.............

Sometimes you want to update all documents in your database. In this case, you
can leave out the query argument:

db.update({'foo': 'bar'})

When passing a dict to db.update(fields, query), it only allows you to
update a document by adding or overwriting its values. But sometimes you may
need to e.g. remove one field or increment its value. In that case you can
pass a function instead of fields:

from tinydb.operations import delete
db.update(delete('key1'), User.name == 'John')

This will remove the key key1 from all matching documents. TinyDB comes
with these operations:

  • delete(key): delete a key from the document
  • increment(key): increment the value of a key
  • decrement(key): decrement the value of a key
  • add(key, value): add value to the value of a key (also works for strings)
  • subtract(key, value): subtract value from the value of a key
  • set(key, value): set key to value

Of course you also can write your own operations:

def your_operation(your_arguments):
... def transform(doc):
... # do something with the document
... # ...
... return transform
...
db.update(your_operation(arguments), query)

In order to perform multiple update operations at once, you can use the
update_multiple method like this:

db.update_multiple([
... ({'int': 2}, where('char') == 'a'),
... ({'int': 4}, where('char') == 'b'),
... ])

You also can mix normal updates with update operations:

db.update_multiple([
... ({'int': 2}, where('char') == 'a'),
... (delete('int'), where('char') == 'b'),
... ])

Data access and modification

Upserting data
..............

In some cases you'll need a mix of both update and insert: upsert.
This operation is provided a document and a query. If it finds any documents
matching the query, they will be updated with the data from the provided document.
On the other hand, if no matching document is found, it inserts the provided
document into the table:

db.upsert({'name': 'John', 'logged-in': True}, User.name == 'John')

This will update all users with the name John to have logged-in set to True.
If no matching user is found, a new document is inserted with both the name set
and the logged-in flag.

To use the ID of the document as matching criterion a :class:~tinydb.table.Document
with doc_id is passed instead of a query:

db.upsert(Document({'name': 'John', 'logged-in': True}, doc_id=12))

Retrieving data
...............

There are several ways to retrieve data from your database. For instance you
can get the number of stored documents:

len(db)
3

.. hint::
This will return the number of documents in the default table
(see the notes on the :ref:default table <default_table>).

Then of course you can use db.search(...) as described in the :doc:getting-started
section. But sometimes you want to get only one matching document. Instead of using

try:
... result = db.search(User.name == 'John')[0]
... except IndexError:
... pass

you can use db.get(...):

db.get(User.name == 'John')
{'name': 'John', 'age': 22}
db.get(User.name == 'Bobby')
None

.. caution::

text
If multiple documents match the query, the first match in insertion order
is returned (not a random document).

Often you don't want to search for documents but only know whether they are
stored in the database. In this case db.contains(...) is your friend:

db.contains(User.name == 'John')

In a similar manner you can look up the number of documents matching a query:

db.count(User.name == 'John')
2

Recap
^^^^^

Let's summarize the ways to handle data:

+-------------------------------+---------------------------------------------------------------+
| Inserting data |
+-------------------------------+---------------------------------------------------------------+
| db.insert_multiple(...) | Insert multiple documents |
+-------------------------------+---------------------------------------------------------------+
| Updating data |
+-------------------------------+---------------------------------------------------------------+
| db.update(operation, ...) | Update all matching documents with a special operation |
+-------------------------------+---------------------------------------------------------------+
| Retrieving data |
+-------------------------------+---------------------------------------------------------------+
| len(db) | Get the number of documents in the database |
+-------------------------------+---------------------------------------------------------------+
| db.get(query) | Get one document matching the query |
+-------------------------------+---------------------------------------------------------------+
| db.contains(query) | Check if the database contains a matching document |
+-------------------------------+---------------------------------------------------------------+
| db.count(query) | Get the number of matching documents |
+-------------------------------+---------------------------------------------------------------+

.. note::

text
This was a new feature in v3.6.0

.. _document_ids:

Using Document IDs

Internally TinyDB associates an ID with every document you insert. It's returned
after inserting a document:

db.insert({'name': 'John', 'age': 22})
3
db.insert_multiple([{...}, {...}, {...}])
[4, 5, 6]

In addition you can get the ID of already inserted documents using
document.doc_id. This works both with get and all:

el = db.get(User.name == 'John')
el.doc_id
3
el = db.all()[0]
el.doc_id
1
el = db.all()[-1]
el.doc_id
12

Different TinyDB methods also work with IDs, namely: update, remove,
contains and get. The first two also return a list of affected IDs.

db.update({'value': 2}, doc_ids=[1, 2])
db.contains(doc_id=1)
True
db.remove(doc_ids=[1, 2])
db.get(doc_id=3)
{...}
db.get(doc_ids=[1, 2])
[{...}, {...}]

Using doc_id/doc_ids instead of Query() again is slightly faster
in operation.

Moving documents between databases
..................................

Documents returned by TinyDB are :class:~tinydb.table.Document instances.
They look like plain dicts, but they also carry the document ID assigned by
the table they came from. If you insert such a Document into another
table or database, TinyDB reuses that ID. When the target table already has
a document with the same ID, insertion fails with ValueError: Document with ID ... already exists.

This often shows up when moving documents from one database to another:

db1 = TinyDB('db1.json')
db2 = TinyDB('db2.json')
document = db1.get(doc_id=1)
db2.insert(document) # reuses doc_id=1 from db1
db1.remove(doc_ids=[1])

To copy the document's data and let the target table assign a new ID, convert
the Document to a plain dict first:

db2.insert(dict(document))

The same applies when moving documents between named tables in a single
database.

Recap
.....

Let's sum up the way TinyDB supports working with IDs:

+-------------------------------------+------------------------------------------------------------+
| Getting a document's ID |
+-------------------------------------+------------------------------------------------------------+
| db.insert(...) | Returns the inserted document's ID |
+-------------------------------------+------------------------------------------------------------+
| db.insert_multiple(...) | Returns the inserted documents' ID |
+-------------------------------------+------------------------------------------------------------+
| document.doc_id | Get the ID of a document fetched from the db |
+-------------------------------------+------------------------------------------------------------+
| Working with IDs |
+-------------------------------------+------------------------------------------------------------+
| db.get(doc_id=...) | Get the document with the given ID |
+-------------------------------------+------------------------------------------------------------+
| db.contains(doc_id=...) | Check if the db contains a document with the given |
| | IDs |
+-------------------------------------+------------------------------------------------------------+
| db.update({...}, doc_ids=[...]) | Update all documents with the given IDs |
+-------------------------------------+------------------------------------------------------------+
| db.remove(doc_ids=[...]) | Remove all documents with the given IDs |
+-------------------------------------+------------------------------------------------------------+

Tables

TinyDB supports working with multiple tables. They behave just the same as
the TinyDB class. To create and use a table, use db.table(name).

table = db.table('table_name')
table.insert({'value': True})
table.all()
[{'value': True}]
for row in table:
print(row)
{'value': True}

To remove a table from a database, use:

db.drop_table('table_name')

If on the other hand you want to remove all tables, use the counterpart:

db.drop_tables()

Finally, you can get a list with the names of all tables in your database:

db.tables()
{'_default', 'table_name'}

.. _default_table:

Default Table
.............

TinyDB uses a table named _default as the default table. All operations
on the database object (like db.insert(...)) operate on this table.
The name of this table can be modified by setting the default_table_name
class variable to modify the default table name for all instances:

#1: for a single instance only
db = TinyDB(storage=SomeStorage)
db.default_table_name = 'my-default'
#2: for all instances
TinyDB.default_table_name = 'my-default'

.. _query_caching:

Query Caching
.............

TinyDB caches query result for performance. That way re-running a query won't
have to read the data from the storage as long as the database hasn't been
modified. You can optimize the query cache size by passing the cache_size
to the table(...) function:

table = db.table('table_name', cache_size=30)

.. hint:: You can set cache_size to None to make the cache unlimited in
size. Also, you can set cache_size to 0 to disable it.

.. hint:: A repeated search with the same query is served from the cache.
The returned list is a fresh copy, but the documents inside it are the
same objects the cache holds — treat them as read-only. Mutating a
returned document (a key or a nested value) corrupts what a later
identical search returns.

.. hint:: It's not possible to open the same table multiple times with different
settings. After the first invocation, all the subsequent calls will return
the same table with the same settings as the first one.

.. hint:: The TinyDB query cache doesn't check if the underlying storage
that the database uses has been modified by an external process. In this
case the query cache may return outdated results. To clear the cache and
read data from the storage again you can use db.clear_cache().

.. hint:: When using an unlimited cache size and test() queries, TinyDB
will store a reference to the test function. As a result of that behavior
long-running applications that use lambda functions as a test function
may experience memory leaks.

Storage & Middleware

Storage Types
.............

TinyDB comes with two storage types: JSON and in-memory. By
default TinyDB stores its data in JSON files so you have to specify the path
where to store it:

from tinydb import TinyDB, where
db = TinyDB('path/to/db.json')

To use the in-memory storage, use:

from tinydb.storages import MemoryStorage
db = TinyDB(storage=MemoryStorage)

.. hint::
All arguments except for the storage argument are forwarded to the
underlying storage. For the JSON storage you can use this to pass
additional keyword arguments to Python's
json.dumps(...) <https://docs.python.org/3/library/json.html#json.dumps>_
method. For example, you can set it to create prettified JSON files like
this:

text
>>> db = TinyDB('db.json', sort_keys=True, indent=4, separators=(',', ': '))

.. note::
JSONStorage forwards **kwargs to json.dumps. Never pass
user-controlled values for callable arguments (e.g. default or
cls) as they are executed in-process on every write operation.

To modify the default storage for all TinyDB instances, set the
default_storage_class class variable:

TinyDB.default_storage_class = MemoryStorage

In case you need to access the storage instance directly, you can use the
storage property of your TinyDB instance. This may be useful to call
method directly on the storage or middleware:

db = TinyDB(storage=CachingMiddleware(MemoryStorage))
<tinydb.middlewares.CachingMiddleware at 0x10991def0>
db.storage.flush()

Middleware
..........

Middleware wraps around existing storage allowing you to customize their
behaviour.

from tinydb.storages import JSONStorage
from tinydb.middlewares import CachingMiddleware
db = TinyDB('/path/to/db.json', storage=CachingMiddleware(JSONStorage))

.. hint::
You can nest middleware:

text
>>> db = TinyDB('/path/to/db.json',
                storage=FirstMiddleware(SecondMiddleware(JSONStorage)))

CachingMiddleware
^^^^^^^^^^^^^^^^^

The CachingMiddleware improves speed by reducing disk I/O. It caches all
read operations and writes data to disk after a configured number of
write operations.

To make sure that all data is safely written when closing the table, use one
of these ways:

.. code-block:: python

text
# Using a context manager:
with database as db:
    # Your operations

.. code-block:: python

text
# Using the close function
db.close()

.. _mypy_type_checking:

MyPy Type Checking

TinyDB comes with type annotations that MyPy can use to make sure you're using
the API correctly. Unfortunately, MyPy doesn't understand all code patterns
that TinyDB uses. For that reason TinyDB ships a MyPy plugin that helps
correctly type checking code that uses TinyDB. To use it, add it to the
plugins list in the MyPy configuration file <https://mypy.readthedocs.io/en/latest/config_file.html>_
(typically located in setup.cfg or mypy.ini):

.. code-block:: ini

text
[mypy]
plugins = tinydb.mypy_plugin

What's next

Congratulations, you've made through the user guide! Now go and build something
awesome or dive deeper into TinyDB with these resources:

  • Want to learn how to customize TinyDB (storages, middlewares) and what
    extensions exist? Check out :doc:extend and :doc:extensions.
  • Want to study the API in detail? Read :doc:api.
  • Interested in contributing to the TinyDB development guide? Go on to the
    :doc:contribute.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#database Browse database catalog → #documentdb Browse documentdb catalog → #json Browse json catalog → #nosql Browse nosql catalog → #python Browse python catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →