# Technical Documentation: msiemens/tinydb
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/msiemens/tinydb)
### Api (`docs/api.rst`)
.. _api_docs:
API Documentation
=================
Architecture Overview
---------------------
TinyDB is organized in three layers: the **core API** you interact with in
application code, a **persistence layer** that serializes database state, and
**supporting components** used internally for caching, updates, and query
hashing.
At the center is :class:`~tinydb.database.TinyDB`, which owns a
:class:`~tinydb.storages.Storage` instance and manages one or more
:class:`~tinydb.table.Table` objects. Each table stores documents as
:class:`~tinydb.table.Document` instances (dict-like objects that also expose
``doc_id``) and executes queries built from :class:`~tinydb.queries.Query` and
:class:`~tinydb.queries.QueryInstance`.
Persistence is handled by subclasses of :class:`~tinydb.storages.Storage` such
as :class:`~tinydb.storages.JSONStorage` and
:class:`~tinydb.storages.MemoryStorage`. You can wrap a storage with
:class:`~tinydb.middlewares.Middleware` implementations like
:class:`~tinydb.middlewares.CachingMiddleware` to change how reads and writes
are processed.
The diagram below summarizes how these pieces relate:
.. image:: /_static/architecture.svg
:alt: TinyDB class diagram showing relationships between TinyDB, Table, Document, Query, Storage, and supporting components
:align: center
**Typical data flow**
1. You call methods on :class:`~tinydb.database.TinyDB` (for example
``insert`` or ``search``). Unless you opened a named table explicitly,
these calls are forwarded to the default table.
2. The :class:`~tinydb.table.Table` evaluates a :class:`~tinydb.queries.QueryInstance`
against its documents. Results may be served from the table's
:class:`~tinydb.utils.LRUCache` query cache.
3. When the table changes, it reads and writes the full database state through
its :class:`~tinydb.storages.Storage` instance.
4. For partial document updates, :mod:`tinydb.operations` provides helper
callables that can be passed to :meth:`~tinydb.table.Table.update`.
If you plan to extend TinyDB, start with :doc:`extend` for custom storages and
middlewares, then use the class reference sections below for method-level
details.
``tinydb.database``
-------------------
.. autoclass:: tinydb.database.TinyDB
:members:
:private-members:
:member-order: bysource
.. _table_api:
``tinydb.table``
----------------
.. autoclass:: tinydb.table.Table
:members:
:special-members:
:exclude-members: __dict__, __weakref__
:member-order: bysource
.. autoclass:: tinydb.table.Document
:members:
:special-members:
:exclude-members: __dict__, __weakref__
:member-order: bysource
.. py:attribute:: doc_id
The document's id
``tinydb.queries``
------------------
.. autoclass:: tinydb.queries.Query
:members:
:special-members:
:exclude-members: __weakref__
:member-order: bysource
.. autoclass:: tinydb.queries.QueryInstance
:members:
:special-members:
:exclude-members: __weakref__
:member-order: bysource
``tinydb.operations``
---------------------
.. automodule:: tinydb.operations
:members:
:special-members:
:exclude-members: __weakref__
:member-order: bysource
``tinydb.storage``
------------------
.. automodule:: tinydb.storages
:members: JSONStorage, MemoryStorage
:special-members:
:exclude-members: __weakref__
.. class:: Storage
The abstract base class for all Storages.
A Storage (de)serializes the current state of the database and stores
it in some place (memory, file on disk, ...).
.. method:: read()
Read the last stored state.
.. method:: write(data)
Write the current state of the database to the storage.
.. method:: close()
Optional: Close open file handles, etc.
``tinydb.middlewares``
----------------------
.. automodule:: tinydb.middlewares
:members: CachingMiddleware
:special-members:
:exclude-members: __weakref__
.. class:: Middleware
The base class for all Middlewares.
Middlewares hook into the read/write process of TinyDB allowing you to
extend the behaviour by adding caching, logging, ...
If ``read()`` or ``write()`` are not overloaded, they will be forwarded
directly to the storage instance.
.. attribute:: storage
:type: :class:`.Storage`
Access to the underlying storage instance.
.. method:: read()
Read the last stored state.
.. method:: write(data)
Write the current state of the database to the storage.
.. method:: close()
Optional: Close open file handles, etc.
``tinydb.utils``
----------------
.. autoclass:: tinydb.utils.LRUCache
:members:
:special-members:
---
### Changelog (`docs/changelog.rst`)
Changelog
=========
Version Numbering
^^^^^^^^^^^^^^^^^
TinyDB follows the SemVer versioning guidelines. For more information,
see `semver.org `_
.. note:: When new methods are added to the ``Query`` API, this may
result in breaking existing code that uses the property syntax
to access document fields (e.g. ``Query().some.nested.field``)
where the field name is equal to the newly added query method.
Thus, breaking changes may occur in feature releases even though
they don't change the public API in a backwards-incompatible
manner.
To prevent this from happening, one can use the dict access
syntax (``Query()['some']['nested']['field']``) that will
not break even when new methods are added to the ``Query`` API.
v4.9.0 (2026-08-06)
^^^^^^^^^^^^^^^^^^^
- **Drop official support for Python 3.8 and 3.9**. Both versions have
reached their official End of Life (Python 3.8 in October 2024,
Python 3.9 in October 2025). TinyDB now requires Python 3.10 or later
- Fix: Make ``Table.update(doc_ids=...)`` and ``Table.remove(doc_ids=...)``
behave consistently with ``Table.get(doc_id=...)`` when given document
IDs that don't exist: missing IDs are now silently skipped and the
returned list only contains IDs that were actually updated/removed.
Previously both methods raised ``KeyError``, and ``update`` could leave
the table in a partially-updated state when a mix of existing and
missing IDs was passed
(see `issue 591 `_).
- Fix: Prevent ``TypeError`` when passing unhashable extra arguments
(like lists or dicts) to ``Query.test()``
(see `issue 517 `_).
- Fix: Prevent ``FileNotFoundError`` when creating a database file in the
current directory without an explicit path prefix
(see `pull request 619 `_).
- Fix: Correctly handle falsy values (like ``0`` or ``None``) stored in
``LRUCache``.
- Improvement: More precise type hints for ``Table.get``
(see `pull request 602 `_).
- Improvement: Correct the type hints for ``Table.update`` and
``tinydb.operations`` to use ``MutableMapping``, so transform functions
that modify documents in place type-check correctly
(see `issue 598 `_).
v4.8.2 (2024-10-12)
^^^^^^^^^^^^^^^^^^^
- Fix: Correctly update query cache when search results have changed
(see `issue 560 `_).
v4.8.1 (2024-10-07)
^^^^^^^^^^^^^^^^^^^
- Feature: Allow persisting empty tables
(see `pull request 518 `_).
- Fix: Make replacing ``doc_id`` type work properly
(see `issue 545 `_).
v4.8.0 (2023-06-12)
^^^^^^^^^^^^^^^^^^^
- Feature: Allow retrieve multiple documents by document ID using
``Table.get(doc_ids=[...])``
(see `pull request 504 `_).
v4.7.1 (2023-01-14)
^^^^^^^^^^^^^^^^^^^
- Improvement: Improve typing annotations
(see `pull request 477 `_).
- Improvement: Fix some typos in the documentation
(see `pull request 479 `_
and `pull request 498 `_).
v4.7.0 (2022-02-19)
^^^^^^^^^^^^^^^^^^^
- Feature: Allow inserting ``Document`` instances using ``Table.insert_multiple``
(see `pull request 455 `_).
- Performance: Only convert document IDs of a table when returning documents.
This improves performance the ``Table.count`` and ``Table.get`` operations
and also for ``Table.search`` when only returning a few documents
(see `pull request 460 `_).
- Internal change: Run all ``Table`` tests ``JSONStorage`` in addition to
``MemoryStorage``.
v4.6.1 (2022-01-18)
^^^^^^^^^^^^^^^^^^^
- Fix: Make using callables as queries work again
(see `issue 454 `__)
v4.6.0 (2022-01-17)
^^^^^^^^^^^^^^^^^^^
- Feature: Add `map()` query operation to apply a transformation
to a document or field when evaluating a query
(see `pull request 445 `_).
**Note**: This may break code that queries for a field named ``map``
using the ``Query`` APIs property access syntax
- Feature: Add support for `typing-extensions `_
v4
- Documentation: Fix a couple of typos in the documentation (see
`pull request 446 `_,
`pull request 449 `_ and
`pull request 453 `_)
v4.5.2 (2021-09-23)
^^^^^^^^^^^^^^^^^^^
- Fix: Make ``Table.delete()``'s argument priorities consistent with
other table methods. This means that if you pass both ``cond`` as
well as ``doc_ids`` to ``Table.delete()``, the latter will be preferred
(see `issue 424 `__)
v4.5.1 (2021-07-17)
^^^^^^^^^^^^^^^^^^^
- Fix: Correctly install ``typing-extensions`` on Python 3.7
(see `issue 413 `__)
v4.5.0 (2021-06-25)
^^^^^^^^^^^^^^^^^^^
- Feature: Better type hinting/IntelliSense for PyCharm, VS Code and MyPy
(see `issue 372 `__).
PyCharm and VS Code should work out of the box, for MyPy see
:ref:`MyPy Type Checking `
v4.4.0 (2021-02-11)
^^^^^^^^^^^^^^^^^^^
- Feature: Add operation for searching for all documents that match a ``dict``
fragment (see `issue 300 `_)
- Fix: Correctly handle queries that use fields that are also Query methods,
e.g. ``Query()['test']`` for searching for documents with a ``test`` field
(see `issue 373 `_)
v4.3.0 (2020-11-14)
^^^^^^^^^^^^^^^^^^^
- Feature: Add operation for updating multiple documents: ``update_multiple``
(see `issue 346 `_)
- Improvement: Expose type information for MyPy typechecking (PEP 561)
(see `pull request 352 `_)
v4.2.0 (2020-10-03)
^^^^^^^^^^^^^^^^^^^
- Feature: Add support for specifying document IDs during insertion
(see `issue 303 `_)
- Internal change: Use ``OrderedDict.move_to_end()`` in the query cache
(see `issue 338 `_)
v4.1.1 (2020-05-08)
^^^^^^^^^^^^^^^^^^^
- Fix: Don't install dev-dependencies when installing from PyPI (see
`issue 315 `_)
v4.1.0 (2020-05-07)
^^^^^^^^^^^^^^^^^^^
- Feature: Add a no-op query ``Query().noop()`` (see
`issue 313 `_)
- Feature: Add a ``access_mode`` flag to ``JSONStorage`` to allow opening
files read-only (see `issue 297 `_)
- Fix: Don't drop the first document that's being inserted when inserting
data on an existing database (see `issue 314
`_)
v4.0.0 (2020-05-02)
^^^^^^^^^^^^^^^^^^^
:ref:`Upgrade Notes `
Breaking Changes
----------------
- Python 2 support has been removed, see `issue 284
`_
for background
- API changes:
- Removed classes: ``DataProxy``, ``StorageProxy``
- Attributes removed from ``TinyDB`` in favor of
customizing ``TinyDB``'s behavior by subclassing it and overloading
``__init__(...)`` and ``table(...)``:
- ``DEFAULT_TABLE``
- ``DEFAULT_TABLE_KWARGS``
- ``DEFAULT_STORAGE``
- Arguments removed from ``TinyDB(...)``:
- ``default_table``: replace with ``TinyDB.default_table_name = 'name'``
- ``table_class``: replace with ``TinyDB.table_class = Class``
- ``TinyDB.contains(...)``'s ``doc_ids`` parameter has been renamed to
``doc_id`` and now only takes a single document ID
- ``TinyDB.purge_tables(...)`` has been renamed to ``TinyDB.drop_tables(...)``
- ``TinyDB.purge_table(...)`` has been renamed to ``TinyDB.drop_table(...)``
- ``TinyDB.write_back(...)`` has been removed
- ``TinyDB.process_elements(...)`` has been removed
- ``Table.purge()`` has been renamed to ``Table.truncate()``
- Evaluating an empty ``Query()`` without any test operators will now result
in an exception, use ``Query().noop()`` (introduced in v4.1.0) instead
- ``ujson`` support has been removed, see `issue 263
`_ and `issue 306
`_ for background
- The deprecated Element ID API has been removed (e.g. using the ``Element``
class or ``eids`` parameter) in favor the Document API, see
`pull request 158 `_ for details
on the replacement
Improvements
------------
- TinyDB's internal architecture has been reworked to be more simple and
streamlined in order to make it easier to customize TinyDB's behavior
- With the new architecture, TinyDB performance will improve for many
applications
Bugfixes
--------
- Don't break the tests when ``ujson`` is installed (see `issue 262
`_)
- Fix performance when reading data (see `issue 250
`_)
- Fix inconsistent purge function names (see `issue 103
`_)
v3.15.1 (2019-10-26)
^^^^^^^^^^^^^^^^^^^^
- Internal change: fix missing values handling for ``LRUCache``
v3.15.0 (2019-10-12)
^^^^^^^^^^^^^^^^^^^^
- Feature: allow setting the parameters of TinyDB's default table
(see `issue 278 `_)
v3.14.2 (2019-09-13)
^^^^^^^^^^^^^^^^^^^^
- Internal change: support correct iteration for ``LRUCache`` objects
v3.14.1 (2019-07-03)
^^^^^^^^^^^^^^^^^^^^
- Internal change: fix Query class to permit subclass creation
(see `pull request 270 `_)
v3.14.0 (2019-06-18)
^^^^^^^^^^^^^^^^^^^^
- Change: support for ``ujson`` is now deprecated
(see `issue 263 `_)
v3.13.0 (2019-03-16)
^^^^^^^^^^^^^^^^^^^^
- Feature: direct access to a TinyDB instance's storage
(see `issue 258 `_)
v3.12.2 (2018-12-12)
^^^^^^^^^^^^^^^^^^^^
- Internal change: convert documents to dicts during insertion
(see `pull request 256 `_)
- Internal change: use tuple literals instead of tuple class/constructor
(see `pull request 247 `_)
- Infra: ensure YAML tests are run
(see `pull request 252 `_)
v3.12.1 (2018-11-09)
^^^^^^^^^^^^^^^^^^^^
- Fix: Don't break when searching the same query multiple times
(see `pull request 249 `_)
- Internal change: allow ``collections.abc.Mutable`` as valid document types
(see `pull request 245 `_)
v3.12.0 (2018-11-06)
^^^^^^^^^^^^^^^^^^^^
- Feature: Add encoding option to ``JSONStorage``
(see `pull request 238 `_)
- Internal change: allow ``collections.abc.Mutable`` as valid document types
(see `pull request 245 `_)
v3.11.1 (2018-09-13)
^^^^^^^^^^^^^^^^^^^^
- Bugfix: Make path queries (``db.search(where('key))``) work again
(see `issue 232 `_)
- Improvement: Add custom ``repr`` representations for main classes
(see `pull request 229 `_)
v3.11.0 (2018-08-20)
^^^^^^^^^^^^^^^^^^^^
- **Drop official support for Python 3.3**. Python 3.3 has reached its
official End Of Life as of September 29, 2017. It will probably continue
to work, but will not be tested against
(`issue 217 `_)
- Feature: Allow extending TinyDB with a custom storage proxy class
(see `pull request 224 `_)
- Bugfix: Return list of document IDs for upsert when creating a new
document (see `issue 223 `_)
v3.10.0 (2018-07-21)
^^^^^^^^^^^^^^^^^^^^
- Feature: Add support for regex flags
(see `pull request 216 `_)
v3.9.0 (2018-04-24)
^^^^^^^^^^^^^^^^^^^
- Feature: Allow setting a table class for single table only
(see `issue 197 `_)
- Internal change: call fsync after flushing ``JSONStorage``
(see `issue 208 `_)
v3.8.1 (2018-03-26)
^^^^^^^^^^^^^^^^^^^
- Bugfix: Don't install tests as a package anymore
(see `pull request #195 `_)
v3.8.0 (2018-03-01)
^^^^^^^^^^^^^^^^^^^
- Feature: Allow disabling the query cache with ``db.table(name, cache_size=0)``
(see `pull request #187 `_)
- Feature: Add ``db.write_back(docs)`` for replacing documents
(see `pull request #184 `_)
v3.7.0 (2017-11-11)
^^^^^^^^^^^^^^^^^^^
- Feature: ``one_of`` for checking if a value is contained in a list
(see `issue 164 `_)
- Feature: Upsert (insert if document doesn't exist, otherwise update;
see https://forum.m-siemens.de/d/30-primary-key-well-sort-of)
- Internal change: don't read from storage twice during initialization
(see https://forum.m-siemens.de/d/28-reads-the-whole-data-file-twice)
v3.6.0 (2017-10-05)
^^^^^^^^^^^^^^^^^^^
- Allow updating all documents using ``db.update(fields)`` (see
`issue #157 `_).
- Rename elements to documents. Document IDs now available with ``doc.doc_id``,
using ``doc.eid`` is now deprecated
(see `pull request #158 `_)
v3.5.0 (2017-08-30)
^^^^^^^^^^^^^^^^^^^
- Expose the table name via ``table.name`` (see
`issue #147 `_).
- Allow better subclassing of the ``TinyDB`` class
(see `pull request #150 `_).
v3.4.1 (2017-08-23)
^^^^^^^^^^^^^^^^^^^
- Expose TinyDB version via ``import tinyb; tinydb.__version__`` (see
`issue #148 `_).
v3.4.0 (2017-08-08)
^^^^^^^^^^^^^^^^^^^
- Add new update operations: ``add(key, value)``, ``subtract(key, value)``,
and ``set(key, value)``
(see `pull request #145 `_).
v3.3.1 (2017-06-27)
^^^^^^^^^^^^^^^^^^^
- Use relative imports to allow vendoring TinyDB in other packages
(see `pull request #142 `_).
v3.3.0 (2017-06-05)
^^^^^^^^^^^^^^^^^^^
- Allow iterating over a database or table yielding all documents
(see `pull request #139 `_).
v3.2.3 (2017-04-22)
^^^^^^^^^^^^^^^^^^^
- Fix bug with accidental modifications to the query cache when modifying
the list of search results (see `issue #132 `_).
v3.2.2 (2017-01-16)
^^^^^^^^^^^^^^^^^^^
- Fix the ``Query`` constructor to prevent wrong usage
(see `issue #117 `_).
v3.2.1 (2016-06-29)
^^^^^^^^^^^^^^^^^^^
- Fix a bug with queries on documents that have a ``path`` key
(see `pull request #107 `_).
- Don't write to the database file needlessly when opening the database
(see `pull request #104 `_).
v3.2.0 (2016-04-25)
^^^^^^^^^^^^^^^^^^^
- Add a way to specify the default table name via :ref:`default_table `
(see `pull request #98 `_).
- Add ``db.purge_table(name)`` to remove a single table
(see `pull request #100 `_).
- Along the way: celebrating 100 issues and pull requests! Thanks everyone for every single contribution!
- Extend API documentation (see `issue #96 `_).
v3.1.3 (2016-02-14)
^^^^^^^^^^^^^^^^^^^
- Fix a bug when using unhashable documents (lists, dicts) with
``Query.any`` or ``Query.all`` queries
(see `a forum post by karibul `_).
v3.1.2 (2016-01-30)
^^^^^^^^^^^^^^^^^^^
- Fix a bug when using unhashable documents (lists, dicts) with
``Query.any`` or ``Query.all`` queries
(see `a forum post by karibul `_).
v3.1.1 (2016-01-23)
^^^^^^^^^^^^^^^^^^^
- Inserting a dictionary with data that is not JSON serializable doesn't
lead to corrupt files anymore (see `issue #89 `_).
- Fix a bug in the LRU cache that may lead to an invalid query cache
(see `issue #87 `_).
v3.1.0 (2015-12-31)
^^^^^^^^^^^^^^^^^^^
- ``db.update(...)`` and ``db.remove(...)`` now return affected document IDs
(see `issue #83 `_).
- Inserting an invalid document (i.e. not a ``dict``) now raises an error
instead of corrupting the database (see
`issue #74 `_).
v3.0.0 (2015-11-13)
^^^^^^^^^^^^^^^^^^^
- Overhauled Query model:
- ``where('...').contains('...')`` has been renamed to
``where('...').search('...')``.
- Support for ORM-like usage:
``User = Query(); db.search(User.name == 'John')``.
- ``where('foo')`` is an alias for ``Query().foo``.
- ``where('foo').has('bar')`` is replaced by either
``where('foo').bar`` or ``Query().foo.bar``.
- In case the key is not a valid Python identifier, array
notation can be used: ``where('a.b.c')`` is now
``Query()['a.b.c']``.
- Checking for the existence of a key has to be done explicitly:
``where('foo').exists()``.
- Migrations from v1 to v2 have been removed.
- ``SmartCacheTable`` has been moved to `msiemens/tinydb-smartcache`_.
- Serialization has been moved to `msiemens/tinydb-serialization`_.
- Empty storages are now expected to return ``None`` instead of raising ``ValueError``.
(see `issue #67 `_.
.. _msiemens/tinydb-smartcache: https://github.com/msiemens/tinydb-smartcache
.. _msiemens/tinydb-serialization: https://github.com/msiemens/tinydb-serialization
v2.4.0 (2015-08-14)
^^^^^^^^^^^^^^^^^^^
- Allow custom parameters for custom test functions
(see `issue #63 `_ and
`pull request #64 `_).
v2.3.2 (2015-05-20)
^^^^^^^^^^^^^^^^^^^
- Fix a forgotten debug output in the ``SerializationMiddleware``
(see `issue #55 `_).
- Fix an "ignored exception" warning when using the ``CachingMiddleware``
(see `pull request #54 `_)
- Fix a problem with symlinks when checking out TinyDB on OSX Yosemite
(see `issue #52 `_).
v2.3.1 (2015-04-30)
^^^^^^^^^^^^^^^^^^^
- Hopefully fix a problem with using TinyDB as a dependency in a ``setup.py`` script
(see `issue #51 `_).
v2.3.0 (2015-04-08)
^^^^^^^^^^^^^^^^^^^
- Added support for custom serialization. That way, you can teach TinyDB
to store ``datetime`` objects in a JSON file :)
(see `issue #48 `_ and
`pull request #50 `_)
- Fixed a performance regression when searching became slower with every search
(see `issue #49 `_)
- Internal code has been cleaned up
v2.2.2 (2015-02-12)
^^^^^^^^^^^^^^^^^^^
- Fixed a data loss when using ``CachingMiddleware`` together with ``JSONStorage``
(see `issue #47 `_)
v2.2.1 (2015-01-09)
^^^^^^^^^^^^^^^^^^^
- Fixed handling of IDs with the JSON backend that converted integers
to strings (see `issue #45 `_)
v2.2.0 (2014-11-10)
^^^^^^^^^^^^^^^^^^^
- Extended ``any`` and ``all`` queries to take lists as conditions
(see `pull request #38 `_)
- Fixed an ``decode error`` when installing TinyDB in a non-UTF-8 environment
(see `pull request #37 `_)
- Fixed some issues with ``CachingMiddleware`` in combination with
``JSONStorage`` (see `pull request #39 `_)
v2.1.0 (2014-10-14)
^^^^^^^^^^^^^^^^^^^
- Added ``where(...).contains(regex)``
(see `issue #32 `_)
- Fixed a bug that corrupted data after reopening a database
(see `issue #34 `_)
v2.0.1 (2014-09-22)
^^^^^^^^^^^^^^^^^^^
- Fixed handling of Unicode data in Python 2
(see `issue #28 `_).
v2.0.0 (2014-09-05)
^^^^^^^^^^^^^^^^^^^
:ref:`Upgrade Notes `
.. warning:: TinyDB changed the way data is stored. You may need to migrate
your databases to the new scheme. Check out the
:ref:`Upgrade Notes ` for details.
- The syntax ``query in db`` has been removed, use ``db.contains`` instead.
- The ``ConcurrencyMiddleware`` has been removed due to a insecure implementation
(see `issue #18 `_). Consider
:ref:`tinyrecord` instead.
- Better support for working with :ref:`Document IDs `.
- Added support for `nested comparisons `_.
- Added ``all`` and ``any`` `comparisons on lists `_.
- Added optional :`_.
- The query cache is now a :ref:`fixed size LRU cache `.
v1.4.0 (2014-07-22)
^^^^^^^^^^^^^^^^^^^
- Added ``insert_multiple`` function
(see `issue #8 `_).
v1.3.0 (2014-07-02)
^^^^^^^^^^^^^^^^^^^
- Fixed `bug #7 `_: IDs not unique.
- Extended the API: ``db.count(where(...))`` and ``db.contains(where(...))``.
- The syntax ``query in db`` is now **deprecated** and replaced
by ``db.contains``.
v1.2.0 (2014-06-19)
^^^^^^^^^^^^^^^^^^^
- Added ``update`` method
(see `issue #6 `_).
v1.1.1 (2014-06-14)
^^^^^^^^^^^^^^^^^^^
- Merged `PR #5 `_: Fix minor
documentation typos and style issues.
v1.1.0 (2014-05-06)
^^^^^^^^^^^^^^^^^^^
- Improved the docs and fixed some typos.
- Refactored some internal code.
- Fixed a bug with multiple ``TinyDB?`` instances.
v1.0.1 (2014-04-26)
^^^^^^^^^^^^^^^^^^^
- Fixed a bug in ``JSONStorage`` that broke the database when removing entries.
v1.0.0 (2013-07-20)
^^^^^^^^^^^^^^^^^^^
- First official release – consider TinyDB stable now.
---
### Contribute (`docs/contribute.rst`)
Contribution Guidelines
#######################
Whether reporting bugs, discussing improvements and new ideas or writing
extensions: Contributions to TinyDB are welcome! Here's how to get started:
1. Check for open issues or open a fresh issue to start a discussion around
a feature idea or a bug
2. Fork `the repository `_ on Github,
create a new branch off the `master` branch and start making your changes
(known as `GitHub Flow `_)
3. Write a test which shows that the bug was fixed or that the feature works
as expected
4. Send a pull request and bug the maintainer until it gets merged and
published :)
Philosophy of TinyDB
********************
TinyDB aims to be simple and fun to use. Therefore two key values are simplicity
and elegance of interfaces and code. These values will contradict each other
from time to time. In these cases , try using as little magic as possible.
In any case don't forget documenting code that isn't clear at first glance.
Code Conventions
****************
In general the TinyDB source should always follow `PEP 8 `_.
Exceptions are allowed in well justified and documented cases. However we make
a small exception concerning docstrings:
When using multiline docstrings, keep the opening and closing triple quotes
on their own lines and add an empty line after it.
.. code-block:: python
def some_function():
"""
Documentation ...
"""
# implementation ...
Version Numbers
***************
TinyDB follows the `SemVer versioning guidelines `_.
This implies that backwards incompatible changes in the API will increment
the major version. So think twice before making such changes.
---
### Extend (`docs/extend.rst`)
How to Extend TinyDB
====================
There are three main ways to extend TinyDB and modify its behaviour:
1. custom storages,
2. custom middlewares,
3. use hooks and overrides, and
4. subclassing ``TinyDB`` and ``Table``.
Let's look at them in this order.
Write a Custom Storage
----------------------
First, we have support for custom storages. By default TinyDB comes with an
in-memory storage and a JSON file storage. But of course you can add your own.
Let's look how you could add a `YAML `_ storage using
`PyYAML `_:
.. code-block:: python
import yaml
class YAMLStorage(Storage):
def __init__(self, filename): # (1)
self.filename = filename
def read(self):
with open(self.filename) as handle:
try:
data = yaml.safe_load(handle.read()) # (2)
return data
except yaml.YAMLError:
return None # (3)
def write(self, data):
with open(self.filename, 'w+') as handle:
yaml.dump(data, handle)
def close(self): # (4)
pass
There are some things we should look closer at:
1. The constructor will receive all arguments passed to TinyDB when creating
the database instance (except ``storage`` which TinyDB itself consumes).
In other words calling ``TinyDB('something', storage=YAMLStorage)`` will
pass ``'something'`` as an argument to ``YAMLStorage``.
If you accept callables or other executable values in your storage
constructor (or elsewhere), do not derive them from untrusted or
user-controlled input.
2. We use ``yaml.safe_load`` as recommended by the
`PyYAML documentation `_
when processing data from a potentially untrusted source.
3. If the storage is uninitialized, TinyDB expects the storage to return
``None`` so it can do any internal initialization that is necessary.
4. If your storage needs any cleanup (like closing file handles) before an
instance is destroyed, you can put it in the ``close()`` method. To run
these, you'll either have to run ``db.close()`` on your ``TinyDB`` instance
or use it as a context manager, like this:
.. code-block:: python
with TinyDB('db.yml', storage=YAMLStorage) as db:
# ...
Finally, using the YAML storage is very straight-forward:
.. code-block:: python
db = TinyDB('db.yml', storage=YAMLStorage)
# ...
Write Custom Middleware
-------------------------
Sometimes you don't want to write a new storage module but rather modify the
behaviour of an existing one. As an example we'll build middleware that filters
out empty items.
Because middleware acts as a wrapper around a storage, they needs a ``read()``
and a ``write(data)`` method. In addition, they can access the underlying storage
via ``self.storage``. Before we start implementing we should look at the structure
of the data that the middleware receives. Here's what the data that goes through
the middleware looks like:
.. code-block:: python
{
'_default': {
1: {'key': 'value'},
2: {'key': 'value'},
# other items
},
# other tables
}
Thus, we'll need two nested loops:
1. Process every table
2. Process every item
Now let's implement that:
.. code-block:: python
class RemoveEmptyItemsMiddleware(Middleware):
def __init__(self, storage_cls):
# Any middleware *has* to call the super constructor
# with storage_cls
super().__init__(storage_cls) # (1)
def read(self):
data = self.storage.read()
for table_name in data:
table_data = data[table_name]
for doc_id in table_data:
item = table_data[doc_id]
if item == {}:
del table_data[doc_id]
return data
def write(self, data):
for table_name in data:
table_data = data[table_name]
for doc_id in table_data:
item = table_data[doc_id]
if item == {}:
del table_data[doc_id]
self.storage.write(data)
def close(self):
self.storage.close()
Note that the constructor calls the middleware constructor (1) and passes
the storage class to the middleware constructor.
To wrap storage with this new middleware, we use it like this:
.. code-block:: python
db = TinyDB(storage=RemoveEmptyItemsMiddleware(SomeStorageClass))
Here ``SomeStorageClass`` should be replaced with the storage you want to use.
If you leave it empty, the default storage will be used (which is the ``JSONStorage``).
Use hooks and overrides
-----------------------
.. _extend_hooks:
There are cases when neither creating a custom storage nor using a custom
middleware will allow you to adapt TinyDB in the way you need. In this case
you can modify TinyDB's behavior by using predefined hooks and override points.
For example you can configure the name of the default table by setting
``TinyDB.default_table_name``:
.. code-block:: python
TinyDB.default_table_name = 'my_table_name'
Both :class:`~tinydb.database.TinyDB` and the :class:`~tinydb.table.Table`
classes allow modifying their behavior using hooks and overrides. To use
``Table``'s overrides, you can access the class using ``TinyDB.table_class``:
.. code-block:: python
TinyDB.table_class.default_query_cache_capacity = 100
Read the :ref:`api_docs` for more details on the available hooks and override
points.
Subclassing ``TinyDB`` and ``Table``
------------------------------------
Finally, there's the last option to modify TinyDB's behavior. That way you
can change how TinyDB itself works more deeply than using the other extension
mechanisms.
When creating a subclass you can use it by using hooks and overrides to override
the default classes that TinyDB uses:
.. code-block:: python
class MyTable(Table):
# Add your method overrides
...
TinyDB.table_class = MyTable
# Continue using TinyDB as usual
TinyDB's source code is documented with extensions in mind, explaining how
everything works even for internal methods and classes. Feel free to dig into
the source and adapt everything you need for your projects.
---
### Extensions (`docs/extensions.rst`)
Extensions
==========
Here are some extensions that might be useful to you.
An extension is listed here if it
* works with TinyDB 4 or newer,
* supports Python 3,
* is published on PyPI, and
* has not been archived by its maintainers.
Extensions that stop meeting these requirements are removed from the list.
``tinydb-rust``
**************
| **Repo:** https://github.com/itsmorninghao/tinydb-rust/
| **Status:** *beta*
| **Description:** A drop-in reimplementation of TinyDB that uses Rust for
better performance.
``tinydbjoin``
**************
| **Repo:** https://github.com/FerdinaKusumah/tinydb-join
| **Status:** *beta*
| **Description:** Joins between TinyDB tables, even when they live in
different databases or storages: inner and left joins, and
nesting related documents into their parent (one-to-many
or belongs-to).
``tinydbstorage``
*****************
| **Repo:** https://github.com/FerdinaKusumah/tinydb-storage
| **Status:** *beta*
| **Description:** Redis and S3 storages for TinyDB, including S3-compatible
services such as MinIO and Cloudflare R2. A write that
would overwrite a change made by another process fails
instead.
``aiotinydb``
*************
| **Repo:** https://github.com/ASMfreaK/aiotinydb
| **Status:** *stable*
| **Description:** asyncio compatibility shim for TinyDB. Enables usage of
TinyDB in asyncio-aware contexts without slow synchronous
IO.
``BetterJSONStorage``
*********************
| **Repo:** https://github.com/MrPigss/BetterJSONStorage
| **Status:** *stable*
| **Description:** BetterJSONStorage is a faster 'Storage Type' for TinyDB. It
uses the faster Orjson library for parsing the JSON and BLOSC
for compression.
``tinydb-serialization``
************************
| **Repo:** https://github.com/msiemens/tinydb-serialization
| **Status:** *stable*
| **Description:** ``tinydb-serialization`` provides serialization for objects
that TinyDB otherwise couldn't handle.
``tinydb-smartcache``
*********************
| **Repo:** https://github.com/msiemens/tinydb-smartcache
| **Status:** *stable*
| **Description:** ``tinydb-smartcache`` provides a smart query cache for
TinyDB. It updates the query cache when
inserting/removing/updating documents so the cache doesn't
get invalidated. It's useful if you perform lots of queries
while the data changes only little.
.. _tinyrecord:
``tinyrecord``
**************
| **Repo:** https://github.com/eugene-eeo/tinyrecord
| **Status:** *stable*
| **Description:** Tinyrecord is a library which implements experimental atomic
transaction support for the TinyDB NoSQL database. It uses a
record-first then execute architecture which allows us to
minimize the time that we are within a thread lock.
``tinydb-appengine``
********************
| **Repo:** https://github.com/imalento/tinydb-appengine
| **Status:** *inactive*
| **Description:** ``tinydb-appengine`` provides TinyDB storage for
App Engine. You can use JSON readonly.
---
### Getting Started (`docs/getting-started.rst`)
:tocdepth: 3
Getting Started
===============
Installing TinyDB
-----------------
To install TinyDB from PyPI, run::
$ pip install tinydb
You can also grab the latest development version from GitHub_. After downloading
and unpacking it, you can install it using::
$ pip install .
Basic Usage
-----------
Let's cover the basics before going more into detail. We'll start by setting up
a TinyDB database:
>>> from tinydb import TinyDB, Query
>>> db = TinyDB('db.json')
You now have a TinyDB database that stores its data in ``db.json``.
What about inserting some data? TinyDB expects the data to be Python ``dict``\s:
>>> db.insert({'type': 'apple', 'count': 7})
>>> db.insert({'type': 'peach', 'count': 3})
.. note:: The ``insert`` method returns the inserted document's ID. Read more
about it here: :ref:`document_ids`.
Now you can get all documents stored in the database by running:
>>> db.all()
[{'count': 7, 'type': 'apple'}, {'count': 3, 'type': 'peach'}]
You can also iter over stored documents:
>>> for item in db:
>>> print(item)
{'count': 7, 'type': 'apple'}
{'count': 3, 'type': 'peach'}
Of course you'll also want to search for specific documents. Let's try:
>>> Fruit = Query()
>>> db.search(Fruit.type == 'peach')
[{'count': 3, 'type': 'peach'}]
>>> db.search(Fruit.count > 5)
[{'count': 7, 'type': 'apple'}]
Next we'll update the ``count`` field of the apples:
>>> db.update({'count': 10}, Fruit.type == 'apple')
>>> db.all()
[{'count': 10, 'type': 'apple'}, {'count': 3, 'type': 'peach'}]
In the same manner you can also remove documents:
>>> db.remove(Fruit.count < 5)
>>> db.all()
[{'count': 10, 'type': 'apple'}]
And of course you can throw away all data to start with an empty database:
>>> db.truncate()
>>> db.all()
[]
Recap
*****
Before we dive deeper, let's recapitulate the basics:
+-------------------------------+---------------------------------------------------------------+
| **Inserting** |
+-------------------------------+---------------------------------------------------------------+
| ``db.insert(...)`` | Insert a document |
+-------------------------------+---------------------------------------------------------------+
| **Getting data** |
+-------------------------------+---------------------------------------------------------------+
| ``db.all()`` | Get all documents |
+-------------------------------+---------------------------------------------------------------+
| ``iter(db)`` | Iter over all documents |
+-------------------------------+---------------------------------------------------------------+
| ``db.search(query)`` | Get a list of documents matching the query |
+-------------------------------+---------------------------------------------------------------+
| **Updating** |
+-------------------------------+---------------------------------------------------------------+
| ``db.update(fields, query)`` | Update all documents matching the query to contain ``fields`` |
+-------------------------------+---------------------------------------------------------------+
| **Removing** |
+-------------------------------+---------------------------------------------------------------+
| ``db.remove(query)`` | Remove all documents matching the query |
+-------------------------------+---------------------------------------------------------------+
| ``db.truncate()`` | Remove all documents |
+-------------------------------+---------------------------------------------------------------+
| **Querying** |
+-------------------------------+---------------------------------------------------------------+
| ``Query()`` | Create a new query object |
+-------------------------------+---------------------------------------------------------------+
| ``Query().field == 2`` | Match any document that has a key ``field`` with value |
| | ``== 2`` (also possible: ``!=``, ``>``, ``>=``, ``<``, ``<=``)|
+-------------------------------+---------------------------------------------------------------+
.. note::
Query comparisons only support literal values on the right-hand side.
Field-to-field comparisons like ``Query().a == Query().b`` are not
supported. Use a callable predicate like
``db.search(lambda doc: doc.get('a') == doc.get('b'))`` for custom logic.
.. note::
Callables passed to query APIs (e.g. ``lambda`` predicates or ``Query().map``)
execute in-process and must **never** be derived from untrusted or user-controlled
input.
.. References
.. _GitHub: http://github.com/msiemens/tinydb/
---
### Index (`docs/index.rst`)
Welcome to TinyDB!
==================
Welcome to TinyDB, your tiny, document oriented database optimized for your
happiness :)
>>> from tinydb import TinyDB, Query
>>> db = TinyDB('path/to/db.json')
>>> User = Query()
>>> db.insert({'name': 'John', 'age': 22})
>>> db.search(User.name == 'John')
[{'name': 'John', 'age': 22}]
User's Guide
------------
.. toctree::
:maxdepth: 2
intro
getting-started
usage
Extending TinyDB
----------------
.. toctree::
:maxdepth: 2
Extending TinyDB
TinyDB Extensions
API Reference
-------------
.. toctree::
:maxdepth: 2
api
Additional Notes
----------------
.. toctree::
:maxdepth: 2
contribute
changelog
Upgrade Notes
---
### Intro (`docs/intro.rst`)
Introduction
============
Great that you've taken time to check out the TinyDB docs! Before we begin
looking at TinyDB itself, let's take some time to see whether you should use
TinyDB.
Why Use TinyDB?
---------------
- **tiny:** The current source code has 1800 lines of code (with about 40%
documentation) and 1600 lines tests.
- **document oriented:** Like MongoDB_, you can store any document
(represented as ``dict``) in TinyDB.
- **optimized for your happiness:** TinyDB is designed to be simple and
fun to use by providing a simple and clean API.
- **written in pure Python:** TinyDB neither needs an external server (as
e.g. `PyMongo `_) nor any dependencies
from PyPI.
- **works on Python 3.5+ and PyPy:** TinyDB works on all modern versions of Python
and PyPy.
- **powerfully extensible:** You can easily extend TinyDB by writing new
storages or modify the behaviour of storages with Middlewares.
- **100% test coverage:** No explanation needed.
In short: If you need a simple database with a clean API that just works
without lots of configuration, TinyDB might be the right choice for you.
Why **Not** Use TinyDB?
-----------------------
- You need **advanced features** like:
- access from multiple processes or threads (e.g. when using Flask!),
- creating indexes for tables,
- an HTTP server,
- managing relationships between tables or similar,
- `ACID guarantees `_.
- You are really concerned about **performance** and need a high speed
database.
To put it plainly: If you need advanced features or high performance, TinyDB
is the wrong database for you – consider using databases like SQLite_, Buzhug_,
CodernityDB_ or MongoDB_.
.. References
.. _Buzhug: https://buzhug.sourceforge.net/
.. _CodernityDB: http://labs.codernity.com/codernitydb/
.. _MongoDB: https://mongodb.org/
.. _SQLite: https://www.sqlite.org/
---
### Upgrade (`docs/upgrade.rst`)
Upgrading to Newer Releases
===========================
Version 4.0
-----------
.. _upgrade_v4_0:
- API changes:
- Replace ``TinyDB.purge_tables(...)`` with ``TinyDB.drop_tables(...)``
- Replace ``TinyDB.purge_table(...)`` with ``TinyDB.drop_table(...)``
- Replace ``Table.purge()`` with ``Table.truncate()``
- Replace ``TinyDB(default_table='name')`` with ``TinyDB.default_table_name = 'name'``
- Replace ``TinyDB(table_class=Class)`` with ``TinyDB.table_class = Class``
- If you were using ``TinyDB.DEFAULT_TABLE``, ``TinyDB.DEFAULT_TABLE_KWARGS``,
or ``TinyDB.DEFAULT_STORAGE``: Use the new methods for customizing TinyDB
described in :ref:`How to Extend TinyDB `
Version 3.0
-----------
.. _upgrade_v3_0:
Breaking API Changes
^^^^^^^^^^^^^^^^^^^^
- Querying (see `Issue #62 `_):
- ``where('...').contains('...')`` has been renamed to
``where('...').search('...')``.
- ``where('foo').has('bar')`` is replaced by either
``where('foo').bar`` or ``Query().foo.bar``.
- In case the key is not a valid Python identifier, array
notation can be used: ``where('a.b.c')`` is now
``Query()['a.b.c']``.
- Checking for the existence of a key has to be done explicitly:
``where('foo').exists()``.
- ``SmartCacheTable`` has been moved to `msiemens/tinydb-smartcache`_.
- Serialization has been moved to `msiemens/tinydb-serialization`_.
- Empty storages are now expected to return ``None`` instead of raising
``ValueError`` (see `Issue #67 `_).
.. _msiemens/tinydb-smartcache: https://github.com/msiemens/tinydb-smartcache
.. _msiemens/tinydb-serialization: https://github.com/msiemens/tinydb-serialization
.. _upgrade_v2_0:
Version 2.0
-----------
Breaking API Changes
^^^^^^^^^^^^^^^^^^^^
- The syntax ``query in db`` is not supported any more. Use ``db.contains(...)``
instead.
- The ``ConcurrencyMiddleware`` has been removed due to a insecure implementation
(see `Issue #18 `_). Consider
:ref:`tinyrecord` instead.
Apart from that the API remains compatible to v1.4 and prior.
For migration from v1 to v2, check out the `v2.0 documentation `_
---
### Usage (`docs/usage.rst`)
:tocdepth: 3
.. toctree::
:maxdepth: 2
Advanced Usage
==============
Remarks on Storage
------------------
Before we dive deeper into the usage of TinyDB, we should stop for a moment
and discuss how TinyDB stores data.
To convert your data to a format that is writable to disk TinyDB uses the
`Python JSON `_ module by default.
It's great when only simple data types are involved but it cannot handle more
complex data types like custom classes. On Python 2 it also converts strings to
Unicode strings upon reading
(described `here `_).
If that causes problems, you can write
:doc:`your own storage `, that uses a more powerful (but also slower)
library like `pickle `_ or
`PyYAML `_.
.. hint:: Opening multiple TinyDB instances on the same data (e.g. with the
``JSONStorage``) may result in unexpected behavior due to query caching.
See query_caching_ on how to disable the query cache.
Queries
-------
With that out of the way, let's start with TinyDB's rich set of queries.
There are two main ways to construct queries. The first one resembles the
syntax of popular ORM tools:
>>> from tinydb import Query
>>> User = Query()
>>> db.search(User.name == 'John')
As you can see, we first create a new Query object and then use it to specify
which fields to check. Searching for nested fields is just as easy:
>>> db.search(User.birthday.year == 1990)
Not all fields can be accessed this way if the field name is not a valid Python
identifier. In this case, you can switch to dict access notation:
>>> # This would be invalid Python syntax:
>>> db.search(User.country-code == 'foo')
>>> # Use this instead:
>>> db.search(User['country-code'] == 'foo')
In addition, you can use arbitrary transform function where a field would be,
for example:
>>> from unidecode import unidecode
>>> db.search(User.name.map(unidecode) == 'Jose')
>>> # will match 'José' etc.
The second, traditional way of constructing queries is as follows:
>>> from tinydb import where
>>> db.search(where('field') == 'value')
Using ``where('field')`` is a shorthand for the following code:
>>> db.search(Query()['field'] == 'value')
Accessing nested fields with this syntax can be achieved like this:
>>> db.search(where('birthday').year == 1900)
>>> db.search(where('birthday')['year'] == 1900)
Advanced queries
................
In the :doc:`getting-started` you've learned about the basic comparisons
(``==``, ``<``, ``>``, ...). In addition to these TinyDB supports the following
queries:
>>> # Existence of a field:
>>> db.search(User.name.exists())
>>> # Regex:
>>> # Full item has to match the regex:
>>> db.search(User.name.matches('[aZ]*'))
>>> # Case insensitive search for 'John':
>>> import re
>>> db.search(User.name.matches('John', flags=re.IGNORECASE))
>>> # Any part of the item has to match the regex:
>>> db.search(User.name.search('b+'))
>>> # Custom test:
>>> test_func = lambda s: s == 'John'
>>> db.search(User.name.test(test_func))
>>> # Custom test with parameters:
>>> def test_func(val, m, n):
>>> return m <= val <= n
>>> db.search(User.age.test(test_func, 0, 21))
>>> db.search(User.age.test(test_func, 21, 99))
Another case is if you have a ``dict`` where you want to find all documents
that match this ``dict``. We call this searching for a fragment:
>>> db.search(Query().fragment({'foo': True, 'bar': False}))
[{'foo': True, 'bar': False, 'foobar: 'yes!'}]
You also can search for documents where a specific field matches the fragment:
>>> db.search(Query().field.fragment({'foo': True, 'bar': False}))
[{'field': {'foo': True, 'bar': False, 'foobar: 'yes!'}]
When a field contains a list, you also can use the ``any`` and ``all`` methods.
There are two ways to use them: with lists of values and with nested queries.
Let's start with the first one. Assuming we have a user object with a groups list
like this:
>>> db.insert({'name': 'user1', 'groups': ['user']})
>>> db.insert({'name': 'user2', 'groups': ['admin', 'user']})
>>> db.insert({'name': 'user3', 'groups': ['sudo', 'user']})
Now we can use the following queries:
>>> # User's groups include at least one value from ['admin', 'sudo']
>>> db.search(User.groups.any(['admin', 'sudo']))
[{'name': 'user2', 'groups': ['admin', 'user']},
{'name': 'user3', 'groups': ['sudo', 'user']}]
>>>
>>> # User's groups include all values from ['admin', 'user']
>>> db.search(User.groups.all(['admin', 'user']))
[{'name': 'user2', 'groups': ['admin', 'user']}]
In some cases you may want to have more complex ``any``/``all`` queries.
This is where nested queries come in as helpful. Let's set up a table like this:
>>> Group = Query()
>>> Permission = Query()
>>> groups = db.table('groups')
>>> groups.insert({
'name': 'user',
'permissions': [{'type': 'read'}]})
>>> groups.insert({
'name': 'sudo',
'permissions': [{'type': 'read'}, {'type': 'sudo'}]})
>>> groups.insert({
'name': 'admin',
'permissions': [{'type': 'read'}, {'type': 'write'}, {'type': 'sudo'}]})
Now let's search this table using nested ``any``/``all`` queries:
>>> # Group has a permission with type 'read'
>>> groups.search(Group.permissions.any(Permission.type == 'read'))
[{'name': 'user', 'permissions': [{'type': 'read'}]},
{'name': 'sudo', 'permissions': [{'type': 'read'}, {'type': 'sudo'}]},
{'name': 'admin', 'permissions':
[{'type': 'read'}, {'type': 'write'}, {'type': 'sudo'}]}]
>>> # Group has ONLY permission 'read'
>>> groups.search(Group.permissions.all(Permission.type == 'read'))
[{'name': 'user', 'permissions': [{'type': 'read'}]}]
As you can see, ``any`` tests if there is *at least one* document matching
the query while ``all`` ensures *all* documents match the query.
The opposite operation, checking if a single item is contained in a list,
is also possible using ``one_of``:
>>> db.search(User.name.one_of(['jane', 'john']))
Query modifiers
...............
TinyDB also allows you to use logical operations to modify and combine
queries:
>>> # Negate a query:
>>> db.search(~ (User.name == 'John'))
>>> # Logical AND:
>>> db.search((User.name == 'John') & (User.age <= 30))
>>> # Logical OR:
>>> db.search((User.name == 'John') | (User.name == 'Bob'))
.. note::
When using ``&`` or ``|``, make sure you wrap the conditions on both sides
with parentheses or Python will mess up the comparison.
Also, when using negation (``~``) you'll have to wrap the query you want
to negate in parentheses.
The reason for these requirements is that Python's binary operators that are
used for query modifiers have a higher operator precedence than comparison
operators. Simply put, ``~ User.name == 'John'`` is parsed by Python as
``(~User.name) == 'John'`` instead of ``~(User.name == 'John')``. See also the
Python `docs on operator precedence
`_
for details.
You can compose queries dynamically by using the no-op query ``Query().noop()``.
Comparisons only support literal values on the right-hand side. Field-to-field
comparisons like ``Query().a == Query().b`` are not supported. Use a callable
predicate like ``db.search(lambda doc: doc.get('a') == doc.get('b'))`` for
custom logic.
.. note::
Callables passed to query APIs (e.g. predicates, ``Query().map``,
``Query().test``) execute in-process and must **never** be derived from
untrusted or user-controlled input.
Recap
.....
Let's review the query operations we've learned:
+-------------------------------------+---------------------------------------------------------------------+
| **Queries** |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.exists()`` | Match any document where a field called ``field`` exists |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.matches(regex)`` | Match any document with the whole field matching the |
| | regular expression |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.search(regex)`` | Match any document with a substring of the field matching |
| | the regular expression |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.test(func, *args)`` | Matches any document for which the function returns |
| | ``True`` |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.all(query | list)`` | If given a query, matches all documents where all documents |
| | in the list ``field`` match the query. |
| | If given a list, matches all documents where all documents |
| | in the list ``field`` are a member of the given list |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.any(query | list)`` | If given a query, matches all documents where at least one |
| | document in the list ``field`` match the query. |
| | If given a list, matches all documents where at least one |
| | documents in the list ``field`` are a member of the given |
| | list |
+-------------------------------------+---------------------------------------------------------------------+
| ``Query().field.one_of(list)`` | Match if the field is contained in the list |
+-------------------------------------+---------------------------------------------------------------------+
| **Logical operations on queries** |
+-------------------------------------+---------------------------------------------------------------------+
| ``~ (query)`` | Match documents that don't match the query (logical NOT) |
+-------------------------------------+---------------------------------------------------------------------+
| ``(query1) & (query2)`` | Match documents that match both queries (logical AND) |
+-------------------------------------+---------------------------------------------------------------------+
| ``(query1) | (query2)`` | Match documents that match at least one of the queries (logical OR) |
+-------------------------------------+---------------------------------------------------------------------+
Handling Data
-------------
Next, let's look at some more ways to insert, update and retrieve data from
your database.
Inserting data
..............
As already described you can insert a document using ``db.insert(...)``.
In case you want to insert multiple documents, you can use ``db.insert_multiple(...)``:
>>> db.insert_multiple([
{'name': 'John', 'age': 22},
{'name': 'John', 'age': 37}])
>>> db.insert_multiple({'int': 1, 'value': i} for i in range(2))
Also in some cases it may be useful to specify the document ID yourself when
inserting data. You can do that by using the :class:`~tinydb.table.Document`
class:
>>> db.insert(Document({'name': 'John', 'age': 22}, doc_id=12))
12
The same is possible when using ``db.insert_multiple(...)``:
>>> db.insert_multiple([
Document({'name': 'John', 'age': 22}, doc_id=12),
Document({'name': 'Jane', 'age': 24}, doc_id=14),
])
[12, 14]
.. note::
Inserting a ``Document`` with an ID that already exists will result
in a ``ValueError`` being raised.
Updating data
.............
Sometimes you want to update all documents in your database. In this case, you
can leave out the ``query`` argument:
>>> db.update({'foo': 'bar'})
When passing a dict to ``db.update(fields, query)``, it only allows you to
update a document by adding or overwriting its values. But sometimes you may
need to e.g. remove one field or increment its value. In that case you can
pass a function instead of ``fields``:
>>> from tinydb.operations import delete
>>> db.update(delete('key1'), User.name == 'John')
This will remove the key ``key1`` from all matching documents. TinyDB comes
with these operations:
- ``delete(key)``: delete a key from the document
- ``increment(key)``: increment the value of a key
- ``decrement(key)``: decrement the value of a key
- ``add(key, value)``: add ``value`` to the value of a key (also works for strings)
- ``subtract(key, value)``: subtract ``value`` from the value of a key
- ``set(key, value)``: set ``key`` to ``value``
Of course you also can write your own operations:
>>> def your_operation(your_arguments):
... def transform(doc):
... # do something with the document
... # ...
... return transform
...
>>> db.update(your_operation(arguments), query)
In order to perform multiple update operations at once, you can use the
``update_multiple`` method like this:
>>> db.update_multiple([
... ({'int': 2}, where('char') == 'a'),
... ({'int': 4}, where('char') == 'b'),
... ])
You also can mix normal updates with update operations:
>>> db.update_multiple([
... ({'int': 2}, where('char') == 'a'),
... (delete('int'), where('char') == 'b'),
... ])
Data access and modification
----------------------------
Upserting data
..............
In some cases you'll need a mix of both ``update`` and ``insert``: ``upsert``.
This operation is provided a document and a query. If it finds any documents
matching the query, they will be updated with the data from the provided document.
On the other hand, if no matching document is found, it inserts the provided
document into the table:
>>> db.upsert({'name': 'John', 'logged-in': True}, User.name == 'John')
This will update all users with the name John to have ``logged-in`` set to ``True``.
If no matching user is found, a new document is inserted with both the name set
and the ``logged-in`` flag.
To use the ID of the document as matching criterion a :class:`~tinydb.table.Document`
with ``doc_id`` is passed instead of a query:
>>> db.upsert(Document({'name': 'John', 'logged-in': True}, doc_id=12))
Retrieving data
...............
There are several ways to retrieve data from your database. For instance you
can get the number of stored documents:
>>> len(db)
3
.. hint::
This will return the number of documents in the default table
(see the notes on the :ref:`default table `).
Then of course you can use ``db.search(...)`` as described in the :doc:`getting-started`
section. But sometimes you want to get only one matching document. Instead of using
>>> try:
... result = db.search(User.name == 'John')[0]
... except IndexError:
... pass
you can use ``db.get(...)``:
>>> db.get(User.name == 'John')
{'name': 'John', 'age': 22}
>>> db.get(User.name == 'Bobby')
None
.. caution::
If multiple documents match the query, the first match in insertion order
is returned (not a random document).
Often you don't want to search for documents but only know whether they are
stored in the database. In this case ``db.contains(...)`` is your friend:
>>> db.contains(User.name == 'John')
In a similar manner you can look up the number of documents matching a query:
>>> db.count(User.name == 'John')
2
Recap
^^^^^
Let's summarize the ways to handle data:
+-------------------------------+---------------------------------------------------------------+
| **Inserting data** |
+-------------------------------+---------------------------------------------------------------+
| ``db.insert_multiple(...)`` | Insert multiple documents |
+-------------------------------+---------------------------------------------------------------+
| **Updating data** |
+-------------------------------+---------------------------------------------------------------+
| ``db.update(operation, ...)`` | Update all matching documents with a special operation |
+-------------------------------+---------------------------------------------------------------+
| **Retrieving data** |
+-------------------------------+---------------------------------------------------------------+
| ``len(db)`` | Get the number of documents in the database |
+-------------------------------+---------------------------------------------------------------+
| ``db.get(query)`` | Get one document matching the query |
+-------------------------------+---------------------------------------------------------------+
| ``db.contains(query)`` | Check if the database contains a matching document |
+-------------------------------+---------------------------------------------------------------+
| ``db.count(query)`` | Get the number of matching documents |
+-------------------------------+---------------------------------------------------------------+
.. note::
This was a new feature in v3.6.0
.. _document_ids:
Using Document IDs
------------------
Internally TinyDB associates an ID with every document you insert. It's returned
after inserting a document:
>>> db.insert({'name': 'John', 'age': 22})
3
>>> db.insert_multiple([{...}, {...}, {...}])
[4, 5, 6]
In addition you can get the ID of already inserted documents using
``document.doc_id``. This works both with ``get`` and ``all``:
>>> el = db.get(User.name == 'John')
>>> el.doc_id
3
>>> el = db.all()[0]
>>> el.doc_id
1
>>> el = db.all()[-1]
>>> el.doc_id
12
Different TinyDB methods also work with IDs, namely: ``update``, ``remove``,
``contains`` and ``get``. The first two also return a list of affected IDs.
>>> db.update({'value': 2}, doc_ids=[1, 2])
>>> db.contains(doc_id=1)
True
>>> db.remove(doc_ids=[1, 2])
>>> db.get(doc_id=3)
{...}
>>> db.get(doc_ids=[1, 2])
[{...}, {...}]
Using ``doc_id``/``doc_ids`` instead of ``Query()`` again is slightly faster
in operation.
Moving documents between databases
..................................
Documents returned by TinyDB are :class:`~tinydb.table.Document` instances.
They look like plain dicts, but they also carry the document ID assigned by
the table they came from. If you insert such a ``Document`` into another
table or database, TinyDB reuses that ID. When the target table already has
a document with the same ID, insertion fails with ``ValueError: Document
with ID ... already exists``.
This often shows up when moving documents from one database to another:
>>> db1 = TinyDB('db1.json')
>>> db2 = TinyDB('db2.json')
>>> document = db1.get(doc_id=1)
>>> db2.insert(document) # reuses doc_id=1 from db1
>>> db1.remove(doc_ids=[1])
To copy the document's data and let the target table assign a new ID, convert
the ``Document`` to a plain dict first:
>>> db2.insert(dict(document))
The same applies when moving documents between named tables in a single
database.
Recap
.....
Let's sum up the way TinyDB supports working with IDs:
+-------------------------------------+------------------------------------------------------------+
| **Getting a document's ID** |
+-------------------------------------+------------------------------------------------------------+
| ``db.insert(...)`` | Returns the inserted document's ID |
+-------------------------------------+------------------------------------------------------------+
| ``db.insert_multiple(...)`` | Returns the inserted documents' ID |
+-------------------------------------+------------------------------------------------------------+
| ``document.doc_id`` | Get the ID of a document fetched from the db |
+-------------------------------------+------------------------------------------------------------+
| **Working with IDs** |
+-------------------------------------+------------------------------------------------------------+
| ``db.get(doc_id=...)`` | Get the document with the given ID |
+-------------------------------------+------------------------------------------------------------+
| ``db.contains(doc_id=...)`` | Check if the db contains a document with the given |
| | IDs |
+-------------------------------------+------------------------------------------------------------+
| ``db.update({...}, doc_ids=[...])`` | Update all documents with the given IDs |
+-------------------------------------+------------------------------------------------------------+
| ``db.remove(doc_ids=[...])`` | Remove all documents with the given IDs |
+-------------------------------------+------------------------------------------------------------+
Tables
------
TinyDB supports working with multiple tables. They behave just the same as
the ``TinyDB`` class. To create and use a table, use ``db.table(name)``.
>>> table = db.table('table_name')
>>> table.insert({'value': True})
>>> table.all()
[{'value': True}]
>>> for row in table:
>>> print(row)
{'value': True}
To remove a table from a database, use:
>>> db.drop_table('table_name')
If on the other hand you want to remove all tables, use the counterpart:
>>> db.drop_tables()
Finally, you can get a list with the names of all tables in your database:
>>> db.tables()
{'_default', 'table_name'}
.. _default_table:
Default Table
.............
TinyDB uses a table named ``_default`` as the default table. All operations
on the database object (like ``db.insert(...)``) operate on this table.
The name of this table can be modified by setting the ``default_table_name``
class variable to modify the default table name for all instances:
>>> #1: for a single instance only
>>> db = TinyDB(storage=SomeStorage)
>>> db.default_table_name = 'my-default'
>>> #2: for all instances
>>> TinyDB.default_table_name = 'my-default'
.. _query_caching:
Query Caching
.............
TinyDB caches query result for performance. That way re-running a query won't
have to read the data from the storage as long as the database hasn't been
modified. You can optimize the query cache size by passing the ``cache_size``
to the ``table(...)`` function:
>>> table = db.table('table_name', cache_size=30)
.. hint:: You can set ``cache_size`` to ``None`` to make the cache unlimited in
size. Also, you can set ``cache_size`` to 0 to disable it.
.. hint:: A repeated search with the same query is served from the cache.
The returned list is a fresh copy, but the documents inside it are the
same objects the cache holds — treat them as read-only. Mutating a
returned document (a key or a nested value) corrupts what a later
identical search returns.
.. hint:: It's not possible to open the same table multiple times with different
settings. After the first invocation, all the subsequent calls will return
the same table with the same settings as the first one.
.. hint:: The TinyDB query cache doesn't check if the underlying storage
that the database uses has been modified by an external process. In this
case the query cache may return outdated results. To clear the cache and
read data from the storage again you can use ``db.clear_cache()``.
.. hint:: When using an unlimited cache size and ``test()`` queries, TinyDB
will store a reference to the test function. As a result of that behavior
long-running applications that use ``lambda`` functions as a test function
may experience memory leaks.
Storage & Middleware
--------------------
Storage Types
.............
TinyDB comes with two storage types: JSON and in-memory. By
default TinyDB stores its data in JSON files so you have to specify the path
where to store it:
>>> from tinydb import TinyDB, where
>>> db = TinyDB('path/to/db.json')
To use the in-memory storage, use:
>>> from tinydb.storages import MemoryStorage
>>> db = TinyDB(storage=MemoryStorage)
.. hint::
All arguments except for the ``storage`` argument are forwarded to the
underlying storage. For the JSON storage you can use this to pass
additional keyword arguments to Python's
`json.dumps(...) `_
method. For example, you can set it to create prettified JSON files like
this:
>>> db = TinyDB('db.json', sort_keys=True, indent=4, separators=(',', ': '))
.. note::
``JSONStorage`` forwards ``**kwargs`` to ``json.dumps``. **Never** pass
user-controlled values for callable arguments (e.g. ``default`` or
``cls``) as they are executed in-process on every write operation.
To modify the default storage for all ``TinyDB`` instances, set the
``default_storage_class`` class variable:
>>> TinyDB.default_storage_class = MemoryStorage
In case you need to access the storage instance directly, you can use the
``storage`` property of your TinyDB instance. This may be useful to call
method directly on the storage or middleware:
>>> db = TinyDB(storage=CachingMiddleware(MemoryStorage))
>>> db.storage.flush()
Middleware
..........
Middleware wraps around existing storage allowing you to customize their
behaviour.
>>> from tinydb.storages import JSONStorage
>>> from tinydb.middlewares import CachingMiddleware
>>> db = TinyDB('/path/to/db.json', storage=CachingMiddleware(JSONStorage))
.. hint::
You can nest middleware:
>>> db = TinyDB('/path/to/db.json',
storage=FirstMiddleware(SecondMiddleware(JSONStorage)))
CachingMiddleware
^^^^^^^^^^^^^^^^^
The ``CachingMiddleware`` improves speed by reducing disk I/O. It caches all
read operations and writes data to disk after a configured number of
write operations.
To make sure that all data is safely written when closing the table, use one
of these ways:
.. code-block:: python
# Using a context manager:
with database as db:
# Your operations
.. code-block:: python
# Using the close function
db.close()
.. _mypy_type_checking:
MyPy Type Checking
------------------
TinyDB comes with type annotations that MyPy can use to make sure you're using
the API correctly. Unfortunately, MyPy doesn't understand all code patterns
that TinyDB uses. For that reason TinyDB ships a MyPy plugin that helps
correctly type checking code that uses TinyDB. To use it, add it to the
plugins list in the `MyPy configuration file
`_
(typically located in ``setup.cfg`` or ``mypy.ini``):
.. code-block:: ini
[mypy]
plugins = tinydb.mypy_plugin
What's next
-----------
Congratulations, you've made through the user guide! Now go and build something
awesome or dive deeper into TinyDB with these resources:
- Want to learn how to customize TinyDB (storages, middlewares) and what
extensions exist? Check out :doc:`extend` and :doc:`extensions`.
- Want to study the API in detail? Read :doc:`api`.
- Interested in contributing to the TinyDB development guide? Go on to the
:doc:`contribute`.
---
--- METRICS ---
- Files Extracted: 10
- Estimated Token Budget: ~20031 tokens
- Canonical Reference: https://codewiki.google/github.com/msiemens/tinydb