# Technical Documentation: logseq/datascript

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/logseq/datascript)

### Overview (`README.md`)

<img src="./extras/logo.svg">

> What if creating a database would be as cheap as creating a Hashmap?

An immutable in-memory database and Datalog query engine in Clojure and ClojureScript.

DataScript is meant to run inside the browser. It is cheap to create, quick to query and ephemeral. You create a database on page load, put some data in it, track changes, do queries and forget about it when the user closes the page.

DataScript databases are immutable and based on persistent data structures. In fact, they’re more like data structures than databases (think Hashmap). Unlike querying a real SQL DB, when you query DataScript, it all comes down to a Hashmap lookup. Or series of lookups. Or array iteration. There’s no particular overhead to it. You put a little data in it, it’s fast. You put in a lot of data, well, at least it has indexes. That should do better than you filtering an array by hand anyway. The thing is really lightweight.

The intention with DataScript is to be a basic building block in client-side applications that needs to track a lot of state during their lifetime. There’s a lot of benefits:

- Central, uniform approach to manage all application state. Clients working with state become decoupled and independent: rendering, server sync, undo/redo do not interfere with each other.
- Immutability simplifies things even in a single-threaded browser environment. Keep track of app state evolution, rewind to any point in time, always render consistent state, sync in background without locking anybody.
- Datalog query engine to answer non-trivial questions about current app state.
- Structured format to track data coming in and out of DB. Datalog queries can be run against it too.

## Latest version [](https://travis-ci.org/tonsky/datascript)

```clj
;; lein
[datascript "1.7.3"]
```
```clj
;; deps.edn
datascript/datascript {:mvn/version "1.7.3"}
```

Important! If you are using shadow-cljs, add

```clj
:compiler-options {:externs ["datascript/externs.js"]}
```

to your build (see [#432](https://github.com/tonsky/datascript/issues/432) [#298](https://github.com/tonsky/datascript/issues/298) [#216](https://github.com/tonsky/datascript/pull/216))

## Support us

<a href="https://patreon.com/tonsky" target="_blank"><img src="./extras/datascript_patreon.png"></a>

## Resources

Support:

- Join [#datascript on Clojurians Slack](https://clojurians.slack.com/messages/C07V8N22C/) (grab invite [here](http://clojurians.net/))

Books:

- [Learning ClojureScript](https://www.packtpub.com/web-development/learning-clojurescript) has a chapter on DataScript

Docs:

- API Docs [](https://cljdoc.org/d/datascript/datascript/CURRENT)
- [docs/queries.md](docs/queries.md)
- [docs/tuples.md](docs/tuples.md)
- [docs/storage.md](docs/storage.md)
- [Getting started](https://github.com/tonsky/datascript/wiki/Getting-started)
- [Tutorials](https://github.com/kristianmandrup/datascript-tutorial)
- [Tips & tricks](https://github.com/tonsky/datascript/wiki/Tips-&-tricks)
- [Triple Stores overview](https://github.com/threatgrid/asami/wiki/Introduction)
- [Quick tutorial into Datalog](http://www.learndatalogtoday.org/)

Posts:

- [How DataScript fits into the current webdev ecosystem](http://tonsky.me/blog/decomposing-web-app-development/)
- [DataScript internals explained](http://tonsky.me/blog/datascript-internals/)
- [Sketch of client/server reactive architecture](http://tonsky.me/blog/the-web-after-tomorrow/)

Talks:

- “Frontend with Joy” talk (FPConf, August 2015): [video](https://www.youtube.com/watch?v=cRWrrHPrk9g) in Russian
- “Programming Web UI with Database in a Browser” talk (PolyConf, July 2015): [slides](http://s.tonsky.me/conferences/2015.07%20polyconf.pdf), [video](https://www.youtube.com/watch?v=1dr-CzMMDD8)
- “DataScript for Web Development” talk (Clojure eXchange, Dec 2014): [slides](http://s.tonsky.me/conferences/2014.12%20clojure%20eXchange.pdf), [video](https://skillsmatter.com/skillscasts/6038-datascript-for-web-development)
- “Building ToDo list with DataScript” webinar (ClojureScript NYC, Dec 2014): [video](http://vimeo.com/114688970), [app](https://github.com/tonsky/datascript-todo)
- DataScript hangout (May 2014, in Russian): [video](http://www.youtube.com/watch?v=jhBC81pczZY)

Projects using DataScript:

- [Roam Research](https://roamresearch.com/), a note-taking graph database
- [LogSeq](http://logseq.com), a local-first, non-linear, outliner notebook
- [Athens Research](https://github.com/athensresearch/athens), a tool for networked thought
- [Hulunote](https://www.hulunote.com/), a tool for networked thought
- [Cognician](https://www.cognician.com/), coaching platform
- [LightMesh](http://lightmesh.com/), datacenter management
- [PartsBox](https://partsbox.io/), electronic parts management
- [I am Fy](https://www.iamfy.co/), accessories e-shop
- [Under Whose Command?](https://myanmar.securityforcemonitor.org/), an interactive report by Security Force Monitor
- [Pinfigurator](https://kevinlynagh.com/pinfigurator/), a microcontroller search tool
- [Precursor](http://precursorapp.com/), collaborative prototyping tool
- [Acha-acha](http://tonsky.me/blog/acha-acha/), GitHub achievements
- [Showkr](http://showkr.solovyov.net), flickr gallery viewer ([sources](https://github.com/piranha/showkr))
- [Zetawar](http://www.zetawar.com), turn-based tactical strategy game
- [Lemmings](https://www.lemmings.io), incubator focused on art & artificial intelligence

Related projects:

- [DataScript-Transit](https://github.com/tonsky/datascript-transit), transit serialization for database and datoms
- [DataScript SQL Storages](https://github.com/tonsky/datascript-storage-sql), durable storage implementations for popular SQL databases
- [Posh](https://github.com/mpdairy/posh), lib that lets you use a single DataScript db to store Reagent app state
- [re-posh](https://github.com/denistakeda/re-posh), use re-frame with DataScript storage
- [DataScript-mori](https://github.com/typeetfunc/datascript-mori), DataScript & Mori wrapper for use from JS
- [DatSync](https://github.com/metasoarous/datsync), Datomic ↔︎ DataScript syncing/replication utilities
- [Intension](https://github.com/alandipert/intension), lib to convert associative structures to in-memory databases for querying them
- [Datamaps](https://github.com/djjolicoeur/datamaps), lib designed to leverage datalog queries to query arbitrary maps.

Demo applications:

- [Localisation Demo with Om Next](http://simonb.com/blog/2016/01/24/om-next-datascript-localisation-demo/)
- ToDo, task manager demo app (persistence via localStorage and transit, filtering, undo/redo): [sources](https://github.com/tonsky/datascript-todo), [live](http://tonsky.me/datascript-todo/)
- CatChat, chat demo app: [sources](https://github.com/tonsky/datascript-chat), [code walkthrough](http://tonsky.me/blog/datascript-chat/), [live](http://tonsky.me/datascript-chat/)
- clj-crud, demo CRUD app: [sources](https://github.com/thegeez/clj-crud), [blog post](http://thegeez.net/2014/04/30/datascript_clojure_web_app.html)
- [OmNext TodoMVC](https://github.com/madvas/todomvc-omnext-datomic-datascript)

## Usage examples

For more examples, see [our acceptance test suite](test/datascript/test/).

```clj
(require '[datascript.core :as d])

;; Implicit join, multi-valued attribute

(let [schema {:aka {:db/cardinality :db.cardinality/many}}
      conn   (d/create-conn schema)]
  (d/transact! conn [ { :db/id -1
                        :name  "Maksim"
                        :age   45
                        :aka   ["Max Otto von Stierlitz", "Jack Ryan"] } ])
  (d/q '[ :find  ?n ?a
          :where [?e :aka "Max Otto von Stierlitz"]
                 [?e :name ?n]
                 [?e :age  ?a] ]
       @conn))

;; => #{ ["Maksim" 45] }


;; Destructuring, function call, predicate call, query over collection

(d/q '[ :find  ?k ?x
        :in    [[?k [?min ?max]] ...] ?range
        :where [(?range ?min ?max) [?x ...]]
               [(even? ?x)] ]
      { :a [1 7], :b [2 4] }
      range)

;; => #{ [:a 2] [:a 4] [:a 6] [:b 2] }


;; Recursive rule

(d/q '[ :find  ?u1 ?u2
        :in    $ %
        :where (follows ?u1 ?u2) ]
      [ [1 :follows 2]
        [2 :follows 3]
        [3 :follows 4] ]
     '[ [(follows ?e1 ?e2)
         [?e1 :follows ?e2]]
        [(follows ?e1 ?e2)
         [?e1 :follows ?t]
         (follows ?t ?e2)] ])

;; => #{ [1 2] [1 3] [1 4]
;;       [2 3] [2 4]
;;       [3 4] }


;; Aggregates

(d/q '[ :find ?color (max ?amount ?x) (min ?amount ?x)
        :in   [[?color ?x]] ?amount ]
     [[:red 10]  [:red 20] [:red 30] [:red 40] [:red 50]
      [:blue 7] [:blue 8]]
     3)

;; => [[:red  [30 40 50] [10 20 30]]
;;     [:blue [7 8] [7 8]]]
```

## Using from vanilla JS

DataScript can be used from any JS engine without additional dependencies:

```html
<script src="https://github.com/tonsky/datascript/releases/download/1.7.3/datascript-1.7.3.min.js"></script>
```

or as a CommonJS module ([npm page](https://www.npmjs.org/package/datascript)):

```
npm install datascript
```

```js
var ds = require('datascript');
```

or as a RequireJS module:

```js
require(['datascript'], function(ds) { ... });
```

Queries:

* Query and rules should be EDN passed as strings
* Results of `q` are returned as regular JS arrays

Entities:

* Entities returned by `entity` call are lazy as in Clojure
* Use `e.get("prop")`, `e.get(":db/id")`, `e.db` to access entity properties
* Entities implement ECMAScript 6 Map interface (has/get/keys/...)

Transactions:

* Use strings such as `":db/id"`, `":db/add"`, etc. instead of db-namespaced keywords
* Use regular JS arrays and objects to pass data to `transact` and `db_with`

Transaction reports:

* `report.tempids` has string keys (`"-1"` for entity tempid `-1`), use `resolve_tempid` to set up a correspondence

Check out [test/js/tests.js](test/js/tests.js) for usage examples.

## Project status

Stable. Most of the features done, expecting non-breaking API additions and performance optimizations. No docs at the moment, use examples & [Datomic documentation](https://docs.datomic.com/on-prem/index.html#sec-3).

The following features are supported:

* Database as a value: each DB is an immutable value. New DBs are created on top of old ones, but old ones stay perfectly valid too
* Triple store model
* EAVT, AEVT and AVET indexes
* Multi-valued attributes via `:db/cardinality :db.cardinality/many`
* Lazy entities and `:db/valueType :db.type/ref` auto-expansion
* Database “mutations” via `transact!`
* Callback-based analogue to txReportQueue via `listen!`
* Direct index lookup and iteration via `datoms` and `seek-datoms`
* Filtered databases via `filter`
* Lookup refs
* Unique constraints, upsert
* Pull API (thx [David Thomas Hume](https://github.com/dthume))

Query engine features:

* Implicit joins
* Query over DB or regular collections
* Parameterized queries via `:in` clause
* Tuple, collection, relation binding forms in `:in` clause
* Query over multiple DB/collections
* Predicates and user functions in query
* Negation and disjunction
* Rules, recursive rules
* Aggregates
* Find specifications

Interface differences:

* Conn is just an atom storing last DB value, use `@conn` instead of `(d/db conn)`
* Instead of `#db/id[:db.part/user -100]` just use `-100` in place of `:db/id` or entity id
* Transactor functions can be called as `[:db.fn/call f args]` where `f` is a function reference and will take db as first argument (thx [@thegeez](https://github.com/thegeez))
* In ClojureScript, custom query functions and aggregates should be passed as source instead of being referenced by symbol (due to lack of `resolve` in CLJS)
* Custom aggregate functions are called via aggregate keyword: `:find (aggregate ?myfn ?e) :in $ ?myfn`
* Additional `:db.fn/retractAttribute` shortcut
* Transactions are not annotated by default with `:db/txInstant`
* When “transaction function” is called, the db that this function receive is a “partial db” relative to it's position in transaction.

## Differences from Datomic

* DataScript is built totally from scratch and is not related by any means to the popular Clojure database Datomic
* Runs in a browser and/or in a JVM
* Simplified schema, not queryable
* Attributes do not have to be declared in advance. Put them to schema only when you need special behaviour from them
* Any type can be used for values
* No `:db/ident` for attributes, keywords are _literally_ attribute values, no integer id behind them
* No schema migrations
* No full-text search, no partitions
* No external dependencies

Aimed at interactive, long-living browser applications, DataScript DBs operate in constant space. If you do not add new entities, just update existing ones, or clean up database from time to time, memory consumption will be limited. This is unlike Datomic which keeps history of all changes, thus grows monotonically. DataScript does not track history by default, but you can do it via your own code if needed.

Some of the features are omitted intentionally. Different apps have different needs in storing/transfering/keeping track of DB state. DataScript is a foundation to build exactly the right storage solution for your needs without selling too much “vision”.

## Contributing

### Testing

Setup

    npm install ws

Running the tests

    clj -M:test -m kaocha.runner

Watching tests:

    ./script/watch.sh

### Benchmarking and Datomic compatibility

`datomic-free` is a dependency not available on Clojars or Maven Central.

1. Download `datomic-free` from https://my.datomic.com/downloads/free
2. Unzip it
3. Inside the unzipped folder run `./bin/maven-install`

Run compatibility checks:

    clj -M:datomic

Benchmark:

    cd bench
    ./bench.clj

## License

Copyright © 2014–2024 Nikita Prokopov

Licensed under Eclipse Public License (see [LICENSE](LICENSE)).

---

### Overview (`release-js/README.md`)

# DataScript

An immutable in-memory database and Datalog query engine for JS engines.

DataScript is meant to run inside the browser. It is cheap to create, quick to query and ephemeral. You create a database on page load, put some data in it, track changes, do queries and forget about it when the user closes the page.

DataScript databases are immutable and based on persistent data structures. In fact, they’re more like data structures than databases (think Hashmap). Unlike querying a real SQL DB, when you query DataScript, it all comes down to a Hashmap lookup. Or series of lookups. Or array iteration. There’s no particular overhead to it. You put a little data in it, it’s fast. You put in a lot of data, well, at least it has indexes. That should do better than you filtering an array by hand anyway. The thing is really lightweight.

The intention with DataScript is to be a basic building block in client-side applications that needs to track a lot of state during their lifetime. There’s a lot of benefits:

- Central, uniform approach to manage all application state. Clients working with state become decoupled and independent: rendering, server sync, undo/redo do not interfere with each other.
- Immutability simplifies things even in a single-threaded browser environment. Keep track of app state evolution, rewind to any point in time, always render consistent state, sync in background without locking anybody.
- Datalog query engine to answer non-trivial questions about current app state.
- Structured format to track data coming in and out of DB. Datalog queries can be run against it too.

For more info, check out [project page](https://github.com/tonsky/datascript)

---

### Queries (`docs/queries.md`)

# Return maps

```
return-map  = (return-keys | return-syms | return-strs)
return-keys = ':keys' symbol+
return-syms = ':syms' symbol+
return-strs = ':strs' symbol+
```

If you need a query to return a set of maps instead set of tuples, specify one of `:keys` / `:syms` / `:strs`, followed by a list of symbols:

```
[:find ?name ?age ?email, ...]
=> #{["Ivan" 10 "ivan@"]
     ["Oleg" 20 "oleg@"]
     ["Sergey" 30 "sergey@"]}

[:find ?name ?age ?email
 :keys name A e-mail
 ...]
=> #{{:name "Ivan",   :A 10, :e-mail "ivan@"}
     {:name "Oleg",   :A 20, :e-mail "oleg@"}
     {:name "Sergey", :A 30, :e-mail "sergey@"}}
```

`:strs` forces map to have string keys, `:syms` for symbols.

Return maps are only compatible with normal find and tuple-returning find:

```
[:find [?name ?age ?email], ...]
=> {:name "Ivan",   :A 10, :e-mail "ivan@"}
```

The amount of keys must match the amount of find elements.

Datomic docs: https://docs.datomic.com/on-prem/query.html#return-maps

# Query functions

## tuple

```
[(tuple ?a ...) ?tup]
```

Given one or more values, the `tuple` function returns a tuple containing each value. See also `untuple`.

```
;; query
[:find ?tup
 :in ?a ?b
 :where [(tuple ?a ?b) ?tup]]

;; inputs
1 2

;; result
#{[[1 2]]}
```

## untuple

```
[(untuple ?tup) [?a ?b]]
```

Given a tuple, the `untuple` function can be used to name each element of the tuple. See also `tuple`.

```
;; query
[:find ?b
 :in ?tup
 :where [(untuple ?tup) [?a ?b]]]

;; inputs
[1 2]

;; result
#{[2]}
```

---

### Storage (`docs/storage.md`)

# Storing DataScript on disk

There are three ways to store DataScript database on disk. First two are actually just serialization: they can turn your DB into string, which you can then store on disk or whenever you want.

## print/read

The oldest and slowest way. Works both for Clojure and ClojureScript. Just

```
(def string
  (pr-str db))
```

and later

```
(read string)
```

or, better, 

```
(clojure.edn/read-string
  {:readers d/data-readers}
  string)
```

Upsides:

- Always works.

Downsides:

- Not incremental, always stores entire db.
- EDN serialization only.
- Reading back is slow.

## serializable/from-serializable

An interesting idea I don’t think I’ve seen anywhere. Bring your own serialization. Basically when you do

```
(d/serializable db)
```

it returns a datastructure that is serialization-friendly, even for JSON (it doesn’t use keywords, for example):

```
{"count" 1, "tx0" 536870912, "max-eid" 1, "max-tx" 536870913, "schema" "nil", "attrs" [":name"], "keywords" [], "eavt" [[1 0 "Ivan" 1]], "aevt" [0], "avet" [], "branching-factor" 512, "ref-type" "soft"}
```

After that, you are free to call

```
(cheshire/generate-string
  (d/serializable db))
```

or `pr-str`, or `transit/write`. Doesn’t matter, really. Up to you. It’s also pretty convenient, because DS itself doesn’t have to depend on these serialization libraries.

When it’s time to restore, you do the same procedure in reverse:

```
(def db
  (d/from-serializable
    (cheshire/parse-string
      string)))
```

What’s good about this apporach is that it’s also faster to deserialize than print/read method. Works both in Clojure and CLJS.

Upsides:

- Works in both Clojure and ClojureScript.
- Can choose your own serialization format and implementation.
- Faster than print/read method.

Downsides:

- Not incremental, always stores entire db.

## Storage (NEW! HOT! BEST!)

This way pretends we are a real database and does things most optimally: incremental and lazy, just like the big boys. But only on JVM, at least for now. The rest of the article will discuss this approach.

Storing database is easy. First, you have to implement `datascript.storage/IStorage` protocol:

```
(def storage
  (reify datascript.storage/IStorage
    (-store [_ addr+data-seq]
      (doseq [[addr data] addr+data-seq]
        ... serialize and store <addr> -> <data> somehow ...))
    (-restore [_ addr]
      ... load and de-serialize <data> stored at <addr>)))
```

`-store` is batched so that you can, you know, wrap it in a transaction or something. `addr` is a 64-bit number. `data` is EDN-serializable data format, containing vectors, maps, integers, keywords and whatever you yourself put into values. So, not JSON-safe, but EDN-safe (as long as you don’t put weird stuff in values).

After you’re done implementing a store, call

```
(d/store db storage)
```

and you are done!

`d/store` is slightly mutable under-the-hood. In a sense that it remembers the storage, which nodes are stored, etc. It shouldn’t affect your regular use of a database, but still, I think you should know.

The good part about this is, if you modify your database, it’ll remember which parts of the B-tree were stored, and on a next store do an incremental update!

```
(let [db' (d/db-with db
            tx-data)]
  (d/store db' storage))
```

The code above will do much less `-store` calls because only some parts of the tress have changed. That’s one of the main propositions of this approach.

BTW, you can also specify storage during db creation, and then call `d/store` without storage argument:

```
(let [schema nil
      db (d/empty-db schema {:storage storage})]
  (d/store db))
```

Just a possibility. Once stored, DB will remember its storage. Actually, storing into different storage is not supported (yet? ever?), so you can just as well start calling `(d/store db)` without storage argument.

Ok, now for the fun stuff. How to read database from disk? Simple:

```
(def db
  (d/restore storage))
```

The fun part is, that `d/restore` does exactly zero reads! That’s right, restoring a database is lazy (and thus, super-fast). Only when you start accessing it, only relevant part will be fetched. E.g. this:

```
(first (d/datoms db))
```

Will do approximately 3-6 reads depending on how deep your db is (~ 2 + log512(datoms)).

Restored DB should work exactly as a normal DB. You can read whatever you want, query as much as you want, `entity`, `pull`, `datoms`, all should work as usual. If you modify restored DB, then `store` it, store will be incremental as well.

And that’s it! That’s all you need to know.

## Storage + conn

Storage also work with `conn`, and, as a convenience, if you specify `:storage` when creating `conn`, it’ll then `store` after each `transact!`:

```
(def conn
  (d/create-conn schema {:storage storage}))

(d/transact! conn
  [[:db/add 1 :name "Ivan"]]) ;; <- will be stored automatically

(d/transact! conn
  [[:db/add 2 :name "Oleg"]]) ;; <- will be stored again
```

You can restore `conn`, too:

```
(def conn
  (d/restore-conn storage))
```

## What is storage?

Important thing to understand is: one storage can only hold one database. This is because it uses constant addr to write its root. If you try to store two or more databases, last one will essentially overwrite everything else.

You can, of course, solve it on your level. If you need to store two or more databases, make a storage that uses two or more different file system directories (file storage) or two or more SQL tables (SQL storage). It’s entirely up to you.

## File Storage

DataScript comes with one default implementation of storage: `file-storage`. It takes a directory and stores everything about a database in it. Use it like this:

```
(def storage
  (d/file-storage "/tmp/db"))

(d/store db storage)

(def db'
  (d/restore storage))
```

It accepts couple of options. If you want your own serializer (it uses EDN by default):

```
:freeze-fn :: (data)   -> String. A serialization function
:thaw-fn   :: (String) -> data. A deserialization function
```

If you want to read/write from/to input/output stream yourself:

```
:write-fn  :: (OutputStream data) -> void. Implement your own writer to FileOutputStream
:read-fn   :: (InputStream) -> Object. Implement your own reader from FileInputStream
```

And finally, if you want to control how addresses are converted to file names:

```
:addr->filename-fn :: (Long) -> String. Construct file name from address
:filename->addr-fn :: (String) -> Long. Reconstruct address from file name
```

All these options are optional.

Also, remember, it’s never safe to write to file system yourself. Always do it through battle-tested layer like SQLite or RocksDB or something. If you put `file-storage` in production, expect problems :)

## SQL Storage

See https://github.com/tonsky/datascript-storage-sql

## Garbage collection

The IStorage protocol actually has two more methods to implement:

```
(-list-addresses [_]
  "Return seq that lists all addresses currently stored in your storage.
   Will be used during GC to remove keys that are no longer used.")
  
(-delete [_ addrs-seq]
  "Delete data stored under `addrs` (seq). Will be called during GC")
```

What’s that about?

Well, when you store your database first time, every node gets an address. Then you add or remove some stuff from it, which creates new tree, which reuses parts of the old tree. You know, good old Clojure persistent data structures (it’s not literally Clojure, I have to roll my own, but the idea is the same).

Now, new tree consists of some old reused nodes that already have addresses and some new nodes that have no addresses yet. You save it again, and new nodes gets their addresses, and old nodes are just skipped, which make the whole process efficient.

Noticed the catch? New tree lost some nodes because it no longer needs them but they are still in storage! That means eventually this kind of garbage will accumulate, and your storage will hold way more nodes than needed to build last version of database.

(you might think: going back in history? but no. That’s not how we go about that)

That’s why garbage collection exists. Your storage needs to provide us with list of all the addresses that are currently in use, and a way to delete them. Then you call:

```
(d/collect-garbage storage)
```

and you are done! It will clean up everything that is not referenced by the current version of DB stored there or any past references that were restored from it and are still alive.

In the current implementation, expect that `d/collect-garbage` might be slow.

## Options

Databases support two primary options:

```
:branching-factor <int>, default 512
```

How wide your B-trees are. By default it means that each node will contain 256...512 keys. You can change it to e.g. 1024 and have less total nodes which are bigger if e.g. writing to storage has a high overhead or something. Or to something like 64 to have 32...64 keys in each node, if you want very high granularity and it’s cheaper to write small keys in your storage.

```
:ref-type :strong | :soft | :weak, default :soft
```

This is harder to explain. DataScript consists of three B-trees. Each one is, well, a tree. We store them per-node, so each node in a tree has an address if it was previously stored (new nodes have null).

Now, if some node was read from a storage (e.g. very first leaf and all the node leading to it, if you requested `(first (d/datoms db :eavt))`, then it will store its address and its value. But the value could, at any moment, be also restore from the storage, with some deserialization penalty.

So what `:ref-type` is controlling is how values in nodes that have both address and value are stored.

`:strong` means just a normal java reference, meaning, node value will never be unloaded from memory once read. You should use it if you are absolutely sure your entire database easily fits into memory. You will still benefit from lazy loading, but once loaded, it will never unload.

`:soft` is a sweet spot. It uses `SoftReference` to store the values. It means that normally values will not be unloaded, but under memory pressure, they just might.

`:weak` is similar to `:soft`, but uses `WeakReference` and will more aggressively unload your nodes. It was intended to be used in conjunction with pluggable caches like LRU, but this have not been yet implemented. TDB.

I think default options are fine for most, but in case you need to change them, you now know how.

## Conclusion

Storing gigabytes of data in DataScript and only loading parts that are need for the query was my initial vision for DataScript from the very beginning. I’m so happy we can finally get closer to that!

Let me know if you implement something cool with it.

---

### Tuples (`docs/tuples.md`)

# Composite tuples

A tuple is a collection of scalar values, represented in memory as a Clojure vector.

Composite tuples are applicable:

- when a domain entity has a multi-attribute key;
- to optimize a query that joins more than one high-population attributes on the same entity.

Composite attributes are entirely managed by DataScript–you never assert or retract them yourself. Whenever you assert or retract any attribute that is part of a composite, DataScript will automatically populate the composite value. If the current value of an entity does not include all attributes of a composite, the missing attributes will be nil.

Some ground rules:

- tuple attribute must be of cardinality one;
- tuple attribute can’t reference cardinality many attributes;
- tuple attribute can’t reference other tuple attributes;
- tuple attributes are indexed by default.

For example, consider the domain with attributes :a, :b, :c and a composite tuple `:a+b+c`

```
{:a+b+c {:db/tupleAttrs [:a :b :c]}}
```

If we transact something like this:

```
[{:db/id 1, :a "a", :b "b"}]
```

you might notice that entity 1 got composite attribute `:a+b+c` populated automatically:

```
(d/pull db '[*] 1)
; => {:db/id 1, :a "a", :b "b", :a+b+c ["a" "b" nil]}
```

If you change attributes, tuple value is updated automatically:

```
(d/transact! conn
  [{:db/id 1, :a "A", :b "B", :c "c"}])
; => {:db/id 1, :a "A", :b "B", :c "c", :a+b+c ["A" "B" "c"]}
```

If you remove ALL attributes that make up a tuple, tuple attribute is retracted too:

```
(d/transact! conn
  [[:db/add 1 :d "d"]
   [:db/retract 1 :a]
   [:db/retract 1 :b]
   [:db/retract 1 :c]])
; => {:db/id 1, :d "d"}
```

Direct update of tuple attributes is not allowed:

```
(d/transact! conn
  [{:db/id 1, :a+b+c ["A" "B" "c"]}])
; => clojure.lang.ExceptionInfo: Can’t modify tuple attrs directly: [:db/add 1 :a+b+c ["A" "B" "c"]]
```

Tuple attributes are automatically indexed:

```
(d/index-range db :a+b+c ["A" "B" "C"] ["a" "b" "c"])
; => [#db/Datom 1 :a+b+c ["a" "b" "c"]]
```

If you mark tuple attribute as `:db/unique :db.unique/value`, you get an uniqueness by composite key.

```
(def conn (d/create-conn {:a+b {:db/tupleAttrs [:a :b]
                                :db/unique :db.unique/value}}))

(d/transact! conn
  [{:db/id 1, :a "A", :b "B"}
   {:db/id 2, :a "A", :b "b"}
   {:db/id 3, :a "a", :b "B"}
   {:db/id 4, :a "a", :b "b"}])
```

Neither `:a` nor `:b` are unique per se, but their combination together is, enforced by DataScript:

```
(d/transact! conn
  [{:db/id 5, :a "A", :b "B"}])
; => clojure.lang.ExceptionInfo: Cannot add #datascript/Datom [5 :a+b ["A" "B"] 536870916 true] because of unique constraint: (#datascript/Datom [1 :a+b ["A" "b"] 536870915 true])
```

If you mark tuple attribute as `:db/unique :db.unique/identity`, you might use it in lookup refs:

```
(def conn (d/create-conn {:a+b {:db/tupleAttrs [:a :b]
                                :db/unique :db.unique/identity}}))

(d/entity (d/db conn) [:a+b ["a" "b"]])
; => {:db/id 4, :a "a", :b "b", :a+b ["a" "b"]}
```

Composite tuples are like normal attributes in most cases. Feel free to use them in lookup refs, upserts, queries, index access.

---

--- METRICS ---
- Files Extracted: 5
- Estimated Token Budget: ~7817 tokens
- Canonical Reference: https://codewiki.google/github.com/logseq/datascript