Doc/BUILDING Copy Section

Building and running your own Tuta Mail web client Copy Chapter

You can build your own Tuta Mail client and run it locally. Remember that you have to update your Tuta Mail client on
your
own. If you prefer the auto-update feature, you can use the official mail client.

Instructions here will produce a release version of the client. For debug version see HACKING.md.

#### Pre-requisites:

* An up-to-date version of Git is installed
* Node.js (check package.json engines field for the version)
* Emscripten 3.1.59
* WASM2JS (included in binaryen, sometimes includes with emscripten)
* Cargo & Rust (1.80 up)

#### Build steps:

1. Clone the repository: git clone https://github.com/tutao/tutanota.git
2. Switch into the repository directory: cd tutanota
3. Checkout the latest web release tag: git checkout tutanota-release-xxx
4. Initialize liboqs and argon2 submodules: git submodule init
5. Synchronize submodules: git submodule sync --recursive
6. Update submodules: git submodule update
7. run npm ci to install dependencies.
8. Build the web part: node webapp prod
9. Switch into the build directory: cd build
10. Run the local server e.g. npx serve build -s -p 9000 or python -m SimpleHTTPServer 9000.
11. Open localhost:9000 with your favorite browser (tested: Firefox, Chrome/Chromium, Safari).

If you try building without initializing the submodules, you might end up with

the following error:

> ``

`

> Just delete the libs/webassembly/include directory and re-build.

Building and running your own Tuta Mail Android app Copy Chapter

If you build and install the Tuta Mail Android app by yourself, keep in mind that you will not get updates
automatically.
If you prefer the auto-update feature, download the app from the Google Play Store or F-Droid.
The APK is also published on the GitHub releases page https://github.com/tutao/tutanota/releases.
You can add https://github.com/tutao/tutanota/releases.atom to your feed reader to get notified about available updates.

#### Pre-requisites:

* An up-to-date version of Git is installed
* Node.js (check package.json engines field for the version)
* An up-to-date version of the Android SDK is installed

#### Build steps:

1. Clone the repository: git clone https://github.com/tutao/tutanota.git
2. Switch into the Tutanota directory: cd tutanota
3. Checkout the latest android release tag: git checkout tutanota-android-release-xxx
4. Initialize liboqs and argon2 submodules: git submodule init
5. Synchronize submodules: git submodule sync --recursive
6. Update submodules: git submodule update
7. Install dependencies: npm ci
8. Create a keystore if you don't have one:
keytool -genkey -noprompt -keystore MyKeystore.jks -alias tutaKey -keyalg RSA -keysize 2048 -validity 10000 -deststoretype pkcs12 -storepass CHANGEME -keypass CHANGEME -dname "CN=com.example"
9. run
APK_SIGN_ALIAS="tutaKey" APK_SIGN_STORE='MyKeystore.jks' APK_SIGN_STORE_PASS="CHANGEME" APK_SIGN_KEY_PASS="CHANGEME" node android

10. Install the app on your device: adb install -r <path-to-apk> (path as printed by the build script)

Building and running your own Tuta Mail Desktop client Copy Chapter

Keep in mind that your own build of Tuta Mail Desktop will not update automatically.

Pre-requisites: Copy Section

* An up-to-date version of Git is installed.
* Node.js (check package.json engines field for the version)

Preparations: Copy Section

0. Open a terminal.
1. Clone the repository: git clone https://github.com/tutao/tutanota.git.
2. Switch into the Tutanota directory: cd tutanota
3. Checkout the latest web release tag: git checkout tutanota-release-xxx
4. Initialize liboqs and argon2 submodules: git submodule init
5. Synchronize submodules: git submodule sync --recursive
6. Update submodules: git submodule update
7. Run npm ci to install dependencies.

Build: Copy Section

1. Run node desktop --custom-desktop-release.

The client for your platform will be in artifacts/desktop/. Note that you can add --unpacked to the build command to
skip the packaging of the installer. This will yield a directory in build/desktop/ containing the client that can be run without
installation.

Extra Notes: Copy Section

The windows client uses a native dependency to enable MAPI Support. The source can be found
at https://github.com/tutao/mapirs. You can build it yourself before building the client and the build process will pick
the artifact up automatically if you structure the projects a such:

parent dir
├── mapirs
└── tutanota-3

Otherwise, the builder will load the current release from https://github.com/tutao/mapirs/releases/latest .

---

Doc/HACKING Copy Section

Hacking on the Tuta Mail client

Basic structure Copy Chapter

* src/: Common part and the desktop client code
* app-android/: Android specific parts
* app-ios/: iOS specific parts
* libs/: "vendor" directory containing our dependencies in non-minified and minified form. May be improved.
We take security seriously so we review diff between each version.
* resources/: some resources (mostly images) which are used in the project. Most of the are embedded
to the code.
* test/: test code
* android.js: script for building Android app
* make.js: script for building dev version
* desktop.js: script for building the release version of the desktop clients
* webapp.js: script for building release versions of the web application
* fdroid-metadata-workaround: is a link inside app-android so that F-Droid can find
our metadata because our Android project is not in the root. Can be removed once it's fixed in F-Droid.
* tutao-pub.pem: public key which is used to verify desktop clients

Code structure Copy Chapter

Web part of the app is split in three parts: client, worker and common. All code in the src/ except for the api/
directory is intended for GUI and system interaction. Code in the api
contains most of the logic for server communication, encryption, indexing etc.

Glossary Copy Section

* SomethingView: Big part of the app, corresponds to the URL, e.g. mail, contact, settings, search
* SomethingListView: Component which displays things in the list, usually in the second column
* SomethingViewer: Component which usually displays one element (e.g. selected email or contact)
* SomethingModel: Logic for some part of the app, lives in the main part
* SomethingController: Something that does some bookkeeping or general action but is not tied to the specific part
* SomethingFacade: Logic for one domain, lives in the api part
* SomethingApp: Something that communicates with native part to execute tasks in certain domain
* app: Part of the bigger domain structure. Currently there's system app for accounts and such and
tutanota app for mails and contacts
* Entity: Object corresponding to the server database entity
* TypeModel: Describes entity type
* TypeRef: Small object which lets us know which entity it is and find TypeModel if needed

Communication Copy Section

Worker, main thread & apps communicate through the messages. Protocol is described in the
RemoteMessageDispatcher.
See WorkerClient
and
WorkerImpl for the client and server part.

Native code communicates through the NativeInterface.

UI code Copy Section

UI code uses Mithril. It is a tiny framework which does routing & implement virtual DOM. It
uses a "hyperscript" language (m(ComponentOrDomElement, {param: value}, [children]). It may seem intimidating at first
but it's actually quite simple.

#### Defining a new Mithril component

Our preferred way of making Mithril components is through the ES6 classes. Before we've been creating instances of these
classes manually but that's not how Mithril should be used. Preferred way is to pass class and attributes
("props" if you come from React) to hyperscript and let it do its thing. Because of that we sometimes have two versions
of the components, newer one has "N" suffix like ButtonN. It is almost always preferable to use new-style components.

Current preferred way looks roughly like that:

// Defining
import { Component } from "mithril"
type Attrs = { param1: string, paramTwo?: number }
class MyComponent implements Component<Attrs> {
	view(vnode: Vnode<Attrs>) {
		return m(".h1", "Hello " + vnode.attrs.param1)
	}
}
// Usage
// ...
m(MyComponent, { param1: "Mithril", param2: 1 })

Network Copy Section

For working with entities it is preferable to use injected EntityWorker whenever possible and not using freestanding
functions. It makes easier to substitute network interfaces when needed.

One level below EntityWorker lays EntityRestInterface which is either EntityRestClient or EntityRestCache
currently. Caches saves requested entities is the memory and updates them with WebSocket events.

If you're listening for WebSocket updates in the worker part (and you should justify doing that) then you should change
EventBus to do that. For the main thread you can subscribe to the
EventController.

EventBus and EntityRestClient make sure that entities are automatically encrypted/decrypted when needed. See
decryptAndMapToInstance().

#### Entity updates

Most of the server database changes are reflected in the EntityUpdates we receive from server. They describe operation
which happened to the entity. Updates are grouped into EntityEventBatches. These batches are ordered and client tries
tp stay up-to-date with the server (for caching and indexing).

Workflow and Testing Copy Chapter

See HACKING for build pre-requisites.

Prepare the project:

1. Clone the repository: git clone https://github.com/tutao/tutanota.git
2. Switch into the repository directory: cd tutanota
3. Initialize liboqs and argon2 submodules: git submodule init
4. Synchronize submodules: git submodule sync --recursive
5. Update submodules: git submodule update
6. Run npm ci to install dependencies.

To build the web client without specific target (will use browser URL as an API endpoint).

node make

You can run node make prod to run it against the production server.

Start any web server serving build directory, and you should be good to go. e.g.

npx serve build -s -p 9000 or python -m SimpleHTTPServer 9000

To build desktop client against the production server:

node make -d prod

Android app Copy Chapter

Prerequisites:

You need to have Android SDK and NDK (28.2.13676358). The simplest way it to use Android studio but anything that can
run Gradle will do.

To build Android app against production server you first need to build webapp (like node make prod) and then build
the Android app like you would normally (e.g. import the project under android-app in Android Studio, run the app
target for the mail app).

For building calendar app run node make prod -a calendar instead.

iOS app Copy Chapter

Prerequisites:

You need XCode, xcodegen.
You might need swiftlint swift-format.
You can install them through homebrew.

To build iOS app, build the web part (node make prod). Then generate iOS projects:

pushd tuta-sdk/ios # go into SDK directory
xcodegen # generate XCode project
popd # go back
mkdir -p build
mkdir -p build-calendar-app
cd app-ios # go into iOS app directory, generate projects for both apps
xcodegen --spec calendar-project.yml
xcodegen --spec mail-project.yml

After that you can open app-ios/tuta.xcworkspace in XCode and build the mail app.

For building calendar app run node make prod -a calenar instead.

Tests Copy Section

To run sdk and node mimimi test:

cargo test --all
Some sdk & node mimimi test requires actual local http server running, as we do not have "mock" server yet

If on dev-machine, to run test including tests that requires local http server running:

cargo test --all --features test-with-local-http-server

To run all TypeScript tests:

npm test

To run only specific tests:

npm run test -- -f 'CalendarModel'

To run only specific tests without npm:

node test -f CalendarModel

To run tests in browser:

npm run test -- -br

To run tests only in browser:

npm run test -- --no-run -br

To show all test options:

npm:run test -- --help

Chunking rules Copy Chapter

- Don't import things statically which you don't want to be bundled together (e.g. importing settings from login will
load whole settings at startup)
- common-min is api/common which is used by main and worker threads and is needed on startup (marked by @bundleInto)
. rest of api/common is just common.
- main is the rest of the main thread code that is not gui related and does not depend on sanitizer/luxon
- date is luxon and everything that depends on it statically
- rest is obvious: login, mail-view, mail-editor, calendar-view, search, settings, worker
- anything can depend on common-min
- anything can depend on common except for common-min and app.js
- anything can depend on app.js except worker, common-min, common
- gui-related things (like login or mail-view) can depend on gui-base. Currently main also depends on gui-base
but it's not good
- don't depend on settings/subscription/login/mail-view/mail-editor/calendar-view/contacts things
statically
- anything that depends on luxon goes into date and is being imported dynamically
- native code is only imported from common code dynamically. Worker is exception for technical reasons.
- contacts and mail-editor depend on sanitizer statically, rest of the app doesn't

You can check if your imports respect chunking by running node webapp local.

---

Doc/TRANSLATING Copy Section

Translating Tuta

The Tuta apps are available in more than 40 languages, most of which have been translated by volunteers.
We are using POEditor to manage the translation project.
Thanks to them for approving the Tuta clients for their generous open source program!

If you're interested in improving one of the available language versions or even start a new one, drop us a message at [email protected]!

---

Doc/Events Copy Section

Entity Events

Entity events are events in a log of mutations on the server. They are generated for most entities.
We download them when we receive websocket messages and also sometimes on start/reconnect.

We rely on entity events a lot: in most situations we reactively update the UI based on entity events.
They are also used on the client to keep the cache up-to-date and to replay events for indexing.

Event logs are separate for each group but this might change in the future.

We have to process events in order in most cases so we use queues for that.

loading
|
--\     EventBus queue
   --> |_|_|_|_|_|_|_|_| --> cache ---> LoginFacade
--/                                |--> MailFacade
|                                  |--> main thread
ws messages                        |--\     Indexer queue
                                       --> |_|_|_|_|_|_|_|_|
                                   |--/
                    Indexer init --|

Indexer has a separate queue for two reasons:
- It is usually slower than the rest of the app so we don't want to wait for it to process the next event
- It loads entity events since the last launch on startup (persistent cache will also do this in the future)

In some cases we can optimize certain operations. E.g. two moves can be considered a single move operation or move and
delete can be considered a single delete. We need to be careful though and not introduce any impossible state on the
client.

If we optimize operations for the cache, we should move them earlier, not later. In the following scenario
it's clear why.

Let's say that C is a create event for the group and U is an update event for the group A.
Dots in the middle are different event. If at the point marked by ! there is an event for
the instance with ownerGroup A, then we should already have the group and if we don't we miss cache at best
and run into error in the worst case.

C ...... U
 C ...... _
 _ ...!.. C

C - create event

U - update event

! - create for instance with ownerGroup = A

_ - skipped event

=== Entities that do not send Events

In some cases the server will not send an event update to the client. This is usually tied to a specific entity type, i.e., no events of any type will be sent for a given type, and the reason is that the client does not need to react to the change or the needed reaction is handled together with some other change.

==== BlobElementEntities

Event updates have not been implemented on the server for this metatype.
Since at the moment there is only one entity type of this kind (MailDetailsBlob), and all necessary updates on these entities should be handled together with updates of the Mail type, the implementation was not needed.
This decision will have to be reevaluated every time we introduce a new BlobElementEntity.

---

Doc/Notifications Copy Section

Notifications

Notifications handling differs slightly between Desktop/Android and iOS, mostly in the
trigger part: Desktop/Android use SSE connection and when they receive "notification" event, they fetch
MissedNotification (described below). iOS is triggered by APNS messages (background or foreground). New email
notifications are fully controlled by the server for iOS, for SSE clients we get email notification in
MissedNotification.

MissedNotification Copy Chapter

###GET

URL: /rest/sys/missednotification/{ID}

ID is the device id (PushIdentifier.identifier) converted to customId. To get
custom ID one converts a string to Base64URL.

Headers:

|Name |Value
|---------------------------|-----
|userIds |comma-separated list of user IDs who are logged in on this device
|lastProcessedNotificationId|Last returned lastProcessedNotificationId (explanation below)
|v |(system) model version
|cv |client version

lastProcessedNotificationId is the last ID which was fetched and processed by the client. It is returned with
MissedNotification. Client doesn't have to understand it but it should just use it for the next request to not get
the same data twice.

Returned

200, MissedNotification. See model definition.

403, not authorized, userIds don't match missed notification, should invalidate local data

404, no missed notification to fetch (may still return empty MissedNotification in some cases)

TTL Copy Section

---

Doc/Search Copy Section

General structure

Specific indexers - for specific types

IndexerCore - encrypt, write to db

DbFacade - access to indexeddb

SearchFacade - run search on indexed data

Mail indexer Copy Chapter

Goes over each mailbox in lockstep:

- load mailsSearchFacade from the specified date range (one day)
- load referenced mail details/attachments
- create search index entries
- extract attributes
- tokenize
- result is map from word to attributes with positions
- encrypt search index entries
- key is encrypted word
- value is attribute + positions, encoded using our binary encoding and encrypted
- result is

{
    //...
    encInstanceIdToElementData: Map<B64EncInstanceId, ElementDataSurrogate>
    // For each word there's a list of entries we want to insert
    indexMap: Map<B64EncIndexKey, Array<EncSearchIndexEntryWithTimestamp>>
}
export type ElementDataSurrogate = {
    listId: Id
    // we store it here instead of SearchIndexEntry to allow moving mails without changing the SearchIndexEntries for the mail
    encWordsB64: Array<B64EncIndexKey>
    ownerGroup: Id
}

- if we collected enough mails (500+), write it to indexedDB
- includes updating timestamp for each group (mailbox)

IndexedDB logical structure (ignoring encryption and DB representation) Copy Chapter

@startuml
object SearchIndexMetaOs {
    id
    --
    word: index
    rows
}
object SearchIndexMetaEntry {
    key
    size
    app
    type
    oldestTimestamp
}
SearchIndexMetaOs "rows" --o{ SearchIndexMetaEntry
object SearchIndexOS {
    id
    --
    attribute
    positions
}
SearchIndexMetaEntry "key" --> "id" SearchIndexOS
object ElementDataOS {
    instanceId
    --
    listId
    words
    ownerGroup
}
SearchIndexMetaOs "words" --- "words" ElementDataOS
object MetaDta {
    userEncDbKey
    encDbIv
    userGRoupKeyVersion
    mailIndexingEnabled
    excludedListIds
    lastEventIndexTimeMs
}
object GroupData {
    lastBatchIds
    indexTimestamp
    groupType
}
@enduml

We have indirection from words to SearchIndex through SearchIndexMeta as an optimization. When the rows
in indexedDB get too big (e.g. for very common terms) they are very slow to update so we split them and do some manual
bookkeeping.

In practice SearchIndexMetaEntrys are encrypted and serialized into one big binary blob. We do concat them into a
single blob because otherwise GC pressure is too high.

Search Copy Chapter

---

Doc/Theming Copy Section

Theming, colors and whitelabel

We have multiple themes built-in. Theme id is saved to the localStorage for web and into the native storage for apps.

Theme can be changed by user from multiple places:

- login screen
- appearance settings
- WhitelabelThemeSettings

Theme can also be applied automatically from:

- storage
- from window.whitelabelCustomizations (see below).
- from URL parameter in apps (see below)
- theme definition from whitelabel data (see below)

Whitelabel Copy Chapter

Custom colors can be defined for a whitelabel domain. When user logs in on a whitelabel domain they are applied to
index.html by the server (as window.whitelabelCustomizations). Custom theme is also stored in app storage.

If the user open custom domain we automatically always apply custom theme. For apps, we show it as one of the options in
the color picker.

Theme application algorithm Copy Chapter

On startup:

- If we have window.whitelabelCustomizations, apply theme from it
- If we have theme passed in URL, use that
- If we have a theme preference saved in storage [^1], resolve [^2] and use that
- Or fall back to default theme

[^1]: Storage can be either localStorage for web or device storage for apps. However, in this case we would be given a
theme if there was one already.

[^2]: System/automatic theme preference means that we need to query the system for the dark/light mode and pick a theme
according to that.

We are listening to the system theme changes and update the theme if needed (see theme.js).

After login (apps only):

- Check if whitelabelConfig exists and has colors defined. If yes:
- Check if color already exists. If no, show dialog suggesting applying a theme.
- Save new theme to storage.
- Otherwise, remove theme for this domain from storage and switch to default theme. This prevents buying whitelabel to
define themes once.

---