syncstorage-rs Documentation

Technical Documentation: mozilla-services/syncstorage-rs

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Syncstorage-rs

Mozilla Sync Storage built with Rust.
For full documentation, please navigate to the service docs here or in the documentation directory in docs/src. Our documentation is generated using mdBook and published to GitHub Pages.

Overview (`tools/README.md`) Copy Section

A collection of Sync Tools and utilities

See each directory for details:

• hawk - a tool for generating test HAWK authorization headers
• spanner - Google Cloud Platform Spanner tools for maintenance and testing
• payload-link-dataflow - custom Dataflow flex template feeding the payload-link reconciliation pipeline
• payload-reconciler - Pub/Sub-driven reconciler that finalizes and garbage-collects offloaded GCS payload objects

Installation Copy Chapter

These tools are mostly written in python. It is recommended that you create a commonly shared virtual environment using something like:

python3 -m venv venv

to create a /venv directory. To activate this, call sh /venv/bin/activate.

Script dependencies can be installed via poetry:

pip install poetry
poetry install

for each tool.

Overview (`tools/examples/README.md`) Copy Section

Example scripts for working with syncstorage-rs

• put.bash - First chmod +x put.bash then ./put.bash to simulate a PUT.

Overview (`tools/hawk/README.md`) Copy Section

Make a Hawk compatible Auth header

Dependencies and Environment Setup: Copy Chapter

To use the syncstorage-rs make_hawk_token.py script, you'll need a Python =>3.10 development environment with Poetry installed. You can also directly call the script using Poetry as described in step 5.

The easiest solution recommended to use pyenv and the pyenv-virtualenv plugin for your virtual environments
as a way to isolate the dependencies from other directories.

1. Install pyenv using the latest documentation for your platform.

2. Follow the instructions to install the pyenv-virtualenv plugin.
   See the pyenv-virtualenv documentation.

3. Ensure you've added pyenv and pyenv-virtualenv to your PATH.

   Ex:

   shell Copy

   export PATH="$HOME/.pyenv/bin:$PATH"
   eval "$(pyenv init -)"
   eval "$(pyenv virtualenv-init -)"

4. Install version, create virtualenv, activate and install dependencies from inside the hawk/ directory.
   Note you can simply install dependencies, not create a virtual environment and invoke the script using poetry run.

   shell Copy

   $ cd syncstorage-rs/tools/hawk
   # pyenv version install
   $ pyenv install 3.14
   
   # creates named, associated virtualenv
   $ pyenv virtualenv 3.14 hawk # or whatever project name you like.
   $ pyenv local hawk # activates virtual env whenever you enter this directory. 
   
   # Install dependencies
   $ pip install poetry
   $ poetry install

5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):
   poetry run python make_hawk_token.py

Create a Token Header: Copy Chapter

You'll need to pass along your SYNC_MASTER_SECRET and the uri you'll be testing in order to generate a valid Hawk Id:

poetry run python make_hawk_token.py --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header

** For testing against uri's using methods other than GET, you'll need to pass along the --method flag to generate your token. Ie, poetry run python make_hawk_token.py --method PUT --uri /1.5/1/storage/meta/global --secret=$SYNC_MASTER_SECRET --as_header. See examples/put.bash for an example of this.

Use -h for help.

By default, with no passed arguments, a default Hawk Id token will be generated mapping to your localhost:8000 with
the path http://localhost:8000/1.5/1/storage/col2/.

Overview (`tools/payload-link-dataflow/README.md`) Copy Section

payload-link-dataflow

Custom Dataflow flex template that reads the payload_link_changes
Spanner change stream, drops records whose every mod has payload_link
NULL on both sides, and publishes the remainder to a Pub/Sub topic.

The downstream consumer is the payload-reconciler cronjob
(tools/payload-reconciler/), which finalizes newly-committed GCS
payload objects (committed=true, customTime=MAX) and deletes
orphaned ones.

Directory layout Copy Chapter

The Java source tree under src/ is a standalone Apache Beam pipeline
we own and build directly. It is intentionally minimal: it uses Beam's
stock SpannerIO.readChangeStream and PubsubIO.writeStrings, plus the
filter and a small JSON serializer. We do not vendor or depend on
the upstream GoogleCloudPlatform/DataflowTemplates Java tree.

Build Copy Chapter

docker build -t payload-link-dataflow:local tools/payload-link-dataflow

mvn package runs the unit tests, so a change that breaks the filter
contract fails the image build.

For a local non-Docker build:

cd tools/payload-link-dataflow
mvn -B package

The bundled jar lands at
target/payload-link-dataflow-bundled-1.0-SNAPSHOT.jar.

Publish + launch (operator workflow; lives in webservices-infra) Copy Chapter

docker tag payload-link-dataflow:local <REGISTRY>/payload-link-dataflow:<TAG>
docker push <REGISTRY>/payload-link-dataflow:<TAG>

gcloud dataflow flex-template build \
  gs://<BUCKET>/templates/payload-link-dataflow.json \
  --image <REGISTRY>/payload-link-dataflow:<TAG> \
  --sdk-language JAVA \
  --metadata-file tools/payload-link-dataflow/metadata.json

gcloud dataflow flex-template run payload-link-dataflow-<DATE> \
  --template-file-gcs-location gs://<BUCKET>/templates/payload-link-dataflow.json \
  --parameters spannerProjectId=<PROJECT> \
  --parameters spannerInstanceId=<INSTANCE> \
  --parameters spannerDatabase=<DATABASE> \
  --parameters spannerMetadataInstanceId=<METADATA_INSTANCE> \
  --parameters spannerMetadataDatabase=<METADATA_DATABASE> \
  --parameters spannerMetadataTableName=Metadata_payload_link \
  --parameters changeStreamName=payload_link_changes \
  --parameters pubsubTopic=projects/<PROJECT>/topics/payload-link-changes

spannerMetadataTableName is optional but strongly recommended for
streaming launches -- see "Job update and restart continuity" below.

The Dataflow job's service account needs:

• roles/spanner.databaseReader on the syncstorage database
• roles/spanner.databaseUser on the metadata database (the change
  stream connector writes its own partition state there)
• roles/pubsub.publisher on the destination topic
• roles/dataflow.worker

Fine-grained access control (optional) Copy Section

Pass spannerDatabaseRole to read the change stream under a Spanner
database role instead of a database-wide IAM grant. payload_link_reader
in syncstorage-spanner/src/schema.ddl is that role: it holds SELECT
on payload_link_changes and EXECUTE on the generated
READ_payload_link_changes table function, and nothing else.

The role applies to the change stream read only. Beam's
MetadataSpannerConfigFactory deliberately does not copy databaseRole
onto the connector's metadata database config, so that connection still
uses the service account's own IAM grants. Two consequences, both from
Spanner's FGAC guidance:

• spannerMetadataDatabase must be a different database than
  spannerDatabase. The connector needs database-level write access for
  its partition state, and that grant would override the role's
  restrictions on the database being read.
• the service account must not hold roles/spanner.databaseUser on
  spannerDatabase -- it bypasses fine-grained access control outright.
  Grant roles/spanner.fineGrainedAccessUser, plus
  roles/spanner.databaseRoleUser conditioned on the role resource,
  instead.

If the metadata tables share the syncstorage database, setting
spannerDatabaseRole buys nothing security-wise: the role is assumed,
but the database-level grant the connector needs supersedes it.
Provision a dedicated metadata database first.

Job update and restart continuity Copy Section

Pass spannerMetadataTableName on any streaming (unbounded) launch so
--update and cancel-then-restart can resume from the previous job's
watermarks. The connector generates the partition metadata table name
inside expand() from a UUID.randomUUID() call, and that string
gets baked into the pipeline graph; if the new job's binary produces
a different UUID, it reinitializes against a fresh empty table and
reads from startTimestamp (defaulting to Timestamp.now()),
silently dropping records committed between the two jobs.

Pick any Spanner-legal identifier -- e.g. Metadata_payload_link -- and
keep it stable across launches. The connector creates the table on
first initialization if it does not already exist, then reuses it.
Beam's own recommendation: "you have to explicitly set the metadata
table name parameter of the new job when updating it. Use the value
of the metadataTable pipeline option from the job you are
updating."

Notes on stop modes:

• --update: preserves Dataflow's checkpointed state but does not
  transfer the metadata table binding -- pin the name or lose state.
• --cancel: leaves the metadata table intact; the next launch with
  the same name resumes cleanly.
• --drain: the change stream connector does not support drain
  (per Google's guidance).
  Use --cancel or --update instead.

Filter behaviour Copy Chapter

PayloadLinkChangesToPubSub.isPayloadLinkActionable keeps a
DataChangeRecord when any of its mods has a non-null payload_link
on either side. It drops only the case where every mod has both
old and new payload_link NULL -- the inert INSERT/DELETE noise from
column-scoped change streams. Malformed records pass through so the
reconciler / DLQ -- not this filter -- surfaces them.

Beyond the filter: other differences from upstream Copy Chapter

upstream-customization.patch expresses only the filter step. Our
pipeline additionally diverges from upstream's
Cloud_Spanner_Change_Streams_to_PubSub template in ways that diff
does not attempt to capture. Run ./generate-full-delta.sh to see
the complete diff against the pinned upstream SHA; the substantive
differences are:

• Sink swap. PubsubIO.writeStrings + a custom serializeRecord,
  not upstream's FileFormatFactorySpannerChangeStreamsToPubSub.
  Emits raw JSON strings; no PubsubMessage attributes.
• JSON wire format. Emits only {commitTimestamp, modType, tableName, mods[]}. Omits fields upstream would emit:
  serverTransactionId, rowType[],
  numberOfRecordsInTransaction,
  numberOfPartitionsInTransaction, partitionToken,
  recordSequence, isLastRecordInTransactionInPartition,
  valueCaptureType.
• No UncaughtExceptionLogger.register(). Uncaught exceptions
  flow to the runner's default handler.
• No @Template / TemplateCategory annotations. Upstream uses
  the com.google.cloud.teleport.metadata annotation-driven
  registrar for flex-template metadata. We register via
  metadata.json + gcloud dataflow flex-template build instead --
  simpler, no annotation-processor dependency.
• No support for these upstream options:
  useSpannerEmulatorHost, spannerHost,
  spannerChangeStreamTvfNameList, outputMessageMetadata,
  outputDataFormat (JSON/Avro switch), pubsubAPI. Our
  PayloadLinkOptions deliberately has a smaller surface (~12
  options vs. upstream's ~25).
• No ValueProvider indirection on options. Upstream wraps
  several config fields in ValueProvider for template
  parameterisation; we take plain strings.
• No explicit streaming=true mutation. Beam infers streaming
  from the source (SpannerIO.readChangeStream is
  @UnboundedPerElement) and DataflowRunner flips the flag at
  translation time.

These deltas were deliberate scope choices, not oversights. The
filter-only patch is kept as the primary review artifact because
it's small enough to eyeball, applies cleanly against the pinned
SHA, and is insensitive to upstream churn on features we don't use.

main() matches upstream on two pipeline-shape invariants that are
not deployment choices for this source: it appends use_runner_v2
to options.experiments and calls setEnableStreamingEngine(true).
BundleFinalizer (used by SpannerIO.readChangeStream) requires the
Dataflow Portable Runner ("Runner V2"), and V2 streaming jobs
require Streaming Engine -- so both settings are pinned in code
rather than left to launch-time flags.

Output wire format Copy Chapter

Each surviving DataChangeRecord is serialized to a JSON Pub/Sub
message of the form:

{
  "commitTimestamp": "2026-06-29T12:34:56.000000000Z",
  "modType": "UPDATE",
  "tableName": "bsos",
  "mods": [
    {
      "keys": "{\"fxa_uid\":\"...\",\"fxa_kid\":\"...\",\"collection_id\":1,\"bso_id\":\"...\"}",
      "oldValues": "{\"payload_link\":\"gs://bucket/u/c/b/uuid-1\"}",
      "newValues": "{\"payload_link\":\"gs://bucket/u/c/b/uuid-2\"}"
    }
  ]
}

Each mod field carries a JSON string (escaped) -- the downstream
Python reconciler reads them with a second json.loads per mod. This
matches the Spanner change-streams wire convention.

Running against the compose e2e stack Copy Chapter

The docker_run_reconciliation_e2e_tests Makefile target brings up
Spanner + Pub/Sub + fake-GCS emulators and a publisher container
locally. There are two publisher variants; both produce the exact same
Pub/Sub wire format so the reconciler consumes either
indistinguishably.

Default: Python publisher Copy Section

docker/docker-compose.e2e.reconciliation.yaml builds and runs the
Python variant at payload-link-publisher-py/. Sub-second startup,
no JVM. Use this for day-to-day iteration.

make docker_run_reconciliation_e2e_tests

Swap-in: this Java publisher Copy Section

docker/docker-compose.e2e.reconciliation.java.yaml overrides the
payload-link-publisher service to build this directory's Docker
image and run the shaded JAR under Beam's DirectRunner (bypassing the
flex-template launcher's Dataflow-submission entrypoint). Use when you
need to reproduce a prod issue against the actual Java pipeline or
verify wire-format parity between the two variants.

docker compose \
  -f docker/docker-compose.spanner.yaml \
  -f docker/docker-compose.e2e.spanner.yaml \
  -f docker/docker-compose.e2e.reconciliation.yaml \
  -f docker/docker-compose.e2e.reconciliation.java.yaml \
  -f docker/docker-compose.e2e.jwk-cache.yaml \
  up --exit-code-from e2e-tests --abort-on-container-exit

Expect a ~15-30s JVM+Beam warmup and a heavier memory footprint than
the Python variant. If Docker's default resource limits are tight on
your dev machine, bump them.

Keeping the reference patch accurate Copy Chapter

upstream-customization.patch is reference documentation, not a build
input. If upstream's pipeline shape evolves or our pipeline drifts in
ways that change the conceptual diff, regenerate it:

git clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git upstream
cd upstream
git checkout <new-sha>
# Manually re-author the filter customization against the current
# upstream SpannerChangeStreamsToPubSub.java (in our src/ we own the
# whole pipeline; here we only need to express the *delta*).
git diff > ../tools/payload-link-dataflow/upstream-customization.patch
echo "<new-sha>" > ../tools/payload-link-dataflow/upstream.txt

After updating the SHA, run ./generate-full-delta.sh and skim the
output to check that our impl's other divergences (sink swap, JSON
shape, dropped options -- see "Beyond the filter" above) still make
sense against upstream's new state. If upstream restructured
something we deliberately dropped, update the "Beyond the filter"
bullets accordingly.

This is purely a documentation refresh -- it does not affect the build.

Overview (`tools/payload-reconciler/README.md`) Copy Section

payload-reconciler

Pub/Sub-driven reconciler that closes the loop on GCS payload offload.

Reads the payload-link-changes Pub/Sub topic (populated by the
tools/payload-link-dataflow/ flex template). For each mod inside each
change record:

• New payload_link non-null → patch the referenced GCS object's
  metadata to committed=true and customTime=MAX. Completes the
  2-phase commit started in syncserver/src/web/payload_offload.rs.
• Old payload_link non-null and ≠ new → delete the referenced GCS
  object. It is unreferenced now (the row's payload_link moved, was
  cleared, or the row was deleted — including TTL row-deletion-policy
  deletes).

Both operations are idempotent (re-setting metadata is a no-op; 404 on
delete is treated as success), so Pub/Sub at-least-once redelivery is
safe and no ordering key is needed.

Files Copy Chapter

Environment Copy Chapter

Local install / tests Copy Chapter

cd tools/payload-reconciler
poetry install
poetry run pytest

Unit tests cover parse_gs_url and the per-message handler. The
drain loop (sync-pull, budget vs. long-running exit conditions) has
no unit coverage — exercise it against the Pub/Sub emulator when
changing it.

Metrics Copy Chapter

Statsd counters (all under the payload_reconciler. prefix):

• finalizes — successful committed=true + customTime=MAX patch
• orphan_deletes — successful delete of an unreferenced object
• noop_skips — message processed with zero ops. Should be ~0 if the
  Dataflow filter is working; non-zero is the alarm bell that the
  filter regressed.
• gcs_404 + op:finalize / op:delete — 404 on the respective op.
  Expected non-zero under at-least-once redelivery (idempotency tax);
  a sharp rise suggests upstream state is gone unexpectedly.
• errors + kind:handler — Sustained non-zero values feed the DLQ.

Deployment Copy Chapter

The reconciler ships in the primary syncserver image, built from
the root Dockerfile. Its script and Python deps are baked in via
that Dockerfile's per-tool poetry-export / pip-wheel / pip-install
chain; K8s picks between the syncserver binary and the reconciler
script via the pod's command:.

Default: K8s cronjob (definition lives in webservices-infra), cadence
~5 min, exports RUN_BUDGET_SECONDS (e.g. 240).

Alternative: long-running K8s Deployment. Omit RUN_BUDGET_SECONDS;
the pod polls forever. Choose this when finalize-flip latency below
the cronjob cadence matters (e.g. an operator wants offloaded
payloads to become committed=true within seconds of upload rather
than up to one cronjob cycle later).

Service account needs the same roles in either mode:

• roles/pubsub.subscriber on payload-link-reconciler-sub
• roles/storage.objectAdmin on the payload bucket (covers both the
  metadata patch and delete operations)

Related Copy Chapter

• Producer: syncserver/src/web/payload_offload.rs
• Change stream definition: syncstorage-spanner/src/schema.ddl
  (look for payload_link_changes)
• Upstream pipeline: tools/payload-link-dataflow/

Overview (`tools/postgres/README.md`) Copy Section

PostgreSQL Utility Scripts

Currently there is one script: purge_ttl.py.

purge_ttl.py Copy Chapter

Purge expired items (BSOs and batches) from a Sync storage PostgreSQL instance.
The script is intended to be invoked regularly, e.g. as a cronjob.

Prerequisites Copy Section

• Python 3
• Poetry

Run Script Locally Copy Section

cd tools/postgres
poetry install
SYNC_SYNCSTORAGE__DATABASE_URL="postgresql://user:pass@localhost/syncstorage poetry run python purge_ttl.py"

Pass -h to see the list of options.

Run Tests Locally Copy Section

cd tools/postgres
poetry install
SYNC_SYNCSTORAGE__DATABASE_URL="postgresql://user:pass@localhost/syncstorage poetry run pytest test_purge_ttl.py -v"

Run with Docker Copy Section

A Docker image with the PostgreSQL build is not published anywhere at the
moment. Once that's available, this readme will be updated to include that
step.

docker run --rm \
  --entrypoint python3 \
  -e SYNC_SYNCSTORAGE__DATABASE_URL="postgresql://user:pass@host:5432/syncstorage" \
  syncstorage-postgres-image:tag \
  /app/tools/postgres/purge_ttl.py

Overview (`tools/spanner/README.md`) Copy Section

Spanner Tools and Scripts

These tools are supplemental scripts for working with the Google Cloud Platform. Follow the general installation instructions, as well as fetch the proper service account credentials file.

Remember, the GOOGLE_APPLICATION_CREDENTIALS environment variable should point to the absolute path location of your service account credential file.

e.g.

GOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/project-id-service-cred.json venv/bin/python count_expired_rows.py

See each script for details about function and use.

Dependencies and Environment Setup: Copy Chapter

To use the syncstorage-rs spanner untilities, you'll need a Python =>3.12 development environment with Poetry installed. These scripts typically run in a deployed GCP workflow, so their Dockerfile config will generally prepare all of this for you. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using Poetry as described in step 5.

The easiest solution recommended to use pyenv and the pyenv-virtualenv plugin for your virtual environments
as a way to isolate the dependencies from other directories.

1. Install pyenv using the latest documentation for your platform.

2. Follow the instructions to install the pyenv-virtualenv plugin.
   See the pyenv-virtualenv documentation.

3. Ensure you've added pyenv and pyenv-virtualenv to your PATH.

   Ex:

   shell Copy

   export PATH="$HOME/.pyenv/bin:$PATH"
   eval "$(pyenv init -)"
   eval "$(pyenv virtualenv-init -)"

4. Install version, create virtualenv, activate and install dependencies from inside the spanner/ directory.
   Note you can simply install dependencies, not create a virtual environment and invoke the script using poetry run.

   shell Copy

   $ cd syncstorage-rs/tools/spanner
   # pyenv version install
   $ pyenv install 3.10
   
   # creates named, associated virtualenv
   $ pyenv virtualenv 3.10 spanner # or whatever project name you like.
   $ pyenv local spanner # activates virtual env whenever you enter this directory.
   
   # Install dependencies
   $ pip install poetry
   $ poetry install

5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):
   Ex. poetry run python count_expired_rows.py

Overview (`tools/syncstorage-loadtest/README.md`) Copy Section

Storage load test

Note: This requires Python version 3.10+

The tests can execute in three modes:

1. Direct access: use a known master secret
2. Firefox accounts OAuth: create test accounts and use OAuth JWTs
3. Self-signed JWT: generate and sign JWTs locally with a given private key

Installation Copy Chapter

Environment Setup Copy Section

To run the syncstorage load tests, you'll need a Python >=3.10 development environment with Poetry installed. You can also directly use poetry run to execute commands as described in the usage examples below.

The easiest solution is to use pyenv and the pyenv-virtualenv plugin for your virtual environments as a way to isolate the dependencies from other projects.

1. Install pyenv using the latest documentation for your platform.

2. Follow the instructions to install the pyenv-virtualenv plugin.
   See the pyenv-virtualenv documentation.

3. Ensure you've added pyenv and pyenv-virtualenv to your PATH.

   Example:

   shell Copy

   export PATH="$HOME/.pyenv/bin:$PATH"
   eval "$(pyenv init -)"
   eval "$(pyenv virtualenv-init -)"

4. Install Python version, create virtualenv, activate and install dependencies from inside the project directory.

   Note: You can skip creating a virtual environment and invoke commands directly using poetry run.

   shell Copy

   $ cd syncstorage-loadtest
   
   # Install Python 3.10+
   $ pyenv install 3.10
   
   # Create named, associated virtualenv
   $ pyenv virtualenv 3.10 syncstorage-loadtest  # or whatever name you prefer
   $ pyenv local syncstorage-loadtest  # activates virtual env whenever you enter this directory
   
   # Install Poetry and dependencies
   $ pip install poetry
   $ poetry install

5. Once you're in your virtual environment, run the load tests using:

   bash Copy

   poetry run molotov [options] loadtest.py

Quick Install Copy Section

If you already have Poetry installed:

poetry install

Generate Key Copy Section

Run the generate-keys.sh script to generate an RSA keypair and derive the public JWK.

Since this script calls get_jwk.py and it has a dependency on autlib, call the shell script using Poetry:

poetry run ./generate-keys.sh

Otherwise, if in built virtual environment with installed Poetry dependencies:

./generate-keys.sh

This script will output two files:

• load_test.pem: The private key to be used by the load tests to create OAuth tokens
• jwk.json: The public JWK associated with the private key. This is a key of the form:

{
   "n": ...,
   "e": ...,
   "kty": "RSA"
}

Mode 1: Direct Access Copy Chapter

With a known syncstorage master secret:

SERVER_URL="http://localhost:8000#secretValue" poetry run molotov --max-runs 5 -cxv loadtest.py

Mode 2: Firefox Accounts OAuth Copy Chapter

With FxA stage accounts:

SERVER_URL="https://token.stage.mozaws.net" \
  FXA_API_HOST="https://api-accounts.stage.mozaws.net" \
  FXA_OAUTH_HOST="https://oauth.stage.mozaws.net" \
  poetry run molotov --workers 3 --duration 60 -v loadtest.py

Mode 3: Self-Signed JWTs Copy Chapter

Assuming an RSA keypair was generated, the private key pem was saved and accessible, and the token server was started with the public key JWK configured. The presence of OAUTH_PRIVATE_KEY_FILE triggers this mode when using OAuth.

Run with self-signed JWTs:

SERVER_URL="http://localhost:8000" \
  OAUTH_PRIVATE_KEY_FILE="/path/to/load_test.pem" \
  poetry run molotov --workers 100 --duration 300 -v loadtest.py

Expanded payloads (GCS offload) Copy Chapter

The batch write path can generate larger-than-default payloads so you can load
test collections whose payloads are offloaded to Google Cloud Storage. These
knobs are opt-in via environment variables; unset, the load test behaves
exactly as before.

LARGE_PAYLOAD_PROB/LARGE_PAYLOAD_SIZE control payload size; whether a
write is offloaded to GCS is a separate, per-collection server setting. To drive
a fully offloaded collection at maximum size, set OFFLOAD_COLLECTIONS to that
collection and LARGE_PAYLOAD_PROB=1.0.

Individual payloads are capped at the target collection's
max_record_payload_bytes, and each batch is kept under its max_post_bytes,
so a large enough payload results in fewer records per request.

Example — every write is a 5 MiB payload sent to the tabs collection:

SERVER_URL="http://localhost:8000" \
  OAUTH_PRIVATE_KEY_FILE="/path/to/load_test.pem" \
  LARGE_PAYLOAD_PROB=1.0 \
  LARGE_PAYLOAD_SIZE=5242880 \
  OFFLOAD_COLLECTIONS=tabs \
  poetry run molotov --workers 100 --duration 300 -v loadtest.py

Server-side prerequisites Copy Section

To actually exercise expanded/offloaded payloads end to end, the target server
must be configured for it:

• gcs_payload_bucket set and the target collection listed in
  gcs_payload_offload_collections (syncstorage settings).
• Raised limits to allow payloads beyond the 2.5 MiB default:
  max_record_payload_bytes, max_post_bytes, max_request_bytes, and
  max_total_bytes. These are read from /info/configuration. A collection
  may also raise its own limits via the collections section of that
  response; the load test resolves limits per target collection, preferring a
  collection's override and falling back to the global value.
• Any front-end proxy (e.g. nginx) request body-size limit raised to match.

Note: many workers each holding multi-MiB payloads is a real memory footprint
on the load-generating host — size the harness accordingly.

Docker Copy Chapter

Build and run this directory's image:

docker build . --tag syncstorage-loadtest:local

docker run --rm --net=host \
  -e SERVER_URL="http://localhost:8000#secretValue" \
  syncstorage-loadtest:local \
  -c "molotov --processes 1 --workers 5 --duration 60 -v loadtest.py"

Full offload pipeline on docker Copy Chapter

For the offload and change-stream load tests (STOR-628, STOR-629) there is a
compose rig that stands up syncserver with offload enabled, a Spanner emulator
carrying the change stream, Pub/Sub, GCS, the publisher, the reconciler and
this load test, then reports whether the pipeline drained clean:

make loadtest-offload-image
make loadtest-offload-handler        # 100% offloaded
make loadtest-offload-report
make loadtest-offload-down

See docs/src/tools/load-testing.md for the runbook, the knobs, and the GCP
dev path.

Happy Breaking!

Overview (`tools/tokenserver/README.md`) Copy Section

Summary of Files in tools/Tokenserver

Overview (`syncserver/src/tokenserver/README.md`) Copy Section

Tokenserver

Tokenserver is used to:

1. Authenticate Sync clients via an OAuth token that clients receive from FxA
2. Direct Sync clients to the appropriate Sync Storage node
3. Present Sync clients with the encryption key necessary to access their Sync Storage nodes

This functionality was previously provided by a Python service. Originally, the intent was to use Tokenserver as a standalone authentication service for use with various, independent microservices. In practice, it is only used for Firefox Sync, so it was rewritten in Rust to be part of the same code repository as the Sync Storage node.

• Configuration
  • Disabling Syncstorage
  • Test Mode
  • Connecting to Firefox
• Database
• Running
• Testing

Configuration Copy Chapter

You can find example settings for Tokenserver in config/local.example.toml. The available settings are described in doc comments here.

Disabling Syncstorage Copy Section

Tokenserver can be run as a standalone service by disabling the Sync Storage endpoints. This can be done simply by setting the disable_syncstorage setting to true. Note that the Sync Storage settings must still be set even when those endpoints are disabled.

Connecting to Firefox Copy Section

1. Visit about:config in Firefox
2. Set identity.sync.tokenserver.uri to http://localhost:8000/1.0/sync/1.5

This will point Firefox to the Tokenserver running alongside your local instance of Sync Storage.

Database Copy Chapter

Prior to using Tokenserver, the migrations must be run. First, install the diesel CLI tool:

cargo install diesel_cli

Then, run the migrations:

diesel --database-url mysql://sample_user:sample_password@localhost/tokenserver_rs migration --migration-dir src/tokenserver/migrations run

You should replace the above database Data Source Name (DSN) with the DSN of the database you are using.

Running Copy Chapter

Tokenserver is run alongside Sync Storage using make run.

Testing Copy Chapter

Tokenserver includes unit tests and a comprehensive suite of integration tests. These tests are run alongside the Sync Storage tests and can be run by following the instructions here and here.

Overview (`tools/payload-link-dataflow/payload-link-publisher-py/README.md`) Copy Section

payload-link-publisher-py

Dev/E2E-only Python variant of the payload-link Dataflow publisher.

Prod uses the Java flex template one directory up. This variant
exists so the docker-compose e2e stack can run without a JVM, without
Dataflow, and without minute-scale iteration times.

Design summary Copy Chapter

• Not Apache Beam. Python's Beam SDK has no Spanner change-stream
  connector; a Beam pipeline in Python would either require a
  cross-language Java expansion (JVM back in) or a custom Source
  (more code than the whole non-Beam approach). Instead we poll the
  READ_payload_link_changes TVF directly via
  google.cloud.spanner.Client.
• Wire-format identical to the Java job. serialize_record
  produces the exact JSON shape PayloadLinkChangesToPubSub.serializeRecord
  produces, so the reconciler consumes either interchangeably.
• Partition tracking. Follows partition splits: reads _root,
  picks up child-partition tokens from any child_partitions_record,
  reads each child from its advertised start_timestamp, and retires
  a parent when a child announces it. Handles OUT_OF_RANGE by
  dropping ended partitions. Required in practice — the Spanner
  emulator routes DataChangeRecords to child partitions almost
  immediately, so a _root-only reader sees zero DCRs.
• In-memory cursor. On restart, resumes from now - POLL_OVERLAP_SECONDS
  on _root. Fine for tests (compose brings the pod up alongside the
  stream); would lose events across restarts in prod.
• Hardcoded row-shape offsets. Rather than introspect the
  READ_<change_stream> TVF's ChangeRecord / DataChangeRecord /
  ChildPartitionsRecord STRUCT layout at runtime, publisher.py
  documents the field positions and accesses them positionally. If a
  future Spanner rev reorders the STRUCT, tests fail loudly and one
  offset map gets updated. Acceptable trade-off for a dev/E2E tool.

Files Copy Chapter

Environment Copy Chapter

Local install / tests Copy Chapter

cd tools/payload-link-dataflow/payload-link-publisher-py
poetry install
poetry run pytest
poetry run mypy .
poetry run ruff check .

poetry.lock is generated on first install (not yet checked in).

Unit tests cover extract_payload_link, is_payload_link_actionable,
serialize_record, and _to_json_string. Not covered by unit tests:
poll_partition (needs a live Spanner emulator) and
publish_if_actionable (needs a Pub/Sub emulator). Both are
exercised end-to-end by the docker-compose reconciliation stack
(make docker_run_reconciliation_e2e_tests).

Metrics Copy Chapter

• payload_link_publisher.published — records published to Pub/Sub
• payload_link_publisher.filtered — inert records dropped by
  is_payload_link_actionable
• payload_link_publisher.errors.publish — publish failures

Related Copy Chapter

• Java flex template (prod): tools/payload-link-dataflow/ (parent dir)
• Change stream definition: syncstorage-spanner/src/schema.ddl
  (payload_link_changes)
• Downstream reconciler: tools/payload-reconciler/
• Runbook: docs/src/tools/payload_link_reconciler.md

Overview (`tools/tokenserver/loadtests/README.md`) Copy Section

Tokenserver Load Tests

This directory contains everything needed to run the suite of load tests for Tokenserver.

Prerequisite Dependency and Environment Setup: Copy Chapter

To use the syncstorage-rs tokenserver load tests, you'll need a Python =>3.12 development environment with Poetry installed. This testing script can be run within a spun up GCP workflow where you can execute the locust commands from the UI. If you're running them ad-hoc, you'll need to follow these instructions. You can also directly call the script using Poetry as described in step 5.

The easiest solution recommended to use pyenv and the pyenv-virtualenv plugin for your virtual environments
as a way to isolate the dependencies from other directories.

1. Install pyenv using the latest documentation for your platform.

2. Follow the instructions to install the pyenv-virtualenv plugin.
   See the pyenv-virtualenv documentation.

3. Ensure you've added pyenv and pyenv-virtualenv to your PATH.

   Ex:

   shell Copy

   export PATH="$HOME/.pyenv/bin:$PATH"
   eval "$(pyenv init -)"
   eval "$(pyenv virtualenv-init -)"

4. Install version, create virtualenv, activate and install dependencies from inside the loadtests/ directory.
   Note you can simply install dependencies, not create a virtual environment and invoke the script using poetry run.

   shell Copy

   $ cd syncstorage-rs/tools/tokenserver/loadtests
   # pyenv version install
   $ pyenv install 3.10
   
   # creates named, associated virtualenv
   $ pyenv virtualenv 3.10 loadtests # or whatever project name you like.
   $ pyenv local loadtests # activates virtual env whenever you enter this directory. 
   
   # Install dependencies
   $ pip install poetry
   $ poetry install

5. In general, to run the script with the Poetry managed dependencies - once you're already in your virtual env - run the following (more details in #3):
   Ex. poetry run python locustfile.py

Building and Running Copy Chapter

1. Install the load testing dependencies as described above:

   sh Copy

   poetry install

2. Run the generate-keys.sh script to generate an RSA keypair and derive the public JWK.

Since this script calls get_jwk.py and it has a dependency on autlib, call the shell script using Poetry:

poetry run ./generate-keys.sh

Otherwise, if in built virtual environment with installed Poetry dependencies:

./generate-keys.sh

This script will output two files:

• load_test.pem: The private key to be used by the load tests to create OAuth tokens
• jwk.json: The public JWK associated with the private key. This is a key of the form:

{
   "n": ...,
   "e": ...,
   "kty": "RSA"
}

3. Set the following environment variables/settings on Tokenserver:

   sh Copy

   # Should be set to the "n" component of the JWK
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__N
   # Should be set to the "e" component of the JWK (this value should almost always be "AQAB")
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__E
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KTY=RSA
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__USE=sig
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__ALG=RS256
   
   # These two environment variables don't affect the load tests, but they need to be set:
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__KID=""
   SYNC_TOKENSERVER__FXA_OAUTH_PRIMARY_JWK__FXA_CREATED_AT=0

   Note that, because these settings cache the JWK used to verify OAuth tokens, no verification requests will be made to FxA, so the value of SYNC_TOKENSERVER__FXA_OAUTH_VERIFIER_URL does not matter; however, Tokenserver expects it to be set, so setting it to something like http://localhost will suffice.

4. Set the following environment variables on the machine that will be running the load tests:

   • OAUTH_PEM_FILE should be set to the location of the private RSA key generated in a previous step

5. Tokenserver uses locust for load testing. To run the load tests, simply run the following command in this directory:

   sh Copy

   locust

6. Navigate your browser to http://localhost:8090, where you'll find the locust GUI. Enter the following information:

   • Number of users: The peak number of Tokenserver users to be used during the load tests
   • Spawn rate: The rate at which new users are spawned
   • Host: The URL of the server to be load tested. Note that this URL must include the protocol (e.g. "http://")

7. Click the "Start swarming" button to begin the load tests.

Populating the Database Copy Chapter

This directory includes an optional populate_db.py script that can be used to add test users to the database en masse. The script can be run like so:

poetry run python populate_db.py <sqlurl> <nodes> <number of users>

Or if in built virtualenv:

python3 populate_db.py <sqlurl> <nodes> <number of users>

where sqluri is the URL of the Tokenserver database, nodes is a comma-separated list of nodes that are already present in the database to which the users will be randomly assigned, and number of users is the number of users to be created.

Src Adr Index (`docs/src/adr/index.md`) Copy Section

Synsctorage-rs ADRs

This directory archives all the Architectural Decision Records (ADRs) for Syncstorage-rs.

Src How To Index (`docs/src/how-to/index.md`) Copy Section

How To Guides

Collection of How To guides for various Sync-related operations.

• Default Configuration for Spanner & MySQL Builds
• Run Your Own Sync-1.5 Server with Docker
• Run Your Own Sync-1.5 Server (legacy)
• Configure Sync Server for TLS (legacy)

Src Payload Offload Overview (`docs/src/payload-offload/overview.md`) Copy Section

Payload Offload

Summary Copy Chapter

A Sync BSO payload normally lives inline in a Spanner column, and a payload
larger than the per-record limit (2.5 MB by default) is rejected. Payload
offload lifts that ceiling for chosen collections: syncserver writes the
payload to a Google Cloud Storage object and stores only a gs:// URL in the
BSO's payload_link column, leaving payload NULL. The payload's byte length
is recorded alongside the URL in payload_size, since the row itself no longer
carries the bytes to measure. Usage and quota accounting reads that column, so
an offloaded payload still counts against the user's quota.

Offload is off by default, opt-in per collection, and supported on the Spanner
backend only. Nothing changes for a collection that is not opted in.

Moving the payload out of the database splits one write into two, and two
writes can disagree. Everything else in this system exists to make them agree
again. A Spanner change stream reports every change to payload_link, and a
reconciler acts on those reports: an object a live BSO row points at is kept
forever, and everything else is deleted.

Tracking epic: STOR-372, "Expand spanner storage beyond 2.5MB".

The two halves Copy Chapter

The system splits cleanly into a synchronous half that a user request waits
on, and an asynchronous half that runs behind it.

flowchart LR
    client(["Sync client"])

    subgraph request["Request path, synchronous"]
        syncserver["syncserver"]
    end

    spanner[("Spanner
    bsos.payload_link")]
    gcs[("GCS
    payload bucket")]

    subgraph cleanup["Cleanup path, asynchronous"]
        stream["Change stream
        payload_link_changes"]
        dataflow["Dataflow
        drop inert records"]
        topic["Pub/Sub
        payload-link-changes"]
        reconciler["Reconciler
        cronjob"]
    end

    client -->|"BSO read and write"| syncserver
    syncserver -->|"step 1: upload payload"| gcs
    syncserver -->|"step 2: commit the link"| spanner
    gcs -->|"resolve payload on read"| syncserver

    spanner --> stream
    stream --> dataflow
    dataflow --> topic
    topic --> reconciler
    reconciler -->|"finalize or delete"| gcs

Synchronous, in the request path. On a write, syncserver uploads the
payload to GCS before it opens the database transaction, then commits a row
carrying the resulting URL. On a read, it fetches the row first, then
downloads the payload from GCS after the database read transaction is
completed, and swaps it back into the payload field so the client sees an
ordinary BSO. Clients never learn that offload exists.

Asynchronous, behind the request. A change stream on the payload_link
column feeds a Dataflow job, which publishes the interesting records to
Pub/Sub, which a reconciler cronjob drains. The reconciler does two things:
it marks freshly committed objects as permanent, and it deletes objects no
row points at any more.

The asynchronous half is not on the critical path. If it stalls, reads and
writes keep working and cleanup falls behind.

Life of a write Copy Chapter

sequenceDiagram
    autonumber
    participant client as Sync client
    participant sync as syncserver
    participant gcs as GCS
    participant db as Spanner
    participant rec as Reconciler

    client->>sync: POST /storage/collection
    Note over sync: collection is opted into offload
    sync->>gcs: write object, committed=false, customTime=now
    gcs-->>sync: gs:// URL
    sync->>db: commit row, payload_link set, payload NULL

    alt commit succeeds
        db-->>sync: modified timestamp
        sync-->>client: 200 OK
        db->>rec: change record, via change stream and Pub/Sub
        rec->>gcs: patch committed=true, customTime=MAX
        Note over gcs: object is now permanent
    else commit fails
        db-->>sync: error
        sync->>gcs: best effort delete
        sync-->>client: error
        Note over gcs: if that delete fails,<br/>the lifecycle policy reaps it at 30 days
    end

A few things worth knowing about that upload step. Uploads for a multi-BSO
POST run concurrently, bounded by gcs_payload_max_concurrency, and they
fail fast: the first upload error abandons the request, and any object already
uploaded is left for the lifecycle policy rather than deleted inline. The
compensating delete only runs when the database transaction fails, and its
result is ignored, because the lifecycle policy is a sufficient second line.
Ignored is not unobserved: the attempt is counted as
storage.gcs.payload.cleanup, tagged with the handler and whether the delete
landed, which is how you find out an object leaked.

Between the upload and the finalize the object exists but is not protected.
That window is normally seconds to a few minutes, set by the reconciler's
cronjob cadence, and it is always far shorter than the 30 day lifecycle
window.

Life of a read Copy Chapter

sequenceDiagram
    autonumber
    participant client as Sync client
    participant sync as syncserver
    participant db as Spanner
    participant gcs as GCS

    client->>sync: GET /storage/collection, full records
    sync->>db: read rows
    db-->>sync: rows, some carrying payload_link
    Note over sync,db: the transaction closes here
    sync->>gcs: download each linked object, concurrently
    gcs-->>sync: payload bytes
    Note over sync: payload_link cleared, payload filled in
    sync-->>client: 200 OK, ordinary BSOs

The order matters. Downloads happen after the transaction closes, so a slow
GCS read never holds a Spanner transaction open. A request that only asks for
BSO ids never touches GCS at all, and neither does a collection with no
offloaded records, since there are no links to resolve.

Life of an object Copy Chapter

Every GCS object is in one of three states, and the whole design is a matter
of which transitions are allowed.

stateDiagram-v2
    [*] --> Uploaded : syncserver writes it
    Uploaded --> Committed : reconciler finalizes it
    Uploaded --> Deleted : rollback cleanup, inline
    Uploaded --> Deleted : lifecycle policy, at 30 days
    Committed --> Deleted : link replaced, or row removed
    Committed --> Committed : finalize again, no-op
    Deleted --> Deleted : delete again, 404 is success
    Deleted --> [*]

    note right of Uploaded
        committed=false
        customTime = upload time
        the lifecycle policy can reach it
    end note

    note right of Committed
        committed=true
        customTime = 2200-12-31T23:59:59Z
        the lifecycle policy cannot reach it
    end note

The important asymmetry: an object in Uploaded has two independent ways to
die, and an object in Committed has exactly one, which only a change stream
record can trigger. Age alone never deletes a committed object, because
pinning customTime to its maximum makes daysSinceCustomTime permanently
negative.

The two self-loops are not decoration. Pub/Sub delivers at least once, so
both operations are written to be safe to repeat, and a 404 NotFound is
treated as success rather than as an error.

Why it holds together Copy Chapter

The design rests on a handful of invariants. Most of the subtlety in the code
is in preserving one of them.

1. The GCS write happens before the Spanner commit. So a committed row
   never points at an object that does not exist. The reverse, an object with
   no row, is possible, and is what the cleanup arm is for.

2. An object is born untrusted. Upload sets custom metadata
   committed=false and customTime to the moment of upload. At that point
   nothing has promised to keep it.

3. Finalizing is what makes an object permanent. The reconciler flips
   committed=true and pins customTime to 2200-12-31T23:59:59Z. That
   pushes daysSinceCustomTime permanently negative, so the bucket's
   lifecycle policy can never reach the object again, no matter how old it
   gets.

4. Only the change stream authorizes a delete. Nothing removes a
   finalized object except a change record saying a row stopped pointing at
   it. There is no scanner, no sweeper, and no age-based deletion of
   finalized objects.

5. A batch commit is a handoff, not a delete. Batched writes land in
   batch_bsos first, and committing the batch moves the link into the
   permanent bsos row and drops the staging row in one transaction. The
   change stream reports that as a row that stopped pointing at the object,
   which under invariant 4 alone would delete a live payload. Syncstorage
   tags that transaction batch_commit and the reconciler skips the delete
   when it sees the tag. Every other batch_bsos removal, TTL expiry and
   user_collections cascade deletes among them, carries no tag and is
   treated as a real delete.

6. Every reconciler operation is idempotent, and 404 counts as success.
   Pub/Sub is at-least-once, so the same record is sometimes handled twice.
   Finalizing an already finalized object and deleting an already deleted one
   are both no-ops.

7. The subscription is the retry queue. The reconciler acks only what it
   handled. An unacked message comes back, and after five failed attempts it
   lands in a dead-letter topic rather than blocking the queue. There is no
   in-process retry loop and no in-window Kubernetes retry.

8. The bucket lifecycle policy is the backstop. Anything uploaded and
   never finalized, because the request died, the transaction rolled back, or
   the inline cleanup failed, is deleted 30 days after upload. That is the
   only thing standing between a crashed write and an object that leaks
   forever.

Where to look Copy Chapter

Behaviour of the asynchronous half is documented in
Reconciliation Pipeline. Which GCP
project each piece lives in, and which repo defines it, is documented in
GCP Infrastructure.

Known gaps Copy Chapter

This is a work in progress. As of this writing:

• Only the dev environment is built out. Stage and prod need the same set of
  resources plus a dedicated Spanner metadata database.
• Change stream storage cost in prod has not been measured.
• The synchronous half is barely instrumented. The rollback cleanup emits
  storage.gcs.payload.cleanup, but there are no counters or timers on
  upload or download, so how much latency offload adds to a request is not
  something metrics can currently answer.
• Load testing of both halves is still in flight, so the numbers above for
  the upload-to-finalize window are reasoned from the cronjob cadence
  rather than observed under load.
• BSOs offloaded before payload_size existed have no recorded size and count
  as 0 bytes toward usage and quota until they are rewritten. There is no
  backfill.

Src Sync Client Overview (`docs/src/sync-client/overview.md`) Copy Section

Sync Client Overview

This section is intended to provide a comprehensive guide to how Firefox Sync clients interact with the server and ultimately with each other to provide the functionality of syncing browser data between clients.

It is a somewhat technical document, but should require no in-depth knowledge. Links to more detailed API docs offer an opportunity to dig deeper.

Introduction Copy Chapter

The purpose of Sync is to exchange browser data (bookmarks, history, open tabs,
passwords, add-ons, and the like) between clients in a manner that respects a
user's security and privacy.

Syncing is facilitated through the use of a server, where data is centrally
stored. This allows for syncing to occur without pairwise interaction between
network-connected clients.

High-level architecture Copy Section

The following diagram was originally expressed using Graphviz.
It has been converted to Mermaid.

flowchart LR
  Client1["Client 1"] --> SyncServer["Sync Server"]
  Client2["Client 2"] --> SyncServer

Sync is different from most storage-in-the-cloud services in that data is
encrypted locally—that is, it cannot be read by other parties—before it is
sent to the cloud. While many services encrypt data only while it is being
transmitted, Sync keeps your data encrypted even after it has arrived at the
server.

This means that the Sync server operators cannot read your data—even if they
wanted to. The only way your data can be read is if someone possesses your
secret Sync Key (sometimes referred to as a Recovery Key). This can occur
if your device is lost or compromised, or if you reveal it to another party.
The important fact to note is that the Sync Key is never made available to the
Sync Server, and without it, your encrypted data is statistically impossible to
recover.

That being said, the server operators do have access to some limited metadata.
This includes logs of when you connected and the types, number, and rough size
of items being synchronized. This type of information is leaked by practically
every network-connected service and should not come as a surprise.

The Sync Server Copy Chapter

The Sync server performs the vital role of storing data, tracking elementary
metadata, and providing authenticated access. The Sync server is effectively a
dumb shared whiteboard—a bit bucket, if you will. It plays a very small role in
the actual syncing process, and it must be this way: since data is encrypted
before being sent to the server, there is not much the server can do to help.

The Sync server infrastructure exposes a secure HTTP interface for:

• user management and node assignment
• storage access

The storage server is a generic service and is not Sync-specific. Sync uses it
with specific semantics for how and where data is stored. These semantics are
fully described in the Sync Storage Formats documentation.

Per-user access to the Sync server is protected via authentication at the HTTP
layer. This can be implemented however the server operator chooses. Since the
bulk of Sync’s security model resides in client-side encryption—and since the
Sync server is typically accessed over transport-level encryption such as
SSL/TLS—primitive authentication schemes such as HTTP Basic Authentication are
sufficient. In fact, Mozilla’s hosted Sync service has historically used HTTP
Basic Authentication.

Collections and Records Copy Chapter

The primary concept behind the Sync server’s storage model is the collection.
Clients store objects, called records, inside collections.

Sync clients take local data, convert it into records, and upload those records
to the Sync server. Downloading data follows the same process in reverse.

Records contain basic public metadata, such as the time they were last
modified. This allows clients to selectively retrieve only the records that
have changed since the last sync operation.

An important observation is that the server has no notion of a “sync” as the
client understands it. From the server’s perspective, there is simply a series
of HTTP requests arriving from various IP addresses, performing storage
operations on a backing store. The client, however, has a well-defined sequence
of actions that together form a logical sync session, which may succeed or fail
as a whole. The server does not track or enforce this notion.

Sync Clients Copy Chapter

A Sync Client is any entity that communicates with servers providing Sync
functionality.

Sync clients come in many different forms and may support different subsets of
features. For example, some clients may be read-only.

A given client typically targets specific versions of the storage service and
specific Sync storage formats.

Storage Limits Copy Chapter

Each Mozilla account is limited to 2.5 GB of data per collection. This limit
applies across all Sync Clients associated with that account.

When a Sync Client attempts to upload more than 2.5 GB of data to a single
collection, the Sync Server will respond with a specific error code indicating a
User over quota condition.

Src SUMMARY (`docs/src/SUMMARY.md`) Copy Section

Summary

• Introduction

• Application Architecture

• Application Configuration

• Testing

  • Load Testing

• Release Process

• Frequently Asked Questions

• Data Types

• OpenAPI Documentation

• GitHub Actions

• Syncstorage API

  • API v1.5
  • API v1.1 (Obsolete)
  • API v1.0 (Obsolete)

• Syncstorage DB - Postgres

• Syncstorage DB - Spanner

• Payload Offload

  • Reconciliation Pipeline
  • GCP Infrastructure
  • Load Testing

• Tokenserver

  • Goals of Tokenserver
  • Tokenserver API
  • Tokenserver DB - Postgres
  • User Flow
  • Purge Old Records
  • Postgres Purge TTL
  • Hawk Token Generator

• Sync Client

  • Life of a Sync
  • Sync Storage Formats
  • Global Storage Version 5
  • Firefox object formats
    Mozilla Accounts Server - FxA

• How To Guides

  • Default Configuration for Spanner & MySQL Builds
  • Run Your Own Sync-1.5 Server with Docker
  • Run Your Own Sync-1.5 Server (legacy)
  • Configure Sync Server for TLS (legacy)

Documentation and MdBook Notes
Glossary of Terms
Response Codes
Terms of Service

Src Architecture (`docs/src/architecture.md`) Copy Section

Architecture

A high-level architectural overview of the Sync Service which includes Sync and Tokenserver.

Syncstorage Copy Chapter

Below is an illustration of a highly-simplified Sync flow:

graph LR

  SignIn["Sign in to FxA"]
  FxA[("FxA")]
  OAuth["Sync client gets OAuth token"]
  PresentToken["OAuth Token presented to Tokenserver"]
  Tokenserver[("Tokenserver")]
  AssignNode["Tokenserver assigns storage node"]
  InfoCollections["info/collections: Do we even need to sync?"]

  MetaGlobal["meta/global: Do we need to start over?"]
  CryptoKeys["crypto/keys: Get keys"]

  GetStorage["GET storage/<collection>: Fetch new data"]
  PostStorage["POST storage/<collection>: Upload new data"]

  %% Main flow
  SignIn --> FxA
  FxA --> OAuth
  OAuth --> PresentToken
  PresentToken --> Tokenserver
  Tokenserver --> AssignNode
  AssignNode --> InfoCollections

  %% Decision / metadata path
  InfoCollections --> MetaGlobal
  MetaGlobal --> CryptoKeys

  %% Sync operations
  CryptoKeys --> GetStorage
  CryptoKeys --> PostStorage

Storage-Client Relationship Copy Section

This high-level diagram illustrates the standard Sync collections and their relationships.

/* Detailed source-code truncated for AI context efficiency. */

Payload Offload Copy Section

On the Spanner backend, a collection can be opted into storing its BSO
payloads in a Google Cloud Storage bucket instead of inline in the database,
which lifts the per-record size ceiling. The BSO row keeps a gs:// URL in a
payload_link column, and syncserver resolves it on read so clients see an
ordinary BSO either way. A Spanner change stream feeds an asynchronous
reconciler that keeps the bucket and the database in agreement.

Offload is off by default and touches nothing for a collection that is not
opted in. See Payload Offload for the flow, and
GCP Infrastructure for where the
pieces run.

Tokenserver Copy Chapter

The intent of this file is inspired by a very sensible blog post many developers are familiar with regarding the necessity to illustrate systems with clarity. Given Sync's complexity and interrelationships with other architectures, this

Src Config (`docs/src/config.md`) Copy Section

Configuration

Rust uses environment variables for a number of configuration options. Some of these include:

In addition, Sync server configuration options can either be specified as environment variables (prefixed with SYNC_*) or in a configuration file using the --config option.

For example the following are equivalent:

$ SYNC_HOST=0.0.0.0 SYNC_MASTER_SECRET="SuperSikkr3t" SYNC_SYNCSTORAGE__DATABASE_URL=mysql://scott:tiger@localhost/syncstorage cargo run

$ cat syncstorage.local.toml
host = "0.0.0.0"
master_secret = "SuperSikkr3t"

[syncstorage]
database_url = "mysql://scott:tiger@localhost/syncstorage"
$ cargo run -- --config syncstorage.local.toml

Options can be mixed between environment variables and configuration. Environment variables have higher precedence.

Options Copy Chapter

The following configuration options are available.

Server Settings Copy Section

CORS Copy Section

Syncstorage Database Copy Section

Syncstorage Limits Copy Section

Syncstorage Features Copy Section

Syncstorage Payload Off-load Copy Section

Off-loads large BSO payloads to a Google Cloud Storage bucket, storing the
object URL in the payload_link column instead of the inline payload
column. Supported on the Spanner backend only: setting
SYNC_SYNCSTORAGE__GCS_PAYLOAD_BUCKET or
SYNC_SYNCSTORAGE__GCS_PAYLOAD_OFFLOAD_COLLECTIONS on a mysql or postgres
backend fails startup, since those backends have no payload_link column and
would silently drop the payload.

Tokenserver Database Copy Section

Tokenserver Features Copy Section

Last-seen reporting requires the last_seen Cargo feature, independently of the
storage backend. Build it with cargo build -p syncserver --features last_seen.
Docker images include this feature only when built with
SYNCSTORAGE_DATABASE_BACKEND=spanner. Reporting still defaults to off at runtime.

Tokenserver+FxA Integration Copy Section

StatsD Metrics Copy Section

Src Data Types (`docs/src/data-types.md`) Copy Section

Firefox Sync data types

The following are important details around each syncing data type.

Users can configure which Firefox data types are synced across all connected Firefoxes. On desktop, this is found on about:preferences#sync; on mobile this is found in Sync Settings under the application menu. Mobile Firefox currently only shows data types that are supported on that platform. Any changes to selected data types on any platform are applied at the account level: they will take effect on all clients connected by that account.

Bookmarks Copy Chapter

• All bookmarks are synced; users cannot specify bookmarks to include or exclude.
• Bookmarks are merged when synced, so that the result is all bookmarks from all connected devices.
• We do not identify the device that a bookmark came from. However, on mobile you will see a "Desktop Bookmarks" folder, and on desktop you will see a "Mobile bookmarks" folder that the respective platforms use by default.
• There is a record per bookmark in sync storage.

History Copy Chapter

• We sync the most recent 5,000 history entries.
• History is merged when synced, so that the result is the most recent history from all connected devices.
• There is a record per history entry in sync storage.

Open tabs Copy Chapter

• The ability to use the Send Tab feature does not require the user to be syncing tabs.
• The ability to remotely close tabs does require the user to be syncing tabs.
• Unlike other synced data, tabs are synced with an associated client/device identifier; they are not merged.
• Tabs were changed with MR 2022 to sync every 5s after a tab change.
• Synced tab data is a subset of the local tab data. We do not sync:
  • Tabs with any URLs matching these schemes. This includes reader view tabs.
  • Any tabs in private windows (which is an intentional decision).
  • The back/forward stack (i.e. the "back" and "forward" buttons are disabled when opening a remote tab).
  • Window groupings. If you have multiple windows open, each with its own tabs, all your tabs will be flattened into a single list in the Synced Tabs views.
  • Whether the tab is pinned or not. We sync pinned tabs, but other devices see them as regular tabs, and they aren't sorted in any particular order.
  • Top sites from the New Tab page. Pinned top sites are synced completely separately, as part of preferences sync. Frequently visited top sites that aren't pinned rely on the frecency from history, which sometimes will mean they become top sites after history syncs. Manual pinning of top sites are not synced.
  • Additional page state. Cookies, local storage, scroll position, and any text entered in form fields on the page are never synced.
• There is a record per device with tabs in sync storage.

Logins & passwords (AKA. Logins) Copy Chapter

• Logins & passwords are merged when synced, so that the result is all logins & passwords from all connected devices.
• There is a record per login & password entry in sync storage.

Addresses Copy Chapter

• Addresses are only enabled for specific countries/geos.
• Addresses are behind a feature flag on Android.
• There is a record per address entry in sync storage.

Payment methods (Credit Cards) Copy Chapter

• Payment methods are merged when synced, so that the result is all payment methods from all connected devices.
• There is a record per payment method (credit card) in sync storage.

Add-ons Copy Chapter

• Add-ons automatically (for now) sync between desktop clients only.
• Add-ons categorically include web extensions, themes, and language packs, but language packs do not sync.
• Themes do sync, but can be adversely affected by Settings sync (see below).
• Web extensions syncing automatically means that installation/removal or enabling/disabling of an extension from one connected device will result in that same action occurring on the other connected device.
• Web extensions syncing does not imply that the extensions share data; extension's ability to share data is based on the extension developer using the web extensions storage.sync API.
• Extensions on any platform cannot use synced storage (i.e. the storage.sync API) unless the user has checked the "Add-ons" option in Sync settings/Choose What To Sync.
• There is a record per add-on in sync storage.

Settings (Prefs) Copy Chapter

• Prefs are not merged when synced. They synced as an entire set, and the latest write wins.
• Settings sync between desktop clients only; there is no mobile analogue for desktop preferences, so no mapping exists.
• By "settings" we mean: a grab-bag of things from about:config (specifically, anything of the form services.sync.prefs.sync.*).
• All of the syncable prefs are synced: users cannot currently choose to sync only a subset of these prefs.
  • Advanced/Adventuresome users can include/exclude certain preferences (see documentation for details).

Other data Copy Chapter

There are four other collections of data that sync. These are special, and they continue syncing even if you uncheck all displayed data types in the Choose What To Sync dialog.

• clients: A list of clients, used in reconciling the list of clients received from Accounts.
• meta: A list of data that allows the client to coordinate syncing (engines declined, syncID, storageVersion, etc).
• crypto: Cryptographic keys and data for encryption/decryption.
• keys: Key management data for sync encryption.

Src Faq (`docs/src/faq.md`) Copy Section

Frequently Asked Questions

What is Sync? Copy Chapter

Sync is a system of both a backend and client engines that are responsible for the syncing of client data to the storage server.

When do things sync? Copy Chapter

Engines do a full sync on a regular period (except for Tabs, see below):

• iOS: Every 15 minutes
• Fenix: Every 4 hours (after an initial delay)
• Desktop:
  • Initial delay:
    • Wake from sleep: 2s after wake
    • After startup: 5 minutes
  • The period varies depending on state:
    • Idle: 1 hour
    • Active: 10 minutes
    • After recently syncing something: 1.5 minutes (if we've synced something new, we temporarily change the sync delay from 10min → 1.5min until we don't have anything left to sync)

Tabs were changed with MR 2022 to sync every 5s after a tab change. This is sometimes called "quick writing."

In what order do things sync? Copy Chapter

The order of the engines is determined per-platform.

• Desktop: engines, prefs, passwords, tabs, bookmarks, addons, form autofill, forms, history, extension storage
• Mobile: Matches desktop, ignoring engines and prefs as they aren't analogous

However, clients can override this.

What happens when I click on "Sync Now" in Firefox? Copy Chapter

A "Sync Now" button or link in the UI on device A only initiates syncing for that same device A – it does not force all of your connected devices to sync. This is still true when "Sync now" is a contextual option (such as the Firefox sidebar) – clicking on what appears to be "Sync now" for a connected mobile device from desktop actually only initiates syncing of that desktop.

Does every platform have its own engines? Copy Chapter

We use shared components as much as possible to avoid every platform having its own engines.

Can Mozilla read/inspect/decrypt a user's synced data? Copy Chapter

No. Client data is encrypted before it leaves the client; we cannot decrypt this outside of the client, by design. Therefore, the server is extremely limited (e.g. no searching, filtering, fancy queries, etc). If you want to do operations on the data, they have to be done on the client.

What is the structure of synced data on the storage server? Copy Chapter

The structure of the synced data is not the same as the structure of the data in the client. You don't sync entire databases. Synced data is a set of JSON key-value stores with no relationships between them.

• In these key-value stores, the key is a GUID and the value is an encrypted blob of JSON. Each of these stores represents a "collection" (bookmarks, tabs, logins, etc).
• The JSON blobs are associated with hashed user and key IDs derived from the user's FxA. There are no identifiable relationships of this data apart from the collection_id foreign key that identifies what type of data it is.
• In most cases, each item in a store is its own record (e.g. each bookmark is one record). Tabs is different, as each device's tabs is in a single record.
• Because the JSON blob is encrypted, the server cannot see the content of any synced data. This means:
  • No relationships are enforced between collections
  • No atomic updates are possible across collections, only within a collection
  • Modeling a relational DB in synced data isn't viable

Where are the user's Sync settings stored? Copy Chapter

The storage server does not store the user's Sync settings, and neither does FxA. When a user sets which types of data to sync, those are sent to the storage server as collections that the indicated user will sync. Subsequent syncs will request to this data as part of the sync process.

Syncing, step by step Copy Chapter

From the client/user perspective:

1. Fetch kSync encryption key and kXCS node assignment token from FxA
2. Obtain token and storage node endpoint from token server
3. Fetch (unencrypted) associated collections (info/collections) and their last-modified time
4. Fetch (unencrypted) global metadata (meta/global) about syncing – IDs, version numbers, etc
5. Compare global sync IDs and storage versions to determine if we can sync – or if we need to start over
6. Fetch encryption and signing keys (crypto/keys) for future syncing
7. Sync the clients engine to download new client records and commands
8. Process incoming client commands
9. Determine and update enabled engines
10. For each enabled engine:
    • Fetch new records from the server
    • Decrypt each record and handle decryption errors
    • Resolve merge conflicts between incoming and changed records from step 10
    • Insert and update new records into the local store
    • Ask the tracker for a list of IDs changed since the last sync, and upload these records to the server
    • If an engine tracks a "backlog", make some progress on that backlog. This is only true for the history engine on desktop
11. Sync the clients engine to upload outgoing commands
12. Update meta/global on the server
13. Run validation for synced engines
14. Schedule next sync

Src Gh Actions (`docs/src/gh-actions.md`) Copy Section

GitHub Actions

We lint, test, build, and deploy Syncserver-rs using GitHub Actions. We have a number of conventions to follow for security and maintainability purposes and this documentation lays this out.

For general information on GitHub Actions, please see the GitHub Actions official documentation.

Guidelines for Maintaining GitHub Actions Copy Chapter

Code Review & Approval Copy Section

• Require code reviews for all workflow changes; enforce this via branch protection rules and CODEOWNERS
• When introducing any new third-party actions, request review from the GitHub Enterprise (GHE) team and Security team. Go to Github Actions and Applications Security Review Changes in our internal mana space to submit or speak to a member of the security team. When organization-level requests are made, the GHE team routes them to the Security team for review and approval before granting access.

The following permission requests are automatically approved by the GHE team without a security review:

• Read-only permissions for all publicly available resources (code, pull requests, issues, etc.) across all public repositories in any Mozilla organization
• Permission removal or decommissioning requests of any kind

The following require security review and approval before access is granted:

• Read-only permissions for non-public resources (members, teams, settings, etc.) in public repositories
• Read-only permissions for private or internal repositories
• Write permissions for any public, private, or internal repository

A list of pre-approved apps and actions is maintained in the (GHE Pre-Approved List)[https://github.com/MoCo-GHE-Admin/Approved-GHE-add-ons/blob/main/GitHub_Applications.md].

Action Pinning & Updates Copy Section

• Pin all actions to a commit hash instead of a version tag — this applies to Mozilla, GitHub, and especially third-party actions
• Ensure GitHub Actions are kept up to date using Dependabot
• Configure a cooldown period of 7 days for Dependabot updates across all ecosystems.

Permissions & Least Privilege Copy Section

• Use least privilege for the GitHub token configured in each workflow.
• Avoid 'write' or 'admin' permissions unless absolutely necessary.
• If no specific permissions are required, set permissions: {} at the job level.
• Explicitly set persist-credentials: false when using the actions/checkout action.
• Disable any unnecessary jobs.

Injection & Script Safety Copy Section

• Review all scripts run in workflows for code injection risk, including both inline and external scripts.
• Pass all parameters to workflows using environment variables — do not use GitHub Actions expressions (${{ }}) for this; applies to github.event.*, github.ref_name, input, and output parameters
• Do not use GitHub Actions expressions for env variables — use $VARIABLE instead of ${{ env.VARIABLE }}

Event Trigger Safety Copy Section

• Avoid using pull_request_target and workflow_run event triggers whenever possible
• If these triggers are necessary, target only trusted branches and do not check out untrusted code from the pull request

Dependabot Merge Validation Copy Section

• When configuring automatic merging or making exceptions for Dependabot, validate the user not the actor:
  • Use github.event.pull_request.user.login == 'dependabot[bot]'
  • Do not use github.actor == 'dependabot[bot]'

Secrets & Publishing Copy Section

• Use Trusted Publishing when publishing packages from GitHub Actions
• Do not use caching in sensitive workflows to prevent cache poisoning
• Avoid using GITHUB_ENV and GITHUB_PATH to pass parameters between steps — use GITHUB_OUTPUT instead

Src Glossary (`docs/src/glossary.md`) Copy Section

Glossary

Auth Token
Used to identify the user after starting a session. Contains the user application ID and the expiration date.

Change Stream
A Spanner feature that emits a record for every change to a watched table or column. Sync watches the payload_link column so an offloaded payload's GCS object can be cleaned up when the row stops pointing at it. See Payload Offload.

Cluster
Group of webheads and storage devices that make up a set of Service Nodes.

Finalize
Marking an offloaded GCS object as permanent, by setting its committed metadata to true and pinning its customTime to the maximum value so the bucket's lifecycle policy can never delete it. Done by the reconciler, not by syncserver.

Generation Number
An integer that may be included in an identity certificate.
The issuing server increases this value whenever the user changes their password. By rejecting assertions with a generation number lower than the previously seen maximum for that user, the Login Server can reject assertions generated using an old password.

Hawk Auth
An HTTP authentication method using a message authentication code (MAC) algorithm to provide cryptographic verification of portions of HTTP requests.

See https://github.com/hueniverse/hawk/

HKDF
HMAC-based Key Derivation Function, a method for deriving multiple secret keys from a single master secret.

See https://tools.ietf.org/html/rfc5869

Login Server
Used to authenticate user, returns tokens that can be used to authenticate to our services.

Master Secret
A secret shared between Login Server and Service Node.
Never used directly, only for deriving other secrets.

Node
A URL that identifies a service, like http://phx345.

Node Assignment Server
A service that can attribute to a user a node.

Orphan
An offloaded GCS object that no BSO row points at any more, because the row's payload_link was replaced or the row was deleted. The reconciler deletes orphans.

Payload Link
The gs:// URL stored in a BSO's payload_link column when its payload lives in GCS rather than inline in Spanner.

Payload Offload
Storing a BSO payload in a GCS object instead of inline in the database, which lifts the per-record size limit. Off by default, opt-in per collection, Spanner only. See Payload Offload.

Service
A service Mozilla provides, like Sync.

Service Node
A server that contains the service, and can be mapped to several Nodes (URLs).

Signing Secret
Derived from the master secret, used to sign the auth token.

Token Secret
Derived from the master secret and auth token, used as secret.
This is the only secret shared with the client and is different for each auth token.

User DB
A database that keeps the user/node relation.

Weave
The original code name for the Firefox Sync service and project.

Src Introduction (`docs/src/introduction.md`) Copy Section

Syncstorage-rs

Mozilla's Sync provides a secure method for users to synchronize their data across Mozilla applications (like Firefox) using a Mozilla account. This project encapsulates the backend of the Sync service. It can be run using either a Postgres, Spanner, or MySQL database backend.

Sync operates by storing a combined version of your data on a remote server, which then synchronizes with the local Firefox copy across all your signed-in instances (referred to as connected devices, linked through a Mozilla account).

Get up and Running Copy Chapter

To get up and running quickly, see Run Your Own Sync with Docker for instructions on deploying with Docker.

For a complete list of available configuration options you'll need to consider, see the Configuration reference.

Below are detailed instructions for other setup configurations, including bootstrapping and migration instructions for Postgres, MySQL, and using the Google Spanner Emulator.

Mozilla Sync Storage built with Rust. Our documentation is generated using mdBook and published to GitHub Pages.

Initial Setup - Bootstrapping Copy Chapter

General PostgreSQL Setup Copy Section

Syncstorage-rs supports PostgreSQL as a database backend. The database connection is specified with a DSN like:

postgres://_user_:_password_@_host_/_database_

This DSN is then used for the SYNC_TOKENSERVER__DATABASE_URL & SYNC_SYNCSTORAGE__DATABASE_URL URLs.

These values are environment variables set for the application. You can view all configurations and environment variables in the Configuration documentation, specifically SYNC_TOKENSERVER__DATABASE_URL and SYNC_SYNCSTORAGE__DATABASE_URL.

Use your preferred methods, however here are some general instructions on how to setup a fresh PostgreSQL database and user:

1. First make sure you have a PostgreSQL server running. On most systems, you can start it with:

   bash Copy

   # On macOS with Homebrew
   brew services start postgresql
   
   # On Ubuntu/Debian
   sudo systemctl start postgresql

2. Create the databases using createdb:

   bash Copy

   createdb -U postgres syncstorage
   createdb -U postgres tokenserver

3. Connect to PostgreSQL to create a user and grant privileges:

   bash Copy

   psql -U postgres -d syncstorage

4. Run the following SQL statements:

   sql Copy

   CREATE USER sample_user WITH PASSWORD 'sample_password';
   GRANT ALL PRIVILEGES ON DATABASE syncstorage TO sample_user;
   GRANT ALL PRIVILEGES ON DATABASE tokenserver TO sample_user;

Connection pattern: The general pattern for connecting to a PostgreSQL database is:

psql -d database_name -U username

The -d flag is a shorter alternative for --dbname while -U is an alternative for --username.

Environment configuration: You can optionally create a .env file with your database URL:

echo "DATABASE_URL=postgres://sample_user:sample_password@localhost/syncstorage" > .env

Or manually create the file:

touch .env

And add:
DATABASE_URL=postgres://sample_user:sample_password@localhost/syncstorage

Important Note about .env files:

We don't tend to use the .env configuration in the production version of Sync, but for some choosing to self host, the .env solution may be useful. The .env file serves different purposes depending on the context:

1. For Diesel CLI migrations: Diesel automatically reads DATABASE_URL from a .env file in the current directory. When running migrations from tokenserver-postgres/ or syncstorage-postgres/, you can create a .env file in that specific directory with the appropriate database URL. This allows you to run diesel migration run without the --database-url flag.

2. For running the application: The syncstorage-rs application uses prefixed environment variables:

   • SYNC_TOKENSERVER__DATABASE_URL for the tokenserver database
   • SYNC_SYNCSTORAGE__DATABASE_URL for the syncstorage database

   These can also be set in a .env file at the project root.

Example .env file for the application (at project root):

SYNC_TOKENSERVER__DATABASE_URL=postgres://sample_user:sample_password@localhost/tokenserver
SYNC_SYNCSTORAGE__DATABASE_URL=postgres://sample_user:sample_password@localhost/syncstorage

Example .env file for diesel migrations (in tokenserver-postgres/ directory):

DATABASE_URL=postgres://sample_user:sample_password@localhost/tokenserver

Example .env file for diesel migrations (in syncstorage-postgres/ directory):

DATABASE_URL=postgres://sample_user:sample_password@localhost/syncstorage

Bootstrapping Tokenserver (Postgres) Copy Section

Tokenserver includes migrations to initialize its database, but they do not run by default. These can be enabled via the setting:

SYNC_TOKENSERVER__RUN_MIGRATIONS=true

Once you have created and defined your database, copy the URL.

SYNC_TOKENSERVER__DATABASE_URL=postgres://<DB URL>

Running Migrations Manually for Tokenserver

If you prefer to run migrations manually instead of using SYNC_TOKENSERVER__RUN_MIGRATIONS=true, you can use Diesel CLI:

Prerequisites:

1. Install diesel_cli with PostgreSQL support:

   bash Copy

   cargo install diesel_cli --no-default-features --features postgres

2. Optional: Install diesel_cli_ext for additional features (schema/model generation):

   bash Copy

   cargo install diesel_cli_ext

   For more information on diesel_cli_ext, see the diesel_cli_ext repository.

Running Migrations:

The migrations are located in the tokenserver-postgres/migrations directory. To run them:

cd tokenserver-postgres
diesel migration run --database-url="postgres://<DB URL>"

Alternatively, if you've set the DATABASE_URL environment variable:

cd tokenserver-postgres
export DATABASE_URL="postgres://<DB URL>"
diesel migration run

Undoing Migrations:

To undo the last migration:

cd tokenserver-postgres
diesel migration redo --database-url="postgres://<DB URL>"

Note: The diesel.toml configuration file in the tokenserver-postgres directory specifies the migrations directory path and schema generation settings.

Note: We have automated support for this in Tokenserver, however the manual query that must run for Tokenserver is as follows:

After migrations run, insert a node entry:

INSERT INTO nodes (id, service, node, available, current_load, capacity, downed, backoff)
VALUES (1, 1, 'https://<SYNCSTORAGE URL HERE>', 100000, 0, 100000, 0, 0)
ON CONFLICT DO NOTHING;

Bootstrapping Syncstorage (Postgres) Copy Section

Syncstorage includes migrations to initialize its database. These run by default (unlike Tokenserver).

Configure the database URL:

SYNC_SYNCSTORAGE__DATABASE_URL=postgres://<DB URL>

Running Migrations Manually for Syncstorage

If you need to run Syncstorage migrations manually, you can use Diesel CLI:

Prerequisites:
Install diesel_cli with PostgreSQL support (if not already installed):

cargo install diesel_cli --no-default-features --features postgres

Running Migrations:

The migrations are located in the syncstorage-postgres/migrations directory. To run them:

cd syncstorage-postgres
diesel migration run --database-url="postgres://<DB URL>"

Or with the DATABASE_URL environment variable:

cd syncstorage-postgres
export DATABASE_URL="postgres://<DB URL>"
diesel migration run

Undoing Migrations:

To undo the last migration:

cd syncstorage-postgres
diesel migration redo --database-url="postgres://<DB URL>"

Note: Both syncstorage-postgres and tokenserver-postgres directories contain their own diesel.toml configuration files and separate migrations directories. Each must be run from its respective directory.

Bootstrapping Tokenserver (MySQL) Copy Section

Tokenserver includes migrations to initialize its database, but they do not run by default. These can be enabled via the setting:

SYNC_TOKENSERVER__RUN_MIGRATIONS=true

NOTE: These migrations don't run with any locking (at least on MySQL), it's probably safest to limit the node count to 1 during the first run.

After migrations run, insert service and node entries:

INSERT INTO services (id, service, pattern)
VALUES (1, 'sync-1.5', '{node}/1.5/{uid}');

INSERT IGNORE INTO nodes (id, service, node, available, current_load, capacity, downed, backoff)
VALUES (1, 1, 'https://ent-dev.sync.nonprod.webservices.mozgcp.net', 100, 0, 100, 0, 0);

Bootstrapping Syncstorage (Cloud Spanner) Copy Section

Syncstorage does not support initializing Cloud Spanner instances; this must be done manually. It does support initializing its MySQL backend and will support initializing the PostgreSQL backend in the future.

The schema DDL is available here: schema.ddl

We include a basic script to create an instance and initialize the schema via Spanner's REST API: prepare-spanner.sh. This script is currently oriented to run against Cloud Spanner emulators, but it may be adapted to run against a real Spanner database.

System Requirements Copy Chapter

• cmake (>= 3.5 and < 3.30)
• gcc
• golang
• libssl-dev
• make
• pkg-config
• Rust stable
• python 3.9+
• MySQL 8.0 (or compatible)
  • libmysqlclient (brew install mysql on macOS, apt install libmysqlclient-dev on Ubuntu, apt install libmariadb-dev-compat on Debian)

Depending on your OS, you may also need to install libgrpcdev,
and protobuf-compiler-grpc.

Local Setup Copy Chapter

1. Follow the instructions below to use either MySQL or Spanner as your DB.

2. Now cp config/local.example.toml config/local.toml. Open config/local.toml and make sure you have the desired settings configured. For a complete list of available configuration options, check out the Configuration reference.

3. To start a local server in debug mode, run either:

   • make run_mysql if using MySQL or,
   • make run_spanner if using spanner.

   The above starts the server in debug mode, using your new local.toml file for config options. Or, simply cargo run with your own config options provided as env vars.

4. Visit http://localhost:8000/__heartbeat__ to make sure the server is running.

MySQL Copy Section

Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like:

mysql://_user_:_password_@_host_/_database_

To setup a fresh MySQL DB and user:

• First make sure that you have a MySQL server running, to do that run: mysqld
• Then, run the following to launch a mysql shell mysql -u root
• Finally, run each of the following SQL statements

CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password";
CREATE DATABASE syncstorage_rs;
CREATE DATABASE tokenserver_rs;

GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost;
GRANT ALL PRIVILEGES on tokenserver_rs.* to sample_user@localhost;

Note that if you are running MySQL with Docker and encountered a socket connection error, change the MySQL DSN from localhost to 127.0.0.1 to use a TCP connection.

Spanner Copy Section

Authenticating via OAuth

The correct way to authenticate with Spanner is by generating an OAuth token and pointing your local application server to the token. In order for this to work, your Google Cloud account must have the correct permissions; contact the Ops team to ensure the correct permissions are added to your account.

First, install the Google Cloud command-line interface by following the instructions for your operating system here. Next, run the following to log in with your Google account (this should be the Google account associated with your Mozilla LDAP credentials):

gcloud auth application-default login

The above command will prompt you to visit a webpage in your browser to complete the login process. Once completed, ensure that a file called application_default_credentials.json has been created in the appropriate directory (on Linux, this directory is $HOME/.config/gcloud/). The Google Cloud SDK knows to check this location for your credentials, so no further configuration is needed.

Key Revocation

Accidents happen, and you may need to revoke the access of a set of credentials if they have been publicly leaked. To do this, run:

gcloud auth application-default revoke

This will revoke the access of the credentials currently stored in the application_default_credentials.json file. If the file in that location does not contain the leaked credentials, you will need to copy the file containing the leaked credentials to that location and re-run the above command. You can ensure that the leaked credentials are no longer active by attempting to connect to Spanner using the credentials. If access has been revoked, your application server should print an error saying that the token has expired or has been revoked.

Authenticating via Service Account

An alternative to authentication via application default credentials is authentication via a service account. Note that this method of authentication is not recommended. Service accounts are intended to be used by other applications or virtual machines and not people. See this article for more information.

Your system administrator will be able to tell you which service account keys have access to the Spanner instance to which you are trying to connect. Once you are given the email identifier of an active key, log into the Google Cloud Console Service Accounts page. Be sure to select the correct project.

• Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row.
• Select "Create Key" from the pop-up menu.
• Select "JSON" from the Dialog Box.

A proper key file will be downloaded to your local directory. It's important to safeguard that key file. For this example, we're going to name the file
service-account.json.

The proper key file is in JSON format. An example file is provided below, with private information replaced by "..."

{
  "type": "service_account",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "...",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}

Note that the name service-account.json must be exactly correct to be ignored by .gitignore.

Connecting to Spanner

To point to a GCP-hosted Spanner instance from your local machine, follow these steps:

1. Authenticate via either of the two methods outlined above.
2. Open local.toml and replace database_url with a link to your spanner instance.
3. Open the Makefile and ensure you've correctly set you PATH_TO_GRPC_CERT.
4. make run_spanner.
5. Visit http://localhost:8000/__heartbeat__ to make sure the server is running.

Note, that unlike MySQL, there is no automatic migrations facility. Currently, the Spanner schema must be hand edited and modified.

Emulator

Google supports an in-memory Spanner emulator, which can run on your local machine for development purposes. You can install the emulator via the gcloud CLI or Docker by following the instructions here. Once the emulator is running, you'll need to create a new instance and a new database.

Updating the emulator version: The emulator version is pinned in two places, each marked with SPANNER_EMULATOR_VER:

• docker/docker-compose.spanner.yaml — used for e2e/integration tests
• .github/workflows/main-workflow.yml — used for CI unit tests

Both should always be set to the same version. When upgrading, update both and verify no regressions appear in CI.

Quick Setup Using prepare-spanner.sh

The easiest way to set up a Spanner emulator database is to use the prepare-spanner.sh script:

SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9020 ./scripts/prepare-spanner.sh

This script will automatically:

1. Create a test instance (test-instance) on a test project (test-project)
2. Create a test database (test-database) with the schema from schema.ddl
3. Apply all DDL statements to set up the database structure

The script looks for schema.ddl in either the current directory or in syncstorage-spanner/src/. Make sure the SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST environment variable points to your emulator's REST API endpoint (typically localhost:9020).

After running the script, make sure that the database_url config variable in your local.toml file reflects the created database (i.e. spanner://projects/test-project/instances/test-instance/databases/test-database).

To run an application server that points to the local Spanner emulator:

SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 make run_spanner

Manual Setup Using curl

If you prefer to manually create the instance and database, or need custom project/instance/database names, you can use the REST API directly. The Spanner emulator exposes a REST API on port 9020. To create an instance, use curl:

curl --request POST \
  "localhost:9020/v1/projects/$PROJECT_ID/instances" \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data "{\"instance\":{\"config\":\"emulator-test-config\",\"nodeCount\":1,\"displayName\":\"Test Instance\"},\"instanceId\":\"$INSTANCE_ID\"}"

Note that you may set PROJECT_ID and INSTANCE_ID to your liking. To create a new database on this instance, you'll need to include information about the database schema. Since we don't have migrations for Spanner, we keep an up-to-date schema in src/db/spanner/schema.ddl. The jq utility allows us to parse this file for use in the JSON body of an HTTP POST request:

DDL_STATEMENTS=$(
  grep -v ^-- schema.ddl \
  | sed -n 's/ \+/ /gp' \
  | tr -d '\n' \
  | sed 's/\(.*\);/\1/' \
  | jq -R -s -c 'split(";")'
)

This command:

• Filters out SQL comments (lines starting with --)
• Normalizes whitespace
• Removes newlines to create a single line
• Removes the trailing semicolon from the concatenated string
• Splits the DDL statements back into an array using jq

Finally, to create the database:

curl -sS --request POST \
  "localhost:9020/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/databases" \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data "{\"createStatement\":\"CREATE DATABASE \`$DATABASE_ID\`\",\"extraStatements\":$DDL_STATEMENTS}"

Note that, again, you may set DATABASE_ID to your liking. Make sure that the database_url config variable in your local.toml file reflects your choice of project name, instance name, and database name (i.e. it should be of the format spanner://projects/<your project ID here>/instances/<your instance ID here>/databases/<your database ID here>).

To run the application server that points to the local Spanner emulator:

SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST=localhost:9010 make run_spanner

Running via Docker Copy Section

This requires access to Google Cloud Rust (raw) crate. Please note that due to interdependencies, you will need to ensure that grpcio and protobuf match the version used by google-cloud-rust-raw.

1. Make sure you have Docker installed locally.

2. Copy the contents of mozilla-rust-sdk into top level root dir here.

3. Comment out the image value under syncserver in either docker/docker-compose.mysql.yaml or docker/docker-compose.spanner.yaml (depending on which database backend you want to run), and add this instead:

   yml Copy

   build:
       context: .

4. If you are using MySQL, adjust the MySQL db credentials in docker/docker-compose.mysql.yaml to match your local setup.

5. make docker_start_mysql or make docker_start_spanner - You can verify it's working by visiting localhost:8000/__heartbeat__

Connecting to Firefox Copy Section

This will walk you through the steps to connect this project to your local copy of Firefox.

1. Follow the steps outlined above for running this project using MySQL or Spanner.
2. In Firefox, go to about:config. Change identity.sync.tokenserver.uri to http://localhost:8000/1.0/sync/1.5.
3. Restart Firefox. Now, try syncing. You should see new BSOs in your MySQL or Spanner instance.

Logging Copy Chapter

Sentry: Copy Section

1. If you want to connect to the existing Sentry project for local development, login to Sentry, and go to the page with api keys. Copy the DSN value.
2. Comment out the human_logs line in your config/local.toml file.
3. You can force an error to appear in Sentry by adding a panic! into main.rs, just before the final Ok(()).
4. Now, SENTRY_DSN={INSERT_DSN_FROM_STEP_1_HERE} make run.
5. You may need to stop the local server after it hits the panic! before errors will appear in Sentry.

RUST_LOG Copy Section

We use env_logger: set the RUST_LOG env var.

The logging of non-Spanner SQL queries is supported in non-optimized builds via RUST_LOG=syncserver=debug.

Troubleshooting Copy Chapter

• rm Cargo.lock; cargo clean; - Try this if you're having problems compiling.

• Some versions of OpenSSL 1.1.1 can conflict with grpcio's built in BoringSSL. These errors can cause syncstorage to fail to run or compile.
  If you see a problem related to libssl you may need to specify the cargo option --features grpcio/openssl to force grpcio to use OpenSSL.

Sentry Copy Section

• If you're having trouble working with Sentry to create releases, try authenticating using their self hosted server option that's outlined here Ie, sentry-cli --url https://selfhosted.url.com/ login. It's also recommended to create a .sentryclirc config file. See this example for the config values you'll need.

• System Requirements

• Local Setup

  • MySQL
  • Spanner
  • Running via Docker
  • Connecting to Firefox

• Logging

  • Sentry:
  • RUST_LOG

• Tests

  • Unit tests
  • End-to-End tests

• Creating Releases

• Troubleshooting

• Related Documentation

System Requirements Copy Chapter

• cmake (>= 3.5 and < 3.30)
• gcc
• golang
• libcurl4-openssl-dev
• libssl-dev
• make
• pkg-config
• Rust stable
• python 3.9+
• MySQL 8.0 (or compatible)
  • libmysqlclient (brew install mysql on macOS, apt install libmysqlclient-dev on Ubuntu, apt install libmariadb-dev-compat on Debian)

Depending on your OS, you may also need to install libgrpcdev,
and protobuf-compiler-grpc. Note: if the code complies cleanly,
but generates a Segmentation Fault within Sentry init, you probably
are missing libcurl4-openssl-dev.

Src Mdbook Doc Notes (`docs/src/mdbook-doc-notes.md`) Copy Section

Documentation Notes and Comments

mdBook is a command line tool to create books with Markdown. It is ideal for creating product or API documentation, tutorials, course materials or anything that requires a clean, easily navigable and customizable presentation. Source: mdBook Documentation.

mdBook Copy Chapter

To build the documentation, install mdBook:

cargo install mdbook

For mermaid diagram support, you also have to install mdbook-mermaid.
Then you need to run an install command to create two minified js files ["mermaid.min.js", "mermaid-init.js"] to render mermaid diagrams:

cargo install mdbook-mermaid
mdbook-mermaid install path/to/book

To have a live interactive instance when working with docs, you can use mdBook's watch feature.

mdbook watch path/to/book

Or use the Makefile utility make doc-watch from the root of syncstorage-rs.

To build documentation locally, run:

mdbook build

This will generate the html files into the ./output directory. You can also run:

mdbook serve

which will serve those files on http://localhost:3000. You can also add the --open flag to the end of
mdbook serve which will open the docs in a browser window.

TIP: We created the handy Makefile utility doc-prev which will clean, build, and open fresh
docs for you in the browser. Just run make doc-prev in your command line.

Testing docs for validity Copy Section

It is highly recommended that any additions/changes to documentation are tested. This ensures there are no syntax issues or invalid links that will break the deployed documentation. MdBook has a useful mdbook test utility for this. We've created the Makefile command make doc-test ease, run from the root of the syncstorage-rs crate.

As Rust's documentation often serves as a method of testing itself, code blocks are evaluated in documentation. Only blocks annotated with rust are tested. To ignore, annotate as follows: rust,ignore at the open of a code block.

Integration with rustdoc Copy Section

mdbook does not cleanly integrate with rustdoc at this time. It's possible (via some fun github actions) to build the docs and include them in the deploy.

Building Pages using Github Actions Copy Chapter

Running Copy Section

You specify triggers within the .github/workflows directory, in the publish-docs.yml file.
This invokes the make_book.sh shell script to build API, mdBook, and cargo docs.

Setup Copy Section

Github Actions allows for various CI-like steps to run. The publish-docs.yaml
has two "jobs": one to do the build, another to deploy the built artifact to Github pages.

Under the repo settings, be sure to set the following settings like below:

• Actions

  • General

    • Actions permissions:
      • ☑ Allow $USER, and select non-$USER, actions and reusable workflows
      • ☑ Allow actions created by GitHub
      • ☑ Allow actions by Marketplace verified creators
    • Artifact and log retention:
      • (can use default)
    • Fork pull request workflows from outside collaborators
      • Require approval for first-time contributors
    • Workflow permission
      • Read and write permissions
      • ☑ Allow GitHub Actions to create and approve pull requests
    • Runners
      • No settings needed

  • Pages

    • Build and deployment:
    • Source: GitHub Actions

Src Mozilla Accounts (`docs/src/mozilla-accounts.md`) Copy Section

Mozilla Accounts Server

The Mozilla accounts server provides a centralized database of all user
accounts for accessing Mozilla-hosted services.

By default, Firefox will use Mozilla's hosted accounts server at
https://accounts.firefox.com. This configuration will work well for most
use cases, including for those who want to
self-host a storage server.

Users who want to minimize their dependency on Mozilla-hosted services may
also self-host an accounts server, but this setup is
incompatible with other Mozilla-hosted services.

Resources Copy Chapter

• Getting Started:
  https://mozilla.github.io/ecosystem-platform/tutorials/development-setup

• Integration with FxA:
  https://mozilla.github.io/ecosystem-platform/relying-parties/tutorials/integration-with-fxa

• API server code:
  https://github.com/mozilla/fxa/blob/main/packages/fxa-auth-server/

• Web interface code:
  https://github.com/mozilla/fxa/blob/main/packages/fxa-content-server/

Src Open Api Docs (`docs/src/open-api-docs.md`) Copy Section

Open API Documentation

OpenAPI / Swagger UI Copy Chapter

This project uses utoipa and utoipa-swagger-ui to provide interactive API documentation.

Accessing and Working With the Documentation Copy Section

It is suggested to use the stage instance of Sync and/or Tokenserver when playing with the API,
though you may also interact with your data in the production instance.

Please take care to select the URL for Tokenserver for Tokenserver requests and the Syncstorage URL for Syncstorage requests.

The Prod and Stage environments below will be available as a drop-down in the SwaggerUI:

• Sync Stage: https://sync-us-west1-g.sync.services.allizom.org.
• Sync Prod: https://sync-1-us-west1-g.sync.services.mozilla.com.
• Tokenserver Stage: https://stage-tokenserver.sync.nonprod.webservices.mozgcp.net.
• Tokenserver Prod: https://prod-tokenserver.sync.prod.webservices.mozgcp.net.

On GitHub Pages (Static Documentation):

The project automatically publishes API documentation to GitHub Pages:

• Main Documentation: https://mozilla-services.github.io/syncstorage-rs/
• Rust API Docs (cargo doc): https://mozilla-services.github.io/syncstorage-rs/api/
• OpenAPI/Swagger UI: https://mozilla-services.github.io/syncstorage-rs/swagger-ui/

When the service is running (live deployment):

URLs for Swagger and OpenAPI Spec:

• Swagger UI (Interactive): https://<your-deployment-url>/swagger-ui/
• OpenAPI Spec (JSON): https://<your-deployment-url>/api-doc/openapi.json

Replace <your-deployment-url> with:

• Production/Stage: [Add your prod/stage URL here]
• Local Development: http://localhost:8000 (or your configured port)

API Endpoints Copy Section

The API is organized into three main categories:

Syncstorage Endpoints

Endpoints for Firefox Sync data storage operations:

• GET /1.5/{uid}/info/collections - Get collection timestamps
• GET /1.5/{uid}/info/collection_counts - Get collection counts
• GET /1.5/{uid}/info/collection_usage - Get collection usage
• GET /1.5/{uid}/info/configuration - Get server configuration
• GET /1.5/{uid}/info/quota - Get quota information
• DELETE /1.5/{uid}/storage - Delete all user data
• GET /1.5/{uid}/storage/{collection} - Get BSOs from a collection
• POST /1.5/{uid}/storage/{collection} - Add or update BSOs
• DELETE /1.5/{uid}/storage/{collection} - Delete a collection or BSOs
• GET /1.5/{uid}/storage/{collection}/{bso} - Get a specific BSO
• PUT /1.5/{uid}/storage/{collection}/{bso} - Create or update a BSO
• DELETE /1.5/{uid}/storage/{collection}/{bso} - Delete a specific BSO

Tokenserver Endpoints

Endpoints for Sync node allocation and authentication:

• GET /1.0/{application}/{version} - Get sync token
• GET /__heartbeat__ - Tokenserver health check

Dockerflow Endpoints

Service health and monitoring endpoints:

• GET /__heartbeat__ - Service health check
• GET /__lbheartbeat__ - Load balancer health check
• GET /__version__ - Service version information

Exploring the Sync API Copy Section

To aid in exploring your own Sync API with Swagger, you may want to acquire your UID and other details about your Sync account. The easiest way to do so is to use the About Sync Extension. Note that this extension only works on Desktop.

Firefox Extensions Page for About Sync
GitHub Repository for About Sync

Maintenance Copy Section

When adding new endpoints:

1. Add #[utoipa::path(...)] annotation to the handler function.
2. Add the handler path to ApiDoc in syncserver/src/server/mod.rs
3. If using custom types, derive ToSchema on request/response structs.
4. Run cargo run --example generate_openapi_spec to verify the spec generates correctly. Follow instructions below.

Generating the OpenAPI Spec Locally Copy Section

If you don't want to compile the Sync server on your machine to view the API docs, follow these instructions:

Use `make api-prev`

We created a handy Makefile command called make api-prev that automatically generates the specification file, runs Swagger in Docker and opens your browser to localhost:8080. See the steps below to understand this process. Note this attempts to be platform agnostic, but might require some adaptation depending on your operating system.

Commands to generate the OpenAPI specification without running the server:

# Generate the spec to stdout
cargo run --example generate_openapi_spec

# Save to a file
cargo run --example generate_openapi_spec > openapi.json

Other options:

1. Use Docker (simplest - used in make api-prev):
   This option requires you to have run cargo run --example generate_openapi_spec > openapi.json.

   bash Copy

   docker run -p 8080:8080 -e SWAGGER_JSON=/openapi.json -v $(pwd)/openapi.json:/openapi.json swaggerapi/swagger-ui

   Then open http://localhost:8080

2. Use online Swagger Editor:

   • Go to https://editor.swagger.io/
   • Copy the contents of openapi.json
   • Paste into the editor
   • View the interactive documentation

3. Use VS Code extension:

   • Install "OpenAPI (Swagger) Editor" extension
   • Open openapi.json in VS Code
   • Click "Preview Swagger" to view interactive docs

Publishing to GitHub Pages Copy Section

The .github/workflows/publish-docs.yaml workflow automatically publishes these docs:

1. Generates the OpenAPI spec using the generate_openapi_spec example file.
2. Downloads Swagger UI from the official GitHub releases.
3. Replaces the default example Swagger API with your Sync API:
   • The default Swagger UI comes configured to display a demo "Pet Store" API
   • We use sed to replace https://petstore.swagger.io/v2/swagger.json with our openapi.json
4. Deploys everything to GitHub Pages at:
   • https://mozilla-services.github.io/syncstorage-rs/swagger-ui/

The workflow runs in parallel:

• build-mdbook job: Builds mdBook docs + Rust cargo docs
• build-openapi job: Generates OpenAPI spec + sets up Swagger UI
• combine-and-prepare job: Combines both outputs
• deploy job: Deploys to GitHub Pages

Src Release Process (`docs/src/release-process.md`) Copy Section

Creating Releases

1. Switch to master branch of syncstorage-rs
2. git pull to ensure that the local copy is up-to-date.
3. git pull origin master to make sure that you've incorporated any changes to the master branch.
4. git diff origin/master to ensure that there are no local staged or uncommited changes.
5. Bump the version number in Cargo.toml (this new version number will be designated as <version> in this checklist)
6. create a git branch for the new version git checkout -b release/<version>
7. cargo build --release - Build with the release profile release mode.
8. clog --setversion <version> - Generate release notes. We're using clog for release notes. The older v0.9.3 version is recommended (cargo install [email protected]), v0.10.0 has a bug interpreting our .clog.toml file options (which can be worked around by running it via clog -r https://github.com/mozilla-services/syncstorage-rs --from-latest-tag --setversion <version>)
9. Review the CHANGELOG.md file and ensure all relevant changes since the last tag are included.
10. git commit -am "chore: tag <version>" to commit the new version and changes
11. git tag -s -m "chore: tag <version>" <version> to create a signed tag of the current HEAD commit for release.
12. git push origin release/<version> to push the commits to a new origin release branch
13. git push --tags origin release/<version> to push the tags to the release branch.
14. Submit a Pull Request (PR) on github to merge the release branch to master.
15. Go to the GitHub release, you should see the new tag with no release information.
16. Click the Draft a new release button.
17. Enter the <version> number for Tag version.
18. Copy and paste the most recent change set from CHANGELOG.md into the release description, omitting the top 2 lines (the name and version)
19. Once your PR merges, click [Publish Release] on the GitHub release page.

Sync server is automatically deployed to STAGE, however QA may need to be notified if testing is required. Once QA signs off, then a bug should be filed to promote the server to PRODUCTION.

Src Response Codes (`docs/src/response-codes.md`) Copy Section

Response codes

These are the error response codes used by various services, including Sync.

Server-produced Response Codes Copy Chapter

Infrastructure-produced Response Codes Copy Chapter

These response codes are generated by the Mozilla Services infrastructure,
particularly the load balancers. They will not occur in self-hosting scenarios.

If you observe these values in a 503 response in Sync logs, please file an issue:
Syncstorage-rs GitHub Issues.

These codes are temporarily a mixture of strings and numeric values. This
inconsistency may be resolved at a future date.

Src Terms Of Service (`docs/src/terms-of-service.md`) Copy Section

Term of Services

By accessing or using the Firefox Sync APIs in connection with the development
of your own client software to access the Firefox Sync services (a “Third Party
Client”), you acknowledge that you will need to install and use a local version
of the Firefox Sync server for multiple account testing and that any use of
Mozilla’s hosted Firefox Sync services is subject to Mozilla’s Firefox Sync
Terms of Service at: https://www.mozilla.org/en-US/about/legal/terms/services/

Further, you agree:

• (a) to maintain and link to (including on websites from which your Third
  Party Client may be downloaded) a separate, conspicuous, and reasonably
  detailed privacy policy detailing how data collected or transmitted by your
  Third Party Client is managed and protected;
• (b) that your Third Party Client will only store data in encrypted form on
  the Firefox Sync servers operated by Mozilla;
• (c) that you and your Third Party Client will use the Firefox Sync APIs
  solely for their intended purpose;
• (d) that your Third Party Client will not hide or mask its identity as it
  uses the Services and/or Firefox Sync APIs, including by failing to follow
  required identification conventions; and
• (e) that you and your Third Party Client will not use the Firefox Sync APIs
  for any application or service that replicates or attempts to replicate the
  Services or Firefox Sync experience unless such use is non-confusing (by
  non-confusing, we mean that people should always know with whom they are
  dealing and where the information or software they are downloading came from).

You may not imply, either directly or by omission, that your Third Party Client
is produced or endorsed by Mozilla. By providing access to the Firefox Sync APIs,
Mozilla is not granting you a license to any of our trademarks.

— The Storage Team

Src Testing (`docs/src/testing.md`) Copy Section

Tests

Unit tests Copy Chapter

Run unit tests for a specific database backend using one of the following make targets:

• MySQL: make test or make test_with_coverage
• Postgres: make postgres_test_with_coverage
• Spanner: make spanner_test_with_coverage

These commands will run the Rust test suite using cargo-nextest and generate coverage reports using cargo-llvm-cov.

You'll need nextest and llvm-cov installed for full unittest and test coverage.

$ cargo install cargo-nextest --locked
  $ cargo install cargo-llvm-cov --locked

• make test - Runs all tests
• make test_with_coverage - This will use llvm-cov to run tests and generate source-based code coverage

If you need to override SYNC_SYNCSTORAGE__DATABASE_URL or SYNC_TOKENSERVER__DATABASE_URL variables, you can modify them in the Makefile or by setting them in your shell:

$ echo 'export SYNC_SYNCSTORAGE__DATABASE_URL="mysql://sample_user:sample_password@localhost/syncstorage_rs"' >> ~/.zshrc
  $ echo 'export SYNC_TOKENSERVER__DATABASE_URL="mysql://sample_user:sample_password@localhost/tokenserver?rs"' >> ~/.zshrc

Debugging unit test state Copy Section

In some cases, it is useful to inspect the mysql state of a failed test. By
default, we use the diesel test_transaction functionality to ensure test data
is not committed to the database. Therefore, there is an environment variable
which can be used to turn off test_transaction.

SYNC_SYNCSTORAGE__DATABASE_USE_TEST_TRANSACTIONS=false make test ARGS="[testname]"

Note that you will almost certainly want to pass a single test name. When running
the entire test suite, data from previous tests will cause future tests to fail.

To reset the database state between test runs, drop and recreate the database
in the mysql client:

drop database syncstorage_rs; create database syncstorage_rs; use syncstorage_rs;

End-to-End tests Copy Chapter

End-to-end (E2E) tests validate the complete integration of syncstorage-rs with a real database backend and mock Firefox Accounts server. These tests run the full Python integration test suite located in tools/integration_tests/.

Running E2E Tests Locally Copy Section

To run E2E tests, you'll need to:

1. Build a Docker image for your target backend using the appropriate Makefile target
2. Run the E2E test suite using docker-compose

The E2E tests are available for three database backends:

MySQL:

make docker_run_mysql_e2e_tests

Postgres:

make docker_run_postgres_e2e_tests

Spanner:

make docker_run_spanner_e2e_tests

Each E2E test run performs two separate docker-compose invocations:

1. No Local JWK run: starts services with no local JWK configured, runs only test_e2e.py against FxA stage
2. Local JWK & Mocked FxA run: runs all integration tests using a mocked local FxA server and local JWK; the local JWK affects only the tests in test_e2e.py
3. Outputs JUnit XML test results for each run

The E2E test configurations are defined in:

• docker/docker-compose.e2e.mysql.yaml - base
• docker/docker-compose.e2e.mysql.jwk-cache.yaml - JWK + mock FxA overlay
• docker/docker-compose.e2e.mysql.no-jwk-cache.yaml - FxA stage overlay
• docker/docker-compose.e2e.postgres.yaml
• docker/docker-compose.e2e.postgres.jwk-cache.yaml
• docker/docker-compose.e2e.postgres.no-jwk-cache.yaml
• docker/docker-compose.e2e.spanner.yaml
• docker/docker-compose.e2e.spanner.jwk-cache.yaml
• docker/docker-compose.e2e.spanner.no-jwk-cache.yaml

These compose files extend the base service definitions from their corresponding docker/docker-compose..yaml files. Syncserver configuration (JWK, FxA OAuth URL, CORS) is defined in the syncserver block of the e2e overlays.

How E2E Tests Work

The E2E tests:

• Run in a containerized environment with all dependencies (database, syncserver, mock FxA)
• Execute integration tests from tools/integration_tests/ using pytest
• Test OAuth token validation with both cached and non-cached JWKs
• Validate tokenserver functionality, including user allocation and token generation
• Test syncstorage operations like BSO creation, retrieval, and deletion

CI/CD

In GitHub Actions, E2E tests run as part of the CI/CD pipeline for each backend:

• .github/workflows/mysql.yml - mysql-e2e-tests job
• .github/workflows/postgres.yml - postgres-e2e-tests job
• .github/workflows/spanner.yml - spanner-e2e-tests job

Each workflow builds a Docker image, runs unit tests, then executes E2E tests using the same make targets described above.

Running E2E Tests Against a Local Server Copy Section

You can run the integration tests against a locally running Sync server. Start your server, then use the run_local_e2e_tests make target:

SYNC_SYNCSTORAGE__DATABASE_URL="postgres://user:pass@localhost/syncstorage" \
SYNC_TOKENSERVER__DATABASE_URL="postgres://user:pass@localhost/tokenserver" \
make run_local_e2e_tests

The target uses the following env vars:

• SYNC_SERVER_URL (default: http://localhost:8000)
• TOKENSERVER_HOST (default: http://localhost:8000)
• SYNC_MASTER_SECRET (default: secret0)
• SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL (default: http://localhost:6000)

SYNC_SYNCSTORAGE__DATABASE_URL and SYNC_TOKENSERVER__DATABASE_URL must be set when make is invoked.

To run a specific test by name:

SYNC_SYNCSTORAGE__DATABASE_URL="..." \
SYNC_TOKENSERVER__DATABASE_URL="..." \
PYTHONPATH=/path/to/syncstorage-rs/tools \
TOKENSERVER_HOST=http://localhost:8000 \
poetry -C /path/to/syncstorage-rs/tools/integration_tests \
  run pytest . -k test_meta_global_sanity

Or by full module path:

SYNC_SYNCSTORAGE__DATABASE_URL="..." \
SYNC_TOKENSERVER__DATABASE_URL="..." \
PYTHONPATH=/path/to/syncstorage-rs/tools \
TOKENSERVER_HOST=http://localhost:8000 \
poetry -C /path/to/syncstorage-rs/tools/integration_tests \
  run pytest tokenserver/test_authorization.py::TestAuthorization::test_authorized_request

HTTP Request and Response Logging

Set SYNC_TEST_LOG_HTTP=1 and pass --log-cli-level=INFO to pytest log HTTP requests and responses to stdout:

SYNC_SYNCSTORAGE__DATABASE_URL="..." \
SYNC_TOKENSERVER__DATABASE_URL="..." \
PYTHONPATH=/path/to/syncstorage-rs/tools \
TOKENSERVER_HOST=http://localhost:8000 \
SYNC_TEST_LOG_HTTP=1 \
poetry -C /path/to/syncstorage-rs/tools/integration_tests \
  run pytest . -k test_meta_global_sanity --log-cli-level=INFO

Src Adr 0001 Daily Active Use Server Side Metrics Glean (`docs/src/adr/0001-daily-active-use-server-side-metrics-glean.md`) Copy Section

Measuring Server-Side Daily Active Use (DAU) With Glean

• Status: approved
• Deciders: Taddes Korris, David Durst, JR Conlin, Phillip Jenvey
• Date: 2024-10-09

Technical Story:
Jira Epic Url

Context and Problem Statement Copy Chapter

There is an organizational requirement for each service to be able to measure its own DAU (Daily Active Users). Sync historically measured DAU via FxA through browser login. With the addition of Relay, device backups/migrations, etc. this will no longer an be accurate reflection of Sync usage, nor a direct measurement originating within Sync. This is due to increased browser sign-ins with the potential of overcounting, or not counting those logging into the browser without Sync enabled. See Decision Brief - Server-Side Sync Usage Attribution from Mozilla Accounts.

The goal is to measure DAU (and subsequently WAU & MAU) by emitting metrics from syncstorage-rs itself. This requires the following data:

• User identifier (hashed_fxa_uid)
• Timestamp
• Platform (Desktop, Fenix, iOS, Other)
• Device Family (Desktop, Mobioe, Tablet, Other)
• Device ID (hashed_device_id) for opt-out/deletion

Decision Drivers Copy Chapter

1. Simplicity of implementation, not only for internal metric emission, but for processing and querying.
2. Reduction of complexity and load on team to process and make sense of data. Ideally there is existing infrastructure to reduce the possible significant work to calculate DAU.
3. Glean is an internal tool successfully used by many teams at Mozilla.
4. The Glean team has extensive experience and can provide considerable support in establishing application metrics.
5. Extensibility: ability to expand scope of metrics to other measurements and apply methodology to other services in Sync ecosystem.

Considered Options Copy Chapter

• A. Glean Parser for Rust - Contribute to glean team repo by implementing Rust server
• B. Custom Glean Implementation - Our own custom implementation, internal only to Sync
• C. StatsD and Grafana

Decision Outcome Copy Chapter

Chosen option:

• A. "Glean Parser for Rust: for server-side measurement of DAU"

In researching possible implementation methods, it became clear that many options did not offer us the ease and flexibility to reconcile the data after emission. This is why Glean is recommended as a clear frontrunner, due to its rich tooling in aggregating, querying, and visualizing data.

However, this left an additional decision to either 1. implement our own custom Glean implementation to emit "Glean-compliant" output, or 2. to contribute to the Glean team's glean_parser repository for server-side metrics. The glean_parser is used for all server implementations of Glean, since the SDK is only available for client-side applications. Currently, Rust is not supported in the server-side.

In consultation with the Glean team, we determined that avoiding our custom implementation and opting for the general-purpose glean_parser for Rust was the ideal solution. The Glean team does/did not have capacity to implement the Rust glean_parser feature, so we had to decide if we were willing to take on the effort.

Agreeing to implement this feature gave us the benefits of a general purpose solution that solved our immediate challenge but also solved for future possible challenges, all while making this feature available to other Mozilla teams.

Pros and Cons of the Options Copy Chapter

A. Glean Parser for Rust Copy Section

Glean is a widely used tool at Mozilla and provides us with a solution to the given issue with possible extensibility in the future. There would be some challenges related to the required development effort and coordination with the Glean team. However, the potential for positive impact within our team and the organization is significant: all Rust server applications will be able to use Glean with full server support, our possible intention to integrate Glean into Push is made easier, and this is done in partnership with the Glean team.

Pros

• Makes Glean compatible for all Rust server applications going forward.
• Preferred option of the Glean team.
• Glean team believes, based on FxA metrics volume, that our volume will not be a problem (180-190K per minute).
• A collection of metrics, emitted as a single "Ping Event" make querying of related data simpler.
• Core of Glean's purpose is to measure user interactions and the rich metadata that accompanies it.
• Capacity for future expansion of application metrics within Sync, beyond DAU.
• Easier to query, have data team support to set up queries.
• Use of standardized Mozilla tooling.
• Establishment of team knowledge of using Glean.
• Establishment of server-side Rust best practices, leading to easier development for backend Rust applications.
• Transparency in data review process with consideration made to minimum collection of data. This is as a result of defining metrics.yaml files that require data steward review to implement.

Cons

• We have to write the Rust-compatible code. The glean_parser tool used by other server-side Glean applications is currently not supported for Rust.

B. Custom Glean Implementation Copy Section

This was originally the desired approach to measuring DAU via Glean. It was predicated on the ease of prototyping a custom implementation that imitated the Glean team's logic to create "Glean-compliant" output. However, in consulting with the Glean team and evaluating the pros and cons of this approach, it became clear this approach had considerably more drawbacks than implementing the glean_parser for Rust. These drawbacks were a lack of testing, validation, less support from the Glean team, and the potential problems with maintenance and adding Glean metrics in the future.

Pros

• Gives team control over implementation and allows us to customize the Glean logging as we see fit.
• Effort is smaller because it doesn't involve our integrating with an existing repo we are not familiar with. Ex. the templating logic and libraries in the glean_parser.
• Easy to prototype and make changes.

Cons

• There is no built-in testing suite or validation, so this would put a larger development burden on us and require the Glean team's review.
• Lack of testing and validation means higher likelihood of bugs.
• If we decide to add new Glean metrics in the future, this may break the custom implementation and impose a greater maintenance overhead.
• Time required to understand the Glean team's implementations in order to replicate behavior and data structures.
• Likely won't have same support from Glean team as it is not related to their implementation.

C. StatsD, Grafana, InfluxDB Copy Section

StatsD and Grafana offer us core application metrics and service health. While we use this frequently, it doesn't neatly fit the measurement requirements we have for DAU and would likely be very difficult to process via queries. This is because such application metrics are geared towards increment counters, response codes, and timers. Given DAU is a user-initiated interaction, and we need to query unique events based on the hashed_fxa_id, this is not suitable for InfluxDB/StatsD. Figuring out how to query this data poses challenges as it has not been implemented for such a use case.

Pros

• Already utilized for core service metrics (status codes, API endpoint counts, cluster health, etc).
• Well understood and used.
• Possible for SRE for more complex queries.

Cons

• StatsD is not a good format for measuring something like user interactions.
• Somewhat opaque and complicated query logic required.
• Significant difficulty in aggregation and reconciliation logic.
• May not scale well given number of events.
• Likely a considerable overhead in understanding how to make sense of data.
• Heavier load for team to manage data processing, as this approach has not been tried.

Links Copy Chapter

• Decision Brief - Server-Side Sync Usage Attribution from Mozilla Accounts
• FxA DAU Metric in Redash
• Working Document
• Sync Ecosystem Infrastructure and Metrics: KPI Metrics
• Integrating Glean
• Glean Metrics

Src Adr Template (`docs/src/adr/template.md`) Copy Section

[short title of solved problem and solution]

• Status: [proposed | rejected | accepted | deprecated | … | superseded by ADR-0005]
• Deciders: [list everyone involved in the decision]
• Date: [YYYY-MM-DD when the decision was last updated]

Technical Story: [description | ticket/issue URL]

Context and Problem Statement Copy Chapter

[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]

Decision Drivers Copy Chapter

1. [primary driver, e.g., a force, facing concern, …]
2. [secondary driver, e.g., a force, facing concern, …]
3. …

Considered Options Copy Chapter

• A. [option A]
• B. [option B]
• C. [option C]
• D. …

Decision Outcome Copy Chapter

Chosen option:

• A. "[option A]"

[justification. e.g., only option, which meets primary decision driver | which resolves a force or facing concern | … | comes out best (see below)].

Positive Consequences Copy Section

• [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
• …

Negative Consequences Copy Section

• [e.g., compromising quality attribute, follow-up decisions required, …]
• …

Pros and Cons of the Options Copy Chapter

[option A] Copy Section

[example | description | pointer to more information | …]

Pros

• [argument for]
• [argument for]
• …

Cons

• [argument against]
• …

[option B] Copy Section

[example | description | pointer to more information | …]

Pros

• [argument for]
• [argument for]
• …

Cons

• [argument against]
• …

[option C] Copy Section

[example | description | pointer to more information | …]

Pros

• [argument for]
• [argument for]
• …

Cons

• [argument against]
• …

Links Copy Chapter

• [Link type] [Link to ADR]
• …

Src How To Default Config (`docs/src/how-to/default-config.md`) Copy Section

Default Configuration for Spanner & MySQL Builds

This page describes the out-of-the-box configuration needed to run the two
standard syncstorage-rs builds:

• MySQL build — the default cargo build / make run_mysql target and the
  syncserver-mysql Docker image. Both Syncstorage and Tokenserver run
  against MySQL.
• Spanner build — make run_spanner and the syncstorage-rs Docker image
  (the Spanner image keeps the unsuffixed name). This mirrors production: Syncstorage runs against Google Cloud
  Spanner while Tokenserver runs against MySQL.

Annotated, copy-paste-ready templates live in the repo:

Copy one to config/local.toml and edit the values marked REQUIRED:

cp config/local.example.toml config/local.toml          # MySQL
# or
cp config/local.example.spanner.toml config/local.toml  # Spanner

Every setting can also be supplied as an environment variable prefixed with
SYNC_ (nested keys use __). For example syncstorage.database_url becomes
SYNC_SYNCSTORAGE__DATABASE_URL. Environment variables take precedence over the
config file. The complete list of options and their defaults is in the
Application Configuration reference, and the source of truth is
the doc-commented Settings structs in the *-settings crates.

Minimum required settings Copy Chapter

Most settings have sensible defaults. Regardless of backend you must supply:

Backend-specific notes:

• MySQL: set tokenserver.node_type = "mysql". Syncstorage MySQL schema
  migrations run automatically at startup; set
  tokenserver.run_migrations = true to apply the Tokenserver schema too.
• Spanner: syncstorage.database_url must use the
  spanner://projects/.../instances/.../databases/... form — the spanner://
  scheme is what selects the Spanner backend. Leave tokenserver.node_type at
  its default ("spanner"). To run against the local emulator instead of GCP,
  set syncstorage.spanner_emulator_host (e.g. localhost:9010); for real
  Spanner, point GOOGLE_APPLICATION_CREDENTIALS at a service-account key.

Run it out of the box with Docker Copy Chapter

For a zero-to-running MySQL stack (database + server, schema applied
automatically, ready to serve), see the
one-shot MySQL docker compose
recipe. It brings up MySQL and the server, runs migrations, and bootstraps the
storage node so that curl http://localhost:8000/__heartbeat__ succeeds with no
further setup.

A Spanner stack cannot be fully zero-config: real Spanner requires GCP
credentials, and the local emulator requires extra wiring (see
docker/docker-compose.spanner.yaml and make run_spanner).

Src How To How To Config Tls (`docs/src/how-to/how-to-config-tls.md`) Copy Section

Configure your Sync server for TLS

Firefox for Android versions 39 and up request the following protocols and
cipher suites, depending on the Android OS version.

The use of AES128 in preference to AES256 is driven by power and CPU
concerns.

Cipher Suites Copy Chapter

Android 20+ Copy Section

• TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA
• TLS_RSA_WITH_AES_128_CBC_SHA

Android 11+ Copy Section

• TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
• TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA
• TLS_RSA_WITH_AES_128_CBC_SHA

Android 9+ (Gingerbread) Copy Section

• TLS_DHE_RSA_WITH_AES_128_CBC_SHA
• TLS_DHE_DSS_WITH_AES_128_CBC_SHA
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA
• TLS_RSA_WITH_AES_128_CBC_SHA

Protocols Copy Chapter

Android API levels 9 through 15 support only TLSv1.0.
Modern versions of Android support all versions of TLS, from TLSv1.0
through TLSv1.2.

We intend to eliminate TLSv1.0 on suitable Android versions as soon as
possible.

No version of Firefox for Android beyond version 38 supports SSLv3 for
Sync.

Src How To How To Run Sync Server (`docs/src/how-to/how-to-run-sync-server.md`) Copy Section

Run your Own Sync-1.5 Server (legacy)

Note: this is for the legacy syncserver. This guide is left here for those still possibly
self-hosting and using this method.

Mozilla does not provide any pre-packaged release of the Firefox Sync server.

The easiest way to install a Sync Server is to checkout our repository
and run a build in-place. Once this is done, Sync can be run behind
any Web Server that supports the WSGI protocol.

Important Notes Copy Chapter

The sync service uses Mozilla accounts
for user authentication, which is a separate service and is not covered by this guide.

Note: By default, a server set up using this guide will defer authentication
to the Mozilla-hosted accounts server at https://accounts.firefox.com.

You can safely use the Mozilla-hosted Mozilla accounts server in combination
with a self-hosted sync storage server. The authentication and encryption
protocols are designed so that the account server does not know the user's
plaintext password, and therefore cannot access their stored sync data.

Alternatively, you can also refer to How To Run an FxA Server to control all aspects of the
system. The process for doing so is currently very experimental and not well
documented.

Prerequisites Copy Chapter

The various parts are using Python 2.7 and Virtualenv. Make sure your
system has them, or install them:

• Python 2.7 downloads: http://python.org/download/releases/2.7.6
• Virtualenv: http://pypi.python.org/pypi/virtualenv

To build and run the server, you will also need to have these packages
installed:

• python-dev
• make
• git
• c and c++ compiler

For example, under a fresh Ubuntu, you can run this command to meet all
requirements:

$ sudo apt-get install python-dev git-core python-virtualenv g++

Building the server Copy Chapter

Get the latest version at https://github.com/mozilla-services/syncserver and
run the build command:

$ git clone https://github.com/mozilla-services/syncserver
$ cd syncserver
$ make build

This command will create an isolated Python environment and pull all the
required dependencies in it. A local/bin directory is created and contains
a gunicorn command that can be used to run the server.

If you like, you can run the testsuite to make sure everything is working
properly:

$ make test

Basic Configuration Copy Chapter

The server is configured using an ini-like file to specify various runtime
settings. The file syncserver.ini will provide a useful starting point.

There is one setting that you must specify before running the server: the
client-visible URL for the service. Open ./syncserver.ini and locate the
following lines:

[syncserver]
public_url = http://localhost:5000/

The default value of public_url will work for testing purposes on your local
machine. For final deployment, change it to the external, publicly-visible URL
of your server.

By default the server will use an in-memory database for storage, meaning that
any sync data will be lost on server restart. You will almost certainly want
to configure a more permanent database, which can be done with the sqluri
setting:

[syncserver]
sqluri = sqlite:////path/to/database/file.db

This setting will accept any SQLAlchemy
database URI; for example the following would connect to a mysql server:

[syncserver]
sqluri = pymysql://username:[email protected]/sync

Running the Server Copy Chapter

Now you can run the server using gunicorn and the provided syncserver.ini
file. The simplest way is to use the Makefile like this:

$ make serve

Or if you'd like to pass additional arguments to gunicorn, like this:

$ local/bin/gunicorn --threads 4 --paste syncserver.ini

Once the server is launched, you need to tell Firefox about its location.

To configure desktop Firefox to talk to your new Sync server, go to
about:config, search for identity.sync.tokenserver.uri and change its value
to be the public URL of your server with a path of token/1.0/sync/1.5:

• identity.sync.tokenserver.uri: http://localhost:5000/token/1.0/sync/1.5

Alternatively, if you're running your own Mozilla accounts server, and running
Firefox 52 or later, see the documentation on how to refer to howto_run_fxa
for how to configure your client for both Sync and Mozilla accounts with a
single preference.

Firefox for Android ("Daylight", versions 79 and later) does support using a
non-Mozilla-hosted Sync server. Before logging in, go to App Menu > Settings

About Firefox and click the logo 5 times. You should see a "debug menu enabled"
notification. Go back to the main menu and you will see two options for a custom
account server and a custom Sync server. Set the Sync server to the URL given
above and then log in.

To configure Android Firefox 44 up to 78 to talk to your new Sync server, just set
the identity.sync.tokenserver.uri exactly as above before signing in to
Mozilla accounts and Sync on your Android device.

Important: after creating the Android account, changes to
identity.sync.tokenserver.uri will be ignored.

If you need to change the URI, delete the Android account using the
Settings > Sync > Disconnect... menu item, update the pref, and sign in again.

Non-default TokenServer URLs are displayed in the Settings > Sync panel in
Firefox for Android, so you should be able to verify your URL there.

Prior to Firefox 44, a custom add-on was needed to configure Firefox for Android.
For Firefox 43 and earlier, see the blog post:
How to connect Firefox for Android to self-hosted Mozilla account and Firefox Sync servers

(Prior to Firefox 42, the TokenServer preference name for Firefox Desktop was
services.sync.tokenServerURI. While the old preference name will work in
Firefox 42 and later, the new preference is recommended as the old preference
name will be reset when the user signs out from Sync causing potential confusion.)

Since Firefox 18, Firefox for iOS has support for custom sync servers. The settings
can be made in the Advanced Sync Settings in the Mozilla account section, which are
visible if you are not signed in with a Mozilla account and have enabled the debug mode
(tap 5 times on the version number). In order to use the custom sync server with Firefox 28,
the token server's url must not contain the path /1.0/sync/1.5. It is also important to
configure a custom account content server (you may use the default https://accounts.firefox.com).

Further Configuration Copy Chapter

Once the server is running and Firefox is syncing successfully, there are
further configuration options you can tweak in the syncserver.ini file.

The secret setting is used by the server to generate cryptographically-signed
authentication tokens. It is blank by default, which means the server will
randomly generate a new secret at startup. For long-lived server installations
this should be set to a persistent value, generated from a good source of
randomness. An easy way to generate such a value on posix-style systems is to do:

$ head -c 20 /dev/urandom | sha1sum
    db8a203aed5fe3e4594d4b75990acb76242efd35  -

Then copy-paste the value into the config file like so:

[syncserver]
...other settings...
secret = db8a203aed5fe3e4594d4b75990acb76242efd35

The identity_provider setting controls which server service can issue identity
assertions for access to the service. By default it will accept identity
assertions from the Mozilla-hosted account server at https://accounts.firefox.com.
If you are hosting your own instance of Mozilla accounts, you should change this
to your own domain:

[syncserver]
...other settings...
identity_provider = https://accounts.example.com

The allow_new_users setting controls whether the server will accept requests
from previously-unseen users. It is allowed by default, but once you have
configured Firefox and successfully synced with your user account, additional
users can be disabled by setting:

[syncserver]
...other settings...
allow_new_users = false

Updating the server Copy Chapter

You should periodically update your code to make sure you've got the latest fixes.
The following commands will update syncserver in place:

$ cd /path/to/syncserver
$ git stash       # to save any local changes to the config file
$ git pull        # to fetch latest updates from github
$ git stash pop   # to re-apply any local changes to the config file
$ make build      # to pull in any updated dependencies

Running behind a Web Server Copy Chapter

The built-in server should not be used in production, as it does not really
support a lot of load.

If you want to set up a production server, you can use different web servers
that are compatible with the WSGI protocol. For example:

• Apache combined with mod_wsgi
• NGinx with Gunicorn or uWSGI

Note: Remember, you must set the syncserver.public_url option to the
client-visible URL of your server.

For example, if your server will be located at http://example.com/ff-sync/,
the public_url should be set to this value in your config file:

[syncserver]
public_url = http://example.com/ff-sync/

Apache + mod_wsgi Copy Section

Here's an example of an Apache 2.2 setup that uses mod_wsgi:

<Directory /path/to/syncserver>
    Order deny,allow
    Allow from all
</Directory>

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /path/to/syncserver
    WSGIProcessGroup sync
    WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25 python-path=/path/to/syncserver/local/lib/python2.7/site-packages/
    WSGIPassAuthorization On
    WSGIScriptAlias / /path/to/syncserver/syncserver.wsgi
    CustomLog /var/log/apache2/example.com-access.log combined
    ErrorLog  /var/log/apache2/example.com-error.log
</VirtualHost>

Here's the equivalent setup for Apache 2.4, which uses a different syntax for access control:

<Directory /path/to/syncserver>
    Require all granted
</Directory>

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /path/to/syncserver
    WSGIProcessGroup sync
    WSGIDaemonProcess sync user=sync group=sync processes=2 threads=25 python-path=/path/to/syncserver/local/lib/python2.7/site-packages/
    WSGIPassAuthorization On
    WSGIScriptAlias / /path/to/syncserver/syncserver.wsgi
    CustomLog /var/log/apache2/example.com-access.log combined
    ErrorLog  /var/log/apache2/example.com-error.log
</VirtualHost>

We provide a syncserver.wsgi file for your convenience in the repository.
Before running Apache, edit the file and check that it loads the right .ini
file with its full path.

Some users have reported issues with outbound TLS connections when running under
Apache. If your server is giving 503 errors and the Apache error log mentions
SysCallError, you may be able to correct the problem by installing the pyopenssl library:

$ local/bin/pip install pyopenssl

Nginx + Gunicorn Copy Section

Tested with debian stable/squeeze

1. First install gunicorn in the syncserver python environment:

$ cd /usr/src/syncserver
$ local/bin/pip install gunicorn

2. Then enable gunicorn in the syncserver.ini file:

[server:main]
use = egg:gunicorn
host = 127.0.0.1
port = 5000
workers = 2
timeout = 60

3. Finally edit your nginx vhost file:

server {
        listen  443 ssl;
        server_name sync.example.com;

        ssl_certificate /path/to/your.crt;
        ssl_certificate_key /path/to/your.key;

        location / {
                proxy_set_header Host $http_host;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_redirect off;
                proxy_read_timeout 120;
                proxy_connect_timeout 10;
                proxy_pass http://127.0.0.1:5000/;
                }
        }

5. After restarting your nginx and syncserver you should be able to use the
   sync server behind your nginx installation

Note: If you see errors about a mismatch between public_url and
application_url, you may need to tell gunicorn that it should trust the
X-Forwarded-Proto header being sent by nginx. Add the following to the
gunicorn configuration in syncserver.ini:

forwarded_allow_ips = *

Note: If you see errors about "client sent too long header line" in your
nginx logs, you may need to configure nginx to allow large client header
buffers by adding this to the nginx config:

large_client_header_buffers 4 8k;

Nginx + uWSGI Copy Section

1. Install uWSGI and its Python 2 plugin
2. Start it with the following options:

uwsgi --plugins python27 --manage-script-name \
        --mount /<location>=/path/to/syncserver/syncserver.wsgi \
        --socket /path/to/uwsgi.sock

3. Use the following nginx configuration:

location /<location>/ {
    include uwsgi_params;
    uwsgi_pass unix:/path/to/uwsgi.sock;
}

Things that still need to be Documented Copy Chapter

• periodic pruning of expired sync data

Asking for help Copy Chapter

Don't hesitate to jump online and ask us for help:

• on Element (https://chat.mozilla.org) in the
  #sync channel

• File an issue: Syncstorage-rs GitHub Issues.

Src How To How To Run With Docker (`docs/src/how-to/how-to-run-with-docker.md`) Copy Section

Use Docker to Deploy Your Own Sync Server

Mozilla publishes Docker images of its
syncstorage-rs builds
on ghcr.io. This guide provides a simple docker compose setup that can act as
a starting point to self-host Sync.

Images are available for both
MySQL
and
PostgreSQL
as the database. Differences in configuration or deployment steps will be
noted.

Tagged release builds are available on ghcr.io. To pin to a specific version,
set SYNCSERVER_VERSION to the desired release tag (e.g., SYNCSERVER_VERSION=v1.45.0)
before running docker compose. Available releases can be found on the
syncstorage-rs releases page.
If SYNCSERVER_VERSION is not set, the compose files below default to latest.

Prerequisites and Presumptions Copy Chapter

• The reader is familiar with the command line interface and docker.
• The reader is going to use Mozilla accounts
  for authentication and authorization.
• The service will be deployed at http://localhost:8000/.

Docker Compose, Sync Services Only Copy Chapter

With a MySQL or PostgreSQL database is already up and running, save the yaml
below into a file, e.g. docker-compose.yaml, and ensure the image field is
using the correct MySQL or PostgreSQL build for the database.

services:
  syncserver:
    image: ghcr.io/mozilla-services/syncstorage-rs/syncserver-mysql:${SYNCSERVER_VERSION:-latest}
    platform: linux/amd64
    container_name: syncserver
    ports:
      - "${SYNC_PORT:-8000}:${SYNC_PORT:-8000}"
    environment:
      SYNC_HOST: "0.0.0.0"
      SYNC_PORT: "${SYNC_PORT:-8000}"
      SYNC_MASTER_SECRET: "${SYNC_MASTER_SECRET}"
      SYNC_SYNCSTORAGE__DATABASE_URL: "${SYNC_SYNCSTORAGE__DATABASE_URL}"
      SYNC_TOKENSERVER__DATABASE_URL: "${SYNC_TOKENSERVER__DATABASE_URL}"
      SYNC_TOKENSERVER__ENABLED: "true"
      SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
      SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: "api.accounts.firefox.com"
      SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: "https://oauth.accounts.firefox.com"
      SYNC_TOKENSERVER__INIT_NODE_URL: "${SYNC_TOKENSERVER__INIT_NODE_URL:-http://localhost:${SYNC_PORT:-8000}}"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:${SYNC_PORT:-8000}/__heartbeat__"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

Note that multiple values will be read from the environment:

• SYNC_MASTER_SECRET: a secret used in cryptographic operations, a passphrase or random character string, e.g. use_your_own_secret_4d3d3d3d
• SYNC_SYNCSTORAGE__DATABASE_URL: database URL for syncstorage, e.g. mysql://sync:[email protected]/syncstorage or postgres://testo:@localhost/syncdb
• SYNC_TOKENSERVER__DATABASE_URL: database URL for tokenserver, e.g. mysql://sync:[email protected]/tokenserver or postgres://testo:@localhost/syncdb
• SYNC_TOKENSERVER__INIT_NODE_URL: the storage node URL (defaults to http://localhost:8000). Replace with the actual URL where clients will access the sync server. Syncserver initializes the Tokenserver nodes table by inserting this URL on startup (if it doesn't already exist).

The values can be directly written into the yaml as well.

Next, start the service with docker compose:

SYNC_MASTER_SECRET=use_your_own_secret_4d3d3d3d \
SYNC_SYNCSTORAGE__DATABASE_URL="mysql://sync:[email protected]/syncstorage" \
SYNC_TOKENSERVER__DATABASE_URL="mysql://sync:[email protected]/tokenserver" \
SYNC_TOKENSERVER__INIT_NODE_URL="http://localhost:8000" \
docker compose -f docker-compose.yaml up -d

Docker Compose, One-Shot Stand-Alone Servers Copy Chapter

The repository ships ready-to-run, stand-alone compose files under docker/
that bring up a complete server — database(s) included — in a single command,
with no manual database setup. Each builds the server from your local checkout,
so they work directly from a clone without a published image. From the repo
root:

For example, for MySQL:

make docker_oneshot_mysql
# equivalently:
docker compose -f docker/docker-compose.one-shot.mysql.yaml up -d --build

Once the syncserver container reports healthy, confirm it is serving:

curl http://localhost:8000/__heartbeat__

Stop and remove a stack with the matching _stop target, e.g.
make docker_oneshot_mysql_stop.

Syncstorage applies its schema migrations at startup,
SYNC_TOKENSERVER__RUN_MIGRATIONS applies the Tokenserver schema, and
SYNC_TOKENSERVER__INIT_NODE_URL bootstraps the sync-1.5 service and storage
node records — so the stack is ready to serve immediately.

Set SYNC_MASTER_SECRET to your own value for anything beyond local
experimentation; the compose files default to a placeholder.

Backend notes Copy Section

• MySQL / PostgreSQL are reasonable starting points for a real self-hosted
  deployment. The MySQL recipe uses a separate database for Syncstorage and
  Tokenserver; the PostgreSQL recipe shares a single database between them for
  simplicity.
• Spanner runs against the Cloud Spanner emulator and is for local
  experimentation only — it is unauthenticated, single-node, and not durable.
  Production Spanner uses a real instance and a service-account key (see
  make run_spanner). The recipe mirrors the production split (Spanner for
  Syncstorage, MySQL for Tokenserver) and includes a one-time setup container
  that provisions the emulator's schema before the server starts.

Using a published image instead of building Copy Section

Mozilla also publishes prebuilt images on ghcr.io. Release builds are tagged
with the release version and latest; builds from master are tagged with the
short commit SHA only. Pin SYNCSERVER_VERSION to a tag listed on the
syncserver-mysql packages page.
To use one, replace the syncserver service's build: block in the compose
file with an image: reference (published images are linux/amd64):

services:
  syncserver:
    image: ghcr.io/mozilla-services/syncstorage-rs/syncserver-mysql:${SYNCSERVER_VERSION:?set SYNCSERVER_VERSION to a published tag}
    platform: linux/amd64
    # ...the remaining syncserver settings are unchanged

SYNCSERVER_VERSION=<published-tag> docker compose -f docker/docker-compose.one-shot.mysql.yaml up -d

Configuring Firefox (Desktop) Copy Chapter

Firefox itself needs to be configured to use the self-hosted Sync server.

1. Go to about:config in Firefox.
2. Find the identity.sync.tokenserver.uri configuration.
3. Change the value to http://localhost:8000/1.0/sync/1.5.
4. Restart Firefox.

Firefox should be using the self-hosted Sync server at this point. That can be
verified by checking the logs in about:sync-log.

Configuring Firefox (Mobile) Copy Chapter

Firefox itself needs to be configured to use the self-hosted Sync server.

1. Go to Settings -> About Firefox
2. Repeadetly press the Firefox logo (six times) to activate the debug menu
3. Go back to the main Setting menu.
4. Click on the "Sync Debug" menu
5. Click on "custom sync server" and change the value to http://localhost:8000/1.0/sync/1.5.
6. After changing the "custom sync server" click on "Stop Firefox" in the same menu so the changes can be applied.

Src Sync Client Fx Object Formats (`docs/src/sync-client/fx-object-formats.md`) Copy Section

Firefox object formats

Decrypted data objects are cleartext JSON strings.

Each collection can have its own object structure. This document describes the
format of each collection.

The object structure is versioned with the version metadata stored in the
meta/global payload.

The following sections, named by the corresponding collection name, describe
the various object formats and how they’re used. Note that object structures
may change in the future and may not be backwards compatible.

In addition to these custom collection object structures, the Encrypted
DataObject adds fields like id and deleted. Also remember that there is
data at the Weave Basic Object (WBO) level as well as id, modified,
sortindex and payload.

Add-ons Copy Chapter

Version 1 Copy Section

Version 1 is likely only affiliated with storage format 5 clients.

• addonID (string): Public identifier of add-on. This is the id
  attribute from an Addon object obtained from the AddonManager.
• applicationID (string): The application ID the add-on belongs to.
• enabled (bool): Indicates whether the add-on is enabled or disabled.
  true means enabled.
• source (string): Where the add-on came from. amo means it came from
  addons.mozilla.org or a trusted site.

Bookmarks Copy Chapter

Version 1 Copy Section

One bookmark record exists for each bookmark item, where an item may actually
be a folder or a separator. Each item will have a type that determines what
other fields are available in the object. The following sections describe the
object format for a given type.

Each bookmark item has a parentid and predecessorid to form a structure
like a tree of linked-lists to provide a hierarchical ordered list of
bookmarks, folders, etc.

bookmark

This describes a regular bookmark that users can click to view a page.

• title (string): name of the bookmark
• bmkUri (string): uri of the page to load
• description (string): extra description if provided
• loadInSidebar (boolean): true if the bookmark should load in the sidebar
• tags (array of strings): tags for the bookmark
• keyword (string): alias to activate the bookmark from the location bar
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "bookmark"

microsummary

Microsummaries allow pages to be summarized for viewing from the toolbar. This
extends bookmark, so the usual bookmark fields apply.

Reference: https://developer.mozilla.org/en/Microsummary_topics

• generatorUri (string): uri that generates the summary
• staticTitle (string): title to show when no summaries are available
• title (string): name of the microsummary
• bmkUri (string): uri of the page to load
• description (string): extra description if provided
• loadInSidebar (boolean): true if the bookmark should load in the sidebar
• tags (array of strings): tags for the bookmark
• keyword (string): alias to activate the bookmark from the location bar
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "microsummary"

query

Place queries are special bookmarks with a place: uri that links to an
existing folder/tag. This extends bookmark, so the usual bookmark fields
apply.

• folderName (string): name of the folder/tag to link to
• queryId (string, optional): identifier of the smart bookmark query
• title (string): name of the query
• bmkUri (string): place: uri query
• description (string): extra description if provided
• loadInSidebar (boolean): true if the bookmark should load in the sidebar
• tags (array of strings): tags for the query
• keyword (string): alias to activate the bookmark from the location bar
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "query"

folder

Folders contain bookmark items like bookmarks and other folders.

• title (string): name of the folder
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "folder"

livemark

Livemarks act like folders with a dynamic list of bookmarks, e.g. an RSS feed.
This extends folder, so the usual folder fields apply.

Reference: https://developer.mozilla.org/en/Using_the_Places_livemark_service

• siteUri (string): site associated with the livemark
• feedUri (string): feed to get items for the livemark
• title (string): name of the livemark
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "livemark"

separator

Separators help split sections of a folder.

• pos (string): position (index) of the separator
• parentid (string): GUID of the containing folder
• parentName (string): name of the containing folder
• predecessorid (string): GUID of the item before this (empty if it’s first)
• type (string): "separator"

Version 2 Copy Section

Same as engine version 1, except:

• the predecessorid is removed from all records;
• instead folder and livemark records have a children attribute which is an
  array of child GUIDs in order of their appearance in the folder:
  • children (array of strings): ordered list of child GUIDs
• the special folders menu and toolbar now have records that are synced,
  purely to maintain order within them according to their children array.
• dateAdded (unix timestamp): The best lower bound on the creation date
  for this record we have. May be missing, in the case of records uploaded by
  older clients when no newer client is available to fix it up.

Version 3 Copy Section

Note: Proposal corresponding with storage format 6.

Same as version 2 except:

• Support for microsummaries is removed
• We use the ASCII URL

TODO: document full format here since diffs are inconvenient to read.

Clients Copy Chapter

Version 1 Copy Section

Client records identify a user’s one or multiple clients that are accessing the
data. The existence of client records can change the behavior of the Firefox
Sync client — multiple clients and/or mobile clients result in syncs to happen
more frequently.

• name (string): name of the client connecting
• type (string): type of the client: "desktop" or "mobile"
• commands (array): commands to be executed upon next sync — see below for more

In Protocol 1.5, client records additionally include:

• version (string): a version indicator for this client, such as "29.0a1". Optional.
• protocols (array): an array of Sync protocol versions supported by this client, such as ["1.1", "1.5"]. Optional.

In Bug 1097222 additional optional fields were added:

• os (string): an OS name, most likely one of "Darwin" (Mac OS X),
  "WINNT" (Windows), "Android", or "iOS".
• appPackage (string): an unambiguous identifier for the client application.
  For Android, this is the package (e.g., org.mozilla.firefox_beta). For desktop
  this is the value of Services.appinfo.ID.
• application (string): a human-readable application name, such as "Nightly"
  or "Firefox".
• formfactor (string): a value such as "phone", "tablet" (or the more
  specific "largetablet", "smalltablet"), "desktop", "laptop", "tv".
• device (string): a description of the hardware that this client uses.
  Currently only supported by Android; returns values like "HTC One".

If these fields are missing, clients are expected to fall back to behaviors
that do not depend on the missing data.

Clients should preserve existing fields if possible when sending commands to
another client.

commands

commands is an array of JSON objects. Each element has the following attributes:

• command (string): The name of the command to execute. Currently
  supported commands include "resetAll", "resetEngine", "wipeAll",
  "wipeEngine", "logout", "displayURI", "repairRequest" and
  "repairResponse", although not all commands are supported by all implementations.
• args (array of strings/objects): Arguments for the command. These
  are specific to the command.
• flowIID (optional, string): A guid used for reporting telemetry. Both
  the sender and receiver of the command should report this ID in telemetry so
  the reliability of the sending and reception of the command can be tracked.

Version 2 (never deployed) Copy Section

Note: Proposal corresponding with storage format 6.

Each client has its own record which it is authoritative for. No other client
should modify another client’s record except in the case where records are
deleted.

The payload of a client record has the following fields:

• name (string): The name of the client. This is a user-facing value and
  may be provided by the user.
• formfactor (string): The form factor of the client. Recognized values
  include phone, tablet, laptop, desktop.
• application (string): String identifying the application behind the
  client. This should only be used for presentation purposes (e.g. choosing what
  logo to display).
• version (string): The version of the client. This is typically the
  version of the application. Again, this should only be used for presentation
  purposes.
• capabilities (object): Denotes the capabilities a client possesses.
  Keys are string capability names. Values are booleans indicating whether the
  capability is enabled. Modifying the capabilities of another client’s record
  should not change the enabled state on that client.
• mpEnabled (bool): Whether master password is enabled on the client. If
  master password is enabled on any client in an account, the current client
  should hesitate before downloading passwords if master password is not
  enabled locally, as this would decrease the security of the passwords locally
  since they wouldn’t be protected with a master password.

Commands Copy Chapter

Version 1 Copy Section

Note: Proposal corresponding with storage format 6.

This collection holds commands for clients to process. The ID of command records
is randomly generated.

Command records contain an extra unencrypted field in the BSO that says which
client ID they belong to. The value is the hash of the client ID with the
commands engine salt.

Command data is an object with the following fields:

• receiverID (string): Client ID of the client that should receive the
  command. This is duplicated inside the payload so it can be verified by the
  HMAC.
• senderID (string): Client ID of the client that sent the command.
• created (number): Integer seconds since Unix epoch that command was
  created.
• action (string): The action to be performed by the command. Each command
  has its own name that uniquely identifies it. It is recommended that actions
  be namespaced using colon-delimited notation. Sync’s commands are all prefixed
  with sync: (e.g. sync:wipe). If a command is versioned, the name is the
  appropriate place to convey that versioning.
• data (object): Additional data associated with command. This is
  dependent on the specific command type being issued.

Forms Copy Chapter

Form data is used to give suggestions for autocomplete for a HTML text input
form. One record is created for each form entry.

• name (string): name of the HTML input field
• value (string): value to suggest for the input

History Copy Chapter

Version 1 Copy Section

Every page a user visits generates a history item/page. One history (page) per record.

• histUri (string): uri of the page
• title (string): title of the page
• visits (array of objects): a number of how and when the page was visited
• date (integer): datetime of the visit
• type (integer): transition type of the visit

Reference: https://developer.mozilla.org/en/nsINavHistoryService#Constants

Version 2 (never deployed) Copy Section

Note: Proposal corresponding with storage format 6.

History visits are now stored as a timeline/stream of visits. The historical
information for a particular site/URL is spread out over N>=1 records.

Payloads have the structure:

{
      "items": [
        "uri": "http://www.mozilla.org/",
        "title": "Mozilla",
        "visits": {
          1: [1340757179.82, 184],
          2: [1340341244.31, 12, 4]
        }
      ]
    }

The bulk of the payload is a list of history items. Each item is both a place
and a set of visits.

• uri (string): URI of the page that was visited.
• title (string): Title of the page that was visited.
• visits (object): Mapping of visit type to visit times.

The keys in visits define the transition type for the visit. They can be:

• 1: A link was followed.
• 2: The URL was typed by the user.
• 3: The user followed a bookmark.
• 4: Some inner content was loaded.
• 5: A permanent redirect was followed.
• 6: A temporary redirect was followed.
• 7: The URL was downloaded.
• 8: User follows a link that was in a frame.

These correspond to nsINavHistoryService’s transition type constants:
https://developer.mozilla.org/en/nsINavHistoryService#Constants

The values for each visit type are arrays which encode the visit time. The
initial element is the wall time of the first visit in seconds since epoch,
typically with millisecond resolution. Each subsequent value is the number of
seconds elapsed since the previous visit. The values: [100000000.000, 10.100, 5.200]

Correspond to the times:

100000000.000
    100000010.100
    100000015.300

The use of deltas to represent times is to minimize serialized size of visits.

Passwords Copy Chapter

Saved passwords help users get back into websites that require a login such as
HTML input/password fields or HTTP auth.

• hostname (string): hostname that password is applicable at
• formSubmitURL (string): submission url (GET/POST url set by <form>)
• httpRealm (string): the HTTP Realm for which the login is valid; if not
  provided by the server, the value is the same as hostname
• username (string): username to log in as
• password (string): password for the username
• usernameField (string): HTML field name of the username
• passwordField (string): HTML field name of the password

If possible, clients should also write fields corresponding to nsILoginMetaInfo:

• timeLastUsed (unsigned long): local Unix timestamp in milliseconds at which
  this password was last used.
  Note that client clocks can be wrong, and thus this time can be dramatically
  earlier or later than the modified time of the record. Consuming clients should
  be careful to handle out of range values.
• timeCreated (unsigned long): as with timeLastUsed, but for creation.
• timePasswordChanged (unsigned long): as with timeLastUsed, but for password change.
• timesUsed (unsigned long): the number of uses of this password.

These fields are optional; clients should expect them to be missing. Clients that
don’t use this data locally are encouraged to pass through when it makes sense
(timeCreated), or wipe when invalidation is the best option (e.g.,
timePasswordChanged).

Clients should use judgment when updating these fields; it’s typically not feasible
to upload new records each time a password is used. During download, a non-matching
timestamp (or missing field) in an otherwise matching local record should not
automatically be treated as a collision. Handling these fields introduces additional
complexities in reconciliation.

The Firefox desktop client began recording this data in Bug 555755.

Preferences Copy Chapter

Version 1 Copy Section

Some preferences used by Firefox will be synced to other clients. There is only
one record for preferences with a GUID "preferences".

• value (array of objects): each object describes a preference entry
• name (string): full name of the preference
• type (string): type of preference (int, string, boolean)
• value (depends on type): value of the preference

Version 2 Copy Section

There is only one record for preferences, using nsIXULAppInfo.ID as the GUID.
Custom preferences can be synced by following these instructions:
https://developer.mozilla.org/en/Firefox_Sync/Syncing_custom_preferences

• value (object): containing name and value of the preferences.

Note: The preferences that determine which preferences are synced are now included as well.

Tabs Copy Chapter

Version 1 Copy Section

Tabs describe the opened tabs on a given client to provide functionality like
get-up-n-go. Each client will provide one record.

• clientName (string): name of the client providing these tabs
• tabs (array of objects): each object describes a tab
• title (string): title of the current page
• urlHistory (array of strings): page urls in the tab’s history
• icon (string): favicon uri of the tab
• lastUsed (integer): Time in seconds since Unix epoch at which the tab was last accessed.
  Preferred format is an integer, but older clients may write floats or stringified floats,
  so clients should be prepared to receive those formats too.

Version 2 Copy Section

Note: Proposal corresponding with storage format 6.

In version 2, each tab is represented by its own record (a change from version 1).

Payload fields:

• clientID (string): ID of the client this tab originated on.
• title (string): Title of page that is active in the tab.
• history (array of strings): URLs in this tab’s history. Initial element is the current URL.
  Subsequent URLs were previously visited.
• lastUsed (number): Time in seconds since Unix epoch that tab was last active.
• icon (string): Base64 encoded favicon image.
• groupName (string): Name of tab group this tab is associated with; usually for presentation
  and typically the same across records in a given tab group.

Src Sync Client Global Storage V5 (`docs/src/sync-client/global-storage-v5.md`) Copy Section

Global Storage Version 5

This document describes version 5 of Sync's global storage format. It describes
not only the technical details of the storage format, but also the semantics for
how clients supporting version 5 should interact with the Sync server.

Overview Copy Chapter

A single unencrypted record called the metaglobal record (because it exists
in the meta collection with the id global) stores essential data used to
instruct clients how to behave.

A special record called the cryptokeys record (because it exists in the
crypto collection with the id keys) holds encrypted keys which are used to
encrypt, decrypt, and verify all other encrypted records on the server.

Cryptography Copy Chapter

Overview Copy Section

Every encrypted record (and all but one record on the server is encrypted) is
encrypted using symmetric key encryption and verified using HMAC hashing.
The symmetric encryption and HMAC verification keys are only available to
client machines; they are not transmitted to the server in any readable form.
This means that the data on the server cannot be read by anyone with access
to the server.

The symmetric encryption key and HMAC key together form a key bundle.
Each key is 256 bits.

Individual records are encrypted with AES-256. The encryption key from a key
bundle is combined with a per-record 16-byte IV and user data is converted into
ciphertext. The ciphertext is then signed with the key bundle’s HMAC key.
The ciphertext, IV, and HMAC value are uploaded to the server.

When Sync is initially configured by signing in with a Mozilla account, the
client obtains a 256-bit encryption key called the Class-B Master Key.
This key is used to derive a special key bundle via HKDF, called the
Sync Key Bundle. The Sync Key Bundle is used to encrypt and decrypt a special
record on the server which holds additional key bundles. These bundled keys are
used to encrypt and decrypt all other records on the server.

Terminology Copy Section

Class-B Master Key
256-bit encryption key obtained from Mozilla accounts, serving as the root key
for Sync.

Key Bundle
A pair of 256-bit keys: one for symmetric encryption and one for HMAC hashing.

Sync Key Bundle
A Key Bundle derived from the Class-B Master Key via HKDF.

HKDF
Cryptographic technique used to derive keys from another key.

Bulk Key Bundle
A collection of Key Bundles used to secure records, encrypted with the Sync Key
Bundle.

Cleartext
The unencrypted form of user data.

Ciphertext
The encrypted form of cleartext.

Encryption Key
The key used to convert cleartext into ciphertext.

HMAC Key
The key used to verify message integrity.

Symmetric Encryption
Encryption and decryption using the same secret key.

HMAC Hashing
A method to verify that ciphertext has not been tampered with.

Class-B Master Key Copy Chapter

All encryption keys used in Sync are ultimately derived from the
Class-B Master Key, which is managed by Mozilla accounts and obtained through
the Accounts/Sync sign-in protocol (referred to as kB).

All clients collaborating via Sync share the same value for this key.
It must never be transmitted to untrusted parties or stored where it can
be accessed by others, including the storage server.

Sync Key Bundle Copy Chapter

The Sync Key Bundle is derived from the Class-B Master Key using SHA-256
HMAC-based HKDF (RFC 5869).

A total of 64 bytes are derived. The first 32 bytes form the encryption key,
and the remaining 32 bytes form the HMAC key.

Pseudo-code:

info = "identity.mozilla.com/picl/v1/oldsync"
prk = HKDF-Extract-SHA256(0x00 * 32, master_key)
okm = HKDF-Expand-SHA256(prk, info, 64)

encryption_key = okm[0:32]
hmac_key = okm[32:64]

Record Encryption Copy Chapter

Each record is encrypted using AES-256 in CBC mode and signed using HMAC-SHA256.

Pseudo-code:

cleartext = "SECRET MESSAGE"
iv = randomBytes(16)
ciphertext = AES256(cleartext, bundle.encryption_key, iv)
hmac = HMACSHA256(bundle.hmac_key, base64(ciphertext))

The ciphertext, IV, and HMAC are stored in the record payload.

Record Decryption Copy Chapter

When retrieving a record, the client verifies the HMAC before attempting
decryption. If verification fails, the record must not be decrypted.

Pseudo-code:

local_hmac = HMACSHA256(hmac_key, base64(ciphertext))
if local_hmac != record_hmac:
    error

cleartext = AESDecrypt(ciphertext, encryption_key, iv)

Metaglobal Record Copy Chapter

The meta/global record contains metadata describing server state, including
storage version and enabled engines. It is not encrypted.

Fields include:

• storageVersion
• syncID
• engines
• declined (Protocol 1.5)

Example:

{
    "syncID": "7vO3Zcdu6V4I",
    "storageVersion": 5,
    "engines": {
    "clients":   {"version":1,"syncID":"Re1DKzUQE2jt"},
    "bookmarks": {"version":2,"syncID":"ApPN6v8VY42s"}
    },
    "declined": ["passwords"]
}

Clients must verify storage version compatibility before modifying data.

crypto/keys Record Copy Chapter

In version 5, all bulk keys are stored in the crypto/keys record.
It is encrypted using the Sync Key Bundle.

Fields:

• default: default key pair
• collections: per-collection key pairs
• collection: always "crypto"

Each key is Base64-encoded.

Collection Records Copy Chapter

All non-special records store encrypted payloads with:

• ciphertext
• IV
• hmac

Example:

{
    "payload": "{\"ciphertext\":\"...\",\"IV\":\"...\",\"hmac\":\"...\"}",
    "id": "GJN0ojnlXXhU",
    "modified": 1332402035.78
}

Encryption Example Copy Chapter

Given cleartext:

{
    "foo": "supersecret",
    "bar": "anothersecret"
}

Pseudo-code:

key_pair = bulk_key_bundle.getKeyPair(collection_name)
iv = randomBytes(16)
ciphertext = AES256(cleartext, key_pair.encryption_key, iv)
hmac = HMACSHA256(base64(ciphertext), key_pair.hmac_key)

payload = {
    "ciphertext": base64(ciphertext),
    "IV": base64(iv),
    "hmac": base64(hmac)
}

Decryption Example Copy Chapter

Pseudo-code:

fields = JSONDecode(record.payload)
ciphertext_b64 = fields.ciphertext

local_hmac = HMACSHA256(ciphertext_b64, hmac_key)
if local_hmac != remote_hmac:
    error

cleartext = AESDecrypt(Base64Decode(ciphertext_b64), encryption_key, iv)
object = JSONDecode(cleartext)

Src Sync Client Life Of A Sync (`docs/src/sync-client/life-of-a-sync.md`) Copy Section

The Life of a Sync

This document essentially describes how to write a Sync client.

Because the Sync server is essentially a dumb storage bucket, most of the
complexity of Sync is the responsibility of the client. This is good for users'
data security. It is bad for people implementing Sync clients. This document
will hopefully alleviate common issues and answer common questions.

Strictly speaking, information in this document applies only to a specific
version of the Sync server storage format. In practice, client behavior is
similar across storage versions. And, since we wish for clients to support
the latest/greatest versions of everything, this document will target that.

Initial Client Configuration Copy Chapter

The process of performing a sync starts with configuring a fresh client. Before
you can even think about performing a sync, the client needs to possess key
pieces of information. These include:

• The URL of the Sync server.
• Credentials used to access the Sync server.

Depending on the versions of the Sync server and global storage version, you
may also need a Sync Key or similar private key which is used to access
encrypted data on an existing account.

Obtaining these pieces of information is highly dependent on the server
instance you will be communicating with, the client in use, and whether you are
creating a new account or joining an existing one.

How Mozilla and Firefox Does It Copy Chapter

For reference, this section describes how Mozilla and Firefox handle initial
client configuration.

Inside Firefox there exists a UI to Set up Firefox Sync. The user chooses
whether she is setting up a new account or whether she wants to connect to an
existing account.

For completely new accounts, the user is presented with a standard sign-up
form. The user enters her email address and selects a password. Behind the
scenes Firefox is talking to a user provisioning service and the account is
created there and a Sync server is assigned (Mozilla exposes many different
Sync server instances to the Internet and the client connects directly to just
one of them). At this time, a new Sync Key encryption key is generated and
stored in Firefox's credential manager (possibly protected behind a master
password).

If the user selects an existing account, the user is presented 12 random
characters. These are entered on another device and the two devices
effectively pair and share the login credentials, Sync Key, and server info.
This is done with J-PAKE, so the data is secure as it is transported between
devices. Even the intermediary agent bridging the connection between the two
devices can't decrypt the data inside.

Performing a Sync Copy Chapter

Settings and State Pre-check Copy Section

To perform a sync, a client will first need to perform some basic checks:

• Do we have all necessary credentials?
  • Storage server HTTP credentials
  • Sync Key
• Are we online (do we have network connectivity)
• Are we prohibited from syncing due to result from a previous sync?
  • The server may have issued a backoff telling us to slow down, etc

If these are all satisfied, the client can move on to the next phase.

Inspect and Reconcile Client and Server State Copy Section

The initial requests performed on the Sync server serve to inspect, verify,
and reconcile high-level state between the client and server.

Fetch info/collections

The first request to the Sync server should be a GET on the info/collections
URI. This reveals which collections exist on the server and when they were
last modified.

If the client has synced before, it should issue a conditional HTTP request
by adding an X-If-Modified-Since header. If the server responds with 304,
no modifications have been made since the last sync. If the client has no new
data to upload, it can stop immediately.

This request also verifies credentials. A 401 or 404 response should be
interpreted as credential failure, possibly requiring reauthentication.

Flow overview (graphviz source, informational only):

flowchart TD
  PREPARE_REQUEST --> HAVE_SYNCED_BEFORE
  HAVE_SYNCED_BEFORE -->|Yes| ADD_IMS
  HAVE_SYNCED_BEFORE -->|No| PERFORM_REQUEST
  ADD_IMS --> PERFORM_REQUEST
  PERFORM_REQUEST --> CHECK_RESPONSE
  CHECK_RESPONSE -->|304| HAVE_OUTGOING
  CHECK_RESPONSE -->|401, 403| REAUTHENTICATE
  HAVE_OUTGOING -->|No| END_SYNC
  HAVE_OUTGOING -->|Yes| NEXT_STEP

Validate meta/global

The client must validate the meta/global record on every sync.

Possible outcomes:

1. The meta collection does not exist.
2. The meta collection has been modified since the last sync.
3. The meta collection has not been modified since the last sync.

If meta does not exist and any collection exists, the client should delete
all server data to ensure a fresh state. If no collections exist, nothing needs
to be deleted.

If meta has not changed and a cached copy exists, no action is required.

Otherwise, fetch meta/global, decode the payload, and inspect the storage
version. If newer than supported, the client should stop and require upgrade.
Clients must never modify data belonging to a newer storage version.

Flow overview (graphviz source, incomplete):

flowchart TD
  CHECK_INFO_COLLECTIONS --> CHECK_ANY_COLLECTIONS
  CHECK_ANY_COLLECTIONS -->|Yes| DELETE_ALL
  CHECK_ANY_COLLECTIONS -->|No| FRESH_START
  DELETE_ALL --> CHECK_DELETE_ALL_RESPONSE
  CHECK_DELETE_ALL_RESPONSE -->|204| FRESH_START
  CHECK_DELETE_ALL_RESPONSE -->|401, 403| START_NEW_SYNC

Validate crypto/keys

The client must ensure valid cryptographic keys exist.

Flow overview (graphviz source):

flowchart TD
  HAVE_KEYS -->|No| CRYPTO_COLLECTION_EXISTS
  CRYPTO_COLLECTION_EXISTS -->|No| GENERATE_KEYS
  CRYPTO_COLLECTION_EXISTS -->|Yes| FETCH_KEYS
  FETCH_KEYS --> VALIDATE_KEYS
  VALIDATE_KEYS -->|OK| NEXT_STEP
  VALIDATE_KEYS -->|Not OK| GENERATE_KEYS
  GENERATE_KEYS --> UPLOAD_KEYS
  UPLOAD_KEYS --> NEXT_STEP

Collections Pre-Sync Copy Section

Once meta/global and cryptographic keys are validated, the client proceeds to
sync regular collections.

The client records last-modified timestamps from info/collections and
requests only records changed since the last sync.

Clients Collection Copy Section

The clients collection is special. Clients always fetch all records from it.
It is used for inter-client commands such as data wipes, which must be processed
before syncing other collections.

Incomplete Content Copy Chapter

The notes below are legacy and require cleanup.

Perform sync (pseudo-code) Copy Section

// - update engine last modified timestamps from info/collections record
// - sync clients engine
//   - clients engine always fetches all records
// - process reset/wipe requests in 'firstSync' preference
// - process any commands, including the 'wipeClient' command
// - infer enabled engines from meta/global
// - sync engines
//   - only stop if 401 is encountered
// - if meta/global has changed, reupload it

Syncing an engine (pseudo-code) Copy Section

// - meta/global
//   - syncID
//   - engine storage format
// - fetch incoming records
     - GET .../storage/<collection>?newer=<last_sync_server_timestamp>&full=1
     - optional but recommended for streaming: Accept: application/newlines
     - deserialize and apply each record:
       - JSON parse WBO
       - JSON parse payload
       - verify HMAC
       - decrypt ciphertext witH IV
       - JSON parse cleartext
       - apply to local storage
         - TODO deduping
    - fetch outgoing records (e.g. via last sync local timestamp,
      or from list of tracked items, ...)
      - serialize each record
        - assemble cleartext record and JSON stringify
        - assemble payload and JSON stringify
          - generate random IV and encrypt cleartext to ciphertext
          - compute HMAC
        - assemble WBO and JSON stringify
        - upload in batches of 100 or 1 MB, whichever comes first
          - POST .../storage/<collection>
            [{record}, {record}, ...]
          - process repsonse body

Src Sync Client Sync Storage Formats (`docs/src/sync-client/sync-storage-formats.md`) Copy Section

Sync Storage Formats

The way that Sync clients store data on a storage server is defined by sets of
integer storage versions. Each storage version defines specific semantics
for how clients are supposed to behave.

Global Storage Version Copy Chapter

There exists a global storage version that defines global semantics. This
global version typically specifies:

• What special records exist on the server and what they contain
• The payload format of encrypted records on the server
• How cryptography of data works

Each Sync client is coded to support one or more global storage formats. If a
client encounters a storage format it does not support, it should generally
stop attempting to consume data.

Under no normal circumstances should a client modify data on a server that is
defined with an unknown or newer storage format. Even if an older client wipes
all server data and uploads data using its own format, newer clients may
transparently upgrade the server data to the storage format they support.

Because changing storage formats can prevent some clients from syncing—since
not all clients may be upgraded at the same time—new global storage versions
are introduced very rarely.

Versions 1, 2, and 3 Copy Chapter

These versions were used by an older version of Sync that was deprecated in
early 2011.

Historical information is available here

These versions should no longer be in active use and should all be upgraded to
a newer storage format.

Version 4 Copy Chapter

This version introduced a new cryptographic model based fully on AES.
Due to a faulty implementation of the cryptography, version 5 was created to
force alpha clients created with the faulty implementation to upgrade.

As a result, version 4 and version 5 are practically identical in design.

Version 5 (Spring 2011 – Current) Copy Chapter

Version 5 replaces version 3’s cryptographic model with one based purely on AES.

A full overview of this format is available in Global Storage Version 5

Historical notes are available here

Collection / Object Format Versions Copy Chapter

The formats of unencrypted records stored on the server are also versioned.
For example, records in the bookmarks collection are all defined to be of a
specific object format version.

Strictly speaking, these versions are tied to a specific global storage version.
However, since all storage formats to date have stored the per-collection
version in a special record, these object format versions effectively apply
across all global storage versions.

These formats are fully documented in Firefox Object Formats.

Src Syncstorage Api 1.0 (`docs/src/syncstorage/api-1.0.md`) Copy Section

Storage API v1.0 (Obsolete)

This document describes the legacy Sync Server Storage API, version 1.0. It has been
superseded by Sync API v1.5.

Weave Basic Object (WBO) Copy Chapter

A Weave Basic Object is the generic wrapper around all items passed into and
out of the Weave server. The Weave Basic Object has the following fields:

Reference: http://www.ecma-international.org/publications/standards/Ecma-262.htm

Weave Basic Objects and all data passed into the Weave Server should be UTF-8 encoded.

Sample Copy Section

{
    "id": "B1549145-55CB-4A6B-9526-70D370821BB5",
    "parentid": "88C3865F-05A6-4E5C-8867-0FAC9AE264FC",
    "modified": "2454725.98",
    "payload": "{\"encryption\":\"http://server/prefix/version/user/crypto-meta/B1549145-55CB-4A6B-9526-70D370821BB5\", \"data\": \"a89sdmawo58aqlva.8vj2w9fmq2af8vamva98fgqamff...\"}"
}

Collections Copy Chapter

Each WBO is assigned to a collection with other related WBOs. Collection names
may only contain alphanumeric characters, period, underscore and hyphen.

Collections supported at this time are:

• bookmarks
• history
• forms
• prefs
• tabs
• passwords

Additionally, the following collections are supported for internal Weave client use:

• clients
• crypto
• keys
• meta

URL Semantics Copy Chapter

Weave URLs follow, for the most part, REST semantics. Request and response
bodies are all JSON-encoded.

The URL for Weave Storage requests is structured as follows:

https://<server name>/<api pathname>/<version>//<further instruction>

Weave uses HTTP basic auth (over SSL). If the auth username does not match the
username in the path, the server will issue an error response.

The Weave API has a set of Weave Response Codes to cover errors in the request
or on the server side.

GET Copy Chapter

info/collections Copy Section

GET /<version>//info/collections

Returns a hash of collections associated with the account, along with the last
modified timestamp for each collection.

info/collection_counts Copy Section

GET /<version>//info/collection_counts

Returns a hash of collections associated with the account, along with the total
number of items for each collection.

info/quota Copy Section

GET /<version>//info/quota

Returns a tuple containing the user's current usage (in K) and quota.

storage/collection Copy Section

GET /<version>//storage/<collection>

Returns a list of the WBO ids contained in a collection.

Optional parameters:

• ids
• predecessorid
• parentid
• older
• newer
• full
• index_above
• index_below
• limit
• offset
• sort (oldest, newest, index)

storage/collection/id Copy Section

GET /<version>//storage/<collection>/

Returns the WBO in the collection corresponding to the requested id.

Alternate Output Formats Copy Chapter

Triggered by the Accept header:

• application/whoisi: each record consists of a 32-bit integer defining the
  length of the record, followed by the JSON record
• application/newlines: each record is a separate JSON object on its own line;
  newlines in the body are replaced by \u000a

APIs Copy Chapter

PUT Copy Section

PUT /<version>//storage/<collection>/

Adds or updates a WBO. Without a payload, only metadata fields are updated.

Returns the modification timestamp.

POST Copy Section

POST /<version>//storage/<collection>

Takes an array of WBOs and performs atomic PUTs with a shared timestamp.

Example response:

{
      "modified": 1233702554.25,
      "success": ["{GXS58IDC}12","{GXS58IDC}13"],
      "failed": {
        "{GXS58IDC}11": ["invalid parentid"]
      }
    }

DELETE Copy Section

DELETE /<version>//storage/<collection>

Deletes the collection or selected items.

DELETE /<version>//storage/<collection>/

Deletes a single WBO.

DELETE /<version>//storage

Deletes all records for the user. Requires X-Confirm-Delete.

All delete operations return a timestamp.

General Weave Headers Copy Chapter

X-Weave-Backoff Copy Section

Indicates server overload. Client should retry after the specified seconds.

X-If-Unmodified-Since Copy Section

Fails write requests if the collection has changed since the given timestamp.

X-Weave-Alert Copy Section

Human-readable warnings or informational messages.

X-Weave-Timestamp Copy Section

Server timestamp; also the modification time for PUT/POST requests.

X-Weave-Records Copy Section

If supported, returns the number of records in a multi-record GET response.

Src Syncstorage Api 1.1 (`docs/src/syncstorage/api-1.1.md`) Copy Section

Storage API v1.1 (Obsolete)

This document describes the legacy Sync Server Storage API, version 1.1. It has been
superseded by Sync API v1.5.

The Storage server provides web services that can be used to store and
retrieve Weave Basic Objects (WBOs) organized into collections.

Weave Basic Object Copy Chapter

A Weave Basic Object (WBO) is the generic JSON wrapper around all items
passed into and out of the storage server. Like all JSON, WBOs must be UTF-8
encoded. WBOs have the following fields:

Notes:

• Deprecated fields are likely to be removed in future versions.
• See ECMA-262 for timestamp definition: http://www.ecma-international.org/publications/standards/Ecma-262.htm

Sample Copy Section

{
    "id": "-F_Szdjg3GzY",
    "modified": 1278109839.96,
    "sortindex": 140,
    "payload": "{\"ciphertext\":\"e2zLWJYX/iTw3WXQqffo00kuuut0Sk3G7erqXD8c65S5QfB85rqolFAU0r72GbbLkS7ZBpcpmAvX6LckEBBhQPyMt7lJzfwCUxIN/uCTpwlf9MvioGX0d4uk3G8h1YZvrEs45hWngKKf7dTqOxaJ6kGp507A6AvCUVuT7jzG70fvTCIFyemV+Rn80rgzHHDlVy4FYti6tDkmhx8t6OMnH9o/ax/3B2cM+6J2Frj6Q83OEW/QBC8Q6/XHgtJJlFi6fKWrG+XtFxS2/AazbkAMWgPfhZvIGVwkM2HeZtiuRLM=\",\"IV\":\"GluQHjEH65G0gPk/d/OGmg==\",\"hmac\":\"c550f20a784cab566f8b2223e546c3abbd52e2709e74e4e9902faad8611aa289\"}"
}```

## Collections

Each WBO is assigned to a collection with related WBOs. Collection names may
only contain alphanumeric characters, period, underscore, and hyphen.

Default Mozilla collections:

- bookmarks
- history
- forms
- prefs
- tabs
- passwords

Internal-use collections:

- clients
- crypto
- keys
- meta

## URL Semantics

Storage URLs generally follow REST semantics. Request and response bodies are
JSON-encoded.

URL structure:

`https://<server name>/<api pathname>/<version>//<further instruction>`

| Component | Mozilla Default | Description |
|----------|-----------------|-------------|
| server name | defined by user account | Hostname of the server |
| pathname | none | Prefix associated with the service |
| version | 1.1 | API version |
| username | none | User identifier |
| further instruction | none | Function-specific path |

Certain functions use HTTP Basic Authentication over SSL. If the authentication
username does not match the username in the path, an error response is returned.

## APIs

### GET

`GET /info/collections`

Returns collections and their last-modified timestamps.

`GET /info/collection_usage`

Returns collections and storage usage (KB).

`GET /info/collection_counts`

Returns collections and item counts.

`GET /info/quota`

Returns current usage and quota (KB).

`GET /storage/<collection>`

Returns WBO ids in a collection. Optional parameters:

- ids
- predecessorid (deprecated)
- parentid (deprecated)
- older
- newer
- full
- index_above
- index_below
- limit
- offset
- sort (oldest, newest, index)

Alternate output formats via `Accept` header:

- application/whoisi
- application/newlines

`GET /storage/<collection>/`

Returns the requested WBO.

### PUT

`PUT /storage/<collection>/`

Adds or updates a WBO. Metadata-only update if no payload is provided.
Returns the modification timestamp.

### POST

`POST /storage/<collection>`

Bulk upload of WBOs with a shared timestamp.

Sample response:
```json
{
    "modified": 1233702554.25,
    "success": ["{GXS58IDC}12", "{GXS58IDC}13"],
    "failed": {
    "{GXS58IDC}11": ["invalid parentid"]
    }
}

DELETE Copy Section

DELETE /storage/<collection>

Deletes a collection or selected items.

DELETE /storage/<collection>/

Deletes a single WBO.

DELETE /storage

Deletes all user records. Requires X-Confirm-Delete.

All delete operations return a timestamp.

Headers Copy Chapter

Retry-After Copy Section

Used with HTTP 503 to indicate maintenance duration.

X-Weave-Backoff Copy Section

Indicates server overload; client should delay sync (usually 1800 seconds).

X-If-Unmodified-Since Copy Section

Fails write requests if the collection was modified since the given timestamp.

X-Weave-Alert Copy Section

Human-readable warning or informational messages.

X-Weave-Timestamp Copy Section

Current server timestamp; also modification time for PUT/POST.

X-Weave-Records Copy Section

If supported, returns the number of records in a multi-record GET response.

HTTP Status Codes Copy Chapter

200 Copy Section

Request processed successfully.

400 Copy Section

Invalid request or data. Response includes a numeric error code.

401 Copy Section

Invalid credentials, possibly due to node reassignment or password change.

404 Copy Section

Resource not found. Returned for missing records or empty collections.

503 Copy Section

Server maintenance or overload. Used with Retry-After.

Src Syncstorage Api 1.5 (`docs/src/syncstorage/api-1.5.md`) Copy Section

SyncStorage API v1.5

The SyncStorage API defines a HTTP web service used to store and retrieve
simple objects called Basic Storage Objects (BSOs), which are organized
into named collections.

Concepts Copy Chapter

Basic Storage Object Copy Section

A Basic Storage Object (BSO) is the generic JSON wrapper around all items
passed into and out of the SyncStorage server. Like all JSON documents, BSOs
are composed of unicode character data rather than raw bytes and must be
encoded for transmission over the network. The SyncStorage service always
encodes BSOs in UTF8.

Basic Storage Objects have the following fields:

Example:

{
    "id": "-F_Szdjg3GzX",
    "modified": 1388635807.41,
    "sortindex": 140,
    "payload": "{ \"this is\": \"an example\" }"
}

Collections Copy Section

Each BSO is assigned to a collection with other related BSOs. Collection names
may be up to 32 characters long, and must contain only characters from the
urlsafe-base64 alphabet (alphanumeric characters, underscore and hyphen) and
the period.

Collections are created implicitly when a BSO is stored in them for the first
time. They continue to exist until explicitly deleted, even if they no longer
contain any BSOs.

The default collections used by Firefox to store sync data are:

• bookmarks
• history
• forms
• prefs
• tabs
• passwords

The following additional collections are used for internal management purposes
by the storage client:

• clients
• crypto
• keys
• meta

Timestamps Copy Section

In order to allow multiple clients to coordinate their changes, the SyncStorage
server associates a last-modified time with the data stored for each user.
This is a server-assigned decimal value, precise to two decimal places, that is
updated from the server's clock with every modification made to the user's data.

The last-modified time is tracked at three levels of nesting:

• The store as a whole has a last-modified time that is updated whenever
  any change is made to the user's data.
• Each collection has a last-modified time that is updated whenever an item
  in that collection is modified or deleted. It will always be less than or
  equal to the overall last-modified time.
• Each BSO has a last-modified time that is updated whenever that specific
  item is modified. It will always be less than or equal to the last-modified
  time of the containing collection.

The last-modified time is guaranteed to be monotonically increasing and can be
used for coordination and conflict management as described in
Syncstorage Concurrency.

Note that the last-modified time of a collection may be larger than that of any
item within it. For example, if all items are deleted from the collection, its
last-modified time will be the timestamp of the last deletion.

API Instructions Copy Chapter

The SyncStorage data for a given user may be accessed via authenticated HTTP
requests to their SyncStorage API endpoint. Request and response bodies are
all UTF8-encoded JSON unless otherwise specified. All requests are to URLs of
the form:

https://<endpoint-url>/<api-instruction>

The user's SyncStorage endpoint URL can be obtained via the tokenserver
authentication flow. All requests must be signed using HAWK Authentication
credentials obtained from the tokenserver.

Error responses generated by the SyncStorage server will, wherever possible,
conform to the respcodes defined for the User API. The format of a successful
response is defined in the appropriate section below.

General Info Copy Chapter

APIs in this section provide high-level interactions with the user's data store
as a whole.

`GET https:///info/collections` Copy Section